mosaicstack/stack
1
0
Fork 1
Code Issues 294 Pull Requests 18 Actions Packages Projects Releases 15 Wiki Activity

greenfield dead end: fleet init writes roster v1, all roster-v2 commands reject v1, migrate-v1 is preview-only #1237

New Issue
Closed
opened 2026-08-16 03:02:41 +00:00 by fred · 5 comments
fred commented 2026-08-16 03:02:41 +00:00
Collaborator
Copy Link
Copy Source

On a freshly-installed 0.0.50-next host there is no path from "no fleet" to "a fleet the roster-v2 commands will operate on". The only roster-creation command writes v1; every roster-v2 command rejects v1; and the migration command is preview-only. The CLI cannot get you out of the state its own installer puts you in.

Measured on mosaic-sbx-canary (Debian 13) rolled back to greenfield, then curl … | bash -s -- --next --yes → rc=0, CLI 0.0.50-next.2413. (fleet init --write also needs #1236 worked around first.)

The three legs of the trap

1. fleet init writes v1. Every preset, including local-canary:

$ head -1 ~/.config/mosaic/fleet/roster.yaml
version: 1

It is the only command that creates a roster.

2. Every roster-v2 command rejects v1. doctor, apply, create, delete, get, plan all route through executeReconcilerCommand, which calls parseRosterV2 on the canonical roster:

$ mosaic fleet doctor
{"error":{"code":"reconcile-failed"}}

The real error is swallowed (filed separately as the diagnosability bug). Recovered by instrumenting writeOutcome on a sandbox copy:

RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration.
    at normalizeRosterV2 (…/dist/fleet/roster-v2.js:209:15)
    at parseRosterV2 (…/dist/fleet/roster-v2.js:186:12)
    at executeReconcilerCommand (…/dist/commands/fleet-reconciler-command.js:36:20)

"use the existing v1 path until migration" is a fair instruction, but it is invisible to the operator, and the v1 path is not what fleet doctor/create/apply are.

3. migrate-v1 cannot write. It has exactly one subcommand:

$ mosaic fleet migrate-v1 --help
Preview a field-complete v1-to-v2 roster migration
Commands:
  preview   Compile migration evidence without writing files or changing runtimes

And preview needs artifacts a greenfield host has no way to produce:

$ mosaic fleet migrate-v1 preview
{"status":"blocked","blockers":[{"code":"missing-migration-preview-option","path":"request.source",…}]}

$ mosaic fleet migrate-v1 preview --source ~/.config/mosaic/fleet/roster.yaml
{"status":"blocked","blockers":[{"code":"missing-migration-preview-option","path":"request.decisions",…}]}

$ … --decisions {} --observations {}
{"status":"blocked","blockers":[{"code":"migration-preview-failed","path":"request","detail":"Migration preview failed without publishable detail."}]}

--decisions and --observations have no documented schema reachable from the CLI, and "failed without publishable detail" is a second swallowed error.

Net effect

fleet init            -> v1 roster
fleet doctor|apply|create|delete|get|plan  -> "v2 compiler rejects v1 input"  (shown as: reconcile-failed)
fleet migrate-v1      -> preview only, and preview is blocked

Roster v2 is unreachable from a greenfield install using shipped commands. I could not find a command that writes one; roster-v2.js exports validateRosterV2, parseRosterV2, renderRosterV2, normalizeRosterV2 — a renderer exists, nothing surfaces it as a write path.

Why this is on the critical path

The north-star goal for web1 requires seats "manageable via mosaic cli tools" and "tracked durably via the Mosaic Fleet system tools". As shipped, a fresh host reaches neither: its roster is v1, and the management surface is v2-only.

What would resolve it

Any one of:

  1. fleet init emits v2 (with --v1 to opt out), so a new host starts where the tooling lives.
  2. migrate-v1 gains a --write, with --decisions/--observations optional and defaulted for the trivial case, so v1 hosts have an exit.
  3. The v2 commands accept a v1 roster by normalising it on read.

(1) looks smallest for greenfield and does nothing for the existing v1 hosts — web1 and sb-it-1-dt are both roster v1 today, so (2) is needed regardless. Whichever way, the "use the existing v1 path until migration" instruction needs to reach the operator instead of being replaced with reconcile-failed.

Related

  • #1236 — installer umask blocks fleet init --write before you can even get here.
  • The error-swallowing in writeOutcome is why this needed a patched binary to diagnose; filed separately.

Reported by fred (orchestrator seat, sb-it-1-dt), measured on canary VMID 1125.

On a freshly-installed 0.0.50-next host there is no path from "no fleet" to "a fleet the roster-v2 commands will operate on". The only roster-creation command writes v1; every roster-v2 command rejects v1; and the migration command is preview-only. The CLI cannot get you out of the state its own installer puts you in. Measured on `mosaic-sbx-canary` (Debian 13) rolled back to `greenfield`, then `curl … | bash -s -- --next --yes` → rc=0, CLI `0.0.50-next.2413`. (`fleet init --write` also needs #1236 worked around first.) ## The three legs of the trap **1. `fleet init` writes v1.** Every preset, including `local-canary`: ``` $ head -1 ~/.config/mosaic/fleet/roster.yaml version: 1 ``` It is the only command that creates a roster. **2. Every roster-v2 command rejects v1.** `doctor`, `apply`, `create`, `delete`, `get`, `plan` all route through `executeReconcilerCommand`, which calls `parseRosterV2` on the canonical roster: ``` $ mosaic fleet doctor {"error":{"code":"reconcile-failed"}} ``` The real error is swallowed (filed separately as the diagnosability bug). Recovered by instrumenting `writeOutcome` on a sandbox copy: ``` RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration. at normalizeRosterV2 (…/dist/fleet/roster-v2.js:209:15) at parseRosterV2 (…/dist/fleet/roster-v2.js:186:12) at executeReconcilerCommand (…/dist/commands/fleet-reconciler-command.js:36:20) ``` "use the existing v1 path until migration" is a fair instruction, but it is invisible to the operator, and the v1 path is not what `fleet doctor`/`create`/`apply` are. **3. `migrate-v1` cannot write.** It has exactly one subcommand: ``` $ mosaic fleet migrate-v1 --help Preview a field-complete v1-to-v2 roster migration Commands: preview Compile migration evidence without writing files or changing runtimes ``` And preview needs artifacts a greenfield host has no way to produce: ``` $ mosaic fleet migrate-v1 preview {"status":"blocked","blockers":[{"code":"missing-migration-preview-option","path":"request.source",…}]} $ mosaic fleet migrate-v1 preview --source ~/.config/mosaic/fleet/roster.yaml {"status":"blocked","blockers":[{"code":"missing-migration-preview-option","path":"request.decisions",…}]} $ … --decisions {} --observations {} {"status":"blocked","blockers":[{"code":"migration-preview-failed","path":"request","detail":"Migration preview failed without publishable detail."}]} ``` `--decisions` and `--observations` have no documented schema reachable from the CLI, and "failed without publishable detail" is a second swallowed error. ## Net effect ``` fleet init -> v1 roster fleet doctor|apply|create|delete|get|plan -> "v2 compiler rejects v1 input" (shown as: reconcile-failed) fleet migrate-v1 -> preview only, and preview is blocked ``` Roster v2 is unreachable from a greenfield install using shipped commands. I could not find a command that writes one; `roster-v2.js` exports `validateRosterV2`, `parseRosterV2`, `renderRosterV2`, `normalizeRosterV2` — a renderer exists, nothing surfaces it as a write path. ## Why this is on the critical path The north-star goal for web1 requires seats "manageable via mosaic cli tools" and "tracked durably via the Mosaic Fleet system tools". As shipped, a fresh host reaches neither: its roster is v1, and the management surface is v2-only. ## What would resolve it Any one of: 1. `fleet init` emits v2 (with `--v1` to opt out), so a new host starts where the tooling lives. 2. `migrate-v1` gains a `--write`, with `--decisions`/`--observations` optional and defaulted for the trivial case, so v1 hosts have an exit. 3. The v2 commands accept a v1 roster by normalising it on read. (1) looks smallest for greenfield and does nothing for the existing v1 hosts — web1 and sb-it-1-dt are both roster v1 today, so (2) is needed regardless. Whichever way, the "use the existing v1 path until migration" instruction needs to reach the operator instead of being replaced with `reconcile-failed`. ## Related - #1236 — installer umask blocks `fleet init --write` before you can even get here. - The error-swallowing in `writeOutcome` is why this needed a patched binary to diagnose; filed separately. Reported by fred (orchestrator seat, sb-it-1-dt), measured on canary VMID 1125.
mos-dt-0 commented 2026-08-16 03:17:30 +00:00
Collaborator
Copy Link
Copy Source

Amendment: I filed this as a v1 dead end. It is worse and more symmetrical than that — there are two fleet implementations sharing one command namespace, and no roster version can reach a working seat.

Also, one softening: I said roster v2 is unreachable. It is unreachable from fleet init. A hand-written v2 roster works, and there is a documented example to copy — docs/fleet/examples/roster-v2.yaml, with a field reference and an executable schema alongside it. I missed that when I filed. Corrected below.

The matrix

Same host, same install, same commands. Only the roster file swapped between the v1 that fleet init --preset general writes and the documented v2 example.

command v2 roster v1 roster (what fleet init writes)
fleet doctor rc=0 rc=1 reconcile-failed
fleet get <name> rc=0 rc=1 mutation-failed
fleet apply --expected-generation 1 rc=1 — but reaches lifecycle, projections: complete rc=1 reconcile-failed
fleet install rc=1 "Fleet roster has unknown field(s): generation." rc=0 — installs and enables 4 units
fleet ps rc=1 "unknown field(s): generation." rc=0
fleet status rc=0 — reconciler JSON rc=3 — systemctl output
fleet start rc=1 lifecycle-precondition-failed rc=0

fleet status is the clearest tell: with a v2 roster it prints reconciler JSON, with a v1 roster it prints systemd status. Two different commands, one name, selected by a field in a config file.

Note also that the v1-side commands reject generation — a field that is required by the v2 schema and present in the shipped v2 example. So it is not that the v1 side is version-agnostic and the v2 side is strict; both sides are strict, in opposite directions.

Why this blocks the goal

  • v2 gives you the reconciler, and fleet install refuses to create the systemd units that fleet apply's lifecycle stage then reports missing. apply gets as far as projections: complete, lifecycle: incomplete, recovery: lifecycle-apply-failed.
  • v1 gives you install + start — the only path that reaches an actual launch — and every reconciler command refuses it.

There is no roster you can write that gets both halves. To get from a fresh install to a running, manageable seat you currently need to swap the roster file between commands.

The scope note in the v2 docs may explain this

docs/fleet/reference/roster-v2-fields.md opens with:

Status: FCM-M1-001 local-tmux structural compiler contract. This document describes parsing, strict structural validation, normalized in-memory representation, and deterministic rendering only. It does not authorize role resolution, lifecycle reconciliation, mutation, migration, remote placement, connector configuration, secret references, arbitrary commands, channels, gateway mapping, or any live-fleet change.

If that is still the intended scope, then apply, create, delete and update — commands whose entire job is mutation and lifecycle — are wired to a compiler that is documented as not authorizing mutation or lifecycle. That would make this a sequencing problem rather than a design disagreement: the M1 structural milestone landed and the commands were pointed at it before the layer above it existed.

I am not asserting that is what happened; I am flagging that the shipped docs and the shipped wiring disagree, and whoever owns the milestone can settle it in a sentence.

Revised resolutions

The three options in the original report still stand, but the matrix changes which is cheapest:

  1. Make the v1-side commands accept v2 (install, ps, start, stop, status, add, remove, restart). Then a hand-written v2 roster is a complete, usable path today and fleet init writing v1 becomes a papercut rather than a wall.
  2. Make fleet init write v2. Necessary eventually, insufficient alone — it would move the wall from the reconciler to install/start, which is arguably worse because the failure comes later.
  3. Finish migrate-v1 so it writes. Still needed, still does not fix the split.

(1) unblocks the goal fastest. (2) and (3) are the real end state. Doing (2) without (1) makes things worse, which is the part I would most want an owner to weigh in on before anyone starts.

Related

  • #1241 — what happens on the v1 path once you do reach a launch. It reports success with every pane dead.
  • #1236 — amended; mutating commands additionally require 0700 on the mosaic home and fleet/, which is what let apply reach lifecycle at all.
  • #1238 — why half of this needed a patched binary to read.

Measured on canary VMID 1125, greenfield --next install, CLI 0.0.50-next.2413.

Amendment: I filed this as a v1 dead end. It is worse and more symmetrical than that — **there are two fleet implementations sharing one command namespace, and no roster version can reach a working seat.** Also, one softening: I said roster v2 is unreachable. It is unreachable *from `fleet init`*. A hand-written v2 roster works, and there is a documented example to copy — `docs/fleet/examples/roster-v2.yaml`, with a field reference and an executable schema alongside it. I missed that when I filed. Corrected below. ## The matrix Same host, same install, same commands. Only the roster file swapped between the v1 that `fleet init --preset general` writes and the documented v2 example. | command | v2 roster | v1 roster (what `fleet init` writes) | |---|---|---| | `fleet doctor` | **rc=0** | rc=1 `reconcile-failed` | | `fleet get <name>` | **rc=0** | rc=1 `mutation-failed` | | `fleet apply --expected-generation 1` | rc=1 — but reaches lifecycle, `projections: complete` | rc=1 `reconcile-failed` | | `fleet install` | rc=1 **"Fleet roster has unknown field(s): generation."** | **rc=0** — installs and enables 4 units | | `fleet ps` | rc=1 **"unknown field(s): generation."** | **rc=0** | | `fleet status` | rc=0 — reconciler JSON | rc=3 — `systemctl` output | | `fleet start` | rc=1 `lifecycle-precondition-failed` | **rc=0** | `fleet status` is the clearest tell: with a v2 roster it prints reconciler JSON, with a v1 roster it prints systemd status. Two different commands, one name, selected by a field in a config file. Note also that the v1-side commands reject `generation` — a field that is **required** by the v2 schema and present in the shipped v2 example. So it is not that the v1 side is version-agnostic and the v2 side is strict; both sides are strict, in opposite directions. ## Why this blocks the goal - **v2 gives you the reconciler**, and `fleet install` refuses to create the systemd units that `fleet apply`'s lifecycle stage then reports missing. `apply` gets as far as `projections: complete, lifecycle: incomplete, recovery: lifecycle-apply-failed`. - **v1 gives you `install` + `start`** — the only path that reaches an actual launch — and every reconciler command refuses it. There is no roster you can write that gets both halves. To get from a fresh install to a running, manageable seat you currently need to swap the roster file between commands. ## The scope note in the v2 docs may explain this `docs/fleet/reference/roster-v2-fields.md` opens with: > **Status:** FCM-M1-001 local-tmux structural compiler contract. This document describes parsing, strict structural validation, normalized in-memory representation, and deterministic rendering only. It **does not authorize** role resolution, **lifecycle reconciliation**, mutation, migration, remote placement, connector configuration, secret references, arbitrary commands, channels, gateway mapping, **or any live-fleet change**. If that is still the intended scope, then `apply`, `create`, `delete` and `update` — commands whose entire job is mutation and lifecycle — are wired to a compiler that is documented as not authorizing mutation or lifecycle. That would make this a sequencing problem rather than a design disagreement: the M1 structural milestone landed and the commands were pointed at it before the layer above it existed. I am not asserting that is what happened; I am flagging that the shipped docs and the shipped wiring disagree, and whoever owns the milestone can settle it in a sentence. ## Revised resolutions The three options in the original report still stand, but the matrix changes which is cheapest: 1. **Make the v1-side commands accept v2** (`install`, `ps`, `start`, `stop`, `status`, `add`, `remove`, `restart`). Then a hand-written v2 roster is a complete, usable path today and `fleet init` writing v1 becomes a papercut rather than a wall. 2. **Make `fleet init` write v2.** Necessary eventually, insufficient alone — it would move the wall from the reconciler to `install`/`start`, which is arguably worse because the failure comes later. 3. **Finish `migrate-v1`** so it writes. Still needed, still does not fix the split. (1) unblocks the goal fastest. (2) and (3) are the real end state. Doing (2) without (1) makes things worse, which is the part I would most want an owner to weigh in on before anyone starts. ## Related - #1241 — what happens on the v1 path once you do reach a launch. It reports success with every pane dead. - #1236 — amended; mutating commands additionally require `0700` on the mosaic home and `fleet/`, which is what let `apply` reach lifecycle at all. - #1238 — why half of this needed a patched binary to read. Measured on canary VMID 1125, greenfield `--next` install, CLI 0.0.50-next.2413.
mos-dt-0 commented 2026-08-16 03:52:37 +00:00
Collaborator
Copy Link
Copy Source

Correction — this is worse than "init writes v1", and one of my three proposed resolutions is not available

Re-measured on canary (Debian 13, CLI 0.0.50-next.2413, framework at fix/1236-installer-dir-modes so the permission defect #1236 is out of the way). Two things I got wrong when I filed this.

1. It is not that fleet init writes the wrong version. Nothing in the product writes v2.

I framed this as a mismatch between one producer and one consumer. It is broader. There are three roster producers and all three emit v1:

producer emits
fleet init --profile <p> --write copies framework/fleet/examples/<p>.yaml verbatim — all seven are version: 1
fleet provision ("Materialize a roster.yaml from a system-type profile") version: 1 hard-coded, fleet-provision.ts:269
the fleet profile / system-type path same v1 shape

And no roster v2 exists anywhere in the installed framework. The only roster-v2.yaml in the repo is docs/fleet/examples/roster-v2.yaml, and docs/ is not part of what install.sh syncs. I proved this by accident: my probe script looked for a v2 example under ~/.config/mosaic on a freshly-installed box, found none, and silently fell through to measuring v1 again.

So the accurate statement is: the entire roster-v2 command surface — create, get, update, delete, plan, apply, reconcile, doctor — is unreachable on a stock install, because the input format it requires cannot be produced or found on the machine. An operator gets a v2 roster only by hand-copying one out of the git repo's docs/ tree.

2. migrate-v1 is not a resolution, and I should not have listed it as one

I offered "finish migrate-v1 so it writes" as option 3. Reading fleet-migration-command.ts: it is migrate-v1 preview only, and it requires three inputs — --source, --decisions, --observations — each mandatory, each a file path. --decisions is an explicit migration-decisions JSON and --observations is "reviewed lifecycle observations". On a fresh box, an operator has no way to author either. It is a tool for a supervised migration of an existing fleet, not a bootstrap path. Even finished, it would not close this.

That leaves two real resolutions, not three.

The measured split, both directions

Same box, same modes, only the roster file swapped:

command v1 roster v2 roster
doctor rc=1 {"error":{"code":"reconcile-failed"}} rc=0, full plan JSON
status rc=4 Unit mosaic-tmux-holder.service could not be found. rc=0, plan JSON
ps rc=0, three agent rows rc=1 Fleet roster has unknown field(s): generation.
install --no-enable rc=0 Installed fleet files for 3 agent(s). rc=1 Fleet roster has unknown field(s): generation.

Neither version drives the whole product. Worth noting the error quality is inverted here: the v1 parser rejecting v2 says exactly what is wrong (unknown field(s): generation), while the reconciler rejecting v1 says reconcile-failed and nothing else — that is #1238, and this table is a good argument for it.

What I propose to do about it, and what is yours to decide

Two separable pieces. The first is not an architecture decision and I intend to build it unless told otherwise; the second is yours.

Piece A — make the v1-side commands accept a v2 roster (mine). Every one of them (install, ps, start, stop, restart, status, add, remove, verify) loads through a single function, loadFleetRoster. Teaching that one function to recognize version: 2 and downshift it in memory to the v1 shape fixes all of them at one site. The fields those consumers actually use — agent name, runtime, class, working directory, tmux socket and holder session, per-runtime reset commands — are all present in v2, so the downshift is total for their purposes and loses nothing they read. It writes no file and changes no format, so it stays compatible with whichever direction you pick below.

Piece B — which producer emits v2, and when (yours). This is a product decision about the roster format going forward and I am not going to make it unilaterally. The options are: change fleet init to emit v2 (needs seven new example files, or rendering v2 from the existing v1 examples at write time), or leave init as the v1 path and make provision the v2 path, or ship v2 examples and let the operator choose. They differ in what an upgrade does to existing installs, which is the part that matters for web1.

Piece A alone is enough to get a working seat, using a hand-placed v2 roster as the interim — which is what the sandbox E2E needs and what is currently blocking it.

— fred

# Correction — this is worse than "init writes v1", and one of my three proposed resolutions is not available Re-measured on canary (Debian 13, CLI `0.0.50-next.2413`, framework at `fix/1236-installer-dir-modes` so the permission defect #1236 is out of the way). Two things I got wrong when I filed this. ## 1. It is not that `fleet init` writes the wrong version. Nothing in the product writes v2. I framed this as a mismatch between one producer and one consumer. It is broader. There are three roster producers and all three emit v1: | producer | emits | |---|---| | `fleet init --profile <p> --write` | copies `framework/fleet/examples/<p>.yaml` verbatim — all seven are `version: 1` | | `fleet provision` ("Materialize a roster.yaml from a system-type profile") | `version: 1` hard-coded, `fleet-provision.ts:269` | | the `fleet profile` / system-type path | same v1 shape | And **no roster v2 exists anywhere in the installed framework.** The only `roster-v2.yaml` in the repo is `docs/fleet/examples/roster-v2.yaml`, and `docs/` is not part of what `install.sh` syncs. I proved this by accident: my probe script looked for a v2 example under `~/.config/mosaic` on a freshly-installed box, found none, and silently fell through to measuring v1 again. So the accurate statement is: **the entire roster-v2 command surface — `create`, `get`, `update`, `delete`, `plan`, `apply`, `reconcile`, `doctor` — is unreachable on a stock install, because the input format it requires cannot be produced or found on the machine.** An operator gets a v2 roster only by hand-copying one out of the git repo's `docs/` tree. ## 2. `migrate-v1` is not a resolution, and I should not have listed it as one I offered "finish `migrate-v1` so it writes" as option 3. Reading `fleet-migration-command.ts`: it is `migrate-v1 preview` only, and it requires three inputs — `--source`, `--decisions`, `--observations` — each mandatory, each a file path. `--decisions` is an explicit migration-decisions JSON and `--observations` is "reviewed lifecycle observations". On a fresh box, an operator has no way to author either. It is a tool for a supervised migration of an existing fleet, not a bootstrap path. Even finished, it would not close this. That leaves two real resolutions, not three. ## The measured split, both directions Same box, same modes, only the roster file swapped: | command | v1 roster | v2 roster | |---|---|---| | `doctor` | rc=1 `{"error":{"code":"reconcile-failed"}}` | **rc=0**, full plan JSON | | `status` | rc=4 `Unit mosaic-tmux-holder.service could not be found.` | **rc=0**, plan JSON | | `ps` | **rc=0**, three agent rows | rc=1 `Fleet roster has unknown field(s): generation.` | | `install --no-enable` | **rc=0** `Installed fleet files for 3 agent(s).` | rc=1 `Fleet roster has unknown field(s): generation.` | Neither version drives the whole product. Worth noting the error quality is inverted here: the v1 parser rejecting v2 says exactly what is wrong (`unknown field(s): generation`), while the reconciler rejecting v1 says `reconcile-failed` and nothing else — that is #1238, and this table is a good argument for it. ## What I propose to do about it, and what is yours to decide Two separable pieces. The first is not an architecture decision and I intend to build it unless told otherwise; the second is yours. **Piece A — make the v1-side commands accept a v2 roster (mine).** Every one of them (`install`, `ps`, `start`, `stop`, `restart`, `status`, `add`, `remove`, `verify`) loads through a single function, `loadFleetRoster`. Teaching that one function to recognize `version: 2` and downshift it in memory to the v1 shape fixes all of them at one site. The fields those consumers actually use — agent name, runtime, class, working directory, tmux socket and holder session, per-runtime reset commands — are all present in v2, so the downshift is total for their purposes and loses nothing they read. It writes no file and changes no format, so it stays compatible with whichever direction you pick below. **Piece B — which producer emits v2, and when (yours).** This is a product decision about the roster format going forward and I am not going to make it unilaterally. The options are: change `fleet init` to emit v2 (needs seven new example files, or rendering v2 from the existing v1 examples at write time), or leave `init` as the v1 path and make `provision` the v2 path, or ship v2 examples and let the operator choose. They differ in what an upgrade does to existing installs, which is the part that matters for web1. Piece A alone is enough to get a working seat, using a hand-placed v2 roster as the interim — which is what the sandbox E2E needs and what is currently blocking it. — fred
mos-dt-0 commented 2026-08-16 03:54:18 +00:00
Collaborator
Copy Link
Copy Source

Correcting my own Piece A — the codebase already has the right pattern and I proposed fighting it

Ten minutes ago I proposed teaching loadFleetRoster to downshift a v2 roster in memory, and said that would fix install, ps, start, stop, restart, status, add, remove, verify "at one site". That was wrong on both counts. I had not yet read far enough.

There is already a version dispatcher in fleet.ts. usesRosterV2ControlPlane(cmd) reads the roster, checks version === 2, and routes to executeReconcilerCommandJson. It is applied to five commands:

  • start, stop, restart (line 1555)
  • status (line 1619)
  • verify (line 1652)

That is why fleet status returns reconciler plan JSON on a v2 roster and a systemd message on a v1 roster — not two implementations by accident, one deliberate fork. The design intent is clear and it is not "make the v1 parser tolerant". Those five commands would never reach a downshift anyway, because the dispatch runs first.

The actual gap is the five commands that were not given the dispatch: ps, install, install-systemd, add, remove. Those are the only ones that fail with unknown field(s): generation, and my measured table only caught two of them because I only probed two.

And they do not all want the same treatment, which is the part that matters:

  • add and remove have direct roster-v2 counterparts already built — create and delete in fleet-agent-crud-command.ts. These want the existing dispatch, exactly like start/stop/restart got. Mechanical.
  • ps and install/install-systemd have no reconciler counterpart. FleetReconcileCommand is plan | apply | reconcile | start | stop | restart | status | verify | doctor — there is no ps and no install, and there should not be: ps is a read-only display and install copies tool files and enables units. Neither is a desired-state reconciliation. These two want a read-only v2→v1 projection for the fields they actually read (agent name, runtime, class, working directory, tmux socket and holder, per-runtime reset commands), which v2 carries in full.

So it is a dispatch fix for three commands and a read-only projection for two, not one adapter for nine. The projection does not contradict the dispatch design — dispatch owns the control-plane verbs, projection serves the two commands that read the roster without being control-plane operations at all.

I am flagging this against myself because the earlier framing would have made Piece A sound like a single tidy change to one function, and someone deciding Piece B on that basis would be working from a wrong picture of the cost. It is still a contained change and still not an architecture decision. It is just five commands, in two shapes, for stated reasons.

Piece B — which producer emits v2 — is unchanged and still yours.

— fred

# Correcting my own Piece A — the codebase already has the right pattern and I proposed fighting it Ten minutes ago I proposed teaching `loadFleetRoster` to downshift a v2 roster in memory, and said that would fix `install`, `ps`, `start`, `stop`, `restart`, `status`, `add`, `remove`, `verify` "at one site". That was wrong on both counts. I had not yet read far enough. **There is already a version dispatcher in `fleet.ts`.** `usesRosterV2ControlPlane(cmd)` reads the roster, checks `version === 2`, and routes to `executeReconcilerCommandJson`. It is applied to five commands: - `start`, `stop`, `restart` (line 1555) - `status` (line 1619) - `verify` (line 1652) That is why `fleet status` returns reconciler plan JSON on a v2 roster and a systemd message on a v1 roster — not two implementations by accident, one deliberate fork. The design intent is clear and it is not "make the v1 parser tolerant". Those five commands would never reach a downshift anyway, because the dispatch runs first. **The actual gap is the five commands that were not given the dispatch:** `ps`, `install`, `install-systemd`, `add`, `remove`. Those are the only ones that fail with `unknown field(s): generation`, and my measured table only caught two of them because I only probed two. **And they do not all want the same treatment**, which is the part that matters: - `add` and `remove` have direct roster-v2 counterparts already built — `create` and `delete` in `fleet-agent-crud-command.ts`. These want the existing dispatch, exactly like `start`/`stop`/`restart` got. Mechanical. - `ps` and `install`/`install-systemd` have **no** reconciler counterpart. `FleetReconcileCommand` is `plan | apply | reconcile | start | stop | restart | status | verify | doctor` — there is no `ps` and no `install`, and there should not be: `ps` is a read-only display and `install` copies tool files and enables units. Neither is a desired-state reconciliation. These two want a read-only v2→v1 projection for the fields they actually read (agent name, runtime, class, working directory, tmux socket and holder, per-runtime reset commands), which v2 carries in full. So it is a dispatch fix for three commands and a read-only projection for two, not one adapter for nine. The projection does not contradict the dispatch design — dispatch owns the control-plane verbs, projection serves the two commands that read the roster without being control-plane operations at all. I am flagging this against myself because the earlier framing would have made Piece A sound like a single tidy change to one function, and someone deciding Piece B on that basis would be working from a wrong picture of the cost. It is still a contained change and still not an architecture decision. It is just five commands, in two shapes, for stated reasons. Piece B — which producer emits v2 — is unchanged and still yours. — fred
mos-dt-0 commented 2026-08-16 04:07:05 +00:00
Collaborator
Copy Link
Copy Source

Correcting Piece A again — and the good news: the v2 control plane already works end to end

Two things in this comment. The first is me correcting my own proposal for the third time. The second is a measurement that makes this issue smaller than it has looked all along.

1. The "v2 → v1 projection" I proposed would break a stated invariant

I proposed that install, install-systemd and ps get a read-only v2→v1 projection so they could keep using the v1 code path. I had not yet read what install actually writes.

installFleet (fleet.ts:2331) ends by writing every agent's generated env file, using the v1 mapping generateAgentEnvValues (fleet.ts:480).

The reconciler writes that same file using a different function, projectRosterV2AgentGeneratedEnv (fleet-reconciler.ts:594), which carries this docstring:

The single roster-v2 → generated-projection mapping. Both the reconciler's apply path and the recovery-framed mosaic fleet regen command derive their generated env from THIS one function, so the two paths can never drift (the #791 single-SSOT invariant).

So if install projected v2 down to v1 and then called the v1 mapping, there would be a third writer of fleet/agents/<name>.env.generated, and it would reach that file through the mapping the invariant was written to exclude. The two mappings are not identical either — v1 has modelHint ?? '', a per-agent socket override, and a workingDirectory fallback to roster.defaults; v2 makes model and workingDirectory required and has no per-agent socket. So the projection is not even lossless in the direction that matters.

My proposal was to work around the architecture. The architecture is right and I was wrong about it.

2. What Piece A should actually be — no projection anywhere

On a v2 roster the reconciler already owns env. Measured on canary just now:

mosaic fleet apply --expected-generation 1   rc=0   projections: complete   (writes all three .env.generated)
mosaic fleet regen                            rc=0   reads the v2 roster, plans [create]/[rebuild] correctly
mosaic fleet regen --write                    rc=0   writes 0600 env from the v2 mapping

regen is already v2-native and is explicitly the recovery path for exactly this file. So:

  • install / install-systemd — on a v2 roster, place the tool files and the unit templates, enable the units, and do not write env at all. Env belongs to apply/regen. The only thing these commands need from a v2 roster is the list of agent names, for enableFleetUnits (which reads nothing but agent.name). Everything else installFleet does is roster-independent.
  • ps — render the reconciler plan. The plan already carries per-agent name / desiredState / enabled / systemd / tmux / drift, which is what ps displays. It does not need a roster read of its own.
  • add / remove — dispatch to the already-built create / delete, as I said before. This part stands.

That is three dispatches and one narrow change to install, with no new compatibility layer and no second env writer. Smaller than what I described last time, and it stops fighting #791 instead of quietly re-opening it.

Two incidental things noticed while reading: install and install-systemd are registered with byte-identical action bodies (fleet.ts:1524-1542), and both load the roster twice — once inside installFleet and once at the call site. Neither is a bug, both are cheap to fold in while touching these lines.

3. The measurement that reframes this issue: v2 works

I had been treating "no live seat on a v2 roster" as evidence the v2 path was unfinished. It is not. With tmux present and the holder enabled by hand — that is, standing in for the one thing install would have done — the whole control plane goes green on a v2 roster:

command (v2 roster) before after holder up
start --expected-generation 1 rc=1 lifecycle-apply-failed rc=0, lifecycle: complete, holder owned
verify rc=1 lifecycle-precondition-failed rc=0, all three agents systemd: active
apply --expected-generation 1 rc=0 rc=0
regen --write — rc=0

The earlier lifecycle-precondition-failed was the missing holder and nothing else, which is the check behaving correctly.

So the v2 control plane is not the problem. What stands between a greenfield box and a live seat is a chain of four separate things, and this issue is only the second link:

  1. #1240 — tmux is not installed by the installer and not checked by doctor. The holder unit fails to start. Nothing downstream can work.
  2. #1237 (this issue) — install rejects a v2 roster, so on a fresh v2 box the unit templates never get placed or enabled.
  3. #1241 — with units placed, start reports lifecycle: complete while every agent pane is dead. start-agent-session.sh creates the session, the launch command dies immediately, the pane-PID lookup fails all five retries, and the script exits 0 having logged it as a heartbeat warning. set -euo pipefail cannot catch this because tmux new-session -d genuinely succeeded — tmux made the session, the command inside it exited, tmux tore it down.
  4. No agent runtime is installed. On canary claude, pi, codex and opencode are all absent, so the launch command in step 3 is mosaic yolo pi … against a pi that does not exist.

I filed #1240 and #1241 earlier and then, in my own notes, filed them mentally under "runtime provisioning" and treated them as downstream of this issue. That was wrong in both directions: #1240 is upstream of this issue, and #1241 is the reason a fix here will look like it worked when it has not. Anyone fixing #1237 should expect a green install followed by a green start and still no seat, until #1241 is fixed too.

Piece B — which producer emits v2, and what upgrade does to existing installs — is unchanged and still yours.

— fred

# Correcting Piece A again — and the good news: the v2 control plane already works end to end Two things in this comment. The first is me correcting my own proposal for the third time. The second is a measurement that makes this issue smaller than it has looked all along. ## 1. The "v2 → v1 projection" I proposed would break a stated invariant I proposed that `install`, `install-systemd` and `ps` get a read-only v2→v1 projection so they could keep using the v1 code path. I had not yet read what `install` actually writes. `installFleet` (`fleet.ts:2331`) ends by writing every agent's generated env file, using the **v1** mapping `generateAgentEnvValues` (`fleet.ts:480`). The reconciler writes that same file using a **different** function, `projectRosterV2AgentGeneratedEnv` (`fleet-reconciler.ts:594`), which carries this docstring: > The single roster-v2 → generated-projection mapping. Both the reconciler's apply path and the recovery-framed `mosaic fleet regen` command derive their generated env from THIS one function, so the two paths can never drift (the #791 single-SSOT invariant). So if `install` projected v2 down to v1 and then called the v1 mapping, there would be a **third** writer of `fleet/agents/<name>.env.generated`, and it would reach that file through the mapping the invariant was written to exclude. The two mappings are not identical either — v1 has `modelHint ?? ''`, a per-agent `socket` override, and a `workingDirectory` fallback to `roster.defaults`; v2 makes model and workingDirectory required and has no per-agent socket. So the projection is not even lossless in the direction that matters. My proposal was to work around the architecture. The architecture is right and I was wrong about it. ## 2. What Piece A should actually be — no projection anywhere On a v2 roster the reconciler already owns env. Measured on canary just now: ``` mosaic fleet apply --expected-generation 1 rc=0 projections: complete (writes all three .env.generated) mosaic fleet regen rc=0 reads the v2 roster, plans [create]/[rebuild] correctly mosaic fleet regen --write rc=0 writes 0600 env from the v2 mapping ``` `regen` is already v2-native and is explicitly the recovery path for exactly this file. So: - **`install` / `install-systemd`** — on a v2 roster, place the tool files and the unit templates, enable the units, and **do not write env at all**. Env belongs to `apply`/`regen`. The only thing these commands need from a v2 roster is the list of agent names, for `enableFleetUnits` (which reads nothing but `agent.name`). Everything else `installFleet` does is roster-independent. - **`ps`** — render the reconciler plan. The plan already carries per-agent `name / desiredState / enabled / systemd / tmux / drift`, which is what `ps` displays. It does not need a roster read of its own. - **`add` / `remove`** — dispatch to the already-built `create` / `delete`, as I said before. This part stands. That is three dispatches and one narrow change to `install`, with no new compatibility layer and no second env writer. Smaller than what I described last time, and it stops fighting #791 instead of quietly re-opening it. Two incidental things noticed while reading: `install` and `install-systemd` are registered with byte-identical action bodies (`fleet.ts:1524-1542`), and both load the roster twice — once inside `installFleet` and once at the call site. Neither is a bug, both are cheap to fold in while touching these lines. ## 3. The measurement that reframes this issue: v2 works I had been treating "no live seat on a v2 roster" as evidence the v2 path was unfinished. It is not. With tmux present and the holder enabled by hand — that is, standing in for the one thing `install` would have done — the whole control plane goes green on a v2 roster: | command (v2 roster) | before | after holder up | |---|---|---| | `start --expected-generation 1` | rc=1 `lifecycle-apply-failed` | **rc=0**, `lifecycle: complete`, holder `owned` | | `verify` | rc=1 `lifecycle-precondition-failed` | **rc=0**, all three agents `systemd: active` | | `apply --expected-generation 1` | rc=0 | rc=0 | | `regen --write` | — | rc=0 | The earlier `lifecycle-precondition-failed` was the missing holder and nothing else, which is the check behaving correctly. **So the v2 control plane is not the problem.** What stands between a greenfield box and a live seat is a chain of four separate things, and this issue is only the second link: 1. **#1240** — tmux is not installed by the installer and not checked by `doctor`. The holder unit fails to start. Nothing downstream can work. 2. **#1237 (this issue)** — `install` rejects a v2 roster, so on a fresh v2 box the unit templates never get placed or enabled. 3. **#1241** — with units placed, `start` reports `lifecycle: complete` while every agent pane is dead. `start-agent-session.sh` creates the session, the launch command dies immediately, the pane-PID lookup fails all five retries, and the script exits 0 having logged it as a heartbeat warning. `set -euo pipefail` cannot catch this because `tmux new-session -d` genuinely succeeded — tmux made the session, the command inside it exited, tmux tore it down. 4. **No agent runtime is installed.** On canary `claude`, `pi`, `codex` and `opencode` are all absent, so the launch command in step 3 is `mosaic yolo pi …` against a `pi` that does not exist. I filed #1240 and #1241 earlier and then, in my own notes, filed them mentally under "runtime provisioning" and treated them as downstream of this issue. That was wrong in both directions: #1240 is *upstream* of this issue, and #1241 is the reason a fix here will look like it worked when it has not. Anyone fixing #1237 should expect a green `install` followed by a green `start` and still no seat, until #1241 is fixed too. Piece B — which producer emits v2, and what upgrade does to existing installs — is unchanged and still yours. — fred
mos-dt-0 commented 2026-08-16 04:38:26 +00:00
Collaborator
Copy Link
Copy Source

Piece A is up: PR #1243 (base next, branch fix/1237-fleet-v2-dispatch).

ps / install / install-systemd now read the roster through a narrow read-only view, so they work on v2. install on v2 places tool files and units and writes no generated env — the reconciler owns that file, and a second writer here would go through the v1 mapping, which is the #791 invariant. add / remove refuse on v2 and name the create/delete → apply sequence.

Canary red→green, same v2 roster, installed 0.0.50-next.2413 vs the branch:

command 0.0.50-next.2413 branch
fleet ps rc=1 unknown field(s): generation rc=0, 3 agents
fleet install --no-enable rc=1 same rc=0
fleet remove <name> rc=1 same rc=1, refusal naming delete + apply
fleet add <name> … rc=1 same rc=1, refusal naming create + apply

Correcting myself on add/remove

I said in this thread that routing add→create and remove→delete was a stable rename. It is not, and I repeated it to the reviewer three times. The v1 pair edits the roster and drives systemd; the v2 pair changes desired state without runtime actions and leaves convergence to apply. add collects four fields where AGENT_REQUEST_KEYS needs eleven — two of the invented ones being provider and toolPolicy. Defaulting an agent's tool policy silently picks its privilege surface for the operator. So they refuse.

⚠️ Attribution warning for anyone verifying this

A correct fix here shows install rc=0, start rc=0, and still no live seat. #1240 (tmux absent, and neither the installer nor doctor says so) is upstream of this one; #1241 (start reports lifecycle: complete over dead panes) and the absent agent runtime are downstream. A dead pane after this change is not a #1237 regression, and #1241 in particular will make a correct fix here look like it failed.

Piece B is not in this PR

Which producer emits a v2 roster, and what upgrade does to an existing v1 install, is a design decision. Not taking it unilaterally.

Review

Reviewed over git comms by @scooby, who has no mosaicstack principal on fomo-lin and will not borrow a token, so their sign-off is carried in the PR body rather than filed as a Gitea review. They found a real boot window I had opened (install enables the unit but writes no env on v2 → a reboot before the first apply fails every seat unit); it is closed with ConditionPathExists on [email protected], and the effect was measured on canary — with the condition, start is rc=0/Result=success/skipped-clean; with it removed by drop-in and nothing else changed, rc=1/ExecMainStatus=64/failed.

**Piece A is up: PR #1243** (base `next`, branch `fix/1237-fleet-v2-dispatch`). `ps` / `install` / `install-systemd` now read the roster through a narrow read-only view, so they work on v2. `install` on v2 places tool files and units and writes **no** generated env — the reconciler owns that file, and a second writer here would go through the v1 mapping, which is the #791 invariant. `add` / `remove` **refuse** on v2 and name the `create`/`delete` → `apply` sequence. Canary red→green, same v2 roster, installed `0.0.50-next.2413` vs the branch: | command | 0.0.50-next.2413 | branch | |---|---|---| | `fleet ps` | rc=1 `unknown field(s): generation` | **rc=0**, 3 agents | | `fleet install --no-enable` | rc=1 same | **rc=0** | | `fleet remove <name>` | rc=1 same | rc=1, refusal naming `delete` + `apply` | | `fleet add <name> …` | rc=1 same | rc=1, refusal naming `create` + `apply` | ### Correcting myself on `add`/`remove` I said in this thread that routing `add`→`create` and `remove`→`delete` was a stable rename. It is not, and I repeated it to the reviewer three times. The v1 pair edits the roster **and** drives systemd; the v2 pair changes desired state without runtime actions and leaves convergence to `apply`. `add` collects four fields where `AGENT_REQUEST_KEYS` needs eleven — two of the invented ones being `provider` and `toolPolicy`. Defaulting an agent's tool policy silently picks its privilege surface for the operator. So they refuse. ### ⚠️ Attribution warning for anyone verifying this **A correct fix here shows `install` rc=0, `start` rc=0, and still no live seat.** #1240 (tmux absent, and neither the installer nor `doctor` says so) is upstream of this one; #1241 (`start` reports `lifecycle: complete` over dead panes) and the absent agent runtime are downstream. A dead pane after this change is not a #1237 regression, and #1241 in particular will make a correct fix here look like it failed. ### Piece B is not in this PR Which producer emits a v2 roster, and what `upgrade` does to an existing v1 install, is a design decision. Not taking it unilaterally. ### Review Reviewed over git comms by **@scooby**, who has no `mosaicstack` principal on fomo-lin and will not borrow a token, so their sign-off is carried in the PR body rather than filed as a Gitea review. They found a real boot window I had opened (install enables the unit but writes no env on v2 → a reboot before the first `apply` fails every seat unit); it is closed with `ConditionPathExists` on `[email protected]`, and the effect was measured on canary — with the condition, start is rc=0/`Result=success`/skipped-clean; with it removed by drop-in and nothing else changed, rc=1/`ExecMainStatus=64`/failed.
fred closed this issue 2026-08-16 18:06:51 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 coder2 coder3 f10-coder fargo fred happy jason.woltje (Jason Woltje) merge-gate pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev0 sanity scooby (Scooby) scrappy shaggy tess tiny velma woodpecker
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1237