mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

fleet init: two presets violate the agent floor, dry-run skips the check, and a failed --write still persists the rejected roster #1239

New Issue
Open
opened 2026-08-16 03:07:40 +00:00 by fred · 1 comment
fred commented 2026-08-16 03:07:40 +00:00
Collaborator
Copy Link
Copy Source

Two defects in mosaic fleet init, both hit on a fresh install, both about the same thing: validation runs after the point of no return.

Measured on canary VMID 1125, CLI 0.0.50-next.2413, greenfield --next install.

1. Two of the six presets cannot be written

fleet init --write enforces a floor of one orchestrator plus one enhancer. Two shipped presets do not meet it:

preset agents orchestrator enhancer --write
general 3 yes yes rc=0
coding 5 yes yes rc=0
research 5 yes yes rc=0
hybrid 5 yes yes rc=0
minimal 1 no no rc=1
local-canary 3 yes no rc=1

local-canary is the preset named for exactly the case I was using it for — a single sandbox host bringing up a local fleet — and it is one of the two that cannot be written. minimal is the obvious first thing anyone tries.

Either the presets are wrong or the floor is wrong. If the floor is right, these two presets should not ship; if they are legitimate shapes, the floor needs to allow them. What is not defensible is shipping both and letting the operator discover the contradiction.

2. Dry-run passes where --write fails

Both failing presets return rc=0 from the dry run:

$ mosaic fleet init --preset local-canary          # no --write
… renders the plan …
rc=0

$ mosaic fleet init --preset local-canary --write
rc=1   (two-agent floor)

The dry run is the mechanism for checking a plan before committing it, and it does not run the check that will reject the plan. Whatever validation --write performs should run identically in preview.

3. A failed --write leaves the rejected roster on disk

This is the part that costs the operator time. The local-canary --write run above exited 1 — and left a roster behind:

-rw------- 1 mosaic mosaic … Aug 16 02:58 /home/mosaic/.config/mosaic/fleet/roster.yaml

mtime 02:58:19, matching the failed run, containing the local-canary agent set. So the command persisted a state its own validator had just rejected, and every subsequent fleet init then refused to run because a roster already exists — the retry is now gated behind --force.

The operator's position after one wrong preset choice: an invalid roster on disk, a command that will not re-run without a flag, and no message saying either of those things happened. The write should be atomic — validate, then write, or do not write at all.

Suggested fixes

  • Run the same validation in dry-run and --write.
  • Write the roster only after validation passes (render to a temp file, validate, rename).
  • Resolve the preset/floor contradiction one way or the other. If local-canary is meant to be a valid single-host shape, that is the one to keep.
  • On the failure path, say what happened: which rule failed, which agents are missing, and whether anything was written.

Related

  • #1236 — fleet init --write also fails on directory permissions on a stock Debian host. Fixing that is what let me get far enough to hit these.
  • #1237 — the roster it writes is v1, which every reconciler command then rejects.
  • #1238 — the reason the failures above took instrumentation to read.

Reported by fred (orchestrator seat, sb-it-1-dt).

Two defects in `mosaic fleet init`, both hit on a fresh install, both about the same thing: validation runs after the point of no return. Measured on canary VMID 1125, CLI 0.0.50-next.2413, greenfield `--next` install. ## 1. Two of the six presets cannot be written `fleet init --write` enforces a floor of one orchestrator plus one enhancer. Two shipped presets do not meet it: | preset | agents | orchestrator | enhancer | `--write` | |---|---|---|---|---| | `general` | 3 | yes | yes | rc=0 | | `coding` | 5 | yes | yes | rc=0 | | `research` | 5 | yes | yes | rc=0 | | `hybrid` | 5 | yes | yes | rc=0 | | `minimal` | 1 | no | no | **rc=1** | | `local-canary` | 3 | yes | **no** | **rc=1** | `local-canary` is the preset named for exactly the case I was using it for — a single sandbox host bringing up a local fleet — and it is one of the two that cannot be written. `minimal` is the obvious first thing anyone tries. Either the presets are wrong or the floor is wrong. If the floor is right, these two presets should not ship; if they are legitimate shapes, the floor needs to allow them. What is not defensible is shipping both and letting the operator discover the contradiction. ## 2. Dry-run passes where `--write` fails Both failing presets return **rc=0 from the dry run**: ``` $ mosaic fleet init --preset local-canary # no --write … renders the plan … rc=0 $ mosaic fleet init --preset local-canary --write rc=1 (two-agent floor) ``` The dry run is the mechanism for checking a plan before committing it, and it does not run the check that will reject the plan. Whatever validation `--write` performs should run identically in preview. ## 3. A failed `--write` leaves the rejected roster on disk This is the part that costs the operator time. The `local-canary --write` run above exited 1 — and left a roster behind: ``` -rw------- 1 mosaic mosaic … Aug 16 02:58 /home/mosaic/.config/mosaic/fleet/roster.yaml ``` mtime `02:58:19`, matching the failed run, containing the `local-canary` agent set. So the command persisted a state its own validator had just rejected, and every subsequent `fleet init` then refused to run because a roster already exists — the retry is now gated behind `--force`. The operator's position after one wrong preset choice: an invalid roster on disk, a command that will not re-run without a flag, and no message saying either of those things happened. The write should be atomic — validate, then write, or do not write at all. ## Suggested fixes - Run the same validation in dry-run and `--write`. - Write the roster only after validation passes (render to a temp file, validate, rename). - Resolve the preset/floor contradiction one way or the other. If `local-canary` is meant to be a valid single-host shape, that is the one to keep. - On the failure path, say what happened: which rule failed, which agents are missing, and whether anything was written. ## Related - #1236 — `fleet init --write` also fails on directory permissions on a stock Debian host. Fixing that is what let me get far enough to hit these. - #1237 — the roster it writes is v1, which every reconciler command then rejects. - #1238 — the reason the failures above took instrumentation to read. Reported by fred (orchestrator seat, sb-it-1-dt).
Ghost commented 2026-08-16 03:34:31 +00:00
Copy Link
Copy Source

Correcting two errors in my own report, and re-measuring the rest on a permission-fixed box

Two things in the body above are wrong and would cost the next person time. Both found by re-running the whole thing after #1236 was fixed, on canary VMID 1125, CLI 0.0.50-next.2413.

Error 1 — the flag is --profile, not --preset

Every command in the body above is unreproducible as written:

$ mosaic fleet init --preset general --write
error: unknown option '--preset'

Usage: mosaic fleet init [options]
  --profile <name>  Roster profile: general, coding, research, hybrid, minimal, local-canary
  --write           Write the roster to Mosaic home
  --force           Overwrite an existing roster when used with --write

Read --profile everywhere the body says --preset. The codebase is itself split on the term — the CLI flag and help text say profile, the error message the failing case prints says preset ("The preset may be corrupted") — which is how I came to write the wrong one down, but the flag an operator has to type is --profile.

Error 2 — minimal writes fine. My table is wrong.

I listed minimal as rc=1 against the two-agent floor. Re-measured, all six profiles, dry-run and --write, each from a removed roster:

profile dry-run rc roster after dry-run --write rc roster after --write
general 0 absent 0 present
coding 0 absent 0 present
research 0 absent 0 present
hybrid 0 absent 0 present
minimal 0 absent 0 present
local-canary 0 absent 1 present

minimal writes, and says so: Initialized minimal fleet: 1 agent(s) (no orchestrator). One agent, no orchestrator, no enhancer — so the floor is not enforced against it at all.

Which makes the contradiction sharper, not softer. Only local-canary fails, and it fails on the enhancer half alone:

Error: Fleet init failed: the "local-canary" roster has no enhancer agent. Every fleet
keeps an orchestrator + enhancer minimum (two-agent floor). The preset may be corrupted —
re-install the framework.

So a shipped profile with one agent and no orchestrator is accepted, and a shipped profile with three agents and an orchestrator is rejected. Whatever the floor is protecting, it is not applied consistently to the profiles that ship alongside it.

And the message tells the operator something false. local-canary is not corrupted and re-installing the framework will not help — it ships in that shape, so the advice sends them to re-run an installer that reproduces the exact file. That line should go regardless of how the floor question is settled.

Claim 3 stands, and the retry gate is confirmed

The roster is left on disk after the failed --write — the table's last column, measured on a run that exited 1. The follow-on gate is real too:

Error: Fleet roster already exists: /home/mosaic/.config/mosaic/fleet/roster.yaml.
Re-run with --force to overwrite.

So: pick the one profile named for a single sandbox host, get an error telling you to re-install the framework, and find your next attempt blocked by a roster you were never told had been written.

Worth being explicit that this is a different failure from #1236, since the two look alike from the outside. #1236 fails in the permission check before anything renders, and leaves no roster behind — I confirmed that on the RED box. This one fails validation after the write. Fixing #1236 is what exposed it.

One more thing the re-run showed

Every one of these paths throws raw. local-canary --write, the already-exists gate, and the #1236 permission failure all print a Node stack trace with a source path and line number:

file:///home/mosaic/.npm-global/lib/node_modules/@mosaicstack/mosaic/dist/commands/fleet.js:1158
            throw new Error(`Fleet init failed: the "${profile}" roster has no enhancer agent. …
                  ^
Error: Fleet init failed: …
    at Command.<anonymous> (…/dist/commands/fleet.js:1158:19)
Node.js v22.23.2

That is the mirror image of #1238, and the pair is worth seeing together: the reconciler commands catch everything and print {"error":{"code":"reconcile-failed"}} with no message, while fleet init catches nothing and prints a stack trace. Neither gives an operator a usable error. Noting it here rather than filing a seventh issue.

— fred

# Correcting two errors in my own report, and re-measuring the rest on a permission-fixed box Two things in the body above are wrong and would cost the next person time. Both found by re-running the whole thing after #1236 was fixed, on canary VMID 1125, CLI `0.0.50-next.2413`. ## Error 1 — the flag is `--profile`, not `--preset` Every command in the body above is unreproducible as written: ``` $ mosaic fleet init --preset general --write error: unknown option '--preset' ``` ``` Usage: mosaic fleet init [options] --profile <name> Roster profile: general, coding, research, hybrid, minimal, local-canary --write Write the roster to Mosaic home --force Overwrite an existing roster when used with --write ``` Read `--profile` everywhere the body says `--preset`. The codebase is itself split on the term — the CLI flag and help text say **profile**, the error message the failing case prints says **preset** ("The preset may be corrupted") — which is how I came to write the wrong one down, but the flag an operator has to type is `--profile`. ## Error 2 — `minimal` writes fine. My table is wrong. I listed `minimal` as rc=1 against the two-agent floor. Re-measured, all six profiles, dry-run and `--write`, each from a removed roster: | profile | dry-run rc | roster after dry-run | `--write` rc | roster after `--write` | |---|---|---|---|---| | general | 0 | absent | 0 | present | | coding | 0 | absent | 0 | present | | research | 0 | absent | 0 | present | | hybrid | 0 | absent | 0 | present | | minimal | 0 | absent | **0** | present | | local-canary | 0 | absent | **1** | **present** | `minimal` writes, and says so: `Initialized minimal fleet: 1 agent(s) (no orchestrator).` One agent, no orchestrator, no enhancer — so the floor is not enforced against it at all. **Which makes the contradiction sharper, not softer.** Only `local-canary` fails, and it fails on the enhancer half alone: ``` Error: Fleet init failed: the "local-canary" roster has no enhancer agent. Every fleet keeps an orchestrator + enhancer minimum (two-agent floor). The preset may be corrupted — re-install the framework. ``` So a shipped profile with one agent and no orchestrator is accepted, and a shipped profile with three agents and an orchestrator is rejected. Whatever the floor is protecting, it is not applied consistently to the profiles that ship alongside it. And the message tells the operator something false. `local-canary` is not corrupted and re-installing the framework will not help — it ships in that shape, so the advice sends them to re-run an installer that reproduces the exact file. That line should go regardless of how the floor question is settled. ## Claim 3 stands, and the retry gate is confirmed The roster is left on disk after the failed `--write` — the table's last column, measured on a run that exited 1. The follow-on gate is real too: ``` Error: Fleet roster already exists: /home/mosaic/.config/mosaic/fleet/roster.yaml. Re-run with --force to overwrite. ``` So: pick the one profile named for a single sandbox host, get an error telling you to re-install the framework, and find your next attempt blocked by a roster you were never told had been written. Worth being explicit that this is a *different* failure from #1236, since the two look alike from the outside. #1236 fails in the permission check before anything renders, and leaves **no** roster behind — I confirmed that on the RED box. This one fails validation after the write. Fixing #1236 is what exposed it. ## One more thing the re-run showed Every one of these paths throws raw. `local-canary --write`, the already-exists gate, and the #1236 permission failure all print a Node stack trace with a source path and line number: ``` file:///home/mosaic/.npm-global/lib/node_modules/@mosaicstack/mosaic/dist/commands/fleet.js:1158 throw new Error(`Fleet init failed: the "${profile}" roster has no enhancer agent. … ^ Error: Fleet init failed: … at Command.<anonymous> (…/dist/commands/fleet.js:1158:19) Node.js v22.23.2 ``` That is the mirror image of #1238, and the pair is worth seeing together: the reconciler commands catch everything and print `{"error":{"code":"reconcile-failed"}}` with no message, while `fleet init` catches nothing and prints a stack trace. Neither gives an operator a usable error. Noting it here rather than filing a seventh issue. — fred
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1239