mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

fleet init writes a v1 roster that fleet doctor cannot read, and writeOutcome() discards the message that says so #1261

New Issue
Open
opened 2026-08-16 21:13:14 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-16 21:13:14 +00:00
Copy Link
Copy Source

mosaic fleet init writes a roster that mosaic fleet doctor cannot read, and the CLI throws away the message that says so

Two defects, one chain. Found on a greenfield sandbox VM by @daphne, reproduced independently on a
second host with a temp --mosaic-home (no faked HOME, no install).

The chain, measured

Installed CLI 0.0.49 on sb-it-1-dt, and CLI 0.0.50-next.2439 on a greenfield Debian 13 canary
(mosaic-sbx-canary, VMID 1125, reverted to the greenfield snapshot). Byte-identical behaviour on both.

$ mosaic fleet init --profile general --write --mosaic-home <tmp>
Initialized general fleet: 1 orchestrator + 1 enhancer(s) + 1 worker(s). Next: mosaic fleet install
rc=0
$ head -1 <tmp>/fleet/roster.yaml
version: 1

$ mosaic fleet doctor --mosaic-home <tmp>
{"error":{"code":"reconcile-failed"}}
rc=1

That is the entire diagnostic an operator gets on a fresh host, from the command immediately after the
one the CLI told them to run.

Defect 1 — the error message is authored, thrown, and discarded

mosaic fleet doctor is the roster-v2 reconciler (fleet-reconciler-command.ts:43). It calls
parseRosterV2() unconditionally on the roster file (:36). Calling that parser directly on the
roster fleet init just wrote:

RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration.

That message is exactly what the operator needs. It never reaches them, because of
fleet-reconciler-command.ts:130-141:

async function writeOutcome(action: () => Promise<void>): Promise<void> {
  try { await action(); } catch (error: unknown) {
    process.exitCode = 1;
    printJson({ error: { code: error instanceof FleetReconcileError ? error.code : 'reconcile-failed' } });
  }
}

Only code is printed, and only FleetReconcileError carries one. Every other error — including
RosterV2ValidationError, whose whole purpose is to explain this exact situation — is flattened to the
string reconcile-failed with no message, no cause, and no stack. The reporter covers the error class
its author enumerated and discards the one the operator actually hits.

Minimum fix: emit error.message alongside error.code for any thrown error, and give
RosterV2ValidationError its own code (roster-generation-mismatch or similar) rather than letting it
fall through to the catch-all.

Defect 2 — init and the v2 command family disagree about what a roster is

fleet init emits version: 1. The entire reconciler family — doctor, create, plan, apply,
get, and the v2 lifecycle verbs — is v2-only and rejects it. So the documented greenfield sequence
writes a roster that its own next step cannot parse. Nothing shipped writes a v2 roster, so on a fresh
host there is no way to reach the v2 commands at all.

What is NOT broken, and this matters for anyone triaging a greenfield host

The v1 path works on that same roster. Measured after the failing doctor:

$ mosaic fleet add probe --runtime pi --class probe --no-start --mosaic-home <tmp>
Added probe (pi/probe) to the fleet.
Agent queued (--no-start); run: mosaic fleet start probe
rc=0
$ mosaic fleet ps --mosaic-home <tmp>     # lists the roster's agents
rc=0

reconcile-failed reads as "the fleet is broken" and stopped a greenfield E2E run cold. It means
"you ran a v2 command against a v1 roster." Those are very different states and the CLI cannot
currently tell them apart out loud.

Two adjacent things found in the same pass

  • mosaic fleet verify crashes with a raw unhandled Node stack when the roster's sessions do not exist
    — Error: can't find session: orchestrator thrown from runChecked (fleet.js:1912), no diagnosis.
    On a freshly-initialized host, no session exists yet, so this is the normal state for verify's
    first run.
  • mosaic fleet status ignores --mosaic-home and reports the host's real systemd holder unit.
    A temp-home invocation printing live-host state is a footgun for exactly the sandbox testing this
    chain was found by.

-- fred (sb-it-1-dt)

## `mosaic fleet init` writes a roster that `mosaic fleet doctor` cannot read, and the CLI throws away the message that says so Two defects, one chain. Found on a greenfield sandbox VM by @daphne, reproduced independently on a second host with a temp `--mosaic-home` (no faked `HOME`, no install). ### The chain, measured Installed CLI **0.0.49** on sb-it-1-dt, and CLI **0.0.50-next.2439** on a greenfield Debian 13 canary (`mosaic-sbx-canary`, VMID 1125, reverted to the `greenfield` snapshot). Byte-identical behaviour on both. ```text $ mosaic fleet init --profile general --write --mosaic-home <tmp> Initialized general fleet: 1 orchestrator + 1 enhancer(s) + 1 worker(s). Next: mosaic fleet install rc=0 $ head -1 <tmp>/fleet/roster.yaml version: 1 $ mosaic fleet doctor --mosaic-home <tmp> {"error":{"code":"reconcile-failed"}} rc=1 ``` That is the entire diagnostic an operator gets on a fresh host, from the command immediately after the one the CLI told them to run. ### Defect 1 — the error message is authored, thrown, and discarded `mosaic fleet doctor` is the roster-v2 reconciler (`fleet-reconciler-command.ts:43`). It calls `parseRosterV2()` unconditionally on the roster file (`:36`). Calling that parser directly on the roster `fleet init` just wrote: ```text RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration. ``` That message is exactly what the operator needs. It never reaches them, because of `fleet-reconciler-command.ts:130-141`: ```ts async function writeOutcome(action: () => Promise<void>): Promise<void> { try { await action(); } catch (error: unknown) { process.exitCode = 1; printJson({ error: { code: error instanceof FleetReconcileError ? error.code : 'reconcile-failed' } }); } } ``` Only `code` is printed, and only `FleetReconcileError` carries one. Every other error — including `RosterV2ValidationError`, whose whole purpose is to explain this exact situation — is flattened to the string `reconcile-failed` with no message, no cause, and no stack. The reporter covers the error class its author enumerated and discards the one the operator actually hits. Minimum fix: emit `error.message` alongside `error.code` for any thrown error, and give `RosterV2ValidationError` its own code (`roster-generation-mismatch` or similar) rather than letting it fall through to the catch-all. ### Defect 2 — `init` and the v2 command family disagree about what a roster is `fleet init` emits `version: 1`. The entire reconciler family — `doctor`, `create`, `plan`, `apply`, `get`, and the v2 lifecycle verbs — is v2-only and rejects it. So the documented greenfield sequence writes a roster that its own next step cannot parse. Nothing shipped writes a v2 roster, so on a fresh host there is no way to reach the v2 commands at all. ### What is NOT broken, and this matters for anyone triaging a greenfield host The **v1 path works on that same roster**. Measured after the failing `doctor`: ```text $ mosaic fleet add probe --runtime pi --class probe --no-start --mosaic-home <tmp> Added probe (pi/probe) to the fleet. Agent queued (--no-start); run: mosaic fleet start probe rc=0 $ mosaic fleet ps --mosaic-home <tmp> # lists the roster's agents rc=0 ``` `reconcile-failed` reads as "the fleet is broken" and stopped a greenfield E2E run cold. It means "you ran a v2 command against a v1 roster." Those are very different states and the CLI cannot currently tell them apart out loud. ### Two adjacent things found in the same pass - `mosaic fleet verify` crashes with a raw unhandled Node stack when the roster's sessions do not exist — `Error: can't find session: orchestrator` thrown from `runChecked` (`fleet.js:1912`), no diagnosis. On a freshly-initialized host, no session exists yet, so this is the normal state for `verify`'s first run. - `mosaic fleet status` ignores `--mosaic-home` and reports the **host's real** systemd holder unit. A temp-home invocation printing live-host state is a footgun for exactly the sandbox testing this chain was found by. -- fred (sb-it-1-dt)
Ghost added the bug label 2026-08-16 21:13:14 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1261