mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

fleet reconciler swallows every error message and stack; 'mosaic fleet doctor' can only ever say reconcile-failed #1238

New Issue
Open
opened 2026-08-16 03:03:04 +00:00 by fred · 0 comments
fred commented 2026-08-16 03:03:04 +00:00
Collaborator
Copy Link
Copy Source

writeOutcome in dist/commands/fleet-reconciler-command.js catches every error, prints a bare code, and discards the message and stack. It wraps fleet doctor — the command whose entire purpose is diagnosis — along with apply, create, delete, get and plan.

async function writeOutcome(action) {
    try {
        await action();
    }
    catch (error) {
        process.exitCode = 1;
        printJson({
            error: {
                code: error instanceof FleetReconcileError ? error.code : 'reconcile-failed',
            },
        });
    }
}

Two separate losses:

  • Known errors lose their message. FleetReconcileError is constructed with one — e.g. new FleetReconcileError('lifecycle-precondition-failed', 'Roster-v2 reconciliation requires the canonical roster path.') — and only error.code is ever printed. The sentence explaining what to do is thrown away.
  • Unknown errors lose everything. Message, stack, and type are replaced with the literal 'reconcile-failed', which names nothing.

What it cost in practice

mosaic fleet doctor on a freshly-installed host:

$ mosaic fleet doctor
{"error":{"code":"reconcile-failed"}}

No stderr, no detail, and no flag that changes it — fleet doctor --help lists only -h, and MOSAIC_DEBUG=1/DEBUG=1 make no difference. There is nothing an operator can do with that output.

Instrumenting the catch on a sandbox copy gave the actual error immediately:

RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration.
    at normalizeRosterV2 (…/dist/fleet/roster-v2.js:209:15)

That message is precisely what the operator needs, and it names the fix. It was one string away from being printed. Diagnosing #1237 required patching an installed binary to see it.

The same pattern shows up elsewhere in the fleet surface — migrate-v1 preview returns {"code":"migration-preview-failed","detail":"Migration preview failed without publishable detail."}, which spends a sentence saying there is no detail.

Suggested fix

Print the message. For an unrecognised error, print the message and stack too, or at least the constructor name:

catch (error) {
    process.exitCode = 1;
    const known = error instanceof FleetReconcileError;
    printJson({
        error: {
            code: known ? error.code : 'reconcile-failed',
            message: error?.message,
            ...(known ? {} : { type: error?.constructor?.name, stack: error?.stack }),
        },
    });
}

If the JSON shape is a contract for machine consumers, add the detail on stderr instead so it does not disturb stdout.

I would treat "an error in the diagnostic command reveals nothing" as its own class of defect. Whatever is decided for the reconciler, a doctor that can only ever say reconcile-failed is not doing its job.

Related

  • #1237 — the defect this concealed.
  • #1234 — a different check that reports the wrong answer confidently. Same theme: the diagnostic layer is where the trust is, and it is the least reliable part of the surface right now.

Reported by fred (orchestrator seat, sb-it-1-dt), measured on canary VMID 1125 at 0.0.50-next.2413.

`writeOutcome` in `dist/commands/fleet-reconciler-command.js` catches every error, prints a bare code, and discards the message and stack. It wraps `fleet doctor` — the command whose entire purpose is diagnosis — along with `apply`, `create`, `delete`, `get` and `plan`. ```js async function writeOutcome(action) { try { await action(); } catch (error) { process.exitCode = 1; printJson({ error: { code: error instanceof FleetReconcileError ? error.code : 'reconcile-failed', }, }); } } ``` Two separate losses: - **Known errors lose their message.** `FleetReconcileError` is constructed with one — e.g. `new FleetReconcileError('lifecycle-precondition-failed', 'Roster-v2 reconciliation requires the canonical roster path.')` — and only `error.code` is ever printed. The sentence explaining what to do is thrown away. - **Unknown errors lose everything.** Message, stack, and type are replaced with the literal `'reconcile-failed'`, which names nothing. ## What it cost in practice `mosaic fleet doctor` on a freshly-installed host: ``` $ mosaic fleet doctor {"error":{"code":"reconcile-failed"}} ``` No stderr, no detail, and no flag that changes it — `fleet doctor --help` lists only `-h`, and `MOSAIC_DEBUG=1`/`DEBUG=1` make no difference. There is nothing an operator can do with that output. Instrumenting the catch on a sandbox copy gave the actual error immediately: ``` RosterV2ValidationError: Roster v2 compiler rejects v1 input; use the existing v1 path until migration. at normalizeRosterV2 (…/dist/fleet/roster-v2.js:209:15) ``` That message is *precisely* what the operator needs, and it names the fix. It was one string away from being printed. Diagnosing #1237 required patching an installed binary to see it. The same pattern shows up elsewhere in the fleet surface — `migrate-v1 preview` returns `{"code":"migration-preview-failed","detail":"Migration preview failed without publishable detail."}`, which spends a sentence saying there is no detail. ## Suggested fix Print the message. For an unrecognised error, print the message and stack too, or at least the constructor name: ```js catch (error) { process.exitCode = 1; const known = error instanceof FleetReconcileError; printJson({ error: { code: known ? error.code : 'reconcile-failed', message: error?.message, ...(known ? {} : { type: error?.constructor?.name, stack: error?.stack }), }, }); } ``` If the JSON shape is a contract for machine consumers, add the detail on stderr instead so it does not disturb stdout. I would treat "an error in the diagnostic command reveals nothing" as its own class of defect. Whatever is decided for the reconciler, a `doctor` that can only ever say `reconcile-failed` is not doing its job. ## Related - #1237 — the defect this concealed. - #1234 — a different check that reports the wrong answer confidently. Same theme: the diagnostic layer is where the trust is, and it is the least reliable part of the surface right now. Reported by fred (orchestrator seat, sb-it-1-dt), measured on canary VMID 1125 at 0.0.50-next.2413.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1238