FCM-M3-002 — Reconciler lifecycle acceptance gates

Task / issue: FCM-M3-002 / Fleet configuration control plane and operator documentation (mosaicstack/stack#758)
Branch: test/758-reconciler-lifecycle-gates
Required starting head: 499090508ef1d768660e4d54e7934cbcf13cb1cd
Required starting tree: 2f1bb7fed48291f3f7ba8b21c2b52491aa14fe2b
Scope: isolated acceptance coverage and card-required evidence/tracking only; no live fleet, systemd, tmux, session, site, migration, canary, deployment, runtime, connector, or remote action.
Budget: use the task estimate of 25K as the working cap; keep the delta to one coherent acceptance suite plus required task/scratchpad evidence. No production change unless a failing reproducer proves an in-scope defect.

Intake evidence

Clean exact local branch/head/tree verified before editing.
origin/test/758-reconciler-lifecycle-gates fetched and verified at the same required head.
The Mosaic PR wrapper reported no open pull requests, so no open-PR branch collision exists.
Parent issue #758 is open and remains intentionally open through M5.
Requirements loaded from docs/PRD.md FCM requirements and AC-FCM-05, docs/TASKS.md FCM DAG, the M3 rows in docs/fleet/FLEET-CONFIG-DOCS-IA-CHECKLIST.md, and the FCM-M3-001 implementation scratchpad.

Objective

Add broad, behavior-oriented acceptance evidence around the shipped local reconciler contracts. Exercise only injected fake systemd/tmux adapters and temporary filesystem fixtures. Prove exact ownership/targeting, persisted stopped-state safety, truthful partial-failure recovery, rollback behavior, and stable command JSON/exit outcomes without touching live services or sessions.

Acceptance mapping and evidence

FCM-M3-002 acceptance concern	Delivered isolated evidence
Systemd/tmux lifecycle	`fleet-reconciler.acceptance.spec.ts` drives apply, reconcile, stop, restart, status, and recovery reconcile through one stateful injected fake host. The fake models exact systemd effects and tmux session observations; no host commands run.
Drift	Canonical roster-v2 YAML drives the Commander `fleet status` boundary and classifies `missing-session`, `unexpected-session`, and `disabled-running`, including combined drift, while asserting observation emits no lifecycle mutation.
Exact default/named socket targeting	Canonical v2 requires an explicit non-empty named socket; the parser rejects missing/empty values and the Commander acceptance path asserts exact `-L mosaic-fleet` targeting. A separate canonical legacy-v1 roster loader plus runtime-transport path proves a socket-less compatibility roster targets the literal tmux default server with no `-L`. No unreachable empty-socket v2 fixture is used.
Unmanaged-session classification	Stateful fixtures report sorted `coder0-shadow`/`unmanaged` sessions, then prove an exact roster stop leaves both sessions and the near-collision service intact.
Crash/partial failure	Injected restart failure is applied after the fake effect to model crash/partial truth: result is `lifecycle: incomplete`, the roster is unchanged, and observed runtime may be active.
Rollback/recovery semantics	M3 has no rollback command and explicitly does not claim automatic rollback. The acceptance workflow proves the bounded recovery contract: inspect, then exact reconcile restores the persisted stopped target without a start or fuzzy effect. M4 migration/canary rollback remains outside this card.
Stopped-state preservation	Stateful apply and reconcile both stop an initially running observed agent whose persisted target is stopped; failed explicit restart leaves desired state stopped; recovery reconcile restores stopped state. No start call is emitted.
Zero fuzzy destructive targeting	Near-collision `coder0-shadow` service/session plus `unmanaged` session remain untouched. The recorded destructive calls contain only exact `mosaic-agent@coder0.service`; no tmux kill action is emitted.
Stable JSON/exit behavior	Temporary canonical roster fixture invokes the CLI boundary and asserts exactly one JSON line, exact partial-result shape, and exit code 1. Existing focused command specs continue to cover clean zero-exit and stable error JSON.
Redacted truthful recovery	Fake stderr includes `PASSWORD=acceptance-secret`; exact CLI JSON contains only bounded recovery metadata and excludes the key, value, and raw diagnostic.

Plan

Inventory existing reconciler and command specs against the table above; avoid duplicating narrow assertions already present.
Add one acceptance-level spec using only fake/injected adapters and temporary files.
If a real defect is exposed, preserve the failing reproducer and make only the smallest FCM-M3-002-required fix; otherwise leave production unchanged.
Reconcile docs/TASKS.md only for delivered M1/M2/M3-001 truth and mark FCM-M3-002 in progress.
Run focused tests, full @mosaicstack/mosaic tests, package/root typecheck and lint, Prettier/format and diff checks, plus adversarial fake-runner cases.
Record exact evidence and leave the tree uncommitted for independent synthetic-tree review.

TDD decision

This card adds acceptance coverage to already-delivered behavior. Test-first applies to any product defect discovered: retain a failing reproducer before an in-scope fix. If the shipped behavior already satisfies the acceptance contract, no production code will be changed and the acceptance suite itself is the deliverable.

Progress

Intake and immutable baseline verification complete.
Existing coverage inventory confirmed strong unit coverage but no stateful cross-command lifecycle acceptance harness.
Added packages/mosaic/src/fleet/fleet-reconciler.acceptance.spec.ts: one injected stateful fake systemd/tmux host, temporary canonical v2 and legacy-v1 roster fixtures, and seven acceptance tests.
Production source is unchanged; no product defect requiring an FCM-M3-002 fix was found.
docs/TASKS.md reconciles only merged M1/M2/M3-001 truth and marks FCM-M3-002 in progress.

Verification evidence

All commands ran from /home/jarvis/src/mosaic-stack-local-reconciler and passed unless explicitly noted.

pnpm --filter @mosaicstack/mosaic exec vitest run src/fleet/fleet-reconciler.acceptance.spec.ts — final remediation run: 1 file, 7 tests passed; canonical v2 named-socket parsing/Commander status, missing/empty v2 rejection, and canonical legacy-v1 default-server runtime targeting are distinct reachable cases.
Focused reconciler/roster/transport command covering acceptance, reconciler, command, CRUD, v2 parser, and runtime transport specs — 8 files, 304 tests passed.
pnpm --filter @mosaicstack/mosaic test — final remediation run: 57 files, 827 tests passed.
pnpm --filter @mosaicstack/mosaic typecheck — passed.
pnpm --filter @mosaicstack/mosaic lint — passed.
pnpm typecheck — 42/42 Turbo tasks successful.
pnpm lint — 23/23 Turbo tasks successful.
pnpm exec prettier --check docs/TASKS.md docs/scratchpads/758-fcm-m3-002-reconciler-lifecycle-gates.md packages/mosaic/src/fleet/fleet-reconciler.acceptance.spec.ts — passed.
pnpm format:check — all matched files use Prettier style.
git diff --check — passed with no output.
Initial scoped Prettier check found style drift in the new spec and tracking table; pnpm exec prettier --write ... remediated it before all final gates above.
No live fleet, systemctl, tmux, process, site, migration, canary, deploy, runtime, connector, or remote command was invoked.

Review boundary

This is an author handoff. No self-review is represented as reviewer-of-record. The uncommitted synthetic tree is intended for independent review.

Risks / blockers

M3 truthfully reports incomplete lifecycle effects and bounded recovery; it does not implement or claim an automatic rollback command. This suite proves stopped-state restoration by the documented exact recovery reconcile. M4 retains migration/canary rollback ownership.
The fake host models only the public systemd/tmux runner contract and temporary roster filesystem boundary. This is intentional under the no-live-effects hold.
Parent issue closure, commit, push, PR, merge, deployment, and branch cleanup remain explicit holds.
No residual implementation blocker.

8.2 KiB Raw Blame History