stack/docs/scratchpads/758-fcm-m3-001-local-reconciler.md

FCM-M3-001 — Local roster-owned reconciliation and lifecycle

• Task / issue: FCM-M3-001 / #758
• Branch / base: feat/758-local-reconciler from origin/main bc5e73629e92c56a80fa6a769ebad17c0177f504
• Base tree: 1b9ebe4fa1a90734b6f81118e120bae5290cd350
• Scope: source, isolated fake-adapter tests, and card documentation only. No live fleet/systemd/tmux action.

Objective

Provide local roster-v2 apply/reconcile and lifecycle/status contracts. The roster remains desired-state authority; projections and runtime observations are derived state.

Red-first evidence

The initial focused reconciler test failed because fleet-reconciler.ts did not exist. The initial new-worktree test invocation also exposed absent dependencies; pnpm install --frozen-lockfile --store-dir /home/jarvis/.local/share/pnpm/store restored local workspace dependencies without changing source.

Design

• A new fleet-reconciler.ts accepts only typed roster-v2 input plus injected command and projection adapters.
• It targets only exact mosaic-agent@<roster-name>.service units and the exact configured tmux socket/session.
• It classifies unowned/unmanaged state and fails mutation closed rather than adopting or killing it.
• It validates the private install-derived holder identity and complete expected tmux global environment before mutating lifecycle state.
• REVIEW-1 remediation: RED review evidence found service-level apply could omit generation and had no mutation lock. Every non-observational command now requires an expected generation; a private exclusive roster-adjacent lock is acquired before effects and released on success or partial failure. Tests cover missing/stale values, concurrent denial, no effects, release, and lock-free observation.
• REVIEW-2 remediation: lock acquisition now validates private real MOSAIC_HOME/fleet ancestors, rejects symlink or unsafe leaves, distinguishes EEXIST concurrency from other I/O, and binds release to the created inode plus random ownership token. A replacement lock is retained and reported, not unlinked. A crash may leave a stale lock for inspection; no stale-lock break is claimed.
• REVIEW-3 remediation: a lock cleanup failure now adds bounded cleanup diagnostics to a known successful or partial effect result without replacing its projection/lifecycle/recovery truth. Cleanup is not claimed as complete, and the retained lock requires inspection before retry.
• REVIEW-4 remediation: command JSON with an additive cleanup diagnostic now exits non-zero even where known effects completed; clean effect and observational JSON remain zero-exit.
• REVIEW-5 remediation: mutating operations acquire the private lock before rereading canonical roster.yaml; the fenced reread, not a caller snapshot, supplies generation validation, plan, projection, and lifecycle authority.
• apply starts only enabled agents whose persisted desired state is running; stopped/default agents are never started by reconciliation.
• Observational commands produce JSON classification only. Partial projection or lifecycle effects report explicit recovery without values.

Boundaries

Excluded: live host actions, remote/SSH/connector lifecycle mutation, migrations, canaries, deployment, gateway changes, arbitrary command/channel/secret inputs, docs/TASKS.md, and orchestration ledgers.

Verification

• Focused reconciler/Commander/CRUD/fleet tests: 4 files / 229 tests passed.
• Full @mosaicstack/mosaic suite: 56 files / 820 tests passed after REVIEW-5 canonical roster fencing remediation.
• Package and root typecheck/lint, root format check, and git diff --check: passed.
• Isolated launcher and systemd template harnesses passed; they use fixtures only. No live fleet, systemd, tmux, session, remote, connector, or runtime action occurred.