feat(fleet): reconcile local roster state (#785)

docs/fleet/how-to/start-stop-restart.md

@@ -0,0 +1,23 @@
+# Safely Reconcile and Control a Local Fleet Agent
+
+Use the canonical local roster-v2 command surface:
+
+```sh
+mosaic fleet apply --expected-generation <n> --dry-run
+mosaic fleet apply --expected-generation <n>
+mosaic fleet reconcile --expected-generation <n>
+mosaic fleet start <name> --expected-generation <n>
+mosaic fleet stop <name> --expected-generation <n>
+mosaic fleet restart <name> --expected-generation <n>
+mosaic fleet status [name]
+mosaic fleet verify
+mosaic fleet doctor
+```
+
+Start with `--dry-run`. It validates roster semantics, deterministic projections, private managed paths, exact holder ownership, and named-socket state without changing files or lifecycle state. `apply` and `reconcile` rebuild derived projections and enforce only persisted roster state: enabled `running` agents may start, while stopped or disabled agents are not started.
+
+`start`, `stop`, and `restart` are explicit one-shot exact-service actions. They do not persist a lifecycle change. Roster CRUD is the only way to change persisted desired state.
+
+Every command prints JSON. Observation commands report drift without mutation; `verify` exits non-zero on ownership mismatch, unmanaged sessions, or drift. A failed apply that wrote some derived projections reports `projections: "incomplete"` with bounded recovery to regenerate from the roster. A lifecycle failure after projections reports incomplete lifecycle work; it is never represented as a rollback or no-op.
+
+These commands are local only. Remote/SSH/connector entries are inventory/validation-only. Commands do not accept arbitrary runtime commands, channels, secrets, generated-file desired state, or arbitrary tmux sockets.

docs/fleet/operations/reconcile-and-recover.md

@@ -0,0 +1,26 @@
+# Reconcile and Recover a Local Fleet
+
+## Safe sequence
+
+1. Read `mosaic fleet doctor` and `mosaic fleet status`.
+2. Run `mosaic fleet apply --expected-generation <n> --dry-run`.
+3. Resolve stale generation, ownership mismatch, unsafe path, projection validation, or unmanaged-session findings before applying.
+4. Run `mosaic fleet apply --expected-generation <n>` only after the plan is understood.
+
+The reconciler uses the exact roster tmux socket, exact holder session, private installation holder identity, and the complete expected global environment. For mutations it acquires its exclusive lock before rereading the canonical roster and fencing its generation; only that under-lock roster drives validation, planning, projections, and lifecycle effects. Before effects, its exclusive lock proves real private `MOSAIC_HOME` and `fleet` ancestors, uses a private `0600` lock leaf, and binds cleanup to the created file identity and ownership token. A fake holder, contaminated global environment, missing identity, unsafe lock path, or unmanaged session fails closed. It does not adopt, kill, or rename any unproven session. A crash can leave a stale lock for explicit operator inspection; reconciliation deliberately does not guess ownership or remove it.
+
+## Partial results
+
+The roster is never changed by reconciliation. If derived projection application partially fails, JSON reports:
+
+```json
+{
+  "applied": false,
+  "authoritativeRoster": "unchanged",
+  "projections": "incomplete",
+  "lifecycle": "not-applied",
+  "recovery": { "code": "projection-apply-failed", "action": "regenerate-projections-from-roster" }
+}
+```
+
+If projections completed but lifecycle work failed, JSON reports `projections: "complete"`, `lifecycle: "incomplete"`, and the bounded action `rerun-after-inspecting-owned-resources`. If lock cleanup cannot be proven after an effect result, it adds `cleanup: { "code": "lock-cleanup-failed", "action": "inspect-lock-before-retry" }` without changing the known projection, lifecycle, or primary recovery truth. Inspect the retained lock before retrying; no rollback, release, or stale-lock removal is implied. Results do not include environment values, secrets, or privileged command content.

docs/fleet/reference/cli.md

@@ -0,0 +1,20 @@
+# Fleet Control-Plane CLI
+
+The local roster-v2 control plane is `mosaic fleet`.
+
+```text
+mosaic fleet apply --expected-generation <n> [--dry-run]
+mosaic fleet reconcile --expected-generation <n> [--dry-run]
+mosaic fleet start [name] --expected-generation <n> [--dry-run]
+mosaic fleet stop [name] --expected-generation <n> [--dry-run]
+mosaic fleet restart [name] --expected-generation <n> [--dry-run]
+mosaic fleet status [name]
+mosaic fleet verify
+mosaic fleet doctor
+```
+
+`apply` and `reconcile` use roster desired state. `start`, `stop`, and `restart` are exact local one-shot lifecycle effects and never persist a desired-state edit. `status`, `verify`, and `doctor` are observational.
+
+Commands emit one JSON object. Handled precondition errors emit `{ "error": { "code": "..." } }` and exit non-zero. Partial derived/lifecycle effects use explicit `authoritativeRoster`, `projections`, `lifecycle`, and bounded `recovery` fields; they never claim rollback. Any additive `cleanup` diagnostic also exits non-zero, even where known effects are complete: it is not a clean completion and the lock requires inspection before retry.
+
+This control plane is separate from the gateway-backed `mosaic agent` catalog. It is local-only and rejects remote/connector lifecycle mutation, arbitrary command/channel/secret input, and unproven tmux ownership.

docs/fleet/reference/lifecycle-transitions.md

@@ -0,0 +1,14 @@
+# Local Fleet Lifecycle Transitions
+
+FCM-M3-001 uses the roster-v2 `lifecycle.enabled` and `lifecycle.desired_state` fields as the only desired-state authority. Systemd, tmux, generated environment files, and heartbeats are derived or observed state.
+
+| Command                           | Desired-state write | Runtime effect                                                                                                             | Preconditions                                                                                                             |
+| --------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| `fleet apply` / `fleet reconcile` | Never               | Rebuilds projections, then starts only enabled agents desired `running`; stops disabled or desired-`stopped` roster agents | Current generation; private managed paths; valid projections; proven holder ownership; no unmanaged named-socket sessions |
+| `fleet start <name>`              | Never               | One-shot exact `mosaic-agent@<name>.service` start                                                                         | Current generation; exact enabled roster name; proven ownership                                                           |
+| `fleet stop <name>`               | Never               | One-shot exact service stop                                                                                                | Current generation; exact roster name; proven ownership                                                                   |
+| `fleet restart <name>`            | Never               | One-shot exact service restart                                                                                             | Current generation; exact roster name; proven ownership                                                                   |
+
+A stopped roster agent is never started by `apply` or `reconcile`. Direct lifecycle commands are explicit one-shot actions and do not change persisted desired state. Use roster CRUD with the explicit persisted-start option to change that desired state.
+
+All mutations require `--expected-generation <n>` and acquire one private roster-adjacent reconciliation lock before projection or lifecycle effects. Missing or stale generations and concurrent writers fail before effects; the lock is released after success, partial failure, or thrown lifecycle failure. Stale, ownership, unmanaged-session, unsupported-runtime, path, projection, and lifecycle-precondition failures return stable redacted JSON errors and a non-zero exit. No command targets a fuzzy tmux name, arbitrary socket, arbitrary command, channel, secret, or generated file as authority.

docs/fleet/reference/status-and-drift.md

@@ -0,0 +1,13 @@
+# Local Fleet Status and Drift
+
+`mosaic fleet status [name]`, `verify`, and `doctor` are observational roster-v2 commands. They emit one JSON result and do not write projections, change desired state, start services, stop services, restart services, or mutate tmux.
+
+The report distinguishes:
+
+- `missing-session`: an enabled agent desired `running` has no exact roster-named tmux session.
+- `unexpected-session`: a desired-`stopped` agent still has its exact session.
+- `disabled-running`: a disabled roster agent has its exact session.
+- `unmanagedSessions`: sessions on the configured named socket that are neither the exact holder nor an exact roster agent.
+- `holder`: `owned`, `missing`, or `ownership-mismatch` after exact holder, private install identity, and complete global tmux environment validation.
+
+`doctor` and `status` classify rather than adopt, destroy, or repair unmanaged state. `verify` is observational too, but exits non-zero if ownership cannot be proven, unmanaged sessions exist, or drift is present. Reconciliation fails closed under those conditions and never kills or adopts an unmanaged session.

docs/scratchpads/758-fcm-m3-001-local-reconciler.md

@@ -0,0 +1,39 @@
+# FCM-M3-001 — Local roster-owned reconciliation and lifecycle
+
+- **Task / issue:** FCM-M3-001 / #758
+- **Branch / base:** `feat/758-local-reconciler` from `origin/main` `bc5e73629e92c56a80fa6a769ebad17c0177f504`
+- **Base tree:** `1b9ebe4fa1a90734b6f81118e120bae5290cd350`
+- **Scope:** source, isolated fake-adapter tests, and card documentation only. No live fleet/systemd/tmux action.
+
+## Objective
+
+Provide local roster-v2 `apply`/`reconcile` and lifecycle/status contracts. The roster remains desired-state authority; projections and runtime observations are derived state.
+
+## Red-first evidence
+
+The initial focused reconciler test failed because `fleet-reconciler.ts` did not exist. The initial new-worktree test invocation also exposed absent dependencies; `pnpm install --frozen-lockfile --store-dir /home/jarvis/.local/share/pnpm/store` restored local workspace dependencies without changing source.
+
+## Design
+
+- A new `fleet-reconciler.ts` accepts only typed roster-v2 input plus injected command and projection adapters.
+- It targets only exact `mosaic-agent@<roster-name>.service` units and the exact configured tmux socket/session.
+- It classifies unowned/unmanaged state and fails mutation closed rather than adopting or killing it.
+- It validates the private install-derived holder identity and complete expected tmux global environment before mutating lifecycle state.
+- REVIEW-1 remediation: RED review evidence found service-level apply could omit generation and had no mutation lock. Every non-observational command now requires an expected generation; a private exclusive roster-adjacent lock is acquired before effects and released on success or partial failure. Tests cover missing/stale values, concurrent denial, no effects, release, and lock-free observation.
+- REVIEW-2 remediation: lock acquisition now validates private real `MOSAIC_HOME`/`fleet` ancestors, rejects symlink or unsafe leaves, distinguishes `EEXIST` concurrency from other I/O, and binds release to the created inode plus random ownership token. A replacement lock is retained and reported, not unlinked. A crash may leave a stale lock for inspection; no stale-lock break is claimed.
+- REVIEW-3 remediation: a lock cleanup failure now adds bounded `cleanup` diagnostics to a known successful or partial effect result without replacing its projection/lifecycle/recovery truth. Cleanup is not claimed as complete, and the retained lock requires inspection before retry.
+- REVIEW-4 remediation: command JSON with an additive cleanup diagnostic now exits non-zero even where known effects completed; clean effect and observational JSON remain zero-exit.
+- REVIEW-5 remediation: mutating operations acquire the private lock before rereading canonical `roster.yaml`; the fenced reread, not a caller snapshot, supplies generation validation, plan, projection, and lifecycle authority.
+- `apply` starts only enabled agents whose persisted desired state is `running`; stopped/default agents are never started by reconciliation.
+- Observational commands produce JSON classification only. Partial projection or lifecycle effects report explicit recovery without values.
+
+## Boundaries
+
+Excluded: live host actions, remote/SSH/connector lifecycle mutation, migrations, canaries, deployment, gateway changes, arbitrary command/channel/secret inputs, `docs/TASKS.md`, and orchestration ledgers.
+
+## Verification
+
+- Focused reconciler/Commander/CRUD/fleet tests: 4 files / 229 tests passed.
+- Full `@mosaicstack/mosaic` suite: 56 files / 820 tests passed after REVIEW-5 canonical roster fencing remediation.
+- Package and root typecheck/lint, root format check, and `git diff --check`: passed.
+- Isolated launcher and systemd template harnesses passed; they use fixtures only. No live fleet, systemd, tmux, session, remote, connector, or runtime action occurred.