feat(fleet): reconcile local roster state (#785)

docs/fleet/how-to/start-stop-restart.md

@@ -0,0 +1,23 @@
+# Safely Reconcile and Control a Local Fleet Agent
+
+Use the canonical local roster-v2 command surface:
+
+```sh
+mosaic fleet apply --expected-generation <n> --dry-run
+mosaic fleet apply --expected-generation <n>
+mosaic fleet reconcile --expected-generation <n>
+mosaic fleet start <name> --expected-generation <n>
+mosaic fleet stop <name> --expected-generation <n>
+mosaic fleet restart <name> --expected-generation <n>
+mosaic fleet status [name]
+mosaic fleet verify
+mosaic fleet doctor
+```
+
+Start with `--dry-run`. It validates roster semantics, deterministic projections, private managed paths, exact holder ownership, and named-socket state without changing files or lifecycle state. `apply` and `reconcile` rebuild derived projections and enforce only persisted roster state: enabled `running` agents may start, while stopped or disabled agents are not started.
+
+`start`, `stop`, and `restart` are explicit one-shot exact-service actions. They do not persist a lifecycle change. Roster CRUD is the only way to change persisted desired state.
+
+Every command prints JSON. Observation commands report drift without mutation; `verify` exits non-zero on ownership mismatch, unmanaged sessions, or drift. A failed apply that wrote some derived projections reports `projections: "incomplete"` with bounded recovery to regenerate from the roster. A lifecycle failure after projections reports incomplete lifecycle work; it is never represented as a rollback or no-op.
+
+These commands are local only. Remote/SSH/connector entries are inventory/validation-only. Commands do not accept arbitrary runtime commands, channels, secrets, generated-file desired state, or arbitrary tmux sockets.

docs/fleet/operations/reconcile-and-recover.md

@@ -0,0 +1,26 @@
+# Reconcile and Recover a Local Fleet
+
+## Safe sequence
+
+1. Read `mosaic fleet doctor` and `mosaic fleet status`.
+2. Run `mosaic fleet apply --expected-generation <n> --dry-run`.
+3. Resolve stale generation, ownership mismatch, unsafe path, projection validation, or unmanaged-session findings before applying.
+4. Run `mosaic fleet apply --expected-generation <n>` only after the plan is understood.
+
+The reconciler uses the exact roster tmux socket, exact holder session, private installation holder identity, and the complete expected global environment. For mutations it acquires its exclusive lock before rereading the canonical roster and fencing its generation; only that under-lock roster drives validation, planning, projections, and lifecycle effects. Before effects, its exclusive lock proves real private `MOSAIC_HOME` and `fleet` ancestors, uses a private `0600` lock leaf, and binds cleanup to the created file identity and ownership token. A fake holder, contaminated global environment, missing identity, unsafe lock path, or unmanaged session fails closed. It does not adopt, kill, or rename any unproven session. A crash can leave a stale lock for explicit operator inspection; reconciliation deliberately does not guess ownership or remove it.
+
+## Partial results
+
+The roster is never changed by reconciliation. If derived projection application partially fails, JSON reports:
+
+```json
+{
+  "applied": false,
+  "authoritativeRoster": "unchanged",
+  "projections": "incomplete",
+  "lifecycle": "not-applied",
+  "recovery": { "code": "projection-apply-failed", "action": "regenerate-projections-from-roster" }
+}
+```
+
+If projections completed but lifecycle work failed, JSON reports `projections: "complete"`, `lifecycle: "incomplete"`, and the bounded action `rerun-after-inspecting-owned-resources`. If lock cleanup cannot be proven after an effect result, it adds `cleanup: { "code": "lock-cleanup-failed", "action": "inspect-lock-before-retry" }` without changing the known projection, lifecycle, or primary recovery truth. Inspect the retained lock before retrying; no rollback, release, or stale-lock removal is implied. Results do not include environment values, secrets, or privileged command content.

docs/fleet/reference/cli.md

@@ -0,0 +1,20 @@
+# Fleet Control-Plane CLI
+
+The local roster-v2 control plane is `mosaic fleet`.
+
+```text
+mosaic fleet apply --expected-generation <n> [--dry-run]
+mosaic fleet reconcile --expected-generation <n> [--dry-run]
+mosaic fleet start [name] --expected-generation <n> [--dry-run]
+mosaic fleet stop [name] --expected-generation <n> [--dry-run]
+mosaic fleet restart [name] --expected-generation <n> [--dry-run]
+mosaic fleet status [name]
+mosaic fleet verify
+mosaic fleet doctor
+```
+
+`apply` and `reconcile` use roster desired state. `start`, `stop`, and `restart` are exact local one-shot lifecycle effects and never persist a desired-state edit. `status`, `verify`, and `doctor` are observational.
+
+Commands emit one JSON object. Handled precondition errors emit `{ "error": { "code": "..." } }` and exit non-zero. Partial derived/lifecycle effects use explicit `authoritativeRoster`, `projections`, `lifecycle`, and bounded `recovery` fields; they never claim rollback. Any additive `cleanup` diagnostic also exits non-zero, even where known effects are complete: it is not a clean completion and the lock requires inspection before retry.
+
+This control plane is separate from the gateway-backed `mosaic agent` catalog. It is local-only and rejects remote/connector lifecycle mutation, arbitrary command/channel/secret input, and unproven tmux ownership.

docs/fleet/reference/lifecycle-transitions.md

@@ -0,0 +1,14 @@
+# Local Fleet Lifecycle Transitions
+
+FCM-M3-001 uses the roster-v2 `lifecycle.enabled` and `lifecycle.desired_state` fields as the only desired-state authority. Systemd, tmux, generated environment files, and heartbeats are derived or observed state.
+
+| Command                           | Desired-state write | Runtime effect                                                                                                             | Preconditions                                                                                                             |
+| --------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| `fleet apply` / `fleet reconcile` | Never               | Rebuilds projections, then starts only enabled agents desired `running`; stops disabled or desired-`stopped` roster agents | Current generation; private managed paths; valid projections; proven holder ownership; no unmanaged named-socket sessions |
+| `fleet start <name>`              | Never               | One-shot exact `mosaic-agent@<name>.service` start                                                                         | Current generation; exact enabled roster name; proven ownership                                                           |
+| `fleet stop <name>`               | Never               | One-shot exact service stop                                                                                                | Current generation; exact roster name; proven ownership                                                                   |
+| `fleet restart <name>`            | Never               | One-shot exact service restart                                                                                             | Current generation; exact roster name; proven ownership                                                                   |
+
+A stopped roster agent is never started by `apply` or `reconcile`. Direct lifecycle commands are explicit one-shot actions and do not change persisted desired state. Use roster CRUD with the explicit persisted-start option to change that desired state.
+
+All mutations require `--expected-generation <n>` and acquire one private roster-adjacent reconciliation lock before projection or lifecycle effects. Missing or stale generations and concurrent writers fail before effects; the lock is released after success, partial failure, or thrown lifecycle failure. Stale, ownership, unmanaged-session, unsupported-runtime, path, projection, and lifecycle-precondition failures return stable redacted JSON errors and a non-zero exit. No command targets a fuzzy tmux name, arbitrary socket, arbitrary command, channel, secret, or generated file as authority.

docs/fleet/reference/status-and-drift.md

@@ -0,0 +1,13 @@
+# Local Fleet Status and Drift
+
+`mosaic fleet status [name]`, `verify`, and `doctor` are observational roster-v2 commands. They emit one JSON result and do not write projections, change desired state, start services, stop services, restart services, or mutate tmux.
+
+The report distinguishes:
+
+- `missing-session`: an enabled agent desired `running` has no exact roster-named tmux session.
+- `unexpected-session`: a desired-`stopped` agent still has its exact session.
+- `disabled-running`: a disabled roster agent has its exact session.
+- `unmanagedSessions`: sessions on the configured named socket that are neither the exact holder nor an exact roster agent.
+- `holder`: `owned`, `missing`, or `ownership-mismatch` after exact holder, private install identity, and complete global tmux environment validation.
+
+`doctor` and `status` classify rather than adopt, destroy, or repair unmanaged state. `verify` is observational too, but exits non-zero if ownership cannot be proven, unmanaged sessions exist, or drift is present. Reconciliation fails closed under those conditions and never kills or adopts an unmanaged session.