feat(fleet): add mosaic fleet regen recovery command
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-16 21:57:38 -05:00
parent 31607a4af6
commit 1eb37f6fd0
10 changed files with 1994 additions and 37 deletions

View File

@@ -98,6 +98,39 @@ Expected results:
that means the unit ran, not that an agent pane is live. Treat tmux
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
## Recovery — rebuild generated env projections
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
deterministic projection of `roster.yaml` (the SSOT) that the launcher
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
wipes or diverges those projections, rebuild them from the roster with
`mosaic fleet regen` — do NOT restart the affected unit first.
```bash
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json # same plan, machine-readable
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
```
`regen` is projection-only and **never restarts an agent** — it has no path to
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
only (never the projected `KEY=value` body). After `--write`, verify each unit
resolves the intended values before restarting one unit at a time. The unit sets
no `EnvironmentFile=``start-agent-session.sh` sources `.env.generated` itself —
so verify the generated file directly and the launcher path, not a nonexistent
`EnvironmentFile` property:
```bash
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>
```
Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
## Release Preflight
Run this checklist before cutting or dogfooding a fleet release: