feat(fleet): add mosaic fleet regen recovery command
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Add `mosaic fleet regen`, a projection-only recovery command that rebuilds each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an upgrade or partial write leaves the generated projections stale or missing. - Dry-run by default; `--write` applies; `--json` for machine output. Reuses the merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv + the generated-env boundary) rather than reimplementing fleet logic. - Structurally NEVER issues a lifecycle/restart call — regen recovers config only; a recordingRunner gate proves no runner invocation ever occurs. - Serializes against agent CRUD and reconcile via BOTH fleet locks (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired mutation-then-reconcile and released in reverse; both are non-blocking `wx` locks that throw on contention, so no deadlock is possible. - Hardens the shared managed-lock helper: ownership-proving tokened lock reused for both locks with per-lock fault labels; init-failure cleanup no longer strands a just-created lock (dev/ino guard, with a persisted-token fallback when the post-create stat itself fails); acquire-unwind surfaces a lock cleanup fault instead of dropping it. - Resolves personas the SAME way reconcile does by forwarding configured rolesDir/overrideDir, so a custom-persona-root deployment cannot have reconcile accept a roster that regen rejects. - Report/output is secrev-safe: paths and counts only, never projected values. - Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note. Tests are TDD red-first with co-located specs (regen spec: 26 tests covering dry-run/write dispositions, the never-restarts gate, all lock regressions, and persona-root wiring). A residual check-then-unlink TOCTOU in the lock release remains (byte-identical to the merged reconcile lock; unreachable within the `wx` writer protocol); its true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a separate follow-up. Part of #791 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -98,6 +98,39 @@ Expected results:
|
||||
that means the unit ran, not that an agent pane is live. Treat tmux
|
||||
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
||||
|
||||
## Recovery — rebuild generated env projections
|
||||
|
||||
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
|
||||
deterministic projection of `roster.yaml` (the SSOT) that the launcher
|
||||
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
|
||||
wipes or diverges those projections, rebuild them from the roster with
|
||||
`mosaic fleet regen` — do NOT restart the affected unit first.
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
|
||||
mosaic fleet regen --json # same plan, machine-readable
|
||||
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
|
||||
```
|
||||
|
||||
`regen` is projection-only and **never restarts an agent** — it has no path to
|
||||
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
|
||||
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
|
||||
only (never the projected `KEY=value` body). After `--write`, verify each unit
|
||||
resolves the intended values before restarting one unit at a time. The unit sets
|
||||
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
|
||||
so verify the generated file directly and the launcher path, not a nonexistent
|
||||
`EnvironmentFile` property:
|
||||
|
||||
```bash
|
||||
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||
systemctl --user restart mosaic-agent@<name>
|
||||
```
|
||||
|
||||
Full recovery runbook and the three-layer #791 protection model (manifest
|
||||
ownership → pre-update snapshot/restore → regen): see
|
||||
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
|
||||
|
||||
## Release Preflight
|
||||
|
||||
Run this checklist before cutting or dogfooding a fleet release:
|
||||
|
||||
Reference in New Issue
Block a user