Files

ci/woodpecker/pr/ci Pipeline was successful

Details

feat(fleet): add mosaic fleet regen recovery command

Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-16 21:57:38 -05:00

6.6 KiB

Raw Blame History

Upgrade Safety & Recovery

How Mosaic protects operator-owned configuration under ~/.config/mosaic across framework upgrades, and how to recover if a projection is ever lost.

A framework upgrade runs install.sh in keep-mode (MOSAIC_INSTALL_MODE=keep, MOSAIC_SYNC_ONLY=1) to refresh framework-owned files in place. The incident this hardening addresses: an upgrade that silently overwrites or deletes a file the operator owns — credentials, personas, a roster, or a generated agent env — with no snapshot to fall back to.

Protection is layered. Each layer is independent; a later layer catches what an earlier one misses.

Layer 1 — Manifest-owned sync (prevention)

The single source of truth for ownership is framework-manifest.txt. Both the bash installer and the TypeScript sync path resolve every path against this one file (parity is enforced by test), so they can never drift.

Ownership is allow-list, deny-wins: a path is framework-owned only if a [framework] glob matches and no [operator] carve-out overrides it.
Unknown paths default to operator (fail-safe): a file the manifest never anticipated is treated as operator-owned and is never pruned.
Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that only ever iterates framework globs — operator and unknown paths are structurally unreachable by the prune.

Result: a correct upgrade cannot touch operator config at all.

Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)

Before any mutation, the installer snapshots the operator-owned surface that exists into:

${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/

0700 directories / 0600 files (umask 077, scoped and restored), outside ~/.config/mosaic and outside any repo.
Fail-open: a snapshot failure warns but never aborts the upgrade it protects.
Retention is MOSAIC_BACKUP_RETENTION snapshots (default 5).

After the sync, a verify net compares each snapshot file against its target and restores (with a loud warning) any operator file the upgrade diverged or removed — a divergence means a manifest bug slipped through Layer 1.

Inspect and restore snapshots with the CLI:

mosaic restore --list                 # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run  # preview a specific restore without writing

mosaic restore reports counts and relative paths only — it never emits file contents, so a secret in tools/_lib/credentials.json is never echoed. Restores are confirmation-gated (--yes or MOSAIC_ASSUME_YES) and write each leaf atomically with O_NOFOLLOW (a symlink swapped in after the snapshot fails closed rather than following out of the managed tree).

Layer 3 — Regeneration from roster SSOT (recovery)

Some operator files are derived and do not need a byte-for-byte snapshot to recover — they can be rebuilt from their source of truth. The fleet's per-agent generated env projections are the prime case:

~/.config/mosaic/fleet/agents/<name>.env.generated is a deterministic projection of ~/.config/mosaic/fleet/roster.yaml.
The launcher (start-agent-session.sh, invoked by mosaic-agent@<name>.service) sources that generated projection to establish each agent's identity, runtime, model, and working directory. If it is missing or wrong, the agent cannot launch with its intended identity.

mosaic fleet regen rebuilds those projections from the roster SSOT:

mosaic fleet regen            # dry-run (default): show what would be rebuilt
mosaic fleet regen --json     # same, machine-readable
mosaic fleet regen --write    # rebuild the projections on disk

Dry-run by default. Nothing is written until you pass --write.
Deterministic and idempotent — the projection is a pure function of the roster, so repeated --write runs produce byte-identical files.
Projection-only. It never restarts an agent. Recovery order forbids restart-before-verify; regen has no path to systemd lifecycle at all.
It rebuilds only <name>.env.generated — it never writes, relocates, or deletes the operator-owned .env / .env.local surface.
It validates the roster the same way reconcile does (persona resolution and protected-class tool-policy match), so a hand-edited or corrupt roster is rejected rather than projected, and a --write takes the shared reconcile lock so it cannot race a concurrent reconcile.
Output is paths and counts only — the rendered KEY=value body is never echoed.

regen uses the exact same roster→env mapping as mosaic fleet reconcile, so a recovered projection matches what a normal reconcile would have written.

Recovery runbook — wiped `fleet/agents/*.env.generated`

If an upgrade (or a manual mistake) has left an agent without its generated projection, do not restart the unit first — a launch against a missing projection fails closed, and any stale state must be corrected before restart, not after.

Prefer a snapshot restore if one exists (byte-exact operator state):
```
mosaic restore --list
mosaic restore --from <UTC-timestamp>
```

Otherwise regenerate the derived projections from the roster SSOT:

mosaic fleet regen            # confirm the plan (create vs rebuild per agent)
mosaic fleet regen --write    # rebuild fleet/agents/<name>.env.generated

Verify each unit will resolve the intended runtime/workdir before any restart. The unit sets no EnvironmentFile= — it launches from a minimal environment and start-agent-session.sh sources .env.generated itself, so verify the generated file directly and confirm the launcher path:

# Confirm fleet/agents/<name>.env.generated exists and carries the intended
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
# Confirm the unit launches the session script that reads it.
systemctl --user cat mosaic-agent@<name> | grep ExecStart

Only then restart, one unit at a time:

systemctl --user restart mosaic-agent@<name>

6.6 KiB Raw Blame History