stack/docs/guides/upgrade-safety-and-recovery.md

Upgrade Safety & Recovery

How Mosaic protects operator-owned configuration under ~/.config/mosaic across framework upgrades, and how to recover if a projection is ever lost.

A framework upgrade runs install.sh in keep-mode (MOSAIC_INSTALL_MODE=keep, MOSAIC_SYNC_ONLY=1) to refresh framework-owned files in place. The incident this hardening addresses: an upgrade that silently overwrites or deletes a file the operator owns — credentials, personas, a roster, or a generated agent env — with no snapshot to fall back to.

Protection is layered. Each layer is independent; a later layer catches what an earlier one misses.

Layer 1 — Manifest-owned sync (prevention)

The single source of truth for ownership is framework-manifest.txt. Both the bash installer and the TypeScript sync path resolve every path against this one file (parity is enforced by test), so they can never drift.

• Ownership is allow-list, deny-wins: a path is framework-owned only if a [framework] glob matches and no [operator] carve-out overrides it.
• Unknown paths default to operator (fail-safe): a file the manifest never anticipated is treated as operator-owned and is never pruned.
• Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that only ever iterates framework globs — operator and unknown paths are structurally unreachable by the prune.

Result: a correct upgrade cannot touch operator config at all.

Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)

Before any mutation, the installer snapshots the operator-owned surface that exists into:

${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/

• 0700 directories / 0600 files (umask 077, scoped and restored), outside ~/.config/mosaic and outside any repo.
• Fail-open: a snapshot failure warns but never aborts the upgrade it protects.
• Retention is MOSAIC_BACKUP_RETENTION snapshots (default 5).

After the sync, a verify net compares each snapshot file against its target and restores (with a loud warning) any operator file the upgrade diverged or removed — a divergence means a manifest bug slipped through Layer 1.

Inspect and restore snapshots with the CLI:

mosaic restore --list                 # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run  # preview a specific restore without writing

mosaic restore reports counts and relative paths only — it never emits file contents, so a secret in tools/_lib/credentials.json is never echoed. Restores are confirmation-gated (--yes or MOSAIC_ASSUME_YES) and write each leaf atomically with O_NOFOLLOW (a symlink swapped in after the snapshot fails closed rather than following out of the managed tree).

Layer 3 — Regeneration from roster SSOT (recovery)

Some operator files are derived and do not need a byte-for-byte snapshot to recover — they can be rebuilt from their source of truth. The fleet's per-agent generated env projections are the prime case:

• ~/.config/mosaic/fleet/agents/<name>.env.generated is a deterministic projection of ~/.config/mosaic/fleet/roster.yaml.
• The launcher (start-agent-session.sh, invoked by mosaic-agent@<name>.service) sources that generated projection to establish each agent's identity, runtime, model, and working directory. If it is missing or wrong, the agent cannot launch with its intended identity.

mosaic fleet regen rebuilds those projections from the roster SSOT:

mosaic fleet regen            # dry-run (default): show what would be rebuilt
mosaic fleet regen --json     # same, machine-readable
mosaic fleet regen --write    # rebuild the projections on disk

• Dry-run by default. Nothing is written until you pass --write.
• Deterministic and idempotent — the projection is a pure function of the roster, so repeated --write runs produce byte-identical files.
• Projection-only. It never restarts an agent. Recovery order forbids restart-before-verify; regen has no path to systemd lifecycle at all.
• It rebuilds only <name>.env.generated — it never writes, relocates, or deletes the operator-owned .env / .env.local surface.
• It validates the roster the same way reconcile does (persona resolution and protected-class tool-policy match), so a hand-edited or corrupt roster is rejected rather than projected, and a --write takes the shared reconcile lock so it cannot race a concurrent reconcile.
• Output is paths and counts only — the rendered KEY=value body is never echoed.

regen uses the exact same roster→env mapping as mosaic fleet reconcile, so a recovered projection matches what a normal reconcile would have written.

Recovery runbook — wiped fleet/agents/*.env.generated

If an upgrade (or a manual mistake) has left an agent without its generated projection, do not restart the unit first — a launch against a missing projection fails closed, and any stale state must be corrected before restart, not after.

1. Prefer a snapshot restore if one exists (byte-exact operator state):

   mosaic restore --list
   mosaic restore --from <UTC-timestamp>
   

2. Otherwise regenerate the derived projections from the roster SSOT:

   mosaic fleet regen            # confirm the plan (create vs rebuild per agent)
   mosaic fleet regen --write    # rebuild fleet/agents/<name>.env.generated
   

3. Verify each unit will resolve the intended runtime/workdir before any restart. The unit sets no EnvironmentFile= — it launches from a minimal environment and start-agent-session.sh sources .env.generated itself, so verify the generated file directly and confirm the launcher path:

   # Confirm fleet/agents/<name>.env.generated exists and carries the intended
   # MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
   test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
   # Confirm the unit launches the session script that reads it.
   systemctl --user cat mosaic-agent@<name> | grep ExecStart
   

4. Only then restart, one unit at a time:

   systemctl --user restart mosaic-agent@<name>
   

See also

• Design: docs/design/791-upgrade-config-protection.md
• Fleet operations: docs/guides/fleet-local-canary.md
• Ownership SSOT: packages/mosaic/framework/framework-manifest.txt