# Upgrade Safety & Recovery

How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
framework upgrades, and how to recover if a projection is ever lost.

A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
this hardening addresses: an upgrade that silently overwrites or deletes a file
the operator owns — credentials, personas, a roster, or a generated agent env —
with no snapshot to fall back to.

Protection is layered. Each layer is independent; a later layer catches what an
earlier one misses.

## Layer 1 — Manifest-owned sync (prevention)

The single source of truth for ownership is
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
Both the bash installer and the TypeScript sync path resolve every path against
this one file (parity is enforced by test), so they can never drift.

- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
  `[framework]` glob matches and no `[operator]` carve-out overrides it.
- **Unknown paths default to operator** (fail-safe): a file the manifest never
  anticipated is treated as operator-owned and is never pruned.
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
  only ever iterates framework globs — operator and unknown paths are
  structurally unreachable by the prune.

Result: a correct upgrade cannot touch operator config at all.

## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)

Before **any** mutation, the installer snapshots the operator-owned surface that
exists into:

```
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
```

- `0700` directories / `0600` files (`umask 077`, scoped and restored),
  outside `~/.config/mosaic` and outside any repo.
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
  protects.
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).

After the sync, a **verify net** compares each snapshot file against its target
and restores (with a loud warning) any operator file the upgrade diverged or
removed — a divergence means a manifest bug slipped through Layer 1.

Inspect and restore snapshots with the CLI:

```bash
mosaic restore --list                 # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run  # preview a specific restore without writing
```

`mosaic restore` reports **counts and relative paths only** — it never emits file
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
closed rather than following out of the managed tree).

## Layer 3 — Regeneration from roster SSOT (recovery)

Some operator files are **derived** and do not need a byte-for-byte snapshot to
recover — they can be rebuilt from their source of truth. The fleet's per-agent
generated env projections are the prime case:

- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
  projection of `~/.config/mosaic/fleet/roster.yaml`.
- The launcher (`start-agent-session.sh`, invoked by
  `mosaic-agent@<name>.service`) sources that generated projection to establish
  each agent's identity, runtime, model, and working directory. If it is missing
  or wrong, the agent cannot launch with its intended identity.

`mosaic fleet regen` rebuilds those projections from the roster SSOT:

```bash
mosaic fleet regen            # dry-run (default): show what would be rebuilt
mosaic fleet regen --json     # same, machine-readable
mosaic fleet regen --write    # rebuild the projections on disk
```

- **Dry-run by default.** Nothing is written until you pass `--write`.
- **Deterministic and idempotent** — the projection is a pure function of the
  roster, so repeated `--write` runs produce byte-identical files.
- **Projection-only. It never restarts an agent.** Recovery order forbids
  restart-before-verify; `regen` has no path to systemd lifecycle at all.
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
  deletes the operator-owned `.env` / `.env.local` surface.
- It **validates the roster the same way `reconcile` does** (persona resolution
  and protected-class tool-policy match), so a hand-edited or corrupt roster is
  rejected rather than projected, and a `--write` takes the shared reconcile
  lock so it cannot race a concurrent reconcile.
- Output is **paths and counts only** — the rendered `KEY=value` body is never
  echoed.

`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
recovered projection matches what a normal reconcile would have written.

## Recovery runbook — wiped `fleet/agents/*.env.generated`

If an upgrade (or a manual mistake) has left an agent without its generated
projection, **do not restart the unit first** — a launch against a missing
projection fails closed, and any stale state must be corrected before restart,
not after.

1. **Prefer a snapshot restore if one exists** (byte-exact operator state):

   ```bash
   mosaic restore --list
   mosaic restore --from <UTC-timestamp>
   ```

2. **Otherwise regenerate the derived projections from the roster SSOT:**

   ```bash
   mosaic fleet regen            # confirm the plan (create vs rebuild per agent)
   mosaic fleet regen --write    # rebuild fleet/agents/<name>.env.generated
   ```

3. **Verify each unit will resolve the intended runtime/workdir _before_ any
   restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
   environment and `start-agent-session.sh` sources `.env.generated` itself, so
   verify the generated file directly and confirm the launcher path:

   ```bash
   # Confirm fleet/agents/<name>.env.generated exists and carries the intended
   # MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
   test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
   # Confirm the unit launches the session script that reads it.
   systemctl --user cat mosaic-agent@<name> | grep ExecStart
   ```

4. **Only then restart, one unit at a time:**

   ```bash
   systemctl --user restart mosaic-agent@<name>
   ```

## See also

- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)