148 lines
6.6 KiB
Markdown
148 lines
6.6 KiB
Markdown
# Upgrade Safety & Recovery
|
|
|
|
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
|
|
framework upgrades, and how to recover if a projection is ever lost.
|
|
|
|
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
|
|
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
|
|
this hardening addresses: an upgrade that silently overwrites or deletes a file
|
|
the operator owns — credentials, personas, a roster, or a generated agent env —
|
|
with no snapshot to fall back to.
|
|
|
|
Protection is layered. Each layer is independent; a later layer catches what an
|
|
earlier one misses.
|
|
|
|
## Layer 1 — Manifest-owned sync (prevention)
|
|
|
|
The single source of truth for ownership is
|
|
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
|
|
Both the bash installer and the TypeScript sync path resolve every path against
|
|
this one file (parity is enforced by test), so they can never drift.
|
|
|
|
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
|
|
`[framework]` glob matches and no `[operator]` carve-out overrides it.
|
|
- **Unknown paths default to operator** (fail-safe): a file the manifest never
|
|
anticipated is treated as operator-owned and is never pruned.
|
|
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
|
|
only ever iterates framework globs — operator and unknown paths are
|
|
structurally unreachable by the prune.
|
|
|
|
Result: a correct upgrade cannot touch operator config at all.
|
|
|
|
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
|
|
|
|
Before **any** mutation, the installer snapshots the operator-owned surface that
|
|
exists into:
|
|
|
|
```
|
|
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
|
|
```
|
|
|
|
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
|
|
outside `~/.config/mosaic` and outside any repo.
|
|
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
|
|
protects.
|
|
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
|
|
|
|
After the sync, a **verify net** compares each snapshot file against its target
|
|
and restores (with a loud warning) any operator file the upgrade diverged or
|
|
removed — a divergence means a manifest bug slipped through Layer 1.
|
|
|
|
Inspect and restore snapshots with the CLI:
|
|
|
|
```bash
|
|
mosaic restore --list # dry-run: enumerate snapshots by timestamp
|
|
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
|
|
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
|
|
```
|
|
|
|
`mosaic restore` reports **counts and relative paths only** — it never emits file
|
|
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
|
|
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
|
|
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
|
|
closed rather than following out of the managed tree).
|
|
|
|
## Layer 3 — Regeneration from roster SSOT (recovery)
|
|
|
|
Some operator files are **derived** and do not need a byte-for-byte snapshot to
|
|
recover — they can be rebuilt from their source of truth. The fleet's per-agent
|
|
generated env projections are the prime case:
|
|
|
|
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
|
|
projection of `~/.config/mosaic/fleet/roster.yaml`.
|
|
- The launcher (`start-agent-session.sh`, invoked by
|
|
`mosaic-agent@<name>.service`) sources that generated projection to establish
|
|
each agent's identity, runtime, model, and working directory. If it is missing
|
|
or wrong, the agent cannot launch with its intended identity.
|
|
|
|
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
|
|
|
|
```bash
|
|
mosaic fleet regen # dry-run (default): show what would be rebuilt
|
|
mosaic fleet regen --json # same, machine-readable
|
|
mosaic fleet regen --write # rebuild the projections on disk
|
|
```
|
|
|
|
- **Dry-run by default.** Nothing is written until you pass `--write`.
|
|
- **Deterministic and idempotent** — the projection is a pure function of the
|
|
roster, so repeated `--write` runs produce byte-identical files.
|
|
- **Projection-only. It never restarts an agent.** Recovery order forbids
|
|
restart-before-verify; `regen` has no path to systemd lifecycle at all.
|
|
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
|
|
deletes the operator-owned `.env` / `.env.local` surface.
|
|
- It **validates the roster the same way `reconcile` does** (persona resolution
|
|
and protected-class tool-policy match), so a hand-edited or corrupt roster is
|
|
rejected rather than projected, and a `--write` takes the shared reconcile
|
|
lock so it cannot race a concurrent reconcile.
|
|
- Output is **paths and counts only** — the rendered `KEY=value` body is never
|
|
echoed.
|
|
|
|
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
|
|
recovered projection matches what a normal reconcile would have written.
|
|
|
|
## Recovery runbook — wiped `fleet/agents/*.env.generated`
|
|
|
|
If an upgrade (or a manual mistake) has left an agent without its generated
|
|
projection, **do not restart the unit first** — a launch against a missing
|
|
projection fails closed, and any stale state must be corrected before restart,
|
|
not after.
|
|
|
|
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
|
|
|
|
```bash
|
|
mosaic restore --list
|
|
mosaic restore --from <UTC-timestamp>
|
|
```
|
|
|
|
2. **Otherwise regenerate the derived projections from the roster SSOT:**
|
|
|
|
```bash
|
|
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
|
|
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
|
|
```
|
|
|
|
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
|
|
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
|
|
environment and `start-agent-session.sh` sources `.env.generated` itself, so
|
|
verify the generated file directly and confirm the launcher path:
|
|
|
|
```bash
|
|
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
|
|
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
|
|
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
|
# Confirm the unit launches the session script that reads it.
|
|
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
|
```
|
|
|
|
4. **Only then restart, one unit at a time:**
|
|
|
|
```bash
|
|
systemctl --user restart mosaic-agent@<name>
|
|
```
|
|
|
|
## See also
|
|
|
|
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
|
|
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
|
|
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)
|