This commit was merged in pull request #813.
This commit is contained in:
@@ -98,6 +98,39 @@ Expected results:
|
||||
that means the unit ran, not that an agent pane is live. Treat tmux
|
||||
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
||||
|
||||
## Recovery — rebuild generated env projections
|
||||
|
||||
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
|
||||
deterministic projection of `roster.yaml` (the SSOT) that the launcher
|
||||
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
|
||||
wipes or diverges those projections, rebuild them from the roster with
|
||||
`mosaic fleet regen` — do NOT restart the affected unit first.
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
|
||||
mosaic fleet regen --json # same plan, machine-readable
|
||||
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
|
||||
```
|
||||
|
||||
`regen` is projection-only and **never restarts an agent** — it has no path to
|
||||
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
|
||||
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
|
||||
only (never the projected `KEY=value` body). After `--write`, verify each unit
|
||||
resolves the intended values before restarting one unit at a time. The unit sets
|
||||
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
|
||||
so verify the generated file directly and the launcher path, not a nonexistent
|
||||
`EnvironmentFile` property:
|
||||
|
||||
```bash
|
||||
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||
systemctl --user restart mosaic-agent@<name>
|
||||
```
|
||||
|
||||
Full recovery runbook and the three-layer #791 protection model (manifest
|
||||
ownership → pre-update snapshot/restore → regen): see
|
||||
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
|
||||
|
||||
## Release Preflight
|
||||
|
||||
Run this checklist before cutting or dogfooding a fleet release:
|
||||
|
||||
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Upgrade Safety & Recovery
|
||||
|
||||
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
|
||||
framework upgrades, and how to recover if a projection is ever lost.
|
||||
|
||||
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
|
||||
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
|
||||
this hardening addresses: an upgrade that silently overwrites or deletes a file
|
||||
the operator owns — credentials, personas, a roster, or a generated agent env —
|
||||
with no snapshot to fall back to.
|
||||
|
||||
Protection is layered. Each layer is independent; a later layer catches what an
|
||||
earlier one misses.
|
||||
|
||||
## Layer 1 — Manifest-owned sync (prevention)
|
||||
|
||||
The single source of truth for ownership is
|
||||
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
|
||||
Both the bash installer and the TypeScript sync path resolve every path against
|
||||
this one file (parity is enforced by test), so they can never drift.
|
||||
|
||||
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
|
||||
`[framework]` glob matches and no `[operator]` carve-out overrides it.
|
||||
- **Unknown paths default to operator** (fail-safe): a file the manifest never
|
||||
anticipated is treated as operator-owned and is never pruned.
|
||||
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
|
||||
only ever iterates framework globs — operator and unknown paths are
|
||||
structurally unreachable by the prune.
|
||||
|
||||
Result: a correct upgrade cannot touch operator config at all.
|
||||
|
||||
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
|
||||
|
||||
Before **any** mutation, the installer snapshots the operator-owned surface that
|
||||
exists into:
|
||||
|
||||
```
|
||||
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
|
||||
```
|
||||
|
||||
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
|
||||
outside `~/.config/mosaic` and outside any repo.
|
||||
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
|
||||
protects.
|
||||
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
|
||||
|
||||
After the sync, a **verify net** compares each snapshot file against its target
|
||||
and restores (with a loud warning) any operator file the upgrade diverged or
|
||||
removed — a divergence means a manifest bug slipped through Layer 1.
|
||||
|
||||
Inspect and restore snapshots with the CLI:
|
||||
|
||||
```bash
|
||||
mosaic restore --list # dry-run: enumerate snapshots by timestamp
|
||||
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
|
||||
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
|
||||
```
|
||||
|
||||
`mosaic restore` reports **counts and relative paths only** — it never emits file
|
||||
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
|
||||
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
|
||||
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
|
||||
closed rather than following out of the managed tree).
|
||||
|
||||
## Layer 3 — Regeneration from roster SSOT (recovery)
|
||||
|
||||
Some operator files are **derived** and do not need a byte-for-byte snapshot to
|
||||
recover — they can be rebuilt from their source of truth. The fleet's per-agent
|
||||
generated env projections are the prime case:
|
||||
|
||||
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
|
||||
projection of `~/.config/mosaic/fleet/roster.yaml`.
|
||||
- The launcher (`start-agent-session.sh`, invoked by
|
||||
`mosaic-agent@<name>.service`) sources that generated projection to establish
|
||||
each agent's identity, runtime, model, and working directory. If it is missing
|
||||
or wrong, the agent cannot launch with its intended identity.
|
||||
|
||||
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # dry-run (default): show what would be rebuilt
|
||||
mosaic fleet regen --json # same, machine-readable
|
||||
mosaic fleet regen --write # rebuild the projections on disk
|
||||
```
|
||||
|
||||
- **Dry-run by default.** Nothing is written until you pass `--write`.
|
||||
- **Deterministic and idempotent** — the projection is a pure function of the
|
||||
roster, so repeated `--write` runs produce byte-identical files.
|
||||
- **Projection-only. It never restarts an agent.** Recovery order forbids
|
||||
restart-before-verify; `regen` has no path to systemd lifecycle at all.
|
||||
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
|
||||
deletes the operator-owned `.env` / `.env.local` surface.
|
||||
- It **validates the roster the same way `reconcile` does** (persona resolution
|
||||
and protected-class tool-policy match), so a hand-edited or corrupt roster is
|
||||
rejected rather than projected, and a `--write` takes the shared reconcile
|
||||
lock so it cannot race a concurrent reconcile.
|
||||
- Output is **paths and counts only** — the rendered `KEY=value` body is never
|
||||
echoed.
|
||||
|
||||
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
|
||||
recovered projection matches what a normal reconcile would have written.
|
||||
|
||||
## Recovery runbook — wiped `fleet/agents/*.env.generated`
|
||||
|
||||
If an upgrade (or a manual mistake) has left an agent without its generated
|
||||
projection, **do not restart the unit first** — a launch against a missing
|
||||
projection fails closed, and any stale state must be corrected before restart,
|
||||
not after.
|
||||
|
||||
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
|
||||
|
||||
```bash
|
||||
mosaic restore --list
|
||||
mosaic restore --from <UTC-timestamp>
|
||||
```
|
||||
|
||||
2. **Otherwise regenerate the derived projections from the roster SSOT:**
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
|
||||
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
|
||||
```
|
||||
|
||||
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
|
||||
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
|
||||
environment and `start-agent-session.sh` sources `.env.generated` itself, so
|
||||
verify the generated file directly and confirm the launcher path:
|
||||
|
||||
```bash
|
||||
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
|
||||
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
|
||||
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||
# Confirm the unit launches the session script that reads it.
|
||||
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||
```
|
||||
|
||||
4. **Only then restart, one unit at a time:**
|
||||
|
||||
```bash
|
||||
systemctl --user restart mosaic-agent@<name>
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
|
||||
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
|
||||
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)
|
||||
Reference in New Issue
Block a user