diff --git a/docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md b/docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md new file mode 100644 index 00000000..832979e3 --- /dev/null +++ b/docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md @@ -0,0 +1,248 @@ +# Upgrade safety and recovery + +> **Supported route:** an already installed `mosaic` CLI using the local PGlite +> configuration. This is a filesystem and CLI runbook; it does not activate a +> service or connect to a database. + +This page covers the supported upgrade, rollback, health, and recovery checks for +Mosaic framework configuration under `MOSAIC_HOME`. It deliberately does not +turn the repository's deployment or PostgreSQL material into an operative +procedure. + +## Support boundary + +Use this runbook only when all of the following are true: + +- `mosaic` resolves to the installed Mosaic CLI (`command -v mosaic`). +- The active application configuration is the local tier: `tier: "local"`, + `storage.type: "pglite"`, and `queue.type: "local"`. +- `DATABASE_URL` is unset. An inherited PostgreSQL DSN is outside this route and + must be removed before continuing. +- `MOSAIC_STORAGE_TIER` is unset or `local`; standalone and federated overrides + are outside this route. +- No Gateway, Web, Compose, or other service activation is required. + +The following routes are **held/non-operative** in this guide: + +| Route or operation | Status | +| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| PostgreSQL-backed `standalone` storage | Held; do not activate or probe it here. | +| Federated storage and peers | Held; do not activate or probe it here. | +| Bare-metal deployment | Held; no service installation or lifecycle action is authorized. | +| Gateway/Web activation or HTTP health checks | Held; `/health`, `/health/ready`, and `mosaic gateway ...` are not evidence for this route. | +| Compose startup | Held; do not start a Compose profile or the full stack. | +| Migration runners and tier migration | Held; do not run `mosaic-db-migrator`, `pnpm --filter @mosaicstack/db db:migrate`, `mosaic storage migrate`, or `mosaic storage migrate-tier`. | + +A command being present in the CLI does not make a held route operative. + +## Storage and path terminology + +Keep these locations separate: + +- **`MOSAIC_HOME`** — the framework/operator configuration directory. It + defaults to `~/.config/mosaic` and can be overridden with `MOSAIC_HOME`. +- **PGlite data** — the local, in-process database. The checked-in local config + uses `.mosaic/storage-pglite` as its storage `dataDir`; `.mosaic/queue` is the + local queue directory. These are project data, not framework configuration. +- **Durable upgrade snapshots** — operator-file snapshots stored under + `${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/`. They are outside + `MOSAIC_HOME` and do not contain a PostgreSQL dump. + +The configuration tier is named `local`; the storage CLI reports the backend as +`pglite`. PGlite is not PostgreSQL and does not require a PostgreSQL server. +Local PGlite schema setup is adapter-owned; this page does not authorize a +separate migration runner. + +## Pre-upgrade health gate + +Run these checks from a shell that has no inherited database DSN: + +```bash +command -v mosaic +mosaic --version +mosaic config path +test -d "$(mosaic config path)" + +env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show +env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status +mosaic restore --list +``` + +For the supported route, the storage checks should report the `pglite` backend +and say that no network check is needed. `mosaic storage status` reports +`PGLITE_DATA_DIR` when that variable is set; otherwise it reports the CLI's +`:memory:` fallback. That output is an inspection of the CLI environment, not a +claim that PGlite contents are healthy or durable. + +If `DATABASE_URL` is set, stop. Do not point it at a local PostgreSQL instance +to make the check pass. If the active configuration is not local/PGlite, stop; +no operative route is documented here. + +The supported health gate is intentionally limited to CLI resolution, framework +configuration path, local storage selection, and available upgrade snapshots. +It does not prove Gateway/Web readiness, provider connectivity, queue health, or +PGlite data integrity. + +## Safe upgrade procedure + +1. **Record the baseline.** Save the output of `mosaic --version`, + `mosaic config path`, and `mosaic restore --list`. Do not copy secrets into a + ticket or report. +2. **Check for updates without installing them:** + + ```bash + mosaic update --check + ``` + + Exit status `0` means no update was reported; status `2` means an update is + available. Other failures are not a successful health result. The check is a + package-registry check and does not start Mosaic services or access storage. + +3. **Run the installed-CLI upgrade when approved:** + + ```bash + mosaic update + ``` + + The command updates the installed `@mosaicstack/*` packages through npm and, + when the framework package changed or framework drift is detected, re-seeds + framework files in keep mode. Leave the default re-seed enabled. Do not use + `--relaunch` in this runbook: that option can restart durable fleet agents + and is outside the supported no-activation route. `--no-reseed` is also not a + normal upgrade path because it intentionally leaves framework files stale. + +4. **Repeat the health gate.** Confirm the CLI version, the same + `MOSAIC_HOME`, the local/PGlite storage selection, and the snapshot listing. + A successful framework upgrade must not be used as evidence that a held + Gateway, Web, PostgreSQL, or federated route is ready. + +Do not replace this procedure with a direct edit of `~/.config/mosaic`, a +repository `tools/install.sh` invocation, a Compose startup, or a migration +command. The supported operator entry point for this page is the installed +`mosaic update` command. + +## What an upgrade protects + +For an existing keep-mode installation, the framework installer takes two +separate snapshots before the file sync: + +1. An ephemeral whole-directory snapshot under `${TMPDIR:-/tmp}/` is used to + restore the previous framework directory if the sync is interrupted or + fails. If that restore cannot complete, the installer prints the retained + temporary snapshot path for manual recovery. +2. A durable snapshot captures the existing operator-owned files before the + upgrade at: + + ```text + ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-/ + ``` + + Snapshot directories are mode `0700`; captured files are mode `0600`. + Retention is five snapshots by default and can be changed with + `MOSAIC_BACKUP_RETENTION`. A durable snapshot failure is a warning and does + not replace the manifest and crash-rollback protections. + +After the keep-mode sync, the installer compares each captured operator file +with its target. If a file was changed or removed unexpectedly, it restores the +snapshot copy and emits a warning. A symlinked parent is not followed; that +case is left for manual recovery from the snapshot path. + +Keep mode is manifest-driven and fail-safe for operator paths, but it does not +mean every file under `MOSAIC_HOME` is user-owned. The framework contract files +`CONSTITUTION.md`, `AGENTS.md`, and `STANDARDS.md` are refreshed by upgrades; +keep local policy in the supported local overlays rather than editing those +contract files as rollback data. A durable snapshot is for operator-owned +configuration, not for the installed npm package, framework-owned contracts, or +PGlite data. + +## Rollback and recovery + +### If the upgrade fails or is interrupted + +The framework sync attempts an automatic rollback from its ephemeral snapshot. +Do not delete `MOSAIC_HOME`, the PGlite data directory, or the temporary snapshot +named in the error. Preserve the command output, then run the health gate. + +If the installed framework reports that the automatic restore did not complete, +use the durable snapshot procedure below. The durable snapshot is the recovery +pointer that survives a successful upgrade and the removal of the temporary +snapshot. + +### Restore operator configuration from a durable snapshot + +`mosaic restore` is confirmation-gated and reports counts and relative paths, +not file contents. First list snapshots, then preview the selected timestamp: + +```bash +mosaic restore --list +mosaic restore --from --dry-run +``` + +If the preview is correct, run the interactive restore: + +```bash +mosaic restore --from +``` + +Use `--yes` only when the overwrite has been explicitly approved: + +```bash +mosaic restore --from --yes +``` + +The timestamp is the value printed by `mosaic restore --list`; the command also +accepts the full `pre-update-` name. For a non-default configuration +home, pass the same target explicitly: + +```bash +mosaic restore --mosaic-home "$MOSAIC_HOME" --from +``` + +After restoring, repeat the health gate. Restore writes only the operator +surface represented by that snapshot. It does not downgrade the installed CLI, +restore framework-owned contract files, restore `.mosaic/storage-pglite`, or +run any migration. + +### If local PGlite data is missing or corrupt + +Do not use a framework snapshot as a database backup. Do not run a migration +runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate. +The current CLI does not provide a wired PGlite export/import operation; +`mosaic storage export` and `mosaic storage import` only print direct-copy +guidance. Preserve the configured PGlite data directory and route data +restoration through the held data-layer procedure rather than improvising a +recursive delete or copy. + +### Recovery decision tree + +- **CLI or framework path is wrong:** stop, verify `command -v mosaic`, + `mosaic --version`, `MOSAIC_HOME`, and `mosaic config path`; do not activate a + held service route. +- **Upgrade failed during framework sync:** use the installer's automatic + rollback first; if it reports incomplete recovery, list and preview the + durable snapshot, then restore it interactively. +- **An operator file changed after a reported-successful upgrade:** select the + pre-upgrade snapshot with `mosaic restore --list`, preview it with `--dry-run`, + and restore only after reviewing the relative-path plan. +- **PGlite data is affected:** preserve the data directory and stop at the + boundary described above. Framework rollback cannot recover database rows. +- **Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or + a migration runner:** stop; that is a held/non-operative route, not a next + command for this runbook. + +## Post-recovery verification + +Run the non-mutating checks again: + +```bash +mosaic --version +mosaic config path +env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show +env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status +mosaic restore --list +``` + +A green result here means the installed CLI resolves, the framework path is +present, the CLI selects local PGlite without a network probe, and snapshots can +be enumerated. It does not certify database contents or any held deployment +route. diff --git a/docs/_old_structure/guides/upgrade-safety-and-recovery.md b/docs/_old_structure/guides/upgrade-safety-and-recovery.md deleted file mode 100644 index 1f755cb7..00000000 --- a/docs/_old_structure/guides/upgrade-safety-and-recovery.md +++ /dev/null @@ -1,147 +0,0 @@ -# Upgrade Safety & Recovery - -How Mosaic protects operator-owned configuration under `~/.config/mosaic` across -framework upgrades, and how to recover if a projection is ever lost. - -A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`, -`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident -this hardening addresses: an upgrade that silently overwrites or deletes a file -the operator owns — credentials, personas, a roster, or a generated agent env — -with no snapshot to fall back to. - -Protection is layered. Each layer is independent; a later layer catches what an -earlier one misses. - -## Layer 1 — Manifest-owned sync (prevention) - -The single source of truth for ownership is -[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt). -Both the bash installer and the TypeScript sync path resolve every path against -this one file (parity is enforced by test), so they can never drift. - -- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a - `[framework]` glob matches and no `[operator]` carve-out overrides it. -- **Unknown paths default to operator** (fail-safe): a file the manifest never - anticipated is treated as operator-owned and is never pruned. -- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that - only ever iterates framework globs — operator and unknown paths are - structurally unreachable by the prune. - -Result: a correct upgrade cannot touch operator config at all. - -## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback) - -Before **any** mutation, the installer snapshots the operator-owned surface that -exists into: - -``` -${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-/ -``` - -- `0700` directories / `0600` files (`umask 077`, scoped and restored), - outside `~/.config/mosaic` and outside any repo. -- **Fail-open**: a snapshot failure warns but never aborts the upgrade it - protects. -- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5). - -After the sync, a **verify net** compares each snapshot file against its target -and restores (with a loud warning) any operator file the upgrade diverged or -removed — a divergence means a manifest bug slipped through Layer 1. - -Inspect and restore snapshots with the CLI: - -```bash -mosaic restore --list # dry-run: enumerate snapshots by timestamp -mosaic restore --from # restore the operator surface from one snapshot -mosaic restore --from --dry-run # preview a specific restore without writing -``` - -`mosaic restore` reports **counts and relative paths only** — it never emits file -contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores -are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf -atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails -closed rather than following out of the managed tree). - -## Layer 3 — Regeneration from roster SSOT (recovery) - -Some operator files are **derived** and do not need a byte-for-byte snapshot to -recover — they can be rebuilt from their source of truth. The fleet's per-agent -generated env projections are the prime case: - -- `~/.config/mosaic/fleet/agents/.env.generated` is a deterministic - projection of `~/.config/mosaic/fleet/roster.yaml`. -- The launcher (`start-agent-session.sh`, invoked by - `mosaic-agent@.service`) sources that generated projection to establish - each agent's identity, runtime, model, and working directory. If it is missing - or wrong, the agent cannot launch with its intended identity. - -`mosaic fleet regen` rebuilds those projections from the roster SSOT: - -```bash -mosaic fleet regen # dry-run (default): show what would be rebuilt -mosaic fleet regen --json # same, machine-readable -mosaic fleet regen --write # rebuild the projections on disk -``` - -- **Dry-run by default.** Nothing is written until you pass `--write`. -- **Deterministic and idempotent** — the projection is a pure function of the - roster, so repeated `--write` runs produce byte-identical files. -- **Projection-only. It never restarts an agent.** Recovery order forbids - restart-before-verify; `regen` has no path to systemd lifecycle at all. -- **It rebuilds only `.env.generated`** — it never writes, relocates, or - deletes the operator-owned `.env` / `.env.local` surface. -- It **validates the roster the same way `reconcile` does** (persona resolution - and protected-class tool-policy match), so a hand-edited or corrupt roster is - rejected rather than projected, and a `--write` takes the shared reconcile - lock so it cannot race a concurrent reconcile. -- Output is **paths and counts only** — the rendered `KEY=value` body is never - echoed. - -`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a -recovered projection matches what a normal reconcile would have written. - -## Recovery runbook — wiped `fleet/agents/*.env.generated` - -If an upgrade (or a manual mistake) has left an agent without its generated -projection, **do not restart the unit first** — a launch against a missing -projection fails closed, and any stale state must be corrected before restart, -not after. - -1. **Prefer a snapshot restore if one exists** (byte-exact operator state): - - ```bash - mosaic restore --list - mosaic restore --from - ``` - -2. **Otherwise regenerate the derived projections from the roster SSOT:** - - ```bash - mosaic fleet regen # confirm the plan (create vs rebuild per agent) - mosaic fleet regen --write # rebuild fleet/agents/.env.generated - ``` - -3. **Verify each unit will resolve the intended runtime/workdir _before_ any - restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal - environment and `start-agent-session.sh` sources `.env.generated` itself, so - verify the generated file directly and confirm the launcher path: - - ```bash - # Confirm fleet/agents/.env.generated exists and carries the intended - # MOSAIC_AGENT_* values (name, runtime, model, workdir, socket). - test -f ~/.config/mosaic/fleet/agents/.env.generated - # Confirm the unit launches the session script that reads it. - systemctl --user cat mosaic-agent@ | grep ExecStart - ``` - -4. **Only then restart, one unit at a time:** - - ```bash - systemctl --user restart mosaic-agent@ - ``` - -## See also - -- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md) -- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md) -- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)