mosaicstack/stack
1
0
Fork 1
Code Issues 245 Pull Requests 7 Actions Packages Projects Releases 15 Wiki Activity

docs: migrate upgrade recovery admin guide

Browse Source
This commit is contained in:
Jason Woltje
2026-08-10 18:13:46 -05:00
parent 00bdf8b28c
commit 39987a5b61
2 changed files with 248 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md
+248
View File
@@ -0,0 +1,248 @@
# Upgrade safety and recovery
> **Supported route:** an already installed `mosaic` CLI using the local PGlite
> configuration. This is a filesystem and CLI runbook; it does not activate a
> service or connect to a database.
This page covers the supported upgrade, rollback, health, and recovery checks for
Mosaic framework configuration under `MOSAIC_HOME`. It deliberately does not
turn the repository's deployment or PostgreSQL material into an operative
procedure.
## Support boundary
Use this runbook only when all of the following are true:
- `mosaic` resolves to the installed Mosaic CLI (`command -v mosaic`).
- The active application configuration is the local tier: `tier: "local"`,
`storage.type: "pglite"`, and `queue.type: "local"`.
- `DATABASE_URL` is unset. An inherited PostgreSQL DSN is outside this route and
must be removed before continuing.
- `MOSAIC_STORAGE_TIER` is unset or `local`; standalone and federated overrides
are outside this route.
- No Gateway, Web, Compose, or other service activation is required.
The following routes are **held/non-operative** in this guide:
| Route or operation | Status |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL-backed `standalone` storage | Held; do not activate or probe it here. |
| Federated storage and peers | Held; do not activate or probe it here. |
| Bare-metal deployment | Held; no service installation or lifecycle action is authorized. |
| Gateway/Web activation or HTTP health checks | Held; `/health`, `/health/ready`, and `mosaic gateway ...` are not evidence for this route. |
| Compose startup | Held; do not start a Compose profile or the full stack. |
| Migration runners and tier migration | Held; do not run `mosaic-db-migrator`, `pnpm --filter @mosaicstack/db db:migrate`, `mosaic storage migrate`, or `mosaic storage migrate-tier`. |
A command being present in the CLI does not make a held route operative.
## Storage and path terminology
Keep these locations separate:
- **`MOSAIC_HOME`** — the framework/operator configuration directory. It
defaults to `~/.config/mosaic` and can be overridden with `MOSAIC_HOME`.
- **PGlite data** — the local, in-process database. The checked-in local config
uses `.mosaic/storage-pglite` as its storage `dataDir`; `.mosaic/queue` is the
local queue directory. These are project data, not framework configuration.
- **Durable upgrade snapshots** — operator-file snapshots stored under
`${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/`. They are outside
`MOSAIC_HOME` and do not contain a PostgreSQL dump.
The configuration tier is named `local`; the storage CLI reports the backend as
`pglite`. PGlite is not PostgreSQL and does not require a PostgreSQL server.
Local PGlite schema setup is adapter-owned; this page does not authorize a
separate migration runner.
## Pre-upgrade health gate
Run these checks from a shell that has no inherited database DSN:
```bash
command -v mosaic
mosaic --version
mosaic config path
test -d "$(mosaic config path)"
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list
```
For the supported route, the storage checks should report the `pglite` backend
and say that no network check is needed. `mosaic storage status` reports
`PGLITE_DATA_DIR` when that variable is set; otherwise it reports the CLI's
`:memory:` fallback. That output is an inspection of the CLI environment, not a
claim that PGlite contents are healthy or durable.
If `DATABASE_URL` is set, stop. Do not point it at a local PostgreSQL instance
to make the check pass. If the active configuration is not local/PGlite, stop;
no operative route is documented here.
The supported health gate is intentionally limited to CLI resolution, framework
configuration path, local storage selection, and available upgrade snapshots.
It does not prove Gateway/Web readiness, provider connectivity, queue health, or
PGlite data integrity.
## Safe upgrade procedure
1. **Record the baseline.** Save the output of `mosaic --version`,
`mosaic config path`, and `mosaic restore --list`. Do not copy secrets into a
ticket or report.
2. **Check for updates without installing them:**
```bash
mosaic update --check
```
Exit status `0` means no update was reported; status `2` means an update is
available. Other failures are not a successful health result. The check is a
package-registry check and does not start Mosaic services or access storage.
3. **Run the installed-CLI upgrade when approved:**
```bash
mosaic update
```
The command updates the installed `@mosaicstack/*` packages through npm and,
when the framework package changed or framework drift is detected, re-seeds
framework files in keep mode. Leave the default re-seed enabled. Do not use
`--relaunch` in this runbook: that option can restart durable fleet agents
and is outside the supported no-activation route. `--no-reseed` is also not a
normal upgrade path because it intentionally leaves framework files stale.
4. **Repeat the health gate.** Confirm the CLI version, the same
`MOSAIC_HOME`, the local/PGlite storage selection, and the snapshot listing.
A successful framework upgrade must not be used as evidence that a held
Gateway, Web, PostgreSQL, or federated route is ready.
Do not replace this procedure with a direct edit of `~/.config/mosaic`, a
repository `tools/install.sh` invocation, a Compose startup, or a migration
command. The supported operator entry point for this page is the installed
`mosaic update` command.
## What an upgrade protects
For an existing keep-mode installation, the framework installer takes two
separate snapshots before the file sync:
1. An ephemeral whole-directory snapshot under `${TMPDIR:-/tmp}/` is used to
restore the previous framework directory if the sync is interrupted or
fails. If that restore cannot complete, the installer prints the retained
temporary snapshot path for manual recovery.
2. A durable snapshot captures the existing operator-owned files before the
upgrade at:
```text
${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
```
Snapshot directories are mode `0700`; captured files are mode `0600`.
Retention is five snapshots by default and can be changed with
`MOSAIC_BACKUP_RETENTION`. A durable snapshot failure is a warning and does
not replace the manifest and crash-rollback protections.
After the keep-mode sync, the installer compares each captured operator file
with its target. If a file was changed or removed unexpectedly, it restores the
snapshot copy and emits a warning. A symlinked parent is not followed; that
case is left for manual recovery from the snapshot path.
Keep mode is manifest-driven and fail-safe for operator paths, but it does not
mean every file under `MOSAIC_HOME` is user-owned. The framework contract files
`CONSTITUTION.md`, `AGENTS.md`, and `STANDARDS.md` are refreshed by upgrades;
keep local policy in the supported local overlays rather than editing those
contract files as rollback data. A durable snapshot is for operator-owned
configuration, not for the installed npm package, framework-owned contracts, or
PGlite data.
## Rollback and recovery
### If the upgrade fails or is interrupted
The framework sync attempts an automatic rollback from its ephemeral snapshot.
Do not delete `MOSAIC_HOME`, the PGlite data directory, or the temporary snapshot
named in the error. Preserve the command output, then run the health gate.
If the installed framework reports that the automatic restore did not complete,
use the durable snapshot procedure below. The durable snapshot is the recovery
pointer that survives a successful upgrade and the removal of the temporary
snapshot.
### Restore operator configuration from a durable snapshot
`mosaic restore` is confirmation-gated and reports counts and relative paths,
not file contents. First list snapshots, then preview the selected timestamp:
```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp> --dry-run
```
If the preview is correct, run the interactive restore:
```bash
mosaic restore --from <UTC-timestamp>
```
Use `--yes` only when the overwrite has been explicitly approved:
```bash
mosaic restore --from <UTC-timestamp> --yes
```
The timestamp is the value printed by `mosaic restore --list`; the command also
accepts the full `pre-update-<timestamp>` name. For a non-default configuration
home, pass the same target explicitly:
```bash
mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>
```
After restoring, repeat the health gate. Restore writes only the operator
surface represented by that snapshot. It does not downgrade the installed CLI,
restore framework-owned contract files, restore `.mosaic/storage-pglite`, or
run any migration.
### If local PGlite data is missing or corrupt
Do not use a framework snapshot as a database backup. Do not run a migration
runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate.
The current CLI does not provide a wired PGlite export/import operation;
`mosaic storage export` and `mosaic storage import` only print direct-copy
guidance. Preserve the configured PGlite data directory and route data
restoration through the held data-layer procedure rather than improvising a
recursive delete or copy.
### Recovery decision tree
- **CLI or framework path is wrong:** stop, verify `command -v mosaic`,
`mosaic --version`, `MOSAIC_HOME`, and `mosaic config path`; do not activate a
held service route.
- **Upgrade failed during framework sync:** use the installer's automatic
rollback first; if it reports incomplete recovery, list and preview the
durable snapshot, then restore it interactively.
- **An operator file changed after a reported-successful upgrade:** select the
pre-upgrade snapshot with `mosaic restore --list`, preview it with `--dry-run`,
and restore only after reviewing the relative-path plan.
- **PGlite data is affected:** preserve the data directory and stop at the
boundary described above. Framework rollback cannot recover database rows.
- **Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or
a migration runner:** stop; that is a held/non-operative route, not a next
command for this runbook.
## Post-recovery verification
Run the non-mutating checks again:
```bash
mosaic --version
mosaic config path
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list
```
A green result here means the installed CLI resolves, the framework path is
present, the CLI selects local PGlite without a network probe, and snapshots can
be enumerated. It does not certify database contents or any held deployment
route.
docs/_old_structure/guides/upgrade-safety-and-recovery.md
-147
View File
@@ -1,147 +0,0 @@
# Upgrade Safety & Recovery
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
framework upgrades, and how to recover if a projection is ever lost.
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
this hardening addresses: an upgrade that silently overwrites or deletes a file
the operator owns — credentials, personas, a roster, or a generated agent env —
with no snapshot to fall back to.
Protection is layered. Each layer is independent; a later layer catches what an
earlier one misses.
## Layer 1 — Manifest-owned sync (prevention)
The single source of truth for ownership is
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
Both the bash installer and the TypeScript sync path resolve every path against
this one file (parity is enforced by test), so they can never drift.
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
`[framework]` glob matches and no `[operator]` carve-out overrides it.
- **Unknown paths default to operator** (fail-safe): a file the manifest never
anticipated is treated as operator-owned and is never pruned.
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
only ever iterates framework globs — operator and unknown paths are
structurally unreachable by the prune.
Result: a correct upgrade cannot touch operator config at all.
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
Before **any** mutation, the installer snapshots the operator-owned surface that
exists into:
```
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
```
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
outside `~/.config/mosaic` and outside any repo.
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
protects.
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
After the sync, a **verify net** compares each snapshot file against its target
and restores (with a loud warning) any operator file the upgrade diverged or
removed — a divergence means a manifest bug slipped through Layer 1.
Inspect and restore snapshots with the CLI:
```bash
mosaic restore --list # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
```
`mosaic restore` reports **counts and relative paths only** — it never emits file
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
closed rather than following out of the managed tree).
## Layer 3 — Regeneration from roster SSOT (recovery)
Some operator files are **derived** and do not need a byte-for-byte snapshot to
recover — they can be rebuilt from their source of truth. The fleet's per-agent
generated env projections are the prime case:
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
projection of `~/.config/mosaic/fleet/roster.yaml`.
- The launcher (`start-agent-session.sh`, invoked by
`mosaic-agent@<name>.service`) sources that generated projection to establish
each agent's identity, runtime, model, and working directory. If it is missing
or wrong, the agent cannot launch with its intended identity.
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
```bash
mosaic fleet regen # dry-run (default): show what would be rebuilt
mosaic fleet regen --json # same, machine-readable
mosaic fleet regen --write # rebuild the projections on disk
```
- **Dry-run by default.** Nothing is written until you pass `--write`.
- **Deterministic and idempotent** — the projection is a pure function of the
roster, so repeated `--write` runs produce byte-identical files.
- **Projection-only. It never restarts an agent.** Recovery order forbids
restart-before-verify; `regen` has no path to systemd lifecycle at all.
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
deletes the operator-owned `.env` / `.env.local` surface.
- It **validates the roster the same way `reconcile` does** (persona resolution
and protected-class tool-policy match), so a hand-edited or corrupt roster is
rejected rather than projected, and a `--write` takes the shared reconcile
lock so it cannot race a concurrent reconcile.
- Output is **paths and counts only** — the rendered `KEY=value` body is never
echoed.
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
recovered projection matches what a normal reconcile would have written.
## Recovery runbook — wiped `fleet/agents/*.env.generated`
If an upgrade (or a manual mistake) has left an agent without its generated
projection, **do not restart the unit first** — a launch against a missing
projection fails closed, and any stale state must be corrected before restart,
not after.
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp>
```
2. **Otherwise regenerate the derived projections from the roster SSOT:**
```bash
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
```
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
environment and `start-agent-session.sh` sources `.env.generated` itself, so
verify the generated file directly and confirm the launcher path:
```bash
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
# Confirm the unit launches the session script that reads it.
systemctl --user cat mosaic-agent@<name> | grep ExecStart
```
4. **Only then restart, one unit at a time:**
```bash
systemctl --user restart mosaic-agent@<name>
```
## See also
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)