Files
stack/docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md
T
veronica f0d2dd9920 docs(W4): stamp kind and status front matter on 104 live documents
Applies the document contract from
docs/plans/2026-08-20_stack-docs-flatten-and-alignment.md section 3, partially:
`kind` and `status` only. `parent` is deliberately held until the flatten in
section 4 lands, so that 127 documents do not have to be re-pointed by hand
when docs/fleet/NORTH_STAR.yaml moves to docs/NORTH_STAR.yaml.

Scope, measured on origin/next at 63069149:

  127 live docs   = all *.md under docs/ minus docs/archive/ minus docs/_old_structure/
  104 stamped     here
   19 held        operator judgement (plan section 9), worklist in the same PR
    3 held        the SUPERSEDED TASKS.md stamps, which cite the moving path
    1 untouched   docs/fleet/FLEET-DOCTRINE.md, already stamped in W1

Kinds applied: 54 guide, 34 record, 9 spec, 6 tracking, 1 projection.
Every row carries a confidence and a one-line rationale in the worklist.

Two collisions with the existing state, both flagged rather than resolved:

1. docs/README.md:150-160 already documents a front-matter convention
   (title/type/audience/status/source_of_truth) with its own allowed values.
   It is applied to 4 of 127 files. Its `status` vocabulary is
   current|draft|deprecated|historical; the new contract's is active|superseded-by.
   The key collides. This commit lets the new contract win and rewrites
   `status: current` to `status: active` on those 4 files, keeping their other
   legacy keys untouched. No code reads any of them: `git grep source_of_truth`
   outside docs/ returns nothing. docs/README.md still prescribes the old
   convention and is an operator row, so it is not edited here.

2. Two of the plan's 20 operator rows are YAML files, not markdown
   (docs/fleet/examples/roster-v2.yaml, docs/openapi-tess.yaml), and the
   contract's front-matter form has no defined meaning for a .yaml document.
   That gap also applies to docs/fleet/NORTH_STAR.yaml, the source of truth
   itself. Raised in the worklist.

A third row from the plan, docs/fleet/north-star.md, no longer exists: W1
renamed it to docs/fleet/FLEET-DOCTRINE.md.

Verification: 104/104 parse with the expected kind and status in front matter;
the check was shown to reject a wrong kind before it was trusted. The diff
removes 4 lines total, all of them `status: current`.
2026-08-20 19:30:25 -05:00

12 KiB

kind, status
kind status
guide active

Upgrade safety and recovery

Supported route: an already installed mosaic CLI using the local PGlite configuration. This is a filesystem and CLI runbook; it does not activate a service or connect to a database.

This page covers the supported upgrade, rollback, health, and recovery checks for Mosaic framework configuration under MOSAIC_HOME. It deliberately does not turn the repository's deployment or PostgreSQL material into an operative procedure.

Support boundary

Use this runbook only when all of the following are true:

  • mosaic resolves to the installed Mosaic CLI (command -v mosaic).
  • The active application configuration is the local tier: tier: "local", storage.type: "pglite", and queue.type: "local".
  • DATABASE_URL is unset. An inherited PostgreSQL DSN is outside this route and must be removed before continuing.
  • MOSAIC_STORAGE_TIER is unset or local; standalone and federated overrides are outside this route.
  • No Gateway, Web, Compose, or other service activation is required.

The following routes are held/non-operative in this guide:

Route or operation Status
PostgreSQL-backed standalone storage Held; do not activate or probe it here.
Federated storage and peers Held; do not activate or probe it here.
Bare-metal deployment Held; no service installation or lifecycle action is authorized.
Gateway/Web activation or HTTP health checks Held; /health, /health/ready, and mosaic gateway ... are not evidence for this route.
Compose startup Held; do not start a Compose profile or the full stack.
Migration runners and tier migration Held; do not run mosaic-db-migrator, pnpm --filter @mosaicstack/db db:migrate, mosaic storage migrate, or mosaic storage migrate-tier.

A command being present in the CLI does not make a held route operative.

Storage and path terminology

Keep these locations separate:

  • MOSAIC_HOME — the framework/operator configuration directory. It defaults to ~/.config/mosaic and can be overridden with MOSAIC_HOME.
  • PGlite data — the local, in-process database. The checked-in local config uses .mosaic/storage-pglite as its storage dataDir; .mosaic/queue is the local queue directory. These are project data, not framework configuration.
  • Durable upgrade snapshots — operator-file snapshots stored under ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/. They are outside MOSAIC_HOME and do not contain a PostgreSQL dump.

The configuration tier is named local; the storage CLI reports the backend as pglite. PGlite is not PostgreSQL and does not require a PostgreSQL server. Local PGlite schema setup is adapter-owned; this page does not authorize a separate migration runner.

Pre-upgrade health gate

Run these checks from a shell that has no inherited database DSN:

command -v mosaic
mosaic --version
mosaic config path
test -d "$(mosaic config path)"

env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list

For the supported route, the storage checks should report the pglite backend and say that no network check is needed. mosaic storage status reports PGLITE_DATA_DIR when that variable is set; otherwise it reports the CLI's :memory: fallback. That output is an inspection of the CLI environment, not a claim that PGlite contents are healthy or durable.

If DATABASE_URL is set, stop. Do not point it at a local PostgreSQL instance to make the check pass. If the active configuration is not local/PGlite, stop; no operative route is documented here.

The supported health gate is intentionally limited to CLI resolution, framework configuration path, local storage selection, and available upgrade snapshots. It does not prove Gateway/Web readiness, provider connectivity, queue health, or PGlite data integrity.

Safe upgrade procedure

  1. Record the baseline. Save the output of mosaic --version, mosaic config path, and mosaic restore --list. Do not copy secrets into a ticket or report.

  2. Check for updates without installing them:

    mosaic update --check
    

    Exit status 0 means no update was reported; status 2 means an update is available. Other failures are not a successful health result. The check is a package-registry check and does not start Mosaic services or access storage.

  3. Run the installed-CLI upgrade when approved:

    mosaic update
    

    The command updates the installed @mosaicstack/* packages through npm and, when the framework package changed or framework drift is detected, re-seeds framework files in keep mode. Leave the default re-seed enabled. Do not use --relaunch in this runbook: that option can restart durable fleet agents and is outside the supported no-activation route. --no-reseed is also not a normal upgrade path because it intentionally leaves framework files stale.

  4. Repeat the health gate. Confirm the CLI version, the same MOSAIC_HOME, the local/PGlite storage selection, and the snapshot listing. A successful framework upgrade must not be used as evidence that a held Gateway, Web, PostgreSQL, or federated route is ready.

Do not replace this procedure with a direct edit of ~/.config/mosaic, a repository tools/install.sh invocation, a Compose startup, or a migration command. The supported operator entry point for this page is the installed mosaic update command.

What an upgrade protects

For an existing keep-mode installation, the framework installer takes two separate snapshots before the file sync:

  1. An ephemeral whole-directory snapshot under ${TMPDIR:-/tmp}/ is used to restore the previous framework directory if the sync is interrupted or fails. If that restore cannot complete, the installer prints the retained temporary snapshot path for manual recovery.

  2. A durable snapshot captures the existing operator-owned files before the upgrade at:

    ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
    

    Snapshot directories are mode 0700; captured files are mode 0600. Retention is five snapshots by default and can be changed with MOSAIC_BACKUP_RETENTION. A durable snapshot failure is a warning and does not replace the manifest and crash-rollback protections.

After the keep-mode sync, the installer compares each captured operator file with its target. If a file was changed or removed unexpectedly, it restores the snapshot copy and emits a warning. A symlinked parent is not followed; that case is left for manual recovery from the snapshot path.

Keep mode is manifest-driven and fail-safe for operator paths, but it does not mean every file under MOSAIC_HOME is user-owned. The framework contract files CONSTITUTION.md, AGENTS.md, and STANDARDS.md are refreshed by upgrades; keep local policy in the supported local overlays rather than editing those contract files as rollback data. A durable snapshot is for operator-owned configuration, not for the installed npm package, framework-owned contracts, or PGlite data.

Rollback and recovery

If the upgrade fails or is interrupted

The framework sync attempts an automatic rollback from its ephemeral snapshot. Do not delete MOSAIC_HOME, the PGlite data directory, or the temporary snapshot named in the error. Preserve the command output, then run the health gate.

If the installed framework reports that the automatic restore did not complete, use the durable snapshot procedure below. The durable snapshot is the recovery pointer that survives a successful upgrade and the removal of the temporary snapshot.

Restore operator configuration from a durable snapshot

mosaic restore is confirmation-gated and reports counts and relative paths, not file contents. First list snapshots, then preview the selected timestamp:

mosaic restore --list
mosaic restore --from <UTC-timestamp> --dry-run

If the preview is correct, run the interactive restore:

mosaic restore --from <UTC-timestamp>

Use --yes only when the overwrite has been explicitly approved:

mosaic restore --from <UTC-timestamp> --yes

The timestamp is the value printed by mosaic restore --list; the command also accepts the full pre-update-<timestamp> name. For a non-default configuration home, pass the same target explicitly:

mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>

After restoring, repeat the health gate. Restore writes only the operator surface represented by that snapshot. It does not downgrade the installed CLI, restore framework-owned contract files, restore .mosaic/storage-pglite, or run any migration.

If local PGlite data is missing or corrupt

Do not use a framework snapshot as a database backup. Do not run a migration runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate. The current CLI does not provide a wired PGlite export/import operation; mosaic storage export and mosaic storage import only print direct-copy guidance. Preserve the configured PGlite data directory and route data restoration through the held data-layer procedure rather than improvising a recursive delete or copy.

Recovery decision tree

  • CLI or framework path is wrong: stop, verify command -v mosaic, mosaic --version, MOSAIC_HOME, and mosaic config path; do not activate a held service route.
  • Upgrade failed during framework sync: use the installer's automatic rollback first; if it reports incomplete recovery, list and preview the durable snapshot, then restore it interactively.
  • An operator file changed after a reported-successful upgrade: select the pre-upgrade snapshot with mosaic restore --list, preview it with --dry-run, and restore only after reviewing the relative-path plan.
  • PGlite data is affected: preserve the data directory and stop at the boundary described above. Framework rollback cannot recover database rows.
  • Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or a migration runner: stop; that is a held/non-operative route, not a next command for this runbook.

Post-recovery verification

Run the non-mutating checks again:

mosaic --version
mosaic config path
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list

A green result here means the installed CLI resolves, the framework path is present, the CLI selects local PGlite without a network probe, and snapshots can be enumerated. It does not certify database contents or any held deployment route.