stack/docs/ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md

Upgrade safety and recovery

Supported route: an already installed mosaic CLI using the local PGlite configuration. This is a filesystem and CLI runbook; it does not activate a service or connect to a database.

This page covers the supported upgrade, rollback, health, and recovery checks for Mosaic framework configuration under MOSAIC_HOME. It deliberately does not turn the repository's deployment or PostgreSQL material into an operative procedure.

Support boundary

Use this runbook only when all of the following are true:

• mosaic resolves to the installed Mosaic CLI (command -v mosaic).
• The active application configuration is the local tier: tier: "local", storage.type: "pglite", and queue.type: "local".
• DATABASE_URL is unset. An inherited PostgreSQL DSN is outside this route and must be removed before continuing.
• MOSAIC_STORAGE_TIER is unset or local; standalone and federated overrides are outside this route.
• No Gateway, Web, Compose, or other service activation is required.

The following routes are held/non-operative in this guide:

Route or operation	Status
PostgreSQL-backed standalone storage	Held; do not activate or probe it here.
Federated storage and peers	Held; do not activate or probe it here.
Bare-metal deployment	Held; no service installation or lifecycle action is authorized.
Gateway/Web activation or HTTP health checks	Held; /health, /health/ready, and mosaic gateway ... are not evidence for this route.
Compose startup	Held; do not start a Compose profile or the full stack.
Migration runners and tier migration	Held; do not run mosaic-db-migrator, pnpm --filter @mosaicstack/db db:migrate, mosaic storage migrate, or mosaic storage migrate-tier.

A command being present in the CLI does not make a held route operative.

Storage and path terminology

Keep these locations separate:

• MOSAIC_HOME — the framework/operator configuration directory. It defaults to ~/.config/mosaic and can be overridden with MOSAIC_HOME.
• PGlite data — the local, in-process database. The checked-in local config uses .mosaic/storage-pglite as its storage dataDir; .mosaic/queue is the local queue directory. These are project data, not framework configuration.
• Durable upgrade snapshots — operator-file snapshots stored under ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/. They are outside MOSAIC_HOME and do not contain a PostgreSQL dump.

The configuration tier is named local; the storage CLI reports the backend as pglite. PGlite is not PostgreSQL and does not require a PostgreSQL server. Local PGlite schema setup is adapter-owned; this page does not authorize a separate migration runner.

Pre-upgrade health gate

Run these checks from a shell that has no inherited database DSN:

command -v mosaic
mosaic --version
mosaic config path
test -d "$(mosaic config path)"

env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list

For the supported route, the storage checks should report the pglite backend and say that no network check is needed. mosaic storage status reports PGLITE_DATA_DIR when that variable is set; otherwise it reports the CLI's :memory: fallback. That output is an inspection of the CLI environment, not a claim that PGlite contents are healthy or durable.

If DATABASE_URL is set, stop. Do not point it at a local PostgreSQL instance to make the check pass. If the active configuration is not local/PGlite, stop; no operative route is documented here.

The supported health gate is intentionally limited to CLI resolution, framework configuration path, local storage selection, and available upgrade snapshots. It does not prove Gateway/Web readiness, provider connectivity, queue health, or PGlite data integrity.

Safe upgrade procedure

1. Record the baseline. Save the output of mosaic --version, mosaic config path, and mosaic restore --list. Do not copy secrets into a ticket or report.

2. Check for updates without installing them:

   mosaic update --check
   

   Exit status 0 means no update was reported; status 2 means an update is available. Other failures are not a successful health result. The check is a package-registry check and does not start Mosaic services or access storage.

3. Run the installed-CLI upgrade when approved:

   mosaic update
   

   The command updates the installed @mosaicstack/* packages through npm and, when the framework package changed or framework drift is detected, re-seeds framework files in keep mode. Leave the default re-seed enabled. Do not use --relaunch in this runbook: that option can restart durable fleet agents and is outside the supported no-activation route. --no-reseed is also not a normal upgrade path because it intentionally leaves framework files stale.

4. Repeat the health gate. Confirm the CLI version, the same MOSAIC_HOME, the local/PGlite storage selection, and the snapshot listing. A successful framework upgrade must not be used as evidence that a held Gateway, Web, PostgreSQL, or federated route is ready.

Do not replace this procedure with a direct edit of ~/.config/mosaic, a repository tools/install.sh invocation, a Compose startup, or a migration command. The supported operator entry point for this page is the installed mosaic update command.

What an upgrade protects

For an existing keep-mode installation, the framework installer takes two separate snapshots before the file sync:

1. An ephemeral whole-directory snapshot under ${TMPDIR:-/tmp}/ is used to restore the previous framework directory if the sync is interrupted or fails. If that restore cannot complete, the installer prints the retained temporary snapshot path for manual recovery.

2. A durable snapshot captures the existing operator-owned files before the upgrade at:

   ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
   

   Snapshot directories are mode 0700; captured files are mode 0600. Retention is five snapshots by default and can be changed with MOSAIC_BACKUP_RETENTION. A durable snapshot failure is a warning and does not replace the manifest and crash-rollback protections.

After the keep-mode sync, the installer compares each captured operator file with its target. If a file was changed or removed unexpectedly, it restores the snapshot copy and emits a warning. A symlinked parent is not followed; that case is left for manual recovery from the snapshot path.

Keep mode is manifest-driven and fail-safe for operator paths, but it does not mean every file under MOSAIC_HOME is user-owned. The framework contract files CONSTITUTION.md, AGENTS.md, and STANDARDS.md are refreshed by upgrades; keep local policy in the supported local overlays rather than editing those contract files as rollback data. A durable snapshot is for operator-owned configuration, not for the installed npm package, framework-owned contracts, or PGlite data.

Rollback and recovery

If the upgrade fails or is interrupted

The framework sync attempts an automatic rollback from its ephemeral snapshot. Do not delete MOSAIC_HOME, the PGlite data directory, or the temporary snapshot named in the error. Preserve the command output, then run the health gate.

If the installed framework reports that the automatic restore did not complete, use the durable snapshot procedure below. The durable snapshot is the recovery pointer that survives a successful upgrade and the removal of the temporary snapshot.

Restore operator configuration from a durable snapshot

mosaic restore is confirmation-gated and reports counts and relative paths, not file contents. First list snapshots, then preview the selected timestamp:

mosaic restore --list
mosaic restore --from <UTC-timestamp> --dry-run

If the preview is correct, run the interactive restore:

mosaic restore --from <UTC-timestamp>

Use --yes only when the overwrite has been explicitly approved:

mosaic restore --from <UTC-timestamp> --yes

The timestamp is the value printed by mosaic restore --list; the command also accepts the full pre-update-<timestamp> name. For a non-default configuration home, pass the same target explicitly:

mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>

After restoring, repeat the health gate. Restore writes only the operator surface represented by that snapshot. It does not downgrade the installed CLI, restore framework-owned contract files, restore .mosaic/storage-pglite, or run any migration.

If local PGlite data is missing or corrupt

Do not use a framework snapshot as a database backup. Do not run a migration runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate. The current CLI does not provide a wired PGlite export/import operation; mosaic storage export and mosaic storage import only print direct-copy guidance. Preserve the configured PGlite data directory and route data restoration through the held data-layer procedure rather than improvising a recursive delete or copy.

Recovery decision tree

• CLI or framework path is wrong: stop, verify command -v mosaic, mosaic --version, MOSAIC_HOME, and mosaic config path; do not activate a held service route.
• Upgrade failed during framework sync: use the installer's automatic rollback first; if it reports incomplete recovery, list and preview the durable snapshot, then restore it interactively.
• An operator file changed after a reported-successful upgrade: select the pre-upgrade snapshot with mosaic restore --list, preview it with --dry-run, and restore only after reviewing the relative-path plan.
• PGlite data is affected: preserve the data directory and stop at the boundary described above. Framework rollback cannot recover database rows.
• Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or a migration runner: stop; that is a held/non-operative route, not a next command for this runbook.

Post-recovery verification

Run the non-mutating checks again:

mosaic --version
mosaic config path
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list

A green result here means the installed CLI resolves, the framework path is present, the CLI selects local PGlite without a network probe, and snapshots can be enumerated. It does not certify database contents or any held deployment route.