---
kind: guide
status: active
---

# Upgrade safety and recovery

> **Supported route:** an already installed `mosaic` CLI using the local PGlite
> configuration. This is a filesystem and CLI runbook; it does not activate a
> service or connect to a database.

This page covers the supported upgrade, rollback, health, and recovery checks for
Mosaic framework configuration under `MOSAIC_HOME`. It deliberately does not
turn the repository's deployment or PostgreSQL material into an operative
procedure.

## Support boundary

Use this runbook only when all of the following are true:

- `mosaic` resolves to the installed Mosaic CLI (`command -v mosaic`).
- The active application configuration is the local tier: `tier: "local"`,
  `storage.type: "pglite"`, and `queue.type: "local"`.
- `DATABASE_URL` is unset. An inherited PostgreSQL DSN is outside this route and
  must be removed before continuing.
- `MOSAIC_STORAGE_TIER` is unset or `local`; standalone and federated overrides
  are outside this route.
- No Gateway, Web, Compose, or other service activation is required.

The following routes are **held/non-operative** in this guide:

| Route or operation                           | Status                                                                                                                                         |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL-backed `standalone` storage       | Held; do not activate or probe it here.                                                                                                        |
| Federated storage and peers                  | Held; do not activate or probe it here.                                                                                                        |
| Bare-metal deployment                        | Held; no service installation or lifecycle action is authorized.                                                                               |
| Gateway/Web activation or HTTP health checks | Held; `/health`, `/health/ready`, and `mosaic gateway ...` are not evidence for this route.                                                    |
| Compose startup                              | Held; do not start a Compose profile or the full stack.                                                                                        |
| Migration runners and tier migration         | Held; do not run `mosaic-db-migrator`, `pnpm --filter @mosaicstack/db db:migrate`, `mosaic storage migrate`, or `mosaic storage migrate-tier`. |

A command being present in the CLI does not make a held route operative.

## Storage and path terminology

Keep these locations separate:

- **`MOSAIC_HOME`** — the framework/operator configuration directory. It
  defaults to `~/.config/mosaic` and can be overridden with `MOSAIC_HOME`.
- **PGlite data** — the local, in-process database. The checked-in local config
  uses `.mosaic/storage-pglite` as its storage `dataDir`; `.mosaic/queue` is the
  local queue directory. These are project data, not framework configuration.
- **Durable upgrade snapshots** — operator-file snapshots stored under
  `${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/`. They are outside
  `MOSAIC_HOME` and do not contain a PostgreSQL dump.

The configuration tier is named `local`; the storage CLI reports the backend as
`pglite`. PGlite is not PostgreSQL and does not require a PostgreSQL server.
Local PGlite schema setup is adapter-owned; this page does not authorize a
separate migration runner.

## Pre-upgrade health gate

Run these checks from a shell that has no inherited database DSN:

```bash
command -v mosaic
mosaic --version
mosaic config path
test -d "$(mosaic config path)"

env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list
```

For the supported route, the storage checks should report the `pglite` backend
and say that no network check is needed. `mosaic storage status` reports
`PGLITE_DATA_DIR` when that variable is set; otherwise it reports the CLI's
`:memory:` fallback. That output is an inspection of the CLI environment, not a
claim that PGlite contents are healthy or durable.

If `DATABASE_URL` is set, stop. Do not point it at a local PostgreSQL instance
to make the check pass. If the active configuration is not local/PGlite, stop;
no operative route is documented here.

The supported health gate is intentionally limited to CLI resolution, framework
configuration path, local storage selection, and available upgrade snapshots.
It does not prove Gateway/Web readiness, provider connectivity, queue health, or
PGlite data integrity.

## Safe upgrade procedure

1. **Record the baseline.** Save the output of `mosaic --version`,
   `mosaic config path`, and `mosaic restore --list`. Do not copy secrets into a
   ticket or report.
2. **Check for updates without installing them:**

   ```bash
   mosaic update --check
   ```

   Exit status `0` means no update was reported; status `2` means an update is
   available. Other failures are not a successful health result. The check is a
   package-registry check and does not start Mosaic services or access storage.

3. **Run the installed-CLI upgrade when approved:**

   ```bash
   mosaic update
   ```

   The command updates the installed `@mosaicstack/*` packages through npm and,
   when the framework package changed or framework drift is detected, re-seeds
   framework files in keep mode. Leave the default re-seed enabled. Do not use
   `--relaunch` in this runbook: that option can restart durable fleet agents
   and is outside the supported no-activation route. `--no-reseed` is also not a
   normal upgrade path because it intentionally leaves framework files stale.

4. **Repeat the health gate.** Confirm the CLI version, the same
   `MOSAIC_HOME`, the local/PGlite storage selection, and the snapshot listing.
   A successful framework upgrade must not be used as evidence that a held
   Gateway, Web, PostgreSQL, or federated route is ready.

Do not replace this procedure with a direct edit of `~/.config/mosaic`, a
repository `tools/install.sh` invocation, a Compose startup, or a migration
command. The supported operator entry point for this page is the installed
`mosaic update` command.

## What an upgrade protects

For an existing keep-mode installation, the framework installer takes two
separate snapshots before the file sync:

1. An ephemeral whole-directory snapshot under `${TMPDIR:-/tmp}/` is used to
   restore the previous framework directory if the sync is interrupted or
   fails. If that restore cannot complete, the installer prints the retained
   temporary snapshot path for manual recovery.
2. A durable snapshot captures the existing operator-owned files before the
   upgrade at:

   ```text
   ${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
   ```

   Snapshot directories are mode `0700`; captured files are mode `0600`.
   Retention is five snapshots by default and can be changed with
   `MOSAIC_BACKUP_RETENTION`. A durable snapshot failure is a warning and does
   not replace the manifest and crash-rollback protections.

After the keep-mode sync, the installer compares each captured operator file
with its target. If a file was changed or removed unexpectedly, it restores the
snapshot copy and emits a warning. A symlinked parent is not followed; that
case is left for manual recovery from the snapshot path.

Keep mode is manifest-driven and fail-safe for operator paths, but it does not
mean every file under `MOSAIC_HOME` is user-owned. The framework contract files
`CONSTITUTION.md`, `AGENTS.md`, and `STANDARDS.md` are refreshed by upgrades;
keep local policy in the supported local overlays rather than editing those
contract files as rollback data. A durable snapshot is for operator-owned
configuration, not for the installed npm package, framework-owned contracts, or
PGlite data.

## Rollback and recovery

### If the upgrade fails or is interrupted

The framework sync attempts an automatic rollback from its ephemeral snapshot.
Do not delete `MOSAIC_HOME`, the PGlite data directory, or the temporary snapshot
named in the error. Preserve the command output, then run the health gate.

If the installed framework reports that the automatic restore did not complete,
use the durable snapshot procedure below. The durable snapshot is the recovery
pointer that survives a successful upgrade and the removal of the temporary
snapshot.

### Restore operator configuration from a durable snapshot

`mosaic restore` is confirmation-gated and reports counts and relative paths,
not file contents. First list snapshots, then preview the selected timestamp:

```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp> --dry-run
```

If the preview is correct, run the interactive restore:

```bash
mosaic restore --from <UTC-timestamp>
```

Use `--yes` only when the overwrite has been explicitly approved:

```bash
mosaic restore --from <UTC-timestamp> --yes
```

The timestamp is the value printed by `mosaic restore --list`; the command also
accepts the full `pre-update-<timestamp>` name. For a non-default configuration
home, pass the same target explicitly:

```bash
mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>
```

After restoring, repeat the health gate. Restore writes only the operator
surface represented by that snapshot. It does not downgrade the installed CLI,
restore framework-owned contract files, restore `.mosaic/storage-pglite`, or
run any migration.

### If local PGlite data is missing or corrupt

Do not use a framework snapshot as a database backup. Do not run a migration
runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate.
The current CLI does not provide a wired PGlite export/import operation;
`mosaic storage export` and `mosaic storage import` only print direct-copy
guidance. Preserve the configured PGlite data directory and route data
restoration through the held data-layer procedure rather than improvising a
recursive delete or copy.

### Recovery decision tree

- **CLI or framework path is wrong:** stop, verify `command -v mosaic`,
  `mosaic --version`, `MOSAIC_HOME`, and `mosaic config path`; do not activate a
  held service route.
- **Upgrade failed during framework sync:** use the installer's automatic
  rollback first; if it reports incomplete recovery, list and preview the
  durable snapshot, then restore it interactively.
- **An operator file changed after a reported-successful upgrade:** select the
  pre-upgrade snapshot with `mosaic restore --list`, preview it with `--dry-run`,
  and restore only after reviewing the relative-path plan.
- **PGlite data is affected:** preserve the data directory and stop at the
  boundary described above. Framework rollback cannot recover database rows.
- **Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or
  a migration runner:** stop; that is a held/non-operative route, not a next
  command for this runbook.

## Post-recovery verification

Run the non-mutating checks again:

```bash
mosaic --version
mosaic config path
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
mosaic restore --list
```

A green result here means the installed CLI resolves, the framework path is
present, the CLI selects local PGlite without a network probe, and snapshots can
be enumerated. It does not certify database contents or any held deployment
route.
