249 lines
12 KiB
Markdown
249 lines
12 KiB
Markdown
# Upgrade safety and recovery
|
|
|
|
> **Supported route:** an already installed `mosaic` CLI using the local PGlite
|
|
> configuration. This is a filesystem and CLI runbook; it does not activate a
|
|
> service or connect to a database.
|
|
|
|
This page covers the supported upgrade, rollback, health, and recovery checks for
|
|
Mosaic framework configuration under `MOSAIC_HOME`. It deliberately does not
|
|
turn the repository's deployment or PostgreSQL material into an operative
|
|
procedure.
|
|
|
|
## Support boundary
|
|
|
|
Use this runbook only when all of the following are true:
|
|
|
|
- `mosaic` resolves to the installed Mosaic CLI (`command -v mosaic`).
|
|
- The active application configuration is the local tier: `tier: "local"`,
|
|
`storage.type: "pglite"`, and `queue.type: "local"`.
|
|
- `DATABASE_URL` is unset. An inherited PostgreSQL DSN is outside this route and
|
|
must be removed before continuing.
|
|
- `MOSAIC_STORAGE_TIER` is unset or `local`; standalone and federated overrides
|
|
are outside this route.
|
|
- No Gateway, Web, Compose, or other service activation is required.
|
|
|
|
The following routes are **held/non-operative** in this guide:
|
|
|
|
| Route or operation | Status |
|
|
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| PostgreSQL-backed `standalone` storage | Held; do not activate or probe it here. |
|
|
| Federated storage and peers | Held; do not activate or probe it here. |
|
|
| Bare-metal deployment | Held; no service installation or lifecycle action is authorized. |
|
|
| Gateway/Web activation or HTTP health checks | Held; `/health`, `/health/ready`, and `mosaic gateway ...` are not evidence for this route. |
|
|
| Compose startup | Held; do not start a Compose profile or the full stack. |
|
|
| Migration runners and tier migration | Held; do not run `mosaic-db-migrator`, `pnpm --filter @mosaicstack/db db:migrate`, `mosaic storage migrate`, or `mosaic storage migrate-tier`. |
|
|
|
|
A command being present in the CLI does not make a held route operative.
|
|
|
|
## Storage and path terminology
|
|
|
|
Keep these locations separate:
|
|
|
|
- **`MOSAIC_HOME`** — the framework/operator configuration directory. It
|
|
defaults to `~/.config/mosaic` and can be overridden with `MOSAIC_HOME`.
|
|
- **PGlite data** — the local, in-process database. The checked-in local config
|
|
uses `.mosaic/storage-pglite` as its storage `dataDir`; `.mosaic/queue` is the
|
|
local queue directory. These are project data, not framework configuration.
|
|
- **Durable upgrade snapshots** — operator-file snapshots stored under
|
|
`${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/`. They are outside
|
|
`MOSAIC_HOME` and do not contain a PostgreSQL dump.
|
|
|
|
The configuration tier is named `local`; the storage CLI reports the backend as
|
|
`pglite`. PGlite is not PostgreSQL and does not require a PostgreSQL server.
|
|
Local PGlite schema setup is adapter-owned; this page does not authorize a
|
|
separate migration runner.
|
|
|
|
## Pre-upgrade health gate
|
|
|
|
Run these checks from a shell that has no inherited database DSN:
|
|
|
|
```bash
|
|
command -v mosaic
|
|
mosaic --version
|
|
mosaic config path
|
|
test -d "$(mosaic config path)"
|
|
|
|
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
|
|
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
|
|
mosaic restore --list
|
|
```
|
|
|
|
For the supported route, the storage checks should report the `pglite` backend
|
|
and say that no network check is needed. `mosaic storage status` reports
|
|
`PGLITE_DATA_DIR` when that variable is set; otherwise it reports the CLI's
|
|
`:memory:` fallback. That output is an inspection of the CLI environment, not a
|
|
claim that PGlite contents are healthy or durable.
|
|
|
|
If `DATABASE_URL` is set, stop. Do not point it at a local PostgreSQL instance
|
|
to make the check pass. If the active configuration is not local/PGlite, stop;
|
|
no operative route is documented here.
|
|
|
|
The supported health gate is intentionally limited to CLI resolution, framework
|
|
configuration path, local storage selection, and available upgrade snapshots.
|
|
It does not prove Gateway/Web readiness, provider connectivity, queue health, or
|
|
PGlite data integrity.
|
|
|
|
## Safe upgrade procedure
|
|
|
|
1. **Record the baseline.** Save the output of `mosaic --version`,
|
|
`mosaic config path`, and `mosaic restore --list`. Do not copy secrets into a
|
|
ticket or report.
|
|
2. **Check for updates without installing them:**
|
|
|
|
```bash
|
|
mosaic update --check
|
|
```
|
|
|
|
Exit status `0` means no update was reported; status `2` means an update is
|
|
available. Other failures are not a successful health result. The check is a
|
|
package-registry check and does not start Mosaic services or access storage.
|
|
|
|
3. **Run the installed-CLI upgrade when approved:**
|
|
|
|
```bash
|
|
mosaic update
|
|
```
|
|
|
|
The command updates the installed `@mosaicstack/*` packages through npm and,
|
|
when the framework package changed or framework drift is detected, re-seeds
|
|
framework files in keep mode. Leave the default re-seed enabled. Do not use
|
|
`--relaunch` in this runbook: that option can restart durable fleet agents
|
|
and is outside the supported no-activation route. `--no-reseed` is also not a
|
|
normal upgrade path because it intentionally leaves framework files stale.
|
|
|
|
4. **Repeat the health gate.** Confirm the CLI version, the same
|
|
`MOSAIC_HOME`, the local/PGlite storage selection, and the snapshot listing.
|
|
A successful framework upgrade must not be used as evidence that a held
|
|
Gateway, Web, PostgreSQL, or federated route is ready.
|
|
|
|
Do not replace this procedure with a direct edit of `~/.config/mosaic`, a
|
|
repository `tools/install.sh` invocation, a Compose startup, or a migration
|
|
command. The supported operator entry point for this page is the installed
|
|
`mosaic update` command.
|
|
|
|
## What an upgrade protects
|
|
|
|
For an existing keep-mode installation, the framework installer takes two
|
|
separate snapshots before the file sync:
|
|
|
|
1. An ephemeral whole-directory snapshot under `${TMPDIR:-/tmp}/` is used to
|
|
restore the previous framework directory if the sync is interrupted or
|
|
fails. If that restore cannot complete, the installer prints the retained
|
|
temporary snapshot path for manual recovery.
|
|
2. A durable snapshot captures the existing operator-owned files before the
|
|
upgrade at:
|
|
|
|
```text
|
|
${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
|
|
```
|
|
|
|
Snapshot directories are mode `0700`; captured files are mode `0600`.
|
|
Retention is five snapshots by default and can be changed with
|
|
`MOSAIC_BACKUP_RETENTION`. A durable snapshot failure is a warning and does
|
|
not replace the manifest and crash-rollback protections.
|
|
|
|
After the keep-mode sync, the installer compares each captured operator file
|
|
with its target. If a file was changed or removed unexpectedly, it restores the
|
|
snapshot copy and emits a warning. A symlinked parent is not followed; that
|
|
case is left for manual recovery from the snapshot path.
|
|
|
|
Keep mode is manifest-driven and fail-safe for operator paths, but it does not
|
|
mean every file under `MOSAIC_HOME` is user-owned. The framework contract files
|
|
`CONSTITUTION.md`, `AGENTS.md`, and `STANDARDS.md` are refreshed by upgrades;
|
|
keep local policy in the supported local overlays rather than editing those
|
|
contract files as rollback data. A durable snapshot is for operator-owned
|
|
configuration, not for the installed npm package, framework-owned contracts, or
|
|
PGlite data.
|
|
|
|
## Rollback and recovery
|
|
|
|
### If the upgrade fails or is interrupted
|
|
|
|
The framework sync attempts an automatic rollback from its ephemeral snapshot.
|
|
Do not delete `MOSAIC_HOME`, the PGlite data directory, or the temporary snapshot
|
|
named in the error. Preserve the command output, then run the health gate.
|
|
|
|
If the installed framework reports that the automatic restore did not complete,
|
|
use the durable snapshot procedure below. The durable snapshot is the recovery
|
|
pointer that survives a successful upgrade and the removal of the temporary
|
|
snapshot.
|
|
|
|
### Restore operator configuration from a durable snapshot
|
|
|
|
`mosaic restore` is confirmation-gated and reports counts and relative paths,
|
|
not file contents. First list snapshots, then preview the selected timestamp:
|
|
|
|
```bash
|
|
mosaic restore --list
|
|
mosaic restore --from <UTC-timestamp> --dry-run
|
|
```
|
|
|
|
If the preview is correct, run the interactive restore:
|
|
|
|
```bash
|
|
mosaic restore --from <UTC-timestamp>
|
|
```
|
|
|
|
Use `--yes` only when the overwrite has been explicitly approved:
|
|
|
|
```bash
|
|
mosaic restore --from <UTC-timestamp> --yes
|
|
```
|
|
|
|
The timestamp is the value printed by `mosaic restore --list`; the command also
|
|
accepts the full `pre-update-<timestamp>` name. For a non-default configuration
|
|
home, pass the same target explicitly:
|
|
|
|
```bash
|
|
mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>
|
|
```
|
|
|
|
After restoring, repeat the health gate. Restore writes only the operator
|
|
surface represented by that snapshot. It does not downgrade the installed CLI,
|
|
restore framework-owned contract files, restore `.mosaic/storage-pglite`, or
|
|
run any migration.
|
|
|
|
### If local PGlite data is missing or corrupt
|
|
|
|
Do not use a framework snapshot as a database backup. Do not run a migration
|
|
runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate.
|
|
The current CLI does not provide a wired PGlite export/import operation;
|
|
`mosaic storage export` and `mosaic storage import` only print direct-copy
|
|
guidance. Preserve the configured PGlite data directory and route data
|
|
restoration through the held data-layer procedure rather than improvising a
|
|
recursive delete or copy.
|
|
|
|
### Recovery decision tree
|
|
|
|
- **CLI or framework path is wrong:** stop, verify `command -v mosaic`,
|
|
`mosaic --version`, `MOSAIC_HOME`, and `mosaic config path`; do not activate a
|
|
held service route.
|
|
- **Upgrade failed during framework sync:** use the installer's automatic
|
|
rollback first; if it reports incomplete recovery, list and preview the
|
|
durable snapshot, then restore it interactively.
|
|
- **An operator file changed after a reported-successful upgrade:** select the
|
|
pre-upgrade snapshot with `mosaic restore --list`, preview it with `--dry-run`,
|
|
and restore only after reviewing the relative-path plan.
|
|
- **PGlite data is affected:** preserve the data directory and stop at the
|
|
boundary described above. Framework rollback cannot recover database rows.
|
|
- **Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or
|
|
a migration runner:** stop; that is a held/non-operative route, not a next
|
|
command for this runbook.
|
|
|
|
## Post-recovery verification
|
|
|
|
Run the non-mutating checks again:
|
|
|
|
```bash
|
|
mosaic --version
|
|
mosaic config path
|
|
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
|
|
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
|
|
mosaic restore --list
|
|
```
|
|
|
|
A green result here means the installed CLI resolves, the framework path is
|
|
present, the CLI selects local PGlite without a network probe, and snapshots can
|
|
be enumerated. It does not certify database contents or any held deployment
|
|
route.
|