feat(mosaic): durable pre-update snapshot + verify net + restore CLI (#791 PR2) #811

2026-07-17T00:05:53Z

jason.woltje commented

2026-07-17 00:05:53 +00:00

Summary

Part of #791. Adds the third, independent recovery layer for keep-mode framework upgrades under ~/.config/mosaic.

PR1 (#802) gave keep-mode two protections: the manifest (a keep-sync only ever writes framework-owned paths — operator config is structurally untouched) and an ephemeral /tmp snapshot that rolls the whole target back if the sync crashes mid-write. This PR adds the case neither covers: a "successful" upgrade that a manifest/logic bug silently let touch an operator file.

What lands

Durable pre-update snapshot (install.sh make_durable_snapshot): before any mutation, copy exactly the operator-owned surface that exists into ${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-ts>/ — outside ~/.config/mosaic and any repo. 0700 dirs / 0600 files (umask 077 scoped and restored). Fail-open (a backup failure never aborts the upgrade it protects). Retention MOSAIC_BACKUP_RETENTION (default 5).
Post-sync verify net (install.sh verify_operator_surface): cmp every snapshot file against its target counterpart; restore any operator file the upgrade diverged/removed and warn loudly (a divergence = manifest bug). Honors intentional migration removals; refuses to restore through a symlink; guards mkdir under set -e.
mosaic restore (restore.ts): --list (default, dry-run) enumerates snapshots by timestamp; --from <ts> restores over the operator surface, confirmation-gated (--yes / MOSAIC_ASSUME_YES), --dry-run. Reports counts and relative paths only — never emits file contents (secrev).

Security (both classes fixed with regression tests)

CWE-59 (symlink write-through): an attacker swapping an operator path for a symlink after the snapshot must not cause restore/verify to write the snapshot's secret out through the link. Bash refuses a symlinked ancestor and drops a symlinked leaf; TS reuses the audited secure-file.ts helpers (assertCanonicalContainment + ensureManagedDirectory) and opens the leaf O_NOFOLLOW|O_CREAT|O_TRUNC 0600 (ELOOP = fail-closed).
CWE-22 (traversal): --from is validated against ^\d{8}T\d{6}Z(?:-\d+)?$ and resolves to exactly join(root, 'pre-update-'+ts) (lstat, reject symlinked snap dir).

Review round 1

Codex code-review (request-changes: 1 blocker + 3 should-fix) and security-review (high: 1 high + 1 medium) both ran on the diff. All 5 deduped defects were fixed forward, each with a red-first regression test whose control neuters exactly the guard under test (verify-net migration skip, symlink leaf guard, umask restore). Independent (author ≠ reviewer) review + a durable Gitea Reviewer-of-Record comment are still required before merge — codex self-review does not satisfy that gate.

Gates

Gate	Result
typecheck / lint / format:check	✓
mosaic vitest	1252 passed
restore.spec.ts	30 passed
bash: durable-snapshot	41 passed
bash: manifest-guard	193 passed
bash: rollback	28 passed
bash: install-migration	21 passed

CI runs the new test-upgrade-durable-snapshot.sh in the upgrade-guard step.

## Summary Part of #791. Adds the **third, independent recovery layer** for keep-mode framework upgrades under `~/.config/mosaic`. PR1 (#802) gave keep-mode two protections: the **manifest** (a keep-sync only ever writes framework-owned paths — operator config is structurally untouched) and an **ephemeral `/tmp` snapshot** that rolls the whole target back if the sync *crashes* mid-write. This PR adds the case neither covers: a **"successful" upgrade that a manifest/logic bug silently let touch an operator file.** ## What lands - **Durable pre-update snapshot** (`install.sh` `make_durable_snapshot`): before any mutation, copy exactly the operator-owned surface that exists into `${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-ts>/` — outside `~/.config/mosaic` and any repo. `0700` dirs / `0600` files (`umask 077` scoped **and restored**). **Fail-open** (a backup failure never aborts the upgrade it protects). Retention `MOSAIC_BACKUP_RETENTION` (default 5). - **Post-sync verify net** (`install.sh` `verify_operator_surface`): `cmp` every snapshot file against its target counterpart; restore any operator file the upgrade diverged/removed and warn loudly (a divergence = manifest bug). Honors intentional migration removals; refuses to restore through a symlink; guards `mkdir` under `set -e`. - **`mosaic restore`** (`restore.ts`): `--list` (default, dry-run) enumerates snapshots by timestamp; `--from <ts>` restores over the operator surface, confirmation-gated (`--yes` / `MOSAIC_ASSUME_YES`), `--dry-run`. Reports **counts and relative paths only** — never emits file contents (secrev). ## Security (both classes fixed with regression tests) - **CWE-59 (symlink write-through):** an attacker swapping an operator path for a symlink after the snapshot must not cause restore/verify to write the snapshot's secret out through the link. Bash refuses a symlinked ancestor and drops a symlinked leaf; TS reuses the audited `secure-file.ts` helpers (`assertCanonicalContainment` + `ensureManagedDirectory`) and opens the leaf `O_NOFOLLOW|O_CREAT|O_TRUNC` `0600` (ELOOP = fail-closed). - **CWE-22 (traversal):** `--from` is validated against `^\d{8}T\d{6}Z(?:-\d+)?$` and resolves to exactly `join(root, 'pre-update-'+ts)` (`lstat`, reject symlinked snap dir). ## Review round 1 Codex code-review (request-changes: 1 blocker + 3 should-fix) and security-review (high: 1 high + 1 medium) both ran on the diff. All 5 deduped defects were fixed **forward**, each with a red-first regression test whose control neuters exactly the guard under test (verify-net migration skip, symlink leaf guard, umask restore). Independent (author ≠ reviewer) review + a durable Gitea Reviewer-of-Record comment are still required before merge — codex self-review does not satisfy that gate. ## Gates | Gate | Result | |------|--------| | typecheck / lint / format:check | ✓ | | mosaic vitest | 1252 passed | | restore.spec.ts | 30 passed | | bash: durable-snapshot | 41 passed | | bash: manifest-guard | 193 passed | | bash: rollback | 28 passed | | bash: install-migration | 21 passed | CI runs the new `test-upgrade-durable-snapshot.sh` in the `upgrade-guard` step.

jason.woltje added 1 commit 2026-07-17 00:05:54 +00:00

feat(mosaic): durable pre-update snapshot + verify net + restore CLI (#791 PR2)

ci/woodpecker/pr/ci Pipeline failed

Details

8bee1b65ea

Third, independent recovery layer for keep-mode upgrades (Part of #791). PR1's
manifest keeps the sync out of operator paths and the ephemeral /tmp snapshot
rolls back a mid-write crash; this adds the case neither covers — a "successful"
upgrade that a manifest/logic bug silently let touch an operator file.

- install.sh make_durable_snapshot(): before any mutation, copy the operator-owned
  surface that exists to $XDG_STATE_HOME/mosaic/backups/pre-update-<UTC-ts>/
  (0700 dirs / 0600 files, umask 077 scoped + restored). Fail-open; retention
  MOSAIC_BACKUP_RETENTION (default 5).
- install.sh verify_operator_surface(): post-sync, restore any operator file the
  upgrade diverged/removed and warn loudly (a divergence = manifest bug). Honors
  intentional migration removals; refuses to restore through a symlink (CWE-59);
  guards mkdir under set -e.
- mosaic restore (TS): --list (default dry-run), --from <ts> confirmation-gated
  restore. Strict timestamp selector (CWE-22), O_NOFOLLOW 0600 writes (CWE-59),
  containment + managed-dir checks via secure-file.ts. Counts/paths only — never
  emits file contents (secrev).
- Gates: typecheck/lint/format ✓; mosaic vitest 1252; restore.spec 30;
  bash gates durable-snapshot 41 / manifest-guard 193 / rollback 28 / migration 21.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

jason.woltje added 1 commit 2026-07-17 00:25:32 +00:00

test(mosaic): make #791 PR2 Part 7 symlink control busybox-portable (#791 PR2)

ci/woodpecker/pr/ci Pipeline was successful

Details

d12c5f7808

CI #1881 failed at head 8bee1b65: the durable-snapshot gate was 40/41, the sole
failure being the Part 7 NEGATIVE control ("secret leaked through the symlink").
Root cause is a test-harness portability gap, not a code defect: node:24-alpine
runs busybox cp as root, and busybox cp REPLACES a symlinked destination instead
of following it, whereas GNU/BSD cp — what a real operator runs `mosaic update`
under — follows the link and leaks. So under the CI harness the CWE-59 leak vector
the control asserts simply cannot occur, and the negative control can't reproduce.

Fix is test-only; install.sh (independently approved at 8bee1b65) and the real
security assertions are untouched. make_symlink_leaf_shim now emulates GNU cp's
follow-through-dest-symlink behavior portably: when the destination is a symlink it
writes the source bytes through the link via redirection (which follows symlinks on
every coreutils, busybox included); otherwise it delegates to the host cp unchanged.
Both the shipped-case and the control run through this single shim, so the ONLY
difference between them remains the SYMLINK-LEAF-GUARD — the control stays
load-bearing and non-tautological, now on busybox too. With the guard present the
symlinked leaf is dropped before this cp runs, so the shipped path is unchanged.

Verified in the exact CI image (node:24-alpine, busybox, root, apk add bash rsync):
durable-snapshot 41/41, manifest-guard 193, rollback 28. GNU host 41/41, shellcheck
clean. Sole tracked delta vs 8bee1b65 = this test file.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

jason.woltje commented

2026-07-17 00:42:42 +00:00

Reviewer-of-Record (RoR) — exact head `d12c5f7808`

VERDICT: APPROVE — secure & mergeable on security/correctness grounds. (Durable PR comment: Gitea blocks a formal APPROVE review by the PR author account under the single-account setup, so this comment is the durable exact-head RoR artifact.)

Author lane: ms-791 · Reviewer lane: independent Opus adversarial/security reviewer (dispatched by MS Team Lead).
Attestation: author != reviewer is PROCESS/LANE-attested — all Gitea activity under single account jason.woltje; separation is by work lane, not provider account.

CI: Woodpecker #1882 = terminal SUCCESS at this exact head (Gitea combined commit-status = success); all 8 steps green.
Scope: CLEAN — git diff --stat 32a0ffba..d12c5f78 = exactly the 8 ratified files; no stray committed secret/snapshot/session.lock.

Findings (all info-level; no blocker/high/med)

CWE-59 (restore.ts:170-184): leaf write guarded — assertCanonicalContainment + ensureManagedDirectory (lstat, rejects symlinked ancestors) + O_WRONLY|O_CREAT|O_TRUNC|O_NOFOLLOW 0o600; swapped symlink leaf fails closed (ELOOP).
CWE-22 (restore.ts:96,155-166): --from matched /^\d{8}T\d{6}Z(?:-\d+)?$/, joined under fixed root, must be real non-symlink dir; spec.ts:320-358 traversal cases fail-closed.
umask (install.sh:232-271): 077 saved + restored on every return path; no global leak.
Migration ordering (install.sh:646->653): VERIFY-NET runs AFTER run_migrations, restore trap disarmed (649); is_migration_removed (136-143,299) prevents healing intentional bin/ removal; Part 6 control.
secrev: restore + all logging emit counts/paths/timestamps/error-messages only; cmp -s / grep -q value-silent; seeded SECRET present in backup yet ABSENT from stdout/stderr (Parts 3/5/7, spec.ts:291-316). PASS.

Part 7 busybox-portability change (test-harness only)

Non-tautological / load-bearing: shipped & control use identical make_symlink_leaf_shim; sole difference = mk_control 'SYMLINK-LEAF-GUARD' stripping install.sh:309 (marker occurs exactly once). Control asserts secret DID leak; shipped asserts it did NOT.
Faithful: shim -L dest && -f src && cat src > dest reproduces GNU cp write-through-symlink (real CWE-59 vector busybox cp does not exhibit); guarded path delegates to real cp identically to a GNU host.
Does NOT touch install.sh or weaken the four real secure-outcome assertions — CONFIRMED at this head.

Gate status -> mergeable

exact-head terminal-green CI OK · fresh independent exact-head RoR APPROVE OK · scope clean OK · based on current main OK · durable RoR persisted to Gitea before queue guard OK.

## Reviewer-of-Record (RoR) — exact head d12c5f78087be749c25833cb702b508749a1335b **VERDICT: APPROVE** — secure & mergeable on security/correctness grounds. (Durable PR comment: Gitea blocks a formal APPROVE review by the PR author account under the single-account setup, so this comment is the durable exact-head RoR artifact.) **Author lane:** ms-791 · **Reviewer lane:** independent Opus adversarial/security reviewer (dispatched by MS Team Lead). **Attestation:** author != reviewer is PROCESS/LANE-attested — all Gitea activity under single account jason.woltje; separation is by work lane, not provider account. **CI:** Woodpecker #1882 = terminal SUCCESS at this exact head (Gitea combined commit-status = success); all 8 steps green. **Scope:** CLEAN — git diff --stat 32a0ffba..d12c5f78 = exactly the 8 ratified files; no stray committed secret/snapshot/session.lock. ### Findings (all info-level; no blocker/high/med) - CWE-59 (restore.ts:170-184): leaf write guarded — assertCanonicalContainment + ensureManagedDirectory (lstat, rejects symlinked ancestors) + O_WRONLY|O_CREAT|O_TRUNC|O_NOFOLLOW 0o600; swapped symlink leaf fails closed (ELOOP). - CWE-22 (restore.ts:96,155-166): --from matched /^\d{8}T\d{6}Z(?:-\d+)?$/, joined under fixed root, must be real non-symlink dir; spec.ts:320-358 traversal cases fail-closed. - umask (install.sh:232-271): 077 saved + restored on every return path; no global leak. - Migration ordering (install.sh:646->653): VERIFY-NET runs AFTER run_migrations, restore trap disarmed (649); is_migration_removed (136-143,299) prevents healing intentional bin/ removal; Part 6 control. - secrev: restore + all logging emit counts/paths/timestamps/error-messages only; cmp -s / grep -q value-silent; seeded SECRET present in backup yet ABSENT from stdout/stderr (Parts 3/5/7, spec.ts:291-316). PASS. ### Part 7 busybox-portability change (test-harness only) - Non-tautological / load-bearing: shipped & control use identical make_symlink_leaf_shim; sole difference = mk_control 'SYMLINK-LEAF-GUARD' stripping install.sh:309 (marker occurs exactly once). Control asserts secret DID leak; shipped asserts it did NOT. - Faithful: shim [[ -L dest && -f src ]] && cat src > dest reproduces GNU cp write-through-symlink (real CWE-59 vector busybox cp does not exhibit); guarded path delegates to real cp identically to a GNU host. - Does NOT touch install.sh or weaken the four real secure-outcome assertions — CONFIRMED at this head. ### Gate status -> mergeable exact-head terminal-green CI OK · fresh independent exact-head RoR APPROVE OK · scope clean OK · based on current main OK · durable RoR persisted to Gitea before queue guard OK.

jason.woltje merged commit 31607a4af6 into main

2026-07-17 00:43:19 +00:00

jason.woltje referenced this issue from a commit

2026-07-17 00:43:20 +00:00

feat(mosaic): durable pre-update snapshot + verify net + restore CLI (#791 PR2) (#811)

jason.woltje referenced this pull request

2026-07-17 02:59:47 +00:00

feat(fleet): mosaic fleet regen — regenerate roster-derived projections (PR3 of #791) #813

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#811