stack/docs/scratchpads/791-upgrade-config-protection.md

Scratchpad — #791 Upgrade config protection (ms-791 worker lane)

Lane: web1:ms-791 → reports to MS-LEAD (web1:mosaic-100). Do NOT contact Jason/Mos directly. Worktree: /home/hermes/agent-work/stack-agents-dir-791, branch feat/791-upgrade-config-protection off origin/main 9745bc3f (verified exact head).

Mission prompt (verbatim intent)

Protect operator-owned config under ~/.config/mosaic from framework-upgrade wipes. Ratified combination (Mos-approved, do NOT re-litigate): (b) strict ownership separation [PRIMARY] + (a) transactional pre-update snapshot [safety net] + (d) regeneration-from-SSOT [recovery]. (c) periodic timer DEFERRED. HARD GATE: unit test that an upgrade run touches NO path outside the manifest. Design-first: write design doc, send to MS-LEAD, WAIT for confirmation before impl.

Session 1 (2026-07-16) — Phase 1 design

Evidence gathered (wipe mechanism, file/line)

• mosaic update → update-checker.ts:509 buildReseedCommand → bash install.sh (MOSAIC_SYNC_ONLY=1, MOSAIC_INSTALL_MODE=keep).
• Wipe = packages/mosaic/framework/install.sh:199 rsync -a --delete + PRESERVE_PATHS denylist (install.sh:47). cp-fallback install.sh:223 find ... -exec rm -rf.
• Denylist gaps → WIPED: agents/*.conf, policy/*.md, *.local.md, harvester/SOP, tools/_lib/credentials.json.
• Stale comment update-checker.ts:492 claims *.local preserved — PRESERVE_PATHS has no such entry.
• TS path file-adapter.ts:157 → file-ops.ts:66 syncDirectory = non-destructive copy-overlay, BUT its preserve list (file-adapter.ts:164) already DRIFTED from install.sh (missing fleet/backlog, fleet/roles.local). Evidence for single shared manifest SSOT.
• Existing snapshot (install.sh:76) = /tmp, crash-trap only, deleted on success → inadequate; no mosaic restore.
• fleet-reconciler.ts:93,234 already has regenerate-projections-from-roster phase separate from lifecycle → mosaic fleet regen = thin projection-only wrapper (no restart), no FCM-M4/M5 preemption.

Design decisions

• (b) Invert to allow-list: shared framework/framework-manifest.json (framework globs + operatorReserved carve-outs); resolve per-path, deny-wins; UNKNOWN ⇒ operator (fail-safe). Mechanism: drop --delete; non-deleting bulk copy + explicit manifest-scoped prune pass (iterate framework globs only → operator/unknown structurally unreachable). Pure prune-planner fn for tests.
• (a) Snapshot to ~/.local/state/mosaic/backups/pre-update-<ts>/ 0700/0600, retention N=5, post-sync verify+restore, mosaic restore --list/--from. No secret values in output.
• (d) mosaic fleet regen projection-only, preview-first, never restart.
• HARD GATE test includes a deliberately-unanticipated operator path to prove fail-safe default.
• PR split: PR1 manifest+guard (root fix, ships alone) → PR2 snapshot/restore (secrev) → PR3 regen+docs. PR2/PR3 depend on PR1.

Status

Design doc written: docs/design/791-upgrade-config-protection.md. Sent to MS-LEAD.

Session 1 (cont.) — MS-LEAD CONFIRMED → Phase 2 GO

All 4 asks approved. Binding conditions:

• TDD tests-first, red-first proof per PR; ≥85% new-code; co-located *.spec.ts; never --no-verify.
• HARD GATE test (§2.4, unanticipated sentinel survives byte-identical + mtime unchanged) = MERGE-BLOCKING for PR1.
• Manifest-completeness test (§6.2) required.
• Bash+TS read ONE shared framework-manifest.json; parity test (§6.1) required (closes #631 drift class).
• UNKNOWN⇒operator (rule 3) non-negotiable. Keep prune-planner PURE.
• fleet regen: NEVER restart; dry-run default, --write to apply; "never issues restart" test mandatory.
• Independent review every PR; PR2 dedicated secrev.
• One PR at a time through DAG. Report PR1 exact head + red→green evidence for review commission.

Now: implementing PR1 (manifest + resolver + non-deleting sync + scoped prune + guard tests).

Session 2 (2026-07-16) — PR1 built, tests-first, red→green proven

Deviation noted to MS-LEAD in PR: manifest is framework-manifest.txt (line-oriented), NOT .json. Rationale: keep the bash installer free of a python3/jq dependency. The "ONE shared file, parity- tested" requirement is honored — manifest-parity.spec.ts drives the bash resolver as a subprocess and asserts byte-identical ownership vs the TS resolver over 34 probe paths spanning every class.

PR1 artifacts

• SSOT: packages/mosaic/framework/framework-manifest.txt ([framework]/[operator], deny-wins, fail-safe).
• TS resolver: src/framework/manifest.ts (pure: parse/matchGlob/resolveOwnership/frameworkSubtreeRoots/ planPrune) + manifest.spec.ts (18 tests incl. planPrune property test + §6.2 completeness).
• Bash resolver: framework/tools/_lib/manifest.sh (compiled globs → fork-free manifest_is_framework; CLI resolve|subtree-roots|classify). Sourced by install.sh.
• HARD GATE (§2.4): framework/tools/quality/scripts/test-upgrade-manifest-guard.sh — keep-mode reseed, 10 operator sentinels (incl. unanticipated unknown-operator-dir/x, harvester/sop.md, fleet/my-fleet.yaml) survive byte-identical + mtime-unchanged; retired framework file pruned; secret value absent from output. RED=31 fail (orig install.sh) → GREEN=48 pass (fixed).
• install.sh: keep mode now manifest-driven (sync_framework_keep, no --delete); overwrite unchanged. PRESERVE_PATHS denylist deleted.
• TS sync: file-ops.syncDirectory gains isOperatorOwned guard; file-adapter.syncFramework derives it from loadManifest — hardcoded (drifted) preservePaths deleted. Fixture uses the REAL manifest.
• Parity: manifest-parity.spec.ts (§6.1) — bash↔TS agree on 34 paths + subtree roots.
• Migration matrix test-install-migration.sh: F6 flipped — my-fleet.yaml now MUST survive (fail-safe).
• CI: new merge-blocking upgrade-guard step (.woodpecker/ci.yml) runs both bash suites (adds rsync).
• update-checker.ts reseed comment corrected to the manifest model.

Gates (all green)

• pnpm typecheck ✓ · pnpm lint ✓ · pnpm format:check ✓
• Full mosaic vitest: 1062 passed (cli-smoke needs pnpm build first — build-artifact dep, not this change).
• HARD GATE 48/48 · migration 21/21 · parity 3/3 · manifest 18/18 · file-adapter 8/8.

PR opened + reported (2026-07-16)

• PR #802 #802 — base main@9745bc3f, head 34e55d4a (commit feat(mosaic): manifest-owned upgrade guard…). 15 files, +1160/-142.
• Reported PR head + red→green evidence to MS-LEAD (web1:mosaic-100); queued (lead busy). Standing by for the independent-review commission at head 34e55d4a.
• TWO items flagged to MS-LEAD for decision (awaiting reply):
  1. Deviation .txt vs .json — confirm accept (parity-tested) or convert to .json+jq.
  2. pr-create -i 791 appended Fixes #791 → would auto-close the tracking issue on PR1 merge while PR2/PR3 remain. Recommended edit to Part of #791; awaiting go-ahead to patch PR body.
• DO NOT start PR2/PR3 until PR1 merges (DAG; one PR at a time).

MS-LEAD ruling → #797 ledger-survival sentinel folded into PR1 (2026-07-16)

MS-LEAD ruled both my decisions: (1) .txt format ACCEPTED (parity must be strict/merge-blocking incl. format edge cases + negative probe); (2) trailer Fixes #791→Part of #791 APPROVED (patched PR #802 body via Gitea API — tracking issue no longer auto-closes on PR1 merge). Plus Mos-ELEVATED merge-blocker (spec ~/agent-work/planning/epic-796/791-ledger-survival-sentinel-SPEC.md): #797 Runtime Session Ledger must survive upgrade. Two coupled deliverables landed in PR1:

• (i) Carve-out: fleet/run/** was ALREADY an explicit [operator] entry — glob matches the spec's pinned fleet/run/** EXACTLY, so NO divergence to route back to planner-opus. Strengthened its comment to name the ledger (fleet/run/sessions/ events.ndjson + ledger.json) so it is unmistakably load-bearing.
• (ii) HARD-GATE sentinel: seeded populated ledger (events.ndjson 3 events + ledger.json node+edge+gen, 0600 under 0700) into test-upgrade-manifest-guard.sh sentinels; asserts byte-identical + mtime-unchanged
  • dir-perms unchanged. Negative control (retired framework file IS pruned) relabeled explicitly. HARD GATE now 58/58 (was 48).
• Decision-1 parity hardening: format-edge fixtures (comments/blanks/whitespace, duplicate+overlapping globs deny-wins, section/glob-ordering independence) + explicit UNKNOWN→operator negative probe, driven through BOTH resolvers via MANIFEST_FILE override. Parity 7/7 (was 3).
• RED-FIRST honesty note: the bash ledger sentinel stays GREEN even against the pre-fix installer (the ledger was incidentally safe from the rsync --delete bug; overall pre-fix run 30/58 as expected). The carve-out's TRUE load-bearing value (deny-wins if framework ownership ever broadens to fleet/**) is isolated by a dedicated resolver-seam red→green in manifest.spec.ts: WITHOUT fleet/run/** operator entry + hypothetical fleet/** framework → ledger resolves framework and planPrune DELETES it (RED); WITH the carve-out → deny-wins → operator, unprunable (GREEN). manifest.spec.ts 21/21 (was 18).
• Gates all green: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1069 passed · HARD GATE 58/58 · migration 21/21. Committing FORWARD on the branch (NOT rebasing 34e55d4a out from under review).

MS-LEAD REQUEST CHANGES @ 0a5e703a → B1/B2/B3 fixed red-first (2026-07-16)

MS-LEAD returned REQUEST CHANGES (routed merge-blockers satisfied; 2 CRITICAL reliability defects from the commissioned independent review). Fixed forward on the branch, red-first:

• B1 (CRITICAL) — dead ERR trap. install.sh had set -euo pipefail (no -E), so the trap restore_snapshot ERR never fired for a failure inside sync_framework_keep() (function body) — a mid-sync abort left a half-written target with NO rollback. Fix: set -Eeuo pipefail (errtrace) + disarm the trap at the top of restore_snapshot() to prevent re-entrancy. New gate test-upgrade-rollback.sh: injects a mid-sync cp EACCES (read-only divergent framework file); Part A asserts the shipped installer rolls back (restore message fires AND target byte-identical to pre-upgrade); Part B control strips -E and asserts the rollback message does NOT fire (dead trap) — self-verifying red→green. 7/7.
• B2/B3 (CRITICAL) — empty/unreadable/malformed manifest divergence. Pre-fix: TS parseManifest('') returned {framework:[],operator:[]} (NO throw) → silent no-op "Installation complete"; bash aborted fragilely (the _manifest_compile "${MANIFEST_OPERATOR[@]:-}" artifact returned 1 with no message) AND the CLI dispatch swallowed manifest_load's rc (no || exit) so resolve exited 0 resolving everything operator. Fix (fail-loud + identical both langs):
  • TS parseManifest: throw on zero framework entries; loadManifest: wrap read error → "Cannot read framework manifest …".
  • bash manifest_load: explicit unreadable guard ([[ ! -r ]]) + zero-[framework] guard, both loud stderr + return 1; _manifest_compile gets explicit return 0 (kills the empty-array artifact); CLI dispatch manifest_load … || exit 1.
  • finalize.ts: wrap syncFramework → spin.stop('Framework sync aborted …') + rethrow (never falls through to "Installation complete"). Tests: manifest.spec.ts +5 fail-closed (empty/comment-only/operator-only/empty-section/missing); manifest-parity.spec.ts +7 failure-mode parity (both reject empty/comment-only/operator-only/ empty-section/entry-before-header/unknown-header/missing — TS throws, bash CLI exits non-zero+stderr); HARD GATE +4 end-to-end fail-closed matrices (empty/operator-only/malformed/missing → abort non-zero, manifest error surfaced, every operator sentinel byte-identical). RED proven by reverting manifest.ts+manifest.sh to HEAD → 12 new tests fail; restore → 40/40 green.
• Non-blocking addressed. MEDIUM install.sh:222 find-empty now warns on a real failure instead of blanket || true. LOW: corrected the "both destructive paths rsync vs cp" overstatement in the HARD GATE header + cp-fallback comment + ci.yml (keep mode is a single cp-based path; the rsync-present vs -absent runs prove rsync-independence). .pre-constitution.bak triage: single-shot backup is intentional (reconcile_framework_files backs up once), no change.
• Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1081 passed (was 1069, +12) · HARD GATE 118/118 (was 58) · rollback 7/7 (new) · migration 21/21. No --no-verify. Rollback test wired into ci.yml upgrade-guard. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).

Codex round 2 (pre-push self-review) → blockers A/B + should-fix C fixed red-first (2026-07-16)

Before committing round 1 I re-ran codex on the change set; it surfaced two fresh reliability defects and one messaging defect on the SAME rollback/manifest path. Fixed forward, red-first:

• Blocker-A (CRITICAL) — signal trap resumed instead of terminating. A bash INT/TERM handler that merely restore_snapshot (returns) does NOT terminate the script — execution RESUMES past the interrupt, cleans the snapshot and reports success, leaving a partial post-interrupt update. Fix: trap 'restore_snapshot; exit 1' ERR INT TERM so both the errtrace (ERR) and signal (INT/TERM) paths exit non-zero. Rollback test Part C: a cp shim that kill -TERM $PPID mid-sync then succeeds (so set -e never fires and only the signal path governs) → asserts abort non-zero + restore fires + does NOT print "file phase complete"; control strips exit 1 and asserts the buggy resume-to-success.
• Blocker-B (CRITICAL) — degenerate [framework] section resolved everything operator. A manifest whose framework entries are all empty / bare-dot (/, ./, ., ..) passed the non-empty guard yet yielded zero usable globs → nothing is framework → a keep-mode sync silently no-ops (bash resolved operator, exit 0). Fix (both langs, parity): reject when no entry has a char other than //. — TS isUsableFrameworkGlob = /[^/.]/.test(normalizeRel(glob)), throws ManifestError; bash mirror loops [[ "$(_manifest_norm "$_g")" =~ [^/.] ]], loud stderr + return 1. Tests: manifest.spec.ts it.each(['/','./','.','..','/\n./']) throw; parity +3 expectBothReject (root-slash/dot-slash/ bare-dot). RED: reverting the guard makes [framework]\n/ resolve operator exit 0.
• Should-fix-C — misleading abort message. finalize.ts printed one generic "may be partially applied" for every sync failure. A ManifestError is a PRE-sync validation abort (manifest is validated before any copy) → nothing was written; conflating it with a mid-copy failure misdirects recovery. Fix: introduce ManifestError (exported from manifest.ts, thrown by every fail-closed parse/load path), and classify in finalize.ts — ManifestError → "no files were changed"; any other → "may be partially applied". New co-located finalize-sync-abort.spec.ts (3 tests) asserts both branches re-throw the original error + the correct message, and that config writes are never reached. RED proven by collapsing the classification → the ManifestError test fails.
• Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 (was 1081, +3 finalize-abort; manifest specs already counted) · HARD GATE 193/193 · rollback 14/14 · migration 21/21.

Codex round 3 (pre-push self-review) → blockers D1/D2 fixed red-first (2026-07-16)

Re-ran codex again; it found two more rollback-path gaps set -E cannot catch. Fixed forward, red-first:

• Blocker-D1 (CRITICAL) — find scan failures swallowed by process substitution. Both the overlay copy and the scoped prune consumed < <(find … -print0). Bash does NOT propagate the producer's exit status to the while, so an EACCES/I/O failure mid-scan truncates the file list yet leaves the loop exiting 0 → a partial upgrade commits and reports success; the ERR/restore trap never fires. Fix: _scan_or_die runs find … -print0 > "$tmp" to completion, checks its status, and returns non-zero (→ ERR trap → restore) on failure; both loops now read from the checked temp file. Rollback test Part D: a find shim that fails every -print0 scan → shipped installer aborts non-zero + restores + emits "Could not enumerate framework files" + target byte-identical; control neuters the # D1-GUARD return 1 → find failure swallowed, upgrade wrongly reports "file phase complete", no rollback.
• Blocker-D2 (CRITICAL) — silent set -e exit on a failed target reset. restore_snapshot did a bare rm -rf "$TARGET_DIR"; mkdir -p "$TARGET_DIR" (trap disarmed, under set -e). If rm/mkdir fails — possibly after rm deleted part of the target — the script exits immediately, skipping the cp AND the recovery pointer, leaving a half-removed target and an orphaned snapshot the operator can't locate. Fix: if ! rm -rf … || ! mkdir -p …; then fail "Snapshot restore could not reset … preserved at: $SNAPSHOT_DIR — copy it back …"; return 1; fi (tested like the cp -a check; snapshot NOT deleted). Rollback test Part E: cp-poison triggers restore + an rm shim fails rm -rf <TARGET> → shipped emits the recovery pointer, the named snapshot dir survives, secret value never leaked; control deletes the recovery line → operator gets no pointer. RED: reverting D1+D2 → 7 shipped/control assertions fail.
• Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 · HARD GATE 193/193 · rollback 28/28 (was 14, +14 for D1/D2 with controls) · migration 21/21. shellcheck clean on new lines. No --no-verify. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).