fix(fleet): harden #791 upgrade rollback against find/reset failures
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Second- and third-round independent-review reliability fixes on the keep-mode
upgrade rollback path, plus accurate abort messaging. All fixed red-first with
self-verifying controls in the rollback gate.

Round 2 (blockers A/B, should-fix C):
- install.sh: `trap 'restore_snapshot; exit 1' ERR INT TERM` so an INT/TERM
  mid-sync terminates instead of resuming past the interrupt and reporting
  success (a bash signal handler that only returns does not terminate).
- manifest.{ts,sh}: reject a degenerate [framework] section whose entries are
  all empty or bare-dot (`/`, `./`, `.`, `..`) — it passed the non-empty guard
  yet yielded zero usable globs, silently resolving everything to operator.
  Parity via a shared `[^/.]` usable-glob test; TS throws ManifestError.
- finalize.ts: classify the sync-abort message — a ManifestError is a pre-sync
  validation abort ("no files were changed"); any other error may be partial.

Round 3 (blockers D1, D2):
- install.sh: enumerate framework files with a checked temp file (_scan_or_die)
  instead of `< <(find …)` — process substitution discards find's exit status,
  so an EACCES/I/O failure mid-scan would truncate the file list yet leave the
  loop exiting 0, committing a partial upgrade as success (ERR trap never fires).
- install.sh: guard the `rm -rf; mkdir -p` target reset inside restore_snapshot
  — a bare reset failing under set -e exits silently after partial deletion,
  never printing the snapshot-recovery pointer. Now checked like the cp -a
  restore: on failure it preserves the snapshot and tells the operator where.

Tests: rollback gate 14→28 (Parts C/D/E with disabled-guard controls);
new finalize-sync-abort.spec.ts (3). No secret value is ever emitted; snapshots
stay 0700. Gates green: typecheck, lint, format:check, full mosaic vitest 1094,
HARD GATE 193, rollback 28, migration 21.

Refs #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-16 17:30:19 -05:00
parent 0a5e703a70
commit af627e7583
11 changed files with 1003 additions and 26 deletions

View File

@@ -123,3 +123,94 @@ must survive upgrade. Two coupled deliverables landed in PR1:
WITH the carve-out → deny-wins → operator, unprunable (GREEN). manifest.spec.ts 21/21 (was 18).
- Gates all green: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1069 passed · HARD GATE 58/58
· migration 21/21. Committing FORWARD on the branch (NOT rebasing 34e55d4a out from under review).
### MS-LEAD REQUEST CHANGES @ 0a5e703a → B1/B2/B3 fixed red-first (2026-07-16)
MS-LEAD returned REQUEST CHANGES (routed merge-blockers satisfied; 2 CRITICAL reliability defects from
the commissioned independent review). Fixed forward on the branch, red-first:
- **B1 (CRITICAL) — dead ERR trap.** install.sh had `set -euo pipefail` (no `-E`), so the
`trap restore_snapshot ERR` never fired for a failure inside sync_framework_keep() (function body) —
a mid-sync abort left a half-written target with NO rollback. Fix: `set -Eeuo pipefail` (errtrace) +
disarm the trap at the top of restore_snapshot() to prevent re-entrancy. New gate
`test-upgrade-rollback.sh`: injects a mid-sync `cp` EACCES (read-only divergent framework file);
Part A asserts the shipped installer rolls back (restore message fires AND target byte-identical to
pre-upgrade); Part B control strips `-E` and asserts the rollback message does NOT fire (dead trap) —
self-verifying red→green. 7/7.
- **B2/B3 (CRITICAL) — empty/unreadable/malformed manifest divergence.** Pre-fix: TS `parseManifest('')`
returned `{framework:[],operator:[]}` (NO throw) → silent no-op "Installation complete"; bash aborted
fragilely (the `_manifest_compile` `"${MANIFEST_OPERATOR[@]:-}"` artifact returned 1 with no message)
AND the CLI dispatch swallowed manifest_load's rc (no `|| exit`) so `resolve` exited 0 resolving
everything operator. Fix (fail-loud + identical both langs):
* TS `parseManifest`: throw on zero framework entries; `loadManifest`: wrap read error →
"Cannot read framework manifest …".
* bash `manifest_load`: explicit unreadable guard (`[[ ! -r ]]`) + zero-`[framework]` guard, both loud
stderr + return 1; `_manifest_compile` gets explicit `return 0` (kills the empty-array artifact);
CLI dispatch `manifest_load … || exit 1`.
* `finalize.ts`: wrap syncFramework → `spin.stop('Framework sync aborted …')` + rethrow (never falls
through to "Installation complete").
Tests: manifest.spec.ts +5 fail-closed (empty/comment-only/operator-only/empty-section/missing);
manifest-parity.spec.ts +7 failure-mode parity (both reject empty/comment-only/operator-only/
empty-section/entry-before-header/unknown-header/missing — TS throws, bash CLI exits non-zero+stderr);
HARD GATE +4 end-to-end fail-closed matrices (empty/operator-only/malformed/missing → abort non-zero,
manifest error surfaced, every operator sentinel byte-identical). RED proven by reverting
manifest.ts+manifest.sh to HEAD → 12 new tests fail; restore → 40/40 green.
- **Non-blocking addressed.** MEDIUM install.sh:222 find-empty now warns on a real failure instead of
blanket `|| true`. LOW: corrected the "both destructive paths rsync vs cp" overstatement in the HARD
GATE header + cp-fallback comment + ci.yml (keep mode is a single cp-based path; the rsync-present vs
-absent runs prove rsync-independence). `.pre-constitution.bak` triage: single-shot backup is
intentional (reconcile_framework_files backs up once), no change.
- Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1081 passed (was 1069, +12) · HARD GATE
118/118 (was 58) · rollback 7/7 (new) · migration 21/21. No --no-verify. Rollback test wired into
ci.yml upgrade-guard. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).
### Codex round 2 (pre-push self-review) → blockers A/B + should-fix C fixed red-first (2026-07-16)
Before committing round 1 I re-ran codex on the change set; it surfaced two fresh reliability defects
and one messaging defect on the SAME rollback/manifest path. Fixed forward, red-first:
- **Blocker-A (CRITICAL) — signal trap resumed instead of terminating.** A bash INT/TERM handler that
merely `restore_snapshot` (returns) does NOT terminate the script — execution RESUMES past the
interrupt, cleans the snapshot and reports success, leaving a partial post-interrupt update. Fix:
`trap 'restore_snapshot; exit 1' ERR INT TERM` so both the errtrace (ERR) and signal (INT/TERM) paths
exit non-zero. Rollback test Part C: a `cp` shim that `kill -TERM $PPID` mid-sync then succeeds (so
set -e never fires and only the signal path governs) → asserts abort non-zero + restore fires + does
NOT print "file phase complete"; control strips `exit 1` and asserts the buggy resume-to-success.
- **Blocker-B (CRITICAL) — degenerate `[framework]` section resolved everything operator.** A manifest
whose framework entries are all empty / bare-dot (`/`, `./`, `.`, `..`) passed the non-empty guard yet
yielded zero usable globs → nothing is framework → a keep-mode sync silently no-ops (bash resolved
`operator`, exit 0). Fix (both langs, parity): reject when no entry has a char other than `/`/`.`
TS `isUsableFrameworkGlob` = `/[^/.]/.test(normalizeRel(glob))`, throws `ManifestError`; bash mirror
loops `[[ "$(_manifest_norm "$_g")" =~ [^/.] ]]`, loud stderr + return 1. Tests: manifest.spec.ts
`it.each(['/','./','.','..','/\n./'])` throw; parity +3 `expectBothReject` (root-slash/dot-slash/
bare-dot). RED: reverting the guard makes `[framework]\n/` resolve `operator` exit 0.
- **Should-fix-C — misleading abort message.** finalize.ts printed one generic "may be partially
applied" for every sync failure. A `ManifestError` is a PRE-sync validation abort (manifest is
validated before any copy) → nothing was written; conflating it with a mid-copy failure misdirects
recovery. Fix: introduce `ManifestError` (exported from manifest.ts, thrown by every fail-closed
parse/load path), and classify in finalize.ts — ManifestError → "no files were changed"; any other →
"may be partially applied". New co-located `finalize-sync-abort.spec.ts` (3 tests) asserts both
branches re-throw the original error + the correct message, and that config writes are never reached.
RED proven by collapsing the classification → the ManifestError test fails.
- Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 (was 1081, +3 finalize-abort;
manifest specs already counted) · HARD GATE 193/193 · rollback 14/14 · migration 21/21.
### Codex round 3 (pre-push self-review) → blockers D1/D2 fixed red-first (2026-07-16)
Re-ran codex again; it found two more rollback-path gaps `set -E` cannot catch. Fixed forward, red-first:
- **Blocker-D1 (CRITICAL) — `find` scan failures swallowed by process substitution.** Both the overlay
copy and the scoped prune consumed `< <(find … -print0)`. Bash does NOT propagate the producer's exit
status to the `while`, so an EACCES/I/O failure mid-scan truncates the file list yet leaves the loop
exiting 0 → a partial upgrade commits and reports success; the ERR/restore trap never fires. Fix:
`_scan_or_die` runs `find … -print0 > "$tmp"` to completion, checks its status, and returns non-zero
(→ ERR trap → restore) on failure; both loops now read from the checked temp file. Rollback test
Part D: a `find` shim that fails every `-print0` scan → shipped installer aborts non-zero + restores +
emits "Could not enumerate framework files" + target byte-identical; control neuters the `# D1-GUARD`
`return 1` → find failure swallowed, upgrade wrongly reports "file phase complete", no rollback.
- **Blocker-D2 (CRITICAL) — silent `set -e` exit on a failed target reset.** restore_snapshot did a bare
`rm -rf "$TARGET_DIR"; mkdir -p "$TARGET_DIR"` (trap disarmed, under set -e). If `rm`/`mkdir` fails —
possibly after `rm` deleted part of the target — the script exits immediately, skipping the cp AND the
recovery pointer, leaving a half-removed target and an orphaned snapshot the operator can't locate.
Fix: `if ! rm -rf … || ! mkdir -p …; then fail "Snapshot restore could not reset … preserved at:
$SNAPSHOT_DIR — copy it back …"; return 1; fi` (tested like the cp -a check; snapshot NOT deleted).
Rollback test Part E: cp-poison triggers restore + an `rm` shim fails `rm -rf <TARGET>` → shipped
emits the recovery pointer, the named snapshot dir survives, secret value never leaked; control deletes
the recovery line → operator gets no pointer. RED: reverting D1+D2 → 7 shipped/control assertions fail.
- Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 · HARD GATE 193/193 ·
rollback 28/28 (was 14, +14 for D1/D2 with controls) · migration 21/21. shellcheck clean on new lines.
No --no-verify. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).