fix(fleet): harden #791 upgrade rollback against find/reset failures
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Second- and third-round independent-review reliability fixes on the keep-mode
upgrade rollback path, plus accurate abort messaging. All fixed red-first with
self-verifying controls in the rollback gate.

Round 2 (blockers A/B, should-fix C):
- install.sh: `trap 'restore_snapshot; exit 1' ERR INT TERM` so an INT/TERM
  mid-sync terminates instead of resuming past the interrupt and reporting
  success (a bash signal handler that only returns does not terminate).
- manifest.{ts,sh}: reject a degenerate [framework] section whose entries are
  all empty or bare-dot (`/`, `./`, `.`, `..`) — it passed the non-empty guard
  yet yielded zero usable globs, silently resolving everything to operator.
  Parity via a shared `[^/.]` usable-glob test; TS throws ManifestError.
- finalize.ts: classify the sync-abort message — a ManifestError is a pre-sync
  validation abort ("no files were changed"); any other error may be partial.

Round 3 (blockers D1, D2):
- install.sh: enumerate framework files with a checked temp file (_scan_or_die)
  instead of `< <(find …)` — process substitution discards find's exit status,
  so an EACCES/I/O failure mid-scan would truncate the file list yet leave the
  loop exiting 0, committing a partial upgrade as success (ERR trap never fires).
- install.sh: guard the `rm -rf; mkdir -p` target reset inside restore_snapshot
  — a bare reset failing under set -e exits silently after partial deletion,
  never printing the snapshot-recovery pointer. Now checked like the cp -a
  restore: on failure it preserves the snapshot and tells the operator where.

Tests: rollback gate 14→28 (Parts C/D/E with disabled-guard controls);
new finalize-sync-abort.spec.ts (3). No secret value is ever emitted; snapshots
stay 0700. Gates green: typecheck, lint, format:check, full mosaic vitest 1094,
HARD GATE 193, rollback 28, migration 21.

Refs #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-16 17:30:19 -05:00
parent 0a5e703a70
commit af627e7583
11 changed files with 1003 additions and 26 deletions

View File

@@ -9,8 +9,16 @@
# must still update, and a retired framework file inside a shipped subtree must
# still be pruned. No operator secret value may appear in installer output.
#
# Runs the whole matrix twice: once with rsync available, once forcing the
# cp-based fallback (both destructive paths in install.sh must obey the manifest).
# Keep mode is a SINGLE code path (sync_framework_keep, a manifest-driven cp
# overlay + scoped prune — no rsync). The matrix still runs twice, once with
# rsync on PATH and once with it hidden, to prove the keep path is genuinely
# rsync-independent: it must obey the manifest identically whether or not rsync
# happens to be installed (rsync --delete is only ever used by overwrite mode,
# which has no operator state to protect).
#
# It also runs a fail-closed matrix (#791 B2/B3): an empty, operator-only,
# malformed, or missing manifest must ABORT the upgrade loudly and leave every
# operator path untouched — never silently no-op to "complete".
#
# Usage: bash test-upgrade-manifest-guard.sh
set -uo pipefail
@@ -131,6 +139,59 @@ run_matrix() {
rm -rf "$H" "$E" "$OUT"
}
# Fail-closed matrix (#791 B2/B3 + blocker-1): run install.sh from a COPY of the
# framework so the shipped manifest can be corrupted. Every corruption must abort
# the upgrade non-zero with a manifest error, leaving all operator sentinels
# byte-identical AND on the SAME inode. The inode check is the load-bearing part:
# manifest validation is hoisted BEFORE make_snapshot/the restore trap, so a bad
# manifest must abort without ever snapshotting, deleting, and restoring the
# target. Were validation still armed under the ERR trap, restore_snapshot would
# rm -rf + rebuild the target — same bytes but a NEW inode (broken hard links,
# changed ctime), which a content-only hash would miss (#791 blocker-1).
run_failclosed() {
local label="$1" mutate="$2"
local SRC H E OUT rc rel before_hash after_hash before_ino after_ino key
SRC=$(mktemp -d); H=$(mktemp -d); E=$(mktemp -d); OUT=$(mktemp)
cp -a "$FW/." "$SRC/"
case "$mutate" in
empty) : > "$SRC/framework-manifest.txt" ;;
operator-only) printf '[operator]\nSOUL.md\n*.local.md\n' > "$SRC/framework-manifest.txt" ;;
malformed) printf 'stray.md\n[framework]\nguides/**\n' > "$SRC/framework-manifest.txt" ;;
degenerate) printf '[framework]\n/\n./\n[operator]\nSOUL.md\n' > "$SRC/framework-manifest.txt" ;;
missing) rm -f "$SRC/framework-manifest.txt" ;;
esac
seed_home "$H"
for rel in "${OPERATOR_SENTINELS[@]}"; do
key=$(echo "$rel" | tr / _)
sha256sum "$H/$rel" | awk '{print $1}' > "$E/$key.hash"
stat -c '%i' "$H/$rel" > "$E/$key.ino"
done
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 bash "$SRC/install.sh" >"$OUT" 2>&1
rc=$?
chk "[fail-closed:$label] upgrade aborts non-zero" "[ '$rc' -ne 0 ]"
chk "[fail-closed:$label] refuses loudly with a manifest error" \
"grep -qi 'manifest' '$OUT'"
for rel in "${OPERATOR_SENTINELS[@]}"; do
key=$(echo "$rel" | tr / _)
before_hash=$(cat "$E/$key.hash")
after_hash=$(sha256sum "$H/$rel" 2>/dev/null | awk '{print $1}')
chk "[fail-closed:$label] operator sentinel untouched: $rel" \
"[ -n '$after_hash' ] && [ '$before_hash' = '$after_hash' ]"
before_ino=$(cat "$E/$key.ino")
after_ino=$(stat -c '%i' "$H/$rel" 2>/dev/null)
chk "[fail-closed:$label] operator sentinel not deleted/recreated (inode stable): $rel" \
"[ -n '$after_ino' ] && [ '$before_ino' = '$after_ino' ]"
done
chk "[fail-closed:$label] operator secret value absent from output" \
"! grep -q '$SECRET' '$OUT'"
chmod -R u+w "$SRC" "$H" 2>/dev/null || true
rm -rf "$SRC" "$H" "$E" "$OUT"
}
echo "#791 upgrade manifest guard (HARD GATE):"
# 1) rsync path (if available on this host).
@@ -140,15 +201,24 @@ else
echo " · rsync not installed — skipping rsync-path matrix"
fi
# 2) cp fallback path — hide rsync behind a scratch PATH so install.sh takes the
# find/rm+cp branch. Provide the coreutils the fallback needs.
# 2) rsync-absent path — hide rsync behind a scratch PATH. Keep mode never calls
# rsync, so this must resolve identically to run (1); it proves the keep path
# does not silently depend on rsync being installed. (Provide the coreutils the
# installer needs on the stripped PATH.)
FBIN=$(mktemp -d)
for t in bash cp find mktemp rm mkdir chmod cmp sed grep cat dirname basename stat sha256sum awk tr; do
p=$(command -v "$t" 2>/dev/null) && ln -s "$p" "$FBIN/$t"
done
run_matrix "cp-fallback" env "PATH=$FBIN"
run_matrix "rsync-absent" env "PATH=$FBIN"
rm -rf "$FBIN"
# 3) fail-closed matrix (#791 B2/B3) — corrupt the shipped manifest four ways.
run_failclosed "empty-manifest" empty
run_failclosed "operator-only" operator-only
run_failclosed "malformed-manifest" malformed
run_failclosed "degenerate-framework" degenerate
run_failclosed "missing-manifest" missing
echo
echo "RESULT: $pass passed, $fail failed"
[ "$fail" -eq 0 ]