fix(fleet): harden #791 upgrade rollback against find/reset failures
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Second- and third-round independent-review reliability fixes on the keep-mode
upgrade rollback path, plus accurate abort messaging. All fixed red-first with
self-verifying controls in the rollback gate.

Round 2 (blockers A/B, should-fix C):
- install.sh: `trap 'restore_snapshot; exit 1' ERR INT TERM` so an INT/TERM
  mid-sync terminates instead of resuming past the interrupt and reporting
  success (a bash signal handler that only returns does not terminate).
- manifest.{ts,sh}: reject a degenerate [framework] section whose entries are
  all empty or bare-dot (`/`, `./`, `.`, `..`) — it passed the non-empty guard
  yet yielded zero usable globs, silently resolving everything to operator.
  Parity via a shared `[^/.]` usable-glob test; TS throws ManifestError.
- finalize.ts: classify the sync-abort message — a ManifestError is a pre-sync
  validation abort ("no files were changed"); any other error may be partial.

Round 3 (blockers D1, D2):
- install.sh: enumerate framework files with a checked temp file (_scan_or_die)
  instead of `< <(find …)` — process substitution discards find's exit status,
  so an EACCES/I/O failure mid-scan would truncate the file list yet leave the
  loop exiting 0, committing a partial upgrade as success (ERR trap never fires).
- install.sh: guard the `rm -rf; mkdir -p` target reset inside restore_snapshot
  — a bare reset failing under set -e exits silently after partial deletion,
  never printing the snapshot-recovery pointer. Now checked like the cp -a
  restore: on failure it preserves the snapshot and tells the operator where.

Tests: rollback gate 14→28 (Parts C/D/E with disabled-guard controls);
new finalize-sync-abort.spec.ts (3). No secret value is ever emitted; snapshots
stay 0700. Gates green: typecheck, lint, format:check, full mosaic vitest 1094,
HARD GATE 193, rollback 28, migration 21.

Refs #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-16 17:30:19 -05:00
parent 0a5e703a70
commit af627e7583
11 changed files with 1003 additions and 26 deletions

View File

@@ -1,5 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
# -E (errtrace): the ERR trap must propagate INTO functions and command
# substitutions. Without it the `trap restore_snapshot ERR` set below is dead
# code for any failure inside sync_framework_keep() (its whole body runs in a
# function) — a mid-sync failure would abort with a half-written target and NO
# rollback (#791 B1). Keep -E first so every later function inherits the trap.
set -Eeuo pipefail
# ─── Mosaic Framework Installer ──────────────────────────────────────────────
#
@@ -62,14 +67,45 @@ step() { echo -e "\n${BOLD}$1${RESET}"; }
SNAPSHOT_DIR=""
make_snapshot() {
is_existing_install || return 0
# mktemp -d creates the dir 0700 — the snapshot (which mirrors operator config,
# possibly including secrets) is never world-readable.
SNAPSHOT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/mosaic-snapshot-XXXXXX")"
cp -a "$TARGET_DIR/." "$SNAPSHOT_DIR/" 2>/dev/null || true
# The snapshot MUST be complete: restore rebuilds the target from it, so a
# partial capture (unreadable file, disk-full, I/O error) would silently
# discard whatever it missed. If cp -a cannot copy the whole tree, abort NOW —
# before the restore trap is armed and before anything is mutated. Fail closed
# rather than proceed with a snapshot we cannot trust (#791 blocker-2).
if ! cp -a "$TARGET_DIR/." "$SNAPSHOT_DIR/"; then
fail "Could not capture a complete pre-upgrade snapshot of $TARGET_DIR — aborting before any changes were made (fail-closed)."
rm -rf "$SNAPSHOT_DIR"; SNAPSHOT_DIR=""
exit 1
fi
}
restore_snapshot() {
# Disarm the trap first: restore runs under `set -e`, and a non-zero step
# inside it must not re-enter this handler (errtrace makes ERR fire in
# functions now). One restore attempt, then let the script exit non-zero.
trap - ERR INT TERM
[[ -n "$SNAPSHOT_DIR" && -d "$SNAPSHOT_DIR" ]] || return 0
fail "Install interrupted/failed — restoring previous state from snapshot"
rm -rf "$TARGET_DIR"; mkdir -p "$TARGET_DIR"
cp -a "$SNAPSHOT_DIR/." "$TARGET_DIR/" 2>/dev/null || true
# Reset the target before rebuilding from the snapshot — but CHECK it. Under
# `set -e` (trap already disarmed) a bare `rm -rf; mkdir -p` that fails would
# exit the whole script immediately, after `rm` may have deleted part of the
# target, WITHOUT ever printing the recovery pointer below — the operator would
# be left with a half-removed target and no idea the snapshot survives in /tmp.
# Test the reset explicitly (like the cp -a below), and on failure keep the
# snapshot and tell the operator where it is (#791 blocker-D2).
if ! rm -rf "$TARGET_DIR" || ! mkdir -p "$TARGET_DIR"; then
fail "Snapshot restore could not reset $TARGET_DIR. Your previous configuration is preserved at: $SNAPSHOT_DIR — copy it back into $TARGET_DIR manually."
return 1
fi
# Surface an incomplete restore instead of swallowing it: the snapshot is the
# last good copy, so if cp cannot fully rebuild the target we must NOT delete
# the snapshot — point the operator at it for manual recovery (#791 blocker-2).
if ! cp -a "$SNAPSHOT_DIR/." "$TARGET_DIR/"; then
fail "Snapshot restore did not complete cleanly. Your previous configuration is preserved at: $SNAPSHOT_DIR — copy it back into $TARGET_DIR manually."
return 1
fi
}
cleanup_snapshot() { [[ -n "$SNAPSHOT_DIR" && -d "$SNAPSHOT_DIR" ]] && rm -rf "$SNAPSHOT_DIR"; SNAPSHOT_DIR=""; }
@@ -174,7 +210,9 @@ sync_framework() {
if [[ "$INSTALL_MODE" == "keep" ]]; then
# The `mosaic update` path. Manifest-driven, never-deleting-outside-framework:
# operator config is structurally protected (#791). No rsync --delete here.
manifest_load
# The manifest is already loaded+validated in main() BEFORE the snapshot/trap
# (a fail-closed manifest must abort without ever restoring over operator
# files — see the pre-flight in main, #791 blocker-1).
sync_framework_keep
return
fi
@@ -184,15 +222,36 @@ sync_framework() {
sync_framework_overwrite
}
# Enumerate a NUL-delimited file list via `find` into the temp file $1, failing
# CLOSED if find errors. We capture to a checked file instead of consuming
# `< <(find …)` directly because a process substitution discards the producer's
# exit status: an EACCES/I/O failure partway through a scan would truncate the
# list yet leave the reading `while` loop exiting 0, so a partial upgrade would
# commit and report success and the ERR/restore trap would never fire. Running
# find to completion first, then checking its status, turns that silent
# truncation into a fail-closed abort that the restore trap can act on (#791
# blocker-D1). $1 after the shift is the scan root — named in the error.
_scan_or_die() {
local out="$1"; shift
if ! find "$@" -print0 > "$out"; then
fail "Could not enumerate framework files under '$1' — aborting before committing an incomplete sync (fail-closed)."
return 1 # D1-GUARD
fi
}
# Keep-mode sync: create/refresh framework-owned files and prune only retired
# framework files inside shipped framework subtrees. Operator-owned and unknown
# paths (fail-safe default) are never written and never deleted — the #791 HARD
# GATE. Single code path (no rsync) so it is byte-for-byte parity-testable.
sync_framework_keep() {
local src="$SOURCE_DIR" dst="$TARGET_DIR" abs rel root
local src="$SOURCE_DIR" dst="$TARGET_DIR" abs rel root list
# 1) Overlay copy — every framework-owned source file, refreshed only when its
# bytes changed (no mtime churn on unchanged files, never on operator files).
# The source scan is captured fail-closed (#791 blocker-D1): a find failure
# aborts the sync (→ ERR trap → restore) rather than silently truncating it.
list="$(mktemp)"
_scan_or_die "$list" "$src" -type f || { rm -f "$list"; return 1; }
while IFS= read -r -d '' abs; do
rel="${abs#"$src"/}"
case "$rel" in
@@ -202,24 +261,34 @@ sync_framework_keep() {
if [[ -f "$dst/$rel" ]] && cmp -s "$abs" "$dst/$rel"; then continue; fi
[[ "$rel" == */* ]] && mkdir -p "$dst/${rel%/*}"
cp "$abs" "$dst/$rel"
done < <(find "$src" -type f -print0)
done < "$list"
rm -f "$list"
# 2) Scoped prune — within each shipped framework subtree root, remove
# framework-owned target files the current source no longer ships. Operator
# carve-outs (e.g. tools/_lib/credentials.json) resolve to operator and are
# skipped; unknown paths resolve to operator too — both are unreachable here.
# Each subtree scan is captured fail-closed for the same reason as the copy.
while IFS= read -r root; do
[[ -n "$root" && -d "$dst/$root" ]] || continue
list="$(mktemp)"
_scan_or_die "$list" "$dst/$root" -type f || { rm -f "$list"; return 1; }
while IFS= read -r -d '' abs; do
rel="${abs#"$dst"/}"
case "$rel" in *.pre-constitution.bak) continue ;; esac
[[ -f "$src/$rel" ]] && continue # still shipped
manifest_is_framework "$rel" || continue
rm -f "$abs"
done < <(find "$dst/$root" -type f -print0)
done < "$list"
rm -f "$list"
# Drop framework dirs left empty by the prune (never touches a dir that still
# holds an operator file — those are never emptied).
find "$dst/$root" -type d -empty -delete 2>/dev/null || true
# holds an operator file — those are never emptied). A genuine find failure
# (unreadable dir) is surfaced as a warning rather than silently swallowed;
# the "directory not empty" races we tolerate are ignored via -delete's own
# rc, not by hiding stderr — so a real error is still visible to the operator.
if ! find "$dst/$root" -type d -empty -delete 2>/dev/null; then
warn "prune: could not fully sweep empty framework dirs under $root (left as-is)"
fi
done < <(manifest_subtree_roots)
}
@@ -307,9 +376,23 @@ else
ok "Install mode: overwrite"
fi
# Pre-flight (keep mode): load + validate the framework manifest BEFORE taking a
# snapshot or arming the restore trap. A fail-closed manifest (missing / empty /
# malformed) must abort here WITHOUT deleting or restoring over operator files —
# the snapshot/restore path exists only for a genuine mid-sync mutation failure,
# not for a validation failure that has touched nothing yet (#791 blocker-1).
if [[ "$INSTALL_MODE" == "keep" ]]; then
manifest_load
fi
# Snapshot before any destructive file operation; restore on interrupt/failure.
# The trap MUST exit after restoring: a bash INT/TERM handler that merely returns
# does NOT terminate the script — execution would resume past the interrupt,
# clear the snapshot, and report success, leaving a partial post-interrupt update
# (#791 blocker-A). `restore_snapshot; exit 1` guarantees a non-zero exit for
# both the errtrace (ERR) and signal (INT/TERM) paths.
make_snapshot
trap 'restore_snapshot' ERR INT TERM
trap 'restore_snapshot; exit 1' ERR INT TERM
sync_framework