test(wake): T9 privilege-invariant fault injection (mountpoint-busy, root-proof) + cursor-path-writable coverage — CI-green
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
T9 (burn-before-enqueue, #908 arrow #1) forced the durable pending-write failure via `chmod 500 "$STATE_DIR"`. That works only as a non-root user: DAC permission checks (including a read-only directory) are bypassed entirely by root, and Woodpecker CI containers run as root — so the write silently succeeded, observed_seq advanced, and all three "failed-write" assertions went spuriously green. The chmod also blocked the cursor's (observed_seq) own mktemp in the same directory, so even under a non-root run T9 could never have caught a premature cursor commit — narrower coverage than its comment claimed. The product code (store.sh) was already correct: the durable pending write is textually first and the cursor write is gated on its success (verified independently, unchanged here). Fix: bind-mount a throwaway file over $STATE_DIR/pending.jsonl (the exact _atomic_write rename TARGET for the durable write) so the atomic write's `mv -f tmp target` hits EBUSY — the kernel refuses to rename any file onto an ACTIVE MOUNT POINT. That is a structural VFS constraint, not a DAC check: no uid, root included, can bypass it short of an explicit umount. observed_seq is left an ordinary writable file, untouched by the mount, so a regression that commits the cursor before/independent of the durable write is now genuinely caught. The mount + forced enqueue run inside a private `unshare --mount --user --map-root-user` namespace so the same mechanism applies whether the harness runs unprivileged (dev) or as root (CI); the bind mount is private to that namespace and vanishes the instant it exits, so no manual cleanup step is needed afterward. sweep: grep chmod/chattr across all packages/mosaic/framework/tools/wake/test-*.sh -> T9's `chmod 500 "$STATE_DIR"` (fixed here) was the ONLY permission-based fault-injection use. Every other chmod hit is `chmod +x` making a mock/ fixture script executable (benign setup, not failure injection, unaffected by root): test-wake-reconcile.sh:68, test-wake-detector.sh:110, test-wake-digest-hmac.sh:164, test-wake-beacon.sh:70/79/88/103/311, test-wake-store-ack.sh:246 (mock curl/wget/nc/ssh binaries). Verified: enqueue under the injection exits non-zero as the current non-root user, with the process reporting uid 0 INSIDE the namespace (genuine kernel-mapped root) and still failing EBUSY — a direct proof the mechanism is root-proof, not just an argument. fakeroot was tried per request but only fakes stat/chown (no real mount capability), so it is not a meaningful vehicle for this injection and was not used as the proof. Coverage-RED reproduction: temporarily moved the observed_seq cursor write in store.sh to before the durable pending write (premature commit) -> T9 went RED (observed_seq wrongly advanced to 2, post-failure allocation wrongly resumed at 3); reverted -> GREEN again, diff-clean against origin. Full `pnpm run test:framework-shell` rc=0 (all harnesses, T9 included, 11/11 wake store/ack groups green); re-run with openssl masked from PATH (command -v openssl confirmed empty) rc=0, wake store/ack harness unaffected (an unrelated harness, test-wake-digest-hmac.sh, gracefully SKIPs without openssl -- pre-existing, orthogonal, HMAC is W3/A5 not W2). shellcheck 0.11 clean (one deliberate SC2016 on nested-shell variable expansion, disabled inline per this repo's existing convention). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
This commit is contained in:
@@ -306,19 +306,57 @@ echo "== T9: burn-before-enqueue — a failed durable write must NOT advance obs
|
||||
[ "$s1" = "1" ] || fail_msg "T9: baseline allocation should be 1, got '$s1'"
|
||||
obs_before="$("$STORE" cursors | sed -n 's/^observed_seq=//p')"
|
||||
[ "$obs_before" = "1" ] || fail_msg "T9: observed_seq should be 1 before the forced failure, got '$obs_before'"
|
||||
# Force the durable pending write to FAIL: make STATE_DIR read-only so the
|
||||
# atomic-write mktemp inside it cannot create its temp file => the durable
|
||||
# pending write fails and the enqueue must abort. The already-existing cursor
|
||||
# FILE stays owner-writable (a read-only DIR blocks new-entry creation, not a
|
||||
# truncating write to an existing file), so a regression that committed the
|
||||
# observed_seq cursor BEFORE/independent of the durable write (the arrow-#1
|
||||
# burn) would still advance it here — the RED signal this test catches. The
|
||||
# lock file + cursor files already exist, so lock acquisition + cursor reads
|
||||
# still work; only the durable pending WRITE fails.
|
||||
chmod 500 "$STATE_DIR"
|
||||
# Force the durable PENDING write to FAIL via a PRIVILEGE-INVARIANT trigger:
|
||||
# bind-mount a throwaway file OVER $STATE_DIR/pending.jsonl (_atomic_write's
|
||||
# exact rename TARGET for the durable write — see _wake-common.sh) so the
|
||||
# atomic write's `mv -f tmp target` hits EBUSY ("Device or resource busy"):
|
||||
# the kernel refuses to rename ANY file onto an ACTIVE MOUNT POINT. That is
|
||||
# a structural VFS constraint, not a DAC permission check — root does NOT
|
||||
# bypass it (no uid can rename over a live mountpoint short of an explicit
|
||||
# umount first). This replaces the old `chmod 500 "$STATE_DIR"` injection: a
|
||||
# CI container running as root BYPASSES DAC checks entirely, so the
|
||||
# "read-only" dir let mktemp/the write through and every assertion below
|
||||
# went spuriously green. The old chmod also blocked the CURSOR's own
|
||||
# mktemp (same dir), so it could never have caught a premature cursor
|
||||
# commit either — narrower coverage than its comment claimed. Here
|
||||
# observed_seq is left an ordinary writable file, never touched by the
|
||||
# mount, so a regression that commits the cursor before/independent of the
|
||||
# durable write is still genuinely caught by the assertions below.
|
||||
#
|
||||
# The mount + forced enqueue run inside a private mount+user namespace
|
||||
# (`unshare --mount --user --map-root-user`) so the mechanism is IDENTICAL
|
||||
# whether this harness runs unprivileged (dev) or as root (CI): creating a
|
||||
# user namespace needs no pre-existing privilege, and the busy-mountpoint
|
||||
# rule inside it is the real kernel rule, unaffected by uid in or out of a
|
||||
# namespace. The bind mount is entirely private to that namespace and
|
||||
# vanishes the instant it exits, so $STATE_DIR/pending.jsonl is a plain
|
||||
# writable file again for every assertion below — no manual cleanup step.
|
||||
rc=0
|
||||
"$STORE" enqueue --class actionable --locators '{"n":2}' >/dev/null 2>&1 || rc=$?
|
||||
chmod 700 "$STATE_DIR"
|
||||
if ! command -v unshare >/dev/null 2>&1; then
|
||||
fail_msg "T9: 'unshare' unavailable — cannot construct the privilege-invariant fault injection in this sandbox"
|
||||
rc=99
|
||||
else
|
||||
ro_src="$TMP_ROOT/t9-ro-src"
|
||||
mkdir -p "$ro_src"
|
||||
: >"$ro_src/pending.jsonl"
|
||||
# shellcheck disable=SC2016 # deliberate: $1/$2/$3 expand in the NESTED
|
||||
# bash (inside the unshare'd namespace), not this outer single-quoted one.
|
||||
unshare --mount --user --map-root-user bash -c '
|
||||
set -u
|
||||
ro_src="$1"
|
||||
state_dir="$2"
|
||||
store="$3"
|
||||
if ! mount --bind "$ro_src/pending.jsonl" "$state_dir/pending.jsonl" 2>/dev/null; then
|
||||
exit 98
|
||||
fi
|
||||
mount -o remount,ro,bind "$state_dir/pending.jsonl" 2>/dev/null || true
|
||||
"$store" enqueue --class actionable --locators "{\"n\":2}" >/dev/null 2>&1
|
||||
' _ "$ro_src" "$STATE_DIR" "$STORE"
|
||||
rc=$?
|
||||
if [ "$rc" -eq 98 ]; then
|
||||
fail_msg "T9: could not bind-mount over pending.jsonl (mount unavailable in this sandbox) — fault injection not constructed"
|
||||
fi
|
||||
fi
|
||||
[ "$rc" -ne 0 ] || fail_msg "T9: an enqueue whose durable write fails must EXIT NON-ZERO (fail loud)"
|
||||
# THE ARROW-#1 ASSERTION: the observed_seq cursor did NOT advance — no seq was
|
||||
# burned by a write that never reached the store. (RED if the cursor bumps
|
||||
|
||||
Reference in New Issue
Block a user