test(wake): T9 privilege-invariant fault injection (mountpoint-busy, root-proof) + cursor-path-writable coverage — CI-green
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

T9 (burn-before-enqueue, #908 arrow #1) forced the durable pending-write
failure via `chmod 500 "$STATE_DIR"`. That works only as a non-root user:
DAC permission checks (including a read-only directory) are bypassed
entirely by root, and Woodpecker CI containers run as root — so the write
silently succeeded, observed_seq advanced, and all three "failed-write"
assertions went spuriously green. The chmod also blocked the cursor's
(observed_seq) own mktemp in the same directory, so even under a non-root
run T9 could never have caught a premature cursor commit — narrower
coverage than its comment claimed. The product code (store.sh) was already
correct: the durable pending write is textually first and the cursor write
is gated on its success (verified independently, unchanged here).

Fix: bind-mount a throwaway file over $STATE_DIR/pending.jsonl (the exact
_atomic_write rename TARGET for the durable write) so the atomic write's
`mv -f tmp target` hits EBUSY — the kernel refuses to rename any file onto
an ACTIVE MOUNT POINT. That is a structural VFS constraint, not a DAC
check: no uid, root included, can bypass it short of an explicit umount.
observed_seq is left an ordinary writable file, untouched by the mount, so
a regression that commits the cursor before/independent of the durable
write is now genuinely caught. The mount + forced enqueue run inside a
private `unshare --mount --user --map-root-user` namespace so the same
mechanism applies whether the harness runs unprivileged (dev) or as root
(CI); the bind mount is private to that namespace and vanishes the instant
it exits, so no manual cleanup step is needed afterward.

sweep: grep chmod/chattr across all packages/mosaic/framework/tools/wake/test-*.sh
-> T9's `chmod 500 "$STATE_DIR"` (fixed here) was the ONLY permission-based
fault-injection use. Every other chmod hit is `chmod +x` making a mock/
fixture script executable (benign setup, not failure injection, unaffected
by root): test-wake-reconcile.sh:68, test-wake-detector.sh:110,
test-wake-digest-hmac.sh:164, test-wake-beacon.sh:70/79/88/103/311,
test-wake-store-ack.sh:246 (mock curl/wget/nc/ssh binaries).

Verified: enqueue under the injection exits non-zero as the current
non-root user, with the process reporting uid 0 INSIDE the namespace
(genuine kernel-mapped root) and still failing EBUSY — a direct proof the
mechanism is root-proof, not just an argument. fakeroot was tried per
request but only fakes stat/chown (no real mount capability), so it is not
a meaningful vehicle for this injection and was not used as the proof.
Coverage-RED reproduction: temporarily moved the observed_seq cursor write
in store.sh to before the durable pending write (premature commit) -> T9
went RED (observed_seq wrongly advanced to 2, post-failure allocation
wrongly resumed at 3); reverted -> GREEN again, diff-clean against origin.
Full `pnpm run test:framework-shell` rc=0 (all harnesses, T9 included,
11/11 wake store/ack groups green); re-run with openssl masked from PATH
(command -v openssl confirmed empty) rc=0, wake store/ack harness
unaffected (an unrelated harness, test-wake-digest-hmac.sh, gracefully
SKIPs without openssl -- pre-existing, orthogonal, HMAC is W3/A5 not W2).
shellcheck 0.11 clean (one deliberate SC2016 on nested-shell variable
expansion, disabled inline per this repo's existing convention).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
This commit is contained in:
mosaic-coder
2026-07-26 00:49:15 -05:00
parent 2ce332db6c
commit eed04d55a5

View File

@@ -306,19 +306,57 @@ echo "== T9: burn-before-enqueue — a failed durable write must NOT advance obs
[ "$s1" = "1" ] || fail_msg "T9: baseline allocation should be 1, got '$s1'" [ "$s1" = "1" ] || fail_msg "T9: baseline allocation should be 1, got '$s1'"
obs_before="$("$STORE" cursors | sed -n 's/^observed_seq=//p')" obs_before="$("$STORE" cursors | sed -n 's/^observed_seq=//p')"
[ "$obs_before" = "1" ] || fail_msg "T9: observed_seq should be 1 before the forced failure, got '$obs_before'" [ "$obs_before" = "1" ] || fail_msg "T9: observed_seq should be 1 before the forced failure, got '$obs_before'"
# Force the durable pending write to FAIL: make STATE_DIR read-only so the # Force the durable PENDING write to FAIL via a PRIVILEGE-INVARIANT trigger:
# atomic-write mktemp inside it cannot create its temp file => the durable # bind-mount a throwaway file OVER $STATE_DIR/pending.jsonl (_atomic_write's
# pending write fails and the enqueue must abort. The already-existing cursor # exact rename TARGET for the durable write — see _wake-common.sh) so the
# FILE stays owner-writable (a read-only DIR blocks new-entry creation, not a # atomic write's `mv -f tmp target` hits EBUSY ("Device or resource busy"):
# truncating write to an existing file), so a regression that committed the # the kernel refuses to rename ANY file onto an ACTIVE MOUNT POINT. That is
# observed_seq cursor BEFORE/independent of the durable write (the arrow-#1 # a structural VFS constraint, not a DAC permission check — root does NOT
# burn) would still advance it here — the RED signal this test catches. The # bypass it (no uid can rename over a live mountpoint short of an explicit
# lock file + cursor files already exist, so lock acquisition + cursor reads # umount first). This replaces the old `chmod 500 "$STATE_DIR"` injection: a
# still work; only the durable pending WRITE fails. # CI container running as root BYPASSES DAC checks entirely, so the
chmod 500 "$STATE_DIR" # "read-only" dir let mktemp/the write through and every assertion below
# went spuriously green. The old chmod also blocked the CURSOR's own
# mktemp (same dir), so it could never have caught a premature cursor
# commit either — narrower coverage than its comment claimed. Here
# observed_seq is left an ordinary writable file, never touched by the
# mount, so a regression that commits the cursor before/independent of the
# durable write is still genuinely caught by the assertions below.
#
# The mount + forced enqueue run inside a private mount+user namespace
# (`unshare --mount --user --map-root-user`) so the mechanism is IDENTICAL
# whether this harness runs unprivileged (dev) or as root (CI): creating a
# user namespace needs no pre-existing privilege, and the busy-mountpoint
# rule inside it is the real kernel rule, unaffected by uid in or out of a
# namespace. The bind mount is entirely private to that namespace and
# vanishes the instant it exits, so $STATE_DIR/pending.jsonl is a plain
# writable file again for every assertion below — no manual cleanup step.
rc=0 rc=0
"$STORE" enqueue --class actionable --locators '{"n":2}' >/dev/null 2>&1 || rc=$? if ! command -v unshare >/dev/null 2>&1; then
chmod 700 "$STATE_DIR" fail_msg "T9: 'unshare' unavailable — cannot construct the privilege-invariant fault injection in this sandbox"
rc=99
else
ro_src="$TMP_ROOT/t9-ro-src"
mkdir -p "$ro_src"
: >"$ro_src/pending.jsonl"
# shellcheck disable=SC2016 # deliberate: $1/$2/$3 expand in the NESTED
# bash (inside the unshare'd namespace), not this outer single-quoted one.
unshare --mount --user --map-root-user bash -c '
set -u
ro_src="$1"
state_dir="$2"
store="$3"
if ! mount --bind "$ro_src/pending.jsonl" "$state_dir/pending.jsonl" 2>/dev/null; then
exit 98
fi
mount -o remount,ro,bind "$state_dir/pending.jsonl" 2>/dev/null || true
"$store" enqueue --class actionable --locators "{\"n\":2}" >/dev/null 2>&1
' _ "$ro_src" "$STATE_DIR" "$STORE"
rc=$?
if [ "$rc" -eq 98 ]; then
fail_msg "T9: could not bind-mount over pending.jsonl (mount unavailable in this sandbox) — fault injection not constructed"
fi
fi
[ "$rc" -ne 0 ] || fail_msg "T9: an enqueue whose durable write fails must EXIT NON-ZERO (fail loud)" [ "$rc" -ne 0 ] || fail_msg "T9: an enqueue whose durable write fails must EXIT NON-ZERO (fail loud)"
# THE ARROW-#1 ASSERTION: the observed_seq cursor did NOT advance — no seq was # THE ARROW-#1 ASSERTION: the observed_seq cursor did NOT advance — no seq was
# burned by a write that never reached the store. (RED if the cursor bumps # burned by a write that never reached the store. (RED if the cursor bumps