From eed04d55a5ecff93aeab1e53a9cb5a7b778d3200 Mon Sep 17 00:00:00 2001 From: mosaic-coder Date: Sun, 26 Jul 2026 00:49:15 -0500 Subject: [PATCH] =?UTF-8?q?test(wake):=20T9=20privilege-invariant=20fault?= =?UTF-8?q?=20injection=20(mountpoint-busy,=20root-proof)=20+=20cursor-pat?= =?UTF-8?q?h-writable=20coverage=20=E2=80=94=20CI-green?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit T9 (burn-before-enqueue, #908 arrow #1) forced the durable pending-write failure via `chmod 500 "$STATE_DIR"`. That works only as a non-root user: DAC permission checks (including a read-only directory) are bypassed entirely by root, and Woodpecker CI containers run as root — so the write silently succeeded, observed_seq advanced, and all three "failed-write" assertions went spuriously green. The chmod also blocked the cursor's (observed_seq) own mktemp in the same directory, so even under a non-root run T9 could never have caught a premature cursor commit — narrower coverage than its comment claimed. The product code (store.sh) was already correct: the durable pending write is textually first and the cursor write is gated on its success (verified independently, unchanged here). Fix: bind-mount a throwaway file over $STATE_DIR/pending.jsonl (the exact _atomic_write rename TARGET for the durable write) so the atomic write's `mv -f tmp target` hits EBUSY — the kernel refuses to rename any file onto an ACTIVE MOUNT POINT. That is a structural VFS constraint, not a DAC check: no uid, root included, can bypass it short of an explicit umount. observed_seq is left an ordinary writable file, untouched by the mount, so a regression that commits the cursor before/independent of the durable write is now genuinely caught. The mount + forced enqueue run inside a private `unshare --mount --user --map-root-user` namespace so the same mechanism applies whether the harness runs unprivileged (dev) or as root (CI); the bind mount is private to that namespace and vanishes the instant it exits, so no manual cleanup step is needed afterward. sweep: grep chmod/chattr across all packages/mosaic/framework/tools/wake/test-*.sh -> T9's `chmod 500 "$STATE_DIR"` (fixed here) was the ONLY permission-based fault-injection use. Every other chmod hit is `chmod +x` making a mock/ fixture script executable (benign setup, not failure injection, unaffected by root): test-wake-reconcile.sh:68, test-wake-detector.sh:110, test-wake-digest-hmac.sh:164, test-wake-beacon.sh:70/79/88/103/311, test-wake-store-ack.sh:246 (mock curl/wget/nc/ssh binaries). Verified: enqueue under the injection exits non-zero as the current non-root user, with the process reporting uid 0 INSIDE the namespace (genuine kernel-mapped root) and still failing EBUSY — a direct proof the mechanism is root-proof, not just an argument. fakeroot was tried per request but only fakes stat/chown (no real mount capability), so it is not a meaningful vehicle for this injection and was not used as the proof. Coverage-RED reproduction: temporarily moved the observed_seq cursor write in store.sh to before the durable pending write (premature commit) -> T9 went RED (observed_seq wrongly advanced to 2, post-failure allocation wrongly resumed at 3); reverted -> GREEN again, diff-clean against origin. Full `pnpm run test:framework-shell` rc=0 (all harnesses, T9 included, 11/11 wake store/ack groups green); re-run with openssl masked from PATH (command -v openssl confirmed empty) rc=0, wake store/ack harness unaffected (an unrelated harness, test-wake-digest-hmac.sh, gracefully SKIPs without openssl -- pre-existing, orthogonal, HMAC is W3/A5 not W2). shellcheck 0.11 clean (one deliberate SC2016 on nested-shell variable expansion, disabled inline per this repo's existing convention). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb --- .../tools/wake/test-wake-store-ack.sh | 62 +++++++++++++++---- 1 file changed, 50 insertions(+), 12 deletions(-) diff --git a/packages/mosaic/framework/tools/wake/test-wake-store-ack.sh b/packages/mosaic/framework/tools/wake/test-wake-store-ack.sh index 6922411d..eceecc7e 100755 --- a/packages/mosaic/framework/tools/wake/test-wake-store-ack.sh +++ b/packages/mosaic/framework/tools/wake/test-wake-store-ack.sh @@ -306,19 +306,57 @@ echo "== T9: burn-before-enqueue — a failed durable write must NOT advance obs [ "$s1" = "1" ] || fail_msg "T9: baseline allocation should be 1, got '$s1'" obs_before="$("$STORE" cursors | sed -n 's/^observed_seq=//p')" [ "$obs_before" = "1" ] || fail_msg "T9: observed_seq should be 1 before the forced failure, got '$obs_before'" - # Force the durable pending write to FAIL: make STATE_DIR read-only so the - # atomic-write mktemp inside it cannot create its temp file => the durable - # pending write fails and the enqueue must abort. The already-existing cursor - # FILE stays owner-writable (a read-only DIR blocks new-entry creation, not a - # truncating write to an existing file), so a regression that committed the - # observed_seq cursor BEFORE/independent of the durable write (the arrow-#1 - # burn) would still advance it here — the RED signal this test catches. The - # lock file + cursor files already exist, so lock acquisition + cursor reads - # still work; only the durable pending WRITE fails. - chmod 500 "$STATE_DIR" + # Force the durable PENDING write to FAIL via a PRIVILEGE-INVARIANT trigger: + # bind-mount a throwaway file OVER $STATE_DIR/pending.jsonl (_atomic_write's + # exact rename TARGET for the durable write — see _wake-common.sh) so the + # atomic write's `mv -f tmp target` hits EBUSY ("Device or resource busy"): + # the kernel refuses to rename ANY file onto an ACTIVE MOUNT POINT. That is + # a structural VFS constraint, not a DAC permission check — root does NOT + # bypass it (no uid can rename over a live mountpoint short of an explicit + # umount first). This replaces the old `chmod 500 "$STATE_DIR"` injection: a + # CI container running as root BYPASSES DAC checks entirely, so the + # "read-only" dir let mktemp/the write through and every assertion below + # went spuriously green. The old chmod also blocked the CURSOR's own + # mktemp (same dir), so it could never have caught a premature cursor + # commit either — narrower coverage than its comment claimed. Here + # observed_seq is left an ordinary writable file, never touched by the + # mount, so a regression that commits the cursor before/independent of the + # durable write is still genuinely caught by the assertions below. + # + # The mount + forced enqueue run inside a private mount+user namespace + # (`unshare --mount --user --map-root-user`) so the mechanism is IDENTICAL + # whether this harness runs unprivileged (dev) or as root (CI): creating a + # user namespace needs no pre-existing privilege, and the busy-mountpoint + # rule inside it is the real kernel rule, unaffected by uid in or out of a + # namespace. The bind mount is entirely private to that namespace and + # vanishes the instant it exits, so $STATE_DIR/pending.jsonl is a plain + # writable file again for every assertion below — no manual cleanup step. rc=0 - "$STORE" enqueue --class actionable --locators '{"n":2}' >/dev/null 2>&1 || rc=$? - chmod 700 "$STATE_DIR" + if ! command -v unshare >/dev/null 2>&1; then + fail_msg "T9: 'unshare' unavailable — cannot construct the privilege-invariant fault injection in this sandbox" + rc=99 + else + ro_src="$TMP_ROOT/t9-ro-src" + mkdir -p "$ro_src" + : >"$ro_src/pending.jsonl" + # shellcheck disable=SC2016 # deliberate: $1/$2/$3 expand in the NESTED + # bash (inside the unshare'd namespace), not this outer single-quoted one. + unshare --mount --user --map-root-user bash -c ' + set -u + ro_src="$1" + state_dir="$2" + store="$3" + if ! mount --bind "$ro_src/pending.jsonl" "$state_dir/pending.jsonl" 2>/dev/null; then + exit 98 + fi + mount -o remount,ro,bind "$state_dir/pending.jsonl" 2>/dev/null || true + "$store" enqueue --class actionable --locators "{\"n\":2}" >/dev/null 2>&1 + ' _ "$ro_src" "$STATE_DIR" "$STORE" + rc=$? + if [ "$rc" -eq 98 ]; then + fail_msg "T9: could not bind-mount over pending.jsonl (mount unavailable in this sandbox) — fault injection not constructed" + fi + fi [ "$rc" -ne 0 ] || fail_msg "T9: an enqueue whose durable write fails must EXIT NON-ZERO (fail loud)" # THE ARROW-#1 ASSERTION: the observed_seq cursor did NOT advance — no seq was # burned by a write that never reached the store. (RED if the cursor bumps