test(wake): de-flake T10 concurrent-enqueue race — deterministic start-barrier + wait-both (#923)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

T10 ("two concurrent store.sh enqueue get DISTINCT seqs via flock") launched
its two background enqueues with plain `cmd & cmd & wait` — no guarantee the
scheduler ever actually overlapped them, so the flock contention the test
exists to prove was sometimes never really exercised (or the assertion could
race a not-yet-durable second write), producing an intermittent CI-load flake.

Replace the launch with a deterministic start barrier (flock'd gate file, no
sleep): both children signal a ready marker then block on a shared exclusive
lock; the parent busy-polls (not sleeps) until both markers exist, THEN
releases the gate so both children's pending shared-lock waits are granted
together and they race into store.sh's own enqueue lock for real. Assertions
(distinct+gapless {1,2} seqs, order-independent, both durably enqueued) are
unchanged and still run only after both PIDs have exited.

Verified: 30/30 sequential foreground runs pass (previously intermittent).
RED-catch reproduced 5/5: with _wake_lock_acquire temporarily stubbed to a
no-op, T10 reliably aliases (both enqueues get seq '1') and fails loud,
proving the de-flaked test still catches a real aliasing regression;
restored and reconfirmed GREEN. shellcheck 0.11.0 clean.

Closes #923; Part of #892
This commit is contained in:
mosaic-coder
2026-07-26 05:04:15 -05:00
parent 937a276208
commit f9175c99b6

View File

@@ -387,13 +387,58 @@ if command -v flock >/dev/null 2>&1; then
"$STORE" init >/dev/null 2>&1 || true
o1="$TMP_ROOT/t10.o1"
o2="$TMP_ROOT/t10.o2"
# Two enqueues racing on the SAME store. Without the lock both read
# observed_seq=0 and allocate 1 (aliasing + a lost write); the lock serializes
# the allocate+write so they get 1 and 2.
"$STORE" enqueue --class actionable --locators '{"c":1}' >"$o1" 2>/dev/null &
# --- deterministic start barrier (#923 de-flake; NOT a sleep) --------------
# A plain `cmd & cmd & wait` gives no guarantee the two enqueues are ever
# truly in-flight together — the scheduler can run one to completion before
# the other is even forked, so the flock contention this test exists to
# prove is never really exercised. Force REAL concurrency: both children
# block on a shared start gate (an flock'd file — no sleep, no timing
# guess) until BOTH have confirmed they are launched and waiting; only then
# does the parent release the gate, so both race into store.sh's own
# enqueue lock at (as close to) the same instant as the scheduler allows.
gate="$TMP_ROOT/t10.gate"
r1="$TMP_ROOT/t10.ready1"
r2="$TMP_ROOT/t10.ready2"
rm -f "$r1" "$r2"
exec 6>"$gate"
flock -x 6 # gate CLOSED: a shared-locker below blocks here until released.
(
: >"$r1" # "child 1 is launched and about to block on the gate"
exec 7<"$gate"
flock -s 7 # blocks until the parent drops its exclusive lock
flock -u 7
exec 7<&-
"$STORE" enqueue --class actionable --locators '{"c":1}'
) >"$o1" 2>/dev/null &
p1=$!
"$STORE" enqueue --class actionable --locators '{"c":2}' >"$o2" 2>/dev/null &
(
: >"$r2"
exec 7<"$gate"
flock -s 7
flock -u 7
exec 7<&-
"$STORE" enqueue --class actionable --locators '{"c":2}'
) >"$o2" 2>/dev/null &
p2=$!
# Busy-poll (NOT a sleep — no timing assumption) until BOTH children have
# confirmed they are launched and blocked on the gate, so the contention
# window below is genuine rather than incidental.
spins=0
while [ ! -e "$r1" ] || [ ! -e "$r2" ]; do
spins=$((spins + 1))
if [ "$spins" -gt 2000000 ]; then
fail_msg "T10: start barrier never saw both children launch — cannot construct genuine concurrency"
break
fi
done
# Release the gate: both children's pending shared-lock acquisitions are
# granted together, so they race into store.sh's OWN enqueue lock for real.
flock -u 6
exec 6>&-
wait "$p1"; wait "$p2"
a="$(tr -d '[:space:]' <"$o1")"
b="$(tr -d '[:space:]' <"$o2")"