fix(wake): #927 enqueue TOCTOU — move stale-tmp cleanup off the hot enqueue path
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
cmd_enqueue called _wake_init_dir() (which reaped EVERY .wake.tmp.*
unconditionally) BEFORE taking the enqueue lock. So a 2nd enqueue's PRE-LOCK
cleanup deleted the LIVE in-flight tmp of a 1st enqueue that held the lock
through its atomic write -> spurious "durable pending write FAILED" abort of a
valid enqueue. Reachable under live co-feed (detector + reconciler concurrently
enqueue).
Fix (scalpel, Mos's lean = off the hot path):
- _wake_init_dir no longer reaps tmps; it only ensures the layout. Nothing on
the enqueue/consume/cursors/ack hot paths can clobber a concurrent live write.
- _wake_clean_stale_tmp is AGE-SCOPED (-mmin +${WAKE_TMP_STALE_MIN:-5}) so it
can only remove demonstrably-orphaned crash-left tmps, never a live (ms-old)
in-flight write — safe even if it ever overlaps a concurrent enqueue.
- Reaping runs as an explicit MAINTENANCE action at store.sh init (daemon-start)
and the detector poll tick, bounding accumulation once-per-pass, not raced
per-enqueue.
#908 seq-integrity is unchanged (single store-side allocator, atomic
allocate+enqueue under flock, arrow-1 no-burn, anti-swallow fail-loud).
reconcile.sh is unchanged: an aborted enqueue advances neither the seen-ledger
nor observed_seq, so the source is re-enumerated next reconcile cycle (no
obligation loss).
Tests: new deterministic RED->GREEN harness test-wake-store-enqueue-race.sh
(PATH-shadowed mv/flock freeze enqueue A at its rename while B runs its pre-lock
cleanup — RED pre-fix: A aborts; GREEN post-fix: both succeed, distinct gapless
seqs {1,2}, zero spurious aborts). T5 updated to assert the hot path no longer
reaps tmps and that the maintenance path reaps a demonstrably-orphaned tmp. All
store-ack groups incl T10 stay green. manifest 0.6.4 -> 0.6.5.
Closes #927
Part of #892
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -170,11 +170,27 @@ echo "== T5: atomic write-tmp+rename survives crash mid-write =="
|
||||
printf '%s\n' "$drained" | jq -e . >/dev/null 2>&1 || fail_msg "T5: drain returned non-JSON — garbage temp corrupted the store"
|
||||
printf '%s\n' "$drained" | stream_any '.observed_seq==1' || fail_msg "T5: committed entry (seq 1) lost after simulated crash"
|
||||
printf '%s\n' "$drained" | grep -q 999 && fail_msg "T5: uncommitted garbage (seq 999) leaked into the live store"
|
||||
# A subsequent real mutation must succeed and clean the stale temp.
|
||||
# A subsequent real mutation must succeed DESPITE the stale temp present.
|
||||
"$STORE" enqueue --seq 2 --class actionable --locators '{"issue":2}' >/dev/null || fail_msg "T5: enqueue after crash temp failed"
|
||||
[ -e "$STATE_DIR/.wake.tmp.crash12" ] && fail_msg "T5: stale crash temp not cleaned by next mutation"
|
||||
# #927: the enqueue HOT PATH must NOT reap tmp files — an unconditional delete
|
||||
# there clobbered a concurrent enqueue's LIVE in-flight tmp mid-write (spurious
|
||||
# "durable pending write FAILED"). So a FRESH stale tmp is intentionally left
|
||||
# untouched by an enqueue; reaping it on the hot path is exactly the bug.
|
||||
[ -e "$STATE_DIR/.wake.tmp.crash12" ] || fail_msg "T5: the enqueue hot path must NOT delete tmp files off its own write (that hot-path reap was the #927 clobber)"
|
||||
d2="$("$STORE" drain)"
|
||||
[ "$(printf '%s\n' "$d2" | grep -c .)" = "2" ] || fail_msg "T5: store not healthy after crash+recovery (expected 2 entries)"
|
||||
# Bounded accumulation is preserved via a MAINTENANCE reap (store.sh init /
|
||||
# detector tick), age-scoped so it only removes DEMONSTRABLY-orphaned tmps
|
||||
# (older than any plausible in-flight write) and never a live one. Age the
|
||||
# crash temp into the past so it is unambiguously orphaned, then run the
|
||||
# maintenance path and confirm it is reaped.
|
||||
touch -d '1 hour ago' "$STATE_DIR/.wake.tmp.crash12" 2>/dev/null ||
|
||||
touch -t "$(date -d '1 hour ago' +%Y%m%d%H%M.%S 2>/dev/null || echo 197001010000)" "$STATE_DIR/.wake.tmp.crash12" 2>/dev/null || true
|
||||
"$STORE" init >/dev/null 2>&1 || fail_msg "T5: store.sh init (maintenance) failed"
|
||||
[ -e "$STATE_DIR/.wake.tmp.crash12" ] && fail_msg "T5: maintenance reap (store.sh init) must remove a demonstrably-orphaned stale temp (bounded accumulation)"
|
||||
d3="$("$STORE" drain)"
|
||||
[ "$(printf '%s\n' "$d3" | grep -c .)" = "2" ] || fail_msg "T5: store not healthy after maintenance reap (expected 2 entries)"
|
||||
printf '%s\n' "$d3" | jq -e . >/dev/null 2>&1 || fail_msg "T5: drain returned non-JSON after maintenance reap"
|
||||
) && ok
|
||||
|
||||
echo "== T5b: _atomic_write is tmp+rename (killing the writer mid-write leaves the target intact) =="
|
||||
|
||||
Reference in New Issue
Block a user