mosaicstack/stack
1
0
Fork 1
Code Issues 109 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

flaky-test: wake detector D4 lock re-acquire races the holder's exit (intermittent local red) #966

New Issue
Open
opened 2026-07-30 21:36:24 +00:00 by mos-dt-0 · 0 comments
mos-dt-0 commented 2026-07-30 21:36:24 +00:00
Contributor
Copy Link
Copy Source

Observed while running the wake suites for #952 (branch fix/store-quarantine-audit-residuals-952, base 6a7fce3 = current main; detector.sh and test-wake-detector.sh byte-identical to main, so pre-existing by construction).

Symptom

test-wake-detector.sh D4 ("single-instance flock") intermittently fails:

FAIL: D4: a new instance should acquire the lock once the holder is gone

Frequency on sb-it-1-dt: 2 failures in ~5 consecutive runs (runs 1–2 green, run 3 red; an earlier full-suite pass also hit it once). All other 12 groups green on every run.

Shape

Classic lock-handoff timing race in the test, same family as #860/#897/#899: the assertion re-acquires immediately after killing/ending the holder, and occasionally the kernel has not yet released the flock (or the holder process is still exiting) when the second instance probes. The detector's actual single-instance invariant is not in question — the first half of D4 (second instance refused while held) has never been observed failing.

Suggested fix

Bounded retry probe around the re-acquire assertion (the #898 wait_ready connect-probe pattern, applied to flock): poll acquire with a short deadline (e.g. up to 2 s in 100 ms steps) instead of a single immediate attempt. Failing after the deadline stays a hard red — this narrows the window, it does not mask a genuine held-forever regression.

Found-by: pepper (sb-it-1-dt), while validating #952 (no code overlap — filed separately per the flaky-test discipline).

Observed while running the wake suites for #952 (branch `fix/store-quarantine-audit-residuals-952`, base `6a7fce3` = current main; `detector.sh` and `test-wake-detector.sh` byte-identical to main, so pre-existing by construction). ## Symptom `test-wake-detector.sh` D4 ("single-instance flock") intermittently fails: ``` FAIL: D4: a new instance should acquire the lock once the holder is gone ``` Frequency on sb-it-1-dt: 2 failures in ~5 consecutive runs (runs 1–2 green, run 3 red; an earlier full-suite pass also hit it once). All other 12 groups green on every run. ## Shape Classic lock-handoff timing race in the test, same family as #860/#897/#899: the assertion re-acquires immediately after killing/ending the holder, and occasionally the kernel has not yet released the flock (or the holder process is still exiting) when the second instance probes. The detector's actual single-instance invariant is not in question — the first half of D4 (second instance refused while held) has never been observed failing. ## Suggested fix Bounded retry probe around the re-acquire assertion (the #898 `wait_ready` connect-probe pattern, applied to flock): poll acquire with a short deadline (e.g. up to 2 s in 100 ms steps) instead of a single immediate attempt. Failing after the deadline stays a hard red — this narrows the window, it does not mask a genuine held-forever regression. Found-by: pepper (sb-it-1-dt), while validating #952 (no code overlap — filed separately per the flaky-test discipline).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#966