mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

wake/store T10 concurrency flake (pre-existing): intermittent fail on parallel-enqueue race — de-flake the synchronization #923

New Issue
Closed
opened 2026-07-26 08:41:03 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-07-26 08:41:03 +00:00
Owner
Copy Link

Pre-existing concurrency flake in the W2 store/ack harness, surfaced (NOT caused) by #922's review — store.sh + test-wake-store-ack.sh are byte-identical to main, and T10 failed once then passed 3/3 on rerun both in the review sandbox AND against main.

T10 ("two concurrent store.sh enqueue get DISTINCT seqs (flock)") in test-wake-store-ack.sh occasionally fails under load (a concurrency/timing race in the test's parallel-enqueue setup or its assertion), then passes on rerun — same non-deterministic class as the T7 timing flake fixed in #918. It can intermittently red-fail an unrelated PR's CI (head-of-line for merges).

FIX (test determinism, do NOT weaken the invariant): make T10's concurrent-enqueue race deterministic — ensure both background enqueues are genuinely started before the barrier, wait on both PIDs, and assert distinct+gapless seqs without a timing-sensitive window; if the flake is in the flock contention window, harden the test's synchronization (e.g. a start-barrier both children block on) rather than the product (store.sh flock is correct — the review confirmed distinct seqs on every non-flaked run). Red-first: the de-flaked T10 still catches a real aliasing regression (disable flock → RED). Priority: normal. Refs: #892, #918 (sibling T7 de-flake precedent), #922 review.

Pre-existing concurrency flake in the W2 store/ack harness, surfaced (NOT caused) by #922's review — store.sh + test-wake-store-ack.sh are byte-identical to main, and T10 failed once then passed 3/3 on rerun both in the review sandbox AND against main. **T10** ("two concurrent `store.sh enqueue` get DISTINCT seqs (flock)") in `test-wake-store-ack.sh` occasionally fails under load (a concurrency/timing race in the test's parallel-enqueue setup or its assertion), then passes on rerun — same non-deterministic class as the T7 timing flake fixed in #918. It can intermittently red-fail an unrelated PR's CI (head-of-line for merges). **FIX (test determinism, do NOT weaken the invariant):** make T10's concurrent-enqueue race deterministic — ensure both background enqueues are genuinely started before the barrier, wait on both PIDs, and assert distinct+gapless seqs without a timing-sensitive window; if the flake is in the flock contention window, harden the test's synchronization (e.g. a start-barrier both children block on) rather than the product (store.sh flock is correct — the review confirmed distinct seqs on every non-flaked run). Red-first: the de-flaked T10 still catches a real aliasing regression (disable flock → RED). Priority: normal. Refs: #892, #918 (sibling T7 de-flake precedent), #922 review.
Mos closed this issue 2026-07-26 10:23:08 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#923
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?