mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

flaky-test: recovery_b1_adversarial_unittest.py wait_ready() TOCTOU race → ECONNREFUSED under CI load #897

New Issue
Closed
opened 2026-07-25 21:44:55 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-07-25 21:44:55 +00:00
Owner
Copy Link

The framework-shell test step intermittently fails with ConnectionRefusedError: [Errno 111] in packages/mosaic/src/lease-broker/recovery_b1_adversarial_unittest.py.

Root cause (from source): setUp() spawns the lease-broker daemon and wait_ready() polls only for the unix-socket FILE to exist (socket_path.exists()), NOT for the daemon to reach listen(). Under concurrent-pipeline runner load the socket file appears before listen() → immediate client connect() gets ECONNREFUSED.

Evidence: origin/main pipeline 2005 test=SUCCESS (1508/1508); PR pipelines 2006/2010/2011 (#893/#894/#895) failed test with vitest still 1508/1508 — only this Python setUp raced; #894 was a trivial 1-line change. Green on main + red on trivial PRs under 4-pipeline load = flaky race, not a defect. OB e945bf77.

Fix (scalpel scope, Mos-ruled): change ONLY wait_ready() — replace file-existence polling with a real connect() retry loop + timeout. NO daemon code, NO assertion changes, NO gate-timeout relaxation (this is the #828/#869 test-locked fail-closed area). Independent review MUST verify every assertion in the recovery suite is byte-unchanged.

The framework-shell `test` step intermittently fails with `ConnectionRefusedError: [Errno 111]` in `packages/mosaic/src/lease-broker/recovery_b1_adversarial_unittest.py`. **Root cause (from source):** `setUp()` spawns the lease-broker daemon and `wait_ready()` polls only for the unix-socket FILE to exist (`socket_path.exists()`), NOT for the daemon to reach `listen()`. Under concurrent-pipeline runner load the socket file appears before `listen()` → immediate client `connect()` gets ECONNREFUSED. **Evidence:** origin/main pipeline 2005 `test`=SUCCESS (1508/1508); PR pipelines 2006/2010/2011 (#893/#894/#895) failed `test` with vitest still 1508/1508 — only this Python setUp raced; #894 was a trivial 1-line change. Green on main + red on trivial PRs under 4-pipeline load = flaky race, not a defect. OB `e945bf77`. **Fix (scalpel scope, Mos-ruled):** change ONLY `wait_ready()` — replace file-existence polling with a real `connect()` retry loop + timeout. NO daemon code, NO assertion changes, NO gate-timeout relaxation (this is the #828/#869 test-locked fail-closed area). Independent review MUST verify every assertion in the recovery suite is byte-unchanged.
Mos closed this issue 2026-07-25 22:10:16 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#897
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?