mosaicstack/stack
1
0
Fork 0
Code Issues 77 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

flake: recovery_runtime_unittest b2 per-test broker-socket ConnectionRefused race (gates FCM-M5-001 completion) #849

New Issue
Closed
opened 2026-07-20 05:52:38 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-07-20 05:52:38 +00:00
Owner
Copy Link

Summary

RecoveryRuntimeBoundaryTest.test_b2_production_observer_promotes_over_private_transport_and_rejects_broker_supplied_message in packages/mosaic/src/lease-broker/recovery_runtime_unittest.py intermittently errors with ConnectionRefusedError: [Errno 111] Connection refused on a per-test Unix-domain-socket connect. This is a pre-existing environmental fixture race, not a content regression. It surfaced as the sole failing case in the completion pipeline for the docs-only merge #789.

Evidence (provider-visible)

  • #1932 push @ 627cf2bb — FAILURE, test step id 49576: b2 case ERROR at recovery_runtime_unittest.py:186 register() → :103 → :36 connection.connect(socket_path) → ConnectionRefusedError [Errno 111]. Same-run sibling marker ..E — 2 of 3 boundary cases PASSED; only b2 errored.
  • #1931 push @ 0582a891 — SUCCESS, test step id 49563: same test invoked and PASSED.
  • #1929 push @ 2509eb76 — SUCCESS, test step id 49541: same test invoked and PASSED.
  • #1907 pull_request @ a39bafb8 (#789 head) — SUCCESS, test step id 49334: recovery_runtime_unittest.py never invoked (PR suite ≠ push suite).

Clincher: git diff 0582a891..627cf2bb -- packages/mosaic/src/lease-broker/ is byte-empty and the test file is byte-identical, yet the identical content PASSED on pushes #1931/#1929 and only b2-errored on #1932. Identical bytes, opposite outcomes → per-test broker-fixture socket-connect race, not a missing service and not a #789 regression.

Required fix (bounded, red-first — do NOT skip/quarantine/mute)

Fix the fixture start/teardown readiness so the test connects deterministically:

  • connect-retry-with-timeout (bounded) and/or an explicit socket-ready wait before register(), and/or serialize per-test broker start so a prior test's teardown cannot race the next test's connect.
  • Preserve the security assertion the test enforces (observer promotes over private transport AND rejects a broker-supplied message). Do not weaken/remove/repoint any assertion.
  • Red-first: demonstrate the race (or a deterministic reproduction of the refused-connect window) fails before the fix and passes after.

Explicitly forbidden: unittest.skip, muting, @expectedFailure, retry-to-launder the completion pipeline. De-flake by fixing, not hiding.

Completion coupling

Per Mos disposition: FCM-M5-001 is CONTENT-DELIVERED & MERGED @ 627cf2bb but completion is PENDING-GREEN — coupled to this fix's descendant-main pipeline running recovery_runtime GREEN (the honest post-merge completion datapoint; 627cf2bb is current tip with no natural later push). No completion is claimed on the current red gate.

Process

Priority p2 (gates a completion + security-surface — above the p3 wrapper bugs). Surface = lease-broker recovery runtime (security-adjacent). terra author ≠ reviewer; independent security review; exact-head CI; Mos merge; then terminal-green descendant-main. Keep OUTSIDE #848 tracking and #812. Single owner/card, no duplicate lane.

## Summary `RecoveryRuntimeBoundaryTest.test_b2_production_observer_promotes_over_private_transport_and_rejects_broker_supplied_message` in `packages/mosaic/src/lease-broker/recovery_runtime_unittest.py` intermittently errors with `ConnectionRefusedError: [Errno 111] Connection refused` on a per-test Unix-domain-socket connect. This is a **pre-existing environmental fixture race**, not a content regression. It surfaced as the sole failing case in the completion pipeline for the docs-only merge #789. ## Evidence (provider-visible) - **#1932** push @ `627cf2bb` — **FAILURE**, `test` step id **49576**: b2 case ERROR at `recovery_runtime_unittest.py:186` `register()` → `:103` → `:36` `connection.connect(socket_path)` → `ConnectionRefusedError [Errno 111]`. Same-run sibling marker `..E` — 2 of 3 boundary cases PASSED; only b2 errored. - **#1931** push @ `0582a891` — SUCCESS, `test` step id **49563**: same test invoked and PASSED. - **#1929** push @ `2509eb76` — SUCCESS, `test` step id **49541**: same test invoked and PASSED. - **#1907** pull_request @ `a39bafb8` (#789 head) — SUCCESS, `test` step id **49334**: `recovery_runtime_unittest.py` **never invoked** (PR suite ≠ push suite). **Clincher:** `git diff 0582a891..627cf2bb -- packages/mosaic/src/lease-broker/` is byte-**empty** and the test file is byte-identical, yet the identical content PASSED on pushes #1931/#1929 and only b2-errored on #1932. Identical bytes, opposite outcomes → per-test broker-fixture socket-connect race, not a missing service and not a #789 regression. ## Required fix (bounded, red-first — do NOT skip/quarantine/mute) Fix the fixture start/teardown readiness so the test connects deterministically: - connect-retry-with-timeout (bounded) and/or an explicit socket-ready wait before `register()`, and/or serialize per-test broker start so a prior test's teardown cannot race the next test's connect. - Preserve the security assertion the test enforces (observer promotes over private transport AND rejects a broker-supplied message). Do not weaken/remove/repoint any assertion. - Red-first: demonstrate the race (or a deterministic reproduction of the refused-connect window) fails before the fix and passes after. **Explicitly forbidden:** `unittest.skip`, muting, `@expectedFailure`, retry-to-launder the completion pipeline. De-flake by fixing, not hiding. ## Completion coupling Per Mos disposition: FCM-M5-001 is CONTENT-DELIVERED & MERGED @ `627cf2bb` but completion is **PENDING-GREEN** — coupled to this fix's descendant-main pipeline running `recovery_runtime` GREEN (the honest post-merge completion datapoint; `627cf2bb` is current tip with no natural later push). No completion is claimed on the current red gate. ## Process Priority **p2** (gates a completion + security-surface — above the p3 wrapper bugs). Surface = lease-broker recovery runtime (security-adjacent). terra author ≠ reviewer; independent security review; exact-head CI; Mos merge; then terminal-green descendant-main. Keep OUTSIDE #848 tracking and #812. Single owner/card, no duplicate lane.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#849
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?