fix(lease-broker): de-flake recovery_runtime b2 broker-socket ConnectionRefused race (#849) #851

2026-07-20T06:27:17Z

jason.woltje commented

2026-07-20 06:27:17 +00:00

Summary

Fixes the environmental per-test broker-socket ConnectionRefused race in recovery_runtime_unittest.py (b2), which surfaced as intermittent RED on descendant-main push CI. Fixture-lifecycle only: the test request() helper now bounds the Unix-socket connect with a 5s deadline retry that catches only ConnectionRefusedError and re-raises the real failure at the deadline.

Closes #849.

Scope

Touches only packages/mosaic/src/lease-broker/recovery_runtime_unittest.py (test fixture). Zero production lease-broker / recovery-runtime source changed.
b2 security assertion + transport-trust boundary byte-preserved — SHA-256 verified identical at base and head.

Review provenance — three distinct lanes (author != code-reviewer != security-reviewer)

CODE APPROVE — RoR sha256 03234bcf295834e7c98f6ade669976456f9bdabfa15b50badb70ac0d415045c8. Detached exact-head review; AC1-6 + C1-5 all PASS; red-first independently reproduced (base helper RED / head GREEN under an injected pre-listen window); 6x stable; 8 sibling unittests pass; Codex approve 0.94 / 0 findings.
FOCUSED SECREV — ATTEST-CLEAN — RoR sha256 cdf4c4d4d143dabfdfcf5005182ce5264644e61f12d1513d7a29f0618ddb206c. S1-S5 all PASS; b2 assertion + WI-6 production transport blobs byte-identical at base/head; retry catches only ConnectionRefusedError; finite deadline re-raises real failures (independently probed).

CI note — read before relying on PR-CI

PR-event CI does not execute recovery_runtime_unittest.py (the pull_request pipeline does not invoke it). PR-CI green here is the general merge gate only. The recovery_runtime GREEN datapoint that validates this fix comes from the post-merge descendant-main push pipeline. Do not treat PR-CI green as proof the flake is fixed.

No self-merge; no completion claim. Merge and FCM-M5-001 completion ownership retained by the coordinator.

## Summary Fixes the environmental per-test broker-socket `ConnectionRefused` race in `recovery_runtime_unittest.py` (b2), which surfaced as intermittent RED on descendant-`main` push CI. **Fixture-lifecycle only:** the test `request()` helper now bounds the Unix-socket connect with a 5s deadline retry that catches **only** `ConnectionRefusedError` and re-raises the real failure at the deadline. Closes #849. ## Scope - Touches **only** `packages/mosaic/src/lease-broker/recovery_runtime_unittest.py` (test fixture). Zero production lease-broker / recovery-runtime source changed. - b2 security assertion + transport-trust boundary **byte-preserved** — SHA-256 verified identical at base and head. ## Review provenance — three distinct lanes (author != code-reviewer != security-reviewer) - **CODE APPROVE** — RoR sha256 `03234bcf295834e7c98f6ade669976456f9bdabfa15b50badb70ac0d415045c8`. Detached exact-head review; AC1-6 + C1-5 all PASS; red-first independently reproduced (base helper RED / head GREEN under an injected pre-listen window); 6x stable; 8 sibling unittests pass; Codex approve 0.94 / 0 findings. - **FOCUSED SECREV — ATTEST-CLEAN** — RoR sha256 `cdf4c4d4d143dabfdfcf5005182ce5264644e61f12d1513d7a29f0618ddb206c`. S1-S5 all PASS; b2 assertion + WI-6 production transport blobs byte-identical at base/head; retry catches only `ConnectionRefusedError`; finite deadline re-raises real failures (independently probed). ## CI note — read before relying on PR-CI PR-event CI does **not** execute `recovery_runtime_unittest.py` (the `pull_request` pipeline does not invoke it). PR-CI green here is the **general merge gate only**. The `recovery_runtime` GREEN datapoint that validates this fix comes from the **post-merge descendant-`main` push** pipeline. Do not treat PR-CI green as proof the flake is fixed. No self-merge; no completion claim. Merge and FCM-M5-001 completion ownership retained by the coordinator.

jason.woltje added 1 commit 2026-07-20 06:27:18 +00:00

test(mosaic): wait for broker socket readiness

ci/woodpecker/pr/ci Pipeline was successful

Details

cbacd10a66

jason.woltje commented

2026-07-20 06:28:23 +00:00

Durable exact-head review binding — do not merge without the coordinator 6-check

Exact head bound: cbacd10a6601b223cb61d5303b4761e0385a54ee

Three distinct review lanes (author != code-reviewer != security-reviewer):

CODE APPROVE — RoR sha256 03234bcf295834e7c98f6ade669976456f9bdabfa15b50badb70ac0d415045c8. Detached exact-head; AC1-6 + C1-5 PASS; red-first independently reproduced (base helper RED / head GREEN under an injected pre-listen window); 6x stable; 8 sibling unittests pass; Codex approve 0.94 / 0 findings.
FOCUSED SECREV — ATTEST-CLEAN — RoR sha256 cdf4c4d4d143dabfdfcf5005182ce5264644e61f12d1513d7a29f0618ddb206c. S1-S5 PASS; b2 assertion + WI-6 production transport blobs byte-identical at base/head; retry catches only ConnectionRefusedError; finite deadline re-raises real failures (independently probed).

CI distinction: PR-event CI does not run recovery_runtime_unittest.py (proven by prior pipeline #1907). PR-CI green = general merge gate only. The recovery_runtime GREEN datapoint that validates this fix arrives from the post-merge descendant-main push pipeline (the FCM-M5-001 completion gate). This binding does not claim FCM-M5-001 done.

Posted via Gitea REST (the framework pr-review.sh comment path is the subject of #812).

### Durable exact-head review binding — do not merge without the coordinator 6-check **Exact head bound:** `cbacd10a6601b223cb61d5303b4761e0385a54ee` Three distinct review lanes (author != code-reviewer != security-reviewer): - **CODE APPROVE** — RoR sha256 `03234bcf295834e7c98f6ade669976456f9bdabfa15b50badb70ac0d415045c8`. Detached exact-head; AC1-6 + C1-5 PASS; red-first independently reproduced (base helper RED / head GREEN under an injected pre-listen window); 6x stable; 8 sibling unittests pass; Codex approve 0.94 / 0 findings. - **FOCUSED SECREV — ATTEST-CLEAN** — RoR sha256 `cdf4c4d4d143dabfdfcf5005182ce5264644e61f12d1513d7a29f0618ddb206c`. S1-S5 PASS; b2 assertion + WI-6 production transport blobs byte-identical at base/head; retry catches only `ConnectionRefusedError`; finite deadline re-raises real failures (independently probed). **CI distinction:** PR-event CI does **not** run `recovery_runtime_unittest.py` (proven by prior pipeline #1907). PR-CI green = general merge gate only. The `recovery_runtime` GREEN datapoint that validates this fix arrives from the **post-merge descendant-`main` push** pipeline (the FCM-M5-001 completion gate). This binding does not claim FCM-M5-001 done. Posted via Gitea REST (the framework `pr-review.sh` comment path is the subject of #812).

jason.woltje merged commit 77c9a82614 into main

2026-07-20 06:44:38 +00:00

jason.woltje referenced this issue from a commit

2026-07-20 06:44:40 +00:00

fix(lease-broker): de-flake recovery_runtime b2 broker-socket ConnectionRefused race (#849) (#851)

jason.woltje referenced this pull request

2026-07-20 07:13:18 +00:00

docs(tasks): FCM-M5-001 done — verified completion evidence (supersedes #848) #853

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#851