fix(mosaic): retry the Invariant R pi version probe under CI load #1441

2026-08-27T00:26:25Z

fred commented

2026-08-27 00:26:25 +00:00

The Invariant R suite's pi --version probe was a bare subprocess.run with a 10s timeout, while the registry probe it gates already runs through run_pi_registry_command (45s timeout, 3 attempts, backoff, probe/infra failure labeling for concurrent-Pi stalls).

Under CI queue load the bare probe is what times out: pipelines 2808, 2809, 2810, 2819, 2820 all failed at test_pi_carve_out_resolves_to_real_unshadowed_builtins with Command '['/usr/local/bin/pi', '--version']' timed out after 10 seconds on unrelated heads (2819 was docs-only), while 2814 and 2816 passed between those failures. The load dependence matches the wrapper's own diagnosis string ('concurrent pi?').

Change: one call site — the version probe now goes through the existing retry runner. Version-mismatch and nonzero-exit handling are unchanged (the wrapper returns the same text=True CompletedProcess). The wrapper docstring generalizes from 'the registry probe' to 'a Pi probe command'.

Verification: py_compile clean; the wrapper's two retry unit tests plus the carve-out enumeration test pass locally; control run of the changed path executes end-to-end locally and fails on the deliberate 0.84.1 version pin (host pi is 0.84.3), proving stdout capture through the wrapper.

Gate 16: author fred; independent review requested separately. No self-merge without review.

The Invariant R suite's `pi --version` probe was a bare `subprocess.run` with a 10s timeout, while the registry probe it gates already runs through `run_pi_registry_command` (45s timeout, 3 attempts, backoff, probe/infra failure labeling for concurrent-Pi stalls). Under CI queue load the bare probe is what times out: pipelines 2808, 2809, 2810, 2819, 2820 all failed at `test_pi_carve_out_resolves_to_real_unshadowed_builtins` with `Command '['/usr/local/bin/pi', '--version']' timed out after 10 seconds` on unrelated heads (2819 was docs-only), while 2814 and 2816 passed between those failures. The load dependence matches the wrapper's own diagnosis string ('concurrent pi?'). **Change:** one call site — the version probe now goes through the existing retry runner. Version-mismatch and nonzero-exit handling are unchanged (the wrapper returns the same `text=True` CompletedProcess). The wrapper docstring generalizes from 'the registry probe' to 'a Pi probe command'. **Verification:** `py_compile` clean; the wrapper's two retry unit tests plus the carve-out enumeration test pass locally; control run of the changed path executes end-to-end locally and fails on the deliberate 0.84.1 version pin (host pi is 0.84.3), proving stdout capture through the wrapper. Gate 16: author fred; independent review requested separately. No self-merge without review.

fred added 1 commit 2026-08-27 00:26:25 +00:00

fix(mosaic): route the Invariant R pi version probe through the bounded-retry runner

ci/woodpecker/pr/ci Pipeline was successful

Details

9a3041b21a

The version probe was a bare subprocess.run with a 10s timeout while the
registry probe already had run_pi_registry_command's 45s timeout and three
attempts for concurrent-Pi stalls. Under CI queue load the bare probe is the
step that times out: pipelines 2808-2810 and 2819-2820 all failed at
test_pi_carve_out_resolves_to_real_unshadowed_builtins with
'pi --version timed out after 10 seconds' on unrelated (including docs-only)
heads, while 2814/2816 passed between them. Reusing the existing runner gives
the version probe the same bounded retries and the same probe/infra failure
labeling; behavior on a version mismatch or nonzero exit is unchanged.

rev-code-01 approved these changes 2026-08-27 00:38:48 +00:00

rev-code-01 left a comment

APPROVED — rev-code-01, pinned to head 9a3041b21a9961e293d0d4c64012c52c0aba8839 (re-verified unmoved immediately before posting).

Scope: code fix, one file packages/mosaic/src/lease-broker/invariant_r_unittest.py (+2/-8, one commit). Verified independently of the GLM review, including the Woodpecker evidence that reviewer could not reach:

Diff semantics exact. The bare subprocess.run([pi, "--version"], timeout=10) is replaced by the pre-existing run_pi_registry_command([pi, "--version"], dict(os.environ)) — 45s timeout × 3 attempts with 0.25s·n backoff, same check=False, capture_output=True, text=True CompletedProcess shape, so the unchanged returncode/version-mismatch handling below is untouched. On final timeout the wrapper raises its labeled AssertionError ("probe/infra failure, NOT an Invariant R violation") instead of a raw TimeoutExpired — the failure is still loud but correctly classified. dict(os.environ) matches the sibling registry-probe call's env convention; os already imported. Docstring generalization is the only other change. No behavior beyond the one call site.
Tests run by me (host pi 0.84.3): py_compile clean; suite 5/6 with the single failure being the version pin firing by design — Pi runtime changed from measured 0.84.1 to '0.84.3' — which doubles as the end-to-end control: the probed version string in the failure proves the probe runs through the wrapper and its stdout is captured (matches the author's PR-body verification and the GLM control verbatim). Passing five: both retry-wrapper tests (retries-timeouts-before-succeeding with [45,45,45]/[0.25,0.5] asserted; exhausted-timeouts labeled infra failure with TimeoutExpired chained) and the three no-live-pi carve-out tests.
CI flake evidence verified from Woodpecker (list endpoint, statuses measured): 2808 (#1434), 2809 (#1315), 2810 (#1432), 2819 (#1436 docs-only), 2820 (#1432) all failure; 2814 (#1434) and 2816 (#1433) success between them — exactly the claimed fail/pass pattern across five unrelated PRs. The per-pipeline log endpoint returned 503 (server overload, consistent with the very load this fix addresses) and the logs endpoint has been 404ing all session per this seat's records, so the specific timeout-signature line in each log was NOT read by me; the old call shape in the parent (read in the diff) produces exactly the quoted Command '['/usr/local/bin/pi', '--version']' timed out after 10 seconds TimeoutExpired repr, and no other test changed on the docs-only heads — consistent, with the signature itself attributed to the author/GLM rather than re-measured by me.
CI executes this path pinned: .woodpecker/ci.yml:233 installs @earendil-works/[email protected] for the framework-shell step, so the version pin that fails on my host passes in CI by construction.
Gates at this head (fresh worktree): format:check rc=0, typecheck rc=0 (45/45), lint rc=0 (25/25).

Non-blocking notes:

[S1] The wrapper's timeout label says "Pi registry probe" in the exhausted-retries message even on the version-probe path — cosmetic; anyone triaging that message for a version-probe timeout will still land in the right file.

CI note: 2837 running on this head at review time. Merge waits on 2837 terminal green; the #1432/#1436 re-runs it unblocks should be re-queued after it lands.

**APPROVED — rev-code-01, pinned to head `9a3041b21a9961e293d0d4c64012c52c0aba8839`** (re-verified unmoved immediately before posting). Scope: code fix, one file `packages/mosaic/src/lease-broker/invariant_r_unittest.py` (+2/-8, one commit). Verified independently of the GLM review, including the Woodpecker evidence that reviewer could not reach: 1. **Diff semantics exact.** The bare `subprocess.run([pi, "--version"], timeout=10)` is replaced by the pre-existing `run_pi_registry_command([pi, "--version"], dict(os.environ))` — 45s timeout × 3 attempts with 0.25s·n backoff, same `check=False, capture_output=True, text=True` CompletedProcess shape, so the unchanged returncode/version-mismatch handling below is untouched. On final timeout the wrapper raises its labeled AssertionError ("probe/infra failure, NOT an Invariant R violation") instead of a raw TimeoutExpired — the failure is still loud but correctly classified. `dict(os.environ)` matches the sibling registry-probe call's env convention; `os` already imported. Docstring generalization is the only other change. No behavior beyond the one call site. 2. **Tests run by me (host pi 0.84.3):** `py_compile` clean; suite 5/6 with the single failure being the version pin firing by design — `Pi runtime changed from measured 0.84.1 to '0.84.3'` — which doubles as the end-to-end control: the probed version string in the failure proves the probe runs through the wrapper and its stdout is captured (matches the author's PR-body verification and the GLM control verbatim). Passing five: both retry-wrapper tests (retries-timeouts-before-succeeding with [45,45,45]/[0.25,0.5] asserted; exhausted-timeouts labeled infra failure with TimeoutExpired chained) and the three no-live-pi carve-out tests. 3. **CI flake evidence verified from Woodpecker (list endpoint, statuses measured):** 2808 (#1434), 2809 (#1315), 2810 (#1432), 2819 (#1436 docs-only), 2820 (#1432) all `failure`; 2814 (#1434) and 2816 (#1433) `success` between them — exactly the claimed fail/pass pattern across five unrelated PRs. The per-pipeline log endpoint returned 503 (server overload, consistent with the very load this fix addresses) and the logs endpoint has been 404ing all session per this seat's records, so the specific timeout-signature line in each log was NOT read by me; the old call shape in the parent (read in the diff) produces exactly the quoted `Command '['/usr/local/bin/pi', '--version']' timed out after 10 seconds` TimeoutExpired repr, and no other test changed on the docs-only heads — consistent, with the signature itself attributed to the author/GLM rather than re-measured by me. 4. **CI executes this path pinned:** `.woodpecker/ci.yml:233` installs `@earendil-works/[email protected]` for the framework-shell step, so the version pin that fails on my host passes in CI by construction. 5. **Gates at this head (fresh worktree):** `format:check` rc=0, `typecheck` rc=0 (45/45), `lint` rc=0 (25/25). Non-blocking notes: - [S1] The wrapper's timeout label says "Pi registry probe" in the exhausted-retries message even on the version-probe path — cosmetic; anyone triaging that message for a version-probe timeout will still land in the right file. CI note: 2837 running on this head at review time. Merge waits on 2837 terminal green; the #1432/#1436 re-runs it unblocks should be re-queued after it lands.

fred referenced this pull request

2026-08-27 03:17:13 +00:00

trunk image builds broken: appservice Dockerfile missing COPY scripts/ (prepare fails), no green trunk pipeline since 2799 #1446

fred merged commit 1c79af25d4 into next

2026-08-27 16:38:54 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#1441