From 93c1de51e113e5a8bafe79a677357e05c03c45d0 Mon Sep 17 00:00:00 2001 From: fred Date: Sun, 16 Aug 2026 17:58:49 -0500 Subject: [PATCH 1/2] fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269) The `test` step has failed on every `next` pipeline since #1017 on exactly one assertion, and it is the same one on unrelated PRs: FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here (framework/tools/fleet/test-start-agent-session.sh:103) Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441 (#1017-quality): exactly one FAIL line in each full log, identical, this line. Control `zzz-not-present-zzz` -> 0 on all three. Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into $FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so on a host that installs those binaries the missing-binary cases cannot be measured and a green run would mean nothing. The guard says so instead of passing. Its own pipeline 2430 was green only because the suite was CI-excluded at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated it and dropped the exclusion. The CI image installs @earendil-works/pi-coding-agent@0.84.1 on purpose, so the precondition is unsatisfiable there. Both commits are mine. The guard is correct and is not being softened. A check that cannot measure its property and reports success is the failure mode this repo has been cataloguing all week; the error was wiring the suite into an image that violates its precondition, so the wiring is what gets reverted. Second effect, which is the reason this cost a day rather than an hour: test:framework-shell is one && chain and this sat at position 44 of 48, so glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh, woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh have not run at all since the merge. The pipeline reported one failure, never "one failure plus four unrun". All four are green when run directly on sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one host, not a CI-image result. Verification, with controls: - enumeration guard OK (population 52, enumerated 36, signed-excluded 16). - control A, exclusion line removed while unwired -> FAIL UNENUMERATED. - control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION. The gate discriminates in both directions, so its OK is load-bearing. - the four formerly-masked suites: rc=0 each, run directly. - the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at the lease-broker Invariant R test, because this host carries the quarantined operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and out of scope here -- CI pins 0.84.1, and the single FAIL line in those three pipelines proves positions 1-43 passed there. Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi from the image. Recorded in the exclusion reason and in #1269. --- .../quality/test-enumeration-exclusions.txt | 16 ++++++++++++++++ packages/mosaic/package.json | 2 +- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt b/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt index 54f6ffe5..dc409e45 100644 --- a/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt +++ b/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt @@ -39,3 +39,19 @@ packages/mosaic/framework/tools/tmux/test-send-message-verdict.sh | requires rea # recorded judgement. These lines ARE that judgement, signed.) packages/mosaic/framework/tools/orchestrator/smoke-test.sh | behavior smoke checks for coord continue/run workflows, run manually by orchestrator seats; unmeasured in CI; #1017 burndown packages/mosaic/framework/tools/wake/validate-973/microtest-wake-assert.sh | #973 instrument self-test, run as a precondition of the validate-973 evidence procedure rather than as a standing CI suite; #1017 burndown candidate + +# --- tools/fleet: precondition is unsatisfiable in the CI image (#1269) --- +# Signed by fred (sb-it-1-dt, 2026-08-16) at origin/next 476db12. +# This suite asserts the launcher's behaviour when `mosaic` and `pi` are MISSING. +# It shims fakes into $FAKE_BIN, but the constructed PANE_PATH always ends in the +# real system path, so on a host that installs those binaries the missing-binary +# cases cannot be measured at all. The suite's own guard (line 103) says so and +# fails rather than reporting a pass it cannot back. That guard is correct. +# The error was wiring the suite into CI: #1017 (c56483eb) enumerated it and +# dropped this exclusion, and the CI image provides `pi` in the system path, so +# it has failed on every pipeline since. Measured 2026-08-16 across pipelines +# 2444 (#1256), 2438 (#1240) and 2441 (#1017-quality): exactly one FAIL line in +# each full log, identical, this assertion; control `zzz-not-present-zzz` -> 0. +# Because test:framework-shell is one && chain and this sat at position 44 of 48, +# the four suites after it had not run at all since the merge. +packages/mosaic/framework/tools/fleet/test-start-agent-session.sh | precondition unsatisfiable in the CI image: asserts missing-binary behaviour, but PANE_PATH always ends in the system path and the image provides `pi` there; guard at line 103 fails by design rather than passing unmeasured. Burn down by controlling the tail of PANE_PATH inside the test. NOT by removing `pi` from the image: the CI image installs @earendil-works/pi-coding-agent@0.84.1 deliberately (measured in pipeline 2444's test-step log), and other suites depend on that pin. See #1269 diff --git a/packages/mosaic/package.json b/packages/mosaic/package.json index 98cf494f..21ce7cf2 100644 --- a/packages/mosaic/package.json +++ b/packages/mosaic/package.json @@ -25,7 +25,7 @@ "lint": "eslint src", "typecheck": "tsc --noEmit", "test": "vitest run --passWithNoTests && pnpm run test:framework-shell", - "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh" + "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh" }, "dependencies": { "@mosaicstack/brain": "workspace:*", From 57a2f2b40e3fa12b29fdde32efb4d2cee0d96353 Mon Sep 17 00:00:00 2001 From: fred Date: Sun, 16 Aug 2026 18:03:02 -0500 Subject: [PATCH 2/2] docs(ci): point the exclusion at tracking issue #1271, not the closed first filing The first PR for this change was filed under the retired mos-dt-0 principal (pr-create.sh has no --login flag and find_tea_login_for_host returns the first host match) and was closed and refiled as #1270. That left in-tree references pointing at a closed duplicate PR rather than at the burn-down issue, which is the wrong target for them anyway: the open design question belongs on #1271. --- .../framework/tools/quality/test-enumeration-exclusions.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt b/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt index dc409e45..bbd1b34d 100644 --- a/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt +++ b/packages/mosaic/framework/tools/quality/test-enumeration-exclusions.txt @@ -40,7 +40,7 @@ packages/mosaic/framework/tools/tmux/test-send-message-verdict.sh | requires rea packages/mosaic/framework/tools/orchestrator/smoke-test.sh | behavior smoke checks for coord continue/run workflows, run manually by orchestrator seats; unmeasured in CI; #1017 burndown packages/mosaic/framework/tools/wake/validate-973/microtest-wake-assert.sh | #973 instrument self-test, run as a precondition of the validate-973 evidence procedure rather than as a standing CI suite; #1017 burndown candidate -# --- tools/fleet: precondition is unsatisfiable in the CI image (#1269) --- +# --- tools/fleet: precondition is unsatisfiable in the CI image (#1271) --- # Signed by fred (sb-it-1-dt, 2026-08-16) at origin/next 476db12. # This suite asserts the launcher's behaviour when `mosaic` and `pi` are MISSING. # It shims fakes into $FAKE_BIN, but the constructed PANE_PATH always ends in the @@ -52,6 +52,7 @@ packages/mosaic/framework/tools/wake/validate-973/microtest-wake-assert.sh | #97 # it has failed on every pipeline since. Measured 2026-08-16 across pipelines # 2444 (#1256), 2438 (#1240) and 2441 (#1017-quality): exactly one FAIL line in # each full log, identical, this assertion; control `zzz-not-present-zzz` -> 0. +# Burn-down and the full measurement are tracked in #1271; unwired by PR #1270. # Because test:framework-shell is one && chain and this sat at position 44 of 48, # the four suites after it had not run at all since the merge. -packages/mosaic/framework/tools/fleet/test-start-agent-session.sh | precondition unsatisfiable in the CI image: asserts missing-binary behaviour, but PANE_PATH always ends in the system path and the image provides `pi` there; guard at line 103 fails by design rather than passing unmeasured. Burn down by controlling the tail of PANE_PATH inside the test. NOT by removing `pi` from the image: the CI image installs @earendil-works/pi-coding-agent@0.84.1 deliberately (measured in pipeline 2444's test-step log), and other suites depend on that pin. See #1269 +packages/mosaic/framework/tools/fleet/test-start-agent-session.sh | precondition unsatisfiable in the CI image: asserts missing-binary behaviour, but PANE_PATH always ends in the system path and the image provides `pi` there; guard at line 103 fails by design rather than passing unmeasured. Burn down by controlling the tail of PANE_PATH inside the test. NOT by removing `pi` from the image: the CI image installs @earendil-works/pi-coding-agent@0.84.1 deliberately (measured in pipeline 2444's test-step log), and other suites depend on that pin. Burn-down tracked in #1271