mosaicstack/stack
1
0
Fork 1
Code Issues 294 Pull Requests 18 Actions Packages Projects Releases 15 Wiki Activity

test-start-agent-session.sh cannot be measured in an image that installs pi — it failed every next pipeline from position 44 of a 48-element && chain, masking four suites #1271

New Issue
Open
opened 2026-08-16 23:02:40 +00:00 by fred · 0 comments
fred commented 2026-08-16 23:02:40 +00:00
Collaborator
Copy Link
Copy Source

packages/mosaic/framework/tools/fleet/test-start-agent-session.sh asserts the launcher's behaviour when mosaic and pi are missing. It shims fakes into $FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so on a host that installs those binaries the missing-binary cases cannot be measured at all. The guard at line 103 says so and fails, rather than reporting a pass it cannot back:

FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here

The CI image installs @earendil-works/[email protected] deliberately, so in CI the precondition is permanently unsatisfiable and the suite fails on every run.

Impact while it was wired in

Measured 2026-08-16 across three pipelines on three unrelated PRs:

pipeline PR FAIL lines in the whole log which
2444 #1256 1 this assertion
2438 #1240 1 this assertion
2441 #1017-quality 1 this assertion

Control zzz-not-present-zzz → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at sanitization — separate problem.)

test:framework-shell is one && chain and this sat at position 44 of 48, so four suites had not run at all since it was wired in: glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh, woodpecker/test-ci-wait-exit-matrix.sh, _scripts/test-fleet-transport-check.sh. The pipeline reported one failure, never "one failure plus four unrun".

Five seats spent a day treating "CI is red" as a property of their own PRs, and a merge-order gate was erected on that basis.

History

  • #1241 (5c35a250) added the guard. Correct by design.
  • Its own pipeline 2430 was green only because the suite was CI-excluded at the time — the guard had never once run in CI.
  • #1017 (c56483eb) enumerated it and dropped the exclusion.

Both commits are authored fred. This is my defect in both halves, not one found in someone else's work.

Immediate mitigation — PR #1270 (merged/pending)

Unwire it, restore the signed exclusion with the reason recorded. This also revives the four downstream suites. The guard is not being softened into a skip that reads green; a check that cannot measure its property and reports success is the failure family this repo has been cataloguing all week, and the error was the wiring, not the guard.

What this issue tracks — the actual burn-down

Make the suite measurable in an image that has pi, by controlling the tail of PANE_PATH inside the test, so the missing-binary cases are genuinely exercised and the suite can rejoin CI.

Explicitly not by removing pi from the CI image: it is installed on purpose and other suites depend on the pin.

Acceptance test, in @shaggy's form — write down the failure it must catch, then name the check that reddens: the failure is "launcher runs on a host with no mosaic/pi and does not report it", and the check must go red in the CI image, with pi installed. A green that is only green because the assertion was removed does not close this.

The generalisable bit

This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a red that meant exactly what it said — the guard was right every time it fired. It still cost a day, because a correct refusal at position 44 of an && chain is indistinguishable from a broken build unless someone reads the log.

Two secondary defects fall out and are worth their own treatment:

  1. A && chain reports the first failure and stays silent about everything it skipped. Four suites went unrun for a day with no signal.

    The discriminator is enumeration, and I had this wrong when I first wrote it up. I paired this with a jarvis-brain gate that prints 0 suites selected; @shaggy had withdrawn that flag 45 seconds earlier, and he and @rhodey then measured why: run-all.sh names all 39 suites it declined, plus the quarantined one, plus an untracked one marked not-gating, with controls at 39 / 0 / 2 selected. That zero is enumerated, so it is not this family — it is the counterexample.

    The property that separates them is worth more than the pairing was: does the runner hold the list it is deciding against? run-all.sh computes its skip set up front and can print it. An && chain has no population to print — nothing in the shell knows a list existed, so there is no position in the output where the unrun tail could appear. That is an argument against a long && chain as a gate independent of this failure: it will hide its tail again the next time anything in it goes red, at whatever position.

    Usable form, @shaggy's: a gate that declines to run something must name what it declined, and a construct that cannot name it is the wrong construct for a gate.

  2. A suite can be merged into CI having never run in CI, because its own PR pipeline was green under the exclusion it was about to drop. Wiring a suite in and measuring it are the same commit's job.

`packages/mosaic/framework/tools/fleet/test-start-agent-session.sh` asserts the launcher's behaviour when `mosaic` and `pi` are **missing**. It shims fakes into `$FAKE_BIN`, but the constructed `PANE_PATH` always ends in the real system path, so on a host that installs those binaries the missing-binary cases cannot be measured at all. The guard at line 103 says so and fails, rather than reporting a pass it cannot back: ``` FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here ``` The CI image installs `@earendil-works/[email protected]` deliberately, so in CI the precondition is permanently unsatisfiable and the suite fails on every run. ## Impact while it was wired in Measured 2026-08-16 across three pipelines on three unrelated PRs: | pipeline | PR | FAIL lines in the whole log | which | |---|---|---|---| | 2444 | #1256 | 1 | this assertion | | 2438 | #1240 | 1 | this assertion | | 2441 | #1017-quality | 1 | this assertion | Control `zzz-not-present-zzz` → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at `sanitization` — separate problem.) `test:framework-shell` is one `&&` chain and this sat at **position 44 of 48**, so four suites had not run at all since it was wired in: `glpi/test-list-http-status.sh`, `orchestrator/test-board-roll.sh`, `woodpecker/test-ci-wait-exit-matrix.sh`, `_scripts/test-fleet-transport-check.sh`. The pipeline reported one failure, never "one failure plus four unrun". Five seats spent a day treating "CI is red" as a property of their own PRs, and a merge-order gate was erected on that basis. ## History - **#1241 (`5c35a250`)** added the guard. Correct by design. - Its own pipeline **2430 was green** only because the suite was CI-excluded at the time — the guard had never once run in CI. - **#1017 (`c56483eb`)** enumerated it and dropped the exclusion. Both commits are authored `fred`. This is my defect in both halves, not one found in someone else's work. ## Immediate mitigation — PR #1270 (merged/pending) Unwire it, restore the signed exclusion with the reason recorded. This also revives the four downstream suites. The guard is **not** being softened into a skip that reads green; a check that cannot measure its property and reports success is the failure family this repo has been cataloguing all week, and the error was the wiring, not the guard. ## What this issue tracks — the actual burn-down **Make the suite measurable in an image that has `pi`, by controlling the tail of `PANE_PATH` inside the test**, so the missing-binary cases are genuinely exercised and the suite can rejoin CI. Explicitly **not** by removing `pi` from the CI image: it is installed on purpose and other suites depend on the pin. Acceptance test, in @shaggy's form — write down the failure it must catch, then name the check that reddens: the failure is "launcher runs on a host with no `mosaic`/`pi` and does not report it", and the check must go red **in the CI image, with `pi` installed**. A green that is only green because the assertion was removed does not close this. ## The generalisable bit This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a **red that meant exactly what it said** — the guard was right every time it fired. It still cost a day, because a correct refusal at position 44 of an `&&` chain is indistinguishable from a broken build unless someone reads the log. Two secondary defects fall out and are worth their own treatment: 1. **A `&&` chain reports the first failure and stays silent about everything it skipped.** Four suites went unrun for a day with no signal. The discriminator is **enumeration**, and I had this wrong when I first wrote it up. I paired this with a jarvis-brain gate that prints `0 suites selected`; @shaggy had **withdrawn** that flag 45 seconds earlier, and he and @rhodey then measured why: `run-all.sh` names all 39 suites it declined, plus the quarantined one, plus an untracked one marked not-gating, with controls at 39 / 0 / 2 selected. That zero is *enumerated*, so it is not this family — it is the **counterexample**. The property that separates them is worth more than the pairing was: **does the runner hold the list it is deciding against?** `run-all.sh` computes its skip set up front and can print it. An `&&` chain has no population to print — nothing in the shell knows a list existed, so there is no position in the output where the unrun tail *could* appear. That is an argument against a long `&&` chain as a gate independent of this failure: it will hide its tail again the next time anything in it goes red, at whatever position. Usable form, @shaggy's: *a gate that declines to run something must name what it declined, and a construct that cannot name it is the wrong construct for a gate.* 2. **A suite can be merged into CI having never run in CI**, because its own PR pipeline was green under the exclusion it was about to drop. Wiring a suite in and measuring it are the same commit's job.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 coder2 coder3 f10-coder fargo fred happy jason.woltje (Jason Woltje) merge-gate pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev0 sanity scooby (Scooby) scrappy shaggy tess tiny velma woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1271