fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion — unblocks every PR on next #1269

2026-08-16T22:59:37Z

mos-dt-0 commented

2026-08-16 22:59:37 +00:00

What this unblocks

Every failing test step on next dies on one assertion, and it is the same one on unrelated PRs:

FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here
packages/mosaic/framework/tools/fleet/test-start-agent-session.sh:103

pipeline	PR	FAIL lines in the whole log	which line
2444	#1256	1	this one
2438	#1240	1	this one
2441	#1017-quality	1	this one

Control zzz-not-present-zzz → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at sanitization and is a genuinely separate problem.)

Cause, and it is mine

#1241 (5c35a250) added the guard. The suite shims fake mosaic/pi/npm into $FAKE_BIN, but the constructed PANE_PATH always ends in the real system path — so on a host that installs those binaries the missing-binary cases cannot be measured, and a green run would mean nothing. The guard says so instead of passing.
Its own pipeline 2430 was green only because the suite was CI-excluded then. The guard had never run in CI.
#1017 (c56483eb) enumerated it and dropped the exclusion.
The CI image installs @earendil-works/[email protected] on purpose — measured in 2444's test-step log. The precondition is unsatisfiable there.

Both commits are authored fred. I wrote the guard and I wired it into the environment that violates its precondition.

What I did not do

I did not soften the guard into a skip that reads green. A check that cannot measure its property and reports success is the failure family this repo has spent the week cataloguing, and I am not adding another instance to fix my own. The guard is correct; the wiring was the error, so the wiring is what reverts.

The second half nobody was looking at

test:framework-shell is one && chain and this sat at position 44 of 48. Four suites after it have not run at all since the merge:

glpi/test-list-http-status.sh
orchestrator/test-board-roll.sh
woodpecker/test-ci-wait-exit-matrix.sh
_scripts/test-fleet-transport-check.sh

The pipeline reported one failure, never "one failure plus four unrun". That is the same rendering defect as a gate printing 0 suites selected and reading green.

Verification, with controls

Enumeration guard OK — population 52, enumerated 36, signed-excluded 16.
Control A — exclusion removed while unwired → FAIL UNENUMERATED.
Control B — exclusion kept while rewired → FAIL CONTRADICTORY EXCLUSION.
The gate reddens in both directions, so its OK is load-bearing rather than decorative.
The four formerly-masked suites: rc=0 each, run directly.
Pushed object re-read from the remote: chain 48 → 47, suite absent, exclusion present; same grep at origin/next → 0.

Stated gap

The full chain cannot be run to completion on sb-it-1-dt. It stops earlier, at the lease-broker Invariant R test, because this host carries the quarantined operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and out of scope here. CI pins 0.84.1, and the single FAIL line in those three pipelines is itself the proof that positions 1–43 passed there. The four suites being green is a one-host result, not a CI-image result — this PR's own pipeline is what measures that.

Burn-down

Control the tail of PANE_PATH inside the test. Not by removing pi from the image — the image installs it deliberately and other suites depend on the pin. Recorded in the exclusion reason.

The bit worth keeping

This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a red that meant exactly what it said — the guard refused to report a pass it could not measure and was right every time it fired. It cost a day anyway, because a correct refusal at position 44 of an && chain is indistinguishable from a broken build unless someone reads the log. Five seats treated "CI is red" as a property of their own PRs.

## What this unblocks Every failing `test` step on `next` dies on **one assertion**, and it is the same one on unrelated PRs: ``` FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here packages/mosaic/framework/tools/fleet/test-start-agent-session.sh:103 ``` | pipeline | PR | FAIL lines in the whole log | which line | |---|---|---|---| | 2444 | #1256 | 1 | this one | | 2438 | #1240 | 1 | this one | | 2441 | #1017-quality | 1 | this one | Control `zzz-not-present-zzz` → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at `sanitization` and is a genuinely separate problem.) ## Cause, and it is mine 1. **#1241 (`5c35a250`)** added the guard. The suite shims fake `mosaic`/`pi`/`npm` into `$FAKE_BIN`, but the constructed `PANE_PATH` always ends in the real system path — so on a host that installs those binaries the missing-binary cases cannot be measured, and a green run would mean nothing. The guard says so instead of passing. 2. Its own pipeline **2430 was green** only because the suite was CI-excluded then. **The guard had never run in CI.** 3. **#1017 (`c56483eb`)** enumerated it and dropped the exclusion. 4. The CI image installs `@earendil-works/[email protected]` **on purpose** — measured in 2444's test-step log. The precondition is unsatisfiable there. Both commits are authored `fred`. I wrote the guard and I wired it into the environment that violates its precondition. ## What I did not do I did not soften the guard into a skip that reads green. A check that cannot measure its property and reports success is the failure family this repo has spent the week cataloguing, and I am not adding another instance to fix my own. The guard is correct; the **wiring** was the error, so the wiring is what reverts. ## The second half nobody was looking at `test:framework-shell` is one `&&` chain and this sat at **position 44 of 48**. Four suites after it have not run at all since the merge: ``` glpi/test-list-http-status.sh orchestrator/test-board-roll.sh woodpecker/test-ci-wait-exit-matrix.sh _scripts/test-fleet-transport-check.sh ``` The pipeline reported *one failure*, never "one failure plus four unrun". That is the same rendering defect as a gate printing `0 suites selected` and reading green. ## Verification, with controls - Enumeration guard **OK** — population 52, enumerated 36, signed-excluded 16. - **Control A** — exclusion removed while unwired → `FAIL UNENUMERATED`. - **Control B** — exclusion kept while rewired → `FAIL CONTRADICTORY EXCLUSION`. The gate reddens in both directions, so its OK is load-bearing rather than decorative. - The four formerly-masked suites: **rc=0 each**, run directly. - Pushed object re-read from the remote: chain 48 → 47, suite absent, exclusion present; same grep at `origin/next` → 0. ### Stated gap The full chain **cannot** be run to completion on `sb-it-1-dt`. It stops earlier, at the lease-broker Invariant R test, because this host carries the quarantined operator-global **pi 0.84.2** against a measured 0.84.1. That is host-specific and out of scope here. CI pins 0.84.1, and the single FAIL line in those three pipelines is itself the proof that positions 1–43 passed there. The four suites being green is a **one-host** result, not a CI-image result — this PR's own pipeline is what measures that. ## Burn-down Control the tail of `PANE_PATH` inside the test. **Not** by removing `pi` from the image — the image installs it deliberately and other suites depend on the pin. Recorded in the exclusion reason. ## The bit worth keeping This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a **red that meant exactly what it said** — the guard refused to report a pass it could not measure and was right every time it fired. It cost a day anyway, because a correct refusal at position 44 of an `&&` chain is indistinguishable from a broken build unless someone reads the log. Five seats treated "CI is red" as a property of their own PRs.

mos-dt-0 added 1 commit 2026-08-16 22:59:38 +00:00

fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269 )

ci/woodpecker/pr/ci Pipeline was canceled

Details

93c1de51e1

The `test` step has failed on every `next` pipeline since #1017 on exactly one
assertion, and it is the same one on unrelated PRs:

    FAIL: host provides 'pi' in the system path; missing-binary cases are not
    measurable here            (framework/tools/fleet/test-start-agent-session.sh:103)

Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441
(#1017-quality): exactly one FAIL line in each full log, identical, this line.
Control `zzz-not-present-zzz` -> 0 on all three.

Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into
$FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so
on a host that installs those binaries the missing-binary cases cannot be
measured and a green run would mean nothing. The guard says so instead of
passing. Its own pipeline 2430 was green only because the suite was CI-excluded
at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated
it and dropped the exclusion. The CI image installs
@earendil-works/[email protected].1 on purpose, so the precondition is
unsatisfiable there. Both commits are mine.

The guard is correct and is not being softened. A check that cannot measure its
property and reports success is the failure mode this repo has been cataloguing
all week; the error was wiring the suite into an image that violates its
precondition, so the wiring is what gets reverted.

Second effect, which is the reason this cost a day rather than an hour:
test:framework-shell is one && chain and this sat at position 44 of 48, so
glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh,
woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh
have not run at all since the merge. The pipeline reported one failure, never
"one failure plus four unrun". All four are green when run directly on
sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one
host, not a CI-image result.

Verification, with controls:
- enumeration guard OK (population 52, enumerated 36, signed-excluded 16).
- control A, exclusion line removed while unwired -> FAIL UNENUMERATED.
- control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION.
  The gate discriminates in both directions, so its OK is load-bearing.
- the four formerly-masked suites: rc=0 each, run directly.
- the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at
  the lease-broker Invariant R test, because this host carries the quarantined
  operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and
  out of scope here -- CI pins 0.84.1, and the single FAIL line in those three
  pipelines proves positions 1-43 passed there.

Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi
from the image. Recorded in the exclusion reason and in #1269.

mos-dt-0 commented

2026-08-16 23:01:42 +00:00

Closing and refiling under @fred. The content and the branch are unchanged — fix/1269-ci-chain-unblock at 93c1de51 stays exactly as pushed, and its commit is authored fred <[email protected]>.

The reason: the PR author field on this one reads mos-dt-0, a retired seat. I did not choose it and did not notice until I read the author back. Two seats have refused that principal tonight on principle; asking either of them to review a PR filed under it is not reasonable.

The mechanism, measured just now — a real defect, not my typo:

get_gitea_login()                           -> mosaicstack-mos-dt-0
GITEA_LOGIN=fred-ms get_gitea_login()       -> fred-ms
GITEA_LOGIN=nope-not-real get_gitea_login() -> mosaicstack-mos-dt-0    <- silent fallback

find_tea_login_for_host returns the first tea login matching the remote host. Three match git.mosaicstack.dev on this box (mosaicstack-mos-dt-0, daphne-ms, fred-ms), none is marked DEFAULT, and the retired one sorts first. pr-create.sh has no --login flag, so there is no in-band way to choose one. The only override is the undocumented GITEA_LOGIN env var — and per the third line above, a wrong value for it does not fail either; it falls back to the same retired principal without a word.

So every seat on this host writes as mos-dt-0 unless it happens to know an undocumented variable. That is #1266's family — a write attributed to a principal nobody chose — and it has now caught me twice (#1264, #1265).

Filing that separately. The author field is immutable, which is why this is a refile and not an edit.

Closing and refiling under `@fred`. **The content and the branch are unchanged** — `fix/1269-ci-chain-unblock` at `93c1de51` stays exactly as pushed, and its commit is authored `fred <[email protected]>`. The reason: the PR author field on this one reads **`mos-dt-0`**, a retired seat. I did not choose it and did not notice until I read the author back. Two seats have refused that principal tonight on principle; asking either of them to review a PR filed under it is not reasonable. **The mechanism, measured just now** — a real defect, not my typo: ``` get_gitea_login() -> mosaicstack-mos-dt-0 GITEA_LOGIN=fred-ms get_gitea_login() -> fred-ms GITEA_LOGIN=nope-not-real get_gitea_login() -> mosaicstack-mos-dt-0 <- silent fallback ``` `find_tea_login_for_host` returns the **first** tea login matching the remote host. Three match `git.mosaicstack.dev` on this box (`mosaicstack-mos-dt-0`, `daphne-ms`, `fred-ms`), **none is marked DEFAULT**, and the retired one sorts first. `pr-create.sh` has no `--login` flag, so there is no in-band way to choose one. The only override is the undocumented `GITEA_LOGIN` env var — and per the third line above, a *wrong* value for it does not fail either; it falls back to the same retired principal without a word. So every seat on this host writes as `mos-dt-0` unless it happens to know an undocumented variable. That is #1266's family — a write attributed to a principal nobody chose — and it has now caught me twice (#1264, #1265). Filing that separately. The author field is immutable, which is why this is a refile and not an edit.

mos-dt-0 closed this pull request

2026-08-16 23:01:49 +00:00

fred referenced this pull request

2026-08-16 23:13:20 +00:00

gitea wrappers resolve the write principal by first-host-match; a non-matching GITEA_LOGIN is discarded silently — two PRs on a production path are authored as Jason #1272

jason.woltje referenced this issue from a commit

2026-08-17 20:45:03 +00:00

fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269)