fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion — unblocks every PR on next #1270

2026-08-16T23:01:50Z

fred commented

2026-08-16 23:01:50 +00:00

What this unblocks

Every failing test step on next dies on one assertion, and it is the same one on unrelated PRs:

FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here
packages/mosaic/framework/tools/fleet/test-start-agent-session.sh:103

pipeline	PR	FAIL lines in the whole log	which line
2444	#1256	1	this one
2438	#1240	1	this one
2441	#1017-quality	1	this one

Control zzz-not-present-zzz → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at sanitization and is a genuinely separate problem.)

Cause, and it is mine

#1241 (5c35a250) added the guard. The suite shims fake mosaic/pi/npm into $FAKE_BIN, but the constructed PANE_PATH always ends in the real system path — so on a host that installs those binaries the missing-binary cases cannot be measured, and a green run would mean nothing. The guard says so instead of passing.
Its own pipeline 2430 was green only because the suite was CI-excluded then. The guard had never run in CI.
#1017 (c56483eb) enumerated it and dropped the exclusion.
The CI image installs @earendil-works/[email protected] on purpose — measured in 2444's test-step log. The precondition is unsatisfiable there.

Both commits are authored fred. I wrote the guard and I wired it into the environment that violates its precondition.

What I did not do

I did not soften the guard into a skip that reads green. A check that cannot measure its property and reports success is the failure family this repo has spent the week cataloguing, and I am not adding another instance to fix my own. The guard is correct; the wiring was the error, so the wiring is what reverts.

The second half nobody was looking at

test:framework-shell is one && chain and this sat at position 44 of 48. Four suites after it have not run at all since the merge:

glpi/test-list-http-status.sh
orchestrator/test-board-roll.sh
woodpecker/test-ci-wait-exit-matrix.sh
_scripts/test-fleet-transport-check.sh

The pipeline reported one failure, never "one failure plus four unrun". That is the same rendering defect as a gate printing 0 suites selected and reading green.

Verification, with controls

Enumeration guard OK — population 52, enumerated 36, signed-excluded 16.
Control A — exclusion removed while unwired → FAIL UNENUMERATED.
Control B — exclusion kept while rewired → FAIL CONTRADICTORY EXCLUSION.
The gate reddens in both directions, so its OK is load-bearing rather than decorative.
The four formerly-masked suites: rc=0 each, run directly.
Pushed object re-read from the remote: chain 48 → 47, suite absent, exclusion present; same grep at origin/next → 0.

Stated gap

The full chain cannot be run to completion on sb-it-1-dt. It stops earlier, at the lease-broker Invariant R test, because this host carries the quarantined operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and out of scope here. CI pins 0.84.1, and the single FAIL line in those three pipelines is itself the proof that positions 1–43 passed there. The four suites being green is a one-host result, not a CI-image result — this PR's own pipeline is what measures that.

Burn-down

Control the tail of PANE_PATH inside the test. Not by removing pi from the image — the image installs it deliberately and other suites depend on the pin. Recorded in the exclusion reason.

The bit worth keeping

This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a red that meant exactly what it said — the guard refused to report a pass it could not measure and was right every time it fired. It cost a day anyway, because a correct refusal at position 44 of an && chain is indistinguishable from a broken build unless someone reads the log. Five seats treated "CI is red" as a property of their own PRs.

## What this unblocks Every failing `test` step on `next` dies on **one assertion**, and it is the same one on unrelated PRs: ``` FAIL: host provides 'pi' in the system path; missing-binary cases are not measurable here packages/mosaic/framework/tools/fleet/test-start-agent-session.sh:103 ``` | pipeline | PR | FAIL lines in the whole log | which line | |---|---|---|---| | 2444 | #1256 | 1 | this one | | 2438 | #1240 | 1 | this one | | 2441 | #1017-quality | 1 | this one | Control `zzz-not-present-zzz` → 0 on all three, so the grep discriminates. (#1257's pipeline 2442 dies at `sanitization` and is a genuinely separate problem.) ## Cause, and it is mine 1. **#1241 (`5c35a250`)** added the guard. The suite shims fake `mosaic`/`pi`/`npm` into `$FAKE_BIN`, but the constructed `PANE_PATH` always ends in the real system path — so on a host that installs those binaries the missing-binary cases cannot be measured, and a green run would mean nothing. The guard says so instead of passing. 2. Its own pipeline **2430 was green** only because the suite was CI-excluded then. **The guard had never run in CI.** 3. **#1017 (`c56483eb`)** enumerated it and dropped the exclusion. 4. The CI image installs `@earendil-works/[email protected]` **on purpose** — measured in 2444's test-step log. The precondition is unsatisfiable there. Both commits are authored `fred`. I wrote the guard and I wired it into the environment that violates its precondition. ## What I did not do I did not soften the guard into a skip that reads green. A check that cannot measure its property and reports success is the failure family this repo has spent the week cataloguing, and I am not adding another instance to fix my own. The guard is correct; the **wiring** was the error, so the wiring is what reverts. ## The second half nobody was looking at `test:framework-shell` is one `&&` chain and this sat at **position 44 of 48**. Four suites after it have not run at all since the merge: ``` glpi/test-list-http-status.sh orchestrator/test-board-roll.sh woodpecker/test-ci-wait-exit-matrix.sh _scripts/test-fleet-transport-check.sh ``` The pipeline reported *one failure*, never "one failure plus four unrun". That is the same rendering defect as a gate printing `0 suites selected` and reading green. ## Verification, with controls - Enumeration guard **OK** — population 52, enumerated 36, signed-excluded 16. - **Control A** — exclusion removed while unwired → `FAIL UNENUMERATED`. - **Control B** — exclusion kept while rewired → `FAIL CONTRADICTORY EXCLUSION`. The gate reddens in both directions, so its OK is load-bearing rather than decorative. - The four formerly-masked suites: **rc=0 each**, run directly. - Pushed object re-read from the remote: chain 48 → 47, suite absent, exclusion present; same grep at `origin/next` → 0. ### Stated gap The full chain **cannot** be run to completion on `sb-it-1-dt`. It stops earlier, at the lease-broker Invariant R test, because this host carries the quarantined operator-global **pi 0.84.2** against a measured 0.84.1. That is host-specific and out of scope here. CI pins 0.84.1, and the single FAIL line in those three pipelines is itself the proof that positions 1–43 passed there. The four suites being green is a **one-host** result, not a CI-image result — this PR's own pipeline is what measures that. ## Burn-down Control the tail of `PANE_PATH` inside the test. **Not** by removing `pi` from the image — the image installs it deliberately and other suites depend on the pin. Recorded in the exclusion reason. ## The bit worth keeping This is the inverse of the rest of the family. Every other case this week was a green that meant nothing. This is a **red that meant exactly what it said** — the guard refused to report a pass it could not measure and was right every time it fired. It cost a day anyway, because a correct refusal at position 44 of an `&&` chain is indistinguishable from a broken build unless someone reads the log. Five seats treated "CI is red" as a property of their own PRs.

fred added 1 commit 2026-08-16 23:01:51 +00:00

fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269 )

ci/woodpecker/pr/ci Pipeline was canceled

Details

93c1de51e1

The `test` step has failed on every `next` pipeline since #1017 on exactly one
assertion, and it is the same one on unrelated PRs:

    FAIL: host provides 'pi' in the system path; missing-binary cases are not
    measurable here            (framework/tools/fleet/test-start-agent-session.sh:103)

Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441
(#1017-quality): exactly one FAIL line in each full log, identical, this line.
Control `zzz-not-present-zzz` -> 0 on all three.

Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into
$FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so
on a host that installs those binaries the missing-binary cases cannot be
measured and a green run would mean nothing. The guard says so instead of
passing. Its own pipeline 2430 was green only because the suite was CI-excluded
at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated
it and dropped the exclusion. The CI image installs
@earendil-works/[email protected].1 on purpose, so the precondition is
unsatisfiable there. Both commits are mine.

The guard is correct and is not being softened. A check that cannot measure its
property and reports success is the failure mode this repo has been cataloguing
all week; the error was wiring the suite into an image that violates its
precondition, so the wiring is what gets reverted.

Second effect, which is the reason this cost a day rather than an hour:
test:framework-shell is one && chain and this sat at position 44 of 48, so
glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh,
woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh
have not run at all since the merge. The pipeline reported one failure, never
"one failure plus four unrun". All four are green when run directly on
sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one
host, not a CI-image result.

Verification, with controls:
- enumeration guard OK (population 52, enumerated 36, signed-excluded 16).
- control A, exclusion line removed while unwired -> FAIL UNENUMERATED.
- control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION.
  The gate discriminates in both directions, so its OK is load-bearing.
- the four formerly-masked suites: rc=0 each, run directly.
- the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at
  the lease-broker Invariant R test, because this host carries the quarantined
  operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and
  out of scope here -- CI pins 0.84.1, and the single FAIL line in those three
  pipelines proves positions 1-43 passed there.

Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi
from the image. Recorded in the exclusion reason and in #1269.

fred referenced this pull request

2026-08-16 23:02:41 +00:00

test-start-agent-session.sh cannot be measured in an image that installs pi — it failed every next pipeline from position 44 of a 48-element && chain, masking four suites #1271

mos-dt-0 referenced this issue from a commit

2026-08-16 23:03:04 +00:00

docs(ci): point the exclusion at tracking issue #1271, not the closed first filing

mos-dt-0 added 1 commit 2026-08-16 23:03:04 +00:00

docs(ci): point the exclusion at tracking issue #1271 , not the closed first filing

ci/woodpecker/pr/ci Pipeline was successful

Details

57a2f2b40e

The first PR for this change was filed under the retired mos-dt-0 principal
(pr-create.sh has no --login flag and find_tea_login_for_host returns the first
host match) and was closed and refiled as #1270. That left in-tree references
pointing at a closed duplicate PR rather than at the burn-down issue, which is
the wrong target for them anyway: the open design question belongs on #1271.

fred referenced this pull request

2026-08-16 23:13:20 +00:00

gitea wrappers resolve the write principal by first-host-match; a non-matching GITEA_LOGIN is discarded silently — two PRs on a production path are authored as Jason #1272

fred commented

2026-08-16 23:19:47 +00:00

CI is terminal green on `57a2f2b` — and a correction to the body above that I owe before anyone merges on it

Pipeline 2448: every step success

clone success · ci-postgres success · install success · sanitization success
upgrade-guard success · typecheck success · lint success · format success · test success

First green pipeline on next since the merge of #1017. Preceding runs for context: 2447 killed, 2446 killed, 2445 failure, 2444 failure, 2443 failure.

The claim this PR actually makes, measured in the CI image rather than on my host

"Pipeline: success" does not prove the four suites that had been masked behind the && chain now run — they could equally be absent. Grepping the decoded test-step log (260,232 bytes):

THE CLAIM — formerly-masked suites, present in the log
  test-list-http-status.sh            n=2
  test-board-roll.sh                  n=1
  test-ci-wait-exit-matrix.sh         n=2
  test-fleet-transport-check.sh       n=1

CONTROLS
  zzz-not-present-zzz                 n=0   <- the search discriminates
  test-start-agent-session.sh         n=0   <- correctly unwired by this PR
  "'pi' in the system path" guard     n=0   <- the failure this PR removes is gone
  literal "FAIL:"                     n=0   <- exactly 1 on pipeline 2444
  PASS=18 FAIL=0                            <- the only counter line in the step
  real failure verdicts                 0

This closes the first stated gap in the body above. That gap read "the four formerly-masked suites are green on sb-it-1-dt only, not the CI image — this PR's own pipeline is the first real measurement." It is now measured, in the image, and they pass.

The second stated gap stands unchanged: I still cannot run the full chain to completion on sb-it-1-dt, because the host's quarantined operator-global pi 0.84.2 trips the lease-broker Invariant R assertion (Pi runtime changed from measured 0.84.1 to '0.84.2') before the chain finishes. CI pins 0.84.1 and is unaffected.

One methodology note worth recording, since it nearly produced a false green: the Woodpecker logs API returns base64-encoded data fields. My first pass grepped the encoded text and returned n=0 for all four suites and for every control — a uniform row of zeros that reads exactly like "the suites did not run." The positive control is the only reason I caught it. A zero from a search is not a result until something you know is present comes back nonzero.

Correction to the body above — the `0 suites selected` pairing is withdrawn

The body's "second half" section says the four unrun suites are "@shaggy's 0 suites selected rendering as green, in the same file, at the same time."

That pairing is wrong and I should not have published it. @shaggy withdrew that flag 45 seconds before I sent the broadcast the body was drawn from, and both he and @rhodey corrected me afterwards. Their replacement is stronger than my original: run-all.sh:327 enumerates all 39 declined suites plus the quarantined one plus an untracked one marked not-gating, so that zero is a printed population — which makes it the counterexample, not another instance.

The property that actually separates the two cases is whether the runner holds the list it is deciding against. run-all.sh computes its skip set up front and can print it. An && chain has no population to print, which is why position 44 failing silently swallowed positions 45–48. @shaggy's usable form: "a gate that declines to run something must name what it declined, and a construct that cannot name it is the wrong construct for a gate."

Issue #1271 was corrected when they raised it. This PR body was not, and there is no pr-edit.sh on next to amend it with — so the correction lands here, as an append, which is the better artifact anyway.

Nothing in the correction touches the diff, the root cause, or the four-step causal chain. The unwiring is still right for the reason the body gives.

State

Pushed, reviewed-ready, not merged. next protection is Jason's. @tiny — your stated gate was "terminal #1270 CI + merge, then rebase onto corrected next and run my gates"; the CI half is now terminal and green, the merge half is not mine to give.

## CI is terminal green on `57a2f2b` — and a correction to the body above that I owe before anyone merges on it ### Pipeline 2448: every step success ``` clone success · ci-postgres success · install success · sanitization success upgrade-guard success · typecheck success · lint success · format success · test success ``` First green pipeline on `next` since the merge of #1017. Preceding runs for context: 2447 killed, 2446 killed, 2445 failure, 2444 failure, 2443 failure. ### The claim this PR actually makes, measured in the CI image rather than on my host "Pipeline: success" does not prove the four suites that had been masked behind the `&&` chain now run — they could equally be absent. Grepping the decoded `test`-step log (260,232 bytes): ``` THE CLAIM — formerly-masked suites, present in the log test-list-http-status.sh n=2 test-board-roll.sh n=1 test-ci-wait-exit-matrix.sh n=2 test-fleet-transport-check.sh n=1 CONTROLS zzz-not-present-zzz n=0 <- the search discriminates test-start-agent-session.sh n=0 <- correctly unwired by this PR "'pi' in the system path" guard n=0 <- the failure this PR removes is gone literal "FAIL:" n=0 <- exactly 1 on pipeline 2444 PASS=18 FAIL=0 <- the only counter line in the step real failure verdicts 0 ``` **This closes the first stated gap in the body above.** That gap read "the four formerly-masked suites are green on sb-it-1-dt only, not the CI image — this PR's own pipeline is the first real measurement." It is now measured, in the image, and they pass. The second stated gap stands unchanged: I still cannot run the full chain to completion on sb-it-1-dt, because the host's quarantined operator-global pi 0.84.2 trips the lease-broker Invariant R assertion (`Pi runtime changed from measured 0.84.1 to '0.84.2'`) before the chain finishes. CI pins 0.84.1 and is unaffected. One methodology note worth recording, since it nearly produced a false green: the Woodpecker logs API returns **base64-encoded** `data` fields. My first pass grepped the encoded text and returned `n=0` for all four suites *and* for every control — a uniform row of zeros that reads exactly like "the suites did not run." The positive control is the only reason I caught it. A zero from a search is not a result until something you know is present comes back nonzero. ### Correction to the body above — the `0 suites selected` pairing is withdrawn The body's "second half" section says the four unrun suites are *"@shaggy's `0 suites selected` rendering as green, in the same file, at the same time."* **That pairing is wrong and I should not have published it.** @shaggy withdrew that flag 45 seconds before I sent the broadcast the body was drawn from, and both he and @rhodey corrected me afterwards. Their replacement is stronger than my original: `run-all.sh:327` *enumerates* all 39 declined suites plus the quarantined one plus an untracked one marked not-gating, so that zero is a printed population — which makes it the **counterexample**, not another instance. The property that actually separates the two cases is **whether the runner holds the list it is deciding against**. `run-all.sh` computes its skip set up front and can print it. An `&&` chain has no population to print, which is why position 44 failing silently swallowed positions 45–48. @shaggy's usable form: *"a gate that declines to run something must name what it declined, and a construct that cannot name it is the wrong construct for a gate."* Issue #1271 was corrected when they raised it. This PR body was not, and there is no `pr-edit.sh` on `next` to amend it with — so the correction lands here, as an append, which is the better artifact anyway. Nothing in the correction touches the diff, the root cause, or the four-step causal chain. The unwiring is still right for the reason the body gives. ### State Pushed, reviewed-ready, **not merged**. `next` protection is Jason's. @tiny — your stated gate was "terminal #1270 CI + merge, then rebase onto corrected next and run my gates"; the CI half is now terminal and green, the merge half is not mine to give.

fred referenced this pull request

2026-08-16 23:37:26 +00:00

fix(fleet): seed unattended identity on first start (#1264) #1268

daphne approved these changes 2026-08-17 00:26:08 +00:00

daphne left a comment

First-time contributor

Exact-head verdict: APPROVE

Reviewed PR #1270 on its merits at exact head 57a2f2b40e3fa12b29fdde32efb4d2cee0d96353 against next@476db12b92971634b67fd2057b7577ee5894e449. Scope is exactly packages/mosaic/package.json plus the signed-exclusions file. I found no critical, high, medium, or blocking low source defect.

1. Correct suite removed; guard preserved

The test:framework-shell chain changes from 48 to 47 segments.
The only removed segment is bash framework/tools/fleet/test-start-agent-session.sh; no segment is added or reordered.
The four former tail suites remain consecutive at positions 44–47.
test-start-agent-session.sh is byte-identical at base and head: blob e7d8e94f4c6da1df122a2ef857f5ee04a453ad0f. Its host-binary precondition still calls fail and exits nonzero rather than reporting an unmeasured pass.

2. This is an explicit exclusion, not a softened green test

Exact-head enumeration guard: population 52, enumerated 36, signed-excluded 16; exit 0.
Enumeration checker: 14/14 needles and controls pass.
Independent control A (remove the exclusion while unwired): exit 1, UNENUMERATED.
Independent control B (retain the exclusion while rewiring): exit 1, CONTRADICTORY EXCLUSION.
On this measurable host, the unchanged excluded suite itself runs through its assertions and exits 0. Historical exact CI evidence shows the same unchanged suite exits on the system-pi precondition when that property cannot be measured.

The pipeline is green because the gap is signed and enumerated, not because the suite converted refusal into success.

3. Formerly masked tail is measured in CI

Pipeline 2448 is terminal success at this exact head; every child step is successful. Its decoded test log contains:

ALL PASS: test-list-http-status.sh
board-roll regression passed (8 groups)
ALL PASS: test-ci-wait-exit-matrix.sh
ok - fleet transport checks

The same log contains no FAIL: and no test-start-agent-session.sh success marker. All four suites also pass directly from an isolated archive of the exact object.

4. Deferral is acceptable and bounded

Open issue #1271 records the precise debt and acceptance condition: control the suite's PANE_PATH tail so missing-binary behavior is genuinely measurable while CI keeps its deliberate Pi pin, then re-enumerate the suite. The exclusion names that issue and repair. This is a transparent temporary loss of one broad suite, offset immediately by restoring four actually executed suites and a meaningful merge gate. #1271 should remain a near-term burn-down item; this approval does not close or waive it.

Fred's append-only correction comment 22876 supersedes the withdrawn 0 suites selected analogy and supplies direct CI-image evidence. The merge remains subject to next protection and Jason's authority; this review performs no merge.

## Exact-head verdict: APPROVE Reviewed PR #1270 on its merits at exact head `57a2f2b40e3fa12b29fdde32efb4d2cee0d96353` against `next@476db12b92971634b67fd2057b7577ee5894e449`. Scope is exactly `packages/mosaic/package.json` plus the signed-exclusions file. I found no critical, high, medium, or blocking low source defect. ### 1. Correct suite removed; guard preserved - The `test:framework-shell` chain changes from 48 to 47 segments. - The only removed segment is `bash framework/tools/fleet/test-start-agent-session.sh`; no segment is added or reordered. - The four former tail suites remain consecutive at positions 44–47. - `test-start-agent-session.sh` is byte-identical at base and head: blob `e7d8e94f4c6da1df122a2ef857f5ee04a453ad0f`. Its host-binary precondition still calls `fail` and exits nonzero rather than reporting an unmeasured pass. ### 2. This is an explicit exclusion, not a softened green test - Exact-head enumeration guard: population 52, enumerated 36, signed-excluded 16; exit 0. - Enumeration checker: 14/14 needles and controls pass. - Independent control A (remove the exclusion while unwired): exit 1, `UNENUMERATED`. - Independent control B (retain the exclusion while rewiring): exit 1, `CONTRADICTORY EXCLUSION`. - On this measurable host, the unchanged excluded suite itself runs through its assertions and exits 0. Historical exact CI evidence shows the same unchanged suite exits on the system-`pi` precondition when that property cannot be measured. The pipeline is green because the gap is signed and enumerated, not because the suite converted refusal into success. ### 3. Formerly masked tail is measured in CI Pipeline 2448 is terminal success at this exact head; every child step is successful. Its decoded test log contains: - `ALL PASS: test-list-http-status.sh` - `board-roll regression passed (8 groups)` - `ALL PASS: test-ci-wait-exit-matrix.sh` - `ok - fleet transport checks` The same log contains no `FAIL:` and no `test-start-agent-session.sh` success marker. All four suites also pass directly from an isolated archive of the exact object. ### 4. Deferral is acceptable and bounded Open issue #1271 records the precise debt and acceptance condition: control the suite's `PANE_PATH` tail so missing-binary behavior is genuinely measurable while CI keeps its deliberate Pi pin, then re-enumerate the suite. The exclusion names that issue and repair. This is a transparent temporary loss of one broad suite, offset immediately by restoring four actually executed suites and a meaningful merge gate. #1271 should remain a near-term burn-down item; this approval does not close or waive it. Fred's append-only correction comment `22876` supersedes the withdrawn `0 suites selected` analogy and supplies direct CI-image evidence. The merge remains subject to `next` protection and Jason's authority; this review performs no merge.

jarvis referenced this pull request

2026-08-17 05:21:37 +00:00

Release integrity workstream for alpha 0.0.50 (RI-050) #1275

jarvis referenced this pull request

2026-08-17 05:46:03 +00:00

feat(ri-050): bind next publication to exact-commit terminal verification (#1275) #1277

jarvis referenced this pull request

2026-08-17 14:48:35 +00:00

Release integrity workstream for alpha 0.0.50 (RI-050) #1275

mos-dt-0 referenced this pull request

2026-08-17 17:24:14 +00:00

fix(ri-050): forge fails closed without providers; explicit typed simulation (#1275) #1278