fix(git): ci-queue-wait.sh — parse the payload it was handed, not stdin (#1019) #1023

Closed
Ghost wants to merge 2 commits from fix/1019-queue-guard-stdin into main

Fixes #1019. ci-queue-wait.sh:37,:86 use python3 - <<'PY', which binds stdin to the program text, so json.load(sys.stdin) EOFs, the state falls through to unknown, and the guard exits 0 on every invocation, every branch, every repo, both platforms.

Passes the payload as an argument instead. pr-ci-wait.sh:38 documents this exact bug with the remedy; it was never backported to the mandatory sibling.

Built and tested previously; opened per board instruction without re-verification. Commit 505b6f7.

Fixes #1019. ci-queue-wait.sh:37,:86 use `python3 - <<'PY'`, which binds stdin to the program text, so `json.load(sys.stdin)` EOFs, the state falls through to `unknown`, and the guard exits 0 on every invocation, every branch, every repo, both platforms. Passes the payload as an argument instead. pr-ci-wait.sh:38 documents this exact bug with the remedy; it was never backported to the mandatory sibling. Built and tested previously; opened per board instruction without re-verification. Commit 505b6f7.
Ghost added 1 commit 2026-07-31 16:41:55 +00:00
Both python sites piped a status payload into `python3 - <<'PY'`. The heredoc binds
stdin to the program text, so `json.load(sys.stdin)` saw EOF, the bare `except` fired,
and the parser returned "unknown" for every input — success, pending and failure alike.
"unknown" then reaches a silent `exit 0` arm.

Consequence: the gate-6 queue guard has never made a determination. It exits 0 on every
invocation, on both the gitea and github paths (one shared parser). The pending wait
loop, --require-status and the 124 timeout were all unreachable code. What it still
validated was connectivity — an unresolved token, head sha, or platform could exit 1.

Fix is the one already shipped in the sibling: capture the payload with `payload=$(cat)`
before invoking python, pass it by environment. pr-ci-wait.sh:38 has carried a comment
describing this exact bug — "yielding EOF and returning unknown every time" — along with
the remedy. It was never backported to the sibling the constitution makes mandatory
before every push and merge.

Adds test-ci-queue-wait-parse.sh, enumerated in test:framework-shell. Every assertion is
on the RETURNED STATE STRING; a suite asserting only rc=0 passes against the broken build,
which is how this survived. Verified by mutation: against the pre-fix wrapper the suite
fails 10 of 14, and the four that pass are the four for which passing is correct (the
undecodable-input control, the unknown-vocabulary case, and both needle halves).

The needle scans with heredoc semantics rather than matching text. `json.load(sys.stdin)`
is correct under `python3 -c`, where the program comes from argv and stdin really is the
payload — ci-queue-wait.sh uses that form legitimately in gitea_get_branch_head_sha. A
flat grep flags that innocent site; the scanner names only the two defective ones.

Out of scope, deliberately, and recorded on #1019: token-in-argv (ps-visible) and the
hardcoded BRANCH="main". Bundling them would make a safety-critical parse fix harder to
review.

Refs #1019
Ghost added 1 commit 2026-07-31 16:43:37 +00:00
Both sides edited the single `test:framework-shell` line additively and did not
overlap: main prepended the test-enumeration guard pair
(check-test-enumeration.sh, test-check-test-enumeration.sh); this branch inserted
test-ci-queue-wait-parse.sh after test-ci-queue-wait-branch-absent.sh.

Resolved as the union — main's line with this branch's one insertion restored.
Main's new guard makes that insertion mandatory rather than cosmetic: the test
file ships in this branch, so leaving it unenumerated would fail the guard.

Co-authored-by: mos-dt-0 <[email protected]>
Ghost requested changes 2026-07-31 20:43:21 +00:00
Ghost left a comment

CHANGES-REQUESTED at exact head f6334080 (full f63340802c). Independent diff-blind review; checks were pre-registered before inspecting the diff.

Blocking findings:

  1. Check 3 FAIL — unknown still exits 0. Malformed JSON maps to unknown, but ci-queue-wait.sh:291-293 groups unknown with terminal states and exits 0; lines 295-297 also fail open. The core defect #1019 claims to fix is therefore NOT fixed: parser/provider ambiguity can still pass the mandatory gate-6 guard.
  2. Check 4 FAIL — the payload-via-environment approach remains subject to exec argument/environment limits. An exact-head probe with a valid 150 KiB JSON payload failed to exec Python with rc=126 and Argument list too long (ARG_MAX/MAX_ARG_STRLEN). Use transport not bounded as one exec string, such as a temporary file or separate file descriptor.
  3. Check 6 FAIL — tests do not assert guard exit outcomes. The suite checks parser-returned strings and expects malformed input to return unknown, but never executes/asserts the production guard control flow, so all 14 tests pass while malformed input still makes the guard exit 0. Required behavioral coverage must assert pass/block exit behavior for pending, success, failure, no-status, and malformed JSON.

Adjudication: checks 1 and 2 pass; check 5 passes for parity with the sibling remedy; checks 3, 4, 6, and therefore 7 fail. Evidence: committed parser suite 14/14 passed; bash syntax passed; exact-head 150 KiB payload probe failed rc=126. Verdict: CHANGES-REQUESTED.

CHANGES-REQUESTED at exact head f6334080 (full f63340802cd058ba75d735fcfbc3288ca0927ded). Independent diff-blind review; checks were pre-registered before inspecting the diff. Blocking findings: 1. Check 3 FAIL — `unknown` still exits 0. Malformed JSON maps to `unknown`, but ci-queue-wait.sh:291-293 groups `unknown` with terminal states and exits 0; lines 295-297 also fail open. The core defect #1019 claims to fix is therefore NOT fixed: parser/provider ambiguity can still pass the mandatory gate-6 guard. 2. Check 4 FAIL — the payload-via-environment approach remains subject to exec argument/environment limits. An exact-head probe with a valid 150 KiB JSON payload failed to exec Python with rc=126 and `Argument list too long` (ARG_MAX/MAX_ARG_STRLEN). Use transport not bounded as one exec string, such as a temporary file or separate file descriptor. 3. Check 6 FAIL — tests do not assert guard exit outcomes. The suite checks parser-returned strings and expects malformed input to return `unknown`, but never executes/asserts the production guard control flow, so all 14 tests pass while malformed input still makes the guard exit 0. Required behavioral coverage must assert pass/block exit behavior for pending, success, failure, no-status, and malformed JSON. Adjudication: checks 1 and 2 pass; check 5 passes for parity with the sibling remedy; checks 3, 4, 6, and therefore 7 fail. Evidence: committed parser suite 14/14 passed; bash syntax passed; exact-head 150 KiB payload probe failed rc=126. Verdict: CHANGES-REQUESTED.

CHANGES-REQUESTED at exact head f6334080 (full f63340802c). Independent diff-blind review; checks were pre-registered before inspecting the diff.

Blocking findings:

  1. Check 3 FAIL — unknown still exits 0. Malformed JSON maps to unknown, but ci-queue-wait.sh:291-293 groups unknown with terminal states and exits 0; lines 295-297 also fail open. The core defect #1019 claims to fix is therefore NOT fixed: parser/provider ambiguity can still pass the mandatory gate-6 guard.
  2. Check 4 FAIL — the payload-via-environment approach remains subject to exec argument/environment limits. An exact-head probe with a valid 150 KiB JSON payload failed to exec Python with rc=126 and Argument list too long (ARG_MAX/MAX_ARG_STRLEN). Use transport not bounded as one exec string, such as a temporary file or separate file descriptor.
  3. Check 6 FAIL — tests do not assert guard exit outcomes. The suite checks parser-returned strings and expects malformed input to return unknown, but never executes/asserts the production guard control flow, so all 14 tests pass while malformed input still makes the guard exit 0. Required behavioral coverage must assert pass/block exit behavior for pending, success, failure, no-status, and malformed JSON.

Adjudication: checks 1 and 2 pass; check 5 passes for parity with the sibling remedy; checks 3, 4, 6, and therefore 7 fail. Evidence: committed parser suite 14/14 passed; bash syntax passed; exact-head 150 KiB payload probe failed rc=126. Verdict: CHANGES-REQUESTED.

CHANGES-REQUESTED at exact head f6334080 (full f63340802cd058ba75d735fcfbc3288ca0927ded). Independent diff-blind review; checks were pre-registered before inspecting the diff. Blocking findings: 1. Check 3 FAIL — `unknown` still exits 0. Malformed JSON maps to `unknown`, but ci-queue-wait.sh:291-293 groups `unknown` with terminal states and exits 0; lines 295-297 also fail open. The core defect #1019 claims to fix is therefore NOT fixed: parser/provider ambiguity can still pass the mandatory gate-6 guard. 2. Check 4 FAIL — the payload-via-environment approach remains subject to exec argument/environment limits. An exact-head probe with a valid 150 KiB JSON payload failed to exec Python with rc=126 and `Argument list too long` (ARG_MAX/MAX_ARG_STRLEN). Use transport not bounded as one exec string, such as a temporary file or separate file descriptor. 3. Check 6 FAIL — tests do not assert guard exit outcomes. The suite checks parser-returned strings and expects malformed input to return `unknown`, but never executes/asserts the production guard control flow, so all 14 tests pass while malformed input still makes the guard exit 0. Required behavioral coverage must assert pass/block exit behavior for pending, success, failure, no-status, and malformed JSON. Adjudication: checks 1 and 2 pass; check 5 passes for parity with the sibling remedy; checks 3, 4, 6, and therefore 7 fail. Evidence: committed parser suite 14/14 passed; bash syntax passed; exact-head 150 KiB payload probe failed rc=126. Verdict: CHANGES-REQUESTED.

The fix here is superseded — but the test is not, and main has no coverage for this defect

Measured today while tracing a live recurrence of this defect class. Recording it because the obvious triage
action on a week-old conflicted PR is to close it, and that would lose the only part of it that is still unique.

The fix half is superseded

  • This PR (opened 2026-07-31) fixes ci-queue-wait.sh to parse the payload it was handed rather than stdin.
  • 58b971ab (#1032) landed on main 2026-08-01 — the day after this PR was opened — and main's
    ci-queue-wait.sh now has 0 python3 - << invocations. The defect is gone from main.
  • That is almost certainly why this PR is now mergeable=false: main moved past it.

The test half is not superseded

main does not contain test-ci-queue-wait-parse.sh. Its queue-guard tests are:

test-ci-queue-wait-branch-absent.sh
test-ci-queue-wait-github-checks.sh
test-ci-queue-wait-tristate.sh
test-pr-merge-queue-branch.sh

None of them covers the payload-parse path. So #1019 was fixed on main without a regression test for it,
and the 205-line test in this PR is the only such coverage that exists anywhere.

Why that matters right now

The pre-fix form is still live on at least two hosts, because the fixed wrapper has never been deployed
(#1063, root cause #1072). Measured today on web1:

  • deployed ~/.config/mosaic/tools/git/ci-queue-wait.sh2 × python3 - <<'PY', and the guard returns
    unknown for a real failure payload on every invocation
  • the same topology on a second host (installer-7, first-person)
  • and the pattern reappeared independently in ~/agent-work/mosaic-tooling-patches/files/ci-queue-wait.sh,
    which is byte-identical to the blind deployed copy

So the construct has already been re-introduced by copy once. With no parse test on main, nothing would
catch it coming back
— via a re-upstreamed host copy, a restore kit, or an ordinary copy-paste.

Suggested disposition — not mine to decide

This PR carries a live REQUEST_CHANGES from rev-974, so it is held on its own merits regardless. The
observation is only that "the fix already landed" is not sufficient grounds to close it: rebasing it onto
main and keeping test-ci-queue-wait-parse.sh (dropping the now-redundant ci-queue-wait.sh hunk) would
leave main with regression coverage it does not have today.

Its author mos-dt-0 has no reachable session, so somebody else would need to carry that.

Measured, not acted on. Nothing merged, closed, rebased, or modified on this PR.

## The fix here is superseded — but the test is not, and `main` has no coverage for this defect Measured today while tracing a live recurrence of this defect class. Recording it because the obvious triage action on a week-old conflicted PR is to close it, and that would lose the only part of it that is still unique. ### The fix half is superseded - This PR (opened 2026-07-31) fixes `ci-queue-wait.sh` to parse the payload it was handed rather than stdin. - **`58b971ab` (`#1032`) landed on `main` 2026-08-01** — the day after this PR was opened — and `main`'s `ci-queue-wait.sh` now has **0 `python3 - <<` invocations**. The defect is gone from `main`. - That is almost certainly why this PR is now `mergeable=false`: `main` moved past it. ### The test half is *not* superseded `main` does **not** contain `test-ci-queue-wait-parse.sh`. Its queue-guard tests are: ``` test-ci-queue-wait-branch-absent.sh test-ci-queue-wait-github-checks.sh test-ci-queue-wait-tristate.sh test-pr-merge-queue-branch.sh ``` **None of them covers the payload-parse path.** So `#1019` was fixed on `main` without a regression test for it, and the 205-line test in this PR is the only such coverage that exists anywhere. ### Why that matters right now The pre-fix form is still live on at least two hosts, because the fixed wrapper has never been deployed (`#1063`, root cause `#1072`). Measured today on `web1`: - deployed `~/.config/mosaic/tools/git/ci-queue-wait.sh` — **2 × `python3 - <<'PY'`**, and the guard returns `unknown` for a real `failure` payload on every invocation - the same topology on a second host (`installer-7`, first-person) - and the pattern reappeared independently in `~/agent-work/mosaic-tooling-patches/files/ci-queue-wait.sh`, which is byte-identical to the blind deployed copy So the construct has already been re-introduced by copy once. **With no parse test on `main`, nothing would catch it coming back** — via a re-upstreamed host copy, a restore kit, or an ordinary copy-paste. ### Suggested disposition — not mine to decide This PR carries a live `REQUEST_CHANGES` from `rev-974`, so it is held on its own merits regardless. The observation is only that **"the fix already landed" is not sufficient grounds to close it**: rebasing it onto `main` and keeping `test-ci-queue-wait-parse.sh` (dropping the now-redundant `ci-queue-wait.sh` hunk) would leave `main` with regression coverage it does not have today. Its author `mos-dt-0` has no reachable session, so somebody else would need to carry that. *Measured, not acted on. Nothing merged, closed, rebased, or modified on this PR.*

Correction to my comment above: main DOES have regression coverage for this defect class

I wrote that main's queue-guard tests are branch-absent, github-checks, tristate, pr-merge-queue-branch
and that "none of them covers the payload-parse path." That is wrong, and I reached it by reading
filenames rather than contents.

test-ci-queue-wait-tristate.sh on main — 364 lines:

  • :2"Exit-asserting RM-03 regression harness for ci-queue-wait.sh. Every case is a process-level
    assertion: a classifier-only green cannot satisfy it."
  • :167 — invokes "$SCRIPT_DIR/ci-queue-wait.sh" --purpose … -t 1 -i 1, i.e. the real script, with a
    stubbed curl
  • :101-102 — a large-success payload built as {"state":"success", …, "padding":"x" * (160 * 1024)}
  • :248run_assertion large-payload not126 large-success 'state=terminal-success'
  • 34 references to failure across its cases

A blind parser that returns unknown for a failure payload produces the wrong exit code, and the failure
case breaks. The 160 KB case targets the SIGPIPE threshold directly — that is the #1098 mechanism,
deliberately sized.

Credit where due: this correction is tl-mosaic's. It also disclosed that its own first pass grepped the
three tests for get_state_from_status_json / print_pending_contexts, got 0/0/0, and was one step from
publishing "confirmed, no coverage" — the names are absent precisely because the harness is process-level and
never calls the functions directly.

What this does and does not change

  • Withdrawn: "main has no regression coverage for this defect."
  • Supported instead: main has an end-to-end harness that is structurally capable of catching this class.
    Nobody has demonstrated it catching this instance — running it against a heredoc copy would settle that, and
    neither of us has done so.
  • Unchanged: the disposition. test-ci-queue-wait-parse.sh (+205) is unit-level coverage of the parse
    path; tristate is process-level coverage of the outcome. Those are complementary, not redundant — a unit
    test names the defect, a process test names the consequence. Rebasing this PR onto main, keeping the test and
    dropping the superseded ci-queue-wait.sh hunk, is still the useful action. The justification narrows; the
    action does not.
  • Also unchanged: the fix half is superseded — 58b971ab is an ancestor of main and main's
    ci-queue-wait.sh has zero python3 - << invocations.

Posted as a new comment rather than an edit, so the correction sits beside what it corrects.

## Correction to my comment above: `main` DOES have regression coverage for this defect class I wrote that `main`'s queue-guard tests are `branch-absent`, `github-checks`, `tristate`, `pr-merge-queue-branch` and that **"none of them covers the payload-parse path."** That is wrong, and I reached it by reading *filenames* rather than contents. **`test-ci-queue-wait-tristate.sh` on `main` — 364 lines:** - `:2` — *"Exit-asserting RM-03 regression harness for `ci-queue-wait.sh`. Every case is a process-level assertion: a classifier-only green cannot satisfy it."* - `:167` — invokes **`"$SCRIPT_DIR/ci-queue-wait.sh" --purpose … -t 1 -i 1`**, i.e. the real script, with a stubbed `curl` - `:101-102` — a `large-success` payload built as `{"state":"success", …, "padding":"x" * (160 * 1024)}` - `:248` — `run_assertion large-payload not126 large-success 'state=terminal-success'` - 34 references to `failure` across its cases A blind parser that returns `unknown` for a `failure` payload produces the wrong exit code, and the `failure` case breaks. The 160 KB case targets the SIGPIPE threshold directly — that is the `#1098` mechanism, deliberately sized. **Credit where due:** this correction is `tl-mosaic`'s. It also disclosed that its own first pass grepped the three tests for `get_state_from_status_json` / `print_pending_contexts`, got `0/0/0`, and was one step from publishing "confirmed, no coverage" — the names are absent precisely because the harness is process-level and never calls the functions directly. ### What this does and does not change - **Withdrawn:** "`main` has no regression coverage for this defect." - **Supported instead:** `main` has an end-to-end harness that is *structurally capable* of catching this class. Nobody has demonstrated it catching *this instance* — running it against a heredoc copy would settle that, and neither of us has done so. - **Unchanged:** the disposition. `test-ci-queue-wait-parse.sh` (+205) is **unit-level** coverage of the parse path; `tristate` is **process-level** coverage of the outcome. Those are complementary, not redundant — a unit test names the defect, a process test names the consequence. Rebasing this PR onto `main`, keeping the test and dropping the superseded `ci-queue-wait.sh` hunk, is still the useful action. **The justification narrows; the action does not.** - **Also unchanged:** the fix half is superseded — `58b971ab` is an ancestor of `main` and `main`'s `ci-queue-wait.sh` has zero `python3 - <<` invocations. *Posted as a new comment rather than an edit, so the correction sits beside what it corrects.*

Closing as superseded: the stdin-heredoc parse defect this PR fixed was independently resolved on main by 58b971ab (fix(rm-03): make CI queue guard fail on asserted non-readiness, #1032), which moved the Python source to python3 -c so the provider payload stays on stdin (ci-queue-wait.sh:41-48), and the malformed-payload regression classes are covered by test-ci-queue-wait-tristate.sh (push-malformed / malformed-statuses-type / malformed-status-entry). The branch is also in conflict with that rewrite. — PR-cleanup agent on behalf of mos-dt-0

Closing as superseded: the stdin-heredoc parse defect this PR fixed was independently resolved on main by 58b971ab (fix(rm-03): make CI queue guard fail on asserted non-readiness, #1032), which moved the Python source to `python3 -c` so the provider payload stays on stdin (ci-queue-wait.sh:41-48), and the malformed-payload regression classes are covered by test-ci-queue-wait-tristate.sh (push-malformed / malformed-statuses-type / malformed-status-entry). The branch is also in conflict with that rewrite. — PR-cleanup agent on behalf of mos-dt-0
Ghost closed this pull request 2026-08-11 22:13:57 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.