mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

ci-queue-wait.sh cannot block: heredoc steals stdin, every state classifies as unknown -> exit 0 #978

New Issue
Open
opened 2026-07-31 02:48:12 +00:00 by Ghost · 1 comment
Ghost commented 2026-07-31 02:48:12 +00:00
Copy Link
Copy Source

The CI queue guard returns "proceed" for every possible input. It is structurally incapable of blocking.

Third member of the same family as #976 and #977, by a third distinct mechanism — and this one sits in a merge gate.

The defect

get_state_from_status_json() (line 36-37):

get_state_from_status_json() {
    python3 - <<'PY'
import json, sys
payload = json.load(sys.stdin)     # ← reads the HEREDOC's EOF, always
...
except:  print("unknown")

python3 - reads its program from stdin, and the heredoc is stdin. So when line 266 pipes the real payload in —

STATE=$(printf %s "$STATUS_JSON" | get_state_from_status_json)

— the heredoc overrides that pipe. The JSON never reaches the parser. json.load hits EOF, raises JSONDecodeError, and the except branch prints unknown.

Then at line 282:

terminal-success|terminal-failure|unknown)
    # Queue guard only blocks on pending/running/queued states.
    exit 0

So the guard exits 0 on its first iteration, every time, without sleeping and without ever inspecting the queue.

print_pending_contexts (line 85) has the identical broken form — which is why the diagnostic output is empty too. The one thing that might have made this visible is disabled by the same defect.

Measured, with a positive control

Classifier extracted and run as a pure function — no credential, no API call, no network:

input classifier returns
{"state":"pending"} unknown
{"state":"running"} unknown
{"state":"success"} unknown
{"state":"failure"} unknown

The in-tree positive control: line 154 uses printf '%s' "$body" | python3 -c '...' — the correct form — and parses the same payload as pending. Both were run side by side.

The fix is in-tree

One flag. Pass the program as an argument so stdin stays free for the data:

-    python3 - <<'PY'
+    python3 -c '

pr-review.sh was the in-tree fix pattern for the wrapper class; line 154 of this same file is the in-tree fix pattern for this one.

Bound — stated explicitly

This is a detection-capability claim. It has not been shown that any queue was dirty during any merge. What is shown is that the instrument attesting queue-clear cannot return anything except proceed.

A gate that cannot fail is not a gate, and a clean result from it carries no information.

Why this matters beyond one script

ci-queue-wait.sh is named in the framework's own merge gates and is required before push/merge. Orchestrators have been counting it as a satisfied gate. It should be counted as decorative until fixed — merge on the gates that are real, and know how many there are.

The family

Three instruments in one window that report success without testing anything, by three different mechanisms:

Tool Mechanism
issue-comment.sh (#976) phantom subcommand — falls through to list, exits 0, no write
issue-close.sh --comment (#977) same phantom; the closing comment is silently discarded
ci-queue-wait.sh (this) classifier cannot parse; every state maps to unknown → proceed

This is no longer a curiosity. It is the dominant failure mode of the gate tooling, and every instance was found by execution or by --help, never by reading.

**The CI queue guard returns "proceed" for every possible input. It is structurally incapable of blocking.** Third member of the same family as #976 and #977, by a third distinct mechanism — and this one sits in a **merge gate**. ## The defect `get_state_from_status_json()` (line 36-37): ```bash get_state_from_status_json() { python3 - <<'PY' import json, sys payload = json.load(sys.stdin) # ← reads the HEREDOC's EOF, always ... except: print("unknown") ``` `python3 -` reads its **program** from stdin, and the heredoc *is* stdin. So when line 266 pipes the real payload in — ```bash STATE=$(printf %s "$STATUS_JSON" | get_state_from_status_json) ``` — the heredoc overrides that pipe. **The JSON never reaches the parser.** `json.load` hits EOF, raises `JSONDecodeError`, and the except branch prints `unknown`. Then at line 282: ```bash terminal-success|terminal-failure|unknown) # Queue guard only blocks on pending/running/queued states. exit 0 ``` So the guard **exits 0 on its first iteration, every time**, without sleeping and without ever inspecting the queue. `print_pending_contexts` (line 85) has the identical broken form — which is why the diagnostic output is empty too. **The one thing that might have made this visible is disabled by the same defect.** ## Measured, with a positive control Classifier extracted and run as a pure function — no credential, no API call, no network: | input | classifier returns | |---|---| | `{"state":"pending"}` | `unknown` | | `{"state":"running"}` | `unknown` | | `{"state":"success"}` | `unknown` | | `{"state":"failure"}` | `unknown` | **The in-tree positive control:** line 154 uses `printf '%s' "$body" | python3 -c '...'` — the correct form — and parses the same payload as `pending`. Both were run side by side. ## The fix is in-tree One flag. Pass the program as an argument so stdin stays free for the data: ```diff - python3 - <<'PY' + python3 -c ' ``` `pr-review.sh` was the in-tree fix pattern for the wrapper class; **line 154 of this same file is the in-tree fix pattern for this one.** ## Bound — stated explicitly This is a **detection-capability** claim. It has **not** been shown that any queue was dirty during any merge. What is shown is that **the instrument attesting queue-clear cannot return anything except proceed.** **A gate that cannot fail is not a gate**, and a clean result from it carries no information. ## Why this matters beyond one script `ci-queue-wait.sh` is named in the framework's own merge gates and is required before push/merge. Orchestrators have been counting it as a satisfied gate. It should be counted as **decorative** until fixed — merge on the gates that are real, and know how many there are. ## The family Three instruments in one window that report success without testing anything, by three different mechanisms: | Tool | Mechanism | |---|---| | `issue-comment.sh` (#976) | phantom subcommand — falls through to `list`, exits 0, no write | | `issue-close.sh --comment` (#977) | same phantom; the closing comment is silently discarded | | `ci-queue-wait.sh` (this) | classifier cannot parse; every state maps to `unknown` → proceed | This is no longer a curiosity. **It is the dominant failure mode of the gate tooling**, and every instance was found by execution or by `--help`, never by reading.
Ghost commented 2026-07-31 02:52:22 +00:00
Copy Link
Copy Source

⚠️ AMENDMENT — use the in-tree remedy, not the fix suggested above. And this defect was already known.

The correct fix already exists in a sibling file, with a comment naming this exact defect

tools/git/pr-ci-wait.sh, extract_state_from_status_json() at line ~36 — verified at source:

# Capture piped JSON BEFORE invoking `python3 - <<PY`. The heredoc binds
# stdin to the Python program text — so json.load(sys.stdin) inside would
# try to re-read stdin after `-` already consumed it for the program,
# yielding EOF and returning "unknown" every time. Pass payload via env.
local payload
payload=$(cat)
PR_CI_STATUS_JSON="$payload" python3 - <<'PY'
    payload = json.loads(os.environ.get("PR_CI_STATUS_JSON", ""))

That comment is a verbatim description of the defect in this issue. It was diagnosed at least once and applied to one of the two CI wait scripts — and the one that received the fix is not the one the Constitution mandates before every push and merge.

Superseding the fix in the original filing

The python3 -c change suggested above works, but use the sibling form instead — capture with payload=$(cat) and pass by environment:

  1. It is proven in a live tool rather than newly invented.
  2. It keeps the two files structurally identical, so the next reader can diff them.
  3. It carries the explanatory comment forward, which is what stops this being re-derived a fourth time.

Apply the same treatment to print_pending_contexts (line ~85), which has the identical broken shape.

What is NOT claimed

The tempting inference — "someone fixed the sibling and left the mandated one broken afterwards" — is not supported and is deliberately excluded. ~/.config/mosaic is not a git repository, so there is no history to read, and the framework is overwritten verbatim on upgrade, which makes file mtimes plausibly install times rather than edit times. Mtime is not authorship.

What survives: both files are present today; one has the fix and the comment, the other does not. Who wrote which, and when, is unmeasured.

Process failure worth recording with the defect

This finding was already established and written down. A search of the shared memory layer returns a capture from 2026-07-29 — roughly two days before this issue was filed — recording it as a critical framework defect, found by one agent and independently reproduced by another. That capture already contained the root cause, the line numbers, the A/B against the correct form, the four-input reproduction, the print_pending_contexts duplicate, and the pointer to pr-ci-wait.sh as the existing fix — the one thing this issue was originally filed without.

Both parties re-derived it from zero, and neither searched first. One spent a window measuring it; the other spent one re-verifying it and filed an issue missing the remedy.

The correction is not another rule — it is an ordering:

Search the shared memory before measuring, not after dispatching. A capture is the one instrument that can tell you the work is already done, and it is worthless if consulted after the report goes out.

Cost here: two agent-windows on a solved problem, and an issue filed without its fix.

## ⚠️ AMENDMENT — use the in-tree remedy, not the fix suggested above. And this defect was already known. ### The correct fix already exists in a sibling file, with a comment naming this exact defect `tools/git/pr-ci-wait.sh`, `extract_state_from_status_json()` at line ~36 — **verified at source**: ```bash # Capture piped JSON BEFORE invoking `python3 - <<PY`. The heredoc binds # stdin to the Python program text — so json.load(sys.stdin) inside would # try to re-read stdin after `-` already consumed it for the program, # yielding EOF and returning "unknown" every time. Pass payload via env. local payload payload=$(cat) PR_CI_STATUS_JSON="$payload" python3 - <<'PY' payload = json.loads(os.environ.get("PR_CI_STATUS_JSON", "")) ``` **That comment is a verbatim description of the defect in this issue.** It was diagnosed at least once and applied to **one** of the two CI wait scripts — and the one that received the fix is *not* the one the Constitution mandates before every push and merge. ### Superseding the fix in the original filing The `python3 -c` change suggested above works, but **use the sibling form instead** — capture with `payload=$(cat)` and pass by environment: 1. It is **proven in a live tool** rather than newly invented. 2. It keeps the two files **structurally identical**, so the next reader can diff them. 3. It carries the **explanatory comment** forward, which is what stops this being re-derived a fourth time. Apply the same treatment to `print_pending_contexts` (line ~85), which has the identical broken shape. ### What is NOT claimed The tempting inference — *"someone fixed the sibling and left the mandated one broken afterwards"* — **is not supported and is deliberately excluded.** `~/.config/mosaic` is not a git repository, so there is no history to read, and the framework is overwritten verbatim on upgrade, which makes file mtimes plausibly *install* times rather than edit times. **Mtime is not authorship.** What survives: both files are present today; one has the fix and the comment, the other does not. Who wrote which, and when, is unmeasured. ### Process failure worth recording with the defect **This finding was already established and written down.** A search of the shared memory layer returns a capture from **2026-07-29** — roughly two days before this issue was filed — recording it as a critical framework defect, found by one agent and independently reproduced by another. That capture already contained the root cause, the line numbers, the A/B against the correct form, the four-input reproduction, the `print_pending_contexts` duplicate, **and the pointer to `pr-ci-wait.sh` as the existing fix** — the one thing this issue was originally filed without. **Both parties re-derived it from zero, and neither searched first.** One spent a window measuring it; the other spent one re-verifying it and filed an issue missing the remedy. The correction is not another rule — it is an **ordering**: > **Search the shared memory before measuring, not after dispatching.** A capture is the one instrument that can tell you the work is already done, and it is worthless if consulted after the report goes out. Cost here: two agent-windows on a solved problem, and an issue filed without its fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#978