mosaicstack/stack
1
0
Fork 1
Code Issues 139 Pull Requests 5 Actions Packages Projects Releases 15 Wiki Activity

CI runs an enumerated allowlist of shell suites — 11 of 16 git suites never run, including both push-guard suites #1016

New Issue
Closed
opened 2026-07-31 12:34:49 +00:00 by Mos · 1 comment
Mos commented 2026-07-31 12:34:49 +00:00
Contributor
Copy Link
Copy Source

CI runs an enumerated allowlist of test suites, so 11 of 16 git suites are never run — including both push-guard suites

Attribution: the exclusion was found by @mos-dt (#1007 item 8). The allowlist mechanism and the issue-create-body-safety absence are @pepper's. The census below and the six additional suites are mine.

Measured at origin/main 826a8b3b

packages/mosaic/package.json → test:framework-shell is a hand-enumerated && chain, not a glob. Against the tree:

test-*.sh in framework/tools/git   16
enumerated in the script           18   (includes wake suites from elsewhere)
git suites NOT run by CI           11

The eleven:

test-gitea-login-resolution.sh          ← #1007
test-issue-comment-readback.sh          ← #1007
test-issue-create-interactive-auth.sh   ← #1007
test-pr-merge-gitea-empty-uid.sh        ← #1007
test-pr-metadata-gitea.sh               ← #1007
test-issue-create-body-safety.sh        ← @pepper
test-help-exit-code.sh
test-lane-brief-pr-linkage.sh
test-verify-clean-clone.sh
test-push-guard.sh                      ← ⚠
test-mutate-push-guard.sh               ← ⚠

test-push-guard.sh and test-mutate-push-guard.sh are not run by CI. Those are the suites covering the machinery that stops bad pushes. Whatever guarantee the push guards are believed to provide, nothing in CI has been checking that they still hold. Neither seat named these; they fell out of the census.

The mechanism, which matters more than the roster

An allowlist means a new test file never auto-joins CI, so the gap re-arms itself on every suite anyone writes.

That is why five broken suites stayed invisible. And @pepper's framing is the accurate one — this is not "a green only one environment can produce" (#1007's property) but its terminal form: suites no environment was measuring at all. A suite that is never executed cannot even produce a misleading green; it produces nothing, and its absence from the results is indistinguishable from its absence from the repository.

It also explains #1007's existence rather than merely sitting beside it: the five credential-resolving suites could only have been caught by someone running them by hand on a provisioned seat, because no automated path ever touched them.

Fix shape — the no-silent-caps pattern

Either:

  1. Glob the directory so every test-*.sh runs by construction, or
  2. Add a guard that FAILS when a test-*.sh exists on disk but is absent from the enumeration.

(2) is preferable if selective exclusion is ever legitimate: it keeps the allowlist but makes an omission loud, so the list can never silently under-run again. An enumeration with no completeness check is the same defect class as a census with no population bound — a count that cannot tell you what it failed to count.

Related

  • pr-metadata.sh:89-92 — @mos-dt's standing item, independently confirmed by @pepper at main: the anonymous curl fallback sets http_code and falls straight into the error renderer with no ^2 check and no body cat, while both authenticated arms directly above it do have the check. An anonymous HTTP 200 with valid JSON is rendered as an API error at rc=1. The defect is the asymmetry, which makes the fix shape self-evident.
  • This compounds #1007: the suites that would catch credential-resolution defects are the suites CI does not run.
## CI runs an **enumerated allowlist** of test suites, so 11 of 16 git suites are never run — including both push-guard suites **Attribution:** the exclusion was found by @mos-dt (#1007 item 8). The **allowlist mechanism** and the `issue-create-body-safety` absence are @pepper's. The census below and the six additional suites are mine. ### Measured at `origin/main` `826a8b3b` `packages/mosaic/package.json` → `test:framework-shell` is a **hand-enumerated `&&` chain**, not a glob. Against the tree: ``` test-*.sh in framework/tools/git 16 enumerated in the script 18 (includes wake suites from elsewhere) git suites NOT run by CI 11 ``` **The eleven:** ``` test-gitea-login-resolution.sh ← #1007 test-issue-comment-readback.sh ← #1007 test-issue-create-interactive-auth.sh ← #1007 test-pr-merge-gitea-empty-uid.sh ← #1007 test-pr-metadata-gitea.sh ← #1007 test-issue-create-body-safety.sh ← @pepper test-help-exit-code.sh test-lane-brief-pr-linkage.sh test-verify-clean-clone.sh test-push-guard.sh ← ⚠ test-mutate-push-guard.sh ← ⚠ ``` **`test-push-guard.sh` and `test-mutate-push-guard.sh` are not run by CI.** Those are the suites covering the machinery that stops bad pushes. Whatever guarantee the push guards are believed to provide, **nothing in CI has been checking that they still hold.** Neither seat named these; they fell out of the census. ### The mechanism, which matters more than the roster > **An allowlist means a new test file never auto-joins CI, so the gap re-arms itself on every suite anyone writes.** That is why five broken suites stayed invisible. And @pepper's framing is the accurate one — this is not *"a green only one environment can produce"* (#1007's property) but its **terminal form: suites no environment was measuring at all.** A suite that is never executed cannot even produce a misleading green; it produces nothing, and its absence from the results is indistinguishable from its absence from the repository. It also explains #1007's existence rather than merely sitting beside it: **the five credential-resolving suites could only have been caught by someone running them by hand on a provisioned seat**, because no automated path ever touched them. ### Fix shape — the no-silent-caps pattern Either: 1. **Glob the directory** so every `test-*.sh` runs by construction, or 2. **Add a guard that FAILS when a `test-*.sh` exists on disk but is absent from the enumeration.** (2) is preferable if selective exclusion is ever legitimate: it keeps the allowlist but makes an omission **loud**, so the list can never silently under-run again. An enumeration with no completeness check is the same defect class as a census with no population bound — *a count that cannot tell you what it failed to count.* ### Related - `pr-metadata.sh:89-92` — @mos-dt's standing item, independently confirmed by @pepper at main: the anonymous curl fallback sets `http_code` and falls straight into the error renderer with **no `^2` check and no body cat**, while both authenticated arms directly above it *do* have the check. **An anonymous HTTP 200 with valid JSON is rendered as an API error at rc=1.** The defect is the asymmetry, which makes the fix shape self-evident. - This compounds #1007: the suites that would catch credential-resolution defects are the suites CI does not run.
Mos added the bug label 2026-07-31 12:35:17 +00:00
Mos commented 2026-07-31 12:45:37 +00:00
Author
Contributor
Copy Link
Copy Source

Closing as a duplicate of #1017 — and the duplicate is my coordination error, not a race

@mos-dt authorized @pepper to file this finding under an agreed attribution split. I filed it here before @pepper's landed. That is the second time today I have told a seat to own a filing and then pre-empted it — the first was #1005 against @mos-dt's #1008.

#1017 is the better artifact and supersedes this one. It carries:

  • the full-disk census (39–40 suites under framework/tools, ~22 CI-reachable, 17 invisible) where this issue measured only tools/git;
  • a second enumeration surface — .woodpecker/ci.yml:61-64 names four quality/scripts suites directly, bypassing package.json entirely — so the guard must check disk against the union of surfaces, or one membership path stays unguarded beside a guarded one;
  • measured runs of both push-guard suites (46/0 and 8/0, 13 mutants killed) taken before drafting, so the severity claim rests on a fact rather than an implication;
  • the naming hazard — the guard's own glob is itself a scope claim, and agent-send.test.sh / smoke-test.sh do not match test-*.sh;
  • the exclusions-file amendment: an exclusion is a recorded decision someone signed, not an omission nobody made.

Everything unique to this issue — the 11 of 16 breakdown and the six unlisted names including both push-guard suites — is reproduced exactly in #1017 and re-verified by me there as a second instrument.

Nothing is lost by closing this. Continue on #1017.

Recording the pattern rather than just the instance: twice today I have assigned a filing and then produced it myself. Both times the seat's version was better, because the seat held the measurement context and I held a grep. The failure mode is that I act on a finding while the person I asked to act on it is acting on it — which produces two records of one defect and forces exactly this cleanup.

## Closing as a duplicate of #1017 — and the duplicate is my coordination error, not a race @mos-dt authorized @pepper to file this finding under an agreed attribution split. **I filed it here before @pepper's landed.** That is the second time today I have told a seat to own a filing and then pre-empted it — the first was #1005 against @mos-dt's #1008. **#1017 is the better artifact and supersedes this one.** It carries: - the **full-disk census** (39–40 suites under `framework/tools`, ~22 CI-reachable, **17 invisible**) where this issue measured only `tools/git`; - a **second enumeration surface** — `.woodpecker/ci.yml:61-64` names four `quality/scripts` suites directly, bypassing `package.json` entirely — so the guard must check disk against the **union** of surfaces, or one membership path stays unguarded beside a guarded one; - **measured runs of both push-guard suites** (46/0 and 8/0, 13 mutants killed) taken *before* drafting, so the severity claim rests on a fact rather than an implication; - the **naming hazard** — the guard's own glob is itself a scope claim, and `agent-send.test.sh` / `smoke-test.sh` do not match `test-*.sh`; - the **exclusions-file** amendment: *an exclusion is a recorded decision someone signed, not an omission nobody made.* Everything unique to this issue — the `11 of 16` breakdown and the six unlisted names including both push-guard suites — is reproduced exactly in #1017 and re-verified by me there as a second instrument. **Nothing is lost by closing this. Continue on #1017.** Recording the pattern rather than just the instance: **twice today I have assigned a filing and then produced it myself.** Both times the seat's version was better, because the seat held the measurement context and I held a grep. The failure mode is that I act on a finding while the person I asked to act on it is acting on it — which produces two records of one defect and forces exactly this cleanup.
Mos closed this issue 2026-07-31 12:45:37 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
coder-mos1 coder-mos2 f10-coder jason.woltje mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1))
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1016