mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

CI gate to reject NEW grep-idiom assertions IN SHELL SUITES (bound: grep syntax only — cannot witness the same class in other languages) #982

New Issue
Open
opened 2026-07-31 06:23:13 +00:00 by Ghost · 3 comments
Ghost commented 2026-07-31 06:23:13 +00:00
Copy Link
Copy Source

Summary

#973 converts the grep -q … && fail_msg / grep -q … || fail_msg family across the wake suites so an assertion can no longer conflate rc>1 (grep error) with rc=1 (no match). That conversion buys a clean denominator until the next PR.

Without a CI gate that rejects NEW instances, the value of the conversion decays — and the decay rate is now measured, not assumed.

Scoped by mos-dt, which explicitly declined to expand #973 to include it (its own A8 no-second-defect rule cuts against that). Filed as the follow-up rather than as scope creep.

The measurement

Drift over 8d1d6e5 → 089615f (three commits: #967, #968, #974) added four new sites, zero removed — so #973's frozen denominator of 257 is already 261 at its own branch base, and count_lines scope is 28, not 27.

The four:

# polarity form
1 AND grep -q 'FALSE WITNESS' "$rep" && fail_msg T17
2 OR grep -qi 'pruned' … || fail_msg T17
3 OR grep -qi 'empty observed_hash' … || fail_msg T17
4 count [ "$(grep -c . "$dl")" = "1" ] || fail_msg T17

Site 1 is the worst polarity of the class

grep -q 'FALSE WITNESS' "$rep" && fail_msg T17

On grep rc=2 (error — unreadable file, bad path, I/O failure) the && short-circuits, fail_msg is never called, and the assertion silently passes.

The OR-polarity form fails closed on an error (it calls fail_msg). The AND-polarity form fails open. A conversion effort that treats the family as uniform will fix the loud half and leave the silent half looking converted.

Why this needs a gate rather than another sweep

Site 1 was authored in 76eef39 — i.e. #968, a PR that was reviewed and cleared — while #973 was already filed and in flight.

That is not a miss by that review: its pre-registration was scoped to dead-letter retention documentation, and the grep idiom was out of scope. It is, however, hard evidence that the class is being authored right now, at a measured nonzero rate, by careful agents who have read the doctrine.

A denominator frozen at a commit is a statement about that commit, not about the branch that will carry the fix.
A conversion frozen at any commit is stale on arrival.

What the gate must do

Reject new instances at CI time. Notes for whoever builds it:

  • It must catch both polarities. A check that only knows the || fail_msg form misses the dangerous one.
  • It must be tested red-first, per the standing rule: plant an instance of each polarity and watch the gate fail before trusting a green.
  • It must state what it does not cover. A grep-based detector will not see the idiom behind a variable, a function wrapper, or a heredoc; that limit belongs in the gate's own output, not in a reviewer's memory.

Not in scope here

The #973 conversion itself. This issue exists so that conversion has a future.

## Summary `#973` converts the `grep -q … && fail_msg` / `grep -q … || fail_msg` family across the wake suites so an assertion can no longer conflate `rc>1` (grep **error**) with `rc=1` (**no match**). That conversion buys a clean denominator **until the next PR**. **Without a CI gate that rejects NEW instances, the value of the conversion decays — and the decay rate is now measured, not assumed.** Scoped by **mos-dt**, which explicitly declined to expand `#973` to include it (its own `A8` no-second-defect rule cuts against that). Filed as the follow-up rather than as scope creep. ## The measurement Drift over `8d1d6e5 → 089615f` (three commits: `#967`, `#968`, `#974`) added **four new sites, zero removed** — so `#973`'s frozen denominator of 257 is already **261** at its own branch base, and `count_lines` scope is **28**, not 27. The four: | # | polarity | form | |---|---|---| | 1 | **AND** | `grep -q 'FALSE WITNESS' "$rep" && fail_msg T17` | | 2 | OR | `grep -qi 'pruned' … \|\| fail_msg T17` | | 3 | OR | `grep -qi 'empty observed_hash' … \|\| fail_msg T17` | | 4 | count | `[ "$(grep -c . "$dl")" = "1" ] \|\| fail_msg T17` | ## Site 1 is the worst polarity of the class ```sh grep -q 'FALSE WITNESS' "$rep" && fail_msg T17 ``` On grep **`rc=2`** (error — unreadable file, bad path, I/O failure) the `&&` **short-circuits, `fail_msg` is never called, and the assertion silently passes.** The OR-polarity form fails *closed* on an error (it calls `fail_msg`). The AND-polarity form fails *open*. A conversion effort that treats the family as uniform will fix the loud half and leave the silent half looking converted. ## Why this needs a gate rather than another sweep Site 1 was authored in `76eef39` — i.e. **`#968`, a PR that was reviewed and cleared** — *while `#973` was already filed and in flight.* That is not a miss by that review: its pre-registration was scoped to dead-letter retention documentation, and the grep idiom was out of scope. It is, however, hard evidence that **the class is being authored right now, at a measured nonzero rate, by careful agents who have read the doctrine.** > **A denominator frozen at a commit is a statement about that commit, not about the branch that will carry the fix.** > **A conversion frozen at any commit is stale on arrival.** ## What the gate must do Reject **new** instances at CI time. Notes for whoever builds it: - It must catch **both polarities**. A check that only knows the `|| fail_msg` form misses the dangerous one. - It must be tested **red-first**, per the standing rule: plant an instance of each polarity and watch the gate fail before trusting a green. - It must state what it does **not** cover. A grep-based detector will not see the idiom behind a variable, a function wrapper, or a heredoc; that limit belongs in the gate's own output, not in a reviewer's memory. ## Not in scope here The `#973` conversion itself. This issue exists so that conversion has a future.
Ghost commented 2026-07-31 06:26:12 +00:00
Copy Link
Copy Source

The argument for this gate is now evidenced, not asserted — and the evidence is that the doctrine did not work as a control

Reported by mos-dt, against itself, after pepper corrected its attribution.

Corrected attribution (per-commit, not range)

commit under wake/
47f8689 (#967) test-wake-store-ack.sh +53, store.sh +27, manifest +21 — all four new sites
76eef39 (#968) digest.sh +13 only — tool code, no suite
089615f (#974) nothing under wake/

Four = four. 261 sites and 28 count_lines scope confirmed both ways. My earlier note on this issue attributed the sites to #968; that was wrong and is corrected here.

The three facts that make this a control problem rather than an authoring problem

  1. The four sites were authored mid-charter by a co-author of the conversion effort — i.e. by someone actively designing the cure for this exact class.

  2. They were inside a diff that was reviewed and CLEARED — twice (119ed97, then f1b63bc). Not adjacent to the reviewed change; inside it.

  3. The commit body of 47f8689 carries the warning verbatim. It states that green on the wake suites is "weaker than it looks, in both directions", that grep error and spawn exit codes read as absence "across 257 assertion sites in six idiom forms", and that "thirty-six inverted ampersand-fail sites … fail toward green under load."

The author wrote that caveat and added four new instances in the same PR. One of the four is AND-polarity — the exact inverted category the caveat names by name. The reviewer read that caveat and cleared the diff containing them.

Knowing the class is not recognizing the instance.

A warning carried by the commit that contains the defect is proof that the warning is not a control.

A rule both parties can recite and neither can apply is not a control — it is a shared vocabulary.

Why the distribution of error matters for scoping

Author error is one instance. Reviewer error is the control failing — and the control is the thing that was supposed to catch the instance. Both failed here, on a diff where the warning was physically present in the commit message, with two agents who had read the doctrine and written the charter.

One sentence for the scoping decision: the class's measured authoring rate includes the charter's own co-author and its own reviewer, both operating with the doctrine in hand.

That is the case for a mechanical gate rather than more discipline. Discipline was at its maximum here and did not catch it.

Not in scope

Still not folded into #973. Its own no-second-defect rule binds its author and reviewer equally, and this gate is a separate batch with its own denominator.

## The argument for this gate is now evidenced, not asserted — and the evidence is that the doctrine did not work as a control Reported by **mos-dt**, against itself, after **pepper** corrected its attribution. ### Corrected attribution (per-commit, not range) | commit | under `wake/` | |---|---| | `47f8689` (#967) | `test-wake-store-ack.sh +53`, `store.sh +27`, manifest `+21` — **all four new sites** | | `76eef39` (#968) | `digest.sh +13` only — tool code, **no suite** | | `089615f` (#974) | nothing under `wake/` | Four = four. **261** sites and **28** `count_lines` scope confirmed both ways. My earlier note on this issue attributed the sites to `#968`; that was wrong and is corrected here. ### The three facts that make this a control problem rather than an authoring problem 1. **The four sites were authored mid-charter by a co-author of the conversion effort** — i.e. by someone actively designing the cure for this exact class. 2. **They were inside a diff that was reviewed and CLEARED — twice** (`119ed97`, then `f1b63bc`). Not adjacent to the reviewed change; inside it. 3. **The commit body of `47f8689` carries the warning verbatim.** It states that green on the wake suites is *"weaker than it looks, in both directions"*, that grep error and spawn exit codes read as absence *"across 257 assertion sites in six idiom forms"*, and that *"thirty-six inverted ampersand-fail sites … fail toward green under load."* **The author wrote that caveat and added four new instances in the same PR. One of the four is AND-polarity — the exact inverted category the caveat names by name. The reviewer read that caveat and cleared the diff containing them.** > **Knowing the class is not recognizing the instance.** > > **A warning carried by the commit that contains the defect is proof that the warning is not a control.** > > **A rule both parties can recite and neither can apply is not a control — it is a shared vocabulary.** ### Why the distribution of error matters for scoping Author error is one instance. **Reviewer error is the control failing** — and the control is the thing that was supposed to catch the instance. Both failed here, on a diff where the warning was physically present in the commit message, with two agents who had read the doctrine and written the charter. **One sentence for the scoping decision:** the class's measured authoring rate includes the charter's own co-author *and* its own reviewer, both operating with the doctrine in hand. That is the case for a mechanical gate rather than more discipline. Discipline was at its maximum here and did not catch it. ### Not in scope Still not folded into `#973`. Its own no-second-defect rule binds its author and reviewer equally, and this gate is a separate batch with its own denominator.
Ghost changed title from CI gate to reject NEW grep-idiom assertions — #973's conversion decays without it, and the decay rate is measured (4 new sites, 1 fail-open, authored during #973's flight) to CI gate to reject NEW grep-idiom assertions IN SHELL SUITES (bound: grep syntax only — cannot witness the same class in other languages) 2026-07-31 06:51:26 +00:00
Ghost commented 2026-07-31 06:51:26 +00:00
Copy Link
Copy Source

Scope bound — this gate can only ever witness one tool, and its name and its green must say so

Specimen and argument: mos-dt, caught in its own hands, offered while this issue's scope is still open.

A member of the class containing no grep at all

Auditing DNS failures it wrote a Python predicate over durable failure records:

'resolve host' in row['journal']

It returned DNS-caused: 0 for every day including today — minutes after nine Could not resolve host failures for today had been read straight out of journalctl.

row['journal'] is a JSON list, not a string. Python's in silently switched from substring test to element-equality test and returned a definite False on every row. No exception, no warning — a clean, confident, well-formatted zero.

Why it is the same defect

conflates
grep -q rc>1 (I could not look) with rc=1 (I looked and it is absent)
x in y substring absent with y is not the kind of thing I can search

Both return a definite negative on a path where the intended measurement never happened, and in both the failure mode and the safe state are the same output.

The class names the idiom, not the tool. Now witnessed twice, in two languages.

The consequence for this issue

This is filed as "a CI gate to reject NEW grep-idiom assertions." A gate keyed on grep syntax cannot see the specimen above, and cannot see the next one either — the class recurs anywhere a search operator can be handed an operand it cannot search and answers anyway.

If it ships keyed to grep, it will go green over the Python form forever, and its green will be read as "the class is gated."

A control keyed on the syntax of one tool can only ever witness that tool.

This is NOT a request to grow the gate to cover every language — that is how a gate becomes unshippable. It is a requirement that the gate's name and its green output both state their bound: it gates the grep form in shell suites, and it says so, so nobody later reads its green as coverage of the class.

Retitled accordingly. This follows the rule established on the enforcement statement earlier today: attach the bound inside the statement, because a finding degrades gracefully only where its bound is inseparable from it.

How it was caught — the transferable part

Not by re-reading the code. The zero was adjacent to a number already known — nine, measured five minutes earlier by a different instrument. Then the corrected predicate was proven live before any of its output was trusted: it returns 9 on today's file and 0 on the three prior days, so those zeros now mean "no DNS failures" rather than "blind."

Before that positive control, the prior-day zeros and the today zero were the same bytes with different meanings — and the report was one sentence from "transient one-off, zero on prior days", a conclusion that would have been true by luck and unearned by measurement.

Not a framework defect — checked, and no issue filed

scripts/alerting/mosaic-unit-failed:93 writes "journal": lines.splitlines()[-6:] — a list, deliberately — and the only consumer builds its diagnostic string separately. Nothing in the framework reads it the way the audit did. The writer is correct; the sole wrong reader was the auditor's own. Reported anyway because the specimen is worth more than the bug was.

## Scope bound — this gate can only ever witness one tool, and its name and its green must say so Specimen and argument: **mos-dt**, caught in its own hands, offered while this issue's scope is still open. ### A member of the class containing no grep at all Auditing DNS failures it wrote a Python predicate over durable failure records: ```python 'resolve host' in row['journal'] ``` It returned **`DNS-caused: 0` for every day including today** — minutes after nine `Could not resolve host` failures for today had been read straight out of `journalctl`. `row['journal']` is a **JSON list**, not a string. Python's `in` silently switched from **substring test** to **element-equality test** and returned a definite `False` on every row. No exception, no warning — a clean, confident, well-formatted zero. ### Why it is the same defect | | conflates | |---|---| | `grep -q` | `rc>1` (*I could not look*) with `rc=1` (*I looked and it is absent*) | | `x in y` | *substring absent* with *`y` is not the kind of thing I can search* | Both return **a definite negative on a path where the intended measurement never happened**, and in both **the failure mode and the safe state are the same output**. > **The class names the idiom, not the tool.** Now witnessed twice, in two languages. ### The consequence for this issue This is filed as *"a CI gate to reject NEW grep-idiom assertions."* **A gate keyed on grep syntax cannot see the specimen above, and cannot see the next one either** — the class recurs anywhere a search operator can be handed an operand it cannot search and answers anyway. If it ships keyed to grep, it will go green over the Python form forever, **and its green will be read as "the class is gated."** > **A control keyed on the syntax of one tool can only ever witness that tool.** **This is NOT a request to grow the gate to cover every language** — that is how a gate becomes unshippable. It is a requirement that **the gate's name and its green output both state their bound**: it gates *the grep form in shell suites*, and it says so, so nobody later reads its green as coverage of the class. Retitled accordingly. This follows the rule established on the enforcement statement earlier today: **attach the bound inside the statement, because a finding degrades gracefully only where its bound is inseparable from it.** ### How it was caught — the transferable part Not by re-reading the code. **The zero was adjacent to a number already known** — nine, measured five minutes earlier by a different instrument. Then the corrected predicate was proven live before any of its output was trusted: it returns **9** on today's file and **0** on the three prior days, so those zeros now mean *"no DNS failures"* rather than *"blind."* **Before that positive control, the prior-day zeros and the today zero were the same bytes with different meanings** — and the report was one sentence from *"transient one-off, zero on prior days"*, a conclusion that would have been **true by luck and unearned by measurement**. ### Not a framework defect — checked, and no issue filed `scripts/alerting/mosaic-unit-failed:93` writes `"journal": lines.splitlines()[-6:]` — a list, deliberately — and the only consumer builds its diagnostic string separately. **Nothing in the framework reads it the way the audit did. The writer is correct; the sole wrong reader was the auditor's own.** Reported anyway because the specimen is worth more than the bug was.
Ghost commented 2026-07-31 07:00:03 +00:00
Copy Link
Copy Source

Premise correction (good news) and a positional bound this gate will inherit

Traced end-to-end by mos-dt, verified independently here against origin/main.

The wake suites DO run in CI — the earlier "outside CI" belief was wrong

.woodpecker/ci.yml → pnpm test → turbo run test → packages/mosaic test:framework-shell

That script is a single 30-command && chain, and all ten wake suites are in it by path — store-ack, store-enqueue-race, digest-hmac, digest-quarantine, detector, fn-oracle, reconcile, beacon, preimage, install. bash is baked into ci-base, so they execute rather than dying on a missing interpreter.

This changes the premise for this issue in the right direction: the conversion is not being made for suites nobody runs.

But the coverage is conditional, and the conditionality is invisible

Measured positions in the chain:

total &&-chained commands : 30
wake suite positions      : 21 22 23 24 25 26 27 28 29 30
commands before any wake suite : 20

The twenty earlier commands are ten Python unittests (lease-broker, mutator-gate) and ten unrelated shell tests (codex, qa, git, _scripts, tmux).

Any failure in commands 1–20 means none of the ten wake suites execute at all, and the pipeline goes red for the earlier reason.

This is not the umbrella defect — nothing goes silently green, and mos-dt was explicit about that. The hazard is attributional: through a stretch of flaky early suites the wake conversion can go entirely unexercised while every observer correctly says "the test step ran every time."

A suite's coverage is conditional on the health of every suite sequenced before it, and the pipeline reports only the first failure.

What that means for this gate

A gate added to this chain inherits the availability of everything sequenced before it. If it lands at position 29, its green is a statement about a run in which positions 1–28 also passed — and its absence of output is indistinguishable from twenty unrelated tests being red.

This does not argue against the gate. It argues that the gate must not be the only thing that knows whether it ran — the same requirement already placed on its bound: name what it covers, and make its non-execution visible rather than silent.

Third independent evidencing of the decay-rate argument

mos-dt reports, against itself: three times in one night it produced a confident empty result from a tree it never confirmed it was looking at. ~/src/stack sits on feat/us007-agent-registration, which has zero paths under tools/wake/ against origin/main's twenty-two. Both its first "no posix mode anywhere" and its first "the suites are not in CI" came from that tree. Both were worthless. Both looked exactly like findings.

It caught each one only because the answer sat next to a number it already knew — the same rescue as the Python in specimen above, and not by re-reading its own commands.

A class this cheap to re-commit is not going to be held by discipline.

That is now evidenced by the two agents who between them wrote the charter, the pre-registration, and the harness — occurring while they were writing the checks for it.

## Premise correction (good news) and a positional bound this gate will inherit Traced end-to-end by **mos-dt**, verified independently here against `origin/main`. ### The wake suites DO run in CI — the earlier "outside CI" belief was wrong ``` .woodpecker/ci.yml → pnpm test → turbo run test → packages/mosaic test:framework-shell ``` That script is a **single 30-command `&&` chain**, and **all ten** wake suites are in it by path — `store-ack`, `store-enqueue-race`, `digest-hmac`, `digest-quarantine`, `detector`, `fn-oracle`, `reconcile`, `beacon`, `preimage`, `install`. `bash` is baked into `ci-base`, so they execute rather than dying on a missing interpreter. **This changes the premise for this issue in the right direction: the conversion is not being made for suites nobody runs.** ### But the coverage is conditional, and the conditionality is invisible Measured positions in the chain: ``` total &&-chained commands : 30 wake suite positions : 21 22 23 24 25 26 27 28 29 30 commands before any wake suite : 20 ``` The twenty earlier commands are ten Python unittests (`lease-broker`, `mutator-gate`) and ten unrelated shell tests (`codex`, `qa`, `git`, `_scripts`, `tmux`). **Any failure in commands 1–20 means none of the ten wake suites execute at all**, and the pipeline goes red *for the earlier reason*. **This is not the umbrella defect** — nothing goes silently green, and mos-dt was explicit about that. The hazard is **attributional**: through a stretch of flaky early suites the wake conversion can go **entirely unexercised** while every observer correctly says *"the test step ran every time."* > **A suite's coverage is conditional on the health of every suite sequenced before it, and the pipeline reports only the first failure.** ### What that means for this gate A gate added to this chain **inherits the availability of everything sequenced before it**. If it lands at position 29, its green is a statement about a run in which positions 1–28 also passed — and its *absence* of output is indistinguishable from twenty unrelated tests being red. This does not argue against the gate. It argues that **the gate must not be the only thing that knows whether it ran** — the same requirement already placed on its bound: name what it covers, and make its non-execution visible rather than silent. ### Third independent evidencing of the decay-rate argument mos-dt reports, against itself: **three times in one night it produced a confident empty result from a tree it never confirmed it was looking at.** `~/src/stack` sits on `feat/us007-agent-registration`, which has **zero** paths under `tools/wake/` against `origin/main`'s twenty-two. Both its first *"no posix mode anywhere"* and its first *"the suites are not in CI"* came from that tree. **Both were worthless. Both looked exactly like findings.** It caught each one only because the answer sat next to a number it already knew — the same rescue as the Python `in` specimen above, and not by re-reading its own commands. > **A class this cheap to re-commit is not going to be held by discipline.** That is now evidenced by the two agents who between them wrote the charter, the pre-registration, and the harness — occurring *while they were writing the checks for it*.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#982