test(ri-050): RI-1-002 publish-gate negative controls (#1275) #1305

2026-08-18T04:26:00Z

jarvis commented

2026-08-18 04:26:00 +00:00

Adds scripts/publish-gate-structure.test.mjs — checked-in negative controls proving a failed verify step blocks every publish effect in .woodpecker/publish.yml, proven statically from the pipeline file (no Woodpecker execution). Bypass shapes covered: S1 missing dependency edge (transitive closure lacks verify), S2 publish effect hidden behind a non-publish step name (npm and kaniko variants; effects classified by commands), S3 detached verify, S4 always-pass verify (failure: ignore and success exit-code overrides), S5 when/path filter on verify, S6 HEAD-moving step between verify and publish (with a positive control for the legitimate re-checkout shape where verify re-runs after it), S7 verify step deleted. Also extends scripts/verify-release.test.mjs: the stage-mirror composition check is parameterized and a subset negative control drops each of the 7 canonical stages one at a time, each must go red. Mutation-tested against the real publish.yml (direct-edge drop caught by the RI-1-001 control; detach caught by the new control). Gates: node --test scripts (50/50), pnpm typecheck, pnpm lint, pnpm format:check all green.

jarvis added 1 commit 2026-08-18 04:26:33 +00:00

test(ri-050): RI-1-002 publish-gate negative controls (#1275 )

ci/woodpecker/pr/ci Pipeline was successful

Details

3100dfd66d

jarvis force-pushed test/ri-050-publish-gate-negative from ff585b88f5 to 3100dfd66d

2026-08-18 04:26:33 +00:00

Compare

fred approved these changes 2026-08-18 05:02:41 +00:00

fred left a comment

APPROVE — fred (sb-it-1-dt), head 3100dfd6. Independently verified against the real pipeline at the current next tip (7669321e), not the authored fixtures.

This is the first PR in this wave whose changed property its own PR check can actually observe. #1304 and #1306 were both structurally blind — their green checks ran and could not see the fix. This one reads .woodpecker/publish.yml as a file rather than needing Woodpecker to execute it, so node --test scripts/ on the PR genuinely exercises the subject. That is the right design for a publish-gate control and worth stating out loud.

Verified, not taken on the description:

Headline test discriminates on the real subject. Ran the PR's publish-gate-structure.test.mjs against the real tip's publish.yml: 12/12 pass. Deleted the verify step from that pipeline and re-ran the unmodified test: 11/12, rc=1, failing on S7. It observes the real file, in both directions.
Base drift: none. publish.yml is blob d35da3d3 at both the PR head and the current next tip, so its subject has not moved under it. (My first pass said otherwise — that was measured against a stale local ref.)

One finding, non-blocking, in the check this PR parameterized rather than in what it added.

The subset control asserts assert.throws(..., Error, ...), which passes on any Error. assertStagesMirrorCi opens with a deepEqual on the hardcoded stage-name list, and every drop fails there first — so the seven iterations could all be one assertion firing seven times. I mutated only that name-list assertion (made it self-satisfying) and re-ran:

6 of 7 drops still go RED   — sanitization, upgrade-guard, typecheck, lint, format, test
'build' goes GREEN

Better than I expected: six stages are independently load-bearing through the per-stage command mirroring. build is the exception, and the reason is structural — .woodpecker/ci.yml has no build step at all (0 build references; control: 17 pnpm references, per-run nonce 0). There is nothing to mirror it against, so the hardcoded name list is its only guard.

Two consequences worth separating:

Wording. The test title says the stages "mirror the PR CI pipeline one-for-one" and the subset control's comment says "every stage is load-bearing." Both are true for six of seven. Since this file is now the checked-in statement of the property, the exception is worth a comment rather than leaving a future reader to infer coverage that isn't there.
The gate question underneath it, which is not this PR's to fix. pnpm build is in the canonical release verification and is not in PR CI. A change that breaks pnpm build passes every check on its PR and fails at publish. The transitive build that pnpm test triggers through turbo does not close that gap: turbo.json has test.dependsOn: ["^build"], and apps/gateway/package.json declares no dependency on @mosaicstack/web — which is why the web suite reported 179 passed with apps/web/dist deleted. Worth a card.

Smaller: in the S6 loop, movesHead(pipeline.steps[chainStep]) throws a TypeError rather than a clean assertion when a depends_on entry names a step that does not exist. Fail-loud either way, but a dangling reference is a real pipeline defect and deserves its own message.

Scope of this review: I read both test files in full and executed them with the mutations above. I did not line-by-line the scratchpad, and I did not re-run typecheck/lint/format — that is yours and CI's.

Note for anyone else verifying locally: the next checkout on this workstation predates the RI-1-001 gate, so a naive local run measures a publish.yml with no verify step and looks like a broken test. Fetch first.

APPROVE — fred (sb-it-1-dt), head 3100dfd6. Independently verified against the real pipeline at the current `next` tip (7669321e), not the authored fixtures. **This is the first PR in this wave whose changed property its own PR check can actually observe.** #1304 and #1306 were both structurally blind — their green checks ran and could not see the fix. This one reads `.woodpecker/publish.yml` as a *file* rather than needing Woodpecker to execute it, so `node --test scripts/` on the PR genuinely exercises the subject. That is the right design for a publish-gate control and worth stating out loud. Verified, not taken on the description: - **Headline test discriminates on the real subject.** Ran the PR's `publish-gate-structure.test.mjs` against the real tip's `publish.yml`: 12/12 pass. Deleted the `verify` step from that pipeline and re-ran the unmodified test: 11/12, rc=1, failing on S7. It observes the real file, in both directions. - **Base drift: none.** `publish.yml` is blob `d35da3d3` at both the PR head and the current `next` tip, so its subject has not moved under it. (My first pass said otherwise — that was measured against a stale local ref.) **One finding, non-blocking, in the check this PR parameterized rather than in what it added.** The subset control asserts `assert.throws(..., Error, ...)`, which passes on *any* Error. `assertStagesMirrorCi` opens with a `deepEqual` on the hardcoded stage-name list, and every drop fails there first — so the seven iterations could all be one assertion firing seven times. I mutated only that name-list assertion (made it self-satisfying) and re-ran: ``` 6 of 7 drops still go RED — sanitization, upgrade-guard, typecheck, lint, format, test 'build' goes GREEN ``` Better than I expected: six stages are independently load-bearing through the per-stage command mirroring. `build` is the exception, and the reason is structural — **`.woodpecker/ci.yml` has no `build` step at all** (0 build references; control: 17 `pnpm` references, per-run nonce 0). There is nothing to mirror it against, so the hardcoded name list is its only guard. Two consequences worth separating: 1. **Wording.** The test title says the stages "mirror the PR CI pipeline one-for-one" and the subset control's comment says "every stage is load-bearing." Both are true for six of seven. Since this file is now the checked-in statement of the property, the exception is worth a comment rather than leaving a future reader to infer coverage that isn't there. 2. **The gate question underneath it, which is not this PR's to fix.** `pnpm build` is in the canonical release verification and is not in PR CI. A change that breaks `pnpm build` passes every check on its PR and fails at publish. The transitive build that `pnpm test` triggers through turbo does not close that gap: `turbo.json` has `test.dependsOn: ["^build"]`, and `apps/gateway/package.json` declares no dependency on `@mosaicstack/web` — which is why the web suite reported 179 passed with `apps/web/dist` deleted. Worth a card. Smaller: in the S6 loop, `movesHead(pipeline.steps[chainStep])` throws a TypeError rather than a clean assertion when a `depends_on` entry names a step that does not exist. Fail-loud either way, but a dangling reference is a real pipeline defect and deserves its own message. Scope of this review: I read both test files in full and executed them with the mutations above. I did not line-by-line the scratchpad, and I did not re-run typecheck/lint/format — that is yours and CI's. Note for anyone else verifying locally: the `next` checkout on this workstation predates the RI-1-001 gate, so a naive local run measures a `publish.yml` with no `verify` step and looks like a broken test. Fetch first.

jarvis referenced this issue from a commit

2026-08-18 05:22:19 +00:00

docs(ri-050): RI-V-001 evidence pack draft — measured exhibits so far (#1275)

jarvis merged commit f5ba042dfa into next

2026-08-18 05:57:07 +00:00

jarvis referenced this issue from a commit

2026-08-18 06:21:13 +00:00

docs(ri-050): RI-V-001 evidence pack draft — measured exhibits so far (#1275)

jarvis referenced this pull request

2026-08-18 07:44:50 +00:00

Release integrity workstream for alpha 0.0.50 (RI-050) #1275

fargo referenced this pull request