feat(verify-release): wire quality-rails evaluator stage into canonical verification (#1275)
ci/woodpecker/pr/ci Pipeline was successful
ci/woodpecker/pr/ci Pipeline was successful
- new canonical-only quality-rails stage (after build) invokes the evaluator CLI on the repo root — QC-19 monorepo subject — instead of duplicating presence logic; no ci.yml mirror (same shape as the build stage) - parity spec updated: stage-name list, evaluator-delegation assertions, and negative controls for inline duplication / dropped command - quality-rails README points at the probe inventory (input doc)
This commit is contained in:
fargo
@@ -22,7 +22,7 @@
|
||||
| RI-1-002 | done | RI-N1 negative control: checked-in tests proving a broken mandatory check blocks every publish step and that DAG edges cannot be bypassed | #1275 | pi-glm-5.3 | mosaicstack/stack | test/ri-050-publish-gate-negative | RI-1-001 | 12K | |
|
||||
| RI-2-001 | done | RI-N2 (Forge): remove stub-executor false success; `--simulate` typed `simulated` results that satisfy nothing; literal-`true` gates and echo-review replaced with real gates or typed waiting-for-authority | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-forge-fail-closed | RI-0-001 | 20K | Independent review APPROVED 2026-08-17 (Gitea review 172 on PR #1278, head 99b8f6ea; reviewing seat fargo — recorded under shared host principal mos-dt-0, provenance correction posted by fred; wrapper gap filed by fred). Executed at head: forge tests 116/116, lint green, typecheck green after building macp dist (minimal-install artifact, not a defect), workspace typecheck 45/45, no external type consumers of the changed interfaces. CI red = known lane-wide fleet-test failure only, carries no information about this change (fred, log-content analysis, pipelines 2456-2458). Non-blocking finding: README L141-143 + skills/mosaic-forge/SKILL.md document bare forge run/resume, which now fails closed — fast-follow docs touch. Merge queued behind #1270. UPDATE 2026-08-18: #1270 merged; CI GREEN at head 4917df1f via serialized retry (pipeline 2477) - root cause of prior reds was CI-agent contention (web SPA timeouts under concurrent pipelines), superseding the fleet-test-failure theory. |
|
||||
| RI-2-002 | done | RI-N2 (MACP): gate runner fails closed on empty commands, stub executors, and unimplemented CI-provider gates unless explicit simulate; typed capability failures | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-macp-fail-closed | RI-0-001 | 15K | PR #1293 (head 2097379e): CI green (pipeline 2465), independent review APPROVED (Gitea review 173, jarvis seat, 2026-08-17) - macp 109/109 verified at head. Merge queued behind #1276/#1277/#1278. |
|
||||
| RI-3-001 | done | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | PR #1302 (head e06a47fac591): CI green (2484), independent review APPROVED (Gitea review 187, fargo seat, 2026-08-18) — 54 rows / 21 canonical checks / dispositions 43-2-9-0 verified by row-count and code spot-checks. Merged by fargo at pinned head. |
|
||||
| RI-3-001 | done | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | PR #1302 (head e06a47fac591): CI green (2484), independent review APPROVED (Gitea review 187, fargo seat, 2026-08-18) — 54 rows / 21 canonical checks / dispositions 43-2-9-0 verified by row-count and code spot-checks. Merged by fargo at pinned head. |
|
||||
| RI-3-002 | not-started | RI-N4: TS evaluator absorbs effective shell probes; typed results (passed/failed/blocked/error/not-applicable) with versioned digested check definitions; shell commands become thin adapters; contract/parity/negative-control tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-qr-evaluator | RI-3-001 | 30K | |
|
||||
| RI-4-001 | in-progress | RI-N3: one PRD application service — `mission --plan` persists mission↔PRD linkage (ids/versions/selected requirements); `mosaic prdy` routes through the service or becomes a named import/export adapter; Markdown is a labeled generated view; explicit conflict-aware import | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-prd-authority | RI-0-001 | 35K | PR #1294 (head 8d258e1d): CI green (pipeline 2466), independent review APPROVED (Gitea review 174, jarvis seat, 2026-08-17) - prdy 20/20 + command specs 9/9 at head. Merge queued behind #1276/#1277/#1278. |
|
||||
| RI-5-001 | done | RI-N5: typed freshness states (current/stale/partial/unknown/unavailable); no failed-fetch-renders-empty; stale derived verdicts → unknown; mutations disabled when stale; failure-matrix tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-web-stale-safety | RI-0-001 | 25K | |
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# @mosaicstack/quality-rails
|
||||
|
||||
Quality-rails scaffolding and — since RI-3-002 (RI-N4) — the **typed
|
||||
quality-rails evaluator**: the single authoritative producer of check verdicts
|
||||
for the checks it owns.
|
||||
|
||||
## Evaluator (RI-N4)
|
||||
|
||||
Every verdict is typed and fail-closed:
|
||||
|
||||
```
|
||||
{ status: 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable',
|
||||
checkId, checkVersion, subject, reason }
|
||||
```
|
||||
|
||||
Missing implementations, missing inputs, unknown check ids, process errors,
|
||||
timeouts, and malformed probe output can never become `passed` or an
|
||||
unqualified skip — they surface as `blocked`/`error` with a reason (vocabulary
|
||||
mirrors MACP's `GateStatus` discipline).
|
||||
|
||||
- Check definitions live as **data** with a version and a sha256 content
|
||||
digest (`definitionDigest`); every recorded verdict names the definition
|
||||
version that produced it.
|
||||
- Check sets are selected **per subject kind** (`node`, `python`, `rust`,
|
||||
`monorepo`, `unknown`) via the versioned, digested check-set policy — this
|
||||
repository (a `monorepo` subject) does not share the node template's file
|
||||
list.
|
||||
- Shell probes stay **thin adapters**: the TS evaluator invokes them and owns
|
||||
the verdict parsing (e.g. QC-20's planted-commit probe).
|
||||
|
||||
### Owned checks
|
||||
|
||||
| check id | canonical check | mechanism |
|
||||
| --------------------------- | --------------- | ---------------------------------------------------------------------- |
|
||||
| `qc-19-rails-files-present` | QC-19 | typed absorption of the former presence-only `check`/`doctor` loop |
|
||||
| `qc-20-enforcement-verify` | QC-20 | thin shell adapter (framework `verify.sh`); verdict parsing owned here |
|
||||
|
||||
The canonical check ids QC-1..QC-21 and their dispositions are defined in
|
||||
`docs/release-integrity/probe-inventory.md` (the RI-3-001 inventory — the
|
||||
evaluator's input, not its output).
|
||||
|
||||
### Usage
|
||||
|
||||
```sh
|
||||
# CLI (same typed report as the programmatic API)
|
||||
node dist/cli.js quality-rails evaluate --project <path> [--probe-path <verify.sh>] [--json]
|
||||
node dist/cli.js quality-rails check --project <path> [--json] # QC-19 only, fail-closed exit
|
||||
```
|
||||
|
||||
```ts
|
||||
import { evaluateSubject } from '@mosaicstack/quality-rails';
|
||||
|
||||
const report = await evaluateSubject({ subjectPath: '/path/to/project' });
|
||||
// report.state: 'passed' | 'failed' | 'blocked' | 'error'
|
||||
```
|
||||
|
||||
`pnpm verify:release` invokes this evaluator as its `quality-rails` stage
|
||||
(canonical-only stage, QC-19 on the monorepo subject).
|
||||
|
||||
## Scaffolding (pre-existing)
|
||||
|
||||
`init` scaffolds rails files per detected kind/profile; `doctor` is advisory
|
||||
and reports typed states.
|
||||
@@ -22,6 +22,12 @@
|
||||
// format | format | pnpm format:check
|
||||
// test | test | pnpm test
|
||||
// build | publish.yml build | pnpm build
|
||||
// quality-rails | (canonical-only) | the TS quality-rails evaluator
|
||||
// | | (RI-N4, QC-19 monorepo subject). Like
|
||||
// | | `build`, this stage has no ci.yml
|
||||
// | | mirror; it is implemented by
|
||||
// | | importing the evaluator CLI rather
|
||||
// | | than duplicating its presence logic.
|
||||
//
|
||||
// Caller-provided prerequisites (kept at the pipeline level — see the comments
|
||||
// in .woodpecker/ci.yml): `bash` + `rsync` for the guard stages, `openssl` and
|
||||
@@ -90,6 +96,15 @@ export const STAGES = [
|
||||
name: 'build',
|
||||
commands: ['pnpm build'],
|
||||
},
|
||||
{
|
||||
// RI-N4 (QC-19, card RI-3-002): the typed quality-rails evaluator, invoked
|
||||
// as the implementation of the check it owns instead of a duplicated
|
||||
// presence loop here. Canonical-only stage (no ci.yml mirror — same shape
|
||||
// as `build`); runs AFTER build so the evaluator's dist/ exists. Subject
|
||||
// is this repository (`.` → monorepo subject kind, per-subject check set).
|
||||
name: 'quality-rails',
|
||||
commands: ['node packages/quality-rails/dist/cli.js quality-rails evaluate --project .'],
|
||||
},
|
||||
];
|
||||
|
||||
export function stageByName(name) {
|
||||
|
||||
@@ -4,7 +4,7 @@ import { createRequire } from 'node:module';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
import { STAGES } from './verify-release.mjs';
|
||||
import { STAGES, stageByName } from './verify-release.mjs';
|
||||
|
||||
// SDLC-D-034 checkout invariant: publication in .woodpecker/publish.yml is
|
||||
// bound to exact-commit terminal verification. This suite parses the real
|
||||
@@ -229,10 +229,21 @@ steps:
|
||||
function assertStagesMirrorCi(stages, ci) {
|
||||
const canonical = Object.fromEntries(stages.map((stage) => [stage.name, stage.commands]));
|
||||
|
||||
// The complete mandatory set, in gate order.
|
||||
// The complete mandatory set, in gate order. `quality-rails` is a
|
||||
// canonical-only stage (RI-N4, QC-19): like `build`, it has no ci.yml
|
||||
// mirror to match — its contract is asserted separately below.
|
||||
assert.deepEqual(
|
||||
stages.map((stage) => stage.name),
|
||||
['sanitization', 'upgrade-guard', 'typecheck', 'lint', 'format', 'test', 'build'],
|
||||
[
|
||||
'sanitization',
|
||||
'upgrade-guard',
|
||||
'typecheck',
|
||||
'lint',
|
||||
'format',
|
||||
'test',
|
||||
'build',
|
||||
'quality-rails',
|
||||
],
|
||||
);
|
||||
|
||||
// Guard stages: ci.yml commands minus its `apk add` environment prep must be
|
||||
@@ -301,3 +312,38 @@ test('the root package.json exposes verify:release as the canonical command', as
|
||||
const packageJson = JSON.parse(await readFile(path.join(process.cwd(), 'package.json'), 'utf8'));
|
||||
assert.match(packageJson.scripts['verify:release'], /scripts\/verify-release\.mjs/);
|
||||
});
|
||||
|
||||
// RI-N4 (card RI-3-002): the `quality-rails` stage must route through the TS
|
||||
// evaluator instead of duplicating its presence logic inline. The evaluator
|
||||
// owns QC-19; this file keeps that delegation honest.
|
||||
function assertEvaluatorStage(stage) {
|
||||
assert.ok(stage, 'canonical stages must include a quality-rails stage');
|
||||
assert.ok(Array.isArray(stage.commands) && stage.commands.length > 0);
|
||||
for (const command of stage.commands) {
|
||||
assert.match(
|
||||
command,
|
||||
/packages\/quality-rails\/dist\/cli\.js.*quality-rails evaluate/,
|
||||
`quality-rails stage command must invoke the evaluator CLI, got: '${command}'`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
test('the quality-rails stage invokes the evaluator rather than duplicating its logic', () => {
|
||||
assertEvaluatorStage(stageByName('quality-rails'));
|
||||
});
|
||||
|
||||
test('a quality-rails stage that re-implements presence logic inline fails the checker', () => {
|
||||
// Negative control: replacing the evaluator invocation with an inline
|
||||
// `test -f` presence loop is exactly the duplication RI-N4 forbids — the
|
||||
// checker must go red on it.
|
||||
const duplicated = {
|
||||
name: 'quality-rails',
|
||||
commands: ['test -f .husky/pre-commit && test -f .husky/pre-push'],
|
||||
};
|
||||
assert.throws(() => assertEvaluatorStage(duplicated), /must invoke the evaluator CLI/);
|
||||
});
|
||||
|
||||
test('a quality-rails stage that silently drops the evaluator command fails the checker', () => {
|
||||
const empty = { name: 'quality-rails', commands: [] };
|
||||
assert.throws(() => assertEvaluatorStage(empty), /commands/);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user