Compare commits

...
Author SHA1 Message Date
fargo 68279d61a1 feat(verify-release): wire quality-rails evaluator stage into canonical verification (#1275)
ci/woodpecker/pr/ci Pipeline was successful
- new canonical-only quality-rails stage (after build) invokes the evaluator
  CLI on the repo root — QC-19 monorepo subject — instead of duplicating
  presence logic; no ci.yml mirror (same shape as the build stage)
- parity spec updated: stage-name list, evaluator-delegation assertions, and
  negative controls for inline duplication / dropped command
- quality-rails README points at the probe inventory (input doc)
2026-08-18 12:35:03 -05:00
fargo 771127d3cd test(quality-rails): evaluator contract, parity oracle, and negative-control specs (#1275)
- CLI evaluate/check --json vs programmatic evaluateSubject: same subject,
  same typed report
- QC-19 parity vs a verbatim copy of the absorbed presence loop (positive and
  negative fixtures, all scaffold kinds) and QC-20 parity vs the real
  framework verify.sh output contract
- negative controls: unknown check id, absent subject, missing probePath,
  spawn error, timeout, nonzero/unexpected exit, malformed output, throwing
  check, unqualified skip — all never passed
2026-08-18 11:17:25 -05:00
fargo ae95e7b853 feat(quality-rails): absorb QC-19 presence loop into evaluator; add typed evaluate/check/doctor (#1275)
- check (QC-19) is now implemented by the evaluator's typed
  qc-19-rails-files-present definition; keeps fail-closed exit and gains --json
- doctor stays advisory but reports typed states
- new evaluate subcommand is the canonical CLI entry point (--probe-path feeds
  QC-20; shell probes remain thin adapters with TS-owned verdict parsing)
2026-08-18 11:17:23 -05:00
fargo 367cb27591 feat(quality-rails): typed evaluator core — statuses, digested definitions, per-subject sets (#1275) 2026-08-18 11:17:15 -05:00
fargo 18905d69e6 docs(ri-050): RI-3-001 done — review 187 + merge evidence recorded (#1275) 2026-08-18 11:01:20 -05:00
13 changed files with 1661 additions and 58 deletions
+1 -1
View File
@@ -22,7 +22,7 @@
| RI-1-002 | done | RI-N1 negative control: checked-in tests proving a broken mandatory check blocks every publish step and that DAG edges cannot be bypassed | #1275 | pi-glm-5.3 | mosaicstack/stack | test/ri-050-publish-gate-negative | RI-1-001 | 12K | |
| RI-2-001 | done | RI-N2 (Forge): remove stub-executor false success; `--simulate` typed `simulated` results that satisfy nothing; literal-`true` gates and echo-review replaced with real gates or typed waiting-for-authority | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-forge-fail-closed | RI-0-001 | 20K | Independent review APPROVED 2026-08-17 (Gitea review 172 on PR #1278, head 99b8f6ea; reviewing seat fargo — recorded under shared host principal mos-dt-0, provenance correction posted by fred; wrapper gap filed by fred). Executed at head: forge tests 116/116, lint green, typecheck green after building macp dist (minimal-install artifact, not a defect), workspace typecheck 45/45, no external type consumers of the changed interfaces. CI red = known lane-wide fleet-test failure only, carries no information about this change (fred, log-content analysis, pipelines 2456-2458). Non-blocking finding: README L141-143 + skills/mosaic-forge/SKILL.md document bare forge run/resume, which now fails closed — fast-follow docs touch. Merge queued behind #1270. UPDATE 2026-08-18: #1270 merged; CI GREEN at head 4917df1f via serialized retry (pipeline 2477) - root cause of prior reds was CI-agent contention (web SPA timeouts under concurrent pipelines), superseding the fleet-test-failure theory. |
| RI-2-002 | done | RI-N2 (MACP): gate runner fails closed on empty commands, stub executors, and unimplemented CI-provider gates unless explicit simulate; typed capability failures | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-macp-fail-closed | RI-0-001 | 15K | PR #1293 (head 2097379e): CI green (pipeline 2465), independent review APPROVED (Gitea review 173, jarvis seat, 2026-08-17) - macp 109/109 verified at head. Merge queued behind #1276/#1277/#1278. |
| RI-3-001 | in-progress | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | |
| RI-3-001 | done | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | PR #1302 (head e06a47fac591): CI green (2484), independent review APPROVED (Gitea review 187, fargo seat, 2026-08-18) — 54 rows / 21 canonical checks / dispositions 43-2-9-0 verified by row-count and code spot-checks. Merged by fargo at pinned head. |
| RI-3-002 | not-started | RI-N4: TS evaluator absorbs effective shell probes; typed results (passed/failed/blocked/error/not-applicable) with versioned digested check definitions; shell commands become thin adapters; contract/parity/negative-control tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-qr-evaluator | RI-3-001 | 30K | |
| RI-4-001 | in-progress | RI-N3: one PRD application service — `mission --plan` persists mission↔PRD linkage (ids/versions/selected requirements); `mosaic prdy` routes through the service or becomes a named import/export adapter; Markdown is a labeled generated view; explicit conflict-aware import | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-prd-authority | RI-0-001 | 35K | PR #1294 (head 8d258e1d): CI green (pipeline 2466), independent review APPROVED (Gitea review 174, jarvis seat, 2026-08-17) - prdy 20/20 + command specs 9/9 at head. Merge queued behind #1276/#1277/#1278. |
| RI-5-001 | done | RI-N5: typed freshness states (current/stale/partial/unknown/unavailable); no failed-fetch-renders-empty; stale derived verdicts → unknown; mutations disabled when stale; failure-matrix tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-web-stale-safety | RI-0-001 | 25K | |
+63
View File
@@ -0,0 +1,63 @@
# @mosaicstack/quality-rails
Quality-rails scaffolding and — since RI-3-002 (RI-N4) — the **typed
quality-rails evaluator**: the single authoritative producer of check verdicts
for the checks it owns.
## Evaluator (RI-N4)
Every verdict is typed and fail-closed:
```
{ status: 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable',
checkId, checkVersion, subject, reason }
```
Missing implementations, missing inputs, unknown check ids, process errors,
timeouts, and malformed probe output can never become `passed` or an
unqualified skip — they surface as `blocked`/`error` with a reason (vocabulary
mirrors MACP's `GateStatus` discipline).
- Check definitions live as **data** with a version and a sha256 content
digest (`definitionDigest`); every recorded verdict names the definition
version that produced it.
- Check sets are selected **per subject kind** (`node`, `python`, `rust`,
`monorepo`, `unknown`) via the versioned, digested check-set policy — this
repository (a `monorepo` subject) does not share the node template's file
list.
- Shell probes stay **thin adapters**: the TS evaluator invokes them and owns
the verdict parsing (e.g. QC-20's planted-commit probe).
### Owned checks
| check id | canonical check | mechanism |
| --------------------------- | --------------- | ---------------------------------------------------------------------- |
| `qc-19-rails-files-present` | QC-19 | typed absorption of the former presence-only `check`/`doctor` loop |
| `qc-20-enforcement-verify` | QC-20 | thin shell adapter (framework `verify.sh`); verdict parsing owned here |
The canonical check ids QC-1..QC-21 and their dispositions are defined in
`docs/release-integrity/probe-inventory.md` (the RI-3-001 inventory — the
evaluator's input, not its output).
### Usage
```sh
# CLI (same typed report as the programmatic API)
node dist/cli.js quality-rails evaluate --project <path> [--probe-path <verify.sh>] [--json]
node dist/cli.js quality-rails check --project <path> [--json] # QC-19 only, fail-closed exit
```
```ts
import { evaluateSubject } from '@mosaicstack/quality-rails';
const report = await evaluateSubject({ subjectPath: '/path/to/project' });
// report.state: 'passed' | 'failed' | 'blocked' | 'error'
```
`pnpm verify:release` invokes this evaluator as its `quality-rails` stage
(canonical-only stage, QC-19 on the monorepo subject).
## Scaffolding (pre-existing)
`init` scaffolds rails files per detected kind/profile; `doctor` is advisory
and reports typed states.
@@ -0,0 +1,194 @@
import { mkdir, mkdtemp, writeFile, chmod } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { createQualityRailsCli } from './cli.js';
import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js';
import { evaluateSubject } from './evaluator/runner.js';
import type { EvaluationReport } from './evaluator/types.js';
// CLI ↔ programmatic contract (RI-3-002): the same subject must produce the
// same typed verdicts through every entry point the card adds — the
// `evaluate`/`check` CLI surfaces and the `evaluateSubject` API.
async function makeTempDir(): Promise<string> {
return mkdtemp(join(tmpdir(), 'qr-cli-'));
}
async function scaffoldNodeFixture(skip: string[] = []): Promise<string> {
const dir = await makeTempDir();
await writeFile(join(dir, 'package.json'), '{}\n', 'utf8');
for (const relativePath of [
'.eslintrc',
'biome.json',
'.githooks/pre-commit',
'PR-CHECKLIST.md',
]) {
if (skip.includes(relativePath)) continue;
await mkdir(join(dir, relativePath, '..'), { recursive: true });
await writeFile(join(dir, relativePath), 'fixture\n', 'utf8');
}
return dir;
}
async function makePassingProbe(dir: string): Promise<string> {
const scriptPath = join(dir, 'probe-pass.sh');
await writeFile(
scriptPath,
[
'#!/bin/bash',
'echo "✅ PASS: Type errors blocked"',
'echo "✅ PASS: Lint errors blocked"',
'echo "Verification Summary"',
'exit 0',
].join('\n') + '\n',
'utf8',
);
await chmod(scriptPath, 0o755);
return scriptPath;
}
describe('CLI entry points vs the programmatic evaluator', () => {
let logSpy: ReturnType<typeof vi.spyOn>;
let previousExitCode: string | number | undefined;
beforeEach(() => {
logSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
previousExitCode = process.exitCode ?? undefined;
});
afterEach(() => {
logSpy.mockRestore();
process.exitCode = previousExitCode;
});
it('evaluate --json produces the SAME typed report as evaluateSubject (full check set + probe)', async () => {
const dir = await scaffoldNodeFixture();
const probePath = await makePassingProbe(dir);
const programmatic = await evaluateSubject({
subjectPath: dir,
inputs: { 'qc-20-enforcement-verify': { probePath } },
});
const program = createQualityRailsCli();
await program.parseAsync([
'node',
'cli.js',
'quality-rails',
'evaluate',
'--project',
dir,
'--probe-path',
probePath,
'--json',
]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
const cliReport = JSON.parse(printed) as EvaluationReport;
expect(cliReport).toEqual(programmatic);
expect(cliReport.state).toBe('passed');
expect(process.exitCode).toBe(0);
});
it('check --json produces the SAME QC-19 verdict as evaluateSubject (absorbed loop)', async () => {
const dir = await scaffoldNodeFixture(['biome.json', '.githooks/pre-commit']);
const programmatic = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
});
expect(programmatic.state).toBe('failed');
const program = createQualityRailsCli();
await program.parseAsync([
'node',
'cli.js',
'quality-rails',
'check',
'--project',
dir,
'--json',
]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
const cliReport = JSON.parse(printed) as EvaluationReport;
expect(cliReport).toEqual(programmatic);
expect(process.exitCode).toBe(1);
});
it('check on a complete subject exits 0 with a passed verdict', async () => {
const dir = await scaffoldNodeFixture();
const program = createQualityRailsCli();
await program.parseAsync([
'node',
'cli.js',
'quality-rails',
'check',
'--project',
dir,
'--json',
]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
const cliReport = JSON.parse(printed) as EvaluationReport;
expect(cliReport.state).toBe('passed');
expect(process.exitCode).toBe(0);
});
it('evaluate with an unknown check id exits 1 and reports error, never passed', async () => {
const dir = await scaffoldNodeFixture();
const program = createQualityRailsCli();
await program.parseAsync([
'node',
'cli.js',
'quality-rails',
'evaluate',
'--project',
dir,
'--check',
'qc-99-bogus',
'--json',
]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
const cliReport = JSON.parse(printed) as EvaluationReport;
expect(cliReport.results).toHaveLength(1);
const first = cliReport.results[0];
expect(first?.status).toBe('error');
expect(first?.reason).toContain('unknown check id');
expect(process.exitCode).toBe(1);
});
it('evaluate on a scaffold subject without --probe-path stays fail-closed (blocked, exit 1)', async () => {
const dir = await scaffoldNodeFixture();
const program = createQualityRailsCli();
await program.parseAsync([
'node',
'cli.js',
'quality-rails',
'evaluate',
'--project',
dir,
'--json',
]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
const cliReport = JSON.parse(printed) as EvaluationReport;
const qc20 = cliReport.results.find((r) => r.checkId === 'qc-20-enforcement-verify');
expect(qc20).toBeDefined();
expect(qc20?.status).toBe('blocked');
expect(qc20?.reason).toContain('probePath');
expect(cliReport.state).toBe('blocked');
expect(process.exitCode).toBe(1);
});
it('doctor stays advisory (no nonzero exit) but reports TYPED states, including blocked', async () => {
const dir = await scaffoldNodeFixture();
const program = createQualityRailsCli();
await program.parseAsync(['node', 'cli.js', 'quality-rails', 'doctor', '--project', dir]);
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
expect(printed).toContain('blocked: qc-20-enforcement-verify');
expect(process.exitCode ?? 0).toBe(0);
});
});
+81 -54
View File
@@ -1,5 +1,3 @@
import { constants } from 'node:fs';
import { access } from 'node:fs/promises';
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
@@ -8,18 +6,12 @@ import { Command } from 'commander';
import { detectProjectKind } from './detect.js';
import { scaffoldQualityRails } from './scaffolder.js';
import type { ProjectKind, QualityProfile, RailsConfig } from './types.js';
import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js';
import { evaluateSubject } from './evaluator/runner.js';
import type { EvaluationReport } from './evaluator/types.js';
const VALID_PROFILES: readonly QualityProfile[] = ['strict', 'standard', 'minimal'];
async function fileExists(filePath: string): Promise<boolean> {
try {
await access(filePath, constants.F_OK);
return true;
} catch {
return false;
}
}
function parseProfile(rawProfile: string): QualityProfile {
if (VALID_PROFILES.includes(rawProfile as QualityProfile)) {
return rawProfile as QualityProfile;
@@ -59,22 +51,6 @@ function defaultFormatters(kind: ProjectKind): string[] {
return [];
}
function expectedFilesForKind(kind: ProjectKind): string[] {
if (kind === 'node') {
return ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
}
if (kind === 'python') {
return ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
}
if (kind === 'rust') {
return ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
}
return ['.githooks/pre-commit', 'PR-CHECKLIST.md'];
}
function printScaffoldResult(
config: RailsConfig,
filesWritten: string[],
@@ -106,6 +82,20 @@ function printScaffoldResult(
}
}
function printEvaluationReport(report: EvaluationReport): void {
console.log(
`[quality-rails] evaluation for ${report.subject.path} (kind=${report.subject.kind}, check-set v${report.checkSetVersion})`,
);
for (const result of report.results) {
const reason = result.reason === undefined ? '' : `${result.reason}`;
const digest = report.definitionDigests[result.checkId] ?? 'no digest';
console.log(
` - ${result.status}: ${result.checkId} (v${result.checkVersion} [${digest}])${reason}`,
);
}
console.log(`[quality-rails] aggregate: ${report.state}`);
}
/**
* Register quality-rails subcommands on an existing Commander program.
* This avoids cross-package Commander version mismatches by using the
@@ -148,56 +138,93 @@ function buildQualityRailsCommand(qualityRails: Command): void {
printScaffoldResult(config, result.filesWritten, result.warnings, result.commandsToRun);
});
// `check` (QC-19) is ABSORBED by the RI-N4 evaluator: the presence loop
// that lived here is now the versioned, digested, typed check definition
// `qc-19-rails-files-present`. The CLI keeps its human surface (missing
// files listed, exit 1) and gains `--json` for the typed verdicts. Exit
// code is fail-closed: any non-green aggregate (failed/blocked/error) is 1.
qualityRails
.command('check')
.requiredOption('--project <path>', 'Project path')
.action(async (options: { project: string }) => {
.option('--json', 'print the typed evaluation report as JSON')
.action(async (options: { project: string; json?: boolean }) => {
const projectPath = resolve(options.project);
const kind = await detectProjectKind(projectPath);
const expected = expectedFilesForKind(kind);
const missing: string[] = [];
const report = await evaluateSubject({
subjectPath: projectPath,
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
});
for (const relativePath of expected) {
const exists = await fileExists(resolve(projectPath, relativePath));
if (!exists) {
missing.push(relativePath);
}
if (options.json) {
console.log(JSON.stringify(report));
} else {
printEvaluationReport(report);
}
if (missing.length > 0) {
console.error('[quality-rails] missing files:');
for (const relativePath of missing) {
console.error(` - ${relativePath}`);
}
process.exitCode = 1;
return;
}
console.log(`[quality-rails] all expected files present for ${kind} project`);
process.exitCode = report.state === 'passed' ? 0 : 1;
});
// `doctor` (QC-19) stays advisory (documented contract: a doctor that
// cannot fail), but now reports TYPED states — a blocked or failing rail is
// visible instead of silently printed as `ok`/`missing`.
qualityRails
.command('doctor')
.requiredOption('--project <path>', 'Project path')
.action(async (options: { project: string }) => {
const projectPath = resolve(options.project);
const kind = await detectProjectKind(projectPath);
const expected = expectedFilesForKind(kind);
const report = await evaluateSubject({ subjectPath: projectPath });
console.log(`[quality-rails] doctor for ${projectPath}`);
console.log(`detected project kind: ${kind}`);
for (const relativePath of expected) {
const exists = await fileExists(resolve(projectPath, relativePath));
console.log(` - ${exists ? 'ok' : 'missing'}: ${relativePath}`);
console.log(`detected project kind: ${report.subject.kind}`);
for (const result of report.results) {
const reason = result.reason === undefined ? '' : `${result.reason}`;
console.log(` - ${result.status}: ${result.checkId}${reason}`);
}
if (kind === 'unknown') {
if (report.subject.kind === 'unknown') {
console.log(
'recommendation: add package.json, pyproject.toml, or Cargo.toml for better defaults.',
);
}
});
// `evaluate` is the canonical RI-N4 evaluator entry point: typed verdicts
// for the subject's full per-kind check set, same results as the
// programmatic API (evaluateSubject).
qualityRails
.command('evaluate')
.description('Run the typed quality-rails evaluator against a subject project')
.requiredOption('--project <path>', 'Project path')
.option('--check <id...>', 'restrict evaluation to these check ids')
.option(
'--probe-path <path>',
'path to the QC-20 behavioral probe script (framework verify.sh)',
)
.option('--json', 'print the typed evaluation report as JSON')
.action(
async (options: {
project: string;
check?: string[];
probePath?: string;
json?: boolean;
}) => {
const projectPath = resolve(options.project);
const report = await evaluateSubject({
subjectPath: projectPath,
checkIds: options.check,
inputs: options.probePath
? { 'qc-20-enforcement-verify': { probePath: options.probePath } }
: undefined,
});
if (options.json) {
console.log(JSON.stringify(report));
} else {
printEvaluationReport(report);
}
process.exitCode = report.state === 'passed' ? 0 : 1;
},
);
}
export async function runQualityRailsCli(argv: string[] = process.argv): Promise<void> {
@@ -0,0 +1,67 @@
import { spawn } from 'node:child_process';
import type { ChildProcess } from 'node:child_process';
import type { AdapterOutcome, AdapterRequest, ProcessAdapter } from './types.js';
/**
* Default thin process adapter (spawn-based). Runs a command to completion with
* a hard timeout and reports exit code + captured output — it owns NO verdict
* logic. Interpreting the outcome is always the check implementation's job.
*/
export function createSpawnProcessAdapter(): ProcessAdapter {
return {
run(request: AdapterRequest): Promise<AdapterOutcome> {
return new Promise((resolve) => {
let child: ChildProcess;
try {
child = spawn(request.file, request.args, {
cwd: request.cwd,
stdio: ['ignore', 'pipe', 'pipe'],
});
} catch (error) {
resolve({
ok: false,
kind: 'spawn-error',
message: error instanceof Error ? error.message : String(error),
});
return;
}
let stdout = '';
let stderr = '';
let settled = false;
const timer = setTimeout(() => {
if (settled) return;
settled = true;
child.kill('SIGKILL');
resolve({
ok: false,
kind: 'timeout',
message: `process timed out after ${request.timeoutMs}ms: ${request.file}`,
});
}, request.timeoutMs);
const settle = (outcome: AdapterOutcome): void => {
if (settled) return;
settled = true;
clearTimeout(timer);
resolve(outcome);
};
child.stdout?.on('data', (chunk: Buffer) => {
stdout += chunk.toString('utf8');
});
child.stderr?.on('data', (chunk: Buffer) => {
stderr += chunk.toString('utf8');
});
child.on('error', (error: Error) => {
settle({ ok: false, kind: 'spawn-error', message: error.message });
});
child.on('close', (code: number | null) => {
settle({ ok: true, exitCode: code, stdout, stderr });
});
});
},
};
}
@@ -0,0 +1,258 @@
import { constants } from 'node:fs';
import { access } from 'node:fs/promises';
import { isAbsolute, resolve } from 'node:path';
import { digestOfPolicy, digestOfSpec } from './digest.js';
import type {
CheckContext,
CheckDefinition,
CheckDefinitionSpec,
CheckOutcome,
CheckSetPolicy,
CheckSetPolicySpec,
SubjectKind,
} from './types.js';
// Check definitions for the RI-N4 evaluator (card RI-3-002). Each definition is
// DATA with a version and a content digest (see digest.ts); the executable
// half is attached via defineCheck. Check-set SELECTION is per subject kind
// (probe-inventory gap 7): this monorepo does not match the node template's
// file list, so the QC-19 definition carries a distinct file set for the
// `monorepo` subject kind and the policy selects checks per kind.
export function defineCheck(
spec: CheckDefinitionSpec,
evaluate: (ctx: CheckContext) => Promise<CheckOutcome>,
): CheckDefinition {
return { ...spec, definitionDigest: digestOfSpec(spec), evaluate };
}
async function fileExists(filePath: string): Promise<boolean> {
try {
await access(filePath, constants.F_OK);
return true;
} catch {
return false;
}
}
// ─── QC-19: downstream rails presence ────────────────────────────────────────
//
// Typed absorption of the former presence-only `quality-rails check` loop in
// cli.ts. The scaffold-kind file lists below are carried over VERBATIM so the
// evaluator's typed verdicts are parity-equivalent with the presence loop on
// the same fixture; the `monorepo` list is new (per-subject check sets).
const qc19Spec: CheckDefinitionSpec = {
id: 'qc-19-rails-files-present',
version: '1.0.0',
canonicalCheck: 'QC-19',
description:
'The subject still carries its quality-rails files. Typed absorption of the former presence-only check loop; presence is necessary, not sufficient (RI-N4).',
appliesTo: ['node', 'python', 'rust', 'monorepo', 'unknown'],
params: {
expectedFilesByKind: {
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
monorepo: [
'.husky/pre-commit',
'.husky/pre-push',
'eslint.config.mjs',
'.prettierrc',
'.lintstagedrc',
],
unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'],
},
},
};
async function evaluateQc19(ctx: CheckContext): Promise<CheckOutcome> {
const byKind = ctx.params['expectedFilesByKind'] as Record<string, readonly string[]> | undefined;
if (byKind === undefined) {
return { status: 'error', reason: 'definition params missing expectedFilesByKind' };
}
const expected = byKind[ctx.subject.kind];
if (expected === undefined) {
// Fail-closed: an undefined file set for a declared subject kind is a
// definition gap, never a green outcome.
return {
status: 'blocked',
reason: `no expected-file set defined for subject kind '${ctx.subject.kind}'`,
};
}
const missing: string[] = [];
for (const relativePath of expected) {
if (!(await fileExists(resolve(ctx.subject.path, relativePath)))) {
missing.push(relativePath);
}
}
if (missing.length > 0) {
return {
status: 'failed',
reason: `missing rails files (${ctx.subject.kind}): ${missing.join(', ')}`,
};
}
return { status: 'passed' };
}
// ─── QC-20: downstream enforcement verification (behavioral probe) ──────────
//
// The planted-commit behavioral probe (framework tools/quality/scripts/verify.sh)
// stays a THIN SHELL ADAPTER: the TS evaluator invokes it and OWNS the verdict
// parsing (RI-N4: grep-on-output verdict logic moves into the typed evaluator).
// Probe contract (verify.sh): exit 0 ⇔ every sub-probe passed, exit 1 ⇔ at
// least one sub-probe failed; sub-probe verdicts appear as `PASS:` / `FAIL:`
// marker lines and the script always prints a `Verification Summary` section.
// Any deviation from that contract (other exit codes, unparseable output,
// missing probe, process failure, timeout) is `error`/`blocked` — never
// `passed`.
const qc20Spec: CheckDefinitionSpec = {
id: 'qc-20-enforcement-verify',
version: '1.0.0',
canonicalCheck: 'QC-20',
description:
'The behavioral planted-commit probe runs against the subject and every sub-probe blocks as intended. The shell probe is a thin adapter; verdict parsing is owned by this evaluator.',
appliesTo: ['node', 'python', 'rust', 'unknown'],
params: {
command: 'bash',
timeoutMs: 120_000,
passMarker: 'PASS:',
failMarker: 'FAIL:',
summaryMarker: 'Verification Summary',
},
};
function linesWith(text: string, marker: string): string[] {
return text
.split('\n')
.map((line) => line.trim())
.filter((line) => line.includes(marker));
}
async function evaluateQc20(ctx: CheckContext): Promise<CheckOutcome> {
const rawProbePath = ctx.inputs['probePath'];
if (typeof rawProbePath !== 'string' || rawProbePath.trim().length === 0) {
return {
status: 'blocked',
reason:
'missing input: probePath — the behavioral probe script must be provided (e.g. the framework verify.sh)',
};
}
const probePath = isAbsolute(rawProbePath)
? rawProbePath
: resolve(ctx.subject.path, rawProbePath);
if (!(await fileExists(probePath))) {
return { status: 'blocked', reason: `probe script not found: ${probePath}` };
}
const command = typeof ctx.params['command'] === 'string' ? ctx.params['command'] : 'bash';
const timeoutMs = typeof ctx.params['timeoutMs'] === 'number' ? ctx.params['timeoutMs'] : 120_000;
const passMarker =
typeof ctx.params['passMarker'] === 'string' ? ctx.params['passMarker'] : 'PASS:';
const failMarker =
typeof ctx.params['failMarker'] === 'string' ? ctx.params['failMarker'] : 'FAIL:';
const summaryMarker =
typeof ctx.params['summaryMarker'] === 'string'
? ctx.params['summaryMarker']
: 'Verification Summary';
const outcome = await ctx.adapter.run({
file: command,
args: [probePath],
cwd: ctx.subject.path,
timeoutMs,
});
if (!outcome.ok) {
// Process error or timeout: the probe never produced a trustworthy result.
return {
status: 'error',
reason: `probe process ${outcome.kind}: ${outcome.message}`,
};
}
const output = `${outcome.stdout}\n${outcome.stderr}`;
const failLines = linesWith(output, failMarker);
const passLines = linesWith(output, passMarker);
if (outcome.exitCode === 0) {
// A green exit must be corroborated by a parseable green transcript:
// at least one pass marker, no fail markers, and the summary section.
if (passLines.length > 0 && failLines.length === 0 && output.includes(summaryMarker)) {
return { status: 'passed' };
}
return {
status: 'error',
reason: `malformed probe output: exit 0 without a parseable pass transcript (${passLines.length} pass markers, ${failLines.length} fail markers, summary ${output.includes(summaryMarker) ? 'present' : 'absent'})`,
};
}
if (outcome.exitCode === 1) {
if (failLines.length === 0) {
return {
status: 'error',
reason: 'malformed probe output: exit 1 without parseable FAIL markers',
};
}
return {
status: 'failed',
reason: `enforcement probe reported ${failLines.length} failing sub-probe(s): ${failLines.join(' | ')}`,
};
}
return {
status: 'error',
reason: `probe exited with unexpected code ${String(outcome.exitCode)} — outcome not interpretable`,
};
}
// ─── Per-subject check-set policy ───────────────────────────────────────────
//
// Gap 7 of the probe inventory: check sets must be selected per subject, not
// one global list. Downstream scaffold kinds get the presence check plus the
// behavioral probe (QC-20 blocks until a probePath input is provided — an
// unverified subject can never evaluate green). The monorepo subject is this
// repository itself: its rails are the husky hooks + shared lint/format
// configs, covered by QC-19; the downstream planted-commit probe does not
// apply to it (this repo's own commit gates are QC-13/QC-14, outside this
// evaluator's owned checks).
const checkSetPolicySpec: CheckSetPolicySpec = {
version: '1.0.0',
byKind: {
node: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
python: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
rust: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
unknown: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
monorepo: ['qc-19-rails-files-present'],
},
};
export const CHECK_SET_POLICY: CheckSetPolicy = {
...checkSetPolicySpec,
policyDigest: digestOfPolicy(checkSetPolicySpec),
};
export const QC_19_RAILS_FILES_PRESENT = defineCheck(qc19Spec, evaluateQc19);
export const QC_20_ENFORCEMENT_VERIFY = defineCheck(qc20Spec, evaluateQc20);
/** Built-in check definitions, keyed by id. */
export function builtInDefinitions(): CheckDefinition[] {
return [QC_19_RAILS_FILES_PRESENT, QC_20_ENFORCEMENT_VERIFY];
}
export function checkSetForKind(
kind: SubjectKind,
policy: CheckSetPolicy = CHECK_SET_POLICY,
): readonly string[] {
const selected = policy.byKind[kind];
if (selected === undefined) {
// Fail-closed selection: an unknown kind yields an EMPTY set only to the
// caller; the runner treats an empty result list as `blocked`, never green.
return [];
}
return selected;
}
@@ -0,0 +1,45 @@
import { createHash } from 'node:crypto';
import type { CheckDefinitionSpec, CheckSetPolicySpec } from './types.js';
// Deterministic JSON: object keys sorted at every level so two specs with the
// same content always produce the same bytes (and thus the same digest).
export function canonicalJson(value: unknown): string {
if (value === null || typeof value !== 'object') {
return JSON.stringify(value);
}
if (Array.isArray(value)) {
return `[${value.map((entry) => canonicalJson(entry)).join(',')}]`;
}
const record = value as Record<string, unknown>;
const keys = Object.keys(record).sort();
return `{${keys.map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`).join(',')}}`;
}
/** sha256 over the canonical JSON of `value`. */
export function digestContent(value: unknown): string {
return createHash('sha256').update(canonicalJson(value), 'utf8').digest('hex');
}
/**
* Content digest of a check definition: covers the declarative spec (id,
* version, canonical check, description, applicability, params) — everything a
* reviewer reasons about — while excluding the executable function object.
* Changing any covered field changes the digest, so a recorded digest always
* identifies exactly which definition content produced a verdict.
*/
export function digestOfSpec(spec: CheckDefinitionSpec): string {
return digestContent({
id: spec.id,
version: spec.version,
canonicalCheck: spec.canonicalCheck,
description: spec.description,
appliesTo: spec.appliesTo,
params: spec.params,
});
}
/** Content digest of the per-subject check-set policy. */
export function digestOfPolicy(spec: CheckSetPolicySpec): string {
return digestContent(spec);
}
@@ -0,0 +1,525 @@
import { mkdtemp, mkdir, writeFile, chmod } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { describe, expect, it } from 'vitest';
import { createSpawnProcessAdapter } from './adapter.js';
import {
builtInDefinitions,
CHECK_SET_POLICY,
checkSetForKind,
defineCheck,
QC_19_RAILS_FILES_PRESENT,
QC_20_ENFORCEMENT_VERIFY,
} from './definitions.js';
import { digestOfSpec } from './digest.js';
import { aggregateState, evaluateSubject } from './runner.js';
import type {
AdapterOutcome,
CheckDefinitionSpec,
CheckResult,
EvaluationReport,
ProcessAdapter,
} from './types.js';
// ─── helpers ────────────────────────────────────────────────────────────────
function firstResult(report: EvaluationReport): CheckResult {
const result = report.results[0];
if (result === undefined) {
throw new Error('expected the report to contain at least one result');
}
return result;
}
async function makeTempDir(): Promise<string> {
return mkdtemp(join(tmpdir(), 'qr-evaluator-'));
}
async function writeProbeScript(dir: string, name: string, body: string): Promise<string> {
const scriptPath = join(dir, name);
await writeFile(scriptPath, `${body}\n`, 'utf8');
await chmod(scriptPath, 0o755);
return scriptPath;
}
/** Adapter stub that always returns the given outcome (no real process). */
function stubAdapter(outcome: AdapterOutcome): ProcessAdapter {
return {
run: async () => outcome,
};
}
// VERBATIM copy of the pre-absorption presence loop (former cli.ts
// expectedFilesForKind + fileExists loop). This is the PARITY ORACLE: the
// evaluator's typed QC-19 verdict must agree with what the absorbed check
// concluded on the same fixture.
const LEGACY_EXPECTED: Record<'node' | 'python' | 'rust' | 'unknown', string[]> = {
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'],
};
async function legacyPresenceLoop(projectPath: string, kind: keyof typeof LEGACY_EXPECTED) {
const missing: string[] = [];
for (const relativePath of LEGACY_EXPECTED[kind]) {
const fs = await import('node:fs/promises');
try {
await fs.access(join(projectPath, relativePath));
} catch {
missing.push(relativePath);
}
}
return missing;
}
async function scaffoldFixture(kind: keyof typeof LEGACY_EXPECTED, skip: string[] = []) {
const dir = await makeTempDir();
if (kind === 'node') {
await writeFile(join(dir, 'package.json'), '{}\n', 'utf8');
}
if (kind === 'python') {
await writeFile(join(dir, 'pyproject.toml'), '[project]\n', 'utf8');
}
if (kind === 'rust') {
await writeFile(join(dir, 'Cargo.toml'), '[package]\n', 'utf8');
}
for (const relativePath of LEGACY_EXPECTED[kind]) {
if (skip.includes(relativePath)) continue;
await mkdir(join(dir, relativePath, '..'), { recursive: true });
await writeFile(join(dir, relativePath), 'fixture\n', 'utf8');
}
return dir;
}
// ─── QC-19 parity: typed verdict == absorbed presence loop ──────────────────
describe('QC-19 parity with the absorbed presence loop', () => {
const kinds: Array<keyof typeof LEGACY_EXPECTED> = ['node', 'python', 'rust', 'unknown'];
it.each(kinds)('positive fixture (%s): loop said ok ⇒ evaluator passed', async (kind) => {
const dir = await scaffoldFixture(kind);
const oracleMissing = await legacyPresenceLoop(dir, kind);
expect(oracleMissing).toEqual([]);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
});
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
expect(result?.status).toBe('passed');
expect(result?.reason).toBeUndefined();
expect(report.state).toBe('passed');
});
it.each(kinds)(
'negative fixture (%s): loop listed missing ⇒ evaluator failed with them',
async (kind) => {
const all = LEGACY_EXPECTED[kind];
const skip = all.slice(0, Math.max(1, all.length - 1)); // leave exactly 1 present
const dir = await scaffoldFixture(kind, skip);
const oracleMissing = await legacyPresenceLoop(dir, kind);
expect(oracleMissing.length).toBeGreaterThan(0);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
});
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
expect(result?.status).toBe('failed');
expect(report.state).toBe('failed');
for (const missingFile of oracleMissing) {
expect(result?.reason).toContain(missingFile);
}
// No false attribution: a present file must not be named in the reason.
const presentFile = all.find((file) => !skip.includes(file));
if (presentFile !== undefined) {
expect(result?.reason).not.toContain(` ${presentFile},`);
}
},
);
});
// ─── per-subject check sets (inventory gap 7) ────────────────────────────────
describe('per-subject check sets', () => {
it('monorepo subject selects only QC-19 with the monorepo file set', async () => {
const dir = await makeTempDir();
await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n - packages/*\n', 'utf8');
for (const file of [
'.husky/pre-commit',
'.husky/pre-push',
'eslint.config.mjs',
'.prettierrc',
'.lintstagedrc',
]) {
await mkdir(join(dir, file, '..'), { recursive: true });
await writeFile(join(dir, file), 'fixture\n', 'utf8');
}
const report = await evaluateSubject({ subjectPath: dir });
expect(report.subject.kind).toBe('monorepo');
expect(report.results.map((r) => r.checkId)).toEqual(['qc-19-rails-files-present']);
expect(report.state).toBe('passed');
});
it('a monorepo missing one of its rails files fails QC-19 (not the node list)', async () => {
const dir = await makeTempDir();
await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n', 'utf8');
const report = await evaluateSubject({ subjectPath: dir });
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
expect(result?.status).toBe('failed');
expect(result?.reason).toContain('.husky/pre-commit');
// The node-template list must NOT be applied to a monorepo subject.
expect(result?.reason).not.toContain('biome.json');
});
it('the policy selects the behavioral probe for scaffold kinds but not monorepo', () => {
expect(checkSetForKind('node')).toContain(QC_20_ENFORCEMENT_VERIFY.id);
expect(checkSetForKind('unknown')).toContain(QC_20_ENFORCEMENT_VERIFY.id);
expect(checkSetForKind('monorepo')).not.toContain(QC_20_ENFORCEMENT_VERIFY.id);
expect(CHECK_SET_POLICY.version).toBe('1.0.0');
});
});
// ─── negative controls (the point of the card) ───────────────────────────────
describe('negative controls', () => {
it('unknown check id ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({
subjectPath: dir,
checkIds: ['qc-99-does-not-exist'],
});
expect(report.results).toHaveLength(1);
const result = firstResult(report);
expect(result.status).toBe('error');
expect(result.reason).toContain("unknown check id 'qc-99-does-not-exist'");
expect(result.status === 'passed').toBe(false);
expect(report.state).toBe('error');
});
it('missing subject (directory absent) ⇒ blocked for every check, never passed', async () => {
const report = await evaluateSubject({
subjectPath: join(tmpdir(), `qr-evaluator-absent-${Date.now()}`),
});
expect(report.results.length).toBeGreaterThan(0);
for (const result of report.results) {
expect(result.status).toBe('blocked');
expect(result.reason).toContain('subject directory does not exist');
}
expect(report.state).toBe('blocked');
});
it('QC-20 without probePath input ⇒ blocked, never passed', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
});
const result = firstResult(report);
expect(result.status).toBe('blocked');
expect(result.reason).toContain('missing input: probePath');
});
it('QC-20 with a nonexistent probe script ⇒ blocked, never passed', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'no-such-probe.sh') } },
});
expect(firstResult(report).status).toBe('blocked');
expect(firstResult(report).reason).toContain('probe script not found');
});
it('adapter process error (spawn failure) ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } },
adapter: stubAdapter({ ok: false, kind: 'spawn-error', message: 'ENOENT bash' }),
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('probe process spawn-error');
expect(firstResult(report).status === 'passed').toBe(false);
});
it('adapter timeout ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } },
adapter: stubAdapter({ ok: false, kind: 'timeout', message: 'timed out after 120000ms' }),
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('probe process timeout');
});
it('probe exit 1 with parseable FAIL markers ⇒ failed (interpretably red), never passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(
dir,
'probe-fail.sh',
`echo "Test 1: ..."\necho "❌ FAIL: Type errors NOT blocked"\necho "Verification Summary"\nexit 1`,
);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('failed');
expect(firstResult(report).reason).toContain('FAIL: Type errors NOT blocked');
expect(report.state).toBe('failed');
});
it('probe exit 1 WITHOUT parseable FAIL markers ⇒ malformed ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(dir, 'probe-mute.sh', `echo "nothing to see"\nexit 1`);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('malformed probe output');
});
it('probe exit 0 without a parseable pass transcript ⇒ malformed ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(dir, 'probe-lie.sh', `echo "all good"\nexit 0`);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('malformed probe output');
expect(firstResult(report).reason).toContain('exit 0');
});
it('probe exit 0 WITH fail markers ⇒ contradictory transcript ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(
dir,
'probe-contradict.sh',
`echo "✅ PASS: one"\necho "❌ FAIL: two"\necho "Verification Summary"\nexit 0`,
);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('error');
});
it('probe unexpected exit code (7) ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(dir, 'probe-crash.sh', `echo "boom"\nexit 7`);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('unexpected code 7');
});
it('check implementation throwing ⇒ error, never passed', async () => {
const dir = await scaffoldFixture('node');
const spec: CheckDefinitionSpec = {
id: 'test-throws',
version: '1.0.0',
canonicalCheck: 'QC-TEST',
description: 'sabotage-shaped definition that always throws',
appliesTo: ['node'],
params: {},
};
const throwing = defineCheck(spec, async () => {
throw new Error('kaboom');
});
const report = await evaluateSubject({
subjectPath: dir,
checkIds: ['test-throws'],
definitions: [throwing],
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('kaboom');
});
it('non-passed verdict without a reason ⇒ upgraded to error, never an unqualified skip', async () => {
const dir = await scaffoldFixture('node');
const spec: CheckDefinitionSpec = {
id: 'test-silent-fail',
version: '1.0.0',
canonicalCheck: 'QC-TEST',
description: 'returns failed without a reason',
appliesTo: ['node'],
params: {},
};
const silent = defineCheck(spec, async () => ({ status: 'failed' }));
const report = await evaluateSubject({
subjectPath: dir,
checkIds: ['test-silent-fail'],
definitions: [silent],
});
expect(firstResult(report).status).toBe('error');
expect(firstResult(report).reason).toContain('without a reason');
});
it('empty result list aggregates to blocked, never passed', () => {
expect(aggregateState([])).toBe('blocked');
});
});
// ─── QC-20 parity: typed verdict == shell probe's own conclusion ────────────
describe('QC-20 parity with the shell probe contract', () => {
it('green transcript (exit 0) ⇒ evaluator passed', async () => {
const dir = await scaffoldFixture('node');
const probe = await writeProbeScript(
dir,
'probe-pass.sh',
[
'echo "✅ PASS: Type errors blocked"',
'echo "✅ PASS: any types blocked"',
'echo "✅ PASS: Lint errors blocked"',
'echo "Verification Summary"',
'echo "✅ Passed: 3"',
'exit 0',
].join('\n'),
);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
adapter: createSpawnProcessAdapter(),
});
expect(firstResult(report).status).toBe('passed');
expect(report.state).toBe('passed');
});
it('the REAL framework verify.sh on a non-git subject concludes failed (exit 1) ⇒ evaluator failed', async () => {
// Real-probe parity: verify.sh without a git repo cannot block planted
// commits, exits 1 with FAIL markers — the evaluator must record exactly
// `failed` with those markers, matching the probe's own conclusion.
const realProbe = fileURLToPath(
new URL('../../../mosaic/framework/tools/quality/scripts/verify.sh', import.meta.url),
);
const dir = await makeTempDir(); // not a git repository, no hooks
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
inputs: { 'qc-20-enforcement-verify': { probePath: realProbe } },
adapter: createSpawnProcessAdapter(),
});
const result = firstResult(report);
expect(result.status).toBe('failed');
expect(result.reason).toMatch(/FAIL:/);
expect(report.state).toBe('failed');
});
});
// ─── version / digest discipline ─────────────────────────────────────────────
describe('versioned, digested check definitions', () => {
it('every verdict records the definition version that produced it', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({ subjectPath: dir });
for (const result of report.results) {
expect(result.checkVersion).toBe('1.0.0');
}
expect(report.checkSetVersion).toBe(CHECK_SET_POLICY.version);
});
it('the report records each definitions content digest', async () => {
const dir = await scaffoldFixture('node');
const report = await evaluateSubject({ subjectPath: dir });
expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe(
QC_19_RAILS_FILES_PRESENT.definitionDigest,
);
expect(report.definitionDigests[QC_20_ENFORCEMENT_VERIFY.id]).toBe(
QC_20_ENFORCEMENT_VERIFY.definitionDigest,
);
});
it('digests are stable for identical content', () => {
const spec = QC_19_RAILS_FILES_PRESENT;
expect(digestOfSpec(spec)).toBe(digestOfSpec(spec));
});
it('changing a definitions content changes its digest', () => {
const base = { ...QC_19_RAILS_FILES_PRESENT } as CheckDefinitionSpec;
const baseDigest = digestOfSpec(base);
const changedParams: CheckDefinitionSpec = {
...base,
params: {
expectedFilesByKind: {
...(base.params['expectedFilesByKind'] as Record<string, string[]>),
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md', 'NEW.md'],
},
},
};
expect(digestOfSpec(changedParams)).not.toBe(baseDigest);
const changedVersion: CheckDefinitionSpec = { ...base, version: '1.1.0' };
expect(digestOfSpec(changedVersion)).not.toBe(baseDigest);
});
it('a definition with changed content produces a different recorded digest and version', async () => {
const dir = await scaffoldFixture('node');
const modified = defineCheck(
{ ...QC_19_RAILS_FILES_PRESENT, version: '2.0.0' } as unknown as CheckDefinitionSpec,
async () => ({ status: 'passed' }),
);
const report = await evaluateSubject({
subjectPath: dir,
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
definitions: [
modified,
...builtInDefinitions().filter((d) => d.id !== QC_19_RAILS_FILES_PRESENT.id),
],
});
const result = firstResult(report);
expect(result.checkVersion).toBe('2.0.0');
expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe(modified.definitionDigest);
expect(modified.definitionDigest).not.toBe(QC_19_RAILS_FILES_PRESENT.definitionDigest);
});
});
// ─── aggregate state ordering (MACP-style discipline) ───────────────────────
describe('aggregate state precedence', () => {
const result = (status: 'passed' | 'failed' | 'blocked' | 'error') => ({
status,
checkId: 'x',
checkVersion: '1.0.0',
subject: '/tmp/x',
});
it('all passed (with not-applicable) ⇒ passed', () => {
expect(
aggregateState([
result('passed'),
{ ...result('passed'), status: 'not-applicable' as const },
]),
).toBe('passed');
});
it('error outranks blocked and failed; blocked outranks failed', () => {
expect(aggregateState([result('blocked'), result('error')])).toBe('error');
expect(aggregateState([result('failed'), result('blocked')])).toBe('blocked');
expect(aggregateState([result('passed'), result('failed')])).toBe('failed');
});
});
@@ -0,0 +1,183 @@
import { constants } from 'node:fs';
import { access, stat } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { createSpawnProcessAdapter } from './adapter.js';
import { builtInDefinitions, CHECK_SET_POLICY, checkSetForKind } from './definitions.js';
import type {
AggregateState,
CheckResult,
CheckStatus,
EvaluateOptions,
EvaluationReport,
ProcessAdapter,
Subject,
SubjectKind,
} from './types.js';
import { detectProjectKind } from '../detect.js';
async function pathExists(targetPath: string): Promise<boolean> {
try {
await access(targetPath, constants.F_OK);
return true;
} catch {
return false;
}
}
async function isDirectory(targetPath: string): Promise<boolean> {
try {
return (await stat(targetPath)).isDirectory();
} catch {
return false;
}
}
/**
* Subject-kind detection for the evaluator. Extends the scaffold detection
* (detect.ts) with the `monorepo` kind: a pnpm workspace is this repository's
* own subject shape and carries a different rails file set (probe-inventory
* gap 7 — check sets are per subject, not one global file list).
*/
export async function detectSubjectKind(subjectPath: string): Promise<SubjectKind> {
if (await pathExists(join(subjectPath, 'pnpm-workspace.yaml'))) {
return 'monorepo';
}
const kind = await detectProjectKind(subjectPath);
return kind;
}
/**
* Aggregate state, MACP-style discipline: `passed` only when at least one
* check produced a verdict AND every verdict is `passed` or an explicitly
* qualified `not-applicable`. Precedence is fail-closed: error > blocked >
* failed > passed; an empty result list aggregates to `blocked`.
*/
export function aggregateState(results: readonly CheckResult[]): AggregateState {
if (results.length === 0) {
return 'blocked';
}
const has = (status: CheckStatus): boolean => results.some((result) => result.status === status);
if (has('error')) {
return 'error';
}
if (has('blocked')) {
return 'blocked';
}
if (has('failed')) {
return 'failed';
}
return 'passed';
}
function reasonFrom(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
/**
* Evaluate one subject against a set of checks, producing typed verdicts.
*
* Fail-closed invariants (RI-N4):
* - unknown check id → `error` (never passed)
* - subject directory absent → every verdict `blocked`
* - check implementation threw → `error`
* - non-passed without a reason → `error` (no unqualified skips)
* - check not applicable → `not-applicable` WITH a reason
*/
export async function evaluateSubject(options: EvaluateOptions): Promise<EvaluationReport> {
const subjectPath = resolve(options.subjectPath);
const subject: Subject = {
path: subjectPath,
kind: await detectSubjectKind(subjectPath),
};
const definitions = options.definitions ?? builtInDefinitions();
const byId = new Map(definitions.map((definition) => [definition.id, definition]));
const requested = options.checkIds ?? checkSetForKind(subject.kind);
const adapter: ProcessAdapter = options.adapter ?? createSpawnProcessAdapter();
const results: CheckResult[] = [];
const definitionDigests: Record<string, string> = {};
for (const checkId of requested) {
const definition = byId.get(checkId);
if (definition === undefined) {
const known = definitions.map((entry) => entry.id).join(', ');
results.push({
status: 'error',
checkId,
checkVersion: 'unknown',
subject: subjectPath,
reason: `unknown check id '${checkId}' — no registered definition (known: ${known})`,
});
continue;
}
definitionDigests[checkId] = definition.definitionDigest;
if (!(await isDirectory(subjectPath))) {
results.push({
status: 'blocked',
checkId,
checkVersion: definition.version,
subject: subjectPath,
reason: `subject directory does not exist: ${subjectPath}`,
});
continue;
}
if (!definition.appliesTo.includes(subject.kind)) {
results.push({
status: 'not-applicable',
checkId,
checkVersion: definition.version,
subject: subjectPath,
reason: `check '${checkId}' does not apply to subject kind '${subject.kind}'`,
});
continue;
}
try {
const inputs = options.inputs?.[checkId] ?? {};
const outcome = await definition.evaluate({
subject,
params: definition.params,
inputs,
adapter,
});
if (outcome.status !== 'passed' && (outcome.reason === undefined || outcome.reason === '')) {
results.push({
status: 'error',
checkId,
checkVersion: definition.version,
subject: subjectPath,
reason: `check returned status '${outcome.status}' without a reason — treated as error`,
});
continue;
}
results.push({
status: outcome.status,
checkId,
checkVersion: definition.version,
subject: subjectPath,
reason: outcome.reason,
});
} catch (error) {
results.push({
status: 'error',
checkId,
checkVersion: definition.version,
subject: subjectPath,
reason: `check implementation threw: ${reasonFrom(error)}`,
});
}
}
return {
subject,
results,
definitionDigests,
checkSetVersion: CHECK_SET_POLICY.version,
state: aggregateState(results),
};
}
@@ -0,0 +1,148 @@
// Evaluator core types — RI-N4 (card RI-3-002, SDLC-D-037 second half).
//
// The quality-rails evaluator is the SOLE authoritative producer of check
// verdicts for the checks it owns. Every verdict is typed and fail-closed:
// missing implementations, missing inputs, unknown check ids, process errors,
// timeouts, and malformed adapter output can never become `passed` or an
// unqualified skip — they surface as `blocked` or `error` with a reason.
// (Vocabulary mirrors MACP's GateStatus discipline from packages/macp.)
/**
* Typed verdict for a single check execution.
*
* - `passed` — the check really ran and its condition held.
* - `failed` — the check really ran and its condition did NOT hold.
* - `blocked` — the check could not run at all (missing subject, missing
* input). Never a green outcome.
* - `error` — the check attempted to run but its outcome cannot be trusted
* (unknown check id, implementation threw, process error, timeout, malformed
* adapter output). Never a green outcome.
* - `not-applicable` — the check definition explicitly declares it does not
* apply to this subject (a qualified skip, always with a reason).
*/
export type CheckStatus = 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable';
/** Aggregate outcome, MACP-style: `passed` only when every result is green. */
export type AggregateState = 'passed' | 'failed' | 'blocked' | 'error';
/** Kinds of subjects the evaluator can assess. */
export type SubjectKind = 'node' | 'python' | 'rust' | 'monorepo' | 'unknown';
/**
* A single check verdict. This is the canonical result shape: `status`,
* `checkId`, `checkVersion`, `subject`, `reason`. `reason` is REQUIRED
* (enforced by the runner) for every status other than `passed`.
*/
export interface CheckResult {
status: CheckStatus;
checkId: string;
checkVersion: string;
subject: string;
reason?: string;
}
/** The project being evaluated. */
export interface Subject {
/** Absolute path. */
path: string;
kind: SubjectKind;
}
/**
* The data half of a check definition. Definitions live as DATA with a version
* and a content digest (see `digestOfSpec`); the executable half is attached
* separately so the digest covers only reviewable, declarative content.
*/
export interface CheckDefinitionSpec {
/** Stable id, e.g. `qc-19-rails-files-present`. */
id: string;
/** Semver of this definition's data+semantics. */
version: string;
/** Canonical check id from docs/release-integrity/probe-inventory.md (QC-n). */
canonicalCheck: string;
description: string;
/** Subject kinds this check can assess (others yield `not-applicable`). */
appliesTo: readonly SubjectKind[];
/** Declarative parameters (file lists, markers, timeouts) — digest-covered. */
params: Record<string, unknown>;
}
/** A fully assembled check definition: spec + digest + implementation. */
export interface CheckDefinition extends CheckDefinitionSpec {
/** sha256 content digest of the spec (canonical JSON projection). */
definitionDigest: string;
evaluate(ctx: CheckContext): Promise<CheckOutcome>;
}
/** What a check implementation returns; the runner stamps id/version/subject. */
export interface CheckOutcome {
status: CheckStatus;
reason?: string;
}
/** Caller-provided inputs for one check invocation (e.g. the QC-20 probe path). */
export type CheckInputs = Record<string, unknown>;
/** Everything a check implementation may use. */
export interface CheckContext {
subject: Subject;
params: Record<string, unknown>;
inputs: CheckInputs;
adapter: ProcessAdapter;
}
/** Outcome of running a shell probe through the thin process adapter. */
export type AdapterOutcome =
| { ok: true; exitCode: number | null; stdout: string; stderr: string }
| { ok: false; kind: 'spawn-error' | 'timeout'; message: string };
/** Request for the process adapter. */
export interface AdapterRequest {
file: string;
args: string[];
cwd: string;
timeoutMs: number;
}
/**
* Thin process adapter: runs a command, owns NO verdict logic. Verdict parsing
* always lives in the check implementation (TS), never in the shell probe.
*/
export interface ProcessAdapter {
run(request: AdapterRequest): Promise<AdapterOutcome>;
}
/** Per-subject-kind check-set selection policy (versioned and digested). */
export interface CheckSetPolicySpec {
version: string;
byKind: Record<SubjectKind, readonly string[]>;
}
export interface CheckSetPolicy extends CheckSetPolicySpec {
/** sha256 content digest of the policy spec. */
policyDigest: string;
}
/** Full typed evaluation report for one subject. */
export interface EvaluationReport {
subject: Subject;
results: CheckResult[];
/** checkId → content digest of the definition that produced the verdicts. */
definitionDigests: Record<string, string>;
/** Version of the check-set policy used for subject selection. */
checkSetVersion: string;
state: AggregateState;
}
/** Options for `evaluateSubject`. */
export interface EvaluateOptions {
subjectPath: string;
/** Restrict to these check ids; defaults to the subject kind's check set. */
checkIds?: string[];
/** Per-check inputs, keyed by check id (e.g. `{ 'qc-20-enforcement-verify': { probePath } }`). */
inputs?: Record<string, CheckInputs>;
/** Replace the built-in definitions (tests / future batches). */
definitions?: CheckDefinition[];
/** Inject a process adapter (tests / instrumentation). */
adapter?: ProcessAdapter;
}
+32
View File
@@ -3,3 +3,35 @@ export * from './detect.js';
export * from './scaffolder.js';
export * from './templates.js';
export * from './types.js';
// RI-N4 evaluator (card RI-3-002): the public, programmatic entry points.
export {
builtInDefinitions,
CHECK_SET_POLICY,
checkSetForKind,
defineCheck,
QC_19_RAILS_FILES_PRESENT,
QC_20_ENFORCEMENT_VERIFY,
} from './evaluator/definitions.js';
export { canonicalJson, digestContent, digestOfPolicy, digestOfSpec } from './evaluator/digest.js';
export { createSpawnProcessAdapter } from './evaluator/adapter.js';
export { aggregateState, detectSubjectKind, evaluateSubject } from './evaluator/runner.js';
export type {
AdapterOutcome,
AdapterRequest,
AggregateState,
CheckContext,
CheckDefinition,
CheckDefinitionSpec,
CheckInputs,
CheckOutcome,
CheckResult,
CheckSetPolicy,
CheckSetPolicySpec,
CheckStatus,
EvaluateOptions,
EvaluationReport,
ProcessAdapter,
Subject,
SubjectKind,
} from './evaluator/types.js';
+15
View File
@@ -22,6 +22,12 @@
// format | format | pnpm format:check
// test | test | pnpm test
// build | publish.yml build | pnpm build
// quality-rails | (canonical-only) | the TS quality-rails evaluator
// | | (RI-N4, QC-19 monorepo subject). Like
// | | `build`, this stage has no ci.yml
// | | mirror; it is implemented by
// | | importing the evaluator CLI rather
// | | than duplicating its presence logic.
//
// Caller-provided prerequisites (kept at the pipeline level — see the comments
// in .woodpecker/ci.yml): `bash` + `rsync` for the guard stages, `openssl` and
@@ -90,6 +96,15 @@ export const STAGES = [
name: 'build',
commands: ['pnpm build'],
},
{
// RI-N4 (QC-19, card RI-3-002): the typed quality-rails evaluator, invoked
// as the implementation of the check it owns instead of a duplicated
// presence loop here. Canonical-only stage (no ci.yml mirror — same shape
// as `build`); runs AFTER build so the evaluator's dist/ exists. Subject
// is this repository (`.` → monorepo subject kind, per-subject check set).
name: 'quality-rails',
commands: ['node packages/quality-rails/dist/cli.js quality-rails evaluate --project .'],
},
];
export function stageByName(name) {
+49 -3
View File
@@ -4,7 +4,7 @@ import { createRequire } from 'node:module';
import path from 'node:path';
import test from 'node:test';
import { STAGES } from './verify-release.mjs';
import { STAGES, stageByName } from './verify-release.mjs';
// SDLC-D-034 checkout invariant: publication in .woodpecker/publish.yml is
// bound to exact-commit terminal verification. This suite parses the real
@@ -229,10 +229,21 @@ steps:
function assertStagesMirrorCi(stages, ci) {
const canonical = Object.fromEntries(stages.map((stage) => [stage.name, stage.commands]));
// The complete mandatory set, in gate order.
// The complete mandatory set, in gate order. `quality-rails` is a
// canonical-only stage (RI-N4, QC-19): like `build`, it has no ci.yml
// mirror to match — its contract is asserted separately below.
assert.deepEqual(
stages.map((stage) => stage.name),
['sanitization', 'upgrade-guard', 'typecheck', 'lint', 'format', 'test', 'build'],
[
'sanitization',
'upgrade-guard',
'typecheck',
'lint',
'format',
'test',
'build',
'quality-rails',
],
);
// Guard stages: ci.yml commands minus its `apk add` environment prep must be
@@ -301,3 +312,38 @@ test('the root package.json exposes verify:release as the canonical command', as
const packageJson = JSON.parse(await readFile(path.join(process.cwd(), 'package.json'), 'utf8'));
assert.match(packageJson.scripts['verify:release'], /scripts\/verify-release\.mjs/);
});
// RI-N4 (card RI-3-002): the `quality-rails` stage must route through the TS
// evaluator instead of duplicating its presence logic inline. The evaluator
// owns QC-19; this file keeps that delegation honest.
function assertEvaluatorStage(stage) {
assert.ok(stage, 'canonical stages must include a quality-rails stage');
assert.ok(Array.isArray(stage.commands) && stage.commands.length > 0);
for (const command of stage.commands) {
assert.match(
command,
/packages\/quality-rails\/dist\/cli\.js.*quality-rails evaluate/,
`quality-rails stage command must invoke the evaluator CLI, got: '${command}'`,
);
}
}
test('the quality-rails stage invokes the evaluator rather than duplicating its logic', () => {
assertEvaluatorStage(stageByName('quality-rails'));
});
test('a quality-rails stage that re-implements presence logic inline fails the checker', () => {
// Negative control: replacing the evaluator invocation with an inline
// `test -f` presence loop is exactly the duplication RI-N4 forbids — the
// checker must go red on it.
const duplicated = {
name: 'quality-rails',
commands: ['test -f .husky/pre-commit && test -f .husky/pre-push'],
};
assert.throws(() => assertEvaluatorStage(duplicated), /must invoke the evaluator CLI/);
});
test('a quality-rails stage that silently drops the evaluator command fails the checker', () => {
const empty = { name: 'quality-rails', commands: [] };
assert.throws(() => assertEvaluatorStage(empty), /commands/);
});