diff --git a/docs/release-integrity/TASKS.md b/docs/release-integrity/TASKS.md index 47341cc8..a77c7350 100644 --- a/docs/release-integrity/TASKS.md +++ b/docs/release-integrity/TASKS.md @@ -22,7 +22,7 @@ | RI-1-002 | done | RI-N1 negative control: checked-in tests proving a broken mandatory check blocks every publish step and that DAG edges cannot be bypassed | #1275 | pi-glm-5.3 | mosaicstack/stack | test/ri-050-publish-gate-negative | RI-1-001 | 12K | | | RI-2-001 | done | RI-N2 (Forge): remove stub-executor false success; `--simulate` typed `simulated` results that satisfy nothing; literal-`true` gates and echo-review replaced with real gates or typed waiting-for-authority | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-forge-fail-closed | RI-0-001 | 20K | Independent review APPROVED 2026-08-17 (Gitea review 172 on PR #1278, head 99b8f6ea; reviewing seat fargo — recorded under shared host principal mos-dt-0, provenance correction posted by fred; wrapper gap filed by fred). Executed at head: forge tests 116/116, lint green, typecheck green after building macp dist (minimal-install artifact, not a defect), workspace typecheck 45/45, no external type consumers of the changed interfaces. CI red = known lane-wide fleet-test failure only, carries no information about this change (fred, log-content analysis, pipelines 2456-2458). Non-blocking finding: README L141-143 + skills/mosaic-forge/SKILL.md document bare forge run/resume, which now fails closed — fast-follow docs touch. Merge queued behind #1270. UPDATE 2026-08-18: #1270 merged; CI GREEN at head 4917df1f via serialized retry (pipeline 2477) - root cause of prior reds was CI-agent contention (web SPA timeouts under concurrent pipelines), superseding the fleet-test-failure theory. | | RI-2-002 | done | RI-N2 (MACP): gate runner fails closed on empty commands, stub executors, and unimplemented CI-provider gates unless explicit simulate; typed capability failures | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-macp-fail-closed | RI-0-001 | 15K | PR #1293 (head 2097379e): CI green (pipeline 2465), independent review APPROVED (Gitea review 173, jarvis seat, 2026-08-17) - macp 109/109 verified at head. Merge queued behind #1276/#1277/#1278. | -| RI-3-001 | in-progress | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | | +| RI-3-001 | done | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | PR #1302 (head e06a47fac591): CI green (2484), independent review APPROVED (Gitea review 187, fargo seat, 2026-08-18) — 54 rows / 21 canonical checks / dispositions 43-2-9-0 verified by row-count and code spot-checks. Merged by fargo at pinned head. | | RI-3-002 | not-started | RI-N4: TS evaluator absorbs effective shell probes; typed results (passed/failed/blocked/error/not-applicable) with versioned digested check definitions; shell commands become thin adapters; contract/parity/negative-control tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-qr-evaluator | RI-3-001 | 30K | | | RI-4-001 | in-progress | RI-N3: one PRD application service — `mission --plan` persists mission↔PRD linkage (ids/versions/selected requirements); `mosaic prdy` routes through the service or becomes a named import/export adapter; Markdown is a labeled generated view; explicit conflict-aware import | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-prd-authority | RI-0-001 | 35K | PR #1294 (head 8d258e1d): CI green (pipeline 2466), independent review APPROVED (Gitea review 174, jarvis seat, 2026-08-17) - prdy 20/20 + command specs 9/9 at head. Merge queued behind #1276/#1277/#1278. | | RI-5-001 | done | RI-N5: typed freshness states (current/stale/partial/unknown/unavailable); no failed-fetch-renders-empty; stale derived verdicts → unknown; mutations disabled when stale; failure-matrix tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-web-stale-safety | RI-0-001 | 25K | | diff --git a/packages/quality-rails/README.md b/packages/quality-rails/README.md new file mode 100644 index 00000000..ba82ad3a --- /dev/null +++ b/packages/quality-rails/README.md @@ -0,0 +1,63 @@ +# @mosaicstack/quality-rails + +Quality-rails scaffolding and — since RI-3-002 (RI-N4) — the **typed +quality-rails evaluator**: the single authoritative producer of check verdicts +for the checks it owns. + +## Evaluator (RI-N4) + +Every verdict is typed and fail-closed: + +``` +{ status: 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable', + checkId, checkVersion, subject, reason } +``` + +Missing implementations, missing inputs, unknown check ids, process errors, +timeouts, and malformed probe output can never become `passed` or an +unqualified skip — they surface as `blocked`/`error` with a reason (vocabulary +mirrors MACP's `GateStatus` discipline). + +- Check definitions live as **data** with a version and a sha256 content + digest (`definitionDigest`); every recorded verdict names the definition + version that produced it. +- Check sets are selected **per subject kind** (`node`, `python`, `rust`, + `monorepo`, `unknown`) via the versioned, digested check-set policy — this + repository (a `monorepo` subject) does not share the node template's file + list. +- Shell probes stay **thin adapters**: the TS evaluator invokes them and owns + the verdict parsing (e.g. QC-20's planted-commit probe). + +### Owned checks + +| check id | canonical check | mechanism | +| --------------------------- | --------------- | ---------------------------------------------------------------------- | +| `qc-19-rails-files-present` | QC-19 | typed absorption of the former presence-only `check`/`doctor` loop | +| `qc-20-enforcement-verify` | QC-20 | thin shell adapter (framework `verify.sh`); verdict parsing owned here | + +The canonical check ids QC-1..QC-21 and their dispositions are defined in +`docs/release-integrity/probe-inventory.md` (the RI-3-001 inventory — the +evaluator's input, not its output). + +### Usage + +```sh +# CLI (same typed report as the programmatic API) +node dist/cli.js quality-rails evaluate --project [--probe-path ] [--json] +node dist/cli.js quality-rails check --project [--json] # QC-19 only, fail-closed exit +``` + +```ts +import { evaluateSubject } from '@mosaicstack/quality-rails'; + +const report = await evaluateSubject({ subjectPath: '/path/to/project' }); +// report.state: 'passed' | 'failed' | 'blocked' | 'error' +``` + +`pnpm verify:release` invokes this evaluator as its `quality-rails` stage +(canonical-only stage, QC-19 on the monorepo subject). + +## Scaffolding (pre-existing) + +`init` scaffolds rails files per detected kind/profile; `doctor` is advisory +and reports typed states. diff --git a/packages/quality-rails/src/cli-evaluate.spec.ts b/packages/quality-rails/src/cli-evaluate.spec.ts new file mode 100644 index 00000000..4bee2b2a --- /dev/null +++ b/packages/quality-rails/src/cli-evaluate.spec.ts @@ -0,0 +1,194 @@ +import { mkdir, mkdtemp, writeFile, chmod } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { createQualityRailsCli } from './cli.js'; +import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js'; +import { evaluateSubject } from './evaluator/runner.js'; +import type { EvaluationReport } from './evaluator/types.js'; + +// CLI ↔ programmatic contract (RI-3-002): the same subject must produce the +// same typed verdicts through every entry point the card adds — the +// `evaluate`/`check` CLI surfaces and the `evaluateSubject` API. + +async function makeTempDir(): Promise { + return mkdtemp(join(tmpdir(), 'qr-cli-')); +} + +async function scaffoldNodeFixture(skip: string[] = []): Promise { + const dir = await makeTempDir(); + await writeFile(join(dir, 'package.json'), '{}\n', 'utf8'); + for (const relativePath of [ + '.eslintrc', + 'biome.json', + '.githooks/pre-commit', + 'PR-CHECKLIST.md', + ]) { + if (skip.includes(relativePath)) continue; + await mkdir(join(dir, relativePath, '..'), { recursive: true }); + await writeFile(join(dir, relativePath), 'fixture\n', 'utf8'); + } + return dir; +} + +async function makePassingProbe(dir: string): Promise { + const scriptPath = join(dir, 'probe-pass.sh'); + await writeFile( + scriptPath, + [ + '#!/bin/bash', + 'echo "✅ PASS: Type errors blocked"', + 'echo "✅ PASS: Lint errors blocked"', + 'echo "Verification Summary"', + 'exit 0', + ].join('\n') + '\n', + 'utf8', + ); + await chmod(scriptPath, 0o755); + return scriptPath; +} + +describe('CLI entry points vs the programmatic evaluator', () => { + let logSpy: ReturnType; + let previousExitCode: string | number | undefined; + + beforeEach(() => { + logSpy = vi.spyOn(console, 'log').mockImplementation(() => {}); + previousExitCode = process.exitCode ?? undefined; + }); + + afterEach(() => { + logSpy.mockRestore(); + process.exitCode = previousExitCode; + }); + + it('evaluate --json produces the SAME typed report as evaluateSubject (full check set + probe)', async () => { + const dir = await scaffoldNodeFixture(); + const probePath = await makePassingProbe(dir); + + const programmatic = await evaluateSubject({ + subjectPath: dir, + inputs: { 'qc-20-enforcement-verify': { probePath } }, + }); + + const program = createQualityRailsCli(); + await program.parseAsync([ + 'node', + 'cli.js', + 'quality-rails', + 'evaluate', + '--project', + dir, + '--probe-path', + probePath, + '--json', + ]); + + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + const cliReport = JSON.parse(printed) as EvaluationReport; + expect(cliReport).toEqual(programmatic); + expect(cliReport.state).toBe('passed'); + expect(process.exitCode).toBe(0); + }); + + it('check --json produces the SAME QC-19 verdict as evaluateSubject (absorbed loop)', async () => { + const dir = await scaffoldNodeFixture(['biome.json', '.githooks/pre-commit']); + + const programmatic = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_19_RAILS_FILES_PRESENT.id], + }); + expect(programmatic.state).toBe('failed'); + + const program = createQualityRailsCli(); + await program.parseAsync([ + 'node', + 'cli.js', + 'quality-rails', + 'check', + '--project', + dir, + '--json', + ]); + + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + const cliReport = JSON.parse(printed) as EvaluationReport; + expect(cliReport).toEqual(programmatic); + expect(process.exitCode).toBe(1); + }); + + it('check on a complete subject exits 0 with a passed verdict', async () => { + const dir = await scaffoldNodeFixture(); + const program = createQualityRailsCli(); + await program.parseAsync([ + 'node', + 'cli.js', + 'quality-rails', + 'check', + '--project', + dir, + '--json', + ]); + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + const cliReport = JSON.parse(printed) as EvaluationReport; + expect(cliReport.state).toBe('passed'); + expect(process.exitCode).toBe(0); + }); + + it('evaluate with an unknown check id exits 1 and reports error, never passed', async () => { + const dir = await scaffoldNodeFixture(); + const program = createQualityRailsCli(); + await program.parseAsync([ + 'node', + 'cli.js', + 'quality-rails', + 'evaluate', + '--project', + dir, + '--check', + 'qc-99-bogus', + '--json', + ]); + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + const cliReport = JSON.parse(printed) as EvaluationReport; + expect(cliReport.results).toHaveLength(1); + const first = cliReport.results[0]; + expect(first?.status).toBe('error'); + expect(first?.reason).toContain('unknown check id'); + expect(process.exitCode).toBe(1); + }); + + it('evaluate on a scaffold subject without --probe-path stays fail-closed (blocked, exit 1)', async () => { + const dir = await scaffoldNodeFixture(); + const program = createQualityRailsCli(); + await program.parseAsync([ + 'node', + 'cli.js', + 'quality-rails', + 'evaluate', + '--project', + dir, + '--json', + ]); + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + const cliReport = JSON.parse(printed) as EvaluationReport; + const qc20 = cliReport.results.find((r) => r.checkId === 'qc-20-enforcement-verify'); + expect(qc20).toBeDefined(); + expect(qc20?.status).toBe('blocked'); + expect(qc20?.reason).toContain('probePath'); + expect(cliReport.state).toBe('blocked'); + expect(process.exitCode).toBe(1); + }); + + it('doctor stays advisory (no nonzero exit) but reports TYPED states, including blocked', async () => { + const dir = await scaffoldNodeFixture(); + const program = createQualityRailsCli(); + await program.parseAsync(['node', 'cli.js', 'quality-rails', 'doctor', '--project', dir]); + + const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n'); + expect(printed).toContain('blocked: qc-20-enforcement-verify'); + expect(process.exitCode ?? 0).toBe(0); + }); +}); diff --git a/packages/quality-rails/src/cli.ts b/packages/quality-rails/src/cli.ts index 0c5908f3..2ab38b72 100644 --- a/packages/quality-rails/src/cli.ts +++ b/packages/quality-rails/src/cli.ts @@ -1,5 +1,3 @@ -import { constants } from 'node:fs'; -import { access } from 'node:fs/promises'; import { resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -8,18 +6,12 @@ import { Command } from 'commander'; import { detectProjectKind } from './detect.js'; import { scaffoldQualityRails } from './scaffolder.js'; import type { ProjectKind, QualityProfile, RailsConfig } from './types.js'; +import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js'; +import { evaluateSubject } from './evaluator/runner.js'; +import type { EvaluationReport } from './evaluator/types.js'; const VALID_PROFILES: readonly QualityProfile[] = ['strict', 'standard', 'minimal']; -async function fileExists(filePath: string): Promise { - try { - await access(filePath, constants.F_OK); - return true; - } catch { - return false; - } -} - function parseProfile(rawProfile: string): QualityProfile { if (VALID_PROFILES.includes(rawProfile as QualityProfile)) { return rawProfile as QualityProfile; @@ -59,22 +51,6 @@ function defaultFormatters(kind: ProjectKind): string[] { return []; } -function expectedFilesForKind(kind: ProjectKind): string[] { - if (kind === 'node') { - return ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md']; - } - - if (kind === 'python') { - return ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md']; - } - - if (kind === 'rust') { - return ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md']; - } - - return ['.githooks/pre-commit', 'PR-CHECKLIST.md']; -} - function printScaffoldResult( config: RailsConfig, filesWritten: string[], @@ -106,6 +82,20 @@ function printScaffoldResult( } } +function printEvaluationReport(report: EvaluationReport): void { + console.log( + `[quality-rails] evaluation for ${report.subject.path} (kind=${report.subject.kind}, check-set v${report.checkSetVersion})`, + ); + for (const result of report.results) { + const reason = result.reason === undefined ? '' : ` — ${result.reason}`; + const digest = report.definitionDigests[result.checkId] ?? 'no digest'; + console.log( + ` - ${result.status}: ${result.checkId} (v${result.checkVersion} [${digest}])${reason}`, + ); + } + console.log(`[quality-rails] aggregate: ${report.state}`); +} + /** * Register quality-rails subcommands on an existing Commander program. * This avoids cross-package Commander version mismatches by using the @@ -148,56 +138,93 @@ function buildQualityRailsCommand(qualityRails: Command): void { printScaffoldResult(config, result.filesWritten, result.warnings, result.commandsToRun); }); + // `check` (QC-19) is ABSORBED by the RI-N4 evaluator: the presence loop + // that lived here is now the versioned, digested, typed check definition + // `qc-19-rails-files-present`. The CLI keeps its human surface (missing + // files listed, exit 1) and gains `--json` for the typed verdicts. Exit + // code is fail-closed: any non-green aggregate (failed/blocked/error) is 1. qualityRails .command('check') .requiredOption('--project ', 'Project path') - .action(async (options: { project: string }) => { + .option('--json', 'print the typed evaluation report as JSON') + .action(async (options: { project: string; json?: boolean }) => { const projectPath = resolve(options.project); - const kind = await detectProjectKind(projectPath); - const expected = expectedFilesForKind(kind); - const missing: string[] = []; + const report = await evaluateSubject({ + subjectPath: projectPath, + checkIds: [QC_19_RAILS_FILES_PRESENT.id], + }); - for (const relativePath of expected) { - const exists = await fileExists(resolve(projectPath, relativePath)); - if (!exists) { - missing.push(relativePath); - } + if (options.json) { + console.log(JSON.stringify(report)); + } else { + printEvaluationReport(report); } - if (missing.length > 0) { - console.error('[quality-rails] missing files:'); - for (const relativePath of missing) { - console.error(` - ${relativePath}`); - } - process.exitCode = 1; - return; - } - - console.log(`[quality-rails] all expected files present for ${kind} project`); + process.exitCode = report.state === 'passed' ? 0 : 1; }); + // `doctor` (QC-19) stays advisory (documented contract: a doctor that + // cannot fail), but now reports TYPED states — a blocked or failing rail is + // visible instead of silently printed as `ok`/`missing`. qualityRails .command('doctor') .requiredOption('--project ', 'Project path') .action(async (options: { project: string }) => { const projectPath = resolve(options.project); - const kind = await detectProjectKind(projectPath); - const expected = expectedFilesForKind(kind); + const report = await evaluateSubject({ subjectPath: projectPath }); console.log(`[quality-rails] doctor for ${projectPath}`); - console.log(`detected project kind: ${kind}`); - - for (const relativePath of expected) { - const exists = await fileExists(resolve(projectPath, relativePath)); - console.log(` - ${exists ? 'ok' : 'missing'}: ${relativePath}`); + console.log(`detected project kind: ${report.subject.kind}`); + for (const result of report.results) { + const reason = result.reason === undefined ? '' : ` — ${result.reason}`; + console.log(` - ${result.status}: ${result.checkId}${reason}`); } - if (kind === 'unknown') { + if (report.subject.kind === 'unknown') { console.log( 'recommendation: add package.json, pyproject.toml, or Cargo.toml for better defaults.', ); } }); + + // `evaluate` is the canonical RI-N4 evaluator entry point: typed verdicts + // for the subject's full per-kind check set, same results as the + // programmatic API (evaluateSubject). + qualityRails + .command('evaluate') + .description('Run the typed quality-rails evaluator against a subject project') + .requiredOption('--project ', 'Project path') + .option('--check ', 'restrict evaluation to these check ids') + .option( + '--probe-path ', + 'path to the QC-20 behavioral probe script (framework verify.sh)', + ) + .option('--json', 'print the typed evaluation report as JSON') + .action( + async (options: { + project: string; + check?: string[]; + probePath?: string; + json?: boolean; + }) => { + const projectPath = resolve(options.project); + const report = await evaluateSubject({ + subjectPath: projectPath, + checkIds: options.check, + inputs: options.probePath + ? { 'qc-20-enforcement-verify': { probePath: options.probePath } } + : undefined, + }); + + if (options.json) { + console.log(JSON.stringify(report)); + } else { + printEvaluationReport(report); + } + + process.exitCode = report.state === 'passed' ? 0 : 1; + }, + ); } export async function runQualityRailsCli(argv: string[] = process.argv): Promise { diff --git a/packages/quality-rails/src/evaluator/adapter.ts b/packages/quality-rails/src/evaluator/adapter.ts new file mode 100644 index 00000000..a731c0bf --- /dev/null +++ b/packages/quality-rails/src/evaluator/adapter.ts @@ -0,0 +1,67 @@ +import { spawn } from 'node:child_process'; +import type { ChildProcess } from 'node:child_process'; + +import type { AdapterOutcome, AdapterRequest, ProcessAdapter } from './types.js'; + +/** + * Default thin process adapter (spawn-based). Runs a command to completion with + * a hard timeout and reports exit code + captured output — it owns NO verdict + * logic. Interpreting the outcome is always the check implementation's job. + */ +export function createSpawnProcessAdapter(): ProcessAdapter { + return { + run(request: AdapterRequest): Promise { + return new Promise((resolve) => { + let child: ChildProcess; + try { + child = spawn(request.file, request.args, { + cwd: request.cwd, + stdio: ['ignore', 'pipe', 'pipe'], + }); + } catch (error) { + resolve({ + ok: false, + kind: 'spawn-error', + message: error instanceof Error ? error.message : String(error), + }); + return; + } + + let stdout = ''; + let stderr = ''; + let settled = false; + + const timer = setTimeout(() => { + if (settled) return; + settled = true; + child.kill('SIGKILL'); + resolve({ + ok: false, + kind: 'timeout', + message: `process timed out after ${request.timeoutMs}ms: ${request.file}`, + }); + }, request.timeoutMs); + + const settle = (outcome: AdapterOutcome): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(outcome); + }; + + child.stdout?.on('data', (chunk: Buffer) => { + stdout += chunk.toString('utf8'); + }); + child.stderr?.on('data', (chunk: Buffer) => { + stderr += chunk.toString('utf8'); + }); + child.on('error', (error: Error) => { + settle({ ok: false, kind: 'spawn-error', message: error.message }); + }); + child.on('close', (code: number | null) => { + settle({ ok: true, exitCode: code, stdout, stderr }); + }); + }); + }, + }; +} diff --git a/packages/quality-rails/src/evaluator/definitions.ts b/packages/quality-rails/src/evaluator/definitions.ts new file mode 100644 index 00000000..796da0e2 --- /dev/null +++ b/packages/quality-rails/src/evaluator/definitions.ts @@ -0,0 +1,258 @@ +import { constants } from 'node:fs'; +import { access } from 'node:fs/promises'; +import { isAbsolute, resolve } from 'node:path'; + +import { digestOfPolicy, digestOfSpec } from './digest.js'; +import type { + CheckContext, + CheckDefinition, + CheckDefinitionSpec, + CheckOutcome, + CheckSetPolicy, + CheckSetPolicySpec, + SubjectKind, +} from './types.js'; + +// Check definitions for the RI-N4 evaluator (card RI-3-002). Each definition is +// DATA with a version and a content digest (see digest.ts); the executable +// half is attached via defineCheck. Check-set SELECTION is per subject kind +// (probe-inventory gap 7): this monorepo does not match the node template's +// file list, so the QC-19 definition carries a distinct file set for the +// `monorepo` subject kind and the policy selects checks per kind. + +export function defineCheck( + spec: CheckDefinitionSpec, + evaluate: (ctx: CheckContext) => Promise, +): CheckDefinition { + return { ...spec, definitionDigest: digestOfSpec(spec), evaluate }; +} + +async function fileExists(filePath: string): Promise { + try { + await access(filePath, constants.F_OK); + return true; + } catch { + return false; + } +} + +// ─── QC-19: downstream rails presence ──────────────────────────────────────── +// +// Typed absorption of the former presence-only `quality-rails check` loop in +// cli.ts. The scaffold-kind file lists below are carried over VERBATIM so the +// evaluator's typed verdicts are parity-equivalent with the presence loop on +// the same fixture; the `monorepo` list is new (per-subject check sets). + +const qc19Spec: CheckDefinitionSpec = { + id: 'qc-19-rails-files-present', + version: '1.0.0', + canonicalCheck: 'QC-19', + description: + 'The subject still carries its quality-rails files. Typed absorption of the former presence-only check loop; presence is necessary, not sufficient (RI-N4).', + appliesTo: ['node', 'python', 'rust', 'monorepo', 'unknown'], + params: { + expectedFilesByKind: { + node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + monorepo: [ + '.husky/pre-commit', + '.husky/pre-push', + 'eslint.config.mjs', + '.prettierrc', + '.lintstagedrc', + ], + unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'], + }, + }, +}; + +async function evaluateQc19(ctx: CheckContext): Promise { + const byKind = ctx.params['expectedFilesByKind'] as Record | undefined; + if (byKind === undefined) { + return { status: 'error', reason: 'definition params missing expectedFilesByKind' }; + } + const expected = byKind[ctx.subject.kind]; + if (expected === undefined) { + // Fail-closed: an undefined file set for a declared subject kind is a + // definition gap, never a green outcome. + return { + status: 'blocked', + reason: `no expected-file set defined for subject kind '${ctx.subject.kind}'`, + }; + } + + const missing: string[] = []; + for (const relativePath of expected) { + if (!(await fileExists(resolve(ctx.subject.path, relativePath)))) { + missing.push(relativePath); + } + } + if (missing.length > 0) { + return { + status: 'failed', + reason: `missing rails files (${ctx.subject.kind}): ${missing.join(', ')}`, + }; + } + return { status: 'passed' }; +} + +// ─── QC-20: downstream enforcement verification (behavioral probe) ────────── +// +// The planted-commit behavioral probe (framework tools/quality/scripts/verify.sh) +// stays a THIN SHELL ADAPTER: the TS evaluator invokes it and OWNS the verdict +// parsing (RI-N4: grep-on-output verdict logic moves into the typed evaluator). +// Probe contract (verify.sh): exit 0 ⇔ every sub-probe passed, exit 1 ⇔ at +// least one sub-probe failed; sub-probe verdicts appear as `PASS:` / `FAIL:` +// marker lines and the script always prints a `Verification Summary` section. +// Any deviation from that contract (other exit codes, unparseable output, +// missing probe, process failure, timeout) is `error`/`blocked` — never +// `passed`. + +const qc20Spec: CheckDefinitionSpec = { + id: 'qc-20-enforcement-verify', + version: '1.0.0', + canonicalCheck: 'QC-20', + description: + 'The behavioral planted-commit probe runs against the subject and every sub-probe blocks as intended. The shell probe is a thin adapter; verdict parsing is owned by this evaluator.', + appliesTo: ['node', 'python', 'rust', 'unknown'], + params: { + command: 'bash', + timeoutMs: 120_000, + passMarker: 'PASS:', + failMarker: 'FAIL:', + summaryMarker: 'Verification Summary', + }, +}; + +function linesWith(text: string, marker: string): string[] { + return text + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.includes(marker)); +} + +async function evaluateQc20(ctx: CheckContext): Promise { + const rawProbePath = ctx.inputs['probePath']; + if (typeof rawProbePath !== 'string' || rawProbePath.trim().length === 0) { + return { + status: 'blocked', + reason: + 'missing input: probePath — the behavioral probe script must be provided (e.g. the framework verify.sh)', + }; + } + const probePath = isAbsolute(rawProbePath) + ? rawProbePath + : resolve(ctx.subject.path, rawProbePath); + if (!(await fileExists(probePath))) { + return { status: 'blocked', reason: `probe script not found: ${probePath}` }; + } + + const command = typeof ctx.params['command'] === 'string' ? ctx.params['command'] : 'bash'; + const timeoutMs = typeof ctx.params['timeoutMs'] === 'number' ? ctx.params['timeoutMs'] : 120_000; + const passMarker = + typeof ctx.params['passMarker'] === 'string' ? ctx.params['passMarker'] : 'PASS:'; + const failMarker = + typeof ctx.params['failMarker'] === 'string' ? ctx.params['failMarker'] : 'FAIL:'; + const summaryMarker = + typeof ctx.params['summaryMarker'] === 'string' + ? ctx.params['summaryMarker'] + : 'Verification Summary'; + + const outcome = await ctx.adapter.run({ + file: command, + args: [probePath], + cwd: ctx.subject.path, + timeoutMs, + }); + + if (!outcome.ok) { + // Process error or timeout: the probe never produced a trustworthy result. + return { + status: 'error', + reason: `probe process ${outcome.kind}: ${outcome.message}`, + }; + } + + const output = `${outcome.stdout}\n${outcome.stderr}`; + const failLines = linesWith(output, failMarker); + const passLines = linesWith(output, passMarker); + + if (outcome.exitCode === 0) { + // A green exit must be corroborated by a parseable green transcript: + // at least one pass marker, no fail markers, and the summary section. + if (passLines.length > 0 && failLines.length === 0 && output.includes(summaryMarker)) { + return { status: 'passed' }; + } + return { + status: 'error', + reason: `malformed probe output: exit 0 without a parseable pass transcript (${passLines.length} pass markers, ${failLines.length} fail markers, summary ${output.includes(summaryMarker) ? 'present' : 'absent'})`, + }; + } + + if (outcome.exitCode === 1) { + if (failLines.length === 0) { + return { + status: 'error', + reason: 'malformed probe output: exit 1 without parseable FAIL markers', + }; + } + return { + status: 'failed', + reason: `enforcement probe reported ${failLines.length} failing sub-probe(s): ${failLines.join(' | ')}`, + }; + } + + return { + status: 'error', + reason: `probe exited with unexpected code ${String(outcome.exitCode)} — outcome not interpretable`, + }; +} + +// ─── Per-subject check-set policy ─────────────────────────────────────────── +// +// Gap 7 of the probe inventory: check sets must be selected per subject, not +// one global list. Downstream scaffold kinds get the presence check plus the +// behavioral probe (QC-20 blocks until a probePath input is provided — an +// unverified subject can never evaluate green). The monorepo subject is this +// repository itself: its rails are the husky hooks + shared lint/format +// configs, covered by QC-19; the downstream planted-commit probe does not +// apply to it (this repo's own commit gates are QC-13/QC-14, outside this +// evaluator's owned checks). + +const checkSetPolicySpec: CheckSetPolicySpec = { + version: '1.0.0', + byKind: { + node: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'], + python: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'], + rust: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'], + unknown: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'], + monorepo: ['qc-19-rails-files-present'], + }, +}; + +export const CHECK_SET_POLICY: CheckSetPolicy = { + ...checkSetPolicySpec, + policyDigest: digestOfPolicy(checkSetPolicySpec), +}; + +export const QC_19_RAILS_FILES_PRESENT = defineCheck(qc19Spec, evaluateQc19); +export const QC_20_ENFORCEMENT_VERIFY = defineCheck(qc20Spec, evaluateQc20); + +/** Built-in check definitions, keyed by id. */ +export function builtInDefinitions(): CheckDefinition[] { + return [QC_19_RAILS_FILES_PRESENT, QC_20_ENFORCEMENT_VERIFY]; +} + +export function checkSetForKind( + kind: SubjectKind, + policy: CheckSetPolicy = CHECK_SET_POLICY, +): readonly string[] { + const selected = policy.byKind[kind]; + if (selected === undefined) { + // Fail-closed selection: an unknown kind yields an EMPTY set only to the + // caller; the runner treats an empty result list as `blocked`, never green. + return []; + } + return selected; +} diff --git a/packages/quality-rails/src/evaluator/digest.ts b/packages/quality-rails/src/evaluator/digest.ts new file mode 100644 index 00000000..dab0d9a6 --- /dev/null +++ b/packages/quality-rails/src/evaluator/digest.ts @@ -0,0 +1,45 @@ +import { createHash } from 'node:crypto'; + +import type { CheckDefinitionSpec, CheckSetPolicySpec } from './types.js'; + +// Deterministic JSON: object keys sorted at every level so two specs with the +// same content always produce the same bytes (and thus the same digest). +export function canonicalJson(value: unknown): string { + if (value === null || typeof value !== 'object') { + return JSON.stringify(value); + } + if (Array.isArray(value)) { + return `[${value.map((entry) => canonicalJson(entry)).join(',')}]`; + } + const record = value as Record; + const keys = Object.keys(record).sort(); + return `{${keys.map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`).join(',')}}`; +} + +/** sha256 over the canonical JSON of `value`. */ +export function digestContent(value: unknown): string { + return createHash('sha256').update(canonicalJson(value), 'utf8').digest('hex'); +} + +/** + * Content digest of a check definition: covers the declarative spec (id, + * version, canonical check, description, applicability, params) — everything a + * reviewer reasons about — while excluding the executable function object. + * Changing any covered field changes the digest, so a recorded digest always + * identifies exactly which definition content produced a verdict. + */ +export function digestOfSpec(spec: CheckDefinitionSpec): string { + return digestContent({ + id: spec.id, + version: spec.version, + canonicalCheck: spec.canonicalCheck, + description: spec.description, + appliesTo: spec.appliesTo, + params: spec.params, + }); +} + +/** Content digest of the per-subject check-set policy. */ +export function digestOfPolicy(spec: CheckSetPolicySpec): string { + return digestContent(spec); +} diff --git a/packages/quality-rails/src/evaluator/evaluator.spec.ts b/packages/quality-rails/src/evaluator/evaluator.spec.ts new file mode 100644 index 00000000..2ccb9a30 --- /dev/null +++ b/packages/quality-rails/src/evaluator/evaluator.spec.ts @@ -0,0 +1,525 @@ +import { mkdtemp, mkdir, writeFile, chmod } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { createSpawnProcessAdapter } from './adapter.js'; +import { + builtInDefinitions, + CHECK_SET_POLICY, + checkSetForKind, + defineCheck, + QC_19_RAILS_FILES_PRESENT, + QC_20_ENFORCEMENT_VERIFY, +} from './definitions.js'; +import { digestOfSpec } from './digest.js'; +import { aggregateState, evaluateSubject } from './runner.js'; +import type { + AdapterOutcome, + CheckDefinitionSpec, + CheckResult, + EvaluationReport, + ProcessAdapter, +} from './types.js'; + +// ─── helpers ──────────────────────────────────────────────────────────────── + +function firstResult(report: EvaluationReport): CheckResult { + const result = report.results[0]; + if (result === undefined) { + throw new Error('expected the report to contain at least one result'); + } + return result; +} + +async function makeTempDir(): Promise { + return mkdtemp(join(tmpdir(), 'qr-evaluator-')); +} + +async function writeProbeScript(dir: string, name: string, body: string): Promise { + const scriptPath = join(dir, name); + await writeFile(scriptPath, `${body}\n`, 'utf8'); + await chmod(scriptPath, 0o755); + return scriptPath; +} + +/** Adapter stub that always returns the given outcome (no real process). */ +function stubAdapter(outcome: AdapterOutcome): ProcessAdapter { + return { + run: async () => outcome, + }; +} + +// VERBATIM copy of the pre-absorption presence loop (former cli.ts +// expectedFilesForKind + fileExists loop). This is the PARITY ORACLE: the +// evaluator's typed QC-19 verdict must agree with what the absorbed check +// concluded on the same fixture. +const LEGACY_EXPECTED: Record<'node' | 'python' | 'rust' | 'unknown', string[]> = { + node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'], + unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'], +}; + +async function legacyPresenceLoop(projectPath: string, kind: keyof typeof LEGACY_EXPECTED) { + const missing: string[] = []; + for (const relativePath of LEGACY_EXPECTED[kind]) { + const fs = await import('node:fs/promises'); + try { + await fs.access(join(projectPath, relativePath)); + } catch { + missing.push(relativePath); + } + } + return missing; +} + +async function scaffoldFixture(kind: keyof typeof LEGACY_EXPECTED, skip: string[] = []) { + const dir = await makeTempDir(); + if (kind === 'node') { + await writeFile(join(dir, 'package.json'), '{}\n', 'utf8'); + } + if (kind === 'python') { + await writeFile(join(dir, 'pyproject.toml'), '[project]\n', 'utf8'); + } + if (kind === 'rust') { + await writeFile(join(dir, 'Cargo.toml'), '[package]\n', 'utf8'); + } + for (const relativePath of LEGACY_EXPECTED[kind]) { + if (skip.includes(relativePath)) continue; + await mkdir(join(dir, relativePath, '..'), { recursive: true }); + await writeFile(join(dir, relativePath), 'fixture\n', 'utf8'); + } + return dir; +} + +// ─── QC-19 parity: typed verdict == absorbed presence loop ────────────────── + +describe('QC-19 parity with the absorbed presence loop', () => { + const kinds: Array = ['node', 'python', 'rust', 'unknown']; + + it.each(kinds)('positive fixture (%s): loop said ok ⇒ evaluator passed', async (kind) => { + const dir = await scaffoldFixture(kind); + const oracleMissing = await legacyPresenceLoop(dir, kind); + expect(oracleMissing).toEqual([]); + + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_19_RAILS_FILES_PRESENT.id], + }); + const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id); + expect(result?.status).toBe('passed'); + expect(result?.reason).toBeUndefined(); + expect(report.state).toBe('passed'); + }); + + it.each(kinds)( + 'negative fixture (%s): loop listed missing ⇒ evaluator failed with them', + async (kind) => { + const all = LEGACY_EXPECTED[kind]; + const skip = all.slice(0, Math.max(1, all.length - 1)); // leave exactly 1 present + const dir = await scaffoldFixture(kind, skip); + const oracleMissing = await legacyPresenceLoop(dir, kind); + expect(oracleMissing.length).toBeGreaterThan(0); + + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_19_RAILS_FILES_PRESENT.id], + }); + const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id); + expect(result?.status).toBe('failed'); + expect(report.state).toBe('failed'); + for (const missingFile of oracleMissing) { + expect(result?.reason).toContain(missingFile); + } + // No false attribution: a present file must not be named in the reason. + const presentFile = all.find((file) => !skip.includes(file)); + if (presentFile !== undefined) { + expect(result?.reason).not.toContain(` ${presentFile},`); + } + }, + ); +}); + +// ─── per-subject check sets (inventory gap 7) ──────────────────────────────── + +describe('per-subject check sets', () => { + it('monorepo subject selects only QC-19 with the monorepo file set', async () => { + const dir = await makeTempDir(); + await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n - packages/*\n', 'utf8'); + for (const file of [ + '.husky/pre-commit', + '.husky/pre-push', + 'eslint.config.mjs', + '.prettierrc', + '.lintstagedrc', + ]) { + await mkdir(join(dir, file, '..'), { recursive: true }); + await writeFile(join(dir, file), 'fixture\n', 'utf8'); + } + + const report = await evaluateSubject({ subjectPath: dir }); + expect(report.subject.kind).toBe('monorepo'); + expect(report.results.map((r) => r.checkId)).toEqual(['qc-19-rails-files-present']); + expect(report.state).toBe('passed'); + }); + + it('a monorepo missing one of its rails files fails QC-19 (not the node list)', async () => { + const dir = await makeTempDir(); + await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n', 'utf8'); + const report = await evaluateSubject({ subjectPath: dir }); + const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id); + expect(result?.status).toBe('failed'); + expect(result?.reason).toContain('.husky/pre-commit'); + // The node-template list must NOT be applied to a monorepo subject. + expect(result?.reason).not.toContain('biome.json'); + }); + + it('the policy selects the behavioral probe for scaffold kinds but not monorepo', () => { + expect(checkSetForKind('node')).toContain(QC_20_ENFORCEMENT_VERIFY.id); + expect(checkSetForKind('unknown')).toContain(QC_20_ENFORCEMENT_VERIFY.id); + expect(checkSetForKind('monorepo')).not.toContain(QC_20_ENFORCEMENT_VERIFY.id); + expect(CHECK_SET_POLICY.version).toBe('1.0.0'); + }); +}); + +// ─── negative controls (the point of the card) ─────────────────────────────── + +describe('negative controls', () => { + it('unknown check id ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: ['qc-99-does-not-exist'], + }); + expect(report.results).toHaveLength(1); + const result = firstResult(report); + expect(result.status).toBe('error'); + expect(result.reason).toContain("unknown check id 'qc-99-does-not-exist'"); + expect(result.status === 'passed').toBe(false); + expect(report.state).toBe('error'); + }); + + it('missing subject (directory absent) ⇒ blocked for every check, never passed', async () => { + const report = await evaluateSubject({ + subjectPath: join(tmpdir(), `qr-evaluator-absent-${Date.now()}`), + }); + expect(report.results.length).toBeGreaterThan(0); + for (const result of report.results) { + expect(result.status).toBe('blocked'); + expect(result.reason).toContain('subject directory does not exist'); + } + expect(report.state).toBe('blocked'); + }); + + it('QC-20 without probePath input ⇒ blocked, never passed', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + }); + const result = firstResult(report); + expect(result.status).toBe('blocked'); + expect(result.reason).toContain('missing input: probePath'); + }); + + it('QC-20 with a nonexistent probe script ⇒ blocked, never passed', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'no-such-probe.sh') } }, + }); + expect(firstResult(report).status).toBe('blocked'); + expect(firstResult(report).reason).toContain('probe script not found'); + }); + + it('adapter process error (spawn failure) ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } }, + adapter: stubAdapter({ ok: false, kind: 'spawn-error', message: 'ENOENT bash' }), + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('probe process spawn-error'); + expect(firstResult(report).status === 'passed').toBe(false); + }); + + it('adapter timeout ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } }, + adapter: stubAdapter({ ok: false, kind: 'timeout', message: 'timed out after 120000ms' }), + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('probe process timeout'); + }); + + it('probe exit 1 with parseable FAIL markers ⇒ failed (interpretably red), never passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript( + dir, + 'probe-fail.sh', + `echo "Test 1: ..."\necho "❌ FAIL: Type errors NOT blocked"\necho "Verification Summary"\nexit 1`, + ); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('failed'); + expect(firstResult(report).reason).toContain('FAIL: Type errors NOT blocked'); + expect(report.state).toBe('failed'); + }); + + it('probe exit 1 WITHOUT parseable FAIL markers ⇒ malformed ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript(dir, 'probe-mute.sh', `echo "nothing to see"\nexit 1`); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('malformed probe output'); + }); + + it('probe exit 0 without a parseable pass transcript ⇒ malformed ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript(dir, 'probe-lie.sh', `echo "all good"\nexit 0`); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('malformed probe output'); + expect(firstResult(report).reason).toContain('exit 0'); + }); + + it('probe exit 0 WITH fail markers ⇒ contradictory transcript ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript( + dir, + 'probe-contradict.sh', + `echo "✅ PASS: one"\necho "❌ FAIL: two"\necho "Verification Summary"\nexit 0`, + ); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('error'); + }); + + it('probe unexpected exit code (7) ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript(dir, 'probe-crash.sh', `echo "boom"\nexit 7`); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('unexpected code 7'); + }); + + it('check implementation throwing ⇒ error, never passed', async () => { + const dir = await scaffoldFixture('node'); + const spec: CheckDefinitionSpec = { + id: 'test-throws', + version: '1.0.0', + canonicalCheck: 'QC-TEST', + description: 'sabotage-shaped definition that always throws', + appliesTo: ['node'], + params: {}, + }; + const throwing = defineCheck(spec, async () => { + throw new Error('kaboom'); + }); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: ['test-throws'], + definitions: [throwing], + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('kaboom'); + }); + + it('non-passed verdict without a reason ⇒ upgraded to error, never an unqualified skip', async () => { + const dir = await scaffoldFixture('node'); + const spec: CheckDefinitionSpec = { + id: 'test-silent-fail', + version: '1.0.0', + canonicalCheck: 'QC-TEST', + description: 'returns failed without a reason', + appliesTo: ['node'], + params: {}, + }; + const silent = defineCheck(spec, async () => ({ status: 'failed' })); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: ['test-silent-fail'], + definitions: [silent], + }); + expect(firstResult(report).status).toBe('error'); + expect(firstResult(report).reason).toContain('without a reason'); + }); + + it('empty result list aggregates to blocked, never passed', () => { + expect(aggregateState([])).toBe('blocked'); + }); +}); + +// ─── QC-20 parity: typed verdict == shell probe's own conclusion ──────────── + +describe('QC-20 parity with the shell probe contract', () => { + it('green transcript (exit 0) ⇒ evaluator passed', async () => { + const dir = await scaffoldFixture('node'); + const probe = await writeProbeScript( + dir, + 'probe-pass.sh', + [ + 'echo "✅ PASS: Type errors blocked"', + 'echo "✅ PASS: any types blocked"', + 'echo "✅ PASS: Lint errors blocked"', + 'echo "Verification Summary"', + 'echo "✅ Passed: 3"', + 'exit 0', + ].join('\n'), + ); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: probe } }, + adapter: createSpawnProcessAdapter(), + }); + expect(firstResult(report).status).toBe('passed'); + expect(report.state).toBe('passed'); + }); + + it('the REAL framework verify.sh on a non-git subject concludes failed (exit 1) ⇒ evaluator failed', async () => { + // Real-probe parity: verify.sh without a git repo cannot block planted + // commits, exits 1 with FAIL markers — the evaluator must record exactly + // `failed` with those markers, matching the probe's own conclusion. + const realProbe = fileURLToPath( + new URL('../../../mosaic/framework/tools/quality/scripts/verify.sh', import.meta.url), + ); + const dir = await makeTempDir(); // not a git repository, no hooks + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_20_ENFORCEMENT_VERIFY.id], + inputs: { 'qc-20-enforcement-verify': { probePath: realProbe } }, + adapter: createSpawnProcessAdapter(), + }); + const result = firstResult(report); + expect(result.status).toBe('failed'); + expect(result.reason).toMatch(/FAIL:/); + expect(report.state).toBe('failed'); + }); +}); + +// ─── version / digest discipline ───────────────────────────────────────────── + +describe('versioned, digested check definitions', () => { + it('every verdict records the definition version that produced it', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ subjectPath: dir }); + for (const result of report.results) { + expect(result.checkVersion).toBe('1.0.0'); + } + expect(report.checkSetVersion).toBe(CHECK_SET_POLICY.version); + }); + + it('the report records each definition’s content digest', async () => { + const dir = await scaffoldFixture('node'); + const report = await evaluateSubject({ subjectPath: dir }); + expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe( + QC_19_RAILS_FILES_PRESENT.definitionDigest, + ); + expect(report.definitionDigests[QC_20_ENFORCEMENT_VERIFY.id]).toBe( + QC_20_ENFORCEMENT_VERIFY.definitionDigest, + ); + }); + + it('digests are stable for identical content', () => { + const spec = QC_19_RAILS_FILES_PRESENT; + expect(digestOfSpec(spec)).toBe(digestOfSpec(spec)); + }); + + it('changing a definition’s content changes its digest', () => { + const base = { ...QC_19_RAILS_FILES_PRESENT } as CheckDefinitionSpec; + const baseDigest = digestOfSpec(base); + + const changedParams: CheckDefinitionSpec = { + ...base, + params: { + expectedFilesByKind: { + ...(base.params['expectedFilesByKind'] as Record), + node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md', 'NEW.md'], + }, + }, + }; + expect(digestOfSpec(changedParams)).not.toBe(baseDigest); + + const changedVersion: CheckDefinitionSpec = { ...base, version: '1.1.0' }; + expect(digestOfSpec(changedVersion)).not.toBe(baseDigest); + }); + + it('a definition with changed content produces a different recorded digest and version', async () => { + const dir = await scaffoldFixture('node'); + const modified = defineCheck( + { ...QC_19_RAILS_FILES_PRESENT, version: '2.0.0' } as unknown as CheckDefinitionSpec, + async () => ({ status: 'passed' }), + ); + const report = await evaluateSubject({ + subjectPath: dir, + checkIds: [QC_19_RAILS_FILES_PRESENT.id], + definitions: [ + modified, + ...builtInDefinitions().filter((d) => d.id !== QC_19_RAILS_FILES_PRESENT.id), + ], + }); + const result = firstResult(report); + expect(result.checkVersion).toBe('2.0.0'); + expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe(modified.definitionDigest); + expect(modified.definitionDigest).not.toBe(QC_19_RAILS_FILES_PRESENT.definitionDigest); + }); +}); + +// ─── aggregate state ordering (MACP-style discipline) ─────────────────────── + +describe('aggregate state precedence', () => { + const result = (status: 'passed' | 'failed' | 'blocked' | 'error') => ({ + status, + checkId: 'x', + checkVersion: '1.0.0', + subject: '/tmp/x', + }); + + it('all passed (with not-applicable) ⇒ passed', () => { + expect( + aggregateState([ + result('passed'), + { ...result('passed'), status: 'not-applicable' as const }, + ]), + ).toBe('passed'); + }); + + it('error outranks blocked and failed; blocked outranks failed', () => { + expect(aggregateState([result('blocked'), result('error')])).toBe('error'); + expect(aggregateState([result('failed'), result('blocked')])).toBe('blocked'); + expect(aggregateState([result('passed'), result('failed')])).toBe('failed'); + }); +}); diff --git a/packages/quality-rails/src/evaluator/runner.ts b/packages/quality-rails/src/evaluator/runner.ts new file mode 100644 index 00000000..71e8e11f --- /dev/null +++ b/packages/quality-rails/src/evaluator/runner.ts @@ -0,0 +1,183 @@ +import { constants } from 'node:fs'; +import { access, stat } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; + +import { createSpawnProcessAdapter } from './adapter.js'; +import { builtInDefinitions, CHECK_SET_POLICY, checkSetForKind } from './definitions.js'; +import type { + AggregateState, + CheckResult, + CheckStatus, + EvaluateOptions, + EvaluationReport, + ProcessAdapter, + Subject, + SubjectKind, +} from './types.js'; +import { detectProjectKind } from '../detect.js'; + +async function pathExists(targetPath: string): Promise { + try { + await access(targetPath, constants.F_OK); + return true; + } catch { + return false; + } +} + +async function isDirectory(targetPath: string): Promise { + try { + return (await stat(targetPath)).isDirectory(); + } catch { + return false; + } +} + +/** + * Subject-kind detection for the evaluator. Extends the scaffold detection + * (detect.ts) with the `monorepo` kind: a pnpm workspace is this repository's + * own subject shape and carries a different rails file set (probe-inventory + * gap 7 — check sets are per subject, not one global file list). + */ +export async function detectSubjectKind(subjectPath: string): Promise { + if (await pathExists(join(subjectPath, 'pnpm-workspace.yaml'))) { + return 'monorepo'; + } + const kind = await detectProjectKind(subjectPath); + return kind; +} + +/** + * Aggregate state, MACP-style discipline: `passed` only when at least one + * check produced a verdict AND every verdict is `passed` or an explicitly + * qualified `not-applicable`. Precedence is fail-closed: error > blocked > + * failed > passed; an empty result list aggregates to `blocked`. + */ +export function aggregateState(results: readonly CheckResult[]): AggregateState { + if (results.length === 0) { + return 'blocked'; + } + const has = (status: CheckStatus): boolean => results.some((result) => result.status === status); + if (has('error')) { + return 'error'; + } + if (has('blocked')) { + return 'blocked'; + } + if (has('failed')) { + return 'failed'; + } + return 'passed'; +} + +function reasonFrom(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +/** + * Evaluate one subject against a set of checks, producing typed verdicts. + * + * Fail-closed invariants (RI-N4): + * - unknown check id → `error` (never passed) + * - subject directory absent → every verdict `blocked` + * - check implementation threw → `error` + * - non-passed without a reason → `error` (no unqualified skips) + * - check not applicable → `not-applicable` WITH a reason + */ +export async function evaluateSubject(options: EvaluateOptions): Promise { + const subjectPath = resolve(options.subjectPath); + const subject: Subject = { + path: subjectPath, + kind: await detectSubjectKind(subjectPath), + }; + + const definitions = options.definitions ?? builtInDefinitions(); + const byId = new Map(definitions.map((definition) => [definition.id, definition])); + const requested = options.checkIds ?? checkSetForKind(subject.kind); + const adapter: ProcessAdapter = options.adapter ?? createSpawnProcessAdapter(); + + const results: CheckResult[] = []; + const definitionDigests: Record = {}; + + for (const checkId of requested) { + const definition = byId.get(checkId); + if (definition === undefined) { + const known = definitions.map((entry) => entry.id).join(', '); + results.push({ + status: 'error', + checkId, + checkVersion: 'unknown', + subject: subjectPath, + reason: `unknown check id '${checkId}' — no registered definition (known: ${known})`, + }); + continue; + } + + definitionDigests[checkId] = definition.definitionDigest; + + if (!(await isDirectory(subjectPath))) { + results.push({ + status: 'blocked', + checkId, + checkVersion: definition.version, + subject: subjectPath, + reason: `subject directory does not exist: ${subjectPath}`, + }); + continue; + } + + if (!definition.appliesTo.includes(subject.kind)) { + results.push({ + status: 'not-applicable', + checkId, + checkVersion: definition.version, + subject: subjectPath, + reason: `check '${checkId}' does not apply to subject kind '${subject.kind}'`, + }); + continue; + } + + try { + const inputs = options.inputs?.[checkId] ?? {}; + const outcome = await definition.evaluate({ + subject, + params: definition.params, + inputs, + adapter, + }); + if (outcome.status !== 'passed' && (outcome.reason === undefined || outcome.reason === '')) { + results.push({ + status: 'error', + checkId, + checkVersion: definition.version, + subject: subjectPath, + reason: `check returned status '${outcome.status}' without a reason — treated as error`, + }); + continue; + } + results.push({ + status: outcome.status, + checkId, + checkVersion: definition.version, + subject: subjectPath, + reason: outcome.reason, + }); + } catch (error) { + results.push({ + status: 'error', + checkId, + checkVersion: definition.version, + subject: subjectPath, + reason: `check implementation threw: ${reasonFrom(error)}`, + }); + } + } + + return { + subject, + results, + definitionDigests, + checkSetVersion: CHECK_SET_POLICY.version, + state: aggregateState(results), + }; +} diff --git a/packages/quality-rails/src/evaluator/types.ts b/packages/quality-rails/src/evaluator/types.ts new file mode 100644 index 00000000..8edd4080 --- /dev/null +++ b/packages/quality-rails/src/evaluator/types.ts @@ -0,0 +1,148 @@ +// Evaluator core types — RI-N4 (card RI-3-002, SDLC-D-037 second half). +// +// The quality-rails evaluator is the SOLE authoritative producer of check +// verdicts for the checks it owns. Every verdict is typed and fail-closed: +// missing implementations, missing inputs, unknown check ids, process errors, +// timeouts, and malformed adapter output can never become `passed` or an +// unqualified skip — they surface as `blocked` or `error` with a reason. +// (Vocabulary mirrors MACP's GateStatus discipline from packages/macp.) + +/** + * Typed verdict for a single check execution. + * + * - `passed` — the check really ran and its condition held. + * - `failed` — the check really ran and its condition did NOT hold. + * - `blocked` — the check could not run at all (missing subject, missing + * input). Never a green outcome. + * - `error` — the check attempted to run but its outcome cannot be trusted + * (unknown check id, implementation threw, process error, timeout, malformed + * adapter output). Never a green outcome. + * - `not-applicable` — the check definition explicitly declares it does not + * apply to this subject (a qualified skip, always with a reason). + */ +export type CheckStatus = 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable'; + +/** Aggregate outcome, MACP-style: `passed` only when every result is green. */ +export type AggregateState = 'passed' | 'failed' | 'blocked' | 'error'; + +/** Kinds of subjects the evaluator can assess. */ +export type SubjectKind = 'node' | 'python' | 'rust' | 'monorepo' | 'unknown'; + +/** + * A single check verdict. This is the canonical result shape: `status`, + * `checkId`, `checkVersion`, `subject`, `reason`. `reason` is REQUIRED + * (enforced by the runner) for every status other than `passed`. + */ +export interface CheckResult { + status: CheckStatus; + checkId: string; + checkVersion: string; + subject: string; + reason?: string; +} + +/** The project being evaluated. */ +export interface Subject { + /** Absolute path. */ + path: string; + kind: SubjectKind; +} + +/** + * The data half of a check definition. Definitions live as DATA with a version + * and a content digest (see `digestOfSpec`); the executable half is attached + * separately so the digest covers only reviewable, declarative content. + */ +export interface CheckDefinitionSpec { + /** Stable id, e.g. `qc-19-rails-files-present`. */ + id: string; + /** Semver of this definition's data+semantics. */ + version: string; + /** Canonical check id from docs/release-integrity/probe-inventory.md (QC-n). */ + canonicalCheck: string; + description: string; + /** Subject kinds this check can assess (others yield `not-applicable`). */ + appliesTo: readonly SubjectKind[]; + /** Declarative parameters (file lists, markers, timeouts) — digest-covered. */ + params: Record; +} + +/** A fully assembled check definition: spec + digest + implementation. */ +export interface CheckDefinition extends CheckDefinitionSpec { + /** sha256 content digest of the spec (canonical JSON projection). */ + definitionDigest: string; + evaluate(ctx: CheckContext): Promise; +} + +/** What a check implementation returns; the runner stamps id/version/subject. */ +export interface CheckOutcome { + status: CheckStatus; + reason?: string; +} + +/** Caller-provided inputs for one check invocation (e.g. the QC-20 probe path). */ +export type CheckInputs = Record; + +/** Everything a check implementation may use. */ +export interface CheckContext { + subject: Subject; + params: Record; + inputs: CheckInputs; + adapter: ProcessAdapter; +} + +/** Outcome of running a shell probe through the thin process adapter. */ +export type AdapterOutcome = + | { ok: true; exitCode: number | null; stdout: string; stderr: string } + | { ok: false; kind: 'spawn-error' | 'timeout'; message: string }; + +/** Request for the process adapter. */ +export interface AdapterRequest { + file: string; + args: string[]; + cwd: string; + timeoutMs: number; +} + +/** + * Thin process adapter: runs a command, owns NO verdict logic. Verdict parsing + * always lives in the check implementation (TS), never in the shell probe. + */ +export interface ProcessAdapter { + run(request: AdapterRequest): Promise; +} + +/** Per-subject-kind check-set selection policy (versioned and digested). */ +export interface CheckSetPolicySpec { + version: string; + byKind: Record; +} + +export interface CheckSetPolicy extends CheckSetPolicySpec { + /** sha256 content digest of the policy spec. */ + policyDigest: string; +} + +/** Full typed evaluation report for one subject. */ +export interface EvaluationReport { + subject: Subject; + results: CheckResult[]; + /** checkId → content digest of the definition that produced the verdicts. */ + definitionDigests: Record; + /** Version of the check-set policy used for subject selection. */ + checkSetVersion: string; + state: AggregateState; +} + +/** Options for `evaluateSubject`. */ +export interface EvaluateOptions { + subjectPath: string; + /** Restrict to these check ids; defaults to the subject kind's check set. */ + checkIds?: string[]; + /** Per-check inputs, keyed by check id (e.g. `{ 'qc-20-enforcement-verify': { probePath } }`). */ + inputs?: Record; + /** Replace the built-in definitions (tests / future batches). */ + definitions?: CheckDefinition[]; + /** Inject a process adapter (tests / instrumentation). */ + adapter?: ProcessAdapter; +} diff --git a/packages/quality-rails/src/index.ts b/packages/quality-rails/src/index.ts index 3f61eb1f..c0c927ef 100644 --- a/packages/quality-rails/src/index.ts +++ b/packages/quality-rails/src/index.ts @@ -3,3 +3,35 @@ export * from './detect.js'; export * from './scaffolder.js'; export * from './templates.js'; export * from './types.js'; + +// RI-N4 evaluator (card RI-3-002): the public, programmatic entry points. +export { + builtInDefinitions, + CHECK_SET_POLICY, + checkSetForKind, + defineCheck, + QC_19_RAILS_FILES_PRESENT, + QC_20_ENFORCEMENT_VERIFY, +} from './evaluator/definitions.js'; +export { canonicalJson, digestContent, digestOfPolicy, digestOfSpec } from './evaluator/digest.js'; +export { createSpawnProcessAdapter } from './evaluator/adapter.js'; +export { aggregateState, detectSubjectKind, evaluateSubject } from './evaluator/runner.js'; +export type { + AdapterOutcome, + AdapterRequest, + AggregateState, + CheckContext, + CheckDefinition, + CheckDefinitionSpec, + CheckInputs, + CheckOutcome, + CheckResult, + CheckSetPolicy, + CheckSetPolicySpec, + CheckStatus, + EvaluateOptions, + EvaluationReport, + ProcessAdapter, + Subject, + SubjectKind, +} from './evaluator/types.js'; diff --git a/scripts/verify-release.mjs b/scripts/verify-release.mjs index f0b518f4..ca7ab100 100644 --- a/scripts/verify-release.mjs +++ b/scripts/verify-release.mjs @@ -22,6 +22,12 @@ // format | format | pnpm format:check // test | test | pnpm test // build | publish.yml build | pnpm build +// quality-rails | (canonical-only) | the TS quality-rails evaluator +// | | (RI-N4, QC-19 monorepo subject). Like +// | | `build`, this stage has no ci.yml +// | | mirror; it is implemented by +// | | importing the evaluator CLI rather +// | | than duplicating its presence logic. // // Caller-provided prerequisites (kept at the pipeline level — see the comments // in .woodpecker/ci.yml): `bash` + `rsync` for the guard stages, `openssl` and @@ -90,6 +96,15 @@ export const STAGES = [ name: 'build', commands: ['pnpm build'], }, + { + // RI-N4 (QC-19, card RI-3-002): the typed quality-rails evaluator, invoked + // as the implementation of the check it owns instead of a duplicated + // presence loop here. Canonical-only stage (no ci.yml mirror — same shape + // as `build`); runs AFTER build so the evaluator's dist/ exists. Subject + // is this repository (`.` → monorepo subject kind, per-subject check set). + name: 'quality-rails', + commands: ['node packages/quality-rails/dist/cli.js quality-rails evaluate --project .'], + }, ]; export function stageByName(name) { diff --git a/scripts/verify-release.test.mjs b/scripts/verify-release.test.mjs index 49e9ff5b..677c65af 100644 --- a/scripts/verify-release.test.mjs +++ b/scripts/verify-release.test.mjs @@ -4,7 +4,7 @@ import { createRequire } from 'node:module'; import path from 'node:path'; import test from 'node:test'; -import { STAGES } from './verify-release.mjs'; +import { STAGES, stageByName } from './verify-release.mjs'; // SDLC-D-034 checkout invariant: publication in .woodpecker/publish.yml is // bound to exact-commit terminal verification. This suite parses the real @@ -229,10 +229,21 @@ steps: function assertStagesMirrorCi(stages, ci) { const canonical = Object.fromEntries(stages.map((stage) => [stage.name, stage.commands])); - // The complete mandatory set, in gate order. + // The complete mandatory set, in gate order. `quality-rails` is a + // canonical-only stage (RI-N4, QC-19): like `build`, it has no ci.yml + // mirror to match — its contract is asserted separately below. assert.deepEqual( stages.map((stage) => stage.name), - ['sanitization', 'upgrade-guard', 'typecheck', 'lint', 'format', 'test', 'build'], + [ + 'sanitization', + 'upgrade-guard', + 'typecheck', + 'lint', + 'format', + 'test', + 'build', + 'quality-rails', + ], ); // Guard stages: ci.yml commands minus its `apk add` environment prep must be @@ -301,3 +312,38 @@ test('the root package.json exposes verify:release as the canonical command', as const packageJson = JSON.parse(await readFile(path.join(process.cwd(), 'package.json'), 'utf8')); assert.match(packageJson.scripts['verify:release'], /scripts\/verify-release\.mjs/); }); + +// RI-N4 (card RI-3-002): the `quality-rails` stage must route through the TS +// evaluator instead of duplicating its presence logic inline. The evaluator +// owns QC-19; this file keeps that delegation honest. +function assertEvaluatorStage(stage) { + assert.ok(stage, 'canonical stages must include a quality-rails stage'); + assert.ok(Array.isArray(stage.commands) && stage.commands.length > 0); + for (const command of stage.commands) { + assert.match( + command, + /packages\/quality-rails\/dist\/cli\.js.*quality-rails evaluate/, + `quality-rails stage command must invoke the evaluator CLI, got: '${command}'`, + ); + } +} + +test('the quality-rails stage invokes the evaluator rather than duplicating its logic', () => { + assertEvaluatorStage(stageByName('quality-rails')); +}); + +test('a quality-rails stage that re-implements presence logic inline fails the checker', () => { + // Negative control: replacing the evaluator invocation with an inline + // `test -f` presence loop is exactly the duplication RI-N4 forbids — the + // checker must go red on it. + const duplicated = { + name: 'quality-rails', + commands: ['test -f .husky/pre-commit && test -f .husky/pre-push'], + }; + assert.throws(() => assertEvaluatorStage(duplicated), /must invoke the evaluator CLI/); +}); + +test('a quality-rails stage that silently drops the evaluator command fails the checker', () => { + const empty = { name: 'quality-rails', commands: [] }; + assert.throws(() => assertEvaluatorStage(empty), /commands/); +});