Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 fargo
|
68279d61a1 | ||
|
fargo
|
771127d3cd | ||
|
fargo
|
ae95e7b853 | ||
|
fargo
|
367cb27591 | ||
|
fargo
|
18905d69e6 | ||
  jarvisandfargo
|
ff45f7b5d0 | ||
|
jarvisandfargo
|
64350892e7 | ||
 jason.woltjeandjarvis
|
6e9df3c640 |
@@ -138,9 +138,9 @@ mosaic brain tasks
|
||||
mosaic brain conversations
|
||||
|
||||
# Agent forge pipeline
|
||||
mosaic forge run
|
||||
mosaic forge run [--simulate] # fails closed (FORGE_NO_EXECUTOR) with no executor wired; --simulate for typed simulated runs
|
||||
mosaic forge status
|
||||
mosaic forge resume
|
||||
mosaic forge resume [--simulate] # same fail-closed rule as forge run
|
||||
mosaic forge personas
|
||||
|
||||
# Structured logging
|
||||
|
||||
@@ -15,18 +15,18 @@
|
||||
> `done` requires: repo quality gates green, independent review recorded, terminal-green CI on
|
||||
> the PR head, squash merge to `next`, and acceptance evidence in notes.
|
||||
|
||||
| id | status | description | issue | agent | repo | branch | depends_on | estimate | notes |
|
||||
| -------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | ----------------- | --------------------------------- | ---------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| RI-0-001 | in-progress | Bootstrap: issue #1275, PRD section, this DAG, scratchpad (docs only) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-mission-bootstrap | — | 6K | |
|
||||
| RI-1-001 | in-progress | RI-N1: canonical terminal verification command + publish-pipeline exact-commit gate (every publish step depends on verify; commit identity check; fail closed) | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-publish-gate | RI-0-001 | 25K | |
|
||||
| RI-1-002 | not-started | RI-N1 negative control: checked-in tests proving a broken mandatory check blocks every publish step and that DAG edges cannot be bypassed | #1275 | pi-glm-5.3 | mosaicstack/stack | test/ri-050-publish-gate-negative | RI-1-001 | 12K | |
|
||||
| RI-2-001 | in-progress | RI-N2 (Forge): remove stub-executor false success; `--simulate` typed `simulated` results that satisfy nothing; literal-`true` gates and echo-review replaced with real gates or typed waiting-for-authority | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-forge-fail-closed | RI-0-001 | 20K | Independent review APPROVED 2026-08-17 (Gitea review 172 on PR #1278, head 99b8f6ea; reviewing seat fargo — recorded under shared host principal mos-dt-0, provenance correction posted by fred; wrapper gap filed by fred). Executed at head: forge tests 116/116, lint green, typecheck green after building macp dist (minimal-install artifact, not a defect), workspace typecheck 45/45, no external type consumers of the changed interfaces. CI red = known lane-wide fleet-test failure only, carries no information about this change (fred, log-content analysis, pipelines 2456-2458). Non-blocking finding: README L141-143 + skills/mosaic-forge/SKILL.md document bare forge run/resume, which now fails closed — fast-follow docs touch. Merge queued behind #1270. |
|
||||
| RI-2-002 | in-progress | RI-N2 (MACP): gate runner fails closed on empty commands, stub executors, and unimplemented CI-provider gates unless explicit simulate; typed capability failures | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-macp-fail-closed | RI-0-001 | 15K | |
|
||||
| RI-3-001 | not-started | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | |
|
||||
| RI-3-002 | not-started | RI-N4: TS evaluator absorbs effective shell probes; typed results (passed/failed/blocked/error/not-applicable) with versioned digested check definitions; shell commands become thin adapters; contract/parity/negative-control tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-qr-evaluator | RI-3-001 | 30K | |
|
||||
| RI-4-001 | in-progress | RI-N3: one PRD application service — `mission --plan` persists mission↔PRD linkage (ids/versions/selected requirements); `mosaic prdy` routes through the service or becomes a named import/export adapter; Markdown is a labeled generated view; explicit conflict-aware import | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-prd-authority | RI-0-001 | 35K | |
|
||||
| RI-5-001 | not-started | RI-N5: typed freshness states (current/stale/partial/unknown/unavailable); no failed-fetch-renders-empty; stale derived verdicts → unknown; mutations disabled when stale; failure-matrix tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-web-stale-safety | RI-0-001 | 25K | |
|
||||
| RI-V-001 | not-started | Final verification + release evidence: all cards verified merged, negative controls demonstrated, real `next` publish run green on exact commit, evidence pack recorded | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-release-evidence | RI-1-002, RI-2-001, RI-2-002, RI-3-002, RI-4-001, RI-5-001 | 10K | |
|
||||
| id | status | description | issue | agent | repo | branch | depends_on | estimate | notes |
|
||||
| -------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | ----------------- | --------------------------------- | ---------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| RI-0-001 | done | Bootstrap: issue #1275, PRD section, this DAG, scratchpad (docs only) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-mission-bootstrap | — | 6K | PR #1276 (head 758659dd): docs-only, CI green (2475). Review requested from fargo. Merges first (no publish run). |
|
||||
| RI-1-001 | done | RI-N1: canonical terminal verification command + publish-pipeline exact-commit gate (every publish step depends on verify; commit identity check; fail closed) | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-publish-gate | RI-0-001 | 25K | PR #1277 (head 46784c8d): CI GREEN at head after serialized retry (pipeline 2476, 2026-08-18) - earlier red was CI-agent contention (web SPA timeouts under concurrent pipelines), not code. Review requested from fargo at pinned head (comms 20260818T021025Z). |
|
||||
| RI-1-002 | done | RI-N1 negative control: checked-in tests proving a broken mandatory check blocks every publish step and that DAG edges cannot be bypassed | #1275 | pi-glm-5.3 | mosaicstack/stack | test/ri-050-publish-gate-negative | RI-1-001 | 12K | |
|
||||
| RI-2-001 | done | RI-N2 (Forge): remove stub-executor false success; `--simulate` typed `simulated` results that satisfy nothing; literal-`true` gates and echo-review replaced with real gates or typed waiting-for-authority | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-forge-fail-closed | RI-0-001 | 20K | Independent review APPROVED 2026-08-17 (Gitea review 172 on PR #1278, head 99b8f6ea; reviewing seat fargo — recorded under shared host principal mos-dt-0, provenance correction posted by fred; wrapper gap filed by fred). Executed at head: forge tests 116/116, lint green, typecheck green after building macp dist (minimal-install artifact, not a defect), workspace typecheck 45/45, no external type consumers of the changed interfaces. CI red = known lane-wide fleet-test failure only, carries no information about this change (fred, log-content analysis, pipelines 2456-2458). Non-blocking finding: README L141-143 + skills/mosaic-forge/SKILL.md document bare forge run/resume, which now fails closed — fast-follow docs touch. Merge queued behind #1270. UPDATE 2026-08-18: #1270 merged; CI GREEN at head 4917df1f via serialized retry (pipeline 2477) - root cause of prior reds was CI-agent contention (web SPA timeouts under concurrent pipelines), superseding the fleet-test-failure theory. |
|
||||
| RI-2-002 | done | RI-N2 (MACP): gate runner fails closed on empty commands, stub executors, and unimplemented CI-provider gates unless explicit simulate; typed capability failures | #1275 | pi-glm-5.3 | mosaicstack/stack | fix/ri-050-macp-fail-closed | RI-0-001 | 15K | PR #1293 (head 2097379e): CI green (pipeline 2465), independent review APPROVED (Gitea review 173, jarvis seat, 2026-08-17) - macp 109/109 verified at head. Merge queued behind #1276/#1277/#1278. |
|
||||
| RI-3-001 | done | RI-N4: complete probe inventory mapping every TS and shell quality-rail check to one canonical check with disposition (preserve/strengthen/retire, each named) | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-qr-probe-inventory | RI-0-001 | 12K | PR #1302 (head e06a47fac591): CI green (2484), independent review APPROVED (Gitea review 187, fargo seat, 2026-08-18) — 54 rows / 21 canonical checks / dispositions 43-2-9-0 verified by row-count and code spot-checks. Merged by fargo at pinned head. |
|
||||
| RI-3-002 | not-started | RI-N4: TS evaluator absorbs effective shell probes; typed results (passed/failed/blocked/error/not-applicable) with versioned digested check definitions; shell commands become thin adapters; contract/parity/negative-control tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-qr-evaluator | RI-3-001 | 30K | |
|
||||
| RI-4-001 | in-progress | RI-N3: one PRD application service — `mission --plan` persists mission↔PRD linkage (ids/versions/selected requirements); `mosaic prdy` routes through the service or becomes a named import/export adapter; Markdown is a labeled generated view; explicit conflict-aware import | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-prd-authority | RI-0-001 | 35K | PR #1294 (head 8d258e1d): CI green (pipeline 2466), independent review APPROVED (Gitea review 174, jarvis seat, 2026-08-17) - prdy 20/20 + command specs 9/9 at head. Merge queued behind #1276/#1277/#1278. |
|
||||
| RI-5-001 | done | RI-N5: typed freshness states (current/stale/partial/unknown/unavailable); no failed-fetch-renders-empty; stale derived verdicts → unknown; mutations disabled when stale; failure-matrix tests | #1275 | pi-glm-5.3 | mosaicstack/stack | feat/ri-050-web-stale-safety | RI-0-001 | 25K | |
|
||||
| RI-V-001 | not-started | Final verification + release evidence: all cards verified merged, negative controls demonstrated, real `next` publish run green on exact commit, evidence pack recorded | #1275 | pi-glm-5.3 | mosaicstack/stack | docs/ri-050-release-evidence | RI-1-002, RI-2-001, RI-2-002, RI-3-002, RI-4-001, RI-5-001 | 10K | |
|
||||
|
||||
## Dispatch waves (max 2 parallel workers)
|
||||
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
# Quality-Rails Probe Inventory — RI-3-001
|
||||
|
||||
- **Task:** RI-3-001 (SDLC-D-037 first half; PRD § Release Integrity Workstream, RI-N4)
|
||||
- **Date:** 2026-08-18
|
||||
- **Base:** `origin/next` @ `8199261c` (branch `docs/ri-050-qr-probe-inventory`)
|
||||
- **Follow-up:** RI-3-002 consumes the dispositions here when building the single TS evaluator.
|
||||
|
||||
## 0. Scope and method
|
||||
|
||||
Every mechanism in this repository that verifies a quality, integrity, safety, or release
|
||||
property — TypeScript checks, shell probes, pipeline steps, git hooks, and installer-side
|
||||
assertions — gets one row. Each row's "what it actually verifies" was written from the
|
||||
probe's **code**, not its name or docs. Framework tool unit/regression suites (git wrappers,
|
||||
wake, tmux, orchestrator, …) are treated as one enforcement surface (`test:framework-shell`)
|
||||
because they test tool behavior rather than repo quality; their wiring integrity is itself
|
||||
guarded by `check-test-enumeration.sh`, and the quality-relevant members are rowed
|
||||
individually.
|
||||
|
||||
**Kinds:** `ts` (TypeScript/Node check), `shell` (bash/python probe), `pipeline-step`
|
||||
(exists only inside a Woodpecker pipeline).
|
||||
|
||||
**Enforcement points:** `local` (operator-invoked), `pre-commit`, `pre-push`,
|
||||
`CI ci.yml#<step>`, `publish.yml#<step>` (CI on push to main/next), `turbo <task>`,
|
||||
`agent-runtime` (framework hooks on an agent host), `installer` (host install path),
|
||||
`unwired`.
|
||||
|
||||
**Dispositions** (recommendations for RI-3-002): `preserve` (keep as-is; already the
|
||||
canonical or a correct guard-of-the-guard), `strengthen` (keep, but a concrete gap must
|
||||
close — usually absorption into the TS evaluator), `strengthen (review)` (viable retirement
|
||||
candidate once the evaluator absorbs it; do not retire yet). Note: RI-N4 requires that
|
||||
effective shell probes be **absorbed before** their independent paths retire — no row here
|
||||
is marked `retire` because no absorption exists yet.
|
||||
|
||||
## 1. Inventory
|
||||
|
||||
### 1.1 Repo-level gate tasks (pnpm / turbo)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------- | ------------------------------------------------------------------------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `pnpm preflight` (checkout preflight) | `scripts/preflight.mjs` | ts | Six gate binaries (eslint, husky, prettier, tsc, turbo, vitest) exist and are executable in `node_modules/.bin` (exit 42 if not); no stale `.mosaic-test-work/web-build.lock` (exit 43); `apps/web/.next` is a real directory (not a symlink), every entry owned by the current uid, and its `.mosaic-source-hash` fingerprint + `.mosaic-symlink-manifest` hash match the certified build written by `scripts/build-web.mjs` | `pre-push`; inside `pnpm typecheck` (→ `CI ci.yml#typecheck`, verify-release `typecheck` stage) | QC-1 Checkout integrity | preserve | Blocks a poisoned/stale generated `.next` from faking a green typecheck (the five-month-stale-`.next` class); trust chain is self-contained per-checkout. |
|
||||
| `pnpm typecheck` | root `package.json` → `turbo run typecheck` | ts | Per-package `tsc --noEmit` (all 20 packages); turbo `typecheck` depends on `^build`, so package builds must succeed first; prefixed by checkout preflight | `CI ci.yml#typecheck`; `pre-push`; verify-release `typecheck` stage; `turbo typecheck` | QC-2 Workspace typecheck | preserve | The single workspace-wide type gate; CI and hooks invoke the same task, no divergent checklist. |
|
||||
| `pnpm lint` | root `package.json` → `turbo run lint` | ts | Per-package `eslint src` under root `eslint.config.mjs` (ignores `dist`, `.next`, `framework/**`, etc.) | `CI ci.yml#lint`; `pre-push`; verify-release `lint` stage; `turbo lint` | QC-3 Workspace lint | preserve | Same-task invocation from every surface; no second lint definition. |
|
||||
| `pnpm format:check` | root `package.json` → `prettier --check` | ts | Prettier parse/format equality over `**/*.{ts,tsx,js,jsx,json,md}` minus `.prettierignore` (generated trees, `docs/scratchpads/`, venvs, …) | `CI ci.yml#format`; `pre-push`; verify-release `format` stage | QC-4 Format check | preserve | Single formatter, single ignore list, enforced identically everywhere. |
|
||||
| `pnpm test` | root `package.json` `test` = `test:checkout` && `turbo run test` && `test:installer` | ts | (a) `node --test scripts/*.test.mjs` — checkout-tool units; (b) per-package `vitest run` (mosaic appends the 47-command `test:framework-shell` chain); (c) `tools/install-next-lane.test.sh`; turbo `test` declares DB env vars and depends on `^build` | `CI ci.yml#test` (with `DATABASE_URL` + `db:migrate` first); verify-release `test` stage; `turbo test` | QC-5 Test suite execution | preserve | One composed test command; the chain property (any link red ⇒ step red) is the gate. |
|
||||
| `pnpm build` | root `package.json` → `turbo run build` | ts | Per-package build (`tsc`/Next) with `^build` dependency and `dist/**` outputs | `publish.yml#build`; verify-release `build` stage; `turbo build` | QC-6 Workspace build | preserve | Publish artifacts derive from the same build task CI verifies. |
|
||||
|
||||
### 1.2 Framework quality shell probes (`packages/mosaic/framework/tools/quality/`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------------- | ----------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| Sanitization gate | `scripts/verify-sanitized.sh` | shell | Built-in self-test first (planted identity/structural/YAML+service fixtures; exit 2 if the regexes or extension coverage break), then: (1) identity denylist grep (`jarvis\|jason\|woltje\|brain.woltje.com\|/home/jwoltje\|\bPDA\b`) over all shipped text files **including** `examples/`; (2) structural grep for private `$HOME/src` defaults in shipped scripts **excluding** `examples/`. Any hit ⇒ exit 1 | `CI ci.yml#sanitization`; verify-release `sanitization` stage | QC-7 Framework sanitization | preserve | Labeled one-time regression guard with a self-test that prevents silent no-op; correctly scoped (identity vs structural) and documented as not a general PII detector. |
|
||||
| Resident-context budget | `scripts/check-resident-budget.sh` (+ `--self-test`) | shell | Self-test of the comparator, then `wc -l` vs per-file ceilings (CONSTITUTION 120, AGENTS 120, each RUNTIME.md 90); missing file ⇒ fail; over ceiling ⇒ exit 1 | `CI ci.yml#sanitization` (both modes); verify-release `sanitization` stage | QC-8 Resident-context budget | preserve | Caps the container (lines), never the wording — the deliberate anti-drift design (DESIGN §7); CI-enforceable half only, by design. |
|
||||
| Test-membership enumeration guard (#1017) | `scripts/check-test-enumeration.sh` + `test-enumeration-exclusions.txt` | shell | Parses surface S1 (`packages/mosaic` `test:framework-shell` via JSON+shlex) and S2 (every `framework/tools/\*.sh | .py`token in`ci.yml`, comment lines stripped); population = `_test_.sh`under`framework/tools`; FAILS on: suite-shaped file on disk neither enumerated nor signed-excluded; surface naming a path missing on disk (both directions); exclusion without reason / stale / outside population / contradicting enumeration. Proves **naming, not reachability** (stated in-file) | `CI ci.yml#sanitization` (direct line); link [0] of `test:framework-shell` (thus `CI ci.yml#test`); verify-release `sanitization` stage | QC-9 Test-membership enumeration | preserve | Makes silent under-run impossible; invoked from both surfaces it audits so severing the chain cannot silence it. |
|
||||
| Enumeration-guard needles | `scripts/test-check-test-enumeration.sh` | shell | Needle/control fixtures driven through `--root`: every promised failure mode must trip the guard **on its own words**, plus controls that must pass (null-case defense); covers commented-out ci.yml lines (F1) and line-range parsing (n2b) | `test:framework-shell` → `CI ci.yml#test`; verify-release `test` stage | QC-9 Test-membership enumeration | preserve | Guard-of-the-guard with both polarities; same canonical check by design. |
|
||||
| Upgrade manifest guard (#791 HARD GATE) | `scripts/test-upgrade-manifest-guard.sh` | shell | Keep-mode `install.sh` upgrade against seeded throwaway `MOSAIC_HOME`: every operator sentinel — including an **unanticipated** one — survives byte-identical with unchanged mtime; framework files still update; retired framework files pruned; matrix run with rsync present AND absent (keep path must be rsync-independent); fail-closed matrix (empty/operator-only/malformed/missing manifest aborts loudly, operator files untouched); operator secret never appears in installer output | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | The operator-data hard gate for the `mosaic update` path; negative controls are load-bearing and documented. |
|
||||
| Upgrade rollback gate (#791 B1) | `scripts/test-upgrade-rollback.sh` | shell | Mid-sync failure (PATH-shadowing `cp` shim) must trigger snapshot restore: restore message fires, corrupted file restored, target byte-identical to pre-upgrade; control installer with `set -E` stripped must NOT roll back (proves errtrace is load-bearing); plus signal/exit-guard controls | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Proves the rollback trap actually fires; the `-E`-stripped control keeps Part A honest. |
|
||||
| Durable-snapshot gate (#791 PR2) | `scripts/test-upgrade-durable-snapshot.sh` | shell | Pre-update snapshot taken before any mutation (0700/0600 perms, secret never logged, retention-pruned); post-sync verify net restores operator files a manifest bug lets the sync touch; CWE-59 symlink-leaf guard proven with a portable cp shim in both polarities (write-through-link must not happen); v1→v2 migration semantics (intended `bin/` removal not healed) | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Covers tampering and leak vectors the manifest guard cannot see; the shim rationale (busybox vs GNU cp) is documented in-file. |
|
||||
| Install migration matrix (v2→v3) | `scripts/test-install-migration.sh` | shell | Fixture matrix running the real installer with `MOSAIC_SYNC_ONLY=1`: fresh install seeds + stamps version 3; legacy user-edited AGENTS overwritten with `.pre-constitution.bak` preserved (and idempotent); tuned STANDARDS overwritten; operator files (SOUL, credentials) preserved. Mirrors the TS suite `packages/mosaic/src/config/file-adapter.test.ts` — both installers must behave identically | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Pins the shell/TS installer parity contract; removal would orphan that parity requirement. |
|
||||
| Enforcement verification probe (bash) | `scripts/verify.sh` | shell | Attempts **real commits** in the target repo: planted type error must produce a commit blocked with `error`; planted `any` must trip `no-explicit-any`; planted lint error must trip `prettier`; gitleaks binary must exist (3a) and detect a planted AWS key via `gitleaks git --pre-commit --staged --redact` (3b). Verdicts are output-grep matches on hook stderr | `local` via installed `mosaic-quality-verify` on scaffolded target projects; **not run in this repo's CI** | QC-20 Downstream enforcement verification | strengthen (review) | Mechanism is genuinely behavioral (stronger than file presence) but verdict logic is grep-on-output and it is unwired here; absorb as the evaluator's enforcement-probe check (the RI-N4 evaluator invokes it or reimplements it) before retiring the shell path. |
|
||||
| Enforcement verification probe (PowerShell) | `scripts/verify.ps1` | shell | Windows port of `verify.sh`: same planted-commit tests with `$output -match` matching; no gitleaks self-test parity beyond the same checks | `local` (Windows operator); no Windows CI runner exists | QC-20 Downstream enforcement verification | strengthen (review) | A hand-maintained twin of `verify.sh` with no CI coverage — exactly the drift shape the single evaluator removes; retire after the TS evaluator owns the probe. |
|
||||
| Quality template installer (bash) | `scripts/install.sh` | shell | Copies template files (`.husky/pre-commit` incl. mandatory gitleaks, `.lintstagedrc.js`, `.eslintrc.js`, `tsconfig.json`, `.woodpecker.yml`, `.gitleaks.toml`) into a target project; **warns** (does not verify) about `package.json` snippet merge; no post-condition check | `local` / via `mosaic-quality-apply` | QC-21 Downstream rails scaffolding | strengthen (review) | Duplicates the TS `quality-rails init` scaffolder for a different template set; converging on one scaffolder (with post-scaffold verification) is prerequisite to retiring this path. |
|
||||
| Quality template installer (PowerShell) | `scripts/install.ps1` | shell | Windows twin of the template copy above | `local` (Windows operator) | QC-21 Downstream rails scaffolding | strengthen (review) | Same twin-drift risk as `verify.ps1`; no runner exercises it. |
|
||||
| `mosaic-quality-verify` adapter | `framework/tools/_scripts/mosaic-quality-verify` | shell | Thin adapter: validates target dir exists, asserts `verify.sh` present+executable, `cd` target, exec it. No verdict logic of its own | `local` (installed framework bin) | QC-20 Downstream enforcement verification | preserve | Already the thin-adapter shape RI-N4 prescribes for shell surfaces. |
|
||||
| `mosaic-quality-apply` adapter | `framework/tools/_scripts/mosaic-quality-apply` | shell | Thin adapter: arg validation then exec of quality `install.sh --template … --target …` | `local` (installed framework bin) | QC-21 Downstream rails scaffolding | preserve | Thin adapter, no separate verdict; disposition follows its target script's convergence. |
|
||||
| Roster schema regression | `scripts/test-roster-schema.py` | shell | jsonschema `Draft202012Validator` over `fleet/roster.schema.json` with valid/invalid connector-kind fixtures (tmux/discord/matrix conditional fields) | **unwired** — not on S1 or S2, not signed-excluded; also outside the enumeration guard's `*.sh` population, so the guard cannot see it | QC-5 Test suite execution | strengthen (review) | A real regression suite that currently runs nowhere; wire it into a CI surface or sign an exclusion — leaving it invisible re-arms the exact gap #1017 closed. |
|
||||
| Framework shell chain (S1) | `packages/mosaic/package.json` `test:framework-shell` | shell | 47-command `&&` chain: enumeration guard + needles, 14 lease-broker/mutator-gate python unitests, `check-runtime-launches.py`, and ~30 framework-tool shell suites (git wrappers, wake, woodpecker, tmux, glpi, orchestrator, `_scripts`). Quality-relevant members rowed separately below | `turbo test` → `CI ci.yml#test`; verify-release `test` stage | QC-5 Test suite execution | preserve | The chain is the execution surface the enumeration guard audits; known residuals: a failing link stops later suites (measured in #1270 — suites after position 44 had not run), and the guard proves naming, not reachability. |
|
||||
|
||||
### 1.3 Framework runtime hooks and their harnesses (agent-host enforcement)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------- | ----------------------------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------- | --------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| QA edit hook seam | `framework/tools/qa/qa-hook-stdin.sh` (+ `qa-hook-handler.sh`) | shell | PostToolUse stdin hook: extracts edited file from the tool JSON (jq or grep fallback), skips non-JS/TS, then the deps-preflight gate — exits 1 with the legible sentinel `deps not installed — run pnpm install` when `node_modules/.bin` is missing/empty (the #856 false-red class); the downstream handler only files QA remediation **report templates** (no verification logic) | `agent-runtime` (framework `runtime/claude/settings.json` PostToolUse); never CI | QC-16 Agent-runtime edit-time checks | strengthen (review) | The sentinel gate is real enforcement; the handler's report-filing adds no verdict and its name promises more than the code does — evaluator absorption should keep the sentinel, drop the report theater. |
|
||||
| Typecheck-on-edit hook | `framework/tools/qa/typecheck-hook.sh` | shell | PostToolUse: for edited `.ts/.tsx`, finds nearest `tsconfig.json` and runs `tsc --noEmit`, surfacing errors nonzero to the agent immediately | `agent-runtime` (framework `runtime/claude/settings.json` PostToolUse) | QC-16 Agent-runtime edit-time checks | strengthen (review) | Edit-time duplicate of QC-2 with independent invocation logic; keep behavior, converge invocation through the evaluator adapter. |
|
||||
| Deps-preflight harness | `framework/tools/qa/test-deps-preflight.sh` | shell | Five assertions against the seam incl. a documented RED control (raw `not found`), sentinel behavior for missing and empty `.bin`, and no-false-positive once populated | `test:framework-shell` → `CI ci.yml#test` | QC-16 Agent-runtime edit-time checks | preserve | Guard-of-the-check with a red control; keeps the sentinel from regressing. |
|
||||
| Prompt-helper RCE regression | `framework/tools/_scripts/test-mosaic-init-rce.sh` | shell | Sources the prompt helpers and proves a literal `$(touch /tmp/pwned)` answer round-trips verbatim and never executes (no `/tmp/pwned` created) | `test:framework-shell` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Cheap, load-bearing security regression on the installer's input path. |
|
||||
| Install-ordering harness (#869 C2) | `framework/tools/_scripts/test-install-ordering-guard.sh` | shell | Drives `mosaic-link-runtime-assets` with a fake `mosaic` on PATH: probe ok ⇒ settings copied + exit 0; probe fail ⇒ exit 1 with degraded outcome but all other runtime files still copied; `--allow-inactive-enforcement` forwarded; no-mosaic-on-PATH ⇒ python3 fallback strips enforcement hooks and exits 1; fallback + flag ⇒ wires as-is, exit 0 | `test:framework-shell` → `CI ci.yml#test` | QC-17 Lease-enforcement wiring safety | preserve | Exercises the shell wiring seam independently of the TS guard's own spec suite (complementary coverage, by design). |
|
||||
| Fleet-transport harness (#1240) | `framework/tools/_scripts/test-fleet-transport-check.sh` | shell | Extracts the shipped `check_fleet_transport`/`fleet_declared_transport` functions **from the shipped scripts** (fails loud if extraction yields nothing) and drives both implementations (mosaic-doctor + `tools/install.sh`) from one case table | `test:framework-shell` → `CI ci.yml#test` | QC-18 Operator-host drift audit | preserve | The anti-drift harness for the one rule shipped twice; extraction-from-source keeps it from testing a stale copy. |
|
||||
| Terminal-green contract (RM-61/#1000) | `framework/tools/woodpecker/test-terminal-green-contract.sh` + `verify-terminal-green.py` | shell | Red-first fixtures: pipeline JSON variants (service failure, step failure, cancelled, etc.) must produce the correct terminal-green verdict; controls must pass | `test:framework-shell` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Keeps the CI-wait wrapper's green-detection honest; a false green here would poison every merge gate that trusts `pr-ci-wait.sh`. |
|
||||
| Lease-gate launch invariant | `framework/tools/lease-broker/check-runtime-launches.py` | shell | Scans production roots (`packages/`, `apps/`, `plugins/`, `tools/`) across sh/py/ts/yaml suffixes for Claude/Pi process launches **outside** the lease gate; allowlist-based; fails CI on violation | `test:framework-shell` → `CI ci.yml#test` | QC-15 Lease-gate architecture invariant | preserve | The only architectural "no ungated launches" rail; grep+allowlist is the right cost/benefit for this invariant. |
|
||||
|
||||
### 1.4 TypeScript quality logic (`@mosaicstack/quality-rails` + mosaic CLI)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `quality-rails check` | `packages/quality-rails/src/cli.ts` (`mosaic quality-rails check --project`) | ts | **Expected-file presence only**: loops `expectedFilesForKind` (node: `.eslintrc`, `biome.json`, `.githooks/pre-commit`, `PR-CHECKLIST.md`; python: `pyproject.toml`+hooks+checklist; rust: `rustfmt.toml`+…) and exits 1 listing missing paths. Does not execute any linter, formatter, hook, or scanner | `local` (operator CLI); **no CI wiring in this repo** | QC-19 Downstream rails presence check | strengthen | This is the RI-N4 evaluator seed. Today presence ≠ parity (explicitly called out by RI-N4): it must grow typed verdicts (`passed/failed/blocked/error/not-applicable`), check versioning/subject/reason, digested definitions, and absorb the effective shell probes (QC-20 first). |
|
||||
| `quality-rails doctor` | `packages/quality-rails/src/cli.ts` | ts | Same presence data as `check`, printed with ok/missing lines; **cannot fail** (no nonzero exit on missing files) | `local` (operator CLI) | QC-19 Downstream rails presence check | strengthen | A doctor that cannot fail is advisory; fold into `check` (or return typed states) when the evaluator lands. |
|
||||
| `quality-rails init` | `packages/quality-rails/src/cli.ts` + `scaffolder.ts`/`templates.ts` | ts | Scaffolds rails files per detected kind/profile (linters/formatters lists are advisory strings; hooks flag always true); writes files, prints follow-ups — no post-condition verification | `local` (operator CLI) | QC-21 Downstream rails scaffolding | strengthen (review) | Second scaffolding path alongside quality `install.sh` (§1.2); converge on one with post-scaffold verification before retiring either. |
|
||||
| Lease activation probe (#869 C1, hidden) | `packages/mosaic/src/commands/lease-activation-probe.ts` | ts | Real capability probe, not file presence: resolves the installed mosaic CLI and requires it to advertise the exact `{name, version}` activation contract; all deps injectable; registered as hidden CLI command and consumed by C2/C5 | `local` (hidden CLI + consumed by C2/C5); spec-tested via `lease-activation-probe.spec.ts` in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | The versioned-contract probe is precisely the fail-closed capability check RI-N2 generalizes; already typed and injectable. |
|
||||
| Install-ordering guard (#869 C2, hidden) | `packages/mosaic/src/commands/install-ordering-guard.ts` | ts | Decides whether enforcement hook entries are written into the `~/.claude/settings.json` the framework reseed ships: not activatable ⇒ strip hooks + nonzero loud outcome (default); explicit per-invocation `--allow-inactive-enforcement` opt-out wires-with-warning. Never touches the runtime gate's own fail-closed behavior | `installer` (framework reseed via `mosaic-link-runtime-assets`); spec + shell harness coverage in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | Correct default-deny with an explicit, non-env opt-out; test-locked from both the TS and shell sides. |
|
||||
| Lease doctor check (#869 C5) | `packages/mosaic/src/commands/lease-doctor-check.ts` | ts | Combines hook-wiring detection in `~/.claude/settings.json` with C1 activatable and C3 broker-supervisor health: wired ∧ (¬activatable ∨ ¬healthy) ⇒ loud `[ERROR]` that forces `mosaic doctor` exit 1 regardless of the bash audit's own exit | `local` (inside `mosaic doctor`); spec coverage in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | Closes the "bricked host looks green" hole; cannot be masked by the bash script — that composition is the point. |
|
||||
| `mosaic doctor` (framework drift audit) | `packages/mosaic/src/commands/launch.ts` (`doctor`) + `framework/tools/_scripts/mosaic-doctor` | shell+ts | Bash audit of the installed framework home: ~40 expected files/dirs present; runtime files are copies (not symlinks) matching source (`cmp`) or composed runtime-contract markers; hard-gates block present in AGENTS.md; sequential-thinking MCP configured; fleet transport binary present per roster (warn); legacy symlink trees gone; skills synced — **warn-based, exit 1 only with `--fail-on-warn`**, plus C5's forced error | `local` (operator audit) | QC-18 Operator-host drift audit | preserve | Host-state audit CI cannot see (user files by design, DESIGN §7); advisory exit is the documented contract — do not silently change it. |
|
||||
| `mosaic gateway doctor` | `packages/mosaic/src/commands/gateway-doctor.ts` | ts | Probes per-service health (PostgreSQL, Valkey, pgvector) via `@mosaicstack/storage`, reports tier and JSON; exit 1 only when at least one **required** service fails (yellow stays 0) | `local` (operator) | QC-18 Operator-host drift audit | preserve | Service health with correct red/yellow exit semantics; JSON mode exists for scripting. |
|
||||
| `mosaic gateway verify` | `packages/mosaic/src/commands/gateway/verify.ts` | ts | Post-install liveness: daemon meta via HTTP with retries, admin token on file, bootstrap endpoint reachable; aggregated pass/fail | `local`; consumed by `tools/e2e-install-test.sh` | QC-18 Operator-host drift audit | preserve | The first-run proof the installer E2E relies on; retry-aware so startup races don't false-red. |
|
||||
| `mosaic fleet doctor` | `packages/mosaic/src/commands/fleet-reconciler-command.ts` | ts | Classifies local roster-owned drift (no mutation) from the parsed v2 roster | `local` (operator) | QC-18 Operator-host drift audit | preserve | Dry-run classification is the correct non-mutating audit shape. |
|
||||
|
||||
### 1.5 Git hooks (developer machine)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------- | --------------------------------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| Pre-commit staged hygiene | `.husky/pre-commit` → `npx lint-staged` (`.lintstagedrc`) | shell | On staged files only: `prettier --write` + `eslint --fix` for ts/tsx/js/jsx; `prettier --write` for json/md/yaml/yml. **Mutating** (fixes and re-stages); commit blocks only if a fixer itself fails | `pre-commit` (every local commit; hooks activated by `install-hooks.mjs` via `core.hooksPath .husky/_`) | QC-13 Staged-change hygiene | preserve | Correct scoped fast gate; note it auto-fixes rather than rejects (deliberate). Gap: no secret scan here — see §3. |
|
||||
| Pre-push gate | `.husky/pre-push` | shell | `pnpm preflight && pnpm typecheck && pnpm lint && pnpm format:check` (no test run — documented in AGENTS.md) | `pre-push` | QC-14 Pre-push gate | preserve | Composes QC-1..4 exactly as specified in AGENTS.md; tests intentionally left to CI. |
|
||||
| Hook installer | `scripts/install-hooks.mjs` (`pnpm prepare`) | ts | Stages husky hooks into a scratch repo first, asserts husky produced its `h` shim, quarantines incomplete previous sets, verifies idempotence via full directory snapshot comparison, then sets `core.hooksPath`; skips cleanly with `HUSKY=0` or no git | `installer` (runs on `pnpm install`) | QC-13 Staged-change hygiene | preserve | Self-verifying wiring for the hook gates — a corrupted half-install cannot silently disable them. |
|
||||
|
||||
### 1.6 CI pipeline steps (`.woodpecker/`)
|
||||
|
||||
Step-to-probe mapping for container steps: `ci.yml#sanitization` = QC-7+QC-8+QC-9 (rows §1.2, plus `apk add bash` env prep); `ci.yml#upgrade-guard` = QC-10 (rows §1.2, plus `apk add rsync`); `ci.yml#typecheck`/`#lint`/`#format`/`#test` = QC-2/3/4/5 (rows §1.1). Rows below are mechanisms that exist only in a pipeline.
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| -------------------------------------- | -------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| Frozen install | `ci.yml#install` | pipeline-step | `pnpm install --frozen-lockfile --prefer-offline` against the baked ci-base store — lockfile supply integrity; a drifted lockfile fails the build before any gate runs | `CI ci.yml#install` | QC-1 Checkout integrity | preserve | Lockfile-pinned dep resolution is the supply-chain floor under every later gate. |
|
||||
| Test-step readiness prelude | `ci.yml#test` prologue | pipeline-step | Installs pinned `@earendil-works/[email protected]` (Invariant R suite requires the real binary) + openssl; waits up to 60×1s on `pg_isready` for the `ci-postgres` service and fails fast if it never comes up; runs `db:migrate` before tests | `CI ci.yml#test` | QC-5 Test suite execution | preserve | Fail-fast environment preconditions — a missing service produces a legible failure, not a wall of red tests. |
|
||||
| Publish verify step (pending RI-1-001) | `publish.yml#verify` (branch `feat/ri-050-publish-gate` @ `46784c8d`, not yet on next) | pipeline-step | (a) Commit identity: fails closed if `CI_COMMIT_SHA` empty, `git rev-parse HEAD` empty, or the two differ; (b) runs the canonical `pnpm verify:release`. **Every publish effect depends on this step; it carries no path filter** | `publish.yml#verify` | QC-11 Terminal release verification | preserve | The RI-N1 exact-commit binding; until it merges, publish steps on next depend on `build` only (see §3 gap 1). |
|
||||
| Publish error classification | `publish.yml#publish-npm` | pipeline-step | Publishes `@mosaicstack/*` (minus web) and classifies outcome: success, or the **only tolerated failure** = already-published (EPUBLISHCONFLICT / "cannot publish over" / "previously published"); explicit fatal on npm `E404/E401/ENEEDAUTH/ECONNREFUSED/ETIMEDOUT/ENOTFOUND` and on any unrecognized failure (replacing the old ` | | echo` that hid a registry 404) | `publish.yml#publish-npm` (main/tags, path-filtered on `packages/**`) | QC-12 Publish-effect integrity | preserve | Converts silent publish fall-on-floor into loud failure; allowlist-of-one error tolerance is the right shape. |
|
||||
| Next-lane publish assertions | `publish.yml#publish-next-npm` | pipeline-step | Guards: branch must be `next`, `CI_PIPELINE_NUMBER` required; registry dist-tags JSON must be usable; walks all manifests, strictly parses stable semver, rewrites `X.Y.(Z+1)-next.<N>`; publishes with `--tag next` (never latest); post-publish asserts `npm view @mosaicstack/mosaic@next` resolves to the exact expected version | `publish.yml#publish-next-npm` (push/manual on next) | QC-12 Publish-effect integrity | preserve | Durable prerelease lane with end-to-end resolution proof — the published artifact is verified, not assumed. |
|
||||
| Image destination policy | `publish.yml#build-gateway` / `#build-appservice` / `#build-web` | pipeline-step | Kaniko builds with destination policy: `next` ⇒ sha-tag only (fatal if a tag event sneaks in); `main` ⇒ sha + `latest`; tag events ⇒ sha + `<tag>`; anything else fatal. Path filters only skip **effects**, never the verify step | `publish.yml#build-*` | QC-12 Publish-effect integrity | preserve | Fail-closed tagging matrix; the exclude-list default-safe design keeps stale images impossible. |
|
||||
|
||||
Adjacent pipeline surface (not a probe): `.woodpecker/ci-image.yml` rebuilds the ci-base image on `pnpm-lock.yaml`/`Dockerfile.ci` change with an immutable `lock-<hash>` tag; pipelines consume `:latest`. Recorded for completeness — no code-quality property is checked.
|
||||
|
||||
### 1.7 Root installer tooling (`tools/`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| --------------------------- | --------------------------------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Next-lane installer test | `tools/install-next-lane.test.sh` (`pnpm test:installer`) | shell | Drives `tools/install.sh --next` with faked `node`/`npm` binaries (no network): Node 20 must be rejected; installs must pin **exact** versions (mutable `@next` forbidden); fast path must not unexpectedly fall back to source; gateway-install failure takes the documented fallback | `turbo`-external tail of `pnpm test` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Hermetic (shimmed) regression net for the installer lane; runs as part of the standard test command. |
|
||||
| Clean-container install E2E | `tools/e2e-install-test.sh` | shell | Full first-run flow in a node:22-alpine container: `install.sh --yes` → `mosaic wizard` (non-interactive) → `mosaic gateway install` → `mosaic gateway verify` exit check (with EXPECTED-SKIP if the installed CLI predates `gateway verify`); skips gracefully without Docker | `local` (manual; requires Docker); **not wired in CI** | QC-5 Test suite execution | strengthen (review) | The only end-to-end proof of the install→verify path; currently operator-initiated only — wire into a periodic/manual CI lane or sign its exclusion explicitly. |
|
||||
| Host installer advisories | `tools/install.sh` (`--check`; `check_fleet_transport`) | shell | `--check` = version comparison only, no install; `check_fleet_transport` warns (non-blocking, by design — tmux is the fleet's dependency, not mosaic's) when the roster-declared transport binary is absent, naming exactly what it blocks; PATH-persistence warnings | `installer` (operator-run) | QC-18 Operator-host drift audit | preserve | Advisory-by-design warnings; the parallel doctor check is drift-tested by §1.3's harness. |
|
||||
|
||||
### 1.8 Pending workstream additions (branch `feat/ri-050-publish-gate` @ `46784c8d`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------- | ---------------------------------------------------- | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------- | ----------- | ---------------------------------------------------------------------------------------------- |
|
||||
| Canonical terminal verification | `scripts/verify-release.mjs` (`pnpm verify:release`) | ts | One command replaying the full mandatory set as stages — sanitization, upgrade-guard, typecheck (incl. preflight), lint, format, test, build — mirroring `ci.yml` step-for-step; fail-fast on first failing command; requires `bash`+`rsync` on PATH; `--stage <name>` for wiring smoke-tests only | `publish.yml#verify` (pending); `local` (`pnpm verify:release`) | QC-11 Terminal release verification | preserve | The RI-N1 canonical command — CI and publication share one semantic checklist by construction. |
|
||||
| Verify-parity contract test | `scripts/verify-release.test.mjs` | ts | Parses the real `ci.yml`/`publish.yml`: stage table must match ci.yml step-for-step; every publish-effect step (name `publish*` or image-pushing) must transitively depend on `verify`; commit-identity assertion must be present; `verify` must carry no path filter | `test:checkout` → `CI ci.yml#test` (once merged) | QC-11 Terminal release verification | preserve | Guard-of-the-guard at checkout time — the two surfaces cannot drift apart silently. |
|
||||
|
||||
## 2. Canonical check set
|
||||
|
||||
The deduplicated checks every row above maps onto. IDs are stable for RI-3-002 to consume.
|
||||
|
||||
- **QC-1 Checkout integrity.** Owns: the checkout can run its gates — frozen-lockfile dependency resolution, required gate binaries present, no stale build lock, and the `apps/web/.next` generated-state trust chain (real directory, uid ownership, certified source fingerprint, certified symlink manifest). Implemented by `scripts/preflight.mjs` + frozen install steps.
|
||||
- **QC-2 Workspace typecheck.** Owns workspace-wide TypeScript soundness: per-package `tsc --noEmit` over built dependencies (`turbo typecheck`). The single definition invoked by CI, pre-push, and terminal verification.
|
||||
- **QC-3 Workspace lint.** Owns static-analysis policy: per-package ESLint under the root config. One config, one task, every surface.
|
||||
- **QC-4 Format check.** Owns formatting uniformity: Prettier check with the repo ignore list. (The pre-commit variant additionally fixes; the verdict form is this check.)
|
||||
- **QC-5 Test suite execution.** Owns execution of all test surfaces: checkout script units (`node --test`), per-package Vitest suites (including the framework shell chain and its python unitests), the installer-lane shim test, and — once wired — `test-roster-schema.py` and container E2E. Also owns guards-of-the-gate that live inside the chain (terminal-green contract, RCE regression).
|
||||
- **QC-6 Workspace build.** Owns artifact buildability: `turbo build` producing the artifacts publication consumes.
|
||||
- **QC-7 Framework sanitization.** Owns the open-source guarantee for the shipped framework package: no operator-identity tokens anywhere (examples included), no private `$HOME` defaults in shipped scripts, with a self-test that keeps the regexes honest.
|
||||
- **QC-8 Resident-context budget.** Owns the line-count ceilings on framework files injected into every agent's context (Constitution, dispatcher, RUNTIME.md slices) — the CI-enforceable half of the resident-prompt budget.
|
||||
- **QC-9 Test-membership enumeration.** Owns the property that no test suite can silently fall out of CI: disk population vs parsed enumeration surfaces, both-directions staleness, and signed exclusions with reasons. Includes its needle/control harness.
|
||||
- **QC-10 Upgrade/install safety.** Owns the #791 family: operator-path byte-identity across keep-mode upgrades (manifest guard), mid-failure rollback (errtrace-proven), durable pre-update snapshot + verify net + CWE-59 leaf guard, and the v2→v3 migration matrix with shell/TS parity.
|
||||
- **QC-11 Terminal release verification.** Owns the RI-N1 exact-commit binding: commit-identity assertion plus one canonical command (`pnpm verify:release`) replaying the complete mandatory set, with every publish effect depending on it; plus the checkout-time parity/DAG contract test that keeps pipeline and command in sync.
|
||||
- **QC-12 Publish-effect integrity.** Owns publication correctness: npm publish error classification (only already-published tolerated), next-lane versioning and post-publish resolution proof, and image destination/tag policy.
|
||||
- **QC-13 Staged-change hygiene.** Owns commit-time hygiene on staged files (prettier/eslint fix-and-restage) and the self-verifying hook wiring that guarantees the gates are actually installed.
|
||||
- **QC-14 Pre-push gate.** Owns the local push composition: preflight + typecheck + lint + format:check (tests deliberately deferred to CI).
|
||||
- **QC-15 Lease-gate architecture invariant.** Owns "no ungated runtime launches in production code": the scan + allowlist over `packages/`, `apps/`, `plugins/`, `tools/`.
|
||||
- **QC-16 Agent-runtime edit-time checks.** Owns edit-time feedback on agent hosts: the deps-preflight legibility sentinel and typecheck-on-edit, plus their regression harnesses.
|
||||
- **QC-17 Lease-enforcement wiring safety.** Owns the #869 C1/C2/C5 trio: activation capability probe (versioned contract), enforcement-hook wiring gate (default-deny with explicit opt-out), and the doctor check that surfaces a bricked host — with their shell/TS harnesses.
|
||||
- **QC-18 Operator-host drift audit.** Owns host-state health CI cannot see: `mosaic doctor` drift audit (+ fleet transport, both implementations), `fleet doctor` roster classification, `gateway doctor`/`gateway verify` service health, and installer advisories. Advisory exits are part of the contract.
|
||||
- **QC-19 Downstream rails presence check.** Owns "does a scaffolded project still carry its rails files" — today the TS `quality-rails check/doctor` presence loop; per RI-N4 this is the seed that must become the typed evaluator (presence alone is explicitly not parity).
|
||||
- **QC-20 Downstream enforcement verification.** Owns "do the rails actually block" on scaffolded projects: the behavioral planted-commit probe (type error, `any`, lint, gitleaks secret) currently in `verify.sh`/`verify.ps1` behind the `mosaic-quality-verify` adapter.
|
||||
- **QC-21 Downstream rails scaffolding.** Owns putting rails files into a target project: the shell template installer (+ PowerShell twin) and the TS `quality-rails init` scaffolder — currently two paths that must converge.
|
||||
|
||||
## 3. Coverage gaps
|
||||
|
||||
Enforced nowhere but implied, or named in docs/tooling but not wired:
|
||||
|
||||
1. **Publication not yet bound to verification on `next`.** At this base (`8199261c`), `publish.yml` publish steps depend on `build` only; the `verify` step and `scripts/verify-release.mjs` exist on `feat/ri-050-publish-gate` (`46784c8d`) but are not merged. Until RI-1-001 lands, AC-RI-1's negative control cannot hold on the real pipeline.
|
||||
2. **Playwright E2E unwired.** `apps/web` ships `test:e2e` (`playwright test`) with real suites (`admin/auth/chat/navigation.spec.ts`); neither `pnpm test` nor any CI step invokes it. The web UI's user flows are verified only when an operator runs them manually.
|
||||
3. **No secret scanning on this repo.** The framework's own template pre-commit makes gitleaks **required**, and `verify.sh` proves detection with a planted key — but this repository's `.husky/pre-commit` (lint-staged only) and CI run no secret scan. The repo ships the control it does not use.
|
||||
4. **No dependency audit.** The quality `.woodpecker.yml` templates and `docs/CI-SETUP.md` specify `npm audit --audit-level=high` as a pipeline stage; nothing equivalent runs for this repo.
|
||||
5. **No coverage thresholds.** Templates enforce 80% Jest coverage thresholds; this repo's Vitest configs collect coverage with no thresholds — coverage is measured nowhere and enforced nowhere.
|
||||
6. **`test-roster-schema.py` invisible.** A real jsonschema regression suite wired to no surface and invisible to the enumeration guard (its population is `*.sh`; the suite is `.py`). Either enumerate it or sign an exclusion — silence here is the #1017 defect shape.
|
||||
7. **Presence-checker expectations ≠ this repo.** `quality-rails check` expects `.eslintrc`, `biome.json`, `.githooks/pre-commit`, `PR-CHECKLIST.md` for node projects — none describe this monorepo (husky, flat eslint config, no biome, no PR-CHECKLIST.md). The evaluator's check set must be per-subject (versioned, digested), not one global file list.
|
||||
8. **Chain-ordering residual (documented).** `test:framework-shell` is one `&&` chain: a failing link skips every later suite while the step still fails (measured in #1270 — four suites after position 44 had not run since a prior merge). The enumeration guard proves naming, not reachability; both residuals are in-file documented but structurally unfixed.
|
||||
9. **Signed-exclusion burndown open.** 16 signed exclusions remain in `test-enumeration-exclusions.txt`; several are "unmeasured in CI image" or blocked on missing CI tooling (tmux, setsid) — tracked under #1017/#1271. Each is an enforcement promise deferred, not delivered.
|
||||
10. **Windows twins unexercised.** `verify.ps1`, `install.ps1`, `mosaic-doctor.ps1` have no runner anywhere (no Windows CI); behavioral drift from their bash twins is undetectable by construction.
|
||||
11. **QA hook name vs behavior.** `qa-hook-handler.sh` files remediation report templates but performs no verification; the seam's actual gate value is only the deps-preflight sentinel. Anything relying on "QA automation hook" as a check is relying on report-filing.
|
||||
12. **Two test paths, one gated.** CI runs tests against ci-postgres (`DATABASE_URL` set); the local PGlite path is the documented default (AGENTS.md) until KBN-101-02/101-05. Only the CI path is enforced by pipeline.
|
||||
|
||||
## 4. Disposition summary
|
||||
|
||||
| disposition | rows | checks |
|
||||
| ------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| preserve | 43 | Every canonical owner (QC-1..QC-18) plus correct guards-of-the-guard and thin adapters: all of §1.1, the CI-invoked framework probes and adapters in §1.2, all of §1.3, the C1/C2/C5 trio and doctors in §1.4, all of §1.5, all pipeline-only steps in §1.6, §1.7 rows 1 and 3, and §1.8. |
|
||||
| strengthen | 2 | `quality-rails check` and `quality-rails doctor` (QC-19) — the RI-N4 evaluator seed: typed verdicts, versioned/digested check definitions, per-subject check sets. |
|
||||
| strengthen (review) | 9 | `verify.sh` + `verify.ps1` (QC-20), quality `install.sh`/`install.ps1` + `quality-rails init` (QC-21 — scaffold-path convergence), `test-roster-schema.py` (QC-5 — wire or sign), `qa-hook-stdin.sh` seam + `typecheck-hook.sh` (QC-16), `tools/e2e-install-test.sh` (QC-5 — CI lane). |
|
||||
| retire | 0 | None meet the bar: RI-N4 requires effective shell probes be **absorbed before** their paths retire, and no absorption exists yet. The `strengthen (review)` rows are the retirement candidates for RI-3-002 once the evaluator owns their behavior. |
|
||||
|
||||
Row total: 54. Canonical checks: 21 (QC-1..QC-21).
|
||||
@@ -51,6 +51,8 @@ fix() { fix_count=$((fix_count + 1)); echo "[FIX] $*"; }
|
||||
|
||||
warn_count=0
|
||||
warn() { warn_count=$((warn_count + 1)); echo "[WARN] $*"; }
|
||||
note() { echo "[NOTE] $*"; return 0; }
|
||||
|
||||
pass() {
|
||||
if [[ $VERBOSE -eq 1 ]]; then
|
||||
echo "[OK] $*"
|
||||
@@ -288,6 +290,18 @@ check_brain_home() {
|
||||
brain="$(resolve_brain_home)"
|
||||
|
||||
if [[ "$brain" == "$MOSAIC_HOME" ]]; then
|
||||
# Implicit-path greenfield case (#1288 comment 23133, fred's trace): nothing
|
||||
# in product code creates ~/.mosaic/fleet/agents — the first fleet write
|
||||
# resolves legacy (generated-env-boundary resolves before creating) and
|
||||
# then manufactures the evidence that keeps the host legacy. On a host with
|
||||
# ~/.mosaic but no fleet/agents, the three operator checks all agree and all
|
||||
# point the wrong way; this doctor is the only one that can disagree, so it
|
||||
# must say it — as a note, not a warn: nothing is broken yet.
|
||||
if [[ "$(cd "$MOSAIC_HOME" 2>/dev/null && pwd -P)" == "$HOME/.config/mosaic" \
|
||||
&& -d "$HOME/.mosaic" && ! -d "$HOME/.mosaic/fleet/agents" ]]; then
|
||||
note "Fleet state home: $MOSAIC_HOME (legacy). NOTE: ~/.mosaic exists but carries no fleet/agents — the first 'mosaic fleet regen' on this host locks in the legacy tree. Create ~/.mosaic/fleet/agents first to adopt the brain."
|
||||
return
|
||||
fi
|
||||
pass "Fleet state home: $MOSAIC_HOME (legacy single-tree; no brain adopted)"
|
||||
return
|
||||
fi
|
||||
|
||||
@@ -39,22 +39,26 @@ eval "$(extract_function check_brain_home)"
|
||||
ROOT=$(mktemp -d)
|
||||
trap 'rm -rf "$ROOT"' EXIT
|
||||
|
||||
# note output is neither [OK] nor [WARN] — assert it directly in the case below.
|
||||
run_case() {
|
||||
# label, expect (ok|warn), then env assignments as arguments.
|
||||
# label, expect (ok|warn|note), then env assignments as arguments.
|
||||
# The check runs under `env` in a subshell, so its warn() also prints a
|
||||
# sentinel the parent counts — a subshell counter would never be visible.
|
||||
local label="$1" expect="$2"
|
||||
shift 2
|
||||
local out warns
|
||||
out=$(env "$@" bash -c "warn() { echo \"[WARN] \$*\"; }; pass() { echo \"[OK] \$*\"; return 0; }; $(extract_function resolve_brain_home); $(extract_function check_brain_home); check_brain_home" 2>&1)
|
||||
local out warns notes
|
||||
out=$(env "$@" bash -c "warn() { echo \"[WARN] \$*\"; }; note() { echo \"[NOTE] \$*\"; return 0; }; pass() { echo \"[OK] \$*\"; return 0; }; $(extract_function resolve_brain_home); $(extract_function check_brain_home); check_brain_home" 2>&1)
|
||||
warns=$(printf '%s\n' "$out" | grep -c '^\[WARN\]' || true)
|
||||
if [[ "$expect" == ok && "$warns" -eq 0 ]]; then
|
||||
notes=$(printf '%s\n' "$out" | grep -c '^\[NOTE\]' || true)
|
||||
if [[ "$expect" == ok && "$warns" -eq 0 && "$notes" -eq 0 ]]; then
|
||||
echo "ok - $label"
|
||||
elif [[ "$expect" == warn && "$warns" -gt 0 ]]; then
|
||||
echo "ok - $label (warned)"
|
||||
elif [[ "$expect" == note && "$notes" -gt 0 ]]; then
|
||||
echo "ok - $label (noted)"
|
||||
else
|
||||
echo "output: $out" >&2
|
||||
fail "$label: expected $expect (warns=$warns)"
|
||||
fail "$label: expected $expect (warns=$warns notes=$notes)"
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -105,4 +109,19 @@ chmod 700 "$ROOT/brain-clean/fleet/agents" "$ROOT/config6/fleet/agents"
|
||||
run_case "empty config-home agents dir alongside brain passes" ok \
|
||||
MOSAIC_HOME="$ROOT/config6" HOME="$ROOT" MOSAIC_BRAIN_HOME="$ROOT/brain-clean"
|
||||
|
||||
# ── greenfield brain-without-agents at the default home → note (#1288) ─────
|
||||
mkdir -p "$ROOT/gf-home/.config/mosaic/fleet" "$ROOT/gf-home/.mosaic"
|
||||
run_case "~/.mosaic without fleet/agents at default home notes the lock-in" note \
|
||||
MOSAIC_HOME="$ROOT/gf-home/.config/mosaic" HOME="$ROOT/gf-home"
|
||||
|
||||
# ── no ~/.mosaic at all at the default home → clean pass ─────────────────
|
||||
mkdir -p "$ROOT/plain-home/.config/mosaic/fleet"
|
||||
run_case "no ~/.mosaic at default home passes silently" ok \
|
||||
MOSAIC_HOME="$ROOT/plain-home/.config/mosaic" HOME="$ROOT/plain-home"
|
||||
|
||||
# ── custom (non-default) home with a stray ~/.mosaic → still silent ──────
|
||||
mkdir -p "$ROOT/custom-home/fleet/agents" "$ROOT/custom-home/.mosaic"
|
||||
run_case "custom home with stray ~/.mosaic stays silent" ok \
|
||||
MOSAIC_HOME="$ROOT/custom-home" HOME="$ROOT/custom-home"
|
||||
|
||||
echo "ok - mosaic-doctor brain-home check"
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# @mosaicstack/quality-rails
|
||||
|
||||
Quality-rails scaffolding and — since RI-3-002 (RI-N4) — the **typed
|
||||
quality-rails evaluator**: the single authoritative producer of check verdicts
|
||||
for the checks it owns.
|
||||
|
||||
## Evaluator (RI-N4)
|
||||
|
||||
Every verdict is typed and fail-closed:
|
||||
|
||||
```
|
||||
{ status: 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable',
|
||||
checkId, checkVersion, subject, reason }
|
||||
```
|
||||
|
||||
Missing implementations, missing inputs, unknown check ids, process errors,
|
||||
timeouts, and malformed probe output can never become `passed` or an
|
||||
unqualified skip — they surface as `blocked`/`error` with a reason (vocabulary
|
||||
mirrors MACP's `GateStatus` discipline).
|
||||
|
||||
- Check definitions live as **data** with a version and a sha256 content
|
||||
digest (`definitionDigest`); every recorded verdict names the definition
|
||||
version that produced it.
|
||||
- Check sets are selected **per subject kind** (`node`, `python`, `rust`,
|
||||
`monorepo`, `unknown`) via the versioned, digested check-set policy — this
|
||||
repository (a `monorepo` subject) does not share the node template's file
|
||||
list.
|
||||
- Shell probes stay **thin adapters**: the TS evaluator invokes them and owns
|
||||
the verdict parsing (e.g. QC-20's planted-commit probe).
|
||||
|
||||
### Owned checks
|
||||
|
||||
| check id | canonical check | mechanism |
|
||||
| --------------------------- | --------------- | ---------------------------------------------------------------------- |
|
||||
| `qc-19-rails-files-present` | QC-19 | typed absorption of the former presence-only `check`/`doctor` loop |
|
||||
| `qc-20-enforcement-verify` | QC-20 | thin shell adapter (framework `verify.sh`); verdict parsing owned here |
|
||||
|
||||
The canonical check ids QC-1..QC-21 and their dispositions are defined in
|
||||
`docs/release-integrity/probe-inventory.md` (the RI-3-001 inventory — the
|
||||
evaluator's input, not its output).
|
||||
|
||||
### Usage
|
||||
|
||||
```sh
|
||||
# CLI (same typed report as the programmatic API)
|
||||
node dist/cli.js quality-rails evaluate --project <path> [--probe-path <verify.sh>] [--json]
|
||||
node dist/cli.js quality-rails check --project <path> [--json] # QC-19 only, fail-closed exit
|
||||
```
|
||||
|
||||
```ts
|
||||
import { evaluateSubject } from '@mosaicstack/quality-rails';
|
||||
|
||||
const report = await evaluateSubject({ subjectPath: '/path/to/project' });
|
||||
// report.state: 'passed' | 'failed' | 'blocked' | 'error'
|
||||
```
|
||||
|
||||
`pnpm verify:release` invokes this evaluator as its `quality-rails` stage
|
||||
(canonical-only stage, QC-19 on the monorepo subject).
|
||||
|
||||
## Scaffolding (pre-existing)
|
||||
|
||||
`init` scaffolds rails files per detected kind/profile; `doctor` is advisory
|
||||
and reports typed states.
|
||||
@@ -0,0 +1,194 @@
|
||||
import { mkdir, mkdtemp, writeFile, chmod } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { createQualityRailsCli } from './cli.js';
|
||||
import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js';
|
||||
import { evaluateSubject } from './evaluator/runner.js';
|
||||
import type { EvaluationReport } from './evaluator/types.js';
|
||||
|
||||
// CLI ↔ programmatic contract (RI-3-002): the same subject must produce the
|
||||
// same typed verdicts through every entry point the card adds — the
|
||||
// `evaluate`/`check` CLI surfaces and the `evaluateSubject` API.
|
||||
|
||||
async function makeTempDir(): Promise<string> {
|
||||
return mkdtemp(join(tmpdir(), 'qr-cli-'));
|
||||
}
|
||||
|
||||
async function scaffoldNodeFixture(skip: string[] = []): Promise<string> {
|
||||
const dir = await makeTempDir();
|
||||
await writeFile(join(dir, 'package.json'), '{}\n', 'utf8');
|
||||
for (const relativePath of [
|
||||
'.eslintrc',
|
||||
'biome.json',
|
||||
'.githooks/pre-commit',
|
||||
'PR-CHECKLIST.md',
|
||||
]) {
|
||||
if (skip.includes(relativePath)) continue;
|
||||
await mkdir(join(dir, relativePath, '..'), { recursive: true });
|
||||
await writeFile(join(dir, relativePath), 'fixture\n', 'utf8');
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
async function makePassingProbe(dir: string): Promise<string> {
|
||||
const scriptPath = join(dir, 'probe-pass.sh');
|
||||
await writeFile(
|
||||
scriptPath,
|
||||
[
|
||||
'#!/bin/bash',
|
||||
'echo "✅ PASS: Type errors blocked"',
|
||||
'echo "✅ PASS: Lint errors blocked"',
|
||||
'echo "Verification Summary"',
|
||||
'exit 0',
|
||||
].join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
await chmod(scriptPath, 0o755);
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
describe('CLI entry points vs the programmatic evaluator', () => {
|
||||
let logSpy: ReturnType<typeof vi.spyOn>;
|
||||
let previousExitCode: string | number | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
logSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
previousExitCode = process.exitCode ?? undefined;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
logSpy.mockRestore();
|
||||
process.exitCode = previousExitCode;
|
||||
});
|
||||
|
||||
it('evaluate --json produces the SAME typed report as evaluateSubject (full check set + probe)', async () => {
|
||||
const dir = await scaffoldNodeFixture();
|
||||
const probePath = await makePassingProbe(dir);
|
||||
|
||||
const programmatic = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath } },
|
||||
});
|
||||
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync([
|
||||
'node',
|
||||
'cli.js',
|
||||
'quality-rails',
|
||||
'evaluate',
|
||||
'--project',
|
||||
dir,
|
||||
'--probe-path',
|
||||
probePath,
|
||||
'--json',
|
||||
]);
|
||||
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
const cliReport = JSON.parse(printed) as EvaluationReport;
|
||||
expect(cliReport).toEqual(programmatic);
|
||||
expect(cliReport.state).toBe('passed');
|
||||
expect(process.exitCode).toBe(0);
|
||||
});
|
||||
|
||||
it('check --json produces the SAME QC-19 verdict as evaluateSubject (absorbed loop)', async () => {
|
||||
const dir = await scaffoldNodeFixture(['biome.json', '.githooks/pre-commit']);
|
||||
|
||||
const programmatic = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
|
||||
});
|
||||
expect(programmatic.state).toBe('failed');
|
||||
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync([
|
||||
'node',
|
||||
'cli.js',
|
||||
'quality-rails',
|
||||
'check',
|
||||
'--project',
|
||||
dir,
|
||||
'--json',
|
||||
]);
|
||||
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
const cliReport = JSON.parse(printed) as EvaluationReport;
|
||||
expect(cliReport).toEqual(programmatic);
|
||||
expect(process.exitCode).toBe(1);
|
||||
});
|
||||
|
||||
it('check on a complete subject exits 0 with a passed verdict', async () => {
|
||||
const dir = await scaffoldNodeFixture();
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync([
|
||||
'node',
|
||||
'cli.js',
|
||||
'quality-rails',
|
||||
'check',
|
||||
'--project',
|
||||
dir,
|
||||
'--json',
|
||||
]);
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
const cliReport = JSON.parse(printed) as EvaluationReport;
|
||||
expect(cliReport.state).toBe('passed');
|
||||
expect(process.exitCode).toBe(0);
|
||||
});
|
||||
|
||||
it('evaluate with an unknown check id exits 1 and reports error, never passed', async () => {
|
||||
const dir = await scaffoldNodeFixture();
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync([
|
||||
'node',
|
||||
'cli.js',
|
||||
'quality-rails',
|
||||
'evaluate',
|
||||
'--project',
|
||||
dir,
|
||||
'--check',
|
||||
'qc-99-bogus',
|
||||
'--json',
|
||||
]);
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
const cliReport = JSON.parse(printed) as EvaluationReport;
|
||||
expect(cliReport.results).toHaveLength(1);
|
||||
const first = cliReport.results[0];
|
||||
expect(first?.status).toBe('error');
|
||||
expect(first?.reason).toContain('unknown check id');
|
||||
expect(process.exitCode).toBe(1);
|
||||
});
|
||||
|
||||
it('evaluate on a scaffold subject without --probe-path stays fail-closed (blocked, exit 1)', async () => {
|
||||
const dir = await scaffoldNodeFixture();
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync([
|
||||
'node',
|
||||
'cli.js',
|
||||
'quality-rails',
|
||||
'evaluate',
|
||||
'--project',
|
||||
dir,
|
||||
'--json',
|
||||
]);
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
const cliReport = JSON.parse(printed) as EvaluationReport;
|
||||
const qc20 = cliReport.results.find((r) => r.checkId === 'qc-20-enforcement-verify');
|
||||
expect(qc20).toBeDefined();
|
||||
expect(qc20?.status).toBe('blocked');
|
||||
expect(qc20?.reason).toContain('probePath');
|
||||
expect(cliReport.state).toBe('blocked');
|
||||
expect(process.exitCode).toBe(1);
|
||||
});
|
||||
|
||||
it('doctor stays advisory (no nonzero exit) but reports TYPED states, including blocked', async () => {
|
||||
const dir = await scaffoldNodeFixture();
|
||||
const program = createQualityRailsCli();
|
||||
await program.parseAsync(['node', 'cli.js', 'quality-rails', 'doctor', '--project', dir]);
|
||||
|
||||
const printed = logSpy.mock.calls.map((call) => String(call[0])).join('\n');
|
||||
expect(printed).toContain('blocked: qc-20-enforcement-verify');
|
||||
expect(process.exitCode ?? 0).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,3 @@
|
||||
import { constants } from 'node:fs';
|
||||
import { access } from 'node:fs/promises';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
@@ -8,18 +6,12 @@ import { Command } from 'commander';
|
||||
import { detectProjectKind } from './detect.js';
|
||||
import { scaffoldQualityRails } from './scaffolder.js';
|
||||
import type { ProjectKind, QualityProfile, RailsConfig } from './types.js';
|
||||
import { QC_19_RAILS_FILES_PRESENT } from './evaluator/definitions.js';
|
||||
import { evaluateSubject } from './evaluator/runner.js';
|
||||
import type { EvaluationReport } from './evaluator/types.js';
|
||||
|
||||
const VALID_PROFILES: readonly QualityProfile[] = ['strict', 'standard', 'minimal'];
|
||||
|
||||
async function fileExists(filePath: string): Promise<boolean> {
|
||||
try {
|
||||
await access(filePath, constants.F_OK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function parseProfile(rawProfile: string): QualityProfile {
|
||||
if (VALID_PROFILES.includes(rawProfile as QualityProfile)) {
|
||||
return rawProfile as QualityProfile;
|
||||
@@ -59,22 +51,6 @@ function defaultFormatters(kind: ProjectKind): string[] {
|
||||
return [];
|
||||
}
|
||||
|
||||
function expectedFilesForKind(kind: ProjectKind): string[] {
|
||||
if (kind === 'node') {
|
||||
return ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
|
||||
}
|
||||
|
||||
if (kind === 'python') {
|
||||
return ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
|
||||
}
|
||||
|
||||
if (kind === 'rust') {
|
||||
return ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'];
|
||||
}
|
||||
|
||||
return ['.githooks/pre-commit', 'PR-CHECKLIST.md'];
|
||||
}
|
||||
|
||||
function printScaffoldResult(
|
||||
config: RailsConfig,
|
||||
filesWritten: string[],
|
||||
@@ -106,6 +82,20 @@ function printScaffoldResult(
|
||||
}
|
||||
}
|
||||
|
||||
function printEvaluationReport(report: EvaluationReport): void {
|
||||
console.log(
|
||||
`[quality-rails] evaluation for ${report.subject.path} (kind=${report.subject.kind}, check-set v${report.checkSetVersion})`,
|
||||
);
|
||||
for (const result of report.results) {
|
||||
const reason = result.reason === undefined ? '' : ` — ${result.reason}`;
|
||||
const digest = report.definitionDigests[result.checkId] ?? 'no digest';
|
||||
console.log(
|
||||
` - ${result.status}: ${result.checkId} (v${result.checkVersion} [${digest}])${reason}`,
|
||||
);
|
||||
}
|
||||
console.log(`[quality-rails] aggregate: ${report.state}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Register quality-rails subcommands on an existing Commander program.
|
||||
* This avoids cross-package Commander version mismatches by using the
|
||||
@@ -148,56 +138,93 @@ function buildQualityRailsCommand(qualityRails: Command): void {
|
||||
printScaffoldResult(config, result.filesWritten, result.warnings, result.commandsToRun);
|
||||
});
|
||||
|
||||
// `check` (QC-19) is ABSORBED by the RI-N4 evaluator: the presence loop
|
||||
// that lived here is now the versioned, digested, typed check definition
|
||||
// `qc-19-rails-files-present`. The CLI keeps its human surface (missing
|
||||
// files listed, exit 1) and gains `--json` for the typed verdicts. Exit
|
||||
// code is fail-closed: any non-green aggregate (failed/blocked/error) is 1.
|
||||
qualityRails
|
||||
.command('check')
|
||||
.requiredOption('--project <path>', 'Project path')
|
||||
.action(async (options: { project: string }) => {
|
||||
.option('--json', 'print the typed evaluation report as JSON')
|
||||
.action(async (options: { project: string; json?: boolean }) => {
|
||||
const projectPath = resolve(options.project);
|
||||
const kind = await detectProjectKind(projectPath);
|
||||
const expected = expectedFilesForKind(kind);
|
||||
const missing: string[] = [];
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: projectPath,
|
||||
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
|
||||
});
|
||||
|
||||
for (const relativePath of expected) {
|
||||
const exists = await fileExists(resolve(projectPath, relativePath));
|
||||
if (!exists) {
|
||||
missing.push(relativePath);
|
||||
}
|
||||
if (options.json) {
|
||||
console.log(JSON.stringify(report));
|
||||
} else {
|
||||
printEvaluationReport(report);
|
||||
}
|
||||
|
||||
if (missing.length > 0) {
|
||||
console.error('[quality-rails] missing files:');
|
||||
for (const relativePath of missing) {
|
||||
console.error(` - ${relativePath}`);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`[quality-rails] all expected files present for ${kind} project`);
|
||||
process.exitCode = report.state === 'passed' ? 0 : 1;
|
||||
});
|
||||
|
||||
// `doctor` (QC-19) stays advisory (documented contract: a doctor that
|
||||
// cannot fail), but now reports TYPED states — a blocked or failing rail is
|
||||
// visible instead of silently printed as `ok`/`missing`.
|
||||
qualityRails
|
||||
.command('doctor')
|
||||
.requiredOption('--project <path>', 'Project path')
|
||||
.action(async (options: { project: string }) => {
|
||||
const projectPath = resolve(options.project);
|
||||
const kind = await detectProjectKind(projectPath);
|
||||
const expected = expectedFilesForKind(kind);
|
||||
const report = await evaluateSubject({ subjectPath: projectPath });
|
||||
|
||||
console.log(`[quality-rails] doctor for ${projectPath}`);
|
||||
console.log(`detected project kind: ${kind}`);
|
||||
|
||||
for (const relativePath of expected) {
|
||||
const exists = await fileExists(resolve(projectPath, relativePath));
|
||||
console.log(` - ${exists ? 'ok' : 'missing'}: ${relativePath}`);
|
||||
console.log(`detected project kind: ${report.subject.kind}`);
|
||||
for (const result of report.results) {
|
||||
const reason = result.reason === undefined ? '' : ` — ${result.reason}`;
|
||||
console.log(` - ${result.status}: ${result.checkId}${reason}`);
|
||||
}
|
||||
|
||||
if (kind === 'unknown') {
|
||||
if (report.subject.kind === 'unknown') {
|
||||
console.log(
|
||||
'recommendation: add package.json, pyproject.toml, or Cargo.toml for better defaults.',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// `evaluate` is the canonical RI-N4 evaluator entry point: typed verdicts
|
||||
// for the subject's full per-kind check set, same results as the
|
||||
// programmatic API (evaluateSubject).
|
||||
qualityRails
|
||||
.command('evaluate')
|
||||
.description('Run the typed quality-rails evaluator against a subject project')
|
||||
.requiredOption('--project <path>', 'Project path')
|
||||
.option('--check <id...>', 'restrict evaluation to these check ids')
|
||||
.option(
|
||||
'--probe-path <path>',
|
||||
'path to the QC-20 behavioral probe script (framework verify.sh)',
|
||||
)
|
||||
.option('--json', 'print the typed evaluation report as JSON')
|
||||
.action(
|
||||
async (options: {
|
||||
project: string;
|
||||
check?: string[];
|
||||
probePath?: string;
|
||||
json?: boolean;
|
||||
}) => {
|
||||
const projectPath = resolve(options.project);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: projectPath,
|
||||
checkIds: options.check,
|
||||
inputs: options.probePath
|
||||
? { 'qc-20-enforcement-verify': { probePath: options.probePath } }
|
||||
: undefined,
|
||||
});
|
||||
|
||||
if (options.json) {
|
||||
console.log(JSON.stringify(report));
|
||||
} else {
|
||||
printEvaluationReport(report);
|
||||
}
|
||||
|
||||
process.exitCode = report.state === 'passed' ? 0 : 1;
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export async function runQualityRailsCli(argv: string[] = process.argv): Promise<void> {
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import type { ChildProcess } from 'node:child_process';
|
||||
|
||||
import type { AdapterOutcome, AdapterRequest, ProcessAdapter } from './types.js';
|
||||
|
||||
/**
|
||||
* Default thin process adapter (spawn-based). Runs a command to completion with
|
||||
* a hard timeout and reports exit code + captured output — it owns NO verdict
|
||||
* logic. Interpreting the outcome is always the check implementation's job.
|
||||
*/
|
||||
export function createSpawnProcessAdapter(): ProcessAdapter {
|
||||
return {
|
||||
run(request: AdapterRequest): Promise<AdapterOutcome> {
|
||||
return new Promise((resolve) => {
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(request.file, request.args, {
|
||||
cwd: request.cwd,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
} catch (error) {
|
||||
resolve({
|
||||
ok: false,
|
||||
kind: 'spawn-error',
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
let settled = false;
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
child.kill('SIGKILL');
|
||||
resolve({
|
||||
ok: false,
|
||||
kind: 'timeout',
|
||||
message: `process timed out after ${request.timeoutMs}ms: ${request.file}`,
|
||||
});
|
||||
}, request.timeoutMs);
|
||||
|
||||
const settle = (outcome: AdapterOutcome): void => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve(outcome);
|
||||
};
|
||||
|
||||
child.stdout?.on('data', (chunk: Buffer) => {
|
||||
stdout += chunk.toString('utf8');
|
||||
});
|
||||
child.stderr?.on('data', (chunk: Buffer) => {
|
||||
stderr += chunk.toString('utf8');
|
||||
});
|
||||
child.on('error', (error: Error) => {
|
||||
settle({ ok: false, kind: 'spawn-error', message: error.message });
|
||||
});
|
||||
child.on('close', (code: number | null) => {
|
||||
settle({ ok: true, exitCode: code, stdout, stderr });
|
||||
});
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
import { constants } from 'node:fs';
|
||||
import { access } from 'node:fs/promises';
|
||||
import { isAbsolute, resolve } from 'node:path';
|
||||
|
||||
import { digestOfPolicy, digestOfSpec } from './digest.js';
|
||||
import type {
|
||||
CheckContext,
|
||||
CheckDefinition,
|
||||
CheckDefinitionSpec,
|
||||
CheckOutcome,
|
||||
CheckSetPolicy,
|
||||
CheckSetPolicySpec,
|
||||
SubjectKind,
|
||||
} from './types.js';
|
||||
|
||||
// Check definitions for the RI-N4 evaluator (card RI-3-002). Each definition is
|
||||
// DATA with a version and a content digest (see digest.ts); the executable
|
||||
// half is attached via defineCheck. Check-set SELECTION is per subject kind
|
||||
// (probe-inventory gap 7): this monorepo does not match the node template's
|
||||
// file list, so the QC-19 definition carries a distinct file set for the
|
||||
// `monorepo` subject kind and the policy selects checks per kind.
|
||||
|
||||
export function defineCheck(
|
||||
spec: CheckDefinitionSpec,
|
||||
evaluate: (ctx: CheckContext) => Promise<CheckOutcome>,
|
||||
): CheckDefinition {
|
||||
return { ...spec, definitionDigest: digestOfSpec(spec), evaluate };
|
||||
}
|
||||
|
||||
async function fileExists(filePath: string): Promise<boolean> {
|
||||
try {
|
||||
await access(filePath, constants.F_OK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ─── QC-19: downstream rails presence ────────────────────────────────────────
|
||||
//
|
||||
// Typed absorption of the former presence-only `quality-rails check` loop in
|
||||
// cli.ts. The scaffold-kind file lists below are carried over VERBATIM so the
|
||||
// evaluator's typed verdicts are parity-equivalent with the presence loop on
|
||||
// the same fixture; the `monorepo` list is new (per-subject check sets).
|
||||
|
||||
const qc19Spec: CheckDefinitionSpec = {
|
||||
id: 'qc-19-rails-files-present',
|
||||
version: '1.0.0',
|
||||
canonicalCheck: 'QC-19',
|
||||
description:
|
||||
'The subject still carries its quality-rails files. Typed absorption of the former presence-only check loop; presence is necessary, not sufficient (RI-N4).',
|
||||
appliesTo: ['node', 'python', 'rust', 'monorepo', 'unknown'],
|
||||
params: {
|
||||
expectedFilesByKind: {
|
||||
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
monorepo: [
|
||||
'.husky/pre-commit',
|
||||
'.husky/pre-push',
|
||||
'eslint.config.mjs',
|
||||
'.prettierrc',
|
||||
'.lintstagedrc',
|
||||
],
|
||||
unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
async function evaluateQc19(ctx: CheckContext): Promise<CheckOutcome> {
|
||||
const byKind = ctx.params['expectedFilesByKind'] as Record<string, readonly string[]> | undefined;
|
||||
if (byKind === undefined) {
|
||||
return { status: 'error', reason: 'definition params missing expectedFilesByKind' };
|
||||
}
|
||||
const expected = byKind[ctx.subject.kind];
|
||||
if (expected === undefined) {
|
||||
// Fail-closed: an undefined file set for a declared subject kind is a
|
||||
// definition gap, never a green outcome.
|
||||
return {
|
||||
status: 'blocked',
|
||||
reason: `no expected-file set defined for subject kind '${ctx.subject.kind}'`,
|
||||
};
|
||||
}
|
||||
|
||||
const missing: string[] = [];
|
||||
for (const relativePath of expected) {
|
||||
if (!(await fileExists(resolve(ctx.subject.path, relativePath)))) {
|
||||
missing.push(relativePath);
|
||||
}
|
||||
}
|
||||
if (missing.length > 0) {
|
||||
return {
|
||||
status: 'failed',
|
||||
reason: `missing rails files (${ctx.subject.kind}): ${missing.join(', ')}`,
|
||||
};
|
||||
}
|
||||
return { status: 'passed' };
|
||||
}
|
||||
|
||||
// ─── QC-20: downstream enforcement verification (behavioral probe) ──────────
|
||||
//
|
||||
// The planted-commit behavioral probe (framework tools/quality/scripts/verify.sh)
|
||||
// stays a THIN SHELL ADAPTER: the TS evaluator invokes it and OWNS the verdict
|
||||
// parsing (RI-N4: grep-on-output verdict logic moves into the typed evaluator).
|
||||
// Probe contract (verify.sh): exit 0 ⇔ every sub-probe passed, exit 1 ⇔ at
|
||||
// least one sub-probe failed; sub-probe verdicts appear as `PASS:` / `FAIL:`
|
||||
// marker lines and the script always prints a `Verification Summary` section.
|
||||
// Any deviation from that contract (other exit codes, unparseable output,
|
||||
// missing probe, process failure, timeout) is `error`/`blocked` — never
|
||||
// `passed`.
|
||||
|
||||
const qc20Spec: CheckDefinitionSpec = {
|
||||
id: 'qc-20-enforcement-verify',
|
||||
version: '1.0.0',
|
||||
canonicalCheck: 'QC-20',
|
||||
description:
|
||||
'The behavioral planted-commit probe runs against the subject and every sub-probe blocks as intended. The shell probe is a thin adapter; verdict parsing is owned by this evaluator.',
|
||||
appliesTo: ['node', 'python', 'rust', 'unknown'],
|
||||
params: {
|
||||
command: 'bash',
|
||||
timeoutMs: 120_000,
|
||||
passMarker: 'PASS:',
|
||||
failMarker: 'FAIL:',
|
||||
summaryMarker: 'Verification Summary',
|
||||
},
|
||||
};
|
||||
|
||||
function linesWith(text: string, marker: string): string[] {
|
||||
return text
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.includes(marker));
|
||||
}
|
||||
|
||||
async function evaluateQc20(ctx: CheckContext): Promise<CheckOutcome> {
|
||||
const rawProbePath = ctx.inputs['probePath'];
|
||||
if (typeof rawProbePath !== 'string' || rawProbePath.trim().length === 0) {
|
||||
return {
|
||||
status: 'blocked',
|
||||
reason:
|
||||
'missing input: probePath — the behavioral probe script must be provided (e.g. the framework verify.sh)',
|
||||
};
|
||||
}
|
||||
const probePath = isAbsolute(rawProbePath)
|
||||
? rawProbePath
|
||||
: resolve(ctx.subject.path, rawProbePath);
|
||||
if (!(await fileExists(probePath))) {
|
||||
return { status: 'blocked', reason: `probe script not found: ${probePath}` };
|
||||
}
|
||||
|
||||
const command = typeof ctx.params['command'] === 'string' ? ctx.params['command'] : 'bash';
|
||||
const timeoutMs = typeof ctx.params['timeoutMs'] === 'number' ? ctx.params['timeoutMs'] : 120_000;
|
||||
const passMarker =
|
||||
typeof ctx.params['passMarker'] === 'string' ? ctx.params['passMarker'] : 'PASS:';
|
||||
const failMarker =
|
||||
typeof ctx.params['failMarker'] === 'string' ? ctx.params['failMarker'] : 'FAIL:';
|
||||
const summaryMarker =
|
||||
typeof ctx.params['summaryMarker'] === 'string'
|
||||
? ctx.params['summaryMarker']
|
||||
: 'Verification Summary';
|
||||
|
||||
const outcome = await ctx.adapter.run({
|
||||
file: command,
|
||||
args: [probePath],
|
||||
cwd: ctx.subject.path,
|
||||
timeoutMs,
|
||||
});
|
||||
|
||||
if (!outcome.ok) {
|
||||
// Process error or timeout: the probe never produced a trustworthy result.
|
||||
return {
|
||||
status: 'error',
|
||||
reason: `probe process ${outcome.kind}: ${outcome.message}`,
|
||||
};
|
||||
}
|
||||
|
||||
const output = `${outcome.stdout}\n${outcome.stderr}`;
|
||||
const failLines = linesWith(output, failMarker);
|
||||
const passLines = linesWith(output, passMarker);
|
||||
|
||||
if (outcome.exitCode === 0) {
|
||||
// A green exit must be corroborated by a parseable green transcript:
|
||||
// at least one pass marker, no fail markers, and the summary section.
|
||||
if (passLines.length > 0 && failLines.length === 0 && output.includes(summaryMarker)) {
|
||||
return { status: 'passed' };
|
||||
}
|
||||
return {
|
||||
status: 'error',
|
||||
reason: `malformed probe output: exit 0 without a parseable pass transcript (${passLines.length} pass markers, ${failLines.length} fail markers, summary ${output.includes(summaryMarker) ? 'present' : 'absent'})`,
|
||||
};
|
||||
}
|
||||
|
||||
if (outcome.exitCode === 1) {
|
||||
if (failLines.length === 0) {
|
||||
return {
|
||||
status: 'error',
|
||||
reason: 'malformed probe output: exit 1 without parseable FAIL markers',
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: 'failed',
|
||||
reason: `enforcement probe reported ${failLines.length} failing sub-probe(s): ${failLines.join(' | ')}`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
status: 'error',
|
||||
reason: `probe exited with unexpected code ${String(outcome.exitCode)} — outcome not interpretable`,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Per-subject check-set policy ───────────────────────────────────────────
|
||||
//
|
||||
// Gap 7 of the probe inventory: check sets must be selected per subject, not
|
||||
// one global list. Downstream scaffold kinds get the presence check plus the
|
||||
// behavioral probe (QC-20 blocks until a probePath input is provided — an
|
||||
// unverified subject can never evaluate green). The monorepo subject is this
|
||||
// repository itself: its rails are the husky hooks + shared lint/format
|
||||
// configs, covered by QC-19; the downstream planted-commit probe does not
|
||||
// apply to it (this repo's own commit gates are QC-13/QC-14, outside this
|
||||
// evaluator's owned checks).
|
||||
|
||||
const checkSetPolicySpec: CheckSetPolicySpec = {
|
||||
version: '1.0.0',
|
||||
byKind: {
|
||||
node: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
|
||||
python: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
|
||||
rust: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
|
||||
unknown: ['qc-19-rails-files-present', 'qc-20-enforcement-verify'],
|
||||
monorepo: ['qc-19-rails-files-present'],
|
||||
},
|
||||
};
|
||||
|
||||
export const CHECK_SET_POLICY: CheckSetPolicy = {
|
||||
...checkSetPolicySpec,
|
||||
policyDigest: digestOfPolicy(checkSetPolicySpec),
|
||||
};
|
||||
|
||||
export const QC_19_RAILS_FILES_PRESENT = defineCheck(qc19Spec, evaluateQc19);
|
||||
export const QC_20_ENFORCEMENT_VERIFY = defineCheck(qc20Spec, evaluateQc20);
|
||||
|
||||
/** Built-in check definitions, keyed by id. */
|
||||
export function builtInDefinitions(): CheckDefinition[] {
|
||||
return [QC_19_RAILS_FILES_PRESENT, QC_20_ENFORCEMENT_VERIFY];
|
||||
}
|
||||
|
||||
export function checkSetForKind(
|
||||
kind: SubjectKind,
|
||||
policy: CheckSetPolicy = CHECK_SET_POLICY,
|
||||
): readonly string[] {
|
||||
const selected = policy.byKind[kind];
|
||||
if (selected === undefined) {
|
||||
// Fail-closed selection: an unknown kind yields an EMPTY set only to the
|
||||
// caller; the runner treats an empty result list as `blocked`, never green.
|
||||
return [];
|
||||
}
|
||||
return selected;
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
import type { CheckDefinitionSpec, CheckSetPolicySpec } from './types.js';
|
||||
|
||||
// Deterministic JSON: object keys sorted at every level so two specs with the
|
||||
// same content always produce the same bytes (and thus the same digest).
|
||||
export function canonicalJson(value: unknown): string {
|
||||
if (value === null || typeof value !== 'object') {
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map((entry) => canonicalJson(entry)).join(',')}]`;
|
||||
}
|
||||
const record = value as Record<string, unknown>;
|
||||
const keys = Object.keys(record).sort();
|
||||
return `{${keys.map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`).join(',')}}`;
|
||||
}
|
||||
|
||||
/** sha256 over the canonical JSON of `value`. */
|
||||
export function digestContent(value: unknown): string {
|
||||
return createHash('sha256').update(canonicalJson(value), 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* Content digest of a check definition: covers the declarative spec (id,
|
||||
* version, canonical check, description, applicability, params) — everything a
|
||||
* reviewer reasons about — while excluding the executable function object.
|
||||
* Changing any covered field changes the digest, so a recorded digest always
|
||||
* identifies exactly which definition content produced a verdict.
|
||||
*/
|
||||
export function digestOfSpec(spec: CheckDefinitionSpec): string {
|
||||
return digestContent({
|
||||
id: spec.id,
|
||||
version: spec.version,
|
||||
canonicalCheck: spec.canonicalCheck,
|
||||
description: spec.description,
|
||||
appliesTo: spec.appliesTo,
|
||||
params: spec.params,
|
||||
});
|
||||
}
|
||||
|
||||
/** Content digest of the per-subject check-set policy. */
|
||||
export function digestOfPolicy(spec: CheckSetPolicySpec): string {
|
||||
return digestContent(spec);
|
||||
}
|
||||
@@ -0,0 +1,525 @@
|
||||
import { mkdtemp, mkdir, writeFile, chmod } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { createSpawnProcessAdapter } from './adapter.js';
|
||||
import {
|
||||
builtInDefinitions,
|
||||
CHECK_SET_POLICY,
|
||||
checkSetForKind,
|
||||
defineCheck,
|
||||
QC_19_RAILS_FILES_PRESENT,
|
||||
QC_20_ENFORCEMENT_VERIFY,
|
||||
} from './definitions.js';
|
||||
import { digestOfSpec } from './digest.js';
|
||||
import { aggregateState, evaluateSubject } from './runner.js';
|
||||
import type {
|
||||
AdapterOutcome,
|
||||
CheckDefinitionSpec,
|
||||
CheckResult,
|
||||
EvaluationReport,
|
||||
ProcessAdapter,
|
||||
} from './types.js';
|
||||
|
||||
// ─── helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
function firstResult(report: EvaluationReport): CheckResult {
|
||||
const result = report.results[0];
|
||||
if (result === undefined) {
|
||||
throw new Error('expected the report to contain at least one result');
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function makeTempDir(): Promise<string> {
|
||||
return mkdtemp(join(tmpdir(), 'qr-evaluator-'));
|
||||
}
|
||||
|
||||
async function writeProbeScript(dir: string, name: string, body: string): Promise<string> {
|
||||
const scriptPath = join(dir, name);
|
||||
await writeFile(scriptPath, `${body}\n`, 'utf8');
|
||||
await chmod(scriptPath, 0o755);
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
/** Adapter stub that always returns the given outcome (no real process). */
|
||||
function stubAdapter(outcome: AdapterOutcome): ProcessAdapter {
|
||||
return {
|
||||
run: async () => outcome,
|
||||
};
|
||||
}
|
||||
|
||||
// VERBATIM copy of the pre-absorption presence loop (former cli.ts
|
||||
// expectedFilesForKind + fileExists loop). This is the PARITY ORACLE: the
|
||||
// evaluator's typed QC-19 verdict must agree with what the absorbed check
|
||||
// concluded on the same fixture.
|
||||
const LEGACY_EXPECTED: Record<'node' | 'python' | 'rust' | 'unknown', string[]> = {
|
||||
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
python: ['pyproject.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
rust: ['rustfmt.toml', '.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
unknown: ['.githooks/pre-commit', 'PR-CHECKLIST.md'],
|
||||
};
|
||||
|
||||
async function legacyPresenceLoop(projectPath: string, kind: keyof typeof LEGACY_EXPECTED) {
|
||||
const missing: string[] = [];
|
||||
for (const relativePath of LEGACY_EXPECTED[kind]) {
|
||||
const fs = await import('node:fs/promises');
|
||||
try {
|
||||
await fs.access(join(projectPath, relativePath));
|
||||
} catch {
|
||||
missing.push(relativePath);
|
||||
}
|
||||
}
|
||||
return missing;
|
||||
}
|
||||
|
||||
async function scaffoldFixture(kind: keyof typeof LEGACY_EXPECTED, skip: string[] = []) {
|
||||
const dir = await makeTempDir();
|
||||
if (kind === 'node') {
|
||||
await writeFile(join(dir, 'package.json'), '{}\n', 'utf8');
|
||||
}
|
||||
if (kind === 'python') {
|
||||
await writeFile(join(dir, 'pyproject.toml'), '[project]\n', 'utf8');
|
||||
}
|
||||
if (kind === 'rust') {
|
||||
await writeFile(join(dir, 'Cargo.toml'), '[package]\n', 'utf8');
|
||||
}
|
||||
for (const relativePath of LEGACY_EXPECTED[kind]) {
|
||||
if (skip.includes(relativePath)) continue;
|
||||
await mkdir(join(dir, relativePath, '..'), { recursive: true });
|
||||
await writeFile(join(dir, relativePath), 'fixture\n', 'utf8');
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
// ─── QC-19 parity: typed verdict == absorbed presence loop ──────────────────
|
||||
|
||||
describe('QC-19 parity with the absorbed presence loop', () => {
|
||||
const kinds: Array<keyof typeof LEGACY_EXPECTED> = ['node', 'python', 'rust', 'unknown'];
|
||||
|
||||
it.each(kinds)('positive fixture (%s): loop said ok ⇒ evaluator passed', async (kind) => {
|
||||
const dir = await scaffoldFixture(kind);
|
||||
const oracleMissing = await legacyPresenceLoop(dir, kind);
|
||||
expect(oracleMissing).toEqual([]);
|
||||
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
|
||||
});
|
||||
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
|
||||
expect(result?.status).toBe('passed');
|
||||
expect(result?.reason).toBeUndefined();
|
||||
expect(report.state).toBe('passed');
|
||||
});
|
||||
|
||||
it.each(kinds)(
|
||||
'negative fixture (%s): loop listed missing ⇒ evaluator failed with them',
|
||||
async (kind) => {
|
||||
const all = LEGACY_EXPECTED[kind];
|
||||
const skip = all.slice(0, Math.max(1, all.length - 1)); // leave exactly 1 present
|
||||
const dir = await scaffoldFixture(kind, skip);
|
||||
const oracleMissing = await legacyPresenceLoop(dir, kind);
|
||||
expect(oracleMissing.length).toBeGreaterThan(0);
|
||||
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
|
||||
});
|
||||
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
|
||||
expect(result?.status).toBe('failed');
|
||||
expect(report.state).toBe('failed');
|
||||
for (const missingFile of oracleMissing) {
|
||||
expect(result?.reason).toContain(missingFile);
|
||||
}
|
||||
// No false attribution: a present file must not be named in the reason.
|
||||
const presentFile = all.find((file) => !skip.includes(file));
|
||||
if (presentFile !== undefined) {
|
||||
expect(result?.reason).not.toContain(` ${presentFile},`);
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
// ─── per-subject check sets (inventory gap 7) ────────────────────────────────
|
||||
|
||||
describe('per-subject check sets', () => {
|
||||
it('monorepo subject selects only QC-19 with the monorepo file set', async () => {
|
||||
const dir = await makeTempDir();
|
||||
await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n - packages/*\n', 'utf8');
|
||||
for (const file of [
|
||||
'.husky/pre-commit',
|
||||
'.husky/pre-push',
|
||||
'eslint.config.mjs',
|
||||
'.prettierrc',
|
||||
'.lintstagedrc',
|
||||
]) {
|
||||
await mkdir(join(dir, file, '..'), { recursive: true });
|
||||
await writeFile(join(dir, file), 'fixture\n', 'utf8');
|
||||
}
|
||||
|
||||
const report = await evaluateSubject({ subjectPath: dir });
|
||||
expect(report.subject.kind).toBe('monorepo');
|
||||
expect(report.results.map((r) => r.checkId)).toEqual(['qc-19-rails-files-present']);
|
||||
expect(report.state).toBe('passed');
|
||||
});
|
||||
|
||||
it('a monorepo missing one of its rails files fails QC-19 (not the node list)', async () => {
|
||||
const dir = await makeTempDir();
|
||||
await writeFile(join(dir, 'pnpm-workspace.yaml'), 'packages:\n', 'utf8');
|
||||
const report = await evaluateSubject({ subjectPath: dir });
|
||||
const result = report.results.find((r) => r.checkId === QC_19_RAILS_FILES_PRESENT.id);
|
||||
expect(result?.status).toBe('failed');
|
||||
expect(result?.reason).toContain('.husky/pre-commit');
|
||||
// The node-template list must NOT be applied to a monorepo subject.
|
||||
expect(result?.reason).not.toContain('biome.json');
|
||||
});
|
||||
|
||||
it('the policy selects the behavioral probe for scaffold kinds but not monorepo', () => {
|
||||
expect(checkSetForKind('node')).toContain(QC_20_ENFORCEMENT_VERIFY.id);
|
||||
expect(checkSetForKind('unknown')).toContain(QC_20_ENFORCEMENT_VERIFY.id);
|
||||
expect(checkSetForKind('monorepo')).not.toContain(QC_20_ENFORCEMENT_VERIFY.id);
|
||||
expect(CHECK_SET_POLICY.version).toBe('1.0.0');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── negative controls (the point of the card) ───────────────────────────────
|
||||
|
||||
describe('negative controls', () => {
|
||||
it('unknown check id ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: ['qc-99-does-not-exist'],
|
||||
});
|
||||
expect(report.results).toHaveLength(1);
|
||||
const result = firstResult(report);
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.reason).toContain("unknown check id 'qc-99-does-not-exist'");
|
||||
expect(result.status === 'passed').toBe(false);
|
||||
expect(report.state).toBe('error');
|
||||
});
|
||||
|
||||
it('missing subject (directory absent) ⇒ blocked for every check, never passed', async () => {
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: join(tmpdir(), `qr-evaluator-absent-${Date.now()}`),
|
||||
});
|
||||
expect(report.results.length).toBeGreaterThan(0);
|
||||
for (const result of report.results) {
|
||||
expect(result.status).toBe('blocked');
|
||||
expect(result.reason).toContain('subject directory does not exist');
|
||||
}
|
||||
expect(report.state).toBe('blocked');
|
||||
});
|
||||
|
||||
it('QC-20 without probePath input ⇒ blocked, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
});
|
||||
const result = firstResult(report);
|
||||
expect(result.status).toBe('blocked');
|
||||
expect(result.reason).toContain('missing input: probePath');
|
||||
});
|
||||
|
||||
it('QC-20 with a nonexistent probe script ⇒ blocked, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'no-such-probe.sh') } },
|
||||
});
|
||||
expect(firstResult(report).status).toBe('blocked');
|
||||
expect(firstResult(report).reason).toContain('probe script not found');
|
||||
});
|
||||
|
||||
it('adapter process error (spawn failure) ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } },
|
||||
adapter: stubAdapter({ ok: false, kind: 'spawn-error', message: 'ENOENT bash' }),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('probe process spawn-error');
|
||||
expect(firstResult(report).status === 'passed').toBe(false);
|
||||
});
|
||||
|
||||
it('adapter timeout ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: join(dir, 'PR-CHECKLIST.md') } },
|
||||
adapter: stubAdapter({ ok: false, kind: 'timeout', message: 'timed out after 120000ms' }),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('probe process timeout');
|
||||
});
|
||||
|
||||
it('probe exit 1 with parseable FAIL markers ⇒ failed (interpretably red), never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(
|
||||
dir,
|
||||
'probe-fail.sh',
|
||||
`echo "Test 1: ..."\necho "❌ FAIL: Type errors NOT blocked"\necho "Verification Summary"\nexit 1`,
|
||||
);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('failed');
|
||||
expect(firstResult(report).reason).toContain('FAIL: Type errors NOT blocked');
|
||||
expect(report.state).toBe('failed');
|
||||
});
|
||||
|
||||
it('probe exit 1 WITHOUT parseable FAIL markers ⇒ malformed ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(dir, 'probe-mute.sh', `echo "nothing to see"\nexit 1`);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('malformed probe output');
|
||||
});
|
||||
|
||||
it('probe exit 0 without a parseable pass transcript ⇒ malformed ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(dir, 'probe-lie.sh', `echo "all good"\nexit 0`);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('malformed probe output');
|
||||
expect(firstResult(report).reason).toContain('exit 0');
|
||||
});
|
||||
|
||||
it('probe exit 0 WITH fail markers ⇒ contradictory transcript ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(
|
||||
dir,
|
||||
'probe-contradict.sh',
|
||||
`echo "✅ PASS: one"\necho "❌ FAIL: two"\necho "Verification Summary"\nexit 0`,
|
||||
);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
});
|
||||
|
||||
it('probe unexpected exit code (7) ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(dir, 'probe-crash.sh', `echo "boom"\nexit 7`);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('unexpected code 7');
|
||||
});
|
||||
|
||||
it('check implementation throwing ⇒ error, never passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const spec: CheckDefinitionSpec = {
|
||||
id: 'test-throws',
|
||||
version: '1.0.0',
|
||||
canonicalCheck: 'QC-TEST',
|
||||
description: 'sabotage-shaped definition that always throws',
|
||||
appliesTo: ['node'],
|
||||
params: {},
|
||||
};
|
||||
const throwing = defineCheck(spec, async () => {
|
||||
throw new Error('kaboom');
|
||||
});
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: ['test-throws'],
|
||||
definitions: [throwing],
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('kaboom');
|
||||
});
|
||||
|
||||
it('non-passed verdict without a reason ⇒ upgraded to error, never an unqualified skip', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const spec: CheckDefinitionSpec = {
|
||||
id: 'test-silent-fail',
|
||||
version: '1.0.0',
|
||||
canonicalCheck: 'QC-TEST',
|
||||
description: 'returns failed without a reason',
|
||||
appliesTo: ['node'],
|
||||
params: {},
|
||||
};
|
||||
const silent = defineCheck(spec, async () => ({ status: 'failed' }));
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: ['test-silent-fail'],
|
||||
definitions: [silent],
|
||||
});
|
||||
expect(firstResult(report).status).toBe('error');
|
||||
expect(firstResult(report).reason).toContain('without a reason');
|
||||
});
|
||||
|
||||
it('empty result list aggregates to blocked, never passed', () => {
|
||||
expect(aggregateState([])).toBe('blocked');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── QC-20 parity: typed verdict == shell probe's own conclusion ────────────
|
||||
|
||||
describe('QC-20 parity with the shell probe contract', () => {
|
||||
it('green transcript (exit 0) ⇒ evaluator passed', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const probe = await writeProbeScript(
|
||||
dir,
|
||||
'probe-pass.sh',
|
||||
[
|
||||
'echo "✅ PASS: Type errors blocked"',
|
||||
'echo "✅ PASS: any types blocked"',
|
||||
'echo "✅ PASS: Lint errors blocked"',
|
||||
'echo "Verification Summary"',
|
||||
'echo "✅ Passed: 3"',
|
||||
'exit 0',
|
||||
].join('\n'),
|
||||
);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: probe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
expect(firstResult(report).status).toBe('passed');
|
||||
expect(report.state).toBe('passed');
|
||||
});
|
||||
|
||||
it('the REAL framework verify.sh on a non-git subject concludes failed (exit 1) ⇒ evaluator failed', async () => {
|
||||
// Real-probe parity: verify.sh without a git repo cannot block planted
|
||||
// commits, exits 1 with FAIL markers — the evaluator must record exactly
|
||||
// `failed` with those markers, matching the probe's own conclusion.
|
||||
const realProbe = fileURLToPath(
|
||||
new URL('../../../mosaic/framework/tools/quality/scripts/verify.sh', import.meta.url),
|
||||
);
|
||||
const dir = await makeTempDir(); // not a git repository, no hooks
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_20_ENFORCEMENT_VERIFY.id],
|
||||
inputs: { 'qc-20-enforcement-verify': { probePath: realProbe } },
|
||||
adapter: createSpawnProcessAdapter(),
|
||||
});
|
||||
const result = firstResult(report);
|
||||
expect(result.status).toBe('failed');
|
||||
expect(result.reason).toMatch(/FAIL:/);
|
||||
expect(report.state).toBe('failed');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── version / digest discipline ─────────────────────────────────────────────
|
||||
|
||||
describe('versioned, digested check definitions', () => {
|
||||
it('every verdict records the definition version that produced it', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({ subjectPath: dir });
|
||||
for (const result of report.results) {
|
||||
expect(result.checkVersion).toBe('1.0.0');
|
||||
}
|
||||
expect(report.checkSetVersion).toBe(CHECK_SET_POLICY.version);
|
||||
});
|
||||
|
||||
it('the report records each definition’s content digest', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const report = await evaluateSubject({ subjectPath: dir });
|
||||
expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe(
|
||||
QC_19_RAILS_FILES_PRESENT.definitionDigest,
|
||||
);
|
||||
expect(report.definitionDigests[QC_20_ENFORCEMENT_VERIFY.id]).toBe(
|
||||
QC_20_ENFORCEMENT_VERIFY.definitionDigest,
|
||||
);
|
||||
});
|
||||
|
||||
it('digests are stable for identical content', () => {
|
||||
const spec = QC_19_RAILS_FILES_PRESENT;
|
||||
expect(digestOfSpec(spec)).toBe(digestOfSpec(spec));
|
||||
});
|
||||
|
||||
it('changing a definition’s content changes its digest', () => {
|
||||
const base = { ...QC_19_RAILS_FILES_PRESENT } as CheckDefinitionSpec;
|
||||
const baseDigest = digestOfSpec(base);
|
||||
|
||||
const changedParams: CheckDefinitionSpec = {
|
||||
...base,
|
||||
params: {
|
||||
expectedFilesByKind: {
|
||||
...(base.params['expectedFilesByKind'] as Record<string, string[]>),
|
||||
node: ['.eslintrc', 'biome.json', '.githooks/pre-commit', 'PR-CHECKLIST.md', 'NEW.md'],
|
||||
},
|
||||
},
|
||||
};
|
||||
expect(digestOfSpec(changedParams)).not.toBe(baseDigest);
|
||||
|
||||
const changedVersion: CheckDefinitionSpec = { ...base, version: '1.1.0' };
|
||||
expect(digestOfSpec(changedVersion)).not.toBe(baseDigest);
|
||||
});
|
||||
|
||||
it('a definition with changed content produces a different recorded digest and version', async () => {
|
||||
const dir = await scaffoldFixture('node');
|
||||
const modified = defineCheck(
|
||||
{ ...QC_19_RAILS_FILES_PRESENT, version: '2.0.0' } as unknown as CheckDefinitionSpec,
|
||||
async () => ({ status: 'passed' }),
|
||||
);
|
||||
const report = await evaluateSubject({
|
||||
subjectPath: dir,
|
||||
checkIds: [QC_19_RAILS_FILES_PRESENT.id],
|
||||
definitions: [
|
||||
modified,
|
||||
...builtInDefinitions().filter((d) => d.id !== QC_19_RAILS_FILES_PRESENT.id),
|
||||
],
|
||||
});
|
||||
const result = firstResult(report);
|
||||
expect(result.checkVersion).toBe('2.0.0');
|
||||
expect(report.definitionDigests[QC_19_RAILS_FILES_PRESENT.id]).toBe(modified.definitionDigest);
|
||||
expect(modified.definitionDigest).not.toBe(QC_19_RAILS_FILES_PRESENT.definitionDigest);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── aggregate state ordering (MACP-style discipline) ───────────────────────
|
||||
|
||||
describe('aggregate state precedence', () => {
|
||||
const result = (status: 'passed' | 'failed' | 'blocked' | 'error') => ({
|
||||
status,
|
||||
checkId: 'x',
|
||||
checkVersion: '1.0.0',
|
||||
subject: '/tmp/x',
|
||||
});
|
||||
|
||||
it('all passed (with not-applicable) ⇒ passed', () => {
|
||||
expect(
|
||||
aggregateState([
|
||||
result('passed'),
|
||||
{ ...result('passed'), status: 'not-applicable' as const },
|
||||
]),
|
||||
).toBe('passed');
|
||||
});
|
||||
|
||||
it('error outranks blocked and failed; blocked outranks failed', () => {
|
||||
expect(aggregateState([result('blocked'), result('error')])).toBe('error');
|
||||
expect(aggregateState([result('failed'), result('blocked')])).toBe('blocked');
|
||||
expect(aggregateState([result('passed'), result('failed')])).toBe('failed');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,183 @@
|
||||
import { constants } from 'node:fs';
|
||||
import { access, stat } from 'node:fs/promises';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import { createSpawnProcessAdapter } from './adapter.js';
|
||||
import { builtInDefinitions, CHECK_SET_POLICY, checkSetForKind } from './definitions.js';
|
||||
import type {
|
||||
AggregateState,
|
||||
CheckResult,
|
||||
CheckStatus,
|
||||
EvaluateOptions,
|
||||
EvaluationReport,
|
||||
ProcessAdapter,
|
||||
Subject,
|
||||
SubjectKind,
|
||||
} from './types.js';
|
||||
import { detectProjectKind } from '../detect.js';
|
||||
|
||||
async function pathExists(targetPath: string): Promise<boolean> {
|
||||
try {
|
||||
await access(targetPath, constants.F_OK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function isDirectory(targetPath: string): Promise<boolean> {
|
||||
try {
|
||||
return (await stat(targetPath)).isDirectory();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Subject-kind detection for the evaluator. Extends the scaffold detection
|
||||
* (detect.ts) with the `monorepo` kind: a pnpm workspace is this repository's
|
||||
* own subject shape and carries a different rails file set (probe-inventory
|
||||
* gap 7 — check sets are per subject, not one global file list).
|
||||
*/
|
||||
export async function detectSubjectKind(subjectPath: string): Promise<SubjectKind> {
|
||||
if (await pathExists(join(subjectPath, 'pnpm-workspace.yaml'))) {
|
||||
return 'monorepo';
|
||||
}
|
||||
const kind = await detectProjectKind(subjectPath);
|
||||
return kind;
|
||||
}
|
||||
|
||||
/**
|
||||
* Aggregate state, MACP-style discipline: `passed` only when at least one
|
||||
* check produced a verdict AND every verdict is `passed` or an explicitly
|
||||
* qualified `not-applicable`. Precedence is fail-closed: error > blocked >
|
||||
* failed > passed; an empty result list aggregates to `blocked`.
|
||||
*/
|
||||
export function aggregateState(results: readonly CheckResult[]): AggregateState {
|
||||
if (results.length === 0) {
|
||||
return 'blocked';
|
||||
}
|
||||
const has = (status: CheckStatus): boolean => results.some((result) => result.status === status);
|
||||
if (has('error')) {
|
||||
return 'error';
|
||||
}
|
||||
if (has('blocked')) {
|
||||
return 'blocked';
|
||||
}
|
||||
if (has('failed')) {
|
||||
return 'failed';
|
||||
}
|
||||
return 'passed';
|
||||
}
|
||||
|
||||
function reasonFrom(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluate one subject against a set of checks, producing typed verdicts.
|
||||
*
|
||||
* Fail-closed invariants (RI-N4):
|
||||
* - unknown check id → `error` (never passed)
|
||||
* - subject directory absent → every verdict `blocked`
|
||||
* - check implementation threw → `error`
|
||||
* - non-passed without a reason → `error` (no unqualified skips)
|
||||
* - check not applicable → `not-applicable` WITH a reason
|
||||
*/
|
||||
export async function evaluateSubject(options: EvaluateOptions): Promise<EvaluationReport> {
|
||||
const subjectPath = resolve(options.subjectPath);
|
||||
const subject: Subject = {
|
||||
path: subjectPath,
|
||||
kind: await detectSubjectKind(subjectPath),
|
||||
};
|
||||
|
||||
const definitions = options.definitions ?? builtInDefinitions();
|
||||
const byId = new Map(definitions.map((definition) => [definition.id, definition]));
|
||||
const requested = options.checkIds ?? checkSetForKind(subject.kind);
|
||||
const adapter: ProcessAdapter = options.adapter ?? createSpawnProcessAdapter();
|
||||
|
||||
const results: CheckResult[] = [];
|
||||
const definitionDigests: Record<string, string> = {};
|
||||
|
||||
for (const checkId of requested) {
|
||||
const definition = byId.get(checkId);
|
||||
if (definition === undefined) {
|
||||
const known = definitions.map((entry) => entry.id).join(', ');
|
||||
results.push({
|
||||
status: 'error',
|
||||
checkId,
|
||||
checkVersion: 'unknown',
|
||||
subject: subjectPath,
|
||||
reason: `unknown check id '${checkId}' — no registered definition (known: ${known})`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
definitionDigests[checkId] = definition.definitionDigest;
|
||||
|
||||
if (!(await isDirectory(subjectPath))) {
|
||||
results.push({
|
||||
status: 'blocked',
|
||||
checkId,
|
||||
checkVersion: definition.version,
|
||||
subject: subjectPath,
|
||||
reason: `subject directory does not exist: ${subjectPath}`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!definition.appliesTo.includes(subject.kind)) {
|
||||
results.push({
|
||||
status: 'not-applicable',
|
||||
checkId,
|
||||
checkVersion: definition.version,
|
||||
subject: subjectPath,
|
||||
reason: `check '${checkId}' does not apply to subject kind '${subject.kind}'`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const inputs = options.inputs?.[checkId] ?? {};
|
||||
const outcome = await definition.evaluate({
|
||||
subject,
|
||||
params: definition.params,
|
||||
inputs,
|
||||
adapter,
|
||||
});
|
||||
if (outcome.status !== 'passed' && (outcome.reason === undefined || outcome.reason === '')) {
|
||||
results.push({
|
||||
status: 'error',
|
||||
checkId,
|
||||
checkVersion: definition.version,
|
||||
subject: subjectPath,
|
||||
reason: `check returned status '${outcome.status}' without a reason — treated as error`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
results.push({
|
||||
status: outcome.status,
|
||||
checkId,
|
||||
checkVersion: definition.version,
|
||||
subject: subjectPath,
|
||||
reason: outcome.reason,
|
||||
});
|
||||
} catch (error) {
|
||||
results.push({
|
||||
status: 'error',
|
||||
checkId,
|
||||
checkVersion: definition.version,
|
||||
subject: subjectPath,
|
||||
reason: `check implementation threw: ${reasonFrom(error)}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
subject,
|
||||
results,
|
||||
definitionDigests,
|
||||
checkSetVersion: CHECK_SET_POLICY.version,
|
||||
state: aggregateState(results),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
// Evaluator core types — RI-N4 (card RI-3-002, SDLC-D-037 second half).
|
||||
//
|
||||
// The quality-rails evaluator is the SOLE authoritative producer of check
|
||||
// verdicts for the checks it owns. Every verdict is typed and fail-closed:
|
||||
// missing implementations, missing inputs, unknown check ids, process errors,
|
||||
// timeouts, and malformed adapter output can never become `passed` or an
|
||||
// unqualified skip — they surface as `blocked` or `error` with a reason.
|
||||
// (Vocabulary mirrors MACP's GateStatus discipline from packages/macp.)
|
||||
|
||||
/**
|
||||
* Typed verdict for a single check execution.
|
||||
*
|
||||
* - `passed` — the check really ran and its condition held.
|
||||
* - `failed` — the check really ran and its condition did NOT hold.
|
||||
* - `blocked` — the check could not run at all (missing subject, missing
|
||||
* input). Never a green outcome.
|
||||
* - `error` — the check attempted to run but its outcome cannot be trusted
|
||||
* (unknown check id, implementation threw, process error, timeout, malformed
|
||||
* adapter output). Never a green outcome.
|
||||
* - `not-applicable` — the check definition explicitly declares it does not
|
||||
* apply to this subject (a qualified skip, always with a reason).
|
||||
*/
|
||||
export type CheckStatus = 'passed' | 'failed' | 'blocked' | 'error' | 'not-applicable';
|
||||
|
||||
/** Aggregate outcome, MACP-style: `passed` only when every result is green. */
|
||||
export type AggregateState = 'passed' | 'failed' | 'blocked' | 'error';
|
||||
|
||||
/** Kinds of subjects the evaluator can assess. */
|
||||
export type SubjectKind = 'node' | 'python' | 'rust' | 'monorepo' | 'unknown';
|
||||
|
||||
/**
|
||||
* A single check verdict. This is the canonical result shape: `status`,
|
||||
* `checkId`, `checkVersion`, `subject`, `reason`. `reason` is REQUIRED
|
||||
* (enforced by the runner) for every status other than `passed`.
|
||||
*/
|
||||
export interface CheckResult {
|
||||
status: CheckStatus;
|
||||
checkId: string;
|
||||
checkVersion: string;
|
||||
subject: string;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/** The project being evaluated. */
|
||||
export interface Subject {
|
||||
/** Absolute path. */
|
||||
path: string;
|
||||
kind: SubjectKind;
|
||||
}
|
||||
|
||||
/**
|
||||
* The data half of a check definition. Definitions live as DATA with a version
|
||||
* and a content digest (see `digestOfSpec`); the executable half is attached
|
||||
* separately so the digest covers only reviewable, declarative content.
|
||||
*/
|
||||
export interface CheckDefinitionSpec {
|
||||
/** Stable id, e.g. `qc-19-rails-files-present`. */
|
||||
id: string;
|
||||
/** Semver of this definition's data+semantics. */
|
||||
version: string;
|
||||
/** Canonical check id from docs/release-integrity/probe-inventory.md (QC-n). */
|
||||
canonicalCheck: string;
|
||||
description: string;
|
||||
/** Subject kinds this check can assess (others yield `not-applicable`). */
|
||||
appliesTo: readonly SubjectKind[];
|
||||
/** Declarative parameters (file lists, markers, timeouts) — digest-covered. */
|
||||
params: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** A fully assembled check definition: spec + digest + implementation. */
|
||||
export interface CheckDefinition extends CheckDefinitionSpec {
|
||||
/** sha256 content digest of the spec (canonical JSON projection). */
|
||||
definitionDigest: string;
|
||||
evaluate(ctx: CheckContext): Promise<CheckOutcome>;
|
||||
}
|
||||
|
||||
/** What a check implementation returns; the runner stamps id/version/subject. */
|
||||
export interface CheckOutcome {
|
||||
status: CheckStatus;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/** Caller-provided inputs for one check invocation (e.g. the QC-20 probe path). */
|
||||
export type CheckInputs = Record<string, unknown>;
|
||||
|
||||
/** Everything a check implementation may use. */
|
||||
export interface CheckContext {
|
||||
subject: Subject;
|
||||
params: Record<string, unknown>;
|
||||
inputs: CheckInputs;
|
||||
adapter: ProcessAdapter;
|
||||
}
|
||||
|
||||
/** Outcome of running a shell probe through the thin process adapter. */
|
||||
export type AdapterOutcome =
|
||||
| { ok: true; exitCode: number | null; stdout: string; stderr: string }
|
||||
| { ok: false; kind: 'spawn-error' | 'timeout'; message: string };
|
||||
|
||||
/** Request for the process adapter. */
|
||||
export interface AdapterRequest {
|
||||
file: string;
|
||||
args: string[];
|
||||
cwd: string;
|
||||
timeoutMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Thin process adapter: runs a command, owns NO verdict logic. Verdict parsing
|
||||
* always lives in the check implementation (TS), never in the shell probe.
|
||||
*/
|
||||
export interface ProcessAdapter {
|
||||
run(request: AdapterRequest): Promise<AdapterOutcome>;
|
||||
}
|
||||
|
||||
/** Per-subject-kind check-set selection policy (versioned and digested). */
|
||||
export interface CheckSetPolicySpec {
|
||||
version: string;
|
||||
byKind: Record<SubjectKind, readonly string[]>;
|
||||
}
|
||||
|
||||
export interface CheckSetPolicy extends CheckSetPolicySpec {
|
||||
/** sha256 content digest of the policy spec. */
|
||||
policyDigest: string;
|
||||
}
|
||||
|
||||
/** Full typed evaluation report for one subject. */
|
||||
export interface EvaluationReport {
|
||||
subject: Subject;
|
||||
results: CheckResult[];
|
||||
/** checkId → content digest of the definition that produced the verdicts. */
|
||||
definitionDigests: Record<string, string>;
|
||||
/** Version of the check-set policy used for subject selection. */
|
||||
checkSetVersion: string;
|
||||
state: AggregateState;
|
||||
}
|
||||
|
||||
/** Options for `evaluateSubject`. */
|
||||
export interface EvaluateOptions {
|
||||
subjectPath: string;
|
||||
/** Restrict to these check ids; defaults to the subject kind's check set. */
|
||||
checkIds?: string[];
|
||||
/** Per-check inputs, keyed by check id (e.g. `{ 'qc-20-enforcement-verify': { probePath } }`). */
|
||||
inputs?: Record<string, CheckInputs>;
|
||||
/** Replace the built-in definitions (tests / future batches). */
|
||||
definitions?: CheckDefinition[];
|
||||
/** Inject a process adapter (tests / instrumentation). */
|
||||
adapter?: ProcessAdapter;
|
||||
}
|
||||
@@ -3,3 +3,35 @@ export * from './detect.js';
|
||||
export * from './scaffolder.js';
|
||||
export * from './templates.js';
|
||||
export * from './types.js';
|
||||
|
||||
// RI-N4 evaluator (card RI-3-002): the public, programmatic entry points.
|
||||
export {
|
||||
builtInDefinitions,
|
||||
CHECK_SET_POLICY,
|
||||
checkSetForKind,
|
||||
defineCheck,
|
||||
QC_19_RAILS_FILES_PRESENT,
|
||||
QC_20_ENFORCEMENT_VERIFY,
|
||||
} from './evaluator/definitions.js';
|
||||
export { canonicalJson, digestContent, digestOfPolicy, digestOfSpec } from './evaluator/digest.js';
|
||||
export { createSpawnProcessAdapter } from './evaluator/adapter.js';
|
||||
export { aggregateState, detectSubjectKind, evaluateSubject } from './evaluator/runner.js';
|
||||
export type {
|
||||
AdapterOutcome,
|
||||
AdapterRequest,
|
||||
AggregateState,
|
||||
CheckContext,
|
||||
CheckDefinition,
|
||||
CheckDefinitionSpec,
|
||||
CheckInputs,
|
||||
CheckOutcome,
|
||||
CheckResult,
|
||||
CheckSetPolicy,
|
||||
CheckSetPolicySpec,
|
||||
CheckStatus,
|
||||
EvaluateOptions,
|
||||
EvaluationReport,
|
||||
ProcessAdapter,
|
||||
Subject,
|
||||
SubjectKind,
|
||||
} from './evaluator/types.js';
|
||||
|
||||
@@ -22,6 +22,12 @@
|
||||
// format | format | pnpm format:check
|
||||
// test | test | pnpm test
|
||||
// build | publish.yml build | pnpm build
|
||||
// quality-rails | (canonical-only) | the TS quality-rails evaluator
|
||||
// | | (RI-N4, QC-19 monorepo subject). Like
|
||||
// | | `build`, this stage has no ci.yml
|
||||
// | | mirror; it is implemented by
|
||||
// | | importing the evaluator CLI rather
|
||||
// | | than duplicating its presence logic.
|
||||
//
|
||||
// Caller-provided prerequisites (kept at the pipeline level — see the comments
|
||||
// in .woodpecker/ci.yml): `bash` + `rsync` for the guard stages, `openssl` and
|
||||
@@ -90,6 +96,15 @@ export const STAGES = [
|
||||
name: 'build',
|
||||
commands: ['pnpm build'],
|
||||
},
|
||||
{
|
||||
// RI-N4 (QC-19, card RI-3-002): the typed quality-rails evaluator, invoked
|
||||
// as the implementation of the check it owns instead of a duplicated
|
||||
// presence loop here. Canonical-only stage (no ci.yml mirror — same shape
|
||||
// as `build`); runs AFTER build so the evaluator's dist/ exists. Subject
|
||||
// is this repository (`.` → monorepo subject kind, per-subject check set).
|
||||
name: 'quality-rails',
|
||||
commands: ['node packages/quality-rails/dist/cli.js quality-rails evaluate --project .'],
|
||||
},
|
||||
];
|
||||
|
||||
export function stageByName(name) {
|
||||
|
||||
@@ -4,7 +4,7 @@ import { createRequire } from 'node:module';
|
||||
import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
import { STAGES } from './verify-release.mjs';
|
||||
import { STAGES, stageByName } from './verify-release.mjs';
|
||||
|
||||
// SDLC-D-034 checkout invariant: publication in .woodpecker/publish.yml is
|
||||
// bound to exact-commit terminal verification. This suite parses the real
|
||||
@@ -229,10 +229,21 @@ steps:
|
||||
function assertStagesMirrorCi(stages, ci) {
|
||||
const canonical = Object.fromEntries(stages.map((stage) => [stage.name, stage.commands]));
|
||||
|
||||
// The complete mandatory set, in gate order.
|
||||
// The complete mandatory set, in gate order. `quality-rails` is a
|
||||
// canonical-only stage (RI-N4, QC-19): like `build`, it has no ci.yml
|
||||
// mirror to match — its contract is asserted separately below.
|
||||
assert.deepEqual(
|
||||
stages.map((stage) => stage.name),
|
||||
['sanitization', 'upgrade-guard', 'typecheck', 'lint', 'format', 'test', 'build'],
|
||||
[
|
||||
'sanitization',
|
||||
'upgrade-guard',
|
||||
'typecheck',
|
||||
'lint',
|
||||
'format',
|
||||
'test',
|
||||
'build',
|
||||
'quality-rails',
|
||||
],
|
||||
);
|
||||
|
||||
// Guard stages: ci.yml commands minus its `apk add` environment prep must be
|
||||
@@ -301,3 +312,38 @@ test('the root package.json exposes verify:release as the canonical command', as
|
||||
const packageJson = JSON.parse(await readFile(path.join(process.cwd(), 'package.json'), 'utf8'));
|
||||
assert.match(packageJson.scripts['verify:release'], /scripts\/verify-release\.mjs/);
|
||||
});
|
||||
|
||||
// RI-N4 (card RI-3-002): the `quality-rails` stage must route through the TS
|
||||
// evaluator instead of duplicating its presence logic inline. The evaluator
|
||||
// owns QC-19; this file keeps that delegation honest.
|
||||
function assertEvaluatorStage(stage) {
|
||||
assert.ok(stage, 'canonical stages must include a quality-rails stage');
|
||||
assert.ok(Array.isArray(stage.commands) && stage.commands.length > 0);
|
||||
for (const command of stage.commands) {
|
||||
assert.match(
|
||||
command,
|
||||
/packages\/quality-rails\/dist\/cli\.js.*quality-rails evaluate/,
|
||||
`quality-rails stage command must invoke the evaluator CLI, got: '${command}'`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
test('the quality-rails stage invokes the evaluator rather than duplicating its logic', () => {
|
||||
assertEvaluatorStage(stageByName('quality-rails'));
|
||||
});
|
||||
|
||||
test('a quality-rails stage that re-implements presence logic inline fails the checker', () => {
|
||||
// Negative control: replacing the evaluator invocation with an inline
|
||||
// `test -f` presence loop is exactly the duplication RI-N4 forbids — the
|
||||
// checker must go red on it.
|
||||
const duplicated = {
|
||||
name: 'quality-rails',
|
||||
commands: ['test -f .husky/pre-commit && test -f .husky/pre-push'],
|
||||
};
|
||||
assert.throws(() => assertEvaluatorStage(duplicated), /must invoke the evaluator CLI/);
|
||||
});
|
||||
|
||||
test('a quality-rails stage that silently drops the evaluator command fails the checker', () => {
|
||||
const empty = { name: 'quality-rails', commands: [] };
|
||||
assert.throws(() => assertEvaluatorStage(empty), /commands/);
|
||||
});
|
||||
|
||||
@@ -133,10 +133,13 @@ When the full `@mosaicstack/forge` package is available, Forge uses MACP task ex
|
||||
|
||||
```bash
|
||||
# Run from CLI
|
||||
mosaic forge run path/to/brief.md
|
||||
# Fails closed with a typed FORGE_NO_EXECUTOR capability error when no real
|
||||
# executor is wired — pass --simulate to opt into explicit typed simulation
|
||||
# (every result carries status `simulated`, which satisfies nothing).
|
||||
mosaic forge run path/to/brief.md [--simulate]
|
||||
|
||||
# Resume interrupted run
|
||||
mosaic forge resume .forge/runs/20260401-143022/
|
||||
# Resume interrupted run (same fail-closed rule as forge run)
|
||||
mosaic forge resume .forge/runs/20260401-143022/ [--simulate]
|
||||
|
||||
# Check status
|
||||
mosaic forge status .forge/runs/20260401-143022/
|
||||
|
||||
Reference in New Issue
Block a user