Files
stack/docs/release-integrity/probe-inventory.md
T
jarvisandfargo 64350892e7
ci/woodpecker/push/publish Pipeline was canceled
docs(ri-050): RI-3-001 complete quality-rails probe inventory (#1275) (#1302)
Co-authored-by: jarvis <[email protected]>
2026-08-18 15:59:34 +00:00

80 KiB
Raw Blame History

Quality-Rails Probe Inventory — RI-3-001

  • Task: RI-3-001 (SDLC-D-037 first half; PRD § Release Integrity Workstream, RI-N4)
  • Date: 2026-08-18
  • Base: origin/next @ 8199261c (branch docs/ri-050-qr-probe-inventory)
  • Follow-up: RI-3-002 consumes the dispositions here when building the single TS evaluator.

0. Scope and method

Every mechanism in this repository that verifies a quality, integrity, safety, or release property — TypeScript checks, shell probes, pipeline steps, git hooks, and installer-side assertions — gets one row. Each row's "what it actually verifies" was written from the probe's code, not its name or docs. Framework tool unit/regression suites (git wrappers, wake, tmux, orchestrator, …) are treated as one enforcement surface (test:framework-shell) because they test tool behavior rather than repo quality; their wiring integrity is itself guarded by check-test-enumeration.sh, and the quality-relevant members are rowed individually.

Kinds: ts (TypeScript/Node check), shell (bash/python probe), pipeline-step (exists only inside a Woodpecker pipeline).

Enforcement points: local (operator-invoked), pre-commit, pre-push, CI ci.yml#<step>, publish.yml#<step> (CI on push to main/next), turbo <task>, agent-runtime (framework hooks on an agent host), installer (host install path), unwired.

Dispositions (recommendations for RI-3-002): preserve (keep as-is; already the canonical or a correct guard-of-the-guard), strengthen (keep, but a concrete gap must close — usually absorption into the TS evaluator), strengthen (review) (viable retirement candidate once the evaluator absorbs it; do not retire yet). Note: RI-N4 requires that effective shell probes be absorbed before their independent paths retire — no row here is marked retire because no absorption exists yet.

1. Inventory

1.1 Repo-level gate tasks (pnpm / turbo)

check location kind what it actually verifies enforcement point canonical check disposition rationale
pnpm preflight (checkout preflight) scripts/preflight.mjs ts Six gate binaries (eslint, husky, prettier, tsc, turbo, vitest) exist and are executable in node_modules/.bin (exit 42 if not); no stale .mosaic-test-work/web-build.lock (exit 43); apps/web/.next is a real directory (not a symlink), every entry owned by the current uid, and its .mosaic-source-hash fingerprint + .mosaic-symlink-manifest hash match the certified build written by scripts/build-web.mjs pre-push; inside pnpm typecheck (→ CI ci.yml#typecheck, verify-release typecheck stage) QC-1 Checkout integrity preserve Blocks a poisoned/stale generated .next from faking a green typecheck (the five-month-stale-.next class); trust chain is self-contained per-checkout.
pnpm typecheck root package.jsonturbo run typecheck ts Per-package tsc --noEmit (all 20 packages); turbo typecheck depends on ^build, so package builds must succeed first; prefixed by checkout preflight CI ci.yml#typecheck; pre-push; verify-release typecheck stage; turbo typecheck QC-2 Workspace typecheck preserve The single workspace-wide type gate; CI and hooks invoke the same task, no divergent checklist.
pnpm lint root package.jsonturbo run lint ts Per-package eslint src under root eslint.config.mjs (ignores dist, .next, framework/**, etc.) CI ci.yml#lint; pre-push; verify-release lint stage; turbo lint QC-3 Workspace lint preserve Same-task invocation from every surface; no second lint definition.
pnpm format:check root package.jsonprettier --check ts Prettier parse/format equality over **/*.{ts,tsx,js,jsx,json,md} minus .prettierignore (generated trees, docs/scratchpads/, venvs, …) CI ci.yml#format; pre-push; verify-release format stage QC-4 Format check preserve Single formatter, single ignore list, enforced identically everywhere.
pnpm test root package.json test = test:checkout && turbo run test && test:installer ts (a) node --test scripts/*.test.mjs — checkout-tool units; (b) per-package vitest run (mosaic appends the 47-command test:framework-shell chain); (c) tools/install-next-lane.test.sh; turbo test declares DB env vars and depends on ^build CI ci.yml#test (with DATABASE_URL + db:migrate first); verify-release test stage; turbo test QC-5 Test suite execution preserve One composed test command; the chain property (any link red ⇒ step red) is the gate.
pnpm build root package.jsonturbo run build ts Per-package build (tsc/Next) with ^build dependency and dist/** outputs publish.yml#build; verify-release build stage; turbo build QC-6 Workspace build preserve Publish artifacts derive from the same build task CI verifies.

1.2 Framework quality shell probes (packages/mosaic/framework/tools/quality/)

check location kind what it actually verifies enforcement point canonical check disposition rationale
Sanitization gate scripts/verify-sanitized.sh shell Built-in self-test first (planted identity/structural/YAML+service fixtures; exit 2 if the regexes or extension coverage break), then: (1) identity denylist grep (jarvis|jason|woltje|brain.woltje.com|/home/jwoltje|\bPDA\b) over all shipped text files including examples/; (2) structural grep for private $HOME/src defaults in shipped scripts excluding examples/. Any hit ⇒ exit 1 CI ci.yml#sanitization; verify-release sanitization stage QC-7 Framework sanitization preserve Labeled one-time regression guard with a self-test that prevents silent no-op; correctly scoped (identity vs structural) and documented as not a general PII detector.
Resident-context budget scripts/check-resident-budget.sh (+ --self-test) shell Self-test of the comparator, then wc -l vs per-file ceilings (CONSTITUTION 120, AGENTS 120, each RUNTIME.md 90); missing file ⇒ fail; over ceiling ⇒ exit 1 CI ci.yml#sanitization (both modes); verify-release sanitization stage QC-8 Resident-context budget preserve Caps the container (lines), never the wording — the deliberate anti-drift design (DESIGN §7); CI-enforceable half only, by design.
Test-membership enumeration guard (#1017) scripts/check-test-enumeration.sh + test-enumeration-exclusions.txt shell Parses surface S1 (packages/mosaic test:framework-shell via JSON+shlex) and S2 (every `framework/tools/*.sh .pytoken inci.yml, comment lines stripped); population = test.shunderframework/tools`; FAILS on: suite-shaped file on disk neither enumerated nor signed-excluded; surface naming a path missing on disk (both directions); exclusion without reason / stale / outside population / contradicting enumeration. Proves naming, not reachability (stated in-file) CI ci.yml#sanitization (direct line); link [0] of test:framework-shell (thus CI ci.yml#test); verify-release sanitization stage QC-9 Test-membership enumeration preserve Makes silent under-run impossible; invoked from both surfaces it audits so severing the chain cannot silence it.
Enumeration-guard needles scripts/test-check-test-enumeration.sh shell Needle/control fixtures driven through --root: every promised failure mode must trip the guard on its own words, plus controls that must pass (null-case defense); covers commented-out ci.yml lines (F1) and line-range parsing (n2b) test:framework-shellCI ci.yml#test; verify-release test stage QC-9 Test-membership enumeration preserve Guard-of-the-guard with both polarities; same canonical check by design.
Upgrade manifest guard (#791 HARD GATE) scripts/test-upgrade-manifest-guard.sh shell Keep-mode install.sh upgrade against seeded throwaway MOSAIC_HOME: every operator sentinel — including an unanticipated one — survives byte-identical with unchanged mtime; framework files still update; retired framework files pruned; matrix run with rsync present AND absent (keep path must be rsync-independent); fail-closed matrix (empty/operator-only/malformed/missing manifest aborts loudly, operator files untouched); operator secret never appears in installer output CI ci.yml#upgrade-guard; verify-release upgrade-guard stage QC-10 Upgrade/install safety preserve The operator-data hard gate for the mosaic update path; negative controls are load-bearing and documented.
Upgrade rollback gate (#791 B1) scripts/test-upgrade-rollback.sh shell Mid-sync failure (PATH-shadowing cp shim) must trigger snapshot restore: restore message fires, corrupted file restored, target byte-identical to pre-upgrade; control installer with set -E stripped must NOT roll back (proves errtrace is load-bearing); plus signal/exit-guard controls CI ci.yml#upgrade-guard; verify-release upgrade-guard stage QC-10 Upgrade/install safety preserve Proves the rollback trap actually fires; the -E-stripped control keeps Part A honest.
Durable-snapshot gate (#791 PR2) scripts/test-upgrade-durable-snapshot.sh shell Pre-update snapshot taken before any mutation (0700/0600 perms, secret never logged, retention-pruned); post-sync verify net restores operator files a manifest bug lets the sync touch; CWE-59 symlink-leaf guard proven with a portable cp shim in both polarities (write-through-link must not happen); v1→v2 migration semantics (intended bin/ removal not healed) CI ci.yml#upgrade-guard; verify-release upgrade-guard stage QC-10 Upgrade/install safety preserve Covers tampering and leak vectors the manifest guard cannot see; the shim rationale (busybox vs GNU cp) is documented in-file.
Install migration matrix (v2→v3) scripts/test-install-migration.sh shell Fixture matrix running the real installer with MOSAIC_SYNC_ONLY=1: fresh install seeds + stamps version 3; legacy user-edited AGENTS overwritten with .pre-constitution.bak preserved (and idempotent); tuned STANDARDS overwritten; operator files (SOUL, credentials) preserved. Mirrors the TS suite packages/mosaic/src/config/file-adapter.test.ts — both installers must behave identically CI ci.yml#upgrade-guard; verify-release upgrade-guard stage QC-10 Upgrade/install safety preserve Pins the shell/TS installer parity contract; removal would orphan that parity requirement.
Enforcement verification probe (bash) scripts/verify.sh shell Attempts real commits in the target repo: planted type error must produce a commit blocked with error; planted any must trip no-explicit-any; planted lint error must trip prettier; gitleaks binary must exist (3a) and detect a planted AWS key via gitleaks git --pre-commit --staged --redact (3b). Verdicts are output-grep matches on hook stderr local via installed mosaic-quality-verify on scaffolded target projects; not run in this repo's CI QC-20 Downstream enforcement verification strengthen (review) Mechanism is genuinely behavioral (stronger than file presence) but verdict logic is grep-on-output and it is unwired here; absorb as the evaluator's enforcement-probe check (the RI-N4 evaluator invokes it or reimplements it) before retiring the shell path.
Enforcement verification probe (PowerShell) scripts/verify.ps1 shell Windows port of verify.sh: same planted-commit tests with $output -match matching; no gitleaks self-test parity beyond the same checks local (Windows operator); no Windows CI runner exists QC-20 Downstream enforcement verification strengthen (review) A hand-maintained twin of verify.sh with no CI coverage — exactly the drift shape the single evaluator removes; retire after the TS evaluator owns the probe.
Quality template installer (bash) scripts/install.sh shell Copies template files (.husky/pre-commit incl. mandatory gitleaks, .lintstagedrc.js, .eslintrc.js, tsconfig.json, .woodpecker.yml, .gitleaks.toml) into a target project; warns (does not verify) about package.json snippet merge; no post-condition check local / via mosaic-quality-apply QC-21 Downstream rails scaffolding strengthen (review) Duplicates the TS quality-rails init scaffolder for a different template set; converging on one scaffolder (with post-scaffold verification) is prerequisite to retiring this path.
Quality template installer (PowerShell) scripts/install.ps1 shell Windows twin of the template copy above local (Windows operator) QC-21 Downstream rails scaffolding strengthen (review) Same twin-drift risk as verify.ps1; no runner exercises it.
mosaic-quality-verify adapter framework/tools/_scripts/mosaic-quality-verify shell Thin adapter: validates target dir exists, asserts verify.sh present+executable, cd target, exec it. No verdict logic of its own local (installed framework bin) QC-20 Downstream enforcement verification preserve Already the thin-adapter shape RI-N4 prescribes for shell surfaces.
mosaic-quality-apply adapter framework/tools/_scripts/mosaic-quality-apply shell Thin adapter: arg validation then exec of quality install.sh --template … --target … local (installed framework bin) QC-21 Downstream rails scaffolding preserve Thin adapter, no separate verdict; disposition follows its target script's convergence.
Roster schema regression scripts/test-roster-schema.py shell jsonschema Draft202012Validator over fleet/roster.schema.json with valid/invalid connector-kind fixtures (tmux/discord/matrix conditional fields) unwired — not on S1 or S2, not signed-excluded; also outside the enumeration guard's *.sh population, so the guard cannot see it QC-5 Test suite execution strengthen (review) A real regression suite that currently runs nowhere; wire it into a CI surface or sign an exclusion — leaving it invisible re-arms the exact gap #1017 closed.
Framework shell chain (S1) packages/mosaic/package.json test:framework-shell shell 47-command && chain: enumeration guard + needles, 14 lease-broker/mutator-gate python unitests, check-runtime-launches.py, and ~30 framework-tool shell suites (git wrappers, wake, woodpecker, tmux, glpi, orchestrator, _scripts). Quality-relevant members rowed separately below turbo testCI ci.yml#test; verify-release test stage QC-5 Test suite execution preserve The chain is the execution surface the enumeration guard audits; known residuals: a failing link stops later suites (measured in #1270 — suites after position 44 had not run), and the guard proves naming, not reachability.

1.3 Framework runtime hooks and their harnesses (agent-host enforcement)

check location kind what it actually verifies enforcement point canonical check disposition rationale
QA edit hook seam framework/tools/qa/qa-hook-stdin.sh (+ qa-hook-handler.sh) shell PostToolUse stdin hook: extracts edited file from the tool JSON (jq or grep fallback), skips non-JS/TS, then the deps-preflight gate — exits 1 with the legible sentinel deps not installed — run pnpm install when node_modules/.bin is missing/empty (the #856 false-red class); the downstream handler only files QA remediation report templates (no verification logic) agent-runtime (framework runtime/claude/settings.json PostToolUse); never CI QC-16 Agent-runtime edit-time checks strengthen (review) The sentinel gate is real enforcement; the handler's report-filing adds no verdict and its name promises more than the code does — evaluator absorption should keep the sentinel, drop the report theater.
Typecheck-on-edit hook framework/tools/qa/typecheck-hook.sh shell PostToolUse: for edited .ts/.tsx, finds nearest tsconfig.json and runs tsc --noEmit, surfacing errors nonzero to the agent immediately agent-runtime (framework runtime/claude/settings.json PostToolUse) QC-16 Agent-runtime edit-time checks strengthen (review) Edit-time duplicate of QC-2 with independent invocation logic; keep behavior, converge invocation through the evaluator adapter.
Deps-preflight harness framework/tools/qa/test-deps-preflight.sh shell Five assertions against the seam incl. a documented RED control (raw not found), sentinel behavior for missing and empty .bin, and no-false-positive once populated test:framework-shellCI ci.yml#test QC-16 Agent-runtime edit-time checks preserve Guard-of-the-check with a red control; keeps the sentinel from regressing.
Prompt-helper RCE regression framework/tools/_scripts/test-mosaic-init-rce.sh shell Sources the prompt helpers and proves a literal $(touch /tmp/pwned) answer round-trips verbatim and never executes (no /tmp/pwned created) test:framework-shellCI ci.yml#test QC-5 Test suite execution preserve Cheap, load-bearing security regression on the installer's input path.
Install-ordering harness (#869 C2) framework/tools/_scripts/test-install-ordering-guard.sh shell Drives mosaic-link-runtime-assets with a fake mosaic on PATH: probe ok ⇒ settings copied + exit 0; probe fail ⇒ exit 1 with degraded outcome but all other runtime files still copied; --allow-inactive-enforcement forwarded; no-mosaic-on-PATH ⇒ python3 fallback strips enforcement hooks and exits 1; fallback + flag ⇒ wires as-is, exit 0 test:framework-shellCI ci.yml#test QC-17 Lease-enforcement wiring safety preserve Exercises the shell wiring seam independently of the TS guard's own spec suite (complementary coverage, by design).
Fleet-transport harness (#1240) framework/tools/_scripts/test-fleet-transport-check.sh shell Extracts the shipped check_fleet_transport/fleet_declared_transport functions from the shipped scripts (fails loud if extraction yields nothing) and drives both implementations (mosaic-doctor + tools/install.sh) from one case table test:framework-shellCI ci.yml#test QC-18 Operator-host drift audit preserve The anti-drift harness for the one rule shipped twice; extraction-from-source keeps it from testing a stale copy.
Terminal-green contract (RM-61/#1000) framework/tools/woodpecker/test-terminal-green-contract.sh + verify-terminal-green.py shell Red-first fixtures: pipeline JSON variants (service failure, step failure, cancelled, etc.) must produce the correct terminal-green verdict; controls must pass test:framework-shellCI ci.yml#test QC-5 Test suite execution preserve Keeps the CI-wait wrapper's green-detection honest; a false green here would poison every merge gate that trusts pr-ci-wait.sh.
Lease-gate launch invariant framework/tools/lease-broker/check-runtime-launches.py shell Scans production roots (packages/, apps/, plugins/, tools/) across sh/py/ts/yaml suffixes for Claude/Pi process launches outside the lease gate; allowlist-based; fails CI on violation test:framework-shellCI ci.yml#test QC-15 Lease-gate architecture invariant preserve The only architectural "no ungated launches" rail; grep+allowlist is the right cost/benefit for this invariant.

1.4 TypeScript quality logic (@mosaicstack/quality-rails + mosaic CLI)

check location kind what it actually verifies enforcement point canonical check disposition rationale
quality-rails check packages/quality-rails/src/cli.ts (mosaic quality-rails check --project) ts Expected-file presence only: loops expectedFilesForKind (node: .eslintrc, biome.json, .githooks/pre-commit, PR-CHECKLIST.md; python: pyproject.toml+hooks+checklist; rust: rustfmt.toml+…) and exits 1 listing missing paths. Does not execute any linter, formatter, hook, or scanner local (operator CLI); no CI wiring in this repo QC-19 Downstream rails presence check strengthen This is the RI-N4 evaluator seed. Today presence ≠ parity (explicitly called out by RI-N4): it must grow typed verdicts (passed/failed/blocked/error/not-applicable), check versioning/subject/reason, digested definitions, and absorb the effective shell probes (QC-20 first).
quality-rails doctor packages/quality-rails/src/cli.ts ts Same presence data as check, printed with ok/missing lines; cannot fail (no nonzero exit on missing files) local (operator CLI) QC-19 Downstream rails presence check strengthen A doctor that cannot fail is advisory; fold into check (or return typed states) when the evaluator lands.
quality-rails init packages/quality-rails/src/cli.ts + scaffolder.ts/templates.ts ts Scaffolds rails files per detected kind/profile (linters/formatters lists are advisory strings; hooks flag always true); writes files, prints follow-ups — no post-condition verification local (operator CLI) QC-21 Downstream rails scaffolding strengthen (review) Second scaffolding path alongside quality install.sh (§1.2); converge on one with post-scaffold verification before retiring either.
Lease activation probe (#869 C1, hidden) packages/mosaic/src/commands/lease-activation-probe.ts ts Real capability probe, not file presence: resolves the installed mosaic CLI and requires it to advertise the exact {name, version} activation contract; all deps injectable; registered as hidden CLI command and consumed by C2/C5 local (hidden CLI + consumed by C2/C5); spec-tested via lease-activation-probe.spec.ts in turbo test QC-17 Lease-enforcement wiring safety preserve The versioned-contract probe is precisely the fail-closed capability check RI-N2 generalizes; already typed and injectable.
Install-ordering guard (#869 C2, hidden) packages/mosaic/src/commands/install-ordering-guard.ts ts Decides whether enforcement hook entries are written into the ~/.claude/settings.json the framework reseed ships: not activatable ⇒ strip hooks + nonzero loud outcome (default); explicit per-invocation --allow-inactive-enforcement opt-out wires-with-warning. Never touches the runtime gate's own fail-closed behavior installer (framework reseed via mosaic-link-runtime-assets); spec + shell harness coverage in turbo test QC-17 Lease-enforcement wiring safety preserve Correct default-deny with an explicit, non-env opt-out; test-locked from both the TS and shell sides.
Lease doctor check (#869 C5) packages/mosaic/src/commands/lease-doctor-check.ts ts Combines hook-wiring detection in ~/.claude/settings.json with C1 activatable and C3 broker-supervisor health: wired ∧ (¬activatable ¬healthy) ⇒ loud [ERROR] that forces mosaic doctor exit 1 regardless of the bash audit's own exit local (inside mosaic doctor); spec coverage in turbo test QC-17 Lease-enforcement wiring safety preserve Closes the "bricked host looks green" hole; cannot be masked by the bash script — that composition is the point.
mosaic doctor (framework drift audit) packages/mosaic/src/commands/launch.ts (doctor) + framework/tools/_scripts/mosaic-doctor shell+ts Bash audit of the installed framework home: ~40 expected files/dirs present; runtime files are copies (not symlinks) matching source (cmp) or composed runtime-contract markers; hard-gates block present in AGENTS.md; sequential-thinking MCP configured; fleet transport binary present per roster (warn); legacy symlink trees gone; skills synced — warn-based, exit 1 only with --fail-on-warn, plus C5's forced error local (operator audit) QC-18 Operator-host drift audit preserve Host-state audit CI cannot see (user files by design, DESIGN §7); advisory exit is the documented contract — do not silently change it.
mosaic gateway doctor packages/mosaic/src/commands/gateway-doctor.ts ts Probes per-service health (PostgreSQL, Valkey, pgvector) via @mosaicstack/storage, reports tier and JSON; exit 1 only when at least one required service fails (yellow stays 0) local (operator) QC-18 Operator-host drift audit preserve Service health with correct red/yellow exit semantics; JSON mode exists for scripting.
mosaic gateway verify packages/mosaic/src/commands/gateway/verify.ts ts Post-install liveness: daemon meta via HTTP with retries, admin token on file, bootstrap endpoint reachable; aggregated pass/fail local; consumed by tools/e2e-install-test.sh QC-18 Operator-host drift audit preserve The first-run proof the installer E2E relies on; retry-aware so startup races don't false-red.
mosaic fleet doctor packages/mosaic/src/commands/fleet-reconciler-command.ts ts Classifies local roster-owned drift (no mutation) from the parsed v2 roster local (operator) QC-18 Operator-host drift audit preserve Dry-run classification is the correct non-mutating audit shape.

1.5 Git hooks (developer machine)

check location kind what it actually verifies enforcement point canonical check disposition rationale
Pre-commit staged hygiene .husky/pre-commitnpx lint-staged (.lintstagedrc) shell On staged files only: prettier --write + eslint --fix for ts/tsx/js/jsx; prettier --write for json/md/yaml/yml. Mutating (fixes and re-stages); commit blocks only if a fixer itself fails pre-commit (every local commit; hooks activated by install-hooks.mjs via core.hooksPath .husky/_) QC-13 Staged-change hygiene preserve Correct scoped fast gate; note it auto-fixes rather than rejects (deliberate). Gap: no secret scan here — see §3.
Pre-push gate .husky/pre-push shell pnpm preflight && pnpm typecheck && pnpm lint && pnpm format:check (no test run — documented in AGENTS.md) pre-push QC-14 Pre-push gate preserve Composes QC-1..4 exactly as specified in AGENTS.md; tests intentionally left to CI.
Hook installer scripts/install-hooks.mjs (pnpm prepare) ts Stages husky hooks into a scratch repo first, asserts husky produced its h shim, quarantines incomplete previous sets, verifies idempotence via full directory snapshot comparison, then sets core.hooksPath; skips cleanly with HUSKY=0 or no git installer (runs on pnpm install) QC-13 Staged-change hygiene preserve Self-verifying wiring for the hook gates — a corrupted half-install cannot silently disable them.

1.6 CI pipeline steps (.woodpecker/)

Step-to-probe mapping for container steps: ci.yml#sanitization = QC-7+QC-8+QC-9 (rows §1.2, plus apk add bash env prep); ci.yml#upgrade-guard = QC-10 (rows §1.2, plus apk add rsync); ci.yml#typecheck/#lint/#format/#test = QC-2/3/4/5 (rows §1.1). Rows below are mechanisms that exist only in a pipeline.

check location kind what it actually verifies enforcement point canonical check disposition rationale
Frozen install ci.yml#install pipeline-step pnpm install --frozen-lockfile --prefer-offline against the baked ci-base store — lockfile supply integrity; a drifted lockfile fails the build before any gate runs CI ci.yml#install QC-1 Checkout integrity preserve Lockfile-pinned dep resolution is the supply-chain floor under every later gate.
Test-step readiness prelude ci.yml#test prologue pipeline-step Installs pinned @earendil-works/[email protected] (Invariant R suite requires the real binary) + openssl; waits up to 60×1s on pg_isready for the ci-postgres service and fails fast if it never comes up; runs db:migrate before tests CI ci.yml#test QC-5 Test suite execution preserve Fail-fast environment preconditions — a missing service produces a legible failure, not a wall of red tests.
Publish verify step (pending RI-1-001) publish.yml#verify (branch feat/ri-050-publish-gate @ 46784c8d, not yet on next) pipeline-step (a) Commit identity: fails closed if CI_COMMIT_SHA empty, git rev-parse HEAD empty, or the two differ; (b) runs the canonical pnpm verify:release. Every publish effect depends on this step; it carries no path filter publish.yml#verify QC-11 Terminal release verification preserve The RI-N1 exact-commit binding; until it merges, publish steps on next depend on build only (see §3 gap 1).
Publish error classification publish.yml#publish-npm pipeline-step Publishes @mosaicstack/* (minus web) and classifies outcome: success, or the only tolerated failure = already-published (EPUBLISHCONFLICT / "cannot publish over" / "previously published"); explicit fatal on npm E404/E401/ENEEDAUTH/ECONNREFUSED/ETIMEDOUT/ENOTFOUND and on any unrecognized failure (replacing the old ` echo` that hid a registry 404) publish.yml#publish-npm (main/tags, path-filtered on packages/**) QC-12 Publish-effect integrity preserve Converts silent publish fall-on-floor into loud failure; allowlist-of-one error tolerance is the right shape.
Next-lane publish assertions publish.yml#publish-next-npm pipeline-step Guards: branch must be next, CI_PIPELINE_NUMBER required; registry dist-tags JSON must be usable; walks all manifests, strictly parses stable semver, rewrites X.Y.(Z+1)-next.<N>; publishes with --tag next (never latest); post-publish asserts npm view @mosaicstack/mosaic@next resolves to the exact expected version publish.yml#publish-next-npm (push/manual on next) QC-12 Publish-effect integrity preserve Durable prerelease lane with end-to-end resolution proof — the published artifact is verified, not assumed.
Image destination policy publish.yml#build-gateway / #build-appservice / #build-web pipeline-step Kaniko builds with destination policy: next ⇒ sha-tag only (fatal if a tag event sneaks in); main ⇒ sha + latest; tag events ⇒ sha + <tag>; anything else fatal. Path filters only skip effects, never the verify step publish.yml#build-* QC-12 Publish-effect integrity preserve Fail-closed tagging matrix; the exclude-list default-safe design keeps stale images impossible.

Adjacent pipeline surface (not a probe): .woodpecker/ci-image.yml rebuilds the ci-base image on pnpm-lock.yaml/Dockerfile.ci change with an immutable lock-<hash> tag; pipelines consume :latest. Recorded for completeness — no code-quality property is checked.

1.7 Root installer tooling (tools/)

check location kind what it actually verifies enforcement point canonical check disposition rationale
Next-lane installer test tools/install-next-lane.test.sh (pnpm test:installer) shell Drives tools/install.sh --next with faked node/npm binaries (no network): Node 20 must be rejected; installs must pin exact versions (mutable @next forbidden); fast path must not unexpectedly fall back to source; gateway-install failure takes the documented fallback turbo-external tail of pnpm testCI ci.yml#test QC-5 Test suite execution preserve Hermetic (shimmed) regression net for the installer lane; runs as part of the standard test command.
Clean-container install E2E tools/e2e-install-test.sh shell Full first-run flow in a node:22-alpine container: install.sh --yesmosaic wizard (non-interactive) → mosaic gateway installmosaic gateway verify exit check (with EXPECTED-SKIP if the installed CLI predates gateway verify); skips gracefully without Docker local (manual; requires Docker); not wired in CI QC-5 Test suite execution strengthen (review) The only end-to-end proof of the install→verify path; currently operator-initiated only — wire into a periodic/manual CI lane or sign its exclusion explicitly.
Host installer advisories tools/install.sh (--check; check_fleet_transport) shell --check = version comparison only, no install; check_fleet_transport warns (non-blocking, by design — tmux is the fleet's dependency, not mosaic's) when the roster-declared transport binary is absent, naming exactly what it blocks; PATH-persistence warnings installer (operator-run) QC-18 Operator-host drift audit preserve Advisory-by-design warnings; the parallel doctor check is drift-tested by §1.3's harness.

1.8 Pending workstream additions (branch feat/ri-050-publish-gate @ 46784c8d)

check location kind what it actually verifies enforcement point canonical check disposition rationale
Canonical terminal verification scripts/verify-release.mjs (pnpm verify:release) ts One command replaying the full mandatory set as stages — sanitization, upgrade-guard, typecheck (incl. preflight), lint, format, test, build — mirroring ci.yml step-for-step; fail-fast on first failing command; requires bash+rsync on PATH; --stage <name> for wiring smoke-tests only publish.yml#verify (pending); local (pnpm verify:release) QC-11 Terminal release verification preserve The RI-N1 canonical command — CI and publication share one semantic checklist by construction.
Verify-parity contract test scripts/verify-release.test.mjs ts Parses the real ci.yml/publish.yml: stage table must match ci.yml step-for-step; every publish-effect step (name publish* or image-pushing) must transitively depend on verify; commit-identity assertion must be present; verify must carry no path filter test:checkoutCI ci.yml#test (once merged) QC-11 Terminal release verification preserve Guard-of-the-guard at checkout time — the two surfaces cannot drift apart silently.

2. Canonical check set

The deduplicated checks every row above maps onto. IDs are stable for RI-3-002 to consume.

  • QC-1 Checkout integrity. Owns: the checkout can run its gates — frozen-lockfile dependency resolution, required gate binaries present, no stale build lock, and the apps/web/.next generated-state trust chain (real directory, uid ownership, certified source fingerprint, certified symlink manifest). Implemented by scripts/preflight.mjs + frozen install steps.
  • QC-2 Workspace typecheck. Owns workspace-wide TypeScript soundness: per-package tsc --noEmit over built dependencies (turbo typecheck). The single definition invoked by CI, pre-push, and terminal verification.
  • QC-3 Workspace lint. Owns static-analysis policy: per-package ESLint under the root config. One config, one task, every surface.
  • QC-4 Format check. Owns formatting uniformity: Prettier check with the repo ignore list. (The pre-commit variant additionally fixes; the verdict form is this check.)
  • QC-5 Test suite execution. Owns execution of all test surfaces: checkout script units (node --test), per-package Vitest suites (including the framework shell chain and its python unitests), the installer-lane shim test, and — once wired — test-roster-schema.py and container E2E. Also owns guards-of-the-gate that live inside the chain (terminal-green contract, RCE regression).
  • QC-6 Workspace build. Owns artifact buildability: turbo build producing the artifacts publication consumes.
  • QC-7 Framework sanitization. Owns the open-source guarantee for the shipped framework package: no operator-identity tokens anywhere (examples included), no private $HOME defaults in shipped scripts, with a self-test that keeps the regexes honest.
  • QC-8 Resident-context budget. Owns the line-count ceilings on framework files injected into every agent's context (Constitution, dispatcher, RUNTIME.md slices) — the CI-enforceable half of the resident-prompt budget.
  • QC-9 Test-membership enumeration. Owns the property that no test suite can silently fall out of CI: disk population vs parsed enumeration surfaces, both-directions staleness, and signed exclusions with reasons. Includes its needle/control harness.
  • QC-10 Upgrade/install safety. Owns the #791 family: operator-path byte-identity across keep-mode upgrades (manifest guard), mid-failure rollback (errtrace-proven), durable pre-update snapshot + verify net + CWE-59 leaf guard, and the v2→v3 migration matrix with shell/TS parity.
  • QC-11 Terminal release verification. Owns the RI-N1 exact-commit binding: commit-identity assertion plus one canonical command (pnpm verify:release) replaying the complete mandatory set, with every publish effect depending on it; plus the checkout-time parity/DAG contract test that keeps pipeline and command in sync.
  • QC-12 Publish-effect integrity. Owns publication correctness: npm publish error classification (only already-published tolerated), next-lane versioning and post-publish resolution proof, and image destination/tag policy.
  • QC-13 Staged-change hygiene. Owns commit-time hygiene on staged files (prettier/eslint fix-and-restage) and the self-verifying hook wiring that guarantees the gates are actually installed.
  • QC-14 Pre-push gate. Owns the local push composition: preflight + typecheck + lint + format:check (tests deliberately deferred to CI).
  • QC-15 Lease-gate architecture invariant. Owns "no ungated runtime launches in production code": the scan + allowlist over packages/, apps/, plugins/, tools/.
  • QC-16 Agent-runtime edit-time checks. Owns edit-time feedback on agent hosts: the deps-preflight legibility sentinel and typecheck-on-edit, plus their regression harnesses.
  • QC-17 Lease-enforcement wiring safety. Owns the #869 C1/C2/C5 trio: activation capability probe (versioned contract), enforcement-hook wiring gate (default-deny with explicit opt-out), and the doctor check that surfaces a bricked host — with their shell/TS harnesses.
  • QC-18 Operator-host drift audit. Owns host-state health CI cannot see: mosaic doctor drift audit (+ fleet transport, both implementations), fleet doctor roster classification, gateway doctor/gateway verify service health, and installer advisories. Advisory exits are part of the contract.
  • QC-19 Downstream rails presence check. Owns "does a scaffolded project still carry its rails files" — today the TS quality-rails check/doctor presence loop; per RI-N4 this is the seed that must become the typed evaluator (presence alone is explicitly not parity).
  • QC-20 Downstream enforcement verification. Owns "do the rails actually block" on scaffolded projects: the behavioral planted-commit probe (type error, any, lint, gitleaks secret) currently in verify.sh/verify.ps1 behind the mosaic-quality-verify adapter.
  • QC-21 Downstream rails scaffolding. Owns putting rails files into a target project: the shell template installer (+ PowerShell twin) and the TS quality-rails init scaffolder — currently two paths that must converge.

3. Coverage gaps

Enforced nowhere but implied, or named in docs/tooling but not wired:

  1. Publication not yet bound to verification on next. At this base (8199261c), publish.yml publish steps depend on build only; the verify step and scripts/verify-release.mjs exist on feat/ri-050-publish-gate (46784c8d) but are not merged. Until RI-1-001 lands, AC-RI-1's negative control cannot hold on the real pipeline.
  2. Playwright E2E unwired. apps/web ships test:e2e (playwright test) with real suites (admin/auth/chat/navigation.spec.ts); neither pnpm test nor any CI step invokes it. The web UI's user flows are verified only when an operator runs them manually.
  3. No secret scanning on this repo. The framework's own template pre-commit makes gitleaks required, and verify.sh proves detection with a planted key — but this repository's .husky/pre-commit (lint-staged only) and CI run no secret scan. The repo ships the control it does not use.
  4. No dependency audit. The quality .woodpecker.yml templates and docs/CI-SETUP.md specify npm audit --audit-level=high as a pipeline stage; nothing equivalent runs for this repo.
  5. No coverage thresholds. Templates enforce 80% Jest coverage thresholds; this repo's Vitest configs collect coverage with no thresholds — coverage is measured nowhere and enforced nowhere.
  6. test-roster-schema.py invisible. A real jsonschema regression suite wired to no surface and invisible to the enumeration guard (its population is *.sh; the suite is .py). Either enumerate it or sign an exclusion — silence here is the #1017 defect shape.
  7. Presence-checker expectations ≠ this repo. quality-rails check expects .eslintrc, biome.json, .githooks/pre-commit, PR-CHECKLIST.md for node projects — none describe this monorepo (husky, flat eslint config, no biome, no PR-CHECKLIST.md). The evaluator's check set must be per-subject (versioned, digested), not one global file list.
  8. Chain-ordering residual (documented). test:framework-shell is one && chain: a failing link skips every later suite while the step still fails (measured in #1270 — four suites after position 44 had not run since a prior merge). The enumeration guard proves naming, not reachability; both residuals are in-file documented but structurally unfixed.
  9. Signed-exclusion burndown open. 16 signed exclusions remain in test-enumeration-exclusions.txt; several are "unmeasured in CI image" or blocked on missing CI tooling (tmux, setsid) — tracked under #1017/#1271. Each is an enforcement promise deferred, not delivered.
  10. Windows twins unexercised. verify.ps1, install.ps1, mosaic-doctor.ps1 have no runner anywhere (no Windows CI); behavioral drift from their bash twins is undetectable by construction.
  11. QA hook name vs behavior. qa-hook-handler.sh files remediation report templates but performs no verification; the seam's actual gate value is only the deps-preflight sentinel. Anything relying on "QA automation hook" as a check is relying on report-filing.
  12. Two test paths, one gated. CI runs tests against ci-postgres (DATABASE_URL set); the local PGlite path is the documented default (AGENTS.md) until KBN-101-02/101-05. Only the CI path is enforced by pipeline.

4. Disposition summary

disposition rows checks
preserve 43 Every canonical owner (QC-1..QC-18) plus correct guards-of-the-guard and thin adapters: all of §1.1, the CI-invoked framework probes and adapters in §1.2, all of §1.3, the C1/C2/C5 trio and doctors in §1.4, all of §1.5, all pipeline-only steps in §1.6, §1.7 rows 1 and 3, and §1.8.
strengthen 2 quality-rails check and quality-rails doctor (QC-19) — the RI-N4 evaluator seed: typed verdicts, versioned/digested check definitions, per-subject check sets.
strengthen (review) 9 verify.sh + verify.ps1 (QC-20), quality install.sh/install.ps1 + quality-rails init (QC-21 — scaffold-path convergence), test-roster-schema.py (QC-5 — wire or sign), qa-hook-stdin.sh seam + typecheck-hook.sh (QC-16), tools/e2e-install-test.sh (QC-5 — CI lane).
retire 0 None meet the bar: RI-N4 requires effective shell probes be absorbed before their paths retire, and no absorption exists yet. The strengthen (review) rows are the retirement candidates for RI-3-002 once the evaluator owns their behavior.

Row total: 54. Canonical checks: 21 (QC-1..QC-21).