Co-authored-by: jarvis <[email protected]>
80 KiB
Quality-Rails Probe Inventory — RI-3-001
- Task: RI-3-001 (SDLC-D-037 first half; PRD § Release Integrity Workstream, RI-N4)
- Date: 2026-08-18
- Base:
origin/next@8199261c(branchdocs/ri-050-qr-probe-inventory) - Follow-up: RI-3-002 consumes the dispositions here when building the single TS evaluator.
0. Scope and method
Every mechanism in this repository that verifies a quality, integrity, safety, or release
property — TypeScript checks, shell probes, pipeline steps, git hooks, and installer-side
assertions — gets one row. Each row's "what it actually verifies" was written from the
probe's code, not its name or docs. Framework tool unit/regression suites (git wrappers,
wake, tmux, orchestrator, …) are treated as one enforcement surface (test:framework-shell)
because they test tool behavior rather than repo quality; their wiring integrity is itself
guarded by check-test-enumeration.sh, and the quality-relevant members are rowed
individually.
Kinds: ts (TypeScript/Node check), shell (bash/python probe), pipeline-step
(exists only inside a Woodpecker pipeline).
Enforcement points: local (operator-invoked), pre-commit, pre-push,
CI ci.yml#<step>, publish.yml#<step> (CI on push to main/next), turbo <task>,
agent-runtime (framework hooks on an agent host), installer (host install path),
unwired.
Dispositions (recommendations for RI-3-002): preserve (keep as-is; already the
canonical or a correct guard-of-the-guard), strengthen (keep, but a concrete gap must
close — usually absorption into the TS evaluator), strengthen (review) (viable retirement
candidate once the evaluator absorbs it; do not retire yet). Note: RI-N4 requires that
effective shell probes be absorbed before their independent paths retire — no row here
is marked retire because no absorption exists yet.
1. Inventory
1.1 Repo-level gate tasks (pnpm / turbo)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
pnpm preflight (checkout preflight) |
scripts/preflight.mjs |
ts | Six gate binaries (eslint, husky, prettier, tsc, turbo, vitest) exist and are executable in node_modules/.bin (exit 42 if not); no stale .mosaic-test-work/web-build.lock (exit 43); apps/web/.next is a real directory (not a symlink), every entry owned by the current uid, and its .mosaic-source-hash fingerprint + .mosaic-symlink-manifest hash match the certified build written by scripts/build-web.mjs |
pre-push; inside pnpm typecheck (→ CI ci.yml#typecheck, verify-release typecheck stage) |
QC-1 Checkout integrity | preserve | Blocks a poisoned/stale generated .next from faking a green typecheck (the five-month-stale-.next class); trust chain is self-contained per-checkout. |
pnpm typecheck |
root package.json → turbo run typecheck |
ts | Per-package tsc --noEmit (all 20 packages); turbo typecheck depends on ^build, so package builds must succeed first; prefixed by checkout preflight |
CI ci.yml#typecheck; pre-push; verify-release typecheck stage; turbo typecheck |
QC-2 Workspace typecheck | preserve | The single workspace-wide type gate; CI and hooks invoke the same task, no divergent checklist. |
pnpm lint |
root package.json → turbo run lint |
ts | Per-package eslint src under root eslint.config.mjs (ignores dist, .next, framework/**, etc.) |
CI ci.yml#lint; pre-push; verify-release lint stage; turbo lint |
QC-3 Workspace lint | preserve | Same-task invocation from every surface; no second lint definition. |
pnpm format:check |
root package.json → prettier --check |
ts | Prettier parse/format equality over **/*.{ts,tsx,js,jsx,json,md} minus .prettierignore (generated trees, docs/scratchpads/, venvs, …) |
CI ci.yml#format; pre-push; verify-release format stage |
QC-4 Format check | preserve | Single formatter, single ignore list, enforced identically everywhere. |
pnpm test |
root package.json test = test:checkout && turbo run test && test:installer |
ts | (a) node --test scripts/*.test.mjs — checkout-tool units; (b) per-package vitest run (mosaic appends the 47-command test:framework-shell chain); (c) tools/install-next-lane.test.sh; turbo test declares DB env vars and depends on ^build |
CI ci.yml#test (with DATABASE_URL + db:migrate first); verify-release test stage; turbo test |
QC-5 Test suite execution | preserve | One composed test command; the chain property (any link red ⇒ step red) is the gate. |
pnpm build |
root package.json → turbo run build |
ts | Per-package build (tsc/Next) with ^build dependency and dist/** outputs |
publish.yml#build; verify-release build stage; turbo build |
QC-6 Workspace build | preserve | Publish artifacts derive from the same build task CI verifies. |
1.2 Framework quality shell probes (packages/mosaic/framework/tools/quality/)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale | |
|---|---|---|---|---|---|---|---|---|
| Sanitization gate | scripts/verify-sanitized.sh |
shell | Built-in self-test first (planted identity/structural/YAML+service fixtures; exit 2 if the regexes or extension coverage break), then: (1) identity denylist grep (jarvis|jason|woltje|brain.woltje.com|/home/jwoltje|\bPDA\b) over all shipped text files including examples/; (2) structural grep for private $HOME/src defaults in shipped scripts excluding examples/. Any hit ⇒ exit 1 |
CI ci.yml#sanitization; verify-release sanitization stage |
QC-7 Framework sanitization | preserve | Labeled one-time regression guard with a self-test that prevents silent no-op; correctly scoped (identity vs structural) and documented as not a general PII detector. | |
| Resident-context budget | scripts/check-resident-budget.sh (+ --self-test) |
shell | Self-test of the comparator, then wc -l vs per-file ceilings (CONSTITUTION 120, AGENTS 120, each RUNTIME.md 90); missing file ⇒ fail; over ceiling ⇒ exit 1 |
CI ci.yml#sanitization (both modes); verify-release sanitization stage |
QC-8 Resident-context budget | preserve | Caps the container (lines), never the wording — the deliberate anti-drift design (DESIGN §7); CI-enforceable half only, by design. | |
| Test-membership enumeration guard (#1017) | scripts/check-test-enumeration.sh + test-enumeration-exclusions.txt |
shell | Parses surface S1 (packages/mosaic test:framework-shell via JSON+shlex) and S2 (every `framework/tools/*.sh |
.pytoken inci.yml, comment lines stripped); population = test.shunderframework/tools`; FAILS on: suite-shaped file on disk neither enumerated nor signed-excluded; surface naming a path missing on disk (both directions); exclusion without reason / stale / outside population / contradicting enumeration. Proves naming, not reachability (stated in-file) |
CI ci.yml#sanitization (direct line); link [0] of test:framework-shell (thus CI ci.yml#test); verify-release sanitization stage |
QC-9 Test-membership enumeration | preserve | Makes silent under-run impossible; invoked from both surfaces it audits so severing the chain cannot silence it. |
| Enumeration-guard needles | scripts/test-check-test-enumeration.sh |
shell | Needle/control fixtures driven through --root: every promised failure mode must trip the guard on its own words, plus controls that must pass (null-case defense); covers commented-out ci.yml lines (F1) and line-range parsing (n2b) |
test:framework-shell → CI ci.yml#test; verify-release test stage |
QC-9 Test-membership enumeration | preserve | Guard-of-the-guard with both polarities; same canonical check by design. | |
| Upgrade manifest guard (#791 HARD GATE) | scripts/test-upgrade-manifest-guard.sh |
shell | Keep-mode install.sh upgrade against seeded throwaway MOSAIC_HOME: every operator sentinel — including an unanticipated one — survives byte-identical with unchanged mtime; framework files still update; retired framework files pruned; matrix run with rsync present AND absent (keep path must be rsync-independent); fail-closed matrix (empty/operator-only/malformed/missing manifest aborts loudly, operator files untouched); operator secret never appears in installer output |
CI ci.yml#upgrade-guard; verify-release upgrade-guard stage |
QC-10 Upgrade/install safety | preserve | The operator-data hard gate for the mosaic update path; negative controls are load-bearing and documented. |
|
| Upgrade rollback gate (#791 B1) | scripts/test-upgrade-rollback.sh |
shell | Mid-sync failure (PATH-shadowing cp shim) must trigger snapshot restore: restore message fires, corrupted file restored, target byte-identical to pre-upgrade; control installer with set -E stripped must NOT roll back (proves errtrace is load-bearing); plus signal/exit-guard controls |
CI ci.yml#upgrade-guard; verify-release upgrade-guard stage |
QC-10 Upgrade/install safety | preserve | Proves the rollback trap actually fires; the -E-stripped control keeps Part A honest. |
|
| Durable-snapshot gate (#791 PR2) | scripts/test-upgrade-durable-snapshot.sh |
shell | Pre-update snapshot taken before any mutation (0700/0600 perms, secret never logged, retention-pruned); post-sync verify net restores operator files a manifest bug lets the sync touch; CWE-59 symlink-leaf guard proven with a portable cp shim in both polarities (write-through-link must not happen); v1→v2 migration semantics (intended bin/ removal not healed) |
CI ci.yml#upgrade-guard; verify-release upgrade-guard stage |
QC-10 Upgrade/install safety | preserve | Covers tampering and leak vectors the manifest guard cannot see; the shim rationale (busybox vs GNU cp) is documented in-file. | |
| Install migration matrix (v2→v3) | scripts/test-install-migration.sh |
shell | Fixture matrix running the real installer with MOSAIC_SYNC_ONLY=1: fresh install seeds + stamps version 3; legacy user-edited AGENTS overwritten with .pre-constitution.bak preserved (and idempotent); tuned STANDARDS overwritten; operator files (SOUL, credentials) preserved. Mirrors the TS suite packages/mosaic/src/config/file-adapter.test.ts — both installers must behave identically |
CI ci.yml#upgrade-guard; verify-release upgrade-guard stage |
QC-10 Upgrade/install safety | preserve | Pins the shell/TS installer parity contract; removal would orphan that parity requirement. | |
| Enforcement verification probe (bash) | scripts/verify.sh |
shell | Attempts real commits in the target repo: planted type error must produce a commit blocked with error; planted any must trip no-explicit-any; planted lint error must trip prettier; gitleaks binary must exist (3a) and detect a planted AWS key via gitleaks git --pre-commit --staged --redact (3b). Verdicts are output-grep matches on hook stderr |
local via installed mosaic-quality-verify on scaffolded target projects; not run in this repo's CI |
QC-20 Downstream enforcement verification | strengthen (review) | Mechanism is genuinely behavioral (stronger than file presence) but verdict logic is grep-on-output and it is unwired here; absorb as the evaluator's enforcement-probe check (the RI-N4 evaluator invokes it or reimplements it) before retiring the shell path. | |
| Enforcement verification probe (PowerShell) | scripts/verify.ps1 |
shell | Windows port of verify.sh: same planted-commit tests with $output -match matching; no gitleaks self-test parity beyond the same checks |
local (Windows operator); no Windows CI runner exists |
QC-20 Downstream enforcement verification | strengthen (review) | A hand-maintained twin of verify.sh with no CI coverage — exactly the drift shape the single evaluator removes; retire after the TS evaluator owns the probe. |
|
| Quality template installer (bash) | scripts/install.sh |
shell | Copies template files (.husky/pre-commit incl. mandatory gitleaks, .lintstagedrc.js, .eslintrc.js, tsconfig.json, .woodpecker.yml, .gitleaks.toml) into a target project; warns (does not verify) about package.json snippet merge; no post-condition check |
local / via mosaic-quality-apply |
QC-21 Downstream rails scaffolding | strengthen (review) | Duplicates the TS quality-rails init scaffolder for a different template set; converging on one scaffolder (with post-scaffold verification) is prerequisite to retiring this path. |
|
| Quality template installer (PowerShell) | scripts/install.ps1 |
shell | Windows twin of the template copy above | local (Windows operator) |
QC-21 Downstream rails scaffolding | strengthen (review) | Same twin-drift risk as verify.ps1; no runner exercises it. |
|
mosaic-quality-verify adapter |
framework/tools/_scripts/mosaic-quality-verify |
shell | Thin adapter: validates target dir exists, asserts verify.sh present+executable, cd target, exec it. No verdict logic of its own |
local (installed framework bin) |
QC-20 Downstream enforcement verification | preserve | Already the thin-adapter shape RI-N4 prescribes for shell surfaces. | |
mosaic-quality-apply adapter |
framework/tools/_scripts/mosaic-quality-apply |
shell | Thin adapter: arg validation then exec of quality install.sh --template … --target … |
local (installed framework bin) |
QC-21 Downstream rails scaffolding | preserve | Thin adapter, no separate verdict; disposition follows its target script's convergence. | |
| Roster schema regression | scripts/test-roster-schema.py |
shell | jsonschema Draft202012Validator over fleet/roster.schema.json with valid/invalid connector-kind fixtures (tmux/discord/matrix conditional fields) |
unwired — not on S1 or S2, not signed-excluded; also outside the enumeration guard's *.sh population, so the guard cannot see it |
QC-5 Test suite execution | strengthen (review) | A real regression suite that currently runs nowhere; wire it into a CI surface or sign an exclusion — leaving it invisible re-arms the exact gap #1017 closed. | |
| Framework shell chain (S1) | packages/mosaic/package.json test:framework-shell |
shell | 47-command && chain: enumeration guard + needles, 14 lease-broker/mutator-gate python unitests, check-runtime-launches.py, and ~30 framework-tool shell suites (git wrappers, wake, woodpecker, tmux, glpi, orchestrator, _scripts). Quality-relevant members rowed separately below |
turbo test → CI ci.yml#test; verify-release test stage |
QC-5 Test suite execution | preserve | The chain is the execution surface the enumeration guard audits; known residuals: a failing link stops later suites (measured in #1270 — suites after position 44 had not run), and the guard proves naming, not reachability. |
1.3 Framework runtime hooks and their harnesses (agent-host enforcement)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
| QA edit hook seam | framework/tools/qa/qa-hook-stdin.sh (+ qa-hook-handler.sh) |
shell | PostToolUse stdin hook: extracts edited file from the tool JSON (jq or grep fallback), skips non-JS/TS, then the deps-preflight gate — exits 1 with the legible sentinel deps not installed — run pnpm install when node_modules/.bin is missing/empty (the #856 false-red class); the downstream handler only files QA remediation report templates (no verification logic) |
agent-runtime (framework runtime/claude/settings.json PostToolUse); never CI |
QC-16 Agent-runtime edit-time checks | strengthen (review) | The sentinel gate is real enforcement; the handler's report-filing adds no verdict and its name promises more than the code does — evaluator absorption should keep the sentinel, drop the report theater. |
| Typecheck-on-edit hook | framework/tools/qa/typecheck-hook.sh |
shell | PostToolUse: for edited .ts/.tsx, finds nearest tsconfig.json and runs tsc --noEmit, surfacing errors nonzero to the agent immediately |
agent-runtime (framework runtime/claude/settings.json PostToolUse) |
QC-16 Agent-runtime edit-time checks | strengthen (review) | Edit-time duplicate of QC-2 with independent invocation logic; keep behavior, converge invocation through the evaluator adapter. |
| Deps-preflight harness | framework/tools/qa/test-deps-preflight.sh |
shell | Five assertions against the seam incl. a documented RED control (raw not found), sentinel behavior for missing and empty .bin, and no-false-positive once populated |
test:framework-shell → CI ci.yml#test |
QC-16 Agent-runtime edit-time checks | preserve | Guard-of-the-check with a red control; keeps the sentinel from regressing. |
| Prompt-helper RCE regression | framework/tools/_scripts/test-mosaic-init-rce.sh |
shell | Sources the prompt helpers and proves a literal $(touch /tmp/pwned) answer round-trips verbatim and never executes (no /tmp/pwned created) |
test:framework-shell → CI ci.yml#test |
QC-5 Test suite execution | preserve | Cheap, load-bearing security regression on the installer's input path. |
| Install-ordering harness (#869 C2) | framework/tools/_scripts/test-install-ordering-guard.sh |
shell | Drives mosaic-link-runtime-assets with a fake mosaic on PATH: probe ok ⇒ settings copied + exit 0; probe fail ⇒ exit 1 with degraded outcome but all other runtime files still copied; --allow-inactive-enforcement forwarded; no-mosaic-on-PATH ⇒ python3 fallback strips enforcement hooks and exits 1; fallback + flag ⇒ wires as-is, exit 0 |
test:framework-shell → CI ci.yml#test |
QC-17 Lease-enforcement wiring safety | preserve | Exercises the shell wiring seam independently of the TS guard's own spec suite (complementary coverage, by design). |
| Fleet-transport harness (#1240) | framework/tools/_scripts/test-fleet-transport-check.sh |
shell | Extracts the shipped check_fleet_transport/fleet_declared_transport functions from the shipped scripts (fails loud if extraction yields nothing) and drives both implementations (mosaic-doctor + tools/install.sh) from one case table |
test:framework-shell → CI ci.yml#test |
QC-18 Operator-host drift audit | preserve | The anti-drift harness for the one rule shipped twice; extraction-from-source keeps it from testing a stale copy. |
| Terminal-green contract (RM-61/#1000) | framework/tools/woodpecker/test-terminal-green-contract.sh + verify-terminal-green.py |
shell | Red-first fixtures: pipeline JSON variants (service failure, step failure, cancelled, etc.) must produce the correct terminal-green verdict; controls must pass | test:framework-shell → CI ci.yml#test |
QC-5 Test suite execution | preserve | Keeps the CI-wait wrapper's green-detection honest; a false green here would poison every merge gate that trusts pr-ci-wait.sh. |
| Lease-gate launch invariant | framework/tools/lease-broker/check-runtime-launches.py |
shell | Scans production roots (packages/, apps/, plugins/, tools/) across sh/py/ts/yaml suffixes for Claude/Pi process launches outside the lease gate; allowlist-based; fails CI on violation |
test:framework-shell → CI ci.yml#test |
QC-15 Lease-gate architecture invariant | preserve | The only architectural "no ungated launches" rail; grep+allowlist is the right cost/benefit for this invariant. |
1.4 TypeScript quality logic (@mosaicstack/quality-rails + mosaic CLI)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
quality-rails check |
packages/quality-rails/src/cli.ts (mosaic quality-rails check --project) |
ts | Expected-file presence only: loops expectedFilesForKind (node: .eslintrc, biome.json, .githooks/pre-commit, PR-CHECKLIST.md; python: pyproject.toml+hooks+checklist; rust: rustfmt.toml+…) and exits 1 listing missing paths. Does not execute any linter, formatter, hook, or scanner |
local (operator CLI); no CI wiring in this repo |
QC-19 Downstream rails presence check | strengthen | This is the RI-N4 evaluator seed. Today presence ≠ parity (explicitly called out by RI-N4): it must grow typed verdicts (passed/failed/blocked/error/not-applicable), check versioning/subject/reason, digested definitions, and absorb the effective shell probes (QC-20 first). |
quality-rails doctor |
packages/quality-rails/src/cli.ts |
ts | Same presence data as check, printed with ok/missing lines; cannot fail (no nonzero exit on missing files) |
local (operator CLI) |
QC-19 Downstream rails presence check | strengthen | A doctor that cannot fail is advisory; fold into check (or return typed states) when the evaluator lands. |
quality-rails init |
packages/quality-rails/src/cli.ts + scaffolder.ts/templates.ts |
ts | Scaffolds rails files per detected kind/profile (linters/formatters lists are advisory strings; hooks flag always true); writes files, prints follow-ups — no post-condition verification | local (operator CLI) |
QC-21 Downstream rails scaffolding | strengthen (review) | Second scaffolding path alongside quality install.sh (§1.2); converge on one with post-scaffold verification before retiring either. |
| Lease activation probe (#869 C1, hidden) | packages/mosaic/src/commands/lease-activation-probe.ts |
ts | Real capability probe, not file presence: resolves the installed mosaic CLI and requires it to advertise the exact {name, version} activation contract; all deps injectable; registered as hidden CLI command and consumed by C2/C5 |
local (hidden CLI + consumed by C2/C5); spec-tested via lease-activation-probe.spec.ts in turbo test |
QC-17 Lease-enforcement wiring safety | preserve | The versioned-contract probe is precisely the fail-closed capability check RI-N2 generalizes; already typed and injectable. |
| Install-ordering guard (#869 C2, hidden) | packages/mosaic/src/commands/install-ordering-guard.ts |
ts | Decides whether enforcement hook entries are written into the ~/.claude/settings.json the framework reseed ships: not activatable ⇒ strip hooks + nonzero loud outcome (default); explicit per-invocation --allow-inactive-enforcement opt-out wires-with-warning. Never touches the runtime gate's own fail-closed behavior |
installer (framework reseed via mosaic-link-runtime-assets); spec + shell harness coverage in turbo test |
QC-17 Lease-enforcement wiring safety | preserve | Correct default-deny with an explicit, non-env opt-out; test-locked from both the TS and shell sides. |
| Lease doctor check (#869 C5) | packages/mosaic/src/commands/lease-doctor-check.ts |
ts | Combines hook-wiring detection in ~/.claude/settings.json with C1 activatable and C3 broker-supervisor health: wired ∧ (¬activatable ∨ ¬healthy) ⇒ loud [ERROR] that forces mosaic doctor exit 1 regardless of the bash audit's own exit |
local (inside mosaic doctor); spec coverage in turbo test |
QC-17 Lease-enforcement wiring safety | preserve | Closes the "bricked host looks green" hole; cannot be masked by the bash script — that composition is the point. |
mosaic doctor (framework drift audit) |
packages/mosaic/src/commands/launch.ts (doctor) + framework/tools/_scripts/mosaic-doctor |
shell+ts | Bash audit of the installed framework home: ~40 expected files/dirs present; runtime files are copies (not symlinks) matching source (cmp) or composed runtime-contract markers; hard-gates block present in AGENTS.md; sequential-thinking MCP configured; fleet transport binary present per roster (warn); legacy symlink trees gone; skills synced — warn-based, exit 1 only with --fail-on-warn, plus C5's forced error |
local (operator audit) |
QC-18 Operator-host drift audit | preserve | Host-state audit CI cannot see (user files by design, DESIGN §7); advisory exit is the documented contract — do not silently change it. |
mosaic gateway doctor |
packages/mosaic/src/commands/gateway-doctor.ts |
ts | Probes per-service health (PostgreSQL, Valkey, pgvector) via @mosaicstack/storage, reports tier and JSON; exit 1 only when at least one required service fails (yellow stays 0) |
local (operator) |
QC-18 Operator-host drift audit | preserve | Service health with correct red/yellow exit semantics; JSON mode exists for scripting. |
mosaic gateway verify |
packages/mosaic/src/commands/gateway/verify.ts |
ts | Post-install liveness: daemon meta via HTTP with retries, admin token on file, bootstrap endpoint reachable; aggregated pass/fail | local; consumed by tools/e2e-install-test.sh |
QC-18 Operator-host drift audit | preserve | The first-run proof the installer E2E relies on; retry-aware so startup races don't false-red. |
mosaic fleet doctor |
packages/mosaic/src/commands/fleet-reconciler-command.ts |
ts | Classifies local roster-owned drift (no mutation) from the parsed v2 roster | local (operator) |
QC-18 Operator-host drift audit | preserve | Dry-run classification is the correct non-mutating audit shape. |
1.5 Git hooks (developer machine)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
| Pre-commit staged hygiene | .husky/pre-commit → npx lint-staged (.lintstagedrc) |
shell | On staged files only: prettier --write + eslint --fix for ts/tsx/js/jsx; prettier --write for json/md/yaml/yml. Mutating (fixes and re-stages); commit blocks only if a fixer itself fails |
pre-commit (every local commit; hooks activated by install-hooks.mjs via core.hooksPath .husky/_) |
QC-13 Staged-change hygiene | preserve | Correct scoped fast gate; note it auto-fixes rather than rejects (deliberate). Gap: no secret scan here — see §3. |
| Pre-push gate | .husky/pre-push |
shell | pnpm preflight && pnpm typecheck && pnpm lint && pnpm format:check (no test run — documented in AGENTS.md) |
pre-push |
QC-14 Pre-push gate | preserve | Composes QC-1..4 exactly as specified in AGENTS.md; tests intentionally left to CI. |
| Hook installer | scripts/install-hooks.mjs (pnpm prepare) |
ts | Stages husky hooks into a scratch repo first, asserts husky produced its h shim, quarantines incomplete previous sets, verifies idempotence via full directory snapshot comparison, then sets core.hooksPath; skips cleanly with HUSKY=0 or no git |
installer (runs on pnpm install) |
QC-13 Staged-change hygiene | preserve | Self-verifying wiring for the hook gates — a corrupted half-install cannot silently disable them. |
1.6 CI pipeline steps (.woodpecker/)
Step-to-probe mapping for container steps: ci.yml#sanitization = QC-7+QC-8+QC-9 (rows §1.2, plus apk add bash env prep); ci.yml#upgrade-guard = QC-10 (rows §1.2, plus apk add rsync); ci.yml#typecheck/#lint/#format/#test = QC-2/3/4/5 (rows §1.1). Rows below are mechanisms that exist only in a pipeline.
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale | ||
|---|---|---|---|---|---|---|---|---|---|
| Frozen install | ci.yml#install |
pipeline-step | pnpm install --frozen-lockfile --prefer-offline against the baked ci-base store — lockfile supply integrity; a drifted lockfile fails the build before any gate runs |
CI ci.yml#install |
QC-1 Checkout integrity | preserve | Lockfile-pinned dep resolution is the supply-chain floor under every later gate. | ||
| Test-step readiness prelude | ci.yml#test prologue |
pipeline-step | Installs pinned @earendil-works/[email protected] (Invariant R suite requires the real binary) + openssl; waits up to 60×1s on pg_isready for the ci-postgres service and fails fast if it never comes up; runs db:migrate before tests |
CI ci.yml#test |
QC-5 Test suite execution | preserve | Fail-fast environment preconditions — a missing service produces a legible failure, not a wall of red tests. | ||
| Publish verify step (pending RI-1-001) | publish.yml#verify (branch feat/ri-050-publish-gate @ 46784c8d, not yet on next) |
pipeline-step | (a) Commit identity: fails closed if CI_COMMIT_SHA empty, git rev-parse HEAD empty, or the two differ; (b) runs the canonical pnpm verify:release. Every publish effect depends on this step; it carries no path filter |
publish.yml#verify |
QC-11 Terminal release verification | preserve | The RI-N1 exact-commit binding; until it merges, publish steps on next depend on build only (see §3 gap 1). |
||
| Publish error classification | publish.yml#publish-npm |
pipeline-step | Publishes @mosaicstack/* (minus web) and classifies outcome: success, or the only tolerated failure = already-published (EPUBLISHCONFLICT / "cannot publish over" / "previously published"); explicit fatal on npm E404/E401/ENEEDAUTH/ECONNREFUSED/ETIMEDOUT/ENOTFOUND and on any unrecognized failure (replacing the old ` |
echo` that hid a registry 404) | publish.yml#publish-npm (main/tags, path-filtered on packages/**) |
QC-12 Publish-effect integrity | preserve | Converts silent publish fall-on-floor into loud failure; allowlist-of-one error tolerance is the right shape. | |
| Next-lane publish assertions | publish.yml#publish-next-npm |
pipeline-step | Guards: branch must be next, CI_PIPELINE_NUMBER required; registry dist-tags JSON must be usable; walks all manifests, strictly parses stable semver, rewrites X.Y.(Z+1)-next.<N>; publishes with --tag next (never latest); post-publish asserts npm view @mosaicstack/mosaic@next resolves to the exact expected version |
publish.yml#publish-next-npm (push/manual on next) |
QC-12 Publish-effect integrity | preserve | Durable prerelease lane with end-to-end resolution proof — the published artifact is verified, not assumed. | ||
| Image destination policy | publish.yml#build-gateway / #build-appservice / #build-web |
pipeline-step | Kaniko builds with destination policy: next ⇒ sha-tag only (fatal if a tag event sneaks in); main ⇒ sha + latest; tag events ⇒ sha + <tag>; anything else fatal. Path filters only skip effects, never the verify step |
publish.yml#build-* |
QC-12 Publish-effect integrity | preserve | Fail-closed tagging matrix; the exclude-list default-safe design keeps stale images impossible. |
Adjacent pipeline surface (not a probe): .woodpecker/ci-image.yml rebuilds the ci-base image on pnpm-lock.yaml/Dockerfile.ci change with an immutable lock-<hash> tag; pipelines consume :latest. Recorded for completeness — no code-quality property is checked.
1.7 Root installer tooling (tools/)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
| Next-lane installer test | tools/install-next-lane.test.sh (pnpm test:installer) |
shell | Drives tools/install.sh --next with faked node/npm binaries (no network): Node 20 must be rejected; installs must pin exact versions (mutable @next forbidden); fast path must not unexpectedly fall back to source; gateway-install failure takes the documented fallback |
turbo-external tail of pnpm test → CI ci.yml#test |
QC-5 Test suite execution | preserve | Hermetic (shimmed) regression net for the installer lane; runs as part of the standard test command. |
| Clean-container install E2E | tools/e2e-install-test.sh |
shell | Full first-run flow in a node:22-alpine container: install.sh --yes → mosaic wizard (non-interactive) → mosaic gateway install → mosaic gateway verify exit check (with EXPECTED-SKIP if the installed CLI predates gateway verify); skips gracefully without Docker |
local (manual; requires Docker); not wired in CI |
QC-5 Test suite execution | strengthen (review) | The only end-to-end proof of the install→verify path; currently operator-initiated only — wire into a periodic/manual CI lane or sign its exclusion explicitly. |
| Host installer advisories | tools/install.sh (--check; check_fleet_transport) |
shell | --check = version comparison only, no install; check_fleet_transport warns (non-blocking, by design — tmux is the fleet's dependency, not mosaic's) when the roster-declared transport binary is absent, naming exactly what it blocks; PATH-persistence warnings |
installer (operator-run) |
QC-18 Operator-host drift audit | preserve | Advisory-by-design warnings; the parallel doctor check is drift-tested by §1.3's harness. |
1.8 Pending workstream additions (branch feat/ri-050-publish-gate @ 46784c8d)
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|---|---|---|---|---|---|---|---|
| Canonical terminal verification | scripts/verify-release.mjs (pnpm verify:release) |
ts | One command replaying the full mandatory set as stages — sanitization, upgrade-guard, typecheck (incl. preflight), lint, format, test, build — mirroring ci.yml step-for-step; fail-fast on first failing command; requires bash+rsync on PATH; --stage <name> for wiring smoke-tests only |
publish.yml#verify (pending); local (pnpm verify:release) |
QC-11 Terminal release verification | preserve | The RI-N1 canonical command — CI and publication share one semantic checklist by construction. |
| Verify-parity contract test | scripts/verify-release.test.mjs |
ts | Parses the real ci.yml/publish.yml: stage table must match ci.yml step-for-step; every publish-effect step (name publish* or image-pushing) must transitively depend on verify; commit-identity assertion must be present; verify must carry no path filter |
test:checkout → CI ci.yml#test (once merged) |
QC-11 Terminal release verification | preserve | Guard-of-the-guard at checkout time — the two surfaces cannot drift apart silently. |
2. Canonical check set
The deduplicated checks every row above maps onto. IDs are stable for RI-3-002 to consume.
- QC-1 Checkout integrity. Owns: the checkout can run its gates — frozen-lockfile dependency resolution, required gate binaries present, no stale build lock, and the
apps/web/.nextgenerated-state trust chain (real directory, uid ownership, certified source fingerprint, certified symlink manifest). Implemented byscripts/preflight.mjs+ frozen install steps. - QC-2 Workspace typecheck. Owns workspace-wide TypeScript soundness: per-package
tsc --noEmitover built dependencies (turbo typecheck). The single definition invoked by CI, pre-push, and terminal verification. - QC-3 Workspace lint. Owns static-analysis policy: per-package ESLint under the root config. One config, one task, every surface.
- QC-4 Format check. Owns formatting uniformity: Prettier check with the repo ignore list. (The pre-commit variant additionally fixes; the verdict form is this check.)
- QC-5 Test suite execution. Owns execution of all test surfaces: checkout script units (
node --test), per-package Vitest suites (including the framework shell chain and its python unitests), the installer-lane shim test, and — once wired —test-roster-schema.pyand container E2E. Also owns guards-of-the-gate that live inside the chain (terminal-green contract, RCE regression). - QC-6 Workspace build. Owns artifact buildability:
turbo buildproducing the artifacts publication consumes. - QC-7 Framework sanitization. Owns the open-source guarantee for the shipped framework package: no operator-identity tokens anywhere (examples included), no private
$HOMEdefaults in shipped scripts, with a self-test that keeps the regexes honest. - QC-8 Resident-context budget. Owns the line-count ceilings on framework files injected into every agent's context (Constitution, dispatcher, RUNTIME.md slices) — the CI-enforceable half of the resident-prompt budget.
- QC-9 Test-membership enumeration. Owns the property that no test suite can silently fall out of CI: disk population vs parsed enumeration surfaces, both-directions staleness, and signed exclusions with reasons. Includes its needle/control harness.
- QC-10 Upgrade/install safety. Owns the #791 family: operator-path byte-identity across keep-mode upgrades (manifest guard), mid-failure rollback (errtrace-proven), durable pre-update snapshot + verify net + CWE-59 leaf guard, and the v2→v3 migration matrix with shell/TS parity.
- QC-11 Terminal release verification. Owns the RI-N1 exact-commit binding: commit-identity assertion plus one canonical command (
pnpm verify:release) replaying the complete mandatory set, with every publish effect depending on it; plus the checkout-time parity/DAG contract test that keeps pipeline and command in sync. - QC-12 Publish-effect integrity. Owns publication correctness: npm publish error classification (only already-published tolerated), next-lane versioning and post-publish resolution proof, and image destination/tag policy.
- QC-13 Staged-change hygiene. Owns commit-time hygiene on staged files (prettier/eslint fix-and-restage) and the self-verifying hook wiring that guarantees the gates are actually installed.
- QC-14 Pre-push gate. Owns the local push composition: preflight + typecheck + lint + format:check (tests deliberately deferred to CI).
- QC-15 Lease-gate architecture invariant. Owns "no ungated runtime launches in production code": the scan + allowlist over
packages/,apps/,plugins/,tools/. - QC-16 Agent-runtime edit-time checks. Owns edit-time feedback on agent hosts: the deps-preflight legibility sentinel and typecheck-on-edit, plus their regression harnesses.
- QC-17 Lease-enforcement wiring safety. Owns the #869 C1/C2/C5 trio: activation capability probe (versioned contract), enforcement-hook wiring gate (default-deny with explicit opt-out), and the doctor check that surfaces a bricked host — with their shell/TS harnesses.
- QC-18 Operator-host drift audit. Owns host-state health CI cannot see:
mosaic doctordrift audit (+ fleet transport, both implementations),fleet doctorroster classification,gateway doctor/gateway verifyservice health, and installer advisories. Advisory exits are part of the contract. - QC-19 Downstream rails presence check. Owns "does a scaffolded project still carry its rails files" — today the TS
quality-rails check/doctorpresence loop; per RI-N4 this is the seed that must become the typed evaluator (presence alone is explicitly not parity). - QC-20 Downstream enforcement verification. Owns "do the rails actually block" on scaffolded projects: the behavioral planted-commit probe (type error,
any, lint, gitleaks secret) currently inverify.sh/verify.ps1behind themosaic-quality-verifyadapter. - QC-21 Downstream rails scaffolding. Owns putting rails files into a target project: the shell template installer (+ PowerShell twin) and the TS
quality-rails initscaffolder — currently two paths that must converge.
3. Coverage gaps
Enforced nowhere but implied, or named in docs/tooling but not wired:
- Publication not yet bound to verification on
next. At this base (8199261c),publish.ymlpublish steps depend onbuildonly; theverifystep andscripts/verify-release.mjsexist onfeat/ri-050-publish-gate(46784c8d) but are not merged. Until RI-1-001 lands, AC-RI-1's negative control cannot hold on the real pipeline. - Playwright E2E unwired.
apps/webshipstest:e2e(playwright test) with real suites (admin/auth/chat/navigation.spec.ts); neitherpnpm testnor any CI step invokes it. The web UI's user flows are verified only when an operator runs them manually. - No secret scanning on this repo. The framework's own template pre-commit makes gitleaks required, and
verify.shproves detection with a planted key — but this repository's.husky/pre-commit(lint-staged only) and CI run no secret scan. The repo ships the control it does not use. - No dependency audit. The quality
.woodpecker.ymltemplates anddocs/CI-SETUP.mdspecifynpm audit --audit-level=highas a pipeline stage; nothing equivalent runs for this repo. - No coverage thresholds. Templates enforce 80% Jest coverage thresholds; this repo's Vitest configs collect coverage with no thresholds — coverage is measured nowhere and enforced nowhere.
test-roster-schema.pyinvisible. A real jsonschema regression suite wired to no surface and invisible to the enumeration guard (its population is*.sh; the suite is.py). Either enumerate it or sign an exclusion — silence here is the #1017 defect shape.- Presence-checker expectations ≠ this repo.
quality-rails checkexpects.eslintrc,biome.json,.githooks/pre-commit,PR-CHECKLIST.mdfor node projects — none describe this monorepo (husky, flat eslint config, no biome, no PR-CHECKLIST.md). The evaluator's check set must be per-subject (versioned, digested), not one global file list. - Chain-ordering residual (documented).
test:framework-shellis one&&chain: a failing link skips every later suite while the step still fails (measured in #1270 — four suites after position 44 had not run since a prior merge). The enumeration guard proves naming, not reachability; both residuals are in-file documented but structurally unfixed. - Signed-exclusion burndown open. 16 signed exclusions remain in
test-enumeration-exclusions.txt; several are "unmeasured in CI image" or blocked on missing CI tooling (tmux, setsid) — tracked under #1017/#1271. Each is an enforcement promise deferred, not delivered. - Windows twins unexercised.
verify.ps1,install.ps1,mosaic-doctor.ps1have no runner anywhere (no Windows CI); behavioral drift from their bash twins is undetectable by construction. - QA hook name vs behavior.
qa-hook-handler.shfiles remediation report templates but performs no verification; the seam's actual gate value is only the deps-preflight sentinel. Anything relying on "QA automation hook" as a check is relying on report-filing. - Two test paths, one gated. CI runs tests against ci-postgres (
DATABASE_URLset); the local PGlite path is the documented default (AGENTS.md) until KBN-101-02/101-05. Only the CI path is enforced by pipeline.
4. Disposition summary
| disposition | rows | checks |
|---|---|---|
| preserve | 43 | Every canonical owner (QC-1..QC-18) plus correct guards-of-the-guard and thin adapters: all of §1.1, the CI-invoked framework probes and adapters in §1.2, all of §1.3, the C1/C2/C5 trio and doctors in §1.4, all of §1.5, all pipeline-only steps in §1.6, §1.7 rows 1 and 3, and §1.8. |
| strengthen | 2 | quality-rails check and quality-rails doctor (QC-19) — the RI-N4 evaluator seed: typed verdicts, versioned/digested check definitions, per-subject check sets. |
| strengthen (review) | 9 | verify.sh + verify.ps1 (QC-20), quality install.sh/install.ps1 + quality-rails init (QC-21 — scaffold-path convergence), test-roster-schema.py (QC-5 — wire or sign), qa-hook-stdin.sh seam + typecheck-hook.sh (QC-16), tools/e2e-install-test.sh (QC-5 — CI lane). |
| retire | 0 | None meet the bar: RI-N4 requires effective shell probes be absorbed before their paths retire, and no absorption exists yet. The strengthen (review) rows are the retirement candidates for RI-3-002 once the evaluator owns their behavior. |
Row total: 54. Canonical checks: 21 (QC-1..QC-21).