diff --git a/docs/ADMIN-GUIDE/quality-gate-registry.md b/docs/ADMIN-GUIDE/quality-gate-registry.md index 312bb351..19c9cd4d 100644 --- a/docs/ADMIN-GUIDE/quality-gate-registry.md +++ b/docs/ADMIN-GUIDE/quality-gate-registry.md @@ -6,7 +6,7 @@ Run `pnpm gate:verify` from a dependency-installed checkout. Exit zero means reg Investigate any of these immediately: -- `GATE VERIFY FAILED` — registry structure, observed behavior, provenance, claim binding, source/deployment identity, or negative-control detection changed. +- `GATE VERIFY FAILED` — registry structure, observed behavior, provenance, claim binding, source/deployment identity, or negative-control detection changed. The verifier aggregates independent phase failures, so repair the responsible stable-ID diagnostics as well as any accompanying stale-fixture error; do not treat the generic error as a substitute. - `unregistered gate` — an executable appeared under a declared gate root without a registry entry. - `no negative control` — a gate has no must-fail case. - `DEPLOYED IDENTITY UNAVAILABLE` — the runner cannot reach the installed enforcing copy. The pinned observation is checked, but live equality is not asserted. diff --git a/docs/DEVELOPER-GUIDE/quality-gate-registry.md b/docs/DEVELOPER-GUIDE/quality-gate-registry.md index 526bf481..dce1ded8 100644 --- a/docs/DEVELOPER-GUIDE/quality-gate-registry.md +++ b/docs/DEVELOPER-GUIDE/quality-gate-registry.md @@ -10,7 +10,7 @@ pnpm gate:verify The registry covers root typecheck, lint, and format checks; RM-01 checkout preflight; the Mosaic CI queue guard; and root Husky pre-commit/pre-push hooks. It does not imply repository-wide coverage. Framework scripts, package-local build/test scripts, templates, and deployment/release scripts remain assigned to RM-54. -Every gate declares exact invocations, observed and required outcomes, criterion bindings, and a single exact inerting mutation. Every must-fail case requires a non-empty reason diagnostic. The verifier rejects a stale, ambiguous, crashing, or ineffective mutation. Fixture and mutation writes reject path traversal and final-component symlinks. This proves detection of the **declared** inerting mutation, not every possible semantic weakening. +Every gate declares exact invocations, observed and required outcomes, criterion bindings, and a single exact inerting mutation. Every must-fail case requires a non-empty reason diagnostic. The verifier rejects a stale, ambiguous, crashing, or ineffective mutation. Fixture and mutation writes reject path traversal and final-component symlinks. Independent validation phases collect labeled failures instead of letting one thrown fixture, claim, discovery, deployment, mutation, or compatibility error mask already-known stable-ID diagnostics. This proves detection of the **declared** inerting mutation, not every possible semantic weakening. ## Required versus actual diff --git a/docs/PRD.md b/docs/PRD.md index 840e3f1a..7ef0c363 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -51,7 +51,7 @@ Existing deterministic gates can return success without enforcing their stated p 2. `RM02-AC-02`: Externally mutate any registered gate at its declared inerting point so its failure path succeeds; verification returns nonzero and names that gate. The verifier's internal meta-control is observed red before its healthy result is trusted. 3. `RM02-AC-03`: An executable added under a declared gate root without an entry returns nonzero and includes `unregistered gate`. 4. `RM02-AC-04`: A gate with zero must-fail cases returns nonzero and includes `no negative control`. -5. `RM02-AC-05`: Unbound or semantically misbound criteria, prose claims bound to unrelated cases, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID. Registered meta-negative controls move a criterion binding, remove meaning provenance, and redirect a prose claim to an unrelated case; each is observed red for its stated reason. +5. `RM02-AC-05`: Unbound or semantically misbound criteria, prose claims bound to unrelated cases, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID. A simultaneous stale fixture or independent phase error SHALL NOT mask responsible stable-ID diagnostics. Registered meta-negative controls move a criterion binding, remove meaning provenance, and redirect a prose claim to an unrelated case; each is observed red for its stated reason. 6. `RM02-AC-06`: CI configuration invokes the verifier unconditionally on every pull request. 7. `RM02-AC-07`: PR output states adjacent `DOES`/`DOES NOT` boundaries: current-tree gates and inerting mutations execute unprivileged and fail-closed; isolated own-tree replay does not execute in repository-controlled PR CI. RM-60/RM-59 are named, retained provider evidence is never inferred, and future protected post-merge detection specifies quarantine/revert rather than claiming pre-merge prevention. diff --git a/docs/scratchpads/1029-rm-02-gate-registry.md b/docs/scratchpads/1029-rm-02-gate-registry.md index 3e6d0b85..631ae905 100644 --- a/docs/scratchpads/1029-rm-02-gate-registry.md +++ b/docs/scratchpads/1029-rm-02-gate-registry.md @@ -39,7 +39,7 @@ Deliver the seven-gate registry and RED-first anti-inert verifier on `feat/rm-02 - CI wiring RED: package script and unconditional Woodpecker step tests both failed before wiring. - History RED: history test failed with missing module before own-tree manifest selection/provider classification was implemented. - `pnpm gate:verify`: exit 0; seven gates each reported `META-NEGATIVE-CONTROL ... observed red`; queue source/deployed drift control observed red; six queue behavior deltas printed as `DEFECT (owner: RM-03)`. -- Focused Node tests: 37/37 pass after review hardening (26 verifier/wiring plus 11 history/provider tests). +- Focused Node tests: 38/38 pass after review hardening (27 verifier/wiring plus 11 history/provider tests). - `pnpm typecheck`: pass (45/45 Turbo tasks). - `pnpm lint`: pass (25/25 Turbo tasks). - `pnpm format:check`: pass. @@ -62,6 +62,7 @@ The queue guard's `get_state_from_status_json` runs `python3 - <<'PY'` while pro - Pipeline #2178 then proved the unprivileged Docker runner cannot establish Bubblewrap namespaces. A privileged experiment remained uncommitted and was rejected after Codex correctly rated it CRITICAL: PR-controlled code executes before an in-repository sandbox and could directly use the granted capability. - `mos-remediation` and `rev-974` independently ruled Option C. RM02-REQ-10 now retains its original text, restatement, and reason: PR CI verifies only the current tree, unprivileged and fail-closed; isolated own-tree replay is deferred to RM-60/#1031's external pre-execution authority, cross-referenced with RM-59. Future protected post-merge replay is detection with quarantine/revert, never pre-merge prevention. - RED-first boundary test proved the old path executed an inert intermediate verifier. The revised path states adjacent `DOES`/`DOES NOT` claims, validates historical manifest provenance without executing it, and infers no replay success. Direct sandbox tests remain hard-fail; unprivileged CI asserts terminal refusal instead of treating replay as success. Pipelines #2179/#2180/#2181 exposed two runner refusal forms: namespace denial as `spawnSync bwrap` with `error.code=EPERM`, and a test image without Bubblewrap as `error.code=ENOENT`. The replay diagnostic now preserves spawn errors. Parent-generated launcher/entry metadata distinguishes refusal before sandbox entry from child-controlled output; the detector recognizes exact `spawnSync bwrap` provenance for `EPERM`/`EACCES`/`ENOENT` and known namespace-refusal text only when the entry command provably did not run. Focused negative assertions reject unrelated `spawnSync git EPERM`, verifier output that merely says `bwrap ENOENT`, and exact namespace-denial impersonation without provenance or after sandbox entry. +- Exact-head independent review at `38f1b249` found one valid diagnostic-masking blocker: canonical verification knew four stable-ID rebinding failures but a thrown stale fixture replacement reached the outer catch first and emitted only the generic error. RED-first reproduction confirmed the canonical path omitted both responsible criterion IDs. Verification now collects labeled failures independently across claims, discovery, deployment, case execution/outcome checks, mutation, and compatibility, preserving structural stable-ID failures alongside the stale-fixture signal. The canonical regression test requires both missing-binding IDs and the generic fixture error. - Exact-head independent review at `9b4d4beb` found two valid blockers. RED-first controls reproduced both: denial-looking child stderr was accepted as sandbox unavailability, and moving meaning/prose criterion IDs to an unrelated type-error case left `gate:verify` green. Bubblewrap execution now emits a parent-generated random entry marker and returns parent-owned launcher/entry metadata; unavailability requires Bubblewrap launcher provenance plus proof entry never ran, so exact denial impersonation from plain or entered-child results is rejected. Criterion objects now declare exact `caseRefs`, checked bidirectionally against case-side `criterionIds`; prose claims declare an exact must-fail `caseRef`. Registered must-fail cases move a criterion binding, remove meaning provenance, and redirect a prose claim, each producing its stable reason. The review freeze was deliberately lifted before remediation. - Option C security review reported no findings. Code review rejected an initial unrelated typecheck binding for the new security criterion. It was replaced with a dedicated registered `privileged-pr-gate` case: the fixture injects a privilege key into the gate step, the wiring control rejects it for that exact reason, and `gate:verify` observes the boundary negative control. Follow-up hardening uses a closed exact gate-step construction, rejects privilege across the entire pipeline, rejects non-canonical/merged YAML keys, and pins the unrestricted PR/main trigger block; quoted/escaped/alias/merge/duplicate/filter bypass tests pass. Final Codex code review approved with no findings. diff --git a/scripts/gate-verify.mjs b/scripts/gate-verify.mjs index a570e1db..1dcd3537 100644 --- a/scripts/gate-verify.mjs +++ b/scripts/gate-verify.mjs @@ -326,8 +326,13 @@ function validateClosedSchema(manifest, failures) { ); if (!Array.isArray(criterion.caseRefs) || criterion.caseRefs.length === 0) { failures.push(`${criterion.id}: no declared exercising cases`); - } else if (new Set(criterion.caseRefs).size !== criterion.caseRefs.length) { - failures.push(`${criterion.id}: duplicate declared exercising case`); + } else { + if (criterion.caseRefs.some((caseRef) => typeof caseRef !== 'string' || !caseRef)) { + failures.push(`${criterion.id}: declared exercising cases must be non-empty strings`); + } + if (new Set(criterion.caseRefs).size !== criterion.caseRefs.length) { + failures.push(`${criterion.id}: duplicate declared exercising case`); + } } } rejectDuplicateIds(manifest.proseClaims, 'prose claim', failures); @@ -683,6 +688,7 @@ async function validateMutation(root, gate, failures, observations) { } function findGateCase(manifest, caseRef) { + if (typeof caseRef !== 'string') return undefined; const separator = caseRef.indexOf('/'); if (separator < 1) return undefined; const gateId = caseRef.slice(0, separator); @@ -772,21 +778,45 @@ export async function verifyRegistry(options) { validateStructure(manifest, failures); if (options.structureOnly) return { failures, manifest, observations }; - await validateClaims(options.root, manifest, failures); - await validateDiscovery(options.root, manifest, failures); + + async function collectPhaseFailure(label, action) { + try { + return await action(); + } catch (error) { + failures.push(`${label}: ${error.message}`); + return undefined; + } + } + + await collectPhaseFailure('governing claim validation failed', () => + validateClaims(options.root, manifest, failures), + ); + await collectPhaseFailure('gate discovery failed', () => + validateDiscovery(options.root, manifest, failures), + ); for (const gate of manifest.gates ?? []) { - await validateDeployment(options.root, gate, failures, observations); + await collectPhaseFailure(`${gate.id}: deployment validation failed`, () => + validateDeployment(options.root, gate, failures, observations), + ); for (const gateCase of gate.cases ?? []) { - const result = await runCase(options.root, gate, gateCase); + const result = await collectPhaseFailure( + `${gate.id}/${gateCase.id}: case execution failed`, + () => runCase(options.root, gate, gateCase), + ); + if (!result) continue; const combined = `${result.stdout ?? ''}\n${result.stderr ?? ''}`; - if (!outcomeMatches(gateCase.actual, result)) { - failures.push( - `${gate.id}/${gateCase.id}: observed exit ${String(result.status)}${result.signal ? ` signal ${result.signal}` : ''}${result.error ? ` error ${result.error.message}` : ''} or output disagrees with registry actual ${JSON.stringify(gateCase.actual)}`, - ); - } - if (gateCase.reasonPattern && !new RegExp(gateCase.reasonPattern, 'm').test(combined)) { - failures.push(`${gate.id}/${gateCase.id}: did not fail for its stated reason`); + try { + if (!outcomeMatches(gateCase.actual, result)) { + failures.push( + `${gate.id}/${gateCase.id}: observed exit ${String(result.status)}${result.signal ? ` signal ${result.signal}` : ''}${result.error ? ` error ${result.error.message}` : ''} or output disagrees with registry actual ${JSON.stringify(gateCase.actual)}`, + ); + } + if (gateCase.reasonPattern && !new RegExp(gateCase.reasonPattern, 'm').test(combined)) { + failures.push(`${gate.id}/${gateCase.id}: did not fail for its stated reason`); + } + } catch (error) { + failures.push(`${gate.id}/${gateCase.id}: outcome validation failed: ${error.message}`); } if (!structuredValuesEqual(gateCase.required, gateCase.actual) && gateCase.defect?.owner) { observations.push( @@ -794,11 +824,15 @@ export async function verifyRegistry(options) { ); } } - await validateMutation(options.root, gate, failures, observations); + await collectPhaseFailure(`${gate.id}: mutation validation failed`, () => + validateMutation(options.root, gate, failures, observations), + ); } for (const scenario of manifest.compatibilityScenarios ?? []) { - await runCompatibilityScenario(options.root, manifest, scenario, failures, observations); + await collectPhaseFailure(`${scenario.id}: compatibility validation failed`, () => + runCompatibilityScenario(options.root, manifest, scenario, failures, observations), + ); } return { failures, manifest, observations }; diff --git a/scripts/gate-verify.test.mjs b/scripts/gate-verify.test.mjs index 1777b151..6e1a1126 100644 --- a/scripts/gate-verify.test.mjs +++ b/scripts/gate-verify.test.mjs @@ -296,6 +296,43 @@ test('moving criterion bindings to unrelated cases is rejected', async () => { assert.match(output(result), /META-CRIT-2.*rejects-second-input.*not bound/i); }); +test('canonical verification preserves binding diagnostics when a case fixture is also stale', async () => { + const root = await fixture('misbinding-plus-stale-fixture'); + await writeGate(root); + const manifest = baseManifest(); + manifest.criteria.push({ + id: 'META-CRIT-2', + originalText: 'The second case rejects its own bad input.', + currentText: 'The second case rejects its own bad input.', + claimType: 'integrity', + source: 'fixture', + meaningChanges: [], + caseRefs: ['meta-fixture/rejects-second-input'], + }); + manifest.gates[0].cases.push({ + ...manifest.gates[0].cases[0], + id: 'rejects-second-input', + criterionIds: ['META-CRIT-1'], + }); + manifest.gates[0].cases[0].criterionIds = ['META-CRIT-2']; + manifest.gates[0].cases[0].fixture = { + replaceFiles: [ + { + path: 'gates/meta-fixture.sh', + find: 'text that is not present', + replace: 'irrelevant', + }, + ], + }; + await writeManifest(root, manifest); + + const result = verify(root); + assert.notEqual(result.status, 0); + assert.match(output(result), /META-CRIT-1.*rejects-bad-input.*not bound/i); + assert.match(output(result), /META-CRIT-2.*rejects-second-input.*not bound/i); + assert.match(output(result), /fixture replace.*stale or ambiguous/i); +}); + test('moving meaning and prose criteria to an unrelated type error is rejected', async () => { const root = await fixture('real-manifest-misbinding'); const manifest = JSON.parse(