fix(quality): prove criterion binding semantics

This commit is contained in:
2026-08-01 09:58:47 -05:00
committed by f10-coder
parent d119635265
commit 8b1b873056
9 changed files with 414 additions and 79 deletions
+3 -3
View File
@@ -14,8 +14,8 @@ Investigate any of these immediately:
## Updating a gate
1. Add or change the criterion and exact case.
2. Observe the case fail for its own stated reason.
1. Add or change the criterion, its exact criterion-side `caseRefs`, and matching case-side `criterionIds`.
2. Observe the must-fail case fail for its own stated reason; moving the binding to any undeclared case must fail verification.
3. Declare an exact inerting mutation and observe the verifier detect it.
4. If required and actual behavior differ, add a tracked remediation owner and justification.
5. If meaning changed, append provenance; never replace the original silently.
@@ -32,4 +32,4 @@ Provider evidence input is an optional JSON array of normalized pipeline records
PR CI executes current-tree verification only, unprivileged and fail-closed. It does not execute isolated own-tree replay: RM-60 must provide a protected launcher or runner-level rootless sandbox before any PR-controlled executable/configuration is evaluated. Repo-only code cannot safely grant itself the capability intended to contain itself.
The deferred replay implementation remains hard-fail when its sandbox cannot be established; it is not silently skipped as a successful replay. When RM-60 activates it under protected authority, it uses frozen own-tree dependencies, namespace/environment isolation, and archived-file identity checks. A post-merge failure triggers quarantine and revert. This is detection, not pre-merge prevention.
The deferred replay implementation remains hard-fail when its sandbox cannot be established; it is not silently skipped as a successful replay. Tests recognize unavailability only from parent-generated Bubblewrap-launch provenance combined with proof that the sandbox entry command did not run. A denial-looking string from child-controlled output is not evidence. When RM-60 activates replay under protected authority, it uses frozen own-tree dependencies, namespace/environment isolation, and archived-file identity checks. A post-merge failure triggers quarantine and revert. This is detection, not pre-merge prevention.
@@ -20,7 +20,9 @@ The queue guard currently has RM-03-owned deltas. In particular, its stdin/hered
## Criteria, prose, and compatibility
Each criterion must bind to a must-fail case. Designated governing prose uses `GATE-CLAIM:<id>` markers; an unbound marker or registered-but-missing marker fails. Orchestrator-owned claims from `TASKS.md` are bound through `docs/remediation/GATE-CLAIMS.md`, which records source headings and anchored text without changing task tracking. Marker completeness still requires RM-54 review because arbitrary English claims cannot be inferred safely.
Each criterion declares exact `caseRefs`; the verifier compares those semantic declarations bidirectionally with case-side `criterionIds` and requires at least one must-fail case. Moving a criterion ID to an unrelated case therefore fails as both a missing declared exercising case and an undeclared binding. Registered meta-negative controls misbind a criterion, remove meaning provenance, and misbind a prose claim, and each must make structure verification red for its stated reason.
Designated governing prose uses `GATE-CLAIM:<id>` markers. Each claim also names the exact must-fail `caseRef` that exercises its criterion; unknown, positive-only, unrelated, unbound, or registered-but-missing claims fail. Orchestrator-owned claims from `TASKS.md` are bound through `docs/remediation/GATE-CLAIMS.md`, which records source headings and anchored text without changing task tracking. Marker completeness still requires RM-54 review because arbitrary English claims cannot be inferred safely.
Compatibility checks detect direct contradictions in declared finite constructions. The verifier combines referenced case fixtures and environments in one isolated tree, rejects conflicting fixture/environment values, executes the construction's exact invocation, and checks its exact outcome. They do not prove semantic consistency of arbitrary natural language.
@@ -36,7 +38,7 @@ A gate with an external installed counterpart declares it explicitly. When the i
**DOES NOT:** Repository-controlled PR CI does not execute a commit's own verifier in an isolated replay. Doing so safely would require granting namespace capability before PR-controlled configuration or code runs; that same PR could consume the capability directly. This is an absent trust boundary, not unfinished hardening. RM-60 owns a runner-level rootless sandbox or protected immutable launcher; RM-59 owns the parallel artifact-integrity anchor.
The replay implementation and abuse-case tests remain fail-closed: when invoked by a future protected authority, inability to establish Bubblewrap is terminal nonzero; controls are never omitted or treated as replay success. On an unprivileged CI runner, sandbox integration tests pass only by asserting that this refusal is nonzero, while capable local/protected environments exercise the full abuse cases. Historical installs use frozen lockfiles, isolated network/PID/IPC/UTS and environment/home boundaries, and authoritative-file snapshots that detect lifecycle rewrites.
The replay implementation and abuse-case tests remain fail-closed: when invoked by a future protected authority, inability to establish Bubblewrap is terminal nonzero; controls are never omitted or treated as replay success. On an unprivileged CI runner, sandbox integration tests pass only when the result carries parent-generated Bubblewrap-launch provenance and proves the sandbox entry command never ran. Child-controlled text that merely reproduces a Bubblewrap denial is not accepted. Capable local/protected environments exercise the full abuse cases. Historical installs use frozen lockfiles, isolated network/PID/IPC/UTS and environment/home boundaries, and authoritative-file snapshots that detect lifecycle rewrites.
Retained provider evidence can assert terminal-success **current-tree** records for prior commits when supplied through `GATE_PROVIDER_EVIDENCE_FILE`. Each normalized record contains `commit`, unique integer pipeline `number`, pipeline `status`, and exactly one `gate-verify` step; the highest-numbered rerun is authoritative. Ambiguous duplicates fail. Absent, expired, or currently-running evidence is reported explicitly and never inferred as success.
+4 -3
View File
@@ -30,7 +30,7 @@ Existing deterministic gates can return success without enforcing their stated p
1. `RM02-REQ-01`: The JSON registry SHALL give every gate and criterion a stable ID and SHALL declare exact invocation, input classes, cases, exact observed and required exit codes, reason diagnostics, and criterion bindings.
2. `RM02-REQ-02`: Every gate SHALL have at least one observed-red must-fail case. The verifier SHALL reject missing, stale, ambiguous, or ineffective declared inert mutations and SHALL name an externally inerted gate.
3. `RM02-REQ-03`: Every acceptance criterion SHALL bind to a case that can fail for that criterion's stated reason. Security/integrity prose claims in the designated governing documents SHALL carry bound `GATE-CLAIM:<id>` markers.
3. `RM02-REQ-03`: Every acceptance criterion SHALL declare exact criterion-side `caseRefs` that match case-side `criterionIds` bidirectionally and include a case that can fail for that criterion's stated reason. Moving a binding to an unrelated case SHALL fail verification. Security/integrity prose claims in the designated governing documents SHALL carry bound `GATE-CLAIM:<id>` markers and declare the exact must-fail case exercising the claim criterion.
4. `RM02-REQ-04`: Declared finite compatibility scenarios SHALL execute together and direct modeled contradictions SHALL fail. This does not claim semantic consistency of arbitrary English.
5. `RM02-REQ-05`: Restated criteria SHALL retain original text, current text, reason, finding/task, and dated meaning-change history.
6. `RM02-REQ-06`: An observed behavior differing from required behavior SHALL be reported as `DEFECT` with a tracked owner; an ownerless delta SHALL fail verification. Such a gate SHALL never be described as passing, green, or OK.
@@ -51,13 +51,14 @@ Existing deterministic gates can return success without enforcing their stated p
2. `RM02-AC-02`: Externally mutate any registered gate at its declared inerting point so its failure path succeeds; verification returns nonzero and names that gate. The verifier's internal meta-control is observed red before its healthy result is trusted.
3. `RM02-AC-03`: An executable added under a declared gate root without an entry returns nonzero and includes `unregistered gate`.
4. `RM02-AC-04`: A gate with zero must-fail cases returns nonzero and includes `no negative control`.
5. `RM02-AC-05`: Unbound criteria, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID.
5. `RM02-AC-05`: Unbound or semantically misbound criteria, prose claims bound to unrelated cases, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID. Registered meta-negative controls move a criterion binding, remove meaning provenance, and redirect a prose claim to an unrelated case; each is observed red for its stated reason.
6. `RM02-AC-06`: CI configuration invokes the verifier unconditionally on every pull request.
7. `RM02-AC-07`: PR output states adjacent `DOES`/`DOES NOT` boundaries: current-tree gates and inerting mutations execute unprivileged and fail-closed; isolated own-tree replay does not execute in repository-controlled PR CI. RM-60/RM-59 are named, retained provider evidence is never inferred, and future protected post-merge detection specifies quarantine/revert rather than claiming pre-merge prevention.
### Risks, dependencies, and verification boundary
- The repository verifier proves declared controls, modeled scenarios, source/deployed equality at execution time, and unprivileged current-tree behavior. It does **not** execute isolated per-commit replay or defend against an actor able to rewrite the gate, registry, verifier, and sandbox entry consistently.
- The repository verifier proves declared controls, modeled scenarios, bidirectional declared criterion/case relationships, source/deployed equality at execution time, and unprivileged current-tree behavior. It does **not** infer arbitrary-English semantics, execute isolated per-commit replay, or defend against an actor able to rewrite the gate, registry, verifier, and sandbox entry consistently.
- Sandbox refusal tests require parent-generated Bubblewrap-launch provenance and proof that the sandbox entry command never ran; child-controlled denial-looking text alone cannot establish unavailability.
- Repo-only code cannot both grant namespace capability to PR configuration and prevent that same PR from using the capability directly. RM-60 owns a runner/provider-controlled pre-execution boundary; RM-59 owns the parallel artifact-integrity anchor.
- Protected post-merge replay, once RM-60 exists, is detection only. Failure requires immediate quarantine of the affected result and revert of the offending merge; it is not equivalent to a pre-merge gate.
- External branch protection and provider CI history supply merge-time current-tree evidence where retained. RM-25 tracks provider-side enforcement.
+3 -2
View File
@@ -39,7 +39,7 @@ Deliver the seven-gate registry and RED-first anti-inert verifier on `feat/rm-02
- CI wiring RED: package script and unconditional Woodpecker step tests both failed before wiring.
- History RED: history test failed with missing module before own-tree manifest selection/provider classification was implemented.
- `pnpm gate:verify`: exit 0; seven gates each reported `META-NEGATIVE-CONTROL ... observed red`; queue source/deployed drift control observed red; six queue behavior deltas printed as `DEFECT (owner: RM-03)`.
- Focused Node tests: 35/35 pass after review hardening (24 verifier/wiring plus 11 history/provider tests).
- Focused Node tests: 37/37 pass after review hardening (26 verifier/wiring plus 11 history/provider tests).
- `pnpm typecheck`: pass (45/45 Turbo tasks).
- `pnpm lint`: pass (25/25 Turbo tasks).
- `pnpm format:check`: pass.
@@ -61,7 +61,8 @@ The queue guard's `get_state_from_status_json` runs `python3 - <<'PY'` while pro
- Initial PR pipeline #2177 exposed Woodpecker's shallow boundary: the activation parent object was present but marked shallow, so `merge-base --is-ancestor` correctly refused to infer ancestry. The unconditional gate step now unshallows before ancestry/provenance checks; its wiring test was observed RED before the CI fix.
- Pipeline #2178 then proved the unprivileged Docker runner cannot establish Bubblewrap namespaces. A privileged experiment remained uncommitted and was rejected after Codex correctly rated it CRITICAL: PR-controlled code executes before an in-repository sandbox and could directly use the granted capability.
- `mos-remediation` and `rev-974` independently ruled Option C. RM02-REQ-10 now retains its original text, restatement, and reason: PR CI verifies only the current tree, unprivileged and fail-closed; isolated own-tree replay is deferred to RM-60/#1031's external pre-execution authority, cross-referenced with RM-59. Future protected post-merge replay is detection with quarantine/revert, never pre-merge prevention.
- RED-first boundary test proved the old path executed an inert intermediate verifier. The revised path states adjacent `DOES`/`DOES NOT` claims, validates historical manifest provenance without executing it, and infers no replay success. Direct sandbox tests remain hard-fail; unprivileged CI asserts terminal refusal instead of treating replay as success. Pipelines #2179/#2180/#2181 exposed two runner refusal forms: namespace denial as `spawnSync bwrap` with `error.code=EPERM`, and a test image without Bubblewrap as `error.code=ENOENT`. The replay diagnostic now preserves spawn errors; the refusal detector recognizes only exact `spawnSync bwrap` provenance for `EPERM`/`EACCES`/`ENOENT`, plus Bubblewrap's known namespace-refusal text. Focused negative assertions reject both unrelated `spawnSync git EPERM` and verifier output that merely says `bwrap ENOENT`.
- RED-first boundary test proved the old path executed an inert intermediate verifier. The revised path states adjacent `DOES`/`DOES NOT` claims, validates historical manifest provenance without executing it, and infers no replay success. Direct sandbox tests remain hard-fail; unprivileged CI asserts terminal refusal instead of treating replay as success. Pipelines #2179/#2180/#2181 exposed two runner refusal forms: namespace denial as `spawnSync bwrap` with `error.code=EPERM`, and a test image without Bubblewrap as `error.code=ENOENT`. The replay diagnostic now preserves spawn errors. Parent-generated launcher/entry metadata distinguishes refusal before sandbox entry from child-controlled output; the detector recognizes exact `spawnSync bwrap` provenance for `EPERM`/`EACCES`/`ENOENT` and known namespace-refusal text only when the entry command provably did not run. Focused negative assertions reject unrelated `spawnSync git EPERM`, verifier output that merely says `bwrap ENOENT`, and exact namespace-denial impersonation without provenance or after sandbox entry.
- Exact-head independent review at `9b4d4beb` found two valid blockers. RED-first controls reproduced both: denial-looking child stderr was accepted as sandbox unavailability, and moving meaning/prose criterion IDs to an unrelated type-error case left `gate:verify` green. Bubblewrap execution now emits a parent-generated random entry marker and returns parent-owned launcher/entry metadata; unavailability requires Bubblewrap launcher provenance plus proof entry never ran, so exact denial impersonation from plain or entered-child results is rejected. Criterion objects now declare exact `caseRefs`, checked bidirectionally against case-side `criterionIds`; prose claims declare an exact must-fail `caseRef`. Registered must-fail cases move a criterion binding, remove meaning provenance, and redirect a prose claim, each producing its stable reason. The review freeze was deliberately lifted before remediation.
- Option C security review reported no findings. Code review rejected an initial unrelated typecheck binding for the new security criterion. It was replaced with a dedicated registered `privileged-pr-gate` case: the fixture injects a privilege key into the gate step, the wiring control rejects it for that exact reason, and `gate:verify` observes the boundary negative control. Follow-up hardening uses a closed exact gate-step construction, rejects privilege across the entire pipeline, rejects non-canonical/merged YAML keys, and pins the unrestricted PR/main trigger block; quoted/escaped/alias/merge/duplicate/filter bypass tests pass. Final Codex code review approved with no findings.
## Documentation checklist
+195 -46
View File
@@ -25,7 +25,20 @@
"currentText": "Every registered check is observed red for its own stated reason before its green counts.",
"claimType": "integrity",
"source": "docs/remediation/MISSION.md#first-class-principle-pre-registration",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": [
"quality-typecheck/type-error",
"quality-lint/invalid-syntax",
"quality-format/unformatted-json",
"checkout-preflight/stale-build-lock",
"checkout-preflight/criterion-misbinding",
"checkout-preflight/missing-meaning-provenance",
"checkout-preflight/prose-claim-misbinding",
"ci-queue-wait/no-status-required",
"ci-queue-wait/unknown-option",
"hook-pre-commit/lint-staged-failure",
"hook-pre-push/typecheck-failure"
]
},
{
"id": "RM02-SET-COVERS",
@@ -33,7 +46,8 @@
"currentText": "Every acceptance criterion is bound to the specific case that exercises it.",
"claimType": "integrity",
"source": "docs/remediation/TASKS.md#d-17",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["checkout-preflight/criterion-misbinding"]
},
{
"id": "RM02-MODELED-CONSISTENCY",
@@ -50,6 +64,14 @@
"task": "RM-54/RM-55",
"date": "2026-08-01"
}
],
"caseRefs": [
"quality-typecheck/clean-tree",
"quality-lint/clean-tree",
"quality-format/clean-tree",
"checkout-preflight/clean-tree",
"hook-pre-commit/lint-staged-failure",
"hook-pre-push/typecheck-failure"
]
},
{
@@ -58,7 +80,8 @@
"currentText": "A restated criterion retains original text, restatement, and reason.",
"claimType": "integrity",
"source": "docs/remediation/TASKS.md#d-18",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["checkout-preflight/missing-meaning-provenance"]
},
{
"id": "RM02-PROSE-CONTROL",
@@ -75,24 +98,26 @@
"task": "RM-54",
"date": "2026-08-01"
}
]
],
"caseRefs": ["checkout-preflight/prose-claim-misbinding"]
},
{
"id": "RM02-CURRENT-TREE-BOUNDARY",
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE not against current main.",
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE \u2014 not against current main.",
"currentText": "PR CI performs unprivileged, fail-closed current-tree verification only; isolated per-commit replay is deferred to RM-60's protected post-merge/main authority as detection with quarantine/revert, not pre-merge prevention.",
"claimType": "security",
"source": "docs/PRD.md#rm02-req-10-meaning-change-provenance",
"meaningChanges": [
{
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE not against current main.",
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE \u2014 not against current main.",
"restatement": "PR CI performs unprivileged, fail-closed current-tree verification only; isolated per-commit replay is deferred to a protected post-merge/main authority as detection with quarantine/revert, not pre-merge prevention.",
"reason": "PR-controlled code would otherwise receive and could directly use the namespace capability intended to contain it; the pre-execution trust boundary is absent at the repository layer.",
"finding": "D-25",
"task": "RM-60/RM-59",
"date": "2026-08-01"
}
]
],
"caseRefs": ["checkout-preflight/privileged-pr-gate"]
},
{
"id": "QUALITY-TYPECHECK",
@@ -100,7 +125,8 @@
"currentText": "The root typecheck rejects a TypeScript type error.",
"claimType": "quality",
"source": "package.json#scripts.typecheck",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["quality-typecheck/clean-tree", "quality-typecheck/type-error"]
},
{
"id": "QUALITY-LINT",
@@ -108,7 +134,8 @@
"currentText": "The root lint gate rejects invalid TypeScript syntax.",
"claimType": "quality",
"source": "package.json#scripts.lint",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["quality-lint/clean-tree", "quality-lint/invalid-syntax"]
},
{
"id": "QUALITY-FORMAT",
@@ -116,7 +143,8 @@
"currentText": "The root format gate rejects an unformatted tracked-format input.",
"claimType": "quality",
"source": "package.json#scripts.format:check",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["quality-format/clean-tree", "quality-format/unformatted-json"]
},
{
"id": "CHECKOUT-PREFLIGHT",
@@ -133,7 +161,8 @@
"task": "RM-59",
"date": "2026-07-31"
}
]
],
"caseRefs": ["checkout-preflight/clean-tree", "checkout-preflight/stale-build-lock"]
},
{
"id": "QUEUE-GUARD",
@@ -150,6 +179,15 @@
"task": "RM-03",
"date": "2026-08-01"
}
],
"caseRefs": [
"ci-queue-wait/terminal-success",
"ci-queue-wait/no-status-required",
"ci-queue-wait/unknown-state",
"ci-queue-wait/malformed-status",
"ci-queue-wait/terminal-failure",
"ci-queue-wait/push-defaults-to-main",
"ci-queue-wait/unknown-option"
]
},
{
@@ -158,7 +196,8 @@
"currentText": "The pre-commit hook propagates lint-staged failure.",
"claimType": "workflow",
"source": ".husky/pre-commit",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["hook-pre-commit/clean-staged-input", "hook-pre-commit/lint-staged-failure"]
},
{
"id": "HOOK-PRE-PUSH",
@@ -166,7 +205,8 @@
"currentText": "The pre-push hook propagates each required gate failure.",
"claimType": "workflow",
"source": ".husky/pre-push",
"meaningChanges": []
"meaningChanges": [],
"caseRefs": ["hook-pre-push/all-subgates-succeed", "hook-pre-push/typecheck-failure"]
},
{
"id": "GATE-SOURCE-DEPLOYMENT",
@@ -183,41 +223,50 @@
"task": "RM-02",
"date": "2026-08-01"
}
]
],
"caseRefs": ["ci-queue-wait/terminal-success", "ci-queue-wait/unknown-option"]
}
],
"proseClaims": [
{
"id": "OBSERVE-PROPERTY",
"criterionId": "RM02-CHECK-RIGHT"
"criterionId": "RM02-CHECK-RIGHT",
"caseRef": "checkout-preflight/criterion-misbinding"
},
{
"id": "PREREGISTRATION-BOUNDARY",
"criterionId": "RM02-SET-COVERS"
"criterionId": "RM02-SET-COVERS",
"caseRef": "checkout-preflight/criterion-misbinding"
},
{
"id": "ARTIFACT-INTEGRITY-BOUNDARY",
"criterionId": "GATE-SOURCE-DEPLOYMENT"
"criterionId": "GATE-SOURCE-DEPLOYMENT",
"caseRef": "ci-queue-wait/unknown-option"
},
{
"id": "IMPOSSIBLE-LAYER-BOUNDARY",
"criterionId": "CHECKOUT-PREFLIGHT"
"criterionId": "CHECKOUT-PREFLIGHT",
"caseRef": "checkout-preflight/stale-build-lock"
},
{
"id": "PROSE-IS-A-CLAIM",
"criterionId": "RM02-PROSE-CONTROL"
"criterionId": "RM02-PROSE-CONTROL",
"caseRef": "checkout-preflight/prose-claim-misbinding"
},
{
"id": "GENERATED-STATE-SCOPE",
"criterionId": "CHECKOUT-PREFLIGHT"
"criterionId": "CHECKOUT-PREFLIGHT",
"caseRef": "checkout-preflight/stale-build-lock"
},
{
"id": "EXECUTION-TRUST-BOUNDARY",
"criterionId": "RM02-CURRENT-TREE-BOUNDARY"
"criterionId": "RM02-CURRENT-TREE-BOUNDARY",
"caseRef": "checkout-preflight/privileged-pr-gate"
},
{
"id": "CRITERION-RESTATEMENT",
"criterionId": "RM02-MEANING-PROVENANCE"
"criterionId": "RM02-MEANING-PROVENANCE",
"caseRef": "checkout-preflight/missing-meaning-provenance"
}
],
"compatibilityScenarios": [
@@ -296,7 +345,7 @@
},
{
"id": "type-error",
"criterionIds": ["QUALITY-TYPECHECK", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
"criterionIds": ["QUALITY-TYPECHECK", "RM02-CHECK-RIGHT"],
"mustFail": true,
"required": {
"exitCode": 2,
@@ -354,7 +403,7 @@
},
{
"id": "invalid-syntax",
"criterionIds": ["QUALITY-LINT", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
"criterionIds": ["QUALITY-LINT", "RM02-CHECK-RIGHT"],
"mustFail": true,
"required": {
"exitCode": 1,
@@ -412,7 +461,7 @@
},
{
"id": "unformatted-json",
"criterionIds": ["QUALITY-FORMAT", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
"criterionIds": ["QUALITY-FORMAT", "RM02-CHECK-RIGHT"],
"mustFail": true,
"required": {
"exitCode": 1,
@@ -469,13 +518,7 @@
},
{
"id": "stale-build-lock",
"criterionIds": [
"CHECKOUT-PREFLIGHT",
"RM02-CHECK-RIGHT",
"RM02-SET-COVERS",
"RM02-MEANING-PROVENANCE",
"RM02-PROSE-CONTROL"
],
"criterionIds": ["CHECKOUT-PREFLIGHT", "RM02-CHECK-RIGHT"],
"mustFail": true,
"required": {
"exitCode": 43,
@@ -519,6 +562,122 @@
}
]
}
},
{
"id": "criterion-misbinding",
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
"mustFail": true,
"invocation": [
"node",
"scripts/gate-verify.mjs",
"--root",
".",
"--manifest",
"gates/gates.manifest.json",
"--structure-only"
],
"required": {
"exitCode": 1,
"outputPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound"
},
"actual": {
"exitCode": 1,
"outputPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound"
},
"reasonPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound",
"fixture": {
"copyPaths": [
"gates/gates.manifest.json",
"scripts/gate-verify.mjs",
"scripts/gate-history.mjs"
],
"replaceFiles": [
{
"path": "gates/gates.manifest.json",
"find": "\"criterionIds\": [\"QUALITY-TYPECHECK\", \"RM02-CHECK-RIGHT\"],",
"replace": "\"criterionIds\": [\"QUALITY-TYPECHECK\", \"RM02-CHECK-RIGHT\", \"RM02-SET-COVERS\"],"
},
{
"path": "gates/gates.manifest.json",
"find": "\"criterionIds\": [\"RM02-CHECK-RIGHT\", \"RM02-SET-COVERS\"],",
"replace": "\"criterionIds\": [\"RM02-CHECK-RIGHT\"],"
}
]
}
},
{
"id": "missing-meaning-provenance",
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-MEANING-PROVENANCE"],
"mustFail": true,
"invocation": [
"node",
"scripts/gate-verify.mjs",
"--root",
".",
"--manifest",
"gates/gates.manifest.json",
"--structure-only"
],
"required": {
"exitCode": 1,
"outputPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance"
},
"actual": {
"exitCode": 1,
"outputPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance"
},
"reasonPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance",
"fixture": {
"copyPaths": [
"gates/gates.manifest.json",
"scripts/gate-verify.mjs",
"scripts/gate-history.mjs"
],
"replaceFiles": [
{
"path": "gates/gates.manifest.json",
"find": "\"currentText\": \"A restated criterion retains original text, restatement, and reason.\",",
"replace": "\"currentText\": \"A restated criterion changed without provenance\","
}
]
}
},
{
"id": "prose-claim-misbinding",
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-PROSE-CONTROL"],
"mustFail": true,
"invocation": [
"node",
"scripts/gate-verify.mjs",
"--root",
".",
"--manifest",
"gates/gates.manifest.json",
"--structure-only"
],
"required": {
"exitCode": 1,
"outputPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL"
},
"actual": {
"exitCode": 1,
"outputPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL"
},
"reasonPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL",
"fixture": {
"copyPaths": [
"gates/gates.manifest.json",
"scripts/gate-verify.mjs",
"scripts/gate-history.mjs"
],
"replaceFiles": [
{
"path": "gates/gates.manifest.json",
"find": "\"caseRef\": \"checkout-preflight/prose-claim-misbinding\"",
"replace": "\"caseRef\": \"quality-typecheck/type-error\""
}
]
}
}
]
},
@@ -607,7 +766,7 @@
},
{
"id": "no-status-required",
"criterionIds": ["QUEUE-GUARD", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
"criterionIds": ["QUEUE-GUARD", "RM02-CHECK-RIGHT"],
"mustFail": true,
"required": {
"exitCode": 1,
@@ -669,7 +828,7 @@
},
{
"id": "unknown-state",
"criterionIds": ["QUEUE-GUARD", "RM02-SET-COVERS"],
"criterionIds": ["QUEUE-GUARD"],
"mustFail": true,
"required": {
"exitCode": 1,
@@ -946,12 +1105,7 @@
},
{
"id": "lint-staged-failure",
"criterionIds": [
"HOOK-PRE-COMMIT",
"RM02-CHECK-RIGHT",
"RM02-SET-COVERS",
"RM02-MODELED-CONSISTENCY"
],
"criterionIds": ["HOOK-PRE-COMMIT", "RM02-CHECK-RIGHT", "RM02-MODELED-CONSISTENCY"],
"mustFail": true,
"required": {
"exitCode": 19,
@@ -1027,12 +1181,7 @@
},
{
"id": "typecheck-failure",
"criterionIds": [
"HOOK-PRE-PUSH",
"RM02-CHECK-RIGHT",
"RM02-SET-COVERS",
"RM02-MODELED-CONSISTENCY"
],
"criterionIds": ["HOOK-PRE-PUSH", "RM02-CHECK-RIGHT", "RM02-MODELED-CONSISTENCY"],
"mustFail": true,
"required": {
"exitCode": 19,
+25 -5
View File
@@ -1,5 +1,5 @@
import { existsSync } from 'node:fs';
import { createHash } from 'node:crypto';
import { createHash, randomUUID } from 'node:crypto';
import { access, lstat, mkdir, mkdtemp, readFile, readdir, readlink, rm } from 'node:fs/promises';
import { spawnSync } from 'node:child_process';
import path from 'node:path';
@@ -40,7 +40,9 @@ async function snapshotAuthoritativeTree(root) {
} else if (stats.isSymbolicLink()) {
snapshot.set(relative, `symlink:${stats.mode}:${await readlink(absolute)}`);
} else if (stats.isFile()) {
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
const digest = createHash('sha256')
.update(await readFile(absolute))
.digest('hex');
snapshot.set(relative, `file:${stats.mode}:${digest}`);
}
}
@@ -59,7 +61,9 @@ async function authoritativeTreeChanges(root, snapshot) {
if (stats.isSymbolicLink()) {
actual = `symlink:${stats.mode}:${await readlink(absolute)}`;
} else if (stats.isFile()) {
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
const digest = createHash('sha256')
.update(await readFile(absolute))
.digest('hex');
actual = `file:${stats.mode}:${digest}`;
} else {
actual = `other:${stats.mode}`;
@@ -108,8 +112,24 @@ function bubblewrap(root, command, args, { storePath, timeout = 300_000 } = {})
);
if (storePath) sandboxArgs.push('--setenv', 'NPM_CONFIG_STORE_DIR', '/pnpm-store');
if (existsSync(corepackHome)) sandboxArgs.push('--setenv', 'COREPACK_HOME', '/corepack');
sandboxArgs.push(command, ...args);
return spawnSync('bwrap', sandboxArgs, { encoding: 'utf8', timeout });
const enteredMarker = `__MOSAIC_BWRAP_ENTERED_${randomUUID()}__`;
sandboxArgs.push(
'/bin/sh',
'-c',
'printf "%s\\n" "$1"; shift; exec "$@"',
'mosaic-bwrap-entry',
enteredMarker,
command,
...args,
);
const result = spawnSync('bwrap', sandboxArgs, { encoding: 'utf8', timeout });
const sandboxEntered = result.stdout?.includes(enteredMarker) === true;
return {
...result,
stdout: (result.stdout ?? '').replace(`${enteredMarker}\n`, ''),
sandboxLauncher: 'bwrap',
sandboxEntered,
};
}
export async function replayCommit(root, commit) {
+32 -4
View File
@@ -15,6 +15,7 @@ import {
const fixtureRoot = path.join(process.cwd(), '.mosaic-test-work', `gate-history-${process.pid}`);
function sandboxUnavailable(result) {
if (result.sandboxLauncher !== 'bwrap' || result.sandboxEntered === true) return false;
const detail = `${result.stdout ?? ''}${result.stderr ?? ''}${result.error?.message ?? ''}`;
const bubblewrapSpawnDenied =
['EPERM', 'EACCES', 'ENOENT'].includes(result.error?.code) &&
@@ -56,6 +57,8 @@ test('sandbox refusal classification requires Bubblewrap provenance', () => {
sandboxUnavailable({
status: null,
error: { code, message: `spawnSync bwrap ${code}` },
sandboxLauncher: 'bwrap',
sandboxEntered: false,
}),
true,
);
@@ -71,6 +74,31 @@ test('sandbox refusal classification requires Bubblewrap provenance', () => {
sandboxUnavailable({ status: 1, stderr: 'historical verifier said bwrap ENOENT' }),
false,
);
assert.equal(
sandboxUnavailable({
status: 1,
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
sandboxLauncher: 'bwrap',
sandboxEntered: false,
}),
true,
);
assert.equal(
sandboxUnavailable({
status: 1,
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
}),
false,
);
assert.equal(
sandboxUnavailable({
status: 1,
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
sandboxLauncher: 'bwrap',
sandboxEntered: true,
}),
false,
);
});
test('prospective history reads each commit own manifest rather than the current tree', async () => {
@@ -261,13 +289,13 @@ test('PR verification states the RM-60 boundary without executing an intermediat
});
assert.deepEqual(result.failures, []);
assert.ok(
result.observations.some(
(observation) => /DOES:.*current tree.*DOES NOT:.*isolated.*RM-60.*RM-59/i.test(observation),
result.observations.some((observation) =>
/DOES:.*current tree.*DOES NOT:.*isolated.*RM-60.*RM-59/i.test(observation),
),
);
assert.ok(
result.observations.some(
(observation) => /INTERMEDIATE REPLAY DEFERRED.*RM-60.*no success is inferred/i.test(observation),
result.observations.some((observation) =>
/INTERMEDIATE REPLAY DEFERRED.*RM-60.*no success is inferred/i.test(observation),
),
);
assert.ok(result.observations.every((observation) => !/INTERMEDIATE INERT/.test(observation)));
+77 -11
View File
@@ -29,12 +29,14 @@ function parseArgs(argv) {
root: process.cwd(),
manifest: 'gates/gates.manifest.json',
skipHistory: false,
structureOnly: false,
};
for (let index = 0; index < argv.length; index += 1) {
const value = argv[index];
if (value === '--root') options.root = path.resolve(argv[++index]);
else if (value === '--manifest') options.manifest = argv[++index];
else if (value === '--skip-history') options.skipHistory = true;
else if (value === '--structure-only') options.structureOnly = true;
else throw new Error(`unknown option: ${value}`);
}
return options;
@@ -307,7 +309,39 @@ function validateClosedSchema(manifest, failures) {
failures,
);
rejectDuplicateIds(manifest.criteria, 'criterion', failures);
for (const criterion of manifest.criteria ?? []) {
rejectUnknownKeys(
criterion,
new Set([
'id',
'originalText',
'currentText',
'claimType',
'source',
'meaningChanges',
'caseRefs',
]),
`criterion ${criterion.id}`,
failures,
);
if (!Array.isArray(criterion.caseRefs) || criterion.caseRefs.length === 0) {
failures.push(`${criterion.id}: no declared exercising cases`);
} else if (new Set(criterion.caseRefs).size !== criterion.caseRefs.length) {
failures.push(`${criterion.id}: duplicate declared exercising case`);
}
}
rejectDuplicateIds(manifest.proseClaims, 'prose claim', failures);
for (const claim of manifest.proseClaims ?? []) {
rejectUnknownKeys(
claim,
new Set(['id', 'criterionId', 'caseRef']),
`prose claim ${claim.id}`,
failures,
);
if (typeof claim.caseRef !== 'string' || claim.caseRef.length === 0) {
failures.push(`GATE-CLAIM:${claim.id} has no declared exercising case`);
}
}
rejectDuplicateIds(manifest.compatibilityScenarios, 'compatibility scenario', failures);
for (const scenario of manifest.compatibilityScenarios ?? []) {
rejectUnknownKeys(
@@ -390,7 +424,9 @@ function validateClosedSchema(manifest, failures) {
failures.push(`${gate.id}/${gateCase.id}: case invocation must be non-empty`);
}
if (gateCase.mustFail === true && !gateCase.reasonPattern?.trim()) {
failures.push(`${gate.id}/${gateCase.id}: must-fail case requires a non-empty reasonPattern`);
failures.push(
`${gate.id}/${gateCase.id}: must-fail case requires a non-empty reasonPattern`,
);
}
}
}
@@ -401,6 +437,7 @@ function validateStructure(manifest, failures) {
const criteria = new Map((manifest.criteria ?? []).map((criterion) => [criterion.id, criterion]));
const boundCriteria = new Set();
const negativeBoundCriteria = new Set();
const observedCaseRefs = new Map();
for (const gate of manifest.gates ?? []) {
const negativeCases = (gate.cases ?? []).filter((gateCase) => gateCase.mustFail === true);
@@ -413,12 +450,12 @@ function validateStructure(manifest, failures) {
failures.push(`${gate.id}/${gateCase.id}: unknown criterion ${criterionId}`);
}
boundCriteria.add(criterionId);
const caseRef = `${gate.id}/${gateCase.id}`;
if (!observedCaseRefs.has(criterionId)) observedCaseRefs.set(criterionId, new Set());
observedCaseRefs.get(criterionId).add(caseRef);
if (gateCase.mustFail === true) negativeBoundCriteria.add(criterionId);
}
if (
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
!gateCase.defect?.owner
) {
if (!structuredValuesEqual(gateCase.required, gateCase.actual) && !gateCase.defect?.owner) {
failures.push(`${gate.id}/${gateCase.id}: behavior delta requires a tracked owner`);
}
}
@@ -427,6 +464,18 @@ function validateStructure(manifest, failures) {
for (const claim of manifest.proseClaims ?? []) {
if (!criteria.has(claim.criterionId)) {
failures.push(`GATE-CLAIM:${claim.id} references unknown criterion ${claim.criterionId}`);
continue;
}
const found = findGateCase(manifest, claim.caseRef ?? '');
if (!found) {
failures.push(`GATE-CLAIM:${claim.id} references missing exercising case ${claim.caseRef}`);
} else if (
found.gateCase.mustFail !== true ||
!found.gateCase.criterionIds?.includes(claim.criterionId)
) {
failures.push(
`GATE-CLAIM:${claim.id} exercising case ${claim.caseRef} does not exercise criterion ${claim.criterionId}`,
);
}
}
@@ -435,6 +484,20 @@ function validateStructure(manifest, failures) {
else if (!negativeBoundCriteria.has(criterion.id)) {
failures.push(`${criterion.id}: no must-fail case exercises this criterion`);
}
const declaredRefs = new Set(criterion.caseRefs ?? []);
const actualRefs = observedCaseRefs.get(criterion.id) ?? new Set();
for (const caseRef of declaredRefs) {
const found = findGateCase(manifest, caseRef);
if (!found) failures.push(`${criterion.id}: declared exercising case ${caseRef} is missing`);
else if (!actualRefs.has(caseRef)) {
failures.push(`${criterion.id}: declared exercising case ${caseRef} is not bound`);
}
}
for (const caseRef of actualRefs) {
if (!declaredRefs.has(caseRef)) {
failures.push(`${criterion.id}: bound to undeclared exercising case ${caseRef}`);
}
}
if (
criterion.originalText !== criterion.currentText &&
(!Array.isArray(criterion.meaningChanges) || criterion.meaningChanges.length === 0)
@@ -634,7 +697,9 @@ async function runCompatibilityScenario(root, manifest, scenario, failures, obse
for (const caseRef of scenario.caseRefs ?? []) {
const found = findGateCase(manifest, caseRef);
if (!found) {
failures.push(`${scenario.id}: compatibility construction references missing case ${caseRef}`);
failures.push(
`${scenario.id}: compatibility construction references missing case ${caseRef}`,
);
} else {
referenced.push({ caseRef, ...found });
}
@@ -671,7 +736,10 @@ async function runCompatibilityScenario(root, manifest, scenario, failures, obse
}
await applyFixture(sandbox, fixture);
}
for (const source of [...referenced.map(({ gateCase }) => gateCase.environment), scenario.environment]) {
for (const source of [
...referenced.map(({ gateCase }) => gateCase.environment),
scenario.environment,
]) {
for (const [key, value] of Object.entries(source ?? {})) {
if (environment[key] !== undefined && environment[key] !== value) {
failures.push(`${scenario.id}: incompatible environment values for ${key}`);
@@ -703,6 +771,7 @@ export async function verifyRegistry(options) {
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
validateStructure(manifest, failures);
if (options.structureOnly) return { failures, manifest, observations };
await validateClaims(options.root, manifest, failures);
await validateDiscovery(options.root, manifest, failures);
@@ -719,10 +788,7 @@ export async function verifyRegistry(options) {
if (gateCase.reasonPattern && !new RegExp(gateCase.reasonPattern, 'm').test(combined)) {
failures.push(`${gate.id}/${gateCase.id}: did not fail for its stated reason`);
}
if (
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
gateCase.defect?.owner
) {
if (!structuredValuesEqual(gateCase.required, gateCase.actual) && gateCase.defect?.owner) {
observations.push(
`DEFECT (owner: ${gateCase.defect.owner}) ${gate.id}/${gateCase.id}: required ${JSON.stringify(gateCase.required)}, actual ${JSON.stringify(gateCase.actual)}`,
);
+71 -3
View File
@@ -30,6 +30,7 @@ function baseManifest() {
claimType: 'integrity',
source: 'fixture',
meaningChanges: [],
caseRefs: ['meta-fixture/rejects-bad-input'],
},
],
compatibilityScenarios: [],
@@ -72,10 +73,18 @@ async function writeManifest(root, manifest) {
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), `${JSON.stringify(manifest)}\n`);
}
function verify(root) {
function verify(root, extraArgs = []) {
return spawnSync(
process.execPath,
[verifier, '--root', root, '--manifest', 'gates/gates.manifest.json', '--skip-history'],
[
verifier,
'--root',
root,
'--manifest',
'gates/gates.manifest.json',
'--skip-history',
...extraArgs,
],
{ cwd: root, encoding: 'utf8', env: { ...process.env, HOME: os.homedir() } },
);
}
@@ -260,6 +269,61 @@ test('a required-versus-actual delta without a tracked owner is rejected', async
assert.match(output(result), /meta-fixture.*delta.*tracked owner/i);
});
test('moving criterion bindings to unrelated cases is rejected', async () => {
const root = await fixture('semantic-misbinding');
await writeGate(root);
const manifest = baseManifest();
manifest.criteria.push({
id: 'META-CRIT-2',
originalText: 'The fixture reports the second rejection reason.',
currentText: 'The fixture reports the second rejection reason.',
claimType: 'integrity',
source: 'fixture',
meaningChanges: [],
caseRefs: ['meta-fixture/rejects-second-input'],
});
manifest.gates[0].cases.push({
...manifest.gates[0].cases[0],
id: 'rejects-second-input',
criterionIds: ['META-CRIT-1'],
});
manifest.gates[0].cases[0].criterionIds = ['META-CRIT-2'];
await writeManifest(root, manifest);
const result = verify(root);
assert.notEqual(result.status, 0);
assert.match(output(result), /META-CRIT-1.*rejects-bad-input.*not bound/i);
assert.match(output(result), /META-CRIT-2.*rejects-second-input.*not bound/i);
});
test('moving meaning and prose criteria to an unrelated type error is rejected', async () => {
const root = await fixture('real-manifest-misbinding');
const manifest = JSON.parse(
await readFile(path.join(process.cwd(), 'gates', 'gates.manifest.json'), 'utf8'),
);
for (const gate of manifest.gates) {
for (const gateCase of gate.cases) {
gateCase.criterionIds = gateCase.criterionIds.filter(
(id) => !['RM02-MEANING-PROVENANCE', 'RM02-PROSE-CONTROL'].includes(id),
);
}
}
const typeError = manifest.gates
.find((gate) => gate.id === 'quality-typecheck')
.cases.find((gateCase) => gateCase.id === 'type-error');
typeError.criterionIds.push('RM02-MEANING-PROVENANCE', 'RM02-PROSE-CONTROL');
await writeManifest(root, manifest);
const result = verify(root, ['--structure-only']);
assert.notEqual(result.status, 0);
assert.match(output(result), /RM02-MEANING-PROVENANCE.*missing-meaning-provenance.*not bound/i);
assert.match(output(result), /RM02-PROSE-CONTROL.*prose-claim-misbinding.*not bound/i);
assert.match(
output(result),
/RM02-(?:MEANING-PROVENANCE|PROSE-CONTROL).*undeclared exercising case quality-typecheck\/type-error/i,
);
});
test('a criterion with no bound case is rejected', async () => {
const root = await fixture('unbound-criterion');
await writeGate(root);
@@ -335,6 +399,7 @@ test('compatibility scenarios execute referenced conditions as one construction'
reasonPattern: 'SECOND_REASON',
environment: { SECOND_REASON: 'SECOND_REASON' },
});
manifest.criteria[0].caseRefs.push('meta-fixture/second-condition');
manifest.gates[0].cases[0].fixture = {
writeFiles: [{ path: 'conditions/first', content: 'present\n' }],
};
@@ -427,7 +492,10 @@ test('deployment drift meta-control fails if the shared comparator is made inert
const root = await fixture('deployment-comparator-inert');
await writeGate(root);
await mkdir(path.join(root, 'deployed'), { recursive: true });
await copyFile(path.join(root, 'gates', 'meta-fixture.sh'), path.join(root, 'deployed', 'meta-fixture.sh'));
await copyFile(
path.join(root, 'gates', 'meta-fixture.sh'),
path.join(root, 'deployed', 'meta-fixture.sh'),
);
const manifest = baseManifest();
manifest.gates[0].deployment = { kind: 'file', path: 'deployed/meta-fixture.sh' };
await writeManifest(root, manifest);