fix(quality): prove criterion binding semantics
This commit is contained in:
@@ -14,8 +14,8 @@ Investigate any of these immediately:
|
||||
|
||||
## Updating a gate
|
||||
|
||||
1. Add or change the criterion and exact case.
|
||||
2. Observe the case fail for its own stated reason.
|
||||
1. Add or change the criterion, its exact criterion-side `caseRefs`, and matching case-side `criterionIds`.
|
||||
2. Observe the must-fail case fail for its own stated reason; moving the binding to any undeclared case must fail verification.
|
||||
3. Declare an exact inerting mutation and observe the verifier detect it.
|
||||
4. If required and actual behavior differ, add a tracked remediation owner and justification.
|
||||
5. If meaning changed, append provenance; never replace the original silently.
|
||||
@@ -32,4 +32,4 @@ Provider evidence input is an optional JSON array of normalized pipeline records
|
||||
|
||||
PR CI executes current-tree verification only, unprivileged and fail-closed. It does not execute isolated own-tree replay: RM-60 must provide a protected launcher or runner-level rootless sandbox before any PR-controlled executable/configuration is evaluated. Repo-only code cannot safely grant itself the capability intended to contain itself.
|
||||
|
||||
The deferred replay implementation remains hard-fail when its sandbox cannot be established; it is not silently skipped as a successful replay. When RM-60 activates it under protected authority, it uses frozen own-tree dependencies, namespace/environment isolation, and archived-file identity checks. A post-merge failure triggers quarantine and revert. This is detection, not pre-merge prevention.
|
||||
The deferred replay implementation remains hard-fail when its sandbox cannot be established; it is not silently skipped as a successful replay. Tests recognize unavailability only from parent-generated Bubblewrap-launch provenance combined with proof that the sandbox entry command did not run. A denial-looking string from child-controlled output is not evidence. When RM-60 activates replay under protected authority, it uses frozen own-tree dependencies, namespace/environment isolation, and archived-file identity checks. A post-merge failure triggers quarantine and revert. This is detection, not pre-merge prevention.
|
||||
|
||||
@@ -20,7 +20,9 @@ The queue guard currently has RM-03-owned deltas. In particular, its stdin/hered
|
||||
|
||||
## Criteria, prose, and compatibility
|
||||
|
||||
Each criterion must bind to a must-fail case. Designated governing prose uses `GATE-CLAIM:<id>` markers; an unbound marker or registered-but-missing marker fails. Orchestrator-owned claims from `TASKS.md` are bound through `docs/remediation/GATE-CLAIMS.md`, which records source headings and anchored text without changing task tracking. Marker completeness still requires RM-54 review because arbitrary English claims cannot be inferred safely.
|
||||
Each criterion declares exact `caseRefs`; the verifier compares those semantic declarations bidirectionally with case-side `criterionIds` and requires at least one must-fail case. Moving a criterion ID to an unrelated case therefore fails as both a missing declared exercising case and an undeclared binding. Registered meta-negative controls misbind a criterion, remove meaning provenance, and misbind a prose claim, and each must make structure verification red for its stated reason.
|
||||
|
||||
Designated governing prose uses `GATE-CLAIM:<id>` markers. Each claim also names the exact must-fail `caseRef` that exercises its criterion; unknown, positive-only, unrelated, unbound, or registered-but-missing claims fail. Orchestrator-owned claims from `TASKS.md` are bound through `docs/remediation/GATE-CLAIMS.md`, which records source headings and anchored text without changing task tracking. Marker completeness still requires RM-54 review because arbitrary English claims cannot be inferred safely.
|
||||
|
||||
Compatibility checks detect direct contradictions in declared finite constructions. The verifier combines referenced case fixtures and environments in one isolated tree, rejects conflicting fixture/environment values, executes the construction's exact invocation, and checks its exact outcome. They do not prove semantic consistency of arbitrary natural language.
|
||||
|
||||
@@ -36,7 +38,7 @@ A gate with an external installed counterpart declares it explicitly. When the i
|
||||
|
||||
**DOES NOT:** Repository-controlled PR CI does not execute a commit's own verifier in an isolated replay. Doing so safely would require granting namespace capability before PR-controlled configuration or code runs; that same PR could consume the capability directly. This is an absent trust boundary, not unfinished hardening. RM-60 owns a runner-level rootless sandbox or protected immutable launcher; RM-59 owns the parallel artifact-integrity anchor.
|
||||
|
||||
The replay implementation and abuse-case tests remain fail-closed: when invoked by a future protected authority, inability to establish Bubblewrap is terminal nonzero; controls are never omitted or treated as replay success. On an unprivileged CI runner, sandbox integration tests pass only by asserting that this refusal is nonzero, while capable local/protected environments exercise the full abuse cases. Historical installs use frozen lockfiles, isolated network/PID/IPC/UTS and environment/home boundaries, and authoritative-file snapshots that detect lifecycle rewrites.
|
||||
The replay implementation and abuse-case tests remain fail-closed: when invoked by a future protected authority, inability to establish Bubblewrap is terminal nonzero; controls are never omitted or treated as replay success. On an unprivileged CI runner, sandbox integration tests pass only when the result carries parent-generated Bubblewrap-launch provenance and proves the sandbox entry command never ran. Child-controlled text that merely reproduces a Bubblewrap denial is not accepted. Capable local/protected environments exercise the full abuse cases. Historical installs use frozen lockfiles, isolated network/PID/IPC/UTS and environment/home boundaries, and authoritative-file snapshots that detect lifecycle rewrites.
|
||||
|
||||
Retained provider evidence can assert terminal-success **current-tree** records for prior commits when supplied through `GATE_PROVIDER_EVIDENCE_FILE`. Each normalized record contains `commit`, unique integer pipeline `number`, pipeline `status`, and exactly one `gate-verify` step; the highest-numbered rerun is authoritative. Ambiguous duplicates fail. Absent, expired, or currently-running evidence is reported explicitly and never inferred as success.
|
||||
|
||||
|
||||
+4
-3
@@ -30,7 +30,7 @@ Existing deterministic gates can return success without enforcing their stated p
|
||||
|
||||
1. `RM02-REQ-01`: The JSON registry SHALL give every gate and criterion a stable ID and SHALL declare exact invocation, input classes, cases, exact observed and required exit codes, reason diagnostics, and criterion bindings.
|
||||
2. `RM02-REQ-02`: Every gate SHALL have at least one observed-red must-fail case. The verifier SHALL reject missing, stale, ambiguous, or ineffective declared inert mutations and SHALL name an externally inerted gate.
|
||||
3. `RM02-REQ-03`: Every acceptance criterion SHALL bind to a case that can fail for that criterion's stated reason. Security/integrity prose claims in the designated governing documents SHALL carry bound `GATE-CLAIM:<id>` markers.
|
||||
3. `RM02-REQ-03`: Every acceptance criterion SHALL declare exact criterion-side `caseRefs` that match case-side `criterionIds` bidirectionally and include a case that can fail for that criterion's stated reason. Moving a binding to an unrelated case SHALL fail verification. Security/integrity prose claims in the designated governing documents SHALL carry bound `GATE-CLAIM:<id>` markers and declare the exact must-fail case exercising the claim criterion.
|
||||
4. `RM02-REQ-04`: Declared finite compatibility scenarios SHALL execute together and direct modeled contradictions SHALL fail. This does not claim semantic consistency of arbitrary English.
|
||||
5. `RM02-REQ-05`: Restated criteria SHALL retain original text, current text, reason, finding/task, and dated meaning-change history.
|
||||
6. `RM02-REQ-06`: An observed behavior differing from required behavior SHALL be reported as `DEFECT` with a tracked owner; an ownerless delta SHALL fail verification. Such a gate SHALL never be described as passing, green, or OK.
|
||||
@@ -51,13 +51,14 @@ Existing deterministic gates can return success without enforcing their stated p
|
||||
2. `RM02-AC-02`: Externally mutate any registered gate at its declared inerting point so its failure path succeeds; verification returns nonzero and names that gate. The verifier's internal meta-control is observed red before its healthy result is trusted.
|
||||
3. `RM02-AC-03`: An executable added under a declared gate root without an entry returns nonzero and includes `unregistered gate`.
|
||||
4. `RM02-AC-04`: A gate with zero must-fail cases returns nonzero and includes `no negative control`.
|
||||
5. `RM02-AC-05`: Unbound criteria, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID.
|
||||
5. `RM02-AC-05`: Unbound or semantically misbound criteria, prose claims bound to unrelated cases, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID. Registered meta-negative controls move a criterion binding, remove meaning provenance, and redirect a prose claim to an unrelated case; each is observed red for its stated reason.
|
||||
6. `RM02-AC-06`: CI configuration invokes the verifier unconditionally on every pull request.
|
||||
7. `RM02-AC-07`: PR output states adjacent `DOES`/`DOES NOT` boundaries: current-tree gates and inerting mutations execute unprivileged and fail-closed; isolated own-tree replay does not execute in repository-controlled PR CI. RM-60/RM-59 are named, retained provider evidence is never inferred, and future protected post-merge detection specifies quarantine/revert rather than claiming pre-merge prevention.
|
||||
|
||||
### Risks, dependencies, and verification boundary
|
||||
|
||||
- The repository verifier proves declared controls, modeled scenarios, source/deployed equality at execution time, and unprivileged current-tree behavior. It does **not** execute isolated per-commit replay or defend against an actor able to rewrite the gate, registry, verifier, and sandbox entry consistently.
|
||||
- The repository verifier proves declared controls, modeled scenarios, bidirectional declared criterion/case relationships, source/deployed equality at execution time, and unprivileged current-tree behavior. It does **not** infer arbitrary-English semantics, execute isolated per-commit replay, or defend against an actor able to rewrite the gate, registry, verifier, and sandbox entry consistently.
|
||||
- Sandbox refusal tests require parent-generated Bubblewrap-launch provenance and proof that the sandbox entry command never ran; child-controlled denial-looking text alone cannot establish unavailability.
|
||||
- Repo-only code cannot both grant namespace capability to PR configuration and prevent that same PR from using the capability directly. RM-60 owns a runner/provider-controlled pre-execution boundary; RM-59 owns the parallel artifact-integrity anchor.
|
||||
- Protected post-merge replay, once RM-60 exists, is detection only. Failure requires immediate quarantine of the affected result and revert of the offending merge; it is not equivalent to a pre-merge gate.
|
||||
- External branch protection and provider CI history supply merge-time current-tree evidence where retained. RM-25 tracks provider-side enforcement.
|
||||
|
||||
@@ -39,7 +39,7 @@ Deliver the seven-gate registry and RED-first anti-inert verifier on `feat/rm-02
|
||||
- CI wiring RED: package script and unconditional Woodpecker step tests both failed before wiring.
|
||||
- History RED: history test failed with missing module before own-tree manifest selection/provider classification was implemented.
|
||||
- `pnpm gate:verify`: exit 0; seven gates each reported `META-NEGATIVE-CONTROL ... observed red`; queue source/deployed drift control observed red; six queue behavior deltas printed as `DEFECT (owner: RM-03)`.
|
||||
- Focused Node tests: 35/35 pass after review hardening (24 verifier/wiring plus 11 history/provider tests).
|
||||
- Focused Node tests: 37/37 pass after review hardening (26 verifier/wiring plus 11 history/provider tests).
|
||||
- `pnpm typecheck`: pass (45/45 Turbo tasks).
|
||||
- `pnpm lint`: pass (25/25 Turbo tasks).
|
||||
- `pnpm format:check`: pass.
|
||||
@@ -61,7 +61,8 @@ The queue guard's `get_state_from_status_json` runs `python3 - <<'PY'` while pro
|
||||
- Initial PR pipeline #2177 exposed Woodpecker's shallow boundary: the activation parent object was present but marked shallow, so `merge-base --is-ancestor` correctly refused to infer ancestry. The unconditional gate step now unshallows before ancestry/provenance checks; its wiring test was observed RED before the CI fix.
|
||||
- Pipeline #2178 then proved the unprivileged Docker runner cannot establish Bubblewrap namespaces. A privileged experiment remained uncommitted and was rejected after Codex correctly rated it CRITICAL: PR-controlled code executes before an in-repository sandbox and could directly use the granted capability.
|
||||
- `mos-remediation` and `rev-974` independently ruled Option C. RM02-REQ-10 now retains its original text, restatement, and reason: PR CI verifies only the current tree, unprivileged and fail-closed; isolated own-tree replay is deferred to RM-60/#1031's external pre-execution authority, cross-referenced with RM-59. Future protected post-merge replay is detection with quarantine/revert, never pre-merge prevention.
|
||||
- RED-first boundary test proved the old path executed an inert intermediate verifier. The revised path states adjacent `DOES`/`DOES NOT` claims, validates historical manifest provenance without executing it, and infers no replay success. Direct sandbox tests remain hard-fail; unprivileged CI asserts terminal refusal instead of treating replay as success. Pipelines #2179/#2180/#2181 exposed two runner refusal forms: namespace denial as `spawnSync bwrap` with `error.code=EPERM`, and a test image without Bubblewrap as `error.code=ENOENT`. The replay diagnostic now preserves spawn errors; the refusal detector recognizes only exact `spawnSync bwrap` provenance for `EPERM`/`EACCES`/`ENOENT`, plus Bubblewrap's known namespace-refusal text. Focused negative assertions reject both unrelated `spawnSync git EPERM` and verifier output that merely says `bwrap ENOENT`.
|
||||
- RED-first boundary test proved the old path executed an inert intermediate verifier. The revised path states adjacent `DOES`/`DOES NOT` claims, validates historical manifest provenance without executing it, and infers no replay success. Direct sandbox tests remain hard-fail; unprivileged CI asserts terminal refusal instead of treating replay as success. Pipelines #2179/#2180/#2181 exposed two runner refusal forms: namespace denial as `spawnSync bwrap` with `error.code=EPERM`, and a test image without Bubblewrap as `error.code=ENOENT`. The replay diagnostic now preserves spawn errors. Parent-generated launcher/entry metadata distinguishes refusal before sandbox entry from child-controlled output; the detector recognizes exact `spawnSync bwrap` provenance for `EPERM`/`EACCES`/`ENOENT` and known namespace-refusal text only when the entry command provably did not run. Focused negative assertions reject unrelated `spawnSync git EPERM`, verifier output that merely says `bwrap ENOENT`, and exact namespace-denial impersonation without provenance or after sandbox entry.
|
||||
- Exact-head independent review at `9b4d4beb` found two valid blockers. RED-first controls reproduced both: denial-looking child stderr was accepted as sandbox unavailability, and moving meaning/prose criterion IDs to an unrelated type-error case left `gate:verify` green. Bubblewrap execution now emits a parent-generated random entry marker and returns parent-owned launcher/entry metadata; unavailability requires Bubblewrap launcher provenance plus proof entry never ran, so exact denial impersonation from plain or entered-child results is rejected. Criterion objects now declare exact `caseRefs`, checked bidirectionally against case-side `criterionIds`; prose claims declare an exact must-fail `caseRef`. Registered must-fail cases move a criterion binding, remove meaning provenance, and redirect a prose claim, each producing its stable reason. The review freeze was deliberately lifted before remediation.
|
||||
- Option C security review reported no findings. Code review rejected an initial unrelated typecheck binding for the new security criterion. It was replaced with a dedicated registered `privileged-pr-gate` case: the fixture injects a privilege key into the gate step, the wiring control rejects it for that exact reason, and `gate:verify` observes the boundary negative control. Follow-up hardening uses a closed exact gate-step construction, rejects privilege across the entire pipeline, rejects non-canonical/merged YAML keys, and pins the unrestricted PR/main trigger block; quoted/escaped/alias/merge/duplicate/filter bypass tests pass. Final Codex code review approved with no findings.
|
||||
|
||||
## Documentation checklist
|
||||
|
||||
+195
-46
@@ -25,7 +25,20 @@
|
||||
"currentText": "Every registered check is observed red for its own stated reason before its green counts.",
|
||||
"claimType": "integrity",
|
||||
"source": "docs/remediation/MISSION.md#first-class-principle-pre-registration",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": [
|
||||
"quality-typecheck/type-error",
|
||||
"quality-lint/invalid-syntax",
|
||||
"quality-format/unformatted-json",
|
||||
"checkout-preflight/stale-build-lock",
|
||||
"checkout-preflight/criterion-misbinding",
|
||||
"checkout-preflight/missing-meaning-provenance",
|
||||
"checkout-preflight/prose-claim-misbinding",
|
||||
"ci-queue-wait/no-status-required",
|
||||
"ci-queue-wait/unknown-option",
|
||||
"hook-pre-commit/lint-staged-failure",
|
||||
"hook-pre-push/typecheck-failure"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RM02-SET-COVERS",
|
||||
@@ -33,7 +46,8 @@
|
||||
"currentText": "Every acceptance criterion is bound to the specific case that exercises it.",
|
||||
"claimType": "integrity",
|
||||
"source": "docs/remediation/TASKS.md#d-17",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["checkout-preflight/criterion-misbinding"]
|
||||
},
|
||||
{
|
||||
"id": "RM02-MODELED-CONSISTENCY",
|
||||
@@ -50,6 +64,14 @@
|
||||
"task": "RM-54/RM-55",
|
||||
"date": "2026-08-01"
|
||||
}
|
||||
],
|
||||
"caseRefs": [
|
||||
"quality-typecheck/clean-tree",
|
||||
"quality-lint/clean-tree",
|
||||
"quality-format/clean-tree",
|
||||
"checkout-preflight/clean-tree",
|
||||
"hook-pre-commit/lint-staged-failure",
|
||||
"hook-pre-push/typecheck-failure"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -58,7 +80,8 @@
|
||||
"currentText": "A restated criterion retains original text, restatement, and reason.",
|
||||
"claimType": "integrity",
|
||||
"source": "docs/remediation/TASKS.md#d-18",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["checkout-preflight/missing-meaning-provenance"]
|
||||
},
|
||||
{
|
||||
"id": "RM02-PROSE-CONTROL",
|
||||
@@ -75,24 +98,26 @@
|
||||
"task": "RM-54",
|
||||
"date": "2026-08-01"
|
||||
}
|
||||
]
|
||||
],
|
||||
"caseRefs": ["checkout-preflight/prose-claim-misbinding"]
|
||||
},
|
||||
{
|
||||
"id": "RM02-CURRENT-TREE-BOUNDARY",
|
||||
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE — not against current main.",
|
||||
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE \u2014 not against current main.",
|
||||
"currentText": "PR CI performs unprivileged, fail-closed current-tree verification only; isolated per-commit replay is deferred to RM-60's protected post-merge/main authority as detection with quarantine/revert, not pre-merge prevention.",
|
||||
"claimType": "security",
|
||||
"source": "docs/PRD.md#rm02-req-10-meaning-change-provenance",
|
||||
"meaningChanges": [
|
||||
{
|
||||
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE — not against current main.",
|
||||
"originalText": "assert that every merged commit passed every required gate, evaluated AGAINST THAT COMMIT'S OWN TREE \u2014 not against current main.",
|
||||
"restatement": "PR CI performs unprivileged, fail-closed current-tree verification only; isolated per-commit replay is deferred to a protected post-merge/main authority as detection with quarantine/revert, not pre-merge prevention.",
|
||||
"reason": "PR-controlled code would otherwise receive and could directly use the namespace capability intended to contain it; the pre-execution trust boundary is absent at the repository layer.",
|
||||
"finding": "D-25",
|
||||
"task": "RM-60/RM-59",
|
||||
"date": "2026-08-01"
|
||||
}
|
||||
]
|
||||
],
|
||||
"caseRefs": ["checkout-preflight/privileged-pr-gate"]
|
||||
},
|
||||
{
|
||||
"id": "QUALITY-TYPECHECK",
|
||||
@@ -100,7 +125,8 @@
|
||||
"currentText": "The root typecheck rejects a TypeScript type error.",
|
||||
"claimType": "quality",
|
||||
"source": "package.json#scripts.typecheck",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["quality-typecheck/clean-tree", "quality-typecheck/type-error"]
|
||||
},
|
||||
{
|
||||
"id": "QUALITY-LINT",
|
||||
@@ -108,7 +134,8 @@
|
||||
"currentText": "The root lint gate rejects invalid TypeScript syntax.",
|
||||
"claimType": "quality",
|
||||
"source": "package.json#scripts.lint",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["quality-lint/clean-tree", "quality-lint/invalid-syntax"]
|
||||
},
|
||||
{
|
||||
"id": "QUALITY-FORMAT",
|
||||
@@ -116,7 +143,8 @@
|
||||
"currentText": "The root format gate rejects an unformatted tracked-format input.",
|
||||
"claimType": "quality",
|
||||
"source": "package.json#scripts.format:check",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["quality-format/clean-tree", "quality-format/unformatted-json"]
|
||||
},
|
||||
{
|
||||
"id": "CHECKOUT-PREFLIGHT",
|
||||
@@ -133,7 +161,8 @@
|
||||
"task": "RM-59",
|
||||
"date": "2026-07-31"
|
||||
}
|
||||
]
|
||||
],
|
||||
"caseRefs": ["checkout-preflight/clean-tree", "checkout-preflight/stale-build-lock"]
|
||||
},
|
||||
{
|
||||
"id": "QUEUE-GUARD",
|
||||
@@ -150,6 +179,15 @@
|
||||
"task": "RM-03",
|
||||
"date": "2026-08-01"
|
||||
}
|
||||
],
|
||||
"caseRefs": [
|
||||
"ci-queue-wait/terminal-success",
|
||||
"ci-queue-wait/no-status-required",
|
||||
"ci-queue-wait/unknown-state",
|
||||
"ci-queue-wait/malformed-status",
|
||||
"ci-queue-wait/terminal-failure",
|
||||
"ci-queue-wait/push-defaults-to-main",
|
||||
"ci-queue-wait/unknown-option"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -158,7 +196,8 @@
|
||||
"currentText": "The pre-commit hook propagates lint-staged failure.",
|
||||
"claimType": "workflow",
|
||||
"source": ".husky/pre-commit",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["hook-pre-commit/clean-staged-input", "hook-pre-commit/lint-staged-failure"]
|
||||
},
|
||||
{
|
||||
"id": "HOOK-PRE-PUSH",
|
||||
@@ -166,7 +205,8 @@
|
||||
"currentText": "The pre-push hook propagates each required gate failure.",
|
||||
"claimType": "workflow",
|
||||
"source": ".husky/pre-push",
|
||||
"meaningChanges": []
|
||||
"meaningChanges": [],
|
||||
"caseRefs": ["hook-pre-push/all-subgates-succeed", "hook-pre-push/typecheck-failure"]
|
||||
},
|
||||
{
|
||||
"id": "GATE-SOURCE-DEPLOYMENT",
|
||||
@@ -183,41 +223,50 @@
|
||||
"task": "RM-02",
|
||||
"date": "2026-08-01"
|
||||
}
|
||||
]
|
||||
],
|
||||
"caseRefs": ["ci-queue-wait/terminal-success", "ci-queue-wait/unknown-option"]
|
||||
}
|
||||
],
|
||||
"proseClaims": [
|
||||
{
|
||||
"id": "OBSERVE-PROPERTY",
|
||||
"criterionId": "RM02-CHECK-RIGHT"
|
||||
"criterionId": "RM02-CHECK-RIGHT",
|
||||
"caseRef": "checkout-preflight/criterion-misbinding"
|
||||
},
|
||||
{
|
||||
"id": "PREREGISTRATION-BOUNDARY",
|
||||
"criterionId": "RM02-SET-COVERS"
|
||||
"criterionId": "RM02-SET-COVERS",
|
||||
"caseRef": "checkout-preflight/criterion-misbinding"
|
||||
},
|
||||
{
|
||||
"id": "ARTIFACT-INTEGRITY-BOUNDARY",
|
||||
"criterionId": "GATE-SOURCE-DEPLOYMENT"
|
||||
"criterionId": "GATE-SOURCE-DEPLOYMENT",
|
||||
"caseRef": "ci-queue-wait/unknown-option"
|
||||
},
|
||||
{
|
||||
"id": "IMPOSSIBLE-LAYER-BOUNDARY",
|
||||
"criterionId": "CHECKOUT-PREFLIGHT"
|
||||
"criterionId": "CHECKOUT-PREFLIGHT",
|
||||
"caseRef": "checkout-preflight/stale-build-lock"
|
||||
},
|
||||
{
|
||||
"id": "PROSE-IS-A-CLAIM",
|
||||
"criterionId": "RM02-PROSE-CONTROL"
|
||||
"criterionId": "RM02-PROSE-CONTROL",
|
||||
"caseRef": "checkout-preflight/prose-claim-misbinding"
|
||||
},
|
||||
{
|
||||
"id": "GENERATED-STATE-SCOPE",
|
||||
"criterionId": "CHECKOUT-PREFLIGHT"
|
||||
"criterionId": "CHECKOUT-PREFLIGHT",
|
||||
"caseRef": "checkout-preflight/stale-build-lock"
|
||||
},
|
||||
{
|
||||
"id": "EXECUTION-TRUST-BOUNDARY",
|
||||
"criterionId": "RM02-CURRENT-TREE-BOUNDARY"
|
||||
"criterionId": "RM02-CURRENT-TREE-BOUNDARY",
|
||||
"caseRef": "checkout-preflight/privileged-pr-gate"
|
||||
},
|
||||
{
|
||||
"id": "CRITERION-RESTATEMENT",
|
||||
"criterionId": "RM02-MEANING-PROVENANCE"
|
||||
"criterionId": "RM02-MEANING-PROVENANCE",
|
||||
"caseRef": "checkout-preflight/missing-meaning-provenance"
|
||||
}
|
||||
],
|
||||
"compatibilityScenarios": [
|
||||
@@ -296,7 +345,7 @@
|
||||
},
|
||||
{
|
||||
"id": "type-error",
|
||||
"criterionIds": ["QUALITY-TYPECHECK", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
|
||||
"criterionIds": ["QUALITY-TYPECHECK", "RM02-CHECK-RIGHT"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 2,
|
||||
@@ -354,7 +403,7 @@
|
||||
},
|
||||
{
|
||||
"id": "invalid-syntax",
|
||||
"criterionIds": ["QUALITY-LINT", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
|
||||
"criterionIds": ["QUALITY-LINT", "RM02-CHECK-RIGHT"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
@@ -412,7 +461,7 @@
|
||||
},
|
||||
{
|
||||
"id": "unformatted-json",
|
||||
"criterionIds": ["QUALITY-FORMAT", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
|
||||
"criterionIds": ["QUALITY-FORMAT", "RM02-CHECK-RIGHT"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
@@ -469,13 +518,7 @@
|
||||
},
|
||||
{
|
||||
"id": "stale-build-lock",
|
||||
"criterionIds": [
|
||||
"CHECKOUT-PREFLIGHT",
|
||||
"RM02-CHECK-RIGHT",
|
||||
"RM02-SET-COVERS",
|
||||
"RM02-MEANING-PROVENANCE",
|
||||
"RM02-PROSE-CONTROL"
|
||||
],
|
||||
"criterionIds": ["CHECKOUT-PREFLIGHT", "RM02-CHECK-RIGHT"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 43,
|
||||
@@ -519,6 +562,122 @@
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "criterion-misbinding",
|
||||
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
|
||||
"mustFail": true,
|
||||
"invocation": [
|
||||
"node",
|
||||
"scripts/gate-verify.mjs",
|
||||
"--root",
|
||||
".",
|
||||
"--manifest",
|
||||
"gates/gates.manifest.json",
|
||||
"--structure-only"
|
||||
],
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound"
|
||||
},
|
||||
"actual": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound"
|
||||
},
|
||||
"reasonPattern": "RM02-SET-COVERS: declared exercising case checkout-preflight/criterion-misbinding is not bound",
|
||||
"fixture": {
|
||||
"copyPaths": [
|
||||
"gates/gates.manifest.json",
|
||||
"scripts/gate-verify.mjs",
|
||||
"scripts/gate-history.mjs"
|
||||
],
|
||||
"replaceFiles": [
|
||||
{
|
||||
"path": "gates/gates.manifest.json",
|
||||
"find": "\"criterionIds\": [\"QUALITY-TYPECHECK\", \"RM02-CHECK-RIGHT\"],",
|
||||
"replace": "\"criterionIds\": [\"QUALITY-TYPECHECK\", \"RM02-CHECK-RIGHT\", \"RM02-SET-COVERS\"],"
|
||||
},
|
||||
{
|
||||
"path": "gates/gates.manifest.json",
|
||||
"find": "\"criterionIds\": [\"RM02-CHECK-RIGHT\", \"RM02-SET-COVERS\"],",
|
||||
"replace": "\"criterionIds\": [\"RM02-CHECK-RIGHT\"],"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "missing-meaning-provenance",
|
||||
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-MEANING-PROVENANCE"],
|
||||
"mustFail": true,
|
||||
"invocation": [
|
||||
"node",
|
||||
"scripts/gate-verify.mjs",
|
||||
"--root",
|
||||
".",
|
||||
"--manifest",
|
||||
"gates/gates.manifest.json",
|
||||
"--structure-only"
|
||||
],
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance"
|
||||
},
|
||||
"actual": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance"
|
||||
},
|
||||
"reasonPattern": "RM02-MEANING-PROVENANCE: missing meaning-change provenance",
|
||||
"fixture": {
|
||||
"copyPaths": [
|
||||
"gates/gates.manifest.json",
|
||||
"scripts/gate-verify.mjs",
|
||||
"scripts/gate-history.mjs"
|
||||
],
|
||||
"replaceFiles": [
|
||||
{
|
||||
"path": "gates/gates.manifest.json",
|
||||
"find": "\"currentText\": \"A restated criterion retains original text, restatement, and reason.\",",
|
||||
"replace": "\"currentText\": \"A restated criterion changed without provenance\","
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "prose-claim-misbinding",
|
||||
"criterionIds": ["RM02-CHECK-RIGHT", "RM02-PROSE-CONTROL"],
|
||||
"mustFail": true,
|
||||
"invocation": [
|
||||
"node",
|
||||
"scripts/gate-verify.mjs",
|
||||
"--root",
|
||||
".",
|
||||
"--manifest",
|
||||
"gates/gates.manifest.json",
|
||||
"--structure-only"
|
||||
],
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL"
|
||||
},
|
||||
"actual": {
|
||||
"exitCode": 1,
|
||||
"outputPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL"
|
||||
},
|
||||
"reasonPattern": "GATE-CLAIM:PROSE-IS-A-CLAIM exercising case quality-typecheck/type-error does not exercise criterion RM02-PROSE-CONTROL",
|
||||
"fixture": {
|
||||
"copyPaths": [
|
||||
"gates/gates.manifest.json",
|
||||
"scripts/gate-verify.mjs",
|
||||
"scripts/gate-history.mjs"
|
||||
],
|
||||
"replaceFiles": [
|
||||
{
|
||||
"path": "gates/gates.manifest.json",
|
||||
"find": "\"caseRef\": \"checkout-preflight/prose-claim-misbinding\"",
|
||||
"replace": "\"caseRef\": \"quality-typecheck/type-error\""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -607,7 +766,7 @@
|
||||
},
|
||||
{
|
||||
"id": "no-status-required",
|
||||
"criterionIds": ["QUEUE-GUARD", "RM02-CHECK-RIGHT", "RM02-SET-COVERS"],
|
||||
"criterionIds": ["QUEUE-GUARD", "RM02-CHECK-RIGHT"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
@@ -669,7 +828,7 @@
|
||||
},
|
||||
{
|
||||
"id": "unknown-state",
|
||||
"criterionIds": ["QUEUE-GUARD", "RM02-SET-COVERS"],
|
||||
"criterionIds": ["QUEUE-GUARD"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 1,
|
||||
@@ -946,12 +1105,7 @@
|
||||
},
|
||||
{
|
||||
"id": "lint-staged-failure",
|
||||
"criterionIds": [
|
||||
"HOOK-PRE-COMMIT",
|
||||
"RM02-CHECK-RIGHT",
|
||||
"RM02-SET-COVERS",
|
||||
"RM02-MODELED-CONSISTENCY"
|
||||
],
|
||||
"criterionIds": ["HOOK-PRE-COMMIT", "RM02-CHECK-RIGHT", "RM02-MODELED-CONSISTENCY"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 19,
|
||||
@@ -1027,12 +1181,7 @@
|
||||
},
|
||||
{
|
||||
"id": "typecheck-failure",
|
||||
"criterionIds": [
|
||||
"HOOK-PRE-PUSH",
|
||||
"RM02-CHECK-RIGHT",
|
||||
"RM02-SET-COVERS",
|
||||
"RM02-MODELED-CONSISTENCY"
|
||||
],
|
||||
"criterionIds": ["HOOK-PRE-PUSH", "RM02-CHECK-RIGHT", "RM02-MODELED-CONSISTENCY"],
|
||||
"mustFail": true,
|
||||
"required": {
|
||||
"exitCode": 19,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { existsSync } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { createHash, randomUUID } from 'node:crypto';
|
||||
import { access, lstat, mkdir, mkdtemp, readFile, readdir, readlink, rm } from 'node:fs/promises';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
@@ -40,7 +40,9 @@ async function snapshotAuthoritativeTree(root) {
|
||||
} else if (stats.isSymbolicLink()) {
|
||||
snapshot.set(relative, `symlink:${stats.mode}:${await readlink(absolute)}`);
|
||||
} else if (stats.isFile()) {
|
||||
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
|
||||
const digest = createHash('sha256')
|
||||
.update(await readFile(absolute))
|
||||
.digest('hex');
|
||||
snapshot.set(relative, `file:${stats.mode}:${digest}`);
|
||||
}
|
||||
}
|
||||
@@ -59,7 +61,9 @@ async function authoritativeTreeChanges(root, snapshot) {
|
||||
if (stats.isSymbolicLink()) {
|
||||
actual = `symlink:${stats.mode}:${await readlink(absolute)}`;
|
||||
} else if (stats.isFile()) {
|
||||
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
|
||||
const digest = createHash('sha256')
|
||||
.update(await readFile(absolute))
|
||||
.digest('hex');
|
||||
actual = `file:${stats.mode}:${digest}`;
|
||||
} else {
|
||||
actual = `other:${stats.mode}`;
|
||||
@@ -108,8 +112,24 @@ function bubblewrap(root, command, args, { storePath, timeout = 300_000 } = {})
|
||||
);
|
||||
if (storePath) sandboxArgs.push('--setenv', 'NPM_CONFIG_STORE_DIR', '/pnpm-store');
|
||||
if (existsSync(corepackHome)) sandboxArgs.push('--setenv', 'COREPACK_HOME', '/corepack');
|
||||
sandboxArgs.push(command, ...args);
|
||||
return spawnSync('bwrap', sandboxArgs, { encoding: 'utf8', timeout });
|
||||
const enteredMarker = `__MOSAIC_BWRAP_ENTERED_${randomUUID()}__`;
|
||||
sandboxArgs.push(
|
||||
'/bin/sh',
|
||||
'-c',
|
||||
'printf "%s\\n" "$1"; shift; exec "$@"',
|
||||
'mosaic-bwrap-entry',
|
||||
enteredMarker,
|
||||
command,
|
||||
...args,
|
||||
);
|
||||
const result = spawnSync('bwrap', sandboxArgs, { encoding: 'utf8', timeout });
|
||||
const sandboxEntered = result.stdout?.includes(enteredMarker) === true;
|
||||
return {
|
||||
...result,
|
||||
stdout: (result.stdout ?? '').replace(`${enteredMarker}\n`, ''),
|
||||
sandboxLauncher: 'bwrap',
|
||||
sandboxEntered,
|
||||
};
|
||||
}
|
||||
|
||||
export async function replayCommit(root, commit) {
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
const fixtureRoot = path.join(process.cwd(), '.mosaic-test-work', `gate-history-${process.pid}`);
|
||||
|
||||
function sandboxUnavailable(result) {
|
||||
if (result.sandboxLauncher !== 'bwrap' || result.sandboxEntered === true) return false;
|
||||
const detail = `${result.stdout ?? ''}${result.stderr ?? ''}${result.error?.message ?? ''}`;
|
||||
const bubblewrapSpawnDenied =
|
||||
['EPERM', 'EACCES', 'ENOENT'].includes(result.error?.code) &&
|
||||
@@ -56,6 +57,8 @@ test('sandbox refusal classification requires Bubblewrap provenance', () => {
|
||||
sandboxUnavailable({
|
||||
status: null,
|
||||
error: { code, message: `spawnSync bwrap ${code}` },
|
||||
sandboxLauncher: 'bwrap',
|
||||
sandboxEntered: false,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
@@ -71,6 +74,31 @@ test('sandbox refusal classification requires Bubblewrap provenance', () => {
|
||||
sandboxUnavailable({ status: 1, stderr: 'historical verifier said bwrap ENOENT' }),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
sandboxUnavailable({
|
||||
status: 1,
|
||||
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
|
||||
sandboxLauncher: 'bwrap',
|
||||
sandboxEntered: false,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
sandboxUnavailable({
|
||||
status: 1,
|
||||
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
sandboxUnavailable({
|
||||
status: 1,
|
||||
stderr: 'bwrap: Creating new namespace failed: Operation not permitted',
|
||||
sandboxLauncher: 'bwrap',
|
||||
sandboxEntered: true,
|
||||
}),
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test('prospective history reads each commit own manifest rather than the current tree', async () => {
|
||||
@@ -261,13 +289,13 @@ test('PR verification states the RM-60 boundary without executing an intermediat
|
||||
});
|
||||
assert.deepEqual(result.failures, []);
|
||||
assert.ok(
|
||||
result.observations.some(
|
||||
(observation) => /DOES:.*current tree.*DOES NOT:.*isolated.*RM-60.*RM-59/i.test(observation),
|
||||
result.observations.some((observation) =>
|
||||
/DOES:.*current tree.*DOES NOT:.*isolated.*RM-60.*RM-59/i.test(observation),
|
||||
),
|
||||
);
|
||||
assert.ok(
|
||||
result.observations.some(
|
||||
(observation) => /INTERMEDIATE REPLAY DEFERRED.*RM-60.*no success is inferred/i.test(observation),
|
||||
result.observations.some((observation) =>
|
||||
/INTERMEDIATE REPLAY DEFERRED.*RM-60.*no success is inferred/i.test(observation),
|
||||
),
|
||||
);
|
||||
assert.ok(result.observations.every((observation) => !/INTERMEDIATE INERT/.test(observation)));
|
||||
|
||||
+77
-11
@@ -29,12 +29,14 @@ function parseArgs(argv) {
|
||||
root: process.cwd(),
|
||||
manifest: 'gates/gates.manifest.json',
|
||||
skipHistory: false,
|
||||
structureOnly: false,
|
||||
};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const value = argv[index];
|
||||
if (value === '--root') options.root = path.resolve(argv[++index]);
|
||||
else if (value === '--manifest') options.manifest = argv[++index];
|
||||
else if (value === '--skip-history') options.skipHistory = true;
|
||||
else if (value === '--structure-only') options.structureOnly = true;
|
||||
else throw new Error(`unknown option: ${value}`);
|
||||
}
|
||||
return options;
|
||||
@@ -307,7 +309,39 @@ function validateClosedSchema(manifest, failures) {
|
||||
failures,
|
||||
);
|
||||
rejectDuplicateIds(manifest.criteria, 'criterion', failures);
|
||||
for (const criterion of manifest.criteria ?? []) {
|
||||
rejectUnknownKeys(
|
||||
criterion,
|
||||
new Set([
|
||||
'id',
|
||||
'originalText',
|
||||
'currentText',
|
||||
'claimType',
|
||||
'source',
|
||||
'meaningChanges',
|
||||
'caseRefs',
|
||||
]),
|
||||
`criterion ${criterion.id}`,
|
||||
failures,
|
||||
);
|
||||
if (!Array.isArray(criterion.caseRefs) || criterion.caseRefs.length === 0) {
|
||||
failures.push(`${criterion.id}: no declared exercising cases`);
|
||||
} else if (new Set(criterion.caseRefs).size !== criterion.caseRefs.length) {
|
||||
failures.push(`${criterion.id}: duplicate declared exercising case`);
|
||||
}
|
||||
}
|
||||
rejectDuplicateIds(manifest.proseClaims, 'prose claim', failures);
|
||||
for (const claim of manifest.proseClaims ?? []) {
|
||||
rejectUnknownKeys(
|
||||
claim,
|
||||
new Set(['id', 'criterionId', 'caseRef']),
|
||||
`prose claim ${claim.id}`,
|
||||
failures,
|
||||
);
|
||||
if (typeof claim.caseRef !== 'string' || claim.caseRef.length === 0) {
|
||||
failures.push(`GATE-CLAIM:${claim.id} has no declared exercising case`);
|
||||
}
|
||||
}
|
||||
rejectDuplicateIds(manifest.compatibilityScenarios, 'compatibility scenario', failures);
|
||||
for (const scenario of manifest.compatibilityScenarios ?? []) {
|
||||
rejectUnknownKeys(
|
||||
@@ -390,7 +424,9 @@ function validateClosedSchema(manifest, failures) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: case invocation must be non-empty`);
|
||||
}
|
||||
if (gateCase.mustFail === true && !gateCase.reasonPattern?.trim()) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: must-fail case requires a non-empty reasonPattern`);
|
||||
failures.push(
|
||||
`${gate.id}/${gateCase.id}: must-fail case requires a non-empty reasonPattern`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -401,6 +437,7 @@ function validateStructure(manifest, failures) {
|
||||
const criteria = new Map((manifest.criteria ?? []).map((criterion) => [criterion.id, criterion]));
|
||||
const boundCriteria = new Set();
|
||||
const negativeBoundCriteria = new Set();
|
||||
const observedCaseRefs = new Map();
|
||||
|
||||
for (const gate of manifest.gates ?? []) {
|
||||
const negativeCases = (gate.cases ?? []).filter((gateCase) => gateCase.mustFail === true);
|
||||
@@ -413,12 +450,12 @@ function validateStructure(manifest, failures) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: unknown criterion ${criterionId}`);
|
||||
}
|
||||
boundCriteria.add(criterionId);
|
||||
const caseRef = `${gate.id}/${gateCase.id}`;
|
||||
if (!observedCaseRefs.has(criterionId)) observedCaseRefs.set(criterionId, new Set());
|
||||
observedCaseRefs.get(criterionId).add(caseRef);
|
||||
if (gateCase.mustFail === true) negativeBoundCriteria.add(criterionId);
|
||||
}
|
||||
if (
|
||||
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
|
||||
!gateCase.defect?.owner
|
||||
) {
|
||||
if (!structuredValuesEqual(gateCase.required, gateCase.actual) && !gateCase.defect?.owner) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: behavior delta requires a tracked owner`);
|
||||
}
|
||||
}
|
||||
@@ -427,6 +464,18 @@ function validateStructure(manifest, failures) {
|
||||
for (const claim of manifest.proseClaims ?? []) {
|
||||
if (!criteria.has(claim.criterionId)) {
|
||||
failures.push(`GATE-CLAIM:${claim.id} references unknown criterion ${claim.criterionId}`);
|
||||
continue;
|
||||
}
|
||||
const found = findGateCase(manifest, claim.caseRef ?? '');
|
||||
if (!found) {
|
||||
failures.push(`GATE-CLAIM:${claim.id} references missing exercising case ${claim.caseRef}`);
|
||||
} else if (
|
||||
found.gateCase.mustFail !== true ||
|
||||
!found.gateCase.criterionIds?.includes(claim.criterionId)
|
||||
) {
|
||||
failures.push(
|
||||
`GATE-CLAIM:${claim.id} exercising case ${claim.caseRef} does not exercise criterion ${claim.criterionId}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -435,6 +484,20 @@ function validateStructure(manifest, failures) {
|
||||
else if (!negativeBoundCriteria.has(criterion.id)) {
|
||||
failures.push(`${criterion.id}: no must-fail case exercises this criterion`);
|
||||
}
|
||||
const declaredRefs = new Set(criterion.caseRefs ?? []);
|
||||
const actualRefs = observedCaseRefs.get(criterion.id) ?? new Set();
|
||||
for (const caseRef of declaredRefs) {
|
||||
const found = findGateCase(manifest, caseRef);
|
||||
if (!found) failures.push(`${criterion.id}: declared exercising case ${caseRef} is missing`);
|
||||
else if (!actualRefs.has(caseRef)) {
|
||||
failures.push(`${criterion.id}: declared exercising case ${caseRef} is not bound`);
|
||||
}
|
||||
}
|
||||
for (const caseRef of actualRefs) {
|
||||
if (!declaredRefs.has(caseRef)) {
|
||||
failures.push(`${criterion.id}: bound to undeclared exercising case ${caseRef}`);
|
||||
}
|
||||
}
|
||||
if (
|
||||
criterion.originalText !== criterion.currentText &&
|
||||
(!Array.isArray(criterion.meaningChanges) || criterion.meaningChanges.length === 0)
|
||||
@@ -634,7 +697,9 @@ async function runCompatibilityScenario(root, manifest, scenario, failures, obse
|
||||
for (const caseRef of scenario.caseRefs ?? []) {
|
||||
const found = findGateCase(manifest, caseRef);
|
||||
if (!found) {
|
||||
failures.push(`${scenario.id}: compatibility construction references missing case ${caseRef}`);
|
||||
failures.push(
|
||||
`${scenario.id}: compatibility construction references missing case ${caseRef}`,
|
||||
);
|
||||
} else {
|
||||
referenced.push({ caseRef, ...found });
|
||||
}
|
||||
@@ -671,7 +736,10 @@ async function runCompatibilityScenario(root, manifest, scenario, failures, obse
|
||||
}
|
||||
await applyFixture(sandbox, fixture);
|
||||
}
|
||||
for (const source of [...referenced.map(({ gateCase }) => gateCase.environment), scenario.environment]) {
|
||||
for (const source of [
|
||||
...referenced.map(({ gateCase }) => gateCase.environment),
|
||||
scenario.environment,
|
||||
]) {
|
||||
for (const [key, value] of Object.entries(source ?? {})) {
|
||||
if (environment[key] !== undefined && environment[key] !== value) {
|
||||
failures.push(`${scenario.id}: incompatible environment values for ${key}`);
|
||||
@@ -703,6 +771,7 @@ export async function verifyRegistry(options) {
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
|
||||
validateStructure(manifest, failures);
|
||||
if (options.structureOnly) return { failures, manifest, observations };
|
||||
await validateClaims(options.root, manifest, failures);
|
||||
await validateDiscovery(options.root, manifest, failures);
|
||||
|
||||
@@ -719,10 +788,7 @@ export async function verifyRegistry(options) {
|
||||
if (gateCase.reasonPattern && !new RegExp(gateCase.reasonPattern, 'm').test(combined)) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: did not fail for its stated reason`);
|
||||
}
|
||||
if (
|
||||
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
|
||||
gateCase.defect?.owner
|
||||
) {
|
||||
if (!structuredValuesEqual(gateCase.required, gateCase.actual) && gateCase.defect?.owner) {
|
||||
observations.push(
|
||||
`DEFECT (owner: ${gateCase.defect.owner}) ${gate.id}/${gateCase.id}: required ${JSON.stringify(gateCase.required)}, actual ${JSON.stringify(gateCase.actual)}`,
|
||||
);
|
||||
|
||||
@@ -30,6 +30,7 @@ function baseManifest() {
|
||||
claimType: 'integrity',
|
||||
source: 'fixture',
|
||||
meaningChanges: [],
|
||||
caseRefs: ['meta-fixture/rejects-bad-input'],
|
||||
},
|
||||
],
|
||||
compatibilityScenarios: [],
|
||||
@@ -72,10 +73,18 @@ async function writeManifest(root, manifest) {
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), `${JSON.stringify(manifest)}\n`);
|
||||
}
|
||||
|
||||
function verify(root) {
|
||||
function verify(root, extraArgs = []) {
|
||||
return spawnSync(
|
||||
process.execPath,
|
||||
[verifier, '--root', root, '--manifest', 'gates/gates.manifest.json', '--skip-history'],
|
||||
[
|
||||
verifier,
|
||||
'--root',
|
||||
root,
|
||||
'--manifest',
|
||||
'gates/gates.manifest.json',
|
||||
'--skip-history',
|
||||
...extraArgs,
|
||||
],
|
||||
{ cwd: root, encoding: 'utf8', env: { ...process.env, HOME: os.homedir() } },
|
||||
);
|
||||
}
|
||||
@@ -260,6 +269,61 @@ test('a required-versus-actual delta without a tracked owner is rejected', async
|
||||
assert.match(output(result), /meta-fixture.*delta.*tracked owner/i);
|
||||
});
|
||||
|
||||
test('moving criterion bindings to unrelated cases is rejected', async () => {
|
||||
const root = await fixture('semantic-misbinding');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.criteria.push({
|
||||
id: 'META-CRIT-2',
|
||||
originalText: 'The fixture reports the second rejection reason.',
|
||||
currentText: 'The fixture reports the second rejection reason.',
|
||||
claimType: 'integrity',
|
||||
source: 'fixture',
|
||||
meaningChanges: [],
|
||||
caseRefs: ['meta-fixture/rejects-second-input'],
|
||||
});
|
||||
manifest.gates[0].cases.push({
|
||||
...manifest.gates[0].cases[0],
|
||||
id: 'rejects-second-input',
|
||||
criterionIds: ['META-CRIT-1'],
|
||||
});
|
||||
manifest.gates[0].cases[0].criterionIds = ['META-CRIT-2'];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /META-CRIT-1.*rejects-bad-input.*not bound/i);
|
||||
assert.match(output(result), /META-CRIT-2.*rejects-second-input.*not bound/i);
|
||||
});
|
||||
|
||||
test('moving meaning and prose criteria to an unrelated type error is rejected', async () => {
|
||||
const root = await fixture('real-manifest-misbinding');
|
||||
const manifest = JSON.parse(
|
||||
await readFile(path.join(process.cwd(), 'gates', 'gates.manifest.json'), 'utf8'),
|
||||
);
|
||||
for (const gate of manifest.gates) {
|
||||
for (const gateCase of gate.cases) {
|
||||
gateCase.criterionIds = gateCase.criterionIds.filter(
|
||||
(id) => !['RM02-MEANING-PROVENANCE', 'RM02-PROSE-CONTROL'].includes(id),
|
||||
);
|
||||
}
|
||||
}
|
||||
const typeError = manifest.gates
|
||||
.find((gate) => gate.id === 'quality-typecheck')
|
||||
.cases.find((gateCase) => gateCase.id === 'type-error');
|
||||
typeError.criterionIds.push('RM02-MEANING-PROVENANCE', 'RM02-PROSE-CONTROL');
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root, ['--structure-only']);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /RM02-MEANING-PROVENANCE.*missing-meaning-provenance.*not bound/i);
|
||||
assert.match(output(result), /RM02-PROSE-CONTROL.*prose-claim-misbinding.*not bound/i);
|
||||
assert.match(
|
||||
output(result),
|
||||
/RM02-(?:MEANING-PROVENANCE|PROSE-CONTROL).*undeclared exercising case quality-typecheck\/type-error/i,
|
||||
);
|
||||
});
|
||||
|
||||
test('a criterion with no bound case is rejected', async () => {
|
||||
const root = await fixture('unbound-criterion');
|
||||
await writeGate(root);
|
||||
@@ -335,6 +399,7 @@ test('compatibility scenarios execute referenced conditions as one construction'
|
||||
reasonPattern: 'SECOND_REASON',
|
||||
environment: { SECOND_REASON: 'SECOND_REASON' },
|
||||
});
|
||||
manifest.criteria[0].caseRefs.push('meta-fixture/second-condition');
|
||||
manifest.gates[0].cases[0].fixture = {
|
||||
writeFiles: [{ path: 'conditions/first', content: 'present\n' }],
|
||||
};
|
||||
@@ -427,7 +492,10 @@ test('deployment drift meta-control fails if the shared comparator is made inert
|
||||
const root = await fixture('deployment-comparator-inert');
|
||||
await writeGate(root);
|
||||
await mkdir(path.join(root, 'deployed'), { recursive: true });
|
||||
await copyFile(path.join(root, 'gates', 'meta-fixture.sh'), path.join(root, 'deployed', 'meta-fixture.sh'));
|
||||
await copyFile(
|
||||
path.join(root, 'gates', 'meta-fixture.sh'),
|
||||
path.join(root, 'deployed', 'meta-fixture.sh'),
|
||||
);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].deployment = { kind: 'file', path: 'deployed/meta-fixture.sh' };
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
Reference in New Issue
Block a user