forked from mosaicstack/stack
feat(quality): add anti-inert gate registry
This commit is contained in:
+14
-3
@@ -68,15 +68,26 @@ steps:
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-durable-snapshot.sh
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh
|
||||
|
||||
# Anti-inert-gate registry. Deliberately unconditional: no path filter and no
|
||||
# step-level `when`, because a gate can be disabled by changes outside its own path.
|
||||
gate-verify:
|
||||
image: *node_image
|
||||
commands:
|
||||
- *enable_pnpm
|
||||
- apk add --no-cache bubblewrap
|
||||
- pnpm gate:verify
|
||||
depends_on:
|
||||
- install
|
||||
- sanitization
|
||||
- upgrade-guard
|
||||
|
||||
typecheck:
|
||||
image: *node_image
|
||||
commands:
|
||||
- *enable_pnpm
|
||||
- pnpm typecheck
|
||||
depends_on:
|
||||
- install
|
||||
- sanitization
|
||||
- upgrade-guard
|
||||
- gate-verify
|
||||
|
||||
# lint, format, and test are independent — run in parallel after typecheck
|
||||
lint:
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# Administrator Guide
|
||||
|
||||
- [Gate registry operations](quality-gate-registry.md)
|
||||
@@ -0,0 +1,33 @@
|
||||
# Gate Registry Operations
|
||||
|
||||
## Routine verification
|
||||
|
||||
Run `pnpm gate:verify` from a dependency-installed checkout. Exit zero means registry observations matched their declared `actual` values; it does **not** assert required-behavior conformance while deltas remain. Open `DEFECT` records are checked descriptions of current behavior with tracked owners, never successful gate outcomes.
|
||||
|
||||
Investigate any of these immediately:
|
||||
|
||||
- `GATE VERIFY FAILED` — registry structure, observed behavior, provenance, claim binding, source/deployment identity, or negative-control detection changed.
|
||||
- `unregistered gate` — an executable appeared under a declared gate root without a registry entry.
|
||||
- `no negative control` — a gate has no must-fail case.
|
||||
- `DEPLOYED IDENTITY UNAVAILABLE` — the runner cannot reach the installed enforcing copy. The pinned observation is checked, but live equality is not asserted.
|
||||
- `PROVIDER EVIDENCE ... ABSENT` — retained external history was unavailable; do not infer merge-time success.
|
||||
|
||||
## Updating a gate
|
||||
|
||||
1. Add or change the criterion and exact case.
|
||||
2. Observe the case fail for its own stated reason.
|
||||
3. Declare an exact inerting mutation and observe the verifier detect it.
|
||||
4. If required and actual behavior differ, add a tracked remediation owner and justification.
|
||||
5. If meaning changed, append provenance; never replace the original silently.
|
||||
6. For an installed counterpart, verify live byte identity and update the observed digest only from measured evidence.
|
||||
7. Run focused verifier tests, `pnpm gate:verify`, and the repository baseline gates.
|
||||
|
||||
Do not add an ownerless exception or describe an open delta as pass/green/OK.
|
||||
|
||||
## CI behavior
|
||||
|
||||
Woodpecker runs `gate-verify` on every pull request and protected-main push without path filtering. This is deliberate: changes outside gate files can make a gate inert.
|
||||
|
||||
Provider evidence input is an optional JSON array of normalized pipeline records containing `commit`, unique integer pipeline `number`, pipeline `status`, and a `gate-verify` step status. The highest numbered rerun is authoritative; ambiguous duplicates fail. Its retention window is provider-controlled and is not overstated by this repository.
|
||||
|
||||
Prior commit replay performs a frozen offline install from each commit's own lockfile before running that commit's verifier. Bubblewrap clears the environment, hides operator-home credentials, and disables networking for historical lifecycle and verifier code. A missing cached dependency or unavailable sandbox fails replay; current dependencies are never substituted.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Developer Guide
|
||||
|
||||
- [Gate registry and negative controls](quality-gate-registry.md)
|
||||
@@ -0,0 +1,41 @@
|
||||
# Gate Registry and Negative Controls
|
||||
|
||||
`gates/gates.manifest.json` is the machine-readable registry for the initial RM-02 gate slice. Run:
|
||||
|
||||
```bash
|
||||
pnpm gate:verify
|
||||
```
|
||||
|
||||
## Registered slice
|
||||
|
||||
The registry covers root typecheck, lint, and format checks; RM-01 checkout preflight; the Mosaic CI queue guard; and root Husky pre-commit/pre-push hooks. It does not imply repository-wide coverage. Framework scripts, package-local build/test scripts, templates, and deployment/release scripts remain assigned to RM-54.
|
||||
|
||||
Every gate declares exact invocations, observed and required outcomes, criterion bindings, and a single exact inerting mutation. Every must-fail case requires a non-empty reason diagnostic. The verifier rejects a stale, ambiguous, crashing, or ineffective mutation. Fixture and mutation writes reject path traversal and final-component symlinks. This proves detection of the **declared** inerting mutation, not every possible semantic weakening.
|
||||
|
||||
## Required versus actual
|
||||
|
||||
A case may record different `required` and `actual` outcomes only with a tracked owner. The verifier checks current reality against `actual`, prints each difference as `DEFECT (owner: ...)`, and fails when behavior changes without a matching registry update. A defect is never described as passing, green, or OK.
|
||||
|
||||
The queue guard currently has RM-03-owned deltas. In particular, its stdin/heredoc classifier does not consume piped status JSON, so terminal-success, no-status, and terminal-failure payloads become `unknown`; unknown and malformed states exit zero; push purpose defaults to `main`. RM-02 records these observations and does not edit the guard.
|
||||
|
||||
## Criteria, prose, and compatibility
|
||||
|
||||
Each criterion must bind to a must-fail case. Designated governing prose uses `GATE-CLAIM:<id>` markers; an unbound marker or registered-but-missing marker fails. Orchestrator-owned claims from `TASKS.md` are bound through `docs/remediation/GATE-CLAIMS.md`, which records source headings and anchored text without changing task tracking. Marker completeness still requires RM-54 review because arbitrary English claims cannot be inferred safely.
|
||||
|
||||
Compatibility checks detect direct contradictions in declared finite constructions. The verifier combines referenced case fixtures and environments in one isolated tree, rejects conflicting fixture/environment values, executes the construction's exact invocation, and checks its exact outcome. They do not prove semantic consistency of arbitrary natural language.
|
||||
|
||||
Restatements preserve original text, current text, reason, finding/task, and date.
|
||||
|
||||
## Source and deployed identity
|
||||
|
||||
A gate with an external installed counterpart declares it explicitly. When the installed queue guard is reachable, its bytes must equal repository source and an internal drift control is observed red. In CI the operator-home installation may be outside the container; the verifier checks the pinned observed source digest, reports `DEPLOYED IDENTITY UNAVAILABLE (owner: RM-04)`, and does not infer live equality.
|
||||
|
||||
## Commit and provider boundary
|
||||
|
||||
The current checkout is evaluated directly. On feature branches and main, each prior prospective commit is archived from Git, receives a frozen offline install from that commit's lockfile, and runs that commit's own verifier and manifest. Missing cached dependencies or an unrunnable historical verifier fail loudly rather than borrowing current-tree dependencies.
|
||||
|
||||
Historical install scripts and verifiers execute inside Bubblewrap with network, PID, IPC, and UTS namespaces isolated; a cleared/allowlisted environment; an isolated home; a writable replay tree; read-only system files; and a read-only pnpm store. Current CI secrets, sibling runner processes, and the operator home are not visible inside that boundary. Because lifecycle scripts are required for faithful installs, the verifier snapshots every archived file before install and fails if any authoritative file changes, disappears, or changes type/mode before replay. Replay fails when this sandbox or integrity check cannot be established.
|
||||
|
||||
Retained provider evidence can assert terminal-success for prior commits when supplied through `GATE_PROVIDER_EVIDENCE_FILE`. Each normalized record contains `commit`, unique integer pipeline `number`, pipeline `status`, and step statuses; the highest-numbered rerun is authoritative. Ambiguous duplicates fail. Absent, expired, or currently-running evidence is reported explicitly and never inferred as success.
|
||||
|
||||
Repository replay proves tree reproducibility under the selected commit's locked dependency graph. It does not prove that CI blocked a merge at the time or resist an actor who can rewrite the verifier, registry, and gate consistently. RM-25/RM-59 own that external authority and trust anchor.
|
||||
+43
@@ -14,6 +14,49 @@
|
||||
|
||||
---
|
||||
|
||||
## RM-02 Gate Registry and Negative-Control Verifier (#1029)
|
||||
|
||||
### Problem and objective
|
||||
|
||||
Existing deterministic gates can return success without enforcing their stated property. RM-02 introduces a machine-readable registry and an unconditional CI verifier that distinguishes required behavior from observed behavior, proves every registered negative control can detect its own failure reason, and makes source/deployment drift visible.
|
||||
|
||||
### Scope
|
||||
|
||||
**In scope:** root typecheck, lint, and format gates; RM-01 checkout preflight; the Mosaic CI queue guard; root Husky pre-commit and pre-push hooks; criterion bindings; modeled compatibility; meaning-change provenance; security/integrity prose claim markers; source-versus-deployed identity; prospective per-commit tree replay; retained provider CI evidence where available.
|
||||
|
||||
**Out of scope:** fixing the queue guard (RM-03); exhaustive registration of every repository executable (RM-54); semantic proof that arbitrary English criteria are mutually satisfiable (RM-54/RM-55); a same-authority trust anchor for repository-authored evidence (RM-25/RM-59).
|
||||
|
||||
### Normative requirements
|
||||
|
||||
1. `RM02-REQ-01`: The JSON registry SHALL give every gate and criterion a stable ID and SHALL declare exact invocation, input classes, cases, exact observed and required exit codes, reason diagnostics, and criterion bindings.
|
||||
2. `RM02-REQ-02`: Every gate SHALL have at least one observed-red must-fail case. The verifier SHALL reject missing, stale, ambiguous, or ineffective declared inert mutations and SHALL name an externally inerted gate.
|
||||
3. `RM02-REQ-03`: Every acceptance criterion SHALL bind to a case that can fail for that criterion's stated reason. Security/integrity prose claims in the designated governing documents SHALL carry bound `GATE-CLAIM:<id>` markers.
|
||||
4. `RM02-REQ-04`: Declared finite compatibility scenarios SHALL execute together and direct modeled contradictions SHALL fail. This does not claim semantic consistency of arbitrary English.
|
||||
5. `RM02-REQ-05`: Restated criteria SHALL retain original text, current text, reason, finding/task, and dated meaning-change history.
|
||||
6. `RM02-REQ-06`: An observed behavior differing from required behavior SHALL be reported as `DEFECT` with a tracked owner; an ownerless delta SHALL fail verification. Such a gate SHALL never be described as passing, green, or OK.
|
||||
7. `RM02-REQ-07`: Executables under declared gate roots SHALL fail with `unregistered gate` when absent from the registry. The initial coverage boundary SHALL explicitly list exclusions and bind the broader inventory to RM-54.
|
||||
8. `RM02-REQ-08`: Every gate with a deployed counterpart SHALL register source/deployed byte identity and a must-fail drift control. Gates without a deployed counterpart SHALL say so explicitly.
|
||||
9. `RM02-REQ-09`: CI SHALL run `pnpm gate:verify` on every pull request without path filtering and on protected-main pushes.
|
||||
10. `RM02-REQ-10`: Verification SHALL replay prospective first-parent commits from the activation boundary against each commit's own tree. It SHALL distinguish reproducibility replay from retained external provider evidence and SHALL state when provider history is absent, expired, or still running.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
1. `RM02-AC-01`: A healthy current tree returns zero while prominently reporting the queue guard's required-versus-actual `DEFECT (owner: RM-03)` delta.
|
||||
2. `RM02-AC-02`: Externally mutate any registered gate at its declared inerting point so its failure path succeeds; verification returns nonzero and names that gate. The verifier's internal meta-control is observed red before its healthy result is trusted.
|
||||
3. `RM02-AC-03`: An executable added under a declared gate root without an entry returns nonzero and includes `unregistered gate`.
|
||||
4. `RM02-AC-04`: A gate with zero must-fail cases returns nonzero and includes `no negative control`.
|
||||
5. `RM02-AC-05`: Unbound criteria, unbound governing prose markers, ownerless behavior deltas, stale mutations, source/deployed drift, and modeled compatibility conflicts each return nonzero with the responsible stable ID.
|
||||
6. `RM02-AC-06`: CI configuration invokes the verifier unconditionally on every pull request.
|
||||
7. `RM02-AC-07`: Per-commit replay uses the selected commit's manifest and tree rather than current main, while external CI evidence is asserted only for the provider-retained window and never inferred when unavailable.
|
||||
|
||||
### Risks, dependencies, and verification boundary
|
||||
|
||||
- The repository verifier proves declared controls, modeled scenarios, source/deployed equality at execution time, and prospective tree reproducibility. It does **not** defend against an actor able to rewrite the gate, registry, and verifier consistently.
|
||||
- External branch protection and provider CI history supply merge-time evidence where retained. RM-25/RM-59 track the authority/trust anchor outside the worktree.
|
||||
- `ASSUMPTION:` RM-54 is the owner for expanding registration and prose-marker coverage beyond this approved seven-gate slice; rationale: the remediation task graph already assigns the fleet-wide inert-gate audit there.
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Jarvis (v0.2.0) is a self-hosted AI assistant with a Python FastAPI backend and Next.js frontend. It handles chat, projects, tasks, and LLM routing but lacks orchestration depth, agent coordination, shared memory, and remote access. The Mosaic framework (`~/.config/mosaic`) provides agent guides, shell-based orchestration tools, and quality rails — but these are loose scripts, not an integrated platform. The `@mosaicstack/*` packages in mosaic-mono-v0 began consolidating these into TypeScript packages (brain, queue, coord, cli, prdy, quality-rails) but have no UI, no auth, and no agent runtime integration.
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# Documentation Sitemap
|
||||
|
||||
## Gate verification
|
||||
|
||||
- [Developer gate registry guide](DEVELOPER-GUIDE/quality-gate-registry.md) — manifest schema, negative controls, defect deltas, modeled boundaries, and commit/provider evidence.
|
||||
- [Gate registry operations](ADMIN-GUIDE/quality-gate-registry.md) — routine verification, failure interpretation, registry updates, and unconditional CI behavior.
|
||||
- [RM-02 governing claim index](remediation/GATE-CLAIMS.md) — marker bindings for orchestrator-owned remediation claims without modifying task tracking.
|
||||
|
||||
## Compaction refresh lease broker
|
||||
|
||||
- [Internal broker protocol](architecture/lease-broker-protocol.md) — kernel identity, ancestry and generation invariants, framed requests, responses, and persisted cycle bindings.
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# RM-02 Gate Registry Implementation Plan
|
||||
|
||||
> **For Pi:** Use test-driven development and execute each task RED → GREEN → refactor.
|
||||
|
||||
**Goal:** Build a machine-readable seven-gate registry and an unconditional CI verifier that detects inert gates, binds criteria to observed negative controls, records defects honestly, and replays prospective commits against their own trees.
|
||||
|
||||
**Architecture:** A dependency-free Node CLI reads `gates/gates.manifest.json`, validates its closed schema and references, then runs typed cases in isolated main-disk fixtures. Gate-specific fixture setup remains declarative; exact invocations and exact observed/required exits stay in JSON. A separate history module selects each first-parent commit's own tree/manifest and reports retained external CI evidence without inferring missing evidence.
|
||||
|
||||
**Tech Stack:** Node.js ESM, `node:test`, JSON, shell gates, pnpm, Woodpecker CI.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Meta-negative-control kernel
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `scripts/gate-verify.test.mjs`
|
||||
- Create: `scripts/gate-verify.mjs`
|
||||
|
||||
1. Write a black-box fixture whose gate failure branch has already been changed to success.
|
||||
2. Run `node --test scripts/gate-verify.test.mjs`; require failure because the absent verifier does not name the inert gate.
|
||||
3. Implement manifest loading, exact process execution, and named mismatch reporting only.
|
||||
4. Re-run and require the external inert mutation to produce verifier nonzero while the test passes.
|
||||
5. Add an internally applied declared mutation and require a healthy fixture to report its negative control armed.
|
||||
|
||||
### Task 2: Registry structural clauses
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `scripts/gate-verify.test.mjs`
|
||||
- Modify: `scripts/gate-verify.mjs`
|
||||
|
||||
Add failing tests, one behavior at a time, for: `unregistered gate`; `no negative control`; unbound criterion; unbound `GATE-CLAIM`; ownerless required/actual delta; stale/ambiguous/ineffective mutation; direct modeled conflict; missing meaning-change provenance. Implement the minimum validator after each observed RED.
|
||||
|
||||
### Task 3: Gate fixtures and seven-gate manifest
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `gates/gates.manifest.json`
|
||||
- Create: `gates/fixtures/quality-case.mjs`
|
||||
- Create: `gates/fixtures/preflight-case.mjs`
|
||||
- Create: `gates/fixtures/queue-case.sh`
|
||||
- Create: `gates/fixtures/hook-case.sh`
|
||||
- Modify: `scripts/gate-verify.test.mjs`
|
||||
|
||||
Register typecheck, lint, format, RM-01 preflight, queue guard, pre-commit, and pre-push. For each, first run its known-bad case against an intentionally inert fixture and observe verifier RED; then restore the real gate behavior and require its exact observed exit/reason. Record queue defects as required/actual deltas owned by RM-03, never as pass/green/OK.
|
||||
|
||||
### Task 4: Source-versus-deployed identity
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `gates/gates.manifest.json`
|
||||
- Modify: `scripts/gate-verify.test.mjs`
|
||||
- Modify: `scripts/gate-verify.mjs`
|
||||
|
||||
Write and observe a failing test with a byte-mutated deployed counterpart. Implement byte equality and internal drift mutation controls. Declare `none` explicitly for gates without deployed counterparts.
|
||||
|
||||
### Task 5: Prose claims and compatibility constructions
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/remediation/MISSION.md`
|
||||
- Modify: `docs/remediation/TASKS.md` only if coordinator authorization overrides the worker prohibition; otherwise place markers in an RM-02 claim index that references immutable source anchors.
|
||||
- Modify: `gates/gates.manifest.json`
|
||||
- Modify: verifier tests/implementation.
|
||||
|
||||
Enumerate current security/integrity claims, bind each marker/id to a negative case, and reject unbound markers. Execute finite compatibility scenarios and clearly document that arbitrary English consistency is outside the model.
|
||||
|
||||
### Task 6: Prospective per-commit replay and provider evidence
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `scripts/gate-history.mjs`
|
||||
- Create: `scripts/gate-history.test.mjs`
|
||||
- Modify: `scripts/gate-verify.mjs`
|
||||
- Modify: `gates/gates.manifest.json`
|
||||
|
||||
Test with a synthetic git repository containing two commits whose manifests differ; prove replay reads each selected commit's tree. Add bounded Gitea/Woodpecker status lookup for prior commits when credentials/history are available. Missing, expired, and currently-running evidence must be explicit states, never inferred success.
|
||||
|
||||
### Task 7: CI and documentation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `package.json`
|
||||
- Modify: `.woodpecker/ci.yml`
|
||||
- Create/update: `docs/DEVELOPER-GUIDE/quality-gate-registry.md`
|
||||
- Create/update: `docs/ADMIN-GUIDE/quality-gate-registry.md`
|
||||
- Modify: `docs/SITEMAP.md`
|
||||
|
||||
Add `gate:verify`; run it in an unconditional PR/main CI step. Document invocation, defect semantics, coverage/exclusions, marker syntax, replay/provider boundaries, and source/deployed identity.
|
||||
|
||||
### Task 8: Verification and delivery
|
||||
|
||||
Run focused tests, `pnpm gate:verify`, checkout tests, typecheck, lint, format, and applicable integration tests. Run Codex code and security review; remediate and re-review. Verify authorship, commit, execute queue guard before push, push, create PR via Mosaic wrapper, and send exact-head review request to rev-974 through the coordinator. Do not merge without coordinator authorization.
|
||||
@@ -0,0 +1,24 @@
|
||||
# RM-02 Governing Claim Index
|
||||
|
||||
This index binds remediation claims that live in orchestrator-owned `TASKS.md` without modifying that file. The source heading and quoted text are the reviewable anchor; `gate:verify` enforces each marker-to-criterion binding. Marker completeness beyond this approved slice remains RM-54.
|
||||
|
||||
## Prose is an enforceable claim
|
||||
|
||||
<!-- GATE-CLAIM:PROSE-IS-A-CLAIM -->
|
||||
|
||||
- Source: `docs/remediation/TASKS.md`, heading `D-20 — the orchestrator's own documentation overclaimed, and a reviewer disproved it empirically`.
|
||||
- Anchored text: “The defect was not in the code — it was in this file.”
|
||||
|
||||
## Generated-state verification scope
|
||||
|
||||
<!-- GATE-CLAIM:GENERATED-STATE-SCOPE -->
|
||||
|
||||
- Source: `docs/remediation/TASKS.md`, heading `D-19 — an integrity property that cannot exist at the layer it was specified`.
|
||||
- Anchored text: “a verifier that cannot detect the attack is not a verifier.”
|
||||
|
||||
## Criterion restatement provenance
|
||||
|
||||
<!-- GATE-CLAIM:CRITERION-RESTATEMENT -->
|
||||
|
||||
- Source: `docs/remediation/TASKS.md`, heading `D-18 — two pre-registered criteria were mutually unsatisfiable, discoverable only at implementation`.
|
||||
- Anchored text: “Both criteria were pre-registered. They cannot both be satisfied.”
|
||||
@@ -12,6 +12,8 @@ gate/program; the LLM handles only genuine judgment.
|
||||
|
||||
### First-class principle — observe the property, not the exit code
|
||||
|
||||
<!-- GATE-CLAIM:OBSERVE-PROPERTY -->
|
||||
|
||||
> **No write is done until the requested PROPERTY is observed. A success exit code is not evidence.**
|
||||
>
|
||||
> **Success output is designed to be believed.** That is the whole reason the inert-gate class exists
|
||||
@@ -30,6 +32,8 @@ gate/program; the LLM handles only genuine judgment.
|
||||
|
||||
### First-class principle — pre-registration prevents retrofitting, and nothing else
|
||||
|
||||
<!-- GATE-CLAIM:PREREGISTRATION-BOUNDARY -->
|
||||
|
||||
> **A pre-registered check set can fail in three distinct ways:**
|
||||
>
|
||||
> | mode | the set is… | found as |
|
||||
@@ -56,6 +60,8 @@ gate/program; the LLM handles only genuine judgment.
|
||||
|
||||
### Corollary — never ship an integrity claim dressed as a property
|
||||
|
||||
<!-- GATE-CLAIM:ARTIFACT-INTEGRITY-BOUNDARY -->
|
||||
|
||||
> A verification artifact that can be forged by whoever it is meant to catch verifies nothing. If a
|
||||
> manifest, marker, ledger, or receipt is writable by the same actor whose behaviour it certifies, it
|
||||
> **certifies the attack.** Such an artifact must sit inside the integrity envelope it belongs to,
|
||||
@@ -68,6 +74,8 @@ gate/program; the LLM handles only genuine judgment.
|
||||
|
||||
### First-class principle — when a property cannot exist at the layer it was specified
|
||||
|
||||
<!-- GATE-CLAIM:IMPOSSIBLE-LAYER-BOUNDARY -->
|
||||
|
||||
> Some required properties are **impossible at the layer that asked for them** — not hard, impossible.
|
||||
> A local check cannot defend against an actor who can rewrite the check itself. When that happens,
|
||||
> there are exactly three honest moves, and all three are mandatory:
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# RM-02 Gate Registry Scratchpad (#1029)
|
||||
|
||||
## Objective
|
||||
|
||||
Deliver the seven-gate registry and RED-first anti-inert verifier on `feat/rm-02-gate-registry`, preserving required-versus-actual defects without laundering them as success.
|
||||
|
||||
## Constraints and boundaries
|
||||
|
||||
- Do not modify `ci-queue-wait.sh`; RM-03 owns that fix.
|
||||
- Do not modify `docs/remediation/TASKS.md`; workers cannot edit orchestrator tracking.
|
||||
- Every declared behavior must be observed, not inferred from an exit code.
|
||||
- Repository-local evidence does not establish same-authority tamper resistance; RM-25/RM-59 own the external trust anchor.
|
||||
- Coverage is seven logical gates; broader inventory is RM-54.
|
||||
- Budget assumption: no explicit token ceiling was supplied. Keep implementation dependency-free (stock Node), avoid repeated full monorepo installs, and keep generated test artifacts under the worktree/main disk.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Update PRD and tracking references.
|
||||
2. Write black-box meta-negative-control first and observe it fail for the missing verifier behavior.
|
||||
3. Implement minimal manifest parser/case runner/mutation detector; observe meta-control succeed.
|
||||
4. Add schema, coverage, provenance, prose marker, compatibility, discovery, deployment identity, and defect-delta tests RED-first.
|
||||
5. Register seven gates with exact cases and run each case.
|
||||
6. Add prospective per-commit replay and bounded provider-evidence reporting.
|
||||
7. Wire unconditional Woodpecker CI and update developer/admin documentation.
|
||||
8. Run situational and baseline verification, independent Codex review, remediate, commit, queue guard, push, PR, coordinator/reviewer handoff.
|
||||
|
||||
## Progress
|
||||
|
||||
- 2026-08-01: Design approved by `mos-remediation`; rulings A-E and source/deployed identity addition incorporated.
|
||||
- 2026-08-01: Isolated worktree created from `origin/main` f65e9ea6; RM-01 f58b3699 verified as ancestor.
|
||||
- 2026-08-01: Git author set to `f10-coder <[email protected]>`; provider issue #1029 created with `MOSAIC_GIT_IDENTITY=f10-coder`.
|
||||
- 2026-08-01: Source/deployed queue-guard SHA-256 observed equal (`19cda2f...`); this observation is not yet an enforced property.
|
||||
|
||||
## Tests and RED-first evidence
|
||||
|
||||
- Meta-negative RED first: `node --test scripts/gate-verify.test.mjs` failed because the absent verifier did not name externally inerted `meta-fixture`.
|
||||
- Structural RED: nine tests failed before implementation for internal mutation, unregistered executable, missing negative control, ownerless delta, unbound criterion/claim, modeled conflict, missing provenance, and deployment drift.
|
||||
- Clause hardening RED: positive-only security criterion and missing registered claim marker both passed incorrectly before validation was added.
|
||||
- CI wiring RED: package script and unconditional Woodpecker step tests both failed before wiring.
|
||||
- History RED: history test failed with missing module before own-tree manifest selection/provider classification was implemented.
|
||||
- `pnpm gate:verify`: exit 0; seven gates each reported `META-NEGATIVE-CONTROL ... observed red`; queue source/deployed drift control observed red; six queue behavior deltas printed as `DEFECT (owner: RM-03)`.
|
||||
- Focused Node tests: 33/33 pass after review hardening (23 verifier/wiring plus 10 history/provider tests).
|
||||
- `pnpm typecheck`: pass (45/45 Turbo tasks).
|
||||
- `pnpm lint`: pass (25/25 Turbo tasks).
|
||||
- `pnpm format:check`: pass.
|
||||
- `pnpm test`: repository suites reached 45/46 Turbo tasks; all application/package tests shown passed, then the known host-specific wake assertion aborted exit 97 (`BASH_LINENO convention violated`, #973/D-16). No test was edited or bypassed. CI remains the authoritative full-suite environment.
|
||||
|
||||
## Self-surfaced defect
|
||||
|
||||
The queue guard's `get_state_from_status_json` runs `python3 - <<'PY'` while provider JSON is piped to the shell function. The heredoc owns stdin, so Python never reads provider JSON. Terminal success, no-status, terminal failure, and malformed payloads all classify as `unknown`; the associated fail-open outcomes are recorded under RM-03. No queue-guard source was modified.
|
||||
|
||||
## Independent review remediation
|
||||
|
||||
- Final pre-commit Codex code review found three blockers: a tautological deployment drift control, independently observed rather than combined compatibility cases, and unauthorized edits to orchestrator-owned `TASKS.md`.
|
||||
- Deployment identity now uses one shared file comparator for both live equality and a temporary drifted deployed copy; a test makes that comparator inert and proves the meta-control fails.
|
||||
- Compatibility scenarios now merge referenced fixtures/environments into one isolated construction and execute an exact scenario invocation; a test proves two independently valid fixtures coexist in the combined run.
|
||||
- `TASKS.md` changes were reverted. `docs/remediation/GATE-CLAIMS.md` binds source headings and anchored text without editing orchestrator tracking.
|
||||
- Codex security review found the Bubblewrap replay shared the runner PID namespace. Replay now unshares PID, IPC, and UTS namespaces, and an abuse-case test proves a sibling runner PID is invisible.
|
||||
- Second review found empty reason diagnostics, final-symlink fixture writes, and lifecycle-script mutation of authoritative history files. Must-fail cases now require a reason pattern; writes use no-follow semantics; and replay snapshots every archived file before install and rejects any changed, deleted, or type/mode-shifted source before executing the verifier. Dedicated negative tests cover all three.
|
||||
- Third code review found ambiguous duplicate provider steps and order-sensitive JSON outcome comparison. Provider evidence now requires exactly one `gate-verify` step in the authoritative rerun, and structural equality normalizes object keys. Both regressions have RED-first tests. Third security review reported no findings.
|
||||
|
||||
## Documentation checklist
|
||||
|
||||
- PRD, developer guide, admin guide, governing claim index, sitemap, plan, and scratchpad updated.
|
||||
- User/API documentation not applicable: no user workflow or API changed.
|
||||
- Independent review documentation check pending rev-974.
|
||||
- Canonical documentation remains in-repository; no external publication requested.
|
||||
|
||||
## Risks/blockers
|
||||
|
||||
- Current queue guard intentionally has required-versus-actual deltas owned by RM-03.
|
||||
- Provider CI cannot report the currently executing pipeline as terminal success; current-commit evidence must be labeled pending and becomes historical evidence only after provider completion.
|
||||
- CI containers may not expose the operator-home deployed queue guard. In that layer the verifier checks the pinned observed digest and reports live identity unavailable under RM-04; it does not infer live equality.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,7 @@
|
||||
"build": "turbo run build",
|
||||
"dev": "turbo run dev",
|
||||
"lint": "turbo run lint",
|
||||
"gate:verify": "node scripts/gate-verify.mjs",
|
||||
"preflight": "node scripts/preflight.mjs",
|
||||
"clean:generated": "node scripts/clean-generated.mjs",
|
||||
"typecheck": "pnpm preflight && turbo run typecheck",
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
import { existsSync } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { access, lstat, mkdir, mkdtemp, readFile, readdir, readlink, rm } from 'node:fs/promises';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
|
||||
function git(root, args, { allowFailure = false } = {}) {
|
||||
const result = spawnSync('git', args, { cwd: root, encoding: 'utf8' });
|
||||
if (result.status !== 0 && !allowFailure) {
|
||||
throw new Error(`git ${args.join(' ')} failed: ${(result.stderr || result.stdout).trim()}`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function listProspectiveCommits(root, activationCommit, head = 'HEAD') {
|
||||
const result = git(root, [
|
||||
'rev-list',
|
||||
'--first-parent',
|
||||
'--reverse',
|
||||
`${activationCommit}..${head}`,
|
||||
]);
|
||||
return result.stdout.trim() ? result.stdout.trim().split('\n') : [];
|
||||
}
|
||||
|
||||
export async function readManifestAtCommit(root, commit) {
|
||||
const result = git(root, ['show', `${commit}:gates/gates.manifest.json`]);
|
||||
return JSON.parse(result.stdout);
|
||||
}
|
||||
|
||||
async function snapshotAuthoritativeTree(root) {
|
||||
const snapshot = new Map();
|
||||
async function walk(current) {
|
||||
for (const child of await readdir(current, { withFileTypes: true })) {
|
||||
if (['.git', '.home', 'node_modules'].includes(child.name)) continue;
|
||||
const absolute = path.join(current, child.name);
|
||||
const relative = path.relative(root, absolute).split(path.sep).join('/');
|
||||
const stats = await lstat(absolute);
|
||||
if (stats.isDirectory()) {
|
||||
await walk(absolute);
|
||||
} else if (stats.isSymbolicLink()) {
|
||||
snapshot.set(relative, `symlink:${stats.mode}:${await readlink(absolute)}`);
|
||||
} else if (stats.isFile()) {
|
||||
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
|
||||
snapshot.set(relative, `file:${stats.mode}:${digest}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
await walk(root);
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
async function authoritativeTreeChanges(root, snapshot) {
|
||||
const changes = [];
|
||||
for (const [relative, expected] of snapshot) {
|
||||
const absolute = path.join(root, relative);
|
||||
let actual;
|
||||
try {
|
||||
const stats = await lstat(absolute);
|
||||
if (stats.isSymbolicLink()) {
|
||||
actual = `symlink:${stats.mode}:${await readlink(absolute)}`;
|
||||
} else if (stats.isFile()) {
|
||||
const digest = createHash('sha256').update(await readFile(absolute)).digest('hex');
|
||||
actual = `file:${stats.mode}:${digest}`;
|
||||
} else {
|
||||
actual = `other:${stats.mode}`;
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
actual = 'missing';
|
||||
}
|
||||
if (actual !== expected) changes.push(relative);
|
||||
}
|
||||
return changes;
|
||||
}
|
||||
|
||||
function bubblewrap(root, command, args, { storePath, timeout = 300_000 } = {}) {
|
||||
const sandboxArgs = [
|
||||
'--unshare-net',
|
||||
'--unshare-pid',
|
||||
'--unshare-ipc',
|
||||
'--unshare-uts',
|
||||
'--die-with-parent',
|
||||
'--new-session',
|
||||
'--clearenv',
|
||||
];
|
||||
for (const systemPath of ['/usr', '/bin', '/lib', '/lib64', '/etc']) {
|
||||
if (existsSync(systemPath)) sandboxArgs.push('--ro-bind', systemPath, systemPath);
|
||||
}
|
||||
sandboxArgs.push('--dev', '/dev', '--proc', '/proc', '--tmpfs', '/tmp', '--bind', root, '/work');
|
||||
if (storePath) sandboxArgs.push('--ro-bind', storePath, '/pnpm-store');
|
||||
const corepackHome = path.join(process.env.HOME ?? '', '.cache', 'node', 'corepack');
|
||||
if (existsSync(corepackHome)) sandboxArgs.push('--ro-bind', corepackHome, '/corepack');
|
||||
sandboxArgs.push(
|
||||
'--chdir',
|
||||
'/work',
|
||||
'--setenv',
|
||||
'HOME',
|
||||
'/work/.home',
|
||||
'--setenv',
|
||||
'PATH',
|
||||
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
|
||||
'--setenv',
|
||||
'LANG',
|
||||
'C.UTF-8',
|
||||
'--setenv',
|
||||
'CI',
|
||||
'true',
|
||||
);
|
||||
if (storePath) sandboxArgs.push('--setenv', 'NPM_CONFIG_STORE_DIR', '/pnpm-store');
|
||||
if (existsSync(corepackHome)) sandboxArgs.push('--setenv', 'COREPACK_HOME', '/corepack');
|
||||
sandboxArgs.push(command, ...args);
|
||||
return spawnSync('bwrap', sandboxArgs, { encoding: 'utf8', timeout });
|
||||
}
|
||||
|
||||
export async function replayCommit(root, commit) {
|
||||
const replayRoot = await mkdtemp(
|
||||
path.join(path.dirname(root), `.gate-history-${commit.slice(0, 12)}-`),
|
||||
);
|
||||
const archive = `${replayRoot}.tar`;
|
||||
try {
|
||||
git(root, ['archive', '--format=tar', `--output=${archive}`, commit]);
|
||||
const extract = spawnSync('tar', ['-xf', archive, '-C', replayRoot], { encoding: 'utf8' });
|
||||
if (extract.status !== 0) {
|
||||
return { status: extract.status, stdout: extract.stdout, stderr: extract.stderr };
|
||||
}
|
||||
const authoritativeSnapshot = await snapshotAuthoritativeTree(replayRoot);
|
||||
await mkdir(path.join(replayRoot, '.home'), { recursive: true });
|
||||
let storePath;
|
||||
try {
|
||||
await access(path.join(replayRoot, 'package.json'));
|
||||
const init = spawnSync('git', ['init', '--quiet', replayRoot], { encoding: 'utf8' });
|
||||
if (init.status !== 0) return init;
|
||||
const store = spawnSync('pnpm', ['store', 'path'], { encoding: 'utf8' });
|
||||
if (store.status !== 0) return store;
|
||||
storePath = store.stdout.trim();
|
||||
const install = bubblewrap(
|
||||
replayRoot,
|
||||
'pnpm',
|
||||
['install', '--frozen-lockfile', '--offline'],
|
||||
{ storePath, timeout: 600_000 },
|
||||
);
|
||||
if (install.status !== 0 || install.error || install.signal) {
|
||||
return {
|
||||
...install,
|
||||
stderr: `historical frozen dependency install failed: ${install.stderr || install.stdout || ''}`,
|
||||
};
|
||||
}
|
||||
const authoritativeChanges = await authoritativeTreeChanges(
|
||||
replayRoot,
|
||||
authoritativeSnapshot,
|
||||
);
|
||||
if (authoritativeChanges.length > 0) {
|
||||
return {
|
||||
status: 1,
|
||||
stdout: '',
|
||||
stderr: `authoritative archived file changed during historical install: ${authoritativeChanges.join(', ')}`,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
}
|
||||
return bubblewrap(
|
||||
replayRoot,
|
||||
process.execPath,
|
||||
[
|
||||
'/work/scripts/gate-verify.mjs',
|
||||
'--root',
|
||||
'/work',
|
||||
'--manifest',
|
||||
'gates/gates.manifest.json',
|
||||
'--skip-history',
|
||||
],
|
||||
{ storePath },
|
||||
);
|
||||
} finally {
|
||||
await rm(archive, { force: true });
|
||||
await rm(replayRoot, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
export function assessProviderEvidence(commit, pipelines) {
|
||||
const matches = pipelines.filter((candidate) => candidate.commit === commit);
|
||||
if (matches.length === 0) {
|
||||
return {
|
||||
state: 'absent',
|
||||
detail:
|
||||
'no retained provider record was supplied; retention expiry and never-ran are not inferred',
|
||||
};
|
||||
}
|
||||
const pipelineStates = new Set(['success', 'failure', 'error', 'pending', 'running', 'queued']);
|
||||
const stepStates = new Set([
|
||||
'success',
|
||||
'failure',
|
||||
'error',
|
||||
'pending',
|
||||
'running',
|
||||
'queued',
|
||||
'skipped',
|
||||
]);
|
||||
const numbers = matches.map((candidate) => candidate.number);
|
||||
const malformed = matches.some(
|
||||
(candidate) =>
|
||||
typeof candidate.commit !== 'string' ||
|
||||
candidate.commit.length === 0 ||
|
||||
!Number.isInteger(candidate.number) ||
|
||||
!pipelineStates.has(candidate.status) ||
|
||||
!Array.isArray(candidate.steps) ||
|
||||
candidate.steps.some(
|
||||
(step) =>
|
||||
typeof step?.name !== 'string' ||
|
||||
typeof step?.status !== 'string' ||
|
||||
!stepStates.has(step.status),
|
||||
),
|
||||
);
|
||||
if (malformed || new Set(numbers).size !== numbers.length) {
|
||||
return {
|
||||
state: 'terminal-failure',
|
||||
detail: 'provider records are malformed or have ambiguous pipeline numbers',
|
||||
};
|
||||
}
|
||||
const pipeline = [...matches].sort((left, right) => right.number - left.number)[0];
|
||||
const gateSteps = (pipeline.steps ?? []).filter((step) => step.name === 'gate-verify');
|
||||
if (gateSteps.length !== 1) {
|
||||
return {
|
||||
state: 'terminal-failure',
|
||||
detail: `provider record has ambiguous gate-verify step count ${gateSteps.length}`,
|
||||
};
|
||||
}
|
||||
const [gateStep] = gateSteps;
|
||||
if (pipeline.status === 'success' && gateStep.status === 'success') {
|
||||
return { state: 'terminal-success', detail: 'pipeline and gate-verify step succeeded' };
|
||||
}
|
||||
if (['pending', 'running', 'queued'].includes(pipeline.status)) {
|
||||
return { state: 'current-running', detail: `pipeline is ${pipeline.status}` };
|
||||
}
|
||||
return {
|
||||
state: 'terminal-failure',
|
||||
detail: `pipeline=${pipeline.status ?? 'unknown'}, gate-verify=${gateStep?.status ?? 'absent'}`,
|
||||
};
|
||||
}
|
||||
|
||||
async function loadProviderEvidence() {
|
||||
const evidenceFile = process.env.GATE_PROVIDER_EVIDENCE_FILE;
|
||||
if (!evidenceFile) return [];
|
||||
const parsed = JSON.parse(await readFile(evidenceFile, 'utf8'));
|
||||
if (!Array.isArray(parsed)) throw new Error('provider evidence file must contain a JSON array');
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function isMainCommit(root, head) {
|
||||
if (process.env.CI_COMMIT_BRANCH === 'main') return true;
|
||||
const result = git(root, ['merge-base', '--is-ancestor', head, 'refs/remotes/origin/main'], {
|
||||
allowFailure: true,
|
||||
});
|
||||
return result.status === 0;
|
||||
}
|
||||
|
||||
export async function verifyHistory({ root, manifest }) {
|
||||
const failures = [];
|
||||
const observations = [];
|
||||
const head = git(root, ['rev-parse', 'HEAD']).stdout.trim();
|
||||
if (!manifest.activationCommit) {
|
||||
failures.push('history activationCommit is missing');
|
||||
return { failures, observations };
|
||||
}
|
||||
const activationCheck = git(
|
||||
root,
|
||||
['merge-base', '--is-ancestor', manifest.activationCommit, head],
|
||||
{ allowFailure: true },
|
||||
);
|
||||
if (activationCheck.status !== 0) {
|
||||
failures.push(
|
||||
`history activation commit ${manifest.activationCommit} is not an ancestor of ${head}`,
|
||||
);
|
||||
return { failures, observations };
|
||||
}
|
||||
const onMain = isMainCommit(root, head);
|
||||
if (!onMain) {
|
||||
observations.push(
|
||||
`PROVIDER ASSERTION DEFERRED ${head}: commit is not yet on main; prospective own-tree replay still runs, while retained merge evidence starts after merge`,
|
||||
);
|
||||
}
|
||||
|
||||
const pipelines = onMain ? await loadProviderEvidence() : [];
|
||||
const commits = await listProspectiveCommits(root, manifest.activationCommit, head);
|
||||
for (const commit of commits) {
|
||||
let commitManifest;
|
||||
try {
|
||||
commitManifest = await readManifestAtCommit(root, commit);
|
||||
} catch (error) {
|
||||
failures.push(`${commit}: own-tree registry cannot be read: ${error.message}`);
|
||||
continue;
|
||||
}
|
||||
if (commitManifest.schemaVersion !== manifest.schemaVersion) {
|
||||
failures.push(`${commit}: own-tree registry schema is not supported`);
|
||||
continue;
|
||||
}
|
||||
const evidence = assessProviderEvidence(commit, pipelines);
|
||||
if (commit === head) {
|
||||
observations.push(
|
||||
`CURRENT TREE EVALUATED ${commit}: all registered cases ran from this checkout; provider evidence=${evidence.state} (${evidence.detail})`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const replay = await replayCommit(root, commit);
|
||||
if (replay.status !== 0 || replay.error || replay.signal) {
|
||||
failures.push(
|
||||
`${commit}: own-tree gate replay failed with exit ${String(replay.status)}${replay.signal ? ` signal ${replay.signal}` : ''}${replay.error ? ` error ${replay.error.message}` : ''}: ${(replay.stderr || replay.stdout || '').trim().slice(0, 500)}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
observations.push(`TREE REPLAY ${commit}: own-tree gate verifier exited 0`);
|
||||
if (!onMain) {
|
||||
observations.push(
|
||||
`PROVIDER EVIDENCE ${commit}: DEFERRED until the commit is on main; no success is inferred`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (evidence.state === 'terminal-failure') {
|
||||
failures.push(
|
||||
`${commit}: retained provider evidence is not terminal-success (${evidence.detail})`,
|
||||
);
|
||||
} else if (evidence.state === 'terminal-success') {
|
||||
observations.push(`PROVIDER EVIDENCE ${commit}: terminal-success (${evidence.detail})`);
|
||||
} else {
|
||||
observations.push(
|
||||
`PROVIDER EVIDENCE ${commit}: ${evidence.state.toUpperCase()} (${evidence.detail}); no merge-time success is inferred`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return { failures, observations };
|
||||
}
|
||||
@@ -0,0 +1,324 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdir, rm, writeFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import test from 'node:test';
|
||||
|
||||
import {
|
||||
assessProviderEvidence,
|
||||
listProspectiveCommits,
|
||||
readManifestAtCommit,
|
||||
replayCommit,
|
||||
verifyHistory,
|
||||
} from './gate-history.mjs';
|
||||
|
||||
const fixtureRoot = path.join(process.cwd(), '.mosaic-test-work', `gate-history-${process.pid}`);
|
||||
|
||||
function git(root, ...args) {
|
||||
const result = spawnSync('git', args, { cwd: root, encoding: 'utf8' });
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
return result.stdout.trim();
|
||||
}
|
||||
|
||||
async function commitManifest(root, marker) {
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
await writeFile(
|
||||
path.join(root, 'gates', 'gates.manifest.json'),
|
||||
`${JSON.stringify({ schemaVersion: 1, marker })}\n`,
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', marker);
|
||||
return git(root, 'rev-parse', 'HEAD');
|
||||
}
|
||||
|
||||
test.after(async () => {
|
||||
await rm(fixtureRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('prospective history reads each commit own manifest rather than the current tree', async () => {
|
||||
await rm(fixtureRoot, { recursive: true, force: true });
|
||||
await mkdir(fixtureRoot, { recursive: true });
|
||||
git(fixtureRoot, 'init', '-q');
|
||||
git(fixtureRoot, 'config', 'user.name', 'gate-test');
|
||||
git(fixtureRoot, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(fixtureRoot, 'activation.txt'), 'activation\n');
|
||||
git(fixtureRoot, 'add', '.');
|
||||
git(fixtureRoot, 'commit', '-m', 'activation');
|
||||
const activation = git(fixtureRoot, 'rev-parse', 'HEAD');
|
||||
const first = await commitManifest(fixtureRoot, 'FIRST');
|
||||
const second = await commitManifest(fixtureRoot, 'SECOND');
|
||||
|
||||
assert.deepEqual(await listProspectiveCommits(fixtureRoot, activation, second), [first, second]);
|
||||
assert.equal((await readManifestAtCommit(fixtureRoot, first)).marker, 'FIRST');
|
||||
assert.equal((await readManifestAtCommit(fixtureRoot, second)).marker, 'SECOND');
|
||||
});
|
||||
|
||||
test('historical replay executes each selected commit verifier from that commit tree', async () => {
|
||||
const root = `${fixtureRoot}-replay`;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(path.join(root, 'scripts'), { recursive: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
git(root, 'init', '-q');
|
||||
git(root, 'config', 'user.name', 'gate-test');
|
||||
git(root, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), '{"schemaVersion":1}\n');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"process.stderr.write('OLD TREE INERT\\n'); process.exitCode = 1;\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'inert historical verifier');
|
||||
const inert = git(root, 'rev-parse', 'HEAD');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"process.stdout.write('NEW TREE VERIFIED\\n');\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'fixed historical verifier');
|
||||
const fixed = git(root, 'rev-parse', 'HEAD');
|
||||
|
||||
const inertResult = await replayCommit(root, inert);
|
||||
const fixedResult = await replayCommit(root, fixed);
|
||||
assert.notEqual(inertResult.status, 0);
|
||||
assert.match(inertResult.stderr, /OLD TREE INERT/);
|
||||
assert.equal(fixedResult.status, 0);
|
||||
assert.match(fixedResult.stdout, /NEW TREE VERIFIED/);
|
||||
});
|
||||
|
||||
test('historical install lifecycle cannot replace an authoritative verifier', async () => {
|
||||
const root = `${fixtureRoot}-install-tamper`;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(path.join(root, 'scripts'), { recursive: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
git(root, 'init', '-q');
|
||||
git(root, 'config', 'user.name', 'gate-test');
|
||||
git(root, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), '{"schemaVersion":1}\n');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"process.stderr.write('ORIGINAL VERIFIER RAN\\n'); process.exitCode = 7;\n",
|
||||
);
|
||||
await writeFile(path.join(root, 'forged.mjs'), "process.stdout.write('FORGED SUCCESS\\n');\n");
|
||||
await writeFile(
|
||||
path.join(root, 'package.json'),
|
||||
`${JSON.stringify({
|
||||
name: 'historical-install-tamper',
|
||||
version: '1.0.0',
|
||||
scripts: { postinstall: 'cp forged.mjs scripts/gate-verify.mjs' },
|
||||
})}\n`,
|
||||
);
|
||||
await writeFile(
|
||||
path.join(root, 'pnpm-lock.yaml'),
|
||||
"lockfileVersion: '9.0'\nsettings:\n autoInstallPeers: true\n excludeLinksFromLockfile: false\nimporters:\n .: {}\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'tampering lifecycle fixture');
|
||||
const commit = git(root, 'rev-parse', 'HEAD');
|
||||
|
||||
const result = await replayCommit(root, commit);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(result.stderr, /authoritative archived file changed.*scripts\/gate-verify\.mjs/i);
|
||||
assert.doesNotMatch(result.stdout, /FORGED SUCCESS/);
|
||||
});
|
||||
|
||||
test('historical verifier receives no current-process secret environment', async () => {
|
||||
const root = `${fixtureRoot}-secretless`;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(path.join(root, 'scripts'), { recursive: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
git(root, 'init', '-q');
|
||||
git(root, 'config', 'user.name', 'gate-test');
|
||||
git(root, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), '{"schemaVersion":1}\n');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"if (process.env.REPLAY_SENTINEL) { process.stderr.write('SECRET LEAKED\\n'); process.exitCode = 9; } else { process.stdout.write('SECRETLESS\\n'); }\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'secretless replay fixture');
|
||||
const commit = git(root, 'rev-parse', 'HEAD');
|
||||
|
||||
process.env.REPLAY_SENTINEL = 'must-not-cross-boundary';
|
||||
try {
|
||||
const result = await replayCommit(root, commit);
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.match(result.stdout, /SECRETLESS/);
|
||||
assert.doesNotMatch(
|
||||
`${result.stdout}${result.stderr}`,
|
||||
/SECRET LEAKED|must-not-cross-boundary/,
|
||||
);
|
||||
} finally {
|
||||
delete process.env.REPLAY_SENTINEL;
|
||||
}
|
||||
});
|
||||
|
||||
test('historical replay cannot observe a sibling process in the runner PID namespace', async () => {
|
||||
const root = `${fixtureRoot}-pidless`;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(path.join(root, 'scripts'), { recursive: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
git(root, 'init', '-q');
|
||||
git(root, 'config', 'user.name', 'gate-test');
|
||||
git(root, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), '{"schemaVersion":1}\n');
|
||||
|
||||
const sleeper = spawn('sleep', ['30'], {
|
||||
env: { ...process.env, REPLAY_PID_SENTINEL: 'must-not-be-visible' },
|
||||
});
|
||||
try {
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
`import { existsSync } from 'node:fs';\nif (existsSync('/proc/${sleeper.pid}/environ')) { process.stderr.write('HOST PID VISIBLE\\n'); process.exitCode = 9; } else { process.stdout.write('PIDLESS\\n'); }\n`,
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'pid-isolated replay fixture');
|
||||
const commit = git(root, 'rev-parse', 'HEAD');
|
||||
const result = await replayCommit(root, commit);
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.match(result.stdout, /PIDLESS/);
|
||||
assert.doesNotMatch(`${result.stdout}${result.stderr}`, /HOST PID VISIBLE/);
|
||||
} finally {
|
||||
sleeper.kill('SIGTERM');
|
||||
}
|
||||
});
|
||||
|
||||
test('feature-branch history replays an inert intermediate commit before the healthy head', async () => {
|
||||
const root = `${fixtureRoot}-feature`;
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(root, { recursive: true });
|
||||
git(root, 'init', '-q');
|
||||
git(root, 'config', 'user.name', 'gate-test');
|
||||
git(root, 'config', 'user.email', '[email protected]');
|
||||
await writeFile(path.join(root, 'activation.txt'), 'activation\n');
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'activation');
|
||||
const activation = git(root, 'rev-parse', 'HEAD');
|
||||
git(root, 'update-ref', 'refs/remotes/origin/main', activation);
|
||||
await mkdir(path.join(root, 'scripts'), { recursive: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), '{"schemaVersion":1}\n');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"process.stderr.write('INTERMEDIATE INERT\\n'); process.exitCode = 1;\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'inert intermediate');
|
||||
await writeFile(
|
||||
path.join(root, 'scripts', 'gate-verify.mjs'),
|
||||
"process.stdout.write('HEAD HEALTHY\\n');\n",
|
||||
);
|
||||
git(root, 'add', '.');
|
||||
git(root, 'commit', '-m', 'healthy head');
|
||||
|
||||
const previousBranch = process.env.CI_COMMIT_BRANCH;
|
||||
process.env.CI_COMMIT_BRANCH = 'feature/rm-02';
|
||||
try {
|
||||
const result = await verifyHistory({
|
||||
root,
|
||||
manifest: { schemaVersion: 1, activationCommit: activation },
|
||||
});
|
||||
assert.ok(result.failures.some((failure) => /INTERMEDIATE INERT/.test(failure)));
|
||||
} finally {
|
||||
if (previousBranch === undefined) delete process.env.CI_COMMIT_BRANCH;
|
||||
else process.env.CI_COMMIT_BRANCH = previousBranch;
|
||||
}
|
||||
});
|
||||
|
||||
test('provider evidence distinguishes retained success, failure, and absent history', () => {
|
||||
const pipelines = [
|
||||
{
|
||||
commit: 'aaa',
|
||||
number: 1,
|
||||
status: 'success',
|
||||
steps: [{ name: 'gate-verify', status: 'success' }],
|
||||
},
|
||||
{
|
||||
commit: 'bbb',
|
||||
number: 2,
|
||||
status: 'failure',
|
||||
steps: [{ name: 'gate-verify', status: 'failure' }],
|
||||
},
|
||||
];
|
||||
assert.deepEqual(assessProviderEvidence('aaa', pipelines), {
|
||||
state: 'terminal-success',
|
||||
detail: 'pipeline and gate-verify step succeeded',
|
||||
});
|
||||
assert.equal(assessProviderEvidence('bbb', pipelines).state, 'terminal-failure');
|
||||
assert.equal(assessProviderEvidence('ccc', pipelines).state, 'absent');
|
||||
});
|
||||
|
||||
test('duplicate gate-verify steps cannot establish provider success', () => {
|
||||
const result = assessProviderEvidence('aaa', [
|
||||
{
|
||||
commit: 'aaa',
|
||||
number: 7,
|
||||
status: 'success',
|
||||
steps: [
|
||||
{ name: 'gate-verify', status: 'success' },
|
||||
{ name: 'gate-verify', status: 'failure' },
|
||||
],
|
||||
},
|
||||
]);
|
||||
assert.equal(result.state, 'terminal-failure');
|
||||
assert.match(result.detail, /ambiguous.*gate-verify/i);
|
||||
});
|
||||
|
||||
test('a malformed single provider record cannot establish success', () => {
|
||||
assert.equal(
|
||||
assessProviderEvidence('aaa', [
|
||||
{ commit: 'aaa', status: 'success', steps: [{ name: 'gate-verify', status: 'success' }] },
|
||||
]).state,
|
||||
'terminal-failure',
|
||||
);
|
||||
assert.equal(
|
||||
assessProviderEvidence('bbb', [
|
||||
{
|
||||
commit: 'bbb',
|
||||
number: 1,
|
||||
status: 'surprising',
|
||||
steps: [{ name: 'gate-verify', status: 'success' }],
|
||||
},
|
||||
]).state,
|
||||
'terminal-failure',
|
||||
);
|
||||
});
|
||||
|
||||
test('provider evidence selects the highest numbered rerun deterministically', () => {
|
||||
const failedThenSucceeded = [
|
||||
{
|
||||
commit: 'aaa',
|
||||
number: 10,
|
||||
status: 'failure',
|
||||
steps: [{ name: 'gate-verify', status: 'failure' }],
|
||||
},
|
||||
{
|
||||
commit: 'aaa',
|
||||
number: 11,
|
||||
status: 'success',
|
||||
steps: [{ name: 'gate-verify', status: 'success' }],
|
||||
},
|
||||
];
|
||||
const succeededThenFailed = [
|
||||
{
|
||||
commit: 'bbb',
|
||||
number: 21,
|
||||
status: 'success',
|
||||
steps: [{ name: 'gate-verify', status: 'success' }],
|
||||
},
|
||||
{
|
||||
commit: 'bbb',
|
||||
number: 22,
|
||||
status: 'failure',
|
||||
steps: [{ name: 'gate-verify', status: 'failure' }],
|
||||
},
|
||||
];
|
||||
assert.equal(assessProviderEvidence('aaa', failedThenSucceeded).state, 'terminal-success');
|
||||
assert.equal(assessProviderEvidence('bbb', succeededThenFailed).state, 'terminal-failure');
|
||||
assert.equal(
|
||||
assessProviderEvidence('ccc', [
|
||||
{ commit: 'ccc', status: 'success', steps: [{ name: 'gate-verify', status: 'success' }] },
|
||||
{ commit: 'ccc', status: 'failure', steps: [{ name: 'gate-verify', status: 'failure' }] },
|
||||
]).state,
|
||||
'terminal-failure',
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,747 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { constants } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
import {
|
||||
access,
|
||||
chmod,
|
||||
copyFile,
|
||||
lstat,
|
||||
mkdir,
|
||||
mkdtemp,
|
||||
open,
|
||||
readFile,
|
||||
readdir,
|
||||
readlink,
|
||||
rm,
|
||||
symlink,
|
||||
writeFile,
|
||||
} from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
import { verifyHistory } from './gate-history.mjs';
|
||||
|
||||
const COPY_SKIP = new Set(['.git', '.mosaic-test-work', '.next', '.turbo', 'coverage', 'dist']);
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {
|
||||
root: process.cwd(),
|
||||
manifest: 'gates/gates.manifest.json',
|
||||
skipHistory: false,
|
||||
};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const value = argv[index];
|
||||
if (value === '--root') options.root = path.resolve(argv[++index]);
|
||||
else if (value === '--manifest') options.manifest = argv[++index];
|
||||
else if (value === '--skip-history') options.skipHistory = true;
|
||||
else throw new Error(`unknown option: ${value}`);
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function runInvocation(root, invocation, extraEnvironment = {}) {
|
||||
if (!Array.isArray(invocation) || invocation.length === 0) {
|
||||
return { status: null, stdout: '', stderr: 'missing invocation' };
|
||||
}
|
||||
return spawnSync(invocation[0], invocation.slice(1), {
|
||||
cwd: root,
|
||||
encoding: 'utf8',
|
||||
env: { ...process.env, GATE_VERIFY: '1', ...extraEnvironment },
|
||||
timeout: 300_000,
|
||||
});
|
||||
}
|
||||
|
||||
async function sandboxPath(root, relativePath, label) {
|
||||
if (typeof relativePath !== 'string' || path.isAbsolute(relativePath)) {
|
||||
throw new Error(`${label}: path escapes sandbox (${String(relativePath)})`);
|
||||
}
|
||||
const resolvedRoot = path.resolve(root);
|
||||
const target = path.resolve(resolvedRoot, relativePath);
|
||||
if (target !== resolvedRoot && !target.startsWith(`${resolvedRoot}${path.sep}`)) {
|
||||
throw new Error(`${label}: path escapes sandbox (${relativePath})`);
|
||||
}
|
||||
const parts = path.relative(resolvedRoot, path.dirname(target)).split(path.sep).filter(Boolean);
|
||||
let current = resolvedRoot;
|
||||
for (const part of parts) {
|
||||
current = path.join(current, part);
|
||||
try {
|
||||
if ((await lstat(current)).isSymbolicLink()) {
|
||||
throw new Error(`${label}: path crosses symbolic link (${relativePath})`);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') break;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
function expand(value, root) {
|
||||
return String(value)
|
||||
.replaceAll('${ROOT}', root)
|
||||
.replaceAll('${HOME}', process.env.HOME ?? '')
|
||||
.replaceAll('${PATH}', process.env.PATH ?? '');
|
||||
}
|
||||
|
||||
function normalizedJson(value) {
|
||||
if (Array.isArray(value)) return value.map(normalizedJson);
|
||||
if (value && typeof value === 'object') {
|
||||
return Object.fromEntries(
|
||||
Object.keys(value)
|
||||
.sort()
|
||||
.map((key) => [key, normalizedJson(value[key])]),
|
||||
);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function structuredValuesEqual(left, right) {
|
||||
return JSON.stringify(normalizedJson(left)) === JSON.stringify(normalizedJson(right));
|
||||
}
|
||||
|
||||
function outcomeMatches(outcome, result) {
|
||||
const combined = `${result.stdout ?? ''}\n${result.stderr ?? ''}`;
|
||||
return (
|
||||
result.status === outcome?.exitCode &&
|
||||
(!outcome?.outputPattern || new RegExp(outcome.outputPattern, 'm').test(combined)) &&
|
||||
(!outcome?.notOutputPattern || !new RegExp(outcome.notOutputPattern, 'm').test(combined))
|
||||
);
|
||||
}
|
||||
|
||||
function resultMatches(gateCase, result) {
|
||||
const combined = `${result.stdout ?? ''}\n${result.stderr ?? ''}`;
|
||||
return (
|
||||
outcomeMatches(gateCase.actual, result) &&
|
||||
(!gateCase.reasonPattern || new RegExp(gateCase.reasonPattern, 'm').test(combined))
|
||||
);
|
||||
}
|
||||
|
||||
async function safeSandboxWrite(root, relativePath, contents, label) {
|
||||
const target = await sandboxPath(root, relativePath, label);
|
||||
await mkdir(path.dirname(target), { recursive: true });
|
||||
try {
|
||||
if ((await lstat(target)).isSymbolicLink()) {
|
||||
throw new Error(`${label}: target is a symbolic link (${relativePath})`);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
}
|
||||
let handle;
|
||||
try {
|
||||
handle = await open(
|
||||
target,
|
||||
constants.O_WRONLY | constants.O_CREAT | constants.O_TRUNC | constants.O_NOFOLLOW,
|
||||
0o666,
|
||||
);
|
||||
await handle.writeFile(contents);
|
||||
} catch (error) {
|
||||
if (error.code === 'ELOOP') {
|
||||
throw new Error(`${label}: target is a symbolic link (${relativePath})`);
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
await handle?.close();
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
async function applyFixture(root, fixture = {}) {
|
||||
for (const entry of fixture.writeFiles ?? []) {
|
||||
const target = await safeSandboxWrite(root, entry.path, entry.content, 'fixture write');
|
||||
if (entry.mode !== undefined) await chmod(target, entry.mode);
|
||||
}
|
||||
for (const relativePath of fixture.removePaths ?? []) {
|
||||
await rm(await sandboxPath(root, relativePath, 'fixture remove'), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function copySandbox(root, destination, selectedPaths) {
|
||||
if (!selectedPaths?.length) {
|
||||
await copyTree(root, destination);
|
||||
return;
|
||||
}
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const relativePath of selectedPaths) {
|
||||
await copyTree(
|
||||
await sandboxPath(root, relativePath, 'sandbox copy source'),
|
||||
await sandboxPath(destination, relativePath, 'sandbox copy destination'),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function runCase(root, gate, gateCase) {
|
||||
let caseRoot = root;
|
||||
if (gateCase.fixture) {
|
||||
caseRoot = await mkdtemp(path.join(path.dirname(root), `.gate-case-${gate.id}-`));
|
||||
await copySandbox(root, caseRoot, gateCase.fixture.copyPaths);
|
||||
await applyFixture(caseRoot, gateCase.fixture);
|
||||
}
|
||||
try {
|
||||
const environment = Object.fromEntries(
|
||||
Object.entries(gateCase.environment ?? {}).map(([key, value]) => [
|
||||
key,
|
||||
expand(value, caseRoot),
|
||||
]),
|
||||
);
|
||||
return runInvocation(caseRoot, gateCase.invocation ?? gate.invocation, environment);
|
||||
} finally {
|
||||
if (caseRoot !== root) await rm(caseRoot, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function copyTree(source, destination) {
|
||||
const stats = await lstat(source);
|
||||
if (stats.isSymbolicLink()) {
|
||||
await symlink(await readlink(source), destination);
|
||||
return;
|
||||
}
|
||||
if (stats.isFile()) {
|
||||
await mkdir(path.dirname(destination), { recursive: true });
|
||||
await copyFile(source, destination);
|
||||
await chmod(destination, stats.mode);
|
||||
return;
|
||||
}
|
||||
if (!stats.isDirectory()) return;
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const child of await readdir(source, { withFileTypes: true })) {
|
||||
if (COPY_SKIP.has(child.name)) continue;
|
||||
const childSource = path.join(source, child.name);
|
||||
const childDestination = path.join(destination, child.name);
|
||||
if (child.name === 'node_modules') {
|
||||
await symlink(childSource, childDestination, 'dir');
|
||||
continue;
|
||||
}
|
||||
await copyTree(childSource, childDestination);
|
||||
}
|
||||
}
|
||||
|
||||
async function executableFiles(root, relativeRoot) {
|
||||
const base = await sandboxPath(root, relativeRoot, 'gate root');
|
||||
const found = [];
|
||||
async function walk(current) {
|
||||
for (const child of await readdir(current, { withFileTypes: true })) {
|
||||
const target = path.join(current, child.name);
|
||||
if (child.isDirectory()) await walk(target);
|
||||
else if (child.isFile()) {
|
||||
try {
|
||||
await access(target, constants.X_OK);
|
||||
found.push(path.relative(root, target).split(path.sep).join('/'));
|
||||
} catch {
|
||||
// Non-executable files are not gates for discovery purposes.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
try {
|
||||
await walk(base);
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowed, label, failures) {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
failures.push(`${label}: expected an object`);
|
||||
return;
|
||||
}
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowed.has(key)) failures.push(`${label}: unknown field ${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
function rejectDuplicateIds(values, label, failures) {
|
||||
const seen = new Set();
|
||||
for (const value of values ?? []) {
|
||||
if (typeof value?.id !== 'string' || value.id.length === 0) {
|
||||
failures.push(`${label}: missing stable id`);
|
||||
} else if (seen.has(value.id)) {
|
||||
failures.push(`duplicate ${label} id ${value.id}`);
|
||||
} else {
|
||||
seen.add(value.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateClosedSchema(manifest, failures) {
|
||||
if (manifest.schemaVersion !== 1)
|
||||
failures.push(`unsupported schemaVersion ${String(manifest.schemaVersion)}`);
|
||||
rejectUnknownKeys(
|
||||
manifest,
|
||||
new Set([
|
||||
'schemaVersion',
|
||||
'activationCommit',
|
||||
'gateRoots',
|
||||
'governingClaimFiles',
|
||||
'coverageBoundary',
|
||||
'criteria',
|
||||
'proseClaims',
|
||||
'compatibilityScenarios',
|
||||
'mergeAssertions',
|
||||
'gates',
|
||||
]),
|
||||
'manifest',
|
||||
failures,
|
||||
);
|
||||
rejectDuplicateIds(manifest.criteria, 'criterion', failures);
|
||||
rejectDuplicateIds(manifest.proseClaims, 'prose claim', failures);
|
||||
rejectDuplicateIds(manifest.compatibilityScenarios, 'compatibility scenario', failures);
|
||||
for (const scenario of manifest.compatibilityScenarios ?? []) {
|
||||
rejectUnknownKeys(
|
||||
scenario,
|
||||
new Set([
|
||||
'id',
|
||||
'construction',
|
||||
'caseRefs',
|
||||
'invocation',
|
||||
'expected',
|
||||
'environment',
|
||||
'fixture',
|
||||
]),
|
||||
`compatibility scenario ${scenario.id}`,
|
||||
failures,
|
||||
);
|
||||
if (!Array.isArray(scenario.caseRefs) || scenario.caseRefs.length === 0) {
|
||||
failures.push(`${scenario.id}: compatibility construction has no referenced conditions`);
|
||||
}
|
||||
if (!Array.isArray(scenario.invocation) || scenario.invocation.length === 0) {
|
||||
failures.push(`${scenario.id}: compatibility construction invocation is missing`);
|
||||
}
|
||||
if (typeof scenario.expected?.exitCode !== 'number') {
|
||||
failures.push(`${scenario.id}: compatibility construction exact expected exit is missing`);
|
||||
}
|
||||
}
|
||||
rejectDuplicateIds(manifest.gates, 'gate', failures);
|
||||
for (const gate of manifest.gates ?? []) {
|
||||
rejectUnknownKeys(
|
||||
gate,
|
||||
new Set([
|
||||
'id',
|
||||
'source',
|
||||
'invocation',
|
||||
'deployment',
|
||||
'inertMutation',
|
||||
'cases',
|
||||
'discoveryAliases',
|
||||
]),
|
||||
`gate ${gate.id}`,
|
||||
failures,
|
||||
);
|
||||
rejectDuplicateIds(gate.cases, `case in gate ${gate.id}`, failures);
|
||||
if (!Array.isArray(gate.invocation) || gate.invocation.length === 0) {
|
||||
failures.push(`${gate.id}: exact invocation is missing`);
|
||||
}
|
||||
if (!['none', 'file'].includes(gate.deployment?.kind)) {
|
||||
failures.push(`${gate.id}: unsupported deployment kind ${String(gate.deployment?.kind)}`);
|
||||
}
|
||||
for (const gateCase of gate.cases ?? []) {
|
||||
rejectUnknownKeys(
|
||||
gateCase,
|
||||
new Set([
|
||||
'id',
|
||||
'criterionIds',
|
||||
'mustFail',
|
||||
'invocation',
|
||||
'required',
|
||||
'actual',
|
||||
'reasonPattern',
|
||||
'environment',
|
||||
'fixture',
|
||||
'defect',
|
||||
]),
|
||||
`${gate.id}/${gateCase.id}`,
|
||||
failures,
|
||||
);
|
||||
if (
|
||||
typeof gateCase.required?.exitCode !== 'number' ||
|
||||
typeof gateCase.actual?.exitCode !== 'number'
|
||||
) {
|
||||
failures.push(
|
||||
`${gate.id}/${gateCase.id}: required and actual exact exit codes are mandatory`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
gateCase.invocation &&
|
||||
(!Array.isArray(gateCase.invocation) || gateCase.invocation.length === 0)
|
||||
) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: case invocation must be non-empty`);
|
||||
}
|
||||
if (gateCase.mustFail === true && !gateCase.reasonPattern?.trim()) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: must-fail case requires a non-empty reasonPattern`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateStructure(manifest, failures) {
|
||||
validateClosedSchema(manifest, failures);
|
||||
const criteria = new Map((manifest.criteria ?? []).map((criterion) => [criterion.id, criterion]));
|
||||
const boundCriteria = new Set();
|
||||
const negativeBoundCriteria = new Set();
|
||||
|
||||
for (const gate of manifest.gates ?? []) {
|
||||
const negativeCases = (gate.cases ?? []).filter((gateCase) => gateCase.mustFail === true);
|
||||
if (negativeCases.length === 0) failures.push(`${gate.id}: no negative control`);
|
||||
if (!gate.deployment?.kind) failures.push(`${gate.id}: deployment identity is not declared`);
|
||||
|
||||
for (const gateCase of gate.cases ?? []) {
|
||||
for (const criterionId of gateCase.criterionIds ?? []) {
|
||||
if (!criteria.has(criterionId)) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: unknown criterion ${criterionId}`);
|
||||
}
|
||||
boundCriteria.add(criterionId);
|
||||
if (gateCase.mustFail === true) negativeBoundCriteria.add(criterionId);
|
||||
}
|
||||
if (
|
||||
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
|
||||
!gateCase.defect?.owner
|
||||
) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: behavior delta requires a tracked owner`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const claim of manifest.proseClaims ?? []) {
|
||||
if (!criteria.has(claim.criterionId)) {
|
||||
failures.push(`GATE-CLAIM:${claim.id} references unknown criterion ${claim.criterionId}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const criterion of criteria.values()) {
|
||||
if (!boundCriteria.has(criterion.id)) failures.push(`${criterion.id}: no bound case`);
|
||||
else if (!negativeBoundCriteria.has(criterion.id)) {
|
||||
failures.push(`${criterion.id}: no must-fail case exercises this criterion`);
|
||||
}
|
||||
if (
|
||||
criterion.originalText !== criterion.currentText &&
|
||||
(!Array.isArray(criterion.meaningChanges) || criterion.meaningChanges.length === 0)
|
||||
) {
|
||||
failures.push(`${criterion.id}: missing meaning-change provenance`);
|
||||
}
|
||||
}
|
||||
|
||||
const byConstruction = new Map();
|
||||
for (const scenario of manifest.compatibilityScenarios ?? []) {
|
||||
const previous = byConstruction.get(scenario.construction);
|
||||
if (previous && !structuredValuesEqual(previous.expected, scenario.expected)) {
|
||||
failures.push(`${previous.id} and ${scenario.id}: modeled compatibility conflict`);
|
||||
} else {
|
||||
byConstruction.set(scenario.construction, scenario);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function validateClaims(root, manifest, failures) {
|
||||
const registered = new Set((manifest.proseClaims ?? []).map((claim) => claim.id));
|
||||
const observed = new Set();
|
||||
for (const relativeFile of manifest.governingClaimFiles ?? []) {
|
||||
const contents = await readFile(
|
||||
await sandboxPath(root, relativeFile, 'governing claim file'),
|
||||
'utf8',
|
||||
);
|
||||
for (const match of contents.matchAll(/GATE-CLAIM:([A-Z0-9][A-Z0-9-]*)/g)) {
|
||||
observed.add(match[1]);
|
||||
if (!registered.has(match[1])) {
|
||||
failures.push(`GATE-CLAIM:${match[1]} is unbound in ${relativeFile}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const claimId of registered) {
|
||||
if (!observed.has(claimId)) failures.push(`GATE-CLAIM:${claimId} marker is missing`);
|
||||
}
|
||||
}
|
||||
|
||||
async function validateDiscovery(root, manifest, failures) {
|
||||
const registered = new Set(
|
||||
(manifest.gates ?? []).flatMap((gate) => [gate.source, ...(gate.discoveryAliases ?? [])]),
|
||||
);
|
||||
for (const gateRoot of manifest.gateRoots ?? []) {
|
||||
for (const executable of await executableFiles(root, gateRoot)) {
|
||||
if (!registered.has(executable)) failures.push(`unregistered gate: ${executable}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function deploymentFilesEqual(sourcePath, deployedPath) {
|
||||
const [source, deployed] = await Promise.all([readFile(sourcePath), readFile(deployedPath)]);
|
||||
return source.equals(deployed);
|
||||
}
|
||||
|
||||
async function validateDeployment(root, gate, failures, observations) {
|
||||
if (gate.deployment?.kind !== 'file') return;
|
||||
const sourcePath = await sandboxPath(
|
||||
root,
|
||||
gate.deployment.source ?? gate.source,
|
||||
`${gate.id} deployment source`,
|
||||
);
|
||||
const deployedPath = path.isAbsolute(expand(gate.deployment.path, root))
|
||||
? expand(gate.deployment.path, root)
|
||||
: path.resolve(root, expand(gate.deployment.path, root));
|
||||
const source = await readFile(sourcePath);
|
||||
let deployed;
|
||||
try {
|
||||
deployed = await readFile(deployedPath);
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
const observedHash = createHash('sha256').update(source).digest('hex');
|
||||
if (
|
||||
!gate.deployment.unavailableOwner ||
|
||||
!gate.deployment.observedSha256 ||
|
||||
gate.deployment.observedSha256 !== observedHash
|
||||
) {
|
||||
failures.push(
|
||||
`${gate.id}: deployed counterpart is unavailable and no current tracked observation matches source`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
observations.push(
|
||||
`DEPLOYED IDENTITY UNAVAILABLE (owner: ${gate.deployment.unavailableOwner}) ${gate.id}: ${deployedPath} is outside this runner; source matches pinned observation ${observedHash}, live equality is not inferred`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (!(await deploymentFilesEqual(sourcePath, deployedPath))) {
|
||||
failures.push(`${gate.id}: source versus deployed copy drift`);
|
||||
return;
|
||||
}
|
||||
|
||||
const controlRoot = await mkdtemp(
|
||||
path.join(path.dirname(root), `.gate-deployment-control-${gate.id}-`),
|
||||
);
|
||||
try {
|
||||
const alteredPath = path.join(controlRoot, 'deployed-copy');
|
||||
await writeFile(
|
||||
alteredPath,
|
||||
Buffer.concat([deployed, Buffer.from('\n# gate deployment drift control\n')]),
|
||||
);
|
||||
if (await deploymentFilesEqual(sourcePath, alteredPath)) {
|
||||
failures.push(`${gate.id}: deployed-copy drift negative control was ineffective`);
|
||||
} else {
|
||||
observations.push(`DEPLOYMENT-NEGATIVE-CONTROL ${gate.id}: observed red`);
|
||||
}
|
||||
} finally {
|
||||
await rm(controlRoot, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function validateMutation(root, gate, failures, observations) {
|
||||
const mutation = gate.inertMutation;
|
||||
if (
|
||||
!mutation?.file ||
|
||||
typeof mutation.find !== 'string' ||
|
||||
typeof mutation.replace !== 'string' ||
|
||||
typeof mutation.expected?.exitCode !== 'number'
|
||||
) {
|
||||
failures.push(`${gate.id}: declared inert mutation is incomplete`);
|
||||
return;
|
||||
}
|
||||
const mutationPath = await sandboxPath(root, mutation.file, `${gate.id} mutation source`);
|
||||
let source;
|
||||
try {
|
||||
source = await readFile(mutationPath, 'utf8');
|
||||
} catch (error) {
|
||||
failures.push(`${gate.id}: mutation source unreadable: ${error.message}`);
|
||||
return;
|
||||
}
|
||||
const occurrences = source.split(mutation.find).length - 1;
|
||||
if (occurrences !== 1) {
|
||||
failures.push(
|
||||
`${gate.id}: declared inert mutation is stale or ambiguous (${occurrences} matches)`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
const targetCase = mutation.caseId
|
||||
? (gate.cases ?? []).find((gateCase) => gateCase.id === mutation.caseId)
|
||||
: (gate.cases ?? []).find((gateCase) => gateCase.mustFail === true);
|
||||
if (!targetCase) {
|
||||
failures.push(
|
||||
mutation.caseId
|
||||
? `${gate.id}: declared mutation case ${mutation.caseId} was not found`
|
||||
: `${gate.id}: declared mutation has no must-fail case`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const sandbox = await mkdtemp(path.join(path.dirname(root), `.gate-sandbox-${gate.id}-`));
|
||||
try {
|
||||
await copySandbox(root, sandbox, mutation.sandboxFiles);
|
||||
await safeSandboxWrite(
|
||||
sandbox,
|
||||
mutation.file,
|
||||
source.replace(mutation.find, mutation.replace),
|
||||
`${gate.id} sandbox mutation write`,
|
||||
);
|
||||
await applyFixture(sandbox, targetCase.fixture);
|
||||
const environment = Object.fromEntries(
|
||||
Object.entries(targetCase.environment ?? {}).map(([key, value]) => [
|
||||
key,
|
||||
expand(value, sandbox),
|
||||
]),
|
||||
);
|
||||
const result = runInvocation(sandbox, targetCase.invocation ?? gate.invocation, environment);
|
||||
if (result.error || result.signal) {
|
||||
failures.push(
|
||||
`${gate.id}: mutation execution crashed${result.signal ? ` with ${result.signal}` : ''}${result.error ? `: ${result.error.message}` : ''}`,
|
||||
);
|
||||
} else if (!outcomeMatches(mutation.expected, result)) {
|
||||
failures.push(
|
||||
`${gate.id}: mutation produced unexpected outcome ${String(result.status)}; expected ${JSON.stringify(mutation.expected)}`,
|
||||
);
|
||||
} else if (resultMatches(targetCase, result)) {
|
||||
failures.push(`${gate.id}: declared inert mutation was ineffective`);
|
||||
} else {
|
||||
observations.push(`META-NEGATIVE-CONTROL ${gate.id}: observed red`);
|
||||
}
|
||||
} finally {
|
||||
await rm(sandbox, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function findGateCase(manifest, caseRef) {
|
||||
const separator = caseRef.indexOf('/');
|
||||
if (separator < 1) return undefined;
|
||||
const gateId = caseRef.slice(0, separator);
|
||||
const caseId = caseRef.slice(separator + 1);
|
||||
const gate = (manifest.gates ?? []).find((candidate) => candidate.id === gateId);
|
||||
const gateCase = (gate?.cases ?? []).find((candidate) => candidate.id === caseId);
|
||||
return gate && gateCase ? { gate, gateCase } : undefined;
|
||||
}
|
||||
|
||||
async function runCompatibilityScenario(root, manifest, scenario, failures, observations) {
|
||||
const referenced = [];
|
||||
for (const caseRef of scenario.caseRefs ?? []) {
|
||||
const found = findGateCase(manifest, caseRef);
|
||||
if (!found) {
|
||||
failures.push(`${scenario.id}: compatibility construction references missing case ${caseRef}`);
|
||||
} else {
|
||||
referenced.push({ caseRef, ...found });
|
||||
}
|
||||
}
|
||||
if (referenced.length !== (scenario.caseRefs ?? []).length) return;
|
||||
|
||||
const sandbox = await mkdtemp(path.join(path.dirname(root), `.gate-compat-${scenario.id}-`));
|
||||
try {
|
||||
await copySandbox(root, sandbox);
|
||||
const environment = {};
|
||||
const writeSignatures = new Map();
|
||||
const removedPaths = new Set();
|
||||
const fixtures = [...referenced.map(({ gateCase }) => gateCase.fixture), scenario.fixture];
|
||||
for (const fixture of fixtures.filter(Boolean)) {
|
||||
for (const entry of fixture.writeFiles ?? []) {
|
||||
const signature = JSON.stringify({ content: entry.content, mode: entry.mode });
|
||||
const previous = writeSignatures.get(entry.path);
|
||||
if (previous !== undefined && previous !== signature) {
|
||||
failures.push(`${scenario.id}: incompatible fixture writes for ${entry.path}`);
|
||||
return;
|
||||
}
|
||||
if (removedPaths.has(entry.path)) {
|
||||
failures.push(`${scenario.id}: fixture both writes and removes ${entry.path}`);
|
||||
return;
|
||||
}
|
||||
writeSignatures.set(entry.path, signature);
|
||||
}
|
||||
for (const removed of fixture.removePaths ?? []) {
|
||||
if (writeSignatures.has(removed)) {
|
||||
failures.push(`${scenario.id}: fixture both writes and removes ${removed}`);
|
||||
return;
|
||||
}
|
||||
removedPaths.add(removed);
|
||||
}
|
||||
await applyFixture(sandbox, fixture);
|
||||
}
|
||||
for (const source of [...referenced.map(({ gateCase }) => gateCase.environment), scenario.environment]) {
|
||||
for (const [key, value] of Object.entries(source ?? {})) {
|
||||
if (environment[key] !== undefined && environment[key] !== value) {
|
||||
failures.push(`${scenario.id}: incompatible environment values for ${key}`);
|
||||
return;
|
||||
}
|
||||
environment[key] = value;
|
||||
}
|
||||
}
|
||||
const expandedEnvironment = Object.fromEntries(
|
||||
Object.entries(environment).map(([key, value]) => [key, expand(value, sandbox)]),
|
||||
);
|
||||
const result = runInvocation(sandbox, scenario.invocation, expandedEnvironment);
|
||||
if (result.error || result.signal || !outcomeMatches(scenario.expected, result)) {
|
||||
failures.push(
|
||||
`${scenario.id}: combined compatibility construction ${scenario.construction} observed ${String(result.status)}${result.signal ? ` signal ${result.signal}` : ''}${result.error ? ` error ${result.error.message}` : ''}; expected ${JSON.stringify(scenario.expected)}`,
|
||||
);
|
||||
} else {
|
||||
observations.push(`COMPATIBILITY ${scenario.id}: observed expected outcome`);
|
||||
}
|
||||
} finally {
|
||||
await rm(sandbox, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
export async function verifyRegistry(options) {
|
||||
const failures = [];
|
||||
const observations = [];
|
||||
const manifestPath = path.resolve(options.root, options.manifest);
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
|
||||
validateStructure(manifest, failures);
|
||||
await validateClaims(options.root, manifest, failures);
|
||||
await validateDiscovery(options.root, manifest, failures);
|
||||
|
||||
for (const gate of manifest.gates ?? []) {
|
||||
await validateDeployment(options.root, gate, failures, observations);
|
||||
for (const gateCase of gate.cases ?? []) {
|
||||
const result = await runCase(options.root, gate, gateCase);
|
||||
const combined = `${result.stdout ?? ''}\n${result.stderr ?? ''}`;
|
||||
if (!outcomeMatches(gateCase.actual, result)) {
|
||||
failures.push(
|
||||
`${gate.id}/${gateCase.id}: observed exit ${String(result.status)}${result.signal ? ` signal ${result.signal}` : ''}${result.error ? ` error ${result.error.message}` : ''} or output disagrees with registry actual ${JSON.stringify(gateCase.actual)}`,
|
||||
);
|
||||
}
|
||||
if (gateCase.reasonPattern && !new RegExp(gateCase.reasonPattern, 'm').test(combined)) {
|
||||
failures.push(`${gate.id}/${gateCase.id}: did not fail for its stated reason`);
|
||||
}
|
||||
if (
|
||||
!structuredValuesEqual(gateCase.required, gateCase.actual) &&
|
||||
gateCase.defect?.owner
|
||||
) {
|
||||
observations.push(
|
||||
`DEFECT (owner: ${gateCase.defect.owner}) ${gate.id}/${gateCase.id}: required ${JSON.stringify(gateCase.required)}, actual ${JSON.stringify(gateCase.actual)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
await validateMutation(options.root, gate, failures, observations);
|
||||
}
|
||||
|
||||
for (const scenario of manifest.compatibilityScenarios ?? []) {
|
||||
await runCompatibilityScenario(options.root, manifest, scenario, failures, observations);
|
||||
}
|
||||
|
||||
return { failures, manifest, observations };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
try {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
const { failures, observations, manifest } = await verifyRegistry(options);
|
||||
if (!options.skipHistory && failures.length === 0) {
|
||||
const history = await verifyHistory({ root: options.root, manifest });
|
||||
failures.push(...history.failures);
|
||||
observations.push(...history.observations);
|
||||
}
|
||||
for (const observation of observations) process.stdout.write(`${observation}\n`);
|
||||
if (failures.length > 0) {
|
||||
for (const failure of failures) process.stderr.write(`GATE VERIFY FAILED: ${failure}\n`);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
const defects = observations.filter((line) => line.startsWith('DEFECT ')).length;
|
||||
process.stdout.write(
|
||||
`registry observations matched; open behavior deltas: ${defects}; required-behavior conformance is not asserted while deltas remain\n`,
|
||||
);
|
||||
} catch (error) {
|
||||
process.stderr.write(`GATE VERIFY FAILED: ${error.message}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === `file://${process.argv[1]}`) await main();
|
||||
@@ -0,0 +1,463 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { chmod, copyFile, mkdir, readFile, rm, symlink, writeFile } from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import test from 'node:test';
|
||||
|
||||
const verifier = path.join(process.cwd(), 'scripts', 'gate-verify.mjs');
|
||||
const fixtureBase = path.join(process.cwd(), '.mosaic-test-work', `gate-verify-${process.pid}`);
|
||||
|
||||
async function fixture(name = 'case') {
|
||||
const root = path.join(fixtureBase, name);
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await mkdir(path.join(root, 'gates'), { recursive: true });
|
||||
return root;
|
||||
}
|
||||
|
||||
function baseManifest() {
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
activationCommit: null,
|
||||
gateRoots: ['gates'],
|
||||
governingClaimFiles: [],
|
||||
coverageBoundary: { included: ['meta fixture'], excluded: [], trackedBy: 'RM-54' },
|
||||
criteria: [
|
||||
{
|
||||
id: 'META-CRIT-1',
|
||||
originalText: 'The fixture rejects its bad input.',
|
||||
currentText: 'The fixture rejects its bad input.',
|
||||
claimType: 'integrity',
|
||||
source: 'fixture',
|
||||
meaningChanges: [],
|
||||
},
|
||||
],
|
||||
compatibilityScenarios: [],
|
||||
proseClaims: [],
|
||||
gates: [
|
||||
{
|
||||
id: 'meta-fixture',
|
||||
source: 'gates/meta-fixture.sh',
|
||||
invocation: ['gates/meta-fixture.sh'],
|
||||
deployment: { kind: 'none', reason: 'test fixture only' },
|
||||
inertMutation: {
|
||||
file: 'gates/meta-fixture.sh',
|
||||
find: 'exit 7',
|
||||
replace: 'exit 0',
|
||||
expected: { exitCode: 0 },
|
||||
},
|
||||
cases: [
|
||||
{
|
||||
id: 'rejects-bad-input',
|
||||
criterionIds: ['META-CRIT-1'],
|
||||
mustFail: true,
|
||||
invocation: ['gates/meta-fixture.sh'],
|
||||
required: { exitCode: 7 },
|
||||
actual: { exitCode: 7 },
|
||||
reasonPattern: 'META_REJECT',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
async function writeGate(root, contents = '#!/bin/sh\necho META_REJECT >&2\nexit 7\n') {
|
||||
const target = path.join(root, 'gates', 'meta-fixture.sh');
|
||||
await writeFile(target, contents);
|
||||
await chmod(target, 0o755);
|
||||
}
|
||||
|
||||
async function writeManifest(root, manifest) {
|
||||
await writeFile(path.join(root, 'gates', 'gates.manifest.json'), `${JSON.stringify(manifest)}\n`);
|
||||
}
|
||||
|
||||
function verify(root) {
|
||||
return spawnSync(
|
||||
process.execPath,
|
||||
[verifier, '--root', root, '--manifest', 'gates/gates.manifest.json', '--skip-history'],
|
||||
{ cwd: root, encoding: 'utf8', env: { ...process.env, HOME: os.homedir() } },
|
||||
);
|
||||
}
|
||||
|
||||
function output(result) {
|
||||
return `${result.stdout}\n${result.stderr}`;
|
||||
}
|
||||
|
||||
test.after(async () => {
|
||||
await rm(fixtureBase, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('an externally inerted failure branch makes verification nonzero and names the gate', async () => {
|
||||
const root = await fixture('external-inert');
|
||||
await writeGate(root, '#!/bin/sh\nexit 0\n');
|
||||
await writeManifest(root, baseManifest());
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture/);
|
||||
});
|
||||
|
||||
test('the verifier applies the declared inert mutation and observes its own control red', async () => {
|
||||
const root = await fixture('internal-meta');
|
||||
await writeGate(root);
|
||||
await writeManifest(root, baseManifest());
|
||||
|
||||
const result = verify(root);
|
||||
assert.equal(result.status, 0, output(result));
|
||||
assert.match(output(result), /META-NEGATIVE-CONTROL.*meta-fixture.*observed red/i);
|
||||
});
|
||||
|
||||
test('a mutation crash is rejected instead of counted as an observed-red control', async () => {
|
||||
const root = await fixture('mutation-crash');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].inertMutation.replace = 'this is not valid shell (';
|
||||
manifest.gates[0].inertMutation.expected = { exitCode: 0 };
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*mutation.*unexpected outcome/i);
|
||||
assert.doesNotMatch(output(result), /META-NEGATIVE-CONTROL.*observed red/i);
|
||||
});
|
||||
|
||||
test('a stale declared mutation case id is rejected instead of falling back', async () => {
|
||||
const root = await fixture('stale-case-id');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].inertMutation.caseId = 'case-that-does-not-exist';
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*case-that-does-not-exist.*not found/i);
|
||||
});
|
||||
|
||||
test('duplicate stable ids and unsupported schema versions are rejected', async () => {
|
||||
const root = await fixture('closed-schema');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.schemaVersion = 99;
|
||||
manifest.criteria.push({ ...manifest.criteria[0] });
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /unsupported schemaVersion 99/i);
|
||||
assert.match(output(result), /duplicate criterion id META-CRIT-1/i);
|
||||
});
|
||||
|
||||
test('manifest-controlled fixture paths cannot escape the sandbox', async () => {
|
||||
const root = await fixture('path-traversal');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].cases[0].fixture = {
|
||||
writeFiles: [{ path: '../../escaped-by-manifest', content: 'bad' }],
|
||||
};
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /path escapes sandbox/i);
|
||||
});
|
||||
|
||||
test('fixture writes reject a final symlink and preserve its outside target', async () => {
|
||||
const root = await fixture('final-symlink');
|
||||
await writeGate(root);
|
||||
const outside = path.join(fixtureBase, 'outside-sentinel');
|
||||
await writeFile(outside, 'preserve-me\n');
|
||||
const linked = path.join(root, 'linked-sentinel');
|
||||
await symlink(outside, linked);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].cases[0].fixture = {
|
||||
writeFiles: [{ path: 'linked-sentinel', content: 'overwritten\n' }],
|
||||
};
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /fixture write.*symbolic link/i);
|
||||
assert.equal(await readFile(outside, 'utf8'), 'preserve-me\n');
|
||||
});
|
||||
|
||||
test('an executable below a gate root without an entry is rejected', async () => {
|
||||
const root = await fixture('unregistered');
|
||||
await writeGate(root);
|
||||
const extra = path.join(root, 'gates', 'forgotten.sh');
|
||||
await writeFile(extra, '#!/bin/sh\nexit 1\n');
|
||||
await chmod(extra, 0o755);
|
||||
await writeManifest(root, baseManifest());
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /unregistered gate.*forgotten\.sh/i);
|
||||
});
|
||||
|
||||
test('a must-fail case without a reason diagnostic is rejected', async () => {
|
||||
const root = await fixture('missing-reason');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].cases[0].reasonPattern = '';
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*rejects-bad-input.*reasonPattern/i);
|
||||
});
|
||||
|
||||
test('a gate with zero must-fail cases is rejected', async () => {
|
||||
const root = await fixture('no-negative');
|
||||
await writeGate(root, '#!/bin/sh\nexit 0\n');
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].inertMutation = {
|
||||
file: 'gates/meta-fixture.sh',
|
||||
find: 'exit 0',
|
||||
replace: 'exit 1',
|
||||
};
|
||||
manifest.gates[0].cases = [
|
||||
{
|
||||
id: 'positive',
|
||||
criterionIds: ['META-CRIT-1'],
|
||||
mustFail: false,
|
||||
invocation: ['gates/meta-fixture.sh'],
|
||||
required: { exitCode: 0 },
|
||||
actual: { exitCode: 0 },
|
||||
reasonPattern: '',
|
||||
},
|
||||
];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*no negative control/i);
|
||||
});
|
||||
|
||||
test('reordered but equivalent outcome fields do not create a false behavior delta', async () => {
|
||||
const root = await fixture('reordered-outcomes');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].cases[0].required = { exitCode: 7, outputPattern: 'META_REJECT' };
|
||||
manifest.gates[0].cases[0].actual = { outputPattern: 'META_REJECT', exitCode: 7 };
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.equal(result.status, 0, output(result));
|
||||
assert.doesNotMatch(output(result), /behavior delta requires|DEFECT \(owner:/);
|
||||
});
|
||||
|
||||
test('a required-versus-actual delta without a tracked owner is rejected', async () => {
|
||||
const root = await fixture('ownerless-delta');
|
||||
await writeGate(root, '#!/bin/sh\nexit 0\n');
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].inertMutation.find = 'exit 0';
|
||||
manifest.gates[0].inertMutation.replace = 'exit 7';
|
||||
manifest.gates[0].cases[0].actual.exitCode = 0;
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*delta.*tracked owner/i);
|
||||
});
|
||||
|
||||
test('a criterion with no bound case is rejected', async () => {
|
||||
const root = await fixture('unbound-criterion');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.criteria.push({
|
||||
id: 'ORPHAN',
|
||||
originalText: 'This criterion is not exercised.',
|
||||
currentText: 'This criterion is not exercised.',
|
||||
claimType: 'quality',
|
||||
source: 'fixture',
|
||||
meaningChanges: [],
|
||||
});
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /ORPHAN.*no bound case/i);
|
||||
});
|
||||
|
||||
test('an unbound governing prose marker is rejected', async () => {
|
||||
const root = await fixture('unbound-prose');
|
||||
await writeGate(root);
|
||||
await mkdir(path.join(root, 'docs'), { recursive: true });
|
||||
await writeFile(path.join(root, 'docs', 'governing.md'), 'GATE-CLAIM:UNBOUND\n');
|
||||
const manifest = baseManifest();
|
||||
manifest.governingClaimFiles = ['docs/governing.md'];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /GATE-CLAIM:UNBOUND.*unbound/i);
|
||||
});
|
||||
|
||||
test('directly contradictory modeled scenarios are rejected', async () => {
|
||||
const root = await fixture('conflict');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.compatibilityScenarios = [
|
||||
{
|
||||
id: 'ONE',
|
||||
construction: 'same-input',
|
||||
caseRefs: ['meta-fixture/rejects-bad-input'],
|
||||
invocation: ['sh', '-c', 'exit 0'],
|
||||
expected: { exitCode: 0 },
|
||||
},
|
||||
{
|
||||
id: 'TWO',
|
||||
construction: 'same-input',
|
||||
caseRefs: ['meta-fixture/rejects-bad-input'],
|
||||
invocation: ['sh', '-c', 'exit 1'],
|
||||
expected: { exitCode: 1 },
|
||||
},
|
||||
];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /ONE.*TWO.*conflict/i);
|
||||
});
|
||||
|
||||
test('compatibility scenarios execute referenced conditions as one construction', async () => {
|
||||
const root = await fixture('combined-compatibility');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].cases.push({
|
||||
id: 'second-condition',
|
||||
criterionIds: ['META-CRIT-1'],
|
||||
mustFail: true,
|
||||
invocation: ['sh', '-c', 'echo "$SECOND_REASON" >&2; exit 7'],
|
||||
fixture: { writeFiles: [{ path: 'conditions/second', content: 'present\n' }] },
|
||||
required: { exitCode: 7 },
|
||||
actual: { exitCode: 7 },
|
||||
reasonPattern: 'SECOND_REASON',
|
||||
environment: { SECOND_REASON: 'SECOND_REASON' },
|
||||
});
|
||||
manifest.gates[0].cases[0].fixture = {
|
||||
writeFiles: [{ path: 'conditions/first', content: 'present\n' }],
|
||||
};
|
||||
manifest.compatibilityScenarios = [
|
||||
{
|
||||
id: 'BOTH-CONDITIONS',
|
||||
construction: 'both-fixtures',
|
||||
caseRefs: ['meta-fixture/rejects-bad-input', 'meta-fixture/second-condition'],
|
||||
invocation: ['sh', '-c', 'test -f conditions/first && test -f conditions/second'],
|
||||
expected: { exitCode: 0 },
|
||||
},
|
||||
];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.equal(result.status, 0, output(result));
|
||||
assert.match(output(result), /COMPATIBILITY BOTH-CONDITIONS: observed expected outcome/i);
|
||||
});
|
||||
|
||||
test('a restated criterion without provenance is rejected', async () => {
|
||||
const root = await fixture('provenance');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.criteria[0].currentText = 'The fixture rejects only malformed input.';
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /META-CRIT-1.*meaning-change provenance/i);
|
||||
});
|
||||
|
||||
test('a security criterion bound only to a positive case is unregistered in substance', async () => {
|
||||
const root = await fixture('positive-only-criterion');
|
||||
await writeGate(root);
|
||||
const manifest = baseManifest();
|
||||
manifest.criteria.push({
|
||||
id: 'POSITIVE-ONLY',
|
||||
originalText: 'A security property.',
|
||||
currentText: 'A security property.',
|
||||
claimType: 'security',
|
||||
source: 'fixture',
|
||||
meaningChanges: [],
|
||||
});
|
||||
manifest.gates[0].cases.push({
|
||||
id: 'positive-only',
|
||||
criterionIds: ['POSITIVE-ONLY'],
|
||||
mustFail: false,
|
||||
invocation: ['gates/meta-fixture.sh'],
|
||||
required: { exitCode: 7 },
|
||||
actual: { exitCode: 7 },
|
||||
reasonPattern: '',
|
||||
});
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /POSITIVE-ONLY.*no must-fail case/i);
|
||||
});
|
||||
|
||||
test('a registered prose claim whose marker is absent is rejected', async () => {
|
||||
const root = await fixture('missing-prose-marker');
|
||||
await writeGate(root);
|
||||
await mkdir(path.join(root, 'docs'), { recursive: true });
|
||||
await writeFile(path.join(root, 'docs', 'governing.md'), 'No marker here.\n');
|
||||
const manifest = baseManifest();
|
||||
manifest.governingClaimFiles = ['docs/governing.md'];
|
||||
manifest.proseClaims = [{ id: 'MISSING-MARKER', criterionId: 'META-CRIT-1' }];
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /GATE-CLAIM:MISSING-MARKER.*missing/i);
|
||||
});
|
||||
|
||||
test('source-versus-deployed byte drift is rejected and names the gate', async () => {
|
||||
const root = await fixture('deployment-drift');
|
||||
await writeGate(root);
|
||||
await mkdir(path.join(root, 'deployed'), { recursive: true });
|
||||
await writeFile(path.join(root, 'deployed', 'meta-fixture.sh'), '#!/bin/sh\nexit 0\n');
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].deployment = { kind: 'file', path: 'deployed/meta-fixture.sh' };
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const result = verify(root);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*source.*deployed.*drift/i);
|
||||
});
|
||||
|
||||
test('deployment drift meta-control fails if the shared comparator is made inert', async () => {
|
||||
const root = await fixture('deployment-comparator-inert');
|
||||
await writeGate(root);
|
||||
await mkdir(path.join(root, 'deployed'), { recursive: true });
|
||||
await copyFile(path.join(root, 'gates', 'meta-fixture.sh'), path.join(root, 'deployed', 'meta-fixture.sh'));
|
||||
const manifest = baseManifest();
|
||||
manifest.gates[0].deployment = { kind: 'file', path: 'deployed/meta-fixture.sh' };
|
||||
await writeManifest(root, manifest);
|
||||
|
||||
const alteredScripts = path.join(root, 'verifier-scripts');
|
||||
await mkdir(alteredScripts, { recursive: true });
|
||||
const verifierSource = await readFile(verifier, 'utf8');
|
||||
const inertSource = verifierSource.replace(
|
||||
'return source.equals(deployed);',
|
||||
'return true; // deliberate test-only inert comparator',
|
||||
);
|
||||
assert.notEqual(inertSource, verifierSource, 'shared deployment comparator mutation went stale');
|
||||
await writeFile(path.join(alteredScripts, 'gate-verify.mjs'), inertSource);
|
||||
await copyFile(
|
||||
path.join(process.cwd(), 'scripts', 'gate-history.mjs'),
|
||||
path.join(alteredScripts, 'gate-history.mjs'),
|
||||
);
|
||||
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
path.join(alteredScripts, 'gate-verify.mjs'),
|
||||
'--root',
|
||||
root,
|
||||
'--manifest',
|
||||
'gates/gates.manifest.json',
|
||||
'--skip-history',
|
||||
],
|
||||
{ cwd: root, encoding: 'utf8', env: { ...process.env, HOME: os.homedir() } },
|
||||
);
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.match(output(result), /meta-fixture.*drift negative control was ineffective/i);
|
||||
});
|
||||
@@ -0,0 +1,22 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import test from 'node:test';
|
||||
|
||||
const root = process.cwd();
|
||||
|
||||
test('package.json exposes the canonical gate:verify command', async () => {
|
||||
const packageJson = JSON.parse(await readFile(`${root}/package.json`, 'utf8'));
|
||||
assert.equal(packageJson.scripts['gate:verify'], 'node scripts/gate-verify.mjs');
|
||||
});
|
||||
|
||||
test('Woodpecker runs gate verification on every pipeline without a path filter', async () => {
|
||||
const pipeline = await readFile(`${root}/.woodpecker/ci.yml`, 'utf8');
|
||||
assert.match(pipeline, /\n gate-verify:\n/);
|
||||
const step =
|
||||
pipeline.match(/\n gate-verify:\n([\s\S]*?)(?=\n [a-z][a-z0-9-]+:|\nservices:)/)?.[1] ?? '';
|
||||
assert.match(
|
||||
step,
|
||||
/commands:\n - \*enable_pnpm\n - apk add --no-cache bubblewrap\n - pnpm gate:verify\n/,
|
||||
);
|
||||
assert.doesNotMatch(step, /\bwhen:|\bpath:/);
|
||||
});
|
||||
Reference in New Issue
Block a user