RM-61: prove ci-postgres teardown discrimination (#1033)

Co-authored-by: coder-mos1 <[email protected]>

docs/scratchpads/1000-rm-61-ci-contract-exemption.md

@@ -0,0 +1,118 @@
+# RM-61 — CI contract exemption for #1000 teardown artifact
+
+**Tracking:** RM-61 / issue #1000
+
+**Branch:** `fix/rm-61-ci-contract-exemption`
+**Owner:** `coder-mos1`
+
+## Objective
+
+Determine, by red-first provider controls, whether the `ci-postgres` pod-not-found teardown signature discriminates from a real PostgreSQL failure. Only if it discriminates may a named, bounded CI-contract exemption be implemented. The exemption must retire when #1000 is fixed; fixing #1000 is the closure path.
+
+## Pre-registered kill criterion
+
+If an injected real `ci-postgres` failure also yields `pods "wp-svc-<ULID>-ci-postgres" not found` as the service's provider-visible failure, the signature does not discriminate. Option B is unsafe; stop exemption implementation and fall to Option A (#1000).
+
+## Plan
+
+1. Capture full `-f json` records for the 11 supplied observations and state counts.
+2. Run one startup-failure control using the real pgvector/PostgreSQL image with an invalid `initdb` argument.
+3. Run one post-readiness crash control using real PostgreSQL, `pg_isready`, and a deliberate postmaster kill while a DB-dependent probe is active.
+4. Compare the raw `ci-postgres` service record independently of failures in dependent steps.
+5. Investigate runner/time/head clustering only as a hypothesis; never encode incidental correlates or retries into policy.
+6. If and only if the controls discriminate, implement and test the exact exemption, document its two-way boundary, and track retirement at #1000.
+
+## Budget
+
+No explicit token cap supplied. Working estimate: 20K–30K tokens. Limit provider controls to the two pre-registered runs; no retries or re-roll policy.
+
+## Initial evidence
+
+Historical JSON saved locally under `.evidence/rm-61/` (not for commit). Supplied pipelines: 11 total. Child-step counts: five pipelines with 9 children and six with 10 children. Seven contain the `ci-postgres` pod-not-found failure (#2170, #2175, #2180, #2181, #2182, #2187, #2188); four do not (#2158, #2167, #2184, #2186). Every observed workflow reports `agent_id=44`, so the available JSON does not separate clean and artifact runs by runner. This refutes runner identity as a discriminator in the sampled record.
+
+## Progress
+
+- [x] Requirements and kill criterion recorded before control implementation.
+- [x] Historical full-JSON records captured.
+- [x] Startup-failure control observed terminal.
+- [x] Post-readiness crash control observed terminal.
+- [x] Discrimination verdict recorded: Option B may proceed.
+- [x] Conditional exemption implementation.
+
+## Tests / evidence
+
+### Control 1 — real startup failure
+
+- Commit: `3931b0e29eb834914f7b17e4db7e221481d436fa`
+- Pipeline: #2189, exact commit match.
+- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
+- `ci-postgres`: `state=failure`, `exit_code=1`, `error=null`, with a five-second execution window.
+- `test`: `state=failure`, `exit_code=1` after the readiness budget expired.
+- Pipeline/workflow: terminal `failure`.
+
+This control is red and its service record differs from #1000 (`exit_code=0` plus pod-not-found). It proves the startup-failure direction only. It does not settle the dangerous post-readiness crash/garbage-collection path.
+
+### Control 2 — real post-readiness crash
+
+- Commit: `25ac59715a94dd1b52ef42577472eb44ecc4b446`
+- Pipeline: #2191, exact commit match.
+- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
+- Service log proves PostgreSQL reached `database system is ready to accept connections`, the test created the arm table, and the service then killed postmaster PID 7.
+- Test log proves a successful `SELECT 1` followed by `Connection refused`; it exited the pre-registered control code 61.
+- `ci-postgres`: `state=failure`, `exit_code=137`, `error=null`, with a 203-second execution window.
+- `test`: `state=failure`, `exit_code=61`.
+- Pipeline/workflow: terminal `failure`.
+
+This is the dangerous post-readiness crash path. Its service record is not pod-not-found and therefore differs from #1000 independently of the dependent test failure.
+
+### Discrimination verdict
+
+Both real failures are provider-visible as process exits (`exit_code=1` startup; `exit_code=137` crash) with no pod-not-found error. The seven observed #1000 artifacts are provider reconciliation misses (`exit_code=0` plus the exact pod-not-found error). The declared kill criterion did not fire, so Option B may proceed with a matcher requiring the full conjunction. This evidence does **not** prove every future Kubernetes failure is distinguishable; it proves these two concrete real-failure classes remain blocking and bounds the exemption to the observed reconciliation shape.
+
+### Unit red-first checkpoint
+
+The nine-case contract harness was written before the verifier. First execution exited 1 because `verify-terminal-green.py` did not exist; no exemption implementation was live. Cases pre-register ordinary green, the exact artifact, both provider controls, near-miss signatures, an independent failure, and a skipped step.
+
+### Control 2 setup attempt — invalid, excluded from evidence
+
+- Commit: `9455cd6a2650b2b7e70f746c07933d96e5cb3d20`
+- Pipeline: #2190, exact commit match.
+- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
+- Service log: `/bin/sh: 0: -c requires an argument`.
+- Root cause: Woodpecker service `commands` did not become the third `sh -c` argument. PostgreSQL never started, so this run is **not** the post-readiness crash control and provides no discrimination evidence.
+- Focused remediation: place the script directly in the third `entrypoint` element and supply `PGPASSWORD` for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict.
+
+## Implementation evidence
+
+- `verify-terminal-green.py` consumes only the full JSON/API record; it performs no fetch, retry, or trigger.
+- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
+- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
+- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
+- Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
+- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
+- Python compile: PASS.
+- `pnpm typecheck`: PASS, 45/45 tasks.
+- `pnpm lint`: PASS, 25/25 tasks.
+- `pnpm format:check`: PASS after moving local evidence outside the repository tree.
+- `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.
+
+## Independent review
+
+- Review 67 / comment 20403 at exact head `e7b29219e11efd0a19395156ac0b154bec0c3a73`: **REQUEST CHANGES**.
+- Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
+- Remediation: require `--expect-commit <full-40>`, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
+- This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
+- Review 69 later approved the commit-binding remediation at exact head `033b2ffb46674b2c0bcc5197273c109b461f62d9`; pipeline #2193 was 9/9 success. Before merge-gate, an independent adjudicator found that Python treats JSON `false == 0`, allowing a non-integer exit value to match. The prior gate-ready state was withdrawn. The type-strict set distinguishes genuine red-first controls (`false`, `0.0`, which wrongly exempted) from regression guards (`true`, `"0"`, `null`, which already blocked). Remediation requires the decoded type to be exactly `int` and excludes `bool` explicitly.
+
+## Documentation checklist
+
+- [x] CI contract documented in the canonical framework CI/CD guide.
+- [x] Operator command documented in the Woodpecker tool README.
+- [x] Merge-gate baseline points to the deterministic verifier and named retirement.
+- [x] Tracking and retirement cite issue #1000.
+- [x] Both positive and negative guarantee boundaries are stated.
+- [x] No API/auth/schema/user-facing navigation change; OpenAPI, user guide, and sitemap are not applicable.
+
+## Risks
+
+The controls establish discrimination for deterministic startup failure and an armed post-readiness postmaster crash on the current Woodpecker Kubernetes provider. They cannot prove that every future Kubernetes failure mode will preserve a non-zero exit before reconciliation. The exact matcher minimizes that residual risk, and issue #1000 remains the mandatory provider-seam closure and retirement trigger.

packages/mosaic/framework/fleet/roles/merge-gate.md

@@ -13,7 +13,14 @@ It is a **gate** role: the one and only merge path.
 2. **Use the wrapped scripts as the ONLY merge path** — the merge-gate merges
    **exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the
    authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green
-   CI before merging). These two scripts are the _only_ sanctioned merge path.
+   CI before merging). Before issuing a verdict, scan the full JSON/API child-step
+   record (including `clone`) with **`verify-terminal-green.py --expect-commit
+<current-provider-PR-head>`** and record the equal expected/observed full-40
+   commits, exact step count, anomalies, and named exemptions. Missing or mismatched
+   commit binding is a hard refusal. The verifier's sole interim
+   exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked
+   by #1000, and retires when #1000 is fixed. These scripts are the _only_
+   sanctioned merge path.
 3. **Never call the raw API** — the merge-gate **does NOT** call `tea`, the raw
    Gitea/forge HTTP API, or any other merge mechanism directly. Only `pr-merge.sh`
    and `pr-ci-wait.sh`.

packages/mosaic/framework/guides/CI-CD-PIPELINES.md

@@ -868,6 +868,38 @@ steps:
 7. **Test on a short-lived non-main branch first** — open a PR and verify quality gates before merging to `main`
 8. **Verify images appear** in Gitea Packages tab after successful pipeline
 
+## Terminal-Green Full-Step Contract
+
+A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`.
+
+```bash
+PR_HEAD=<full-40-hex-provider-head>
+~/.config/mosaic/tools/woodpecker/pipeline-status.sh \
+  -r mosaicstack/stack -n <pipeline-number> -f json \
+  | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py \
+      --expect-commit "$PR_HEAD" -
+```
+
+`PR_HEAD` MUST come from the current provider PR metadata and MUST be the full 40-hex head, not a local branch guess. The verifier fails if the argument is missing, malformed, absent from the pipeline record, or differs from that record.
+
+The verifier reports the expected and observed commits, total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means the commit binding or at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the invocation or JSON input could not be verified.
+
+### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN`
+
+Only this exact conjunction is exempted:
+
+- pipeline and workflow state are `success`;
+- exactly one non-success child exists;
+- its name is `ci-postgres` and type is `service`;
+- its state is `failure`, exit code is the JSON integer `0` (not boolean, float, string, or null); and
+- its error exactly matches `pods "wp-svc-<ULID>-ci-postgres" not found`.
+
+Every near miss remains blocking, including non-zero service exits, startup failures, post-readiness crashes, connection errors, image-pull errors, skipped steps, another failed child, malformed pod names, duplicate matches, or a non-success pipeline/workflow.
+
+**Boundary in both directions:** this exemption recognizes the observed Woodpecker Kubernetes reconciliation miss after an otherwise-successful run. It does not prove that every future PostgreSQL or Kubernetes failure is distinguishable. It does prove, through provider controls, that a deterministic startup failure (`exit_code=1`) and an armed post-readiness postmaster crash (`exit_code=137`, dependent probe `Connection refused`) do not match and remain red.
+
+**Tracking and retirement:** [mosaicstack/stack#1000](https://git.mosaicstack.dev/mosaicstack/stack/issues/1000) owns the provider-seam fix. This exemption MUST be removed when #1000 is fixed. It is not authority to retry or re-trigger a pipeline, and no per-PR re-roll is part of the contract.
+
 ## Post-Merge CI Monitoring (Hard Rule)
 
 For source-code delivery, completion is not allowed at "PR opened" stage.

packages/mosaic/framework/tools/woodpecker/README.md

@@ -26,12 +26,13 @@ A Woodpecker API token is required. To configure:
 
 ## Scripts
 
-| Script                | Purpose                                      |
+| Script                     | Purpose                                                        |
-| --------------------- | -------------------------------------------- |
+| -------------------------- | -------------------------------------------------------------- |
-| `pipeline-list.sh`    | List recent pipelines for a repo             |
+| `pipeline-list.sh`         | List recent pipelines for a repo                               |
-| `pipeline-status.sh`  | Get status of a specific or latest pipeline  |
+| `pipeline-status.sh`       | Get status of a specific or latest pipeline                    |
-| `pipeline-trigger.sh` | Trigger a new pipeline build                 |
+| `pipeline-trigger.sh`      | Trigger a new pipeline build                                   |
-| `ci-wait.sh`          | Block until pipeline(s) reach terminal state |
+| `ci-wait.sh`               | Block until pipeline(s) reach terminal state                   |
+| `verify-terminal-green.py` | Verify every JSON/API child step under the bounded CI contract |
 
 ## Common Options
 
@@ -59,4 +60,9 @@ A Woodpecker API token is required. To configure:
 
 # Block until one or more pipelines finish (event-driven CI wait)
 ~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918
+
+# Verify the full JSON child-step record; do not use the text summary for this gate
+PR_HEAD=<full-40-hex-provider-head>
+~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \
+  | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py --expect-commit "$PR_HEAD" -
 ```

packages/mosaic/framework/tools/woodpecker/test-terminal-green-contract.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+# Red-first contract harness for RM-61 / #1000.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VERIFIER="$SCRIPT_DIR/verify-terminal-green.py"
+EXPECTED_COMMIT=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+write_fixture() {
+    local file="$1" pipeline_status="$2" postgres_state="$3" postgres_exit="$4" postgres_error="$5" test_state="$6"
+    python3 - "$file" "$pipeline_status" "$postgres_state" "$postgres_exit" "$postgres_error" "$test_state" <<'PY'
+import json, sys
+path, pipeline_status, pg_state, pg_exit, pg_error, test_state = sys.argv[1:]
+steps = [
+    {"name": "clone", "type": "clone", "state": "success", "exit_code": 0, "error": None},
+    {"name": "ci-postgres", "type": "service", "state": pg_state, "exit_code": int(pg_exit), "error": pg_error or None},
+    {"name": "test", "type": "commands", "state": test_state, "exit_code": 0 if test_state == "success" else 1, "error": None},
+]
+json.dump({
+    "number": 9999,
+    "status": pipeline_status,
+    "commit": "a" * 40,
+    "workflows": [{"name": "ci", "state": pipeline_status, "children": steps}],
+}, open(path, "w"))
+PY
+}
+
+expect_exit() {
+    local expected_exit="$1" label="$2" file="$3" expected_commit="${4:-$EXPECTED_COMMIT}"
+    set +e
+    output=$(python3 "$VERIFIER" --expect-commit "$expected_commit" "$file" 2>&1)
+    actual=$?
+    set -e
+    if [[ "$actual" -ne "$expected_exit" ]]; then
+        printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected_exit" "$actual" "$output" >&2
+        exit 1
+    fi
+    printf 'PASS %s\n' "$label"
+    printf '%s' "$output"
+}
+
+# Ordinary terminal green.
+write_fixture "$TMP/green.json" success success 0 '' success
+out=$(expect_exit 0 green "$TMP/green.json")
+grep -q '"total_steps": 3' <<<"$out"
+grep -q '"exempted_steps": 0' <<<"$out"
+
+# Exact, named #1000 teardown artifact: the only permitted non-success child.
+artifact='pods "wp-svc-01kyxzjhdf6w81swsnbfzh85z9-ci-postgres" not found'
+write_fixture "$TMP/artifact.json" success failure 0 "$artifact" success
+out=$(expect_exit 0 exact-artifact "$TMP/artifact.json")
+grep -q '"exemption_id": "WP-K8S-1000-CI-POSTGRES-TEARDOWN"' <<<"$out"
+grep -q '"exempted_steps": 1' <<<"$out"
+
+# Negative controls: both real PostgreSQL failures must remain red.
+write_fixture "$TMP/startup.json" failure failure 1 '' failure
+expect_exit 1 startup-failure "$TMP/startup.json" >/dev/null
+write_fixture "$TMP/crash.json" failure failure 137 '' failure
+expect_exit 1 post-readiness-crash "$TMP/crash.json" >/dev/null
+
+# The exemption is signature-scoped, not step-scoped.
+write_fixture "$TMP/wrong-error.json" success failure 0 'connection refused' success
+expect_exit 1 other-postgres-error "$TMP/wrong-error.json" >/dev/null
+write_fixture "$TMP/wrong-pod.json" success failure 0 'pods "other-ci-postgres" not found' success
+expect_exit 1 wrong-pod-signature "$TMP/wrong-pod.json" >/dev/null
+write_fixture "$TMP/nonzero-artifact.json" success failure 137 "$artifact" success
+expect_exit 1 nonzero-with-artifact-text "$TMP/nonzero-artifact.json" >/dev/null
+
+# JSON booleans and non-integer zero look equal to 0 in Python but are not exit codes.
+python3 - "$TMP/artifact.json" "$TMP" <<'PY'
+import json, os, sys
+record = json.load(open(sys.argv[1]))
+for label, value in (("false", False), ("true", True), ("float", 0.0), ("string", "0"), ("null", None)):
+    changed = json.loads(json.dumps(record))
+    changed["workflows"][0]["children"][1]["exit_code"] = value
+    json.dump(changed, open(os.path.join(sys.argv[2], f"exit-{label}.json"), "w"))
+PY
+for label in false true float string null; do
+    expect_exit 1 "non-integer-exit-$label" "$TMP/exit-$label.json" >/dev/null
+done
+
+# Exact artifact cannot mask any independent failure or non-success pipeline.
+write_fixture "$TMP/artifact-plus-failure.json" failure failure 0 "$artifact" failure
+expect_exit 1 artifact-plus-real-failure "$TMP/artifact-plus-failure.json" >/dev/null
+write_fixture "$TMP/skipped.json" success success 0 '' skipped
+expect_exit 1 skipped-step "$TMP/skipped.json" >/dev/null
+
+# The scanned pipeline must be bound to an explicit, full PR-head commit.
+set +e
+missing_output=$(python3 "$VERIFIER" "$TMP/artifact.json" 2>&1)
+missing_rc=$?
+set -e
+if [[ "$missing_rc" -ne 2 ]] || ! grep -q -- '--expect-commit' <<<"$missing_output"; then
+    printf 'FAIL missing-expected-commit: expected usage exit 2\n%s\n' "$missing_output" >&2
+    exit 1
+fi
+expect_exit 1 mismatched-expected-commit "$TMP/artifact.json" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >/dev/null
+
+python3 - "$TMP/artifact.json" "$TMP/missing-record-commit.json" <<'PY'
+import json, sys
+record = json.load(open(sys.argv[1]))
+record.pop("commit")
+json.dump(record, open(sys.argv[2], "w"))
+PY
+expect_exit 1 missing-record-commit "$TMP/missing-record-commit.json" >/dev/null
+
+printf 'terminal-green contract harness: PASS (17 cases)\n'

packages/mosaic/framework/tools/woodpecker/verify-terminal-green.py

@@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+"""Verify Mosaic's full-step Woodpecker terminal-green contract.
+
+RM-61 permits one named, signature-scoped exception for issue #1000. The
+exception retires when #1000 is fixed; all other non-success states block.
+This program consumes the JSON/API record emitted by pipeline-status.sh -f json.
+It does not fetch, retry, or re-trigger pipelines.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+EXEMPTION_ID = "WP-K8S-1000-CI-POSTGRES-TEARDOWN"
+EXEMPTION_ISSUE = "https://git.mosaicstack.dev/mosaicstack/stack/issues/1000"
+POD_NOT_FOUND = re.compile(
+    r'^pods "wp-svc-[0-9a-hjkmnp-tv-z]{26}-ci-postgres" not found$'
+)
+
+
+def fail_usage(message: str) -> int:
+    print(f"terminal-green contract input error: {message}", file=sys.stderr)
+    return 2
+
+
+def load_record(argument: str | None) -> dict[str, Any]:
+    if argument in (None, "-"):
+        value = json.load(sys.stdin)
+    else:
+        with Path(argument).open(encoding="utf-8") as handle:
+            value = json.load(handle)
+    if not isinstance(value, dict):
+        raise ValueError("pipeline record must be a JSON object")
+    return value
+
+
+def is_issue_1000_artifact(step: dict[str, Any]) -> bool:
+    error = step.get("error")
+    exit_code = step.get("exit_code")
+    return (
+        step.get("name") == "ci-postgres"
+        and step.get("type") == "service"
+        and step.get("state") == "failure"
+        and type(exit_code) is int
+        and not isinstance(exit_code, bool)
+        and exit_code == 0
+        and isinstance(error, str)
+        and POD_NOT_FOUND.fullmatch(error) is not None
+    )
+
+
+def verify(record: dict[str, Any], expected_commit: str) -> tuple[int, dict[str, Any]]:
+    anomalies: list[dict[str, Any]] = []
+    candidates: list[dict[str, Any]] = []
+    steps: list[dict[str, Any]] = []
+
+    pipeline_status = record.get("status")
+    actual_commit = record.get("commit")
+    if actual_commit != expected_commit:
+        anomalies.append(
+            {
+                "scope": "pipeline",
+                "name": str(record.get("number", "unknown")),
+                "state": pipeline_status,
+                "reason": "pipeline commit does not equal the expected PR head",
+                "expected_commit": expected_commit,
+                "actual_commit": actual_commit,
+            }
+        )
+    if pipeline_status != "success":
+        anomalies.append(
+            {
+                "scope": "pipeline",
+                "name": str(record.get("number", "unknown")),
+                "state": pipeline_status,
+                "reason": "pipeline status is not success",
+            }
+        )
+
+    workflows = record.get("workflows")
+    if not isinstance(workflows, list) or not workflows:
+        anomalies.append(
+            {
+                "scope": "pipeline",
+                "name": str(record.get("number", "unknown")),
+                "state": pipeline_status,
+                "reason": "workflows are missing or empty",
+            }
+        )
+        workflows = []
+
+    for workflow_index, workflow in enumerate(workflows):
+        if not isinstance(workflow, dict):
+            anomalies.append(
+                {
+                    "scope": "workflow",
+                    "name": str(workflow_index),
+                    "state": None,
+                    "reason": "workflow is not an object",
+                }
+            )
+            continue
+        workflow_name = str(workflow.get("name", workflow_index))
+        if workflow.get("state") != "success":
+            anomalies.append(
+                {
+                    "scope": "workflow",
+                    "name": workflow_name,
+                    "state": workflow.get("state"),
+                    "reason": "workflow state is not success",
+                }
+            )
+        children = workflow.get("children")
+        if not isinstance(children, list) or not children:
+            anomalies.append(
+                {
+                    "scope": "workflow",
+                    "name": workflow_name,
+                    "state": workflow.get("state"),
+                    "reason": "child-step list is missing or empty",
+                }
+            )
+            continue
+        for child_index, child in enumerate(children):
+            if not isinstance(child, dict):
+                anomalies.append(
+                    {
+                        "scope": "step",
+                        "name": f"{workflow_name}[{child_index}]",
+                        "state": None,
+                        "reason": "step is not an object",
+                    }
+                )
+                continue
+            steps.append(child)
+            if child.get("state") == "success":
+                continue
+            if is_issue_1000_artifact(child):
+                candidates.append(child)
+                continue
+            anomalies.append(
+                {
+                    "scope": "step",
+                    "name": child.get("name"),
+                    "type": child.get("type"),
+                    "state": child.get("state"),
+                    "exit_code": child.get("exit_code"),
+                    "error": child.get("error"),
+                    "reason": "non-success step does not match the #1000 teardown signature",
+                }
+            )
+
+    if len(candidates) > 1:
+        anomalies.append(
+            {
+                "scope": "exemption",
+                "name": EXEMPTION_ID,
+                "state": "invalid",
+                "reason": "the #1000 exemption may apply to exactly one step",
+            }
+        )
+
+    exemption_applies = len(candidates) == 1 and not anomalies
+    state_counts = Counter(str(step.get("state", "missing")) for step in steps)
+    result: dict[str, Any] = {
+        "schema_version": "mosaic-terminal-green/v1",
+        "verdict": "terminal-green" if not anomalies else "not-terminal-green",
+        "pipeline_number": record.get("number"),
+        "commit": actual_commit,
+        "expected_commit": expected_commit,
+        "pipeline_status": pipeline_status,
+        "total_steps": len(steps),
+        "state_counts": dict(sorted(state_counts.items())),
+        "exempted_steps": 1 if exemption_applies else 0,
+        "anomalies": anomalies,
+    }
+    if exemption_applies:
+        candidate = candidates[0]
+        result["exemptions"] = [
+            {
+                "exemption_id": EXEMPTION_ID,
+                "step": candidate.get("name"),
+                "signature": candidate.get("error"),
+                "tracking_issue": EXEMPTION_ISSUE,
+                "retires_when": "issue #1000 is fixed",
+            }
+        ]
+    else:
+        result["exemptions"] = []
+
+    return (0 if not anomalies else 1), result
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="verify the full Woodpecker terminal-green child-step contract"
+    )
+    parser.add_argument(
+        "--expect-commit",
+        required=True,
+        metavar="FULL_SHA",
+        help="full 40-hex PR-head commit that the pipeline record must match",
+    )
+    parser.add_argument("record", nargs="?", default="-", help="pipeline JSON file or -")
+    arguments = parser.parse_args()
+    if re.fullmatch(r"[0-9a-fA-F]{40}", arguments.expect_commit) is None:
+        parser.error("--expect-commit must be a full 40-hex commit")
+    arguments.expect_commit = arguments.expect_commit.lower()
+    return arguments
+
+
+def main() -> int:
+    arguments = parse_arguments()
+    try:
+        record = load_record(arguments.record)
+    except (OSError, ValueError, json.JSONDecodeError) as error:
+        return fail_usage(str(error))
+    code, result = verify(record, arguments.expect_commit)
+    print(json.dumps(result, indent=2, sort_keys=True))
+    return code
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

packages/mosaic/package.json

@@ -25,7 +25,7 @@
     "lint": "eslint src",
     "typecheck": "tsc --noEmit",
     "test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
-    "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
+    "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
   },
   "dependencies": {
     "@mosaicstack/brain": "workspace:*",