fix(ci): bind terminal-green evidence to PR head
ci/woodpecker/pr/ci Pipeline was successful

This commit is contained in:
2026-08-01 08:53:36 -05:00
parent e7b29219e1
commit 033b2ffb46
6 changed files with 82 additions and 19 deletions
@@ -88,7 +88,7 @@ The nine-case contract harness was written before the verifier. First execution
- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption. - Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist. - Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted. - Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
- Unit harness: 9/9 cases pass after the red-first checkpoint. - Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present. - Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
- Python compile: PASS. - Python compile: PASS.
- `pnpm typecheck`: PASS, 45/45 tasks. - `pnpm typecheck`: PASS, 45/45 tasks.
@@ -96,6 +96,13 @@ The nine-case contract harness was written before the verifier. First execution
- `pnpm format:check`: PASS after moving local evidence outside the repository tree. - `pnpm format:check`: PASS after moving local evidence outside the repository tree.
- `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed. - `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.
## Independent review
- Review 67 / comment 20403 at exact head `e7b29219e11efd0a19395156ac0b154bec0c3a73`: **REQUEST CHANGES**.
- Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
- Remediation: require `--expect-commit <full-40>`, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
- This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
## Documentation checklist ## Documentation checklist
- [x] CI contract documented in the canonical framework CI/CD guide. - [x] CI contract documented in the canonical framework CI/CD guide.
@@ -14,8 +14,10 @@ It is a **gate** role: the one and only merge path.
**exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the **exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the
authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green
CI before merging). Before issuing a verdict, scan the full JSON/API child-step CI before merging). Before issuing a verdict, scan the full JSON/API child-step
record (including `clone`) with **`verify-terminal-green.py`** and record its record (including `clone`) with **`verify-terminal-green.py --expect-commit
exact step count, anomalies, and named exemptions. The verifier's sole interim <current-provider-PR-head>`** and record the equal expected/observed full-40
commits, exact step count, anomalies, and named exemptions. Missing or mismatched
commit binding is a hard refusal. The verifier's sole interim
exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked
by #1000, and retires when #1000 is fixed. These scripts are the _only_ by #1000, and retires when #1000 is fixed. These scripts are the _only_
sanctioned merge path. sanctioned merge path.
@@ -873,12 +873,16 @@ steps:
A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`. A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`.
```bash ```bash
PR_HEAD=<full-40-hex-provider-head>
~/.config/mosaic/tools/woodpecker/pipeline-status.sh \ ~/.config/mosaic/tools/woodpecker/pipeline-status.sh \
-r mosaicstack/stack -n <pipeline-number> -f json \ -r mosaicstack/stack -n <pipeline-number> -f json \
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py - | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py \
--expect-commit "$PR_HEAD" -
``` ```
The verifier reports the total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the JSON input could not be verified. `PR_HEAD` MUST come from the current provider PR metadata and MUST be the full 40-hex head, not a local branch guess. The verifier fails if the argument is missing, malformed, absent from the pipeline record, or differs from that record.
The verifier reports the expected and observed commits, total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means the commit binding or at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the invocation or JSON input could not be verified.
### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN` ### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN`
@@ -62,6 +62,7 @@ A Woodpecker API token is required. To configure:
~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918 ~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918
# Verify the full JSON child-step record; do not use the text summary for this gate # Verify the full JSON child-step record; do not use the text summary for this gate
PR_HEAD=<full-40-hex-provider-head>
~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \ ~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py - | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py --expect-commit "$PR_HEAD" -
``` ```
@@ -4,6 +4,7 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VERIFIER="$SCRIPT_DIR/verify-terminal-green.py" VERIFIER="$SCRIPT_DIR/verify-terminal-green.py"
EXPECTED_COMMIT=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
TMP=$(mktemp -d) TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT trap 'rm -rf "$TMP"' EXIT
@@ -27,14 +28,13 @@ PY
} }
expect_exit() { expect_exit() {
local expected="$1" label="$2" file="$3" local expected_exit="$1" label="$2" file="$3" expected_commit="${4:-$EXPECTED_COMMIT}"
shift 3
set +e set +e
output=$(python3 "$VERIFIER" "$file" "$@" 2>&1) output=$(python3 "$VERIFIER" --expect-commit "$expected_commit" "$file" 2>&1)
actual=$? actual=$?
set -e set -e
if [[ "$actual" -ne "$expected" ]]; then if [[ "$actual" -ne "$expected_exit" ]]; then
printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected" "$actual" "$output" >&2 printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected_exit" "$actual" "$output" >&2
exit 1 exit 1
fi fi
printf 'PASS %s\n' "$label" printf 'PASS %s\n' "$label"
@@ -74,4 +74,23 @@ expect_exit 1 artifact-plus-real-failure "$TMP/artifact-plus-failure.json" >/dev
write_fixture "$TMP/skipped.json" success success 0 '' skipped write_fixture "$TMP/skipped.json" success success 0 '' skipped
expect_exit 1 skipped-step "$TMP/skipped.json" >/dev/null expect_exit 1 skipped-step "$TMP/skipped.json" >/dev/null
printf 'terminal-green contract harness: PASS (9 cases)\n' # The scanned pipeline must be bound to an explicit, full PR-head commit.
set +e
missing_output=$(python3 "$VERIFIER" "$TMP/artifact.json" 2>&1)
missing_rc=$?
set -e
if [[ "$missing_rc" -ne 2 ]] || ! grep -q -- '--expect-commit' <<<"$missing_output"; then
printf 'FAIL missing-expected-commit: expected usage exit 2\n%s\n' "$missing_output" >&2
exit 1
fi
expect_exit 1 mismatched-expected-commit "$TMP/artifact.json" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >/dev/null
python3 - "$TMP/artifact.json" "$TMP/missing-record-commit.json" <<'PY'
import json, sys
record = json.load(open(sys.argv[1]))
record.pop("commit")
json.dump(record, open(sys.argv[2], "w"))
PY
expect_exit 1 missing-record-commit "$TMP/missing-record-commit.json" >/dev/null
printf 'terminal-green contract harness: PASS (12 cases)\n'
@@ -9,6 +9,7 @@ It does not fetch, retry, or re-trigger pipelines.
from __future__ import annotations from __future__ import annotations
import argparse
import json import json
import re import re
import sys import sys
@@ -51,12 +52,24 @@ def is_issue_1000_artifact(step: dict[str, Any]) -> bool:
) )
def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]: def verify(record: dict[str, Any], expected_commit: str) -> tuple[int, dict[str, Any]]:
anomalies: list[dict[str, Any]] = [] anomalies: list[dict[str, Any]] = []
candidates: list[dict[str, Any]] = [] candidates: list[dict[str, Any]] = []
steps: list[dict[str, Any]] = [] steps: list[dict[str, Any]] = []
pipeline_status = record.get("status") pipeline_status = record.get("status")
actual_commit = record.get("commit")
if actual_commit != expected_commit:
anomalies.append(
{
"scope": "pipeline",
"name": str(record.get("number", "unknown")),
"state": pipeline_status,
"reason": "pipeline commit does not equal the expected PR head",
"expected_commit": expected_commit,
"actual_commit": actual_commit,
}
)
if pipeline_status != "success": if pipeline_status != "success":
anomalies.append( anomalies.append(
{ {
@@ -156,7 +169,8 @@ def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]:
"schema_version": "mosaic-terminal-green/v1", "schema_version": "mosaic-terminal-green/v1",
"verdict": "terminal-green" if not anomalies else "not-terminal-green", "verdict": "terminal-green" if not anomalies else "not-terminal-green",
"pipeline_number": record.get("number"), "pipeline_number": record.get("number"),
"commit": record.get("commit"), "commit": actual_commit,
"expected_commit": expected_commit,
"pipeline_status": pipeline_status, "pipeline_status": pipeline_status,
"total_steps": len(steps), "total_steps": len(steps),
"state_counts": dict(sorted(state_counts.items())), "state_counts": dict(sorted(state_counts.items())),
@@ -180,15 +194,31 @@ def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]:
return (0 if not anomalies else 1), result return (0 if not anomalies else 1), result
def parse_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="verify the full Woodpecker terminal-green child-step contract"
)
parser.add_argument(
"--expect-commit",
required=True,
metavar="FULL_SHA",
help="full 40-hex PR-head commit that the pipeline record must match",
)
parser.add_argument("record", nargs="?", default="-", help="pipeline JSON file or -")
arguments = parser.parse_args()
if re.fullmatch(r"[0-9a-fA-F]{40}", arguments.expect_commit) is None:
parser.error("--expect-commit must be a full 40-hex commit")
arguments.expect_commit = arguments.expect_commit.lower()
return arguments
def main() -> int: def main() -> int:
if len(sys.argv) > 2 or (len(sys.argv) == 2 and sys.argv[1] in {"-h", "--help"}): arguments = parse_arguments()
print(f"usage: {Path(sys.argv[0]).name} [pipeline.json|-]", file=sys.stderr)
return 0 if len(sys.argv) == 2 else 2
try: try:
record = load_record(sys.argv[1] if len(sys.argv) == 2 else None) record = load_record(arguments.record)
except (OSError, ValueError, json.JSONDecodeError) as error: except (OSError, ValueError, json.JSONDecodeError) as error:
return fail_usage(str(error)) return fail_usage(str(error))
code, result = verify(record) code, result = verify(record, arguments.expect_commit)
print(json.dumps(result, indent=2, sort_keys=True)) print(json.dumps(result, indent=2, sort_keys=True))
return code return code