fix(ci): bind terminal-green evidence to PR head
ci/woodpecker/pr/ci Pipeline was successful

This commit is contained in:
2026-08-01 08:53:36 -05:00
parent e7b29219e1
commit 033b2ffb46
6 changed files with 82 additions and 19 deletions
@@ -88,7 +88,7 @@ The nine-case contract harness was written before the verifier. First execution
- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
- Unit harness: 9/9 cases pass after the red-first checkpoint.
- Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
- Python compile: PASS.
- `pnpm typecheck`: PASS, 45/45 tasks.
@@ -96,6 +96,13 @@ The nine-case contract harness was written before the verifier. First execution
- `pnpm format:check`: PASS after moving local evidence outside the repository tree.
- `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.
## Independent review
- Review 67 / comment 20403 at exact head `e7b29219e11efd0a19395156ac0b154bec0c3a73`: **REQUEST CHANGES**.
- Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
- Remediation: require `--expect-commit <full-40>`, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
- This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
## Documentation checklist
- [x] CI contract documented in the canonical framework CI/CD guide.
@@ -14,8 +14,10 @@ It is a **gate** role: the one and only merge path.
**exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the
authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green
CI before merging). Before issuing a verdict, scan the full JSON/API child-step
record (including `clone`) with **`verify-terminal-green.py`** and record its
exact step count, anomalies, and named exemptions. The verifier's sole interim
record (including `clone`) with **`verify-terminal-green.py --expect-commit
<current-provider-PR-head>`** and record the equal expected/observed full-40
commits, exact step count, anomalies, and named exemptions. Missing or mismatched
commit binding is a hard refusal. The verifier's sole interim
exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked
by #1000, and retires when #1000 is fixed. These scripts are the _only_
sanctioned merge path.
@@ -873,12 +873,16 @@ steps:
A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`.
```bash
PR_HEAD=<full-40-hex-provider-head>
~/.config/mosaic/tools/woodpecker/pipeline-status.sh \
-r mosaicstack/stack -n <pipeline-number> -f json \
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py -
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py \
--expect-commit "$PR_HEAD" -
```
The verifier reports the total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the JSON input could not be verified.
`PR_HEAD` MUST come from the current provider PR metadata and MUST be the full 40-hex head, not a local branch guess. The verifier fails if the argument is missing, malformed, absent from the pipeline record, or differs from that record.
The verifier reports the expected and observed commits, total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means the commit binding or at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the invocation or JSON input could not be verified.
### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN`
@@ -62,6 +62,7 @@ A Woodpecker API token is required. To configure:
~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918
# Verify the full JSON child-step record; do not use the text summary for this gate
PR_HEAD=<full-40-hex-provider-head>
~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py -
| ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py --expect-commit "$PR_HEAD" -
```
@@ -4,6 +4,7 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VERIFIER="$SCRIPT_DIR/verify-terminal-green.py"
EXPECTED_COMMIT=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
@@ -27,14 +28,13 @@ PY
}
expect_exit() {
local expected="$1" label="$2" file="$3"
shift 3
local expected_exit="$1" label="$2" file="$3" expected_commit="${4:-$EXPECTED_COMMIT}"
set +e
output=$(python3 "$VERIFIER" "$file" "$@" 2>&1)
output=$(python3 "$VERIFIER" --expect-commit "$expected_commit" "$file" 2>&1)
actual=$?
set -e
if [[ "$actual" -ne "$expected" ]]; then
printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected" "$actual" "$output" >&2
if [[ "$actual" -ne "$expected_exit" ]]; then
printf 'FAIL %s: expected exit %s, got %s\n%s\n' "$label" "$expected_exit" "$actual" "$output" >&2
exit 1
fi
printf 'PASS %s\n' "$label"
@@ -74,4 +74,23 @@ expect_exit 1 artifact-plus-real-failure "$TMP/artifact-plus-failure.json" >/dev
write_fixture "$TMP/skipped.json" success success 0 '' skipped
expect_exit 1 skipped-step "$TMP/skipped.json" >/dev/null
printf 'terminal-green contract harness: PASS (9 cases)\n'
# The scanned pipeline must be bound to an explicit, full PR-head commit.
set +e
missing_output=$(python3 "$VERIFIER" "$TMP/artifact.json" 2>&1)
missing_rc=$?
set -e
if [[ "$missing_rc" -ne 2 ]] || ! grep -q -- '--expect-commit' <<<"$missing_output"; then
printf 'FAIL missing-expected-commit: expected usage exit 2\n%s\n' "$missing_output" >&2
exit 1
fi
expect_exit 1 mismatched-expected-commit "$TMP/artifact.json" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb >/dev/null
python3 - "$TMP/artifact.json" "$TMP/missing-record-commit.json" <<'PY'
import json, sys
record = json.load(open(sys.argv[1]))
record.pop("commit")
json.dump(record, open(sys.argv[2], "w"))
PY
expect_exit 1 missing-record-commit "$TMP/missing-record-commit.json" >/dev/null
printf 'terminal-green contract harness: PASS (12 cases)\n'
@@ -9,6 +9,7 @@ It does not fetch, retry, or re-trigger pipelines.
from __future__ import annotations
import argparse
import json
import re
import sys
@@ -51,12 +52,24 @@ def is_issue_1000_artifact(step: dict[str, Any]) -> bool:
)
def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]:
def verify(record: dict[str, Any], expected_commit: str) -> tuple[int, dict[str, Any]]:
anomalies: list[dict[str, Any]] = []
candidates: list[dict[str, Any]] = []
steps: list[dict[str, Any]] = []
pipeline_status = record.get("status")
actual_commit = record.get("commit")
if actual_commit != expected_commit:
anomalies.append(
{
"scope": "pipeline",
"name": str(record.get("number", "unknown")),
"state": pipeline_status,
"reason": "pipeline commit does not equal the expected PR head",
"expected_commit": expected_commit,
"actual_commit": actual_commit,
}
)
if pipeline_status != "success":
anomalies.append(
{
@@ -156,7 +169,8 @@ def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]:
"schema_version": "mosaic-terminal-green/v1",
"verdict": "terminal-green" if not anomalies else "not-terminal-green",
"pipeline_number": record.get("number"),
"commit": record.get("commit"),
"commit": actual_commit,
"expected_commit": expected_commit,
"pipeline_status": pipeline_status,
"total_steps": len(steps),
"state_counts": dict(sorted(state_counts.items())),
@@ -180,15 +194,31 @@ def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]:
return (0 if not anomalies else 1), result
def parse_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="verify the full Woodpecker terminal-green child-step contract"
)
parser.add_argument(
"--expect-commit",
required=True,
metavar="FULL_SHA",
help="full 40-hex PR-head commit that the pipeline record must match",
)
parser.add_argument("record", nargs="?", default="-", help="pipeline JSON file or -")
arguments = parser.parse_args()
if re.fullmatch(r"[0-9a-fA-F]{40}", arguments.expect_commit) is None:
parser.error("--expect-commit must be a full 40-hex commit")
arguments.expect_commit = arguments.expect_commit.lower()
return arguments
def main() -> int:
if len(sys.argv) > 2 or (len(sys.argv) == 2 and sys.argv[1] in {"-h", "--help"}):
print(f"usage: {Path(sys.argv[0]).name} [pipeline.json|-]", file=sys.stderr)
return 0 if len(sys.argv) == 2 else 2
arguments = parse_arguments()
try:
record = load_record(sys.argv[1] if len(sys.argv) == 2 else None)
record = load_record(arguments.record)
except (OSError, ValueError, json.JSONDecodeError) as error:
return fail_usage(str(error))
code, result = verify(record)
code, result = verify(record, arguments.expect_commit)
print(json.dumps(result, indent=2, sort_keys=True))
return code