diff --git a/docs/scratchpads/1000-rm-61-ci-contract-exemption.md b/docs/scratchpads/1000-rm-61-ci-contract-exemption.md index 2818566a..bef73dd3 100644 --- a/docs/scratchpads/1000-rm-61-ci-contract-exemption.md +++ b/docs/scratchpads/1000-rm-61-ci-contract-exemption.md @@ -36,7 +36,7 @@ Historical JSON saved locally under `.evidence/rm-61/` (not for commit). Supplie - [x] Startup-failure control observed terminal. - [x] Post-readiness crash control observed terminal. - [x] Discrimination verdict recorded: Option B may proceed. -- [ ] Conditional exemption implementation. +- [x] Conditional exemption implementation. ## Tests / evidence @@ -81,6 +81,26 @@ The nine-case contract harness was written before the verifier. First execution - Root cause: Woodpecker service `commands` did not become the third `sh -c` argument. PostgreSQL never started, so this run is **not** the post-readiness crash control and provides no discrimination evidence. - Focused remediation: place the script directly in the third `entrypoint` element and supply `PGPASSWORD` for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict. +## Implementation evidence + +- `verify-terminal-green.py` consumes only the full JSON/API record; it performs no fetch, retry, or trigger. +- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption. +- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist. +- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted. +- Unit harness: 9/9 cases pass after the red-first checkpoint. +- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present. +- Python compile: PASS. +- Changed-file Prettier check: PASS after formatting the Woodpecker README. + +## Documentation checklist + +- [x] CI contract documented in the canonical framework CI/CD guide. +- [x] Operator command documented in the Woodpecker tool README. +- [x] Merge-gate baseline points to the deterministic verifier and named retirement. +- [x] Tracking and retirement cite issue #1000. +- [x] Both positive and negative guarantee boundaries are stated. +- [x] No API/auth/schema/user-facing navigation change; OpenAPI, user guide, and sitemap are not applicable. + ## Risks -The Woodpecker Kubernetes backend may garbage-collect both genuinely failed and successfully used service pods before reconciliation. If so, the provider-visible service signature cannot safely support Option B. +The controls establish discrimination for deterministic startup failure and an armed post-readiness postmaster crash on the current Woodpecker Kubernetes provider. They cannot prove that every future Kubernetes failure mode will preserve a non-zero exit before reconciliation. The exact matcher minimizes that residual risk, and issue #1000 remains the mandatory provider-seam closure and retirement trigger. diff --git a/packages/mosaic/framework/fleet/roles/merge-gate.md b/packages/mosaic/framework/fleet/roles/merge-gate.md index 7227084a..77b0759b 100644 --- a/packages/mosaic/framework/fleet/roles/merge-gate.md +++ b/packages/mosaic/framework/fleet/roles/merge-gate.md @@ -13,7 +13,12 @@ It is a **gate** role: the one and only merge path. 2. **Use the wrapped scripts as the ONLY merge path** — the merge-gate merges **exclusively** by calling **`pr-merge.sh`** (the merge action, which carries the authoritative forbidden-path guard) and **`pr-ci-wait.sh`** (to wait for green - CI before merging). These two scripts are the _only_ sanctioned merge path. + CI before merging). Before issuing a verdict, scan the full JSON/API child-step + record (including `clone`) with **`verify-terminal-green.py`** and record its + exact step count, anomalies, and named exemptions. The verifier's sole interim + exemption is `WP-K8S-1000-CI-POSTGRES-TEARDOWN`; it is signature-scoped, tracked + by #1000, and retires when #1000 is fixed. These scripts are the _only_ + sanctioned merge path. 3. **Never call the raw API** — the merge-gate **does NOT** call `tea`, the raw Gitea/forge HTTP API, or any other merge mechanism directly. Only `pr-merge.sh` and `pr-ci-wait.sh`. diff --git a/packages/mosaic/framework/guides/CI-CD-PIPELINES.md b/packages/mosaic/framework/guides/CI-CD-PIPELINES.md index 3766b14c..91802c13 100644 --- a/packages/mosaic/framework/guides/CI-CD-PIPELINES.md +++ b/packages/mosaic/framework/guides/CI-CD-PIPELINES.md @@ -868,6 +868,34 @@ steps: 7. **Test on a short-lived non-main branch first** — open a PR and verify quality gates before merging to `main` 8. **Verify images appear** in Gitea Packages tab after successful pipeline +## Terminal-Green Full-Step Contract + +A successful pipeline summary is not sufficient: verification MUST consume the full JSON/API child-step record, including `clone`. + +```bash +~/.config/mosaic/tools/woodpecker/pipeline-status.sh \ + -r mosaicstack/stack -n -f json \ + | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py - +``` + +The verifier reports the total step count, state counts, anomalies, and any applied exemption. Exit `0` means the record satisfies the contract; exit `1` means at least one pipeline, workflow, or child-step state blocks terminal-green; exit `2` means the JSON input could not be verified. + +### Named interim exemption: `WP-K8S-1000-CI-POSTGRES-TEARDOWN` + +Only this exact conjunction is exempted: + +- pipeline and workflow state are `success`; +- exactly one non-success child exists; +- its name is `ci-postgres` and type is `service`; +- its state is `failure`, exit code is `0`; and +- its error exactly matches `pods "wp-svc--ci-postgres" not found`. + +Every near miss remains blocking, including non-zero service exits, startup failures, post-readiness crashes, connection errors, image-pull errors, skipped steps, another failed child, malformed pod names, duplicate matches, or a non-success pipeline/workflow. + +**Boundary in both directions:** this exemption recognizes the observed Woodpecker Kubernetes reconciliation miss after an otherwise-successful run. It does not prove that every future PostgreSQL or Kubernetes failure is distinguishable. It does prove, through provider controls, that a deterministic startup failure (`exit_code=1`) and an armed post-readiness postmaster crash (`exit_code=137`, dependent probe `Connection refused`) do not match and remain red. + +**Tracking and retirement:** [mosaicstack/stack#1000](https://git.mosaicstack.dev/mosaicstack/stack/issues/1000) owns the provider-seam fix. This exemption MUST be removed when #1000 is fixed. It is not authority to retry or re-trigger a pipeline, and no per-PR re-roll is part of the contract. + ## Post-Merge CI Monitoring (Hard Rule) For source-code delivery, completion is not allowed at "PR opened" stage. diff --git a/packages/mosaic/framework/tools/woodpecker/README.md b/packages/mosaic/framework/tools/woodpecker/README.md index 7e7a614a..80a4813a 100644 --- a/packages/mosaic/framework/tools/woodpecker/README.md +++ b/packages/mosaic/framework/tools/woodpecker/README.md @@ -26,12 +26,13 @@ A Woodpecker API token is required. To configure: ## Scripts -| Script | Purpose | -| --------------------- | -------------------------------------------- | -| `pipeline-list.sh` | List recent pipelines for a repo | -| `pipeline-status.sh` | Get status of a specific or latest pipeline | -| `pipeline-trigger.sh` | Trigger a new pipeline build | -| `ci-wait.sh` | Block until pipeline(s) reach terminal state | +| Script | Purpose | +| -------------------------- | -------------------------------------------------------------- | +| `pipeline-list.sh` | List recent pipelines for a repo | +| `pipeline-status.sh` | Get status of a specific or latest pipeline | +| `pipeline-trigger.sh` | Trigger a new pipeline build | +| `ci-wait.sh` | Block until pipeline(s) reach terminal state | +| `verify-terminal-green.py` | Verify every JSON/API child step under the bounded CI contract | ## Common Options @@ -59,4 +60,8 @@ A Woodpecker API token is required. To configure: # Block until one or more pipelines finish (event-driven CI wait) ~/.config/mosaic/tools/woodpecker/ci-wait.sh -r usc/uconnect -n 3917 -n 3918 + +# Verify the full JSON child-step record; do not use the text summary for this gate +~/.config/mosaic/tools/woodpecker/pipeline-status.sh -r mosaicstack/stack -n 2188 -f json \ + | ~/.config/mosaic/tools/woodpecker/verify-terminal-green.py - ``` diff --git a/packages/mosaic/framework/tools/woodpecker/verify-terminal-green.py b/packages/mosaic/framework/tools/woodpecker/verify-terminal-green.py new file mode 100644 index 00000000..f5677aff --- /dev/null +++ b/packages/mosaic/framework/tools/woodpecker/verify-terminal-green.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Verify Mosaic's full-step Woodpecker terminal-green contract. + +RM-61 permits one named, signature-scoped exception for issue #1000. The +exception retires when #1000 is fixed; all other non-success states block. +This program consumes the JSON/API record emitted by pipeline-status.sh -f json. +It does not fetch, retry, or re-trigger pipelines. +""" + +from __future__ import annotations + +import json +import re +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +EXEMPTION_ID = "WP-K8S-1000-CI-POSTGRES-TEARDOWN" +EXEMPTION_ISSUE = "https://git.mosaicstack.dev/mosaicstack/stack/issues/1000" +POD_NOT_FOUND = re.compile( + r'^pods "wp-svc-[0-9a-hjkmnp-tv-z]{26}-ci-postgres" not found$' +) + + +def fail_usage(message: str) -> int: + print(f"terminal-green contract input error: {message}", file=sys.stderr) + return 2 + + +def load_record(argument: str | None) -> dict[str, Any]: + if argument in (None, "-"): + value = json.load(sys.stdin) + else: + with Path(argument).open(encoding="utf-8") as handle: + value = json.load(handle) + if not isinstance(value, dict): + raise ValueError("pipeline record must be a JSON object") + return value + + +def is_issue_1000_artifact(step: dict[str, Any]) -> bool: + error = step.get("error") + return ( + step.get("name") == "ci-postgres" + and step.get("type") == "service" + and step.get("state") == "failure" + and step.get("exit_code") == 0 + and isinstance(error, str) + and POD_NOT_FOUND.fullmatch(error) is not None + ) + + +def verify(record: dict[str, Any]) -> tuple[int, dict[str, Any]]: + anomalies: list[dict[str, Any]] = [] + candidates: list[dict[str, Any]] = [] + steps: list[dict[str, Any]] = [] + + pipeline_status = record.get("status") + if pipeline_status != "success": + anomalies.append( + { + "scope": "pipeline", + "name": str(record.get("number", "unknown")), + "state": pipeline_status, + "reason": "pipeline status is not success", + } + ) + + workflows = record.get("workflows") + if not isinstance(workflows, list) or not workflows: + anomalies.append( + { + "scope": "pipeline", + "name": str(record.get("number", "unknown")), + "state": pipeline_status, + "reason": "workflows are missing or empty", + } + ) + workflows = [] + + for workflow_index, workflow in enumerate(workflows): + if not isinstance(workflow, dict): + anomalies.append( + { + "scope": "workflow", + "name": str(workflow_index), + "state": None, + "reason": "workflow is not an object", + } + ) + continue + workflow_name = str(workflow.get("name", workflow_index)) + if workflow.get("state") != "success": + anomalies.append( + { + "scope": "workflow", + "name": workflow_name, + "state": workflow.get("state"), + "reason": "workflow state is not success", + } + ) + children = workflow.get("children") + if not isinstance(children, list) or not children: + anomalies.append( + { + "scope": "workflow", + "name": workflow_name, + "state": workflow.get("state"), + "reason": "child-step list is missing or empty", + } + ) + continue + for child_index, child in enumerate(children): + if not isinstance(child, dict): + anomalies.append( + { + "scope": "step", + "name": f"{workflow_name}[{child_index}]", + "state": None, + "reason": "step is not an object", + } + ) + continue + steps.append(child) + if child.get("state") == "success": + continue + if is_issue_1000_artifact(child): + candidates.append(child) + continue + anomalies.append( + { + "scope": "step", + "name": child.get("name"), + "type": child.get("type"), + "state": child.get("state"), + "exit_code": child.get("exit_code"), + "error": child.get("error"), + "reason": "non-success step does not match the #1000 teardown signature", + } + ) + + if len(candidates) > 1: + anomalies.append( + { + "scope": "exemption", + "name": EXEMPTION_ID, + "state": "invalid", + "reason": "the #1000 exemption may apply to exactly one step", + } + ) + + exemption_applies = len(candidates) == 1 and not anomalies + state_counts = Counter(str(step.get("state", "missing")) for step in steps) + result: dict[str, Any] = { + "schema_version": "mosaic-terminal-green/v1", + "verdict": "terminal-green" if not anomalies else "not-terminal-green", + "pipeline_number": record.get("number"), + "commit": record.get("commit"), + "pipeline_status": pipeline_status, + "total_steps": len(steps), + "state_counts": dict(sorted(state_counts.items())), + "exempted_steps": 1 if exemption_applies else 0, + "anomalies": anomalies, + } + if exemption_applies: + candidate = candidates[0] + result["exemptions"] = [ + { + "exemption_id": EXEMPTION_ID, + "step": candidate.get("name"), + "signature": candidate.get("error"), + "tracking_issue": EXEMPTION_ISSUE, + "retires_when": "issue #1000 is fixed", + } + ] + else: + result["exemptions"] = [] + + return (0 if not anomalies else 1), result + + +def main() -> int: + if len(sys.argv) > 2 or (len(sys.argv) == 2 and sys.argv[1] in {"-h", "--help"}): + print(f"usage: {Path(sys.argv[0]).name} [pipeline.json|-]", file=sys.stderr) + return 0 if len(sys.argv) == 2 else 2 + try: + record = load_record(sys.argv[1] if len(sys.argv) == 2 else None) + except (OSError, ValueError, json.JSONDecodeError) as error: + return fail_usage(str(error)) + code, result = verify(record) + print(json.dumps(result, indent=2, sort_keys=True)) + return code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/mosaic/package.json b/packages/mosaic/package.json index 56898aa8..401d0015 100644 --- a/packages/mosaic/package.json +++ b/packages/mosaic/package.json @@ -25,7 +25,7 @@ "lint": "eslint src", "typecheck": "tsc --noEmit", "test": "vitest run --passWithNoTests && pnpm run test:framework-shell", - "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh" + "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh" }, "dependencies": { "@mosaicstack/brain": "workspace:*",