stack/docs/scratchpads/1000-rm-61-ci-contract-exemption.md

RM-61 — CI contract exemption for #1000 teardown artifact

Tracking: RM-61 / issue #1000

Branch: fix/rm-61-ci-contract-exemption Owner: coder-mos1

Objective

Determine, by red-first provider controls, whether the ci-postgres pod-not-found teardown signature discriminates from a real PostgreSQL failure. Only if it discriminates may a named, bounded CI-contract exemption be implemented. The exemption must retire when #1000 is fixed; fixing #1000 is the closure path.

Pre-registered kill criterion

If an injected real ci-postgres failure also yields pods "wp-svc-<ULID>-ci-postgres" not found as the service's provider-visible failure, the signature does not discriminate. Option B is unsafe; stop exemption implementation and fall to Option A (#1000).

Plan

1. Capture full -f json records for the 11 supplied observations and state counts.
2. Run one startup-failure control using the real pgvector/PostgreSQL image with an invalid initdb argument.
3. Run one post-readiness crash control using real PostgreSQL, pg_isready, and a deliberate postmaster kill while a DB-dependent probe is active.
4. Compare the raw ci-postgres service record independently of failures in dependent steps.
5. Investigate runner/time/head clustering only as a hypothesis; never encode incidental correlates or retries into policy.
6. If and only if the controls discriminate, implement and test the exact exemption, document its two-way boundary, and track retirement at #1000.

Budget

No explicit token cap supplied. Working estimate: 20K–30K tokens. Limit provider controls to the two pre-registered runs; no retries or re-roll policy.

Initial evidence

Historical JSON saved locally under .evidence/rm-61/ (not for commit). Supplied pipelines: 11 total. Child-step counts: five pipelines with 9 children and six with 10 children. Seven contain the ci-postgres pod-not-found failure (#2170, #2175, #2180, #2181, #2182, #2187, #2188); four do not (#2158, #2167, #2184, #2186). Every observed workflow reports agent_id=44, so the available JSON does not separate clean and artifact runs by runner. This refutes runner identity as a discriminator in the sampled record.

Progress

• Requirements and kill criterion recorded before control implementation.
• Historical full-JSON records captured.
• Startup-failure control observed terminal.
• Post-readiness crash control observed terminal.
• Discrimination verdict recorded: Option B may proceed.
• Conditional exemption implementation.

Tests / evidence

Control 1 — real startup failure

• Commit: 3931b0e29eb834914f7b17e4db7e221481d436fa
• Pipeline: #2189, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• ci-postgres: state=failure, exit_code=1, error=null, with a five-second execution window.
• test: state=failure, exit_code=1 after the readiness budget expired.
• Pipeline/workflow: terminal failure.

This control is red and its service record differs from #1000 (exit_code=0 plus pod-not-found). It proves the startup-failure direction only. It does not settle the dangerous post-readiness crash/garbage-collection path.

Control 2 — real post-readiness crash

• Commit: 25ac59715a94dd1b52ef42577472eb44ecc4b446
• Pipeline: #2191, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• Service log proves PostgreSQL reached database system is ready to accept connections, the test created the arm table, and the service then killed postmaster PID 7.
• Test log proves a successful SELECT 1 followed by Connection refused; it exited the pre-registered control code 61.
• ci-postgres: state=failure, exit_code=137, error=null, with a 203-second execution window.
• test: state=failure, exit_code=61.
• Pipeline/workflow: terminal failure.

This is the dangerous post-readiness crash path. Its service record is not pod-not-found and therefore differs from #1000 independently of the dependent test failure.

Discrimination verdict

Both real failures are provider-visible as process exits (exit_code=1 startup; exit_code=137 crash) with no pod-not-found error. The seven observed #1000 artifacts are provider reconciliation misses (exit_code=0 plus the exact pod-not-found error). The declared kill criterion did not fire, so Option B may proceed with a matcher requiring the full conjunction. This evidence does not prove every future Kubernetes failure is distinguishable; it proves these two concrete real-failure classes remain blocking and bounds the exemption to the observed reconciliation shape.

Unit red-first checkpoint

The nine-case contract harness was written before the verifier. First execution exited 1 because verify-terminal-green.py did not exist; no exemption implementation was live. Cases pre-register ordinary green, the exact artifact, both provider controls, near-miss signatures, an independent failure, and a skipped step.

Control 2 setup attempt — invalid, excluded from evidence

• Commit: 9455cd6a2650b2b7e70f746c07933d96e5cb3d20
• Pipeline: #2190, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• Service log: /bin/sh: 0: -c requires an argument.
• Root cause: Woodpecker service commands did not become the third sh -c argument. PostgreSQL never started, so this run is not the post-readiness crash control and provides no discrimination evidence.
• Focused remediation: place the script directly in the third entrypoint element and supply PGPASSWORD for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict.

Implementation evidence

• verify-terminal-green.py consumes only the full JSON/API record; it performs no fetch, retry, or trigger.
• Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
• Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
• Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
• Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
• Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
• Python compile: PASS.
• pnpm typecheck: PASS, 45/45 tasks.
• pnpm lint: PASS, 25/25 tasks.
• pnpm format:check: PASS after moving local evidence outside the repository tree.
• test:framework-shell: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports BASH_LINENO [3 5] where that suite requires [3 4]. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.

Independent review

• Review 67 / comment 20403 at exact head e7b29219e11efd0a19395156ac0b154bec0c3a73: REQUEST CHANGES.
• Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
• Remediation: require --expect-commit <full-40>, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
• This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
• Review 69 later approved the commit-binding remediation at exact head 033b2ffb46674b2c0bcc5197273c109b461f62d9; pipeline #2193 was 9/9 success. Before merge-gate, an independent adjudicator found that Python treats JSON false == 0, allowing a non-integer exit value to match. The prior gate-ready state was withdrawn. The type-strict set distinguishes genuine red-first controls (false, 0.0, which wrongly exempted) from regression guards (true, "0", null, which already blocked). Remediation requires the decoded type to be exactly int and excludes bool explicitly.

Documentation checklist

• CI contract documented in the canonical framework CI/CD guide.
• Operator command documented in the Woodpecker tool README.
• Merge-gate baseline points to the deterministic verifier and named retirement.
• Tracking and retirement cite issue #1000.
• Both positive and negative guarantee boundaries are stated.
• No API/auth/schema/user-facing navigation change; OpenAPI, user guide, and sitemap are not applicable.

Risks

The controls establish discrimination for deterministic startup failure and an armed post-readiness postmaster crash on the current Woodpecker Kubernetes provider. They cannot prove that every future Kubernetes failure mode will preserve a non-zero exit before reconciliation. The exact matcher minimizes that residual risk, and issue #1000 remains the mandatory provider-seam closure and retirement trigger.