stack/docs/scratchpads/1000-rm-61-ci-contract-exemption.md

RM-61 — CI contract exemption for #1000 teardown artifact

Tracking: RM-61 / issue #1000
Branch: fix/rm-61-ci-contract-exemption
Owner: coder-mos1

Objective

Determine, by red-first provider controls, whether the ci-postgres pod-not-found teardown signature discriminates from a real PostgreSQL failure. Only if it discriminates may a named, bounded CI-contract exemption be implemented. The exemption must retire when #1000 is fixed; fixing #1000 is the closure path.

Pre-registered kill criterion

If an injected real ci-postgres failure also yields pods "wp-svc-<ULID>-ci-postgres" not found as the service's provider-visible failure, the signature does not discriminate. Option B is unsafe; stop exemption implementation and fall to Option A (#1000).

Plan

1. Capture full -f json records for the 11 supplied observations and state counts.
2. Run one startup-failure control using the real pgvector/PostgreSQL image with an invalid initdb argument.
3. Run one post-readiness crash control using real PostgreSQL, pg_isready, and a deliberate postmaster kill while a DB-dependent probe is active.
4. Compare the raw ci-postgres service record independently of failures in dependent steps.
5. Investigate runner/time/head clustering only as a hypothesis; never encode incidental correlates or retries into policy.
6. If and only if the controls discriminate, implement and test the exact exemption, document its two-way boundary, and track retirement at #1000.

Budget

No explicit token cap supplied. Working estimate: 20K–30K tokens. Limit provider controls to the two pre-registered runs; no retries or re-roll policy.

Initial evidence

Historical JSON saved locally under .evidence/rm-61/ (not for commit). Supplied pipelines: 11 total. Child-step counts: five pipelines with 9 children and six with 10 children. Seven contain the ci-postgres pod-not-found failure (#2170, #2175, #2180, #2181, #2182, #2187, #2188); four do not (#2158, #2167, #2184, #2186). Every observed workflow reports agent_id=44, so the available JSON does not separate clean and artifact runs by runner. This refutes runner identity as a discriminator in the sampled record.

Progress

• Requirements and kill criterion recorded before control implementation.
• Historical full-JSON records captured.
• Startup-failure control observed terminal.
• Post-readiness crash control observed terminal.
• Discrimination verdict recorded: Option B may proceed.
• Conditional exemption implementation.

Tests / evidence

Control 1 — real startup failure

• Commit: 3931b0e29eb834914f7b17e4db7e221481d436fa
• Pipeline: #2189, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• ci-postgres: state=failure, exit_code=1, error=null, with a five-second execution window.
• test: state=failure, exit_code=1 after the readiness budget expired.
• Pipeline/workflow: terminal failure.

This control is red and its service record differs from #1000 (exit_code=0 plus pod-not-found). It proves the startup-failure direction only. It does not settle the dangerous post-readiness crash/garbage-collection path.

Control 2 — real post-readiness crash

• Commit: 25ac59715a94dd1b52ef42577472eb44ecc4b446
• Pipeline: #2191, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• Service log proves PostgreSQL reached database system is ready to accept connections, the test created the arm table, and the service then killed postmaster PID 7.
• Test log proves a successful SELECT 1 followed by Connection refused; it exited the pre-registered control code 61.
• ci-postgres: state=failure, exit_code=137, error=null, with a 203-second execution window.
• test: state=failure, exit_code=61.
• Pipeline/workflow: terminal failure.

This is the dangerous post-readiness crash path. Its service record is not pod-not-found and therefore differs from #1000 independently of the dependent test failure.

Discrimination verdict

Both real failures are provider-visible as process exits (exit_code=1 startup; exit_code=137 crash) with no pod-not-found error. The seven observed #1000 artifacts are provider reconciliation misses (exit_code=0 plus the exact pod-not-found error). The declared kill criterion did not fire, so Option B may proceed with a matcher requiring the full conjunction. This evidence does not prove every future Kubernetes failure is distinguishable; it proves these two concrete real-failure classes remain blocking and bounds the exemption to the observed reconciliation shape.

Unit red-first checkpoint

The nine-case contract harness was written before the verifier. First execution exited 1 because verify-terminal-green.py did not exist; no exemption implementation was live. Cases pre-register ordinary green, the exact artifact, both provider controls, near-miss signatures, an independent failure, and a skipped step.

Control 2 setup attempt — invalid, excluded from evidence

• Commit: 9455cd6a2650b2b7e70f746c07933d96e5cb3d20
• Pipeline: #2190, exact commit match.
• Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
• Service log: /bin/sh: 0: -c requires an argument.
• Root cause: Woodpecker service commands did not become the third sh -c argument. PostgreSQL never started, so this run is not the post-readiness crash control and provides no discrimination evidence.
• Focused remediation: place the script directly in the third entrypoint element and supply PGPASSWORD for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict.

Risks

The Woodpecker Kubernetes backend may garbage-collect both genuinely failed and successfully used service pods before reconciliation. If so, the provider-visible service signature cannot safely support Option B.