mos-dt-0/stack
1
0
Fork 0
forked from mosaicstack/stack
Code Pull Requests Activity

RM-61: prove ci-postgres teardown discrimination (#1033)

Browse Source 
Co-authored-by: coder-mos1 <[email protected]>
This commit is contained in:
coder-mos1
2026-08-01 14:54:04 +00:00
committed by mos-dt-0
parent f65e9ea656
commit f4fd5967fc
7 changed files with 510 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/scratchpads/1000-rm-61-ci-contract-exemption.md
+118
View File
@@ -0,0 +1,118 @@
# RM-61 — CI contract exemption for #1000 teardown artifact
**Tracking:** RM-61 / issue #1000
**Branch:** `fix/rm-61-ci-contract-exemption`
**Owner:** `coder-mos1`
## Objective
Determine, by red-first provider controls, whether the `ci-postgres` pod-not-found teardown signature discriminates from a real PostgreSQL failure. Only if it discriminates may a named, bounded CI-contract exemption be implemented. The exemption must retire when #1000 is fixed; fixing #1000 is the closure path.
## Pre-registered kill criterion
If an injected real `ci-postgres` failure also yields `pods "wp-svc-<ULID>-ci-postgres" not found` as the service's provider-visible failure, the signature does not discriminate. Option B is unsafe; stop exemption implementation and fall to Option A (#1000).
## Plan
1. Capture full `-f json` records for the 11 supplied observations and state counts.
2. Run one startup-failure control using the real pgvector/PostgreSQL image with an invalid `initdb` argument.
3. Run one post-readiness crash control using real PostgreSQL, `pg_isready`, and a deliberate postmaster kill while a DB-dependent probe is active.
4. Compare the raw `ci-postgres` service record independently of failures in dependent steps.
5. Investigate runner/time/head clustering only as a hypothesis; never encode incidental correlates or retries into policy.
6. If and only if the controls discriminate, implement and test the exact exemption, document its two-way boundary, and track retirement at #1000.
## Budget
No explicit token cap supplied. Working estimate: 20K–30K tokens. Limit provider controls to the two pre-registered runs; no retries or re-roll policy.
## Initial evidence
Historical JSON saved locally under `.evidence/rm-61/` (not for commit). Supplied pipelines: 11 total. Child-step counts: five pipelines with 9 children and six with 10 children. Seven contain the `ci-postgres` pod-not-found failure (#2170, #2175, #2180, #2181, #2182, #2187, #2188); four do not (#2158, #2167, #2184, #2186). Every observed workflow reports `agent_id=44`, so the available JSON does not separate clean and artifact runs by runner. This refutes runner identity as a discriminator in the sampled record.
## Progress
- [x] Requirements and kill criterion recorded before control implementation.
- [x] Historical full-JSON records captured.
- [x] Startup-failure control observed terminal.
- [x] Post-readiness crash control observed terminal.
- [x] Discrimination verdict recorded: Option B may proceed.
- [x] Conditional exemption implementation.
## Tests / evidence
### Control 1 — real startup failure
- Commit: `3931b0e29eb834914f7b17e4db7e221481d436fa`
- Pipeline: #2189, exact commit match.
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
- `ci-postgres`: `state=failure`, `exit_code=1`, `error=null`, with a five-second execution window.
- `test`: `state=failure`, `exit_code=1` after the readiness budget expired.
- Pipeline/workflow: terminal `failure`.
This control is red and its service record differs from #1000 (`exit_code=0` plus pod-not-found). It proves the startup-failure direction only. It does not settle the dangerous post-readiness crash/garbage-collection path.
### Control 2 — real post-readiness crash
- Commit: `25ac59715a94dd1b52ef42577472eb44ecc4b446`
- Pipeline: #2191, exact commit match.
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
- Service log proves PostgreSQL reached `database system is ready to accept connections`, the test created the arm table, and the service then killed postmaster PID 7.
- Test log proves a successful `SELECT 1` followed by `Connection refused`; it exited the pre-registered control code 61.
- `ci-postgres`: `state=failure`, `exit_code=137`, `error=null`, with a 203-second execution window.
- `test`: `state=failure`, `exit_code=61`.
- Pipeline/workflow: terminal `failure`.
This is the dangerous post-readiness crash path. Its service record is not pod-not-found and therefore differs from #1000 independently of the dependent test failure.
### Discrimination verdict
Both real failures are provider-visible as process exits (`exit_code=1` startup; `exit_code=137` crash) with no pod-not-found error. The seven observed #1000 artifacts are provider reconciliation misses (`exit_code=0` plus the exact pod-not-found error). The declared kill criterion did not fire, so Option B may proceed with a matcher requiring the full conjunction. This evidence does **not** prove every future Kubernetes failure is distinguishable; it proves these two concrete real-failure classes remain blocking and bounds the exemption to the observed reconciliation shape.
### Unit red-first checkpoint
The nine-case contract harness was written before the verifier. First execution exited 1 because `verify-terminal-green.py` did not exist; no exemption implementation was live. Cases pre-register ordinary green, the exact artifact, both provider controls, near-miss signatures, an independent failure, and a skipped step.
### Control 2 setup attempt — invalid, excluded from evidence
- Commit: `9455cd6a2650b2b7e70f746c07933d96e5cb3d20`
- Pipeline: #2190, exact commit match.
- Full JSON child scan: 9 total — 7 success, 2 failure, 0 skipped/pending/running.
- Service log: `/bin/sh: 0: -c requires an argument`.
- Root cause: Woodpecker service `commands` did not become the third `sh -c` argument. PostgreSQL never started, so this run is **not** the post-readiness crash control and provides no discrimination evidence.
- Focused remediation: place the script directly in the third `entrypoint` element and supply `PGPASSWORD` for the marker query. This is a control-fixture correction, not a retry of #1000 and not evidence for either verdict.
## Implementation evidence
- `verify-terminal-green.py` consumes only the full JSON/API record; it performs no fetch, retry, or trigger.
- Exact #2188 record: exit 0, 10 children, 9 success + 1 named exemption.
- Historical set: #2158/#2167/#2184/#2186 pass with no exemption; #2170/#2175/#2182/#2187/#2188 pass with one named exemption; #2180/#2181 remain red because independent failures exist.
- Provider controls: #2189 and #2191 both exit 1 under the verifier; neither is exempted.
- Unit harness: initial 9/9 cases passed after the red-first checkpoint; review remediation expands this to 12 cases with expected-head match/missing/mismatch coverage.
- Test-membership guard: PASS, population 45; 26 enumerated, 19 signed exclusions; all 39 surface paths present.
- Python compile: PASS.
- `pnpm typecheck`: PASS, 45/45 tasks.
- `pnpm lint`: PASS, 25/25 tasks.
- `pnpm format:check`: PASS after moving local evidence outside the repository tree.
- `test:framework-shell`: RM-61 and all preceding suites passed, then the pre-existing wake assertion aborted with exit 97 because this host's Bash 5.2.15 reports `BASH_LINENO [3 5]` where that suite requires `[3 4]`. RM-61 does not modify the wake suite; the command is not fully runnable on this host as written and no substitute result is claimed.
## Independent review
- Review 67 / comment 20403 at exact head `e7b29219e11efd0a19395156ac0b154bec0c3a73`: **REQUEST CHANGES**.
- Blocker: the verifier echoed the pipeline commit but did not bind it to the current PR head; mutating only #2188's commit still returned terminal-green.
- Remediation: require `--expect-commit <full-40>`, add a pipeline anomaly on missing/mismatched record commits, emit expected and observed values, wire both CI documentation and the merge-gate baseline to pass provider PR head, and add match/missing/mismatch tests.
- This binding is not prohibited head-based clustering policy: it proves the evidence belongs to the commit under verdict. Runner/node/time/head correlation remains excluded from the teardown signature itself.
- Review 69 later approved the commit-binding remediation at exact head `033b2ffb46674b2c0bcc5197273c109b461f62d9`; pipeline #2193 was 9/9 success. Before merge-gate, an independent adjudicator found that Python treats JSON `false == 0`, allowing a non-integer exit value to match. The prior gate-ready state was withdrawn. The type-strict set distinguishes genuine red-first controls (`false`, `0.0`, which wrongly exempted) from regression guards (`true`, `"0"`, `null`, which already blocked). Remediation requires the decoded type to be exactly `int` and excludes `bool` explicitly.
## Documentation checklist
- [x] CI contract documented in the canonical framework CI/CD guide.
- [x] Operator command documented in the Woodpecker tool README.
- [x] Merge-gate baseline points to the deterministic verifier and named retirement.
- [x] Tracking and retirement cite issue #1000.
- [x] Both positive and negative guarantee boundaries are stated.
- [x] No API/auth/schema/user-facing navigation change; OpenAPI, user guide, and sitemap are not applicable.
## Risks
The controls establish discrimination for deterministic startup failure and an armed post-readiness postmaster crash on the current Woodpecker Kubernetes provider. They cannot prove that every future Kubernetes failure mode will preserve a non-zero exit before reconciliation. The exact matcher minimizes that residual risk, and issue #1000 remains the mandatory provider-seam closure and retirement trigger.