mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

CI: type=service step records state=failure from pod absence at teardown (exit_code 0) on otherwise-green pipelines #1000

New Issue
Open
opened 2026-07-31 09:41:55 +00:00 by Ghost · 0 comments
Ghost commented 2026-07-31 09:41:55 +00:00
Copy Link
Copy Source

CI: a type=service step records state=failure from the pod's absence at teardown — exit_code 0, on otherwise-fully-green pipelines

The tell, first

ci-postgres on pipeline 2139 (mosaicstack/stack, repo 47):

state       = failure
exit_code   = 0          <-- nothing reported a non-zero status
start_time  = null
end_time    = null
error       = pods "wp-svc-01kyvq0w222pgvfvm69pdr39gx-ci-postgres" not found

An exit_code of 0 under a failure state means the state was recorded from the pod's absence, not from anything the pod did. The Kubernetes backend went looking for the service pod during reconciliation, did not find it, and wrote failure from that miss. Null start/end times are consistent: no execution window was ever recorded for the thing being judged.

The pipeline still reports status=success, because service steps do not gate the verdict. So the visible artifact is a green pipeline containing a red row that no one can attribute.

Base rate — it is not any one branch

Across the last 40 repo-47 pipelines carrying a ci-postgres step:

count
pipelines with the step 40
ci-postgres = failure 3
…of those, on otherwise fully green runs 2

The two otherwise-green cases are 2139 and #2101, with the identical pods "wp-svc-…-ci-postgres" not found error string. #2101 predates the branch that surfaced this. Roughly 5% of runs, pre-existing, branch-independent.

The service was live — this is not a postgres fault

The test step opens with a pg_isready retry loop (60 iterations) against the service before doing anything else. From the decoded step log for 2139:

+ ready=0
  for i in $(seq 1 60); do if pg_isready -h ci-postgres -p 5432 -U ********; then
    ready=1
ci-postgres:5432 - accepting connections

It answered. No ECONNREFUSED, no exhausted retry budget, no skipped tests. All eight command steps returned success / exit 0. Whatever the orchestrator failed to find at teardown, the database was serving while the tests ran against it.

Why both halves are stated

Neither measurement is sufficient alone, and I want the reasoning on the record rather than the conclusion:

  • The base rate alone would be a dismissal — "it happens sometimes" is how a real service fault gets waved through on the third occurrence.
  • The liveness alone would be an anecdote — one green readiness probe does not establish that the red row is harmless in general.

Together they support the actual claim: the failure is backend bookkeeping at teardown, and the tests that depended on the service were not affected on this run.

Why it is worth fixing rather than tolerating

The failure mode and the safe state produce the same output. A service that genuinely died mid-run and a service whose pod record vanished at teardown both land as state=failure, exit_code=0 on a green pipeline. Today that ambiguity is resolved by hand — decoding the dependent step's log to see whether the service ever answered. That is not a check anyone will run on the hundredth occurrence, and the one time it matters is the time it gets skipped.

The concrete cost is that this row trains readers to ignore a red service step, which is precisely the signal that would report a real service outage.

Suggested direction

Distinguish "the step ran and failed" from "the step's record could not be found at reconciliation" — a distinct state, or at minimum an exit_code that is not 0, so the two are separable without reading logs. A reconciliation miss on a step that was never observed to start is arguably not a step failure at all.

Bounds

Filed at @Mos's scoping call rather than folded into the PR that surfaced it. I have not attempted a fix and have not examined the Woodpecker Kubernetes backend beyond the API records and step logs cited above. The 40-pipeline window is what I sampled; I did not scan further back, so the ~5% figure is a rate over that window and not a long-run estimate.

— mos-dt (sb-it-1-dt). Signed in body; shared account on this host, so the signature is a labelled claim, never provenance.

## CI: a `type=service` step records `state=failure` from the pod's *absence* at teardown — `exit_code 0`, on otherwise-fully-green pipelines ### The tell, first `ci-postgres` on pipeline **2139** (`mosaicstack/stack`, repo 47): ``` state = failure exit_code = 0 <-- nothing reported a non-zero status start_time = null end_time = null error = pods "wp-svc-01kyvq0w222pgvfvm69pdr39gx-ci-postgres" not found ``` **An `exit_code` of 0 under a `failure` state means the state was recorded from the pod's absence, not from anything the pod did.** The Kubernetes backend went looking for the service pod during reconciliation, did not find it, and wrote `failure` from that miss. Null start/end times are consistent: no execution window was ever recorded for the thing being judged. The pipeline still reports `status=success`, because service steps do not gate the verdict. So the visible artifact is a green pipeline containing a red row that no one can attribute. ### Base rate — it is not any one branch Across the **last 40** repo-47 pipelines carrying a `ci-postgres` step: | | count | |---|---| | pipelines with the step | 40 | | `ci-postgres` = `failure` | 3 | | …of those, on **otherwise fully green** runs | **2** | The two otherwise-green cases are **2139** and **#2101**, with the **identical** `pods "wp-svc-…-ci-postgres" not found` error string. #2101 predates the branch that surfaced this. Roughly **5% of runs**, pre-existing, branch-independent. ### The service was live — this is not a postgres fault The `test` step opens with a `pg_isready` retry loop (60 iterations) against the service before doing anything else. From the decoded step log for 2139: ``` + ready=0 for i in $(seq 1 60); do if pg_isready -h ci-postgres -p 5432 -U ********; then ready=1 ci-postgres:5432 - accepting connections ``` It answered. **No `ECONNREFUSED`, no exhausted retry budget, no skipped tests.** All eight command steps returned `success` / `exit 0`. Whatever the orchestrator failed to find at teardown, the database was serving while the tests ran against it. ### Why both halves are stated Neither measurement is sufficient alone, and I want the reasoning on the record rather than the conclusion: - **The base rate alone would be a dismissal** — "it happens sometimes" is how a real service fault gets waved through on the third occurrence. - **The liveness alone would be an anecdote** — one green readiness probe does not establish that the red row is harmless in general. Together they support the actual claim: the `failure` is backend bookkeeping at teardown, and the tests that depended on the service were not affected on this run. ### Why it is worth fixing rather than tolerating The failure mode and the safe state produce the same output. A service that genuinely died mid-run and a service whose pod record vanished at teardown **both** land as `state=failure, exit_code=0` on a green pipeline. Today that ambiguity is resolved by hand — decoding the dependent step's log to see whether the service ever answered. That is not a check anyone will run on the hundredth occurrence, and the one time it matters is the time it gets skipped. The concrete cost is that this row **trains readers to ignore a red service step**, which is precisely the signal that would report a real service outage. ### Suggested direction Distinguish "the step ran and failed" from "the step's record could not be found at reconciliation" — a distinct state, or at minimum an `exit_code` that is not `0`, so the two are separable without reading logs. A reconciliation miss on a step that was never observed to start is arguably not a step failure at all. ### Bounds Filed at @Mos's scoping call rather than folded into the PR that surfaced it. I have **not** attempted a fix and have not examined the Woodpecker Kubernetes backend beyond the API records and step logs cited above. The 40-pipeline window is what I sampled; I did not scan further back, so the ~5% figure is a rate over that window and not a long-run estimate. — mos-dt (sb-it-1-dt). Signed in body; shared account on this host, so the signature is a labelled claim, never provenance.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1000