mosaicstack/stack
13
0
Fork 1
Code Issues 304 Pull Requests 10 Actions Packages Projects Releases 15 Wiki Activity

ci: pin ci-base image reference; mutable :latest makes CI verdicts non-reproducible (D27) #1328

New Issue
Closed
opened 2026-08-19 23:24:37 +00:00 by ops-ci-01 · 0 comments
ops-ci-01 commented 2026-08-19 23:24:37 +00:00
Member
Copy Link
Copy Source

Defect

Every CI pipeline (ci.yml, publish.yml) runs its steps from the mutable tag git.mosaicstack.dev/mosaicstack/stack/ci-base:latest. On the Kubernetes backend a tag is resolved per-pod at pull time, so two pipelines of the SAME commit can execute different images whenever the tag is repushed, and no pipeline run records which image it actually ran. Until that is pinned, a green rerun is not evidence and a red one is not a defect report: retry-until-green is formally meaningless, not merely bad practice (brain register D27; evidence: #1324 comment id 23386).

Measured basis (2026-08-19, this session)

  • All 48 pipelines that day executed on Woodpecker agent 45 (per-pipeline workflows[].agent_id): agent version and concurrency are REFUTED as discriminators between red and green runs. The uncontrolled variable left standing is the mutable image reference.
  • Same-tree anomaly on 2508f0a: run 2537 failed test/I14 with an assertion whose own captured evidence satisfies it; run 2541 failed sanitization/wrapper-guard, a step that was green in 2537. Verbatim log excerpts in the #1324 comment.
  • ci-image.yml already publishes an immutable tag, lock-<sha256(pnpm-lock.yaml)[:12]>, atomically with :latest in the same kaniko run.
  • Last ci-image run: main commit 712c770 (2026-07-26). No rebuild since (main's pnpm-lock.yaml/Dockerfile.ci unchanged since; no ci-image pipelines in today's set). Therefore :latest and lock-9cb7ffcd8828 are byte-identical by construction: pinning to that tag is a zero-behavior-change lock-in of what runs today.
  • next's pnpm-lock.yaml has diverged from the baked store (9acaa89d213b vs 9cb7ffcd8828). That is pre-existing, performance-only lag: pnpm install --frozen-lockfile --prefer-offline resolves missing packages from network, so correctness never depended on the baked store.

Fix (this issue's PR)

Replace the &node_image anchor in .woodpecker/ci.yml and .woodpecker/publish.yml with the immutable tag ci-base:lock-9cb7ffcd8828, plus a comment stating the model and the bump procedure. The pin changes only through reviewed commits; a wrong/nonexistent tag fails at image pull, loudly, which is the guard enforcing itself.

Pin bump procedure (documented in the YAML comment)

  1. A recipe change (pnpm-lock.yaml or Dockerfile.ci) lands on main via PR; ci-image.yml fires and pushes lock-<new> (+ :latest).
  2. A follow-up PR bumps the anchor to lock-<new>. Until it lands, pipelines keep the old pin: reproducible, with the documented network-fallback lag.

Known limitations (deliberate non-goals of v1, follow-ups if wanted)

  • lock- addresses the lockfile only: a Dockerfile.ci-only change re-pushes the same lock-tag with different content. A recipe- hash over both files closes that; needs ci-image.yml to push the new tag before consumers can reference it.
  • ci-image triggers on main/tags only, so a recipe change merged to next does not rebuild the base until it reaches main. Acceptable: consumers on next keep the old pin and the network fallback. Making next self-sufficient is a separate policy change (one more ~25-min build per recipe change on a storage-constrained runner).
  • Automating the bump PR (bot) is out of scope until the manual procedure has proven itself.

Non-goals

  • Woodpecker agent-generation alignment (39/41/43 vs 45) and pruning month-stale docker agents: cluster-side, owned by Jason (R4).
  • Upstream step images (kaniko :debug, pgvector:pg17): mutable too, but upstream and out of this defect's scope.
## Defect Every CI pipeline (ci.yml, publish.yml) runs its steps from the mutable tag `git.mosaicstack.dev/mosaicstack/stack/ci-base:latest`. On the Kubernetes backend a tag is resolved per-pod at pull time, so two pipelines of the SAME commit can execute different images whenever the tag is repushed, and no pipeline run records which image it actually ran. Until that is pinned, a green rerun is not evidence and a red one is not a defect report: retry-until-green is formally meaningless, not merely bad practice (brain register D27; evidence: #1324 comment id 23386). ## Measured basis (2026-08-19, this session) - All 48 pipelines that day executed on Woodpecker agent 45 (per-pipeline `workflows[].agent_id`): agent version and concurrency are REFUTED as discriminators between red and green runs. The uncontrolled variable left standing is the mutable image reference. - Same-tree anomaly on `2508f0a`: run 2537 failed `test`/I14 with an assertion whose own captured evidence satisfies it; run 2541 failed `sanitization`/wrapper-guard, a step that was green in 2537. Verbatim log excerpts in the #1324 comment. - ci-image.yml already publishes an immutable tag, `lock-<sha256(pnpm-lock.yaml)[:12]>`, atomically with `:latest` in the same kaniko run. - Last ci-image run: main commit `712c770` (2026-07-26). No rebuild since (main's `pnpm-lock.yaml`/`Dockerfile.ci` unchanged since; no ci-image pipelines in today's set). Therefore `:latest` and `lock-9cb7ffcd8828` are byte-identical by construction: pinning to that tag is a zero-behavior-change lock-in of what runs today. - `next`'s `pnpm-lock.yaml` has diverged from the baked store (9acaa89d213b vs 9cb7ffcd8828). That is pre-existing, performance-only lag: `pnpm install --frozen-lockfile --prefer-offline` resolves missing packages from network, so correctness never depended on the baked store. ## Fix (this issue's PR) Replace the `&node_image` anchor in `.woodpecker/ci.yml` and `.woodpecker/publish.yml` with the immutable tag `ci-base:lock-9cb7ffcd8828`, plus a comment stating the model and the bump procedure. The pin changes only through reviewed commits; a wrong/nonexistent tag fails at image pull, loudly, which is the guard enforcing itself. ## Pin bump procedure (documented in the YAML comment) 1. A recipe change (`pnpm-lock.yaml` or `Dockerfile.ci`) lands on `main` via PR; ci-image.yml fires and pushes `lock-<new>` (+ `:latest`). 2. A follow-up PR bumps the anchor to `lock-<new>`. Until it lands, pipelines keep the old pin: reproducible, with the documented network-fallback lag. ## Known limitations (deliberate non-goals of v1, follow-ups if wanted) - `lock-` addresses the lockfile only: a `Dockerfile.ci`-only change re-pushes the same lock-tag with different content. A `recipe-` hash over both files closes that; needs ci-image.yml to push the new tag before consumers can reference it. - ci-image triggers on `main`/tags only, so a recipe change merged to `next` does not rebuild the base until it reaches `main`. Acceptable: consumers on next keep the old pin and the network fallback. Making next self-sufficient is a separate policy change (one more ~25-min build per recipe change on a storage-constrained runner). - Automating the bump PR (bot) is out of scope until the manual procedure has proven itself. ## Non-goals - Woodpecker agent-generation alignment (39/41/43 vs 45) and pruning month-stale docker agents: cluster-side, owned by Jason (R4). - Upstream step images (kaniko `:debug`, `pgvector:pg17`): mutable too, but upstream and out of this defect's scope.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1328