mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

main is red: container image publishing fails registry auth (dormant behind a path filter, revealed by #1094) #1095

New Issue
Open
opened 2026-08-07 05:50:13 +00:00 by Ghost · 2 comments
Ghost commented 2026-08-07 05:50:13 +00:00
Copy Link
Copy Source

main's container image publishing is broken, and has been dormant behind a path filter

All three container builds fail on main with a registry authentication error. This was revealed, not caused, by merging #1094.

The failure

Pipeline 2265 (main @ 42ac19af), publish workflow:

build-gateway     failure  exit=1
build-appservice  failure  exit=1
build-web         failure  exit=1

All three, identically:

/kaniko/executor --context . --dockerfile docker/gateway.Dockerfile $DESTINATIONS
error checking push permissions -- make sure you entered the correct tag name,
and that you are authenticated correctly, and try again

This is not a code failure. The build never gets past the registry auth check.

Why nobody has seen it

.woodpecker/publish.yml:16 gates container builds on a path exclude list:

- &image_build_when
  - event: tag
  - event: [push, manual]
    branch: main
    path:
      exclude: ['packages/mosaic/**', 'docs/**', '**/*.md', '.woodpecker/**']

Recent main traffic has been framework-only. Verified:

pipeline commit build-* steps
2239 80a45b1e not triggered
2261 8ff7aac0 not triggered
2265 42ac19af triggered → all 3 failed

#1094 touched apps/gateway/**, which is outside the exclude list. It is the first main push in some time to reach this path.

Two credential sets, one broken

publish-npm        from_secret: gitea_token                      → works (green on 2239, 2261, 2263)
build-gateway      from_secret: gitea_username + gitea_password  → FAILS
build-appservice   same                                          → FAILS
build-web          same                                          → FAILS

The npm-registry path authenticates fine. The container-registry path does not. That is consistent with gitea_username/gitea_password being unset, expired, or lacking write:package — but I have not inspected the secrets and am not going to.

Impact

  • main is red on push/publish right now.
  • Any future main push touching apps/**, src/**, docker/** or anything else outside the exclude list will fail the same way.
  • No container image has been published from main for however long this has been broken. I have not established how long — only that it did not run on the two prior pushes I checked.

What this needs

An owner for the container-registry credential (gitea_username / gitea_password). This is an access problem, not a code problem, and it is outside what I can fix or verify without handling secrets that are not mine.

Not affected

#1085 touches .woodpecker/ci.yml and packages/mosaic/framework/tools/git/** — both excluded. Merging it will not trigger container builds and will not compound this.

Method note

This is the same shape as the USC deploy brake: a failure sitting dormant behind a filter, invisible until something crossed the filter. The sequencing step that reads main after each merge is what surfaced it — a hash-and-review gate would have reported the merge clean.

## `main`'s container image publishing is broken, and has been dormant behind a path filter All three container builds fail on `main` with a registry authentication error. This was **revealed, not caused**, by merging #1094. ### The failure Pipeline **2265** (`main` @ `42ac19af`), `publish` workflow: ``` build-gateway failure exit=1 build-appservice failure exit=1 build-web failure exit=1 ``` All three, identically: ``` /kaniko/executor --context . --dockerfile docker/gateway.Dockerfile $DESTINATIONS error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again ``` **This is not a code failure.** The build never gets past the registry auth check. ### Why nobody has seen it `.woodpecker/publish.yml:16` gates container builds on a path **exclude** list: ```yaml - &image_build_when - event: tag - event: [push, manual] branch: main path: exclude: ['packages/mosaic/**', 'docs/**', '**/*.md', '.woodpecker/**'] ``` Recent `main` traffic has been framework-only. Verified: | pipeline | commit | `build-*` steps | |---|---|---| | 2239 | `80a45b1e` | **not triggered** | | 2261 | `8ff7aac0` | **not triggered** | | 2265 | `42ac19af` | **triggered → all 3 failed** | `#1094` touched `apps/gateway/**`, which is outside the exclude list. It is the first `main` push in some time to reach this path. ### Two credential sets, one broken ``` publish-npm from_secret: gitea_token → works (green on 2239, 2261, 2263) build-gateway from_secret: gitea_username + gitea_password → FAILS build-appservice same → FAILS build-web same → FAILS ``` The npm-registry path authenticates fine. **The container-registry path does not.** That is consistent with `gitea_username`/`gitea_password` being unset, expired, or lacking `write:package` — but I have not inspected the secrets and am not going to. ### Impact - **`main` is red on `push/publish` right now.** - Any future `main` push touching `apps/**`, `src/**`, `docker/**` or anything else outside the exclude list will fail the same way. - No container image has been published from `main` for however long this has been broken. **I have not established how long** — only that it did not run on the two prior pushes I checked. ### What this needs An owner for the container-registry credential (`gitea_username` / `gitea_password`). This is an access problem, not a code problem, and it is outside what I can fix or verify without handling secrets that are not mine. ### Not affected #1085 touches `.woodpecker/ci.yml` and `packages/mosaic/framework/tools/git/**` — **both excluded**. Merging it will not trigger container builds and will not compound this. ### Method note This is the same shape as the USC deploy brake: a failure sitting dormant behind a filter, invisible until something crossed the filter. The sequencing step that reads `main` after each merge is what surfaced it — a hash-and-review gate would have reported the merge clean.
Ghost commented 2026-08-07 06:09:37 +00:00
Copy Link
Copy Source

⚠ main is green again. That is masking, not a fix. Do not close this on a green main.

Predicted before the merge that caused it, and confirmed after.

What happened

#1085 merged to main at aa0a7b5f. Its paths — .woodpecker/ci.yml and packages/mosaic/framework/tools/git/** — are both on publish.yml's exclude list, so the container builds did not run.

pipeline 2265 (42ac19af)  publish workflow: build-gateway FAIL · build-appservice FAIL · build-web FAIL
pipeline 2267 (aa0a7b5f)  publish workflow: publish-npm success   ← build-* steps NOT PRESENT AT ALL
main @aa0a7b5f            push/publish = SUCCESS

The failing steps were not fixed, retried, or skipped-with-a-reason. They were never scheduled.

Why this matters more than the failure itself

A reader checking main right now sees push/publish = success and would reasonably conclude the container registry works. It does not. The next main push touching anything outside packages/mosaic/**, docs/**, **/*.md, .woodpecker/** will fail exactly as pipeline 2265 did.

This is the same shape as the dormancy that hid it in the first place: the green is a property of what was scheduled, not of what works.

The test that actually discriminates

Do not use main's status. Use either:

  1. a main push (or manual run) touching a non-excluded path, and check build-gateway; or
  2. a direct check that gitea_username / gitea_password can authenticate to the container registry with push scope.

Both answer the question. main being green answers a different one.

Unchanged

Still needs a credential owner. Still outside what I can fix or verify without handling secrets that are not mine. I have not inspected them and have not established how long this has been broken.

## ⚠ `main` is green again. That is masking, not a fix. Do not close this on a green `main`. Predicted before the merge that caused it, and confirmed after. ### What happened `#1085` merged to `main` at `aa0a7b5f`. Its paths — `.woodpecker/ci.yml` and `packages/mosaic/framework/tools/git/**` — are **both on `publish.yml`'s exclude list**, so the container builds did not run. ``` pipeline 2265 (42ac19af) publish workflow: build-gateway FAIL · build-appservice FAIL · build-web FAIL pipeline 2267 (aa0a7b5f) publish workflow: publish-npm success ← build-* steps NOT PRESENT AT ALL main @aa0a7b5f push/publish = SUCCESS ``` The failing steps were not fixed, retried, or skipped-with-a-reason. **They were never scheduled.** ### Why this matters more than the failure itself A reader checking `main` right now sees `push/publish = success` and would reasonably conclude the container registry works. It does not. The next `main` push touching anything outside `packages/mosaic/**`, `docs/**`, `**/*.md`, `.woodpecker/**` will fail exactly as pipeline 2265 did. This is the same shape as the dormancy that hid it in the first place: **the green is a property of what was scheduled, not of what works.** ### The test that actually discriminates Do not use `main`'s status. Use either: 1. a `main` push (or manual run) touching a non-excluded path, and check `build-gateway`; or 2. a direct check that `gitea_username` / `gitea_password` can authenticate to the container registry with push scope. Both answer the question. `main` being green answers a different one. ### Unchanged Still needs a credential owner. Still outside what I can fix or verify without handling secrets that are not mine. I have not inspected them and have not established how long this has been broken.
Ghost commented 2026-08-07 09:47:24 +00:00
Copy Link
Copy Source

This is no longer dormant — it fired on main today, and here is the log

#1105 merged as 3a1203b2f83d. Its merged-main run (pipeline 2275) failed all three image builds:

publish/build-gateway      failure
publish/build-appservice   failure
publish/build-web          failure

Cause confirmed from the step log, not inferred:

+ echo "{\"auths\":{\"git.********stack.dev\":{\"username\":\"$REGISTRY_USER\",\"password\":\"$REGISTRY_PASS\"}}}" > /kaniko/.docker/config.json
error checking push permissions -- make sure you entered the correct tag name, and that you are
authenticated correctly, and try again: checking push permissions

kaniko writes the registry credential and then cannot authenticate to push. That is this issue's defect,
reproducing.

Why it has looked quiet — and why that was never evidence

This issue records the defect as "dormant behind a path filter." That reading is now confirmed and the
mechanism is precise: publish.yml excludes packages/mosaic/**, docs/**, **/*.md and
.woodpecker/**, so any merge touching only those paths never runs the image builds at all.

Recent merges did exactly that:

merge changed paths publish steps run image builds
#1100 → df4c591a packages/mosaic/**, docs/** 4 absent
#1105 → 3a1203b2f83d tools/*.sh, scripts/* 7 ran, all 3 failed

Before merging #1105 I stated in advance that its run would exercise the container path and that a red
there would be this issue surfacing rather than a defect in that PR. That is why the result is usable: it
was declared non-evidence about #1105 before it existed. The 4-vs-7 step count is the measured
discriminator — "this run exercises the path" is checkable, not asserted.

What this changes

  • main is red on ci/woodpecker/push/publish right now at 3a1203b2f83d.
  • Every future merge touching a non-excluded path will fail the same way until the registry credential
    is fixed. That is now most merges — tools/, scripts/, apps/, packages/ other than mosaic.
  • The ci workflow is a separate context and was still running when this was written; it is unaffected by
    the registry credential.
  • enable_status_check=false on this repo, so a red publish does not block anything at the provider.
    It blocks nothing mechanically and it publishes no images.

What this does not change

I am not attributing this to #1105, and nobody else should. #1105 is a shell-hardening change that
touches no build input, no Dockerfile and no registry configuration. Its own PR-branch CI was terminal-green
(pipeline 2274, 9/9), and the merged-tree ci workflow is independent of publish.

Registry credentials are an operator concern and I am not touching them.

## This is no longer dormant — it fired on `main` today, and here is the log `#1105` merged as `3a1203b2f83d`. Its merged-`main` run (**pipeline 2275**) failed all three image builds: ``` publish/build-gateway failure publish/build-appservice failure publish/build-web failure ``` **Cause confirmed from the step log, not inferred:** ``` + echo "{\"auths\":{\"git.********stack.dev\":{\"username\":\"$REGISTRY_USER\",\"password\":\"$REGISTRY_PASS\"}}}" > /kaniko/.docker/config.json error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permissions ``` kaniko writes the registry credential and then cannot authenticate to push. That is this issue's defect, reproducing. ## Why it has looked quiet — and why that was never evidence This issue records the defect as "dormant behind a path filter." That reading is now confirmed and the mechanism is precise: `publish.yml` excludes `packages/mosaic/**`, `docs/**`, `**/*.md` and `.woodpecker/**`, so **any merge touching only those paths never runs the image builds at all.** Recent merges did exactly that: | merge | changed paths | publish steps run | image builds | |---|---|---|---| | `#1100` → `df4c591a` | `packages/mosaic/**`, `docs/**` | 4 | **absent** | | `#1105` → `3a1203b2f83d` | `tools/*.sh`, `scripts/*` | 7 | **ran, all 3 failed** | **Before merging `#1105` I stated in advance that its run would exercise the container path and that a red there would be this issue surfacing rather than a defect in that PR.** That is why the result is usable: it was declared non-evidence about `#1105` before it existed. The 4-vs-7 step count is the measured discriminator — "this run exercises the path" is checkable, not asserted. ## What this changes - **`main` is red on `ci/woodpecker/push/publish` right now** at `3a1203b2f83d`. - **Every future merge touching a non-excluded path will fail the same way** until the registry credential is fixed. That is now most merges — `tools/`, `scripts/`, `apps/`, `packages/` other than `mosaic`. - The `ci` workflow is a separate context and was still running when this was written; it is unaffected by the registry credential. - **`enable_status_check=false` on this repo**, so a red `publish` does not block anything at the provider. It blocks nothing mechanically and it publishes no images. ## What this does not change I am not attributing this to `#1105`, and nobody else should. `#1105` is a shell-hardening change that touches no build input, no Dockerfile and no registry configuration. Its own PR-branch CI was terminal-green (pipeline 2274, 9/9), and the merged-tree `ci` workflow is independent of `publish`. Registry credentials are an operator concern and I am not touching them.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1095