mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

publish.yml: next lane runs build-gateway on docs-only merges (no path exclusion) + no step-log wrapper for attribution #1133

New Issue
Open
opened 2026-08-09 10:42:06 +00:00 by Ghost · 3 comments
Ghost commented 2026-08-09 10:42:06 +00:00
Copy Link
Copy Source

Post-merge pipeline 2289 on next head 24bbd40d (docs-only merge #1131, two .md files) went terminal-failure on aggregate context ci/woodpecker/push/publish.

Step results (via pipeline-status.sh): clone/install/build/publish-next-npm all success; build-gateway (Kaniko): failure, exit 1.

Two defects:

  1. .woodpecker/publish.yml: image_build_when excludes docs/Markdown paths for main, but the next entry has NO path exclusion — every docs-only next merge rebuilds the gateway image (and fails, currently). Pre-existing: base 4df478cd carries the identical failure, so this is a lane condition, not introduced by any recent merge. Make next-lane image gating path-aware like main (or document why every next push must rebuild).
  2. Attribution gap: Mosaic Woodpecker wrappers expose step STATE but not step LOGS, so the actual Kaniko failure cause (registry auth vs build vs runner infra) cannot be determined without raw API work. Add an audited log-read wrapper (e.g. pipeline-logs.sh -r <repo> -n <pipeline> -s <step>).

Operational impact: every next push trips --require-status merge guards on this red context until fixed.

Evidence: fred base-vs-merge provider status comparison + velma pipeline-status.sh step binding, 2026-08-09.

Post-merge pipeline 2289 on `next` head `24bbd40d` (docs-only merge #1131, two .md files) went terminal-failure on aggregate context `ci/woodpecker/push/publish`. **Step results (via pipeline-status.sh):** clone/install/build/publish-next-npm all success; **`build-gateway` (Kaniko): failure, exit 1**. **Two defects:** 1. `.woodpecker/publish.yml`: `image_build_when` excludes docs/Markdown paths for `main`, but the `next` entry has NO path exclusion — every docs-only next merge rebuilds the gateway image (and fails, currently). Pre-existing: base `4df478cd` carries the identical failure, so this is a lane condition, not introduced by any recent merge. Make next-lane image gating path-aware like main (or document why every next push must rebuild). 2. **Attribution gap:** Mosaic Woodpecker wrappers expose step STATE but not step LOGS, so the actual Kaniko failure cause (registry auth vs build vs runner infra) cannot be determined without raw API work. Add an audited log-read wrapper (e.g. `pipeline-logs.sh -r <repo> -n <pipeline> -s <step>`). **Operational impact:** every next push trips `--require-status` merge guards on this red context until fixed. Evidence: fred base-vs-merge provider status comparison + velma `pipeline-status.sh` step binding, 2026-08-09.
Ghost commented 2026-08-09 20:02:19 +00:00
Copy Link
Copy Source

Scope expansion — the stall is NOT next-lane-only (evidence: shaggy, mosaic.woltje.com deployment trace, 2026-08-09).

Newest images in mosaicstack/stack/{web,gateway,appservice} are sha-712c770 dated 2026-07-26, while main is b0f7d26d (2026-08-07) — 41 commits later, and those commits touch buildable paths (apps/gateway, apps/web/package.json, scripts/build-web.mjs, package.json). The main lane's docs/md path exclusion therefore does NOT explain the gap: image publishing appears stalled on main as well, consistent with the same build-gateway (Kaniko exit 1) failure this issue documents on next.

Downstream impact: any redeploy of mosaic.woltje.com is capped at ~2-weeks-stale images until this is fixed. Full trace: jarvis-brain docs/scratchpads/mosaic-woltje-deployment-trace.md.

— posted by fred on behalf of shaggy (read-only trace, no deployment changes made)

**Scope expansion — the stall is NOT next-lane-only (evidence: shaggy, mosaic.woltje.com deployment trace, 2026-08-09).** Newest images in `mosaicstack/stack/{web,gateway,appservice}` are `sha-712c770` dated 2026-07-26, while `main` is `b0f7d26d` (2026-08-07) — 41 commits later, and those commits touch buildable paths (`apps/gateway`, `apps/web/package.json`, `scripts/build-web.mjs`, `package.json`). The `main` lane's docs/md path exclusion therefore does NOT explain the gap: image publishing appears stalled on `main` as well, consistent with the same `build-gateway` (Kaniko exit 1) failure this issue documents on `next`. Downstream impact: any redeploy of mosaic.woltje.com is capped at ~2-weeks-stale images until this is fixed. Full trace: jarvis-brain `docs/scratchpads/mosaic-woltje-deployment-trace.md`. — posted by fred on behalf of shaggy (read-only trace, no deployment changes made)
Ghost commented 2026-08-09 20:29:04 +00:00
Copy Link
Copy Source

ROOT CAUSE FOUND — the image builds are deterministically broken at source; path gating was never the cause of the exit-1

Credit: shaggy (reproduced locally on a clean git archive extract of origin/main, both Dockerfiles); independently verified by fred against origin/main.

Primary break — every image build, gateway and web

Root package.json has "prepare": "node scripts/install-hooks.mjs" (introduced by f58b3699 / #1027, 2026-08-01). Both docker/gateway.Dockerfile and docker/web.Dockerfile run pnpm install --frozen-lockfile (gateway line 13) before COPY . . (line 14) — so scripts/ is not in the image when pnpm runs the root prepare lifecycle:

. prepare$ node scripts/install-hooks.mjs
. prepare: Error: Cannot find module '/app/scripts/install-hooks.mjs' (MODULE_NOT_FOUND)
ELIFECYCLE Command failed with exit code 1

Timeline is conclusive: last published image sha-712c770 = 2026-07-26; git merge-base --is-ancestor f58b3699 712c770b → NO. The hook landed after the last good image and nothing has published since. This is the Kaniko build-gateway exit-1 on both main and next.

Second, independent break — web only

docker/web.Dockerfile:13 runs pnpm --filter @mosaic/web build, but the package is named @mosaicstack/web (both branches). pnpm treats a zero-match filter as success (exit 0), so nothing builds and the failure surfaces two stages later at COPY --from=builder /app/apps/web/.next/standalone → not found. Fixing the prepare break alone still leaves web broken.

Fix recommendation

  1. Make scripts/install-hooks.mjs a no-op (exit 0) when git is absent or cwd is not inside a work tree. A prepare hook whose job is installing git hooks must not fail container builds or bare CI checkouts. Note: merely adding COPY scripts/ does NOT work — node:22-alpine has no git and there is no .git in the build context, so the hook then fails with spawn git ENOENT (shaggy tested this).
  2. docker/web.Dockerfile: rename the filter to @mosaicstack/web.
  3. gateway/appservice filter their correct names and should recover with fix 1 alone.

pnpm install --ignore-scripts in the Dockerfiles is the fallback if 1 is rejected; shaggy is running a full build with it now to confirm no downstream postinstall is load-bearing.

Scope of the rest of this issue

The two original defects stand but are now secondary: publish.yml's missing path exclusions on next (docs-only merges trigger doomed rebuilds) and the missing step-log wrapper (which is why this root cause had to be found by local reproduction instead of by reading CI logs).

## ROOT CAUSE FOUND — the image builds are deterministically broken at source; path gating was never the cause of the exit-1 Credit: shaggy (reproduced locally on a clean `git archive` extract of origin/main, both Dockerfiles); independently verified by fred against origin/main. ### Primary break — every image build, gateway and web Root `package.json` has `"prepare": "node scripts/install-hooks.mjs"` (introduced by `f58b3699` / #1027, 2026-08-01). Both `docker/gateway.Dockerfile` and `docker/web.Dockerfile` run `pnpm install --frozen-lockfile` (gateway line 13) **before** `COPY . .` (line 14) — so `scripts/` is not in the image when pnpm runs the root prepare lifecycle: ``` . prepare$ node scripts/install-hooks.mjs . prepare: Error: Cannot find module '/app/scripts/install-hooks.mjs' (MODULE_NOT_FOUND) ELIFECYCLE Command failed with exit code 1 ``` **Timeline is conclusive**: last published image `sha-712c770` = 2026-07-26; `git merge-base --is-ancestor f58b3699 712c770b` → NO. The hook landed after the last good image and nothing has published since. This is the Kaniko `build-gateway` exit-1 on both `main` and `next`. ### Second, independent break — web only `docker/web.Dockerfile:13` runs `pnpm --filter @mosaic/web build`, but the package is named `@mosaicstack/web` (both branches). pnpm treats a zero-match filter as success (exit 0), so nothing builds and the failure surfaces two stages later at `COPY --from=builder /app/apps/web/.next/standalone` → not found. Fixing the prepare break alone still leaves web broken. ### Fix recommendation 1. **Make `scripts/install-hooks.mjs` a no-op (exit 0) when git is absent or cwd is not inside a work tree.** A prepare hook whose job is installing git hooks must not fail container builds or bare CI checkouts. Note: merely adding `COPY scripts/` does NOT work — node:22-alpine has no git and there is no `.git` in the build context, so the hook then fails with `spawn git ENOENT` (shaggy tested this). 2. **`docker/web.Dockerfile`: rename the filter to `@mosaicstack/web`.** 3. gateway/appservice filter their correct names and should recover with fix 1 alone. `pnpm install --ignore-scripts` in the Dockerfiles is the fallback if 1 is rejected; shaggy is running a full build with it now to confirm no downstream postinstall is load-bearing. ### Scope of the rest of this issue The two original defects stand but are now secondary: `publish.yml`'s missing path exclusions on `next` (docs-only merges trigger doomed rebuilds) and the missing step-log wrapper (which is why this root cause had to be found by local reproduction instead of by reading CI logs).
Ghost commented 2026-08-09 20:41:13 +00:00
Copy Link
Copy Source

Follow-up evidence (shaggy): the --ignore-scripts alternative is confirmed insufficient as a one-line fix — and both images now build green locally.

  1. Second trigger site: patching --ignore-scripts onto pnpm install (gateway.Dockerfile:13) still fails — pnpm --filter @mosaicstack/gateway --prod deploy --legacy (line 19) is a separate pnpm invocation and runs the root prepare again (spawn git ENOENT). Only patching BOTH invocations goes green. So the per-Dockerfile variant means chasing every pnpm call in every Dockerfile forever — this is the negative result on option (a), same class as the earlier COPY-scripts/ negative result. The one-site fix stands: scripts/install-hooks.mjs should exit 0 when git is absent or cwd is not a work tree (the script already has the error handling and remediation string; it just exits non-zero).
  2. Both images verified buildable and bootable with the two workarounds applied: web (Next 16.1.6 boots, HTTP 307) and gateway (boots into Nest bootstrap). This also confirms no downstream postinstall script is load-bearing for either image, removing the last doubt about the no-op fix.
  3. Gateway boot surfaced two separate runtime defects, filed independently: step-ca hard requirement (FederationModule unconditional) and the 768/1536 embedding-dimension default mismatch.
**Follow-up evidence (shaggy): the `--ignore-scripts` alternative is confirmed insufficient as a one-line fix — and both images now build green locally.** 1. **Second trigger site**: patching `--ignore-scripts` onto `pnpm install` (gateway.Dockerfile:13) still fails — `pnpm --filter @mosaicstack/gateway --prod deploy --legacy` (line 19) is a separate pnpm invocation and runs the root `prepare` again (`spawn git ENOENT`). Only patching BOTH invocations goes green. So the per-Dockerfile variant means chasing every pnpm call in every Dockerfile forever — this is the negative result on option (a), same class as the earlier COPY-scripts/ negative result. The one-site fix stands: `scripts/install-hooks.mjs` should exit 0 when git is absent or cwd is not a work tree (the script already has the error handling and remediation string; it just exits non-zero). 2. **Both images verified buildable and bootable** with the two workarounds applied: web (Next 16.1.6 boots, HTTP 307) and gateway (boots into Nest bootstrap). This also confirms no downstream postinstall script is load-bearing for either image, removing the last doubt about the no-op fix. 3. Gateway boot surfaced two separate runtime defects, filed independently: step-ca hard requirement (FederationModule unconditional) and the 768/1536 embedding-dimension default mismatch.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1133