stack/agents/sage/work/vikunja-setup/536-design.md

Design for #536, Woodpecker caching, revised under mos-cve-120 and mos-cve-123, with Mos's rulings R1-R5 from mos-cve-126 folded in. It's design only: no pipeline runs, no cluster changes and no MinIO deletes. Packages go to Gitea only, since Harbor is gone. Trust covers two repos and no others: dyor-foundation/dyor-stack and sage/sage-phr.

Measurement limits. I couldn't read the pipeline logs. The ci.mosaicstack.dev API answers 401 without a token, and I didn't use anyone else's. So the numbers come from the two pipeline files on main plus Mos's upload sizes, not from per-step logs. Treat them as estimates.

What a run downloads today. Both pipelines restore and save the pnpm store through drone-cache to MinIO, keyed on the pnpm-lock checksum. The saves never complete (#527), so every install is cold. dyor-stack's install step fetches the whole store once per workflow. On main and develop pushes, each of its four kaniko builds, gateway, market, web and analytics, runs another cold pnpm install inside its Dockerfile. sage-phr does the same with three builds: api, web and landing. Mos measured store uploads of 597,715,196 B in pipeline 535 and 1,127,269,785 B in 536. sage-phr's save also tars every node_modules directory, so I think the larger one is sage-phr. The logs would have to confirm that.

Estimate. Call one store download S. A pull request costs about one S plus step images. A main or develop push costs about 5 S for dyor-stack and 4 S for sage-phr. With S between 0.6 and 1.1 GB, a push costs roughly 3 to 6 GB. Step images are smaller and mostly already sit on the node: node:22-alpine, node:22-bookworm, postgres:17 pinned by digest, the kaniko debug image and meltwater/drone-cache. dyor-stack also runs apk add postgresql17 twice per run, and npx prisma generate fetches Prisma engines every run, outside the pnpm store.

The npm proxy. Mos's belief holds. Gitea's npm registry hosts packages published to it, and its docs describe no proxy or upstream for npmjs. O1 covers repeat installs, so we don't need an npm proxy.

Order. O1 with O4 first, then O2. O3 is conditional, and the YAML already says pushes will meet the condition. After O1, the kaniko installs are 4 of 5 store downloads on a dyor-stack push and 3 of 4 on a sage-phr push. I still want one measured push after O1 before the O3 PRs go in, as mos-cve-123 asks.

O1, a persistent pnpm store. One PVC per repo and per trust level, so four: pnpm-store-dyor-main, pnpm-store-dyor-pr, pnpm-store-phr-main and pnpm-store-phr-pr. The install step mounts the right one at /pnpm-store and runs pnpm install --frozen-lockfile --store-dir /pnpm-store/v10. COREPACK_HOME lives on the same volume, so steps stop fetching pnpm itself. dyor-stack points its Prisma engine cache there too. The restore-cache and rebuild-cache steps go away. That stops writes to MinIO but deletes nothing, so MinIO retirement still waits for Jason. pnpm can't hardlink across volumes into the workspace, so it copies. That costs local disk IO, not network.

Storage class, the open cluster question. Each workflow already gets a strict-local, one-replica longhorn-ci PVC. A step pod that needs that volume and a strict-local store PVC bound on a different node can't schedule. If k8s-mosaic has one tenant=mosaic node, strict-local RWO works, because RWO lets several pods on one node share the volume and MAX_WORKFLOWS is 2. With more than one node, the store PVCs need RWX through Longhorn's share manager, or every CI pod pins to one node. ops-01 reports the node count first.

Size and the disk gate. 5 GiB each, about three times the larger upload. Longhorn allocates as data arrives, so 20 GiB for the four is the ceiling, not the start. The four PVCs are created only when k8s-mosaic shows at least 30 GiB free, through the #504 disk growth or Jason's #534 D4. This design doesn't count on the leftover wp- volumes being deleted; that's a data deletion and Jason's call in D4. If the space isn't there, the stores shrink, the PR stores first, or creation waits. A weekly Woodpecker cron pipeline in each repo runs pnpm store prune on both of its stores.

O4, PR and main split. Mos's text says caches save on main only and PRs only restore. A PVC store has no separate save step, so I split by volume. Push and tag events on main and develop mount the main store, and pull requests mount the PR store. A PR never writes the store main builds use. Woodpecker's kubernetes backend documents no read-only volume mount, and pnpm must write when a PR adds a package, so a read-only PR mount isn't an option. Mos, this departs from your wording, so please rule on it.

Trust residual. Named volumes need the repo marked trusted, and only these two repos get that. The pipeline file enforces the split, and a PR's pipeline file comes from the PR branch. So a PR that edits .woodpecker files could mount the main store, or the other repo's PVC in the same namespace. Content addressing doesn't stop this, because code in a step can rewrite both a package's index entry and its files. Mos accepted the split and the residual (R1). Review rule for both repos: pull request pipelines need approval, and the approver reads every .woodpecker diff in the PR before approving. Anyone who can push a branch to these repos can still reach the main store through an approved PR. That's the residual.

The rest of O4. The disk alert in #389 goes to someone who reads it. Cleanup of the leftover wp- pods and volumes stays in #534 D4 for Jason.

O2, next. Pin every step image by digest, so a digest already on the node isn't pulled again. Build one CI base image into Gitea packages with node 22.23.4, pnpm 10.25.0 through corepack, the postgresql17 client and the tools the steps install today. That removes corepack enable and corepack prepare from every step and both apk add postgresql17 calls in dyor-stack. Each repo gets one PR to switch its steps to the base image.

O3, conditional. My preferred form drops the install inside kaniko. A CI step runs pnpm deploy --filter APP --prod into the workspace from the warm store, and each Dockerfile copies that output into its runtime image. The runtime image's libc must match the step that built the output: musl on alpine for dyor-stack, glibc on bookworm for sage-phr. dyor-stack's web image also installs [email protected].2 in its Dockerfile, and the engine target has to match the runtime image. If that route fails, the fallback is Mos's BuildKit or registry layer cache in Gitea. Gitea's docs call its registry OCI-compliant but say nothing about cache manifests, so one verification push comes before any PR relies on it. As far as I know, upstream kaniko was archived in 2025. ops-01 confirms that before PR3 (R5).

PR plan. PR1 in each repo is O1 and O4: store mounts, the PR and main split, drone-cache removed, the prune cron. PR2 in each is O2: digests and the base image. PR3 in each is O3 by pnpm deploy, after one measured push. One more PR builds the base image, in a repo ops-01 picks. Cluster parts, each on Mos's go, with anything that deletes going to Jason (R4): C1 report the node count, storage classes and free space. C2 create the four PVCs once the 30 GiB gate holds. C3 mark the two repos trusted and turn on PR approval. C4 wire the #389 alert. The dyor-stack and sage-phr pipeline hold stands, so PR1 can be built and reviewed but doesn't run on main until Mos lifts it.

Expected savings, estimated. After PR1, a PR run downloads little beyond new packages, down from one full store. A push saves one store of about five for dyor-stack and one of four for sage-phr. After PR3, a push also stops paying for the kaniko installs: roughly 2.4 to 4.4 GB per dyor-stack push and 1.8 to 3.3 GB per sage-phr push, for S between 0.6 and 1.1 GB.

Open items. 1. Nobody in the fleet can read the 535 or 536 logs (R3), so the measurement comes from the first push after O1. PR1 keeps pnpm's resolved, reused and downloaded lines in the install step output for that. 2. ops-01's node count decides RWO or RWX. 3. The 30 GiB gate decides when the PVCs appear. 4. MinIO retirement and the wp- cleanup wait for Jason.