mosaicstack/stack
1
0
Fork 0
Code Issues 34 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

fix(db): stop pglite migration tests flaking CI (timeout + WASM OOM) #647

Merged
jason.woltje merged 2 commits from fix/db-pglite-test-flake into main 2026-06-24 05:04:29 +00:00
Conversation 0 Commits 2 Files Changed 3 +25
jason.woltje commented 2026-06-24 04:30:22 +00:00
Owner
Copy Link

Problem

CI has been failing non-deterministically across PRs — the same SHA passes pr/ci and fails push/ci, then on a re-run the failure jumps to the other pipeline. Every failure is the same:

FAIL  packages/db/src/migrate.test.ts > runPgliteMigrations > ...
  Error: Test timed out in 5000ms.
  → memory access out of bounds   (wasm:/wasm/01edd1ba:...)

migrate.test.ts spins up a real PGlite (WASM Postgres) instance per test and applies the full drizzle migration set. Each case takes ~3-5s locally (RSS ~705 MB for this file alone) and longer on CI, where turbo runs ~20 packages’ suites concurrently. Two compounding causes:

  1. Timeout — vitest’s default per-test timeout is 5000 ms, but a migration legitimately needs 5-46s. Under CI load it expires mid-migration → phantom Test timed out in 5000ms.
  2. WASM OOM — each PGlite heap is multi-hundred-MB; running test files in parallel forks multiplies peak memory and tips the runner into memory access out of bounds.

This blocks every PR intermittently (observed on #645 across pipelines 1447/1452).

Fix (packages/db/vitest.config.ts)

  • testTimeout / hookTimeout → 120 s so slow-but-correct migrations finish.
  • pool: 'forks' + singleFork: true so only one PGlite WASM instance is resident at a time — deterministic, modest slowdown.

Plus register packages/db/vitest.config.ts in eslint’s allowDefaultProject (next to the existing gateway/storage vitest configs) so the typed lint can parse the now-non-trivial config.

Verification

Full packages/db suite run 3× locally with the new config: all green, ~13 s each, no timeouts, no OOM. eslint clean on both changed files.

🤖 Generated with Claude Code

## Problem CI has been failing **non-deterministically** across PRs — the same SHA passes `pr/ci` and fails `push/ci`, then on a re-run the failure jumps to the *other* pipeline. Every failure is the same: ``` FAIL packages/db/src/migrate.test.ts > runPgliteMigrations > ... Error: Test timed out in 5000ms. → memory access out of bounds (wasm:/wasm/01edd1ba:...) ``` `migrate.test.ts` spins up a real **PGlite** (WASM Postgres) instance per test and applies the full drizzle migration set. Each case takes ~3-5s locally (RSS ~705 MB for this file alone) and longer on CI, where turbo runs ~20 packages’ suites concurrently. Two compounding causes: 1. **Timeout** — vitest’s default per-test timeout is **5000 ms**, but a migration legitimately needs 5-46s. Under CI load it expires mid-migration → phantom `Test timed out in 5000ms`. 2. **WASM OOM** — each PGlite heap is multi-hundred-MB; running test files in parallel forks multiplies peak memory and tips the runner into `memory access out of bounds`. This blocks **every** PR intermittently (observed on #645 across pipelines 1447/1452). ## Fix (`packages/db/vitest.config.ts`) - `testTimeout` / `hookTimeout` → **120 s** so slow-but-correct migrations finish. - `pool: 'forks'` + `singleFork: true` so only one PGlite WASM instance is resident at a time — deterministic, modest slowdown. Plus register `packages/db/vitest.config.ts` in eslint’s `allowDefaultProject` (next to the existing gateway/storage vitest configs) so the typed lint can parse the now-non-trivial config. ## Verification Full `packages/db` suite run **3× locally** with the new config: all green, ~13 s each, **no timeouts, no OOM**. `eslint` clean on both changed files. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
jason.woltje added 1 commit 2026-06-24 04:30:23 +00:00
fix(db): stop pglite migration tests flaking CI on timeout + WASM OOM
Some checks failed
ci/woodpecker/push/ci Pipeline failed
Details
ci/woodpecker/pr/ci Pipeline was successful
Details
80570f7040 
packages/db's migrate.test.ts spins up a real PGlite (WASM Postgres)
instance per test and applies the full drizzle migration set. Each case
takes ~3-5s locally and longer on CI, where turbo runs ~20 packages'
suites concurrently. Two failure modes resulted, bouncing between the
push/ci and pr/ci pipelines on identical SHAs:

  FAIL src/migrate.test.ts > runPgliteMigrations > ...
    Error: Test timed out in 5000ms.
    -> memory access out of bounds  (wasm:/wasm/...)

1. The 5s vitest default timeout expires mid-migration -> phantom
   'Test timed out in 5000ms'. Raise testTimeout/hookTimeout to 120s so
   legitimately-slow migrations finish.
2. Each PGlite WASM heap is multi-hundred-MB (RSS ~705MB for this file
   alone); parallel forks multiply the peak and tip the runner into the
   WASM OOM. Pin the package to a single fork so only one instance is
   resident at a time.

Also register packages/db/vitest.config.ts in eslint's allowDefaultProject
(alongside the gateway/storage vitest configs) so the typed lint can parse
the now-non-trivial config.

Verified: full db suite green 3x locally with the new config; each run
~13s, no timeouts, no OOM. eslint clean on both files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jason.woltje added 1 commit 2026-06-24 04:37:57 +00:00
fix(ci): gitignore vite/vitest *.timestamp-*.mjs to stop turbo traversal race
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Details
ci/woodpecker/push/ci Pipeline was successful
Details
7210b7391a 
The push/ci lint step intermittently failed with:

  x Package traversal error: .../packages/macp/vitest.config.ts.timestamp-
    <n>.mjs: IO error ... No such file or directory (os error 2)

vite/vitest/esbuild write a transient *.timestamp-*.mjs next to a TS
config while loading it, then unlink it. The files were untracked but not
ignored, so turbo's package traversal hashed them and raced the unlink.
Ignoring them excludes them from turbo's input set and removes the race.

Same class of fix as the pglite timeout/OOM change in this PR: transient
test tooling artifacts destabilising CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jason.woltje merged commit 6980e40e51 into main 2026-06-24 05:04:29 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#647
Delete Branch "fix/db-pglite-test-flake"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?