fix(db): stop pglite migration tests flaking CI (timeout + WASM OOM) #647

Merged

jason.woltje merged 2 commits from fix/db-pglite-test-flake into main 2026-06-24 05:04:29 +00:00

Conversation 0 Commits 2 Files Changed 3

+25

jason.woltje commented 2026-06-24 04:30:22 +00:00

Owner

Copy Link

Copy Source

Problem

CI has been failing non-deterministically across PRs — the same SHA passes pr/ci and fails push/ci, then on a re-run the failure jumps to the other pipeline. Every failure is the same:

FAIL  packages/db/src/migrate.test.ts > runPgliteMigrations > ...
  Error: Test timed out in 5000ms.
  → memory access out of bounds   (wasm:/wasm/01edd1ba:...)

migrate.test.ts spins up a real PGlite (WASM Postgres) instance per test and applies the full drizzle migration set. Each case takes ~3-5s locally (RSS ~705 MB for this file alone) and longer on CI, where turbo runs ~20 packages’ suites concurrently. Two compounding causes:

1. Timeout — vitest’s default per-test timeout is 5000 ms, but a migration legitimately needs 5-46s. Under CI load it expires mid-migration → phantom Test timed out in 5000ms.
2. WASM OOM — each PGlite heap is multi-hundred-MB; running test files in parallel forks multiplies peak memory and tips the runner into memory access out of bounds.

This blocks every PR intermittently (observed on #645 across pipelines 1447/1452).

Fix (packages/db/vitest.config.ts)

• testTimeout / hookTimeout → 120 s so slow-but-correct migrations finish.
• pool: 'forks' + singleFork: true so only one PGlite WASM instance is resident at a time — deterministic, modest slowdown.

Plus register packages/db/vitest.config.ts in eslint’s allowDefaultProject (next to the existing gateway/storage vitest configs) so the typed lint can parse the now-non-trivial config.

Verification

Full packages/db suite run 3× locally with the new config: all green, ~13 s each, no timeouts, no OOM. eslint clean on both changed files.

🤖 Generated with Claude Code

## Problem CI has been failing **non-deterministically** across PRs — the same SHA passes `pr/ci` and fails `push/ci`, then on a re-run the failure jumps to the *other* pipeline. Every failure is the same: ``` FAIL packages/db/src/migrate.test.ts > runPgliteMigrations > ... Error: Test timed out in 5000ms. → memory access out of bounds (wasm:/wasm/01edd1ba:...) ``` `migrate.test.ts` spins up a real **PGlite** (WASM Postgres) instance per test and applies the full drizzle migration set. Each case takes ~3-5s locally (RSS ~705 MB for this file alone) and longer on CI, where turbo runs ~20 packages’ suites concurrently. Two compounding causes: 1. **Timeout** — vitest’s default per-test timeout is **5000 ms**, but a migration legitimately needs 5-46s. Under CI load it expires mid-migration → phantom `Test timed out in 5000ms`. 2. **WASM OOM** — each PGlite heap is multi-hundred-MB; running test files in parallel forks multiplies peak memory and tips the runner into `memory access out of bounds`. This blocks **every** PR intermittently (observed on #645 across pipelines 1447/1452). ## Fix (`packages/db/vitest.config.ts`) - `testTimeout` / `hookTimeout` → **120 s** so slow-but-correct migrations finish. - `pool: 'forks'` + `singleFork: true` so only one PGlite WASM instance is resident at a time — deterministic, modest slowdown. Plus register `packages/db/vitest.config.ts` in eslint’s `allowDefaultProject` (next to the existing gateway/storage vitest configs) so the typed lint can parse the now-non-trivial config. ## Verification Full `packages/db` suite run **3× locally** with the new config: all green, ~13 s each, **no timeouts, no OOM**. `eslint` clean on both changed files. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jason.woltje added 1 commit 2026-06-24 04:30:23 +00:00

fix(db): stop pglite migration tests flaking CI on timeout + WASM OOM ... 80570f7040

packages/db's migrate.test.ts spins up a real PGlite (WASM Postgres)
instance per test and applies the full drizzle migration set. Each case
takes ~3-5s locally and longer on CI, where turbo runs ~20 packages'
suites concurrently. Two failure modes resulted, bouncing between the
push/ci and pr/ci pipelines on identical SHAs:

  FAIL src/migrate.test.ts > runPgliteMigrations > ...
    Error: Test timed out in 5000ms.
    -> memory access out of bounds  (wasm:/wasm/...)

1. The 5s vitest default timeout expires mid-migration -> phantom
   'Test timed out in 5000ms'. Raise testTimeout/hookTimeout to 120s so
   legitimately-slow migrations finish.
2. Each PGlite WASM heap is multi-hundred-MB (RSS ~705MB for this file
   alone); parallel forks multiply the peak and tip the runner into the
   WASM OOM. Pin the package to a single fork so only one instance is
   resident at a time.

Also register packages/db/vitest.config.ts in eslint's allowDefaultProject
(alongside the gateway/storage vitest configs) so the typed lint can parse
the now-non-trivial config.

Verified: full db suite green 3x locally with the new config; each run
~13s, no timeouts, no OOM. eslint clean on both files.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

jason.woltje added 1 commit 2026-06-24 04:37:57 +00:00

fix(ci): gitignore vite/vitest *.timestamp-*.mjs to stop turbo traversal race ... 7210b7391a

The push/ci lint step intermittently failed with:

  x Package traversal error: .../packages/macp/vitest.config.ts.timestamp-
    <n>.mjs: IO error ... No such file or directory (os error 2)

vite/vitest/esbuild write a transient *.timestamp-*.mjs next to a TS
config while loading it, then unlink it. The files were untracked but not
ignored, so turbo's package traversal hashed them and raced the unlink.
Ignoring them excludes them from turbo's input set and removes the race.

Same class of fix as the pglite timeout/OOM change in this PR: transient
test tooling artifacts destabilising CI.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

jason.woltje merged commit 6980e40e51 into main 2026-06-24 05:04:29 +00:00

jason.woltje referenced this issue from a commit 2026-06-24 05:04:31 +00:00

fix(db): stop pglite migration tests flaking CI (timeout + WASM OOM) (#647)

jason.woltje referenced this issue from a commit 2026-06-24 05:18:03 +00:00

chore(release): mosaic CLI 0.0.42

jason.woltje referenced this pull request 2026-06-24 05:18:39 +00:00

chore(release): mosaic CLI 0.0.42 #648

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Defect in framework tooling or delivered code

fleet-enhancement

Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 f10-coder jason.woltje (Jason Woltje) merge-gate mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-security-02 scooby (Scooby) velma

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#647