Files
stack/packages/db/src/migrate.ts
jason.woltje ac5650d9f9 fix(db): bootstrap migrations on local-tier gateway startup
Fresh `mosaic gateway install` (npm) left the gateway DB schema empty —
sign-in 500'd with `relation "users" does not exist`, and every entry
point (auth, bootstrap setup) failed because they all query the users
table first. Five stacked bugs on the local (PGlite) tier:

1. `packages/db/package.json` `files: ["dist"]` excluded the `drizzle/`
   SQL migrations from the published tarball.
2. `runMigrations()` only supports postgres-js — unusable for embedded
   PGlite.
3. `apps/gateway/src/database/database.module.ts` never invoked
   migrations at startup.
4. `createPgliteDb` didn't load pgvector, so migration 0001's
   `CREATE EXTENSION vector` failed.
5. Drizzle's PG migrator wraps every migration in one outer
   transaction, which trips Postgres' `check_safe_enum_use` on
   migration 0009 (`ALTER TYPE ADD VALUE 'pending'` → `SET DEFAULT
   'pending'` in the same tx).

Changes:
- Ship `drizzle/` in the published tarball.
- `createPgliteDb` loads `@electric-sql/pglite/vector`.
- New `runPgliteMigrations(handle)` walks the Drizzle journal and
  runs each statement-breakpoint chunk through PGlite's `client.exec()`
  (autocommit per statement). Records into `drizzle.__drizzle_migrations`
  for interop with the postgres-js path. Per-statement try/catch
  surfaces which statement of which migration failed.
- `DatabaseModule` runs migrations in `OnModuleInit` before
  `app.listen()`. Local tier: explicit `runPgliteMigrations` then
  `storageAdapter.migrate()`. Postgres tier: just `storageAdapter.migrate()`,
  which already calls `runMigrations(url)` internally — no double-call.
- Removed `packages/storage/src/test-utils/pglite-with-vector.ts`. The
  "intentionally not exported" rationale is moot now that migration
  0001 forces pgvector load anyway. The integration test uses
  `createPgliteDb` + `runPgliteMigrations` from `@mosaicstack/db`.

Tests: BetterAuth tables exist after migrate; idempotent (re-runs 0009);
partial-failure surfaces statement-level context and leaves no ledger row.

QA on a fresh PGlite install:
- `Applying PGlite schema migrations...` then `Initializing storage
  adapter (pglite)...` in startup log.
- `GET /api/bootstrap/status` → `{"needsSetup":true}` HTTP 200 (was 500).
- `POST /api/bootstrap/setup` reaches Zod validator (was 500).

Scope: this PR fixes the local (PGlite) tier. Postgres-tier first
install still has the outer-transaction problem and a journal ordering
bug (0009's `when` < 0008's). Documented inline as TODO and in the
scratchpad — needs a separate change with real-Postgres validation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 17:06:50 -05:00

110 lines
4.7 KiB
TypeScript

import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { sql } from 'drizzle-orm';
import { drizzle as drizzlePostgres } from 'drizzle-orm/postgres-js';
import { migrate as migratePostgres } from 'drizzle-orm/postgres-js/migrator';
import { readMigrationFiles } from 'drizzle-orm/migrator';
import postgres from 'postgres';
import { DEFAULT_DATABASE_URL } from './defaults.js';
import type { DbHandle } from './client.js';
interface PgliteExecutor {
exec(query: string): Promise<unknown>;
}
interface ExecuteRows<T> {
rows: T[];
}
function migrationsFolder(): string {
const here = dirname(fileURLToPath(import.meta.url));
return resolve(here, '../drizzle');
}
export async function runMigrations(url?: string): Promise<void> {
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
const sqlClient = postgres(connectionString, { max: 1 });
const db = drizzlePostgres(sqlClient);
try {
// TODO: postgres-tier first-install also fails because (a) Drizzle wraps every
// migration in one transaction (breaks 0009's ALTER TYPE ADD VALUE → SET DEFAULT
// sequence) and (b) drizzle/meta/_journal.json has 0009 ordered before 0008,
// which the postgres-js migrator skips by `created_at < folderMillis`. The
// PGlite path below sidesteps both. A follow-up should either share the
// per-statement loop (see runPgliteMigrations) or fix the journal ordering.
await migratePostgres(db, { migrationsFolder: migrationsFolder() });
} finally {
await sqlClient.end();
}
}
// Apply Drizzle migrations against an embedded PGlite database.
//
// We don't reuse drizzle's pglite migrator because it wraps ALL migrations in
// one outer transaction, which breaks Postgres' `check_safe_enum_use` rule —
// e.g. migration 0009 does `ALTER TYPE ADD VALUE 'pending'` then references
// `'pending'` as a default in the same tx. PGlite's `exec()` runs each
// statement under the Simple Query protocol, autocommitting between them.
//
// We still write to the standard `drizzle.__drizzle_migrations` ledger so the
// result is interoperable with `runMigrations()` on a postgres-backed deploy
// (modulo the journal-ordering bug noted above).
//
// We skip-by-hash rather than skip-by-folderMillis (which is what Drizzle's
// postgres-js migrator does). That's deliberate — out-of-order timestamps in
// `_journal.json` won't silently drop migrations.
//
// Failure model: each statement autocommits, and the ledger row is written
// only after all statements in a migration succeed. A crash mid-migration
// leaves the prefix applied with no ledger entry, so the next boot will
// replay those statements and fail loudly on "already exists". Recovery:
// drop the partially-applied objects, or insert the migration's hash into
// `drizzle.__drizzle_migrations` manually. The error log identifies which
// statement of which migration was the culprit.
export async function runPgliteMigrations(handle: DbHandle): Promise<void> {
const client = (handle.db as unknown as { $client?: PgliteExecutor }).$client;
if (!client || typeof client.exec !== 'function') {
throw new Error('runPgliteMigrations: handle.db is not backed by a PGlite client');
}
await client.exec('CREATE SCHEMA IF NOT EXISTS drizzle');
await client.exec(`
CREATE TABLE IF NOT EXISTS drizzle.__drizzle_migrations (
id SERIAL PRIMARY KEY,
hash text NOT NULL,
created_at bigint
)
`);
const appliedRows = (await handle.db.execute(
sql`SELECT hash FROM drizzle.__drizzle_migrations`,
)) as unknown as ExecuteRows<{ hash: string }>;
const applied = new Set(appliedRows.rows.map((r) => r.hash));
const migrations = readMigrationFiles({ migrationsFolder: migrationsFolder() });
for (const migration of migrations) {
if (applied.has(migration.hash)) continue;
// Run each statement-breakpoint chunk in its own exec() call so PGlite
// commits between statements — this is what lets `ALTER TYPE ADD VALUE`
// become visible before a subsequent statement references the new value.
for (const [stmtIdx, stmt] of migration.sql.entries()) {
const trimmed = stmt.trim();
if (!trimmed) continue;
try {
await client.exec(trimmed);
} catch (err) {
const cause = err instanceof Error ? err.message : String(err);
throw new Error(
`runPgliteMigrations: migration hash=${migration.hash} statement #${stmtIdx} failed: ${cause}\n` +
`Statement: ${trimmed.slice(0, 200)}${trimmed.length > 200 ? '…' : ''}`,
{ cause: err },
);
}
}
await handle.db.execute(
sql`INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES (${migration.hash}, ${migration.folderMillis})`,
);
}
}