fix(db): bootstrap migrations on local-tier gateway startup
Fresh `mosaic gateway install` (npm) left the gateway DB schema empty —
sign-in 500'd with `relation "users" does not exist`, and every entry
point (auth, bootstrap setup) failed because they all query the users
table first. Five stacked bugs on the local (PGlite) tier:
1. `packages/db/package.json` `files: ["dist"]` excluded the `drizzle/`
SQL migrations from the published tarball.
2. `runMigrations()` only supports postgres-js — unusable for embedded
PGlite.
3. `apps/gateway/src/database/database.module.ts` never invoked
migrations at startup.
4. `createPgliteDb` didn't load pgvector, so migration 0001's
`CREATE EXTENSION vector` failed.
5. Drizzle's PG migrator wraps every migration in one outer
transaction, which trips Postgres' `check_safe_enum_use` on
migration 0009 (`ALTER TYPE ADD VALUE 'pending'` → `SET DEFAULT
'pending'` in the same tx).
Changes:
- Ship `drizzle/` in the published tarball.
- `createPgliteDb` loads `@electric-sql/pglite/vector`.
- New `runPgliteMigrations(handle)` walks the Drizzle journal and
runs each statement-breakpoint chunk through PGlite's `client.exec()`
(autocommit per statement). Records into `drizzle.__drizzle_migrations`
for interop with the postgres-js path. Per-statement try/catch
surfaces which statement of which migration failed.
- `DatabaseModule` runs migrations in `OnModuleInit` before
`app.listen()`. Local tier: explicit `runPgliteMigrations` then
`storageAdapter.migrate()`. Postgres tier: just `storageAdapter.migrate()`,
which already calls `runMigrations(url)` internally — no double-call.
- Removed `packages/storage/src/test-utils/pglite-with-vector.ts`. The
"intentionally not exported" rationale is moot now that migration
0001 forces pgvector load anyway. The integration test uses
`createPgliteDb` + `runPgliteMigrations` from `@mosaicstack/db`.
Tests: BetterAuth tables exist after migrate; idempotent (re-runs 0009);
partial-failure surfaces statement-level context and leaves no ledger row.
QA on a fresh PGlite install:
- `Applying PGlite schema migrations...` then `Initializing storage
adapter (pglite)...` in startup log.
- `GET /api/bootstrap/status` → `{"needsSetup":true}` HTTP 200 (was 500).
- `POST /api/bootstrap/setup` reaches Zod validator (was 500).
Scope: this PR fixes the local (PGlite) tier. Postgres-tier first
install still has the outer-transaction problem and a journal ordering
bug (0009's `when` < 0008's). Documented inline as TODO and in the
scratchpad — needs a separate change with real-Postgres validation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,21 @@
|
|||||||
import { mkdirSync } from 'node:fs';
|
import { mkdirSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
import { Global, Inject, Module, type OnApplicationShutdown } from '@nestjs/common';
|
import {
|
||||||
import { createDb, createPgliteDb, type Db, type DbHandle } from '@mosaicstack/db';
|
Global,
|
||||||
|
Inject,
|
||||||
|
Logger,
|
||||||
|
Module,
|
||||||
|
type OnApplicationShutdown,
|
||||||
|
type OnModuleInit,
|
||||||
|
} from '@nestjs/common';
|
||||||
|
import {
|
||||||
|
createDb,
|
||||||
|
createPgliteDb,
|
||||||
|
runPgliteMigrations,
|
||||||
|
type Db,
|
||||||
|
type DbHandle,
|
||||||
|
} from '@mosaicstack/db';
|
||||||
import { createStorageAdapter, type StorageAdapter } from '@mosaicstack/storage';
|
import { createStorageAdapter, type StorageAdapter } from '@mosaicstack/storage';
|
||||||
import type { MosaicConfig } from '@mosaicstack/config';
|
import type { MosaicConfig } from '@mosaicstack/config';
|
||||||
import { MOSAIC_CONFIG } from '../config/config.module.js';
|
import { MOSAIC_CONFIG } from '../config/config.module.js';
|
||||||
@@ -39,12 +52,37 @@ export const STORAGE_ADAPTER = 'STORAGE_ADAPTER';
|
|||||||
],
|
],
|
||||||
exports: [DB, STORAGE_ADAPTER],
|
exports: [DB, STORAGE_ADAPTER],
|
||||||
})
|
})
|
||||||
export class DatabaseModule implements OnApplicationShutdown {
|
export class DatabaseModule implements OnApplicationShutdown, OnModuleInit {
|
||||||
|
private readonly logger = new Logger(DatabaseModule.name);
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
@Inject(DB_HANDLE) private readonly handle: DbHandle,
|
@Inject(DB_HANDLE) private readonly handle: DbHandle,
|
||||||
@Inject(STORAGE_ADAPTER) private readonly storageAdapter: StorageAdapter,
|
@Inject(STORAGE_ADAPTER) private readonly storageAdapter: StorageAdapter,
|
||||||
|
@Inject(MOSAIC_CONFIG) private readonly config: MosaicConfig,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
|
// Migrations must complete before any module that injects DB starts serving
|
||||||
|
// requests. NestJS awaits onModuleInit before app.listen(), and modules that
|
||||||
|
// inject DB are initialized after this one — so all DB-dependent code sees a
|
||||||
|
// populated schema before the first HTTP request lands.
|
||||||
|
//
|
||||||
|
// Local (PGlite) tier: we run gateway-DB migrations explicitly here. The
|
||||||
|
// storage adapter writes to a separate PGlite directory and only manages its
|
||||||
|
// own KV tables, so we still call its migrate() afterwards.
|
||||||
|
//
|
||||||
|
// Postgres tier: PostgresAdapter.migrate() already calls runMigrations() on
|
||||||
|
// the same DATABASE_URL, so a single call covers both the gateway DB and
|
||||||
|
// the storage tables. We deliberately do NOT call runMigrations() here to
|
||||||
|
// avoid opening a second short-lived connection and doubling startup cost.
|
||||||
|
async onModuleInit(): Promise<void> {
|
||||||
|
if (this.config.tier === 'local') {
|
||||||
|
this.logger.log('Applying PGlite schema migrations...');
|
||||||
|
await runPgliteMigrations(this.handle);
|
||||||
|
}
|
||||||
|
this.logger.log(`Initializing storage adapter (${this.storageAdapter.name})...`);
|
||||||
|
await this.storageAdapter.migrate();
|
||||||
|
}
|
||||||
|
|
||||||
async onApplicationShutdown(): Promise<void> {
|
async onApplicationShutdown(): Promise<void> {
|
||||||
await Promise.all([this.handle.close(), this.storageAdapter.close()]);
|
await Promise.all([this.handle.close(), this.storageAdapter.close()]);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -42,6 +42,7 @@
|
|||||||
"access": "public"
|
"access": "public"
|
||||||
},
|
},
|
||||||
"files": [
|
"files": [
|
||||||
"dist"
|
"dist",
|
||||||
|
"drizzle"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
import { PGlite } from '@electric-sql/pglite';
|
import { PGlite } from '@electric-sql/pglite';
|
||||||
|
import { vector } from '@electric-sql/pglite/vector';
|
||||||
import { drizzle } from 'drizzle-orm/pglite';
|
import { drizzle } from 'drizzle-orm/pglite';
|
||||||
import * as schema from './schema.js';
|
import * as schema from './schema.js';
|
||||||
import type { DbHandle } from './client.js';
|
import type { DbHandle } from './client.js';
|
||||||
|
|
||||||
export function createPgliteDb(dataDir: string): DbHandle {
|
export function createPgliteDb(dataDir: string): DbHandle {
|
||||||
const client = new PGlite(dataDir);
|
// pgvector extension is required by migration 0001 (insights.embedding column).
|
||||||
|
const client = new PGlite(dataDir, { extensions: { vector } });
|
||||||
const db = drizzle(client, { schema });
|
const db = drizzle(client, { schema });
|
||||||
return {
|
return {
|
||||||
db: db as unknown as DbHandle['db'],
|
db: db as unknown as DbHandle['db'],
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
export { createDb, type Db, type DbHandle } from './client.js';
|
export { createDb, type Db, type DbHandle } from './client.js';
|
||||||
export { createPgliteDb } from './client-pglite.js';
|
export { createPgliteDb } from './client-pglite.js';
|
||||||
export { runMigrations } from './migrate.js';
|
export { runMigrations, runPgliteMigrations } from './migrate.js';
|
||||||
export * from './schema.js';
|
export * from './schema.js';
|
||||||
export * from './federation.js';
|
export * from './federation.js';
|
||||||
export {
|
export {
|
||||||
|
|||||||
70
packages/db/src/migrate.test.ts
Normal file
70
packages/db/src/migrate.test.ts
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
import { mkdtempSync, rmSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { sql } from 'drizzle-orm';
|
||||||
|
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||||
|
import { createPgliteDb } from './client-pglite.js';
|
||||||
|
import { runPgliteMigrations } from './migrate.js';
|
||||||
|
import type { DbHandle } from './client.js';
|
||||||
|
|
||||||
|
interface PgliteExec {
|
||||||
|
exec(query: string): Promise<unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('runPgliteMigrations', () => {
|
||||||
|
let dataDir: string;
|
||||||
|
let handle: DbHandle;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
dataDir = mkdtempSync(join(tmpdir(), 'mosaic-db-migrate-test-'));
|
||||||
|
handle = createPgliteDb(dataDir);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await handle.close();
|
||||||
|
rmSync(dataDir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('creates the BetterAuth tables required by the gateway', async () => {
|
||||||
|
await runPgliteMigrations(handle);
|
||||||
|
|
||||||
|
const result = (await handle.db.execute(sql`
|
||||||
|
SELECT table_name FROM information_schema.tables
|
||||||
|
WHERE table_schema = 'public'
|
||||||
|
ORDER BY table_name
|
||||||
|
`)) as unknown as { rows: Array<{ table_name: string }> };
|
||||||
|
|
||||||
|
const tables = result.rows.map((r) => r.table_name);
|
||||||
|
|
||||||
|
// Auth tables — required for sign-in / bootstrap to function.
|
||||||
|
expect(tables).toContain('users');
|
||||||
|
expect(tables).toContain('sessions');
|
||||||
|
expect(tables).toContain('accounts');
|
||||||
|
expect(tables).toContain('verifications');
|
||||||
|
|
||||||
|
// Schema sanity check — admin token table consumed by mosaic gateway config.
|
||||||
|
expect(tables).toContain('admin_tokens');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is idempotent — running twice does not error', async () => {
|
||||||
|
await runPgliteMigrations(handle);
|
||||||
|
await expect(runPgliteMigrations(handle)).resolves.toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('surfaces statement-level error context on failure and leaves no ledger row', async () => {
|
||||||
|
// Pre-create a `users` table that conflicts with migration 0000's CREATE TABLE,
|
||||||
|
// forcing it to fail without IF NOT EXISTS.
|
||||||
|
const client = (handle.db as unknown as { $client: PgliteExec }).$client;
|
||||||
|
await client.exec('CREATE TABLE users (sentinel text)');
|
||||||
|
|
||||||
|
await expect(runPgliteMigrations(handle)).rejects.toThrow(
|
||||||
|
/migration hash=[a-f0-9]+ statement #\d+ failed/,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Ledger should be empty — partial application must not pretend to be complete.
|
||||||
|
const ledger = (await handle.db.execute(
|
||||||
|
sql`SELECT count(*)::int AS count FROM drizzle.__drizzle_migrations`,
|
||||||
|
)) as unknown as { rows: Array<{ count: number }> };
|
||||||
|
expect(ledger.rows[0]?.count).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,18 +1,109 @@
|
|||||||
import { dirname, resolve } from 'node:path';
|
import { dirname, resolve } from 'node:path';
|
||||||
import { fileURLToPath } from 'node:url';
|
import { fileURLToPath } from 'node:url';
|
||||||
import { drizzle } from 'drizzle-orm/postgres-js';
|
import { sql } from 'drizzle-orm';
|
||||||
import { migrate } from 'drizzle-orm/postgres-js/migrator';
|
import { drizzle as drizzlePostgres } from 'drizzle-orm/postgres-js';
|
||||||
|
import { migrate as migratePostgres } from 'drizzle-orm/postgres-js/migrator';
|
||||||
|
import { readMigrationFiles } from 'drizzle-orm/migrator';
|
||||||
import postgres from 'postgres';
|
import postgres from 'postgres';
|
||||||
import { DEFAULT_DATABASE_URL } from './defaults.js';
|
import { DEFAULT_DATABASE_URL } from './defaults.js';
|
||||||
|
import type { DbHandle } from './client.js';
|
||||||
|
|
||||||
|
interface PgliteExecutor {
|
||||||
|
exec(query: string): Promise<unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ExecuteRows<T> {
|
||||||
|
rows: T[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function migrationsFolder(): string {
|
||||||
|
const here = dirname(fileURLToPath(import.meta.url));
|
||||||
|
return resolve(here, '../drizzle');
|
||||||
|
}
|
||||||
|
|
||||||
export async function runMigrations(url?: string): Promise<void> {
|
export async function runMigrations(url?: string): Promise<void> {
|
||||||
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
|
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
|
||||||
const sql = postgres(connectionString, { max: 1 });
|
const sqlClient = postgres(connectionString, { max: 1 });
|
||||||
const db = drizzle(sql);
|
const db = drizzlePostgres(sqlClient);
|
||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
||||||
try {
|
try {
|
||||||
await migrate(db, { migrationsFolder: resolve(__dirname, '../drizzle') });
|
// TODO: postgres-tier first-install also fails because (a) Drizzle wraps every
|
||||||
|
// migration in one transaction (breaks 0009's ALTER TYPE ADD VALUE → SET DEFAULT
|
||||||
|
// sequence) and (b) drizzle/meta/_journal.json has 0009 ordered before 0008,
|
||||||
|
// which the postgres-js migrator skips by `created_at < folderMillis`. The
|
||||||
|
// PGlite path below sidesteps both. A follow-up should either share the
|
||||||
|
// per-statement loop (see runPgliteMigrations) or fix the journal ordering.
|
||||||
|
await migratePostgres(db, { migrationsFolder: migrationsFolder() });
|
||||||
} finally {
|
} finally {
|
||||||
await sql.end();
|
await sqlClient.end();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply Drizzle migrations against an embedded PGlite database.
|
||||||
|
//
|
||||||
|
// We don't reuse drizzle's pglite migrator because it wraps ALL migrations in
|
||||||
|
// one outer transaction, which breaks Postgres' `check_safe_enum_use` rule —
|
||||||
|
// e.g. migration 0009 does `ALTER TYPE ADD VALUE 'pending'` then references
|
||||||
|
// `'pending'` as a default in the same tx. PGlite's `exec()` runs each
|
||||||
|
// statement under the Simple Query protocol, autocommitting between them.
|
||||||
|
//
|
||||||
|
// We still write to the standard `drizzle.__drizzle_migrations` ledger so the
|
||||||
|
// result is interoperable with `runMigrations()` on a postgres-backed deploy
|
||||||
|
// (modulo the journal-ordering bug noted above).
|
||||||
|
//
|
||||||
|
// We skip-by-hash rather than skip-by-folderMillis (which is what Drizzle's
|
||||||
|
// postgres-js migrator does). That's deliberate — out-of-order timestamps in
|
||||||
|
// `_journal.json` won't silently drop migrations.
|
||||||
|
//
|
||||||
|
// Failure model: each statement autocommits, and the ledger row is written
|
||||||
|
// only after all statements in a migration succeed. A crash mid-migration
|
||||||
|
// leaves the prefix applied with no ledger entry, so the next boot will
|
||||||
|
// replay those statements and fail loudly on "already exists". Recovery:
|
||||||
|
// drop the partially-applied objects, or insert the migration's hash into
|
||||||
|
// `drizzle.__drizzle_migrations` manually. The error log identifies which
|
||||||
|
// statement of which migration was the culprit.
|
||||||
|
export async function runPgliteMigrations(handle: DbHandle): Promise<void> {
|
||||||
|
const client = (handle.db as unknown as { $client?: PgliteExecutor }).$client;
|
||||||
|
if (!client || typeof client.exec !== 'function') {
|
||||||
|
throw new Error('runPgliteMigrations: handle.db is not backed by a PGlite client');
|
||||||
|
}
|
||||||
|
|
||||||
|
await client.exec('CREATE SCHEMA IF NOT EXISTS drizzle');
|
||||||
|
await client.exec(`
|
||||||
|
CREATE TABLE IF NOT EXISTS drizzle.__drizzle_migrations (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
hash text NOT NULL,
|
||||||
|
created_at bigint
|
||||||
|
)
|
||||||
|
`);
|
||||||
|
|
||||||
|
const appliedRows = (await handle.db.execute(
|
||||||
|
sql`SELECT hash FROM drizzle.__drizzle_migrations`,
|
||||||
|
)) as unknown as ExecuteRows<{ hash: string }>;
|
||||||
|
const applied = new Set(appliedRows.rows.map((r) => r.hash));
|
||||||
|
|
||||||
|
const migrations = readMigrationFiles({ migrationsFolder: migrationsFolder() });
|
||||||
|
for (const migration of migrations) {
|
||||||
|
if (applied.has(migration.hash)) continue;
|
||||||
|
|
||||||
|
// Run each statement-breakpoint chunk in its own exec() call so PGlite
|
||||||
|
// commits between statements — this is what lets `ALTER TYPE ADD VALUE`
|
||||||
|
// become visible before a subsequent statement references the new value.
|
||||||
|
for (const [stmtIdx, stmt] of migration.sql.entries()) {
|
||||||
|
const trimmed = stmt.trim();
|
||||||
|
if (!trimmed) continue;
|
||||||
|
try {
|
||||||
|
await client.exec(trimmed);
|
||||||
|
} catch (err) {
|
||||||
|
const cause = err instanceof Error ? err.message : String(err);
|
||||||
|
throw new Error(
|
||||||
|
`runPgliteMigrations: migration hash=${migration.hash} statement #${stmtIdx} failed: ${cause}\n` +
|
||||||
|
`Statement: ${trimmed.slice(0, 200)}${trimmed.length > 200 ? '…' : ''}`,
|
||||||
|
{ cause: err },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await handle.db.execute(
|
||||||
|
sql`INSERT INTO drizzle.__drizzle_migrations (hash, created_at) VALUES (${migration.hash}, ${migration.folderMillis})`,
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,8 +16,15 @@ import fs from 'node:fs/promises';
|
|||||||
import os from 'node:os';
|
import os from 'node:os';
|
||||||
import path from 'node:path';
|
import path from 'node:path';
|
||||||
|
|
||||||
import { users, teams, teamMembers, conversations, messages } from '@mosaicstack/db';
|
import {
|
||||||
import { createPgliteDbWithVector, runPgliteMigrations } from './test-utils/pglite-with-vector.js';
|
users,
|
||||||
|
teams,
|
||||||
|
teamMembers,
|
||||||
|
conversations,
|
||||||
|
messages,
|
||||||
|
createPgliteDb,
|
||||||
|
runPgliteMigrations,
|
||||||
|
} from '@mosaicstack/db';
|
||||||
|
|
||||||
import postgres from 'postgres';
|
import postgres from 'postgres';
|
||||||
import { afterAll, describe, expect, it } from 'vitest';
|
import { afterAll, describe, expect, it } from 'vitest';
|
||||||
@@ -102,11 +109,8 @@ describe.skipIf(!run)('migrate-tier — PGlite → federated PG', () => {
|
|||||||
/* ---- 1. Create a temp PGlite db ---------------------------------- */
|
/* ---- 1. Create a temp PGlite db ---------------------------------- */
|
||||||
|
|
||||||
pgliteDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fed-m1-08-'));
|
pgliteDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fed-m1-08-'));
|
||||||
const handle = createPgliteDbWithVector(pgliteDataDir);
|
const handle = createPgliteDb(pgliteDataDir);
|
||||||
|
await runPgliteMigrations(handle);
|
||||||
// Run Drizzle migrations against PGlite.
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
||||||
await runPgliteMigrations(handle.db as any);
|
|
||||||
|
|
||||||
/* ---- 2. Seed representative data --------------------------------- */
|
/* ---- 2. Seed representative data --------------------------------- */
|
||||||
|
|
||||||
|
|||||||
@@ -1,52 +0,0 @@
|
|||||||
/**
|
|
||||||
* Test-only helpers for creating a PGlite database with the pgvector extension
|
|
||||||
* and running Drizzle migrations against it.
|
|
||||||
*
|
|
||||||
* These are intentionally NOT exported from @mosaicstack/db to avoid pulling
|
|
||||||
* the WASM vector bundle into the public API surface.
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { createRequire } from 'node:module';
|
|
||||||
import { dirname, resolve } from 'node:path';
|
|
||||||
|
|
||||||
import { PGlite } from '@electric-sql/pglite';
|
|
||||||
import { vector } from '@electric-sql/pglite/vector';
|
|
||||||
import { drizzle } from 'drizzle-orm/pglite';
|
|
||||||
import { migrate as migratePglite } from 'drizzle-orm/pglite/migrator';
|
|
||||||
import type { PgliteDatabase } from 'drizzle-orm/pglite';
|
|
||||||
import * as schema from '@mosaicstack/db';
|
|
||||||
import type { DbHandle } from '@mosaicstack/db';
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Create a PGlite DB handle with the pgvector extension loaded.
|
|
||||||
* Required for running Drizzle migrations that include `CREATE EXTENSION vector`.
|
|
||||||
*/
|
|
||||||
export function createPgliteDbWithVector(dataDir: string): DbHandle {
|
|
||||||
const client = new PGlite(dataDir, { extensions: { vector } });
|
|
||||||
const db = drizzle(client, { schema });
|
|
||||||
return {
|
|
||||||
db: db as unknown as DbHandle['db'],
|
|
||||||
close: async () => {
|
|
||||||
await client.close();
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Run Drizzle migrations against an already-open PGlite database handle.
|
|
||||||
* Resolves the migrations folder from @mosaicstack/db's installed location.
|
|
||||||
*
|
|
||||||
* @param db A PgliteDatabase instance (from drizzle-orm/pglite).
|
|
||||||
*/
|
|
||||||
export async function runPgliteMigrations(
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
||||||
db: PgliteDatabase<any>,
|
|
||||||
): Promise<void> {
|
|
||||||
// Resolve @mosaicstack/db package root to locate its drizzle migrations folder.
|
|
||||||
const _require = createRequire(import.meta.url);
|
|
||||||
const dbPkgMain = _require.resolve('@mosaicstack/db');
|
|
||||||
// dbPkgMain → …/packages/db/dist/index.js → dirname = dist/
|
|
||||||
// go up one level from dist/ to find the sibling drizzle/ folder
|
|
||||||
const migrationsFolder = resolve(dirname(dbPkgMain), '../drizzle');
|
|
||||||
await migratePglite(db, { migrationsFolder });
|
|
||||||
}
|
|
||||||
125
scratchpads/fix-db-bootstrap-migrations.md
Normal file
125
scratchpads/fix-db-bootstrap-migrations.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
# fix(db): bootstrap migrations on local-tier gateway startup
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Fresh `mosaic gateway install` (npm-installed) leaves the gateway DB schema empty:
|
||||||
|
|
||||||
|
```
|
||||||
|
relation "users" does not exist
|
||||||
|
```
|
||||||
|
|
||||||
|
Sign-in 500s, `auth users create` says "Not signed in", `admin/bootstrap setup`
|
||||||
|
also fails — every entry point queries `users` before doing anything else.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This PR fixes the **local (PGlite) tier** end-to-end. The postgres-tier path
|
||||||
|
has additional pre-existing bugs (see "Known issues, out of scope" below) and
|
||||||
|
needs a separate change with real Postgres validation.
|
||||||
|
|
||||||
|
## Root causes addressed (5 stacked bugs on the local-tier path)
|
||||||
|
|
||||||
|
1. **`packages/db/package.json` `files: ["dist"]`** — the `drizzle/` SQL
|
||||||
|
migrations folder is excluded from the published tarball. Even if a
|
||||||
|
migrate runner existed, it would have nothing to apply.
|
||||||
|
|
||||||
|
2. **`packages/db/src/migrate.ts`** only supports `postgres-js`. Local-tier
|
||||||
|
gateways use embedded PGlite, which can't be reached over a postgres wire
|
||||||
|
protocol — so `runMigrations()` is unusable for the local tier.
|
||||||
|
|
||||||
|
3. **`apps/gateway/src/database/database.module.ts`** never invokes
|
||||||
|
migrations at startup. The module creates the DB handle and storage
|
||||||
|
adapter, but no consumer calls `.migrate()` on either. `mosaic storage
|
||||||
|
migrate` CLI even claims "pglite runs schema setup automatically on first
|
||||||
|
connection via `adapter.migrate()`" — but `adapter.migrate()` is only
|
||||||
|
called by tests, never at runtime.
|
||||||
|
|
||||||
|
4. **`createPgliteDb` does not load the pgvector extension.** Migration 0001
|
||||||
|
declares `CREATE EXTENSION IF NOT EXISTS vector;` for the
|
||||||
|
`insights.embedding` column. Bare PGlite has no pgvector — the migration
|
||||||
|
fails on extension control file lookup.
|
||||||
|
|
||||||
|
5. **Drizzle's PG migrator wraps every migration in one outer transaction.**
|
||||||
|
Migration 0009 does `ALTER TYPE grant_status ADD VALUE 'pending'` and then
|
||||||
|
`ALTER TABLE federation_grants ALTER COLUMN status SET DEFAULT 'pending'`.
|
||||||
|
Postgres' `check_safe_enum_use` rejects the second statement because the
|
||||||
|
new enum value isn't committed yet. Splitting the migration into two
|
||||||
|
files doesn't help — drizzle batches all migrations into one outer tx.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
- `packages/db/package.json` — ship `drizzle/` in `files`.
|
||||||
|
- `packages/db/src/client-pglite.ts` — load `@electric-sql/pglite/vector`.
|
||||||
|
- `packages/db/src/migrate.ts` — add `runPgliteMigrations(handle)`. Walks the
|
||||||
|
Drizzle journal and runs each statement-breakpoint chunk through PGlite's
|
||||||
|
`client.exec()` (Simple Query protocol → autocommit per statement). Writes
|
||||||
|
to the standard `drizzle.__drizzle_migrations` ledger so the result is
|
||||||
|
interoperable with `runMigrations()` on a postgres-backed deployment.
|
||||||
|
Per-statement try/catch surfaces which statement of which migration failed
|
||||||
|
and the ledger row is only written on full success.
|
||||||
|
- `packages/db/src/index.ts` — re-export.
|
||||||
|
- `apps/gateway/src/database/database.module.ts` — implement `OnModuleInit`:
|
||||||
|
- Local tier → `runPgliteMigrations(handle)`, then `storageAdapter.migrate()`
|
||||||
|
(the local storage adapter has its own kv tables in a separate PGlite dir).
|
||||||
|
- Postgres tier → `storageAdapter.migrate()` only, since
|
||||||
|
`PostgresAdapter.migrate()` already calls `runMigrations(url)` against
|
||||||
|
the same DATABASE_URL — we deliberately don't double-call.
|
||||||
|
|
||||||
|
NestJS awaits `onModuleInit` before `app.listen()`, so DB-dependent modules
|
||||||
|
see a populated schema before any HTTP traffic is accepted.
|
||||||
|
|
||||||
|
- `packages/storage/src/test-utils/pglite-with-vector.ts` — **deleted**.
|
||||||
|
The "intentionally not exported" rationale is moot now that migration 0001
|
||||||
|
forces pgvector load anyway. `migrate-tier.integration.test.ts` switched
|
||||||
|
to `createPgliteDb` + `runPgliteMigrations` from `@mosaicstack/db`.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
`packages/db/src/migrate.test.ts`:
|
||||||
|
|
||||||
|
- Verifies `runPgliteMigrations` creates the BetterAuth tables (the original
|
||||||
|
failure mode).
|
||||||
|
- Idempotence (transitively re-runs migration 0009).
|
||||||
|
- Partial-failure: pre-creates a conflicting `users` table, asserts the
|
||||||
|
thrown error includes statement context (`hash=… statement #N failed`)
|
||||||
|
and that no ledger row was written.
|
||||||
|
|
||||||
|
## QA evidence
|
||||||
|
|
||||||
|
End-to-end on a fresh PGlite install:
|
||||||
|
|
||||||
|
- `[DatabaseModule] Applying PGlite schema migrations...` then
|
||||||
|
`Initializing storage adapter (pglite)...` in startup log.
|
||||||
|
- `GET /api/bootstrap/status` → `{"needsSetup":true}` HTTP 200 (was 500
|
||||||
|
with `relation "users" does not exist`).
|
||||||
|
- `POST /api/bootstrap/setup` with empty body → HTTP 400 with Zod
|
||||||
|
validation error (was 500), confirming the request reached the
|
||||||
|
validator past the table-existence check.
|
||||||
|
|
||||||
|
## Known issues, out of scope (file separately)
|
||||||
|
|
||||||
|
- **Postgres-tier first install is still broken.** `runMigrations()` uses
|
||||||
|
Drizzle's `migratePostgres`, which has the same outer-transaction problem
|
||||||
|
as PGlite's migrator. A fresh standalone-tier install would also fail at
|
||||||
|
migration 0009. Inline TODO in `migrate.ts:31-35` flags this. Fixing it
|
||||||
|
needs either (a) a shared per-statement loop reused for both drivers, or
|
||||||
|
(b) splitting migration 0009.
|
||||||
|
- **`drizzle/meta/_journal.json` has 0009 ordered before 0008** (`when`
|
||||||
|
values `1745280000000` < `1776822435828`). `migratePostgres` skips by
|
||||||
|
`created_at < folderMillis`, so on a postgres deployment that already
|
||||||
|
applied 0008, 0009 would be skipped forever. Our hash-based skip in the
|
||||||
|
PGlite path sidesteps this.
|
||||||
|
- **No advisory lock around the migration loop.** Two gateway processes
|
||||||
|
pointed at the same DATABASE_URL would race. PGlite is single-process by
|
||||||
|
file lock so the local tier is fine; postgres-tier deployments should add
|
||||||
|
`pg_advisory_lock(<deterministic-id>)` around the loop in a follow-up.
|
||||||
|
- **`mosaic storage migrate` CLI message is misleading** — it claims
|
||||||
|
"automatic on first connection via adapter.migrate()" but the adapter
|
||||||
|
doesn't self-migrate. With this PR the gateway invokes it explicitly, but
|
||||||
|
the CLI message could still be tightened.
|
||||||
|
- **Crash mid-migration leaves a partial-state PGlite DB without a ledger
|
||||||
|
row.** Detected loudly on next boot (the replay errors on "already
|
||||||
|
exists"), but recovery is manual (drop the partially-applied objects or
|
||||||
|
insert the migration hash into `drizzle.__drizzle_migrations`). A robust
|
||||||
|
fix would add a "started_at" column to a sidecar table to detect
|
||||||
|
half-applied state and refuse to start with actionable guidance.
|
||||||
Reference in New Issue
Block a user