feat(storage): mosaic storage migrate-tier with dry-run + idempotency (FED-M1-05) #474

jason.woltje · 2026-04-20T00:31:25Z

jason.woltje commented

2026-04-20 00:31:25 +00:00

Summary

FED-M1-05 — Adds mosaic storage migrate-tier --to federated for one-way migration from local (PGlite) or standalone (Postgres without pgvector) → federated (Postgres + pgvector).

Architectural correction during review

The first implementation read source data via PgliteAdapter (flat (id, data jsonb) collection schema). Independent code review surfaced this as a P0 bug: the gateway writes domain data via the normalized Drizzle schema (createPgliteDb() → DB injection), not via the storage adapter. Reading from the flat collections would have silently produced empty federated databases.

Fix: introduced DrizzleMigrationSource that queries the same normalized schema the gateway writes through. Both local and standalone sources now route through Drizzle (createPgliteDb / createDb from @mosaicstack/db), guaranteeing the migration captures real domain data.

Behaviors

--dry-run — prints per-table row counts; no writes
--allow-non-empty — required to write into a populated target (default refuses)
Per-table transactions — partial failure does not leave half-migrated tables
Idempotent — INSERT ... ON CONFLICT (id) DO UPDATE keyed on primary key
Topological FK ordering verified against schema.references()
Skipped tables: sessions, verifications, admin_tokens (TTL'd / one-time / env-bound)
insights.embedding projected to NULL when source lacks pgvector (column is nullable per schema)

Tests

32 unit tests in packages/storage/src/migrate-tier.spec.ts:

Topological order correctness
Dry-run path takes no writes
Idempotency (re-run produces same target state)
Empty-target precondition gating --allow-non-empty
Skipped tables not in migration order
Embedding-null projection for non-pgvector sources

Test Files  1 passed (1)
Tests       32 passed (32)

Test plan

CI green (typecheck + lint + format + test gates)
Real PG ↔ PGlite migration verified by FED-M1-08 (next task)
Manual smoke test once doctor (FED-M1-06) and integration test (FED-M1-07) land

Refs #460

## Summary FED-M1-05 — Adds `mosaic storage migrate-tier --to federated` for one-way migration from `local` (PGlite) or `standalone` (Postgres without pgvector) → `federated` (Postgres + pgvector). ## Architectural correction during review The first implementation read source data via `PgliteAdapter` (flat `(id, data jsonb)` collection schema). Independent code review surfaced this as a P0 bug: the gateway writes domain data via the **normalized Drizzle schema** (`createPgliteDb()` → `DB` injection), not via the storage adapter. Reading from the flat collections would have silently produced empty federated databases. Fix: introduced `DrizzleMigrationSource` that queries the same normalized schema the gateway writes through. Both `local` and `standalone` sources now route through Drizzle (`createPgliteDb` / `createDb` from `@mosaicstack/db`), guaranteeing the migration captures real domain data. ## Behaviors - `--dry-run` — prints per-table row counts; no writes - `--allow-non-empty` — required to write into a populated target (default refuses) - Per-table transactions — partial failure does not leave half-migrated tables - Idempotent — `INSERT ... ON CONFLICT (id) DO UPDATE` keyed on primary key - Topological FK ordering verified against `schema.references()` - Skipped tables: `sessions`, `verifications`, `admin_tokens` (TTL'd / one-time / env-bound) - `insights.embedding` projected to NULL when source lacks pgvector (column is nullable per schema) ## Tests 32 unit tests in `packages/storage/src/migrate-tier.spec.ts`: - Topological order correctness - Dry-run path takes no writes - Idempotency (re-run produces same target state) - Empty-target precondition gating `--allow-non-empty` - Skipped tables not in migration order - Embedding-null projection for non-pgvector sources ``` Test Files 1 passed (1) Tests 32 passed (32) ``` ## Test plan - [ ] CI green (typecheck + lint + format + test gates) - [ ] Real PG ↔ PGlite migration verified by FED-M1-08 (next task) - [ ] Manual smoke test once doctor (FED-M1-06) and integration test (FED-M1-07) land Refs #460

jason.woltje added 1 commit 2026-04-20 00:31:26 +00:00

feat(storage): mosaic storage migrate-tier with dry-run + idempotency (FED-M1-05)

ci/woodpecker/push/ci Pipeline was successful

Details

ci/woodpecker/pr/ci Pipeline was successful

Details

8d30a47286

Adds `mosaic storage migrate-tier --to federated` for one-way migration
from `local` (PGlite) or `standalone` (PG without pgvector) to
`federated` (PG + pgvector). Source is read via DrizzleMigrationSource
which queries the normalized schema (the same path the gateway writes
through), so the migration captures all real domain data — not just
the flat key/value collections.

Key behaviors:
- `--dry-run` prints per-table row counts without writes
- `--allow-non-empty` required to write into a populated target (default
  refuses to mix data sets)
- Per-table transactions; partial failure does not leave half-migrated
  tables
- Idempotent on re-run via INSERT ... ON CONFLICT DO UPDATE keyed on id
- Topological table order respects FK dependencies (verified against
  schema.references())
- Skips sessions, verifications, admin_tokens (TTL'd / one-time / env-bound)
- For sources without pgvector, insights.embedding is projected to NULL
  on read, then upserted as NULL (column is nullable)

32 unit tests cover ordering, dry-run, idempotency, empty-target
preconditions, table skipping, and embedding-null projection. Integration
test against real PG/PGlite is FED-M1-08.

Refs #460

jason.woltje merged commit ccad30dd27 into main

2026-04-20 00:35:09 +00:00

jason.woltje referenced this issue from a commit

2026-04-20 00:35:10 +00:00

feat(storage): mosaic storage migrate-tier with dry-run + idempotency (FED-M1-05) (#474)

jason.woltje referenced this pull request

2026-04-20 01:39:56 +00:00

test(storage): integration test for migrate-tier (FED-M1-08) + camelCase column fix #477

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#474