148 lines
5.1 KiB
Markdown
148 lines
5.1 KiB
Markdown
# Migrating to the Federated Tier
|
|
|
|
Step-by-step guide to migrate from `local` (PGlite) or `standalone` (PostgreSQL without pgvector) to `federated` (PostgreSQL 17 + pgvector + Valkey).
|
|
|
|
## When to migrate
|
|
|
|
Migrate to federated tier when:
|
|
|
|
- Scaling from single-user to multi-user deployments
|
|
- Adding vector embeddings or RAG features
|
|
- Running Mosaic across multiple hosts
|
|
- Requires distributed task queueing and caching
|
|
- Moving to production with high availability
|
|
|
|
## Prerequisites
|
|
|
|
- Federated stack running and healthy (see [Federated Tier Setup](../federation/SETUP.md))
|
|
- Source database accessible and empty target database at the federated URL
|
|
- Backup of source database (recommended before any migration)
|
|
|
|
## Dry-run first
|
|
|
|
Always run a dry-run to validate the migration:
|
|
|
|
```bash
|
|
mosaic storage migrate-tier --to federated \
|
|
--target-url postgresql://mosaic:mosaic@localhost:5433/mosaic \
|
|
--dry-run
|
|
```
|
|
|
|
Expected output (partial example):
|
|
|
|
```
|
|
[migrate-tier] Analyzing source tier: pglite
|
|
[migrate-tier] Analyzing target tier: federated
|
|
[migrate-tier] Precondition: target is empty ✓
|
|
users: 5 rows
|
|
teams: 2 rows
|
|
conversations: 12 rows
|
|
messages: 187 rows
|
|
... (all tables listed)
|
|
[migrate-tier] NOTE: Source tier has no pgvector support. insights.embedding will be NULL on all migrated rows.
|
|
[migrate-tier] DRY-RUN COMPLETE (no data written). 206 total rows would be migrated.
|
|
```
|
|
|
|
Review the output. If it shows an error (e.g., target not empty), address it before proceeding.
|
|
|
|
## Run the migration
|
|
|
|
When ready, run without `--dry-run`:
|
|
|
|
```bash
|
|
mosaic storage migrate-tier --to federated \
|
|
--target-url postgresql://mosaic:mosaic@localhost:5433/mosaic \
|
|
--yes
|
|
```
|
|
|
|
The `--yes` flag skips the confirmation prompt (required in non-TTY environments like CI).
|
|
|
|
The command will:
|
|
|
|
1. Acquire an advisory lock (blocks concurrent invocations)
|
|
2. Copy data from source to target in dependency order
|
|
3. Report rows migrated per table
|
|
4. Display any warnings (e.g., null vector embeddings)
|
|
|
|
## What gets migrated
|
|
|
|
All persistent, user-bound data is migrated in dependency order:
|
|
|
|
- **users, teams, team_members** — user and team ownership
|
|
- **accounts** — OAuth provider tokens (durable credentials)
|
|
- **projects, agents, missions, tasks** — all project and agent definitions
|
|
- **conversations, messages** — all chat history
|
|
- **preferences, insights, agent_logs** — preferences and observability
|
|
- **provider_credentials** — stored API keys and secrets
|
|
- **tickets, events, skills, routing_rules, appreciations** — auxiliary records
|
|
|
|
Full order is defined in code (`MIGRATION_ORDER` in `packages/storage/src/migrate-tier.ts`).
|
|
|
|
## What gets skipped and why
|
|
|
|
Three tables are intentionally not migrated:
|
|
|
|
| Table | Reason |
|
|
| ----------------- | ----------------------------------------------------------------------------------------------- |
|
|
| **sessions** | TTL'd auth sessions from the old environment; they will fail JWT verification on the new target |
|
|
| **verifications** | One-time tokens (email verify, password reset) that have either expired or been consumed |
|
|
| **admin_tokens** | Hashed tokens bound to the old environment's secret keys; must be re-issued |
|
|
|
|
**Note on accounts and provider_credentials:** These durable credentials ARE migrated because they are user-bound and required for resuming agent work on the target environment. After migration to a multi-tenant federated deployment, operators may want to audit or wipe these if users are untrusted or credentials should not be shared.
|
|
|
|
## Idempotency and concurrency
|
|
|
|
The migration is **idempotent**:
|
|
|
|
- Re-running is safe (uses `ON CONFLICT DO UPDATE` internally)
|
|
- Ideal for retries on transient failures
|
|
- Concurrent invocations are blocked by a Postgres advisory lock; the second caller will wait
|
|
|
|
If a previous run is stuck, check for advisory locks:
|
|
|
|
```sql
|
|
SELECT * FROM pg_locks WHERE locktype='advisory';
|
|
```
|
|
|
|
If you need to force-unlock (dangerous):
|
|
|
|
```sql
|
|
SELECT pg_advisory_unlock(<lock_id>);
|
|
```
|
|
|
|
## Verify the migration
|
|
|
|
After migration completes, spot-check the target:
|
|
|
|
```bash
|
|
# Count rows on a few critical tables
|
|
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
|
"SELECT 'users' as table, COUNT(*) FROM users UNION ALL
|
|
SELECT 'conversations' as table, COUNT(*) FROM conversations UNION ALL
|
|
SELECT 'messages' as table, COUNT(*) FROM messages;"
|
|
```
|
|
|
|
Verify a known user or project exists by ID:
|
|
|
|
```bash
|
|
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
|
"SELECT id, email FROM users WHERE email='<your-email>';"
|
|
```
|
|
|
|
Ensure vector embeddings are NULL (if source was PGlite) or populated (if source was postgres + pgvector):
|
|
|
|
```bash
|
|
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
|
"SELECT embedding IS NOT NULL as has_vector FROM insights LIMIT 5;"
|
|
```
|
|
|
|
## Rollback
|
|
|
|
There is no in-place rollback. If the migration fails:
|
|
|
|
1. Restore the target database from a pre-migration backup
|
|
2. Investigate the failure logs
|
|
3. Rerun the migration
|
|
|
|
Always test migrations in a staging environment first.
|