docs(kbn): freeze KBN-101 database role split contract (#774)
This commit was merged in pull request #774.
This commit is contained in:
@@ -1,147 +1,98 @@
|
||||
# Migrating to the Federated Tier
|
||||
|
||||
Step-by-step guide to migrate from `local` (PGlite) or `standalone` (PostgreSQL without pgvector) to `federated` (PostgreSQL 17 + pgvector + Valkey).
|
||||
> **KBN-101-07 ownership:** This active documentation is a **non-operative KBN-101
|
||||
> contract** with no current command authority until KBN-101-00, KBN-101-02, KBN-101-03, KBN-101-05, and KBN-101-06 land and
|
||||
> KBN-101-08 activates an exact reviewed release. The commands below describe the produced interface only. Do not run them on the
|
||||
> current branch or replace them with direct PostgreSQL, raw SQL, legacy storage migration, or
|
||||
> credential-on-argv procedures.
|
||||
|
||||
## When to migrate
|
||||
## Held future procedure
|
||||
|
||||
Migrate to federated tier when:
|
||||
This section is non-operative and grants no current command authority until KBN-101-00, KBN-101-03, and KBN-101-05 land.
|
||||
|
||||
- Scaling from single-user to multi-user deployments
|
||||
- Adding vector embeddings or RAG features
|
||||
- Running Mosaic across multiple hosts
|
||||
- Requires distributed task queueing and caching
|
||||
- Moving to production with high availability
|
||||
The deployment control plane executes the complete held future procedure, in order: external bootstrap → TLS/roles → `mosaic-db-migrator --run` → `mosaic-db-migrator --verify` → Gateway/Compose readiness. The
|
||||
runner is the only attestation producer after its verified TLS, identity, manifest, and schema
|
||||
checks. A data importer is never a schema bootstrap, extension installer, repair command, or DDL
|
||||
consumer.
|
||||
|
||||
## Prerequisites
|
||||
## Target material contract
|
||||
|
||||
- Federated stack running and healthy (see [Federated Tier Setup](../federation/SETUP.md))
|
||||
- Source database accessible and empty target database at the federated URL
|
||||
- Backup of source database (recommended before any migration)
|
||||
KBN-101-05 obtains the target URL from Vault KV-v2
|
||||
`secret-{env}/mosaic-stack/database/importer`, key `url`, and reads its authenticated version from
|
||||
the same successful response `data.metadata.version`. A hash or DSN byte sequence is not a
|
||||
provider version. The renderer treats URL bytes and provider version as one generation, writes a
|
||||
temporary generation directory with fsync plus atomic rename, and creates separate immutable
|
||||
consumer mounts. Swarm uses distinct versioned secret/config references. A deployment cannot mix
|
||||
generations.
|
||||
|
||||
## Dry-run first
|
||||
| Consumer | Permitted material |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Migrator-attestation producer (`10003:10003`) | Its own migration URL/CA; read-only `/run/secrets/mosaic-migrate-target-url` and `/run/secrets/mosaic-migrate-target-version`, each `0400`, solely to bind; producer-only attestation output at `/run/mosaic-attestations-producer/migrate-target.v1.json`; root-wrapper-only signing key. It never connects with, uses, exports, logs, or forwards the importer URL/version. |
|
||||
| Privileged deployment handoff controller | After runner success and before importer creation, it receives only root-owned non-secret expected provider-version/URL-SHA-256/generation descriptor and pinned public verifier key—not URL bytes or private key. It safe-opens/verifies descriptor and producer artifact, copies exact bytes to a new importer-only mount with fsync/atomic rename, sets `10002:10002` `0400`, seals it read-only, and refuses importer start on any partial/wrong-generation/wrong-owner/mode result. |
|
||||
| Importer (`10002:10002`) | Its own immutable `0400` copies at the same URL/version paths; CA at exact `DATABASE_TLS_CA_CERT_PATH=/run/secrets/mosaic-db-ca.crt`; pinned Ed25519 public key; read-only `/run/mosaic-attestations/migrate-target.v1.json` supplied only by the sealed handoff. |
|
||||
| Gateway/runtime/unrelated container | No importer URL/version, importer artifact, attestation private key, or unrelated CA mount. |
|
||||
|
||||
Always run a dry-run to validate the migration:
|
||||
The migrator and importer safe-open URL, provider-version, attestation, and public-key files only
|
||||
with `O_RDONLY|O_CLOEXEC|O_NOFOLLOW`; they validate from the opened fd that the file is regular,
|
||||
has its expected owner/mode and link count one. The migrator digests only that URL fd for binding,
|
||||
then zeroizes/closes it. The importer reads URL bytes once into protected memory, validates the
|
||||
signed binding and exact CA before connecting from those same bytes, then zeroizes/closes every
|
||||
fd. It neither logs nor exposes a URL/version/attestation/key oracle.
|
||||
|
||||
## Produced command interface
|
||||
|
||||
After activation and only after approved target preparation, the future interface is:
|
||||
|
||||
```bash
|
||||
# Deployment control plane has already completed the held runner procedure above.
|
||||
mosaic storage migrate-tier --to federated \
|
||||
--target-url postgresql://mosaic:mosaic@localhost:5433/mosaic \
|
||||
--target-url-file /run/secrets/mosaic-migrate-target-url \
|
||||
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
Expected output (partial example):
|
||||
The provider-version file is fixed deployment material, not argv. This connecting dry-run consumes its nonce; before an actual copy, the deployment control plane must provide fresh runner verification and a new sealed handoff. The runner uses its migration
|
||||
identity; the importer connects only as non-DDL `mosaic_data_importer` and only after all
|
||||
pre-connect validation. After verified TLS and before DML it compares PostgreSQL system ID,
|
||||
database OID, `current_user`, CA/SPKI, and manifest/schema fingerprints to the artifact.
|
||||
|
||||
```
|
||||
[migrate-tier] Analyzing source tier: pglite
|
||||
[migrate-tier] Analyzing target tier: federated
|
||||
[migrate-tier] Precondition: target is empty ✓
|
||||
users: 5 rows
|
||||
teams: 2 rows
|
||||
conversations: 12 rows
|
||||
messages: 187 rows
|
||||
... (all tables listed)
|
||||
[migrate-tier] NOTE: Source tier has no pgvector support. insights.embedding will be NULL on all migrated rows.
|
||||
[migrate-tier] DRY-RUN COMPLETE (no data written). 206 total rows would be migrated.
|
||||
```
|
||||
## Required refusals and evidence
|
||||
|
||||
Review the output. If it shows an error (e.g., target not empty), address it before proceeding.
|
||||
KBN-101-02/-03/-05/-06 must prove, with stable sanitized errors, that no target connection occurs
|
||||
for missing/unsafe URL/version/attestation/public-key files; symlink, hardlink, owner, mode, or
|
||||
TOCTOU violations; mixed URL/version generations; missing/wrong CA mount; stale/replayed/tampered
|
||||
or revoked-key artifacts; provider rotation/revocation; wrong TLS/server/database/role/manifest
|
||||
binding; raw `--target-url`; `DATABASE_URL` fallback; runtime/owner identity; consumer leakage;
|
||||
or any DDL attempt. Post-connect identity mismatch closes with zero DML/DDL. Tests also prove no
|
||||
forwarding, child environment, logging, or error oracle leaks URL/version/key/artifact contents.
|
||||
|
||||
## Run the migration
|
||||
The attestation is credential-free JCS with detached Ed25519 signature and binds issued/expiry,
|
||||
nonce, authenticated provider version, exact URL-fd SHA-256, TLS host/port/database, CA/SPKI,
|
||||
PostgreSQL system ID/database OID, importer role, manifest/schema, and producer identity. Provider
|
||||
version rotation invalidates an old artifact and requires a fresh rendered generation plus runner
|
||||
verification.
|
||||
|
||||
When ready, run without `--dry-run`:
|
||||
## Actual copy after dry-run
|
||||
|
||||
After reviewed dry-run, obtain the required fresh verification/attestation generation, then use:
|
||||
|
||||
```bash
|
||||
# Deployment control plane has supplied fresh runner verification and attestation.
|
||||
mosaic storage migrate-tier --to federated \
|
||||
--target-url postgresql://mosaic:mosaic@localhost:5433/mosaic \
|
||||
--target-url-file /run/secrets/mosaic-migrate-target-url \
|
||||
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
|
||||
--yes
|
||||
```
|
||||
|
||||
The `--yes` flag skips the confirmation prompt (required in non-TTY environments like CI).
|
||||
The dry-run artifact is terminally replayed and must be rejected; `--yes` bypasses no file,
|
||||
generation, signature, TLS, identity, or DDL control.
|
||||
|
||||
The command will:
|
||||
## Data boundary and recovery
|
||||
|
||||
1. Acquire an advisory lock (blocks concurrent invocations)
|
||||
2. Copy data from source to target in dependency order
|
||||
3. Report rows migrated per table
|
||||
4. Display any warnings (e.g., null vector embeddings)
|
||||
The importer has only an allowlisted mutable-table DML registry. It has no grant for immutable KBN
|
||||
relations, schemas, roles, memberships, extensions, catalogs, or the Drizzle ledger. Source PGlite
|
||||
uses its explicit local directory and does not make a PostgreSQL URL fallback valid.
|
||||
|
||||
## What gets migrated
|
||||
|
||||
All persistent, user-bound data is migrated in dependency order:
|
||||
|
||||
- **users, teams, team_members** — user and team ownership
|
||||
- **accounts** — OAuth provider tokens (durable credentials)
|
||||
- **projects, agents, missions, tasks** — all project and agent definitions
|
||||
- **conversations, messages** — all chat history
|
||||
- **preferences, insights, agent_logs** — preferences and observability
|
||||
- **provider_credentials** — stored API keys and secrets
|
||||
- **tickets, events, skills, routing_rules, appreciations** — auxiliary records
|
||||
|
||||
Full order is defined in code (`MIGRATION_ORDER` in `packages/storage/src/migrate-tier.ts`).
|
||||
|
||||
## What gets skipped and why
|
||||
|
||||
Three tables are intentionally not migrated:
|
||||
|
||||
| Table | Reason |
|
||||
| ----------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| **sessions** | TTL'd auth sessions from the old environment; they will fail JWT verification on the new target |
|
||||
| **verifications** | One-time tokens (email verify, password reset) that have either expired or been consumed |
|
||||
| **admin_tokens** | Hashed tokens bound to the old environment's secret keys; must be re-issued |
|
||||
|
||||
**Note on accounts and provider_credentials:** These durable credentials ARE migrated because they are user-bound and required for resuming agent work on the target environment. After migration to a multi-tenant federated deployment, operators may want to audit or wipe these if users are untrusted or credentials should not be shared.
|
||||
|
||||
## Idempotency and concurrency
|
||||
|
||||
The migration is **idempotent**:
|
||||
|
||||
- Re-running is safe (uses `ON CONFLICT DO UPDATE` internally)
|
||||
- Ideal for retries on transient failures
|
||||
- Concurrent invocations are blocked by a Postgres advisory lock; the second caller will wait
|
||||
|
||||
If a previous run is stuck, check for advisory locks:
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_locks WHERE locktype='advisory';
|
||||
```
|
||||
|
||||
If you need to force-unlock (dangerous):
|
||||
|
||||
```sql
|
||||
SELECT pg_advisory_unlock(<lock_id>);
|
||||
```
|
||||
|
||||
## Verify the migration
|
||||
|
||||
After migration completes, spot-check the target:
|
||||
|
||||
```bash
|
||||
# Count rows on a few critical tables
|
||||
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
||||
"SELECT 'users' as table, COUNT(*) FROM users UNION ALL
|
||||
SELECT 'conversations' as table, COUNT(*) FROM conversations UNION ALL
|
||||
SELECT 'messages' as table, COUNT(*) FROM messages;"
|
||||
```
|
||||
|
||||
Verify a known user or project exists by ID:
|
||||
|
||||
```bash
|
||||
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
||||
"SELECT id, email FROM users WHERE email='<your-email>';"
|
||||
```
|
||||
|
||||
Ensure vector embeddings are NULL (if source was PGlite) or populated (if source was postgres + pgvector):
|
||||
|
||||
```bash
|
||||
psql postgresql://mosaic:mosaic@localhost:5433/mosaic -c \
|
||||
"SELECT embedding IS NOT NULL as has_vector FROM insights LIMIT 5;"
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
There is no in-place rollback. If the migration fails:
|
||||
|
||||
1. Restore the target database from a pre-migration backup
|
||||
2. Investigate the failure logs
|
||||
3. Rerun the migration
|
||||
|
||||
Always test migrations in a staging environment first.
|
||||
A failed or ambiguous migration is a control-plane incident: preserve sanitized evidence, retain
|
||||
the approved backup/rollback state, and retry only after independent review. Never inspect,
|
||||
unlock, repair, or initialize the target with ad hoc SQL or copied credentials.
|
||||
|
||||
Reference in New Issue
Block a user