177 lines
8.4 KiB
Markdown
177 lines
8.4 KiB
Markdown
# Migrating to the Federated Tier
|
|
|
|
> **KBN-101-07 ownership:** this is an **active** operator guide, exclusively owned by
|
|
> KBN-101-07. It is not a historical document. The KBN-101-06 finite operator-document
|
|
> inventory records this route and its secure-interface fields.
|
|
|
|
This guide is active documentation but a **non-operative KBN-101 contract** until
|
|
KBN-101-02, -03, and -06 land and KBN-101-08 activates the reviewed artifacts. The command
|
|
blocks below specify the produced interface; they are not available on the current branch and
|
|
MUST NOT be attempted until the activation certificate names the exact release. Until then, do
|
|
not use a direct PostgreSQL target URL, raw SQL, or any legacy storage migration command as a
|
|
production procedure.
|
|
|
|
## Non-negotiable boundary
|
|
|
|
The PostgreSQL destination schema is prepared separately before data migration:
|
|
|
|
1. The dedicated DDL runner executes `mosaic-db-migrator --run` using its migration-only
|
|
identity.
|
|
2. The same runner succeeds at `mosaic-db-migrator --verify` against the exact destination.
|
|
3. Only then may `mosaic storage migrate-tier` connect as dedicated non-DDL
|
|
`mosaic_data_importer` to copy data.
|
|
|
|
The data migration never executes DDL or changes PostgreSQL roles, memberships, schemas,
|
|
extensions, extension members, system catalogs, or the Drizzle ledger. Its only target writes are
|
|
an allowlisted data-copy DML registry: inserts and idempotent conflict updates on declared mutable
|
|
application tables. It has no DML grant for immutable KBN relations, extension/catalog objects, or
|
|
any schema authority. It is not a schema bootstrap, extension installer, or repair procedure.
|
|
PostgreSQL is always pre-migrated and runner-verified, including when the source is PGlite.
|
|
|
|
## Prerequisites
|
|
|
|
- Federated stack is running and healthy (see [Federated Tier Setup](../federation/SETUP.md)).
|
|
- An approved source backup exists and the target is identified in the KBN-101 change record.
|
|
- The target passed the runner `--run` and `--verify` sequence above.
|
|
- The operator has an approved, dedicated importer credential file at
|
|
`/run/secrets/mosaic_migrate_target_url`; the **path reference** is non-secret, but its
|
|
contents are never printed, copied, committed, or placed on argv.
|
|
- KBN-101-02 validates that file before any target connection: it must be a regular,
|
|
non-symlink file, owned by the expected importer UID:GID, mode `0600`, readable only by
|
|
that importer process, and its value must be redacted from all output.
|
|
|
|
### Source modes
|
|
|
|
- **Local PGlite:** the source is the explicit `PGLITE_DATA_DIR`; it does not read or
|
|
interpret a PostgreSQL URL.
|
|
- **PostgreSQL source:** this active route does not accept `DATABASE_URL` as a source or
|
|
target fallback. A PostgreSQL-source procedure needs its own independently approved,
|
|
non-secret credential-reference contract before it is enabled.
|
|
|
|
## Dry-run first
|
|
|
|
Run the migration only after the target schema is prepared and verified:
|
|
|
|
```bash
|
|
mosaic-db-migrator --run
|
|
mosaic-db-migrator --verify
|
|
mosaic storage migrate-tier --to federated \
|
|
--target-url-file /run/secrets/mosaic_migrate_target_url \
|
|
--dry-run
|
|
```
|
|
|
|
The frozen KBN-101-02 option is exactly
|
|
`--target-url-file /run/secrets/mosaic_migrate_target_url`. It is a file reference, not a
|
|
URL option. The runner and importer use distinct connections: the runner has migration
|
|
DDL authority; the importer has only approved data-copy DML authority.
|
|
|
|
Expected dry-run output is sanitized and contains no connection string, username, host,
|
|
database name, SQL, or credential value:
|
|
|
|
```text
|
|
[migrate-tier] Source tier: pglite
|
|
[migrate-tier] Target tier: federated
|
|
[migrate-tier] Target schema: verified by mosaic-db-migrator
|
|
[migrate-tier] Target connection: dedicated importer
|
|
[migrate-tier] Precondition: target is ready
|
|
users: 5 rows
|
|
teams: 2 rows
|
|
conversations: 12 rows
|
|
messages: 187 rows
|
|
... (all tables listed)
|
|
[migrate-tier] NOTE: Source tier has no pgvector support. insights.embedding will be NULL on all migrated rows.
|
|
[migrate-tier] DRY-RUN COMPLETE (no data written). 206 total rows would be migrated.
|
|
```
|
|
|
|
Review the output. If it reports a failed runner verification, unexpected role, unsafe
|
|
secret-file owner/mode, non-empty target, or any connection/DDL precondition error, stop
|
|
and remediate through the KBN-101 control plane.
|
|
|
|
## Run the data migration
|
|
|
|
After a reviewed dry-run, repeat the same verified route with `--yes`:
|
|
|
|
```bash
|
|
mosaic-db-migrator --verify
|
|
mosaic storage migrate-tier --to federated \
|
|
--target-url-file /run/secrets/mosaic_migrate_target_url \
|
|
--yes
|
|
```
|
|
|
|
`--yes` only confirms data copy in non-TTY automation. It does not bypass schema
|
|
verification, target-file validation, role checks, redaction, or the no-DDL boundary.
|
|
|
|
The command will:
|
|
|
|
1. Verify the runner-prepared target schema before opening the importer target connection.
|
|
2. Validate the credential-file reference and use the dedicated importer identity.
|
|
3. Copy data in dependency order without DDL.
|
|
4. Report sanitized row counts and warnings.
|
|
|
|
## Rejected interfaces and failure-before-connect rules
|
|
|
|
KBN-101-02 and KBN-101-06 must prove these failures occur before a target connection or
|
|
DDL attempt:
|
|
|
|
- Raw `--target-url` or any URL/credential value on argv.
|
|
- `DATABASE_URL` as a target or source fallback.
|
|
- A runtime owner, migrator, schema owner, or extension owner used as the data importer.
|
|
- Missing, non-regular, symlinked, wrong-owner, or wrong-mode target credential file.
|
|
- Any mode other than the runner-prepared/verified PostgreSQL destination route.
|
|
- Any direct DDL attempt by the migration command or its target connection.
|
|
|
|
KBN-101-02 SHALL add `packages/storage/src/migrate-tier.spec.ts::secureTargetRoute` and
|
|
its integration counterpart. They prove no target connection is opened on each interface
|
|
rejection. The finite operator-document scanner rejects an unowned route or a guide that
|
|
reintroduces a credential-bearing command.
|
|
|
|
## What gets migrated
|
|
|
|
All persistent, user-bound data is migrated in dependency order:
|
|
|
|
- **users, teams, team_members** — user and team ownership
|
|
- **accounts** — OAuth provider tokens (durable credentials)
|
|
- **projects, agents, missions, tasks** — all project and agent definitions
|
|
- **conversations, messages** — all chat history
|
|
- **preferences, insights, agent_logs** — preferences and observability
|
|
- **provider_credentials** — stored API keys and secrets
|
|
- **tickets, events, skills, routing_rules, appreciations** — auxiliary records
|
|
|
|
Full order is defined in code (`MIGRATION_ORDER` in `packages/storage/src/migrate-tier.ts`).
|
|
|
|
## What gets skipped and why
|
|
|
|
Three tables are intentionally not migrated:
|
|
|
|
| Table | Reason |
|
|
| ----------------- | ----------------------------------------------------------------------------------------------- |
|
|
| **sessions** | TTL'd auth sessions from the old environment; they will fail JWT verification on the new target |
|
|
| **verifications** | One-time tokens (email verify, password reset) that have either expired or been consumed |
|
|
| **admin_tokens** | Hashed tokens bound to the old environment's secret keys; must be re-issued |
|
|
|
|
**Note on accounts and provider_credentials:** These durable credentials are migrated because
|
|
they are user-bound and required for resuming agent work on the target. After migration to a
|
|
multi-tenant federated deployment, operators may want to audit or wipe these if users are
|
|
untrusted or credentials should not be shared.
|
|
|
|
## Idempotency, concurrency, and verification
|
|
|
|
The data copy is idempotent and uses an importer-scoped advisory lock. It may be retried only
|
|
through the same runner-verified, file-reference route. Do not inspect, unlock, or repair the
|
|
target with ad hoc SQL; a failed or ambiguous run is a control-plane incident with backup and
|
|
rollback evidence.
|
|
|
|
After a successful migration, use the KBN-101-approved sanitized verification command or
|
|
read-only operator report. Do not paste a target connection string into a shell command,
|
|
terminal history, ticket, or documentation.
|
|
|
|
## Rollback
|
|
|
|
There is no in-place rollback. If data migration fails:
|
|
|
|
1. Keep the previous target state and restore the approved pre-migration backup where required.
|
|
2. Preserve sanitized failure and audit evidence; do not expose the credential file or contents.
|
|
3. Investigate through the KBN-101 control plane and rerun only after independent review.
|
|
|
|
Always rehearse this route in a staging environment first.
|