stack/docs/guides/migrate-tier.md

Migrating to the Federated Tier

KBN-101-07 ownership: this is an active operator guide, exclusively owned by KBN-101-07. It is not a historical document. The KBN-101-06 finite operator-document inventory records this route and its secure-interface fields.

This guide is active documentation but a non-operative KBN-101 contract until KBN-101-02, -03, and -06 land and KBN-101-08 activates the reviewed artifacts. The command blocks below specify the produced interface; they are not available on the current branch and MUST NOT be attempted until the activation certificate names the exact release. Until then, do not use a direct PostgreSQL target URL, raw SQL, or any legacy storage migration command as a production procedure.

Non-negotiable boundary

The PostgreSQL destination schema is prepared separately before data migration:

1. The dedicated DDL runner executes mosaic-db-migrator --run using its migration-only identity.
2. The same runner succeeds at mosaic-db-migrator --verify against the exact destination and, only then, emits a signed target-bound attestation.
3. Only then may mosaic storage migrate-tier verify that attestation and connect as dedicated non-DDL mosaic_data_importer to copy data.

The data migration never executes DDL or changes PostgreSQL roles, memberships, schemas, extensions, extension members, system catalogs, or the Drizzle ledger. Its only target writes are an allowlisted data-copy DML registry: inserts and idempotent conflict updates on declared mutable application tables. It has no DML grant for immutable KBN relations, extension/catalog objects, or any schema authority. It is not a schema bootstrap, extension installer, or repair procedure. PostgreSQL is always pre-migrated and runner-verified, including when the source is PGlite.

Prerequisites

• Federated stack is running and healthy (see Federated Tier Setup).
• An approved source backup exists and the target is identified in the KBN-101 change record.
• The target passed the runner --run and --verify sequence above; --verify is the only trusted attestation producer after verified TLS, identity, manifest, and schema checks.
• The operator has an approved, dedicated high-entropy importer credential file at /run/secrets/mosaic_migrate_target_url and its paired non-secret attestation artifact at /run/mosaic-attestations/migrate-target.v1.json. Neither file contents are printed, copied, committed, or placed on argv.
• KBN-101-02 validates both references before any target connection: each is a regular, non-symlink file with the exact required owner/mode. The importer gets only pinned public verification key(s) and the attestation; the runner-only Ed25519 private key never reaches importer/runtime. The URL file is opened, digested, parsed, and connected from the same in-memory bytes with no reread.

Source modes

• Local PGlite: the source is the explicit PGLITE_DATA_DIR; it does not read or interpret a PostgreSQL URL.
• PostgreSQL source: this active route does not accept DATABASE_URL as a source or target fallback. A PostgreSQL-source procedure needs its own independently approved, non-secret credential-reference contract before it is enabled.

Dry-run first

Run the migration only after the target schema is prepared and verified:

mosaic-db-migrator --run
mosaic-db-migrator --verify
mosaic storage migrate-tier --to federated \
  --target-url-file /run/secrets/mosaic_migrate_target_url \
  --target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
  --dry-run

The frozen KBN-101-02 options are exactly --target-url-file /run/secrets/mosaic_migrate_target_url and --target-attestation-file /run/mosaic-attestations/migrate-target.v1.json. They are non-secret file references, not URL options. The runner and importer use distinct connections: the runner has migration DDL authority; the importer has only approved data-copy DML authority.

Expected dry-run output is sanitized and contains no connection string, username, host, database name, SQL, or credential value:

[migrate-tier] Source tier: pglite
[migrate-tier] Target tier: federated
[migrate-tier] Target schema: verified by mosaic-db-migrator
[migrate-tier] Target connection: dedicated importer
[migrate-tier] Precondition: target is ready
  users: 5 rows
  teams: 2 rows
  conversations: 12 rows
  messages: 187 rows
  ... (all tables listed)
[migrate-tier] NOTE: Source tier has no pgvector support. insights.embedding will be NULL on all migrated rows.
[migrate-tier] DRY-RUN COMPLETE (no data written). 206 total rows would be migrated.

Review the output. If it reports a failed runner verification, unexpected role, unsafe secret-file owner/mode, non-empty target, or any connection/DDL precondition error, stop and remediate through the KBN-101 control plane.

Run the data migration

After a reviewed dry-run, repeat the same verified route with --yes:

mosaic-db-migrator --verify
mosaic storage migrate-tier --to federated \
  --target-url-file /run/secrets/mosaic_migrate_target_url \
  --target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
  --yes

--yes only confirms data copy in non-TTY automation. It does not bypass schema verification, target-file validation, role checks, redaction, or the no-DDL boundary.

The command will:

1. Before target connection, validate both files, JCS/Ed25519 signature and pinned key ID, expiry, nonce replay, secret version/digest, canonical TLS host/port/database, CA/SPKI, expected importer role, and manifest/schema bindings.
2. Connect only from the once-opened in-memory URL bytes, then before transaction/DML compare PostgreSQL system identifier, database OID, current_user, CA/SPKI, and manifest/schema fingerprints to the attestation; mismatch closes with zero DML/DDL.
3. Copy data in dependency order without DDL.
4. Report sanitized row counts and warnings.

Rejected interfaces and failure-before-connect rules

KBN-101-02 and KBN-101-06 must prove these failures occur before a target connection or DDL attempt:

• Raw --target-url or any URL/credential value on argv.
• DATABASE_URL as a target or source fallback.
• A runtime owner, migrator, schema owner, or extension owner used as the data importer.
• Missing, non-regular, symlinked, wrong-owner, wrong-mode, substituted, or changed-after-check credential file.
• Missing, stale, replayed, tampered, wrong-key, revoked-key, expired, malformed, or wrong-binding attestation file.
• Wrong host, database, CA/SPKI, PostgreSQL system identifier, database OID, importer role, manifest, schema, or credential secret-version/digest.
• Any mode other than the runner-prepared/verified PostgreSQL destination route.
• Any direct DDL attempt by the migration command or its target connection.

The signed attestation has canonical UTF-8/JCS bytes and detached Ed25519 signature. Its credential-free v1 payload binds format/version/key ID; issued/expiry/nonce; exact credential secret version and SHA-256; canonical TLS host/port/database; CA/SPKI; PostgreSQL system identifier/database OID; importer role; manifest/schema; and producer invocation/build/image/ correlation. Artifacts are atomically written/renamed 0444; the replay cache survives until expiry; public-key overlap/revocation and credential rotation force a fresh runner verification.

KBN-101-02 SHALL add packages/storage/src/migrate-tier.spec.ts::secureTargetRouteAndAttestation and its integration counterpart. They prove zero target connection for pre-connect failures and zero DML/DDL for post-connect identity mismatch. The finite operator-document scanner rejects an unowned route or a guide that reintroduces a credential-bearing command.

What gets migrated

All persistent, user-bound data is migrated in dependency order:

• users, teams, team_members — user and team ownership
• accounts — OAuth provider tokens (durable credentials)
• projects, agents, missions, tasks — all project and agent definitions
• conversations, messages — all chat history
• preferences, insights, agent_logs — preferences and observability
• provider_credentials — stored API keys and secrets
• tickets, events, skills, routing_rules, appreciations — auxiliary records

Full order is defined in code (MIGRATION_ORDER in packages/storage/src/migrate-tier.ts).

What gets skipped and why

Three tables are intentionally not migrated:

Table	Reason
sessions	TTL'd auth sessions from the old environment; they will fail JWT verification on the new target
verifications	One-time tokens (email verify, password reset) that have either expired or been consumed
admin_tokens	Hashed tokens bound to the old environment's secret keys; must be re-issued

Note on accounts and provider_credentials: These durable credentials are migrated because they are user-bound and required for resuming agent work on the target. After migration to a multi-tenant federated deployment, operators may want to audit or wipe these if users are untrusted or credentials should not be shared.

Idempotency, concurrency, and verification

The data copy is idempotent and uses an importer-scoped advisory lock. It may be retried only through the same runner-verified, file-reference route. Do not inspect, unlock, or repair the target with ad hoc SQL; a failed or ambiguous run is a control-plane incident with backup and rollback evidence.

After a successful migration, use the KBN-101-approved sanitized verification command or read-only operator report. Do not paste a target connection string into a shell command, terminal history, ticket, or documentation.

Rollback

There is no in-place rollback. If data migration fails:

1. Keep the previous target state and restore the approved pre-migration backup where required.
2. Preserve sanitized failure and audit evidence; do not expose the credential file or contents.
3. Investigate through the KBN-101 control plane and rerun only after independent review.

Always rehearse this route in a staging environment first.