docs(#771): harden importer deployment contract
This commit is contained in:
Hermes Agent
@@ -1,200 +1,98 @@
|
||||
# Migrating to the Federated Tier
|
||||
|
||||
> **KBN-101-07 ownership:** this is an **active** operator guide, exclusively owned by
|
||||
> KBN-101-07. It is not a historical document. The KBN-101-06 finite operator-document
|
||||
> inventory records this route and its secure-interface fields.
|
||||
> **KBN-101-07 ownership:** This active documentation is a **non-operative KBN-101
|
||||
> contract** until KBN-101-02/-03/-05/-06 land and KBN-101-08 activates an exact reviewed
|
||||
> release. The commands below describe the produced interface only. Do not run them on the
|
||||
> current branch or replace them with direct PostgreSQL, raw SQL, legacy storage migration, or
|
||||
> credential-on-argv procedures.
|
||||
|
||||
This guide is active documentation but a **non-operative KBN-101 contract** until
|
||||
KBN-101-02, -03, and -06 land and KBN-101-08 activates the reviewed artifacts. The command
|
||||
blocks below specify the produced interface; they are not available on the current branch and
|
||||
MUST NOT be attempted until the activation certificate names the exact release. Until then, do
|
||||
not use a direct PostgreSQL target URL, raw SQL, or any legacy storage migration command as a
|
||||
production procedure.
|
||||
## Required activation sequence
|
||||
|
||||
## Non-negotiable boundary
|
||||
The deployment control plane executes, in order: external privileged bootstrap; verified TLS and
|
||||
roles; `mosaic-db-migrator --run`; `mosaic-db-migrator --verify`; then Gateway readiness. The
|
||||
runner is the only attestation producer after its verified TLS, identity, manifest, and schema
|
||||
checks. A data importer is never a schema bootstrap, extension installer, repair command, or DDL
|
||||
consumer.
|
||||
|
||||
The PostgreSQL destination schema is prepared separately before data migration:
|
||||
## Target material contract
|
||||
|
||||
1. The dedicated DDL runner executes `mosaic-db-migrator --run` using its migration-only
|
||||
identity.
|
||||
2. The same runner succeeds at `mosaic-db-migrator --verify` against the exact destination and,
|
||||
only then, emits a signed target-bound attestation.
|
||||
3. Only then may `mosaic storage migrate-tier` verify that attestation and connect as dedicated
|
||||
non-DDL `mosaic_data_importer` to copy data.
|
||||
KBN-101-05 obtains the target URL from Vault KV-v2
|
||||
`secret-{env}/mosaic-stack/database/importer`, key `url`, and reads its authenticated version from
|
||||
the same successful response `data.metadata.version`. A hash or DSN byte sequence is not a
|
||||
provider version. The renderer treats URL bytes and provider version as one generation, writes a
|
||||
temporary generation directory with fsync plus atomic rename, and creates separate immutable
|
||||
consumer mounts. Swarm uses distinct versioned secret/config references. A deployment cannot mix
|
||||
generations.
|
||||
|
||||
The data migration never executes DDL or changes PostgreSQL roles, memberships, schemas,
|
||||
extensions, extension members, system catalogs, or the Drizzle ledger. Its only target writes are
|
||||
an allowlisted data-copy DML registry: inserts and idempotent conflict updates on declared mutable
|
||||
application tables. It has no DML grant for immutable KBN relations, extension/catalog objects, or
|
||||
any schema authority. It is not a schema bootstrap, extension installer, or repair procedure.
|
||||
PostgreSQL is always pre-migrated and runner-verified, including when the source is PGlite.
|
||||
| Consumer | Permitted material |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Migrator-attestation producer (`10003:10003`) | Its own migration URL/CA; read-only `/run/secrets/mosaic-migrate-target-url` and `/run/secrets/mosaic-migrate-target-version`, each `0400`, solely to bind; producer-only attestation output at `/run/mosaic-attestations-producer/migrate-target.v1.json`; root-wrapper-only signing key. It never connects with, uses, exports, logs, or forwards the importer URL/version. |
|
||||
| Privileged deployment handoff controller | After runner success and before importer creation, it receives only root-owned non-secret expected provider-version/URL-SHA-256/generation descriptor and pinned public verifier key—not URL bytes or private key. It safe-opens/verifies descriptor and producer artifact, copies exact bytes to a new importer-only mount with fsync/atomic rename, sets `10002:10002` `0400`, seals it read-only, and refuses importer start on any partial/wrong-generation/wrong-owner/mode result. |
|
||||
| Importer (`10002:10002`) | Its own immutable `0400` copies at the same URL/version paths; CA at exact `DATABASE_TLS_CA_CERT_PATH=/run/secrets/mosaic-db-ca.crt`; pinned Ed25519 public key; read-only `/run/mosaic-attestations/migrate-target.v1.json` supplied only by the sealed handoff. |
|
||||
| Gateway/runtime/unrelated container | No importer URL/version, importer artifact, attestation private key, or unrelated CA mount. |
|
||||
|
||||
## Prerequisites
|
||||
The migrator and importer safe-open URL, provider-version, attestation, and public-key files only
|
||||
with `O_RDONLY|O_CLOEXEC|O_NOFOLLOW`; they validate from the opened fd that the file is regular,
|
||||
has its expected owner/mode and link count one. The migrator digests only that URL fd for binding,
|
||||
then zeroizes/closes it. The importer reads URL bytes once into protected memory, validates the
|
||||
signed binding and exact CA before connecting from those same bytes, then zeroizes/closes every
|
||||
fd. It neither logs nor exposes a URL/version/attestation/key oracle.
|
||||
|
||||
- Federated stack is running and healthy (see [Federated Tier Setup](../federation/SETUP.md)).
|
||||
- An approved source backup exists and the target is identified in the KBN-101 change record.
|
||||
- The target passed the runner `--run` and `--verify` sequence above; `--verify` is the only
|
||||
trusted attestation producer after verified TLS, identity, manifest, and schema checks.
|
||||
- The operator has an approved, dedicated high-entropy importer credential file at
|
||||
`/run/secrets/mosaic_migrate_target_url` and its paired non-secret attestation artifact at
|
||||
`/run/mosaic-attestations/migrate-target.v1.json`. Neither file contents are printed, copied,
|
||||
committed, or placed on argv.
|
||||
- KBN-101-02 validates both references before any target connection: each is a regular,
|
||||
non-symlink file with the exact required owner/mode. The importer gets only pinned public
|
||||
verification key(s) and the attestation; the runner-only Ed25519 private key never reaches
|
||||
importer/runtime. The URL file is opened, digested, parsed, and connected from the same
|
||||
in-memory bytes with no reread.
|
||||
## Produced command interface
|
||||
|
||||
### Source modes
|
||||
|
||||
- **Local PGlite:** the source is the explicit `PGLITE_DATA_DIR`; it does not read or
|
||||
interpret a PostgreSQL URL.
|
||||
- **PostgreSQL source:** this active route does not accept `DATABASE_URL` as a source or
|
||||
target fallback. A PostgreSQL-source procedure needs its own independently approved,
|
||||
non-secret credential-reference contract before it is enabled.
|
||||
|
||||
## Dry-run first
|
||||
|
||||
Run the migration only after the target schema is prepared and verified:
|
||||
After activation and only after approved target preparation, the future interface is:
|
||||
|
||||
```bash
|
||||
mosaic-db-migrator --run
|
||||
mosaic-db-migrator --verify
|
||||
mosaic storage migrate-tier --to federated \
|
||||
--target-url-file /run/secrets/mosaic_migrate_target_url \
|
||||
--target-url-file /run/secrets/mosaic-migrate-target-url \
|
||||
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
The frozen KBN-101-02 options are exactly
|
||||
`--target-url-file /run/secrets/mosaic_migrate_target_url` and
|
||||
`--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json`. They are
|
||||
non-secret file references, not URL options. The runner and importer use distinct connections:
|
||||
the runner has migration DDL authority; the importer has only approved data-copy DML authority.
|
||||
The provider-version file is fixed deployment material, not argv. This connecting dry-run consumes its nonce; before an actual copy, obtain fresh `mosaic-db-migrator --verify` and a new sealed handoff. The runner uses its migration
|
||||
identity; the importer connects only as non-DDL `mosaic_data_importer` and only after all
|
||||
pre-connect validation. After verified TLS and before DML it compares PostgreSQL system ID,
|
||||
database OID, `current_user`, CA/SPKI, and manifest/schema fingerprints to the artifact.
|
||||
|
||||
Expected dry-run output is sanitized and contains no connection string, username, host,
|
||||
database name, SQL, or credential value:
|
||||
## Required refusals and evidence
|
||||
|
||||
```text
|
||||
[migrate-tier] Source tier: pglite
|
||||
[migrate-tier] Target tier: federated
|
||||
[migrate-tier] Target schema: verified by mosaic-db-migrator
|
||||
[migrate-tier] Target connection: dedicated importer
|
||||
[migrate-tier] Precondition: target is ready
|
||||
users: 5 rows
|
||||
teams: 2 rows
|
||||
conversations: 12 rows
|
||||
messages: 187 rows
|
||||
... (all tables listed)
|
||||
[migrate-tier] NOTE: Source tier has no pgvector support. insights.embedding will be NULL on all migrated rows.
|
||||
[migrate-tier] DRY-RUN COMPLETE (no data written). 206 total rows would be migrated.
|
||||
```
|
||||
KBN-101-02/-03/-05/-06 must prove, with stable sanitized errors, that no target connection occurs
|
||||
for missing/unsafe URL/version/attestation/public-key files; symlink, hardlink, owner, mode, or
|
||||
TOCTOU violations; mixed URL/version generations; missing/wrong CA mount; stale/replayed/tampered
|
||||
or revoked-key artifacts; provider rotation/revocation; wrong TLS/server/database/role/manifest
|
||||
binding; raw `--target-url`; `DATABASE_URL` fallback; runtime/owner identity; consumer leakage;
|
||||
or any DDL attempt. Post-connect identity mismatch closes with zero DML/DDL. Tests also prove no
|
||||
forwarding, child environment, logging, or error oracle leaks URL/version/key/artifact contents.
|
||||
|
||||
Review the output. If it reports a failed runner verification, unexpected role, unsafe
|
||||
secret-file owner/mode, non-empty target, or any connection/DDL precondition error, stop
|
||||
and remediate through the KBN-101 control plane.
|
||||
The attestation is credential-free JCS with detached Ed25519 signature and binds issued/expiry,
|
||||
nonce, authenticated provider version, exact URL-fd SHA-256, TLS host/port/database, CA/SPKI,
|
||||
PostgreSQL system ID/database OID, importer role, manifest/schema, and producer identity. Provider
|
||||
version rotation invalidates an old artifact and requires a fresh rendered generation plus runner
|
||||
verification.
|
||||
|
||||
## Run the data migration
|
||||
## Actual copy after dry-run
|
||||
|
||||
After a reviewed dry-run, repeat the same verified route with `--yes`:
|
||||
After reviewed dry-run, obtain the required fresh verification/attestation generation, then use:
|
||||
|
||||
```bash
|
||||
mosaic-db-migrator --verify
|
||||
mosaic storage migrate-tier --to federated \
|
||||
--target-url-file /run/secrets/mosaic_migrate_target_url \
|
||||
--target-url-file /run/secrets/mosaic-migrate-target-url \
|
||||
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
|
||||
--yes
|
||||
```
|
||||
|
||||
`--yes` only confirms data copy in non-TTY automation. It does not bypass schema
|
||||
verification, target-file validation, role checks, redaction, or the no-DDL boundary.
|
||||
The dry-run artifact is terminally replayed and must be rejected; `--yes` bypasses no file,
|
||||
generation, signature, TLS, identity, or DDL control.
|
||||
|
||||
The command will:
|
||||
## Data boundary and recovery
|
||||
|
||||
1. Before target connection, validate both files, JCS/Ed25519 signature and pinned key ID,
|
||||
expiry, nonce replay, secret version/digest, canonical TLS host/port/database, CA/SPKI,
|
||||
expected importer role, and manifest/schema bindings.
|
||||
2. Connect only from the once-opened in-memory URL bytes, then before transaction/DML compare
|
||||
PostgreSQL system identifier, database OID, `current_user`, CA/SPKI, and manifest/schema
|
||||
fingerprints to the attestation; mismatch closes with zero DML/DDL.
|
||||
3. Copy data in dependency order without DDL.
|
||||
4. Report sanitized row counts and warnings.
|
||||
The importer has only an allowlisted mutable-table DML registry. It has no grant for immutable KBN
|
||||
relations, schemas, roles, memberships, extensions, catalogs, or the Drizzle ledger. Source PGlite
|
||||
uses its explicit local directory and does not make a PostgreSQL URL fallback valid.
|
||||
|
||||
## Rejected interfaces and failure-before-connect rules
|
||||
|
||||
KBN-101-02 and KBN-101-06 must prove these failures occur before a target connection or
|
||||
DDL attempt:
|
||||
|
||||
- Raw `--target-url` or any URL/credential value on argv.
|
||||
- `DATABASE_URL` as a target or source fallback.
|
||||
- A runtime owner, migrator, schema owner, or extension owner used as the data importer.
|
||||
- Missing, non-regular, symlinked, wrong-owner, wrong-mode, substituted, or changed-after-check
|
||||
credential file.
|
||||
- Missing, stale, replayed, tampered, wrong-key, revoked-key, expired, malformed, or wrong-binding
|
||||
attestation file.
|
||||
- Wrong host, database, CA/SPKI, PostgreSQL system identifier, database OID, importer role,
|
||||
manifest, schema, or credential secret-version/digest.
|
||||
- Any mode other than the runner-prepared/verified PostgreSQL destination route.
|
||||
- Any direct DDL attempt by the migration command or its target connection.
|
||||
|
||||
The signed attestation has canonical UTF-8/JCS bytes and detached Ed25519 signature. Its
|
||||
credential-free v1 payload binds format/version/key ID; issued/expiry/nonce; exact credential
|
||||
secret version and SHA-256; canonical TLS host/port/database; CA/SPKI; PostgreSQL system
|
||||
identifier/database OID; importer role; manifest/schema; and producer invocation/build/image/
|
||||
correlation. Artifacts are atomically written/renamed `0444`; the replay cache survives until
|
||||
expiry; public-key overlap/revocation and credential rotation force a fresh runner verification.
|
||||
|
||||
KBN-101-02 SHALL add `packages/storage/src/migrate-tier.spec.ts::secureTargetRouteAndAttestation`
|
||||
and its integration counterpart. They prove zero target connection for pre-connect failures and
|
||||
zero DML/DDL for post-connect identity mismatch. The finite operator-document scanner rejects an
|
||||
unowned route or a guide that reintroduces a credential-bearing command.
|
||||
|
||||
## What gets migrated
|
||||
|
||||
All persistent, user-bound data is migrated in dependency order:
|
||||
|
||||
- **users, teams, team_members** — user and team ownership
|
||||
- **accounts** — OAuth provider tokens (durable credentials)
|
||||
- **projects, agents, missions, tasks** — all project and agent definitions
|
||||
- **conversations, messages** — all chat history
|
||||
- **preferences, insights, agent_logs** — preferences and observability
|
||||
- **provider_credentials** — stored API keys and secrets
|
||||
- **tickets, events, skills, routing_rules, appreciations** — auxiliary records
|
||||
|
||||
Full order is defined in code (`MIGRATION_ORDER` in `packages/storage/src/migrate-tier.ts`).
|
||||
|
||||
## What gets skipped and why
|
||||
|
||||
Three tables are intentionally not migrated:
|
||||
|
||||
| Table | Reason |
|
||||
| ----------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| **sessions** | TTL'd auth sessions from the old environment; they will fail JWT verification on the new target |
|
||||
| **verifications** | One-time tokens (email verify, password reset) that have either expired or been consumed |
|
||||
| **admin_tokens** | Hashed tokens bound to the old environment's secret keys; must be re-issued |
|
||||
|
||||
**Note on accounts and provider_credentials:** These durable credentials are migrated because
|
||||
they are user-bound and required for resuming agent work on the target. After migration to a
|
||||
multi-tenant federated deployment, operators may want to audit or wipe these if users are
|
||||
untrusted or credentials should not be shared.
|
||||
|
||||
## Idempotency, concurrency, and verification
|
||||
|
||||
The data copy is idempotent and uses an importer-scoped advisory lock. It may be retried only
|
||||
through the same runner-verified, file-reference route. Do not inspect, unlock, or repair the
|
||||
target with ad hoc SQL; a failed or ambiguous run is a control-plane incident with backup and
|
||||
rollback evidence.
|
||||
|
||||
After a successful migration, use the KBN-101-approved sanitized verification command or
|
||||
read-only operator report. Do not paste a target connection string into a shell command,
|
||||
terminal history, ticket, or documentation.
|
||||
|
||||
## Rollback
|
||||
|
||||
There is no in-place rollback. If data migration fails:
|
||||
|
||||
1. Keep the previous target state and restore the approved pre-migration backup where required.
|
||||
2. Preserve sanitized failure and audit evidence; do not expose the credential file or contents.
|
||||
3. Investigate through the KBN-101 control plane and rerun only after independent review.
|
||||
|
||||
Always rehearse this route in a staging environment first.
|
||||
A failed or ambiguous migration is a control-plane incident: preserve sanitized evidence, retain
|
||||
the approved backup/rollback state, and retry only after independent review. Never inspect,
|
||||
unlock, repair, or initialize the target with ad hoc SQL or copied credentials.
|
||||
|
||||
Reference in New Issue
Block a user