stack/docs/native-kanban-sot/KBN-101-DB-ROLE-SPLIT.md

KBN-101 — Database Runtime/Migration Role Split

Status: frozen implementation contract; documentation-only planning for #771

Version: 1.0.0-rc.5

Dependency: KBN-010 → KBN-101 foundation → KBN-100 → KBN-101 deployed-role certification → KBN-105 Scope: PostgreSQL standalone/federated runtime identity, migration identity, readiness, deployment handoff, and evidence. No database, secret, deployment, CI, or runtime mutation is authorized by this card.

1. Decision and current-state evidence

Current main inspection found one DATABASE_URL used by packages/db/src/client.ts, packages/db/src/migrate.ts, Drizzle Kit, storage PostgresAdapter.migrate(), Gateway startup, installer output, Portainer federated stack, and CI. DatabaseModule.onModuleInit() invokes storageAdapter.migrate() for PostgreSQL, so each Gateway starts migrations under its application identity. The current config path also supplies PostgreSQL defaults. This prevents a production proof that the deployed runtime identity cannot update/delete KBN immutable tables.

This contract separates those duties. It does not claim that existing synthetic roles, current defaults, or current single-URL deployments satisfy the target model.

Frozen names and modes

Item	Exact name / shape	Rule
Runtime URL variable	DATABASE_URL	PostgreSQL runtime non-owner connection. Required for standalone and federated; runtime-only containers/services receive this variable and never DATABASE_MIGRATION_URL.
Migration URL variable	DATABASE_MIGRATION_URL	PostgreSQL migrator connection. Required only by an explicit migration Job/one-shot operator phase in standalone and federated; it is forbidden in Gateway runtime service environments.
Runtime config DTO	DatabaseRuntimeConnectionConfigDto	Internal configuration DTO: tier, databaseUrl, runtimeMode. It maps only DATABASE_URL; URL values are never returned, logged, traced, or placed in public DTOs.
Migration config DTO	DatabaseMigrationConnectionConfigDto	Internal migration-command DTO: tier, migrationDatabaseUrl, migrationMode. It maps only DATABASE_MIGRATION_URL; it cannot be constructed by the Gateway runtime bootstrap.
Readiness DTO	DatabaseSchemaReadinessDto	Sanitized internal/admin health shape: state, expectedSchemaVersion, observedSchemaVersion, migrationRequired, roleCheck, checkedAt. It contains neither URL nor password nor host.
TLS variable / DTO	DATABASE_TLS_CA_CERT_PATH; DatabaseTlsConfigDto	Required in PostgreSQL production-like modes. Internal DTO is caCertificatePath, rejectUnauthorized: true, serverName; it derives server name from the URL host and validates the mounted CA file without serializing its content.
Modes	local, standalone, federated	local is the PGlite exception. standalone and federated are PostgreSQL production-like modes and must enforce this contract.

No fallback: in standalone or federated, a missing DATABASE_MIGRATION_URL is a migration-phase configuration error. The migration command must not fall back to DATABASE_URL, a config-file URL, a default URL, or an inferred URL. A missing DATABASE_URL or DATABASE_TLS_CA_CERT_PATH is a runtime startup error. Both PostgreSQL URLs must be parsed with authenticated TLS configured from DatabaseTlsConfigDto; reject sslmode=disable, allow, prefer, require, no-verify, certificate-unverified configuration, and URL/host mismatch. PGlite uses its configured data directory and its explicit local migration routine; it neither requires nor interprets PostgreSQL connection/TLS variables.

ASSUMPTION K101-A1: existing production-like tiers are standalone and federated, as defined by MosaicConfig. If a new PostgreSQL tier is introduced, it inherits the production-like contract until a versioned amendment says otherwise.

2. Explicit migration phase and fail-closed readiness

1. Deployment supplies DATABASE_MIGRATION_URL only to a dedicated migration command/Job. That command opens one migration session, uses pg_try_advisory_lock(hashtext('mosaic-schema-migration-v1')), returns DATABASE_MIGRATION_LOCKED immediately if unavailable, runs the ordered migration set as the migrator role, verifies the final schema version, releases the lock on that same session, and exits.
2. The application Gateway never calls runMigrations(), drizzle-kit migrate, adapter migration, extension creation, or any schema DDL on PostgreSQL startup. Replica startup can only perform runtime identity and schema readiness checks.
3. KBN-101-03 repairs the current PostgreSQL migration-runner/journal ordering defect before foundation certification: it must use a hash-complete ordered runner, repair the _journal.json order, and prove on a clean PostgreSQL database that every shipped migration hash (including 0009) is applied exactly once. expectedSchemaVersion is the deterministic SHA-256 of the resulting ordered shipped Drizzle migration journal entries and migration hashes. observedSchemaVersion is calculated from drizzle.__drizzle_migrations; equality requires the same ordered complete set—no missing, duplicate, unknown, or out-of-order ledger entry. This is the readiness contract, not a best-effort count.
4. The Gateway starts/advertises readiness only when DatabaseSchemaReadinessDto.state === 'ready', observed equals expected, and the role/grant checks in §4 pass. Any pending, partial, unknown, corrupt, or mismatched schema is migration_required or schema_mismatch, emits a sanitized operator code, and fails closed before serving requests.
5. Replicas do not wait for or race to migrate. They remain unready until the deployment controller observes a successful migration phase and each replica independently verifies the exact version/role checks. Migration concurrency is prevented by the non-blocking advisory lock; deployment sequencing permits no application replicas before its successful exit.
6. PostgreSQL extension provisioning requiring elevated database privilege (including any provider-specific extension prerequisite) is a separately audited platform/bootstrap step. It is not performed by Gateway runtime or accepted as a reason to grant the migration role superuser.

3. PostgreSQL role and privilege model

Role creation, passwords, membership, database ownership, and Vault values are platform/IaC/operator work—not Drizzle/application migrations. Application SQL must not issue CREATE ROLE, ALTER ROLE ... PASSWORD, GRANT ... TO LOGIN credential creation, or embed credentials.

Role	Attributes and ownership	Membership / session use
mosaic_platform_database_owner	NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; platform-only database owner after bootstrap. It owns the database but is never granted to application roles.	Outside application runtime/migration paths and separately audited.
external platform bootstrap actor	Provider/operator/IaC-controlled privileged identity, outside the Mosaic role graph and never placed in Vault/app configuration.	Creates a fresh database and roles, then transfers database ownership to mosaic_platform_database_owner; for an existing database it performs the audited ownership/grant transition. It is not used after bootstrap.
mosaic_schema_owner	NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; owns only the Mosaic application schema and drizzle migration-ledger schema plus their objects. It does not own the database and lacks database CREATE, ALTER DATABASE, and extension-install authority.	Never used as an application login.
mosaic_migrator	LOGIN NOINHERIT NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; has a separately stored migration credential.	Is granted only mosaic_schema_owner. The migration runner verifies session_user = mosaic_migrator, executes SET ROLE mosaic_schema_owner, and verifies current_user = mosaic_schema_owner before DDL.
mosaic_runtime_capability	NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; owns no schema/object.	Holds the application table/sequence/schema privileges below.
mosaic_runtime	LOGIN INHERIT NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS; owns no schema/object.	Is granted only mosaic_runtime_capability WITH INHERIT TRUE, SET FALSE, ADMIN FALSE; it is never member of either owner/migrator role and cannot SET ROLE to them. Its inherited capability grants are effective without changing current_user.

The external platform bootstrap actor creates the database and role graph, transfers database ownership to mosaic_platform_database_owner, creates the application schema and drizzle ledger schema, makes mosaic_schema_owner their owner, revokes CREATE and function EXECUTE on both from PUBLIC, revokes database-level create/connect/TEMPORARY privileges not explicitly needed (including REVOKE TEMPORARY ON DATABASE <database> FROM PUBLIC), and grants CONNECT only to the named login roles. The runtime receives USAGE on the application schema, USAGE, SELECT on required sequences, and table rights only through mosaic_runtime_capability. It receives only USAGE on drizzle plus SELECT on drizzle.__drizzle_migrations; INSERT, UPDATE, DELETE, and TRUNCATE on that ledger are explicitly revoked. The migration role/owner retains schema DDL authority; ordinary runtime has no CREATE, TEMP, schema ownership, role-administration, extension, or database-administration privilege. Production search_path is explicitly pg_catalog, <mosaic_application_schema>; current-main public is the N-1 schema name until a separately versioned schema relocation is complete. Runtime startup verifies it and rejects untrusted writable schemas before application queries.

For ordinary mutable application tables, grant SELECT, INSERT, UPDATE, DELETE through mosaic_runtime_capability. ALTER DEFAULT PRIVILEGES FOR ROLE mosaic_schema_owner IN SCHEMA <mosaic_application_schema> grants that baseline to future ordinary tables and sequence rights to future sequences; it also revokes EXECUTE ON FUNCTIONS FROM PUBLIC. The KBN-101 N-1 implementation uses current-main public as that schema name until the separately versioned relocation. SECURITY DEFINER functions are forbidden by default; a separately versioned reviewed exception must pin search path, revoke PUBLIC execute, grant only the named capability role, and add a negative privilege test. Each migration creating an immutable table must then apply the immutable override in the same migration and verify it:

KBN immutable relation	Runtime rights	Explicitly absent
task_events	SELECT, INSERT	UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER
artifacts	SELECT, INSERT	UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER
task_checkpoints	SELECT, INSERT	UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER
task_checkpoint_artifacts	SELECT, INSERT	UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER
approval_decision_artifacts	SELECT, INSERT	UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER

The named list matches the KBN v1 contract’s event/checkpoint/artifact/evidence relations. KBN-100 must retain RESTRICT parent deletion semantics; grants alone do not prevent a privileged owner or a cascading FK from erasing evidence. Any future immutable relation must be added to this table in a versioned amendment before code ships.

4. Effective-identity verification and operator errors

Before accepting traffic, the runtime uses its already-open DATABASE_URL connection to query sanitized identity/privilege metadata (session_user, current_user, current_database, current_schema, pg_has_role, has_schema_privilege, has_table_privilege, rolsuper, rolcreaterole, rolcreatedb, rolbypassrls) and the migration ledger. It must fail closed when any condition holds:

• session_user and current_user resolve to the same effective identity as mosaic_platform_database_owner, mosaic_schema_owner, or mosaic_migrator, or runtime can assume any of them;
• the runtime role is a superuser, CREATEROLE, CREATEDB, BYPASSRLS, database/table/schema owner, has database TEMPORARY, or has schema/DDL/extension/role-management authority;
• required runtime inherited-capability membership, grants, sequence rights, read-only Drizzle-ledger access, no unexpected PUBLIC/runtime function EXECUTE, search_path, authenticated-TLS configuration, database/schema identity, or schema readiness are missing/mismatched;
• after KBN-100 creates the immutable relations, immutable INSERT/SELECT rights or immutable UPDATE/DELETE denials are missing/mismatched, or an expected immutable relation is absent. The KBN-101 foundation verifies only the role/schema boundary; its post-KBN-100 certification verifies the immutable relation matrix.

The migration command performs the reciprocal preflight: it requires the dedicated migration URL, the named migrator session user, schema owner after SET ROLE, no superuser/CREATEROLE/database-owner/database-CREATE requirement, the approved database/schema target, and an acquired advisory lock. It fails before DDL otherwise.

Errors have stable sanitized codes: DATABASE_RUNTIME_URL_REQUIRED, DATABASE_MIGRATION_URL_REQUIRED, DATABASE_ROLE_UNSAFE, DATABASE_ROLE_GRANT_MISMATCH, DATABASE_SEARCH_PATH_UNSAFE, DATABASE_SCHEMA_MISMATCH, DATABASE_MIGRATION_LOCKED, and DATABASE_MIGRATION_IDENTITY_UNSAFE. Operator logs/metrics may contain code, tier, schema-version fingerprints, role class (runtime/migrator/owner), and correlation ID—not a DSN, username, host, database name, SQL parameter, secret, or raw catalog result. External health returns unavailable/not-ready without role detail.

5. N-1 rollout, compatibility, and rollback

1. Prepare (N-1, no production mutation in KBN-101): publish this contract, enumerate all consumers of DATABASE_URL, identify installer/compose/Portainer/CI/Vault handoff owners, and add implementation tests. Existing deployments remain one-URL only until the implementation card’s compatibility release.
2. Compatibility release: N-1 compatibility preserves the DATABASE_URL variable/config shape, not an owner-runtime exception. A current single-URL deployment remains on its prior release until roles are provisioned. The KBN-101 runtime release starts only after DATABASE_URL has been changed to the non-owner runtime role; it refuses owner/migrator identity, does not auto-migrate, and never treats that URL as DATABASE_MIGRATION_URL.
3. Provision and migrate: platform operators create the roles and Vault records, issue distinct URLs, run the explicit migration phase using DATABASE_MIGRATION_URL, verify exact schema and grants, then deploy runtime with only DATABASE_URL.
4. Enforce: after deployed-role certification is green, require non-owner DATABASE_URL for production-like runtime. Remove the N-1 compatibility path only in a separately versioned contract/release after fleet inventory proves no single-URL deployment remains.
5. Rollback: before any incompatible KBN-100 schema switch, stop/scale runtime replicas, restore the previous runtime release and its known schema-compatible backup/manifest, and re-run only the approved prior migration/restore procedure under the migration identity. Never put the migration URL into the application to make rollback work. After KBN-100 canonical mutation/switch, its existing freeze/reconciliation authority governs data rollback; role separation remains in force.

ASSUMPTION K101-A2: a controlled migration Job/one-shot command exists or will be added by the deployment slice. If a target cannot schedule one, its deployment is not eligible for federated/production-like KBN certification.

6. Secrets, observability, and deployment handoff

Vault owns three distinct values, with least-privilege policies and no plaintext copies: secret-{env}/mosaic-stack/database/runtime (url) for the runtime role, secret-{env}/mosaic-stack/database/migrator (url) for the migration role, and secret-{env}/mosaic-stack/database/tls-ca (certificate) mounted at DATABASE_TLS_CA_CERT_PATH for both authorized PostgreSQL phases. Exact existing mount/service naming must be verified by the deployment owner before creation; this naming is a planned canonical target, not a claim that those paths already exist. CI uses isolated disposable non-production roles/URLs and must model both variables without promoting test credentials.

Rotation is independent: rotate the runtime credential first with overlapping database validity and rolling runtime restart; rotate the migration credential only while no migration is active; rotate the CA only with verified trust overlap and rolling consumers; revoke old material after connection/pipeline evidence. Rotation never downgrades a runtime role or causes a Gateway to receive the migration secret. Secret injection maps runtime secret only to Gateway and migration secret only to migration Job; both are redacted by config/error/logging/OTEL processors. Secret names may be logged; values, DSNs, URI components, and decoded credentials may not.

Readiness/metrics distinguish database connectivity, exact schema readiness, identity safety, immutable-grant verification, and migration-phase outcome. They expose codes and versions only. Runbooks must tell operators to inspect secret binding/role grants/schema phase through privileged deployment tooling—not to paste URLs into shell history, issues, logs, or chat.

Deployment-specific ownership is deliberately handed off: installer/wizard, local compose, federated compose, Portainer stack, CI, Vault policy/provisioning, and runtime deployment manifests each require a separate one-card/one-PR implementation slice below. KBN-101 performs none of those mutations.

7. Implementation DAG and file manifest

Each card is one PR and has exclusive ownership. No card absorbs another card’s files.

Card	Depends on	Exact implementation ownership	Required result
KBN-101-00 platform bootstrap / IaC	KBN-101 contract	infra/pg-init/01-extensions.sql, new infra/pg-init/02-kbn101-role-bootstrap.sql, platform deployment/IaC change set, and KBN-101-07 transition runbook/evidence	External bootstrap actor creates fresh DB/roles then transfers database ownership; existing DB transition inventories owners, creates/rotates roles, transfers schema/table/ledger ownership, applies grants/default privileges, provisions Vault paths, and records reversible evidence. It is not an application migration and no login credential is hardcoded.
KBN-101-01 runtime config/DB boundary	KBN-101-00	packages/config/src/mosaic-config.ts, .spec.ts; packages/db/src/client.ts, defaults.ts, relevant DB tests; apps/gateway/src/database/database.module.ts and tests	Typed split DTO/config, authenticated TLS/CA validation, no production defaults/fallback, no PostgreSQL startup migration, readiness/identity verifier seam.
KBN-101-02 storage migration separation	KBN-101-01	packages/db/drizzle.config.ts; packages/storage/src/adapters/postgres.ts, .spec.ts; packages/storage/src/{factory,types,cli,migrate-tier,tier-detection}.ts and focused tests; apps/gateway/src/main.ts and startup tests; packages/mosaic/src/commands/fleet-backlog.ts and tests	Storage adapter/runtime cannot migrate on Gateway startup; Drizzle Kit/migration-capable commands use only migration DTO/URL and fail if absent; no runtime URL fallback. Replace all runtime CREATE EXTENSION probes with read-only extension/catalog checks; prohibit PostgreSQL fleet-backlog auto-migration or route it to the explicit migration command.
KBN-101-03 role foundation + declarative snapshot	KBN-101-00, KBN-101-01	packages/db/src/{migrate,index}.ts, schema.ts, packages/db/drizzle/**, migration tests, generated Drizzle snapshot/journal	Platform/schema ownership boundary, baseline grants/default privileges/function-execute revocation, ordered hash-complete PostgreSQL migration runner/journal repair, lock/version behavior; clean-PG proof every shipped hash applies exactly once; generated declarations and snapshot exact consistency. It creates no future KBN immutable relation.
KBN-101-04 installer/wizard/config migration	KBN-101-01	packages/mosaic/src/stages/gateway-config.ts, .spec.ts, relevant wizard/integration tests; tools/install.sh only if it owns emitted variables	Production-like installer/config output stores no PostgreSQL DSN in generated .env or mosaic.config.json; it records only non-secret deployment/Vault references and accepts injected runtime variables. Tests prove neither runtime nor migration URL is written, echoed, or logged; production-like config loading rejects persisted/config-file DSN fallback.
KBN-101-05 compose/Portainer deployment handoff	KBN-101-00, KBN-101-01, KBN-101-03	docker-compose.yml, docker-compose.federated.yml, deploy/portainer/federated-test.stack.yml, corresponding deployment docs/runbooks	Dedicated migration phase and secret binding boundary; runtime receives non-owner only.
KBN-101-06 CI/test topology	KBN-101-02, KBN-101-03	.woodpecker/ci.yml, CI-only test helpers, targeted integration tests	Ephemeral owner/migrator/runtime identities; migration then runtime grant/denial evidence; no production credential.
KBN-101-07 Vault/runbook/observability	KBN-101-00, KBN-101-04, KBN-101-05, KBN-101-06	deployment/Vault/runbook/operations docs and approved observability configuration only	Provision/rotation/incident/rollback instructions, redaction and handoff evidence.
KBN-101-08 foundation certification gate	KBN-101-00…07	integration evidence only; no opportunistic source scope	Independent security review and deployed runtime role/schema-boundary certificate, including platform-bootstrap fresh/existing transition evidence; no immutable KBN table claim before KBN-100.
KBN-101-09 post-KBN-100 certification gate	KBN-100, KBN-101-08	integration evidence only; no opportunistic source scope	Real deployed-role INSERT/SELECT and UPDATE/DELETE-denial evidence for every KBN-100 immutable relation; independent security/Ultron approval.

KBN-100 starts only after KBN-101-08 passes. It must rebase on the resulting main head, restore Drizzle generated declaration/snapshot/journal consistency, and keep its procedural immutable-table grant/trigger/backfill additions bounded to KBN-100’s schema migration ownership. It may not create roles, passwords, Vault entries, deployment secrets, or mutate environment wiring. KBN-101-09 performs real deployed-role certification after those relations exist and must pass before KBN-105.

8. Threat, tests, and acceptance traceability

Threat / failure	Frozen control	Required evidence
Gateway role owns tables or runs DDL	Separate runtime/migration URLs; no runtime DDL path, including extension probes and fleet-backlog; owner/migrator checks	Runtime startup tests prove migration/extension calls impossible; deployed runtime catalog certificate.
Missing migration secret falls back to runtime URL	Explicit migration DTO requires DATABASE_MIGRATION_URL	Unit/integration negative proving absent migration URL fails before connection/DDL.
Replica migration race	Dedicated phase plus advisory lock; replicas readiness-only	Concurrent migration invocation: one lock holder, other DATABASE_MIGRATION_LOCKED; replicas do no DDL.
Role escalation / unsafe grant	Inherited runtime capability with SET/ADMIN denied; no owner membership; catalog privilege preflight fails closed	Superuser/CREATEROLE/owner/SET ROLE/TEMP/ledger/function-EXECUTE mismatch negatives.
Evidence rewrite/deletion	KBN-100 immutable grants plus RESTRICT FKs	After KBN-100, real mosaic_runtime can INSERT/SELECT and receives permission denial for UPDATE/DELETE on every listed relation; owner-only administrative path separately audited.
Schema drift / partial migration	Hash-complete ordered ledger fingerprint readiness	Clean-PG every-hash-once, missing/unknown/out-of-order/partial ledger, and stale replica tests fail unready.
Dsn/credential leakage / on-path attack	DTO boundary, redaction rules, secret separation, authenticated TLS	Error/log/OTEL snapshots contain no URL/user/host/password; runtime/migrator reject missing/unverified CA, insecure sslmode, and host mismatch; rotation test preserves redaction.
Rollback reintroduces owner runtime	Ordered rollback and secret binding controls	Rollback rehearsal verifies restored runtime gets only runtime URL and remains non-owner.

Acceptance criterion	Evidence needed before KBN-101 closes
AC-K101-01 Split connection contract	Exact variable/DTO tests for local, standalone, federated; no production fallback/default.
AC-K101-02 Explicit migration/readiness	Clean PostgreSQL every-hash-once runner/journal test; dedicated migration command/lock/version tests; Gateway/replica non-migration test.
AC-K101-03 Least-privilege PostgreSQL roles	Catalog assertions for ownership, inherited-only capability with SET/ADMIN denial, search path, ledger read-only grant, TEMP/function-EXECUTE denial, grants/default privileges, unsafe attributes.
AC-K101-04 Immutable KBN privilege proof	KBN-101-09, after KBN-100, proves deployed mosaic_runtime INSERT/SELECT allowed and UPDATE/DELETE denied for every immutable relation.
AC-K101-05 N-1 and recovery	Single-URL compatibility diagnostic; staged rollout and rollback rehearsal with no production mutation in this planning card.
AC-K101-06 Secrets/operations	Vault/injection/rotation/redaction/runbook review, health/metrics codes, operator failure paths.
AC-K101-07 Delivery integrity	One-card/one-PR DAG; strict contract/docs/link checks; independent author≠reviewer security review; terminal-green CI.

9. Non-goals and residual authority

KBN-101 planning does not execute a production role change, create Vault paths, change compose/CI/installer files, alter database objects, or certify any deployed identity. It does not replace KBN-100’s data migration, immutable retention design, tenant constraints, or KBN-105 endpoint freeze. A database superuser or break-glass operator remains outside application containment and requires separately audited platform controls, backup/audit evidence, and drills.