Scratchpad — KBN-101 DB runtime/migration role split (#771)

Branch: docs/771-kbn101-db-role-split
Base: main e9c4aa3
Scope: planning/documentation only; authorized files are PRD, Native Kanban task/shared/index docs, sitemap, this scratchpad, and the new KBN-101 contract.
Explicit exclusions: source/runtime/config/deployment/secret/migration/compose/CI/lock/package/KBN-100 branch edits; no production mutation.

Objective

Freeze an implementation-ready PostgreSQL role/connection split so the Gateway uses a non-owner runtime identity and only a dedicated migration phase uses an owner/migrator identity. Make real deployed-role certification—not synthetic role tests—a serial prerequisite of KBN-100 and KBN-105.

Intake and current-state evidence

Mission MVP is active; W3 Native Kanban/SOT is planning-complete. The task state shows KBN-010 as the predecessor and KBN-100 as the current schema slice.
Current branch started at e9c4aa3; .mosaic/orchestrator/{mission.json,session.lock} were already runtime-modified and remain untouched.
packages/db/src/client.ts, migrate.ts, and drizzle.config.ts resolve one DATABASE_URL (with default fallback). packages/storage/src/adapters/postgres.ts calls runMigrations(this.url).
apps/gateway/src/database/database.module.ts calls storageAdapter.migrate() at startup for PostgreSQL; this is the owner-runtime defect to remove in KBN-101 implementation.
packages/config/src/mosaic-config.ts, installer wizard, local/federated compose, Portainer test stack, and .woodpecker/ci.yml currently expose one URL. PGlite has an existing explicit local migration path.
Current KBN contract requires immutable events/checkpoints/artifacts/evidence, RESTRICT, and KBN-100 generated Drizzle consistency. It did not establish a deployable runtime identity split.

Frozen decisions

DATABASE_URL is the non-owner runtime URL; DATABASE_MIGRATION_URL is migration-only. Both are required in their respective PostgreSQL phases; PGlite is the explicit local exception; migration never falls back to runtime/default/config URL.
PostgreSQL Gateway runtime never auto-runs migration/DDL. Dedicated migrator uses pg_try_advisory_lock(hashtext('mosaic-schema-migration-v1')); replicas only check exact ordered Drizzle-ledger readiness and fail closed.
Roles are non-login mosaic_platform_database_owner, non-login mosaic_schema_owner, login/noinherit mosaic_migrator, non-login mosaic_runtime_capability, and login/inherit mosaic_runtime. Runtime inherits only its capability role with SET/ADMIN denied, has no owner/migrator membership, no unsafe attributes/ownership/DDL authority, and an explicit trusted search path.
Runtime gets mutable DML only as needed, but INSERT/SELECT only on task_events, artifacts, task_checkpoints, task_checkpoint_artifacts, and approval_decision_artifacts. KBN-100 still enforces RESTRICT/no-cascade.
Startup verifies effective role/ownership/attributes/inherited capability/TEMP/function-execute/ledger grants/search path/immutable denials/schema fingerprint without DSN exposure. It also requires authenticated CA/hostname-verified TLS. Stable sanitized errors and redaction rules are required.
N-1 retains single runtime URL only as a non-certified compatibility release; staged role provisioning/migration/runtime deployment then enforces the split. Rollback never injects migration URL into Gateway.
Vault target paths, rotation, deployment injection, CI, installer, compose, Portainer, and observability are separate one-card/one-PR handoffs. The migration-only file manifest includes packages/db/drizzle.config.ts; KBN-101 repairs the known PostgreSQL runner/journal ordering defect and proves a clean database applies every hash once before its foundation certificate. No application migration creates roles/passwords or hardcodes credentials.
KBN-101 foundation merges/certifies first. KBN-100 then rebases, restores Drizzle declaration/snapshot/journal consistency, and bounds procedural immutable-table grant/trigger/backfill work to its own slice. Because those immutable relations do not exist until KBN-100, KBN-101’s real deployed-role immutable-operation certificate follows KBN-100 and is the serial gate before KBN-105.

Assumptions

standalone and federated are all current PostgreSQL production-like modes; a future PostgreSQL tier inherits this contract unless versioned otherwise.
Deployment will support a dedicated migration Job/one-shot command. A target that cannot run it cannot receive production/federated KBN certification.
Canonical Vault target paths require deployment-owner verification before provisioning; the planning document does not claim they already exist.

Documentation produced

docs/PRD.md: bounded KBN-101 requirements and acceptance criteria.
docs/native-kanban-sot/KBN-101-DB-ROLE-SPLIT.md: normative rc.5 implementation, threat, migration/rollback, evidence, and exact file DAG contract.
docs/native-kanban-sot/SHARED-CONTRACT.md: rc.5 amendment preserving rc.4.
docs/native-kanban-sot/TASKS.md: KBN-101 inserted before and blocks KBN-100; KBN-105 held.
Native Kanban index and root sitemap links.

Validation plan

Prettier only for changed Markdown.
Markdown link target/check checks scoped to modified docs.
Strict contract check with pnpm exec tsc --noEmit -p docs/native-kanban-sot/tsconfig.json (the frozen TypeScript contracts remain unchanged).
Diff allowlist proves only authorized documentation files changed, apart from pre-existing Mosaic runtime state.
Independent documentation/security self-review: role escalation, fallback, startup DDL, schema readiness, grants/default privileges, immutable tables, secret leakage, deployment and KBN-100 boundaries.

Review corrections

Independent Codex review found two blockers and security review found two medium defects; all were remediated in the frozen contract:

Split the KBN-101 certificate into a foundation role/schema-boundary certificate (before KBN-100) and real immutable-operation certificate (after KBN-100, before KBN-105). This preserves the requested KBN-100 block without requiring evidence for tables not yet created.
Removed the legacy owner-runtime exception. N-1 compatibility preserves variable/config shape only; the KBN-101 runtime refuses owner/migrator identity and current single-URL installs remain on their previous release until role cutover.
Introduced mosaic_platform_database_owner as a separate non-login platform role. mosaic_schema_owner owns application/ledger schemas only, not the database and has no database CREATE/ALTER/extension authority.
Replaced blocking pg_advisory_lock with pg_try_advisory_lock and the deterministic DATABASE_MIGRATION_LOCKED failure.
Review also flagged active .mosaic/orchestrator state. It was pre-existing launcher state and remains unstaged/uncommitted.
Second review added packages/db/drizzle.config.ts to the migration-only slice, mandates DATABASE_MIGRATION_URL with a missing-variable negative, grants runtime only USAGE plus SELECT on drizzle.__drizzle_migrations, and verifies/revokes its ledger writes.
Security review added DATABASE_TLS_CA_CERT_PATH / DatabaseTlsConfigDto with authenticated TLS and hostname/CA verification in production-like modes, explicit database TEMPORARY revocation/catalog denial testing, and default-PUBLIC function EXECUTE revocation with SECURITY DEFINER prohibited by default.
Final review corrected the runtime login to inherit only its capability role with SET/ADMIN denied, and moved the known hash-complete migration-runner/journal repair into KBN-101-03 before the foundation certificate.
Final manifest review added all live runtime DDL paths (packages/storage/src/tier-detection.ts, Gateway startup, and fleet-backlog) to KBN-101-02, requiring read-only extension probes and no PostgreSQL runtime auto-migration. It also requires KBN-101-04 to stop persisting either DSN into generated .env/mosaic.config.json, using only Vault/deployment references and injected variables.
Provisioning review separated the external privileged platform bootstrap actor from the NOCREATEDB database-owner role and added KBN-101-00. That IaC/bootstrap card owns fresh/existing database role/ownership/grant/Vault transition evidence and is a foundation-certificate dependency.

Results

pnpm exec prettier --check on every authorized Markdown file: PASS.
Markdown link and whitespace checker on all seven authorized Markdown files: PASS.
pnpm exec tsc --noEmit -p docs/native-kanban-sot/tsconfig.json: PASS (frozen strict contracts unchanged).
Codex code review iterated through role inheritability, hash-complete migration ordering, all reachable runtime DDL entrypoints, installer DSN persistence, and platform-bootstrap ownership; each finding was incorporated into the final frozen contract/DAG. The last security review found no new KBN-101 vulnerability; its sole low finding is the pre-existing unstaged Mosaic session-lock metadata, which is excluded from this commit.
Commit: 82ce3252df38a687c50485f8d048b53ca8db5989 (docs(#771): freeze database runtime role split).
Pre-push queue guard: ci-queue-wait.sh --purpose push -B main returned state=unknown without failure. The push hook ran repository pnpm typecheck, pnpm lint, and pnpm format:check: PASS.
Pushed branch docs/771-kbn101-db-role-split at the exact commit above; no PR was opened, merged, or closed. web1:mosaic-100 received the handoff with head, decisions, DAG, and validation.
Awaiting independent security/Ultron review.

2026-07-15 — rc.6 exact-head remediation session

Objective / correction: Replace the prior planning-author handoff and close every finding in the independent exact-head report against da742ca2da4a2ff466916c818fe275c4f7ffd384. The report is a verbatim durable copy of the task-supplied review artifact; scope remains documentation-only, .mosaic is excluded, and source/config/compose/CI/deployment/secrets/migrations remain untouched.
Finding 1 — closed DDL control plane: rc.6 names mosaic-db-migrator as the sole application/CI/test PostgreSQL DDL runner, requires DATABASE_MIGRATION_URL before connection/DDL, inventories runMigrations, Drizzle config/scripts, db:push, storage CLI, adapter/Gateway startup, fleet-backlog, extension probes/bootstrap, direct federated integration DDL, CI, and future scripts, and specifies route/deny/test disposition for each. Tests use runner-prepared disposable PostgreSQL or invoke that runner; db:push is local-disposable-only and rejects production-like URLs.
Finding 2 — deployable TLS: rc.6 freezes distinct runtime/migrator URL and CA/server leaf Vault/compose/Swarm secret identifiers, 0400 key and 0600 URL/cert/CA mount requirements, actual compose/Swarm service-DNS SANs, PostgreSQL TLS settings, legacy-client drain/termination plus hostssl enforcement, verified-TLS readiness ordering, fresh/existing transition, CA-overlap rotation/TLS-only rollback, and standalone plus federated/Swarm positive and missing/wrong CA/SAN/downgrade negatives. PGlite is explicitly non-PostgreSQL evidence.
Finding 3 — manifest/0009: rc.6 defines manifest v1 canonical UTF-8 serialization and raw SQL-byte SHA-256, logical journal order, manifest ownership/grants, exact one-to-one observed hash tuple mapping, non-normative physical insertion order, safe original-0009 conditions, ambiguous-effect fail-closed recovery, and the full required reconciliation/backup test matrix. It preserves shipped 0009 bytes and forbids manual ledger adoption/insertion.
Finding 4 — advisory lock: replaced hashtext with fixed signed-int4-safe (1297044289,1262636593) (MOSA,KBN1), one max:1 runner session, close-on-crash semantics, and contention/crash/readiness/unrelated-key evidence.
Finding 5 — identifiers/search path: selects mosaic, exact pg_catalog,mosaic per pooled connection and SET LOCAL transactions, plans audited public-object/extension/Drizzle relocation, forbids config-derived identifiers, limits bootstrap quoting to server-side %I on fixed allowlist, and requires injection/pool-reset negatives.
Finding 6 — safe DAG: cards 00–07 are inactive prepared capability while owner-runtime remains N-1; KBN-101-08 is the one atomic activation release after platform roles/TLS and compatible code. Mosaic control plane/Jason alone can activate/rollback; no force-on-red, runtime bypass, or temporary compatibility survives the gate. The approved role graph, post-KBN-100 immutable certification, and KBN-105 gate remain unchanged.
Review remediation: Codex review found the legacy plaintext cutover gap, missing URL-secret bindings, historical public migration incompatibility, non-reproducible checkout-byte hashing, CONNECT allowlisting regression, and undocumented direct-DDL operator instructions. rc.6 now requires drain/scale-to-zero, residual non-TLS session termination, hostssl with no host rule, zero plaintext-session proof, TLS-only post-enforcement rollback, distinct named runtime/migrator secret consumers, canonical Git-blob/LF manifest bytes, a runner-only owner-controlled legacy-public bootstrap followed by mosaic relocation, explicit CONNECT/TEMP revocation, and KBN-101-07 replacement of direct-DDL documentation. It also required the durable exact-head report link above. Pre-existing .mosaic runtime state remains excluded.
Validation: Prettier on all changed Markdown, repository Markdown link/whitespace check, and strict native-kanban contract TypeScript passed before final staging; the final staged diff excludes .mosaic. No source-code TDD applies because this is contract-only remediation.

2026-07-15 — rc.7 residual remediation session

Objective / correction: Close every residual in the independent exact-head rc.6 re-review at /home/hermes/agent-work/reviews/771-kbn101-contract-rereview-45ba3d6.md for 45ba3d6ad4d5383f457a303c05bc816144cfa48a, without changing source, compose, CI, deployment, migration, or secret artifacts. Only the existing authorized planning/documentation paths are eligible; pre-existing .mosaic state remains excluded.
Source-backed scope confirmed: the active federated-pgvector.integration.test.ts executes CREATE TEMP TABLE; tracked docker/init-db.sql and infra/pg-init/01-extensions.sql both create vector; migrate-tier.ts advertises raw CREATE EXTENSION; and tools/federation-harness/docker-compose.two-gateways.yml is current plaintext two-PostgreSQL/two-Gateway topology. Current schema.ts has 36 default-schema pgTable declarations, 6 default pgEnum declarations, and an unqualified vector custom type; historical migrations contain public references.
Plan: (1) make the finite DDL/static-bypass inventory and DATABASE_URL-only denial matrix exact, including the runner-prepared persistent pgvector fixture and migrated two-gateway harness; (2) freeze executable public-to-mosaic and mosaic_extensions transition, Drizzle ownership, object-catalog classes/order, eligibility and rollback tests; (3) bind repository/control-plane ownership, UID/GID validation, exact artifact/mount rules, and both gateway TLS topology; (4) correct PRD acceptance mapping and cross-document rc.7 status; then run formatting, link/contract, source-path, diff, review, commit, queue guard, and push.
Independent review closure: initial Codex review found Gateway-key consumer wording, CLAUDE.md omission, final schema-owner set, and placeholder SANs; all are now explicit. Re-review found the legacy 0001 vector-type resolution problem and docs/federation/SETUP.md raw-DDL instruction; the legacy runner now uses only its fixed non-writable pg_catalog,public,mosaic_extensions history path, while runtime remains pg_catalog,mosaic, and the federation setup path is assigned to KBN-101-07/static inventory. Security review final verdict: no confident vulnerability. The review also repeated the pre-existing tracked .mosaic session-state concern; it remains deliberately unstaged/excluded by this task.
Completion evidence: changed Markdown is Prettier-formatted; local links and strict native-kanban TypeScript passed; source-path inventory confirmed all current referenced paths (the new apps/gateway/Dockerfile is explicitly a planned KBN-101-05 artifact); diff check and authorized-doc allowlist passed. No source-code TDD applies to this documentation-only remediation.

2026-07-15 — rc.8 exact residual remediation session

Objective / correction: Close all three HIGH findings in the independent rc.7 exact-head re-review at /home/hermes/agent-work/reviews/771-kbn101-contract-rereview2-0778eba.md for 0778eba2db3c2dfbaca3af352b12ba0389d3552b. Scope remains documentation-only: no source, config, Compose, CI, deployment, secret, or migration artifact changed; pre-existing .mosaic/orchestrator state remains excluded.
Finite authority closure: KBN-101-06 now classifies exact current source/scripts/package bins, operator docs, and deploy manifests. packages/db/src/index.ts has explicit removal/compile-import negative ownership; docs/fleet/backlog-conventions.md and docs/PERFORMANCE.md now remove first-use/direct-Drizzle/Gateway-startup migration instructions and point to sole runner/readiness. Byte-immutable historical SQL, PGlite-only routines, negative-test literals, vendored/generated artifacts, and labeled historical reports are exact-path/category reviewed allowlists; unknown hits fail. The contract explicitly rejects relying on a naive token scan alone.
Executable and exclusive handoff closure: KBN-101-03 exclusively owns the published mosaic-db-migrator bin, packages/db/src/cli.ts, private migrator modules, docker/db-migrator.Dockerfile, exact --run|--verify|--help, environment/argv limits, sanitized exits, and command/order tests. KBN-101-00 exclusively owns infra/pg-bootstrap/roles.sql, infra/pg-bootstrap/extensions.sql, infra/pg-bootstrap/README.md, and bootstrap tests. KBN-101-05 exclusively owns tools/db/render-postgres-secrets.ts, its tests, and deployment declarations, consuming the versioned bootstrap interface without overlap.
pgvector owner closure: mosaic_extension_owner is dedicated NOLOGIN, available only to the external bootstrap actor during bootstrap; fresh vector/member ownership remains there. The contract records PostgreSQL's unsupported extension-owner transfer and forbids catalog mutation, ownership adoption, and DROP CASCADE. Approved-owner existing extensions use verified ALTER EXTENSION ... SET SCHEMA; legacy runtime-owned extensions fail closed to a controlled backup/shadow/runner/copy-evidence/quiesce/final-delta/atomic-switch/read-only-rollback migration. It requires pg_extension.extowner, member/schema/version, and runtime/migrator/schema-owner ALTER/DROP/member-update denial tests across clean, approved-owner, legacy shadow, partial/resume/rollback, and N-1.
Cross-document state: PRD, KBN contract, shared contract, task decomposition, index, sitemap, current operator docs, and this scratchpad are rc.8-consistent. The only intended next action is a fresh independent exact-head re-review after validation/push.
Validation / review: Prettier passed for all nine changed Markdown documents; local links passed (9 documents); pnpm exec tsc --noEmit -p docs/native-kanban-sot/tsconfig.json passed; source-path inventory passed (20 paths: 8 current, 12 explicitly planned); finite-authority requirement checklist and git diff --check passed. Manual documentation/security review checked the three requested paths, private-only runner boundary/exit contract, non-overlapping 00/03/05 ownership, extension-owner denial and shadow path, and .mosaic exclusion. No source-code TDD applies because this is contract-only remediation.
Delivery evidence: committed 1423c2ad02b5471eab006fb4c878808e5b29c387 as docs(#771): close role split rc.8 residuals. Push queue guard returned state=unknown without error; push hook ran repository pnpm typecheck, pnpm lint, and pnpm format:check, all PASS; branch push succeeded. This final evidence append is committed next, then the exact remote head is verified. The only intended next action is a fresh independent exact-head re-review.

2026-07-15 — rc.9 final residual remediation session

Objective / correction: Close the three findings in /home/hermes/agent-work/reviews/771-kbn101-contract-rereview3-9cf5d2f.md against exact head 9cf5d2f6641b14082dc3294e2a84d1fb4ccc019d: move mosaic_extensions schema ownership to mosaic_extension_owner; replace all broad/conflicting KBN-101 card ownership with a complete disjoint exact-path/test manifest; and classify the current architecture-plan db:migrate instruction with pinned scanner mechanics. Scope remains documentation-only; no source/config/Compose/CI/deployment/secret/migration artifact and no .mosaic path may be modified.
Plan: inspect current tracked source topology to name only existing paths; update the normative contract first and synchronize PRD/shared/task/index/sitemap/version language; run Prettier, changed-doc links, strict contract TypeScript, source/path and manifest-overlap checks, diff allowlist, independent documentation/security review; then stage docs only, commit, queue-guard, push, and verify exact remote SHA. No source-code TDD applies because this is contract-only remediation.
Closure implemented: rc.9 makes mosaic_extension_owner create and own mosaic_extensions, vector, and members; the external bootstrap actor alone temporarily SET ROLEs for fresh/approved-owner work, while schema owner has only USAGE for legacy type resolution and never temporary CREATE. Catalog/default-ACL plus direct DDL/member denials now cover runtime, migrator, and schema owner through fresh, relocation, shadow/resume, and rollback evidence.
Delivery decomposition: Replaced broad ownership with complete disjoint 00–09 manifests, exact tests/evidence, producer-before-consumer edges, and an explicit no-intermediate-deploy N-1 activation statement. packages/storage/src/{cli,migrate-tier}.ts belongs only to -02; the current tracked init artifacts are retired by -02 as direct-DLL closure; -03 owns all runner/index/migrate/config/schema assets and exact compiled-bin/image mapping; -07 owns docs only; -08/-09 own evidence only.
Classifier closure: -06 has exact scanner/inventory/matrix paths, canonical inventory fields, classes/dispositions, fixed token/rule set, exact allowlist categories/restrictions, and self-test requirements for unknown, duplicate-owner, ownerless, missing-path, and historical masking cases. The architecture plan now marks direct db:migrate superseded and uses mosaic-db-migrator --run.
Validation: Prettier check, strict native-kanban contract TypeScript, changed-document local-link resolution, git diff --check, and an automated manifest-overlap/owner/current-source-path check passed. Targeted documentation/security review verified role ownership/default privileges/search path/preflight/legacy/shadow/rollback consistency, disjoint manifests/DAG/activation, scanner mechanics, exact bin/entrypoint, and no .mosaic staging intent. No source-code TDD applies because this is documentation-only remediation.
Next: stage documentation only, commit, queue-guard, push, verify exact remote SHA, then wait for fresh exact-head review.
Delivery evidence: committed 8cbad2bcd9bc7507052f74f35670ef7c8e39e44e as docs(#771): close role split rc.9 residuals; ci-queue-wait.sh --purpose push -B main returned state=unknown without error; push-hook pnpm typecheck, pnpm lint, and pnpm format:check all passed; push succeeded and origin/docs/771-kbn101-db-role-split resolved to that exact SHA. Pre-existing .mosaic/orchestrator/{mission.json,session.lock} remains modified but intentionally unstaged/excluded. The only next action is a fresh independent exact-head re-review.

2026-07-15 — rc.10 Ultron NO-GO remediation intake

Objective / scope: Close only HIGH-1 and HIGH-2 in /home/hermes/agent-work/reviews/771-kbn101-ultron-11f09a1.md against exact head 11f09a15e43e72afda6a0374668996b4bda9e536. Documentation only: preserve approved content; no source/config/Compose/CI/deploy/secret/migration edits and no .mosaic staging.
Plan: (1) replace the impossible NOLOGIN NOSUPERUSER extension owner with the exact non-login, zero-member NOLOGIN SUPERUSER external-control exception and document the non-delegable superuser residual; (2) require the audited external superuser session to SET ROLE/RESET ROLE for fresh, approved-owner, and shadow extension work, then prove catalog ownership and all service-role denial; (3) assign the active migrate-tier guide exclusively to KBN-101-07, add its active secure route to the -06 inventory/matrix/scanner schema, and freeze --target-url-file /run/secrets/mosaic_migrate_target_url plus pre-migrated target/dedicated non-DDL importer requirements; (4) synchronize PRD/shared/tasks/index/sitemap/guide and rc.10 status; (5) validate formatting, links, contracts, source paths, finite operator inventory, diff, review, commit, queue guard, push, and exact remote SHA.
Target-image evidence before edits: local pgvector/pgvector:pg17 control file reports default_version = '0.8.2', relocatable = true, and no trusted/superuser override (untrusted PostgreSQL extension). An isolated PostgreSQL 17 container proved a NOLOGIN SUPERUSER mosaic_extension_owner can create mosaic_extensions and vector under external-superuser SET ROLE, returns to the external session after RESET ROLE, has rolcanlogin=false, rolsuper=true, zero role members, exact extension/schema and owner-bearing-member ownership, and denies SET ROLE, ALTER EXTENSION, DROP EXTENSION, and schema ownership changes to runtime, migrator, schema owner, and data importer. Ownerless PostgreSQL catalog member classes (pg_am, pg_cast) were intentionally not misrepresented as ownable members.
TDD decision: skipped as not applicable: this is a documentation-only contract remediation. Future KBN-101-00/-02/-06 tests are specified as the situational evidence; no source/test artifact is permitted in this task.
Final scope correction: Per control-plane direction, remediation remains bounded to the two Ultron findings. The active guide is explicitly a non-operative KBN-101 contract until its owned implementation/activation lands; no additional design, source, CI, deployment, secret, or test artifact was added.
Validation evidence: target-image/container role proof PASS (pgvector 0.8.2, relocatable=true, trusted absent/untrusted; external-superuser SET ROLE/RESET ROLE; exact extension/schema/owner-bearing-member ownership; zero membership and service-role denials). Changed-doc Prettier, strict native-kanban contract TypeScript, local-link resolver (8 docs), finite operator-doc inventory (one active KBN-101-07 route with no credential argv in executable blocks), source-path check (6 current paths), and git diff --check PASS. Pre-existing .mosaic/orchestrator/{mission.json,session.lock} remains intentionally excluded.

27 KiB Raw Blame History Unescape Escape