mosaicstack/stack
1
0
Fork 0
Code Issues 58 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

docs(#771): bind tier importer to verified target

Browse Source
This commit is contained in:
Hermes Agent
2026-07-15 03:58:02 -05:00
parent f60144eb3e
commit 6227f076c8
18 changed files with 260 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/guides/admin-guide.md
View File

@@ -223,10 +223,10 @@ external clients. Authentication requires a valid BetterAuth session (cookie or
### Required
| Variable | Description |
| -------------------- | ----------------------------------------------------------------------------------------- |
| `BETTER_AUTH_SECRET` | Secret key for BetterAuth session signing. Must be set or gateway will not start. |
| `DATABASE_URL` | PostgreSQL connection string. Default: `postgresql://mosaic:mosaic@localhost:5433/mosaic` |
| Variable | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------- |
| `BETTER_AUTH_SECRET` | Secret key for BetterAuth session signing. Must be set or gateway will not start. |
| `DATABASE_URL` | Runtime-only PostgreSQL connection injected from the dedicated deployment secret; no default or inline DSN. |
### Gateway

58
docs/guides/deployment.md
View File

@@ -61,9 +61,13 @@ pnpm install
### 4. Initialize the database
```bash
pnpm --filter @mosaicstack/db db:migrate
mosaic-db-migrator --run
mosaic-db-migrator --verify
```
This is the sole PostgreSQL DDL/readiness control plane. It receives migration-only credentials
through deployment secret mounts and accepts no URL, SQL, schema, or role argv.
### 5. Build all packages
```bash
@@ -118,34 +122,25 @@ sudo apt-get install -y postgresql-17 postgresql-17-pgvector
# Follow https://valkey.io/download/ for your distribution
```
### Step 2 — Create the database
### Step 2 — Provision the database
```sql
-- Run as the postgres superuser
CREATE USER mosaic WITH PASSWORD 'change-me';
CREATE DATABASE mosaic OWNER mosaic;
\c mosaic
CREATE EXTENSION IF NOT EXISTS vector;
```
Do not run ad hoc SQL, create application roles, or create extensions from this guide. The
external reviewed bootstrap artifact provisions roles/pgvector, then the migration Job runs
`mosaic-db-migrator --run` and `mosaic-db-migrator --verify` over verified TLS. This guide does
not authorize a plaintext, owner-runtime, or manual-extension bootstrap.
### Step 3 — Clone and configure
```bash
git clone <repo-url> /opt/mosaic
cd /opt/mosaic
cp .env.example .env
```
Edit `/opt/mosaic/.env`. Required fields:
```dotenv
DATABASE_URL=postgresql://mosaic:<password>@localhost:5432/mosaic
VALKEY_URL=redis://localhost:6379
BETTER_AUTH_SECRET=<openssl rand -base64 32>
BETTER_AUTH_URL=https://your-domain.example.com
GATEWAY_CORS_ORIGIN=https://your-domain.example.com
NEXT_PUBLIC_GATEWAY_URL=https://your-domain.example.com
```
Production deployment does **not** read a committed or local `.env` file. Use the reviewed
secret renderer/deployment control plane: it injects runtime secrets only at process exec,
migration secrets only to the one-shot runner, and non-secret endpoint configuration separately.
Validate required values at startup; never write a DSN or BetterAuth secret into this guide or a
production filesystem `.env`.
### Step 4 — Install dependencies and build
@@ -157,9 +152,12 @@ pnpm build
### Step 5 — Run database migrations
```bash
pnpm --filter @mosaicstack/db db:migrate
mosaic-db-migrator --run
mosaic-db-migrator --verify
```
The one-shot migrator alone receives the migration secret; Gateway/runtime never receives it.
### Step 6 — Start the gateway
```bash
@@ -340,20 +338,12 @@ MaxRetentionSec=30day
`BETTER_AUTH_SECRET` is missing or empty. Set it in `.env` and restart.
### `DATABASE_URL` connection refused
### Runtime database connection refused
Verify PostgreSQL is running and the port matches. The Docker Compose default is `5433`; bare-metal typically uses `5432`.
```bash
psql "$DATABASE_URL" -c '\conninfo'
```
### pgvector extension missing
```sql
\c mosaic
CREATE EXTENSION IF NOT EXISTS vector;
```
Verify PostgreSQL service/TLS readiness and the deployment-mounted runtime secret reference. Do
not print or test a DSN from a shell. Run the sanitized `mosaic-db-migrator --verify` readiness
procedure; if pgvector or schema verification fails, stop and remediate through the reviewed
bootstrap/runner control plane. Manual extension SQL is unavailable.
### Valkey / Redis connection refused

29
docs/guides/dev-guide.md
View File

@@ -106,7 +106,8 @@ This starts:
Create a `.env` file in the monorepo root:
```env
# Database (matches docker-compose defaults)
# Runtime database connection (local-development placeholder only; never a production secret)
# KBN-101 production-like runtime injects DATABASE_URL from its dedicated secret mount.
DATABASE_URL=postgresql://mosaic:mosaic@localhost:5433/mosaic
# Auth (required — generate a random 32+ char string)
@@ -127,14 +128,16 @@ OLLAMA_MODELS=llama3.2
The gateway loads `.env` from the monorepo root via `dotenv` at startup
(`apps/gateway/src/main.ts`).
### 4. Push the Database Schema
### 4. Prepare the PostgreSQL Schema
```bash
pnpm --filter @mosaicstack/db db:push
mosaic-db-migrator --run
mosaic-db-migrator --verify
```
This applies the Drizzle schema directly to the database (development only; use
migrations in production).
KBN-101 makes this the sole PostgreSQL DDL/readiness sequence. Direct schema push is not an
operator workflow; a disposable local experiment is separately guarded and cannot use a
production-like tier or URL.
### 5. Start the Gateway
@@ -355,12 +358,13 @@ The schema lives in a single file:
The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.
### Development: Push Schema
### Development: Prepare Schema
Apply schema changes directly to the dev database (no migration files created):
Prepare a PostgreSQL development target through the same sole runner:
```bash
pnpm --filter @mosaicstack/db db:push
mosaic-db-migrator --run
mosaic-db-migrator --verify
```
### Generating Migrations
@@ -376,7 +380,8 @@ This creates a new SQL migration file in `packages/db/drizzle/`.
### Running Migrations
```bash
pnpm --filter @mosaicstack/db db:migrate
mosaic-db-migrator --run
mosaic-db-migrator --verify
```
### Drizzle Config
@@ -388,9 +393,9 @@ directory are defined there.
1. Add the table definition to `packages/db/src/schema.ts`.
2. Export it from `packages/db/src/index.ts`.
3. Run `pnpm --filter @mosaicstack/db db:push` (dev) or
`pnpm --filter @mosaicstack/db db:generate && pnpm --filter @mosaicstack/db db:migrate`
(production).
3. Generate the offline artifact with `pnpm --filter @mosaicstack/db db:generate`, then run
`mosaic-db-migrator --run` and `mosaic-db-migrator --verify`. Direct schema push is not a
production-like workflow and the migration runner accepts no URL/SQL/schema/role argv.
---

66
docs/guides/migrate-tier.md
View File

@@ -17,9 +17,10 @@ The PostgreSQL destination schema is prepared separately before data migration:
1. The dedicated DDL runner executes `mosaic-db-migrator --run` using its migration-only
identity.
2. The same runner succeeds at `mosaic-db-migrator --verify` against the exact destination.
3. Only then may `mosaic storage migrate-tier` connect as dedicated non-DDL
`mosaic_data_importer` to copy data.
2. The same runner succeeds at `mosaic-db-migrator --verify` against the exact destination and,
only then, emits a signed target-bound attestation.
3. Only then may `mosaic storage migrate-tier` verify that attestation and connect as dedicated
non-DDL `mosaic_data_importer` to copy data.
The data migration never executes DDL or changes PostgreSQL roles, memberships, schemas,
extensions, extension members, system catalogs, or the Drizzle ledger. Its only target writes are
@@ -32,13 +33,17 @@ PostgreSQL is always pre-migrated and runner-verified, including when the source
- Federated stack is running and healthy (see [Federated Tier Setup](../federation/SETUP.md)).
- An approved source backup exists and the target is identified in the KBN-101 change record.
- The target passed the runner `--run` and `--verify` sequence above.
- The operator has an approved, dedicated importer credential file at
`/run/secrets/mosaic_migrate_target_url`; the **path reference** is non-secret, but its
contents are never printed, copied, committed, or placed on argv.
- KBN-101-02 validates that file before any target connection: it must be a regular,
non-symlink file, owned by the expected importer UID:GID, mode `0600`, readable only by
that importer process, and its value must be redacted from all output.
- The target passed the runner `--run` and `--verify` sequence above; `--verify` is the only
trusted attestation producer after verified TLS, identity, manifest, and schema checks.
- The operator has an approved, dedicated high-entropy importer credential file at
`/run/secrets/mosaic_migrate_target_url` and its paired non-secret attestation artifact at
`/run/mosaic-attestations/migrate-target.v1.json`. Neither file contents are printed, copied,
committed, or placed on argv.
- KBN-101-02 validates both references before any target connection: each is a regular,
non-symlink file with the exact required owner/mode. The importer gets only pinned public
verification key(s) and the attestation; the runner-only Ed25519 private key never reaches
importer/runtime. The URL file is opened, digested, parsed, and connected from the same
in-memory bytes with no reread.
### Source modes
@@ -57,13 +62,15 @@ mosaic-db-migrator --run
mosaic-db-migrator --verify
mosaic storage migrate-tier --to federated \
--target-url-file /run/secrets/mosaic_migrate_target_url \
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
--dry-run
```
The frozen KBN-101-02 option is exactly
`--target-url-file /run/secrets/mosaic_migrate_target_url`. It is a file reference, not a
URL option. The runner and importer use distinct connections: the runner has migration
DDL authority; the importer has only approved data-copy DML authority.
The frozen KBN-101-02 options are exactly
`--target-url-file /run/secrets/mosaic_migrate_target_url` and
`--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json`. They are
non-secret file references, not URL options. The runner and importer use distinct connections:
the runner has migration DDL authority; the importer has only approved data-copy DML authority.
Expected dry-run output is sanitized and contains no connection string, username, host,
database name, SQL, or credential value:
@@ -95,6 +102,7 @@ After a reviewed dry-run, repeat the same verified route with `--yes`:
mosaic-db-migrator --verify
mosaic storage migrate-tier --to federated \
--target-url-file /run/secrets/mosaic_migrate_target_url \
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
--yes
```
@@ -103,8 +111,12 @@ verification, target-file validation, role checks, redaction, or the no-DDL boun
The command will:
1. Verify the runner-prepared target schema before opening the importer target connection.
2. Validate the credential-file reference and use the dedicated importer identity.
1. Before target connection, validate both files, JCS/Ed25519 signature and pinned key ID,
expiry, nonce replay, secret version/digest, canonical TLS host/port/database, CA/SPKI,
expected importer role, and manifest/schema bindings.
2. Connect only from the once-opened in-memory URL bytes, then before transaction/DML compare
PostgreSQL system identifier, database OID, `current_user`, CA/SPKI, and manifest/schema
fingerprints to the attestation; mismatch closes with zero DML/DDL.
3. Copy data in dependency order without DDL.
4. Report sanitized row counts and warnings.
@@ -116,14 +128,26 @@ DDL attempt:
- Raw `--target-url` or any URL/credential value on argv.
- `DATABASE_URL` as a target or source fallback.
- A runtime owner, migrator, schema owner, or extension owner used as the data importer.
- Missing, non-regular, symlinked, wrong-owner, or wrong-mode target credential file.
- Missing, non-regular, symlinked, wrong-owner, wrong-mode, substituted, or changed-after-check
credential file.
- Missing, stale, replayed, tampered, wrong-key, revoked-key, expired, malformed, or wrong-binding
attestation file.
- Wrong host, database, CA/SPKI, PostgreSQL system identifier, database OID, importer role,
manifest, schema, or credential secret-version/digest.
- Any mode other than the runner-prepared/verified PostgreSQL destination route.
- Any direct DDL attempt by the migration command or its target connection.
KBN-101-02 SHALL add `packages/storage/src/migrate-tier.spec.ts::secureTargetRoute` and
its integration counterpart. They prove no target connection is opened on each interface
rejection. The finite operator-document scanner rejects an unowned route or a guide that
reintroduces a credential-bearing command.
The signed attestation has canonical UTF-8/JCS bytes and detached Ed25519 signature. Its
credential-free v1 payload binds format/version/key ID; issued/expiry/nonce; exact credential
secret version and SHA-256; canonical TLS host/port/database; CA/SPKI; PostgreSQL system
identifier/database OID; importer role; manifest/schema; and producer invocation/build/image/
correlation. Artifacts are atomically written/renamed `0444`; the replay cache survives until
expiry; public-key overlap/revocation and credential rotation force a fresh runner verification.
KBN-101-02 SHALL add `packages/storage/src/migrate-tier.spec.ts::secureTargetRouteAndAttestation`
and its integration counterpart. They prove zero target connection for pre-connect failures and
zero DML/DDL for post-connect identity mismatch. The finite operator-document scanner rejects an
unowned route or a guide that reintroduces a credential-bearing command.
## What gets migrated

9
docs/guides/user-guide.md
View File

@@ -522,8 +522,13 @@ mosaic storage export --bucket agent-artifacts --output ./artifacts.tar.gz
# Import data into storage
mosaic storage import --bucket agent-artifacts --input ./artifacts.tar.gz
# Migrate data between tiers
mosaic storage migrate --from hot --to cold --older-than 30d
# Schema migration (compatibility wrapper only; not a tier data copy)
mosaic storage migrate --run # delegates only to mosaic-db-migrator
# Tier data copy is unavailable until KBN-101 activation. Its only produced interface is:
# mosaic storage migrate-tier --target-url-file /run/secrets/mosaic_migrate_target_url \
# --target-attestation-file /run/mosaic-attestations/migrate-target.v1.json
# Never use a legacy --from/--to storage-migrate command or pass a credential on argv.
```
---