stack/docs/federation/SETUP.md

# Federated Tier Setup Guide

## What is the federated tier?

The federated tier is designed for multi-user and multi-host deployments. It consists of PostgreSQL 17 with pgvector extension (for embeddings and RAG), Valkey for distributed task queueing and caching, and a shared configuration across multiple Mosaic gateway instances. Use this tier when running Mosaic in production or when scaling beyond a single-host deployment.

## Prerequisites

- Docker and Docker Compose installed
- Ports 5433 (PostgreSQL) and 6380 (Valkey) available on your host (or adjust environment variables)
- At least 2 GB free disk space for data volumes

## Start the federated stack

Run the federated overlay:

```bash
docker compose -f docker-compose.federated.yml --profile federated up -d
```

This starts PostgreSQL 17 with pgvector and Valkey 8. The pgvector extension is created automatically on first boot.

Verify the services are running:

```bash
docker compose -f docker-compose.federated.yml ps
```

Expected output shows `postgres-federated` and `valkey-federated` both healthy.

## Configure mosaic for federated tier

Create or update your `mosaic.config.json`:

```json
{
  "tier": "federated",
  "database": "postgresql://mosaic:mosaic@localhost:5433/mosaic",
  "queue": "redis://localhost:6380"
}
```

If you're using environment variables instead:

```bash
export DATABASE_URL="postgresql://mosaic:mosaic@localhost:5433/mosaic"
export REDIS_URL="redis://localhost:6380"
```

## Verify health

Run the health check:

```bash
mosaic gateway doctor
```

Expected output (green):

```
Tier: federated  Config: mosaic.config.json
  ✓ postgres         localhost:5433       (42ms)
  ✓ valkey           localhost:6380       (8ms)
  ✓ pgvector         (embedded)           (15ms)
```

For JSON output (useful in CI/automation):

```bash
mosaic gateway doctor --json
```

## Step 2: Step-CA Bootstrap

Step-CA is a certificate authority that issues X.509 certificates for federation peers. In Mosaic federation, it signs peer certificates with custom OIDs that embed grant and user identities, enforcing authorization at the certificate level.

### Prerequisites for Step-CA

Before starting the CA, you must set up the dev password:

```bash
cp infra/step-ca/dev-password.example infra/step-ca/dev-password
# Edit dev-password and set your CA password (minimum 16 characters)
```

The password is required for the CA to boot and derive the provisioner key used by the gateway.

### Start the Step-CA service

Add the step-ca service to your federated stack:

```bash
docker compose -f docker-compose.federated.yml --profile federated up -d step-ca
```

On first boot, the init script (`infra/step-ca/init.sh`) runs automatically. It:

- Generates the CA root key and certificate in the Docker volume
- Creates the `mosaic-fed` JWK provisioner
- Applies the X.509 template from `infra/step-ca/templates/federation.tpl`

The volume is persistent, so subsequent boots reuse the existing CA keys.

Verify the CA is healthy:

```bash
curl https://localhost:9000/health --cacert /tmp/step-ca-root.crt
```

(If the root cert file doesn't exist yet, see the extraction steps below.)

### Extract credentials for the gateway

The gateway requires two credentials from the running CA:

**1. Provisioner key (for `STEP_CA_PROVISIONER_KEY_JSON`)**

```bash
docker exec $(docker ps -qf name=step-ca) cat /home/step/secrets/mosaic-fed.json > /tmp/step-ca-provisioner.json
```

This JSON file contains the JWK public and private keys for the `mosaic-fed` provisioner. Store it securely and pass its contents to the gateway via the `STEP_CA_PROVISIONER_KEY_JSON` environment variable.

**2. Root certificate (for `STEP_CA_ROOT_CERT_PATH`)**

```bash
docker cp $(docker ps -qf name=step-ca):/home/step/certs/root_ca.crt /tmp/step-ca-root.crt
```

This PEM file is the CA's root certificate, used to verify peer certificates issued by step-ca. Pass its path to the gateway via `STEP_CA_ROOT_CERT_PATH`.

### Custom OID Registry

Federation certificates include custom OIDs in the certificate extension. These encode authorization metadata:

| OID                 | Name                   | Description           |
| ------------------- | ---------------------- | --------------------- |
| 1.3.6.1.4.1.99999.1 | mosaic_grant_id        | Federation grant UUID |
| 1.3.6.1.4.1.99999.2 | mosaic_subject_user_id | Subject user UUID     |

These OIDs are verified by the gateway after the CSR is signed, ensuring the certificate was issued with the correct grant and user context.

### Environment Variables

Configure the gateway with the following environment variables before startup:

| Variable                       | Required | Description                                                                                               |
| ------------------------------ | -------- | --------------------------------------------------------------------------------------------------------- |
| `STEP_CA_URL`                  | Yes      | Base URL of the step-ca instance, e.g. `https://step-ca:9000` (use `https://localhost:9000` in local dev) |
| `STEP_CA_PROVISIONER_KEY_JSON` | Yes      | JSON-encoded JWK from `/home/step/secrets/mosaic-fed.json`                                                |
| `STEP_CA_ROOT_CERT_PATH`       | Yes      | Absolute path to the root CA certificate (e.g. `/tmp/step-ca-root.crt`)                                   |
| `BETTER_AUTH_SECRET`           | Yes      | Secret used to seal peer private keys at rest; already required for M1                                    |

Example environment setup:

```bash
export STEP_CA_URL="https://localhost:9000"
export STEP_CA_PROVISIONER_KEY_JSON="$(cat /tmp/step-ca-provisioner.json)"
export STEP_CA_ROOT_CERT_PATH="/tmp/step-ca-root.crt"
export BETTER_AUTH_SECRET="<your-secret>"
```

## Troubleshooting

### Port conflicts

**Symptom:** `bind: address already in use`

**Fix:** Stop the base dev stack first:

```bash
docker compose down
docker compose -f docker-compose.federated.yml --profile federated up -d
```

Or change the host port with an environment variable:

```bash
PG_FEDERATED_HOST_PORT=5434 VALKEY_FEDERATED_HOST_PORT=6381 \
  docker compose -f docker-compose.federated.yml --profile federated up -d
```

### pgvector extension error

**Symptom:** `ERROR: could not open extension control file`

**Fix:** pgvector is created at first boot. Check logs:

```bash
docker compose -f docker-compose.federated.yml logs postgres-federated | grep -i vector
```

If missing, exec into the container and create it manually:

```bash
docker exec <postgres-federated-id> psql -U mosaic -d mosaic -c "CREATE EXTENSION vector;"
```

### Valkey connection refused

**Symptom:** `Error: connect ECONNREFUSED 127.0.0.1:6380`

**Fix:** Check service health:

```bash
docker compose -f docker-compose.federated.yml logs valkey-federated
```

If Valkey is running, verify your firewall allows 6380. On macOS, Docker Desktop may require binding to `host.docker.internal` instead of `localhost`.

## Key rotation (deferred)

Federation peer private keys (`federation_peers.client_key_pem`) are sealed at rest using AES-256-GCM with a key derived from `BETTER_AUTH_SECRET` via SHA-256. If `BETTER_AUTH_SECRET` is rotated, all sealed `client_key_pem` values in the database become unreadable and must be re-sealed with the new key before rotation completes.

The full key rotation procedure (decrypt all rows with old key, re-encrypt with new key, atomically swap the secret) is out of scope for M2. Operators must not rotate `BETTER_AUTH_SECRET` without a migration plan for all sealed federation peer keys.

## OID Assignments — Mosaic Internal OID Arc

Mosaic uses the private enterprise arc `1.3.6.1.4.1.99999` for custom X.509
certificate extensions in federation grant certificates.

**IMPORTANT:** This is a development/internal OID arc. Before deploying to a
production environment accessible by external parties, register a proper IANA
Private Enterprise Number (PEN) at <https://pen.iana.org/pen/PenApplication.page>
and update these assignments accordingly.

### Assigned OIDs

| OID                   | Symbolic name                     | Description                                               |
| --------------------- | --------------------------------- | --------------------------------------------------------- |
| `1.3.6.1.4.1.99999.1` | `mosaic.federation.grantId`       | UUID of the `federation_grants` row authorising this cert |
| `1.3.6.1.4.1.99999.2` | `mosaic.federation.subjectUserId` | UUID of the local user on whose behalf the cert is issued |

### Encoding

Each extension value is DER-encoded as an ASN.1 **UTF8String**:

```
Tag    0x0C        (UTF8String)
Length 0x24        (36 decimal — fixed length of a UUID string)
Value  <36 ASCII bytes of the UUID>
```

The step-ca X.509 template at `infra/step-ca/templates/federation.tpl`
produces this encoding via the Go template expression:

```
{{ printf "\x0c\x24%s" .Token.mosaic_grant_id | b64enc }}
```

The resulting base64 value is passed as the `value` field of the extension
object in the template JSON.

### CA Environment Variables

The `CaService` (`apps/gateway/src/federation/ca.service.ts`) requires the
following environment variables at gateway startup:

| Variable                       | Required | Description                                                          |
| ------------------------------ | -------- | -------------------------------------------------------------------- |
| `STEP_CA_URL`                  | Yes      | Base URL of the step-ca instance, e.g. `https://step-ca:9000`        |
| `STEP_CA_PROVISIONER_PASSWORD` | Yes      | JWK provisioner password for the `mosaic-fed` provisioner            |
| `STEP_CA_PROVISIONER_KEY_JSON` | Yes      | JSON-encoded JWK (public + private) for the `mosaic-fed` provisioner |
| `STEP_CA_ROOT_CERT_PATH`       | Yes      | Absolute path to the step-ca root CA certificate PEM file            |

Set these variables in your environment or secret manager before starting
the gateway. In the federated Docker Compose stack they are expected to be
injected via Docker secrets and environment variable overrides.

### Fail-loud contract

The CA service (and the X.509 template) are designed to fail loudly if the
custom OIDs cannot be embedded:

- The template produces a malformed extension value (zero-length UTF8String
  body) when the JWT claims `mosaic_grant_id` or `mosaic_subject_user_id` are
  absent. step-ca rejects the CSR rather than issuing a cert without the OIDs.
- `CaService.issueCert()` throws a `CaServiceError` on every error path with
  a human-readable `remediation` string. It never silently returns a cert that
  may be missing the required extensions.