stack/docs/federation/SETUP.md

# Federated Tier Setup Guide

## What is the federated tier?

The federated tier is designed for multi-user and multi-host deployments. It consists of PostgreSQL 17 with pgvector extension (for embeddings and RAG), Valkey for distributed task queueing and caching, and a shared configuration across multiple Mosaic gateway instances. Use this tier when running Mosaic in production or when scaling beyond a single-host deployment.

## Prerequisites

- Docker and Docker Compose installed
- Ports 5433 (PostgreSQL) and 6380 (Valkey) available on your host (or adjust environment variables)
- At least 2 GB free disk space for data volumes

## Start the federated stack

Run the federated overlay:

```bash
docker compose -f docker-compose.federated.yml --profile federated up -d
```

This starts PostgreSQL 17 with pgvector and Valkey 8. The pgvector extension is created automatically on first boot.

Verify the services are running:

```bash
docker compose -f docker-compose.federated.yml ps
```

Expected output shows `postgres-federated` and `valkey-federated` both healthy.

## Configure mosaic for federated tier

Create or update your `mosaic.config.json`:

```json
{
  "tier": "federated",
  "database": "postgresql://mosaic:mosaic@localhost:5433/mosaic",
  "queue": "redis://localhost:6380"
}
```

If you're using environment variables instead:

```bash
export DATABASE_URL="postgresql://mosaic:mosaic@localhost:5433/mosaic"
export REDIS_URL="redis://localhost:6380"
```

## Verify health

Run the health check:

```bash
mosaic gateway doctor
```

Expected output (green):

```
Tier: federated  Config: mosaic.config.json
  ✓ postgres         localhost:5433       (42ms)
  ✓ valkey           localhost:6380       (8ms)
  ✓ pgvector         (embedded)           (15ms)
```

For JSON output (useful in CI/automation):

```bash
mosaic gateway doctor --json
```

## Troubleshooting

### Port conflicts

**Symptom:** `bind: address already in use`

**Fix:** Stop the base dev stack first:

```bash
docker compose down
docker compose -f docker-compose.federated.yml --profile federated up -d
```

Or change the host port with an environment variable:

```bash
PG_FEDERATED_HOST_PORT=5434 VALKEY_FEDERATED_HOST_PORT=6381 \
  docker compose -f docker-compose.federated.yml --profile federated up -d
```

### pgvector extension error

**Symptom:** `ERROR: could not open extension control file`

**Fix:** pgvector is created at first boot. Check logs:

```bash
docker compose -f docker-compose.federated.yml logs postgres-federated | grep -i vector
```

If missing, exec into the container and create it manually:

```bash
docker exec <postgres-federated-id> psql -U mosaic -d mosaic -c "CREATE EXTENSION vector;"
```

### Valkey connection refused

**Symptom:** `Error: connect ECONNREFUSED 127.0.0.1:6380`

**Fix:** Check service health:

```bash
docker compose -f docker-compose.federated.yml logs valkey-federated
```

If Valkey is running, verify your firewall allows 6380. On macOS, Docker Desktop may require binding to `host.docker.internal` instead of `localhost`.

## Key rotation (deferred)

Federation peer private keys (`federation_peers.client_key_pem`) are sealed at rest using AES-256-GCM with a key derived from `BETTER_AUTH_SECRET` via SHA-256. If `BETTER_AUTH_SECRET` is rotated, all sealed `client_key_pem` values in the database become unreadable and must be re-sealed with the new key before rotation completes.

The full key rotation procedure (decrypt all rows with old key, re-encrypt with new key, atomically swap the secret) is out of scope for M2. Operators must not rotate `BETTER_AUTH_SECRET` without a migration plan for all sealed federation peer keys.

## OID Assignments — Mosaic Internal OID Arc

Mosaic uses the private enterprise arc `1.3.6.1.4.1.99999` for custom X.509
certificate extensions in federation grant certificates.

**IMPORTANT:** This is a development/internal OID arc. Before deploying to a
production environment accessible by external parties, register a proper IANA
Private Enterprise Number (PEN) at <https://pen.iana.org/pen/PenApplication.page>
and update these assignments accordingly.

### Assigned OIDs

| OID                   | Symbolic name                     | Description                                               |
| --------------------- | --------------------------------- | --------------------------------------------------------- |
| `1.3.6.1.4.1.99999.1` | `mosaic.federation.grantId`       | UUID of the `federation_grants` row authorising this cert |
| `1.3.6.1.4.1.99999.2` | `mosaic.federation.subjectUserId` | UUID of the local user on whose behalf the cert is issued |

### Encoding

Each extension value is DER-encoded as an ASN.1 **UTF8String**:

```
Tag    0x0C        (UTF8String)
Length 0x24        (36 decimal — fixed length of a UUID string)
Value  <36 ASCII bytes of the UUID>
```

The step-ca X.509 template at `infra/step-ca/templates/federation.tpl`
produces this encoding via the Go template expression:

```
{{ printf "\x0c\x24%s" .Token.mosaic_grant_id | b64enc }}
```

The resulting base64 value is passed as the `value` field of the extension
object in the template JSON.

### CA Environment Variables

The `CaService` (`apps/gateway/src/federation/ca.service.ts`) requires the
following environment variables at gateway startup:

| Variable                       | Required | Description                                                          |
| ------------------------------ | -------- | -------------------------------------------------------------------- |
| `STEP_CA_URL`                  | Yes      | Base URL of the step-ca instance, e.g. `https://step-ca:9000`        |
| `STEP_CA_PROVISIONER_PASSWORD` | Yes      | JWK provisioner password for the `mosaic-fed` provisioner            |
| `STEP_CA_PROVISIONER_KEY_JSON` | Yes      | JSON-encoded JWK (public + private) for the `mosaic-fed` provisioner |
| `STEP_CA_ROOT_CERT_PATH`       | Yes      | Absolute path to the step-ca root CA certificate PEM file            |

Set these variables in your environment or secret manager before starting
the gateway. In the federated Docker Compose stack they are expected to be
injected via Docker secrets and environment variable overrides.

### Fail-loud contract

The CA service (and the X.509 template) are designed to fail loudly if the
custom OIDs cannot be embedded:

- The template produces a malformed extension value (zero-length UTF8String
  body) when the JWT claims `mosaic_grant_id` or `mosaic_subject_user_id` are
  absent. step-ca rejects the CSR rather than issuing a cert without the OIDs.
- `CaService.issueCert()` throws a `CaServiceError` on every error path with
  a human-readable `remediation` string. It never silently returns a cert that
  may be missing the required extensions.