stack/docs/federation/SETUP.md

Federated Tier Setup Guide

What is the federated tier?

The federated tier is designed for multi-user and multi-host deployments. It consists of PostgreSQL 17 with pgvector extension (for embeddings and RAG), Valkey for distributed task queueing and caching, and a shared configuration across multiple Mosaic gateway instances. Use this tier when running Mosaic in production or when scaling beyond a single-host deployment.

Prerequisites

• Docker and Docker Compose installed
• Ports 5433 (PostgreSQL) and 6380 (Valkey) available on your host (or adjust environment variables)
• At least 2 GB free disk space for data volumes

Start the federated stack

Run the federated overlay:

docker compose -f docker-compose.federated.yml --profile federated up -d

This starts PostgreSQL 17 with pgvector and Valkey 8. The pgvector extension is created automatically on first boot.

Verify the services are running:

docker compose -f docker-compose.federated.yml ps

Expected output shows postgres-federated and valkey-federated both healthy.

Configure mosaic for federated tier

Create or update your mosaic.config.json:

{
  "tier": "federated",
  "database": "postgresql://mosaic:mosaic@localhost:5433/mosaic",
  "queue": "redis://localhost:6380"
}

If you're using environment variables instead:

export DATABASE_URL="postgresql://mosaic:mosaic@localhost:5433/mosaic"
export REDIS_URL="redis://localhost:6380"

Verify health

Run the health check:

mosaic gateway doctor

Expected output (green):

Tier: federated  Config: mosaic.config.json
  ✓ postgres         localhost:5433       (42ms)
  ✓ valkey           localhost:6380       (8ms)
  ✓ pgvector         (embedded)           (15ms)

For JSON output (useful in CI/automation):

mosaic gateway doctor --json

Troubleshooting

Port conflicts

Symptom: bind: address already in use

Fix: Stop the base dev stack first:

docker compose down
docker compose -f docker-compose.federated.yml --profile federated up -d

Or change the host port with an environment variable:

PG_FEDERATED_HOST_PORT=5434 VALKEY_FEDERATED_HOST_PORT=6381 \
  docker compose -f docker-compose.federated.yml --profile federated up -d

pgvector extension error

Symptom: ERROR: could not open extension control file

Fix: pgvector is created at first boot. Check logs:

docker compose -f docker-compose.federated.yml logs postgres-federated | grep -i vector

If missing, exec into the container and create it manually:

docker exec <postgres-federated-id> psql -U mosaic -d mosaic -c "CREATE EXTENSION vector;"

Valkey connection refused

Symptom: Error: connect ECONNREFUSED 127.0.0.1:6380

Fix: Check service health:

docker compose -f docker-compose.federated.yml logs valkey-federated

If Valkey is running, verify your firewall allows 6380. On macOS, Docker Desktop may require binding to host.docker.internal instead of localhost.

Key rotation (deferred)

Federation peer private keys (federation_peers.client_key_pem) are sealed at rest using AES-256-GCM with a key derived from BETTER_AUTH_SECRET via SHA-256. If BETTER_AUTH_SECRET is rotated, all sealed client_key_pem values in the database become unreadable and must be re-sealed with the new key before rotation completes.

The full key rotation procedure (decrypt all rows with old key, re-encrypt with new key, atomically swap the secret) is out of scope for M2. Operators must not rotate BETTER_AUTH_SECRET without a migration plan for all sealed federation peer keys.

OID Assignments — Mosaic Internal OID Arc

Mosaic uses the private enterprise arc 1.3.6.1.4.1.99999 for custom X.509 certificate extensions in federation grant certificates.

IMPORTANT: This is a development/internal OID arc. Before deploying to a production environment accessible by external parties, register a proper IANA Private Enterprise Number (PEN) at https://pen.iana.org/pen/PenApplication.page and update these assignments accordingly.

Assigned OIDs

OID	Symbolic name	Description
1.3.6.1.4.1.99999.1	mosaic.federation.grantId	UUID of the federation_grants row authorising this cert
1.3.6.1.4.1.99999.2	mosaic.federation.subjectUserId	UUID of the local user on whose behalf the cert is issued

Encoding

Each extension value is DER-encoded as an ASN.1 UTF8String:

Tag    0x0C        (UTF8String)
Length 0x24        (36 decimal — fixed length of a UUID string)
Value  <36 ASCII bytes of the UUID>

The step-ca X.509 template at infra/step-ca/templates/federation.tpl produces this encoding via the Go template expression:

{{ printf "\x0c\x24%s" .Token.mosaic_grant_id | b64enc }}

The resulting base64 value is passed as the value field of the extension object in the template JSON.

CA Environment Variables

The CaService (apps/gateway/src/federation/ca.service.ts) requires the following environment variables at gateway startup:

Variable	Required	Description
STEP_CA_URL	Yes	Base URL of the step-ca instance, e.g. https://step-ca:9000
STEP_CA_PROVISIONER_PASSWORD	Yes	JWK provisioner password for the mosaic-fed provisioner
STEP_CA_PROVISIONER_KEY_JSON	Yes	JSON-encoded JWK (public + private) for the mosaic-fed provisioner
STEP_CA_ROOT_CERT_PATH	Yes	Absolute path to the step-ca root CA certificate PEM file

Set these variables in your environment or secret manager before starting the gateway. In the federated Docker Compose stack they are expected to be injected via Docker secrets and environment variable overrides.

Fail-loud contract

The CA service (and the X.509 template) are designed to fail loudly if the custom OIDs cannot be embedded:

• The template produces a malformed extension value (zero-length UTF8String body) when the JWT claims mosaic_grant_id or mosaic_subject_user_id are absent. step-ca rejects the CSR rather than issuing a cert without the OIDs.
• CaService.issueCert() throws a CaServiceError on every error path with a human-readable remediation string. It never silently returns a cert that may be missing the required extensions.