stack/docs/federation/SETUP.md

Federated Tier Setup Guide

What is the federated tier?

The federated tier is designed for multi-user and multi-host deployments. It consists of PostgreSQL 17 with pgvector extension (for embeddings and RAG), Valkey for distributed task queueing and caching, and a shared configuration across multiple Mosaic gateway instances. Use this tier when running Mosaic in production or when scaling beyond a single-host deployment.

Prerequisites

• Docker and Docker Compose installed
• Ports 5433 (PostgreSQL) and 6380 (Valkey) available on your host (or adjust environment variables)
• At least 2 GB free disk space for data volumes

Start the federated stack

Run the federated overlay:

docker compose -f docker-compose.federated.yml --profile federated up -d

This starts PostgreSQL 17 with pgvector and Valkey 8. The pgvector extension is created automatically on first boot.

Verify the services are running:

docker compose -f docker-compose.federated.yml ps

Expected output shows postgres-federated and valkey-federated both healthy.

Configure mosaic for federated tier

Create or update your mosaic.config.json:

{
  "tier": "federated",
  "database": "postgresql://mosaic:mosaic@localhost:5433/mosaic",
  "queue": "redis://localhost:6380"
}

If you're using environment variables instead:

export DATABASE_URL="postgresql://mosaic:mosaic@localhost:5433/mosaic"
export REDIS_URL="redis://localhost:6380"

Verify health

Run the health check:

mosaic gateway doctor

Expected output (green):

Tier: federated  Config: mosaic.config.json
  ✓ postgres         localhost:5433       (42ms)
  ✓ valkey           localhost:6380       (8ms)
  ✓ pgvector         (embedded)           (15ms)

For JSON output (useful in CI/automation):

mosaic gateway doctor --json

Step 2: Step-CA Bootstrap

Step-CA is a certificate authority that issues X.509 certificates for federation peers. In Mosaic federation, it signs peer certificates with custom OIDs that embed grant and user identities, enforcing authorization at the certificate level.

Prerequisites for Step-CA

Before starting the CA, you must set up the dev password:

cp infra/step-ca/dev-password.example infra/step-ca/dev-password
# Edit dev-password and set your CA password (minimum 16 characters)

The password is required for the CA to boot and derive the provisioner key used by the gateway.

Start the Step-CA service

Add the step-ca service to your federated stack:

docker compose -f docker-compose.federated.yml --profile federated up -d step-ca

On first boot, the init script (infra/step-ca/init.sh) runs automatically. It:

• Generates the CA root key and certificate in the Docker volume
• Creates the mosaic-fed JWK provisioner
• Applies the X.509 template from infra/step-ca/templates/federation.tpl

The volume is persistent, so subsequent boots reuse the existing CA keys.

Verify the CA is healthy:

curl https://localhost:9000/health --cacert /tmp/step-ca-root.crt

(If the root cert file doesn't exist yet, see the extraction steps below.)

Extract credentials for the gateway

The gateway requires two credentials from the running CA:

1. Provisioner key (for STEP_CA_PROVISIONER_KEY_JSON)

docker exec $(docker ps -qf name=step-ca) cat /home/step/secrets/mosaic-fed.json > /tmp/step-ca-provisioner.json

This JSON file contains the JWK public and private keys for the mosaic-fed provisioner. Store it securely and pass its contents to the gateway via the STEP_CA_PROVISIONER_KEY_JSON environment variable.

2. Root certificate (for STEP_CA_ROOT_CERT_PATH)

docker cp $(docker ps -qf name=step-ca):/home/step/certs/root_ca.crt /tmp/step-ca-root.crt

This PEM file is the CA's root certificate, used to verify peer certificates issued by step-ca. Pass its path to the gateway via STEP_CA_ROOT_CERT_PATH.

Custom OID Registry

Federation certificates include custom OIDs in the certificate extension. These encode authorization metadata:

OID	Name	Description
1.3.6.1.4.1.99999.1	mosaic_grant_id	Federation grant UUID
1.3.6.1.4.1.99999.2	mosaic_subject_user_id	Subject user UUID

These OIDs are verified by the gateway after the CSR is signed, ensuring the certificate was issued with the correct grant and user context.

Environment Variables

Configure the gateway with the following environment variables before startup:

Variable	Required	Description
STEP_CA_URL	Yes	Base URL of the step-ca instance, e.g. https://step-ca:9000 (use https://localhost:9000 in local dev)
STEP_CA_PROVISIONER_KEY_JSON	Yes	JSON-encoded JWK from /home/step/secrets/mosaic-fed.json
STEP_CA_ROOT_CERT_PATH	Yes	Absolute path to the root CA certificate (e.g. /tmp/step-ca-root.crt)
BETTER_AUTH_SECRET	Yes	Secret used to seal peer private keys at rest; already required for M1

Example environment setup:

export STEP_CA_URL="https://localhost:9000"
export STEP_CA_PROVISIONER_KEY_JSON="$(cat /tmp/step-ca-provisioner.json)"
export STEP_CA_ROOT_CERT_PATH="/tmp/step-ca-root.crt"
export BETTER_AUTH_SECRET="<your-secret>"

Troubleshooting

Port conflicts

Symptom: bind: address already in use

Fix: Stop the base dev stack first:

docker compose down
docker compose -f docker-compose.federated.yml --profile federated up -d

Or change the host port with an environment variable:

PG_FEDERATED_HOST_PORT=5434 VALKEY_FEDERATED_HOST_PORT=6381 \
  docker compose -f docker-compose.federated.yml --profile federated up -d

pgvector extension error

Symptom: ERROR: could not open extension control file

Fix: pgvector is created at first boot. Check logs:

docker compose -f docker-compose.federated.yml logs postgres-federated | grep -i vector

If missing, exec into the container and create it manually:

docker exec <postgres-federated-id> psql -U mosaic -d mosaic -c "CREATE EXTENSION vector;"

Valkey connection refused

Symptom: Error: connect ECONNREFUSED 127.0.0.1:6380

Fix: Check service health:

docker compose -f docker-compose.federated.yml logs valkey-federated

If Valkey is running, verify your firewall allows 6380. On macOS, Docker Desktop may require binding to host.docker.internal instead of localhost.

Key rotation (deferred)

Federation peer private keys (federation_peers.client_key_pem) are sealed at rest using AES-256-GCM with a key derived from BETTER_AUTH_SECRET via SHA-256. If BETTER_AUTH_SECRET is rotated, all sealed client_key_pem values in the database become unreadable and must be re-sealed with the new key before rotation completes.

The full key rotation procedure (decrypt all rows with old key, re-encrypt with new key, atomically swap the secret) is out of scope for M2. Operators must not rotate BETTER_AUTH_SECRET without a migration plan for all sealed federation peer keys.

OID Assignments — Mosaic Internal OID Arc

Mosaic uses the private enterprise arc 1.3.6.1.4.1.99999 for custom X.509 certificate extensions in federation grant certificates.

IMPORTANT: This is a development/internal OID arc. Before deploying to a production environment accessible by external parties, register a proper IANA Private Enterprise Number (PEN) at https://pen.iana.org/pen/PenApplication.page and update these assignments accordingly.

Assigned OIDs

OID	Symbolic name	Description
1.3.6.1.4.1.99999.1	mosaic.federation.grantId	UUID of the federation_grants row authorising this cert
1.3.6.1.4.1.99999.2	mosaic.federation.subjectUserId	UUID of the local user on whose behalf the cert is issued

Encoding

Each extension value is DER-encoded as an ASN.1 UTF8String:

Tag    0x0C        (UTF8String)
Length 0x24        (36 decimal — fixed length of a UUID string)
Value  <36 ASCII bytes of the UUID>

The step-ca X.509 template at infra/step-ca/templates/federation.tpl produces this encoding via the Go template expression:

{{ printf "\x0c\x24%s" .Token.mosaic_grant_id | b64enc }}

The resulting base64 value is passed as the value field of the extension object in the template JSON.

CA Environment Variables

The CaService (apps/gateway/src/federation/ca.service.ts) requires the following environment variables at gateway startup:

Variable	Required	Description
STEP_CA_URL	Yes	Base URL of the step-ca instance, e.g. https://step-ca:9000
STEP_CA_PROVISIONER_PASSWORD	Yes	JWK provisioner password for the mosaic-fed provisioner
STEP_CA_PROVISIONER_KEY_JSON	Yes	JSON-encoded JWK (public + private) for the mosaic-fed provisioner
STEP_CA_ROOT_CERT_PATH	Yes	Absolute path to the step-ca root CA certificate PEM file

Set these variables in your environment or secret manager before starting the gateway. In the federated Docker Compose stack they are expected to be injected via Docker secrets and environment variable overrides.

Fail-loud contract

The CA service (and the X.509 template) are designed to fail loudly if the custom OIDs cannot be embedded:

• The template produces a malformed extension value (zero-length UTF8String body) when the JWT claims mosaic_grant_id or mosaic_subject_user_id are absent. step-ca rejects the CSR rather than issuing a cert without the OIDs.
• CaService.issueCert() throws a CaServiceError on every error path with a human-readable remediation string. It never silently returns a cert that may be missing the required extensions.