stack/docs/federation/SETUP.md

# Federated Tier Setup Guide

## What is the federated tier?

The federated tier is designed for multi-user and multi-host deployments. It consists of PostgreSQL 17 with pgvector extension (for embeddings and RAG), Valkey for distributed task queueing and caching, and a shared configuration across multiple Mosaic gateway instances. Use this tier when running Mosaic in production or when scaling beyond a single-host deployment.

## Prerequisites

- Docker and Docker Compose installed
- Ports 5433 (PostgreSQL) and 6380 (Valkey) available on your host (or adjust environment variables)
- At least 2 GB free disk space for data volumes

## Start the federated stack

Run the federated overlay:

```bash
docker compose -f docker-compose.federated.yml --profile federated up -d
```

This starts PostgreSQL 17 with pgvector and Valkey 8. The pgvector extension is created automatically on first boot.

Verify the services are running:

```bash
docker compose -f docker-compose.federated.yml ps
```

Expected output shows `postgres-federated` and `valkey-federated` both healthy.

## Configure mosaic for federated tier

Create or update your `mosaic.config.json`:

```json
{
  "tier": "federated",
  "database": "postgresql://mosaic:mosaic@localhost:5433/mosaic",
  "queue": "redis://localhost:6380"
}
```

If you're using environment variables instead:

```bash
export DATABASE_URL="postgresql://mosaic:mosaic@localhost:5433/mosaic"
export REDIS_URL="redis://localhost:6380"
```

## Verify health

Run the health check:

```bash
mosaic gateway doctor
```

Expected output (green):

```
Tier: federated  Config: mosaic.config.json
  ✓ postgres         localhost:5433       (42ms)
  ✓ valkey           localhost:6380       (8ms)
  ✓ pgvector         (embedded)           (15ms)
```

For JSON output (useful in CI/automation):

```bash
mosaic gateway doctor --json
```

## Troubleshooting

### Port conflicts

**Symptom:** `bind: address already in use`

**Fix:** Stop the base dev stack first:

```bash
docker compose down
docker compose -f docker-compose.federated.yml --profile federated up -d
```

Or change the host port with an environment variable:

```bash
PG_FEDERATED_HOST_PORT=5434 VALKEY_FEDERATED_HOST_PORT=6381 \
  docker compose -f docker-compose.federated.yml --profile federated up -d
```

### pgvector extension error

**Symptom:** `ERROR: could not open extension control file`

**Fix:** pgvector is created at first boot. Check logs:

```bash
docker compose -f docker-compose.federated.yml logs postgres-federated | grep -i vector
```

If missing, exec into the container and create it manually:

```bash
docker exec <postgres-federated-id> psql -U mosaic -d mosaic -c "CREATE EXTENSION vector;"
```

### Valkey connection refused

**Symptom:** `Error: connect ECONNREFUSED 127.0.0.1:6380`

**Fix:** Check service health:

```bash
docker compose -f docker-compose.federated.yml logs valkey-federated
```

If Valkey is running, verify your firewall allows 6380. On macOS, Docker Desktop may require binding to `host.docker.internal` instead of `localhost`.

## Key rotation (deferred)

Federation peer private keys (`federation_peers.client_key_pem`) are sealed at rest using AES-256-GCM with a key derived from `BETTER_AUTH_SECRET` via SHA-256. If `BETTER_AUTH_SECRET` is rotated, all sealed `client_key_pem` values in the database become unreadable and must be re-sealed with the new key before rotation completes.

The full key rotation procedure (decrypt all rows with old key, re-encrypt with new key, atomically swap the secret) is out of scope for M2. Operators must not rotate `BETTER_AUTH_SECRET` without a migration plan for all sealed federation peer keys.