# Docker Swarm Deployment Guide

## Prerequisites

### 1. Initialize Docker Swarm

```bash
# On the manager node (10.1.1.90)
docker swarm init --advertise-addr 10.1.1.90

# For multi-node swarm, join worker nodes:
# docker swarm join --token <token> 10.1.1.90:2377
```

### 2. Create External Networks

```bash
# Create traefik-public network (required for ingress)
docker network create --driver=overlay traefik-public
```

### 3. Verify Swarm Status

```bash
docker node ls          # Should show nodes in Ready state
docker network ls       # Should show traefik-public overlay network
```

## Configuration

### 1. Create Environment File

```bash
cd /opt/mosaic/stack  # Or your deployment directory

# Copy example configuration
cp .env.swarm.example .env

# Edit configuration
nano .env
```

**Required variables:**

- `POSTGRES_PASSWORD` - Strong password for PostgreSQL
- `JWT_SECRET` - Random secret (min 32 chars)
- `BETTER_AUTH_SECRET` - Random secret (min 32 chars)
- `ENCRYPTION_KEY` - 64-char hex string (generate with `openssl rand -hex 32`)
- `OIDC_CLIENT_ID` - From your Authentik/OIDC provider
- `OIDC_CLIENT_SECRET` - From your Authentik/OIDC provider
- `OIDC_ISSUER` - Your OIDC provider URL (must end with `/`)
- `IMAGE_TAG` - `dev` or `latest` or specific commit SHA

### 2. Configure for External Services (Optional)

**For external Authentik:** Edit `docker-compose.swarm.yml` and comment out:

```yaml
# Comment out these services if using external Authentik
#  authentik-postgres:
#    ...
#  authentik-redis:
#    ...
#  authentik-server:
#    ...
#  authentik-worker:
#    ...
```

**For external Ollama:** Update `.env`:

```bash
OLLAMA_ENDPOINT=http://your-ollama-server:11434
```

Then comment out in `docker-compose.swarm.yml`:

```yaml
#  ollama:
#    ...
```

**For external PostgreSQL/Valkey:** Comment them out and update `.env`:

```bash
DATABASE_URL=postgresql://user:pass@external-db:5432/mosaic
VALKEY_URL=redis://external-cache:6379
```

### 3. Registry Authentication

```bash
# Login to Gitea container registry
docker login git.mosaicstack.dev

# Enter your Gitea username and password/token
```

## Deployment

### Deploy the Stack

```bash
cd /opt/mosaic/stack

# Using the deploy script (recommended)
./scripts/deploy-swarm.sh mosaic

# Or manually
IMAGE_TAG=dev docker stack deploy \
  -c docker-compose.swarm.yml \
  --with-registry-auth mosaic
```

### Verify Deployment

```bash
# Check stack services
docker stack services mosaic

# Expected output - all services should show 1/1 replicas:
# ID             NAME                         MODE         REPLICAS   IMAGE
# abc123         mosaic_postgres              replicated   1/1        git.mosaicstack.dev/mosaic/stack-postgres:dev
# def456         mosaic_valkey                replicated   1/1        valkey/valkey:8-alpine
# ghi789         mosaic_openbao               replicated   1/1        git.mosaicstack.dev/mosaic/stack-openbao:dev
# jkl012         mosaic_api                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-api:dev
# mno345         mosaic_web                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-web:dev
# pqr678         mosaic_orchestrator          replicated   1/1        git.mosaicstack.dev/mosaic/stack-orchestrator:dev

# Check detailed task status
docker stack ps mosaic --no-trunc

# Follow logs for specific services
docker service logs mosaic_api --follow
docker service logs mosaic_web --follow
```

## Post-Deployment Configuration

### 1. Initialize OpenBao (CRITICAL)

**Important:** Unlike docker-compose, the swarm file does NOT include the `openbao-init` sidecar. You must manually initialize OpenBao:

```bash
# Wait for OpenBao to be healthy
sleep 30

# Get the OpenBao container ID
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)

# Initialize OpenBao
docker exec $OPENBAO_CONTAINER bao operator init -key-shares=1 -key-threshold=1

# Save the output - you'll need the Unseal Key and Root Token!
# Example output:
# Unseal Key 1: abc123...
# Initial Root Token: xyz789...

# Unseal OpenBao
docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key-from-above>

# Enable Transit engine
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao secrets enable transit

# Create encryption key
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao write -f transit/keys/mosaic-credentials

# Create AppRole for API
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao auth enable approle

docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao write auth/approle/role/mosaic-api \
    token_policies="default" \
    token_ttl=1h \
    token_max_ttl=4h

# Get Role ID and Secret ID
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao read auth/approle/role/mosaic-api/role-id

docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
  bao write -f auth/approle/role/mosaic-api/secret-id

# Update API service with AppRole credentials (optional - or store in volume)
docker service update mosaic_api \
  --env-add OPENBAO_ROLE_ID=<role-id> \
  --env-add OPENBAO_SECRET_ID=<secret-id>
```

**Automated Alternative:** Create an init script and run it as a Docker container:

```bash
# See docker/openbao/init.sh for reference
# Create a one-time task that runs initialization
docker service create --name openbao-init \
  --network mosaic_internal \
  --restart-condition=none \
  --mount type=volume,source=mosaic_openbao_init,target=/openbao/init \
  git.mosaicstack.dev/mosaic/stack-openbao:dev \
  /openbao/init.sh
```

### 2. Run Database Migrations

```bash
# Get API container ID
API_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api)

# Run Prisma migrations
docker exec $API_CONTAINER pnpm prisma migrate deploy

# Seed initial data (optional)
docker exec $API_CONTAINER pnpm prisma db seed
```

### 3. Verify Services

```bash
# Check API health
curl http://api.mosaicstack.dev/health

# Check Web UI
curl http://mosaic.mosaicstack.dev

# Check Authentik (if bundled)
curl http://auth.mosaicstack.dev/-/health/live/

# Check OpenBao status
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
docker exec $OPENBAO_CONTAINER bao status
```

## Scaling Services

```bash
# Scale web frontend
docker service scale mosaic_web=3

# Scale API
docker service scale mosaic_api=2

# Verify scaling
docker service ls
```

## Updates and Rollbacks

### Update Services to New Image Tag

```bash
# Update all services to new tag
IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic

# Or update individual service
docker service update --image git.mosaicstack.dev/mosaic/stack-api:latest mosaic_api
```

### Rollback Service

```bash
# Rollback to previous version
docker service rollback mosaic_api
```

## Maintenance

### View Logs

```bash
# API logs
docker service logs mosaic_api --tail 100 --follow

# Web logs
docker service logs mosaic_web --tail 100 --follow

# PostgreSQL logs
docker service logs mosaic_postgres --tail 100

# All services
for service in $(docker stack services mosaic --format '{{.Name}}'); do
  echo "=== $service ==="
  docker service logs $service --tail 20
done
```

### Backup Volumes

```bash
# Backup PostgreSQL
docker run --rm \
  -v mosaic_postgres_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .

# Backup OpenBao
docker run --rm \
  -v mosaic_openbao_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/openbao-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
```

### Restore Volumes

```bash
# Stop services first
docker service scale mosaic_postgres=0

# Restore
docker run --rm \
  -v mosaic_postgres_data:/data \
  -v $(pwd):/backup \
  alpine sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data'

# Restart service
docker service scale mosaic_postgres=1
```

## Troubleshooting

### Service Won't Start

```bash
# Check service tasks for errors
docker service ps mosaic_api --no-trunc

# View detailed logs
docker service logs mosaic_api --tail 100

# Inspect service configuration
docker service inspect mosaic_api
```

### Network Issues

```bash
# Verify overlay networks
docker network ls --filter driver=overlay

# Inspect traefik-public network
docker network inspect traefik-public

# Check service network connectivity
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
  ping postgres
```

### OpenBao Issues

```bash
# Check OpenBao status
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
docker exec $OPENBAO_CONTAINER bao status

# Re-unseal if sealed
docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key>

# Check logs
docker service logs mosaic_openbao --tail 100
```

### Complete Stack Restart

```bash
# Remove stack (keeps volumes)
docker stack rm mosaic

# Wait for cleanup
sleep 30

# Redeploy
./scripts/deploy-swarm.sh mosaic
```

## External Services Configuration

### Using External Authentik

1. Comment out Authentik services in `docker-compose.swarm.yml`
2. Configure in `.env`:
   ```bash
   OIDC_ISSUER=https://auth.diversecanvas.com/application/o/mosaic-stack/
   OIDC_CLIENT_ID=your-client-id
   OIDC_CLIENT_SECRET=your-client-secret
   OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik
   ```

### Using External PostgreSQL

1. Comment out `postgres` service in `docker-compose.swarm.yml`
2. Configure in `.env`:
   ```bash
   DATABASE_URL=postgresql://user:pass@rds.amazonaws.com:5432/mosaic
   ```

### Using External Valkey/Redis

1. Comment out `valkey` service in `docker-compose.swarm.yml`
2. Configure in `.env`:
   ```bash
   VALKEY_URL=redis://elasticache.amazonaws.com:6379
   ```

### Using External OpenBao/Vault

1. Comment out `openbao` service in `docker-compose.swarm.yml`
2. Configure in `.env`:
   ```bash
   OPENBAO_ADDR=https://vault.example.com:8200
   OPENBAO_ROLE_ID=your-role-id
   OPENBAO_SECRET_ID=your-secret-id
   ```

## Security Considerations

1. **Never commit `.env` to version control** - Contains secrets
2. **Use strong passwords** - Generate with `openssl rand -base64 32`
3. **Store OpenBao unseal keys securely** - Required for disaster recovery
4. **Enable TLS/SSL in production** - Configure Traefik with Let's Encrypt
5. **Regular backups** - Backup volumes and OpenBao keys
6. **Monitor logs** - Watch for security events and errors
7. **Update regularly** - Pull latest images with `IMAGE_TAG=latest`

## Quick Reference

See `docs/SWARM-QUICKREF.md` for quick command reference.

## See Also

- [Docker Compose Guide](../docker/DOCKER-COMPOSE-GUIDE.md) - Regular docker-compose deployment
- [OpenBao Documentation](OPENBAO.md) - Secrets management setup
- [Configuration Guide](CONFIGURATION.md) - All environment variables