stack/docs/SWARM-DEPLOYMENT.md

Docker Swarm Deployment Guide

Prerequisites

1. Initialize Docker Swarm

# On the manager node (10.1.1.90)
docker swarm init --advertise-addr 10.1.1.90

# For multi-node swarm, join worker nodes:
# docker swarm join --token <token> 10.1.1.90:2377

2. Create External Networks

# Create traefik-public network (required for ingress)
docker network create --driver=overlay traefik-public

3. Verify Swarm Status

docker node ls          # Should show nodes in Ready state
docker network ls       # Should show traefik-public overlay network

Configuration

1. Create Environment File

cd /opt/mosaic/stack  # Or your deployment directory

# Copy example configuration
cp .env.swarm.example .env

# Edit configuration
nano .env

Required variables:

• POSTGRES_PASSWORD - Strong password for PostgreSQL
• JWT_SECRET - Random secret (min 32 chars)
• BETTER_AUTH_SECRET - Random secret (min 32 chars)
• ENCRYPTION_KEY - 64-char hex string (generate with openssl rand -hex 32)
• OIDC_CLIENT_ID - From your Authentik/OIDC provider
• OIDC_CLIENT_SECRET - From your Authentik/OIDC provider
• OIDC_ISSUER - Your OIDC provider URL (must end with /)
• IMAGE_TAG - dev or latest or specific commit SHA

2. Configure for External Services (Optional)

For external Authentik: Edit docker-compose.swarm.yml and comment out:

# Comment out these services if using external Authentik
#  authentik-postgres:
#    ...
#  authentik-redis:
#    ...
#  authentik-server:
#    ...
#  authentik-worker:
#    ...

For external Ollama: Update .env:

OLLAMA_ENDPOINT=http://your-ollama-server:11434

Then comment out in docker-compose.swarm.yml:

#  ollama:
#    ...

For external PostgreSQL/Valkey: Comment them out and update .env:

DATABASE_URL=postgresql://user:pass@external-db:5432/mosaic
VALKEY_URL=redis://external-cache:6379

3. Registry Authentication

# Login to Gitea container registry
docker login git.mosaicstack.dev

# Enter your Gitea username and password/token

Deployment

Step 1: Deploy OpenBao (Standalone)

CRITICAL: OpenBao CANNOT run in swarm mode due to port binding conflicts. Deploy it as a standalone container first.

cd /opt/mosaic/stack

# Deploy OpenBao standalone
docker compose -f docker-compose.openbao.yml up -d

# Verify OpenBao is running
docker ps | grep openbao

# Check initialization (should complete in ~30 seconds)
docker logs mosaic-openbao-init --follow

# Wait for OpenBao to be ready
sleep 30

Alternative Options:

• External Vault: Skip this step and configure OPENBAO_ADDR to point to your external Vault instance
• Fallback Mode: Skip this step - API will use AES-256-GCM encryption with ENCRYPTION_KEY

See OpenBao Deployment Guide for detailed options.

Step 2: Deploy the Swarm Stack

cd /opt/mosaic/stack

# Using the deploy script (recommended)
IMAGE_TAG=dev ./scripts/deploy-swarm.sh mosaic

# Or manually
IMAGE_TAG=dev docker stack deploy \
  -c docker-compose.swarm.yml \
  --with-registry-auth mosaic

Verify Deployment

# Check stack services
docker stack services mosaic

# Expected output - all services should show 1/1 replicas:
# ID             NAME                         MODE         REPLICAS   IMAGE
# abc123         mosaic_postgres              replicated   1/1        git.mosaicstack.dev/mosaic/stack-postgres:dev
# def456         mosaic_valkey                replicated   1/1        valkey/valkey:8-alpine
# jkl012         mosaic_api                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-api:dev
# mno345         mosaic_web                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-web:dev
# pqr678         mosaic_orchestrator          replicated   1/1        git.mosaicstack.dev/mosaic/stack-orchestrator:dev
#
# Note: OpenBao runs as standalone container, not in swarm stack

# Check detailed task status
docker stack ps mosaic --no-trunc

# Follow logs for specific services
docker service logs mosaic_api --follow
docker service logs mosaic_web --follow

Post-Deployment Configuration

1. Verify OpenBao Initialization

OpenBao was deployed as a standalone container with automatic initialization. Verify it completed:

# Check OpenBao container is running
docker ps | grep openbao

# Verify initialization completed
docker logs mosaic-openbao-init

# Expected output:
# ✓ OpenBao initialized successfully
# ✓ Transit engine enabled
# ✓ Encryption key created: mosaic-credentials
# ✓ AppRole authentication configured

# Test OpenBao connectivity from host
curl http://localhost:8200/v1/sys/health

If using external Vault: Skip this step - verify your external Vault is configured per OpenBao Deployment Guide.

2. Run Database Migrations

# Get API container ID
API_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api)

# Run Prisma migrations
docker exec $API_CONTAINER pnpm prisma migrate deploy

# Seed initial data (optional)
docker exec $API_CONTAINER pnpm prisma db seed

3. Verify Services

# Check API health
curl http://api.mosaicstack.dev/health

# Check Web UI
curl http://mosaic.mosaicstack.dev

# Check Authentik (if bundled)
curl http://auth.mosaicstack.dev/-/health/live/

# Check OpenBao status
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
docker exec $OPENBAO_CONTAINER bao status

Scaling Services

# Scale web frontend
docker service scale mosaic_web=3

# Scale API
docker service scale mosaic_api=2

# Verify scaling
docker service ls

Updates and Rollbacks

Update Services to New Image Tag

# Update all services to new tag
IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic

# Or update individual service
docker service update --image git.mosaicstack.dev/mosaic/stack-api:latest mosaic_api

Rollback Service

# Rollback to previous version
docker service rollback mosaic_api

Maintenance

View Logs

# API logs
docker service logs mosaic_api --tail 100 --follow

# Web logs
docker service logs mosaic_web --tail 100 --follow

# PostgreSQL logs
docker service logs mosaic_postgres --tail 100

# All services
for service in $(docker stack services mosaic --format '{{.Name}}'); do
  echo "=== $service ==="
  docker service logs $service --tail 20
done

Backup Volumes

# Backup PostgreSQL
docker run --rm \
  -v mosaic_postgres_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .

# Backup OpenBao
docker run --rm \
  -v mosaic_openbao_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/openbao-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .

Restore Volumes

# Stop services first
docker service scale mosaic_postgres=0

# Restore
docker run --rm \
  -v mosaic_postgres_data:/data \
  -v $(pwd):/backup \
  alpine sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data'

# Restart service
docker service scale mosaic_postgres=1

Troubleshooting

Service Won't Start

# Check service tasks for errors
docker service ps mosaic_api --no-trunc

# View detailed logs
docker service logs mosaic_api --tail 100

# Inspect service configuration
docker service inspect mosaic_api

Network Issues

# Verify overlay networks
docker network ls --filter driver=overlay

# Inspect traefik-public network
docker network inspect traefik-public

# Check service network connectivity
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
  ping postgres

OpenBao Issues

# Check OpenBao status
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
docker exec $OPENBAO_CONTAINER bao status

# Re-unseal if sealed
docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key>

# Check logs
docker service logs mosaic_openbao --tail 100

Complete Stack Restart

# Remove stack (keeps volumes)
docker stack rm mosaic

# Wait for cleanup
sleep 30

# Redeploy
./scripts/deploy-swarm.sh mosaic

External Services Configuration

Using External Authentik

1. Comment out Authentik services in docker-compose.swarm.yml
2. Configure in .env:

   OIDC_ISSUER=https://auth.diversecanvas.com/application/o/mosaic-stack/
   OIDC_CLIENT_ID=your-client-id
   OIDC_CLIENT_SECRET=your-client-secret
   OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik
   

Using External PostgreSQL

1. Comment out postgres service in docker-compose.swarm.yml
2. Configure in .env:

   DATABASE_URL=postgresql://user:pass@rds.amazonaws.com:5432/mosaic
   

Using External Valkey/Redis

1. Comment out valkey service in docker-compose.swarm.yml
2. Configure in .env:

   VALKEY_URL=redis://elasticache.amazonaws.com:6379
   

Using External OpenBao/Vault

1. Comment out openbao service in docker-compose.swarm.yml
2. Configure in .env:

   OPENBAO_ADDR=https://vault.example.com:8200
   OPENBAO_ROLE_ID=your-role-id
   OPENBAO_SECRET_ID=your-secret-id
   

Security Considerations

1. Never commit .env to version control - Contains secrets
2. Use strong passwords - Generate with openssl rand -base64 32
3. Store OpenBao unseal keys securely - Required for disaster recovery
4. Enable TLS/SSL in production - Configure Traefik with Let's Encrypt
5. Regular backups - Backup volumes and OpenBao keys
6. Monitor logs - Watch for security events and errors
7. Update regularly - Pull latest images with IMAGE_TAG=latest

Quick Reference

See docs/SWARM-QUICKREF.md for quick command reference.

See Also

• Docker Compose Guide - Regular docker-compose deployment
• OpenBao Documentation - Secrets management setup
• Configuration Guide - All environment variables