docs(swarm): comprehensive Docker Swarm deployment documentation
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- Update docker-compose.swarm.yml with external Authentik configuration - Comment out Authentik services (using external OIDC provider) - Comment out Authentik volumes - Add header with deployment instructions and current configuration - Create comprehensive SWARM-DEPLOYMENT.md guide - Prerequisites and swarm initialization - Manual OpenBao initialization (critical - no auto-init in swarm) - External service configuration examples - Scaling, updates, rollbacks - Troubleshooting and maintenance procedures - Backup and restore instructions - Update .env.swarm.example - Add note about external vs internal Authentik - Update default OIDC_ISSUER to use https - Clarify which variables are needed for internal Authentik - Update README.md Docker Swarm section - Fix deploy script path (./scripts/deploy-swarm.sh) - Add note about manual OpenBao initialization - Add warning about no profile support in swarm - Update documentation references to docs/ directory - Update documentation cross-references - Add deprecation notice to old DOCKER-SWARM.md - Add deployment guide reference to SWARM-QUICKREF.md - Update DOCKER-COMPOSE-GUIDE.md See Also section Key changes for swarm deployment: - Swarm does NOT support docker-compose profiles - External services must be manually commented out - OpenBao requires manual initialization (no sidecar) - All documentation updated with correct paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,7 @@
|
||||
# Mosaic Stack - Docker Swarm Deployment
|
||||
|
||||
**⚠️ This guide has been superseded. Please see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md) for the complete, up-to-date deployment guide.**
|
||||
|
||||
This guide covers deploying Mosaic Stack to a Docker Swarm cluster with Traefik reverse proxy integration.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
434
docs/SWARM-DEPLOYMENT.md
Normal file
434
docs/SWARM-DEPLOYMENT.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# Docker Swarm Deployment Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Initialize Docker Swarm
|
||||
|
||||
```bash
|
||||
# On the manager node (10.1.1.90)
|
||||
docker swarm init --advertise-addr 10.1.1.90
|
||||
|
||||
# For multi-node swarm, join worker nodes:
|
||||
# docker swarm join --token <token> 10.1.1.90:2377
|
||||
```
|
||||
|
||||
### 2. Create External Networks
|
||||
|
||||
```bash
|
||||
# Create traefik-public network (required for ingress)
|
||||
docker network create --driver=overlay traefik-public
|
||||
```
|
||||
|
||||
### 3. Verify Swarm Status
|
||||
|
||||
```bash
|
||||
docker node ls # Should show nodes in Ready state
|
||||
docker network ls # Should show traefik-public overlay network
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Create Environment File
|
||||
|
||||
```bash
|
||||
cd /opt/mosaic/stack # Or your deployment directory
|
||||
|
||||
# Copy example configuration
|
||||
cp .env.swarm.example .env
|
||||
|
||||
# Edit configuration
|
||||
nano .env
|
||||
```
|
||||
|
||||
**Required variables:**
|
||||
|
||||
- `POSTGRES_PASSWORD` - Strong password for PostgreSQL
|
||||
- `JWT_SECRET` - Random secret (min 32 chars)
|
||||
- `BETTER_AUTH_SECRET` - Random secret (min 32 chars)
|
||||
- `ENCRYPTION_KEY` - 64-char hex string (generate with `openssl rand -hex 32`)
|
||||
- `OIDC_CLIENT_ID` - From your Authentik/OIDC provider
|
||||
- `OIDC_CLIENT_SECRET` - From your Authentik/OIDC provider
|
||||
- `OIDC_ISSUER` - Your OIDC provider URL (must end with `/`)
|
||||
- `IMAGE_TAG` - `dev` or `latest` or specific commit SHA
|
||||
|
||||
### 2. Configure for External Services (Optional)
|
||||
|
||||
**For external Authentik:** Edit `docker-compose.swarm.yml` and comment out:
|
||||
|
||||
```yaml
|
||||
# Comment out these services if using external Authentik
|
||||
# authentik-postgres:
|
||||
# ...
|
||||
# authentik-redis:
|
||||
# ...
|
||||
# authentik-server:
|
||||
# ...
|
||||
# authentik-worker:
|
||||
# ...
|
||||
```
|
||||
|
||||
**For external Ollama:** Update `.env`:
|
||||
|
||||
```bash
|
||||
OLLAMA_ENDPOINT=http://your-ollama-server:11434
|
||||
```
|
||||
|
||||
Then comment out in `docker-compose.swarm.yml`:
|
||||
|
||||
```yaml
|
||||
# ollama:
|
||||
# ...
|
||||
```
|
||||
|
||||
**For external PostgreSQL/Valkey:** Comment them out and update `.env`:
|
||||
|
||||
```bash
|
||||
DATABASE_URL=postgresql://user:pass@external-db:5432/mosaic
|
||||
VALKEY_URL=redis://external-cache:6379
|
||||
```
|
||||
|
||||
### 3. Registry Authentication
|
||||
|
||||
```bash
|
||||
# Login to Gitea container registry
|
||||
docker login git.mosaicstack.dev
|
||||
|
||||
# Enter your Gitea username and password/token
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Deploy the Stack
|
||||
|
||||
```bash
|
||||
cd /opt/mosaic/stack
|
||||
|
||||
# Using the deploy script (recommended)
|
||||
./scripts/deploy-swarm.sh mosaic
|
||||
|
||||
# Or manually
|
||||
IMAGE_TAG=dev docker stack deploy \
|
||||
-c docker/docker-compose.swarm.yml \
|
||||
--with-registry-auth mosaic
|
||||
```
|
||||
|
||||
### Verify Deployment
|
||||
|
||||
```bash
|
||||
# Check stack services
|
||||
docker stack services mosaic
|
||||
|
||||
# Expected output - all services should show 1/1 replicas:
|
||||
# ID NAME MODE REPLICAS IMAGE
|
||||
# abc123 mosaic_postgres replicated 1/1 git.mosaicstack.dev/mosaic/stack-postgres:dev
|
||||
# def456 mosaic_valkey replicated 1/1 valkey/valkey:8-alpine
|
||||
# ghi789 mosaic_openbao replicated 1/1 git.mosaicstack.dev/mosaic/stack-openbao:dev
|
||||
# jkl012 mosaic_api replicated 1/1 git.mosaicstack.dev/mosaic/stack-api:dev
|
||||
# mno345 mosaic_web replicated 1/1 git.mosaicstack.dev/mosaic/stack-web:dev
|
||||
# pqr678 mosaic_orchestrator replicated 1/1 git.mosaicstack.dev/mosaic/stack-orchestrator:dev
|
||||
|
||||
# Check detailed task status
|
||||
docker stack ps mosaic --no-trunc
|
||||
|
||||
# Follow logs for specific services
|
||||
docker service logs mosaic_api --follow
|
||||
docker service logs mosaic_web --follow
|
||||
```
|
||||
|
||||
## Post-Deployment Configuration
|
||||
|
||||
### 1. Initialize OpenBao (CRITICAL)
|
||||
|
||||
**Important:** Unlike docker-compose, the swarm file does NOT include the `openbao-init` sidecar. You must manually initialize OpenBao:
|
||||
|
||||
```bash
|
||||
# Wait for OpenBao to be healthy
|
||||
sleep 30
|
||||
|
||||
# Get the OpenBao container ID
|
||||
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
|
||||
|
||||
# Initialize OpenBao
|
||||
docker exec $OPENBAO_CONTAINER bao operator init -key-shares=1 -key-threshold=1
|
||||
|
||||
# Save the output - you'll need the Unseal Key and Root Token!
|
||||
# Example output:
|
||||
# Unseal Key 1: abc123...
|
||||
# Initial Root Token: xyz789...
|
||||
|
||||
# Unseal OpenBao
|
||||
docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key-from-above>
|
||||
|
||||
# Enable Transit engine
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao secrets enable transit
|
||||
|
||||
# Create encryption key
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao write -f transit/keys/mosaic-credentials
|
||||
|
||||
# Create AppRole for API
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao auth enable approle
|
||||
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao write auth/approle/role/mosaic-api \
|
||||
token_policies="default" \
|
||||
token_ttl=1h \
|
||||
token_max_ttl=4h
|
||||
|
||||
# Get Role ID and Secret ID
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao read auth/approle/role/mosaic-api/role-id
|
||||
|
||||
docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
|
||||
bao write -f auth/approle/role/mosaic-api/secret-id
|
||||
|
||||
# Update API service with AppRole credentials (optional - or store in volume)
|
||||
docker service update mosaic_api \
|
||||
--env-add OPENBAO_ROLE_ID=<role-id> \
|
||||
--env-add OPENBAO_SECRET_ID=<secret-id>
|
||||
```
|
||||
|
||||
**Automated Alternative:** Create an init script and run it as a Docker container:
|
||||
|
||||
```bash
|
||||
# See docker/openbao/init.sh for reference
|
||||
# Create a one-time task that runs initialization
|
||||
docker service create --name openbao-init \
|
||||
--network mosaic_internal \
|
||||
--restart-condition=none \
|
||||
--mount type=volume,source=mosaic_openbao_init,target=/openbao/init \
|
||||
git.mosaicstack.dev/mosaic/stack-openbao:dev \
|
||||
/openbao/init.sh
|
||||
```
|
||||
|
||||
### 2. Run Database Migrations
|
||||
|
||||
```bash
|
||||
# Get API container ID
|
||||
API_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api)
|
||||
|
||||
# Run Prisma migrations
|
||||
docker exec $API_CONTAINER pnpm prisma migrate deploy
|
||||
|
||||
# Seed initial data (optional)
|
||||
docker exec $API_CONTAINER pnpm prisma db seed
|
||||
```
|
||||
|
||||
### 3. Verify Services
|
||||
|
||||
```bash
|
||||
# Check API health
|
||||
curl http://api.mosaicstack.dev/health
|
||||
|
||||
# Check Web UI
|
||||
curl http://mosaic.mosaicstack.dev
|
||||
|
||||
# Check Authentik (if bundled)
|
||||
curl http://auth.mosaicstack.dev/-/health/live/
|
||||
|
||||
# Check OpenBao status
|
||||
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
|
||||
docker exec $OPENBAO_CONTAINER bao status
|
||||
```
|
||||
|
||||
## Scaling Services
|
||||
|
||||
```bash
|
||||
# Scale web frontend
|
||||
docker service scale mosaic_web=3
|
||||
|
||||
# Scale API
|
||||
docker service scale mosaic_api=2
|
||||
|
||||
# Verify scaling
|
||||
docker service ls
|
||||
```
|
||||
|
||||
## Updates and Rollbacks
|
||||
|
||||
### Update Services to New Image Tag
|
||||
|
||||
```bash
|
||||
# Update all services to new tag
|
||||
IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic
|
||||
|
||||
# Or update individual service
|
||||
docker service update --image git.mosaicstack.dev/mosaic/stack-api:latest mosaic_api
|
||||
```
|
||||
|
||||
### Rollback Service
|
||||
|
||||
```bash
|
||||
# Rollback to previous version
|
||||
docker service rollback mosaic_api
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# API logs
|
||||
docker service logs mosaic_api --tail 100 --follow
|
||||
|
||||
# Web logs
|
||||
docker service logs mosaic_web --tail 100 --follow
|
||||
|
||||
# PostgreSQL logs
|
||||
docker service logs mosaic_postgres --tail 100
|
||||
|
||||
# All services
|
||||
for service in $(docker stack services mosaic --format '{{.Name}}'); do
|
||||
echo "=== $service ==="
|
||||
docker service logs $service --tail 20
|
||||
done
|
||||
```
|
||||
|
||||
### Backup Volumes
|
||||
|
||||
```bash
|
||||
# Backup PostgreSQL
|
||||
docker run --rm \
|
||||
-v mosaic_postgres_data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
|
||||
|
||||
# Backup OpenBao
|
||||
docker run --rm \
|
||||
-v mosaic_openbao_data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine tar czf /backup/openbao-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
|
||||
```
|
||||
|
||||
### Restore Volumes
|
||||
|
||||
```bash
|
||||
# Stop services first
|
||||
docker service scale mosaic_postgres=0
|
||||
|
||||
# Restore
|
||||
docker run --rm \
|
||||
-v mosaic_postgres_data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data'
|
||||
|
||||
# Restart service
|
||||
docker service scale mosaic_postgres=1
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
```bash
|
||||
# Check service tasks for errors
|
||||
docker service ps mosaic_api --no-trunc
|
||||
|
||||
# View detailed logs
|
||||
docker service logs mosaic_api --tail 100
|
||||
|
||||
# Inspect service configuration
|
||||
docker service inspect mosaic_api
|
||||
```
|
||||
|
||||
### Network Issues
|
||||
|
||||
```bash
|
||||
# Verify overlay networks
|
||||
docker network ls --filter driver=overlay
|
||||
|
||||
# Inspect traefik-public network
|
||||
docker network inspect traefik-public
|
||||
|
||||
# Check service network connectivity
|
||||
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
|
||||
ping postgres
|
||||
```
|
||||
|
||||
### OpenBao Issues
|
||||
|
||||
```bash
|
||||
# Check OpenBao status
|
||||
OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
|
||||
docker exec $OPENBAO_CONTAINER bao status
|
||||
|
||||
# Re-unseal if sealed
|
||||
docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key>
|
||||
|
||||
# Check logs
|
||||
docker service logs mosaic_openbao --tail 100
|
||||
```
|
||||
|
||||
### Complete Stack Restart
|
||||
|
||||
```bash
|
||||
# Remove stack (keeps volumes)
|
||||
docker stack rm mosaic
|
||||
|
||||
# Wait for cleanup
|
||||
sleep 30
|
||||
|
||||
# Redeploy
|
||||
./scripts/deploy-swarm.sh mosaic
|
||||
```
|
||||
|
||||
## External Services Configuration
|
||||
|
||||
### Using External Authentik
|
||||
|
||||
1. Comment out Authentik services in `docker-compose.swarm.yml`
|
||||
2. Configure in `.env`:
|
||||
```bash
|
||||
OIDC_ISSUER=https://auth.diversecanvas.com/application/o/mosaic-stack/
|
||||
OIDC_CLIENT_ID=your-client-id
|
||||
OIDC_CLIENT_SECRET=your-client-secret
|
||||
OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik
|
||||
```
|
||||
|
||||
### Using External PostgreSQL
|
||||
|
||||
1. Comment out `postgres` service in `docker-compose.swarm.yml`
|
||||
2. Configure in `.env`:
|
||||
```bash
|
||||
DATABASE_URL=postgresql://user:pass@rds.amazonaws.com:5432/mosaic
|
||||
```
|
||||
|
||||
### Using External Valkey/Redis
|
||||
|
||||
1. Comment out `valkey` service in `docker-compose.swarm.yml`
|
||||
2. Configure in `.env`:
|
||||
```bash
|
||||
VALKEY_URL=redis://elasticache.amazonaws.com:6379
|
||||
```
|
||||
|
||||
### Using External OpenBao/Vault
|
||||
|
||||
1. Comment out `openbao` service in `docker-compose.swarm.yml`
|
||||
2. Configure in `.env`:
|
||||
```bash
|
||||
OPENBAO_ADDR=https://vault.example.com:8200
|
||||
OPENBAO_ROLE_ID=your-role-id
|
||||
OPENBAO_SECRET_ID=your-secret-id
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Never commit `.env` to version control** - Contains secrets
|
||||
2. **Use strong passwords** - Generate with `openssl rand -base64 32`
|
||||
3. **Store OpenBao unseal keys securely** - Required for disaster recovery
|
||||
4. **Enable TLS/SSL in production** - Configure Traefik with Let's Encrypt
|
||||
5. **Regular backups** - Backup volumes and OpenBao keys
|
||||
6. **Monitor logs** - Watch for security events and errors
|
||||
7. **Update regularly** - Pull latest images with `IMAGE_TAG=latest`
|
||||
|
||||
## Quick Reference
|
||||
|
||||
See `docs/SWARM-QUICKREF.md` for quick command reference.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Docker Compose Guide](../docker/DOCKER-COMPOSE-GUIDE.md) - Regular docker-compose deployment
|
||||
- [OpenBao Documentation](OPENBAO.md) - Secrets management setup
|
||||
- [Configuration Guide](CONFIGURATION.md) - All environment variables
|
||||
@@ -1,5 +1,9 @@
|
||||
# Docker Swarm Quick Reference
|
||||
|
||||
**For complete deployment instructions, see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md)**
|
||||
|
||||
This is a quick reference for common Docker Swarm commands after your stack is deployed.
|
||||
|
||||
## Initial Setup
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user