Jason Woltje
f8477d5052
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- Update docker-compose.swarm.yml with external Authentik configuration - Comment out Authentik services (using external OIDC provider) - Comment out Authentik volumes - Add header with deployment instructions and current configuration - Create comprehensive SWARM-DEPLOYMENT.md guide - Prerequisites and swarm initialization - Manual OpenBao initialization (critical - no auto-init in swarm) - External service configuration examples - Scaling, updates, rollbacks - Troubleshooting and maintenance procedures - Backup and restore instructions - Update .env.swarm.example - Add note about external vs internal Authentik - Update default OIDC_ISSUER to use https - Clarify which variables are needed for internal Authentik - Update README.md Docker Swarm section - Fix deploy script path (./scripts/deploy-swarm.sh) - Add note about manual OpenBao initialization - Add warning about no profile support in swarm - Update documentation references to docs/ directory - Update documentation cross-references - Add deprecation notice to old DOCKER-SWARM.md - Add deployment guide reference to SWARM-QUICKREF.md - Update DOCKER-COMPOSE-GUIDE.md See Also section Key changes for swarm deployment: - Swarm does NOT support docker-compose profiles - External services must be manually commented out - OpenBao requires manual initialization (no sidecar) - All documentation updated with correct paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
328 lines
7.0 KiB
Markdown
328 lines
7.0 KiB
Markdown
# Docker Swarm Quick Reference
|
|
|
|
**For complete deployment instructions, see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md)**
|
|
|
|
This is a quick reference for common Docker Swarm commands after your stack is deployed.
|
|
|
|
## Initial Setup
|
|
|
|
```bash
|
|
# 1. Configure environment
|
|
cp .env.swarm.example .env
|
|
nano .env # Set passwords, API keys, domains
|
|
|
|
# 2. Create Traefik network (if needed)
|
|
docker network create --driver=overlay traefik-public
|
|
|
|
# 3. Deploy stack
|
|
./scripts/deploy-swarm.sh mosaic
|
|
```
|
|
|
|
## Common Commands
|
|
|
|
### Stack Management
|
|
|
|
```bash
|
|
# Deploy/update stack
|
|
docker stack deploy -c docker-compose.swarm.yml mosaic
|
|
|
|
# List all stacks
|
|
docker stack ls
|
|
|
|
# Remove stack
|
|
docker stack rm mosaic
|
|
|
|
# List services in stack
|
|
docker stack services mosaic
|
|
|
|
# List tasks in stack
|
|
docker stack ps mosaic
|
|
```
|
|
|
|
### Service Management
|
|
|
|
```bash
|
|
# List all services
|
|
docker service ls
|
|
|
|
# Inspect service
|
|
docker service inspect mosaic_api
|
|
|
|
# View service logs
|
|
docker service logs mosaic_api --tail 100 --follow
|
|
|
|
# Scale service
|
|
docker service scale mosaic_web=3
|
|
|
|
# Update service (force redeploy)
|
|
docker service update --force mosaic_api
|
|
|
|
# Update service image
|
|
docker service update --image mosaic-stack-api:latest mosaic_api
|
|
|
|
# Rollback service
|
|
docker service rollback mosaic_api
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
```bash
|
|
# Watch service status
|
|
watch -n 2 'docker service ls'
|
|
|
|
# Service resource usage
|
|
docker stats $(docker ps --filter label=com.docker.swarm.service.name=mosaic_api -q)
|
|
|
|
# Check service placement
|
|
docker service ps mosaic_api --format "table {{.Name}}\t{{.Node}}\t{{.CurrentState}}"
|
|
```
|
|
|
|
### Debugging
|
|
|
|
```bash
|
|
# Check why service failed
|
|
docker service ps mosaic_api --no-trunc
|
|
|
|
# View recent logs with timestamps
|
|
docker service logs mosaic_api --timestamps --tail 50
|
|
|
|
# Follow logs in real-time
|
|
docker service logs mosaic_api --follow
|
|
|
|
# Exec into running container
|
|
docker exec -it $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) sh
|
|
```
|
|
|
|
### Network Management
|
|
|
|
```bash
|
|
# List networks
|
|
docker network ls
|
|
|
|
# Inspect traefik-public network
|
|
docker network inspect traefik-public
|
|
|
|
# List containers on traefik-public
|
|
docker network inspect traefik-public --format '{{range .Containers}}{{.Name}} {{end}}'
|
|
```
|
|
|
|
### Volume Management
|
|
|
|
```bash
|
|
# List volumes
|
|
docker volume ls --filter label=com.docker.stack.namespace=mosaic
|
|
|
|
# Inspect volume
|
|
docker volume inspect mosaic_postgres_data
|
|
|
|
# Backup volume
|
|
docker run --rm -v mosaic_postgres_data:/data -v $(pwd):/backup alpine \
|
|
tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
|
|
|
|
# Restore volume
|
|
docker run --rm -v mosaic_postgres_data:/data -v $(pwd):/backup alpine \
|
|
tar xzf /backup/postgres-20260208-143022.tar.gz -C /data
|
|
```
|
|
|
|
## Service-Specific Commands
|
|
|
|
### Database (PostgreSQL)
|
|
|
|
```bash
|
|
# Connect to database
|
|
docker exec -it $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_postgres) \
|
|
psql -U mosaic -d mosaic
|
|
|
|
# Run migrations (from API container)
|
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
|
|
pnpm prisma migrate deploy
|
|
|
|
# View database logs
|
|
docker service logs mosaic_postgres --tail 100
|
|
```
|
|
|
|
### API Service
|
|
|
|
```bash
|
|
# View API logs
|
|
docker service logs mosaic_api --follow
|
|
|
|
# Check API health
|
|
curl http://api.mosaicstack.dev/health
|
|
|
|
# Force API redeploy
|
|
docker service update --force mosaic_api
|
|
```
|
|
|
|
### Web Service
|
|
|
|
```bash
|
|
# View web logs
|
|
docker service logs mosaic_web --follow
|
|
|
|
# Scale web to 3 replicas
|
|
docker service scale mosaic_web=3
|
|
|
|
# Check web health
|
|
curl http://mosaic.mosaicstack.dev
|
|
```
|
|
|
|
### Authentik
|
|
|
|
```bash
|
|
# View Authentik logs
|
|
docker service logs mosaic_authentik-server --follow
|
|
docker service logs mosaic_authentik-worker --follow
|
|
|
|
# Access Authentik UI
|
|
open http://auth.mosaicstack.dev
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Service Won't Start
|
|
|
|
```bash
|
|
# 1. Check service tasks
|
|
docker service ps mosaic_api --no-trunc
|
|
|
|
# 2. View service logs
|
|
docker service logs mosaic_api --tail 100
|
|
|
|
# 3. Check if image exists
|
|
docker images | grep mosaic-stack-api
|
|
|
|
# 4. Rebuild and update
|
|
docker compose -f docker-compose.swarm.yml build api
|
|
docker service update --image mosaic-stack-api:latest mosaic_api
|
|
```
|
|
|
|
### Traefik Not Routing
|
|
|
|
```bash
|
|
# 1. Verify service is on traefik-public network
|
|
docker service inspect mosaic_web | grep -A 10 Networks
|
|
|
|
# 2. Check Traefik labels
|
|
docker service inspect mosaic_web --format '{{json .Spec.Labels}}' | jq
|
|
|
|
# 3. Verify DNS resolution
|
|
ping mosaic.mosaicstack.dev
|
|
|
|
# 4. Check Traefik logs (if Traefik is a service)
|
|
docker service logs traefik --tail 50
|
|
```
|
|
|
|
### Database Connection Failed
|
|
|
|
```bash
|
|
# 1. Check postgres is running
|
|
docker service ls | grep postgres
|
|
|
|
# 2. Check postgres health
|
|
docker service ps mosaic_postgres
|
|
|
|
# 3. View postgres logs
|
|
docker service logs mosaic_postgres --tail 50
|
|
|
|
# 4. Test connection from API container
|
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
|
|
sh -c 'nc -zv postgres 5432'
|
|
```
|
|
|
|
### Out of Memory / Resources
|
|
|
|
```bash
|
|
# Check node resources
|
|
docker node ls
|
|
docker node inspect self --format '{{json .Description.Resources}}' | jq
|
|
|
|
# Check service resource limits
|
|
docker service inspect mosaic_api --format '{{json .Spec.TaskTemplate.Resources}}' | jq
|
|
|
|
# Update resource limits
|
|
docker service update --limit-memory 1g --reserve-memory 512m mosaic_api
|
|
```
|
|
|
|
## Useful Aliases
|
|
|
|
Add to `~/.bashrc` or `~/.zshrc`:
|
|
|
|
```bash
|
|
# Stack shortcuts
|
|
alias dss='docker stack services'
|
|
alias dsp='docker stack ps'
|
|
alias dsl='docker service logs'
|
|
alias dsi='docker service inspect'
|
|
alias dsu='docker service update'
|
|
|
|
# Mosaic-specific
|
|
alias mosaic-logs='docker service logs mosaic_api --follow'
|
|
alias mosaic-status='docker stack services mosaic'
|
|
alias mosaic-ps='docker stack ps mosaic'
|
|
alias mosaic-deploy='./scripts/deploy-swarm.sh mosaic'
|
|
```
|
|
|
|
## Emergency Procedures
|
|
|
|
### Complete Stack Restart
|
|
|
|
```bash
|
|
# 1. Remove stack (keeps volumes)
|
|
docker stack rm mosaic
|
|
|
|
# 2. Wait for cleanup (30 seconds)
|
|
sleep 30
|
|
|
|
# 3. Redeploy
|
|
./scripts/deploy-swarm.sh mosaic
|
|
```
|
|
|
|
### Database Recovery
|
|
|
|
```bash
|
|
# 1. Stop API to prevent writes
|
|
docker service scale mosaic_api=0
|
|
|
|
# 2. Backup current database
|
|
docker run --rm -v mosaic_postgres_data:/data -v $(pwd):/backup alpine \
|
|
tar czf /backup/postgres-emergency-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
|
|
|
|
# 3. Stop postgres
|
|
docker service scale mosaic_postgres=0
|
|
|
|
# 4. Restore from backup
|
|
docker run --rm -v mosaic_postgres_data:/data -v $(pwd):/backup alpine \
|
|
sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data'
|
|
|
|
# 5. Restart postgres
|
|
docker service scale mosaic_postgres=1
|
|
|
|
# 6. Wait for postgres healthy
|
|
sleep 10
|
|
|
|
# 7. Restart API
|
|
docker service scale mosaic_api=1
|
|
```
|
|
|
|
## Health Checks
|
|
|
|
```bash
|
|
# API
|
|
curl http://api.mosaicstack.dev/health
|
|
|
|
# Web
|
|
curl http://mosaic.mosaicstack.dev
|
|
|
|
# Authentik
|
|
curl http://auth.mosaicstack.dev/-/health/live/
|
|
|
|
# Postgres (from API container)
|
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
|
|
sh -c 'nc -zv postgres 5432'
|
|
|
|
# Valkey (from API container)
|
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
|
|
sh -c 'nc -zv valkey 6379'
|
|
```
|