From f8477d5052c67be050c9cb8f7fc704d0c5078901 Mon Sep 17 00:00:00 2001 From: Jason Woltje Date: Sun, 8 Feb 2026 17:12:49 -0600 Subject: [PATCH] docs(swarm): comprehensive Docker Swarm deployment documentation - Update docker-compose.swarm.yml with external Authentik configuration - Comment out Authentik services (using external OIDC provider) - Comment out Authentik volumes - Add header with deployment instructions and current configuration - Create comprehensive SWARM-DEPLOYMENT.md guide - Prerequisites and swarm initialization - Manual OpenBao initialization (critical - no auto-init in swarm) - External service configuration examples - Scaling, updates, rollbacks - Troubleshooting and maintenance procedures - Backup and restore instructions - Update .env.swarm.example - Add note about external vs internal Authentik - Update default OIDC_ISSUER to use https - Clarify which variables are needed for internal Authentik - Update README.md Docker Swarm section - Fix deploy script path (./scripts/deploy-swarm.sh) - Add note about manual OpenBao initialization - Add warning about no profile support in swarm - Update documentation references to docs/ directory - Update documentation cross-references - Add deprecation notice to old DOCKER-SWARM.md - Add deployment guide reference to SWARM-QUICKREF.md - Update DOCKER-COMPOSE-GUIDE.md See Also section Key changes for swarm deployment: - Swarm does NOT support docker-compose profiles - External services must be manually commented out - OpenBao requires manual initialization (no sidecar) - All documentation updated with correct paths Co-Authored-By: Claude Opus 4.6 --- .env.swarm.example | 12 +- README.md | 18 +- docker/DOCKER-COMPOSE-GUIDE.md | 7 +- docker/docker-compose.swarm.yml | 383 ++++++++++++++++++++++++++++ docs/DOCKER-SWARM.md | 2 + docs/SWARM-DEPLOYMENT.md | 434 ++++++++++++++++++++++++++++++++ docs/SWARM-QUICKREF.md | 4 + 7 files changed, 850 insertions(+), 10 deletions(-) create mode 100644 docker/docker-compose.swarm.yml create mode 100644 docs/SWARM-DEPLOYMENT.md diff --git a/.env.swarm.example b/.env.swarm.example index 6bc6fc4..a1a37ca 100644 --- a/.env.swarm.example +++ b/.env.swarm.example @@ -54,18 +54,24 @@ KNOWLEDGE_CACHE_TTL=300 # ====================== # Authentication (Authentik OIDC) # ====================== +# NOTE: Authentik services are COMMENTED OUT in docker-compose.swarm.yml by default +# Uncomment those services if you want to run Authentik internally +# Otherwise, use external Authentik by configuring OIDC_* variables below + +# External Authentik Configuration (default) OIDC_ENABLED=true -OIDC_ISSUER=http://auth.mosaicstack.dev/application/o/mosaic-stack/ +OIDC_ISSUER=https://auth.example.com/application/o/mosaic-stack/ OIDC_CLIENT_ID=your-client-id-here OIDC_CLIENT_SECRET=your-client-secret-here -OIDC_REDIRECT_URI=http://api.mosaicstack.dev/auth/callback/authentik +OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik +# Internal Authentik Configuration (only needed if uncommenting Authentik services) # Authentik PostgreSQL Database AUTHENTIK_POSTGRES_USER=authentik AUTHENTIK_POSTGRES_PASSWORD=REPLACE_WITH_SECURE_PASSWORD AUTHENTIK_POSTGRES_DB=authentik -# Authentik Configuration +# Authentik Server Configuration AUTHENTIK_SECRET_KEY=REPLACE_WITH_RANDOM_SECRET_MINIMUM_50_CHARS AUTHENTIK_ERROR_REPORTING=false AUTHENTIK_BOOTSTRAP_PASSWORD=REPLACE_WITH_SECURE_PASSWORD diff --git a/README.md b/README.md index 7a1d077..035bc04 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,7 @@ Deploy to a Docker Swarm cluster with integrated Traefik reverse proxy: ```bash # 1. Initialize swarm (if not already done) -docker swarm init +docker swarm init --advertise-addr # 2. Create Traefik network docker network create --driver=overlay traefik-public @@ -174,16 +174,19 @@ cp .env.swarm.example .env nano .env # Configure domains, passwords, API keys # 4. Deploy stack -./deploy-swarm.sh mosaic +./scripts/deploy-swarm.sh mosaic # 5. Check deployment status docker stack services mosaic docker stack ps mosaic +# 6. CRITICAL: Initialize OpenBao manually (see docs) +# Unlike docker-compose, swarm requires manual OpenBao initialization + # Access services via Traefik # Web: http://mosaic.mosaicstack.dev # API: http://api.mosaicstack.dev -# Auth: http://auth.mosaicstack.dev +# Auth: http://auth.mosaicstack.dev (if using bundled Authentik) ``` **Key features:** @@ -193,8 +196,15 @@ docker stack ps mosaic - Built-in health checks and rolling updates - Horizontal scaling for web and API services - Zero-downtime deployments +- Service orchestration across multiple nodes -See [Docker Swarm Deployment Guide](DOCKER-SWARM.md) and [Quick Reference](SWARM-QUICKREF.md) for complete documentation. +**Important Notes:** + +- Swarm does NOT support docker-compose profiles +- To use external services (PostgreSQL, Authentik, etc.), manually comment them out in `docker/docker-compose.swarm.yml` +- OpenBao requires manual initialization (no auto-init sidecar in swarm mode) + +See [Docker Swarm Deployment Guide](docs/SWARM-DEPLOYMENT.md) and [Quick Reference](docs/SWARM-QUICKREF.md) for complete documentation. ## Project Structure diff --git a/docker/DOCKER-COMPOSE-GUIDE.md b/docker/DOCKER-COMPOSE-GUIDE.md index 9d8e295..13f2e89 100644 --- a/docker/DOCKER-COMPOSE-GUIDE.md +++ b/docker/DOCKER-COMPOSE-GUIDE.md @@ -260,6 +260,7 @@ Bundles database/cache/AI locally, uses external auth/secrets. ## See Also -- [Deployment Guide](docs/DOCKER.md) - Full Docker deployment documentation -- [Configuration Guide](docs/CONFIGURATION.md) - Environment variable reference -- [CI/CD Pipeline](.woodpecker.yml) - Automated build and deployment +- [Docker Swarm Deployment](../docs/SWARM-DEPLOYMENT.md) - Production deployment with Docker Swarm +- [Swarm Quick Reference](../docs/SWARM-QUICKREF.md) - Common swarm commands +- [Configuration Guide](../docs/CONFIGURATION.md) - Environment variable reference +- [OpenBao Documentation](../docs/OPENBAO.md) - Secrets management setup diff --git a/docker/docker-compose.swarm.yml b/docker/docker-compose.swarm.yml new file mode 100644 index 0000000..60babf3 --- /dev/null +++ b/docker/docker-compose.swarm.yml @@ -0,0 +1,383 @@ +# ============================================== +# Mosaic Stack - Docker Swarm Deployment +# ============================================== +# +# IMPORTANT: Docker Swarm does NOT support docker-compose profiles +# To disable services (e.g., for external alternatives), manually comment them out +# +# Current Configuration: +# - PostgreSQL: ENABLED (internal) +# - Valkey: ENABLED (internal) +# - OpenBao: ENABLED (internal) +# - Authentik: DISABLED (commented out - using external OIDC) +# - Ollama: ENABLED (internal) +# +# For detailed deployment instructions, see: +# docs/SWARM-DEPLOYMENT.md +# +# Quick Start: +# 1. cp .env.swarm.example .env +# 2. nano .env # Configure environment +# 3. ./scripts/deploy-swarm.sh mosaic +# 4. Initialize OpenBao manually (see docs/SWARM-DEPLOYMENT.md) +# +# ============================================== + +services: + # ====================== + # PostgreSQL Database + # ====================== + postgres: + image: git.mosaicstack.dev/mosaic/stack-postgres:${IMAGE_TAG:-latest} + env_file: .env + environment: + POSTGRES_USER: ${POSTGRES_USER:-mosaic} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mosaic_dev_password} + POSTGRES_DB: ${POSTGRES_DB:-mosaic} + POSTGRES_SHARED_BUFFERS: ${POSTGRES_SHARED_BUFFERS:-256MB} + POSTGRES_EFFECTIVE_CACHE_SIZE: ${POSTGRES_EFFECTIVE_CACHE_SIZE:-1GB} + POSTGRES_MAX_CONNECTIONS: ${POSTGRES_MAX_CONNECTIONS:-100} + volumes: + - postgres_data:/var/lib/postgresql/data + - ./docker/postgres/init-scripts:/docker-entrypoint-initdb.d:ro + healthcheck: + test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-mosaic} -d ${POSTGRES_DB:-mosaic}"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - internal + deploy: + restart_policy: + condition: on-failure + + # ====================== + # Valkey Cache + # ====================== + valkey: + image: valkey/valkey:8-alpine + env_file: .env + command: + - valkey-server + - --maxmemory ${VALKEY_MAXMEMORY:-256mb} + - --maxmemory-policy allkeys-lru + - --appendonly yes + volumes: + - valkey_data:/data + healthcheck: + test: ["CMD", "valkey-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 10s + networks: + - internal + deploy: + restart_policy: + condition: on-failure + + # ====================== + # OpenBao Secrets Vault + # ====================== + openbao: + image: git.mosaicstack.dev/mosaic/stack-openbao:${IMAGE_TAG:-latest} + env_file: .env + environment: + OPENBAO_ADDR: ${OPENBAO_ADDR:-http://0.0.0.0:8200} + OPENBAO_DEV_ROOT_TOKEN_ID: ${OPENBAO_DEV_ROOT_TOKEN_ID:-root} + volumes: + - openbao_data:/openbao/data + - openbao_logs:/openbao/logs + - openbao_init:/openbao/init + cap_add: + - IPC_LOCK + healthcheck: + test: + ["CMD", "wget", "--spider", "--quiet", "http://localhost:8200/v1/sys/health?standbyok=true"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - internal + deploy: + restart_policy: + condition: on-failure + + # ====================== + # Authentik - COMMENTED OUT (Using External Authentik) + # ====================== + # Uncomment these services if you want to run Authentik internally + # For external Authentik, configure OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET in .env + # + # authentik-postgres: + # image: postgres:17-alpine + # env_file: .env + # environment: + # POSTGRES_USER: ${AUTHENTIK_POSTGRES_USER:-authentik} + # POSTGRES_PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password} + # POSTGRES_DB: ${AUTHENTIK_POSTGRES_DB:-authentik} + # volumes: + # - authentik_postgres_data:/var/lib/postgresql/data + # healthcheck: + # test: ["CMD-SHELL", "pg_isready -U ${AUTHENTIK_POSTGRES_USER:-authentik}"] + # interval: 10s + # timeout: 5s + # retries: 5 + # start_period: 20s + # networks: + # - internal + # deploy: + # restart_policy: + # condition: on-failure + # + # authentik-redis: + # image: valkey/valkey:8-alpine + # env_file: .env + # command: valkey-server --save 60 1 --loglevel warning + # volumes: + # - authentik_redis_data:/data + # healthcheck: + # test: ["CMD", "valkey-cli", "ping"] + # interval: 10s + # timeout: 5s + # retries: 5 + # start_period: 10s + # networks: + # - internal + # deploy: + # restart_policy: + # condition: on-failure + # + # authentik-server: + # image: ghcr.io/goauthentik/server:2024.12.1 + # env_file: .env + # command: server + # environment: + # AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:-change-this-to-a-random-secret} + # AUTHENTIK_ERROR_REPORTING__ENABLED: ${AUTHENTIK_ERROR_REPORTING:-false} + # AUTHENTIK_POSTGRESQL__HOST: authentik-postgres + # AUTHENTIK_POSTGRESQL__PORT: 5432 + # AUTHENTIK_POSTGRESQL__NAME: ${AUTHENTIK_POSTGRES_DB:-authentik} + # AUTHENTIK_POSTGRESQL__USER: ${AUTHENTIK_POSTGRES_USER:-authentik} + # AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password} + # AUTHENTIK_REDIS__HOST: authentik-redis + # AUTHENTIK_REDIS__PORT: 6379 + # AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD:-admin} + # AUTHENTIK_BOOTSTRAP_EMAIL: ${AUTHENTIK_BOOTSTRAP_EMAIL:-admin@localhost} + # AUTHENTIK_COOKIE_DOMAIN: ${AUTHENTIK_COOKIE_DOMAIN:-.mosaicstack.dev} + # volumes: + # - authentik_media:/media + # - authentik_templates:/templates + # healthcheck: + # test: + # [ + # "CMD", + # "wget", + # "--no-verbose", + # "--tries=1", + # "--spider", + # "http://localhost:9000/-/health/live/", + # ] + # interval: 30s + # timeout: 10s + # retries: 3 + # start_period: 90s + # networks: + # - internal + # - traefik-public + # deploy: + # restart_policy: + # condition: on-failure + # labels: + # - "traefik.enable=true" + # - "traefik.http.routers.mosaic-auth.rule=Host(`${MOSAIC_AUTH_DOMAIN:-auth.mosaicstack.dev}`)" + # - "traefik.http.routers.mosaic-auth.entrypoints=web" + # - "traefik.http.services.mosaic-auth.loadbalancer.server.port=9000" + # + # authentik-worker: + # image: ghcr.io/goauthentik/server:2024.12.1 + # env_file: .env + # command: worker + # environment: + # AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:-change-this-to-a-random-secret} + # AUTHENTIK_ERROR_REPORTING__ENABLED: ${AUTHENTIK_ERROR_REPORTING:-false} + # AUTHENTIK_POSTGRESQL__HOST: authentik-postgres + # AUTHENTIK_POSTGRESQL__PORT: 5432 + # AUTHENTIK_POSTGRESQL__NAME: ${AUTHENTIK_POSTGRES_DB:-authentik} + # AUTHENTIK_POSTGRESQL__USER: ${AUTHENTIK_POSTGRES_USER:-authentik} + # AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password} + # AUTHENTIK_REDIS__HOST: authentik-redis + # AUTHENTIK_REDIS__PORT: 6379 + # volumes: + # - authentik_media:/media + # - authentik_certs:/certs + # - authentik_templates:/templates + # networks: + # - internal + # deploy: + # restart_policy: + # condition: on-failure + + # ====================== + # Ollama (Optional AI Service) + # ====================== + ollama: + image: ollama/ollama:latest + env_file: .env + volumes: + - ollama_data:/root/.ollama + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + networks: + - internal + deploy: + restart_policy: + condition: on-failure + + # ====================== + # Mosaic API + # ====================== + api: + image: git.mosaicstack.dev/mosaic/stack-api:${IMAGE_TAG:-latest} + env_file: .env + environment: + NODE_ENV: production + PORT: ${API_PORT:-3001} + API_HOST: ${API_HOST:-0.0.0.0} + DATABASE_URL: postgresql://${POSTGRES_USER:-mosaic}:${POSTGRES_PASSWORD:-mosaic_dev_password}@postgres:5432/${POSTGRES_DB:-mosaic} + VALKEY_URL: redis://valkey:6379 + OIDC_ISSUER: ${OIDC_ISSUER} + OIDC_CLIENT_ID: ${OIDC_CLIENT_ID} + OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET} + OIDC_REDIRECT_URI: ${OIDC_REDIRECT_URI:-http://localhost:3001/auth/callback} + JWT_SECRET: ${JWT_SECRET:-change-this-to-a-random-secret} + JWT_EXPIRATION: ${JWT_EXPIRATION:-24h} + OLLAMA_ENDPOINT: ${OLLAMA_ENDPOINT:-http://ollama:11434} + OPENBAO_ADDR: ${OPENBAO_ADDR:-http://openbao:8200} + ENCRYPTION_KEY: ${ENCRYPTION_KEY} + healthcheck: + test: + [ + "CMD-SHELL", + 'node -e "require(''http'').get(''http://localhost:${API_PORT:-3001}/health'', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"', + ] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + networks: + - internal + - traefik-public + deploy: + restart_policy: + condition: on-failure + labels: + - "traefik.enable=true" + - "traefik.http.routers.mosaic-api.rule=Host(`${MOSAIC_API_DOMAIN:-api.mosaicstack.dev}`)" + - "traefik.http.routers.mosaic-api.entrypoints=web" + - "traefik.http.services.mosaic-api.loadbalancer.server.port=${API_PORT:-3001}" + + # ====================== + # Mosaic Orchestrator + # ====================== + orchestrator: + image: git.mosaicstack.dev/mosaic/stack-orchestrator:${IMAGE_TAG:-latest} + env_file: .env + user: "1000:1000" + environment: + NODE_ENV: production + ORCHESTRATOR_PORT: 3001 + VALKEY_URL: redis://valkey:6379 + CLAUDE_API_KEY: ${CLAUDE_API_KEY} + DOCKER_SOCKET: /var/run/docker.sock + GIT_USER_NAME: "Mosaic Orchestrator" + GIT_USER_EMAIL: "orchestrator@mosaicstack.dev" + KILLSWITCH_ENABLED: "true" + SANDBOX_ENABLED: "true" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - orchestrator_workspace:/workspace + healthcheck: + test: + ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3001/health || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + networks: + - internal + # Note: security_opt not supported in swarm mode + # Security hardening done via cap_drop/cap_add + cap_drop: + - ALL + cap_add: + - NET_BIND_SERVICE + tmpfs: + - /tmp:noexec,nosuid,size=100m + deploy: + restart_policy: + condition: on-failure + + # ====================== + # Mosaic Web + # ====================== + web: + image: git.mosaicstack.dev/mosaic/stack-web:${IMAGE_TAG:-latest} + env_file: .env + environment: + NODE_ENV: production + PORT: ${WEB_PORT:-3000} + NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL:-http://localhost:3001} + healthcheck: + test: + [ + "CMD-SHELL", + 'node -e "require(''http'').get(''http://localhost:${WEB_PORT:-3000}'', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"', + ] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + networks: + - traefik-public + deploy: + restart_policy: + condition: on-failure + labels: + - "traefik.enable=true" + - "traefik.http.routers.mosaic-web.rule=Host(`${MOSAIC_WEB_DOMAIN:-mosaic.mosaicstack.dev}`)" + - "traefik.http.routers.mosaic-web.entrypoints=web" + - "traefik.http.services.mosaic-web.loadbalancer.server.port=${WEB_PORT:-3000}" + +# ====================== +# Volumes +# ====================== +volumes: + postgres_data: + valkey_data: + openbao_data: + openbao_logs: + openbao_init: + # Authentik volumes - commented out (using external Authentik) + # authentik_postgres_data: + # authentik_redis_data: + # authentik_media: + # authentik_certs: + # authentik_templates: + ollama_data: + orchestrator_workspace: + +# ====================== +# Networks +# ====================== +networks: + internal: + driver: overlay + traefik-public: + external: true diff --git a/docs/DOCKER-SWARM.md b/docs/DOCKER-SWARM.md index acd8e6a..77c69ee 100644 --- a/docs/DOCKER-SWARM.md +++ b/docs/DOCKER-SWARM.md @@ -1,5 +1,7 @@ # Mosaic Stack - Docker Swarm Deployment +**⚠️ This guide has been superseded. Please see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md) for the complete, up-to-date deployment guide.** + This guide covers deploying Mosaic Stack to a Docker Swarm cluster with Traefik reverse proxy integration. ## Prerequisites diff --git a/docs/SWARM-DEPLOYMENT.md b/docs/SWARM-DEPLOYMENT.md new file mode 100644 index 0000000..551de8d --- /dev/null +++ b/docs/SWARM-DEPLOYMENT.md @@ -0,0 +1,434 @@ +# Docker Swarm Deployment Guide + +## Prerequisites + +### 1. Initialize Docker Swarm + +```bash +# On the manager node (10.1.1.90) +docker swarm init --advertise-addr 10.1.1.90 + +# For multi-node swarm, join worker nodes: +# docker swarm join --token 10.1.1.90:2377 +``` + +### 2. Create External Networks + +```bash +# Create traefik-public network (required for ingress) +docker network create --driver=overlay traefik-public +``` + +### 3. Verify Swarm Status + +```bash +docker node ls # Should show nodes in Ready state +docker network ls # Should show traefik-public overlay network +``` + +## Configuration + +### 1. Create Environment File + +```bash +cd /opt/mosaic/stack # Or your deployment directory + +# Copy example configuration +cp .env.swarm.example .env + +# Edit configuration +nano .env +``` + +**Required variables:** + +- `POSTGRES_PASSWORD` - Strong password for PostgreSQL +- `JWT_SECRET` - Random secret (min 32 chars) +- `BETTER_AUTH_SECRET` - Random secret (min 32 chars) +- `ENCRYPTION_KEY` - 64-char hex string (generate with `openssl rand -hex 32`) +- `OIDC_CLIENT_ID` - From your Authentik/OIDC provider +- `OIDC_CLIENT_SECRET` - From your Authentik/OIDC provider +- `OIDC_ISSUER` - Your OIDC provider URL (must end with `/`) +- `IMAGE_TAG` - `dev` or `latest` or specific commit SHA + +### 2. Configure for External Services (Optional) + +**For external Authentik:** Edit `docker-compose.swarm.yml` and comment out: + +```yaml +# Comment out these services if using external Authentik +# authentik-postgres: +# ... +# authentik-redis: +# ... +# authentik-server: +# ... +# authentik-worker: +# ... +``` + +**For external Ollama:** Update `.env`: + +```bash +OLLAMA_ENDPOINT=http://your-ollama-server:11434 +``` + +Then comment out in `docker-compose.swarm.yml`: + +```yaml +# ollama: +# ... +``` + +**For external PostgreSQL/Valkey:** Comment them out and update `.env`: + +```bash +DATABASE_URL=postgresql://user:pass@external-db:5432/mosaic +VALKEY_URL=redis://external-cache:6379 +``` + +### 3. Registry Authentication + +```bash +# Login to Gitea container registry +docker login git.mosaicstack.dev + +# Enter your Gitea username and password/token +``` + +## Deployment + +### Deploy the Stack + +```bash +cd /opt/mosaic/stack + +# Using the deploy script (recommended) +./scripts/deploy-swarm.sh mosaic + +# Or manually +IMAGE_TAG=dev docker stack deploy \ + -c docker/docker-compose.swarm.yml \ + --with-registry-auth mosaic +``` + +### Verify Deployment + +```bash +# Check stack services +docker stack services mosaic + +# Expected output - all services should show 1/1 replicas: +# ID NAME MODE REPLICAS IMAGE +# abc123 mosaic_postgres replicated 1/1 git.mosaicstack.dev/mosaic/stack-postgres:dev +# def456 mosaic_valkey replicated 1/1 valkey/valkey:8-alpine +# ghi789 mosaic_openbao replicated 1/1 git.mosaicstack.dev/mosaic/stack-openbao:dev +# jkl012 mosaic_api replicated 1/1 git.mosaicstack.dev/mosaic/stack-api:dev +# mno345 mosaic_web replicated 1/1 git.mosaicstack.dev/mosaic/stack-web:dev +# pqr678 mosaic_orchestrator replicated 1/1 git.mosaicstack.dev/mosaic/stack-orchestrator:dev + +# Check detailed task status +docker stack ps mosaic --no-trunc + +# Follow logs for specific services +docker service logs mosaic_api --follow +docker service logs mosaic_web --follow +``` + +## Post-Deployment Configuration + +### 1. Initialize OpenBao (CRITICAL) + +**Important:** Unlike docker-compose, the swarm file does NOT include the `openbao-init` sidecar. You must manually initialize OpenBao: + +```bash +# Wait for OpenBao to be healthy +sleep 30 + +# Get the OpenBao container ID +OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao) + +# Initialize OpenBao +docker exec $OPENBAO_CONTAINER bao operator init -key-shares=1 -key-threshold=1 + +# Save the output - you'll need the Unseal Key and Root Token! +# Example output: +# Unseal Key 1: abc123... +# Initial Root Token: xyz789... + +# Unseal OpenBao +docker exec $OPENBAO_CONTAINER bao operator unseal + +# Enable Transit engine +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao secrets enable transit + +# Create encryption key +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao write -f transit/keys/mosaic-credentials + +# Create AppRole for API +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao auth enable approle + +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao write auth/approle/role/mosaic-api \ + token_policies="default" \ + token_ttl=1h \ + token_max_ttl=4h + +# Get Role ID and Secret ID +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao read auth/approle/role/mosaic-api/role-id + +docker exec -e BAO_TOKEN= $OPENBAO_CONTAINER \ + bao write -f auth/approle/role/mosaic-api/secret-id + +# Update API service with AppRole credentials (optional - or store in volume) +docker service update mosaic_api \ + --env-add OPENBAO_ROLE_ID= \ + --env-add OPENBAO_SECRET_ID= +``` + +**Automated Alternative:** Create an init script and run it as a Docker container: + +```bash +# See docker/openbao/init.sh for reference +# Create a one-time task that runs initialization +docker service create --name openbao-init \ + --network mosaic_internal \ + --restart-condition=none \ + --mount type=volume,source=mosaic_openbao_init,target=/openbao/init \ + git.mosaicstack.dev/mosaic/stack-openbao:dev \ + /openbao/init.sh +``` + +### 2. Run Database Migrations + +```bash +# Get API container ID +API_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) + +# Run Prisma migrations +docker exec $API_CONTAINER pnpm prisma migrate deploy + +# Seed initial data (optional) +docker exec $API_CONTAINER pnpm prisma db seed +``` + +### 3. Verify Services + +```bash +# Check API health +curl http://api.mosaicstack.dev/health + +# Check Web UI +curl http://mosaic.mosaicstack.dev + +# Check Authentik (if bundled) +curl http://auth.mosaicstack.dev/-/health/live/ + +# Check OpenBao status +OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao) +docker exec $OPENBAO_CONTAINER bao status +``` + +## Scaling Services + +```bash +# Scale web frontend +docker service scale mosaic_web=3 + +# Scale API +docker service scale mosaic_api=2 + +# Verify scaling +docker service ls +``` + +## Updates and Rollbacks + +### Update Services to New Image Tag + +```bash +# Update all services to new tag +IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic + +# Or update individual service +docker service update --image git.mosaicstack.dev/mosaic/stack-api:latest mosaic_api +``` + +### Rollback Service + +```bash +# Rollback to previous version +docker service rollback mosaic_api +``` + +## Maintenance + +### View Logs + +```bash +# API logs +docker service logs mosaic_api --tail 100 --follow + +# Web logs +docker service logs mosaic_web --tail 100 --follow + +# PostgreSQL logs +docker service logs mosaic_postgres --tail 100 + +# All services +for service in $(docker stack services mosaic --format '{{.Name}}'); do + echo "=== $service ===" + docker service logs $service --tail 20 +done +``` + +### Backup Volumes + +```bash +# Backup PostgreSQL +docker run --rm \ + -v mosaic_postgres_data:/data \ + -v $(pwd):/backup \ + alpine tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data . + +# Backup OpenBao +docker run --rm \ + -v mosaic_openbao_data:/data \ + -v $(pwd):/backup \ + alpine tar czf /backup/openbao-$(date +%Y%m%d-%H%M%S).tar.gz -C /data . +``` + +### Restore Volumes + +```bash +# Stop services first +docker service scale mosaic_postgres=0 + +# Restore +docker run --rm \ + -v mosaic_postgres_data:/data \ + -v $(pwd):/backup \ + alpine sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data' + +# Restart service +docker service scale mosaic_postgres=1 +``` + +## Troubleshooting + +### Service Won't Start + +```bash +# Check service tasks for errors +docker service ps mosaic_api --no-trunc + +# View detailed logs +docker service logs mosaic_api --tail 100 + +# Inspect service configuration +docker service inspect mosaic_api +``` + +### Network Issues + +```bash +# Verify overlay networks +docker network ls --filter driver=overlay + +# Inspect traefik-public network +docker network inspect traefik-public + +# Check service network connectivity +docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \ + ping postgres +``` + +### OpenBao Issues + +```bash +# Check OpenBao status +OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao) +docker exec $OPENBAO_CONTAINER bao status + +# Re-unseal if sealed +docker exec $OPENBAO_CONTAINER bao operator unseal + +# Check logs +docker service logs mosaic_openbao --tail 100 +``` + +### Complete Stack Restart + +```bash +# Remove stack (keeps volumes) +docker stack rm mosaic + +# Wait for cleanup +sleep 30 + +# Redeploy +./scripts/deploy-swarm.sh mosaic +``` + +## External Services Configuration + +### Using External Authentik + +1. Comment out Authentik services in `docker-compose.swarm.yml` +2. Configure in `.env`: + ```bash + OIDC_ISSUER=https://auth.diversecanvas.com/application/o/mosaic-stack/ + OIDC_CLIENT_ID=your-client-id + OIDC_CLIENT_SECRET=your-client-secret + OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik + ``` + +### Using External PostgreSQL + +1. Comment out `postgres` service in `docker-compose.swarm.yml` +2. Configure in `.env`: + ```bash + DATABASE_URL=postgresql://user:pass@rds.amazonaws.com:5432/mosaic + ``` + +### Using External Valkey/Redis + +1. Comment out `valkey` service in `docker-compose.swarm.yml` +2. Configure in `.env`: + ```bash + VALKEY_URL=redis://elasticache.amazonaws.com:6379 + ``` + +### Using External OpenBao/Vault + +1. Comment out `openbao` service in `docker-compose.swarm.yml` +2. Configure in `.env`: + ```bash + OPENBAO_ADDR=https://vault.example.com:8200 + OPENBAO_ROLE_ID=your-role-id + OPENBAO_SECRET_ID=your-secret-id + ``` + +## Security Considerations + +1. **Never commit `.env` to version control** - Contains secrets +2. **Use strong passwords** - Generate with `openssl rand -base64 32` +3. **Store OpenBao unseal keys securely** - Required for disaster recovery +4. **Enable TLS/SSL in production** - Configure Traefik with Let's Encrypt +5. **Regular backups** - Backup volumes and OpenBao keys +6. **Monitor logs** - Watch for security events and errors +7. **Update regularly** - Pull latest images with `IMAGE_TAG=latest` + +## Quick Reference + +See `docs/SWARM-QUICKREF.md` for quick command reference. + +## See Also + +- [Docker Compose Guide](../docker/DOCKER-COMPOSE-GUIDE.md) - Regular docker-compose deployment +- [OpenBao Documentation](OPENBAO.md) - Secrets management setup +- [Configuration Guide](CONFIGURATION.md) - All environment variables diff --git a/docs/SWARM-QUICKREF.md b/docs/SWARM-QUICKREF.md index ae8fb66..e3ab900 100644 --- a/docs/SWARM-QUICKREF.md +++ b/docs/SWARM-QUICKREF.md @@ -1,5 +1,9 @@ # Docker Swarm Quick Reference +**For complete deployment instructions, see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md)** + +This is a quick reference for common Docker Swarm commands after your stack is deployed. + ## Initial Setup ```bash