docs(swarm): comprehensive Docker Swarm deployment documentation

- Update docker-compose.swarm.yml with external Authentik configuration
  - Comment out Authentik services (using external OIDC provider)
  - Comment out Authentik volumes
  - Add header with deployment instructions and current configuration

- Create comprehensive SWARM-DEPLOYMENT.md guide
  - Prerequisites and swarm initialization
  - Manual OpenBao initialization (critical - no auto-init in swarm)
  - External service configuration examples
  - Scaling, updates, rollbacks
  - Troubleshooting and maintenance procedures
  - Backup and restore instructions

- Update .env.swarm.example
  - Add note about external vs internal Authentik
  - Update default OIDC_ISSUER to use https
  - Clarify which variables are needed for internal Authentik

- Update README.md Docker Swarm section
  - Fix deploy script path (./scripts/deploy-swarm.sh)
  - Add note about manual OpenBao initialization
  - Add warning about no profile support in swarm
  - Update documentation references to docs/ directory

- Update documentation cross-references
  - Add deprecation notice to old DOCKER-SWARM.md
  - Add deployment guide reference to SWARM-QUICKREF.md
  - Update DOCKER-COMPOSE-GUIDE.md See Also section

Key changes for swarm deployment:
- Swarm does NOT support docker-compose profiles
- External services must be manually commented out
- OpenBao requires manual initialization (no sidecar)
- All documentation updated with correct paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.env.swarm.example

@@ -54,18 +54,24 @@ KNOWLEDGE_CACHE_TTL=300
 # ======================
 # Authentication (Authentik OIDC)
 # ======================
+# NOTE: Authentik services are COMMENTED OUT in docker-compose.swarm.yml by default
+# Uncomment those services if you want to run Authentik internally
+# Otherwise, use external Authentik by configuring OIDC_* variables below
+
+# External Authentik Configuration (default)
 OIDC_ENABLED=true
-OIDC_ISSUER=http://auth.mosaicstack.dev/application/o/mosaic-stack/
+OIDC_ISSUER=https://auth.example.com/application/o/mosaic-stack/
 OIDC_CLIENT_ID=your-client-id-here
 OIDC_CLIENT_SECRET=your-client-secret-here
-OIDC_REDIRECT_URI=http://api.mosaicstack.dev/auth/callback/authentik
+OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik
 
+# Internal Authentik Configuration (only needed if uncommenting Authentik services)
 # Authentik PostgreSQL Database
 AUTHENTIK_POSTGRES_USER=authentik
 AUTHENTIK_POSTGRES_PASSWORD=REPLACE_WITH_SECURE_PASSWORD
 AUTHENTIK_POSTGRES_DB=authentik
 
-# Authentik Configuration
+# Authentik Server Configuration
 AUTHENTIK_SECRET_KEY=REPLACE_WITH_RANDOM_SECRET_MINIMUM_50_CHARS
 AUTHENTIK_ERROR_REPORTING=false
 AUTHENTIK_BOOTSTRAP_PASSWORD=REPLACE_WITH_SECURE_PASSWORD

README.md

@@ -164,7 +164,7 @@ Deploy to a Docker Swarm cluster with integrated Traefik reverse proxy:
 
 ```bash
 # 1. Initialize swarm (if not already done)
-docker swarm init
+docker swarm init --advertise-addr <your-ip>
 
 # 2. Create Traefik network
 docker network create --driver=overlay traefik-public
@@ -174,16 +174,19 @@ cp .env.swarm.example .env
 nano .env  # Configure domains, passwords, API keys
 
 # 4. Deploy stack
-./deploy-swarm.sh mosaic
+./scripts/deploy-swarm.sh mosaic
 
 # 5. Check deployment status
 docker stack services mosaic
 docker stack ps mosaic
 
+# 6. CRITICAL: Initialize OpenBao manually (see docs)
+# Unlike docker-compose, swarm requires manual OpenBao initialization
+
 # Access services via Traefik
 # Web:  http://mosaic.mosaicstack.dev
 # API:  http://api.mosaicstack.dev
-# Auth: http://auth.mosaicstack.dev
+# Auth: http://auth.mosaicstack.dev (if using bundled Authentik)
 ```
 
 **Key features:**
@@ -193,8 +196,15 @@ docker stack ps mosaic
 - Built-in health checks and rolling updates
 - Horizontal scaling for web and API services
 - Zero-downtime deployments
+- Service orchestration across multiple nodes
 
-See [Docker Swarm Deployment Guide](DOCKER-SWARM.md) and [Quick Reference](SWARM-QUICKREF.md) for complete documentation.
+**Important Notes:**
+
+- Swarm does NOT support docker-compose profiles
+- To use external services (PostgreSQL, Authentik, etc.), manually comment them out in `docker/docker-compose.swarm.yml`
+- OpenBao requires manual initialization (no auto-init sidecar in swarm mode)
+
+See [Docker Swarm Deployment Guide](docs/SWARM-DEPLOYMENT.md) and [Quick Reference](docs/SWARM-QUICKREF.md) for complete documentation.
 
 ## Project Structure
 

docker/DOCKER-COMPOSE-GUIDE.md

@@ -260,6 +260,7 @@ Bundles database/cache/AI locally, uses external auth/secrets.
 
 ## See Also
 
-- [Deployment Guide](docs/DOCKER.md) - Full Docker deployment documentation
+- [Docker Swarm Deployment](../docs/SWARM-DEPLOYMENT.md) - Production deployment with Docker Swarm
-- [Configuration Guide](docs/CONFIGURATION.md) - Environment variable reference
+- [Swarm Quick Reference](../docs/SWARM-QUICKREF.md) - Common swarm commands
-- [CI/CD Pipeline](.woodpecker.yml) - Automated build and deployment
+- [Configuration Guide](../docs/CONFIGURATION.md) - Environment variable reference
+- [OpenBao Documentation](../docs/OPENBAO.md) - Secrets management setup

docker/docker-compose.swarm.yml

@@ -0,0 +1,383 @@
+# ==============================================
+# Mosaic Stack - Docker Swarm Deployment
+# ==============================================
+#
+# IMPORTANT: Docker Swarm does NOT support docker-compose profiles
+# To disable services (e.g., for external alternatives), manually comment them out
+#
+# Current Configuration:
+#   - PostgreSQL: ENABLED (internal)
+#   - Valkey: ENABLED (internal)
+#   - OpenBao: ENABLED (internal)
+#   - Authentik: DISABLED (commented out - using external OIDC)
+#   - Ollama: ENABLED (internal)
+#
+# For detailed deployment instructions, see:
+#   docs/SWARM-DEPLOYMENT.md
+#
+# Quick Start:
+#   1. cp .env.swarm.example .env
+#   2. nano .env  # Configure environment
+#   3. ./scripts/deploy-swarm.sh mosaic
+#   4. Initialize OpenBao manually (see docs/SWARM-DEPLOYMENT.md)
+#
+# ==============================================
+
+services:
+  # ======================
+  # PostgreSQL Database
+  # ======================
+  postgres:
+    image: git.mosaicstack.dev/mosaic/stack-postgres:${IMAGE_TAG:-latest}
+    env_file: .env
+    environment:
+      POSTGRES_USER: ${POSTGRES_USER:-mosaic}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mosaic_dev_password}
+      POSTGRES_DB: ${POSTGRES_DB:-mosaic}
+      POSTGRES_SHARED_BUFFERS: ${POSTGRES_SHARED_BUFFERS:-256MB}
+      POSTGRES_EFFECTIVE_CACHE_SIZE: ${POSTGRES_EFFECTIVE_CACHE_SIZE:-1GB}
+      POSTGRES_MAX_CONNECTIONS: ${POSTGRES_MAX_CONNECTIONS:-100}
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+      - ./docker/postgres/init-scripts:/docker-entrypoint-initdb.d:ro
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-mosaic} -d ${POSTGRES_DB:-mosaic}"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    networks:
+      - internal
+    deploy:
+      restart_policy:
+        condition: on-failure
+
+  # ======================
+  # Valkey Cache
+  # ======================
+  valkey:
+    image: valkey/valkey:8-alpine
+    env_file: .env
+    command:
+      - valkey-server
+      - --maxmemory ${VALKEY_MAXMEMORY:-256mb}
+      - --maxmemory-policy allkeys-lru
+      - --appendonly yes
+    volumes:
+      - valkey_data:/data
+    healthcheck:
+      test: ["CMD", "valkey-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 10s
+    networks:
+      - internal
+    deploy:
+      restart_policy:
+        condition: on-failure
+
+  # ======================
+  # OpenBao Secrets Vault
+  # ======================
+  openbao:
+    image: git.mosaicstack.dev/mosaic/stack-openbao:${IMAGE_TAG:-latest}
+    env_file: .env
+    environment:
+      OPENBAO_ADDR: ${OPENBAO_ADDR:-http://0.0.0.0:8200}
+      OPENBAO_DEV_ROOT_TOKEN_ID: ${OPENBAO_DEV_ROOT_TOKEN_ID:-root}
+    volumes:
+      - openbao_data:/openbao/data
+      - openbao_logs:/openbao/logs
+      - openbao_init:/openbao/init
+    cap_add:
+      - IPC_LOCK
+    healthcheck:
+      test:
+        ["CMD", "wget", "--spider", "--quiet", "http://localhost:8200/v1/sys/health?standbyok=true"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    networks:
+      - internal
+    deploy:
+      restart_policy:
+        condition: on-failure
+
+  # ======================
+  # Authentik - COMMENTED OUT (Using External Authentik)
+  # ======================
+  # Uncomment these services if you want to run Authentik internally
+  # For external Authentik, configure OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET in .env
+  #
+  # authentik-postgres:
+  #   image: postgres:17-alpine
+  #   env_file: .env
+  #   environment:
+  #     POSTGRES_USER: ${AUTHENTIK_POSTGRES_USER:-authentik}
+  #     POSTGRES_PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password}
+  #     POSTGRES_DB: ${AUTHENTIK_POSTGRES_DB:-authentik}
+  #   volumes:
+  #     - authentik_postgres_data:/var/lib/postgresql/data
+  #   healthcheck:
+  #     test: ["CMD-SHELL", "pg_isready -U ${AUTHENTIK_POSTGRES_USER:-authentik}"]
+  #     interval: 10s
+  #     timeout: 5s
+  #     retries: 5
+  #     start_period: 20s
+  #   networks:
+  #     - internal
+  #   deploy:
+  #     restart_policy:
+  #       condition: on-failure
+  #
+  # authentik-redis:
+  #   image: valkey/valkey:8-alpine
+  #   env_file: .env
+  #   command: valkey-server --save 60 1 --loglevel warning
+  #   volumes:
+  #     - authentik_redis_data:/data
+  #   healthcheck:
+  #     test: ["CMD", "valkey-cli", "ping"]
+  #     interval: 10s
+  #     timeout: 5s
+  #     retries: 5
+  #     start_period: 10s
+  #   networks:
+  #     - internal
+  #   deploy:
+  #     restart_policy:
+  #       condition: on-failure
+  #
+  # authentik-server:
+  #   image: ghcr.io/goauthentik/server:2024.12.1
+  #   env_file: .env
+  #   command: server
+  #   environment:
+  #     AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:-change-this-to-a-random-secret}
+  #     AUTHENTIK_ERROR_REPORTING__ENABLED: ${AUTHENTIK_ERROR_REPORTING:-false}
+  #     AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
+  #     AUTHENTIK_POSTGRESQL__PORT: 5432
+  #     AUTHENTIK_POSTGRESQL__NAME: ${AUTHENTIK_POSTGRES_DB:-authentik}
+  #     AUTHENTIK_POSTGRESQL__USER: ${AUTHENTIK_POSTGRES_USER:-authentik}
+  #     AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password}
+  #     AUTHENTIK_REDIS__HOST: authentik-redis
+  #     AUTHENTIK_REDIS__PORT: 6379
+  #     AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD:-admin}
+  #     AUTHENTIK_BOOTSTRAP_EMAIL: ${AUTHENTIK_BOOTSTRAP_EMAIL:-admin@localhost}
+  #     AUTHENTIK_COOKIE_DOMAIN: ${AUTHENTIK_COOKIE_DOMAIN:-.mosaicstack.dev}
+  #   volumes:
+  #     - authentik_media:/media
+  #     - authentik_templates:/templates
+  #   healthcheck:
+  #     test:
+  #       [
+  #         "CMD",
+  #         "wget",
+  #         "--no-verbose",
+  #         "--tries=1",
+  #         "--spider",
+  #         "http://localhost:9000/-/health/live/",
+  #       ]
+  #     interval: 30s
+  #     timeout: 10s
+  #     retries: 3
+  #     start_period: 90s
+  #   networks:
+  #     - internal
+  #     - traefik-public
+  #   deploy:
+  #     restart_policy:
+  #       condition: on-failure
+  #     labels:
+  #       - "traefik.enable=true"
+  #       - "traefik.http.routers.mosaic-auth.rule=Host(`${MOSAIC_AUTH_DOMAIN:-auth.mosaicstack.dev}`)"
+  #       - "traefik.http.routers.mosaic-auth.entrypoints=web"
+  #       - "traefik.http.services.mosaic-auth.loadbalancer.server.port=9000"
+  #
+  # authentik-worker:
+  #   image: ghcr.io/goauthentik/server:2024.12.1
+  #   env_file: .env
+  #   command: worker
+  #   environment:
+  #     AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:-change-this-to-a-random-secret}
+  #     AUTHENTIK_ERROR_REPORTING__ENABLED: ${AUTHENTIK_ERROR_REPORTING:-false}
+  #     AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
+  #     AUTHENTIK_POSTGRESQL__PORT: 5432
+  #     AUTHENTIK_POSTGRESQL__NAME: ${AUTHENTIK_POSTGRES_DB:-authentik}
+  #     AUTHENTIK_POSTGRESQL__USER: ${AUTHENTIK_POSTGRES_USER:-authentik}
+  #     AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_POSTGRES_PASSWORD:-authentik_password}
+  #     AUTHENTIK_REDIS__HOST: authentik-redis
+  #     AUTHENTIK_REDIS__PORT: 6379
+  #   volumes:
+  #     - authentik_media:/media
+  #     - authentik_certs:/certs
+  #     - authentik_templates:/templates
+  #   networks:
+  #     - internal
+  #   deploy:
+  #     restart_policy:
+  #       condition: on-failure
+
+  # ======================
+  # Ollama (Optional AI Service)
+  # ======================
+  ollama:
+    image: ollama/ollama:latest
+    env_file: .env
+    volumes:
+      - ollama_data:/root/.ollama
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+    networks:
+      - internal
+    deploy:
+      restart_policy:
+        condition: on-failure
+
+  # ======================
+  # Mosaic API
+  # ======================
+  api:
+    image: git.mosaicstack.dev/mosaic/stack-api:${IMAGE_TAG:-latest}
+    env_file: .env
+    environment:
+      NODE_ENV: production
+      PORT: ${API_PORT:-3001}
+      API_HOST: ${API_HOST:-0.0.0.0}
+      DATABASE_URL: postgresql://${POSTGRES_USER:-mosaic}:${POSTGRES_PASSWORD:-mosaic_dev_password}@postgres:5432/${POSTGRES_DB:-mosaic}
+      VALKEY_URL: redis://valkey:6379
+      OIDC_ISSUER: ${OIDC_ISSUER}
+      OIDC_CLIENT_ID: ${OIDC_CLIENT_ID}
+      OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET}
+      OIDC_REDIRECT_URI: ${OIDC_REDIRECT_URI:-http://localhost:3001/auth/callback}
+      JWT_SECRET: ${JWT_SECRET:-change-this-to-a-random-secret}
+      JWT_EXPIRATION: ${JWT_EXPIRATION:-24h}
+      OLLAMA_ENDPOINT: ${OLLAMA_ENDPOINT:-http://ollama:11434}
+      OPENBAO_ADDR: ${OPENBAO_ADDR:-http://openbao:8200}
+      ENCRYPTION_KEY: ${ENCRYPTION_KEY}
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'node -e "require(''http'').get(''http://localhost:${API_PORT:-3001}/health'', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"',
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    networks:
+      - internal
+      - traefik-public
+    deploy:
+      restart_policy:
+        condition: on-failure
+      labels:
+        - "traefik.enable=true"
+        - "traefik.http.routers.mosaic-api.rule=Host(`${MOSAIC_API_DOMAIN:-api.mosaicstack.dev}`)"
+        - "traefik.http.routers.mosaic-api.entrypoints=web"
+        - "traefik.http.services.mosaic-api.loadbalancer.server.port=${API_PORT:-3001}"
+
+  # ======================
+  # Mosaic Orchestrator
+  # ======================
+  orchestrator:
+    image: git.mosaicstack.dev/mosaic/stack-orchestrator:${IMAGE_TAG:-latest}
+    env_file: .env
+    user: "1000:1000"
+    environment:
+      NODE_ENV: production
+      ORCHESTRATOR_PORT: 3001
+      VALKEY_URL: redis://valkey:6379
+      CLAUDE_API_KEY: ${CLAUDE_API_KEY}
+      DOCKER_SOCKET: /var/run/docker.sock
+      GIT_USER_NAME: "Mosaic Orchestrator"
+      GIT_USER_EMAIL: "orchestrator@mosaicstack.dev"
+      KILLSWITCH_ENABLED: "true"
+      SANDBOX_ENABLED: "true"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - orchestrator_workspace:/workspace
+    healthcheck:
+      test:
+        ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3001/health || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    networks:
+      - internal
+    # Note: security_opt not supported in swarm mode
+    # Security hardening done via cap_drop/cap_add
+    cap_drop:
+      - ALL
+    cap_add:
+      - NET_BIND_SERVICE
+    tmpfs:
+      - /tmp:noexec,nosuid,size=100m
+    deploy:
+      restart_policy:
+        condition: on-failure
+
+  # ======================
+  # Mosaic Web
+  # ======================
+  web:
+    image: git.mosaicstack.dev/mosaic/stack-web:${IMAGE_TAG:-latest}
+    env_file: .env
+    environment:
+      NODE_ENV: production
+      PORT: ${WEB_PORT:-3000}
+      NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL:-http://localhost:3001}
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'node -e "require(''http'').get(''http://localhost:${WEB_PORT:-3000}'', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"',
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    networks:
+      - traefik-public
+    deploy:
+      restart_policy:
+        condition: on-failure
+      labels:
+        - "traefik.enable=true"
+        - "traefik.http.routers.mosaic-web.rule=Host(`${MOSAIC_WEB_DOMAIN:-mosaic.mosaicstack.dev}`)"
+        - "traefik.http.routers.mosaic-web.entrypoints=web"
+        - "traefik.http.services.mosaic-web.loadbalancer.server.port=${WEB_PORT:-3000}"
+
+# ======================
+# Volumes
+# ======================
+volumes:
+  postgres_data:
+  valkey_data:
+  openbao_data:
+  openbao_logs:
+  openbao_init:
+  # Authentik volumes - commented out (using external Authentik)
+  # authentik_postgres_data:
+  # authentik_redis_data:
+  # authentik_media:
+  # authentik_certs:
+  # authentik_templates:
+  ollama_data:
+  orchestrator_workspace:
+
+# ======================
+# Networks
+# ======================
+networks:
+  internal:
+    driver: overlay
+  traefik-public:
+    external: true

docs/DOCKER-SWARM.md

@@ -1,5 +1,7 @@
 # Mosaic Stack - Docker Swarm Deployment
 
+**⚠️ This guide has been superseded. Please see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md) for the complete, up-to-date deployment guide.**
+
 This guide covers deploying Mosaic Stack to a Docker Swarm cluster with Traefik reverse proxy integration.
 
 ## Prerequisites

docs/SWARM-DEPLOYMENT.md

@@ -0,0 +1,434 @@
+# Docker Swarm Deployment Guide
+
+## Prerequisites
+
+### 1. Initialize Docker Swarm
+
+```bash
+# On the manager node (10.1.1.90)
+docker swarm init --advertise-addr 10.1.1.90
+
+# For multi-node swarm, join worker nodes:
+# docker swarm join --token <token> 10.1.1.90:2377
+```
+
+### 2. Create External Networks
+
+```bash
+# Create traefik-public network (required for ingress)
+docker network create --driver=overlay traefik-public
+```
+
+### 3. Verify Swarm Status
+
+```bash
+docker node ls          # Should show nodes in Ready state
+docker network ls       # Should show traefik-public overlay network
+```
+
+## Configuration
+
+### 1. Create Environment File
+
+```bash
+cd /opt/mosaic/stack  # Or your deployment directory
+
+# Copy example configuration
+cp .env.swarm.example .env
+
+# Edit configuration
+nano .env
+```
+
+**Required variables:**
+
+- `POSTGRES_PASSWORD` - Strong password for PostgreSQL
+- `JWT_SECRET` - Random secret (min 32 chars)
+- `BETTER_AUTH_SECRET` - Random secret (min 32 chars)
+- `ENCRYPTION_KEY` - 64-char hex string (generate with `openssl rand -hex 32`)
+- `OIDC_CLIENT_ID` - From your Authentik/OIDC provider
+- `OIDC_CLIENT_SECRET` - From your Authentik/OIDC provider
+- `OIDC_ISSUER` - Your OIDC provider URL (must end with `/`)
+- `IMAGE_TAG` - `dev` or `latest` or specific commit SHA
+
+### 2. Configure for External Services (Optional)
+
+**For external Authentik:** Edit `docker-compose.swarm.yml` and comment out:
+
+```yaml
+# Comment out these services if using external Authentik
+#  authentik-postgres:
+#    ...
+#  authentik-redis:
+#    ...
+#  authentik-server:
+#    ...
+#  authentik-worker:
+#    ...
+```
+
+**For external Ollama:** Update `.env`:
+
+```bash
+OLLAMA_ENDPOINT=http://your-ollama-server:11434
+```
+
+Then comment out in `docker-compose.swarm.yml`:
+
+```yaml
+#  ollama:
+#    ...
+```
+
+**For external PostgreSQL/Valkey:** Comment them out and update `.env`:
+
+```bash
+DATABASE_URL=postgresql://user:pass@external-db:5432/mosaic
+VALKEY_URL=redis://external-cache:6379
+```
+
+### 3. Registry Authentication
+
+```bash
+# Login to Gitea container registry
+docker login git.mosaicstack.dev
+
+# Enter your Gitea username and password/token
+```
+
+## Deployment
+
+### Deploy the Stack
+
+```bash
+cd /opt/mosaic/stack
+
+# Using the deploy script (recommended)
+./scripts/deploy-swarm.sh mosaic
+
+# Or manually
+IMAGE_TAG=dev docker stack deploy \
+  -c docker/docker-compose.swarm.yml \
+  --with-registry-auth mosaic
+```
+
+### Verify Deployment
+
+```bash
+# Check stack services
+docker stack services mosaic
+
+# Expected output - all services should show 1/1 replicas:
+# ID             NAME                         MODE         REPLICAS   IMAGE
+# abc123         mosaic_postgres              replicated   1/1        git.mosaicstack.dev/mosaic/stack-postgres:dev
+# def456         mosaic_valkey                replicated   1/1        valkey/valkey:8-alpine
+# ghi789         mosaic_openbao               replicated   1/1        git.mosaicstack.dev/mosaic/stack-openbao:dev
+# jkl012         mosaic_api                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-api:dev
+# mno345         mosaic_web                   replicated   1/1        git.mosaicstack.dev/mosaic/stack-web:dev
+# pqr678         mosaic_orchestrator          replicated   1/1        git.mosaicstack.dev/mosaic/stack-orchestrator:dev
+
+# Check detailed task status
+docker stack ps mosaic --no-trunc
+
+# Follow logs for specific services
+docker service logs mosaic_api --follow
+docker service logs mosaic_web --follow
+```
+
+## Post-Deployment Configuration
+
+### 1. Initialize OpenBao (CRITICAL)
+
+**Important:** Unlike docker-compose, the swarm file does NOT include the `openbao-init` sidecar. You must manually initialize OpenBao:
+
+```bash
+# Wait for OpenBao to be healthy
+sleep 30
+
+# Get the OpenBao container ID
+OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
+
+# Initialize OpenBao
+docker exec $OPENBAO_CONTAINER bao operator init -key-shares=1 -key-threshold=1
+
+# Save the output - you'll need the Unseal Key and Root Token!
+# Example output:
+# Unseal Key 1: abc123...
+# Initial Root Token: xyz789...
+
+# Unseal OpenBao
+docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key-from-above>
+
+# Enable Transit engine
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao secrets enable transit
+
+# Create encryption key
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao write -f transit/keys/mosaic-credentials
+
+# Create AppRole for API
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao auth enable approle
+
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao write auth/approle/role/mosaic-api \
+    token_policies="default" \
+    token_ttl=1h \
+    token_max_ttl=4h
+
+# Get Role ID and Secret ID
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao read auth/approle/role/mosaic-api/role-id
+
+docker exec -e BAO_TOKEN=<root-token> $OPENBAO_CONTAINER \
+  bao write -f auth/approle/role/mosaic-api/secret-id
+
+# Update API service with AppRole credentials (optional - or store in volume)
+docker service update mosaic_api \
+  --env-add OPENBAO_ROLE_ID=<role-id> \
+  --env-add OPENBAO_SECRET_ID=<secret-id>
+```
+
+**Automated Alternative:** Create an init script and run it as a Docker container:
+
+```bash
+# See docker/openbao/init.sh for reference
+# Create a one-time task that runs initialization
+docker service create --name openbao-init \
+  --network mosaic_internal \
+  --restart-condition=none \
+  --mount type=volume,source=mosaic_openbao_init,target=/openbao/init \
+  git.mosaicstack.dev/mosaic/stack-openbao:dev \
+  /openbao/init.sh
+```
+
+### 2. Run Database Migrations
+
+```bash
+# Get API container ID
+API_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api)
+
+# Run Prisma migrations
+docker exec $API_CONTAINER pnpm prisma migrate deploy
+
+# Seed initial data (optional)
+docker exec $API_CONTAINER pnpm prisma db seed
+```
+
+### 3. Verify Services
+
+```bash
+# Check API health
+curl http://api.mosaicstack.dev/health
+
+# Check Web UI
+curl http://mosaic.mosaicstack.dev
+
+# Check Authentik (if bundled)
+curl http://auth.mosaicstack.dev/-/health/live/
+
+# Check OpenBao status
+OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
+docker exec $OPENBAO_CONTAINER bao status
+```
+
+## Scaling Services
+
+```bash
+# Scale web frontend
+docker service scale mosaic_web=3
+
+# Scale API
+docker service scale mosaic_api=2
+
+# Verify scaling
+docker service ls
+```
+
+## Updates and Rollbacks
+
+### Update Services to New Image Tag
+
+```bash
+# Update all services to new tag
+IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic
+
+# Or update individual service
+docker service update --image git.mosaicstack.dev/mosaic/stack-api:latest mosaic_api
+```
+
+### Rollback Service
+
+```bash
+# Rollback to previous version
+docker service rollback mosaic_api
+```
+
+## Maintenance
+
+### View Logs
+
+```bash
+# API logs
+docker service logs mosaic_api --tail 100 --follow
+
+# Web logs
+docker service logs mosaic_web --tail 100 --follow
+
+# PostgreSQL logs
+docker service logs mosaic_postgres --tail 100
+
+# All services
+for service in $(docker stack services mosaic --format '{{.Name}}'); do
+  echo "=== $service ==="
+  docker service logs $service --tail 20
+done
+```
+
+### Backup Volumes
+
+```bash
+# Backup PostgreSQL
+docker run --rm \
+  -v mosaic_postgres_data:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/postgres-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
+
+# Backup OpenBao
+docker run --rm \
+  -v mosaic_openbao_data:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/openbao-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
+```
+
+### Restore Volumes
+
+```bash
+# Stop services first
+docker service scale mosaic_postgres=0
+
+# Restore
+docker run --rm \
+  -v mosaic_postgres_data:/data \
+  -v $(pwd):/backup \
+  alpine sh -c 'rm -rf /data/* && tar xzf /backup/postgres-20260208.tar.gz -C /data'
+
+# Restart service
+docker service scale mosaic_postgres=1
+```
+
+## Troubleshooting
+
+### Service Won't Start
+
+```bash
+# Check service tasks for errors
+docker service ps mosaic_api --no-trunc
+
+# View detailed logs
+docker service logs mosaic_api --tail 100
+
+# Inspect service configuration
+docker service inspect mosaic_api
+```
+
+### Network Issues
+
+```bash
+# Verify overlay networks
+docker network ls --filter driver=overlay
+
+# Inspect traefik-public network
+docker network inspect traefik-public
+
+# Check service network connectivity
+docker exec $(docker ps -q -f label=com.docker.swarm.service.name=mosaic_api) \
+  ping postgres
+```
+
+### OpenBao Issues
+
+```bash
+# Check OpenBao status
+OPENBAO_CONTAINER=$(docker ps -q -f label=com.docker.swarm.service.name=mosaic_openbao)
+docker exec $OPENBAO_CONTAINER bao status
+
+# Re-unseal if sealed
+docker exec $OPENBAO_CONTAINER bao operator unseal <unseal-key>
+
+# Check logs
+docker service logs mosaic_openbao --tail 100
+```
+
+### Complete Stack Restart
+
+```bash
+# Remove stack (keeps volumes)
+docker stack rm mosaic
+
+# Wait for cleanup
+sleep 30
+
+# Redeploy
+./scripts/deploy-swarm.sh mosaic
+```
+
+## External Services Configuration
+
+### Using External Authentik
+
+1. Comment out Authentik services in `docker-compose.swarm.yml`
+2. Configure in `.env`:
+   ```bash
+   OIDC_ISSUER=https://auth.diversecanvas.com/application/o/mosaic-stack/
+   OIDC_CLIENT_ID=your-client-id
+   OIDC_CLIENT_SECRET=your-client-secret
+   OIDC_REDIRECT_URI=https://api.mosaicstack.dev/auth/callback/authentik
+   ```
+
+### Using External PostgreSQL
+
+1. Comment out `postgres` service in `docker-compose.swarm.yml`
+2. Configure in `.env`:
+   ```bash
+   DATABASE_URL=postgresql://user:pass@rds.amazonaws.com:5432/mosaic
+   ```
+
+### Using External Valkey/Redis
+
+1. Comment out `valkey` service in `docker-compose.swarm.yml`
+2. Configure in `.env`:
+   ```bash
+   VALKEY_URL=redis://elasticache.amazonaws.com:6379
+   ```
+
+### Using External OpenBao/Vault
+
+1. Comment out `openbao` service in `docker-compose.swarm.yml`
+2. Configure in `.env`:
+   ```bash
+   OPENBAO_ADDR=https://vault.example.com:8200
+   OPENBAO_ROLE_ID=your-role-id
+   OPENBAO_SECRET_ID=your-secret-id
+   ```
+
+## Security Considerations
+
+1. **Never commit `.env` to version control** - Contains secrets
+2. **Use strong passwords** - Generate with `openssl rand -base64 32`
+3. **Store OpenBao unseal keys securely** - Required for disaster recovery
+4. **Enable TLS/SSL in production** - Configure Traefik with Let's Encrypt
+5. **Regular backups** - Backup volumes and OpenBao keys
+6. **Monitor logs** - Watch for security events and errors
+7. **Update regularly** - Pull latest images with `IMAGE_TAG=latest`
+
+## Quick Reference
+
+See `docs/SWARM-QUICKREF.md` for quick command reference.
+
+## See Also
+
+- [Docker Compose Guide](../docker/DOCKER-COMPOSE-GUIDE.md) - Regular docker-compose deployment
+- [OpenBao Documentation](OPENBAO.md) - Secrets management setup
+- [Configuration Guide](CONFIGURATION.md) - All environment variables

docs/SWARM-QUICKREF.md

@@ -1,5 +1,9 @@
 # Docker Swarm Quick Reference
 
+**For complete deployment instructions, see [SWARM-DEPLOYMENT.md](SWARM-DEPLOYMENT.md)**
+
+This is a quick reference for common Docker Swarm commands after your stack is deployed.
+
 ## Initial Setup
 
 ```bash