stack/docs/OPENBAO-DEPLOYMENT.md

OpenBao Deployment Guide

Overview

OpenBao provides Transit encryption for sensitive credentials in Mosaic Stack. Due to the stateful nature of secrets management and port binding requirements, OpenBao has specific deployment constraints.

Deployment Options

Option 1: Standalone Container (Recommended for Swarm)

When to use:

• Docker Swarm deployments
• You want OpenBao isolated from the main stack
• You need guaranteed port availability

How to deploy:

# Deploy OpenBao as standalone container
docker compose -f docker-compose.openbao.yml up -d

# Check status
docker ps | grep openbao

# View logs
docker logs mosaic-openbao
docker logs mosaic-openbao-init

# The init container auto-initializes OpenBao on first run
# Check initialization status
docker logs mosaic-openbao-init

Configuration:

The standalone deployment:

• Binds to 127.0.0.1:8200 (localhost only for security)
• Auto-initializes via openbao-init sidecar
• Connects to swarm stack via mosaic_internal network
• Uses named volumes for persistence

File: docker-compose.openbao.yml

Option 2: Bundled (Standalone Docker Compose Only)

When to use:

• Standalone docker-compose deployment (NOT swarm)
• Development/testing environments
• All-in-one turnkey setup

How to deploy:

# Use docker-compose.yml with full profile
export COMPOSE_PROFILES=full
docker compose up -d

Configuration:

OpenBao runs as part of the main stack with:

• openbao and openbao-init services
• Automatic initialization on first startup
• Integrated with other services

File: docker-compose.yml (with COMPOSE_PROFILES=full or COMPOSE_PROFILES=openbao)

Option 3: External HashiCorp Vault

When to use:

• Production environments with existing Vault infrastructure
• Managed secrets service (HashiCorp Cloud Platform, AWS Secrets Manager + Vault)
• Multi-region deployments
• High availability requirements

How to configure:

1. Set environment variables:

   OPENBAO_ADDR=https://vault.example.com:8200
   OPENBAO_ROLE_ID=your-approle-role-id
   OPENBAO_SECRET_ID=your-approle-secret-id
   

2. Ensure external Vault has:

   • Transit secrets engine enabled
   • Encryption key created: mosaic-credentials
   • AppRole configured for Mosaic API authentication

3. Comment out OpenBao in all compose files

4. API will automatically connect to external Vault

Option 4: Fallback Mode (No Vault)

When to use:

• Development/testing without secrets infrastructure
• Simplified deployments
• Gradual migration to Vault

How to configure:

1. Comment out or don't deploy OpenBao
2. Set ENCRYPTION_KEY in .env (required!)
3. API automatically falls back to AES-256-GCM encryption

Configuration:

# Generate encryption key
ENCRYPTION_KEY=$(openssl rand -hex 32)
echo "ENCRYPTION_KEY=$ENCRYPTION_KEY" >> .env

Note: This provides graceful degradation but lacks the key management features of OpenBao/Vault.

Why OpenBao Can't Run in Swarm

OpenBao is a stateful service that binds to a specific port (8200). In Docker Swarm:

1. Port binding conflicts: Swarm services use overlay networks and can't reliably bind to host ports
2. State management: OpenBao maintains unsealed state and encryption keys that don't work with Swarm's task model
3. Multiple replicas: Swarm tries to create multiple replicas, causing port conflicts

Solution: Deploy OpenBao as a standalone container that connects to the swarm network.

Deployment Workflows

Swarm + Standalone OpenBao

# 1. Deploy OpenBao standalone
docker compose -f docker-compose.openbao.yml up -d

# 2. Wait for initialization
sleep 30
docker logs mosaic-openbao-init

# 3. Deploy swarm stack
IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic

# 4. Verify API connects to OpenBao
docker service logs mosaic_api | grep -i openbao

Standalone Compose + Bundled OpenBao

# 1. Set profile
export COMPOSE_PROFILES=full

# 2. Deploy everything
docker compose up -d

# 3. Verify OpenBao initialization
docker logs mosaic-openbao-init

External Vault

# 1. Configure .env with external Vault URL and credentials
# OPENBAO_ADDR=https://vault.example.com:8200
# OPENBAO_ROLE_ID=...
# OPENBAO_SECRET_ID=...

# 2. Deploy stack (no OpenBao)
IMAGE_TAG=latest ./scripts/deploy-swarm.sh mosaic

# 3. Verify API connects to external Vault
docker service logs mosaic_api | grep -i vault

Network Configuration

Standalone OpenBao with Swarm Stack

The standalone OpenBao container connects to the swarm stack's internal network:

networks:
  mosaic_internal:
    external: true
    name: mosaic_internal

This allows services in the swarm stack to reach OpenBao at http://openbao:8200.

Port Exposure

• Standalone: 127.0.0.1:8200 (localhost only)
• Bundled: No port exposure (internal network only)
• External: Your Vault URL (typically HTTPS on 8200)

Security Note: Never expose OpenBao to the public internet without proper TLS and authentication.

Initialization and Unsealing

Auto-Initialization (Standalone & Bundled)

The openbao-init sidecar automatically:

1. Initializes OpenBao on first run
2. Creates unseal keys and root token
3. Unseals OpenBao
4. Enables Transit secrets engine
5. Creates mosaic-credentials encryption key
6. Sets up AppRole authentication

Credentials location: /openbao/init/approle-credentials (mounted volume)

Manual Initialization (External Vault)

For external Vault, you must manually:

# 1. Enable transit engine
vault secrets enable transit

# 2. Create encryption key
vault write -f transit/keys/mosaic-credentials

# 3. Enable AppRole
vault auth enable approle

# 4. Create role for Mosaic API
vault write auth/approle/role/mosaic-api \
  token_policies="default" \
  token_ttl=1h \
  token_max_ttl=4h

# 5. Get credentials
vault read auth/approle/role/mosaic-api/role-id
vault write -f auth/approle/role/mosaic-api/secret-id

# 6. Set in .env
# OPENBAO_ROLE_ID=<role-id>
# OPENBAO_SECRET_ID=<secret-id>

Troubleshooting

OpenBao Won't Start in Swarm

Symptom: Error initializing listener: bind: address already in use

Cause: OpenBao cannot run in swarm mode

Fix: Deploy as standalone container:

docker compose -f docker-compose.openbao.yml up -d

API Can't Connect to OpenBao

Symptom: API logs show connection errors to OpenBao

Check:

1. OpenBao is running: docker ps | grep openbao
2. Network connectivity: docker exec mosaic_api curl http://openbao:8200/v1/sys/health
3. Environment variable: OPENBAO_ADDR=http://openbao:8200

Swarm specific: Ensure standalone OpenBao is connected to mosaic_internal network

Initialization Failed

Symptom: openbao-init container exits with errors

Check:

1. OpenBao is healthy: docker logs mosaic-openbao
2. Init logs: docker logs mosaic-openbao-init
3. Volume permissions: docker volume inspect mosaic-openbao-init

Fix: Remove volumes and redeploy:

docker compose -f docker-compose.openbao.yml down -v
docker compose -f docker-compose.openbao.yml up -d

Fallback Mode Active (Unintended)

Symptom: API logs show "Using fallback encryption"

Cause: OpenBao not reachable

Fix:

1. Verify OpenBao deployment
2. Check OPENBAO_ADDR in .env
3. Test connectivity from API container

Security Considerations

1. Never commit unseal keys or root tokens to version control
2. Store credentials securely - Use a password manager or secrets management system
3. Rotate AppRole secrets regularly - At least every 90 days
4. Use TLS for external Vault - Never use HTTP in production
5. Restrict port exposure - OpenBao should only be accessible from API/services
6. Monitor access logs - Review Vault audit logs regularly
7. Backup regularly - Backup OpenBao volumes and external Vault state

See Also

• OpenBao Documentation - Detailed OpenBao configuration and usage
• Swarm Deployment Guide - Complete swarm deployment instructions
• Configuration Guide - All environment variables
• Credential Security Design - Architecture and security model