stack/docs/OPENBAO-DEPLOYMENT.md

# OpenBao Deployment Guide

## Overview

OpenBao provides Transit encryption for sensitive credentials in Mosaic Stack. Due to the stateful nature of secrets management and port binding requirements, OpenBao has specific deployment constraints.

## Deployment Options

### Option 1: Standalone Container (Recommended for Swarm)

**When to use:**

- Docker Swarm deployments
- You want OpenBao isolated from the main stack
- You need guaranteed port availability

**How to deploy:**

```bash
# Deploy OpenBao as standalone container
docker compose -f docker-compose.openbao.yml up -d

# Check status
docker ps | grep openbao

# View logs
docker logs mosaic-openbao
docker logs mosaic-openbao-init

# The init container auto-initializes OpenBao on first run
# Check initialization status
docker logs mosaic-openbao-init
```

**Configuration:**

The standalone deployment:

- Binds to `127.0.0.1:8200` (localhost only for security)
- Auto-initializes via `openbao-init` sidecar
- Connects to swarm stack via `mosaic_internal` network
- Uses named volumes for persistence

**File:** `docker-compose.openbao.yml`

### Option 2: Bundled (Standalone Docker Compose Only)

**When to use:**

- Standalone docker-compose deployment (NOT swarm)
- Development/testing environments
- All-in-one turnkey setup

**How to deploy:**

```bash
# Use docker-compose.yml with full profile
export COMPOSE_PROFILES=full
docker compose up -d
```

**Configuration:**

OpenBao runs as part of the main stack with:

- `openbao` and `openbao-init` services
- Automatic initialization on first startup
- Integrated with other services

**File:** `docker-compose.yml` (with `COMPOSE_PROFILES=full` or `COMPOSE_PROFILES=openbao`)

### Option 3: External HashiCorp Vault

**When to use:**

- Production environments with existing Vault infrastructure
- Managed secrets service (HashiCorp Cloud Platform, AWS Secrets Manager + Vault)
- Multi-region deployments
- High availability requirements

**How to configure:**

1. Set environment variables:

   ```bash
   OPENBAO_ADDR=https://vault.example.com:8200
   OPENBAO_ROLE_ID=your-approle-role-id
   OPENBAO_SECRET_ID=your-approle-secret-id
   ```

2. Ensure external Vault has:
   - Transit secrets engine enabled
   - Encryption key created: `mosaic-credentials`
   - AppRole configured for Mosaic API authentication

3. Comment out OpenBao in all compose files

4. API will automatically connect to external Vault

### Option 4: Fallback Mode (No Vault)

**When to use:**

- Development/testing without secrets infrastructure
- Simplified deployments
- Gradual migration to Vault

**How to configure:**

1. Comment out or don't deploy OpenBao
2. Set `ENCRYPTION_KEY` in `.env` (required!)
3. API automatically falls back to AES-256-GCM encryption

**Configuration:**

```bash
# Generate encryption key
ENCRYPTION_KEY=$(openssl rand -hex 32)
echo "ENCRYPTION_KEY=$ENCRYPTION_KEY" >> .env
```

**Note:** This provides graceful degradation but lacks the key management features of OpenBao/Vault.

## Why OpenBao Can't Run in Swarm

OpenBao is a **stateful service** that binds to a specific port (`8200`). In Docker Swarm:

1. **Port binding conflicts:** Swarm services use overlay networks and can't reliably bind to host ports
2. **State management:** OpenBao maintains unsealed state and encryption keys that don't work with Swarm's task model
3. **Multiple replicas:** Swarm tries to create multiple replicas, causing port conflicts

**Solution:** Deploy OpenBao as a standalone container that connects to the swarm network.

## Deployment Workflows

### Swarm + Standalone OpenBao

```bash
# 1. Deploy OpenBao standalone
docker compose -f docker-compose.openbao.yml up -d

# 2. Wait for initialization
sleep 30
docker logs mosaic-openbao-init

# 3. Deploy swarm stack
IMAGE_TAG=dev ./scripts/deploy-swarm.sh mosaic

# 4. Verify API connects to OpenBao
docker service logs mosaic_api | grep -i openbao
```

### Standalone Compose + Bundled OpenBao

```bash
# 1. Set profile
export COMPOSE_PROFILES=full

# 2. Deploy everything
docker compose up -d

# 3. Verify OpenBao initialization
docker logs mosaic-openbao-init
```

### External Vault

```bash
# 1. Configure .env with external Vault URL and credentials
# OPENBAO_ADDR=https://vault.example.com:8200
# OPENBAO_ROLE_ID=...
# OPENBAO_SECRET_ID=...

# 2. Deploy stack (no OpenBao)
IMAGE_TAG=dev ./scripts/deploy-swarm.sh mosaic

# 3. Verify API connects to external Vault
docker service logs mosaic_api | grep -i vault
```

## Network Configuration

### Standalone OpenBao with Swarm Stack

The standalone OpenBao container connects to the swarm stack's internal network:

```yaml
networks:
  mosaic_internal:
    external: true
    name: mosaic_internal
```

This allows services in the swarm stack to reach OpenBao at `http://openbao:8200`.

### Port Exposure

- **Standalone:** `127.0.0.1:8200` (localhost only)
- **Bundled:** No port exposure (internal network only)
- **External:** Your Vault URL (typically HTTPS on 8200)

**Security Note:** Never expose OpenBao to the public internet without proper TLS and authentication.

## Initialization and Unsealing

### Auto-Initialization (Standalone & Bundled)

The `openbao-init` sidecar automatically:

1. Initializes OpenBao on first run
2. Creates unseal keys and root token
3. Unseals OpenBao
4. Enables Transit secrets engine
5. Creates `mosaic-credentials` encryption key
6. Sets up AppRole authentication

**Credentials location:** `/openbao/init/approle-credentials` (mounted volume)

### Manual Initialization (External Vault)

For external Vault, you must manually:

```bash
# 1. Enable transit engine
vault secrets enable transit

# 2. Create encryption key
vault write -f transit/keys/mosaic-credentials

# 3. Enable AppRole
vault auth enable approle

# 4. Create role for Mosaic API
vault write auth/approle/role/mosaic-api \
  token_policies="default" \
  token_ttl=1h \
  token_max_ttl=4h

# 5. Get credentials
vault read auth/approle/role/mosaic-api/role-id
vault write -f auth/approle/role/mosaic-api/secret-id

# 6. Set in .env
# OPENBAO_ROLE_ID=<role-id>
# OPENBAO_SECRET_ID=<secret-id>
```

## Troubleshooting

### OpenBao Won't Start in Swarm

**Symptom:** `Error initializing listener: bind: address already in use`

**Cause:** OpenBao cannot run in swarm mode

**Fix:** Deploy as standalone container:

```bash
docker compose -f docker-compose.openbao.yml up -d
```

### API Can't Connect to OpenBao

**Symptom:** API logs show connection errors to OpenBao

**Check:**

1. OpenBao is running: `docker ps | grep openbao`
2. Network connectivity: `docker exec mosaic_api curl http://openbao:8200/v1/sys/health`
3. Environment variable: `OPENBAO_ADDR=http://openbao:8200`

**Swarm specific:** Ensure standalone OpenBao is connected to `mosaic_internal` network

### Initialization Failed

**Symptom:** `openbao-init` container exits with errors

**Check:**

1. OpenBao is healthy: `docker logs mosaic-openbao`
2. Init logs: `docker logs mosaic-openbao-init`
3. Volume permissions: `docker volume inspect mosaic-openbao-init`

**Fix:** Remove volumes and redeploy:

```bash
docker compose -f docker-compose.openbao.yml down -v
docker compose -f docker-compose.openbao.yml up -d
```

### Fallback Mode Active (Unintended)

**Symptom:** API logs show "Using fallback encryption"

**Cause:** OpenBao not reachable

**Fix:**

1. Verify OpenBao deployment
2. Check `OPENBAO_ADDR` in `.env`
3. Test connectivity from API container

## Security Considerations

1. **Never commit unseal keys or root tokens** to version control
2. **Store credentials securely** - Use a password manager or secrets management system
3. **Rotate AppRole secrets regularly** - At least every 90 days
4. **Use TLS for external Vault** - Never use HTTP in production
5. **Restrict port exposure** - OpenBao should only be accessible from API/services
6. **Monitor access logs** - Review Vault audit logs regularly
7. **Backup regularly** - Backup OpenBao volumes and external Vault state

## See Also

- [OpenBao Documentation](OPENBAO.md) - Detailed OpenBao configuration and usage
- [Swarm Deployment Guide](SWARM-DEPLOYMENT.md) - Complete swarm deployment instructions
- [Configuration Guide](CONFIGURATION.md) - All environment variables
- [Credential Security Design](design/credential-security.md) - Architecture and security model