stack/docs/OPENBAO.md

OpenBao Integration Guide

Version: 0.0.9 Status: Production Ready Related Issues: #346, #357, #353

Table of Contents

1. Overview
2. Architecture
3. Default Turnkey Setup
4. Environment Variables
5. Transit Encryption Keys
6. Production Hardening
7. Operations
8. Troubleshooting
9. Disaster Recovery

---

Overview

Why OpenBao?

OpenBao is an open-source secrets management platform forked from HashiCorp Vault after HashiCorp changed to the Business Source License. Key benefits:

• Truly open-source - Linux Foundation project with OSI-approved license
• Drop-in Vault replacement - API-compatible with HashiCorp Vault
• Production-ready - v2.0+ with active development and community support
• Transit encryption - Encrypt data at rest without storing plaintext in OpenBao

What is Transit Encryption?

The Transit secrets engine provides "encryption as a service":

• Encryption/decryption operations via API calls
• Key versioning - Rotate keys without re-encrypting existing data
• No plaintext storage - OpenBao never stores your plaintext data
• Ciphertext format - Versioned format (vault:v1:...) enables seamless key rotation

Use Case: Encrypt sensitive data (OAuth tokens, API keys, credentials) before storing in PostgreSQL. If the database is compromised, attackers only get encrypted ciphertext.

---

Architecture

┌──────────────────────────────────────────────────────────────┐
│                     Mosaic Stack API                         │
│                                                              │
│  ┌────────────────┐        ┌──────────────────┐            │
│  │ VaultService   │───────>│  CryptoService   │            │
│  │ (Primary)      │        │  (Fallback)      │            │
│  └────────┬───────┘        └──────────────────┘            │
│           │                                                 │
│           │ Transit API                                     │
│           │ (encrypt/decrypt)                               │
└───────────┼─────────────────────────────────────────────────┘
            │
            │ HTTP (localhost:8200)
            ▼
┌───────────────────────────────────────────────────────────────┐
│                         OpenBao                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  Transit Secrets Engine                            │      │
│  │                                                     │      │
│  │  ┌──────────────┐  ┌──────────────┐               │      │
│  │  │ mosaic-      │  │ mosaic-      │  + 2 more     │      │
│  │  │ credentials  │  │ account-     │  keys         │      │
│  │  │              │  │ tokens       │               │      │
│  │  └──────────────┘  └──────────────┘               │      │
│  └────────────────────────────────────────────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  AppRole Authentication                            │      │
│  │  - Role: mosaic-transit                            │      │
│  │  - Policy: Transit encrypt/decrypt only            │      │
│  └────────────────────────────────────────────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  File Storage Backend                              │      │
│  │  /openbao/data (Docker volume)                     │      │
│  └────────────────────────────────────────────────────┘      │
└───────────────────────────────────────────────────────────────┘
            │
            │ Auto-init / Auto-unseal
            ▼
┌───────────────────────────────────────────────────────────────┐
│               OpenBao Init Sidecar                            │
│  - Initializes OpenBao on first run                          │
│  - Auto-unseals on container restart                         │
│  - Creates Transit keys and AppRole                          │
│  - Runs continuously with 30s check loop                     │
└───────────────────────────────────────────────────────────────┘

Data Flow: Encrypt

1. API receives plaintext credential (e.g., OAuth access token)
2. VaultService encrypts via Transit: POST /v1/transit/encrypt/mosaic-account-tokens
3. OpenBao returns ciphertext: vault:v1:8SDd3WHDOjf8Fz5MSLXjL...
4. Ciphertext stored in PostgreSQL
5. Plaintext never touches disk

Data Flow: Decrypt

1. API reads ciphertext from PostgreSQL
2. VaultService decrypts via Transit: POST /v1/transit/decrypt/mosaic-account-tokens
3. OpenBao returns plaintext
4. API uses plaintext for OAuth/API requests
5. Plaintext cleared from memory after use

Fallback Behavior

When OpenBao is unavailable:

• Encryption: VaultService falls back to AES-256-GCM (CryptoService)
• Decryption: Auto-detects format (vault:v1: vs AES) and uses appropriate service
• Logging: ERROR-level logs for visibility into infrastructure issues
• Backward Compatibility: Existing AES-encrypted data remains decryptable

Production Recommendation: Set OPENBAO_REQUIRED=true to fail startup if OpenBao is unavailable (prevents accidental fallback).

---

Default Turnkey Setup

What Happens on First docker compose up

The openbao-init sidecar automatically:

1. Waits for OpenBao server to be healthy
2. Initializes OpenBao with 1-of-1 Shamir key split (development mode)
3. Unseals OpenBao using the stored unseal key
4. Enables Transit secrets engine at /transit
5. Creates 4 named encryption keys:
   • mosaic-credentials - User-provided credentials (API keys, tokens)
   • mosaic-account-tokens - OAuth tokens from BetterAuth accounts
   • mosaic-federation - Federation private keys
   • mosaic-llm-config - LLM provider API keys
6. Creates AppRole mosaic-transit with Transit-only policy
7. Generates AppRole credentials (role_id, secret_id)
8. Saves credentials to /openbao/init/approle-credentials
9. Watches continuously - auto-unseals on container restart every 30s

File Locations

Docker Volumes:
├── mosaic-openbao-data          # OpenBao database (encrypted storage)
├── mosaic-openbao-init          # Unseal key, root token, AppRole credentials
└── docker/openbao/config.hcl    # Server configuration (bind mount)

Credentials Storage

Unseal Key: /openbao/init/unseal-key (plaintext, volume-only access) Root Token: /openbao/init/root-token (plaintext, volume-only access) AppRole Credentials: /openbao/init/approle-credentials (JSON, read by API)

Example /openbao/init/approle-credentials:

{
  "role_id": "42474304-cb07-edff-f5c8-cae494ba6a51",
  "secret_id": "39ccdb6b-4570-a279-022c-36220739ebcf"
}

Security Note: These files are only accessible within the Docker internal network. The OpenBao API is bound to 127.0.0.1:8200 (localhost only).

---

Environment Variables

Required

None - the turnkey setup works with defaults.

Optional

Variable	Default	Purpose
OPENBAO_ADDR	http://openbao:8200	OpenBao API address (Docker internal)
OPENBAO_PORT	8200	Port for localhost binding in docker-compose
OPENBAO_ROLE_ID	(read from file)	Override AppRole role_id
OPENBAO_SECRET_ID	(read from file)	Override AppRole secret_id
OPENBAO_REQUIRED	false	Fail startup if OpenBao unavailable

Production Configuration

Development:

OPENBAO_ADDR=http://openbao:8200
OPENBAO_REQUIRED=false

Production:

OPENBAO_ADDR=https://vault.internal.corp:8200
OPENBAO_REQUIRED=true
# AppRole credentials provided by external Vault deployment
OPENBAO_ROLE_ID=<from-external-vault>
OPENBAO_SECRET_ID=<from-external-vault>

Deployment Scenarios

OpenBao can be deployed in three modes using Docker Compose profiles:

Bundled OpenBao (Development/Turnkey)

Use Case: Local development, testing, demo environments

# .env
COMPOSE_PROFILES=full  # or openbao
OPENBAO_ADDR=http://openbao:8200

# Start services
docker compose up -d

OpenBao automatically initializes with 4 Transit keys and AppRole authentication. API reads credentials from /openbao/init/approle-credentials volume.

External OpenBao/Vault (Production)

Use Case: Production with managed HashiCorp Vault or external OpenBao

# .env
COMPOSE_PROFILES=  # Empty - disable bundled OpenBao
OPENBAO_ADDR=https://vault.example.com:8200
OPENBAO_ROLE_ID=your-role-id
OPENBAO_SECRET_ID=your-secret-id
OPENBAO_REQUIRED=true  # Fail startup if unavailable

# Or use docker-compose.example.external.yml
cp docker/docker-compose.example.external.yml docker-compose.override.yml

# Start services
docker compose up -d

Requirements for External Vault:

• Transit secrets engine enabled at /transit
• Four named encryption keys created (see Transit Encryption Keys section)
• AppRole authentication configured with Transit-only policy
• Network connectivity from API container to Vault endpoint

Fallback Mode (No OpenBao)

Use Case: Development without secrets management, testing graceful degradation

# .env
COMPOSE_PROFILES=database,cache  # Exclude openbao profile
ENCRYPTION_KEY=your-64-char-hex-key  # For AES-256-GCM fallback

# Start services
docker compose up -d

API automatically falls back to AES-256-GCM encryption using ENCRYPTION_KEY. This provides encryption at rest without Transit infrastructure. Logs will show ERROR-level warnings about OpenBao unavailability.

Note: Fallback mode uses aes:iv:tag:encrypted ciphertext format instead of vault:v1:... format.

---

Transit Encryption Keys

Named Keys

Key Name	Purpose	Used By	Example Data
mosaic-credentials	User-provided credentials	UserCredential model	GitHub PAT, AWS access keys
mosaic-account-tokens	OAuth tokens	BetterAuth accounts table	access_token, refresh_token, id_token
mosaic-federation	Federation private keys	FederatedInstance model	Ed25519 private keys
mosaic-llm-config	LLM provider credentials	LLMProviderInstance model	OpenAI API key, Anthropic API key

Key Properties

• Algorithm: AES-256-GCM96
• Versioning: Enabled (transparent key rotation)
• Deletion: Not allowed (exportable = false)
• Min Decryption Version: 1 (all versions can decrypt)
• Latest Version: Auto-increments on rotation

Key Rotation

OpenBao Transit keys support transparent rotation:

1. Rotate key: bao write -f transit/keys/mosaic-credentials/rotate
2. New version created (e.g., v2)
3. New encryptions use v2
4. Old ciphertexts still decrypt with v1
5. No data re-encryption needed

Recommendation: Rotate keys annually or after suspected compromise.

Ciphertext Format

vault:v1:8SDd3WHDOjf8Fz5MSLXjLx0AzTdYhLMAAp4W
       ^^  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       │   └─ Base64-encoded ciphertext
       └───── Key version

Version prefix enables seamless rotation - OpenBao knows which key version to use for decryption.

---

Production Hardening

Security Checklist

CRITICAL - Do not deploy to production without these:

Task	Command	Priority
1. Upgrade Shamir key splitting	bao operator rekey -key-shares=5 -key-threshold=3	P0
2. Enable TLS on listener	Update config.hcl: tls_disable = 0	P0
3. Revoke root token	bao token revoke -self	P0
4. Enable audit logging	bao audit enable file file_path=/bao/logs/audit.log	P0
5. Use external auto-unseal	AWS KMS, GCP CKMS, Azure Key Vault	P1
6. HA storage backend	Raft or Consul	P1
7. Bind to internal network only	Update docker-compose.yml: remove port exposure	P1
8. Rate limiting	Use reverse proxy (nginx/Traefik) with rate limits	P2
9. Rotate AppRole credentials	Regenerate secret_id monthly	P2
10. Backup automation	Snapshot Raft or file storage daily	P2

1. Shamir Key Splitting

Current (Development): 1-of-1 key shares (single unseal key) Production: 3-of-5 key shares (requires 3 of 5 keys to unseal)

Procedure:

# SSH into OpenBao container
docker compose exec openbao sh

# Set environment
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat /openbao/init/root-token)

# Rekey to 3-of-5
bao operator rekey -init -key-shares=5 -key-threshold=3

# Follow prompts to generate new keys
# Distribute 5 keys to different key holders
# Require 3 keys to unseal

Key Management Best Practices:

• Store each key with a different person/system
• Use hardware security modules (HSMs) for key storage
• Document key recovery procedures
• Test unseal process regularly

2. TLS Configuration

Update docker/openbao/config.hcl:

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0  # Changed from 1
  tls_cert_file = "/openbao/config/server.crt"
  tls_key_file = "/openbao/config/server.key"
  tls_min_version = "tls12"
}

Generate TLS certificates:

# Production: Use certificates from your CA
# Development: Generate self-signed
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \
  -days 365 -nodes -subj "/CN=openbao.internal"

Update API configuration:

OPENBAO_ADDR=https://openbao.internal:8200

3. Revoke Root Token

After initial setup, the root token should be revoked:

export VAULT_TOKEN=$(cat /openbao/init/root-token)
bao token revoke -self

# Remove root token file
rm /openbao/init/root-token

Recovery: Use the Shamir keys to generate a new root token if needed:

bao operator generate-root -init

4. Audit Logging

Enable audit logging to track all API activity:

bao audit enable file file_path=/openbao/logs/audit.log

# Verify
bao audit list

Mount logs volume in docker-compose.yml:

openbao:
  volumes:
    - openbao_logs:/openbao/logs

Log Format: JSON, one entry per API request Retention: Rotate daily, retain 90 days minimum

5. External Auto-Unseal

Replace file-based unseal with KMS-based auto-unseal:

AWS KMS Example:

# config.hcl
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc123"
}

Benefits:

• No manual unseal required
• Unseal key never touches disk
• Audit trail in KMS service
• Automatic rotation support

6. High Availability

File Storage (Current) - Single node, no HA Raft Storage (Recommended) - Multi-node with leader election

Raft configuration:

storage "raft" {
  path = "/openbao/data"
  node_id = "node1"
}

Run 3-5 OpenBao nodes with Raft for production HA.

7. Network Isolation

Current: Port bound to 127.0.0.1:8200 (localhost only) Production: Use internal network only, no public exposure

docker-compose.yml:

openbao:
  # Remove port mapping
  # ports:
  #   - "127.0.0.1:8200:8200"
  networks:
    - internal

API access: Use reverse proxy with authentication/authorization.

8. Rate Limiting

Prevent brute-force attacks on AppRole authentication:

nginx reverse proxy:

http {
  limit_req_zone $binary_remote_addr zone=vault_auth:10m rate=10r/s;

  server {
    listen 8200 ssl;

    location /v1/auth/approle/login {
      limit_req zone=vault_auth burst=20 nodelay;
      proxy_pass http://openbao:8200;
    }

    location / {
      proxy_pass http://openbao:8200;
    }
  }
}

---

Operations

Health Checks

OpenBao server:

curl http://localhost:8200/v1/sys/health

Response:

{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "server_time_utc": 1738938524
}

VaultService health (API endpoint):

curl http://localhost:3000/health

Response:

{
  "status": "ok",
  "info": {
    "openbao": {
      "status": "up"
    }
  }
}

Manual Unseal

If auto-unseal fails:

# Get unseal key
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/unseal-key

# Unseal OpenBao
docker compose exec openbao bao operator unseal <key>

Check Transit Keys

# Get root token
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)

# List Transit keys
docker compose exec -e VAULT_TOKEN openbao bao list transit/keys

# Read key details
docker compose exec -e VAULT_TOKEN openbao bao read transit/keys/mosaic-credentials

Test Encryption

# Encrypt test data
PLAINTEXT=$(echo -n "test-secret" | base64)
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  transit/encrypt/mosaic-credentials plaintext=$PLAINTEXT

# Decrypt
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  transit/decrypt/mosaic-credentials ciphertext=<ciphertext>

Monitor Logs

# OpenBao server logs
docker compose logs -f openbao

# Init sidecar logs
docker compose logs -f openbao-init

# VaultService logs (API)
docker compose logs -f api | grep VaultService

---

Troubleshooting

OpenBao Won't Start

Symptoms: Container restarts repeatedly, "connection refused" errors

Diagnosis:

docker compose logs openbao

Common Causes:

1. Port conflict: Another service on 8200
2. Volume permission issues: Can't write to /openbao/data
3. Invalid config: Syntax error in config.hcl

Solutions:

# Check port usage
netstat -tuln | grep 8200

# Fix volume permissions
docker compose down -v
docker volume rm mosaic-openbao-data
docker compose up -d

# Validate config
docker compose exec openbao bao server -config=/openbao/config/config.hcl -test

OpenBao Sealed

Symptoms: API returns 503, "sealed": true in health check

Diagnosis:

docker compose exec openbao bao status

Solution:

# Auto-unseal should handle this
# Force restart init container
docker compose restart openbao-init

# Manual unseal if needed
docker compose exec openbao bao operator unseal <key>

VaultService Falls Back to AES

Symptoms: Logs show "OpenBao unavailable, falling back to AES-256-GCM"

Diagnosis:

# Check OpenBao health
curl http://localhost:8200/v1/sys/health

# Check AppRole credentials
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/approle-credentials

Common Causes:

1. OpenBao not running: docker compose ps openbao
2. Wrong OPENBAO_ADDR: Check environment variable
3. AppRole credentials missing: Reinitialize with docker compose restart openbao-init
4. Network issue: Check Docker network connectivity

Solutions:

# Restart OpenBao stack
docker compose restart openbao openbao-init

# Verify connectivity from API container
docker compose exec api curl http://openbao:8200/v1/sys/health

Authentication Failures

Symptoms: "Token renewal failed", "Authentication failed"

Diagnosis:

# Check AppRole
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)
docker compose exec -e VAULT_TOKEN openbao bao read auth/approle/role/mosaic-transit

Solutions:

# Regenerate AppRole credentials
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  -f auth/approle/role/mosaic-transit/secret-id > new-credentials.json

# Update credentials file
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/host alpine \
  cp /host/new-credentials.json /data/approle-credentials

# Restart API
docker compose restart api

Decrypt Failures

Symptoms: "Failed to decrypt data", encrypted tokens visible in database

Diagnosis:

# Check ciphertext format
psql -h localhost -U mosaic -d mosaic \
  -c "SELECT access_token FROM accounts LIMIT 1;"

Ciphertext formats:

• vault:v1:... - Transit encryption (requires OpenBao)
• iv:tag:encrypted - AES fallback (works without OpenBao)

Solutions:

1. Transit ciphertext + OpenBao unavailable: Start OpenBao
2. Corrupted ciphertext: Check database integrity
3. Wrong encryption key: Verify ENCRYPTION_KEY hasn't changed (for AES)

---

Disaster Recovery

Backup

Critical Data:

1. Unseal key: /openbao/init/unseal-key (Shamir keys in production)
2. Root token: /openbao/init/root-token (revoke in production, document recovery)
3. AppRole credentials: /openbao/init/approle-credentials
4. OpenBao data: /openbao/data (encrypted storage)

Backup Procedure:

# Backup volumes
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
  alpine tar czf /backup/openbao-init-$(date +%Y%m%d).tar.gz /data

docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
  alpine tar czf /backup/openbao-data-$(date +%Y%m%d).tar.gz /data

# Backup to remote storage
aws s3 cp openbao-*.tar.gz s3://backups/openbao/

Schedule: Daily automated backups with 30-day retention.

Restore

Full Recovery:

# Download backups
aws s3 cp s3://backups/openbao/openbao-init-20260207.tar.gz .
aws s3 cp s3://backups/openbao/openbao-data-20260207.tar.gz .

# Stop containers
docker compose down

# Restore volumes
docker volume create mosaic-openbao-init
docker volume create mosaic-openbao-data

docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
  alpine tar xzf /backup/openbao-init-20260207.tar.gz -C /data --strip-components=1

docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
  alpine tar xzf /backup/openbao-data-20260207.tar.gz -C /data --strip-components=1

# Restart
docker compose up -d

Key Recovery

Lost Unseal Key:

• Development (1-of-1): Restore from backup
• Production (3-of-5): Collect 3 of 5 Shamir keys from key holders

Lost Root Token:

# Generate new root token using Shamir keys
bao operator generate-root -init
bao operator generate-root -nonce=<nonce> <key1>
bao operator generate-root -nonce=<nonce> <key2>
bao operator generate-root -nonce=<nonce> <key3>

Lost AppRole Credentials:

# Use root token to regenerate
export VAULT_TOKEN=<root-token>
bao write -format=json -f auth/approle/role/mosaic-transit/secret-id \
  > new-approle-credentials.json

Testing Recovery

Quarterly DR test:

1. Backup production OpenBao
2. Restore to test environment
3. Verify all Transit keys accessible
4. Test encrypt/decrypt operations
5. Document any issues

---

Additional Resources

• OpenBao Documentation: https://openbao.org/docs/
• Transit Secrets Engine: https://openbao.org/docs/secrets/transit/
• AppRole Auth: https://openbao.org/docs/auth/approle/
• Production Hardening: https://openbao.org/docs/guides/production/
• Design Document: docs/design/credential-security.md

Support

For issues with Mosaic Stack's OpenBao integration:

• Repository: https://git.mosaicstack.dev/mosaic/stack
• Issues: https://git.mosaicstack.dev/mosaic/stack/issues
• Milestone: M9-CredentialSecurity

For OpenBao itself:

• GitHub: https://github.com/openbao/openbao
• Discussions: https://github.com/openbao/openbao/discussions