stack/docs/OPENBAO.md

# OpenBao Integration Guide

**Version:** 0.0.9
**Status:** Production Ready
**Related Issues:** [#346](https://git.mosaicstack.dev/mosaic/stack/issues/346), [#357](https://git.mosaicstack.dev/mosaic/stack/issues/357), [#353](https://git.mosaicstack.dev/mosaic/stack/issues/353)

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Default Turnkey Setup](#default-turnkey-setup)
4. [Environment Variables](#environment-variables)
5. [Transit Encryption Keys](#transit-encryption-keys)
6. [Production Hardening](#production-hardening)
7. [Operations](#operations)
8. [Troubleshooting](#troubleshooting)
9. [Disaster Recovery](#disaster-recovery)

---

## Overview

### Why OpenBao?

OpenBao is an open-source secrets management platform forked from HashiCorp Vault after HashiCorp changed to the Business Source License. Key benefits:

- **Truly open-source** - Linux Foundation project with OSI-approved license
- **Drop-in Vault replacement** - API-compatible with HashiCorp Vault
- **Production-ready** - v2.0+ with active development and community support
- **Transit encryption** - Encrypt data at rest without storing plaintext in OpenBao

### What is Transit Encryption?

The Transit secrets engine provides "encryption as a service":

- **Encryption/decryption operations** via API calls
- **Key versioning** - Rotate keys without re-encrypting existing data
- **No plaintext storage** - OpenBao never stores your plaintext data
- **Ciphertext format** - Versioned format (`vault:v1:...`) enables seamless key rotation

**Use Case**: Encrypt sensitive data (OAuth tokens, API keys, credentials) before storing in PostgreSQL. If the database is compromised, attackers only get encrypted ciphertext.

---

## Architecture

```
┌──────────────────────────────────────────────────────────────┐
│                     Mosaic Stack API                         │
│                                                              │
│  ┌────────────────┐        ┌──────────────────┐            │
│  │ VaultService   │───────>│  CryptoService   │            │
│  │ (Primary)      │        │  (Fallback)      │            │
│  └────────┬───────┘        └──────────────────┘            │
│           │                                                 │
│           │ Transit API                                     │
│           │ (encrypt/decrypt)                               │
└───────────┼─────────────────────────────────────────────────┘
            │
            │ HTTP (localhost:8200)
            ▼
┌───────────────────────────────────────────────────────────────┐
│                         OpenBao                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  Transit Secrets Engine                            │      │
│  │                                                     │      │
│  │  ┌──────────────┐  ┌──────────────┐               │      │
│  │  │ mosaic-      │  │ mosaic-      │  + 2 more     │      │
│  │  │ credentials  │  │ account-     │  keys         │      │
│  │  │              │  │ tokens       │               │      │
│  │  └──────────────┘  └──────────────┘               │      │
│  └────────────────────────────────────────────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  AppRole Authentication                            │      │
│  │  - Role: mosaic-transit                            │      │
│  │  - Policy: Transit encrypt/decrypt only            │      │
│  └────────────────────────────────────────────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────┐      │
│  │  File Storage Backend                              │      │
│  │  /openbao/data (Docker volume)                     │      │
│  └────────────────────────────────────────────────────┘      │
└───────────────────────────────────────────────────────────────┘
            │
            │ Auto-init / Auto-unseal
            ▼
┌───────────────────────────────────────────────────────────────┐
│               OpenBao Init Sidecar                            │
│  - Initializes OpenBao on first run                          │
│  - Auto-unseals on container restart                         │
│  - Creates Transit keys and AppRole                          │
│  - Runs continuously with 30s check loop                     │
└───────────────────────────────────────────────────────────────┘
```

### Data Flow: Encrypt

1. API receives plaintext credential (e.g., OAuth access token)
2. VaultService encrypts via Transit: `POST /v1/transit/encrypt/mosaic-account-tokens`
3. OpenBao returns ciphertext: `vault:v1:8SDd3WHDOjf8Fz5MSLXjL...`
4. Ciphertext stored in PostgreSQL
5. Plaintext never touches disk

### Data Flow: Decrypt

1. API reads ciphertext from PostgreSQL
2. VaultService decrypts via Transit: `POST /v1/transit/decrypt/mosaic-account-tokens`
3. OpenBao returns plaintext
4. API uses plaintext for OAuth/API requests
5. Plaintext cleared from memory after use

### Fallback Behavior

When OpenBao is unavailable:

- **Encryption**: VaultService falls back to AES-256-GCM (`CryptoService`)
- **Decryption**: Auto-detects format (`vault:v1:` vs AES) and uses appropriate service
- **Logging**: ERROR-level logs for visibility into infrastructure issues
- **Backward Compatibility**: Existing AES-encrypted data remains decryptable

**Production Recommendation**: Set `OPENBAO_REQUIRED=true` to fail startup if OpenBao is unavailable (prevents accidental fallback).

---

## Default Turnkey Setup

### What Happens on First `docker compose up`

The `openbao-init` sidecar automatically:

1. **Waits** for OpenBao server to be healthy
2. **Initializes** OpenBao with 1-of-1 Shamir key split (development mode)
3. **Unseals** OpenBao using the stored unseal key
4. **Enables** Transit secrets engine at `/transit`
5. **Creates** 4 named encryption keys:
   - `mosaic-credentials` - User-provided credentials (API keys, tokens)
   - `mosaic-account-tokens` - OAuth tokens from BetterAuth accounts
   - `mosaic-federation` - Federation private keys
   - `mosaic-llm-config` - LLM provider API keys
6. **Creates** AppRole `mosaic-transit` with Transit-only policy
7. **Generates** AppRole credentials (role_id, secret_id)
8. **Saves** credentials to `/openbao/init/approle-credentials`
9. **Watches** continuously - auto-unseals on container restart every 30s

### File Locations

```
Docker Volumes:
├── mosaic-openbao-data          # OpenBao database (encrypted storage)
├── mosaic-openbao-init          # Unseal key, root token, AppRole credentials
└── docker/openbao/config.hcl    # Server configuration (bind mount)
```

### Credentials Storage

**Unseal Key**: `/openbao/init/unseal-key` (plaintext, volume-only access)
**Root Token**: `/openbao/init/root-token` (plaintext, volume-only access)
**AppRole Credentials**: `/openbao/init/approle-credentials` (JSON, read by API)

Example `/openbao/init/approle-credentials`:

```json
{
  "role_id": "42474304-cb07-edff-f5c8-cae494ba6a51",
  "secret_id": "39ccdb6b-4570-a279-022c-36220739ebcf"
}
```

**Security Note**: These files are only accessible within the Docker internal network. The OpenBao API is bound to `127.0.0.1:8200` (localhost only).

---

## Environment Variables

### Required

None - the turnkey setup works with defaults.

### Optional

| Variable            | Default               | Purpose                                      |
| ------------------- | --------------------- | -------------------------------------------- |
| `OPENBAO_ADDR`      | `http://openbao:8200` | OpenBao API address (Docker internal)        |
| `OPENBAO_PORT`      | `8200`                | Port for localhost binding in docker-compose |
| `OPENBAO_ROLE_ID`   | (read from file)      | Override AppRole role_id                     |
| `OPENBAO_SECRET_ID` | (read from file)      | Override AppRole secret_id                   |
| `OPENBAO_REQUIRED`  | `false`               | Fail startup if OpenBao unavailable          |

### Production Configuration

**Development**:

```bash
OPENBAO_ADDR=http://openbao:8200
OPENBAO_REQUIRED=false
```

**Production**:

```bash
OPENBAO_ADDR=https://vault.internal.corp:8200
OPENBAO_REQUIRED=true
# AppRole credentials provided by external Vault deployment
OPENBAO_ROLE_ID=<from-external-vault>
OPENBAO_SECRET_ID=<from-external-vault>
```

---

## Transit Encryption Keys

### Named Keys

| Key Name                | Purpose                   | Used By                   | Example Data                          |
| ----------------------- | ------------------------- | ------------------------- | ------------------------------------- |
| `mosaic-credentials`    | User-provided credentials | UserCredential model      | GitHub PAT, AWS access keys           |
| `mosaic-account-tokens` | OAuth tokens              | BetterAuth accounts table | access_token, refresh_token, id_token |
| `mosaic-federation`     | Federation private keys   | FederatedInstance model   | Ed25519 private keys                  |
| `mosaic-llm-config`     | LLM provider credentials  | LLMProviderInstance model | OpenAI API key, Anthropic API key     |

### Key Properties

- **Algorithm**: AES-256-GCM96
- **Versioning**: Enabled (transparent key rotation)
- **Deletion**: Not allowed (exportable = false)
- **Min Decryption Version**: 1 (all versions can decrypt)
- **Latest Version**: Auto-increments on rotation

### Key Rotation

OpenBao Transit keys support **transparent rotation**:

1. Rotate key: `bao write -f transit/keys/mosaic-credentials/rotate`
2. New version created (e.g., v2)
3. **New encryptions** use v2
4. **Old ciphertexts** still decrypt with v1
5. **No data re-encryption needed**

**Recommendation**: Rotate keys annually or after suspected compromise.

### Ciphertext Format

```
vault:v1:8SDd3WHDOjf8Fz5MSLXjLx0AzTdYhLMAAp4W
       ^^  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       │   └─ Base64-encoded ciphertext
       └───── Key version
```

**Version prefix** enables seamless rotation - OpenBao knows which key version to use for decryption.

---

## Production Hardening

### Security Checklist

**CRITICAL - Do not deploy to production without these:**

| Task                                 | Command                                               | Priority |
| ------------------------------------ | ----------------------------------------------------- | -------- |
| **1. Upgrade Shamir key splitting**  | `bao operator rekey -key-shares=5 -key-threshold=3`   | P0       |
| **2. Enable TLS on listener**        | Update `config.hcl`: `tls_disable = 0`                | P0       |
| **3. Revoke root token**             | `bao token revoke -self`                              | P0       |
| **4. Enable audit logging**          | `bao audit enable file file_path=/bao/logs/audit.log` | P0       |
| **5. Use external auto-unseal**      | AWS KMS, GCP CKMS, Azure Key Vault                    | P1       |
| **6. HA storage backend**            | Raft or Consul                                        | P1       |
| **7. Bind to internal network only** | Update docker-compose.yml: remove port exposure       | P1       |
| **8. Rate limiting**                 | Use reverse proxy (nginx/Traefik) with rate limits    | P2       |
| **9. Rotate AppRole credentials**    | Regenerate secret_id monthly                          | P2       |
| **10. Backup automation**            | Snapshot Raft or file storage daily                   | P2       |

### 1. Shamir Key Splitting

**Current (Development)**: 1-of-1 key shares (single unseal key)
**Production**: 3-of-5 key shares (requires 3 of 5 keys to unseal)

**Procedure**:

```bash
# SSH into OpenBao container
docker compose exec openbao sh

# Set environment
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat /openbao/init/root-token)

# Rekey to 3-of-5
bao operator rekey -init -key-shares=5 -key-threshold=3

# Follow prompts to generate new keys
# Distribute 5 keys to different key holders
# Require 3 keys to unseal
```

**Key Management Best Practices**:

- Store each key with a different person/system
- Use hardware security modules (HSMs) for key storage
- Document key recovery procedures
- Test unseal process regularly

### 2. TLS Configuration

**Update `docker/openbao/config.hcl`**:

```hcl
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0  # Changed from 1
  tls_cert_file = "/openbao/config/server.crt"
  tls_key_file = "/openbao/config/server.key"
  tls_min_version = "tls12"
}
```

**Generate TLS certificates**:

```bash
# Production: Use certificates from your CA
# Development: Generate self-signed
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \
  -days 365 -nodes -subj "/CN=openbao.internal"
```

**Update API configuration**:

```bash
OPENBAO_ADDR=https://openbao.internal:8200
```

### 3. Revoke Root Token

After initial setup, the root token should be revoked:

```bash
export VAULT_TOKEN=$(cat /openbao/init/root-token)
bao token revoke -self

# Remove root token file
rm /openbao/init/root-token
```

**Recovery**: Use the Shamir keys to generate a new root token if needed:

```bash
bao operator generate-root -init
```

### 4. Audit Logging

Enable audit logging to track all API activity:

```bash
bao audit enable file file_path=/openbao/logs/audit.log

# Verify
bao audit list
```

**Mount logs volume in docker-compose.yml**:

```yaml
openbao:
  volumes:
    - openbao_logs:/openbao/logs
```

**Log Format**: JSON, one entry per API request
**Retention**: Rotate daily, retain 90 days minimum

### 5. External Auto-Unseal

Replace file-based unseal with KMS-based auto-unseal:

**AWS KMS Example**:

```hcl
# config.hcl
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc123"
}
```

**Benefits**:

- No manual unseal required
- Unseal key never touches disk
- Audit trail in KMS service
- Automatic rotation support

### 6. High Availability

**File Storage (Current)** - Single node, no HA
**Raft Storage (Recommended)** - Multi-node with leader election

**Raft configuration**:

```hcl
storage "raft" {
  path = "/openbao/data"
  node_id = "node1"
}
```

**Run 3-5 OpenBao nodes** with Raft for production HA.

### 7. Network Isolation

**Current**: Port bound to `127.0.0.1:8200` (localhost only)
**Production**: Use internal network only, no public exposure

**docker-compose.yml**:

```yaml
openbao:
  # Remove port mapping
  # ports:
  #   - "127.0.0.1:8200:8200"
  networks:
    - internal
```

**API access**: Use reverse proxy with authentication/authorization.

### 8. Rate Limiting

Prevent brute-force attacks on AppRole authentication:

**nginx reverse proxy**:

```nginx
http {
  limit_req_zone $binary_remote_addr zone=vault_auth:10m rate=10r/s;

  server {
    listen 8200 ssl;

    location /v1/auth/approle/login {
      limit_req zone=vault_auth burst=20 nodelay;
      proxy_pass http://openbao:8200;
    }

    location / {
      proxy_pass http://openbao:8200;
    }
  }
}
```

---

## Operations

### Health Checks

**OpenBao server**:

```bash
curl http://localhost:8200/v1/sys/health
```

**Response**:

```json
{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "server_time_utc": 1738938524
}
```

**VaultService health** (API endpoint):

```bash
curl http://localhost:3000/health
```

**Response**:

```json
{
  "status": "ok",
  "info": {
    "openbao": {
      "status": "up"
    }
  }
}
```

### Manual Unseal

If auto-unseal fails:

```bash
# Get unseal key
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/unseal-key

# Unseal OpenBao
docker compose exec openbao bao operator unseal <key>
```

### Check Transit Keys

```bash
# Get root token
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)

# List Transit keys
docker compose exec -e VAULT_TOKEN openbao bao list transit/keys

# Read key details
docker compose exec -e VAULT_TOKEN openbao bao read transit/keys/mosaic-credentials
```

### Test Encryption

```bash
# Encrypt test data
PLAINTEXT=$(echo -n "test-secret" | base64)
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  transit/encrypt/mosaic-credentials plaintext=$PLAINTEXT

# Decrypt
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  transit/decrypt/mosaic-credentials ciphertext=<ciphertext>
```

### Monitor Logs

```bash
# OpenBao server logs
docker compose logs -f openbao

# Init sidecar logs
docker compose logs -f openbao-init

# VaultService logs (API)
docker compose logs -f api | grep VaultService
```

---

## Troubleshooting

### OpenBao Won't Start

**Symptoms**: Container restarts repeatedly, "connection refused" errors

**Diagnosis**:

```bash
docker compose logs openbao
```

**Common Causes**:

1. **Port conflict**: Another service on 8200
2. **Volume permission issues**: Can't write to `/openbao/data`
3. **Invalid config**: Syntax error in `config.hcl`

**Solutions**:

```bash
# Check port usage
netstat -tuln | grep 8200

# Fix volume permissions
docker compose down -v
docker volume rm mosaic-openbao-data
docker compose up -d

# Validate config
docker compose exec openbao bao server -config=/openbao/config/config.hcl -test
```

### OpenBao Sealed

**Symptoms**: API returns 503, `"sealed": true` in health check

**Diagnosis**:

```bash
docker compose exec openbao bao status
```

**Solution**:

```bash
# Auto-unseal should handle this
# Force restart init container
docker compose restart openbao-init

# Manual unseal if needed
docker compose exec openbao bao operator unseal <key>
```

### VaultService Falls Back to AES

**Symptoms**: Logs show "OpenBao unavailable, falling back to AES-256-GCM"

**Diagnosis**:

```bash
# Check OpenBao health
curl http://localhost:8200/v1/sys/health

# Check AppRole credentials
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/approle-credentials
```

**Common Causes**:

1. **OpenBao not running**: `docker compose ps openbao`
2. **Wrong OPENBAO_ADDR**: Check environment variable
3. **AppRole credentials missing**: Reinitialize with `docker compose restart openbao-init`
4. **Network issue**: Check Docker network connectivity

**Solutions**:

```bash
# Restart OpenBao stack
docker compose restart openbao openbao-init

# Verify connectivity from API container
docker compose exec api curl http://openbao:8200/v1/sys/health
```

### Authentication Failures

**Symptoms**: "Token renewal failed", "Authentication failed"

**Diagnosis**:

```bash
# Check AppRole
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)
docker compose exec -e VAULT_TOKEN openbao bao read auth/approle/role/mosaic-transit
```

**Solutions**:

```bash
# Regenerate AppRole credentials
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
  -f auth/approle/role/mosaic-transit/secret-id > new-credentials.json

# Update credentials file
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/host alpine \
  cp /host/new-credentials.json /data/approle-credentials

# Restart API
docker compose restart api
```

### Decrypt Failures

**Symptoms**: "Failed to decrypt data", encrypted tokens visible in database

**Diagnosis**:

```bash
# Check ciphertext format
psql -h localhost -U mosaic -d mosaic \
  -c "SELECT access_token FROM accounts LIMIT 1;"
```

**Ciphertext formats**:

- `vault:v1:...` - Transit encryption (requires OpenBao)
- `iv:tag:encrypted` - AES fallback (works without OpenBao)

**Solutions**:

1. **Transit ciphertext + OpenBao unavailable**: Start OpenBao
2. **Corrupted ciphertext**: Check database integrity
3. **Wrong encryption key**: Verify `ENCRYPTION_KEY` hasn't changed (for AES)

---

## Disaster Recovery

### Backup

**Critical Data**:

1. **Unseal key**: `/openbao/init/unseal-key` (Shamir keys in production)
2. **Root token**: `/openbao/init/root-token` (revoke in production, document recovery)
3. **AppRole credentials**: `/openbao/init/approle-credentials`
4. **OpenBao data**: `/openbao/data` (encrypted storage)

**Backup Procedure**:

```bash
# Backup volumes
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
  alpine tar czf /backup/openbao-init-$(date +%Y%m%d).tar.gz /data

docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
  alpine tar czf /backup/openbao-data-$(date +%Y%m%d).tar.gz /data

# Backup to remote storage
aws s3 cp openbao-*.tar.gz s3://backups/openbao/
```

**Schedule**: Daily automated backups with 30-day retention.

### Restore

**Full Recovery**:

```bash
# Download backups
aws s3 cp s3://backups/openbao/openbao-init-20260207.tar.gz .
aws s3 cp s3://backups/openbao/openbao-data-20260207.tar.gz .

# Stop containers
docker compose down

# Restore volumes
docker volume create mosaic-openbao-init
docker volume create mosaic-openbao-data

docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
  alpine tar xzf /backup/openbao-init-20260207.tar.gz -C /data --strip-components=1

docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
  alpine tar xzf /backup/openbao-data-20260207.tar.gz -C /data --strip-components=1

# Restart
docker compose up -d
```

### Key Recovery

**Lost Unseal Key**:

- **Development (1-of-1)**: Restore from backup
- **Production (3-of-5)**: Collect 3 of 5 Shamir keys from key holders

**Lost Root Token**:

```bash
# Generate new root token using Shamir keys
bao operator generate-root -init
bao operator generate-root -nonce=<nonce> <key1>
bao operator generate-root -nonce=<nonce> <key2>
bao operator generate-root -nonce=<nonce> <key3>
```

**Lost AppRole Credentials**:

```bash
# Use root token to regenerate
export VAULT_TOKEN=<root-token>
bao write -format=json -f auth/approle/role/mosaic-transit/secret-id \
  > new-approle-credentials.json
```

### Testing Recovery

**Quarterly DR test**:

1. Backup production OpenBao
2. Restore to test environment
3. Verify all Transit keys accessible
4. Test encrypt/decrypt operations
5. Document any issues

---

## Additional Resources

- **OpenBao Documentation**: https://openbao.org/docs/
- **Transit Secrets Engine**: https://openbao.org/docs/secrets/transit/
- **AppRole Auth**: https://openbao.org/docs/auth/approle/
- **Production Hardening**: https://openbao.org/docs/guides/production/
- **Design Document**: `docs/design/credential-security.md`

## Support

For issues with Mosaic Stack's OpenBao integration:

- **Repository**: https://git.mosaicstack.dev/mosaic/stack
- **Issues**: https://git.mosaicstack.dev/mosaic/stack/issues
- **Milestone**: M9-CredentialSecurity

For OpenBao itself:

- **GitHub**: https://github.com/openbao/openbao
- **Discussions**: https://github.com/openbao/openbao/discussions