mosaic/stack
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

docs(#354): Add comprehensive OpenBao integration guide
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Details

Browse Source 
Complete documentation for OpenBao Transit encryption covering setup,
architecture, production hardening, and operations.

Sections:
- Overview: Why OpenBao, Transit encryption explained
- Architecture: Data flow diagrams, fallback behavior
- Default Setup: Turnkey auto-init/unseal, file locations
- Environment Variables: Configuration options
- Transit Keys: Named keys, rotation procedures
- Production Hardening: 10-point security checklist
- Operations: Health checks, manual procedures, monitoring
- Troubleshooting: Common issues and solutions
- Disaster Recovery: Backup/restore procedures

Key Topics:
- Shamir key splitting upgrade (1-of-1 → 3-of-5)
- TLS configuration for production
- Audit logging enablement
- HA storage backends (Raft/Consul)
- External auto-unseal with KMS
- Rate limiting via reverse proxy
- Network isolation best practices
- Key rotation procedures
- Backup automation

Closes #354

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Jason Woltje
2026-02-07 16:16:51 -06:00
parent dd171b287f
commit 40f7e7e4c0
1 changed files with 795 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

795
docs/OPENBAO.md Normal file
View File

@@ -0,0 +1,795 @@
# OpenBao Integration Guide
**Version:** 0.0.9
**Status:** Production Ready
**Related Issues:** [#346](https://git.mosaicstack.dev/mosaic/stack/issues/346), [#357](https://git.mosaicstack.dev/mosaic/stack/issues/357), [#353](https://git.mosaicstack.dev/mosaic/stack/issues/353)
## Table of Contents
1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Default Turnkey Setup](#default-turnkey-setup)
4. [Environment Variables](#environment-variables)
5. [Transit Encryption Keys](#transit-encryption-keys)
6. [Production Hardening](#production-hardening)
7. [Operations](#operations)
8. [Troubleshooting](#troubleshooting)
9. [Disaster Recovery](#disaster-recovery)
---
## Overview
### Why OpenBao?
OpenBao is an open-source secrets management platform forked from HashiCorp Vault after HashiCorp changed to the Business Source License. Key benefits:
- **Truly open-source** - Linux Foundation project with OSI-approved license
- **Drop-in Vault replacement** - API-compatible with HashiCorp Vault
- **Production-ready** - v2.0+ with active development and community support
- **Transit encryption** - Encrypt data at rest without storing plaintext in OpenBao
### What is Transit Encryption?
The Transit secrets engine provides "encryption as a service":
- **Encryption/decryption operations** via API calls
- **Key versioning** - Rotate keys without re-encrypting existing data
- **No plaintext storage** - OpenBao never stores your plaintext data
- **Ciphertext format** - Versioned format (`vault:v1:...`) enables seamless key rotation
**Use Case**: Encrypt sensitive data (OAuth tokens, API keys, credentials) before storing in PostgreSQL. If the database is compromised, attackers only get encrypted ciphertext.
---
## Architecture
```
┌──────────────────────────────────────────────────────────────┐
│ Mosaic Stack API │
│ │
│ ┌────────────────┐ ┌──────────────────┐ │
│ │ VaultService │───────>│ CryptoService │ │
│ │ (Primary) │ │ (Fallback) │ │
│ └────────┬───────┘ └──────────────────┘ │
│ │ │
│ │ Transit API │
│ │ (encrypt/decrypt) │
└───────────┼─────────────────────────────────────────────────┘
│
│ HTTP (localhost:8200)
▼
┌───────────────────────────────────────────────────────────────┐
│ OpenBao │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Transit Secrets Engine │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ mosaic- │ │ mosaic- │ + 2 more │ │
│ │ │ credentials │ │ account- │ keys │ │
│ │ │ │ │ tokens │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ AppRole Authentication │ │
│ │ - Role: mosaic-transit │ │
│ │ - Policy: Transit encrypt/decrypt only │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ File Storage Backend │ │
│ │ /openbao/data (Docker volume) │ │
│ └────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
│
│ Auto-init / Auto-unseal
▼
┌───────────────────────────────────────────────────────────────┐
│ OpenBao Init Sidecar │
│ - Initializes OpenBao on first run │
│ - Auto-unseals on container restart │
│ - Creates Transit keys and AppRole │
│ - Runs continuously with 30s check loop │
└───────────────────────────────────────────────────────────────┘
```
### Data Flow: Encrypt
1. API receives plaintext credential (e.g., OAuth access token)
2. VaultService encrypts via Transit: `POST /v1/transit/encrypt/mosaic-account-tokens`
3. OpenBao returns ciphertext: `vault:v1:8SDd3WHDOjf8Fz5MSLXjL...`
4. Ciphertext stored in PostgreSQL
5. Plaintext never touches disk
### Data Flow: Decrypt
1. API reads ciphertext from PostgreSQL
2. VaultService decrypts via Transit: `POST /v1/transit/decrypt/mosaic-account-tokens`
3. OpenBao returns plaintext
4. API uses plaintext for OAuth/API requests
5. Plaintext cleared from memory after use
### Fallback Behavior
When OpenBao is unavailable:
- **Encryption**: VaultService falls back to AES-256-GCM (`CryptoService`)
- **Decryption**: Auto-detects format (`vault:v1:` vs AES) and uses appropriate service
- **Logging**: ERROR-level logs for visibility into infrastructure issues
- **Backward Compatibility**: Existing AES-encrypted data remains decryptable
**Production Recommendation**: Set `OPENBAO_REQUIRED=true` to fail startup if OpenBao is unavailable (prevents accidental fallback).
---
## Default Turnkey Setup
### What Happens on First `docker compose up`
The `openbao-init` sidecar automatically:
1. **Waits** for OpenBao server to be healthy
2. **Initializes** OpenBao with 1-of-1 Shamir key split (development mode)
3. **Unseals** OpenBao using the stored unseal key
4. **Enables** Transit secrets engine at `/transit`
5. **Creates** 4 named encryption keys:
- `mosaic-credentials` - User-provided credentials (API keys, tokens)
- `mosaic-account-tokens` - OAuth tokens from BetterAuth accounts
- `mosaic-federation` - Federation private keys
- `mosaic-llm-config` - LLM provider API keys
6. **Creates** AppRole `mosaic-transit` with Transit-only policy
7. **Generates** AppRole credentials (role_id, secret_id)
8. **Saves** credentials to `/openbao/init/approle-credentials`
9. **Watches** continuously - auto-unseals on container restart every 30s
### File Locations
```
Docker Volumes:
├── mosaic-openbao-data # OpenBao database (encrypted storage)
├── mosaic-openbao-init # Unseal key, root token, AppRole credentials
└── docker/openbao/config.hcl # Server configuration (bind mount)
```
### Credentials Storage
**Unseal Key**: `/openbao/init/unseal-key` (plaintext, volume-only access)
**Root Token**: `/openbao/init/root-token` (plaintext, volume-only access)
**AppRole Credentials**: `/openbao/init/approle-credentials` (JSON, read by API)
Example `/openbao/init/approle-credentials`:
```json
{
"role_id": "42474304-cb07-edff-f5c8-cae494ba6a51",
"secret_id": "39ccdb6b-4570-a279-022c-36220739ebcf"
}
```
**Security Note**: These files are only accessible within the Docker internal network. The OpenBao API is bound to `127.0.0.1:8200` (localhost only).
---
## Environment Variables
### Required
None - the turnkey setup works with defaults.
### Optional
| Variable | Default | Purpose |
| ------------------- | --------------------- | -------------------------------------------- |
| `OPENBAO_ADDR` | `http://openbao:8200` | OpenBao API address (Docker internal) |
| `OPENBAO_PORT` | `8200` | Port for localhost binding in docker-compose |
| `OPENBAO_ROLE_ID` | (read from file) | Override AppRole role_id |
| `OPENBAO_SECRET_ID` | (read from file) | Override AppRole secret_id |
| `OPENBAO_REQUIRED` | `false` | Fail startup if OpenBao unavailable |
### Production Configuration
**Development**:
```bash
OPENBAO_ADDR=http://openbao:8200
OPENBAO_REQUIRED=false
```
**Production**:
```bash
OPENBAO_ADDR=https://vault.internal.corp:8200
OPENBAO_REQUIRED=true
# AppRole credentials provided by external Vault deployment
OPENBAO_ROLE_ID=<from-external-vault>
OPENBAO_SECRET_ID=<from-external-vault>
```
---
## Transit Encryption Keys
### Named Keys
| Key Name | Purpose | Used By | Example Data |
| ----------------------- | ------------------------- | ------------------------- | ------------------------------------- |
| `mosaic-credentials` | User-provided credentials | UserCredential model | GitHub PAT, AWS access keys |
| `mosaic-account-tokens` | OAuth tokens | BetterAuth accounts table | access_token, refresh_token, id_token |
| `mosaic-federation` | Federation private keys | FederatedInstance model | Ed25519 private keys |
| `mosaic-llm-config` | LLM provider credentials | LLMProviderInstance model | OpenAI API key, Anthropic API key |
### Key Properties
- **Algorithm**: AES-256-GCM96
- **Versioning**: Enabled (transparent key rotation)
- **Deletion**: Not allowed (exportable = false)
- **Min Decryption Version**: 1 (all versions can decrypt)
- **Latest Version**: Auto-increments on rotation
### Key Rotation
OpenBao Transit keys support **transparent rotation**:
1. Rotate key: `bao write -f transit/keys/mosaic-credentials/rotate`
2. New version created (e.g., v2)
3. **New encryptions** use v2
4. **Old ciphertexts** still decrypt with v1
5. **No data re-encryption needed**
**Recommendation**: Rotate keys annually or after suspected compromise.
### Ciphertext Format
```
vault:v1:8SDd3WHDOjf8Fz5MSLXjLx0AzTdYhLMAAp4W
^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ └─ Base64-encoded ciphertext
└───── Key version
```
**Version prefix** enables seamless rotation - OpenBao knows which key version to use for decryption.
---
## Production Hardening
### Security Checklist
**CRITICAL - Do not deploy to production without these:**
| Task | Command | Priority |
| ------------------------------------ | ----------------------------------------------------- | -------- |
| **1. Upgrade Shamir key splitting** | `bao operator rekey -key-shares=5 -key-threshold=3` | P0 |
| **2. Enable TLS on listener** | Update `config.hcl`: `tls_disable = 0` | P0 |
| **3. Revoke root token** | `bao token revoke -self` | P0 |
| **4. Enable audit logging** | `bao audit enable file file_path=/bao/logs/audit.log` | P0 |
| **5. Use external auto-unseal** | AWS KMS, GCP CKMS, Azure Key Vault | P1 |
| **6. HA storage backend** | Raft or Consul | P1 |
| **7. Bind to internal network only** | Update docker-compose.yml: remove port exposure | P1 |
| **8. Rate limiting** | Use reverse proxy (nginx/Traefik) with rate limits | P2 |
| **9. Rotate AppRole credentials** | Regenerate secret_id monthly | P2 |
| **10. Backup automation** | Snapshot Raft or file storage daily | P2 |
### 1. Shamir Key Splitting
**Current (Development)**: 1-of-1 key shares (single unseal key)
**Production**: 3-of-5 key shares (requires 3 of 5 keys to unseal)
**Procedure**:
```bash
# SSH into OpenBao container
docker compose exec openbao sh
# Set environment
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat /openbao/init/root-token)
# Rekey to 3-of-5
bao operator rekey -init -key-shares=5 -key-threshold=3
# Follow prompts to generate new keys
# Distribute 5 keys to different key holders
# Require 3 keys to unseal
```
**Key Management Best Practices**:
- Store each key with a different person/system
- Use hardware security modules (HSMs) for key storage
- Document key recovery procedures
- Test unseal process regularly
### 2. TLS Configuration
**Update `docker/openbao/config.hcl`**:
```hcl
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 0 # Changed from 1
tls_cert_file = "/openbao/config/server.crt"
tls_key_file = "/openbao/config/server.key"
tls_min_version = "tls12"
}
```
**Generate TLS certificates**:
```bash
# Production: Use certificates from your CA
# Development: Generate self-signed
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \
-days 365 -nodes -subj "/CN=openbao.internal"
```
**Update API configuration**:
```bash
OPENBAO_ADDR=https://openbao.internal:8200
```
### 3. Revoke Root Token
After initial setup, the root token should be revoked:
```bash
export VAULT_TOKEN=$(cat /openbao/init/root-token)
bao token revoke -self
# Remove root token file
rm /openbao/init/root-token
```
**Recovery**: Use the Shamir keys to generate a new root token if needed:
```bash
bao operator generate-root -init
```
### 4. Audit Logging
Enable audit logging to track all API activity:
```bash
bao audit enable file file_path=/openbao/logs/audit.log
# Verify
bao audit list
```
**Mount logs volume in docker-compose.yml**:
```yaml
openbao:
volumes:
- openbao_logs:/openbao/logs
```
**Log Format**: JSON, one entry per API request
**Retention**: Rotate daily, retain 90 days minimum
### 5. External Auto-Unseal
Replace file-based unseal with KMS-based auto-unseal:
**AWS KMS Example**:
```hcl
# config.hcl
seal "awskms" {
region = "us-east-1"
kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc123"
}
```
**Benefits**:
- No manual unseal required
- Unseal key never touches disk
- Audit trail in KMS service
- Automatic rotation support
### 6. High Availability
**File Storage (Current)** - Single node, no HA
**Raft Storage (Recommended)** - Multi-node with leader election
**Raft configuration**:
```hcl
storage "raft" {
path = "/openbao/data"
node_id = "node1"
}
```
**Run 3-5 OpenBao nodes** with Raft for production HA.
### 7. Network Isolation
**Current**: Port bound to `127.0.0.1:8200` (localhost only)
**Production**: Use internal network only, no public exposure
**docker-compose.yml**:
```yaml
openbao:
# Remove port mapping
# ports:
# - "127.0.0.1:8200:8200"
networks:
- internal
```
**API access**: Use reverse proxy with authentication/authorization.
### 8. Rate Limiting
Prevent brute-force attacks on AppRole authentication:
**nginx reverse proxy**:
```nginx
http {
limit_req_zone $binary_remote_addr zone=vault_auth:10m rate=10r/s;
server {
listen 8200 ssl;
location /v1/auth/approle/login {
limit_req zone=vault_auth burst=20 nodelay;
proxy_pass http://openbao:8200;
}
location / {
proxy_pass http://openbao:8200;
}
}
}
```
---
## Operations
### Health Checks
**OpenBao server**:
```bash
curl http://localhost:8200/v1/sys/health
```
**Response**:
```json
{
"initialized": true,
"sealed": false,
"standby": false,
"server_time_utc": 1738938524
}
```
**VaultService health** (API endpoint):
```bash
curl http://localhost:3000/health
```
**Response**:
```json
{
"status": "ok",
"info": {
"openbao": {
"status": "up"
}
}
}
```
### Manual Unseal
If auto-unseal fails:
```bash
# Get unseal key
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/unseal-key
# Unseal OpenBao
docker compose exec openbao bao operator unseal <key>
```
### Check Transit Keys
```bash
# Get root token
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)
# List Transit keys
docker compose exec -e VAULT_TOKEN openbao bao list transit/keys
# Read key details
docker compose exec -e VAULT_TOKEN openbao bao read transit/keys/mosaic-credentials
```
### Test Encryption
```bash
# Encrypt test data
PLAINTEXT=$(echo -n "test-secret" | base64)
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
transit/encrypt/mosaic-credentials plaintext=$PLAINTEXT
# Decrypt
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
transit/decrypt/mosaic-credentials ciphertext=<ciphertext>
```
### Monitor Logs
```bash
# OpenBao server logs
docker compose logs -f openbao
# Init sidecar logs
docker compose logs -f openbao-init
# VaultService logs (API)
docker compose logs -f api | grep VaultService
```
---
## Troubleshooting
### OpenBao Won't Start
**Symptoms**: Container restarts repeatedly, "connection refused" errors
**Diagnosis**:
```bash
docker compose logs openbao
```
**Common Causes**:
1. **Port conflict**: Another service on 8200
2. **Volume permission issues**: Can't write to `/openbao/data`
3. **Invalid config**: Syntax error in `config.hcl`
**Solutions**:
```bash
# Check port usage
netstat -tuln | grep 8200
# Fix volume permissions
docker compose down -v
docker volume rm mosaic-openbao-data
docker compose up -d
# Validate config
docker compose exec openbao bao server -config=/openbao/config/config.hcl -test
```
### OpenBao Sealed
**Symptoms**: API returns 503, `"sealed": true` in health check
**Diagnosis**:
```bash
docker compose exec openbao bao status
```
**Solution**:
```bash
# Auto-unseal should handle this
# Force restart init container
docker compose restart openbao-init
# Manual unseal if needed
docker compose exec openbao bao operator unseal <key>
```
### VaultService Falls Back to AES
**Symptoms**: Logs show "OpenBao unavailable, falling back to AES-256-GCM"
**Diagnosis**:
```bash
# Check OpenBao health
curl http://localhost:8200/v1/sys/health
# Check AppRole credentials
docker run --rm -v mosaic-openbao-init:/data alpine cat /data/approle-credentials
```
**Common Causes**:
1. **OpenBao not running**: `docker compose ps openbao`
2. **Wrong OPENBAO_ADDR**: Check environment variable
3. **AppRole credentials missing**: Reinitialize with `docker compose restart openbao-init`
4. **Network issue**: Check Docker network connectivity
**Solutions**:
```bash
# Restart OpenBao stack
docker compose restart openbao openbao-init
# Verify connectivity from API container
docker compose exec api curl http://openbao:8200/v1/sys/health
```
### Authentication Failures
**Symptoms**: "Token renewal failed", "Authentication failed"
**Diagnosis**:
```bash
# Check AppRole
export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token)
docker compose exec -e VAULT_TOKEN openbao bao read auth/approle/role/mosaic-transit
```
**Solutions**:
```bash
# Regenerate AppRole credentials
docker compose exec -e VAULT_TOKEN openbao bao write -format=json \
-f auth/approle/role/mosaic-transit/secret-id > new-credentials.json
# Update credentials file
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/host alpine \
cp /host/new-credentials.json /data/approle-credentials
# Restart API
docker compose restart api
```
### Decrypt Failures
**Symptoms**: "Failed to decrypt data", encrypted tokens visible in database
**Diagnosis**:
```bash
# Check ciphertext format
psql -h localhost -U mosaic -d mosaic \
-c "SELECT access_token FROM accounts LIMIT 1;"
```
**Ciphertext formats**:
- `vault:v1:...` - Transit encryption (requires OpenBao)
- `iv:tag:encrypted` - AES fallback (works without OpenBao)
**Solutions**:
1. **Transit ciphertext + OpenBao unavailable**: Start OpenBao
2. **Corrupted ciphertext**: Check database integrity
3. **Wrong encryption key**: Verify `ENCRYPTION_KEY` hasn't changed (for AES)
---
## Disaster Recovery
### Backup
**Critical Data**:
1. **Unseal key**: `/openbao/init/unseal-key` (Shamir keys in production)
2. **Root token**: `/openbao/init/root-token` (revoke in production, document recovery)
3. **AppRole credentials**: `/openbao/init/approle-credentials`
4. **OpenBao data**: `/openbao/data` (encrypted storage)
**Backup Procedure**:
```bash
# Backup volumes
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
alpine tar czf /backup/openbao-init-$(date +%Y%m%d).tar.gz /data
docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
alpine tar czf /backup/openbao-data-$(date +%Y%m%d).tar.gz /data
# Backup to remote storage
aws s3 cp openbao-*.tar.gz s3://backups/openbao/
```
**Schedule**: Daily automated backups with 30-day retention.
### Restore
**Full Recovery**:
```bash
# Download backups
aws s3 cp s3://backups/openbao/openbao-init-20260207.tar.gz .
aws s3 cp s3://backups/openbao/openbao-data-20260207.tar.gz .
# Stop containers
docker compose down
# Restore volumes
docker volume create mosaic-openbao-init
docker volume create mosaic-openbao-data
docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \
alpine tar xzf /backup/openbao-init-20260207.tar.gz -C /data --strip-components=1
docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \
alpine tar xzf /backup/openbao-data-20260207.tar.gz -C /data --strip-components=1
# Restart
docker compose up -d
```
### Key Recovery
**Lost Unseal Key**:
- **Development (1-of-1)**: Restore from backup
- **Production (3-of-5)**: Collect 3 of 5 Shamir keys from key holders
**Lost Root Token**:
```bash
# Generate new root token using Shamir keys
bao operator generate-root -init
bao operator generate-root -nonce=<nonce> <key1>
bao operator generate-root -nonce=<nonce> <key2>
bao operator generate-root -nonce=<nonce> <key3>
```
**Lost AppRole Credentials**:
```bash
# Use root token to regenerate
export VAULT_TOKEN=<root-token>
bao write -format=json -f auth/approle/role/mosaic-transit/secret-id \
> new-approle-credentials.json
```
### Testing Recovery
**Quarterly DR test**:
1. Backup production OpenBao
2. Restore to test environment
3. Verify all Transit keys accessible
4. Test encrypt/decrypt operations
5. Document any issues
---
## Additional Resources
- **OpenBao Documentation**: https://openbao.org/docs/
- **Transit Secrets Engine**: https://openbao.org/docs/secrets/transit/
- **AppRole Auth**: https://openbao.org/docs/auth/approle/
- **Production Hardening**: https://openbao.org/docs/guides/production/
- **Design Document**: `docs/design/credential-security.md`
## Support
For issues with Mosaic Stack's OpenBao integration:
- **Repository**: https://git.mosaicstack.dev/mosaic/stack
- **Issues**: https://git.mosaicstack.dev/mosaic/stack/issues
- **Milestone**: M9-CredentialSecurity
For OpenBao itself:
- **GitHub**: https://github.com/openbao/openbao
- **Discussions**: https://github.com/openbao/openbao/discussions