diff --git a/docs/OPENBAO.md b/docs/OPENBAO.md new file mode 100644 index 0000000..402ee44 --- /dev/null +++ b/docs/OPENBAO.md @@ -0,0 +1,795 @@ +# OpenBao Integration Guide + +**Version:** 0.0.9 +**Status:** Production Ready +**Related Issues:** [#346](https://git.mosaicstack.dev/mosaic/stack/issues/346), [#357](https://git.mosaicstack.dev/mosaic/stack/issues/357), [#353](https://git.mosaicstack.dev/mosaic/stack/issues/353) + +## Table of Contents + +1. [Overview](#overview) +2. [Architecture](#architecture) +3. [Default Turnkey Setup](#default-turnkey-setup) +4. [Environment Variables](#environment-variables) +5. [Transit Encryption Keys](#transit-encryption-keys) +6. [Production Hardening](#production-hardening) +7. [Operations](#operations) +8. [Troubleshooting](#troubleshooting) +9. [Disaster Recovery](#disaster-recovery) + +--- + +## Overview + +### Why OpenBao? + +OpenBao is an open-source secrets management platform forked from HashiCorp Vault after HashiCorp changed to the Business Source License. Key benefits: + +- **Truly open-source** - Linux Foundation project with OSI-approved license +- **Drop-in Vault replacement** - API-compatible with HashiCorp Vault +- **Production-ready** - v2.0+ with active development and community support +- **Transit encryption** - Encrypt data at rest without storing plaintext in OpenBao + +### What is Transit Encryption? + +The Transit secrets engine provides "encryption as a service": + +- **Encryption/decryption operations** via API calls +- **Key versioning** - Rotate keys without re-encrypting existing data +- **No plaintext storage** - OpenBao never stores your plaintext data +- **Ciphertext format** - Versioned format (`vault:v1:...`) enables seamless key rotation + +**Use Case**: Encrypt sensitive data (OAuth tokens, API keys, credentials) before storing in PostgreSQL. If the database is compromised, attackers only get encrypted ciphertext. + +--- + +## Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Mosaic Stack API │ +│ │ +│ ┌────────────────┐ ┌──────────────────┐ │ +│ │ VaultService │───────>│ CryptoService │ │ +│ │ (Primary) │ │ (Fallback) │ │ +│ └────────┬───────┘ └──────────────────┘ │ +│ │ │ +│ │ Transit API │ +│ │ (encrypt/decrypt) │ +└───────────┼─────────────────────────────────────────────────┘ + │ + │ HTTP (localhost:8200) + ▼ +┌───────────────────────────────────────────────────────────────┐ +│ OpenBao │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ Transit Secrets Engine │ │ +│ │ │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ mosaic- │ │ mosaic- │ + 2 more │ │ +│ │ │ credentials │ │ account- │ keys │ │ +│ │ │ │ │ tokens │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ AppRole Authentication │ │ +│ │ - Role: mosaic-transit │ │ +│ │ - Policy: Transit encrypt/decrypt only │ │ +│ └────────────────────────────────────────────────────┘ │ +│ │ +│ ┌────────────────────────────────────────────────────┐ │ +│ │ File Storage Backend │ │ +│ │ /openbao/data (Docker volume) │ │ +│ └────────────────────────────────────────────────────┘ │ +└───────────────────────────────────────────────────────────────┘ + │ + │ Auto-init / Auto-unseal + ▼ +┌───────────────────────────────────────────────────────────────┐ +│ OpenBao Init Sidecar │ +│ - Initializes OpenBao on first run │ +│ - Auto-unseals on container restart │ +│ - Creates Transit keys and AppRole │ +│ - Runs continuously with 30s check loop │ +└───────────────────────────────────────────────────────────────┘ +``` + +### Data Flow: Encrypt + +1. API receives plaintext credential (e.g., OAuth access token) +2. VaultService encrypts via Transit: `POST /v1/transit/encrypt/mosaic-account-tokens` +3. OpenBao returns ciphertext: `vault:v1:8SDd3WHDOjf8Fz5MSLXjL...` +4. Ciphertext stored in PostgreSQL +5. Plaintext never touches disk + +### Data Flow: Decrypt + +1. API reads ciphertext from PostgreSQL +2. VaultService decrypts via Transit: `POST /v1/transit/decrypt/mosaic-account-tokens` +3. OpenBao returns plaintext +4. API uses plaintext for OAuth/API requests +5. Plaintext cleared from memory after use + +### Fallback Behavior + +When OpenBao is unavailable: + +- **Encryption**: VaultService falls back to AES-256-GCM (`CryptoService`) +- **Decryption**: Auto-detects format (`vault:v1:` vs AES) and uses appropriate service +- **Logging**: ERROR-level logs for visibility into infrastructure issues +- **Backward Compatibility**: Existing AES-encrypted data remains decryptable + +**Production Recommendation**: Set `OPENBAO_REQUIRED=true` to fail startup if OpenBao is unavailable (prevents accidental fallback). + +--- + +## Default Turnkey Setup + +### What Happens on First `docker compose up` + +The `openbao-init` sidecar automatically: + +1. **Waits** for OpenBao server to be healthy +2. **Initializes** OpenBao with 1-of-1 Shamir key split (development mode) +3. **Unseals** OpenBao using the stored unseal key +4. **Enables** Transit secrets engine at `/transit` +5. **Creates** 4 named encryption keys: + - `mosaic-credentials` - User-provided credentials (API keys, tokens) + - `mosaic-account-tokens` - OAuth tokens from BetterAuth accounts + - `mosaic-federation` - Federation private keys + - `mosaic-llm-config` - LLM provider API keys +6. **Creates** AppRole `mosaic-transit` with Transit-only policy +7. **Generates** AppRole credentials (role_id, secret_id) +8. **Saves** credentials to `/openbao/init/approle-credentials` +9. **Watches** continuously - auto-unseals on container restart every 30s + +### File Locations + +``` +Docker Volumes: +├── mosaic-openbao-data # OpenBao database (encrypted storage) +├── mosaic-openbao-init # Unseal key, root token, AppRole credentials +└── docker/openbao/config.hcl # Server configuration (bind mount) +``` + +### Credentials Storage + +**Unseal Key**: `/openbao/init/unseal-key` (plaintext, volume-only access) +**Root Token**: `/openbao/init/root-token` (plaintext, volume-only access) +**AppRole Credentials**: `/openbao/init/approle-credentials` (JSON, read by API) + +Example `/openbao/init/approle-credentials`: + +```json +{ + "role_id": "42474304-cb07-edff-f5c8-cae494ba6a51", + "secret_id": "39ccdb6b-4570-a279-022c-36220739ebcf" +} +``` + +**Security Note**: These files are only accessible within the Docker internal network. The OpenBao API is bound to `127.0.0.1:8200` (localhost only). + +--- + +## Environment Variables + +### Required + +None - the turnkey setup works with defaults. + +### Optional + +| Variable | Default | Purpose | +| ------------------- | --------------------- | -------------------------------------------- | +| `OPENBAO_ADDR` | `http://openbao:8200` | OpenBao API address (Docker internal) | +| `OPENBAO_PORT` | `8200` | Port for localhost binding in docker-compose | +| `OPENBAO_ROLE_ID` | (read from file) | Override AppRole role_id | +| `OPENBAO_SECRET_ID` | (read from file) | Override AppRole secret_id | +| `OPENBAO_REQUIRED` | `false` | Fail startup if OpenBao unavailable | + +### Production Configuration + +**Development**: + +```bash +OPENBAO_ADDR=http://openbao:8200 +OPENBAO_REQUIRED=false +``` + +**Production**: + +```bash +OPENBAO_ADDR=https://vault.internal.corp:8200 +OPENBAO_REQUIRED=true +# AppRole credentials provided by external Vault deployment +OPENBAO_ROLE_ID= +OPENBAO_SECRET_ID= +``` + +--- + +## Transit Encryption Keys + +### Named Keys + +| Key Name | Purpose | Used By | Example Data | +| ----------------------- | ------------------------- | ------------------------- | ------------------------------------- | +| `mosaic-credentials` | User-provided credentials | UserCredential model | GitHub PAT, AWS access keys | +| `mosaic-account-tokens` | OAuth tokens | BetterAuth accounts table | access_token, refresh_token, id_token | +| `mosaic-federation` | Federation private keys | FederatedInstance model | Ed25519 private keys | +| `mosaic-llm-config` | LLM provider credentials | LLMProviderInstance model | OpenAI API key, Anthropic API key | + +### Key Properties + +- **Algorithm**: AES-256-GCM96 +- **Versioning**: Enabled (transparent key rotation) +- **Deletion**: Not allowed (exportable = false) +- **Min Decryption Version**: 1 (all versions can decrypt) +- **Latest Version**: Auto-increments on rotation + +### Key Rotation + +OpenBao Transit keys support **transparent rotation**: + +1. Rotate key: `bao write -f transit/keys/mosaic-credentials/rotate` +2. New version created (e.g., v2) +3. **New encryptions** use v2 +4. **Old ciphertexts** still decrypt with v1 +5. **No data re-encryption needed** + +**Recommendation**: Rotate keys annually or after suspected compromise. + +### Ciphertext Format + +``` +vault:v1:8SDd3WHDOjf8Fz5MSLXjLx0AzTdYhLMAAp4W + ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + │ └─ Base64-encoded ciphertext + └───── Key version +``` + +**Version prefix** enables seamless rotation - OpenBao knows which key version to use for decryption. + +--- + +## Production Hardening + +### Security Checklist + +**CRITICAL - Do not deploy to production without these:** + +| Task | Command | Priority | +| ------------------------------------ | ----------------------------------------------------- | -------- | +| **1. Upgrade Shamir key splitting** | `bao operator rekey -key-shares=5 -key-threshold=3` | P0 | +| **2. Enable TLS on listener** | Update `config.hcl`: `tls_disable = 0` | P0 | +| **3. Revoke root token** | `bao token revoke -self` | P0 | +| **4. Enable audit logging** | `bao audit enable file file_path=/bao/logs/audit.log` | P0 | +| **5. Use external auto-unseal** | AWS KMS, GCP CKMS, Azure Key Vault | P1 | +| **6. HA storage backend** | Raft or Consul | P1 | +| **7. Bind to internal network only** | Update docker-compose.yml: remove port exposure | P1 | +| **8. Rate limiting** | Use reverse proxy (nginx/Traefik) with rate limits | P2 | +| **9. Rotate AppRole credentials** | Regenerate secret_id monthly | P2 | +| **10. Backup automation** | Snapshot Raft or file storage daily | P2 | + +### 1. Shamir Key Splitting + +**Current (Development)**: 1-of-1 key shares (single unseal key) +**Production**: 3-of-5 key shares (requires 3 of 5 keys to unseal) + +**Procedure**: + +```bash +# SSH into OpenBao container +docker compose exec openbao sh + +# Set environment +export VAULT_ADDR=http://localhost:8200 +export VAULT_TOKEN=$(cat /openbao/init/root-token) + +# Rekey to 3-of-5 +bao operator rekey -init -key-shares=5 -key-threshold=3 + +# Follow prompts to generate new keys +# Distribute 5 keys to different key holders +# Require 3 keys to unseal +``` + +**Key Management Best Practices**: + +- Store each key with a different person/system +- Use hardware security modules (HSMs) for key storage +- Document key recovery procedures +- Test unseal process regularly + +### 2. TLS Configuration + +**Update `docker/openbao/config.hcl`**: + +```hcl +listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = 0 # Changed from 1 + tls_cert_file = "/openbao/config/server.crt" + tls_key_file = "/openbao/config/server.key" + tls_min_version = "tls12" +} +``` + +**Generate TLS certificates**: + +```bash +# Production: Use certificates from your CA +# Development: Generate self-signed +openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt \ + -days 365 -nodes -subj "/CN=openbao.internal" +``` + +**Update API configuration**: + +```bash +OPENBAO_ADDR=https://openbao.internal:8200 +``` + +### 3. Revoke Root Token + +After initial setup, the root token should be revoked: + +```bash +export VAULT_TOKEN=$(cat /openbao/init/root-token) +bao token revoke -self + +# Remove root token file +rm /openbao/init/root-token +``` + +**Recovery**: Use the Shamir keys to generate a new root token if needed: + +```bash +bao operator generate-root -init +``` + +### 4. Audit Logging + +Enable audit logging to track all API activity: + +```bash +bao audit enable file file_path=/openbao/logs/audit.log + +# Verify +bao audit list +``` + +**Mount logs volume in docker-compose.yml**: + +```yaml +openbao: + volumes: + - openbao_logs:/openbao/logs +``` + +**Log Format**: JSON, one entry per API request +**Retention**: Rotate daily, retain 90 days minimum + +### 5. External Auto-Unseal + +Replace file-based unseal with KMS-based auto-unseal: + +**AWS KMS Example**: + +```hcl +# config.hcl +seal "awskms" { + region = "us-east-1" + kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abc123" +} +``` + +**Benefits**: + +- No manual unseal required +- Unseal key never touches disk +- Audit trail in KMS service +- Automatic rotation support + +### 6. High Availability + +**File Storage (Current)** - Single node, no HA +**Raft Storage (Recommended)** - Multi-node with leader election + +**Raft configuration**: + +```hcl +storage "raft" { + path = "/openbao/data" + node_id = "node1" +} +``` + +**Run 3-5 OpenBao nodes** with Raft for production HA. + +### 7. Network Isolation + +**Current**: Port bound to `127.0.0.1:8200` (localhost only) +**Production**: Use internal network only, no public exposure + +**docker-compose.yml**: + +```yaml +openbao: + # Remove port mapping + # ports: + # - "127.0.0.1:8200:8200" + networks: + - internal +``` + +**API access**: Use reverse proxy with authentication/authorization. + +### 8. Rate Limiting + +Prevent brute-force attacks on AppRole authentication: + +**nginx reverse proxy**: + +```nginx +http { + limit_req_zone $binary_remote_addr zone=vault_auth:10m rate=10r/s; + + server { + listen 8200 ssl; + + location /v1/auth/approle/login { + limit_req zone=vault_auth burst=20 nodelay; + proxy_pass http://openbao:8200; + } + + location / { + proxy_pass http://openbao:8200; + } + } +} +``` + +--- + +## Operations + +### Health Checks + +**OpenBao server**: + +```bash +curl http://localhost:8200/v1/sys/health +``` + +**Response**: + +```json +{ + "initialized": true, + "sealed": false, + "standby": false, + "server_time_utc": 1738938524 +} +``` + +**VaultService health** (API endpoint): + +```bash +curl http://localhost:3000/health +``` + +**Response**: + +```json +{ + "status": "ok", + "info": { + "openbao": { + "status": "up" + } + } +} +``` + +### Manual Unseal + +If auto-unseal fails: + +```bash +# Get unseal key +docker run --rm -v mosaic-openbao-init:/data alpine cat /data/unseal-key + +# Unseal OpenBao +docker compose exec openbao bao operator unseal +``` + +### Check Transit Keys + +```bash +# Get root token +export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token) + +# List Transit keys +docker compose exec -e VAULT_TOKEN openbao bao list transit/keys + +# Read key details +docker compose exec -e VAULT_TOKEN openbao bao read transit/keys/mosaic-credentials +``` + +### Test Encryption + +```bash +# Encrypt test data +PLAINTEXT=$(echo -n "test-secret" | base64) +docker compose exec -e VAULT_TOKEN openbao bao write -format=json \ + transit/encrypt/mosaic-credentials plaintext=$PLAINTEXT + +# Decrypt +docker compose exec -e VAULT_TOKEN openbao bao write -format=json \ + transit/decrypt/mosaic-credentials ciphertext= +``` + +### Monitor Logs + +```bash +# OpenBao server logs +docker compose logs -f openbao + +# Init sidecar logs +docker compose logs -f openbao-init + +# VaultService logs (API) +docker compose logs -f api | grep VaultService +``` + +--- + +## Troubleshooting + +### OpenBao Won't Start + +**Symptoms**: Container restarts repeatedly, "connection refused" errors + +**Diagnosis**: + +```bash +docker compose logs openbao +``` + +**Common Causes**: + +1. **Port conflict**: Another service on 8200 +2. **Volume permission issues**: Can't write to `/openbao/data` +3. **Invalid config**: Syntax error in `config.hcl` + +**Solutions**: + +```bash +# Check port usage +netstat -tuln | grep 8200 + +# Fix volume permissions +docker compose down -v +docker volume rm mosaic-openbao-data +docker compose up -d + +# Validate config +docker compose exec openbao bao server -config=/openbao/config/config.hcl -test +``` + +### OpenBao Sealed + +**Symptoms**: API returns 503, `"sealed": true` in health check + +**Diagnosis**: + +```bash +docker compose exec openbao bao status +``` + +**Solution**: + +```bash +# Auto-unseal should handle this +# Force restart init container +docker compose restart openbao-init + +# Manual unseal if needed +docker compose exec openbao bao operator unseal +``` + +### VaultService Falls Back to AES + +**Symptoms**: Logs show "OpenBao unavailable, falling back to AES-256-GCM" + +**Diagnosis**: + +```bash +# Check OpenBao health +curl http://localhost:8200/v1/sys/health + +# Check AppRole credentials +docker run --rm -v mosaic-openbao-init:/data alpine cat /data/approle-credentials +``` + +**Common Causes**: + +1. **OpenBao not running**: `docker compose ps openbao` +2. **Wrong OPENBAO_ADDR**: Check environment variable +3. **AppRole credentials missing**: Reinitialize with `docker compose restart openbao-init` +4. **Network issue**: Check Docker network connectivity + +**Solutions**: + +```bash +# Restart OpenBao stack +docker compose restart openbao openbao-init + +# Verify connectivity from API container +docker compose exec api curl http://openbao:8200/v1/sys/health +``` + +### Authentication Failures + +**Symptoms**: "Token renewal failed", "Authentication failed" + +**Diagnosis**: + +```bash +# Check AppRole +export VAULT_TOKEN=$(docker run --rm -v mosaic-openbao-init:/data alpine cat /data/root-token) +docker compose exec -e VAULT_TOKEN openbao bao read auth/approle/role/mosaic-transit +``` + +**Solutions**: + +```bash +# Regenerate AppRole credentials +docker compose exec -e VAULT_TOKEN openbao bao write -format=json \ + -f auth/approle/role/mosaic-transit/secret-id > new-credentials.json + +# Update credentials file +docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/host alpine \ + cp /host/new-credentials.json /data/approle-credentials + +# Restart API +docker compose restart api +``` + +### Decrypt Failures + +**Symptoms**: "Failed to decrypt data", encrypted tokens visible in database + +**Diagnosis**: + +```bash +# Check ciphertext format +psql -h localhost -U mosaic -d mosaic \ + -c "SELECT access_token FROM accounts LIMIT 1;" +``` + +**Ciphertext formats**: + +- `vault:v1:...` - Transit encryption (requires OpenBao) +- `iv:tag:encrypted` - AES fallback (works without OpenBao) + +**Solutions**: + +1. **Transit ciphertext + OpenBao unavailable**: Start OpenBao +2. **Corrupted ciphertext**: Check database integrity +3. **Wrong encryption key**: Verify `ENCRYPTION_KEY` hasn't changed (for AES) + +--- + +## Disaster Recovery + +### Backup + +**Critical Data**: + +1. **Unseal key**: `/openbao/init/unseal-key` (Shamir keys in production) +2. **Root token**: `/openbao/init/root-token` (revoke in production, document recovery) +3. **AppRole credentials**: `/openbao/init/approle-credentials` +4. **OpenBao data**: `/openbao/data` (encrypted storage) + +**Backup Procedure**: + +```bash +# Backup volumes +docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \ + alpine tar czf /backup/openbao-init-$(date +%Y%m%d).tar.gz /data + +docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \ + alpine tar czf /backup/openbao-data-$(date +%Y%m%d).tar.gz /data + +# Backup to remote storage +aws s3 cp openbao-*.tar.gz s3://backups/openbao/ +``` + +**Schedule**: Daily automated backups with 30-day retention. + +### Restore + +**Full Recovery**: + +```bash +# Download backups +aws s3 cp s3://backups/openbao/openbao-init-20260207.tar.gz . +aws s3 cp s3://backups/openbao/openbao-data-20260207.tar.gz . + +# Stop containers +docker compose down + +# Restore volumes +docker volume create mosaic-openbao-init +docker volume create mosaic-openbao-data + +docker run --rm -v mosaic-openbao-init:/data -v $(pwd):/backup \ + alpine tar xzf /backup/openbao-init-20260207.tar.gz -C /data --strip-components=1 + +docker run --rm -v mosaic-openbao-data:/data -v $(pwd):/backup \ + alpine tar xzf /backup/openbao-data-20260207.tar.gz -C /data --strip-components=1 + +# Restart +docker compose up -d +``` + +### Key Recovery + +**Lost Unseal Key**: + +- **Development (1-of-1)**: Restore from backup +- **Production (3-of-5)**: Collect 3 of 5 Shamir keys from key holders + +**Lost Root Token**: + +```bash +# Generate new root token using Shamir keys +bao operator generate-root -init +bao operator generate-root -nonce= +bao operator generate-root -nonce= +bao operator generate-root -nonce= +``` + +**Lost AppRole Credentials**: + +```bash +# Use root token to regenerate +export VAULT_TOKEN= +bao write -format=json -f auth/approle/role/mosaic-transit/secret-id \ + > new-approle-credentials.json +``` + +### Testing Recovery + +**Quarterly DR test**: + +1. Backup production OpenBao +2. Restore to test environment +3. Verify all Transit keys accessible +4. Test encrypt/decrypt operations +5. Document any issues + +--- + +## Additional Resources + +- **OpenBao Documentation**: https://openbao.org/docs/ +- **Transit Secrets Engine**: https://openbao.org/docs/secrets/transit/ +- **AppRole Auth**: https://openbao.org/docs/auth/approle/ +- **Production Hardening**: https://openbao.org/docs/guides/production/ +- **Design Document**: `docs/design/credential-security.md` + +## Support + +For issues with Mosaic Stack's OpenBao integration: + +- **Repository**: https://git.mosaicstack.dev/mosaic/stack +- **Issues**: https://git.mosaicstack.dev/mosaic/stack/issues +- **Milestone**: M9-CredentialSecurity + +For OpenBao itself: + +- **GitHub**: https://github.com/openbao/openbao +- **Discussions**: https://github.com/openbao/openbao/discussions