feat(#37-41): Add domains, ideas, relationships, agents, widgets schema
Schema additions for issues #37-41: New models: - Domain (#37): Life domains (work, marriage, homelab, etc.) - Idea (#38): Brain dumps with pgvector embeddings - Relationship (#39): Generic entity linking (blocks, depends_on) - Agent (#40): ClawdBot agent tracking with metrics - AgentSession (#40): Conversation session tracking - WidgetDefinition (#41): HUD widget registry - UserLayout (#41): Per-user dashboard configuration Updated models: - Task, Event, Project: Added domainId foreign key - User, Workspace: Added new relations New enums: - IdeaStatus: CAPTURED, PROCESSING, ACTIONABLE, ARCHIVED, DISCARDED - RelationshipType: BLOCKS, BLOCKED_BY, DEPENDS_ON, etc. - AgentStatus: IDLE, WORKING, WAITING, ERROR, TERMINATED - EntityType: Added IDEA, DOMAIN Migration: 20260129182803_add_domains_ideas_agents_widgets
This commit is contained in:
Jason Woltje
416
docs/1-getting-started/4-docker-deployment/README.md
Normal file
416
docs/1-getting-started/4-docker-deployment/README.md
Normal file
@@ -0,0 +1,416 @@
|
||||
# Docker Deployment
|
||||
|
||||
Complete guide for deploying Mosaic Stack using Docker Compose.
|
||||
|
||||
## Overview
|
||||
|
||||
Mosaic Stack provides a turnkey Docker Compose setup that includes:
|
||||
|
||||
- **PostgreSQL 17** with pgvector extension
|
||||
- **Valkey** (Redis-compatible cache)
|
||||
- **Traefik** (optional reverse proxy)
|
||||
- **Authentik** (optional OIDC provider)
|
||||
- **Ollama** (optional AI service)
|
||||
- **Mosaic API** (NestJS backend)
|
||||
- **Mosaic Web** (Next.js frontend)
|
||||
|
||||
## Quick Start
|
||||
|
||||
Start the entire stack with one command:
|
||||
|
||||
```bash
|
||||
# Copy environment configuration
|
||||
cp .env.example .env
|
||||
|
||||
# Edit .env with your settings
|
||||
nano .env
|
||||
|
||||
# Start core services (PostgreSQL, Valkey, API, Web)
|
||||
docker compose up -d
|
||||
|
||||
# Check service status
|
||||
docker compose ps
|
||||
|
||||
# View logs
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
That's it! Your Mosaic Stack is now running:
|
||||
- Web: http://localhost:3000
|
||||
- API: http://localhost:3001
|
||||
- PostgreSQL: localhost:5432
|
||||
- Valkey: localhost:6379
|
||||
|
||||
## Service Profiles
|
||||
|
||||
Mosaic Stack uses Docker Compose profiles to enable optional services:
|
||||
|
||||
### Core Services (Always Active)
|
||||
- `postgres` - PostgreSQL database
|
||||
- `valkey` - Valkey cache
|
||||
- `api` - Mosaic API
|
||||
- `web` - Mosaic Web
|
||||
|
||||
### Optional Services (Profiles)
|
||||
|
||||
#### Traefik (Reverse Proxy)
|
||||
```bash
|
||||
# Start with bundled Traefik
|
||||
docker compose --profile traefik-bundled up -d
|
||||
|
||||
# Or set in .env
|
||||
COMPOSE_PROFILES=traefik-bundled
|
||||
```
|
||||
|
||||
Services included:
|
||||
- `traefik` - Traefik reverse proxy with dashboard (http://localhost:8080)
|
||||
|
||||
See [Traefik Integration Guide](traefik.md) for detailed configuration options including upstream mode.
|
||||
|
||||
#### Authentik (OIDC Provider)
|
||||
```bash
|
||||
# Start with Authentik
|
||||
docker compose --profile authentik up -d
|
||||
|
||||
# Or set in .env
|
||||
COMPOSE_PROFILES=authentik
|
||||
```
|
||||
|
||||
Services included:
|
||||
- `authentik-postgres` - Authentik database
|
||||
- `authentik-redis` - Authentik cache
|
||||
- `authentik-server` - Authentik OIDC server (http://localhost:9000)
|
||||
- `authentik-worker` - Authentik background worker
|
||||
|
||||
#### Ollama (AI Service)
|
||||
```bash
|
||||
# Start with Ollama
|
||||
docker compose --profile ollama up -d
|
||||
|
||||
# Or set in .env
|
||||
COMPOSE_PROFILES=ollama
|
||||
```
|
||||
|
||||
Services included:
|
||||
- `ollama` - Ollama LLM service (http://localhost:11434)
|
||||
|
||||
#### All Services
|
||||
```bash
|
||||
# Start everything
|
||||
docker compose --profile full up -d
|
||||
|
||||
# Or set in .env
|
||||
COMPOSE_PROFILES=full
|
||||
```
|
||||
|
||||
## Deployment Modes
|
||||
|
||||
### Turnkey Deployment (Recommended for Development)
|
||||
|
||||
Uses all bundled services:
|
||||
|
||||
```bash
|
||||
# Start core services
|
||||
docker compose up -d
|
||||
|
||||
# Or with optional services
|
||||
docker compose --profile full up -d
|
||||
```
|
||||
|
||||
### Customized Deployment (Production)
|
||||
|
||||
Use external services for production:
|
||||
|
||||
1. Copy override template:
|
||||
```bash
|
||||
cp docker-compose.override.yml.example docker-compose.override.yml
|
||||
```
|
||||
|
||||
2. Edit `docker-compose.override.yml` to:
|
||||
- Disable bundled services
|
||||
- Point to external services
|
||||
- Add custom configuration
|
||||
|
||||
3. Start stack:
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Docker automatically merges `docker-compose.yml` and `docker-compose.override.yml`.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Network Configuration
|
||||
|
||||
Mosaic Stack uses two Docker networks to organize service communication:
|
||||
|
||||
#### mosaic-internal (Backend Services)
|
||||
- **Purpose**: Isolates database and cache services
|
||||
- **Services**:
|
||||
- PostgreSQL (main database)
|
||||
- Valkey (main cache)
|
||||
- Authentik PostgreSQL
|
||||
- Authentik Redis
|
||||
- Ollama (when using bundled service)
|
||||
- **Connectivity**:
|
||||
- Not marked as `internal: true` to allow API to reach external services
|
||||
- API can connect to external Authentik and Ollama instances
|
||||
- Database and cache services only accessible within Docker network
|
||||
- No direct external access to database/cache ports (unless explicitly exposed)
|
||||
|
||||
#### mosaic-public (Frontend Services)
|
||||
- **Purpose**: Services that need external network access
|
||||
- **Services**:
|
||||
- Mosaic API (needs to reach Authentik OIDC and external Ollama)
|
||||
- Mosaic Web
|
||||
- Authentik Server (receives OIDC callbacks)
|
||||
- **Connectivity**: Full external network access for API integrations
|
||||
|
||||
#### Network Security Notes
|
||||
|
||||
1. **Why mosaic-internal is not marked internal**: The API service needs to:
|
||||
- Connect to external Authentik servers for OIDC authentication
|
||||
- Connect to external Ollama services when using remote AI
|
||||
- Make outbound HTTP requests for integrations
|
||||
|
||||
2. **Database/Cache Protection**: Even though the network allows external access:
|
||||
- PostgreSQL and Valkey are NOT exposed on host ports by default
|
||||
- Only accessible via internal Docker DNS (postgres:5432, valkey:6379)
|
||||
- To expose for development, explicitly set ports in `.env`
|
||||
|
||||
3. **Production Recommendations**:
|
||||
- Use firewall rules to restrict container egress traffic
|
||||
- Use reverse proxy (Traefik, nginx) for API/Web with TLS
|
||||
- Use external managed PostgreSQL and Valkey services
|
||||
- Implement network policies in orchestration platforms (Kubernetes)
|
||||
|
||||
### Volume Management
|
||||
|
||||
Persistent data volumes:
|
||||
|
||||
- `mosaic-postgres-data` - PostgreSQL data
|
||||
- `mosaic-valkey-data` - Valkey persistence
|
||||
- `mosaic-authentik-postgres-data` - Authentik database
|
||||
- `mosaic-authentik-redis-data` - Authentik cache
|
||||
- `mosaic-authentik-media` - Authentik media files
|
||||
- `mosaic-authentik-certs` - Authentik certificates
|
||||
- `mosaic-authentik-templates` - Authentik templates
|
||||
- `mosaic-ollama-data` - Ollama models
|
||||
|
||||
### Health Checks
|
||||
|
||||
All services include health checks with automatic restart:
|
||||
|
||||
- PostgreSQL: `pg_isready` check every 10s
|
||||
- Valkey: `valkey-cli ping` every 10s
|
||||
- API: HTTP GET /health every 30s
|
||||
- Web: HTTP GET / every 30s
|
||||
- Authentik: HTTP GET /-/health/live/ every 30s
|
||||
|
||||
## Common Operations
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# All services
|
||||
docker compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker compose logs -f api
|
||||
|
||||
# Last 100 lines
|
||||
docker compose logs --tail=100 api
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
|
||||
```bash
|
||||
# Restart all
|
||||
docker compose restart
|
||||
|
||||
# Restart specific service
|
||||
docker compose restart api
|
||||
```
|
||||
|
||||
### Stop Services
|
||||
|
||||
```bash
|
||||
# Stop all (keeps data)
|
||||
docker compose down
|
||||
|
||||
# Stop and remove volumes (WARNING: deletes all data)
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
### Execute Commands in Containers
|
||||
|
||||
```bash
|
||||
# PostgreSQL shell
|
||||
docker compose exec postgres psql -U mosaic -d mosaic
|
||||
|
||||
# API shell
|
||||
docker compose exec api sh
|
||||
|
||||
# Run Prisma migrations
|
||||
docker compose exec api pnpm prisma:migrate:prod
|
||||
```
|
||||
|
||||
### Update Services
|
||||
|
||||
```bash
|
||||
# Pull latest images
|
||||
docker compose pull
|
||||
|
||||
# Rebuild and restart
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
See [Configuration Guide](../3-configuration/README.md) for detailed environment variable documentation.
|
||||
|
||||
### Key Environment Variables
|
||||
|
||||
```bash
|
||||
# Application Ports
|
||||
API_PORT=3001
|
||||
WEB_PORT=3000
|
||||
|
||||
# Database
|
||||
DATABASE_URL=postgresql://mosaic:password@postgres:5432/mosaic
|
||||
POSTGRES_USER=mosaic
|
||||
POSTGRES_PASSWORD=change-me
|
||||
|
||||
# Cache
|
||||
VALKEY_URL=redis://valkey:6379
|
||||
|
||||
# Authentication (if using Authentik)
|
||||
OIDC_ISSUER=https://auth.example.com/application/o/mosaic-stack/
|
||||
OIDC_CLIENT_ID=your-client-id
|
||||
OIDC_CLIENT_SECRET=your-client-secret
|
||||
|
||||
# JWT
|
||||
JWT_SECRET=change-this-to-a-random-secret
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
Check logs:
|
||||
```bash
|
||||
docker compose logs <service-name>
|
||||
```
|
||||
|
||||
Common issues:
|
||||
- Port already in use: Change port in `.env`
|
||||
- Health check failing: Wait longer or check service logs
|
||||
- Missing environment variables: Check `.env` file
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
1. Verify PostgreSQL is healthy:
|
||||
```bash
|
||||
docker compose ps postgres
|
||||
```
|
||||
|
||||
2. Check database logs:
|
||||
```bash
|
||||
docker compose logs postgres
|
||||
```
|
||||
|
||||
3. Test connection:
|
||||
```bash
|
||||
docker compose exec postgres psql -U mosaic -d mosaic -c "SELECT 1;"
|
||||
```
|
||||
|
||||
### Performance Issues
|
||||
|
||||
1. Adjust PostgreSQL settings in `.env`:
|
||||
```bash
|
||||
POSTGRES_SHARED_BUFFERS=512MB
|
||||
POSTGRES_EFFECTIVE_CACHE_SIZE=2GB
|
||||
POSTGRES_MAX_CONNECTIONS=200
|
||||
```
|
||||
|
||||
2. Adjust Valkey memory:
|
||||
```bash
|
||||
VALKEY_MAXMEMORY=512mb
|
||||
```
|
||||
|
||||
3. Check resource usage:
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
### Reset Everything
|
||||
|
||||
```bash
|
||||
# Stop and remove all containers, networks, and volumes
|
||||
docker compose down -v
|
||||
|
||||
# Remove all Mosaic images
|
||||
docker images | grep mosaic | awk '{print $3}' | xargs docker rmi
|
||||
|
||||
# Start fresh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### Security
|
||||
|
||||
1. **Change default passwords** in `.env`:
|
||||
- `POSTGRES_PASSWORD`
|
||||
- `AUTHENTIK_POSTGRES_PASSWORD`
|
||||
- `AUTHENTIK_SECRET_KEY`
|
||||
- `JWT_SECRET`
|
||||
|
||||
2. **Use secrets management**:
|
||||
- Docker secrets
|
||||
- External secret manager (Vault, AWS Secrets Manager)
|
||||
|
||||
3. **Network security**:
|
||||
- Use reverse proxy (see [Traefik Integration](traefik.md))
|
||||
- Enable HTTPS/TLS
|
||||
- Restrict port exposure
|
||||
|
||||
4. **Regular updates**:
|
||||
- Keep images updated
|
||||
- Monitor security advisories
|
||||
|
||||
### Backup
|
||||
|
||||
Backup volumes regularly:
|
||||
|
||||
```bash
|
||||
# Backup PostgreSQL
|
||||
docker compose exec postgres pg_dump -U mosaic mosaic > backup.sql
|
||||
|
||||
# Backup volumes
|
||||
docker run --rm -v mosaic-postgres-data:/data -v $(pwd):/backup alpine tar czf /backup/postgres-data.tar.gz /data
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
Consider adding:
|
||||
- Prometheus for metrics
|
||||
- Grafana for dashboards
|
||||
- Loki for log aggregation
|
||||
- Alertmanager for alerts
|
||||
|
||||
### Scaling
|
||||
|
||||
For production scaling:
|
||||
- Use external PostgreSQL (managed service)
|
||||
- Use external Redis/Valkey cluster
|
||||
- Load balance multiple API instances
|
||||
- Use CDN for static assets
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Traefik Integration](traefik.md) - Reverse proxy setup and configuration
|
||||
- [Installation Guide](../2-installation/README.md) - Detailed setup instructions
|
||||
- [Configuration Guide](../3-configuration/README.md) - Environment variables
|
||||
- [Development Guide](../../2-development/README.md) - Development workflow
|
||||
- [API Documentation](../../4-api/README.md) - API reference
|
||||
Reference in New Issue
Block a user