stack/README.md

# Mosaic Stack

Multi-tenant personal assistant platform with PostgreSQL backend, Authentik SSO, and MoltBot integration.

## Overview

Mosaic Stack is a modern, PDA-friendly platform designed to help users manage their personal and professional lives with:

- **Multi-user workspaces** with team collaboration
- **Knowledge management** with wiki-style linking and version history
- **Task management** with flexible organization
- **Event & calendar** integration
- **Project tracking** with Gantt charts and Kanban boards
- **MoltBot integration** for natural language interactions
- **Authentik OIDC** for secure, enterprise-grade authentication

**Version:** 0.0.1 (Pre-MVP)
**Repository:** https://git.mosaicstack.dev/mosaic/stack

## Technology Stack

| Layer          | Technology                                   |
| -------------- | -------------------------------------------- |
| **Frontend**   | Next.js 16 + React + TailwindCSS + Shadcn/ui |
| **Backend**    | NestJS + Prisma ORM                          |
| **Database**   | PostgreSQL 17 + pgvector                     |
| **Cache**      | Valkey (Redis-compatible)                    |
| **Auth**       | Authentik (OIDC) via BetterAuth              |
| **AI**         | Ollama (local or remote)                     |
| **Messaging**  | MoltBot (stock + plugins)                    |
| **Real-time**  | WebSockets (Socket.io)                       |
| **Monorepo**   | pnpm workspaces + TurboRepo                  |
| **Testing**    | Vitest + Playwright                          |
| **Deployment** | Docker + docker-compose                      |

## Quick Start

### Prerequisites

- Node.js 20+ and pnpm 9+
- PostgreSQL 17+ (or use Docker)
- Docker & Docker Compose (optional, for turnkey deployment)

### Installation

```bash
# Clone the repository
git clone https://git.mosaicstack.dev/mosaic/stack mosaic-stack
cd mosaic-stack

# Install dependencies
pnpm install

# Copy environment file
cp .env.example .env

# Configure environment variables (see CONFIGURATION.md)
# Edit .env with your database and auth settings

# Generate Prisma client
pnpm prisma:generate

# Run database migrations
pnpm prisma:migrate

# Seed development data (optional)
pnpm prisma:seed

# Start development servers
pnpm dev
```

### Docker Deployment

**Recommended for quick setup and production deployments.**

#### Development (Turnkey - All Services Bundled)

```bash
# Clone repository
git clone https://git.mosaicstack.dev/mosaic/stack mosaic-stack
cd mosaic-stack

# Copy and configure environment
cp .env.example .env
# Set COMPOSE_PROFILES=full in .env

# Start all services (PostgreSQL, Valkey, OpenBao, Authentik, Ollama, API, Web)
docker compose up -d

# View logs
docker compose logs -f

# Access services
# Web:  http://localhost:3000
# API:  http://localhost:3001
# Auth: http://localhost:9000
```

#### Production (External Managed Services)

```bash
# Clone repository
git clone https://git.mosaicstack.dev/mosaic/stack mosaic-stack
cd mosaic-stack

# Copy environment template and example
cp .env.example .env
cp docker/docker-compose.example.external.yml docker-compose.override.yml

# Edit .env with external service URLs:
# - DATABASE_URL=postgresql://... (RDS, Cloud SQL, etc.)
# - VALKEY_URL=redis://... (ElastiCache, Memorystore, etc.)
# - OPENBAO_ADDR=https://... (HashiCorp Vault, etc.)
# - OIDC_ISSUER=https://... (Auth0, Okta, etc.)
# - Set COMPOSE_PROFILES= (empty)

# Start API and Web only
docker compose up -d

# View logs
docker compose logs -f
```

#### Hybrid (Mix of Bundled and External)

```bash
# Use bundled database/cache, external auth/secrets
cp docker/docker-compose.example.hybrid.yml docker-compose.override.yml

# Edit .env:
# - COMPOSE_PROFILES=database,cache,ollama
# - OPENBAO_ADDR=https://... (external vault)
# - OIDC_ISSUER=https://... (external auth)

# Start mixed deployment
docker compose up -d
```

**Stop services:**

```bash
docker compose down
```

**What's included:**

- PostgreSQL 17 with pgvector extension
- Valkey (Redis-compatible cache)
- Mosaic API (NestJS)
- Mosaic Web (Next.js)
- Mosaic Orchestrator (Agent lifecycle management)
- Mosaic Coordinator (Task assignment & monitoring)
- Authentik OIDC (optional, use `--profile authentik`)
- Ollama AI (optional, use `--profile ollama`)

See [Docker Deployment Guide](docs/1-getting-started/4-docker-deployment/) for complete documentation.

### Docker Swarm Deployment (Production)

**Recommended for production deployments with high availability and auto-scaling.**

Deploy to a Docker Swarm cluster with integrated Traefik reverse proxy:

```bash
# 1. Initialize swarm (if not already done)
docker swarm init --advertise-addr <your-ip>

# 2. Create Traefik network
docker network create --driver=overlay traefik-public

# 3. Configure environment for swarm
cp .env.swarm.example .env
nano .env  # Configure domains, passwords, API keys

# 4. CRITICAL: Deploy OpenBao standalone FIRST
# OpenBao cannot run in swarm mode - deploy as standalone container
docker compose -f docker-compose.openbao.yml up -d
sleep 30  # Wait for auto-initialization

# 5. Deploy swarm stack
IMAGE_TAG=dev ./scripts/deploy-swarm.sh mosaic

# 6. Check deployment status
docker stack services mosaic
docker stack ps mosaic

# Access services via Traefik
# Web:  http://mosaic.mosaicstack.dev
# API:  http://api.mosaicstack.dev
# Auth: http://auth.mosaicstack.dev (if using bundled Authentik)
```

**Key features:**

- Automatic Traefik integration for routing
- Overlay networking for multi-host deployments
- Built-in health checks and rolling updates
- Horizontal scaling for web and API services
- Zero-downtime deployments
- Service orchestration across multiple nodes

**Important Notes:**

- **OpenBao Requirement:** OpenBao MUST be deployed as standalone container (not in swarm). Use `docker-compose.openbao.yml` or external Vault.
- Swarm does NOT support docker-compose profiles
- To use external services (PostgreSQL, Authentik, etc.), manually comment them out in `docker-compose.swarm.yml`

See [Docker Swarm Deployment Guide](docs/SWARM-DEPLOYMENT.md) and [Quick Reference](docs/SWARM-QUICKREF.md) for complete documentation.

### Portainer Deployment

**Recommended for GUI-based stack management.**

Portainer provides a web UI for managing Docker containers and stacks. Use the Portainer-optimized compose file:

**File:** `docker-compose.portainer.yml`

**Key differences from standard compose:**

- No `env_file` directive (define variables in Portainer UI)
- Port exposed on all interfaces (Portainer limitation)
- Optimized for Portainer's stack parser

**Quick Steps:**

1. Create `mosaic_internal` overlay network in Portainer
2. Deploy `mosaic-openbao` stack with `docker-compose.portainer.yml`
3. Deploy `mosaic` swarm stack with `docker-compose.swarm.yml`
4. Configure environment variables in Portainer UI

See [Portainer Deployment Guide](docs/PORTAINER-DEPLOYMENT.md) for detailed instructions.

## Project Structure

```
mosaic-stack/
├── apps/
│   ├── api/                      # NestJS backend API
│   │   ├── src/
│   │   │   ├── auth/             # BetterAuth + Authentik OIDC
│   │   │   ├── prisma/           # Database service
│   │   │   ├── coordinator-integration/ # Coordinator API client
│   │   │   └── app.module.ts     # Main application module
│   │   ├── prisma/
│   │   │   └── schema.prisma     # Database schema
│   │   └── Dockerfile
│   ├── web/                      # Next.js 16 frontend
│   │   ├── app/
│   │   ├── components/
│   │   │   └── widgets/          # HUD widgets (agent status, etc.)
│   │   └── Dockerfile
│   ├── orchestrator/             # Agent lifecycle & spawning (NestJS)
│   │   ├── src/
│   │   │   ├── spawner/          # Agent spawning service
│   │   │   ├── queue/            # Valkey-backed task queue
│   │   │   ├── monitor/          # Health monitoring
│   │   │   ├── git/              # Git worktree management
│   │   │   └── killswitch/       # Emergency agent termination
│   │   └── Dockerfile
│   └── coordinator/              # Task assignment & monitoring (FastAPI)
│       ├── src/
│       │   ├── webhook.py        # Gitea webhook receiver
│       │   ├── parser.py         # Issue metadata parser
│       │   └── security.py       # HMAC signature verification
│       └── Dockerfile
├── packages/
│   ├── shared/                   # Shared types & utilities
│   │   └── src/types/
│   │       ├── auth.types.ts     # Auth types (AuthUser, Session, etc.)
│   │       ├── database.types.ts # DB entity types
│   │       └── enums.ts          # Shared enums
│   ├── ui/                       # Shared UI components (planned)
│   └── config/                   # Shared configuration (planned)
├── plugins/                      # MoltBot skills (planned)
│   ├── mosaic-plugin-brain/      # API query skill
│   ├── mosaic-plugin-calendar/   # Calendar skill
│   └── mosaic-plugin-tasks/      # Task management skill
├── docker/
│   ├── docker-compose.yml        # Production deployment
│   └── init-scripts/             # PostgreSQL initialization
├── docs/
│   ├── SETUP.md                  # Installation guide
│   ├── CONFIGURATION.md          # Environment configuration
│   ├── DESIGN-PRINCIPLES.md      # PDA-friendly design patterns
│   ├── API.md                    # API documentation
│   ├── TYPE-SHARING.md           # Type sharing strategy
│   └── scratchpads/              # Development notes
├── .env.example                  # Environment template
├── turbo.json                    # TurboRepo configuration
└── pnpm-workspace.yaml           # Workspace configuration
```

## Agent Orchestration Layer (v0.0.6)

Mosaic Stack includes a sophisticated agent orchestration system for autonomous task execution:

- **Orchestrator Service** (NestJS) - Manages agent lifecycle, spawning, and health monitoring
- **Coordinator Service** (FastAPI) - Receives Gitea webhooks, assigns tasks to agents
- **Task Queue** - Valkey-backed queue for distributed task management
- **Git Worktrees** - Isolated workspaces for parallel agent execution
- **Killswitch** - Emergency stop mechanism for runaway agents
- **Agent Dashboard** - Real-time monitoring UI with status widgets

See [Agent Orchestration Design](docs/design/agent-orchestration.md) for architecture details.

## Current Implementation Status

### ✅ Completed (v0.0.1-0.0.6)

- **M1-Foundation:** Project scaffold, PostgreSQL 17 + pgvector, Prisma ORM
- **M2-MultiTenant:** Workspace isolation with RLS, team management
- **M3-Features:** Knowledge management, tasks, calendar, authentication
- **M4-MoltBot:** Bot integration architecture (in progress)
- **M6-AgentOrchestration:** Orchestrator service, coordinator, agent dashboard ✅

**Test Coverage:** 2168+ tests passing

### 🚧 In Progress (v0.0.x)

- Agent orchestration E2E testing
- Usage budget management
- Performance optimization

### 📋 Planned Features (v0.1.0 MVP)

- Event/calendar management
- Project tracking with Gantt charts
- MoltBot integration
- WebSocket real-time updates
- Data migration from jarvis-brain

See the [issue tracker](https://git.mosaicstack.dev/mosaic/stack/issues) for complete roadmap.

## Knowledge Module

The **Knowledge Module** is a powerful personal wiki and knowledge management system built into Mosaic Stack. Create interconnected notes, organize with tags, track changes over time, and visualize relationships.

### Features

- **📝 Markdown-based entries** — Write using familiar Markdown syntax
- **🔗 Wiki-style linking** — Connect entries using `[[wiki-links]]`
- **🏷️ Tag organization** — Categorize and filter with flexible tagging
- **📜 Full version history** — Every edit is tracked and recoverable
- **🔍 Powerful search** — Full-text search across titles and content
- **📊 Knowledge graph** — Visualize relationships between entries
- **📤 Import/Export** — Bulk import/export for portability
- **⚡ Valkey caching** — High-performance caching for fast access

### Quick Examples

**Create an entry:**

```bash
curl -X POST http://localhost:3001/api/knowledge/entries \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "x-workspace-id: WORKSPACE_ID" \
  -d '{
    "title": "React Hooks Guide",
    "content": "# React Hooks\n\nSee [[Component Patterns]] for more.",
    "tags": ["react", "frontend"],
    "status": "PUBLISHED"
  }'
```

**Search entries:**

```bash
curl -X GET 'http://localhost:3001/api/knowledge/search?q=react+hooks' \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "x-workspace-id: WORKSPACE_ID"
```

**Export knowledge base:**

```bash
curl -X GET 'http://localhost:3001/api/knowledge/export?format=markdown' \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "x-workspace-id: WORKSPACE_ID" \
  -o knowledge-export.zip
```

### Documentation

- **[User Guide](KNOWLEDGE_USER_GUIDE.md)** — Getting started, features, and workflows
- **[API Documentation](KNOWLEDGE_API.md)** — Complete REST API reference with examples
- **[Developer Guide](KNOWLEDGE_DEV.md)** — Architecture, implementation, and contributing

### Key Concepts

**Wiki-links**
Connect entries using double-bracket syntax:

```markdown
See [[Entry Title]] or [[entry-slug]] for details.
Use [[Page|custom text]] for custom display text.
```

**Version History**
Every edit creates a new version. View history, compare changes, and restore previous versions:

```bash
# List versions
GET /api/knowledge/entries/:slug/versions

# Get specific version
GET /api/knowledge/entries/:slug/versions/:version

# Restore version
POST /api/knowledge/entries/:slug/restore/:version
```

**Backlinks**
Automatically discover entries that link to a given entry:

```bash
GET /api/knowledge/entries/:slug/backlinks
```

**Tags**
Organize entries with tags:

```bash
# Create tag
POST /api/knowledge/tags
{ "name": "React", "color": "#61dafb" }

# Find entries with tags
GET /api/knowledge/search/by-tags?tags=react,frontend
```

### Performance

With Valkey caching enabled:

- **Entry retrieval:** ~2-5ms (vs ~50ms uncached)
- **Search queries:** ~2-5ms (vs ~200ms uncached)
- **Graph traversals:** ~2-5ms (vs ~400ms uncached)
- **Cache hit rates:** 70-90% for active workspaces

Configure caching via environment variables:

```bash
VALKEY_URL=redis://localhost:6379
KNOWLEDGE_CACHE_ENABLED=true
KNOWLEDGE_CACHE_TTL=300  # 5 minutes
```

## Development Workflow

### Branch Strategy

- `main` — Stable releases only
- `develop` — Active development (default working branch)
- `feature/*` — Feature branches from develop
- `fix/*` — Bug fix branches

### Running Locally

```bash
# Start all services (requires Docker for PostgreSQL)
pnpm dev              # All apps

# Or run individually
pnpm dev:api          # API only (http://localhost:3001)
pnpm dev:web          # Web only (http://localhost:3000)

# Database tools
pnpm prisma:studio    # Open Prisma Studio
pnpm prisma:migrate   # Run migrations
pnpm prisma:generate  # Regenerate Prisma client
```

### Testing

```bash
pnpm test             # All tests
pnpm test:api         # API tests only
pnpm test:web         # Web tests only
pnpm test:e2e         # E2E tests with Playwright
pnpm test:coverage    # Generate coverage report
```

### Building

```bash
pnpm build            # Build all apps
pnpm build:api        # Build API only
pnpm build:web        # Build web only
```

## Design Philosophy

Mosaic Stack follows strict **PDA-friendly design principles**:

### Language Guidelines

We **never** use demanding or stressful language:

| ❌ NEVER    | ✅ ALWAYS            |
| ----------- | -------------------- |
| OVERDUE     | Target passed        |
| URGENT      | Approaching target   |
| MUST DO     | Scheduled for        |
| CRITICAL    | High priority        |
| YOU NEED TO | Consider / Option to |
| REQUIRED    | Recommended          |

### Visual Principles

- **10-second scannability** — Key info visible immediately
- **Visual chunking** — Clear sections with headers
- **Single-line items** — Compact, scannable lists
- **Calm colors** — No aggressive reds for status indicators
- **Progressive disclosure** — Details on click, not upfront

See [Design Principles](docs/3-architecture/3-design-principles/1-pda-friendly.md) for complete guidelines.

## API Conventions

### Endpoints

```
GET    /api/{resource}           # List (with pagination, filters)
GET    /api/{resource}/:id       # Get single item
POST   /api/{resource}           # Create new item
PATCH  /api/{resource}/:id       # Update item
DELETE /api/{resource}/:id       # Delete item
```

### Authentication

All authenticated endpoints require a Bearer token:

```http
Authorization: Bearer {session_token}
```

See [API Reference](docs/4-api/) for complete API documentation.

## Configuration

Key environment variables:

```bash
# Database
DATABASE_URL=postgresql://mosaic:password@localhost:5432/mosaic

# Authentik OIDC
OIDC_ISSUER=https://auth.example.com/application/o/mosaic-stack/
OIDC_CLIENT_ID=your-client-id
OIDC_CLIENT_SECRET=your-client-secret

# JWT Session
JWT_SECRET=change-this-to-a-random-secret-in-production
JWT_EXPIRATION=24h

# Application
NEXT_PUBLIC_APP_URL=http://localhost:3000
```

See [Configuration](docs/1-getting-started/3-configuration/1-environment.md) for all configuration options.

## Caching

Mosaic Stack uses **Valkey** (Redis-compatible) for high-performance caching, significantly improving response times for frequently accessed data.

### Knowledge Module Caching

The Knowledge module implements intelligent caching for:

- **Entry Details** - Individual knowledge entries (GET `/api/knowledge/entries/:slug`)
- **Search Results** - Full-text search queries with filters
- **Graph Queries** - Knowledge graph traversals with depth limits

### Cache Configuration

Configure caching via environment variables:

```bash
# Valkey connection
VALKEY_URL=redis://localhost:6379

# Knowledge cache settings
KNOWLEDGE_CACHE_ENABLED=true        # Set to false to disable caching (dev mode)
KNOWLEDGE_CACHE_TTL=300             # Time-to-live in seconds (default: 5 minutes)
```

### Cache Invalidation Strategy

Caches are automatically invalidated on data changes:

- **Entry Updates** - Invalidates entry cache, search caches, and related graph caches
- **Entry Creation** - Invalidates search caches and graph caches
- **Entry Deletion** - Invalidates entry cache, search caches, and graph caches
- **Link Changes** - Invalidates graph caches for affected entries

### Cache Statistics & Management

Monitor and manage caches via REST endpoints:

```bash
# Get cache statistics (hits, misses, hit rate)
GET /api/knowledge/cache/stats

# Clear all caches for a workspace (admin only)
POST /api/knowledge/cache/clear

# Reset cache statistics (admin only)
POST /api/knowledge/cache/stats/reset
```

**Example response:**

```json
{
  "enabled": true,
  "stats": {
    "hits": 1250,
    "misses": 180,
    "sets": 195,
    "deletes": 15,
    "hitRate": 0.874
  }
}
```

### Performance Benefits

- **Entry retrieval:** ~10-50ms → ~2-5ms (80-90% improvement)
- **Search queries:** ~100-300ms → ~2-5ms (95-98% improvement)
- **Graph traversals:** ~200-500ms → ~2-5ms (95-99% improvement)

Cache hit rates typically stabilize at 70-90% for active workspaces.

## Type Sharing

Types used by both frontend and backend live in `@mosaic/shared`:

```typescript
import type { AuthUser, Task, Event } from "@mosaic/shared";

// Frontend
function UserProfile({ user }: { user: AuthUser }) {
  return <div>{user.name}</div>;
}

// Backend
@Get("profile")
@UseGuards(AuthGuard)
getProfile(@CurrentUser() user: AuthUser) {
  return user;
}
```

See [Type Sharing Strategy](docs/2-development/3-type-sharing/1-strategy.md) for the complete type sharing strategy.

## Contributing

1. Check the [issue tracker](https://git.mosaicstack.dev/mosaic/stack/issues) for open issues
2. Create a feature branch: `git checkout -b feature/my-feature develop`
3. Make your changes with tests (minimum 85% coverage required)
4. Run tests: `pnpm test`
5. Build: `pnpm build`
6. Commit with conventional format: `feat(#issue): Description`
7. Push and create a pull request to `develop`

### Commit Format

```
<type>(#issue): Brief description

Detailed explanation if needed.

Fixes #123
```

**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `chore`

## Testing Requirements

- **Minimum 85% coverage** for new code
- **TDD approach** — Write tests before implementation
- **All tests pass** before PR merge
- Use Vitest for unit/integration tests
- Use Playwright for E2E tests

## Documentation

Complete documentation is organized in a Bookstack-compatible structure in the `docs/` directory.

### 📚 Getting Started

- **[Quick Start](docs/1-getting-started/1-quick-start/1-overview.md)** — Get running in 5 minutes
- **[Installation](docs/1-getting-started/2-installation/)** — Prerequisites, local setup, Docker deployment
- **[Configuration](docs/1-getting-started/3-configuration/)** — Environment variables and Authentik OIDC

### 💻 Development

- **[Workflow](docs/2-development/1-workflow/)** — Branching strategy, testing requirements, commit guidelines
- **[Database](docs/2-development/2-database/)** — Schema design, migrations, Prisma usage
- **[Type Sharing](docs/2-development/3-type-sharing/1-strategy.md)** — Shared types across monorepo

### 🏗️ Architecture

- **[Overview](docs/3-architecture/1-overview/)** — System design and components
- **[Authentication](docs/3-architecture/2-authentication/)** — BetterAuth and OIDC integration
- **[Design Principles](docs/3-architecture/3-design-principles/1-pda-friendly.md)** — PDA-friendly patterns (non-negotiable)

### 🔌 API Reference

- **[Conventions](docs/4-api/1-conventions/1-endpoints.md)** — REST patterns, pagination, filtering
- **[Authentication](docs/4-api/2-authentication/1-endpoints.md)** — Auth endpoints and flows

**Browse all documentation:** [docs/](docs/)

## Related Projects

- **jarvis-brain** — Original JSON-based personal assistant (migration source)
- **MoltBot** — Stock messaging gateway for multi-platform integration

## Security

- **Session-based authentication** with secure JWT tokens
- **Row-level security** ready for multi-tenant isolation
- **OIDC integration** with Authentik for enterprise SSO
- **Secure error handling** — No sensitive data in logs or responses
- **Type-safe validation** — TypeScript catches issues at compile time

## License

[Add license information]

## Support

- **Issues:** https://git.mosaicstack.dev/mosaic/stack/issues
- **Documentation:** https://git.mosaicstack.dev/mosaic/stack/wiki

---

**Mosaic Stack v0.0.1** — Building the future of personal assistants.