Compare commits
2 Commits
docs/ms22-
...
chore/ms22
| Author | SHA1 | Date | |
|---|---|---|---|
| fab833a710 | |||
| 4294deda49 |
@@ -71,3 +71,21 @@ Remaining estimate: ~143K tokens (Codex budget).
|
||||
| MS22-SKILL-001 | done | p0-knowledge | OpenClaw mosaic skill (agents read/write findings/memory) | TASKS:P0 | stack | feat/ms22-openclaw-skill | MS22-API-001,MS22-API-002 | MS22-VER-P0 | — | — | — | 15K | — | Skill in ~/.agents/skills/mosaic/ |
|
||||
| MS22-INGEST-001 | done | p0-knowledge | Session log ingestion pipeline (OpenClaw logs → ConvArchive) | TASKS:P0 | stack | feat/ms22-ingest | MS22-API-004 | MS22-VER-P0 | — | — | — | 20K | — | Script to batch-ingest existing logs |
|
||||
| MS22-VER-P0 | done | p0-knowledge | Phase 0 verification: all modules deployed + smoke tested | TASKS:P0 | stack | — | MS22-TEST-001,MS22-SKILL-001,MS22-INGEST-001,MS22-API-003 | — | — | — | — | 5K | — | |
|
||||
|
||||
## MS22 Phase 1: DB-Centric Agent Fleet (reworked)
|
||||
|
||||
Design doc: `docs/design/MS22-DB-CENTRIC-ARCHITECTURE.md`
|
||||
|
||||
| Task ID | Status | Phase | Description | Issue | Scope | Branch | Depends On | Blocks | Assigned Worker | Started | Completed | Est Tokens | Act Tokens | Notes |
|
||||
| -------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------- | ----- | ------- | ---------------------------- | ---------- | --------------- | --------------- | ------- | --------- | ---------- | ---------- | ----- |
|
||||
| MS22-P1a | not-started | phase-1a | Prisma schema: SystemConfig, BreakglassUser, LlmProvider, UserContainer, SystemContainer, UserAgentConfig + migration | — | api | feat/ms22-p1a-schema | — | P1b,P1c,P1d,P1e | — | — | — | 20K | — | |
|
||||
| MS22-P1b | not-started | phase-1b | Encryption service (AES-256-GCM) for API keys and tokens | — | api | feat/ms22-p1b-crypto | — | P1c,P1e,P1g | — | — | — | 15K | — | |
|
||||
| MS22-P1c | not-started | phase-1c | Internal config endpoint: assemble openclaw.json from DB | — | api | feat/ms22-p1c-config-api | P1a,P1b | P1i,P1j | — | — | — | 20K | — | |
|
||||
| MS22-P1d | not-started | phase-1d | ContainerLifecycleService: Docker API (dockerode) start/stop/health/reap | — | api | feat/ms22-p1d-container-mgr | P1a | P1i,P1k | — | — | — | 25K | — | |
|
||||
| MS22-P1e | not-started | phase-1e | Onboarding API: breakglass, OIDC, provider, agents, complete | — | api | feat/ms22-p1e-onboarding-api | P1a,P1b | P1f | — | — | — | 20K | — | |
|
||||
| MS22-P1f | not-started | phase-1f | Onboarding wizard WebUI (multi-step form) | — | web | feat/ms22-p1f-onboarding-ui | P1e | — | — | — | — | 25K | — | |
|
||||
| MS22-P1g | not-started | phase-1g | Settings API: CRUD providers, agent config, OIDC, breakglass | — | api | feat/ms22-p1g-settings-api | P1a,P1b | P1h | — | — | — | 20K | — | |
|
||||
| MS22-P1h | not-started | phase-1h | Settings UI: Providers, Agent Config, Auth pages | — | web | feat/ms22-p1h-settings-ui | P1g | — | — | — | — | 25K | — | |
|
||||
| MS22-P1i | not-started | phase-1i | Chat proxy: route WebUI chat to user's OpenClaw container (SSE) | — | api+web | feat/ms22-p1i-chat-proxy | P1c,P1d | — | — | — | — | 20K | — | |
|
||||
| MS22-P1j | not-started | phase-1j | Docker entrypoint + health checks + core compose | — | docker | feat/ms22-p1j-docker | P1c | — | — | — | — | 10K | — | |
|
||||
| MS22-P1k | not-started | phase-1k | Idle reaper cron: stop inactive user containers | — | api | feat/ms22-p1k-idle-reaper | P1d | — | — | — | — | 10K | — | |
|
||||
|
||||
413
docs/design/MS22-DB-CENTRIC-ARCHITECTURE.md
Normal file
413
docs/design/MS22-DB-CENTRIC-ARCHITECTURE.md
Normal file
@@ -0,0 +1,413 @@
|
||||
# MS22 Phase 1: DB-Centric Multi-User Agent Architecture
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **2 env vars to bootstrap** — `DATABASE_URL` + `MOSAIC_SECRET_KEY`
|
||||
2. **DB-centric config** — All runtime config in Postgres, managed via WebUI
|
||||
3. **Mosaic is the gatekeeper** — Users authenticate to Mosaic, never to OpenClaw directly
|
||||
4. **Per-user agent isolation** — Each user gets their own OpenClaw container(s) with their own credentials
|
||||
5. **Onboarding-first** — Breakglass user + wizard on first boot
|
||||
6. **Generic product** — No hardcoded names, models, providers, or endpoints
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ MOSAIC WEBUI │
|
||||
│ (Auth: breakglass local + OIDC via settings) │
|
||||
└──────────────────────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ MOSAIC API │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌────────────────┐ ┌─────────┐ │
|
||||
│ │ Onboarding │ │ Container │ │ Config │ │
|
||||
│ │ Wizard │ │ Lifecycle Mgr │ │ Store │ │
|
||||
│ └──────────────┘ └───────┬────────┘ └─────────┘ │
|
||||
│ │ │
|
||||
└────────────────────────────┼────────────────────────┘
|
||||
│ Docker API
|
||||
┌──────────────────┼──────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ OpenClaw │ │ OpenClaw │ │ OpenClaw │
|
||||
│ User A │ │ User B │ │ System │
|
||||
│ │ │ │ │ (admin) │
|
||||
│ Claude Max │ │ Z.ai key │ │ Shared key │
|
||||
│ own memory │ │ own memory │ │ monitoring │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
Scale to zero Scale to zero Always on
|
||||
after idle after idle
|
||||
```
|
||||
|
||||
## Container Lifecycle
|
||||
|
||||
### User containers (on-demand)
|
||||
|
||||
1. User logs in → Mosaic checks `UserContainer` table
|
||||
2. No running container → Mosaic calls Docker API to create one
|
||||
3. Injects user's encrypted API keys via config endpoint
|
||||
4. Routes chat requests to user's container
|
||||
5. Idle timeout (configurable, default 30min) → scale to zero
|
||||
6. State volume persists (sessions, memory, auth tokens)
|
||||
7. Next request → container restarts, picks up state from volume
|
||||
|
||||
### System containers (always-on, optional)
|
||||
|
||||
- Admin-provisioned for system tasks (monitoring, scheduled jobs)
|
||||
- Use admin-configured shared API keys
|
||||
- Not tied to any user
|
||||
|
||||
## Auth Layers
|
||||
|
||||
| Flow | Method |
|
||||
| ------------------------------- | ---------------------------------------------------------------------- |
|
||||
| User → Mosaic WebUI | Breakglass (local) or OIDC (configured in settings) |
|
||||
| Mosaic API → OpenClaw container | Bearer token (auto-generated per container, stored encrypted in DB) |
|
||||
| OpenClaw → LLM providers | User's own API keys (delivered via config endpoint, decrypted from DB) |
|
||||
| Admin → System settings | RBAC (admin role required) |
|
||||
| Internal config endpoint | Bearer token (container authenticates to fetch its config) |
|
||||
|
||||
## Database Schema
|
||||
|
||||
### System Tables
|
||||
|
||||
```prisma
|
||||
model SystemConfig {
|
||||
id String @id @default(cuid())
|
||||
key String @unique // "oidc.issuerUrl", "oidc.clientId", "onboarding.completed"
|
||||
value String // plaintext or encrypted (prefix: "enc:")
|
||||
encrypted Boolean @default(false)
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
|
||||
model BreakglassUser {
|
||||
id String @id @default(cuid())
|
||||
username String @unique
|
||||
passwordHash String // bcrypt
|
||||
isActive Boolean @default(true)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
### Provider Tables (per-user)
|
||||
|
||||
```prisma
|
||||
model LlmProvider {
|
||||
id String @id @default(cuid())
|
||||
userId String // owner — each user manages their own providers
|
||||
name String // "my-zai", "work-openai", "local-ollama"
|
||||
displayName String // "Z.ai", "OpenAI (Work)", "Local Ollama"
|
||||
type String // "zai" | "openai" | "anthropic" | "ollama" | "custom"
|
||||
baseUrl String? // null for built-in, URL for custom/ollama
|
||||
apiKey String? // encrypted
|
||||
apiType String @default("openai-completions")
|
||||
models Json @default("[]") // [{id, name, contextWindow, maxTokens}]
|
||||
isActive Boolean @default(true)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
@@unique([userId, name])
|
||||
}
|
||||
```
|
||||
|
||||
### Container Tables
|
||||
|
||||
```prisma
|
||||
model UserContainer {
|
||||
id String @id @default(cuid())
|
||||
userId String @unique
|
||||
containerId String? // Docker container ID (null = not running)
|
||||
containerName String // "mosaic-user-{userId}"
|
||||
gatewayPort Int? // assigned port (null = not running)
|
||||
gatewayToken String // encrypted — auto-generated
|
||||
status String @default("stopped") // "running" | "stopped" | "starting" | "error"
|
||||
lastActiveAt DateTime?
|
||||
idleTimeoutMin Int @default(30)
|
||||
config Json @default("{}") // cached openclaw.json for this user
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
|
||||
model SystemContainer {
|
||||
id String @id @default(cuid())
|
||||
name String @unique // "mosaic-system-ops", "mosaic-system-monitor"
|
||||
role String // "operations" | "monitor" | "scheduler"
|
||||
containerId String?
|
||||
gatewayPort Int?
|
||||
gatewayToken String // encrypted
|
||||
status String @default("stopped")
|
||||
providerId String? // references admin-level LlmProvider
|
||||
primaryModel String // "zai/glm-5", etc.
|
||||
isActive Boolean @default(true)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
### User Agent Preferences
|
||||
|
||||
```prisma
|
||||
model UserAgentConfig {
|
||||
id String @id @default(cuid())
|
||||
userId String @unique
|
||||
primaryModel String? // user's preferred model
|
||||
fallbackModels Json @default("[]")
|
||||
personality String? // custom SOUL.md content
|
||||
providerId String? // default provider for this user
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
## Internal Config Endpoint
|
||||
|
||||
`GET /api/internal/agent-config/:containerType/:id`
|
||||
|
||||
- Auth: Bearer token (container's own gateway token)
|
||||
- Returns: Complete `openclaw.json` generated from DB
|
||||
- For user containers: includes user's providers, model prefs, personality
|
||||
- For system containers: includes admin provider config
|
||||
|
||||
Response assembles openclaw.json dynamically:
|
||||
|
||||
```json
|
||||
{
|
||||
"gateway": { "mode": "local", "port": 18789, "bind": "lan", "auth": { "mode": "token" } ... },
|
||||
"agents": { "defaults": { "model": { "primary": "<from UserAgentConfig>" } } },
|
||||
"models": { "providers": { "<from LlmProvider rows>": { ... } } }
|
||||
}
|
||||
```
|
||||
|
||||
## Container Lifecycle Manager
|
||||
|
||||
NestJS service that manages Docker containers:
|
||||
|
||||
```typescript
|
||||
class ContainerLifecycleService {
|
||||
// Create and start a user's OpenClaw container
|
||||
async ensureRunning(userId: string): Promise<{ url: string; token: string }>;
|
||||
|
||||
// Stop idle containers (called by cron/scheduler)
|
||||
async reapIdle(): Promise<number>;
|
||||
|
||||
// Stop a specific user's container
|
||||
async stop(userId: string): Promise<void>;
|
||||
|
||||
// Health check all running containers
|
||||
async healthCheckAll(): Promise<HealthStatus[]>;
|
||||
|
||||
// Restart container with updated config
|
||||
async restart(userId: string): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Uses Docker Engine API (`/var/run/docker.sock` or TCP) via `dockerode` npm package.
|
||||
|
||||
## Onboarding Wizard
|
||||
|
||||
### First-Boot Detection
|
||||
|
||||
- API checks: `SystemConfig.get("onboarding.completed")` → null = first boot
|
||||
- WebUI redirects to `/onboarding` if not completed
|
||||
|
||||
### Steps
|
||||
|
||||
**Step 1: Create Breakglass Admin**
|
||||
|
||||
- Username + password → bcrypt → `BreakglassUser` table
|
||||
- This user always works, even if OIDC is misconfigured
|
||||
|
||||
**Step 2: Configure Authentication (optional)**
|
||||
|
||||
- OIDC: provider URL, client ID, client secret → encrypted in `SystemConfig`
|
||||
- Skip = breakglass-only auth (can add OIDC later in settings)
|
||||
|
||||
**Step 3: Add Your First LLM Provider**
|
||||
|
||||
- Pick type → enter API key/endpoint → test connection → save to `LlmProvider`
|
||||
- This becomes the admin's default provider
|
||||
|
||||
**Step 4: System Agents (optional)**
|
||||
|
||||
- Configure always-on system agents for monitoring/ops
|
||||
- Or skip — users can just use their own personal agents
|
||||
|
||||
**Step 5: Complete**
|
||||
|
||||
- Sets `SystemConfig("onboarding.completed") = true`
|
||||
- Redirects to dashboard
|
||||
|
||||
### Post-Onboarding: User Self-Service
|
||||
|
||||
- Each user adds their own LLM providers in profile settings
|
||||
- Each user configures their preferred model, personality
|
||||
- First chat request triggers container creation
|
||||
|
||||
## Docker Compose (final)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mosaic-api:
|
||||
image: mosaic/api:latest
|
||||
environment:
|
||||
DATABASE_URL: ${DATABASE_URL}
|
||||
MOSAIC_SECRET_KEY: ${MOSAIC_SECRET_KEY}
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock # Docker API access
|
||||
networks:
|
||||
- internal
|
||||
|
||||
mosaic-web:
|
||||
image: mosaic/web:latest
|
||||
environment:
|
||||
NEXT_PUBLIC_API_URL: http://mosaic-api:4000
|
||||
networks:
|
||||
- internal
|
||||
|
||||
postgres:
|
||||
image: postgres:17
|
||||
environment:
|
||||
POSTGRES_DB: mosaic
|
||||
POSTGRES_USER: mosaic
|
||||
POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
networks:
|
||||
- internal
|
||||
|
||||
# System agent (optional, admin-provisioned)
|
||||
# mosaic-system:
|
||||
# image: alpine/openclaw:latest
|
||||
# ... (managed by ContainerLifecycleService)
|
||||
|
||||
# User containers are NOT in this file —
|
||||
# they are dynamically created by ContainerLifecycleService
|
||||
# via the Docker API at runtime.
|
||||
|
||||
networks:
|
||||
internal:
|
||||
driver: overlay
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
```
|
||||
|
||||
Note: User OpenClaw containers are **not** defined in docker-compose. They are
|
||||
created dynamically by the `ContainerLifecycleService` when users start chatting.
|
||||
|
||||
## Entrypoint (for dynamically created containers)
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
set -e
|
||||
: "${MOSAIC_API_URL:?required}"
|
||||
: "${AGENT_TOKEN:?required}"
|
||||
: "${AGENT_ID:?required}"
|
||||
|
||||
# Fetch config from Mosaic API
|
||||
curl -sf "${MOSAIC_API_URL}/api/internal/agent-config/${AGENT_ID}" \
|
||||
-H "Authorization: Bearer ${AGENT_TOKEN}" \
|
||||
-o /tmp/openclaw.json
|
||||
|
||||
export OPENCLAW_CONFIG_PATH=/tmp/openclaw.json
|
||||
exec openclaw gateway run --bind lan --auth token
|
||||
```
|
||||
|
||||
Container env vars (injected by ContainerLifecycleService):
|
||||
|
||||
- `MOSAIC_API_URL` — internal API URL
|
||||
- `AGENT_TOKEN` — this container's bearer token (from DB)
|
||||
- `AGENT_ID` — container ID for config lookup
|
||||
|
||||
## Config Update Strategy
|
||||
|
||||
When a user changes settings (model, provider, personality):
|
||||
|
||||
1. Mosaic API updates DB
|
||||
2. API calls `ContainerLifecycleService.restart(userId)`
|
||||
3. Container restarts, fetches fresh config from API
|
||||
4. OpenClaw gateway starts with new config
|
||||
5. State volume preserves sessions/memory across restarts
|
||||
|
||||
## Task Breakdown
|
||||
|
||||
| Task | Phase | Scope | Dependencies |
|
||||
| -------- | -------------- | --------------------------------------------------------------------------------------------------------------------- | ------------ |
|
||||
| MS22-P1a | Schema | Prisma models: SystemConfig, BreakglassUser, LlmProvider, UserContainer, SystemContainer, UserAgentConfig. Migration. | — |
|
||||
| MS22-P1b | Crypto | Encryption service for API keys/tokens (AES-256-GCM using MOSAIC_SECRET_KEY) | P1a |
|
||||
| MS22-P1c | Config API | Internal config endpoint: assembles openclaw.json from DB | P1a, P1b |
|
||||
| MS22-P1d | Container Mgr | ContainerLifecycleService: Docker API integration (dockerode), start/stop/health/reap | P1a |
|
||||
| MS22-P1e | Onboarding API | Onboarding endpoints: breakglass, OIDC, provider, complete | P1a, P1b |
|
||||
| MS22-P1f | Onboarding UI | Multi-step wizard in WebUI | P1e |
|
||||
| MS22-P1g | Settings API | CRUD: providers, agent config, OIDC, breakglass | P1a, P1b |
|
||||
| MS22-P1h | Settings UI | Settings pages: Providers, Agent Config, Auth | P1g |
|
||||
| MS22-P1i | Chat Proxy | Route WebUI chat → user's OpenClaw container (SSE) | P1c, P1d |
|
||||
| MS22-P1j | Docker | Entrypoint script, health checks, compose for core services | P1c |
|
||||
| MS22-P1k | Idle Reaper | Cron service to stop idle user containers | P1d |
|
||||
|
||||
## Open Questions (Resolved)
|
||||
|
||||
1. ~~Config updates → restart?~~ **Yes.** Mosaic restarts the container, fresh config on boot.
|
||||
2. ~~CLI alternative for breakglass?~~ **Yes.** Both WebUI wizard and CLI (`mosaic admin create-breakglass`).
|
||||
3. ~~Config cache TTL?~~ **Yes.** Config fetched once at startup, changes trigger restart.
|
||||
|
||||
## Security Isolation Model
|
||||
|
||||
### Core Principle: ZERO cross-user access
|
||||
|
||||
Every user is fully sandboxed. No exceptions.
|
||||
|
||||
### Container Isolation
|
||||
|
||||
- Each user gets their **own** OpenClaw container (separate process, PID namespace)
|
||||
- Each container has its **own** Docker volume (sessions, memory, workspace)
|
||||
- Containers run on an **internal-only** Docker network — no external exposure
|
||||
- Users NEVER talk to OpenClaw directly — Mosaic proxies all requests
|
||||
- Container gateway tokens are unique per-user and single-purpose
|
||||
|
||||
### Data Isolation (enforced at API + DB level)
|
||||
|
||||
| Data | Isolation | Enforcement |
|
||||
| ---------------- | ------------------------- | --------------------------------------------------------------------------------- |
|
||||
| LLM API keys | Per-user, encrypted | `LlmProvider.userId` — all queries scoped by authenticated user |
|
||||
| Chat history | Per-user container volume | Separate Docker volume per user, not shared |
|
||||
| Agent memory | Per-user container volume | Separate Docker volume per user |
|
||||
| Agent config | Per-user | `UserAgentConfig.userId` — scoped queries |
|
||||
| Container access | Per-user | `UserContainer.userId` — Mosaic validates user owns the container before proxying |
|
||||
|
||||
### API Enforcement
|
||||
|
||||
- **All user-facing endpoints** include `WHERE userId = authenticatedUser.id`
|
||||
- **No admin endpoint** exposes another user's API keys (even to admins)
|
||||
- **Chat proxy** validates: authenticated user → owns target container → forwards request
|
||||
- **Config endpoint** validates: container token matches the container requesting config
|
||||
- **Provider CRUD** is fully user-scoped — User A cannot list, read, or modify User B's providers
|
||||
|
||||
### What admins CAN see
|
||||
|
||||
- Container status (running/stopped) — not contents
|
||||
- User list and roles
|
||||
- System-level config (OIDC, system agents)
|
||||
- Aggregate usage metrics (not individual conversations)
|
||||
|
||||
### What admins CANNOT see
|
||||
|
||||
- Other users' API keys (encrypted, no decrypt endpoint)
|
||||
- Other users' chat history (in container volumes, not in Mosaic DB)
|
||||
- Other users' agent memory/workspace contents
|
||||
|
||||
### Future: Team Workspaces (NOT in scope)
|
||||
|
||||
Team/shared workspaces are a potential future feature where users opt-in to
|
||||
shared agent contexts. This requires explicit consent, shared-key management,
|
||||
and a different isolation model. **Not designed here. Not built now.**
|
||||
|
||||
### Attack Surface Notes
|
||||
|
||||
- Docker socket access (`/var/run/docker.sock`) is required by Mosaic API for container management. This is a privileged operation — the Mosaic API container must be trusted.
|
||||
- `MOSAIC_SECRET_KEY` is the root of trust for encryption. Rotation requires re-encrypting all secrets in DB.
|
||||
- Container-to-container communication is blocked by default (no shared network between user containers unless explicitly configured).
|
||||
Reference in New Issue
Block a user