mosaicstack/stack
19
0
Fork 1
Code Issues 335 Pull Requests 17 Actions Packages Projects Releases 15 Wiki Activity
Files
8d28e163a4316ab13e9f614033f5b7523aef8654
stack/docs/PRD-MS22.md
T

18 KiB
Raw Blame History

PRD: Fleet Evolution — Multi-Agent Knowledge Layer & Management Plane

Metadata

  • Owner: Jason Woltje
  • Date: 2026-02-28
  • Status: draft
  • Best-Guess Mode: true
  • Mission: MS22
  • Planning Docs:
    • ~/src/jarvis-brain/docs/planning/FLEET-EVOLUTION-PLAN.md
    • ~/src/jarvis-brain/docs/planning/matrix-agent-communication-RESOLVED.md

Problem Statement

Mosaic Stack manages projects, users, and infrastructure but has no integration with the AI agents that do the actual work. OpenClaw runs those agents but has no persistent knowledge layer — context is lost on every compaction and session boundary. Agents cannot collaborate, share findings, or build on each other's work. There is no unified interface for users to interact with agents, manage LLM providers, or observe agent activity.

Why now: MS21 (multi-tenant RBAC) is complete. The platform has users, roles, and workspaces. The next logical layer is the intelligence that operates within them.

Objectives

  1. Eliminate context loss across agent sessions via a persistent, searchable knowledge layer in Postgres + pgvector
  2. Enable inter-agent collaboration through structured Task and Findings APIs (not chat)
  3. Provide a WebUI for users to chat with agents, manage tasks, and search findings
  4. Support multi-LLM provider configuration with per-agent model assignment
  5. Mirror agent activity to Matrix rooms for observability and audit
  6. Make the entire system installable via a scripted setup process

Scope

In Scope

  1. Knowledge layer schema (findings, agent_memory, tasks enhancements, conversation_archive)
  2. Task API with state machine, assignment, and parent/child relationships
  3. Findings API with vector similarity search
  4. Agent memory API (per-agent key/value, cross-agent readable)
  5. OpenClaw session log ingestion pipeline (JSONL → Postgres → pgvector embeddings)
  6. OpenClaw mosaic skill (CLI wrapper for agents to access the knowledge layer)
  7. OpenClaw multi-agent provisioning (jarvis, builder, medic — with SOUL.md, model config, tool permissions)
  8. Discord channel bindings for agent interaction
  9. Matrix observation room auto-provisioning and async mirroring (via existing bridge module)
  10. WebSocket chat endpoint (Mosaic API → OpenClaw session proxy)
  11. WebUI chat component with agent selector
  12. Task management UI (create, assign, track, view findings)
  13. Agent registry UI (status, model, channel, last active)
  14. Findings search UI (semantic search across all agent output)
  15. LLM provider management UI (add/remove/test providers, per-agent assignment, fallback chains)
  16. Usage/cost dashboard per provider and per agent
  17. Provider health monitoring (latency, errors, rate limits)
  18. Scripted installer (openclaw skills install mosaic-fleet or equivalent)
  19. Interactive setup wizard (detect stack, provision agents, bind channels, configure providers)
  20. Data retention policy (configurable per-workspace in WebUI, stored in DB)

Out of Scope

  1. Matrix federation between Mosaic Stack deployments (federation stays at Mosaic level)
  2. More than 3 initial agents (additional agents added organically when workflows justify)
  3. Agent-to-agent direct chat (collaboration is via Task/Findings API)
  4. Rebuilding agent execution logic in Mosaic (OpenClaw is the runtime)
  5. Per-agent Docker containers (agents are OpenClaw workspace configs, not separate images)
  6. Mobile-native apps (Discord serves as mobile surface)
  7. Billing/payment integration for LLM providers
  8. Voice/audio agent interaction

User/Stakeholder Requirements

  1. Jason (admin/power user): Must be able to configure agents, providers, and channels from WebUI without editing config files
  2. Non-technical user (e.g., spouse): Must be able to chat with agents via WebUI or Discord using natural language — no commands or technical knowledge required
  3. Agents: Must be able to read/write findings, query task state, and access other agents' public memory via the mosaic skill — with zero knowledge of Matrix or Discord internals
  4. System: Must persist all agent-produced data (findings, memory, conversations) in Postgres with vector embeddings for semantic search. Must survive compaction, session restarts, and agent reprovisioning.

Functional Requirements

FR-01: Knowledge Layer Schema

  • findings table: id (UUID), task_id (FK nullable), agent_id (string), type (string), data (JSONB), embedding (vector), created_at, updated_at
  • agent_memory table: id (UUID), agent_id (string), key (string), value (JSONB), updated_at. Unique constraint on (agent_id, key).
  • tasks table enhancements: status enum with state machine (not-started → in-progress → awaiting-verification → done | blocked | cancelled), assigned_agent (string), parent_task_id (FK nullable), priority (enum), findings (relation)
  • conversation_archive table: id (UUID), session_id (string), agent_id (string), messages (JSONB), summary (text), embedding (vector), created_at
  • All tables use pgvector for embedding columns

FR-02: Task API

  • POST /api/tasks — Create task with title, description, type, priority, assigned_agent, parent_task_id
  • GET /api/tasks — List with filters (status, agent, type, parent)
  • GET /api/tasks/:id — Get task with findings and sub-tasks
  • PATCH /api/tasks/:id — Update status (enforces state machine), assignment, priority
  • State transitions: not-started → in-progress → awaiting-verification → done; any → blocked; any → cancelled
  • Webhook/event emission on state change (for triggering downstream agent sessions)

FR-03: Findings API

  • POST /api/findings — Write finding with task_id, agent_id, type, data (auto-generates embedding)
  • GET /api/findings — List with filters (agent, type, task, date range)
  • POST /api/findings/search — Vector similarity search with query text, optional filters (agent, type, date range), returns ranked results with scores
  • Embedding generation: use configured embedding model (OpenAI text-embedding-3-small or local alternative)

FR-04: Agent Memory API

  • PUT /api/agents/:id/memory/:key — Write/update a memory entry
  • GET /api/agents/:id/memory — List all memory entries for an agent
  • GET /api/agents/:id/memory/:key — Get specific entry
  • DELETE /api/agents/:id/memory/:key — Remove entry
  • Cross-agent read: any agent can read any other agent's memory (no write across agents)

FR-05: Conversation Archive API

  • POST /api/conversations/ingest — Accept OpenClaw JSONL session data, parse, embed, store
  • POST /api/conversations/search — Semantic search across archived conversations
  • GET /api/conversations — List with filters (agent, date range, session_id)
  • Ingestion pipeline: scheduled job or webhook that watches OpenClaw session log directory

FR-06: Mosaic Skill for OpenClaw

  • OpenClaw skill installed in each agent's workspace
  • SKILL.md documenting all available commands
  • CLI commands wrapping Task, Findings, Memory, and Conversation APIs
  • Authentication via API key or token stored in agent's workspace config
  • Commands:
    • mosaic task list|get|update
    • mosaic finding write|search
    • mosaic agent memory read|write
    • mosaic agent status
    • mosaic conversation search

FR-07: Agent Provisioning

  • OpenClaw multi-agent configuration for 3 initial agents: jarvis, builder, medic
  • Each agent has: SOUL.md (personality), AGENTS.md (operational rules), USER.md (shared user context), TOOLS.md, MEMORY.md, memory/
  • Per-agent model config: jarvis=opus, builder=codex/sonnet, medic=haiku
  • Per-agent tool permissions: medic=exec(SSH)+cron, builder=exec+github+read+write, jarvis=all
  • mosaic skill installed in each workspace

FR-08: Channel Bindings

  • Discord: agent-specific channels (#jarvis, #builder, #medic-alerts, #agent-status)
  • Matrix: per-agent observation rooms, per-project rooms, #mosaic-ops system room
  • Matrix rooms auto-created when agents provisioned or projects created
  • Matrix rooms E2E encrypted by default
  • Async fire-and-forget mirroring: Mosaic API mirrors findings/status to Matrix rooms (no agent inference cost)

FR-09: WebSocket Chat

  • WS /api/chat — WebSocket endpoint accepting agent_id, message, returning streamed response
  • Proxies to OpenClaw agent session via OpenClaw's session API
  • Message persistence in conversation_archive
  • Support for multiple concurrent chats (one per agent)

FR-10: WebUI Components

  • Chat component: agent selector dropdown, message input, streaming response display, conversation history
  • Task management: create task form, kanban or list view, task detail with findings, assign agent
  • Agent registry: card/list view showing agent name, role, model, status, last active, channel
  • Findings search: search bar with semantic query, filters (agent, type, date), ranked results
  • Data retention settings: per-workspace retention_days config

FR-11: LLM Provider Management

  • Provider CRUD: add/remove/configure providers (Anthropic, OpenAI, GLM/Z.ai, Ollama, MiniMax, OpenRouter)
  • Per-provider config: API key/endpoint, model list, rate limits, enabled/disabled
  • Connection test: verify provider is reachable and authenticated
  • Per-agent model assignment: primary model + fallback chain
  • ASSUMPTION: Provider config stored in Mosaic DB, synced to OpenClaw at runtime via API call. Rationale: single source of truth, no file sync needed.

FR-12: Usage Dashboard

  • Token count tracking per agent per provider
  • Cost estimation (where provider publishes pricing)
  • Rate limit status display (for providers that expose it)
  • Time-series charts: usage over time (daily/weekly/monthly)
  • ASSUMPTION: Usage data collected from OpenClaw session logs during ingestion. Rationale: OpenClaw already tracks token counts in JSONL logs.

FR-13: Provider Health Monitoring

  • Endpoint latency tracking (per request average, p95, p99)
  • Error rate tracking (4xx, 5xx, timeouts)
  • Status indicator per provider (healthy/degraded/down)
  • ASSUMPTION: Health data collected passively from OpenClaw request logs, not active probing. Rationale: avoids unnecessary API calls and billing.

FR-14: Installer

  • Single command to install: openclaw skills install mosaic-fleet or npx mosaic-fleet init
  • Interactive wizard:
    • Detect existing Mosaic Stack (URL, API key)
    • Provision agent workspaces (create directories, write SOUL.md files)
    • Configure channel bindings (Discord bot token, Matrix homeserver)
    • Set up LLM providers (API keys, endpoints)
    • Verify connectivity (Mosaic API, Discord, Matrix, LLM providers)
  • Idempotent: safe to re-run for updates or adding agents

Non-Functional Requirements

  1. Security:

    • Agent API keys stored encrypted in Mosaic DB (existing credential management)
    • LLM provider API keys encrypted at rest
    • Matrix rooms E2E encrypted
    • Cross-agent memory read-only (agents cannot write to other agents' memory)
    • No Matrix federation (local bus only)
    • Retention policies enforced via scheduled Postgres cleanup job

  2. Performance:

    • Findings vector search: < 200ms for 95th percentile (pgvector with HNSW index)
    • WebSocket chat: first token < 2s (dependent on LLM provider latency)
    • Matrix mirroring: async, fire-and-forget, < 500ms overhead on finding writes
    • Conversation ingestion: batch processing, not blocking agent sessions

  3. Reliability:

    • Knowledge layer is Postgres — same backup/HA as existing Mosaic DB
    • Agent sessions managed by OpenClaw (existing restart/reconnect logic)
    • Provider failover: if primary model fails, automatically try fallback chain
    • Valkey cache for hot task state; Postgres for durability

  4. Observability:

    • Matrix rooms serve as human-readable audit trail
    • Agent activity logged in conversation_archive
    • Provider health metrics exposed via dashboard
    • Task state changes logged with timestamps and actor

Acceptance Criteria

  1. An agent can write a finding via the mosaic skill and another agent can search for and retrieve it
  2. Task state changes trigger downstream agent sessions (e.g., Builder completes → Medic verifies)
  3. A user can chat with any agent via the WebUI and see responses streamed in real-time
  4. A user can search across all agent findings and conversation history semantically
  5. LLM providers can be added, configured, and tested from the WebUI
  6. Per-agent model assignment works: changing an agent's model in WebUI changes its behavior
  7. Usage dashboard shows accurate token counts per agent per provider
  8. Matrix observation rooms show mirrored agent activity within 5 seconds of the finding being written
  9. The installer provisions a working 3-agent fleet from scratch in < 10 minutes
  10. Data retention policies are enforced: findings older than retention_days are purged on schedule
  11. Context survives: an agent's finding from day 1 is still searchable on day 30 via the mosaic skill

Constraints and Dependencies

  1. OpenClaw multi-agent support — must support per-agent workspaces, model config, tool permissions, and channel bindings. Current version appears to support this; verify during Phase 1.
  2. pgvector extension — must be enabled in production Postgres (already available in standard Postgres images)
  3. Embedding model — need an embedding provider configured for vector generation. ASSUMPTION: Use OpenAI text-embedding-3-small initially; support local alternatives (Ollama) later. Rationale: most reliable, cheapest per-token for embeddings.
  4. Existing Mosaic Stack modules — Task API extends existing task infrastructure; Matrix bridge extends existing apps/api/src/bridge/matrix/ module; LLM provider management extends existing apps/api/src/llm/providers/
  5. Discord bot — already configured via OpenClaw; channel bindings extend existing setup
  6. Synapse — already deployed; disable federation, enable E2E encryption
  7. Woodpecker CI — existing CI pipeline; docs-only PRs may need path trigger fix from MS21 leftovers

Risks and Open Questions

  1. Risk: OpenClaw multi-agent API may not support all needed features (tool restrictions, per-agent model override). Mitigation: verify early in Phase 1; fall back to separate OpenClaw instances if needed.
  2. Risk: pgvector embedding quality affects search relevance. Mitigation: test with representative queries; tune embedding model and similarity thresholds.
  3. Risk: WebSocket chat proxy may add latency or reliability issues. Mitigation: benchmark during Phase 2; consider SSE as alternative.
  4. Open Question: How does OpenClaw expose session creation API for Mosaic to proxy chat? Research during Phase 1.
  5. Open Question: What triggers a Medic agent session on task state change? Options: (a) Mosaic webhook → OpenClaw, (b) OpenClaw cron/heartbeat polls Mosaic, (c) Mosaic directly invokes OpenClaw session API. ASSUMPTION: Option (a) — Mosaic sends webhook to OpenClaw on task state change. Rationale: event-driven is more efficient than polling.
  6. Open Question: Embedding generation — synchronous on write or async batch? ASSUMPTION: Async via queue/cron. Rationale: don't block finding writes on embedding API calls.
  7. Risk: Conversation archive could grow large. Mitigation: retention policies + partitioning by date.

Testing and Verification Expectations

  1. Baseline checks: pnpm lint && pnpm build && pnpm test must pass (existing quality gates)
  2. API tests: Each new endpoint (tasks, findings, memory, conversations, chat) must have integration tests
  3. E2E tests: Agent writes finding → another agent searches and finds it (round-trip via mosaic skill)
  4. WebUI tests: Chat component renders, sends message, receives streamed response
  5. Installer test: Run installer on clean environment, verify all 3 agents respond correctly
  6. Evidence format: PR descriptions include test results, CI green confirmation

Milestone / Delivery Intent

MS22: Fleet Evolution

Phase Name Target Definition of Done
P0 Knowledge Layer + Mosaic Skill v0.1.0 Schema deployed, APIs working, mosaic skill installed, session log ingestion running
P1 Agent Fleet Standup v0.2.0 3 agents responding in Discord, reading/writing findings, Matrix rooms active
P2 WebUI Chat + Task Management v0.3.0 Users can chat with agents in WebUI, create/track tasks, search findings
P3 LLM Provider Management v0.4.0 Providers configurable in WebUI, per-agent model assignment, usage dashboard
P4 Installer + Polish v0.5.0 Single-command install, interactive wizard, documentation complete

Target version: v0.5.0 (full fleet) Definition of done: A non-technical user can install the fleet, chat with agents in WebUI and Discord, and agents collaborate via shared knowledge layer with zero context loss.

Assumptions

  1. ASSUMPTION: Provider config stored in Mosaic DB, synced to OpenClaw via API. Rationale: single source of truth, avoids config file drift.
  2. ASSUMPTION: Use OpenAI text-embedding-3-small for embeddings initially. Rationale: reliable, cheap, well-supported. Can swap to Ollama later.
  3. ASSUMPTION: Mosaic webhook → OpenClaw for task state change triggers. Rationale: event-driven > polling, lower latency.
  4. ASSUMPTION: Embedding generation is async (queue/cron, not synchronous on write). Rationale: don't block agent work on external API calls.
  5. ASSUMPTION: Usage data harvested from OpenClaw session logs during ingestion. Rationale: data already exists in JSONL, no separate telemetry needed.
  6. ASSUMPTION: Provider health data collected passively from request logs. Rationale: avoids unnecessary API calls and billing.
  7. ASSUMPTION: Matrix rooms are local only (no federation). Rationale: per Jason's decision — federation at Mosaic Stack level, not Matrix level.
  8. ASSUMPTION: Start with 3 agents (jarvis, builder, medic). Rationale: per Jason's direction — add agents when workflows justify, not speculatively.
Reference in New Issue View Git Blame Copy Permalink