516 lines
17 KiB
Markdown
516 lines
17 KiB
Markdown
# Mosaic Stack — Developer Guide
|
|
|
|
## Table of Contents
|
|
|
|
1. [Architecture Overview](#architecture-overview)
|
|
2. [Local Development Setup](#local-development-setup)
|
|
3. [Building and Testing](#building-and-testing)
|
|
4. [Adding New Agent Tools](#adding-new-agent-tools)
|
|
5. [Adding New MCP Tools](#adding-new-mcp-tools)
|
|
6. [Database Schema and Migrations](#database-schema-and-migrations)
|
|
7. [API Endpoint Reference](#api-endpoint-reference)
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
Mosaic Stack is a TypeScript monorepo managed with **pnpm workspaces** and
|
|
**Turborepo**.
|
|
|
|
```
|
|
mosaic-mono-v1/
|
|
├── apps/
|
|
│ ├── gateway/ # NestJS + Fastify API server
|
|
│ └── web/ # Next.js 16 + React 19 web dashboard
|
|
├── packages/
|
|
│ ├── agent/ # Agent session types (shared)
|
|
│ ├── auth/ # BetterAuth configuration
|
|
│ ├── brain/ # Structured data layer (projects, tasks, missions)
|
|
│ ├── cli/ # mosaic CLI and TUI (Ink)
|
|
│ ├── coord/ # Mission coordination engine
|
|
│ ├── db/ # Drizzle ORM schema, migrations, client
|
|
│ ├── design-tokens/ # Shared design system tokens
|
|
│ ├── log/ # Agent log ingestion and tiering
|
|
│ ├── memory/ # Preference and insight storage
|
|
│ ├── mosaic/ # Install wizard and bootstrap utilities
|
|
│ ├── prdy/ # PRD wizard CLI
|
|
│ ├── quality-rails/ # Code quality scaffolder CLI
|
|
│ ├── queue/ # Valkey-backed task queue
|
|
│ └── types/ # Shared TypeScript types
|
|
├── docker/ # Dockerfile(s) for containerized deployment
|
|
├── infra/ # Infra config (OTEL collector, pg-init scripts)
|
|
├── docker-compose.yml # Local services (Postgres, Valkey, OTEL, Jaeger)
|
|
└── CLAUDE.md # Project conventions for AI coding agents
|
|
```
|
|
|
|
### Key Technology Choices
|
|
|
|
| Concern | Technology |
|
|
| ----------------- | ---------------------------------------- |
|
|
| API framework | NestJS with Fastify adapter |
|
|
| Web framework | Next.js 16 (App Router), React 19 |
|
|
| ORM | Drizzle ORM |
|
|
| Database | PostgreSQL 17 + pgvector extension |
|
|
| Auth | BetterAuth |
|
|
| Agent harness | Pi SDK (`@mariozechner/pi-coding-agent`) |
|
|
| Queue | Valkey 8 (Redis-compatible) |
|
|
| Build | pnpm workspaces + Turborepo |
|
|
| CI | Woodpecker CI |
|
|
| Observability | OpenTelemetry → Jaeger |
|
|
| Module resolution | NodeNext (ESM everywhere) |
|
|
|
|
### Module System
|
|
|
|
All packages use `"type": "module"` and NodeNext resolution. Import paths must
|
|
include the `.js` extension even when the source file is `.ts`.
|
|
|
|
NestJS `@Inject()` decorators must be used explicitly because `tsx`/`esbuild`
|
|
does not support `emitDecoratorMetadata`.
|
|
|
|
---
|
|
|
|
## Local Development Setup
|
|
|
|
### Prerequisites
|
|
|
|
- Node.js 20+
|
|
- pnpm 9+
|
|
- Docker and Docker Compose
|
|
|
|
### 1. Clone and Install Dependencies
|
|
|
|
```bash
|
|
git clone <repo-url> mosaic-mono-v1
|
|
cd mosaic-mono-v1
|
|
pnpm install
|
|
```
|
|
|
|
### 2. Start Infrastructure Services
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
This starts:
|
|
|
|
| Service | Port | Description |
|
|
| ------------------------ | -------------- | -------------------- |
|
|
| PostgreSQL 17 + pgvector | `5433` (host) | Primary database |
|
|
| Valkey 8 | `6380` (host) | Queue and cache |
|
|
| OpenTelemetry Collector | `4317`, `4318` | OTEL gRPC and HTTP |
|
|
| Jaeger | `16686` | Distributed trace UI |
|
|
|
|
### 3. Configure Environment
|
|
|
|
Create a `.env` file in the monorepo root:
|
|
|
|
```env
|
|
# Database (matches docker-compose defaults)
|
|
DATABASE_URL=postgresql://mosaic:mosaic@localhost:5433/mosaic
|
|
|
|
# Auth (required — generate a random 32+ char string)
|
|
BETTER_AUTH_SECRET=change-me-to-a-random-secret
|
|
|
|
# Gateway
|
|
GATEWAY_PORT=14242
|
|
GATEWAY_CORS_ORIGIN=http://localhost:3000
|
|
|
|
# Web
|
|
NEXT_PUBLIC_GATEWAY_URL=http://localhost:14242
|
|
|
|
# Optional: Ollama
|
|
OLLAMA_BASE_URL=http://localhost:11434
|
|
OLLAMA_MODELS=llama3.2
|
|
```
|
|
|
|
The gateway loads `.env` from the monorepo root via `dotenv` at startup
|
|
(`apps/gateway/src/main.ts`).
|
|
|
|
### 4. Push the Database Schema
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/db db:push
|
|
```
|
|
|
|
This applies the Drizzle schema directly to the database (development only; use
|
|
migrations in production).
|
|
|
|
### 5. Start the Gateway
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/gateway exec tsx src/main.ts
|
|
```
|
|
|
|
The gateway starts on port `14242` by default.
|
|
|
|
### 6. Start the Web App
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/web dev
|
|
```
|
|
|
|
The web app starts on port `3000` by default.
|
|
|
|
---
|
|
|
|
## Building and Testing
|
|
|
|
### TypeScript Typecheck
|
|
|
|
```bash
|
|
pnpm typecheck
|
|
```
|
|
|
|
Runs `tsc --noEmit` across all packages in dependency order via Turborepo.
|
|
|
|
### Lint
|
|
|
|
```bash
|
|
pnpm lint
|
|
```
|
|
|
|
Runs ESLint across all packages. Config is in `eslint.config.mjs` at the root.
|
|
|
|
### Format Check
|
|
|
|
```bash
|
|
pnpm format:check
|
|
```
|
|
|
|
Runs Prettier in check mode. To auto-fix:
|
|
|
|
```bash
|
|
pnpm format
|
|
```
|
|
|
|
### Tests
|
|
|
|
```bash
|
|
pnpm test
|
|
```
|
|
|
|
Runs Vitest across all packages. The workspace config is at
|
|
`vitest.workspace.ts`.
|
|
|
|
### Build
|
|
|
|
```bash
|
|
pnpm build
|
|
```
|
|
|
|
Builds all packages and apps in dependency order.
|
|
|
|
### Pre-Push Gates (MANDATORY)
|
|
|
|
All three must pass before any push:
|
|
|
|
```bash
|
|
pnpm format:check && pnpm typecheck && pnpm lint
|
|
```
|
|
|
|
A pre-push hook enforces this mechanically.
|
|
|
|
---
|
|
|
|
## Adding New Agent Tools
|
|
|
|
Agent tools are Pi SDK `ToolDefinition` objects registered in
|
|
`apps/gateway/src/agent/agent.service.ts`.
|
|
|
|
### 1. Create a Tool Factory File
|
|
|
|
Add a new file in `apps/gateway/src/agent/tools/`:
|
|
|
|
```typescript
|
|
// apps/gateway/src/agent/tools/my-tools.ts
|
|
import { Type } from '@sinclair/typebox';
|
|
import type { ToolDefinition } from '@mariozechner/pi-coding-agent';
|
|
|
|
export function createMyTools(): ToolDefinition[] {
|
|
const myTool: ToolDefinition = {
|
|
name: 'my_tool_name',
|
|
label: 'Human Readable Label',
|
|
description: 'What this tool does.',
|
|
parameters: Type.Object({
|
|
input: Type.String({ description: 'The input parameter' }),
|
|
}),
|
|
async execute(_toolCallId, params) {
|
|
const { input } = params as { input: string };
|
|
const result = `Processed: ${input}`;
|
|
return {
|
|
content: [{ type: 'text' as const, text: result }],
|
|
details: undefined,
|
|
};
|
|
},
|
|
};
|
|
|
|
return [myTool];
|
|
}
|
|
```
|
|
|
|
### 2. Register the Tools in AgentService
|
|
|
|
In `apps/gateway/src/agent/agent.service.ts`, import and call your factory
|
|
alongside the existing tool registrations:
|
|
|
|
```typescript
|
|
import { createMyTools } from './tools/my-tools.js';
|
|
|
|
// Inside the session creation logic where tools are assembled:
|
|
const tools: ToolDefinition[] = [
|
|
...createBrainTools(this.brain),
|
|
...createCoordTools(this.coordService),
|
|
...createMemoryTools(this.memory, this.embeddingService),
|
|
...createFileTools(sandboxDir),
|
|
...createGitTools(sandboxDir),
|
|
...createShellTools(sandboxDir),
|
|
...createWebTools(),
|
|
...createMyTools(), // Add this line
|
|
...mcpTools,
|
|
...skillTools,
|
|
];
|
|
```
|
|
|
|
### 3. Export from the Tools Index
|
|
|
|
Add an export to `apps/gateway/src/agent/tools/index.ts`:
|
|
|
|
```typescript
|
|
export { createMyTools } from './my-tools.js';
|
|
```
|
|
|
|
### 4. Typecheck and Test
|
|
|
|
```bash
|
|
pnpm typecheck
|
|
pnpm test
|
|
```
|
|
|
|
---
|
|
|
|
## Adding New MCP Tools
|
|
|
|
Mosaic connects to external MCP servers via `McpClientService`. To expose tools
|
|
from a new MCP server:
|
|
|
|
### 1. Run an MCP Server
|
|
|
|
Implement a standard MCP server that exposes tools via the streamable HTTP
|
|
transport or SSE transport. The server must accept connections at a `/mcp`
|
|
endpoint.
|
|
|
|
### 2. Configure `MCP_SERVERS`
|
|
|
|
In your `.env`:
|
|
|
|
```env
|
|
MCP_SERVERS='[{"name":"my-server","url":"http://localhost:3001/mcp"}]'
|
|
```
|
|
|
|
With authentication:
|
|
|
|
```env
|
|
MCP_SERVERS='[{"name":"secure-server","url":"http://my-server/mcp","headers":{"Authorization":"Bearer token"}}]'
|
|
```
|
|
|
|
### 3. Restart the Gateway
|
|
|
|
On startup, `McpClientService` (`apps/gateway/src/mcp-client/mcp-client.service.ts`)
|
|
connects to each configured server, calls `tools/list`, and bridges the results
|
|
to Pi SDK `ToolDefinition` format. These tools become available in all new agent
|
|
sessions.
|
|
|
|
### Tool Naming
|
|
|
|
Bridged MCP tool names are taken directly from the MCP server's tool manifest.
|
|
Ensure names do not conflict with built-in tools (check
|
|
`apps/gateway/src/agent/tools/`).
|
|
|
|
---
|
|
|
|
## Database Schema and Migrations
|
|
|
|
The schema lives in a single file:
|
|
`packages/db/src/schema.ts`
|
|
|
|
### Schema Overview
|
|
|
|
| Table | Purpose |
|
|
| -------------------- | ------------------------------------------------- |
|
|
| `users` | User accounts (BetterAuth-compatible) |
|
|
| `sessions` | Auth sessions |
|
|
| `accounts` | OAuth accounts |
|
|
| `verifications` | Email verification tokens |
|
|
| `projects` | Project records |
|
|
| `missions` | Mission records (linked to projects) |
|
|
| `tasks` | Task records (linked to projects and/or missions) |
|
|
| `conversations` | Chat conversation metadata |
|
|
| `messages` | Individual chat messages |
|
|
| `preferences` | Per-user key-value preference store |
|
|
| `insights` | Vector-embedded memory insights |
|
|
| `agent_logs` | Agent interaction logs (hot/warm/cold tiers) |
|
|
| `skills` | Installed agent skills |
|
|
| `summarization_jobs` | Log summarization job tracking |
|
|
|
|
The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.
|
|
|
|
### Development: Push Schema
|
|
|
|
Apply schema changes directly to the dev database (no migration files created):
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/db db:push
|
|
```
|
|
|
|
### Generating Migrations
|
|
|
|
For production-safe, versioned changes:
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/db db:generate
|
|
```
|
|
|
|
This creates a new SQL migration file in `packages/db/drizzle/`.
|
|
|
|
### Running Migrations
|
|
|
|
```bash
|
|
pnpm --filter @mosaic/db db:migrate
|
|
```
|
|
|
|
### Drizzle Config
|
|
|
|
Config is at `packages/db/drizzle.config.ts`. The schema file path and output
|
|
directory are defined there.
|
|
|
|
### Adding a New Table
|
|
|
|
1. Add the table definition to `packages/db/src/schema.ts`.
|
|
2. Export it from `packages/db/src/index.ts`.
|
|
3. Run `pnpm --filter @mosaic/db db:push` (dev) or
|
|
`pnpm --filter @mosaic/db db:generate && pnpm --filter @mosaic/db db:migrate`
|
|
(production).
|
|
|
|
---
|
|
|
|
## API Endpoint Reference
|
|
|
|
All endpoints are served by the gateway at `http://localhost:14242` by default.
|
|
|
|
### Authentication
|
|
|
|
Authentication uses BetterAuth session cookies. The auth handler is mounted at
|
|
`/api/auth/*` via a Fastify low-level hook in
|
|
`apps/gateway/src/auth/auth.controller.ts`.
|
|
|
|
| Endpoint | Method | Description |
|
|
| ------------------------- | ------ | -------------------------------- |
|
|
| `/api/auth/sign-in/email` | POST | Sign in with email/password |
|
|
| `/api/auth/sign-up/email` | POST | Register a new account |
|
|
| `/api/auth/sign-out` | POST | Sign out (clears session cookie) |
|
|
| `/api/auth/get-session` | GET | Returns the current session |
|
|
|
|
### Chat
|
|
|
|
WebSocket namespace `/chat` (Socket.IO). Authentication via session cookie.
|
|
|
|
Events sent by the client:
|
|
|
|
| Event | Payload | Description |
|
|
| --------- | --------------------------------------------------- | -------------- |
|
|
| `message` | `{ content, conversationId?, provider?, modelId? }` | Send a message |
|
|
|
|
Events emitted by the server:
|
|
|
|
| Event | Payload | Description |
|
|
| ------- | --------------------------- | ---------------------- |
|
|
| `token` | `{ token, conversationId }` | Streaming token |
|
|
| `end` | `{ conversationId }` | Stream complete |
|
|
| `error` | `{ message }` | Error during streaming |
|
|
|
|
HTTP endpoints (`apps/gateway/src/chat/chat.controller.ts`):
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| -------------------------------------- | ------ | ---- | ------------------------------- |
|
|
| `/api/chat/conversations` | GET | User | List conversations |
|
|
| `/api/chat/conversations/:id/messages` | GET | User | Get messages for a conversation |
|
|
|
|
### Admin
|
|
|
|
All admin endpoints require `role = admin`.
|
|
|
|
| Endpoint | Method | Description |
|
|
| --------------------------------- | ------ | -------------------- |
|
|
| `GET /api/admin/users` | GET | List all users |
|
|
| `GET /api/admin/users/:id` | GET | Get a single user |
|
|
| `POST /api/admin/users` | POST | Create a user |
|
|
| `PATCH /api/admin/users/:id/role` | PATCH | Update user role |
|
|
| `POST /api/admin/users/:id/ban` | POST | Ban a user |
|
|
| `POST /api/admin/users/:id/unban` | POST | Unban a user |
|
|
| `DELETE /api/admin/users/:id` | DELETE | Delete a user |
|
|
| `GET /api/admin/health` | GET | System health status |
|
|
|
|
### Agent / Providers
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------------------ | ------ | ---- | ----------------------------------- |
|
|
| `GET /api/agent/providers` | GET | User | List all providers and their models |
|
|
| `GET /api/agent/providers/models` | GET | User | List available models |
|
|
| `POST /api/agent/providers/:id/test` | POST | User | Test provider connectivity |
|
|
|
|
### Projects / Brain
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| -------------------------------- | ------ | ---- | ---------------- |
|
|
| `GET /api/brain/projects` | GET | User | List projects |
|
|
| `POST /api/brain/projects` | POST | User | Create a project |
|
|
| `GET /api/brain/projects/:id` | GET | User | Get a project |
|
|
| `PATCH /api/brain/projects/:id` | PATCH | User | Update a project |
|
|
| `DELETE /api/brain/projects/:id` | DELETE | User | Delete a project |
|
|
| `GET /api/brain/tasks` | GET | User | List tasks |
|
|
| `POST /api/brain/tasks` | POST | User | Create a task |
|
|
| `GET /api/brain/tasks/:id` | GET | User | Get a task |
|
|
| `PATCH /api/brain/tasks/:id` | PATCH | User | Update a task |
|
|
| `DELETE /api/brain/tasks/:id` | DELETE | User | Delete a task |
|
|
|
|
### Memory / Preferences
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ----------------------------- | ------ | ---- | -------------------- |
|
|
| `GET /api/memory/preferences` | GET | User | Get user preferences |
|
|
| `PUT /api/memory/preferences` | PUT | User | Upsert a preference |
|
|
|
|
### MCP Server (Gateway-side)
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ----------- | ------ | --------------------------------------------- | ----------------------------- |
|
|
| `POST /mcp` | POST | User (session cookie or Authorization header) | MCP streamable HTTP transport |
|
|
| `GET /mcp` | GET | User | MCP SSE stream reconnect |
|
|
|
|
### Skills
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------ | ------ | ----- | --------------------- |
|
|
| `GET /api/skills` | GET | User | List installed skills |
|
|
| `POST /api/skills` | POST | Admin | Install a skill |
|
|
| `PATCH /api/skills/:id` | PATCH | Admin | Update a skill |
|
|
| `DELETE /api/skills/:id` | DELETE | Admin | Remove a skill |
|
|
|
|
### Coord (Mission Coordination)
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------------- | ------ | ---- | ---------------- |
|
|
| `GET /api/coord/missions` | GET | User | List missions |
|
|
| `POST /api/coord/missions` | POST | User | Create a mission |
|
|
| `GET /api/coord/missions/:id` | GET | User | Get a mission |
|
|
| `PATCH /api/coord/missions/:id` | PATCH | User | Update a mission |
|
|
|
|
### Observability
|
|
|
|
OpenTelemetry traces are exported to the OTEL collector (`OTEL_EXPORTER_OTLP_ENDPOINT`).
|
|
View traces in Jaeger at `http://localhost:16686`.
|
|
|
|
Tracing is initialized before NestJS bootstrap in
|
|
`apps/gateway/src/tracing.ts`. The import order in `apps/gateway/src/main.ts`
|
|
is intentional: `import './tracing.js'` must come before any NestJS imports.
|