476 lines
18 KiB
Markdown
476 lines
18 KiB
Markdown
# Mosaic Stack — Developer Guide
|
|
|
|
## Table of Contents
|
|
|
|
1. [Architecture Overview](#architecture-overview)
|
|
2. [Local Development Setup](#local-development-setup)
|
|
3. [Building and Testing](#building-and-testing)
|
|
4. [Adding New Agent Tools](#adding-new-agent-tools)
|
|
5. [Adding New MCP Tools](#adding-new-mcp-tools)
|
|
6. [Database Schema and Migrations](#database-schema-and-migrations)
|
|
7. [API Endpoint Reference](#api-endpoint-reference)
|
|
8. [Local Fleet Canary](./fleet-local-canary.md)
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
Mosaic Stack is a TypeScript monorepo managed with **pnpm workspaces** and
|
|
**Turborepo**.
|
|
|
|
```
|
|
mosaic-mono-v1/
|
|
├── apps/
|
|
│ ├── gateway/ # NestJS + Fastify API server
|
|
│ └── web/ # Next.js 16 + React 19 web dashboard
|
|
├── packages/
|
|
│ ├── agent/ # Agent session types (shared)
|
|
│ ├── auth/ # BetterAuth configuration
|
|
│ ├── brain/ # Structured data layer (projects, tasks, missions)
|
|
│ ├── cli/ # mosaic CLI and TUI (Ink)
|
|
│ ├── coord/ # Mission coordination engine
|
|
│ ├── db/ # Drizzle ORM schema, migrations, client
|
|
│ ├── design-tokens/ # Shared design system tokens
|
|
│ ├── log/ # Agent log ingestion and tiering
|
|
│ ├── memory/ # Preference and insight storage
|
|
│ ├── mosaic/ # Install wizard and bootstrap utilities
|
|
│ ├── prdy/ # PRD wizard CLI
|
|
│ ├── quality-rails/ # Code quality scaffolder CLI
|
|
│ ├── queue/ # Valkey-backed task queue
|
|
│ └── types/ # Shared TypeScript types
|
|
├── docker/ # Dockerfile(s) for containerized deployment
|
|
├── infra/ # Infrastructure configuration (for example, OTEL collector)
|
|
├── docker-compose.yml # Local services (Postgres, Valkey, OTEL, Jaeger)
|
|
└── CLAUDE.md # Project conventions for AI coding agents
|
|
```
|
|
|
|
### Key Technology Choices
|
|
|
|
| Concern | Technology |
|
|
| ----------------- | ---------------------------------------- |
|
|
| API framework | NestJS with Fastify adapter |
|
|
| Web framework | Next.js 16 (App Router), React 19 |
|
|
| ORM | Drizzle ORM |
|
|
| Database | PostgreSQL 17 + pgvector extension |
|
|
| Auth | BetterAuth |
|
|
| Agent harness | Pi SDK (`@mariozechner/pi-coding-agent`) |
|
|
| Queue | Valkey 8 (Redis-compatible) |
|
|
| Build | pnpm workspaces + Turborepo |
|
|
| CI | Woodpecker CI |
|
|
| Observability | OpenTelemetry → Jaeger |
|
|
| Module resolution | NodeNext (ESM everywhere) |
|
|
|
|
### Module System
|
|
|
|
All packages use `"type": "module"` and NodeNext resolution. Import paths must
|
|
include the `.js` extension even when the source file is `.ts`.
|
|
|
|
NestJS `@Inject()` decorators must be used explicitly because `tsx`/`esbuild`
|
|
does not support `emitDecoratorMetadata`.
|
|
|
|
---
|
|
|
|
## Local Development Setup
|
|
|
|
### Prerequisites
|
|
|
|
- Node.js 20+
|
|
- pnpm 9+
|
|
- Docker and Docker Compose
|
|
|
|
### 1. Clone and Install Dependencies
|
|
|
|
```bash
|
|
git clone <repo-url> mosaic-mono-v1
|
|
cd mosaic-mono-v1
|
|
pnpm install
|
|
```
|
|
|
|
### 2. Use the local PGlite tier
|
|
|
|
The supported local tier is in-process PGlite and requires no PostgreSQL service. Leave
|
|
`DATABASE_URL` unset for this route. Its default local configuration uses PGlite and performs no
|
|
external database probe.
|
|
|
|
If a local queue service is useful, start only that non-PostgreSQL service:
|
|
|
|
```bash
|
|
docker compose up -d valkey
|
|
```
|
|
|
|
Do not use the current Compose PostgreSQL service: it mounts legacy `infra/pg-init` SQL and is
|
|
not qualified for KBN-101. Start OTEL Collector or Jaeger individually only when needed and
|
|
without starting PostgreSQL.
|
|
|
|
### 3. Gateway/Web local process (held)
|
|
|
|
Do not start the current Gateway or web process as a local PGlite route. Gateway first loads the
|
|
daemon configuration and then project environment files without a tier guard; a pre-existing
|
|
`DATABASE_URL` can select PostgreSQL, where current startup still reaches runtime DDL/migrations.
|
|
Creating a root `.env` that omits `DATABASE_URL` does not make this safe, so neither a local
|
|
credential file nor a web environment file is a current developer procedure.
|
|
|
|
PGlite remains the supported in-process data-layer implementation, and the optional Valkey command
|
|
above remains safe because it does not start PostgreSQL. A safe Gateway/Web local procedure is held
|
|
until KBN-101-02 rejects a daemon, inherited, root, or app-local PostgreSQL DSN and any non-local
|
|
tier before connection or DDL; KBN-101-05 then supplies the production renderer/Vault process-exec
|
|
or `LoadCredential` boundary.
|
|
|
|
### Held future procedure
|
|
|
|
PostgreSQL local and federated deployment are held until KBN-101-00 (external bootstrap),
|
|
KBN-101-03 (runner), and KBN-101-05 (renderer-backed deployment) land. The following is the
|
|
**held, non-operative future activation order with no current command authority**:
|
|
|
|
external bootstrap → TLS/roles → `mosaic-db-migrator --run` →
|
|
`mosaic-db-migrator --verify` → Gateway/Compose readiness.
|
|
|
|
Neither current Compose nor this development guide authorizes PostgreSQL initialization SQL,
|
|
manual DDL, or a pre-runner start.
|
|
|
|
### 5. Gateway/Web start (held)
|
|
|
|
No Gateway/Web start command is currently authorized for the local PGlite route. Do not use root
|
|
`pnpm dev` as a workaround: it additionally starts configured integrations and cannot establish the
|
|
required local-tier/DSN isolation. Resume this section only after KBN-101-02 provides its
|
|
fail-closed local-startup evidence.
|
|
|
|
---
|
|
|
|
## Building and Testing
|
|
|
|
### TypeScript Typecheck
|
|
|
|
```bash
|
|
pnpm typecheck
|
|
```
|
|
|
|
Runs `tsc --noEmit` across all packages in dependency order via Turborepo.
|
|
|
|
### Lint
|
|
|
|
```bash
|
|
pnpm lint
|
|
```
|
|
|
|
Runs ESLint across all packages. Config is in `eslint.config.mjs` at the root.
|
|
|
|
### Format Check
|
|
|
|
```bash
|
|
pnpm format:check
|
|
```
|
|
|
|
Runs Prettier in check mode. To auto-fix:
|
|
|
|
```bash
|
|
pnpm format
|
|
```
|
|
|
|
### Tests
|
|
|
|
```bash
|
|
pnpm test
|
|
```
|
|
|
|
Runs Vitest across all packages. The workspace config is at
|
|
`vitest.workspace.ts`.
|
|
|
|
### Build
|
|
|
|
```bash
|
|
pnpm build
|
|
```
|
|
|
|
Builds all packages and apps in dependency order.
|
|
|
|
### Pre-Push Gates (MANDATORY)
|
|
|
|
All three must pass before any push:
|
|
|
|
```bash
|
|
pnpm format:check && pnpm typecheck && pnpm lint
|
|
```
|
|
|
|
A pre-push hook enforces this mechanically.
|
|
|
|
---
|
|
|
|
## Adding New Agent Tools
|
|
|
|
Agent tools are Pi SDK `ToolDefinition` objects registered in
|
|
`apps/gateway/src/agent/agent.service.ts`.
|
|
|
|
### 1. Create a Tool Factory File
|
|
|
|
Add a new file in `apps/gateway/src/agent/tools/`:
|
|
|
|
```typescript
|
|
// apps/gateway/src/agent/tools/my-tools.ts
|
|
import { Type } from '@sinclair/typebox';
|
|
import type { ToolDefinition } from '@mariozechner/pi-coding-agent';
|
|
|
|
export function createMyTools(): ToolDefinition[] {
|
|
const myTool: ToolDefinition = {
|
|
name: 'my_tool_name',
|
|
label: 'Human Readable Label',
|
|
description: 'What this tool does.',
|
|
parameters: Type.Object({
|
|
input: Type.String({ description: 'The input parameter' }),
|
|
}),
|
|
async execute(_toolCallId, params) {
|
|
const { input } = params as { input: string };
|
|
const result = `Processed: ${input}`;
|
|
return {
|
|
content: [{ type: 'text' as const, text: result }],
|
|
details: undefined,
|
|
};
|
|
},
|
|
};
|
|
|
|
return [myTool];
|
|
}
|
|
```
|
|
|
|
### 2. Register the Tools in AgentService
|
|
|
|
In `apps/gateway/src/agent/agent.service.ts`, import and call your factory
|
|
alongside the existing tool registrations:
|
|
|
|
```typescript
|
|
import { createMyTools } from './tools/my-tools.js';
|
|
|
|
// Inside the session creation logic where tools are assembled:
|
|
const tools: ToolDefinition[] = [
|
|
...createBrainTools(this.brain),
|
|
...createCoordTools(this.coordService),
|
|
...createMemoryTools(this.memory, this.embeddingService),
|
|
...createFileTools(sandboxDir),
|
|
...createGitTools(sandboxDir),
|
|
...createShellTools(sandboxDir),
|
|
...createWebTools(),
|
|
...createMyTools(), // Add this line
|
|
...mcpTools,
|
|
...skillTools,
|
|
];
|
|
```
|
|
|
|
### 3. Export from the Tools Index
|
|
|
|
Add an export to `apps/gateway/src/agent/tools/index.ts`:
|
|
|
|
```typescript
|
|
export { createMyTools } from './my-tools.js';
|
|
```
|
|
|
|
### 4. Typecheck and Test
|
|
|
|
```bash
|
|
pnpm typecheck
|
|
pnpm test
|
|
```
|
|
|
|
---
|
|
|
|
## Adding New MCP Tools
|
|
|
|
Mosaic connects to external MCP servers via `McpClientService`. To expose tools
|
|
from a new MCP server:
|
|
|
|
### 1. Run an MCP Server
|
|
|
|
Implement a standard MCP server that exposes tools via the streamable HTTP
|
|
transport or SSE transport. The server must accept connections at a `/mcp`
|
|
endpoint.
|
|
|
|
### 2. Gateway MCP configuration (held)
|
|
|
|
Do not configure MCP endpoint credentials, write them to a local environment file, or restart the
|
|
Gateway from this guide. Gateway/Web startup is held until KBN-101-02 supplies fail-closed
|
|
local-tier/DSN isolation and KBN-101-05 supplies the renderer/Vault process-exec or
|
|
`LoadCredential` secret-consumer interface. The future authenticated MCP route requires verified
|
|
HTTPS and certificate validation; plaintext bearer-token examples are forbidden.
|
|
|
|
### Tool Naming
|
|
|
|
Bridged MCP tool names are taken directly from the MCP server's tool manifest.
|
|
Ensure names do not conflict with built-in tools (check
|
|
`apps/gateway/src/agent/tools/`).
|
|
|
|
---
|
|
|
|
## Database Schema and Migrations
|
|
|
|
The schema lives in a single file:
|
|
`packages/db/src/schema.ts`
|
|
|
|
### Schema Overview
|
|
|
|
| Table | Purpose |
|
|
| -------------------- | ------------------------------------------------- |
|
|
| `users` | User accounts (BetterAuth-compatible) |
|
|
| `sessions` | Auth sessions |
|
|
| `accounts` | OAuth accounts |
|
|
| `verifications` | Email verification tokens |
|
|
| `projects` | Project records |
|
|
| `missions` | Mission records (linked to projects) |
|
|
| `tasks` | Task records (linked to projects and/or missions) |
|
|
| `conversations` | Chat conversation metadata |
|
|
| `messages` | Individual chat messages |
|
|
| `preferences` | Per-user key-value preference store |
|
|
| `insights` | Vector-embedded memory insights |
|
|
| `agent_logs` | Agent interaction logs (hot/warm/cold tiers) |
|
|
| `skills` | Installed agent skills |
|
|
| `summarization_jobs` | Log summarization job tracking |
|
|
|
|
The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.
|
|
|
|
### PostgreSQL schema work (held)
|
|
|
|
Do not prepare or run a PostgreSQL target from this branch. The sole runner, bootstrap, and
|
|
renderer are future KBN-101 artifacts, not current commands. When KBN-101-00/-03/-05 land, the
|
|
owned activation documentation will require external bootstrap → TLS/roles → runner `--run` →
|
|
runner `--verify` → Gateway/Compose readiness.
|
|
|
|
### Generating migration artifacts
|
|
|
|
`pnpm --filter @mosaicstack/db db:generate` is an offline artifact-generation command. It does
|
|
not authorize connecting to or initializing PostgreSQL. A future reviewed PostgreSQL procedure
|
|
will determine when its output is applied.
|
|
|
|
### Drizzle Config
|
|
|
|
Config is at `packages/db/drizzle.config.ts`. The schema file path and output directory are
|
|
defined there.
|
|
|
|
### Adding a New Table
|
|
|
|
1. Add the table definition to `packages/db/src/schema.ts`.
|
|
2. Export it from `packages/db/src/index.ts`.
|
|
3. Generate the offline artifact with `pnpm --filter @mosaicstack/db db:generate`.
|
|
4. Do not apply it to PostgreSQL until the future KBN-101 activation artifacts and their owned
|
|
procedure are available. Direct schema push is not a production-like workflow.
|
|
|
|
---
|
|
|
|
## API Endpoint Reference
|
|
|
|
All endpoints are served by the gateway at `http://localhost:14242` by default.
|
|
|
|
### Authentication
|
|
|
|
Authentication uses BetterAuth session cookies. The auth handler is mounted at
|
|
`/api/auth/*` via a Fastify low-level hook in
|
|
`apps/gateway/src/auth/auth.controller.ts`.
|
|
|
|
| Endpoint | Method | Description |
|
|
| ------------------------- | ------ | -------------------------------- |
|
|
| `/api/auth/sign-in/email` | POST | Sign in with email/password |
|
|
| `/api/auth/sign-up/email` | POST | Register a new account |
|
|
| `/api/auth/sign-out` | POST | Sign out (clears session cookie) |
|
|
| `/api/auth/get-session` | GET | Returns the current session |
|
|
|
|
### Chat
|
|
|
|
WebSocket namespace `/chat` (Socket.IO). Authentication via session cookie.
|
|
|
|
Events sent by the client:
|
|
|
|
| Event | Payload | Description |
|
|
| --------- | --------------------------------------------------- | -------------- |
|
|
| `message` | `{ content, conversationId?, provider?, modelId? }` | Send a message |
|
|
|
|
Events emitted by the server:
|
|
|
|
| Event | Payload | Description |
|
|
| ------- | --------------------------- | ---------------------- |
|
|
| `token` | `{ token, conversationId }` | Streaming token |
|
|
| `end` | `{ conversationId }` | Stream complete |
|
|
| `error` | `{ message }` | Error during streaming |
|
|
|
|
HTTP endpoints (`apps/gateway/src/chat/chat.controller.ts`):
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| -------------------------------------- | ------ | ---- | ------------------------------- |
|
|
| `/api/chat/conversations` | GET | User | List conversations |
|
|
| `/api/chat/conversations/:id/messages` | GET | User | Get messages for a conversation |
|
|
|
|
### Admin
|
|
|
|
All admin endpoints require `role = admin`.
|
|
|
|
| Endpoint | Method | Description |
|
|
| --------------------------------- | ------ | -------------------- |
|
|
| `GET /api/admin/users` | GET | List all users |
|
|
| `GET /api/admin/users/:id` | GET | Get a single user |
|
|
| `POST /api/admin/users` | POST | Create a user |
|
|
| `PATCH /api/admin/users/:id/role` | PATCH | Update user role |
|
|
| `POST /api/admin/users/:id/ban` | POST | Ban a user |
|
|
| `POST /api/admin/users/:id/unban` | POST | Unban a user |
|
|
| `DELETE /api/admin/users/:id` | DELETE | Delete a user |
|
|
| `GET /api/admin/health` | GET | System health status |
|
|
|
|
### Agent / Providers
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------------------ | ------ | ---- | ----------------------------------- |
|
|
| `GET /api/agent/providers` | GET | User | List all providers and their models |
|
|
| `GET /api/agent/providers/models` | GET | User | List available models |
|
|
| `POST /api/agent/providers/:id/test` | POST | User | Test provider connectivity |
|
|
|
|
### Projects / Brain
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| -------------------------------- | ------ | ---- | ---------------- |
|
|
| `GET /api/brain/projects` | GET | User | List projects |
|
|
| `POST /api/brain/projects` | POST | User | Create a project |
|
|
| `GET /api/brain/projects/:id` | GET | User | Get a project |
|
|
| `PATCH /api/brain/projects/:id` | PATCH | User | Update a project |
|
|
| `DELETE /api/brain/projects/:id` | DELETE | User | Delete a project |
|
|
| `GET /api/brain/tasks` | GET | User | List tasks |
|
|
| `POST /api/brain/tasks` | POST | User | Create a task |
|
|
| `GET /api/brain/tasks/:id` | GET | User | Get a task |
|
|
| `PATCH /api/brain/tasks/:id` | PATCH | User | Update a task |
|
|
| `DELETE /api/brain/tasks/:id` | DELETE | User | Delete a task |
|
|
|
|
### Memory / Preferences
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ----------------------------- | ------ | ---- | -------------------- |
|
|
| `GET /api/memory/preferences` | GET | User | Get user preferences |
|
|
| `PUT /api/memory/preferences` | PUT | User | Upsert a preference |
|
|
|
|
### MCP Server (Gateway-side)
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ----------- | ------ | --------------------------------------------- | ----------------------------- |
|
|
| `POST /mcp` | POST | User (session cookie or Authorization header) | MCP streamable HTTP transport |
|
|
| `GET /mcp` | GET | User | MCP SSE stream reconnect |
|
|
|
|
### Skills
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------ | ------ | ----- | --------------------- |
|
|
| `GET /api/skills` | GET | User | List installed skills |
|
|
| `POST /api/skills` | POST | Admin | Install a skill |
|
|
| `PATCH /api/skills/:id` | PATCH | Admin | Update a skill |
|
|
| `DELETE /api/skills/:id` | DELETE | Admin | Remove a skill |
|
|
|
|
### Coord (Mission Coordination)
|
|
|
|
| Endpoint | Method | Auth | Description |
|
|
| ------------------------------- | ------ | ---- | ---------------- |
|
|
| `GET /api/coord/missions` | GET | User | List missions |
|
|
| `POST /api/coord/missions` | POST | User | Create a mission |
|
|
| `GET /api/coord/missions/:id` | GET | User | Get a mission |
|
|
| `PATCH /api/coord/missions/:id` | PATCH | User | Update a mission |
|
|
|
|
### Observability
|
|
|
|
OpenTelemetry traces are exported to the OTEL collector (`OTEL_EXPORTER_OTLP_ENDPOINT`).
|
|
View traces in Jaeger at `http://localhost:16686`.
|
|
|
|
Tracing is initialized before NestJS bootstrap in
|
|
`apps/gateway/src/tracing.ts`. The import order in `apps/gateway/src/main.ts`
|
|
is intentional: `import './tracing.js'` must come before any NestJS imports.
|