stack/docs/guides/dev-guide.md

# Mosaic Stack — Developer Guide

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Local Development Setup](#local-development-setup)
3. [Building and Testing](#building-and-testing)
4. [Adding New Agent Tools](#adding-new-agent-tools)
5. [Adding New MCP Tools](#adding-new-mcp-tools)
6. [Database Schema and Migrations](#database-schema-and-migrations)
7. [API Endpoint Reference](#api-endpoint-reference)

---

## Architecture Overview

Mosaic Stack is a TypeScript monorepo managed with **pnpm workspaces** and
**Turborepo**.

```
mosaic-mono-v1/
├── apps/
│   ├── gateway/          # NestJS + Fastify API server
│   └── web/              # Next.js 16 + React 19 web dashboard
├── packages/
│   ├── agent/            # Agent session types (shared)
│   ├── auth/             # BetterAuth configuration
│   ├── brain/            # Structured data layer (projects, tasks, missions)
│   ├── cli/              # mosaic CLI and TUI (Ink)
│   ├── coord/            # Mission coordination engine
│   ├── db/               # Drizzle ORM schema, migrations, client
│   ├── design-tokens/    # Shared design system tokens
│   ├── log/              # Agent log ingestion and tiering
│   ├── memory/           # Preference and insight storage
│   ├── mosaic/           # Install wizard and bootstrap utilities
│   ├── prdy/             # PRD wizard CLI
│   ├── quality-rails/    # Code quality scaffolder CLI
│   ├── queue/            # Valkey-backed task queue
│   └── types/            # Shared TypeScript types
├── docker/               # Dockerfile(s) for containerized deployment
├── infra/                # Infra config (OTEL collector, pg-init scripts)
├── docker-compose.yml    # Local services (Postgres, Valkey, OTEL, Jaeger)
└── CLAUDE.md             # Project conventions for AI coding agents
```

### Key Technology Choices

| Concern           | Technology                               |
| ----------------- | ---------------------------------------- |
| API framework     | NestJS with Fastify adapter              |
| Web framework     | Next.js 16 (App Router), React 19        |
| ORM               | Drizzle ORM                              |
| Database          | PostgreSQL 17 + pgvector extension       |
| Auth              | BetterAuth                               |
| Agent harness     | Pi SDK (`@mariozechner/pi-coding-agent`) |
| Queue             | Valkey 8 (Redis-compatible)              |
| Build             | pnpm workspaces + Turborepo              |
| CI                | Woodpecker CI                            |
| Observability     | OpenTelemetry → Jaeger                   |
| Module resolution | NodeNext (ESM everywhere)                |

### Module System

All packages use `"type": "module"` and NodeNext resolution. Import paths must
include the `.js` extension even when the source file is `.ts`.

NestJS `@Inject()` decorators must be used explicitly because `tsx`/`esbuild`
does not support `emitDecoratorMetadata`.

---

## Local Development Setup

### Prerequisites

- Node.js 20+
- pnpm 9+
- Docker and Docker Compose

### 1. Clone and Install Dependencies

```bash
git clone <repo-url> mosaic-mono-v1
cd mosaic-mono-v1
pnpm install
```

### 2. Start Infrastructure Services

```bash
docker compose up -d
```

This starts:

| Service                  | Port           | Description          |
| ------------------------ | -------------- | -------------------- |
| PostgreSQL 17 + pgvector | `5433` (host)  | Primary database     |
| Valkey 8                 | `6380` (host)  | Queue and cache      |
| OpenTelemetry Collector  | `4317`, `4318` | OTEL gRPC and HTTP   |
| Jaeger                   | `16686`        | Distributed trace UI |

### 3. Configure Environment

Create a `.env` file in the monorepo root:

```env
# Database (matches docker-compose defaults)
DATABASE_URL=postgresql://mosaic:mosaic@localhost:5433/mosaic

# Auth (required — generate a random 32+ char string)
BETTER_AUTH_SECRET=change-me-to-a-random-secret

# Gateway
GATEWAY_PORT=14242
GATEWAY_CORS_ORIGIN=http://localhost:3000

# Web
NEXT_PUBLIC_GATEWAY_URL=http://localhost:14242

# Optional: Ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODELS=llama3.2
```

The gateway loads `.env` from the monorepo root via `dotenv` at startup
(`apps/gateway/src/main.ts`).

### 4. Push the Database Schema

```bash
pnpm --filter @mosaicstack/db db:push
```

This applies the Drizzle schema directly to the database (development only; use
migrations in production).

### 5. Start the Gateway

```bash
pnpm --filter @mosaicstack/gateway exec tsx src/main.ts
```

The gateway starts on port `14242` by default.

### 6. Start the Web App

```bash
pnpm --filter @mosaicstack/web dev
```

The web app starts on port `3000` by default.

---

## Building and Testing

### TypeScript Typecheck

```bash
pnpm typecheck
```

Runs `tsc --noEmit` across all packages in dependency order via Turborepo.

### Lint

```bash
pnpm lint
```

Runs ESLint across all packages. Config is in `eslint.config.mjs` at the root.

### Format Check

```bash
pnpm format:check
```

Runs Prettier in check mode. To auto-fix:

```bash
pnpm format
```

### Tests

```bash
pnpm test
```

Runs Vitest across all packages. The workspace config is at
`vitest.workspace.ts`.

### Build

```bash
pnpm build
```

Builds all packages and apps in dependency order.

### Pre-Push Gates (MANDATORY)

All three must pass before any push:

```bash
pnpm format:check && pnpm typecheck && pnpm lint
```

A pre-push hook enforces this mechanically.

---

## Adding New Agent Tools

Agent tools are Pi SDK `ToolDefinition` objects registered in
`apps/gateway/src/agent/agent.service.ts`.

### 1. Create a Tool Factory File

Add a new file in `apps/gateway/src/agent/tools/`:

```typescript
// apps/gateway/src/agent/tools/my-tools.ts
import { Type } from '@sinclair/typebox';
import type { ToolDefinition } from '@mariozechner/pi-coding-agent';

export function createMyTools(): ToolDefinition[] {
  const myTool: ToolDefinition = {
    name: 'my_tool_name',
    label: 'Human Readable Label',
    description: 'What this tool does.',
    parameters: Type.Object({
      input: Type.String({ description: 'The input parameter' }),
    }),
    async execute(_toolCallId, params) {
      const { input } = params as { input: string };
      const result = `Processed: ${input}`;
      return {
        content: [{ type: 'text' as const, text: result }],
        details: undefined,
      };
    },
  };

  return [myTool];
}
```

### 2. Register the Tools in AgentService

In `apps/gateway/src/agent/agent.service.ts`, import and call your factory
alongside the existing tool registrations:

```typescript
import { createMyTools } from './tools/my-tools.js';

// Inside the session creation logic where tools are assembled:
const tools: ToolDefinition[] = [
  ...createBrainTools(this.brain),
  ...createCoordTools(this.coordService),
  ...createMemoryTools(this.memory, this.embeddingService),
  ...createFileTools(sandboxDir),
  ...createGitTools(sandboxDir),
  ...createShellTools(sandboxDir),
  ...createWebTools(),
  ...createMyTools(), // Add this line
  ...mcpTools,
  ...skillTools,
];
```

### 3. Export from the Tools Index

Add an export to `apps/gateway/src/agent/tools/index.ts`:

```typescript
export { createMyTools } from './my-tools.js';
```

### 4. Typecheck and Test

```bash
pnpm typecheck
pnpm test
```

---

## Adding New MCP Tools

Mosaic connects to external MCP servers via `McpClientService`. To expose tools
from a new MCP server:

### 1. Run an MCP Server

Implement a standard MCP server that exposes tools via the streamable HTTP
transport or SSE transport. The server must accept connections at a `/mcp`
endpoint.

### 2. Configure `MCP_SERVERS`

In your `.env`:

```env
MCP_SERVERS='[{"name":"my-server","url":"http://localhost:3001/mcp"}]'
```

With authentication:

```env
MCP_SERVERS='[{"name":"secure-server","url":"http://my-server/mcp","headers":{"Authorization":"Bearer token"}}]'
```

### 3. Restart the Gateway

On startup, `McpClientService` (`apps/gateway/src/mcp-client/mcp-client.service.ts`)
connects to each configured server, calls `tools/list`, and bridges the results
to Pi SDK `ToolDefinition` format. These tools become available in all new agent
sessions.

### Tool Naming

Bridged MCP tool names are taken directly from the MCP server's tool manifest.
Ensure names do not conflict with built-in tools (check
`apps/gateway/src/agent/tools/`).

---

## Database Schema and Migrations

The schema lives in a single file:
`packages/db/src/schema.ts`

### Schema Overview

| Table                | Purpose                                           |
| -------------------- | ------------------------------------------------- |
| `users`              | User accounts (BetterAuth-compatible)             |
| `sessions`           | Auth sessions                                     |
| `accounts`           | OAuth accounts                                    |
| `verifications`      | Email verification tokens                         |
| `projects`           | Project records                                   |
| `missions`           | Mission records (linked to projects)              |
| `tasks`              | Task records (linked to projects and/or missions) |
| `conversations`      | Chat conversation metadata                        |
| `messages`           | Individual chat messages                          |
| `preferences`        | Per-user key-value preference store               |
| `insights`           | Vector-embedded memory insights                   |
| `agent_logs`         | Agent interaction logs (hot/warm/cold tiers)      |
| `skills`             | Installed agent skills                            |
| `summarization_jobs` | Log summarization job tracking                    |

The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.

### Development: Push Schema

Apply schema changes directly to the dev database (no migration files created):

```bash
pnpm --filter @mosaicstack/db db:push
```

### Generating Migrations

For production-safe, versioned changes:

```bash
pnpm --filter @mosaicstack/db db:generate
```

This creates a new SQL migration file in `packages/db/drizzle/`.

### Running Migrations

```bash
pnpm --filter @mosaicstack/db db:migrate
```

### Drizzle Config

Config is at `packages/db/drizzle.config.ts`. The schema file path and output
directory are defined there.

### Adding a New Table

1. Add the table definition to `packages/db/src/schema.ts`.
2. Export it from `packages/db/src/index.ts`.
3. Run `pnpm --filter @mosaicstack/db db:push` (dev) or
   `pnpm --filter @mosaicstack/db db:generate && pnpm --filter @mosaicstack/db db:migrate`
   (production).

---

## API Endpoint Reference

All endpoints are served by the gateway at `http://localhost:14242` by default.

### Authentication

Authentication uses BetterAuth session cookies. The auth handler is mounted at
`/api/auth/*` via a Fastify low-level hook in
`apps/gateway/src/auth/auth.controller.ts`.

| Endpoint                  | Method | Description                      |
| ------------------------- | ------ | -------------------------------- |
| `/api/auth/sign-in/email` | POST   | Sign in with email/password      |
| `/api/auth/sign-up/email` | POST   | Register a new account           |
| `/api/auth/sign-out`      | POST   | Sign out (clears session cookie) |
| `/api/auth/get-session`   | GET    | Returns the current session      |

### Chat

WebSocket namespace `/chat` (Socket.IO). Authentication via session cookie.

Events sent by the client:

| Event     | Payload                                             | Description    |
| --------- | --------------------------------------------------- | -------------- |
| `message` | `{ content, conversationId?, provider?, modelId? }` | Send a message |

Events emitted by the server:

| Event   | Payload                     | Description            |
| ------- | --------------------------- | ---------------------- |
| `token` | `{ token, conversationId }` | Streaming token        |
| `end`   | `{ conversationId }`        | Stream complete        |
| `error` | `{ message }`               | Error during streaming |

HTTP endpoints (`apps/gateway/src/chat/chat.controller.ts`):

| Endpoint                               | Method | Auth | Description                     |
| -------------------------------------- | ------ | ---- | ------------------------------- |
| `/api/chat/conversations`              | GET    | User | List conversations              |
| `/api/chat/conversations/:id/messages` | GET    | User | Get messages for a conversation |

### Admin

All admin endpoints require `role = admin`.

| Endpoint                          | Method | Description          |
| --------------------------------- | ------ | -------------------- |
| `GET /api/admin/users`            | GET    | List all users       |
| `GET /api/admin/users/:id`        | GET    | Get a single user    |
| `POST /api/admin/users`           | POST   | Create a user        |
| `PATCH /api/admin/users/:id/role` | PATCH  | Update user role     |
| `POST /api/admin/users/:id/ban`   | POST   | Ban a user           |
| `POST /api/admin/users/:id/unban` | POST   | Unban a user         |
| `DELETE /api/admin/users/:id`     | DELETE | Delete a user        |
| `GET /api/admin/health`           | GET    | System health status |

### Agent / Providers

| Endpoint                             | Method | Auth | Description                         |
| ------------------------------------ | ------ | ---- | ----------------------------------- |
| `GET /api/agent/providers`           | GET    | User | List all providers and their models |
| `GET /api/agent/providers/models`    | GET    | User | List available models               |
| `POST /api/agent/providers/:id/test` | POST   | User | Test provider connectivity          |

### Projects / Brain

| Endpoint                         | Method | Auth | Description      |
| -------------------------------- | ------ | ---- | ---------------- |
| `GET /api/brain/projects`        | GET    | User | List projects    |
| `POST /api/brain/projects`       | POST   | User | Create a project |
| `GET /api/brain/projects/:id`    | GET    | User | Get a project    |
| `PATCH /api/brain/projects/:id`  | PATCH  | User | Update a project |
| `DELETE /api/brain/projects/:id` | DELETE | User | Delete a project |
| `GET /api/brain/tasks`           | GET    | User | List tasks       |
| `POST /api/brain/tasks`          | POST   | User | Create a task    |
| `GET /api/brain/tasks/:id`       | GET    | User | Get a task       |
| `PATCH /api/brain/tasks/:id`     | PATCH  | User | Update a task    |
| `DELETE /api/brain/tasks/:id`    | DELETE | User | Delete a task    |

### Memory / Preferences

| Endpoint                      | Method | Auth | Description          |
| ----------------------------- | ------ | ---- | -------------------- |
| `GET /api/memory/preferences` | GET    | User | Get user preferences |
| `PUT /api/memory/preferences` | PUT    | User | Upsert a preference  |

### MCP Server (Gateway-side)

| Endpoint    | Method | Auth                                          | Description                   |
| ----------- | ------ | --------------------------------------------- | ----------------------------- |
| `POST /mcp` | POST   | User (session cookie or Authorization header) | MCP streamable HTTP transport |
| `GET /mcp`  | GET    | User                                          | MCP SSE stream reconnect      |

### Skills

| Endpoint                 | Method | Auth  | Description           |
| ------------------------ | ------ | ----- | --------------------- |
| `GET /api/skills`        | GET    | User  | List installed skills |
| `POST /api/skills`       | POST   | Admin | Install a skill       |
| `PATCH /api/skills/:id`  | PATCH  | Admin | Update a skill        |
| `DELETE /api/skills/:id` | DELETE | Admin | Remove a skill        |

### Coord (Mission Coordination)

| Endpoint                        | Method | Auth | Description      |
| ------------------------------- | ------ | ---- | ---------------- |
| `GET /api/coord/missions`       | GET    | User | List missions    |
| `POST /api/coord/missions`      | POST   | User | Create a mission |
| `GET /api/coord/missions/:id`   | GET    | User | Get a mission    |
| `PATCH /api/coord/missions/:id` | PATCH  | User | Update a mission |

### Observability

OpenTelemetry traces are exported to the OTEL collector (`OTEL_EXPORTER_OTLP_ENDPOINT`).
View traces in Jaeger at `http://localhost:16686`.

Tracing is initialized before NestJS bootstrap in
`apps/gateway/src/tracing.ts`. The import order in `apps/gateway/src/main.ts`
is intentional: `import './tracing.js'` must come before any NestJS imports.