mosaicstack/stack
21
0
Fork 1
Code Issues 338 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

chore: consolidate new foundation and archive v1 (#1495)

Browse Source
This commit is contained in:
jason.woltje
2026-09-07 12:32:57 -05:00
parent 9a5fbdbda7 5d27700026
commit 127a54fdff
3511 changed files with 727899 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
v1/docs/guides/dev-guide.md
+617
View File
@@ -0,0 +1,617 @@
---
kind: guide
status: active
---
# Mosaic Stack — Developer Guide
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Local Development Setup](#local-development-setup)
3. [Building and Testing](#building-and-testing)
4. [Adding New Agent Tools](#adding-new-agent-tools)
5. [Adding New MCP Tools](#adding-new-mcp-tools)
6. [Database Schema and Migrations](#database-schema-and-migrations)
7. [Claude Code Skill Bridge](#claude-code-skill-bridge)
8. [Pi Persistent Goal Extension](#pi-persistent-goal-extension)
9. [API Endpoint Reference](#api-endpoint-reference)
10. [Local Fleet Canary](./fleet-local-canary.md)
---
## Architecture Overview
Mosaic Stack is a TypeScript monorepo managed with **pnpm workspaces** and
**Turborepo**.
```
mosaic-mono-v1/
├── apps/
│ ├── gateway/ # NestJS + Fastify API server
│ └── web/ # Next.js 16 + React 19 web dashboard
├── packages/
│ ├── agent/ # Agent session types (shared)
│ ├── auth/ # BetterAuth configuration
│ ├── brain/ # Structured data layer (projects, tasks, missions)
│ ├── cli/ # mosaic CLI and TUI (Ink)
│ ├── coord/ # Mission coordination engine
│ ├── db/ # Drizzle ORM schema, migrations, client
│ ├── design-tokens/ # Shared design system tokens
│ ├── log/ # Agent log ingestion and tiering
│ ├── memory/ # Preference and insight storage
│ ├── mosaic/ # Install wizard and bootstrap utilities
│ ├── prdy/ # PRD wizard CLI
│ ├── quality-rails/ # Code quality scaffolder CLI
│ ├── queue/ # Valkey-backed task queue
│ └── types/ # Shared TypeScript types
├── docker/ # Dockerfile(s) for containerized deployment
├── infra/ # Infrastructure configuration (for example, OTEL collector)
├── docker-compose.yml # Local services (Postgres, Valkey, OTEL, Jaeger)
└── CLAUDE.md # Project conventions for AI coding agents
```
### Key Technology Choices
| Concern | Technology |
| ----------------- | ---------------------------------------- |
| API framework | NestJS with Fastify adapter |
| Web framework | Next.js 16 (App Router), React 19 |
| ORM | Drizzle ORM |
| Database | PostgreSQL 17 + pgvector extension |
| Auth | BetterAuth |
| Agent harness | Pi SDK (`@mariozechner/pi-coding-agent`) |
| Queue | Valkey 8 (Redis-compatible) |
| Build | pnpm workspaces + Turborepo |
| CI | Woodpecker CI |
| Observability | OpenTelemetry → Jaeger |
| Module resolution | NodeNext (ESM everywhere) |
### Module System
All packages use `"type": "module"` and NodeNext resolution. Import paths must
include the `.js` extension even when the source file is `.ts`.
NestJS `@Inject()` decorators must be used explicitly because `tsx`/`esbuild`
does not support `emitDecoratorMetadata`.
---
## Local Development Setup
### Prerequisites
- Node.js 20+
- pnpm 9+
- Docker and Docker Compose
### 1. Clone and Install Dependencies
```bash
git clone <repo-url> mosaic-mono-v1
cd mosaic-mono-v1
pnpm install
```
### 2. Use the local PGlite tier
The supported local tier is in-process PGlite and requires no PostgreSQL service. Leave
`DATABASE_URL` unset for this route. Its default local configuration uses PGlite and performs no
external database probe.
If a local queue service is useful, start only that non-PostgreSQL service:
```bash
docker compose up -d valkey
```
Do not use the current Compose PostgreSQL service: it mounts legacy `infra/pg-init` SQL and is
not qualified for KBN-101. Start OTEL Collector or Jaeger individually only when needed and
without starting PostgreSQL.
### 3. Gateway/Web local process (held)
Do not start the current Gateway or web process as a local PGlite route. Gateway first loads the
daemon configuration and then project environment files without a tier guard; a pre-existing
`DATABASE_URL` can select PostgreSQL, where current startup still reaches runtime DDL/migrations.
Creating a root `.env` that omits `DATABASE_URL` does not make this safe, so neither a local
credential file nor a web environment file is a current developer procedure.
PGlite remains the supported in-process data-layer implementation, and the optional Valkey command
above remains safe because it does not start PostgreSQL. A safe Gateway/Web local procedure is held
until KBN-101-02 rejects a daemon, inherited, root, or app-local PostgreSQL DSN and any non-local
tier before connection or DDL; KBN-101-05 then supplies the production renderer/Vault process-exec
or `LoadCredential` boundary.
### Held future procedure
PostgreSQL local and federated deployment are held until KBN-101-00 (external bootstrap),
KBN-101-03 (runner), and KBN-101-05 (renderer-backed deployment) land. The following is the
**held, non-operative future activation order with no current command authority**:
external bootstrap → TLS/roles → `mosaic-db-migrator --run` →
`mosaic-db-migrator --verify` → Gateway/Compose readiness.
Neither current Compose nor this development guide authorizes PostgreSQL initialization SQL,
manual DDL, or a pre-runner start.
### 5. Gateway/Web start (held)
No Gateway/Web start command is currently authorized for the local PGlite route. Do not use root
`pnpm dev` as a workaround: it additionally starts configured integrations and cannot establish the
required local-tier/DSN isolation. Resume this section only after KBN-101-02 provides its
fail-closed local-startup evidence.
---
## Building and Testing
### TypeScript Typecheck
```bash
pnpm typecheck
```
Runs `tsc --noEmit` across all packages in dependency order via Turborepo.
### Lint
```bash
pnpm lint
```
Runs ESLint across all packages. Config is in `eslint.config.mjs` at the root.
### Format Check
```bash
pnpm format:check
```
Runs Prettier in check mode. To auto-fix:
```bash
pnpm format
```
### Tests
```bash
pnpm test
```
Runs Vitest across all packages. The workspace config is at
`vitest.workspace.ts`.
### Build
```bash
pnpm build
```
Builds all packages and apps in dependency order.
### Pre-Push Gates (MANDATORY)
All three must pass before any push:
```bash
pnpm format:check && pnpm typecheck && pnpm lint
```
A pre-push hook enforces this mechanically.
### CI Publish Channels
Woodpecker `.woodpecker/publish.yml` keeps stable and integration-line artifacts separate:
| Source | npm packages | Gateway image |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `main` push/manual or release tag | committed package versions published to Gitea npm without changing the dist-tag workflow | `gateway:sha-<short>` plus `gateway:latest` on `main`, and the release tag on tag events |
| `next` push/manual | CI-computed prereleases, `<target-stable>-next.<CI_PIPELINE_NUMBER>`, published with `npm publish --tag next` | `gateway:sha-<short>` only |
`next` never publishes npm `latest` or Docker `latest`. The next npm publish step verifies that `@mosaicstack/mosaic@next` resolves to the computed prerelease before the pipeline can pass.
### E2E Gate (#1445, P6)
Trunk publish pipelines run a headless Playwright suite (`e2e` step) before any image publishes: the built gateway `dist` boots on a throwaway embedded PGlite database (isolated via a fresh `HOME`), serves the built SPA bundle through `WEB_DIST_DIR` — the same serving path the gateway image ships — and the suite runs against it inside the pinned `mcr.microsoft.com/playwright` image. `E2E_REQUIRE_SEEDED_AUTH=1` makes login failures hard failures (the skip-when-login-fails guards are a live-environment affordance only). Both image build steps depend on this gate.
Reproduce locally (Ubuntu-based environments; Fedora's headless-shell rendering is broken):
```bash
pnpm build
BETTER_AUTH_SECRET="$(head -c 32 /dev/urandom | base64)" GATEWAY_PORT=14242 \
WEB_DIST_DIR="$PWD/apps/web/dist" HOME="$(mktemp -d)" node apps/gateway/dist/main.js &
E2E_REQUIRE_SEEDED_AUTH=1 PLAYWRIGHT_BASE_URL=http://localhost:14242 \
pnpm --filter @mosaicstack/web exec playwright test
```
---
## Adding New Agent Tools
Agent tools are Pi SDK `ToolDefinition` objects registered in
`apps/gateway/src/agent/agent.service.ts`.
### 1. Create a Tool Factory File
Add a new file in `apps/gateway/src/agent/tools/`:
```typescript
// apps/gateway/src/agent/tools/my-tools.ts
import { Type } from '@sinclair/typebox';
import type { ToolDefinition } from '@mariozechner/pi-coding-agent';
export function createMyTools(): ToolDefinition[] {
const myTool: ToolDefinition = {
name: 'my_tool_name',
label: 'Human Readable Label',
description: 'What this tool does.',
parameters: Type.Object({
input: Type.String({ description: 'The input parameter' }),
}),
async execute(_toolCallId, params) {
const { input } = params as { input: string };
const result = `Processed: ${input}`;
return {
content: [{ type: 'text' as const, text: result }],
details: undefined,
};
},
};
return [myTool];
}
```
### 2. Register the Tools in AgentService
In `apps/gateway/src/agent/agent.service.ts`, import and call your factory
alongside the existing tool registrations:
```typescript
import { createMyTools } from './tools/my-tools.js';
// Inside the session creation logic where tools are assembled:
const tools: ToolDefinition[] = [
...createBrainTools(this.brain),
...createCoordTools(this.coordService),
...createMemoryTools(this.memory, this.embeddingService),
...createFileTools(sandboxDir),
...createGitTools(sandboxDir),
...createShellTools(sandboxDir),
...createWebTools(),
...createMyTools(), // Add this line
...mcpTools,
...skillTools,
];
```
### 3. Export from the Tools Index
Add an export to `apps/gateway/src/agent/tools/index.ts`:
```typescript
export { createMyTools } from './my-tools.js';
```
### 4. Typecheck and Test
```bash
pnpm typecheck
pnpm test
```
---
## Adding New MCP Tools
Mosaic connects to external MCP servers via `McpClientService`. To expose tools
from a new MCP server:
### 1. Run an MCP Server
Implement a standard MCP server that exposes tools via the streamable HTTP
transport or SSE transport. The server must accept connections at a `/mcp`
endpoint.
### 2. Gateway MCP configuration (held)
Do not configure MCP endpoint credentials, write them to a local environment file, or restart the
Gateway from this guide. Gateway/Web startup is held until KBN-101-02 supplies fail-closed
local-tier/DSN isolation and KBN-101-05 supplies the renderer/Vault process-exec or
`LoadCredential` secret-consumer interface. The future authenticated MCP route requires verified
HTTPS and certificate validation; plaintext bearer-token examples are forbidden.
### Tool Naming
Bridged MCP tool names are taken directly from the MCP server's tool manifest.
Ensure names do not conflict with built-in tools (check
`apps/gateway/src/agent/tools/`).
---
## Database Schema and Migrations
The schema lives in a single file:
`packages/db/src/schema.ts`
### Schema Overview
| Table | Purpose |
| -------------------- | ------------------------------------------------- |
| `users` | User accounts (BetterAuth-compatible) |
| `sessions` | Auth sessions |
| `accounts` | OAuth accounts |
| `verifications` | Email verification tokens |
| `projects` | Project records |
| `missions` | Mission records (linked to projects) |
| `tasks` | Task records (linked to projects and/or missions) |
| `conversations` | Chat conversation metadata |
| `messages` | Individual chat messages |
| `preferences` | Per-user key-value preference store |
| `insights` | Vector-embedded memory insights |
| `agent_logs` | Agent interaction logs (hot/warm/cold tiers) |
| `skills` | Installed agent skills |
| `summarization_jobs` | Log summarization job tracking |
The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.
### PostgreSQL schema work (held)
Do not prepare or run a PostgreSQL target from this branch. The sole runner, bootstrap, and
renderer are future KBN-101 artifacts, not current commands. When KBN-101-00/-03/-05 land, the
owned activation documentation will require external bootstrap → TLS/roles → runner `--run` →
runner `--verify` → Gateway/Compose readiness.
### Generating migration artifacts
`pnpm --filter @mosaicstack/db db:generate` is an offline artifact-generation command. It does
not authorize connecting to or initializing PostgreSQL. A future reviewed PostgreSQL procedure
will determine when its output is applied.
### Drizzle Config
Config is at `packages/db/drizzle.config.ts`. The schema file path and output directory are
defined there.
### Adding a New Table
1. Add the table definition to `packages/db/src/schema.ts`.
2. Export it from `packages/db/src/index.ts`.
3. Generate the offline artifact with `pnpm --filter @mosaicstack/db db:generate`.
4. Do not apply it to PostgreSQL until the future KBN-101 activation artifacts and their owned
procedure are available. Direct schema push is not a production-like workflow.
---
## Claude Code Skill Bridge
The framework's canonical skill root is `~/.config/mosaic/skills/`; Claude Code
requires registrations under `~/.claude/skills/`. The implementation in
`packages/mosaic/src/commands/skill.ts` owns only direct-child symlinks whose
resolved target remains inside the canonical root.
Security invariants:
1. Validate the user-supplied name before filesystem access against
`[A-Za-z0-9][A-Za-z0-9._-]*`. Separators, control characters, whitespace,
`..`, absolute paths, and leading `-` are invalid; filesystem-derived invalid
names are escaped before terminal output.
2. Never replace a real file, directory, foreign symlink, or live misdirected
symlink in the Claude skill directory.
3. Repair a dangling link only when its lexical target is inside the canonical
Mosaic skills root.
4. Unregister only a symlink pointing inside that root.
5. Enumerate canonical directories at runtime; never hardcode framework skill
names.
`finalizeStage` reconciles after wizard/framework synchronization, and
`runFrameworkReseed` reconciles after the sync-only `mosaic update` path. A
foreign conflict is reported but does not prevent unrelated canonical skills
from registering. Filesystem tests use injected temporary roots in
`skill.spec.ts`, `finalize-skills.spec.ts`, and `update-checker.reseed.spec.ts`.
M1 intentionally manages Claude Code only. Pi's Mosaic launcher can discover the
canonical root directly. Codex still relies on the existing full skill-sync
linker and needs separate parity analysis before this lifecycle API is extended.
## Pi Persistent Goal Extension
The source of the Mosaic-owned Pi goal controller is:
```text
packages/mosaic/framework/runtime/pi/goal-extension.ts
```
The framework manifest classifies `runtime/**` as framework-owned. Both the bash installer and the
TypeScript file adapter therefore deploy the same reviewed source to:
```text
$MOSAIC_HOME/runtime/pi/goal-extension.ts
# default: ~/.config/mosaic/runtime/pi/goal-extension.ts
```
Do not copy or link this extension into `~/.pi/agent/extensions/`. The launcher function
`discoverPiExtensionArgs()` emits the core `mosaic-extension.ts` first and the optional
`goal-extension.ts` second, preserving compatibility with an older installed framework that does
not have the goal file yet.
### Lifecycle design
| Pi API | Goal-controller responsibility |
| ------------------------------ | --------------------------------------------------------------------------------- |
| `registerCommand('goal')` | Set, inspect, pause, resume, or cancel one branch-specific goal |
| `registerTool(...)` | Record a terminating structured progress report with evidence |
| `context` | Inject the active goal contract before every provider request |
| `turn_end` | Record every turn, reject mixed final reports, and enforce the turn bound |
| `agent_settled` | Start one deduplicated continuation only after Pi has no retry/compact/queue work |
| `session_compact` | Record the compact check, reset provisional verification, and defer idle work |
| `session_start`/`session_tree` | Rebuild state from custom entries on the active branch |
| `session_shutdown` | Invalidate deferred callbacks and clear UI state |
State is appended as `mosaic-goal-state` custom entries, which do not enter model context. The
`context` hook creates a fresh hidden `mosaic-goal-context` message for each request instead of
trusting compaction summaries. The `mosaic_goal_report` result uses `terminate: true`; when it is the
sole final tool call, Pi avoids an unnecessary model response before the controller decides whether
to verify, continue, or stop.
Before state is appended or displayed, the controller applies bounded credential-pattern redaction
to the goal statement, report summary/evidence/next step, and stop reason. Fingerprints are computed
over redacted report content. Pi session entries are append-only, so a credential-bearing legacy
entry cannot honestly be erased by the extension: restoration fails closed, emits a warning, and
requires removal of the affected session before setting a new goal. This is defense-in-depth rather
than a secret-storage contract, and it does not rewrite Pi's separate model-message/tool-call
history. Goal prompts tell the agent not to submit credentials or raw sensitive output, and tests use
canaries to prove known forms do not reach new custom entries, status text, context, or tool details
while ordinary typed fields such as `token: string` remain intact.
Completion remains evidence-gated but semantic: two consecutive `achieved` reports are required,
and the second run is explicitly a verification pass. This avoids an extra judge-model request after
every turn. Deterministic validator commands are intentionally not accepted as `/goal` input in this
slice, so never describe this mechanism as proof of arbitrary natural-language completion.
### Tests and local smoke workflow
```bash
pnpm --filter @mosaicstack/mosaic exec vitest run \
src/runtime/pi-goal-extension.spec.ts \
src/commands/launch.spec.ts \
src/config/file-adapter.test.ts
bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh
```
For an additive local smoke test without reseeding unrelated live framework files:
```bash
install -D -m 0644 \
packages/mosaic/framework/runtime/pi/goal-extension.ts \
~/.config/mosaic/runtime/pi/goal-extension.ts
pi --extension ~/.config/mosaic/runtime/pi/goal-extension.ts
```
Use `/goal help`, `/goal set ...`, and `/goal status` in that test session. A released framework
sync installs the file, and a released Mosaic CLI loads it automatically through `mosaic pi`.
## API Endpoint Reference
All endpoints are served by the gateway at `http://localhost:14242` by default.
### Authentication
Authentication uses BetterAuth session cookies. The auth handler is mounted at
`/api/auth/*` via a Fastify low-level hook in
`apps/gateway/src/auth/auth.controller.ts`.
| Endpoint | Method | Description |
| ------------------------- | ------ | -------------------------------- |
| `/api/auth/sign-in/email` | POST | Sign in with email/password |
| `/api/auth/sign-up/email` | POST | Register a new account |
| `/api/auth/sign-out` | POST | Sign out (clears session cookie) |
| `/api/auth/get-session` | GET | Returns the current session |
### Chat
WebSocket namespace `/chat` (Socket.IO). Authentication via session cookie.
Events sent by the client:
| Event | Payload | Description |
| --------- | --------------------------------------------------- | -------------- |
| `message` | `{ content, conversationId?, provider?, modelId? }` | Send a message |
Events emitted by the server:
| Event | Payload | Description |
| ------- | --------------------------- | ---------------------- |
| `token` | `{ token, conversationId }` | Streaming token |
| `end` | `{ conversationId }` | Stream complete |
| `error` | `{ message }` | Error during streaming |
HTTP endpoints (`apps/gateway/src/chat/chat.controller.ts`):
| Endpoint | Method | Auth | Description |
| -------------------------------------- | ------ | ---- | ------------------------------- |
| `/api/chat/conversations` | GET | User | List conversations |
| `/api/chat/conversations/:id/messages` | GET | User | Get messages for a conversation |
### Admin
All admin endpoints require `role = admin`.
| Endpoint | Method | Description |
| --------------------------------- | ------ | -------------------- |
| `GET /api/admin/users` | GET | List all users |
| `GET /api/admin/users/:id` | GET | Get a single user |
| `POST /api/admin/users` | POST | Create a user |
| `PATCH /api/admin/users/:id/role` | PATCH | Update user role |
| `POST /api/admin/users/:id/ban` | POST | Ban a user |
| `POST /api/admin/users/:id/unban` | POST | Unban a user |
| `DELETE /api/admin/users/:id` | DELETE | Delete a user |
| `GET /api/admin/health` | GET | System health status |
### Agent / Providers
| Endpoint | Method | Auth | Description |
| ------------------------------------ | ------ | ---- | ----------------------------------- |
| `GET /api/agent/providers` | GET | User | List all providers and their models |
| `GET /api/agent/providers/models` | GET | User | List available models |
| `POST /api/agent/providers/:id/test` | POST | User | Test provider connectivity |
### Projects / Brain
| Endpoint | Method | Auth | Description |
| -------------------------------- | ------ | ---- | ---------------- |
| `GET /api/brain/projects` | GET | User | List projects |
| `POST /api/brain/projects` | POST | User | Create a project |
| `GET /api/brain/projects/:id` | GET | User | Get a project |
| `PATCH /api/brain/projects/:id` | PATCH | User | Update a project |
| `DELETE /api/brain/projects/:id` | DELETE | User | Delete a project |
| `GET /api/brain/tasks` | GET | User | List tasks |
| `POST /api/brain/tasks` | POST | User | Create a task |
| `GET /api/brain/tasks/:id` | GET | User | Get a task |
| `PATCH /api/brain/tasks/:id` | PATCH | User | Update a task |
| `DELETE /api/brain/tasks/:id` | DELETE | User | Delete a task |
### Memory / Preferences
| Endpoint | Method | Auth | Description |
| ----------------------------- | ------ | ---- | -------------------- |
| `GET /api/memory/preferences` | GET | User | Get user preferences |
| `PUT /api/memory/preferences` | PUT | User | Upsert a preference |
### MCP Server (Gateway-side)
| Endpoint | Method | Auth | Description |
| ----------- | ------ | --------------------------------------------- | ----------------------------- |
| `POST /mcp` | POST | User (session cookie or Authorization header) | MCP streamable HTTP transport |
| `GET /mcp` | GET | User | MCP SSE stream reconnect |
### Skills
| Endpoint | Method | Auth | Description |
| ------------------------ | ------ | ----- | --------------------- |
| `GET /api/skills` | GET | User | List installed skills |
| `POST /api/skills` | POST | Admin | Install a skill |
| `PATCH /api/skills/:id` | PATCH | Admin | Update a skill |
| `DELETE /api/skills/:id` | DELETE | Admin | Remove a skill |
### Coord (Mission Coordination)
| Endpoint | Method | Auth | Description |
| ------------------------------- | ------ | ---- | ---------------- |
| `GET /api/coord/missions` | GET | User | List missions |
| `POST /api/coord/missions` | POST | User | Create a mission |
| `GET /api/coord/missions/:id` | GET | User | Get a mission |
| `PATCH /api/coord/missions/:id` | PATCH | User | Update a mission |
### Observability
OpenTelemetry traces are exported to the OTEL collector (`OTEL_EXPORTER_OTLP_ENDPOINT`).
View traces in Jaeger at `http://localhost:16686`.
Tracing is initialized before NestJS bootstrap in
`apps/gateway/src/tracing.ts`. The import order in `apps/gateway/src/main.ts`
is intentional: `import './tracing.js'` must come before any NestJS imports.