mosaicstack/stack
21
0
Fork 1
Code Issues 338 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

chore: consolidate new foundation and archive v1 (#1495)

Browse Source
This commit is contained in:
jason.woltje
2026-09-07 12:32:57 -05:00
parent 9a5fbdbda7 5d27700026
commit 127a54fdff
3511 changed files with 727899 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
v1/docs/guides/deployment.md
+72
View File
@@ -0,0 +1,72 @@
---
kind: guide
status: active
---
# Deployment Guide
> **Status: non-operative for PostgreSQL, federated, and bare-metal production.** The checked-in
> Compose PostgreSQL service mounts legacy initialization SQL and the KBN-101 bootstrap, runner,
> secret-renderer, and process-exec interfaces do not exist yet. This page does not authorize a
> production deployment, database initialization, manual DDL, secret provisioning, or service
> activation.
## Current safe local route
Use PGlite only for current in-process data-layer work; it requires no PostgreSQL. A Gateway/Web
local process is held because its unguarded dotenv loader can inherit a daemon PostgreSQL DSN and
reach runtime DDL. If a local queue service is useful, start only Valkey:
```bash
docker compose up -d valkey
```
This command intentionally does not start PostgreSQL. Do not run a broad Compose start, use its
PostgreSQL initialization mount, infer that current Compose is a production/federated route, or
start Gateway/Web until KBN-101-02 supplies fail-closed local-tier/DSN isolation.
## Held future procedure
PostgreSQL local, federated, Compose, and bare-metal production activation are held until these
artifacts land and pass their independent gates:
1. **KBN-101-00** external privileged bootstrap artifact;
2. **KBN-101-03** sole `mosaic-db-migrator` runner and verified-readiness artifact; and
3. **KBN-101-05** Vault/secret-renderer-backed deployment and consumer-isolation artifact.
The required future order is external bootstrap → TLS/roles → `mosaic-db-migrator --run` → `mosaic-db-migrator --verify` → Gateway/Compose readiness.
This is a held, non-operative future activation specification with no current command authority. Do not invoke the named
runner, start PostgreSQL, or substitute a Compose/init/manual-SQL route until the owned artifacts
are implemented and reviewed.
## Future production secret and unit boundary (schematic only)
No current bare-metal production unit or command is published. KBN-101-05 must supply a reviewed,
generation-pinned Vault renderer and a process-exec or systemd `LoadCredential` interface before
production units can exist. The interface must preserve these exact consumer boundaries:
| Consumer | May receive | Must never receive |
| ----------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Gateway/runtime | Its own runtime URL and DB client CA at process exec | Migrator URL, importer URL/version, attestation material, signing key, PostgreSQL private key |
| One-shot migrator | Its own migration URL, DB client CA, and runner-only signing capability | Runtime URL, importer consumer copy, Gateway/private PostgreSQL keys |
| Data importer | Its own immutable URL/version copies, importer CA, pinned public key, and sealed attestation | Runtime/migrator URLs, signing key, shared writable mount |
| PostgreSQL | Its own server certificate/key and only its approved server material | Application, migrator, importer, or Gateway secrets |
A future unit specification is non-executable until KBN-101-05 supplies it. It must obtain
credentials through the renderer’s Vault generation and process-exec/`LoadCredential` boundary;
it must not place credentials in a production environment file, a monorepo auto-load path, a shell
export, command arguments, logs, or a manual secret-activation lifecycle instruction. Rotation and
process replacement semantics must be delivered by the reviewed renderer/interface with generation,
consumer-isolation, mode/owner, and no-mixed-generation evidence—not improvised in this guide.
## Readiness and troubleshooting status
Until the future procedure is implemented, do not diagnose PostgreSQL with ad hoc SQL, connection
strings, or initialization scripts. The future sanitized runner-verification readiness artifact is
the required PostgreSQL readiness authority after its bootstrap/TLS prerequisites pass.
For local PGlite development, diagnose application behavior without introducing a PostgreSQL
connection.
Non-database local services may be inspected with their ordinary local health/log tools. Those
checks do not certify PostgreSQL, federated deployment, or production readiness.
v1/docs/guides/dev-guide.md
+617
View File
@@ -0,0 +1,617 @@
---
kind: guide
status: active
---
# Mosaic Stack — Developer Guide
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Local Development Setup](#local-development-setup)
3. [Building and Testing](#building-and-testing)
4. [Adding New Agent Tools](#adding-new-agent-tools)
5. [Adding New MCP Tools](#adding-new-mcp-tools)
6. [Database Schema and Migrations](#database-schema-and-migrations)
7. [Claude Code Skill Bridge](#claude-code-skill-bridge)
8. [Pi Persistent Goal Extension](#pi-persistent-goal-extension)
9. [API Endpoint Reference](#api-endpoint-reference)
10. [Local Fleet Canary](./fleet-local-canary.md)
---
## Architecture Overview
Mosaic Stack is a TypeScript monorepo managed with **pnpm workspaces** and
**Turborepo**.
```
mosaic-mono-v1/
├── apps/
│ ├── gateway/ # NestJS + Fastify API server
│ └── web/ # Next.js 16 + React 19 web dashboard
├── packages/
│ ├── agent/ # Agent session types (shared)
│ ├── auth/ # BetterAuth configuration
│ ├── brain/ # Structured data layer (projects, tasks, missions)
│ ├── cli/ # mosaic CLI and TUI (Ink)
│ ├── coord/ # Mission coordination engine
│ ├── db/ # Drizzle ORM schema, migrations, client
│ ├── design-tokens/ # Shared design system tokens
│ ├── log/ # Agent log ingestion and tiering
│ ├── memory/ # Preference and insight storage
│ ├── mosaic/ # Install wizard and bootstrap utilities
│ ├── prdy/ # PRD wizard CLI
│ ├── quality-rails/ # Code quality scaffolder CLI
│ ├── queue/ # Valkey-backed task queue
│ └── types/ # Shared TypeScript types
├── docker/ # Dockerfile(s) for containerized deployment
├── infra/ # Infrastructure configuration (for example, OTEL collector)
├── docker-compose.yml # Local services (Postgres, Valkey, OTEL, Jaeger)
└── CLAUDE.md # Project conventions for AI coding agents
```
### Key Technology Choices
| Concern | Technology |
| ----------------- | ---------------------------------------- |
| API framework | NestJS with Fastify adapter |
| Web framework | Next.js 16 (App Router), React 19 |
| ORM | Drizzle ORM |
| Database | PostgreSQL 17 + pgvector extension |
| Auth | BetterAuth |
| Agent harness | Pi SDK (`@mariozechner/pi-coding-agent`) |
| Queue | Valkey 8 (Redis-compatible) |
| Build | pnpm workspaces + Turborepo |
| CI | Woodpecker CI |
| Observability | OpenTelemetry → Jaeger |
| Module resolution | NodeNext (ESM everywhere) |
### Module System
All packages use `"type": "module"` and NodeNext resolution. Import paths must
include the `.js` extension even when the source file is `.ts`.
NestJS `@Inject()` decorators must be used explicitly because `tsx`/`esbuild`
does not support `emitDecoratorMetadata`.
---
## Local Development Setup
### Prerequisites
- Node.js 20+
- pnpm 9+
- Docker and Docker Compose
### 1. Clone and Install Dependencies
```bash
git clone <repo-url> mosaic-mono-v1
cd mosaic-mono-v1
pnpm install
```
### 2. Use the local PGlite tier
The supported local tier is in-process PGlite and requires no PostgreSQL service. Leave
`DATABASE_URL` unset for this route. Its default local configuration uses PGlite and performs no
external database probe.
If a local queue service is useful, start only that non-PostgreSQL service:
```bash
docker compose up -d valkey
```
Do not use the current Compose PostgreSQL service: it mounts legacy `infra/pg-init` SQL and is
not qualified for KBN-101. Start OTEL Collector or Jaeger individually only when needed and
without starting PostgreSQL.
### 3. Gateway/Web local process (held)
Do not start the current Gateway or web process as a local PGlite route. Gateway first loads the
daemon configuration and then project environment files without a tier guard; a pre-existing
`DATABASE_URL` can select PostgreSQL, where current startup still reaches runtime DDL/migrations.
Creating a root `.env` that omits `DATABASE_URL` does not make this safe, so neither a local
credential file nor a web environment file is a current developer procedure.
PGlite remains the supported in-process data-layer implementation, and the optional Valkey command
above remains safe because it does not start PostgreSQL. A safe Gateway/Web local procedure is held
until KBN-101-02 rejects a daemon, inherited, root, or app-local PostgreSQL DSN and any non-local
tier before connection or DDL; KBN-101-05 then supplies the production renderer/Vault process-exec
or `LoadCredential` boundary.
### Held future procedure
PostgreSQL local and federated deployment are held until KBN-101-00 (external bootstrap),
KBN-101-03 (runner), and KBN-101-05 (renderer-backed deployment) land. The following is the
**held, non-operative future activation order with no current command authority**:
external bootstrap → TLS/roles → `mosaic-db-migrator --run` →
`mosaic-db-migrator --verify` → Gateway/Compose readiness.
Neither current Compose nor this development guide authorizes PostgreSQL initialization SQL,
manual DDL, or a pre-runner start.
### 5. Gateway/Web start (held)
No Gateway/Web start command is currently authorized for the local PGlite route. Do not use root
`pnpm dev` as a workaround: it additionally starts configured integrations and cannot establish the
required local-tier/DSN isolation. Resume this section only after KBN-101-02 provides its
fail-closed local-startup evidence.
---
## Building and Testing
### TypeScript Typecheck
```bash
pnpm typecheck
```
Runs `tsc --noEmit` across all packages in dependency order via Turborepo.
### Lint
```bash
pnpm lint
```
Runs ESLint across all packages. Config is in `eslint.config.mjs` at the root.
### Format Check
```bash
pnpm format:check
```
Runs Prettier in check mode. To auto-fix:
```bash
pnpm format
```
### Tests
```bash
pnpm test
```
Runs Vitest across all packages. The workspace config is at
`vitest.workspace.ts`.
### Build
```bash
pnpm build
```
Builds all packages and apps in dependency order.
### Pre-Push Gates (MANDATORY)
All three must pass before any push:
```bash
pnpm format:check && pnpm typecheck && pnpm lint
```
A pre-push hook enforces this mechanically.
### CI Publish Channels
Woodpecker `.woodpecker/publish.yml` keeps stable and integration-line artifacts separate:
| Source | npm packages | Gateway image |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `main` push/manual or release tag | committed package versions published to Gitea npm without changing the dist-tag workflow | `gateway:sha-<short>` plus `gateway:latest` on `main`, and the release tag on tag events |
| `next` push/manual | CI-computed prereleases, `<target-stable>-next.<CI_PIPELINE_NUMBER>`, published with `npm publish --tag next` | `gateway:sha-<short>` only |
`next` never publishes npm `latest` or Docker `latest`. The next npm publish step verifies that `@mosaicstack/mosaic@next` resolves to the computed prerelease before the pipeline can pass.
### E2E Gate (#1445, P6)
Trunk publish pipelines run a headless Playwright suite (`e2e` step) before any image publishes: the built gateway `dist` boots on a throwaway embedded PGlite database (isolated via a fresh `HOME`), serves the built SPA bundle through `WEB_DIST_DIR` — the same serving path the gateway image ships — and the suite runs against it inside the pinned `mcr.microsoft.com/playwright` image. `E2E_REQUIRE_SEEDED_AUTH=1` makes login failures hard failures (the skip-when-login-fails guards are a live-environment affordance only). Both image build steps depend on this gate.
Reproduce locally (Ubuntu-based environments; Fedora's headless-shell rendering is broken):
```bash
pnpm build
BETTER_AUTH_SECRET="$(head -c 32 /dev/urandom | base64)" GATEWAY_PORT=14242 \
WEB_DIST_DIR="$PWD/apps/web/dist" HOME="$(mktemp -d)" node apps/gateway/dist/main.js &
E2E_REQUIRE_SEEDED_AUTH=1 PLAYWRIGHT_BASE_URL=http://localhost:14242 \
pnpm --filter @mosaicstack/web exec playwright test
```
---
## Adding New Agent Tools
Agent tools are Pi SDK `ToolDefinition` objects registered in
`apps/gateway/src/agent/agent.service.ts`.
### 1. Create a Tool Factory File
Add a new file in `apps/gateway/src/agent/tools/`:
```typescript
// apps/gateway/src/agent/tools/my-tools.ts
import { Type } from '@sinclair/typebox';
import type { ToolDefinition } from '@mariozechner/pi-coding-agent';
export function createMyTools(): ToolDefinition[] {
const myTool: ToolDefinition = {
name: 'my_tool_name',
label: 'Human Readable Label',
description: 'What this tool does.',
parameters: Type.Object({
input: Type.String({ description: 'The input parameter' }),
}),
async execute(_toolCallId, params) {
const { input } = params as { input: string };
const result = `Processed: ${input}`;
return {
content: [{ type: 'text' as const, text: result }],
details: undefined,
};
},
};
return [myTool];
}
```
### 2. Register the Tools in AgentService
In `apps/gateway/src/agent/agent.service.ts`, import and call your factory
alongside the existing tool registrations:
```typescript
import { createMyTools } from './tools/my-tools.js';
// Inside the session creation logic where tools are assembled:
const tools: ToolDefinition[] = [
...createBrainTools(this.brain),
...createCoordTools(this.coordService),
...createMemoryTools(this.memory, this.embeddingService),
...createFileTools(sandboxDir),
...createGitTools(sandboxDir),
...createShellTools(sandboxDir),
...createWebTools(),
...createMyTools(), // Add this line
...mcpTools,
...skillTools,
];
```
### 3. Export from the Tools Index
Add an export to `apps/gateway/src/agent/tools/index.ts`:
```typescript
export { createMyTools } from './my-tools.js';
```
### 4. Typecheck and Test
```bash
pnpm typecheck
pnpm test
```
---
## Adding New MCP Tools
Mosaic connects to external MCP servers via `McpClientService`. To expose tools
from a new MCP server:
### 1. Run an MCP Server
Implement a standard MCP server that exposes tools via the streamable HTTP
transport or SSE transport. The server must accept connections at a `/mcp`
endpoint.
### 2. Gateway MCP configuration (held)
Do not configure MCP endpoint credentials, write them to a local environment file, or restart the
Gateway from this guide. Gateway/Web startup is held until KBN-101-02 supplies fail-closed
local-tier/DSN isolation and KBN-101-05 supplies the renderer/Vault process-exec or
`LoadCredential` secret-consumer interface. The future authenticated MCP route requires verified
HTTPS and certificate validation; plaintext bearer-token examples are forbidden.
### Tool Naming
Bridged MCP tool names are taken directly from the MCP server's tool manifest.
Ensure names do not conflict with built-in tools (check
`apps/gateway/src/agent/tools/`).
---
## Database Schema and Migrations
The schema lives in a single file:
`packages/db/src/schema.ts`
### Schema Overview
| Table | Purpose |
| -------------------- | ------------------------------------------------- |
| `users` | User accounts (BetterAuth-compatible) |
| `sessions` | Auth sessions |
| `accounts` | OAuth accounts |
| `verifications` | Email verification tokens |
| `projects` | Project records |
| `missions` | Mission records (linked to projects) |
| `tasks` | Task records (linked to projects and/or missions) |
| `conversations` | Chat conversation metadata |
| `messages` | Individual chat messages |
| `preferences` | Per-user key-value preference store |
| `insights` | Vector-embedded memory insights |
| `agent_logs` | Agent interaction logs (hot/warm/cold tiers) |
| `skills` | Installed agent skills |
| `summarization_jobs` | Log summarization job tracking |
The `insights` table uses a `vector(1536)` column (pgvector) for semantic search.
### PostgreSQL schema work (held)
Do not prepare or run a PostgreSQL target from this branch. The sole runner, bootstrap, and
renderer are future KBN-101 artifacts, not current commands. When KBN-101-00/-03/-05 land, the
owned activation documentation will require external bootstrap → TLS/roles → runner `--run` →
runner `--verify` → Gateway/Compose readiness.
### Generating migration artifacts
`pnpm --filter @mosaicstack/db db:generate` is an offline artifact-generation command. It does
not authorize connecting to or initializing PostgreSQL. A future reviewed PostgreSQL procedure
will determine when its output is applied.
### Drizzle Config
Config is at `packages/db/drizzle.config.ts`. The schema file path and output directory are
defined there.
### Adding a New Table
1. Add the table definition to `packages/db/src/schema.ts`.
2. Export it from `packages/db/src/index.ts`.
3. Generate the offline artifact with `pnpm --filter @mosaicstack/db db:generate`.
4. Do not apply it to PostgreSQL until the future KBN-101 activation artifacts and their owned
procedure are available. Direct schema push is not a production-like workflow.
---
## Claude Code Skill Bridge
The framework's canonical skill root is `~/.config/mosaic/skills/`; Claude Code
requires registrations under `~/.claude/skills/`. The implementation in
`packages/mosaic/src/commands/skill.ts` owns only direct-child symlinks whose
resolved target remains inside the canonical root.
Security invariants:
1. Validate the user-supplied name before filesystem access against
`[A-Za-z0-9][A-Za-z0-9._-]*`. Separators, control characters, whitespace,
`..`, absolute paths, and leading `-` are invalid; filesystem-derived invalid
names are escaped before terminal output.
2. Never replace a real file, directory, foreign symlink, or live misdirected
symlink in the Claude skill directory.
3. Repair a dangling link only when its lexical target is inside the canonical
Mosaic skills root.
4. Unregister only a symlink pointing inside that root.
5. Enumerate canonical directories at runtime; never hardcode framework skill
names.
`finalizeStage` reconciles after wizard/framework synchronization, and
`runFrameworkReseed` reconciles after the sync-only `mosaic update` path. A
foreign conflict is reported but does not prevent unrelated canonical skills
from registering. Filesystem tests use injected temporary roots in
`skill.spec.ts`, `finalize-skills.spec.ts`, and `update-checker.reseed.spec.ts`.
M1 intentionally manages Claude Code only. Pi's Mosaic launcher can discover the
canonical root directly. Codex still relies on the existing full skill-sync
linker and needs separate parity analysis before this lifecycle API is extended.
## Pi Persistent Goal Extension
The source of the Mosaic-owned Pi goal controller is:
```text
packages/mosaic/framework/runtime/pi/goal-extension.ts
```
The framework manifest classifies `runtime/**` as framework-owned. Both the bash installer and the
TypeScript file adapter therefore deploy the same reviewed source to:
```text
$MOSAIC_HOME/runtime/pi/goal-extension.ts
# default: ~/.config/mosaic/runtime/pi/goal-extension.ts
```
Do not copy or link this extension into `~/.pi/agent/extensions/`. The launcher function
`discoverPiExtensionArgs()` emits the core `mosaic-extension.ts` first and the optional
`goal-extension.ts` second, preserving compatibility with an older installed framework that does
not have the goal file yet.
### Lifecycle design
| Pi API | Goal-controller responsibility |
| ------------------------------ | --------------------------------------------------------------------------------- |
| `registerCommand('goal')` | Set, inspect, pause, resume, or cancel one branch-specific goal |
| `registerTool(...)` | Record a terminating structured progress report with evidence |
| `context` | Inject the active goal contract before every provider request |
| `turn_end` | Record every turn, reject mixed final reports, and enforce the turn bound |
| `agent_settled` | Start one deduplicated continuation only after Pi has no retry/compact/queue work |
| `session_compact` | Record the compact check, reset provisional verification, and defer idle work |
| `session_start`/`session_tree` | Rebuild state from custom entries on the active branch |
| `session_shutdown` | Invalidate deferred callbacks and clear UI state |
State is appended as `mosaic-goal-state` custom entries, which do not enter model context. The
`context` hook creates a fresh hidden `mosaic-goal-context` message for each request instead of
trusting compaction summaries. The `mosaic_goal_report` result uses `terminate: true`; when it is the
sole final tool call, Pi avoids an unnecessary model response before the controller decides whether
to verify, continue, or stop.
Before state is appended or displayed, the controller applies bounded credential-pattern redaction
to the goal statement, report summary/evidence/next step, and stop reason. Fingerprints are computed
over redacted report content. Pi session entries are append-only, so a credential-bearing legacy
entry cannot honestly be erased by the extension: restoration fails closed, emits a warning, and
requires removal of the affected session before setting a new goal. This is defense-in-depth rather
than a secret-storage contract, and it does not rewrite Pi's separate model-message/tool-call
history. Goal prompts tell the agent not to submit credentials or raw sensitive output, and tests use
canaries to prove known forms do not reach new custom entries, status text, context, or tool details
while ordinary typed fields such as `token: string` remain intact.
Completion remains evidence-gated but semantic: two consecutive `achieved` reports are required,
and the second run is explicitly a verification pass. This avoids an extra judge-model request after
every turn. Deterministic validator commands are intentionally not accepted as `/goal` input in this
slice, so never describe this mechanism as proof of arbitrary natural-language completion.
### Tests and local smoke workflow
```bash
pnpm --filter @mosaicstack/mosaic exec vitest run \
src/runtime/pi-goal-extension.spec.ts \
src/commands/launch.spec.ts \
src/config/file-adapter.test.ts
bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh
```
For an additive local smoke test without reseeding unrelated live framework files:
```bash
install -D -m 0644 \
packages/mosaic/framework/runtime/pi/goal-extension.ts \
~/.config/mosaic/runtime/pi/goal-extension.ts
pi --extension ~/.config/mosaic/runtime/pi/goal-extension.ts
```
Use `/goal help`, `/goal set ...`, and `/goal status` in that test session. A released framework
sync installs the file, and a released Mosaic CLI loads it automatically through `mosaic pi`.
## API Endpoint Reference
All endpoints are served by the gateway at `http://localhost:14242` by default.
### Authentication
Authentication uses BetterAuth session cookies. The auth handler is mounted at
`/api/auth/*` via a Fastify low-level hook in
`apps/gateway/src/auth/auth.controller.ts`.
| Endpoint | Method | Description |
| ------------------------- | ------ | -------------------------------- |
| `/api/auth/sign-in/email` | POST | Sign in with email/password |
| `/api/auth/sign-up/email` | POST | Register a new account |
| `/api/auth/sign-out` | POST | Sign out (clears session cookie) |
| `/api/auth/get-session` | GET | Returns the current session |
### Chat
WebSocket namespace `/chat` (Socket.IO). Authentication via session cookie.
Events sent by the client:
| Event | Payload | Description |
| --------- | --------------------------------------------------- | -------------- |
| `message` | `{ content, conversationId?, provider?, modelId? }` | Send a message |
Events emitted by the server:
| Event | Payload | Description |
| ------- | --------------------------- | ---------------------- |
| `token` | `{ token, conversationId }` | Streaming token |
| `end` | `{ conversationId }` | Stream complete |
| `error` | `{ message }` | Error during streaming |
HTTP endpoints (`apps/gateway/src/chat/chat.controller.ts`):
| Endpoint | Method | Auth | Description |
| -------------------------------------- | ------ | ---- | ------------------------------- |
| `/api/chat/conversations` | GET | User | List conversations |
| `/api/chat/conversations/:id/messages` | GET | User | Get messages for a conversation |
### Admin
All admin endpoints require `role = admin`.
| Endpoint | Method | Description |
| --------------------------------- | ------ | -------------------- |
| `GET /api/admin/users` | GET | List all users |
| `GET /api/admin/users/:id` | GET | Get a single user |
| `POST /api/admin/users` | POST | Create a user |
| `PATCH /api/admin/users/:id/role` | PATCH | Update user role |
| `POST /api/admin/users/:id/ban` | POST | Ban a user |
| `POST /api/admin/users/:id/unban` | POST | Unban a user |
| `DELETE /api/admin/users/:id` | DELETE | Delete a user |
| `GET /api/admin/health` | GET | System health status |
### Agent / Providers
| Endpoint | Method | Auth | Description |
| ------------------------------------ | ------ | ---- | ----------------------------------- |
| `GET /api/agent/providers` | GET | User | List all providers and their models |
| `GET /api/agent/providers/models` | GET | User | List available models |
| `POST /api/agent/providers/:id/test` | POST | User | Test provider connectivity |
### Projects / Brain
| Endpoint | Method | Auth | Description |
| -------------------------------- | ------ | ---- | ---------------- |
| `GET /api/brain/projects` | GET | User | List projects |
| `POST /api/brain/projects` | POST | User | Create a project |
| `GET /api/brain/projects/:id` | GET | User | Get a project |
| `PATCH /api/brain/projects/:id` | PATCH | User | Update a project |
| `DELETE /api/brain/projects/:id` | DELETE | User | Delete a project |
| `GET /api/brain/tasks` | GET | User | List tasks |
| `POST /api/brain/tasks` | POST | User | Create a task |
| `GET /api/brain/tasks/:id` | GET | User | Get a task |
| `PATCH /api/brain/tasks/:id` | PATCH | User | Update a task |
| `DELETE /api/brain/tasks/:id` | DELETE | User | Delete a task |
### Memory / Preferences
| Endpoint | Method | Auth | Description |
| ----------------------------- | ------ | ---- | -------------------- |
| `GET /api/memory/preferences` | GET | User | Get user preferences |
| `PUT /api/memory/preferences` | PUT | User | Upsert a preference |
### MCP Server (Gateway-side)
| Endpoint | Method | Auth | Description |
| ----------- | ------ | --------------------------------------------- | ----------------------------- |
| `POST /mcp` | POST | User (session cookie or Authorization header) | MCP streamable HTTP transport |
| `GET /mcp` | GET | User | MCP SSE stream reconnect |
### Skills
| Endpoint | Method | Auth | Description |
| ------------------------ | ------ | ----- | --------------------- |
| `GET /api/skills` | GET | User | List installed skills |
| `POST /api/skills` | POST | Admin | Install a skill |
| `PATCH /api/skills/:id` | PATCH | Admin | Update a skill |
| `DELETE /api/skills/:id` | DELETE | Admin | Remove a skill |
### Coord (Mission Coordination)
| Endpoint | Method | Auth | Description |
| ------------------------------- | ------ | ---- | ---------------- |
| `GET /api/coord/missions` | GET | User | List missions |
| `POST /api/coord/missions` | POST | User | Create a mission |
| `GET /api/coord/missions/:id` | GET | User | Get a mission |
| `PATCH /api/coord/missions/:id` | PATCH | User | Update a mission |
### Observability
OpenTelemetry traces are exported to the OTEL collector (`OTEL_EXPORTER_OTLP_ENDPOINT`).
View traces in Jaeger at `http://localhost:16686`.
Tracing is initialized before NestJS bootstrap in
`apps/gateway/src/tracing.ts`. The import order in `apps/gateway/src/main.ts`
is intentional: `import './tracing.js'` must come before any NestJS imports.
v1/docs/guides/fleet-local-canary.md
+182
View File
@@ -0,0 +1,182 @@
---
kind: guide
status: active
---
# Local Fleet Canary
The local fleet canary runs a small tmux-backed Mosaic agent fleet on an
isolated tmux socket. The default socket is `mosaic-fleet`; the commands do
not use or stop the default tmux server.
## Files
Product-owned defaults:
- `packages/mosaic/framework/fleet/roster.schema.json`
- `packages/mosaic/framework/fleet/examples/minimal.yaml`
- `packages/mosaic/framework/fleet/examples/local-canary.yaml`
- `packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service`
- `packages/mosaic/framework/systemd/user/[email protected]`
- `packages/mosaic/framework/tools/fleet/start-agent-session.sh`
- `packages/mosaic/framework/tools/tmux/agent-send.sh`
- `packages/mosaic/framework/tools/tmux/send-message.sh`
These files are published through `packages/mosaic/package.json`, whose `files`
allowlist includes `framework` along with `dist`.
Site-owned local roster:
```text
~/.config/mosaic/fleet/roster.yaml
```
Do not put a host-specific full roster into product defaults. Start from an
example and edit the local roster after `mosaic fleet init --write`.
## Install
Minimal canary:
```bash
mosaic fleet init --profile minimal --write
# If a site-owned roster already exists, inspect it first; overwrite only explicitly:
# mosaic fleet init --profile minimal --write --force
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet verify
```
Small dogfood roster:
```bash
mosaic fleet init --profile local-canary --write
# Use --force only after preserving any site-owned roster changes.
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet status
```
## Agent Operations
```bash
mosaic agent roster
mosaic agent status
mosaic agent status canary-pi
mosaic agent send canary-pi --message "status check"
mosaic agent reset canary-pi --new
mosaic agent tail canary-pi -n 80
```
These commands read the roster and target the configured tmux socket. The
generated systemd agent services use `start-agent-session.sh`; message delivery
uses the tmux send tools with `-L mosaic-fleet`.
`mosaic agent send` is operator-origin traffic unless a caller explicitly says
otherwise. The CLI always passes a deterministic source label to
`agent-send.sh` with `-S`, defaulting to `<hostname>:operator`, so it does not
query the target tmux socket and accidentally identify as an active agent pane.
Use `--source-label <label>` or `--source <label>` only when deliberately
impersonating a known handoff lane. The lower-level inter-agent wrapper
`agent-send.sh -S <label>` remains the explicit source override for scripts.
## Verification
Use these checks before expanding the roster:
```bash
tmux -L mosaic-fleet ls
tmux ls
mosaic fleet verify
systemctl --user status mosaic-tmux-holder.service
```
Expected results:
- `tmux -L mosaic-fleet ls` shows `_holder` and roster agent sessions.
- `tmux ls` shows only the default tmux server sessions and is not changed by
fleet start/stop operations.
- `mosaic fleet verify` checks exact session targets on the isolated socket.
- `systemctl --user status ...` may show `active (exited)` for oneshot units;
that means the unit ran, not that an agent pane is live. Treat tmux
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
## Recovery — rebuild generated env projections
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
deterministic projection of `roster.yaml` (the SSOT) that the launcher
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
wipes or diverges those projections, rebuild them from the roster with
`mosaic fleet regen` — do NOT restart the affected unit first.
```bash
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json # same plan, machine-readable
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
```
`regen` is projection-only and **never restarts an agent** — it has no path to
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
only (never the projected `KEY=value` body). After `--write`, verify each unit
resolves the intended values before restarting one unit at a time. The unit sets
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
so verify the generated file directly and the launcher path, not a nonexistent
`EnvironmentFile` property:
```bash
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>
```
Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](../ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md).
## Release Preflight
Run this checklist before cutting or dogfooding a fleet release:
- Real AI dogfood: send at least one task through `mosaic agent send`, then
confirm the agent accepted/responded using pane, process, or log evidence.
- Restart/stop/idempotency: run `mosaic fleet start`, `restart`, `stop`, and a
repeated `start` against the named socket; verify the default tmux server is
unchanged.
- Liveness verification: run `mosaic fleet verify` and confirm roster sessions
with `tmux -L mosaic-fleet ls` or exact `has-session` checks.
- Package dry-run: run `npm pack --dry-run --json` from `packages/mosaic` and
confirm `framework/fleet`, `framework/systemd/user`,
`framework/tools/fleet`, and `framework/tools/tmux` assets are included.
- Mosaic update test: install or upgrade from the packed artifact in a temporary
Mosaic home and confirm `mosaic update` or the release upgrade path does not
remove local roster/config files.
## Rollback
Stop the local canary:
```bash
mosaic fleet stop
systemctl --user disable [email protected]
systemctl --user disable mosaic-tmux-holder.service
systemctl --user daemon-reload
```
For a full local cleanup of generated canary files:
```bash
rm -f ~/.config/systemd/user/[email protected]
rm -f ~/.config/systemd/user/mosaic-tmux-holder.service
rm -rf ~/.config/mosaic/fleet
rm -rf ~/.config/mosaic/tools/fleet
```
This rollback leaves the default tmux server untouched. If a canary session is
still present after service stop, remove only the isolated socket server:
```bash
tmux -L mosaic-fleet kill-server
```
v1/docs/guides/migrate-tier.md
+103
View File
@@ -0,0 +1,103 @@
---
kind: guide
status: active
---
# Migrating to the Federated Tier
> **KBN-101-07 ownership:** This active documentation is a **non-operative KBN-101
> contract** with no current command authority until KBN-101-00, KBN-101-02, KBN-101-03, KBN-101-05, and KBN-101-06 land and
> KBN-101-08 activates an exact reviewed release. The commands below describe the produced interface only. Do not run them on the
> current branch or replace them with direct PostgreSQL, raw SQL, legacy storage migration, or
> credential-on-argv procedures.
## Held future procedure
This section is non-operative and grants no current command authority until KBN-101-00, KBN-101-03, and KBN-101-05 land.
The deployment control plane executes the complete held future procedure, in order: external bootstrap → TLS/roles → `mosaic-db-migrator --run` → `mosaic-db-migrator --verify` → Gateway/Compose readiness. The
runner is the only attestation producer after its verified TLS, identity, manifest, and schema
checks. A data importer is never a schema bootstrap, extension installer, repair command, or DDL
consumer.
## Target material contract
KBN-101-05 obtains the target URL from Vault KV-v2
`secret-{env}/mosaic-stack/database/importer`, key `url`, and reads its authenticated version from
the same successful response `data.metadata.version`. A hash or DSN byte sequence is not a
provider version. The renderer treats URL bytes and provider version as one generation, writes a
temporary generation directory with fsync plus atomic rename, and creates separate immutable
consumer mounts. Swarm uses distinct versioned secret/config references. A deployment cannot mix
generations.
| Consumer | Permitted material |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Migrator-attestation producer (`10003:10003`) | Its own migration URL/CA; read-only `/run/secrets/mosaic-migrate-target-url` and `/run/secrets/mosaic-migrate-target-version`, each `0400`, solely to bind; producer-only attestation output at `/run/mosaic-attestations-producer/migrate-target.v1.json`; root-wrapper-only signing key. It never connects with, uses, exports, logs, or forwards the importer URL/version. |
| Privileged deployment handoff controller | After runner success and before importer creation, it receives only root-owned non-secret expected provider-version/URL-SHA-256/generation descriptor and pinned public verifier key—not URL bytes or private key. It safe-opens/verifies descriptor and producer artifact, copies exact bytes to a new importer-only mount with fsync/atomic rename, sets `10002:10002` `0400`, seals it read-only, and refuses importer start on any partial/wrong-generation/wrong-owner/mode result. |
| Importer (`10002:10002`) | Its own immutable `0400` copies at the same URL/version paths; CA at exact `DATABASE_TLS_CA_CERT_PATH=/run/secrets/mosaic-db-ca.crt`; pinned Ed25519 public key; read-only `/run/mosaic-attestations/migrate-target.v1.json` supplied only by the sealed handoff. |
| Gateway/runtime/unrelated container | No importer URL/version, importer artifact, attestation private key, or unrelated CA mount. |
The migrator and importer safe-open URL, provider-version, attestation, and public-key files only
with `O_RDONLY|O_CLOEXEC|O_NOFOLLOW`; they validate from the opened fd that the file is regular,
has its expected owner/mode and link count one. The migrator digests only that URL fd for binding,
then zeroizes/closes it. The importer reads URL bytes once into protected memory, validates the
signed binding and exact CA before connecting from those same bytes, then zeroizes/closes every
fd. It neither logs nor exposes a URL/version/attestation/key oracle.
## Produced command interface
After activation and only after approved target preparation, the future interface is:
```bash
# Deployment control plane has already completed the held runner procedure above.
mosaic storage migrate-tier --to federated \
--target-url-file /run/secrets/mosaic-migrate-target-url \
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
--dry-run
```
The provider-version file is fixed deployment material, not argv. This connecting dry-run consumes its nonce; before an actual copy, the deployment control plane must provide fresh runner verification and a new sealed handoff. The runner uses its migration
identity; the importer connects only as non-DDL `mosaic_data_importer` and only after all
pre-connect validation. After verified TLS and before DML it compares PostgreSQL system ID,
database OID, `current_user`, CA/SPKI, and manifest/schema fingerprints to the artifact.
## Required refusals and evidence
KBN-101-02/-03/-05/-06 must prove, with stable sanitized errors, that no target connection occurs
for missing/unsafe URL/version/attestation/public-key files; symlink, hardlink, owner, mode, or
TOCTOU violations; mixed URL/version generations; missing/wrong CA mount; stale/replayed/tampered
or revoked-key artifacts; provider rotation/revocation; wrong TLS/server/database/role/manifest
binding; raw `--target-url`; `DATABASE_URL` fallback; runtime/owner identity; consumer leakage;
or any DDL attempt. Post-connect identity mismatch closes with zero DML/DDL. Tests also prove no
forwarding, child environment, logging, or error oracle leaks URL/version/key/artifact contents.
The attestation is credential-free JCS with detached Ed25519 signature and binds issued/expiry,
nonce, authenticated provider version, exact URL-fd SHA-256, TLS host/port/database, CA/SPKI,
PostgreSQL system ID/database OID, importer role, manifest/schema, and producer identity. Provider
version rotation invalidates an old artifact and requires a fresh rendered generation plus runner
verification.
## Actual copy after dry-run
After reviewed dry-run, obtain the required fresh verification/attestation generation, then use:
```bash
# Deployment control plane has supplied fresh runner verification and attestation.
mosaic storage migrate-tier --to federated \
--target-url-file /run/secrets/mosaic-migrate-target-url \
--target-attestation-file /run/mosaic-attestations/migrate-target.v1.json \
--yes
```
The dry-run artifact is terminally replayed and must be rejected; `--yes` bypasses no file,
generation, signature, TLS, identity, or DDL control.
## Data boundary and recovery
The importer has only an allowlisted mutable-table DML registry. It has no grant for immutable KBN
relations, schemas, roles, memberships, extensions, catalogs, or the Drizzle ledger. Source PGlite
uses its explicit local directory and does not make a PostgreSQL URL fallback valid.
A failed or ambiguous migration is a control-plane incident: preserve sanitized evidence, retain
the approved backup/rollback state, and retry only after independent review. Never inspect,
unlock, repair, or initialize the target with ad hoc SQL or copied credentials.