docs: establish canonical documentation architecture (#1210)
ci/woodpecker/push/publish Pipeline failed
ci/woodpecker/push/publish Pipeline failed
This commit was merged in pull request #1210.
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Archived planning records
|
||||
|
||||
> **Status:** Historical planning index. These records preserve prior intent and review context; they are not current requirements, task state, implementation evidence, or operational authority.
|
||||
|
||||
## Monorepo consolidation
|
||||
|
||||
- [Planning bundle](monorepo-consolidation/README.md) — historical brief, board review, and Forge/MACP/framework-plugin work-package specifications.
|
||||
|
||||
## Legacy plans and deferred stubs
|
||||
|
||||
- [Legacy planning index](legacy/README.md) — unreferenced implementation plans, a superseded SSO setup record, and explicitly deferred design stubs.
|
||||
- [Archived Matrix/MACP proposals](matrix-macp/README.md) — historical draft communications and deployment RFCs for functionality not established by current source/tests.
|
||||
- [Archived standalone designs](designs/README.md) — historical prerelease-pipeline and storage-abstraction designs.
|
||||
|
||||
Use current package source, tests, audience guides, and approved control documents for present behavior and status.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Archived standalone designs
|
||||
|
||||
> **Status:** Historical design records. These files moved byte-identically from migration quarantine on 2026-08-10 and are not current implementation or release contracts.
|
||||
|
||||
- [npm prerelease `@next` lane](prerelease-next-dist-tag-pipeline.md) — prior release-pipeline design; verify current CI and package scripts before use.
|
||||
- [Storage and queue abstraction](storage-abstraction-middleware.md) — prior middleware/tier design. Current storage abstractions exist, but this record does not prove its complete target architecture or operational procedures.
|
||||
|
||||
Use current package source, manifests, tests, and canonical safety guidance for present behavior. The coupled #791 upgrade design and normative framework constitution remain in migration quarantine pending their owning workstreams.
|
||||
@@ -0,0 +1,63 @@
|
||||
# npm `@next` prerelease lane
|
||||
|
||||
Status: **IMPLEMENTED**
|
||||
|
||||
## Current behavior
|
||||
|
||||
`tools/install.sh --next` provides the prerelease integration lane for the permanent `next` branch.
|
||||
|
||||
The lane is fast-by-default:
|
||||
|
||||
1. Install framework files from the `next` source archive.
|
||||
2. Resolve the Gitea npm registry `next` dist-tag for the globally installed packages:
|
||||
|
||||
```bash
|
||||
npm view @mosaicstack/gateway@next version
|
||||
npm view @mosaicstack/mosaic@next version
|
||||
```
|
||||
|
||||
3. Require both resolved versions to share the same `next.<pipeline>` suffix, then install the exact resolved versions.
|
||||
4. If either `@next` package is missing, unreachable, mismatched, or fails to install, fall back to the source-build path at `next`.
|
||||
|
||||
`--next` never hard-fails solely because the prerelease npm dist-tag is unavailable.
|
||||
|
||||
## Published packages
|
||||
|
||||
The `next` publish pipeline publishes non-private `@mosaicstack/*` packages to the Mosaic Gitea npm registry:
|
||||
|
||||
```text
|
||||
https://git.mosaicstack.dev/api/packages/mosaicstack/npm/
|
||||
```
|
||||
|
||||
Observed `next` dist-tags after enabling the pipeline:
|
||||
|
||||
```text
|
||||
@mosaicstack/mosaic@next -> 0.0.49-next.1633
|
||||
@mosaicstack/gateway@next -> 0.0.7-next.1633
|
||||
```
|
||||
|
||||
The gateway also publishes a Docker image as `gateway:sha-<short>` on `next` merges. The installer fast path uses the npm gateway package when available; the Docker image is for deployed gateway/runtime harness flows.
|
||||
|
||||
## Explicit source lanes
|
||||
|
||||
Source builds remain available and are still the authority for explicit ref validation:
|
||||
|
||||
- `--dev` always builds from source.
|
||||
- `--ref <ref>` / `MOSAIC_REF=<ref>` wins over `--next` and uses the source path for that exact ref.
|
||||
|
||||
## Pipeline shape
|
||||
|
||||
1. Trigger on `next` merges.
|
||||
2. Compute the next prerelease version from the upcoming stable version plus the Woodpecker pipeline number (`<target-stable>-next.<CI_PIPELINE_NUMBER>`).
|
||||
3. Build and publish non-private packages in CI.
|
||||
4. Publish to the Mosaic Gitea npm registry with dist-tag `next`.
|
||||
5. Keep `latest` untouched; only main/release promotion can update `latest`.
|
||||
6. Publish gateway Docker images from `next` as `gateway:sha-<short>` only.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- `@next` is mutable prerelease convenience, not a deployment pin.
|
||||
- Stable installs continue to use `@latest`.
|
||||
- Contributor validation remains available through `--dev --ref <branch>`.
|
||||
- Pipeline output traces every prerelease package back to the source commit on `next`.
|
||||
- The installer falls back to source rather than hard-failing on prerelease registry issues.
|
||||
@@ -0,0 +1,559 @@
|
||||
# Storage & Queue Abstraction — Middleware Architecture
|
||||
|
||||
Design
|
||||
Status: Design (retrofit required)
|
||||
date: 2026-04-02
|
||||
context: Agents coupled directly to infrastructure backends, bypassing intended middleware layer
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
Current packages are **direct adapters**, not **middleware**:
|
||||
| Package | Current State | Intended Design |
|
||||
|---------|---------------|-----------------|
|
||||
| `@mosaicstack/queue` | `ioredis` hardcoded | Interface → BullMQ OR local-files |
|
||||
| `@mosaicstack/db` | Drizzle + Postgres hardcoded | Interface → Postgres OR SQLite OR JSON/MD |
|
||||
| `@mosaicstack/memory` | pgvector required | Interface → pgvector OR sqlite-vec OR keyword-search |
|
||||
|
||||
## The gateway and TUI import these packages directly, which means they they're coupled to specific infrastructure. Users cannot run Mosaic Stack without Postgres + Valkey.
|
||||
|
||||
## The Intended Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Gateway / TUI / CLI │
|
||||
│ (agnostic of storage backend, talks to middleware) │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
│ │ │
|
||||
▼─────────────────┴─────────────────┴─────────────────┘
|
||||
| | | |
|
||||
▼─────────────────┴───────────────────┴─────────────────┘
|
||||
| | | |
|
||||
Queue Storage Memory
|
||||
| | | |
|
||||
┌─────────┬─────────┬─────────┬─────────────────────────────────┐
|
||||
| BullMQ | | Local | | Postgres | SQLite | JSON/MD | pgvector | sqlite-vec | keyword |
|
||||
|(Valkey)| |(files) | | | | | |
|
||||
└─────────┴─────────┴─────────┴─────────────────────────────────┘
|
||||
```
|
||||
|
||||
The gateway imports the interface, not the backend. At startup it reads config and instantiates the correct adapter.
|
||||
|
||||
## The Drift
|
||||
|
||||
```typescript
|
||||
// What should have happened:
|
||||
gateway/queue.service.ts → @mosaicstack/queue (interface) → queue.adapter.ts
|
||||
|
||||
// What actually happened:
|
||||
gateway/queue.service.ts → @mosaicstack/queue → ioredis (hardcoded)
|
||||
```
|
||||
|
||||
## The Current State Analysis
|
||||
|
||||
### `@mosaicstack/queue` (packages/queue/src/queue.ts)
|
||||
|
||||
```typescript
|
||||
import Redis from 'ioredis'; // ← Direct import of backend
|
||||
|
||||
export function createQueue(config?: QueueConfig): QueueHandle {
|
||||
const url = config?.url ?? process.env['VALKEY_URL'] ?? DEFAULT_VALKEY_URL;
|
||||
const redis = new Redis(url, { maxRetriesPerRequest: 3 });
|
||||
// ...queue ops directly on redis...
|
||||
}
|
||||
```
|
||||
|
||||
**Problem:** `ioredis` is imported in the package, not the adapter interface. Consumers cannot swap backends.
|
||||
|
||||
### `@mosaicstack/db` (packages/db/src/client.ts)
|
||||
|
||||
> **Historical design specimen — status-only, not an operator instruction.** KBN-101 supersedes
|
||||
> this pre-split `DATABASE_URL` fallback shape; it cannot authorize runtime migration, DDL, or a
|
||||
> connection-string fallback. See the KBN-101 runner/role contract for the produced interface.
|
||||
|
||||
```typescript
|
||||
import { drizzle, type PostgresJsDatabase } from 'drizzle-orm/postgres-js';
|
||||
import postgres from 'postgres';
|
||||
|
||||
export function createDb(url?: string): DbHandle {
|
||||
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
|
||||
const sql = postgres(connectionString, { max: 20, idle_timeout: 30, connect_timeout: 5 });
|
||||
const db = drizzle(sql, { schema });
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Problem:** Drizzle + Postgres is hardcoded. No SQLite, JSON, or file-based options.
|
||||
|
||||
### `@mosaicstack/memory` (packages/memory/src/memory.ts)
|
||||
|
||||
```typescript
|
||||
import type { Db } from '@mosaicstack/db'; // ← Depends on Drizzle/PG
|
||||
|
||||
export function createMemory(db: Db): Memory {
|
||||
return {
|
||||
preferences: createPreferencesRepo(db),
|
||||
insights: createInsightsRepo(db),
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Problem:** Memory package is tightly coupled to `@mosaicstack/db` (which is Postgres-only). No alternative storage backends.
|
||||
|
||||
## The Target Interfaces
|
||||
|
||||
### Queue Interface
|
||||
|
||||
```typescript
|
||||
// packages/queue/src/types.ts
|
||||
export interface QueueAdapter {
|
||||
readonly name: string;
|
||||
|
||||
enqueue(queueName: string, payload: TaskPayload): Promise<void>;
|
||||
dequeue(queueName: string): Promise<TaskPayload | null>;
|
||||
length(queueName: string): Promise<number>;
|
||||
publish(channel: string, message: string): Promise<void>;
|
||||
subscribe(channel: string, handler: (message: string) => void): () => void;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface TaskPayload {
|
||||
id: string;
|
||||
type: string;
|
||||
data: Record<string, unknown>;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface QueueConfig {
|
||||
type: 'bullmq' | 'local';
|
||||
url?: string; // For bullmq: Valkey/Redis URL
|
||||
dataDir?: string; // For local: directory for JSON persistence
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Interface
|
||||
|
||||
```typescript
|
||||
// packages/storage/src/types.ts
|
||||
export interface StorageAdapter {
|
||||
readonly name: string;
|
||||
|
||||
// Entity CRUD
|
||||
create<T>(collection: string, data: O): Promise<T>;
|
||||
read<T>(collection: string, id: string): Promise<T | null>;
|
||||
update<T>(collection: string, id: string, data: Partial<O>): Promise<T | null>;
|
||||
delete(collection: string, id: string): Promise<boolean>;
|
||||
|
||||
// Queries
|
||||
find<T>(collection: string, filter: Record<string, unknown>): Promise<T[]>;
|
||||
findOne<T>(collection: string, filter: Record<string, unknown): Promise<T | null>;
|
||||
|
||||
// Bulk operations
|
||||
createMany<T>(collection: string, items: O[]): Promise<T[]>;
|
||||
updateMany<T>(collection: string, ids: string[], data: Partial<O>): Promise<number>;
|
||||
deleteMany(collection: string, ids: string[]): Promise<number>;
|
||||
|
||||
// Raw queries (for complex queries)
|
||||
query<T>(collection: string, query: string, params?: unknown[]): Promise<T[]>;
|
||||
|
||||
// Transaction support
|
||||
transaction<T>(fn: (tx: StorageTransaction) => Promise<T>): Promise<T>;
|
||||
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface StorageTransaction {
|
||||
commit(): Promise<void>;
|
||||
rollback(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface StorageConfig {
|
||||
type: 'postgres' | 'sqlite' | 'files';
|
||||
url?: string; // For postgres
|
||||
path?: string; // For sqlite/files
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Interface (Vector + Preferences)
|
||||
|
||||
```typescript
|
||||
// packages/memory/src/types.ts
|
||||
export interface MemoryAdapter {
|
||||
readonly name: string;
|
||||
|
||||
// Preferences (key-value storage)
|
||||
getPreference(userId: string, key: string): Promise<unknown | null>;
|
||||
setPreference(userId: string, key: string, value: unknown): Promise<void>;
|
||||
deletePreference(userId: string, key: string): Promise<boolean>;
|
||||
listPreferences(
|
||||
userId: string,
|
||||
category?: string,
|
||||
): Promise<Array<{ key: string; value: unknown }>>;
|
||||
|
||||
// Insights (with optional vector search)
|
||||
storeInsight(insight: NewInsight): Promise<Insight>;
|
||||
getInsight(id: string): Promise<Insight | null>;
|
||||
searchInsights(query: string, limit?: number, filter?: InsightFilter): Promise<SearchResult[]>;
|
||||
deleteInsight(id: string): Promise<boolean>;
|
||||
|
||||
// Embedding provider (optional, null = no vector search)
|
||||
readonly embedder?: EmbeddingProvider | null;
|
||||
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface NewInsight {
|
||||
id: string;
|
||||
userId: string;
|
||||
content: string;
|
||||
embedding?: number[]; // If embedder is available
|
||||
source: 'agent' | 'user' | 'summarization' | 'system';
|
||||
category: 'decision' | 'learning' | 'preference' | 'fact' | 'pattern' | 'general';
|
||||
relevanceScore: number;
|
||||
metadata?: Record<string, unknown>;
|
||||
createdAt: Date;
|
||||
decayedAt?: Date;
|
||||
}
|
||||
|
||||
export interface InsightFilter {
|
||||
userId?: string;
|
||||
category?: string;
|
||||
source?: string;
|
||||
minRelevance?: number;
|
||||
fromDate?: Date;
|
||||
toDate?: Date;
|
||||
}
|
||||
|
||||
export interface SearchResult {
|
||||
documentId: string;
|
||||
content: string;
|
||||
distance: number;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface MemoryConfig {
|
||||
type: 'pgvector' | 'sqlite-vec' | 'keyword';
|
||||
storage: StorageAdapter;
|
||||
embedder?: EmbeddingProvider;
|
||||
}
|
||||
|
||||
export interface EmbeddingProvider {
|
||||
embed(text: string): Promise<number[]>;
|
||||
embedBatch(texts: string[]): Promise<number[][]>;
|
||||
readonly dimensions: number;
|
||||
}
|
||||
```
|
||||
|
||||
## Three Tiers
|
||||
|
||||
### Tier 1: Local (Zero Dependencies)
|
||||
|
||||
**Target:** Single user, single machine, no external services
|
||||
|
||||
| Component | Backend | Storage |
|
||||
| --------- | --------------------------------------------- | ------------ |
|
||||
| Queue | In-process + JSON files in `~/.mosaic/queue/` |
|
||||
| Storage | SQLite (better-sqlite3) `~/.mosaic/data.db` |
|
||||
| Memory | Keyword search | SQLite table |
|
||||
| Vector | None | N/A |
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
- `better-sqlite3` (bundled)
|
||||
- No Postgres, No Valkey, No pgvector
|
||||
|
||||
**Upgrade path:**
|
||||
|
||||
1. Run `mosaic gateway configure` → select "local" tier
|
||||
2. Gateway starts with SQLite database
|
||||
3. Optional: run `mosaic gateway upgrade --tier team` to migrate to Postgres
|
||||
|
||||
### Tier 2: Team (Postgres + Valkey)
|
||||
|
||||
**Target:** Multiple users, shared server, CI/CD environments
|
||||
|
||||
| Component | Backend | Storage |
|
||||
| --------- | -------------- | ------------------------------ |
|
||||
| Queue | BullMQ | Valkey |
|
||||
| Storage | Postgres | Shared PG instance |
|
||||
| Memory | pgvector | Postgres with vector extension |
|
||||
| Vector | LLM embeddings | Configured provider |
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
- PostgreSQL 17+ with pgvector extension
|
||||
- Valkey (Redis-compatible)
|
||||
- LLM provider for embeddings
|
||||
|
||||
**Migration from Local → Team:**
|
||||
|
||||
1. `mosaic gateway backup` → creates dump of SQLite database
|
||||
2. `mosaic gateway upgrade --tier team` → restores to Postgres
|
||||
3. Queue replays from BullMQ (may need manual reconciliation for in-flight jobs)
|
||||
4. Memory embeddings regenerated if vector search was new
|
||||
|
||||
### Tier 3: Enterprise (Clustered)
|
||||
|
||||
**Target:** Large teams, multi-region, high availability
|
||||
|
||||
| Component | Backend | Storage |
|
||||
| --------- | --------------------------- | ----------------------------- |
|
||||
| Queue | BullMQ cluster | Multiple Valkey nodes |
|
||||
| Storage | Postgres cluster | Primary + replicas |
|
||||
| Memory | Dedicated vector DB | Qdrant, Pinecone, or pgvector |
|
||||
| Vector | Dedicated embedding service | Separate microservice |
|
||||
|
||||
## MarkdownDB Integration
|
||||
|
||||
For file-based storage, we use [MarkdownDB](https://markdowndb.com) to parse MD files into queryable data.
|
||||
|
||||
**What it provides:**
|
||||
|
||||
- Parses frontmatter (YAML/JSON/TOML)
|
||||
- Extracts links, tags, metadata
|
||||
- Builds index in JSON or SQLite
|
||||
- Queryable via SQL-like interface
|
||||
|
||||
**Usage in Mosaic:**
|
||||
|
||||
```typescript
|
||||
// Local tier with MD files for documents
|
||||
const storage = createStorageAdapter({
|
||||
type: 'files',
|
||||
path: path.join(mosaicHome, 'docs'),
|
||||
markdowndb: {
|
||||
parseFrontmatter: true,
|
||||
extractLinks: true,
|
||||
indexFile: 'index.json',
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Dream Mode — Memory Consolidation
|
||||
|
||||
Automated equivalent to Claude Code's "Dream: Memory Consolidation" cycle
|
||||
|
||||
**Trigger:** Every 24 hours (if 5+ sessions active)
|
||||
|
||||
**Phases:**
|
||||
|
||||
1. **Orient** — What happened, what's the current state
|
||||
- Scan recent session logs
|
||||
- Identify active tasks, missions, conversations
|
||||
- Calculate time window (last 24h)
|
||||
|
||||
2. **Gather** — Pull in relevant context
|
||||
- Load conversations, decisions, agent logs
|
||||
- Extract key interactions and outcomes
|
||||
- Identify patterns and learnings
|
||||
|
||||
3. **Consolidate** — Summarize and compress
|
||||
- Generate summary of the last 24h
|
||||
- Extract key decisions and their rationale
|
||||
- Identify recurring patterns
|
||||
- Compress verbose logs into concise insights
|
||||
|
||||
4. **Prune** — Archive and cleanup
|
||||
- Archive raw session files to dated folders
|
||||
- Delete redundant/temporary data
|
||||
- Update MEMORY.md with consolidated content
|
||||
- Update insight relevance scores
|
||||
|
||||
**Implementation:**
|
||||
|
||||
```typescript
|
||||
// In @mosaicstack/dream (new package)
|
||||
export async function runDreamCycle(config: DreamConfig): Promise<DreamResult> {
|
||||
const memory = await loadMemoryAdapter(config.storage);
|
||||
|
||||
// Orient
|
||||
const sessions = await memory.getRecentSessions(24 * 60 * 60 * 1000);
|
||||
if (sessions.length < 5) return { skipped: true, reason: 'insufficient_sessions' };
|
||||
|
||||
// Gather
|
||||
const context = await gatherContext(memory, sessions);
|
||||
|
||||
// Consolidate
|
||||
const consolidated = await consolidateWithLLM(context, config.llm);
|
||||
|
||||
// Prune
|
||||
await pruneArchivedData(memory, config.retention);
|
||||
|
||||
// Store consolidated insights
|
||||
await memory.storeInsights(consolidated.insights);
|
||||
|
||||
return {
|
||||
sessionsProcessed: sessions.length,
|
||||
insightsCreated: consolidated.insights.length,
|
||||
bytesPruned: consolidated.bytesRemoved,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retrofit Plan
|
||||
|
||||
### Phase 1: Interface Extraction (2-3 days)
|
||||
|
||||
**Goal:** Define interfaces without changing existing behavior
|
||||
|
||||
1. Create `packages/queue/src/types.ts` with `QueueAdapter` interface
|
||||
2. Create `packages/storage/src/types.ts` with `StorageAdapter` interface
|
||||
3. Create `packages/memory/src/types.ts` with `MemoryAdapter` interface (refactor existing)
|
||||
4. Add adapter registry pattern to each package
|
||||
5. No breaking changes — existing code continues to work
|
||||
|
||||
### Phase 2: Refactor Existing to Adapters (3-5 days)
|
||||
|
||||
**Goal:** Move existing implementations behind adapters
|
||||
|
||||
#### 2.1 Queue Refactor
|
||||
|
||||
1. Rename `packages/queue/src/queue.ts` → `packages/queue/src/adapters/bullmq.ts`
|
||||
2. Create `packages/queue/src/index.ts` to export factory function
|
||||
3. Factory function reads config, instantiates correct adapter
|
||||
4. Update gateway imports to use factory
|
||||
|
||||
#### 2.2 Storage Refactor
|
||||
|
||||
1. Create `packages/storage/` (new package)
|
||||
2. Move Drizzle logic to `packages/storage/src/adapters/postgres.ts`
|
||||
3. Create SQLite adapter in `packages/storage/src/adapters/sqlite.ts`
|
||||
4. Update gateway to use storage factory
|
||||
5. Deprecate direct `@mosaicstack/db` imports
|
||||
|
||||
#### 2.3 Memory Refactor
|
||||
|
||||
1. Extract existing logic to `packages/memory/src/adapters/pgvector.ts`
|
||||
2. Create keyword adapter in `packages/memory/src/adapters/keyword.ts`
|
||||
3. Update vector-store.ts to be adapter-agnostic
|
||||
|
||||
### Phase 3: Local Tier Implementation (2-3 days)
|
||||
|
||||
**Goal:** Zero-dependency baseline
|
||||
|
||||
1. Implement `packages/queue/src/adapters/local.ts` (in-process + JSON persistence)
|
||||
2. Implement `packages/storage/src/adapters/files.ts` (JSON + MD via MarkdownDB)
|
||||
3. Implement `packages/memory/src/adapters/keyword.ts` (TF-IDF search)
|
||||
4. Add `packages/dream/` for consolidation cycle
|
||||
5. Wire up local tier in gateway startup
|
||||
|
||||
### Phase 4: Configuration System (1-2 days)
|
||||
|
||||
**Goal:** Runtime backend selection
|
||||
|
||||
1. Create `packages/config/src/storage.ts` for storage configuration
|
||||
2. Add `mosaic.config.ts` schema with storage tier settings
|
||||
3. Update gateway to read config on startup
|
||||
4. Add `mosaic gateway configure` CLI command
|
||||
5. Add tier migration commands (`mosaic gateway upgrade`)
|
||||
|
||||
### Phase 5: Testing & Documentation (2-3 days)
|
||||
|
||||
1. Unit tests for each adapter
|
||||
2. Integration tests for factory pattern
|
||||
3. Migration tests (local → team)
|
||||
4. Update README and architecture docs
|
||||
5. Add configuration guide
|
||||
|
||||
---
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
### New Files
|
||||
|
||||
```
|
||||
packages/
|
||||
├── config/
|
||||
│ └── src/
|
||||
│ ├── storage.ts # Storage config schema
|
||||
│ └── index.ts
|
||||
├── dream/ # NEW: Dream mode consolidation
|
||||
│ ├── src/
|
||||
│ │ ├── index.ts
|
||||
│ │ ├── orient.ts
|
||||
│ │ ├── gather.ts
|
||||
│ │ ├── consolidate.ts
|
||||
│ │ └── prune.ts
|
||||
│ └── package.json
|
||||
├── queue/
|
||||
│ └── src/
|
||||
│ ├── types.ts # NEW: QueueAdapter interface
|
||||
│ ├── index.ts # NEW: Factory function
|
||||
│ └── adapters/
|
||||
│ ├── bullmq.ts # MOVED from queue.ts
|
||||
│ └── local.ts # NEW: In-process adapter
|
||||
├── storage/ # NEW: Storage abstraction
|
||||
│ ├── src/
|
||||
│ │ ├── types.ts # StorageAdapter interface
|
||||
│ │ ├── index.ts # Factory function
|
||||
│ │ └── adapters/
|
||||
│ │ ├── postgres.ts # MOVED from @mosaicstack/db
|
||||
│ │ ├── sqlite.ts # NEW: SQLite adapter
|
||||
│ │ └── files.ts # NEW: JSON/MD adapter
|
||||
│ └── package.json
|
||||
└── memory/
|
||||
└── src/
|
||||
├── types.ts # UPDATED: MemoryAdapter interface
|
||||
├── index.ts # UPDATED: Factory function
|
||||
└── adapters/
|
||||
├── pgvector.ts # EXTRACTED from existing code
|
||||
├── sqlite-vec.ts # NEW: SQLite with vectors
|
||||
└── keyword.ts # NEW: TF-IDF search
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
```
|
||||
packages/
|
||||
├── db/ # DEPRECATED: Logic moved to storage adapters
|
||||
├── queue/
|
||||
│ └── src/
|
||||
│ └── queue.ts # → adapters/bullmq.ts
|
||||
├── memory/
|
||||
│ ├── src/
|
||||
│ │ ├── memory.ts # → use factory
|
||||
│ │ ├── insights.ts # → use factory
|
||||
│ │ └── preferences.ts # → use factory
|
||||
│ └── package.json # Remove pgvector from dependencies
|
||||
└── gateway/
|
||||
└── src/
|
||||
├── database/
|
||||
│ └── database.module.ts # Update to use storage factory
|
||||
├── memory/
|
||||
│ └── memory.module.ts # Update to use memory factory
|
||||
└── queue/
|
||||
└── queue.module.ts # Update to use queue factory
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
1. **`@mosaicstack/db`** → **`@mosaicstack/storage`** (with migration guide)
|
||||
2. Direct `ioredis` imports → Use `@mosaicstack/queue` factory
|
||||
3. Direct `pgvector` queries → Use `@mosaicstack/memory` factory
|
||||
4. Gateway startup now requires storage config (defaults to local)
|
||||
|
||||
## Non-Breaking Migration Path
|
||||
|
||||
1. Existing deployments with Postgres/Valkey continue to work (default config)
|
||||
2. New deployments can choose local tier
|
||||
3. Migration commands available when ready to upgrade
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Local tier runs with zero external dependencies
|
||||
- [ ] All three tiers (local, team, enterprise) work correctly
|
||||
- [ ] Factory pattern correctly selects backend at runtime
|
||||
- [ ] Migration from local → team preserves all data
|
||||
- [ ] Dream mode consolidates 24h of sessions
|
||||
- [ ] Documentation covers all three tiers and migration paths
|
||||
- [ ] All existing tests pass
|
||||
- [ ] New adapters have >80% coverage
|
||||
@@ -0,0 +1,98 @@
|
||||
# Gateway Security Hardening Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Finish the requested gateway security hardening fixes in the existing `fix/gateway-security` worktree and produce a PR-ready branch.
|
||||
|
||||
**Architecture:** Tighten NestJS gateway boundaries in-place by enforcing auth guards, session validation, ownership checks, DTO validation, and Fastify security defaults. Preserve the current module structure and existing ESM import conventions.
|
||||
|
||||
**Tech Stack:** NestJS 11, Fastify, Socket.IO, Better Auth, class-validator, Vitest, pnpm, TypeScript ESM
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Reconcile Security Tests
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `apps/gateway/src/chat/__tests__/chat-security.test.ts`
|
||||
- Modify: `apps/gateway/src/__tests__/resource-ownership.test.ts`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
- Encode the requested DTO constraints and socket-auth contract exactly.
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/gateway test -- src/chat/__tests__/chat-security.test.ts src/__tests__/resource-ownership.test.ts`
|
||||
|
||||
Expected: FAIL on current DTO/helper mismatch.
|
||||
|
||||
**Step 3: Write minimal implementation**
|
||||
|
||||
- Update DTO/helper/controller code only where tests prove a gap.
|
||||
|
||||
**Step 4: Run test to verify it passes**
|
||||
|
||||
Run the same command and require green.
|
||||
|
||||
### Task 2: Align Gateway Runtime Hardening
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `apps/gateway/src/conversations/conversations.dto.ts`
|
||||
- Modify: `apps/gateway/src/chat/chat.dto.ts`
|
||||
- Modify: `apps/gateway/src/chat/chat.gateway-auth.ts`
|
||||
- Modify: `apps/gateway/src/chat/chat.gateway.ts`
|
||||
- Modify: `apps/gateway/src/main.ts`
|
||||
- Modify: `apps/gateway/src/app.module.ts`
|
||||
|
||||
**Step 1: Verify remaining requested deltas**
|
||||
|
||||
- Confirm code matches requested guard, rate limit, helmet, body limit, env validation, and CORS settings.
|
||||
|
||||
**Step 2: Apply minimal patch**
|
||||
|
||||
- Keep changes scoped to requested behavior only.
|
||||
|
||||
**Step 3: Run targeted tests**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/gateway test -- src/chat/__tests__/chat-security.test.ts src/__tests__/resource-ownership.test.ts`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
### Task 3: Verification, Review, and Delivery
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/reports/code-review/gateway-security-20260313.md`
|
||||
- Create: `docs/reports/qa/gateway-security-20260313.md`
|
||||
- Modify: `docs/scratchpads/gateway-security-20260313.md`
|
||||
|
||||
**Step 1: Run baseline gates**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
pnpm typecheck
|
||||
pnpm lint
|
||||
```
|
||||
|
||||
**Step 2: Perform manual code review**
|
||||
|
||||
- Record correctness/security/testing/doc findings.
|
||||
|
||||
**Step 3: Commit and publish**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(gateway): security hardening — auth guards, ownership checks, validation, rate limiting"
|
||||
git push origin fix/gateway-security
|
||||
```
|
||||
|
||||
**Step 4: Open PR and notify**
|
||||
|
||||
- Open PR titled `fix(gateway): security hardening — auth guards, ownership checks, validation, rate limiting`
|
||||
- Run `openclaw system event --text "PR ready: mosaic-mono-v1 fix/gateway-security — 7 security fixes" --mode now`
|
||||
- Remove worktree after PR is created.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,998 @@
|
||||
# Wave 2 — TUI Layout & Navigation Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Add conversation sidebar, keybindings, scrollable message history, and search to the Mosaic TUI.
|
||||
|
||||
**Architecture:** The TUI gains a sidebar panel for conversation management (list/create/switch) fetched via REST from the gateway. A `useConversations` hook manages REST calls. A `useScrollableViewport` hook wraps the message list with virtual viewport logic. An app-level focus/mode state machine (`useAppMode`) controls which panel receives input. All new socket events for conversation listing use the existing REST API (`GET /api/conversations`).
|
||||
|
||||
**Tech Stack:** Ink 5, React 18, socket.io-client, fetch (for REST), @mosaicstack/types
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
TUI-010 (scrollable history) ← TUI-011 (search)
|
||||
TUI-008 (sidebar) ← TUI-009 (keybindings)
|
||||
```
|
||||
|
||||
TUI-008 and TUI-010 are independent — can be built in parallel.
|
||||
TUI-009 depends on TUI-008. TUI-011 depends on TUI-010.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: TUI-010 — Scrollable Message History
|
||||
|
||||
### 1A: Create `use-viewport` hook
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/cli/src/tui/hooks/use-viewport.ts`
|
||||
|
||||
**Step 1: Write the hook**
|
||||
|
||||
This hook tracks a scroll offset and viewport height for the message list.
|
||||
Ink's `useStdout` gives us terminal rows. We calculate visible slice.
|
||||
|
||||
```ts
|
||||
import { useState, useCallback, useMemo } from 'react';
|
||||
import { useStdout } from 'ink';
|
||||
|
||||
export interface UseViewportOptions {
|
||||
/** Total number of renderable lines (message count as proxy) */
|
||||
totalItems: number;
|
||||
/** Lines reserved for chrome (top bar, input bar, bottom bar) */
|
||||
reservedLines?: number;
|
||||
}
|
||||
|
||||
export interface UseViewportReturn {
|
||||
/** Index of first visible item (0-based) */
|
||||
scrollOffset: number;
|
||||
/** Number of items that fit in viewport */
|
||||
viewportSize: number;
|
||||
/** Whether user has scrolled up from bottom */
|
||||
isScrolledUp: boolean;
|
||||
/** Scroll to bottom (auto-follow mode) */
|
||||
scrollToBottom: () => void;
|
||||
/** Scroll by delta (negative = up, positive = down) */
|
||||
scrollBy: (delta: number) => void;
|
||||
/** Scroll to a specific offset */
|
||||
scrollTo: (offset: number) => void;
|
||||
/** Whether we can scroll up/down */
|
||||
canScrollUp: boolean;
|
||||
canScrollDown: boolean;
|
||||
}
|
||||
|
||||
export function useViewport(opts: UseViewportOptions): UseViewportReturn {
|
||||
const { totalItems, reservedLines = 10 } = opts;
|
||||
const { stdout } = useStdout();
|
||||
const terminalRows = stdout?.rows ?? 24;
|
||||
|
||||
// Viewport = terminal height minus chrome
|
||||
const viewportSize = Math.max(1, terminalRows - reservedLines);
|
||||
|
||||
const maxOffset = Math.max(0, totalItems - viewportSize);
|
||||
|
||||
const [scrollOffset, setScrollOffset] = useState(0);
|
||||
// Track if user explicitly scrolled up
|
||||
const [autoFollow, setAutoFollow] = useState(true);
|
||||
|
||||
// Effective offset: if auto-following, always show latest
|
||||
const effectiveOffset = autoFollow ? maxOffset : Math.min(scrollOffset, maxOffset);
|
||||
|
||||
const scrollBy = useCallback(
|
||||
(delta: number) => {
|
||||
setAutoFollow(false);
|
||||
setScrollOffset((prev) => {
|
||||
const next = Math.max(0, Math.min(prev + delta, maxOffset));
|
||||
// If scrolled to bottom, re-enable auto-follow
|
||||
if (next >= maxOffset) {
|
||||
setAutoFollow(true);
|
||||
}
|
||||
return next;
|
||||
});
|
||||
},
|
||||
[maxOffset],
|
||||
);
|
||||
|
||||
const scrollToBottom = useCallback(() => {
|
||||
setAutoFollow(true);
|
||||
setScrollOffset(maxOffset);
|
||||
}, [maxOffset]);
|
||||
|
||||
const scrollTo = useCallback(
|
||||
(offset: number) => {
|
||||
const clamped = Math.max(0, Math.min(offset, maxOffset));
|
||||
setAutoFollow(clamped >= maxOffset);
|
||||
setScrollOffset(clamped);
|
||||
},
|
||||
[maxOffset],
|
||||
);
|
||||
|
||||
return {
|
||||
scrollOffset: effectiveOffset,
|
||||
viewportSize,
|
||||
isScrolledUp: !autoFollow,
|
||||
scrollToBottom,
|
||||
scrollBy,
|
||||
scrollTo,
|
||||
canScrollUp: effectiveOffset > 0,
|
||||
canScrollDown: effectiveOffset < maxOffset,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Typecheck**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck`
|
||||
Expected: PASS
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/hooks/use-viewport.ts
|
||||
git commit -m "feat(cli): add use-viewport hook for scrollable message history"
|
||||
```
|
||||
|
||||
### 1B: Integrate viewport into MessageList and wire PgUp/PgDn
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/cli/src/tui/components/message-list.tsx`
|
||||
- Modify: `packages/cli/src/tui/app.tsx`
|
||||
|
||||
**Step 1: Update MessageList to accept viewport props and slice messages**
|
||||
|
||||
In `message-list.tsx`, add viewport props and render only the visible slice. Add a scroll indicator when scrolled up.
|
||||
|
||||
```tsx
|
||||
// Add to MessageListProps:
|
||||
export interface MessageListProps {
|
||||
messages: Message[];
|
||||
isStreaming: boolean;
|
||||
currentStreamText: string;
|
||||
currentThinkingText: string;
|
||||
activeToolCalls: ToolCall[];
|
||||
// New viewport props
|
||||
scrollOffset: number;
|
||||
viewportSize: number;
|
||||
isScrolledUp: boolean;
|
||||
}
|
||||
```
|
||||
|
||||
In the component body, slice messages:
|
||||
|
||||
```tsx
|
||||
const visibleMessages = messages.slice(scrollOffset, scrollOffset + viewportSize);
|
||||
```
|
||||
|
||||
Replace `messages.map(...)` with `visibleMessages.map(...)`. Add a scroll-up indicator at the top:
|
||||
|
||||
```tsx
|
||||
{
|
||||
isScrolledUp && (
|
||||
<Box justifyContent="center">
|
||||
<Text dimColor>↑ {scrollOffset} more messages ↑</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Wire viewport hook + keybindings in app.tsx**
|
||||
|
||||
In `app.tsx`:
|
||||
|
||||
1. Import and call `useViewport({ totalItems: socket.messages.length })`
|
||||
2. Pass viewport props to `<MessageList>`
|
||||
3. Add PgUp/PgDn/Home/End keybindings in the existing `useInput`:
|
||||
- `key.pageUp` → `viewport.scrollBy(-viewport.viewportSize)`
|
||||
- `key.pageDown` → `viewport.scrollBy(viewport.viewportSize)`
|
||||
- Shift+Up → `viewport.scrollBy(-1)` (line scroll)
|
||||
- Shift+Down → `viewport.scrollBy(1)` (line scroll)
|
||||
|
||||
Note: Ink's `useInput` key object supports `pageUp`, `pageDown`. For Home/End, check `key.meta && ch === '<'` / `key.meta && ch === '>'` as Ink doesn't have built-in home/end.
|
||||
|
||||
**Step 3: Typecheck and lint**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck && pnpm --filter @mosaicstack/cli lint`
|
||||
Expected: PASS
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/components/message-list.tsx packages/cli/src/tui/app.tsx
|
||||
git commit -m "feat(cli): scrollable message history with PgUp/PgDn viewport"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: TUI-008 — Conversation Sidebar
|
||||
|
||||
### 2A: Create `use-conversations` hook (REST client)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/cli/src/tui/hooks/use-conversations.ts`
|
||||
|
||||
**Step 1: Write the hook**
|
||||
|
||||
This hook fetches conversations from the gateway REST API and provides create/switch actions.
|
||||
|
||||
```ts
|
||||
import { useState, useEffect, useCallback, useRef } from 'react';
|
||||
|
||||
export interface ConversationSummary {
|
||||
id: string;
|
||||
title: string | null;
|
||||
archived: boolean;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface UseConversationsOptions {
|
||||
gatewayUrl: string;
|
||||
sessionCookie?: string;
|
||||
/** Currently active conversation ID from socket */
|
||||
activeConversationId: string | undefined;
|
||||
}
|
||||
|
||||
export interface UseConversationsReturn {
|
||||
conversations: ConversationSummary[];
|
||||
loading: boolean;
|
||||
error: string | null;
|
||||
refresh: () => Promise<void>;
|
||||
createConversation: (title?: string) => Promise<ConversationSummary | null>;
|
||||
deleteConversation: (id: string) => Promise<boolean>;
|
||||
renameConversation: (id: string, title: string) => Promise<boolean>;
|
||||
}
|
||||
|
||||
export function useConversations(opts: UseConversationsOptions): UseConversationsReturn {
|
||||
const { gatewayUrl, sessionCookie } = opts;
|
||||
const [conversations, setConversations] = useState<ConversationSummary[]>([]);
|
||||
const [loading, setLoading] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const mountedRef = useRef(true);
|
||||
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': 'application/json',
|
||||
...(sessionCookie ? { Cookie: sessionCookie } : {}),
|
||||
};
|
||||
|
||||
const refresh = useCallback(async () => {
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
try {
|
||||
const res = await fetch(`${gatewayUrl}/api/conversations`, { headers });
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const data = (await res.json()) as ConversationSummary[];
|
||||
if (mountedRef.current) {
|
||||
setConversations(data);
|
||||
}
|
||||
} catch (err) {
|
||||
if (mountedRef.current) {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
if (mountedRef.current) setLoading(false);
|
||||
}
|
||||
}, [gatewayUrl, sessionCookie]);
|
||||
|
||||
const createConversation = useCallback(
|
||||
async (title?: string): Promise<ConversationSummary | null> => {
|
||||
try {
|
||||
const res = await fetch(`${gatewayUrl}/api/conversations`, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: JSON.stringify({ title: title ?? 'New Conversation' }),
|
||||
});
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const conv = (await res.json()) as ConversationSummary;
|
||||
await refresh();
|
||||
return conv;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
},
|
||||
[gatewayUrl, sessionCookie, refresh],
|
||||
);
|
||||
|
||||
const deleteConversation = useCallback(
|
||||
async (id: string): Promise<boolean> => {
|
||||
try {
|
||||
const res = await fetch(`${gatewayUrl}/api/conversations/${id}`, {
|
||||
method: 'DELETE',
|
||||
headers,
|
||||
});
|
||||
if (!res.ok) return false;
|
||||
await refresh();
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
[gatewayUrl, sessionCookie, refresh],
|
||||
);
|
||||
|
||||
const renameConversation = useCallback(
|
||||
async (id: string, title: string): Promise<boolean> => {
|
||||
try {
|
||||
const res = await fetch(`${gatewayUrl}/api/conversations/${id}`, {
|
||||
method: 'PATCH',
|
||||
headers,
|
||||
body: JSON.stringify({ title }),
|
||||
});
|
||||
if (!res.ok) return false;
|
||||
await refresh();
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
[gatewayUrl, sessionCookie, refresh],
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
mountedRef.current = true;
|
||||
void refresh();
|
||||
return () => {
|
||||
mountedRef.current = false;
|
||||
};
|
||||
}, []);
|
||||
|
||||
return {
|
||||
conversations,
|
||||
loading,
|
||||
error,
|
||||
refresh,
|
||||
createConversation,
|
||||
deleteConversation,
|
||||
renameConversation,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Typecheck**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck`
|
||||
Expected: PASS
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/hooks/use-conversations.ts
|
||||
git commit -m "feat(cli): add use-conversations hook for REST conversation management"
|
||||
```
|
||||
|
||||
### 2B: Create `use-app-mode` hook (focus/mode state machine)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/cli/src/tui/hooks/use-app-mode.ts`
|
||||
|
||||
**Step 1: Write the hook**
|
||||
|
||||
This manages which panel has focus and the current UI mode.
|
||||
|
||||
```ts
|
||||
import { useState, useCallback } from 'react';
|
||||
|
||||
export type AppMode = 'chat' | 'sidebar' | 'search';
|
||||
|
||||
export interface UseAppModeReturn {
|
||||
mode: AppMode;
|
||||
setMode: (mode: AppMode) => void;
|
||||
toggleSidebar: () => void;
|
||||
/** Whether sidebar panel should be visible */
|
||||
sidebarOpen: boolean;
|
||||
}
|
||||
|
||||
export function useAppMode(): UseAppModeReturn {
|
||||
const [mode, setModeState] = useState<AppMode>('chat');
|
||||
const [sidebarOpen, setSidebarOpen] = useState(false);
|
||||
|
||||
const setMode = useCallback((m: AppMode) => {
|
||||
setModeState(m);
|
||||
if (m === 'sidebar') setSidebarOpen(true);
|
||||
}, []);
|
||||
|
||||
const toggleSidebar = useCallback(() => {
|
||||
setSidebarOpen((prev) => {
|
||||
const next = !prev;
|
||||
if (!next) {
|
||||
// Closing sidebar → return to chat mode
|
||||
setModeState('chat');
|
||||
} else {
|
||||
setModeState('sidebar');
|
||||
}
|
||||
return next;
|
||||
});
|
||||
}, []);
|
||||
|
||||
return { mode, setMode, toggleSidebar, sidebarOpen };
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Typecheck**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck`
|
||||
Expected: PASS
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/hooks/use-app-mode.ts
|
||||
git commit -m "feat(cli): add use-app-mode hook for panel focus state machine"
|
||||
```
|
||||
|
||||
### 2C: Create `Sidebar` component
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/cli/src/tui/components/sidebar.tsx`
|
||||
|
||||
**Step 1: Write the component**
|
||||
|
||||
The sidebar shows a scrollable list of conversations with the active one highlighted. It handles keyboard navigation when focused.
|
||||
|
||||
```tsx
|
||||
import React from 'react';
|
||||
import { Box, Text, useInput } from 'ink';
|
||||
import type { ConversationSummary } from '../hooks/use-conversations.js';
|
||||
|
||||
export interface SidebarProps {
|
||||
conversations: ConversationSummary[];
|
||||
activeConversationId: string | undefined;
|
||||
selectedIndex: number;
|
||||
onSelectIndex: (index: number) => void;
|
||||
onSwitchConversation: (id: string) => void;
|
||||
onDeleteConversation: (id: string) => void;
|
||||
loading: boolean;
|
||||
focused: boolean;
|
||||
width: number;
|
||||
}
|
||||
|
||||
function truncate(str: string, maxLen: number): string {
|
||||
if (str.length <= maxLen) return str;
|
||||
return str.slice(0, maxLen - 1) + '…';
|
||||
}
|
||||
|
||||
function formatDate(iso: string): string {
|
||||
const d = new Date(iso);
|
||||
const now = new Date();
|
||||
const diffMs = now.getTime() - d.getTime();
|
||||
const diffDays = Math.floor(diffMs / (1000 * 60 * 60 * 24));
|
||||
if (diffDays === 0) {
|
||||
return d.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: false });
|
||||
}
|
||||
if (diffDays < 7) return `${diffDays}d ago`;
|
||||
return d.toLocaleDateString('en-US', { month: 'short', day: 'numeric' });
|
||||
}
|
||||
|
||||
export function Sidebar({
|
||||
conversations,
|
||||
activeConversationId,
|
||||
selectedIndex,
|
||||
onSelectIndex,
|
||||
onSwitchConversation,
|
||||
onDeleteConversation,
|
||||
loading,
|
||||
focused,
|
||||
width,
|
||||
}: SidebarProps) {
|
||||
useInput(
|
||||
(ch, key) => {
|
||||
if (!focused) return;
|
||||
|
||||
if (key.upArrow) {
|
||||
onSelectIndex(Math.max(0, selectedIndex - 1));
|
||||
} else if (key.downArrow) {
|
||||
onSelectIndex(Math.min(conversations.length - 1, selectedIndex + 1));
|
||||
} else if (key.return) {
|
||||
const conv = conversations[selectedIndex];
|
||||
if (conv) onSwitchConversation(conv.id);
|
||||
} else if (ch === 'd' || ch === 'D') {
|
||||
const conv = conversations[selectedIndex];
|
||||
if (conv && conv.id !== activeConversationId) {
|
||||
onDeleteConversation(conv.id);
|
||||
}
|
||||
}
|
||||
},
|
||||
{ isActive: focused },
|
||||
);
|
||||
|
||||
const titleWidth = width - 4; // padding + borders
|
||||
|
||||
return (
|
||||
<Box
|
||||
flexDirection="column"
|
||||
width={width}
|
||||
borderStyle="single"
|
||||
borderColor={focused ? 'cyan' : 'gray'}
|
||||
>
|
||||
<Box paddingX={1}>
|
||||
<Text bold color={focused ? 'cyan' : undefined}>
|
||||
Conversations
|
||||
</Text>
|
||||
{loading && <Text dimColor> …</Text>}
|
||||
</Box>
|
||||
|
||||
{conversations.length === 0 && (
|
||||
<Box paddingX={1}>
|
||||
<Text dimColor>No conversations</Text>
|
||||
</Box>
|
||||
)}
|
||||
|
||||
{conversations.map((conv, i) => {
|
||||
const isActive = conv.id === activeConversationId;
|
||||
const isSelected = i === selectedIndex && focused;
|
||||
const title = conv.title ?? `Untitled (${conv.id.slice(0, 6)})`;
|
||||
const displayTitle = truncate(title, titleWidth);
|
||||
|
||||
return (
|
||||
<Box key={conv.id} paddingX={1}>
|
||||
<Text
|
||||
bold={isActive}
|
||||
color={isSelected ? 'cyan' : isActive ? 'green' : undefined}
|
||||
inverse={isSelected}
|
||||
>
|
||||
{isActive ? '● ' : ' '}
|
||||
{displayTitle}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
})}
|
||||
|
||||
{focused && (
|
||||
<Box paddingX={1} marginTop={1}>
|
||||
<Text dimColor>↑↓ navigate · ↵ switch · d delete</Text>
|
||||
</Box>
|
||||
)}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Typecheck**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck`
|
||||
Expected: PASS
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/components/sidebar.tsx
|
||||
git commit -m "feat(cli): add conversation sidebar component"
|
||||
```
|
||||
|
||||
### 2D: Wire sidebar + conversation switching into app.tsx
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/cli/src/tui/app.tsx`
|
||||
- Modify: `packages/cli/src/tui/hooks/use-socket.ts`
|
||||
|
||||
**Step 1: Add `switchConversation` to useSocket**
|
||||
|
||||
In `use-socket.ts`, add a method to switch conversations. When switching, clear local messages and set the new conversation ID. The socket will pick up the new conversation on next `message` emit.
|
||||
|
||||
Add to `UseSocketReturn`:
|
||||
|
||||
```ts
|
||||
switchConversation: (id: string) => void;
|
||||
clearMessages: () => void;
|
||||
```
|
||||
|
||||
Implementation:
|
||||
|
||||
```ts
|
||||
const switchConversation = useCallback((id: string) => {
|
||||
setConversationId(id);
|
||||
setMessages([]);
|
||||
setIsStreaming(false);
|
||||
setCurrentStreamText('');
|
||||
setCurrentThinkingText('');
|
||||
setActiveToolCalls([]);
|
||||
}, []);
|
||||
|
||||
const clearMessages = useCallback(() => {
|
||||
setMessages([]);
|
||||
}, []);
|
||||
```
|
||||
|
||||
**Step 2: Update app.tsx layout to include sidebar**
|
||||
|
||||
1. Import `useAppMode`, `useConversations`, `Sidebar`
|
||||
2. Add `useAppMode()` call
|
||||
3. Add `useConversations({ gatewayUrl, sessionCookie, activeConversationId: socket.conversationId })`
|
||||
4. Track `sidebarSelectedIndex` state
|
||||
5. Wrap the main content area in a horizontal `<Box>`:
|
||||
|
||||
```tsx
|
||||
<Box flexDirection="row" flexGrow={1}>
|
||||
{appMode.sidebarOpen && (
|
||||
<Sidebar
|
||||
conversations={convos.conversations}
|
||||
activeConversationId={socket.conversationId}
|
||||
selectedIndex={sidebarSelectedIndex}
|
||||
onSelectIndex={setSidebarSelectedIndex}
|
||||
onSwitchConversation={(id) => {
|
||||
socket.switchConversation(id);
|
||||
appMode.setMode('chat');
|
||||
}}
|
||||
onDeleteConversation={(id) => void convos.deleteConversation(id)}
|
||||
loading={convos.loading}
|
||||
focused={appMode.mode === 'sidebar'}
|
||||
width={30}
|
||||
/>
|
||||
)}
|
||||
<Box flexDirection="column" flexGrow={1}>
|
||||
<MessageList ... />
|
||||
</Box>
|
||||
</Box>
|
||||
```
|
||||
|
||||
6. InputBar should be disabled (or readonly placeholder) when mode is not 'chat'
|
||||
|
||||
**Step 3: Typecheck and lint**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck && pnpm --filter @mosaicstack/cli lint`
|
||||
Expected: PASS
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/app.tsx packages/cli/src/tui/hooks/use-socket.ts
|
||||
git commit -m "feat(cli): wire conversation sidebar with create/switch/delete"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: TUI-009 — Keybinding System
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/cli/src/tui/app.tsx`
|
||||
|
||||
**Step 1: Add global keybindings in the existing useInput**
|
||||
|
||||
Add these bindings to the `useInput` in `app.tsx`:
|
||||
|
||||
| Binding | Action |
|
||||
| ----------- | ----------------------------------------- |
|
||||
| `Ctrl+L` | Toggle sidebar visibility |
|
||||
| `Ctrl+N` | Create new conversation + switch to it |
|
||||
| `Ctrl+K` | Toggle search mode (TUI-011) |
|
||||
| `Escape` | Return to chat mode from any panel |
|
||||
| `Ctrl+T` | Cycle thinking level (already exists) |
|
||||
| `PgUp/PgDn` | Scroll viewport (already wired in Task 1) |
|
||||
|
||||
```ts
|
||||
useInput((ch, key) => {
|
||||
if (key.ctrl && ch === 'c') {
|
||||
exit();
|
||||
return;
|
||||
}
|
||||
|
||||
// Global keybindings (work in any mode)
|
||||
if (key.ctrl && ch === 'l') {
|
||||
appMode.toggleSidebar();
|
||||
if (!appMode.sidebarOpen) void convos.refresh();
|
||||
return;
|
||||
}
|
||||
|
||||
if (key.ctrl && ch === 'n') {
|
||||
void convos.createConversation().then((conv) => {
|
||||
if (conv) {
|
||||
socket.switchConversation(conv.id);
|
||||
appMode.setMode('chat');
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (key.ctrl && ch === 'k') {
|
||||
appMode.setMode(appMode.mode === 'search' ? 'chat' : 'search');
|
||||
return;
|
||||
}
|
||||
|
||||
if (key.escape) {
|
||||
if (appMode.mode !== 'chat') {
|
||||
appMode.setMode('chat');
|
||||
return;
|
||||
}
|
||||
// In chat mode, Escape could scroll to bottom
|
||||
viewport.scrollToBottom();
|
||||
return;
|
||||
}
|
||||
|
||||
// Ctrl+T: cycle thinking (existing)
|
||||
if (key.ctrl && ch === 't') {
|
||||
const levels = socket.availableThinkingLevels;
|
||||
if (levels.length > 0) {
|
||||
const currentIdx = levels.indexOf(socket.thinkingLevel);
|
||||
const nextIdx = (currentIdx + 1) % levels.length;
|
||||
const next = levels[nextIdx];
|
||||
if (next) socket.setThinkingLevel(next);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Viewport scrolling (only in chat mode)
|
||||
if (appMode.mode === 'chat') {
|
||||
if (key.pageUp) {
|
||||
viewport.scrollBy(-viewport.viewportSize);
|
||||
} else if (key.pageDown) {
|
||||
viewport.scrollBy(viewport.viewportSize);
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Step 2: Add keybinding hints to bottom bar**
|
||||
|
||||
In `bottom-bar.tsx`, add a hints line above the status lines (or integrate into line 1):
|
||||
|
||||
```tsx
|
||||
<Text dimColor>^L sidebar · ^N new · ^K search · ^T thinking · PgUp/Dn scroll</Text>
|
||||
```
|
||||
|
||||
**Step 3: Typecheck and lint**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck && pnpm --filter @mosaicstack/cli lint`
|
||||
Expected: PASS
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/app.tsx packages/cli/src/tui/components/bottom-bar.tsx
|
||||
git commit -m "feat(cli): keybinding system — Ctrl+L sidebar, Ctrl+N new, Ctrl+K search, Escape"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: TUI-011 — Message Search
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/cli/src/tui/components/search-bar.tsx`
|
||||
- Create: `packages/cli/src/tui/hooks/use-search.ts`
|
||||
- Modify: `packages/cli/src/tui/app.tsx`
|
||||
- Modify: `packages/cli/src/tui/components/message-list.tsx`
|
||||
|
||||
### 4A: Create `use-search` hook
|
||||
|
||||
**Step 1: Write the hook**
|
||||
|
||||
```ts
|
||||
import { useState, useCallback, useMemo } from 'react';
|
||||
import type { Message } from './use-socket.js';
|
||||
|
||||
export interface SearchMatch {
|
||||
messageIndex: number;
|
||||
/** Character offset within message content */
|
||||
charOffset: number;
|
||||
}
|
||||
|
||||
export interface UseSearchReturn {
|
||||
query: string;
|
||||
setQuery: (q: string) => void;
|
||||
matches: SearchMatch[];
|
||||
currentMatchIndex: number;
|
||||
nextMatch: () => void;
|
||||
prevMatch: () => void;
|
||||
clear: () => void;
|
||||
/** Total match count */
|
||||
totalMatches: number;
|
||||
}
|
||||
|
||||
export function useSearch(messages: Message[]): UseSearchReturn {
|
||||
const [query, setQuery] = useState('');
|
||||
const [currentMatchIndex, setCurrentMatchIndex] = useState(0);
|
||||
|
||||
const matches = useMemo(() => {
|
||||
if (!query || query.length < 2) return [];
|
||||
const q = query.toLowerCase();
|
||||
const result: SearchMatch[] = [];
|
||||
for (let i = 0; i < messages.length; i++) {
|
||||
const content = messages[i]!.content.toLowerCase();
|
||||
let pos = 0;
|
||||
while ((pos = content.indexOf(q, pos)) !== -1) {
|
||||
result.push({ messageIndex: i, charOffset: pos });
|
||||
pos += q.length;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}, [query, messages]);
|
||||
|
||||
const nextMatch = useCallback(() => {
|
||||
if (matches.length === 0) return;
|
||||
setCurrentMatchIndex((prev) => (prev + 1) % matches.length);
|
||||
}, [matches.length]);
|
||||
|
||||
const prevMatch = useCallback(() => {
|
||||
if (matches.length === 0) return;
|
||||
setCurrentMatchIndex((prev) => (prev - 1 + matches.length) % matches.length);
|
||||
}, [matches.length]);
|
||||
|
||||
const clear = useCallback(() => {
|
||||
setQuery('');
|
||||
setCurrentMatchIndex(0);
|
||||
}, []);
|
||||
|
||||
return {
|
||||
query,
|
||||
setQuery,
|
||||
matches,
|
||||
currentMatchIndex: matches.length > 0 ? currentMatchIndex % matches.length : 0,
|
||||
nextMatch,
|
||||
prevMatch,
|
||||
clear,
|
||||
totalMatches: matches.length,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/hooks/use-search.ts
|
||||
git commit -m "feat(cli): add use-search hook for message search"
|
||||
```
|
||||
|
||||
### 4B: Create SearchBar component
|
||||
|
||||
**Step 1: Write the component**
|
||||
|
||||
```tsx
|
||||
import React from 'react';
|
||||
import { Box, Text } from 'ink';
|
||||
import TextInput from 'ink-text-input';
|
||||
|
||||
export interface SearchBarProps {
|
||||
query: string;
|
||||
onQueryChange: (q: string) => void;
|
||||
totalMatches: number;
|
||||
currentMatch: number;
|
||||
onNext: () => void;
|
||||
onPrev: () => void;
|
||||
onClose: () => void;
|
||||
focused: boolean;
|
||||
}
|
||||
|
||||
export function SearchBar({
|
||||
query,
|
||||
onQueryChange,
|
||||
totalMatches,
|
||||
currentMatch,
|
||||
onClose,
|
||||
focused,
|
||||
}: SearchBarProps) {
|
||||
return (
|
||||
<Box paddingX={1} borderStyle="single" borderColor={focused ? 'yellow' : 'gray'}>
|
||||
<Text color="yellow">🔍 </Text>
|
||||
<TextInput
|
||||
value={query}
|
||||
onChange={onQueryChange}
|
||||
placeholder="search messages…"
|
||||
focus={focused}
|
||||
/>
|
||||
<Box marginLeft={1}>
|
||||
{query.length >= 2 ? (
|
||||
<Text dimColor>
|
||||
{totalMatches > 0 ? `${currentMatch + 1}/${totalMatches}` : 'no matches'}
|
||||
{' · ↑↓ navigate · Esc close'}
|
||||
</Text>
|
||||
) : (
|
||||
<Text dimColor>type to search…</Text>
|
||||
)}
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/components/search-bar.tsx
|
||||
git commit -m "feat(cli): add search bar component"
|
||||
```
|
||||
|
||||
### 4C: Wire search into app.tsx and message-list
|
||||
|
||||
**Step 1: Integrate**
|
||||
|
||||
In `app.tsx`:
|
||||
|
||||
1. Import `useSearch` and `SearchBar`
|
||||
2. Call `useSearch(socket.messages)`
|
||||
3. When mode is 'search', render `<SearchBar>` above `<InputBar>`
|
||||
4. In search mode, Up/Down arrows call `search.nextMatch()`/`search.prevMatch()` and scroll the viewport to the matched message
|
||||
5. Pass `searchHighlights` to `MessageList` — the set of message indices that match
|
||||
|
||||
In `message-list.tsx`:
|
||||
|
||||
1. Add optional `highlightedMessageIndices?: Set<number>` and `currentHighlightIndex?: number` props
|
||||
2. Highlighted messages get a yellow left border or background tint
|
||||
3. The current match gets a brighter highlight
|
||||
|
||||
**Step 2: Typecheck and lint**
|
||||
|
||||
Run: `pnpm --filter @mosaicstack/cli typecheck && pnpm --filter @mosaicstack/cli lint`
|
||||
Expected: PASS
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/cli/src/tui/app.tsx packages/cli/src/tui/components/message-list.tsx packages/cli/src/tui/components/search-bar.tsx
|
||||
git commit -m "feat(cli): wire message search with highlight and viewport scroll"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Final Integration & Quality Gates
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/TASKS-TUI_Improvements.md` (update status)
|
||||
|
||||
**Step 1: Full typecheck across all affected packages**
|
||||
|
||||
```bash
|
||||
pnpm --filter @mosaicstack/cli typecheck && pnpm --filter @mosaicstack/cli lint
|
||||
pnpm --filter @mosaicstack/types typecheck
|
||||
```
|
||||
|
||||
Expected: All PASS
|
||||
|
||||
**Step 2: Manual smoke test (held)**
|
||||
|
||||
This historical TUI smoke test is unavailable until KBN-101-02 supplies a fail-closed Gateway local
|
||||
startup route. Do not start current Compose PostgreSQL or infer a local Gateway from PGlite support.
|
||||
A future reviewed test must use the correct Mosaic CLI package and an independently verified Gateway.
|
||||
|
||||
Verify:
|
||||
|
||||
- [ ] Messages scroll with PgUp/PgDn
|
||||
- [ ] Ctrl+L opens/closes sidebar
|
||||
- [ ] Sidebar shows conversations from REST API
|
||||
- [ ] Arrow keys navigate sidebar when focused
|
||||
- [ ] Enter switches conversation, clears messages
|
||||
- [ ] Ctrl+N creates new conversation
|
||||
- [ ] Ctrl+K opens search bar
|
||||
- [ ] Typing in search highlights matches
|
||||
- [ ] Up/Down in search mode cycles through matches
|
||||
- [ ] Escape returns to chat from any mode
|
||||
- [ ] Ctrl+T still cycles thinking levels
|
||||
- [ ] Auto-scroll follows new messages at bottom
|
||||
|
||||
**Step 3: Update task tracker**
|
||||
|
||||
Mark TUI-008, TUI-009, TUI-010, TUI-011 as ✅ done in `docs/TASKS-TUI_Improvements.md`
|
||||
|
||||
**Step 4: Commit and push**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "docs: mark Wave 2 tasks complete"
|
||||
git push
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Summary
|
||||
|
||||
| Action | Path |
|
||||
| ------ | -------------------------------------------------- |
|
||||
| Create | `packages/cli/src/tui/hooks/use-viewport.ts` |
|
||||
| Create | `packages/cli/src/tui/hooks/use-conversations.ts` |
|
||||
| Create | `packages/cli/src/tui/hooks/use-app-mode.ts` |
|
||||
| Create | `packages/cli/src/tui/hooks/use-search.ts` |
|
||||
| Create | `packages/cli/src/tui/components/sidebar.tsx` |
|
||||
| Create | `packages/cli/src/tui/components/search-bar.tsx` |
|
||||
| Modify | `packages/cli/src/tui/app.tsx` |
|
||||
| Modify | `packages/cli/src/tui/hooks/use-socket.ts` |
|
||||
| Modify | `packages/cli/src/tui/components/message-list.tsx` |
|
||||
| Modify | `packages/cli/src/tui/components/bottom-bar.tsx` |
|
||||
| Modify | `docs/TASKS-TUI_Improvements.md` |
|
||||
@@ -0,0 +1,238 @@
|
||||
# Hermes-Mosaic Alignment Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Package Mosaic's mechanical coordination primitives as a native Hermes toolset so any Hermes profile gets mission management, task decomposition, handoff, and session continuity without depending on the Mosaic gateway or OpenClaw runtime.
|
||||
|
||||
**Architecture:** Extract the coordination logic from Mosaic's `packages/coord` (TypeScript, file-first) into a Hermes Python toolset that wraps the same file conventions. The Mosaic Stack repo remains the canonical upstream for the file formats (TASKS.md schema, mission.json schema, handoff packet schema). Hermes implements native Python tools that read/write those same files, plus tool-calls for churn detection and handoff generation that have no Mosaic equivalent today.
|
||||
|
||||
**Tech Stack:** Python (Hermes toolset), SQLite (Hermes Kanban), JSON + Markdown (Mosaic file conventions)
|
||||
|
||||
---
|
||||
|
||||
## Alignment Map
|
||||
|
||||
### What Mosaic has that Hermes needs
|
||||
|
||||
| Mosaic Component | What it does | Natural Hermes home | Why |
|
||||
| -------------------------------- | --------------------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `packages/coord` (mission.ts) | Mission CRUD, session tracking, milestone state | **Hermes toolset: `mission`** | Mission state is session-scoped, not gateway-scoped. Hermes sessions already have identity, process tracking, and context windows. |
|
||||
| `packages/coord` (tasks-file.ts) | Parse/write TASKS.md tables | **Hermes toolset: `mission`** (same) | Hermes already reads/writes files. The TASKS.md parser is ~300 lines of pure string manipulation — trivial Python port. |
|
||||
| `packages/coord` (runner.ts) | Spawn claude/codex workers with continuation prompts | **Already covered by `delegate_task`** | Hermes delegate_task already does isolated subagent spawning with restricted toolsets. The runner's "find next task and build continuation prompt" logic moves into a tool-call. |
|
||||
| `packages/coord` (status.ts) | Mission health, task progress, next task | **Hermes toolset: `mission`** (same) | Status readout fits naturally as a tool-call. No gateway needed. |
|
||||
| `packages/prdy` | PRD generation wizard | **Hermes skill: `prdy`** | PRD generation is a prompt + template problem, not infrastructure. A Hermes skill with templates is the right fit. |
|
||||
| `plugins/mosaic-framework` | before_agent_start + subagent_spawning hooks | **Hermes system prompt injection** | Hermes already injects system context via skills and config. The framework preamble and worktree rules become standard Hermes skills loaded by the orchestrator profile. |
|
||||
| `plugins/macp` | OpenClaw ACP bridge (spawn codex/claude) | **Already covered by `delegate_task` + ACP** | Hermes already has ACP support and delegate_task. The MACP bridge is redundant when running natively in Hermes. |
|
||||
| Churn detection (planned) | Detect compaction loops, repeated tool calls, no progress | **Hermes middleware** | This needs to live inside Hermes's turn loop where it can observe tool-call patterns. Mosaic can't see this from outside. |
|
||||
| Handoff packet (planned) | Structured context summary for session rotation | **Hermes toolset: `mission`** | Handoff is a serialization of mission + session state. Hermes owns the session, so it should own the handoff. |
|
||||
|
||||
### What Hermes already has that replaces Mosaic infrastructure
|
||||
|
||||
| Mosaic concept | Hermes equivalent | Notes |
|
||||
| -------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| Gateway (NestJS) | Hermes gateway | Hermes already has a gateway with WebSocket, Discord, Telegram, CLI. No need for a second one. |
|
||||
| Pi SDK agent runtime | Hermes agent loop | Hermes IS the agent runtime. OpenClaw's Pi SDK is a different runtime that Mosaic targets. |
|
||||
| MACP ACP bridge | `delegate_task` + ACP tools | Same capability, already native. |
|
||||
| Session identity | Hermes session IDs + process_registry | Hermes already tracks session identity, PIDs, and background processes. |
|
||||
| Task execution board | Hermes Kanban | Fully functional SQLite-backed Kanban with dispatcher, triage, events, comments. |
|
||||
| Worker spawning | Hermes dispatcher + cron | Kanban dispatcher + cron already handle this. |
|
||||
| Context injection | Hermes skills + system prompt | Skills are loaded at session start and injected into context. Exactly what mosaic-framework plugin does. |
|
||||
| File checkpoints | Hermes checkpoint_manager | Already tracks file mutations with shadow git. |
|
||||
|
||||
### What Mosaic keeps as its own entity
|
||||
|
||||
| Component | Why it stays in Mosaic |
|
||||
| --------------------- | --------------------------------------------------- |
|
||||
| `apps/gateway` | NestJS API surface — Mosaic's web platform offering |
|
||||
| `apps/web` | Next.js dashboard — Mosaic's UI offering |
|
||||
| `packages/types` | Shared TS contracts for Mosaic gateway plugins |
|
||||
| `packages/db` | Drizzle ORM + PG — Mosaic's data layer |
|
||||
| `packages/auth` | BetterAuth — Mosaic's auth system |
|
||||
| `packages/brain` | PG-backed data layer for Mosaic web app |
|
||||
| `packages/queue` | Valkey task queue for Mosaic gateway |
|
||||
| `plugins/discord` | OpenClaw Discord plugin |
|
||||
| `plugins/telegram` | OpenClaw Telegram plugin |
|
||||
| `packages/mosaic` CLI | The `mosaic` CLI — Mosaic's own command surface |
|
||||
|
||||
---
|
||||
|
||||
## Architecture: `mission` Toolset for Hermes
|
||||
|
||||
### New files under `/opt/hermes/tools/`
|
||||
|
||||
```
|
||||
mission_tools.py — Tool-call surface (mission_create, mission_status,
|
||||
mission_next_task, mission_update_task, mission_handoff,
|
||||
mission_resume)
|
||||
mission_state.py — State management (read/write mission.json, parse TASKS.md,
|
||||
parse MISSION-MANIFEST.md)
|
||||
mission_churn.py — Churn detection (tool-loop counter, compaction counter,
|
||||
progress scorer)
|
||||
mission_handoff.py — Handoff packet generation and loading
|
||||
```
|
||||
|
||||
### Tool-calls exposed to the agent
|
||||
|
||||
| Tool | What it does | When the agent calls it |
|
||||
| --------------------- | --------------------------------------------------------------------------------- | ------------------------------------------- |
|
||||
| `mission_create` | Initialize mission.json + TASKS.md + MISSION-MANIFEST.md in a project dir | When starting a new mission |
|
||||
| `mission_status` | Read current mission state, milestone progress, next task, active session | At session start, or when checking progress |
|
||||
| `mission_next_task` | Find the next `not-started` task whose dependencies are met, return its full spec | When the agent needs work to do |
|
||||
| `mission_update_task` | Update a task row status in TASKS.md | When completing or blocking a task |
|
||||
| `mission_handoff` | Generate a handoff packet from current session context + mission state | Before session rotation or at session end |
|
||||
| `mission_resume` | Load a handoff packet and inject it as context for the new session | At session start after rotation |
|
||||
|
||||
### Toolset registration
|
||||
|
||||
The `mission` toolset follows the same pattern as `kanban`:
|
||||
|
||||
1. **Gating**: Tools are available when:
|
||||
- The profile has `mission` in its toolsets config, OR
|
||||
- A `HERMES_MISSION_DIR` env var is set (cron/dispatcher spawned workers)
|
||||
2. **File conventions**: The toolset reads/writes the same file formats as Mosaic `packages/coord`:
|
||||
- `.mosaic/orchestrator/mission.json` — mission state
|
||||
- `docs/TASKS.md` — task table
|
||||
- `docs/MISSION-MANIFEST.md` — mission manifest
|
||||
- `docs/scratchpads/<id>.md` — session scratchpad
|
||||
|
||||
3. **Kanban bridge**: Optional bidirectional sync between mission TASKS.md rows and Kanban task cards, so the dashboard sees mission tasks.
|
||||
|
||||
### Churn detection (middleware)
|
||||
|
||||
Churn detection lives in Hermes's turn loop, NOT as a tool-call. It observes:
|
||||
|
||||
- Repeated compaction events (context window pressure)
|
||||
- Identical tool-call sequences (loop detection)
|
||||
- No file state changes across N turns
|
||||
- Repeated permission denials
|
||||
|
||||
When churn score exceeds threshold:
|
||||
|
||||
1. `mission_handoff` is called automatically
|
||||
2. Session is rotated (fresh context window)
|
||||
3. `mission_resume` is called in the new session
|
||||
|
||||
This is new infrastructure that only Hermes can provide (Mosaic runs outside the agent loop).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Tasks
|
||||
|
||||
### Phase 1: Core state management (Python port of coord)
|
||||
|
||||
| Task | Files | Estimate |
|
||||
| -------------------------------------------------- | ----------------------------- | -------- |
|
||||
| 1.1 Port mission.json read/write to Python | `mission_state.py` | 2h |
|
||||
| 1.2 Port TASKS.md parser to Python | `mission_state.py` | 2h |
|
||||
| 1.3 Port MISSION-MANIFEST.md reader to Python | `mission_state.py` | 1h |
|
||||
| 1.4 Implement `mission_create` tool-call | `mission_tools.py` | 1h |
|
||||
| 1.5 Implement `mission_status` tool-call | `mission_tools.py` | 1h |
|
||||
| 1.6 Implement `mission_next_task` tool-call | `mission_tools.py` | 1h |
|
||||
| 1.7 Implement `mission_update_task` tool-call | `mission_tools.py` | 1h |
|
||||
| 1.8 Register `mission` toolset in Hermes registry | `tools/registry.py` | 30m |
|
||||
| 1.9 Add `mission` to orchestrator profile toolsets | `config.yaml` | 10m |
|
||||
| 1.10 Write unit tests for mission_state | `tests/test_mission_state.py` | 2h |
|
||||
| 1.11 Write unit tests for TASKS.md parser | `tests/test_tasks_parser.py` | 1h |
|
||||
|
||||
**Phase 1 estimate:** ~13h
|
||||
|
||||
### Phase 2: Handoff and session continuity
|
||||
|
||||
| Task | Files | Estimate |
|
||||
| ------------------------------------------------- | ---------------------------------------- | -------- |
|
||||
| 2.1 Define handoff packet schema (JSON) | `mission_handoff.py` | 1h |
|
||||
| 2.2 Implement `mission_handoff` tool-call | `mission_handoff.py`, `mission_tools.py` | 2h |
|
||||
| 2.3 Implement `mission_resume` tool-call | `mission_handoff.py`, `mission_tools.py` | 2h |
|
||||
| 2.4 Wire handoff into session start (auto-resume) | agent loop hook | 2h |
|
||||
| 2.5 Write tests for handoff round-trip | `tests/test_mission_handoff.py` | 1h |
|
||||
|
||||
**Phase 2 estimate:** ~8h
|
||||
|
||||
### Phase 3: Churn detection
|
||||
|
||||
| Task | Files | Estimate |
|
||||
| -------------------------------------------------------------- | ----------------------------- | -------- |
|
||||
| 3.1 Define churn signal weights and thresholds | `mission_churn.py` | 1h |
|
||||
| 3.2 Implement tool-loop detector (consecutive identical calls) | `mission_churn.py` | 2h |
|
||||
| 3.3 Implement compaction pressure detector | `mission_churn.py` | 1h |
|
||||
| 3.4 Implement progress scorer (file state delta) | `mission_churn.py` | 2h |
|
||||
| 3.5 Wire churn scoring into agent turn loop | agent loop middleware | 2h |
|
||||
| 3.6 Implement auto-rotation trigger | agent loop + handoff | 2h |
|
||||
| 3.7 Write tests for churn scoring | `tests/test_mission_churn.py` | 1h |
|
||||
|
||||
**Phase 3 estimate:** ~11h
|
||||
|
||||
### Phase 4: Kanban bridge + CLI surface
|
||||
|
||||
| Task | Files | Estimate |
|
||||
| ---------------------------------------------------- | ------------------------ | -------- |
|
||||
| 4.1 Implement TASKS.md → Kanban sync (one-way first) | `mission_kanban_sync.py` | 2h |
|
||||
| 4.2 Add `hermes mission` CLI subcommand | `mission_cli.py` | 2h |
|
||||
| 4.3 Add `hermes mission status` command | `mission_cli.py` | 1h |
|
||||
| 4.4 Add `hermes mission init` command | `mission_cli.py` | 1h |
|
||||
| 4.5 Add `hermes mission handoff` command | `mission_cli.py` | 1h |
|
||||
| 4.6 Add `hermes mission resume` command | `mission_cli.py` | 1h |
|
||||
|
||||
**Phase 4 estimate:** ~8h
|
||||
|
||||
---
|
||||
|
||||
## File Format Compatibility
|
||||
|
||||
The Python implementation MUST read and write the exact same file formats as Mosaic's TypeScript `packages/coord`. This means:
|
||||
|
||||
1. **mission.json** schema is identical to `Mission` type in `packages/coord/src/types.ts`
|
||||
2. **TASKS.md** table format is identical to what `packages/coord/src/tasks-file.ts` parses
|
||||
3. **MISSION-MANIFEST.md** is free-form markdown (no parser needed — just read the file)
|
||||
4. **Handoff packets** are a new JSON format defined in this toolset (Mosaic doesn't have them yet)
|
||||
|
||||
This way a project can use Hermes mission tools OR Mosaic `mosaic coord` commands interchangeably. The files are the contract.
|
||||
|
||||
---
|
||||
|
||||
## Relationship Diagram
|
||||
|
||||
```
|
||||
Mosaic Stack (TypeScript) Hermes Agent (Python)
|
||||
┌─────────────────────────┐ ┌─────────────────────────┐
|
||||
│ packages/coord │ │ tools/mission_tools.py │
|
||||
│ ├─ mission.ts │◄──────►│ ├─ mission_state.py │
|
||||
│ ├─ tasks-file.ts │ same │ ├─ mission_handoff.py │
|
||||
│ ├─ status.ts │ files │ ├─ mission_churn.py │
|
||||
│ └─ runner.ts │ │ └─ mission_tools.py │
|
||||
│ │ │ │
|
||||
│ packages/prdy │ │ skills/prdy/ │
|
||||
│ └─ templates, wizard │◄──────►│ └─ SKILL.md + templates │
|
||||
│ │ │ │
|
||||
│ plugins/mosaic-framework│ │ skills/ (existing) │
|
||||
│ └─ context injection │◄──────►│ └─ kanban-orchestrator │
|
||||
│ │ │ + mosaic-coding-* │
|
||||
│ plugins/macp │ │ tools/delegate_task.py │
|
||||
│ └─ ACP bridge │◄──────►│ └─ already covers this │
|
||||
│ │ │ │
|
||||
│ (stays in Mosaic) │ │ tools/kanban_tools.py │
|
||||
│ apps/gateway │ │ └─ Hermes Kanban DB │
|
||||
│ apps/web │ │ │
|
||||
│ packages/db │ │ tools/cronjob_tools.py │
|
||||
│ packages/queue │ │ └─ already covers cron │
|
||||
└─────────────────────────┘ └─────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Should the `mission` toolset ship with Hermes core, or as a plugin?**
|
||||
- Recommendation: ship as a **built-in toolset** (like `kanban`) since mission coordination is a core agent capability, not an optional integration. The file formats are stable and the code is small.
|
||||
|
||||
2. **Should churn detection be per-profile configurable?**
|
||||
- Recommendation: yes. Add `mission.churn_threshold` and `mission.churn_weights` to profile config.yaml. Default threshold = 5 consecutive no-progress turns.
|
||||
|
||||
3. **Should handoff packets live in the project dir or in Hermes home?**
|
||||
- Recommendation: **project dir** (`.mosaic/handoffs/<session-id>.json`). This keeps them version-controlled and accessible regardless of which agent runtime picks up the project.
|
||||
|
||||
4. **Bidirectional Kanban sync?**
|
||||
- Recommendation: **one-way first** (TASKS.md → Kanban). Bidirectional adds conflict resolution complexity. Ship one-way, add reverse sync in v2 if needed.
|
||||
|
||||
5. **PRD generation — skill or tool-call?**
|
||||
- Recommendation: **skill** (`prdy`). PRD generation is a prompt engineering problem with templates. Skills already handle this pattern perfectly.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Mosaic Stack ↔ Hermes Coordination Resilience
|
||||
|
||||
> Purpose: document the self-healing coordination patterns that emerged while implementing the Hermes mission toolset, distress-card protocol, and auto-heal watchers, so the same mechanics can be reimplemented in Mosaic Stack or any similar agent platform.
|
||||
|
||||
## Summary
|
||||
|
||||
The coordination layer should be treated as a system of mechanical recovery loops rather than a single interactive agent session.
|
||||
|
||||
## SIBKISS operational summary
|
||||
|
||||
- mission on
|
||||
- heartbeat always
|
||||
- resume from packet
|
||||
- block with `[BLOCKED]`
|
||||
- reassign
|
||||
- keep tasks tiny
|
||||
- auto-heal dead workers
|
||||
|
||||
The design has four parts:
|
||||
|
||||
1. Atomic task decomposition — workers operate only within a small, explicit scope.
|
||||
2. Distress signaling — workers create a standardized `[BLOCKED]` card when they encounter a blocker outside their scope.
|
||||
3. Mechanical fallback — if the worker cannot phone home because of rate limits or dead context, a cron-style watcher synthesizes the distress card for them.
|
||||
4. Auto-heal / reassignment — stale workers are reaped, crash-loops are reset, and rate-limited work is reassigned to a different profile/provider.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Observed failure modes:
|
||||
|
||||
- Scope creep: a worker completes the target fix, then spends the rest of its budget chasing downstream cascade work.
|
||||
- Silent failure / dead worker: the worker PID is gone, but the task remains running or blocked.
|
||||
- Rate-limited worker: the worker is too constrained to create a help card itself, so it spins or fails without a clean handoff.
|
||||
|
||||
The answer is not to raise iteration caps or ask the worker to keep trying longer. The answer is to make the coordination layer self-healing and the work items atomic.
|
||||
|
||||
## Core workflow
|
||||
|
||||
### 1) Atomic task boundaries
|
||||
|
||||
Every task should have:
|
||||
|
||||
- one concern
|
||||
- explicit files/packages in scope
|
||||
- explicit files/packages out of scope
|
||||
- a maximum file count if possible
|
||||
- a stated expected iteration budget
|
||||
|
||||
When a worker discovers work outside scope, it must stop fixing it and hand off.
|
||||
|
||||
### 2) Worker-authored distress card
|
||||
|
||||
If the worker can still report status, it creates a card like:
|
||||
|
||||
- Title: `[BLOCKED] t_<source_id> <blocker_type>`
|
||||
- Assignee: `tuesday` / orchestrator role
|
||||
- Status: `ready`
|
||||
- Body: standardized distress template with source task, blocker type, completed work, cannot-touch scope, and needed action
|
||||
|
||||
The orchestrator receives the card, acts on it, and closes the loop.
|
||||
|
||||
## Routing rules
|
||||
|
||||
### Distress card routing
|
||||
|
||||
- Title: `[BLOCKED] t_<source_id> <blocker_type>`
|
||||
- Assignee: `tuesday` / orchestrator role
|
||||
- Status: `ready`
|
||||
- Body: standardized distress template with source task, blocker type, completed work, cannot-touch scope, and needed action
|
||||
- Source task stays linked to the distress card so the recovery trail is auditable
|
||||
|
||||
The orchestrator receives the card, acts on it, and closes the loop.
|
||||
|
||||
### 3) Mechanical fallback for rate-limited workers
|
||||
|
||||
If the worker is too rate-limited or unstable to create the distress card itself, a no-agent watcher must synthesize the card from the task row and failure metadata.
|
||||
|
||||
That watcher should:
|
||||
|
||||
- inspect running / blocked tasks
|
||||
- detect repeated 429 / 503 / overload errors
|
||||
- create the same standardized `[BLOCKED]` card on behalf of the worker
|
||||
- link the distress card to the source task
|
||||
- add a comment to the source task
|
||||
- allow the dispatcher to pick up the new card immediately
|
||||
|
||||
This is the key fix for the logic issue: the worker does not need to be able to phone home if the watcher can do it mechanically.
|
||||
|
||||
### 4) Auto-heal for dead workers
|
||||
|
||||
A separate no-agent watcher should:
|
||||
|
||||
- reap dead PIDs stuck in `running`
|
||||
- reset crash-loops whose failures are infrastructure-related
|
||||
- escalate tasks that have been reset too many times
|
||||
|
||||
This watcher prevents stale tasks from clogging the board and keeps the dispatch queue moving.
|
||||
|
||||
## Distress card contract
|
||||
|
||||
### Canonical title
|
||||
|
||||
```text
|
||||
[BLOCKED] t_<source_task_id> <blocker_type>
|
||||
```
|
||||
|
||||
### Canonical blocker types
|
||||
|
||||
- `scope_boundary`
|
||||
- `env_blocker`
|
||||
- `credential_failure`
|
||||
- `dependency`
|
||||
- `iteration_budget`
|
||||
- `rate_limited`
|
||||
|
||||
### Canonical body
|
||||
|
||||
```markdown
|
||||
## Distress Signal
|
||||
|
||||
- Blocked task: t_xxx
|
||||
- Worker: <profile_name>
|
||||
- Branch: <git_branch_name>
|
||||
- Workspace: <path>
|
||||
- Blocker type: <type>
|
||||
- Completed: <what was done>
|
||||
- Cannot touch: <out-of-scope packages/files>
|
||||
- Needs: <what the orchestrator should do>
|
||||
- State: committed | uncommitted | stashed(<stash_name>)
|
||||
|
||||
## Scope Guard
|
||||
|
||||
DO NOT touch: anything outside diagnosing and remediating the blocker described above
|
||||
Only fix: assign, split, reassign, or unblock the source task
|
||||
```
|
||||
|
||||
## Routing rules
|
||||
|
||||
### Distress card routing
|
||||
|
||||
- `[BLOCKED]` title prefix should bypass normal triage.
|
||||
- The card should go directly to the orchestration profile.
|
||||
- The orchestrator should start from a clean session each time.
|
||||
|
||||
### Rate-limit fallback
|
||||
|
||||
When the source task is rate-limited:
|
||||
|
||||
- do not keep retrying in the worker
|
||||
- let the watcher synthesize the distress card
|
||||
- have the orchestrator reassign the source task to a different profile/provider combo
|
||||
|
||||
### Provider fallback principle
|
||||
|
||||
Never reassign rate-limited work back to the same provider if the failure was provider pressure. Use a different provider when possible.
|
||||
|
||||
### Suggested fallback order
|
||||
|
||||
1. Keep the current task body and scope guards intact.
|
||||
2. Reassign to a different profile on a different provider.
|
||||
3. If that is impossible, reassign to a different profile on the same provider only for non-rate-limit blockers.
|
||||
4. If repeated failures continue, split the task into a narrower atomic card.
|
||||
|
||||
## Related recovery docs
|
||||
|
||||
- Mission packet recovery contract: `/opt/hermes/docs/mission-toolset-heartbeat.md`
|
||||
- Hermes mission implementation plan: `/opt/hermes/docs/plans/mission-toolset-implementation.md`
|
||||
- The same packet-first resume rule applies: inspect the latest packet before re-reading mission files.
|
||||
- New-session trigger: when a profile config changes, start a fresh session or `/reset` so the updated toolset is actually loaded.
|
||||
|
||||
## Watchers to implement
|
||||
|
||||
### Auto-heal watcher
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- reap stale workers
|
||||
- reset dead-PID crash loops
|
||||
- track reset counts
|
||||
- escalate after repeated resets
|
||||
|
||||
### Distress synthesizer watcher
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- detect rate-limited / stuck workers
|
||||
- create `[BLOCKED]` cards mechanically
|
||||
- link the card to the source task
|
||||
- leave a comment for traceability
|
||||
|
||||
### Iteration-budget watcher
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- detect long-running tasks and repeated failure patterns
|
||||
- recommend splits when a task is clearly over-scoped
|
||||
- report tasks that need human review after multiple resets
|
||||
|
||||
## Operational principle
|
||||
|
||||
If a task cannot cleanly finish within its atomic scope, the right response is to surface a smaller coordination problem, not to keep burning context.
|
||||
|
||||
This is what makes the system robust across compaction, rate limits, and dead workers.
|
||||
|
||||
## Suggested implementation order
|
||||
|
||||
1. Atomic task metadata in task bodies
|
||||
2. Worker-authored distress card protocol
|
||||
3. Mechanical distress synthesizer watcher
|
||||
4. Auto-heal watcher for dead workers
|
||||
5. Orchestrator routing rules for `[BLOCKED]`
|
||||
6. Rate-limit fallback / model reassignment table
|
||||
|
||||
## Where this fits in Hermes
|
||||
|
||||
- Kanban = durable work graph and status engine
|
||||
- Watchers = mechanical healing and distress synthesis
|
||||
- Orchestrator = split / reassign / unblock decision-maker
|
||||
- Workers = execution inside atomic task boundaries
|
||||
|
||||
## Where this fits in Mosaic Stack
|
||||
|
||||
- PRD / coordination infra should encode the same patterns
|
||||
- Mosaic can use the same distress-card contract and watcher logic
|
||||
- The coordination model should be runtime-agnostic: any agent system can use it if it can write a task card and react to a ready queue
|
||||
|
||||
## Cross-project takeaway
|
||||
|
||||
The important pattern is not the specific tool names. It is the mechanical feedback loop:
|
||||
|
||||
- detect failure without requiring the failing worker to succeed
|
||||
- create a standardized help artifact
|
||||
- route that artifact to a fresh orchestrator context
|
||||
- repair the assignment graph
|
||||
- continue the mission
|
||||
|
||||
That pattern is reusable anywhere.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Legacy plans and deferred design stubs
|
||||
|
||||
> **Status:** Historical planning archive. These files preserve prior proposals and implementation approaches; they are not proof of shipped behavior or authority to run commands.
|
||||
|
||||
The records below moved byte-identically from migration quarantine on 2026-08-10. Validate every claim against current source, tests, configuration, and safety policy before reuse.
|
||||
|
||||
## Implementation plans
|
||||
|
||||
- [Gateway security hardening](2026-03-13-gateway-security-hardening.md)
|
||||
- [Agent platform architecture](2026-03-15-agent-platform-architecture.md)
|
||||
- [Wave 2 TUI layout and navigation](2026-03-15-wave2-tui-layout-navigation.md)
|
||||
- [Hermes–Mosaic alignment](2026-05-06-hermes-mosaic-alignment.md)
|
||||
- [Coordination resilience](2026-05-07-coordination-resilience.md)
|
||||
- [Gateway token recovery](gateway-token-recovery.md)
|
||||
|
||||
## Setup record
|
||||
|
||||
- [Authentik SSO setup](authentik-sso-setup.md) — superseded for current administration by the canonical [SSO provider guide](../../../ADMIN-GUIDE/security/sso-providers.md).
|
||||
|
||||
## Explicitly deferred stubs
|
||||
|
||||
- [Chroot agent sandboxing](chroot-sandboxing.md)
|
||||
- [Gatekeeper service](gatekeeper-service.md)
|
||||
- [Task queue unification](task-queue-unification.md)
|
||||
|
||||
## Exclusions
|
||||
|
||||
The Agent Reflection PRD remains in quarantine because a live MACP test names its intended canonical path. The WebUI/Fleet Claude bridge draft remains authority-gated and coupled to Fleet decisions. Neither was moved in this archival slice.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Authentik SSO Setup
|
||||
|
||||
## Create the Authentik application
|
||||
|
||||
1. In Authentik, create an OAuth2/OpenID Provider.
|
||||
2. Create an Application and link it to that provider.
|
||||
3. Copy the generated client ID and client secret.
|
||||
|
||||
## Required environment variables
|
||||
|
||||
Set these values for the gateway/auth runtime:
|
||||
|
||||
```bash
|
||||
AUTHENTIK_CLIENT_ID=your-client-id
|
||||
AUTHENTIK_CLIENT_SECRET=your-client-secret
|
||||
AUTHENTIK_ISSUER=https://authentik.example.com
|
||||
```
|
||||
|
||||
`AUTHENTIK_ISSUER` should be the Authentik base URL, for example `https://authentik.example.com`.
|
||||
|
||||
## Redirect URI
|
||||
|
||||
Configure this redirect URI in the Authentik provider/application:
|
||||
|
||||
```text
|
||||
{BETTER_AUTH_URL}/api/auth/callback/authentik
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
https://mosaic.example.com/api/auth/callback/authentik
|
||||
```
|
||||
|
||||
## Test the flow
|
||||
|
||||
1. Start the gateway with `BETTER_AUTH_URL` and the Authentik environment variables set.
|
||||
2. Open the Mosaic login flow and choose the Authentik provider.
|
||||
3. Complete the Authentik login.
|
||||
4. Confirm the browser returns to Mosaic and a session is created successfully.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Chroot Agent Sandboxing — Process Isolation for Agent Tool Execution
|
||||
|
||||
> **Status:** Stub — deferred. Referenced from `2026-03-15-agent-platform-architecture.md` (Phase 7 Workspaces → Chroot Agent Sandboxing).
|
||||
> Implement after Workspaces (P8-015) is complete. Requires workspace directory structure and `WorkspaceService` to be operational.
|
||||
|
||||
**Date:** 2026-03-15
|
||||
**Packages:** `apps/gateway`
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Agent sessions can use file, git, and shell tools. Path validation in tools is defense-in-depth but insufficient alone — an agent with shell access can run `cat /opt/mosaic/.workspaces/other_user/...` and bypass gateway RBAC.
|
||||
|
||||
Chroot provides OS-level enforcement: tool processes literally cannot see outside their workspace directory.
|
||||
|
||||
---
|
||||
|
||||
## Design (Sweet Spot)
|
||||
|
||||
Chroot strikes the balance between full container isolation (too heavy per session) and path validation only (escape-prone):
|
||||
|
||||
- Gateway spawns tool processes inside a chroot rooted at the session's `sandboxDir`
|
||||
- Requires `CAP_SYS_CHROOT` capability on the gateway process (not full root)
|
||||
- Chroot environment provisioned by `WorkspaceService` on workspace creation (minimal deps: git, shell utils, language runtimes as needed)
|
||||
- Alternative for Docker deployments: Linux `unshare` namespaces (lighter, no chroot env setup)
|
||||
|
||||
---
|
||||
|
||||
## Scope (To Be Designed)
|
||||
|
||||
- [ ] Chroot environment provisioning — `WorkspaceService.provisionChroot(workspacePath)` on project creation
|
||||
- [ ] Minimal chroot deps — identify required binaries/libs per tool type (file: none; git: git binary; shell: bash, common utils)
|
||||
- [ ] Gateway capability — document `CAP_SYS_CHROOT` requirement; Dockerfile and docker-compose.yml changes
|
||||
- [ ] Tool process spawning — modify `createShellTools`, `createFileTools`, `createGitTools` to spawn via chroot wrapper
|
||||
- [ ] Docker alternative — `unshare --mount --pid --user` namespace wrapper as fallback for environments without chroot capability
|
||||
- [ ] Defense-in-depth layering — chroot + path validation both active; neither alone is sufficient
|
||||
- [ ] Chroot cleanup — integrate with `SessionGCService` / workspace deletion
|
||||
- [ ] AppArmor/SELinux profiles (v2) — restrict gateway process file access patterns for multi-tenant hardening
|
||||
|
||||
---
|
||||
|
||||
## Security Constraints
|
||||
|
||||
- What lives **inside** the chroot (agent-accessible): workspace files, git repo, language runtimes
|
||||
- What lives **outside** the chroot (gateway-only, never agent-accessible): Valkey connection, PG connection, other users' workspaces, gateway config, OTEL endpoint, credentials
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Workspaces (P8-015) — chroot is rooted at workspace directory; workspace must exist first
|
||||
- Tool hardening (P8-016) — path validation stays active as defense-in-depth alongside chroot
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Original design context: `docs/plans/2026-03-15-agent-platform-architecture.md` → "Chroot Agent Sandboxing" section
|
||||
- Current tool implementations: `apps/gateway/src/agent/tools/`
|
||||
@@ -0,0 +1,53 @@
|
||||
# Gatekeeper Service — PR Review, Quality Gates & Merge Authority
|
||||
|
||||
> **Status:** Stub — deferred. Referenced from `2026-03-15-agent-platform-architecture.md` (Phase 7 Workspaces).
|
||||
> Implement after Workspaces (P8-015) is complete and the workspace/git infrastructure is operational.
|
||||
|
||||
**Date:** 2026-03-15
|
||||
**Packages:** `apps/gateway`, `packages/types`, `packages/agent`
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Project agents create PRs but cannot review or merge their own work. A separate, isolated agent service with read-only code access and quality gate enforcement is needed to act as the authoritative merge authority.
|
||||
|
||||
The Gatekeeper existed in the old Mosaic codebase and must be ported/redesigned for mosaic-mono-v1.
|
||||
|
||||
---
|
||||
|
||||
## Key Design Constraints
|
||||
|
||||
- **Isolated trust boundary** — project agents cannot invoke Gatekeeper directly; it listens for PR events from the git provider
|
||||
- **`isSystem: true`** — system agent, not editable by users
|
||||
- **Read-only code access** — reads diffs and runs checks; cannot commit or push
|
||||
- **Quality gates required before merge** — lint, typecheck, test results must pass
|
||||
- **Cannot self-approve** — the agent that authored the PR cannot be the Gatekeeper for that PR
|
||||
|
||||
---
|
||||
|
||||
## Scope (To Be Designed)
|
||||
|
||||
- [ ] Gatekeeper agent bootstrap — system agent config, tool set, prompt engineering
|
||||
- [ ] PR event listener — Gitea/GitHub webhook integration (PR opened/updated/ready)
|
||||
- [ ] Quality gate runner — trigger CI checks, poll for results, enforce pass criteria
|
||||
- [ ] Review generation — LLM-driven code review comment generation
|
||||
- [ ] Merge execution — approve + merge when gates pass; reject with comments when they fail
|
||||
- [ ] Configurable strictness — per-project required checks, review depth
|
||||
- [ ] Trust boundary enforcement — gateway rejects Gatekeeper tool calls that exceed read-only scope
|
||||
- [ ] Audit trail — OTEL spans for all Gatekeeper decisions (approve/reject/merge)
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Workspaces (P8-015) — Gatekeeper needs project workspace layout to locate code
|
||||
- Git provider API tools — PR creation/review/merge API (Gitea/GitHub/GitLab)
|
||||
- CI/CD tool integration — Woodpecker pipeline status polling
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Original design context: `docs/plans/2026-03-15-agent-platform-architecture.md` → "Gatekeeper Service" section
|
||||
- Workspace RBAC and agent trust model: same document → "RBAC & Filesystem Security"
|
||||
@@ -0,0 +1,193 @@
|
||||
# Gateway Admin Token Recovery — Implementation Plan
|
||||
|
||||
**Mission:** `cli-unification-20260404`
|
||||
**Task:** `CU-03-01` (planning only — no runtime code changes)
|
||||
**Status:** Design locked (Session 1) — BetterAuth cookie-based recovery
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
The gateway installer strands operators when the admin user exists but the admin
|
||||
API token is missing. Concrete trigger:
|
||||
|
||||
- `~/.config/mosaic/gateway/meta.json` was deleted / regenerated.
|
||||
- The installer was re-run after a previous successful bootstrap.
|
||||
|
||||
Flow today (`packages/mosaic/src/commands/gateway/install.ts:375-400`):
|
||||
|
||||
1. `bootstrapFirstUser` hits `GET /api/bootstrap/status`.
|
||||
2. Server returns `needsSetup: false` because `users` count > 0.
|
||||
3. Installer logs `Admin user already exists — skipping setup. (No admin token on file — sign in via the web UI to manage tokens.)` and returns.
|
||||
4. The operator now has:
|
||||
- No token in `meta.json`.
|
||||
- No CLI path to mint a new one (`mosaic gateway <anything>` that needs the token fails).
|
||||
- `POST /api/bootstrap/setup` locked out — it only runs when `users` count is zero (`apps/gateway/src/admin/bootstrap.controller.ts:34-37`).
|
||||
- `POST /api/admin/tokens` gated by `AdminGuard` — requires either a bearer token (which they don't have) or a BetterAuth session (which they don't have in the CLI).
|
||||
|
||||
Dead end. The web UI is the only escape hatch today, and for headless installs even that may be inaccessible.
|
||||
|
||||
## 2. Design Summary
|
||||
|
||||
The BetterAuth session cookie is the authority. The operator runs
|
||||
`mosaic gateway login` to sign in with email/password, which persists a session
|
||||
cookie via `saveSession` (reusing `packages/mosaic/src/auth.ts`). With a valid
|
||||
session, `mosaic gateway config recover-token` (stranded-operator entry point)
|
||||
and `mosaic gateway config rotate-token` call the existing authenticated admin
|
||||
endpoint `POST /api/admin/tokens` using the cookie, then persist the returned
|
||||
plaintext to `meta.json` via `writeMeta`. **No new server endpoints are
|
||||
required** — `AdminGuard` already accepts BetterAuth session cookies via its
|
||||
`validateSession` path (`apps/gateway/src/admin/admin.guard.ts:90-120`).
|
||||
|
||||
## 3. Surface Contract
|
||||
|
||||
### 3.1 Server — no changes required
|
||||
|
||||
| Endpoint | Status | Notes |
|
||||
| ------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `POST /api/admin/tokens` | **Reuse as-is** | `admin-tokens.controller.ts:46-72`. Returns `{ id, label, scope, expiresAt, lastUsedAt, createdAt, plaintext }`. |
|
||||
| `GET /api/admin/tokens` | **Reuse** | Useful for `mosaic gateway config tokens list` follow-on (out of scope for CU-03-01, but trivial once auth path exists). |
|
||||
| `DELETE /api/admin/tokens/:id` | **Reuse** | Used by rotate flow for optional old-token revocation. |
|
||||
| `POST /api/bootstrap/setup` | **Unchanged** | Remains first-user-only; not part of recovery. |
|
||||
|
||||
`AdminGuard.validateSession` takes BetterAuth cookies from `request.raw.headers`
|
||||
via `fromNodeHeaders` and calls `auth.api.getSession({ headers })`. It also
|
||||
enforces `role === 'admin'`. This is exactly the path the CLI will hit with
|
||||
`Cookie: better-auth.session_token=...`.
|
||||
|
||||
**Confirmed feasible** during CU-03-01 investigation.
|
||||
|
||||
### 3.2 `mosaic gateway login`
|
||||
|
||||
Thin wrapper over the existing top-level `mosaic login`
|
||||
(`packages/mosaic/src/cli.ts:42-76`) with gateway-specific defaults pulled from
|
||||
`readMeta()`.
|
||||
|
||||
| Aspect | Behavior |
|
||||
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Default gateway URL | `http://${meta.host}:${meta.port}` from `readMeta()`, fallback `http://localhost:14242`. |
|
||||
| Flow | Prompt email + password -> `signIn()` -> `saveSession()`. |
|
||||
| Persistence | `~/.mosaic/session.json` via existing `saveSession` (7-day expiry). |
|
||||
| Decision | **Thin wrapper**, not alias. Rationale: defaults differ (reads `meta.json`), and discoverability under `mosaic gateway --help`. |
|
||||
| Implementation | Share the sign-in logic by extracting a small `runLogin(gatewayUrl, email?, password?)` helper; both commands call it. |
|
||||
|
||||
### 3.3 `mosaic gateway config rotate-token`
|
||||
|
||||
| Aspect | Behavior |
|
||||
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| Precondition | Valid session (via `loadSession` + `validateSession`). On failure, print: "Not signed in — run `mosaic gateway login`" and exit non-zero. |
|
||||
| Request | `POST ${gatewayUrl}/api/admin/tokens` with header `Cookie: <session>`, body `{ label: "CLI token (rotated YYYY-MM-DD)" }`. |
|
||||
| On success | Read meta via `readMeta()`, set `meta.adminToken = plaintext`, `writeMeta(meta)`. Print the token banner (reuse `printAdminTokenBanner` shape). |
|
||||
| Old token | **Optional `--revoke-old`** flag. When set and a previous `meta.adminToken` existed, call `DELETE /api/admin/tokens/:id` after rotation. Requires listing first to find the id; punt to CU-03-02 decision. Document as nice-to-have. |
|
||||
| Exit codes | `0` success; `1` network error; `2` auth error; `3` server rejection. |
|
||||
|
||||
### 3.4 `mosaic gateway config recover-token`
|
||||
|
||||
Superset of `rotate-token` with an inline login nudge — the "stranded operator"
|
||||
entry point.
|
||||
|
||||
| Step | Action |
|
||||
| ---- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | `readMeta()` — derive gateway URL. If meta is missing entirely, fall back to `--gateway` flag or default. |
|
||||
| 2 | `loadSession(gatewayUrl)` then `validateSession`. If either fails, prompt inline: email + password -> `signIn` -> `saveSession`. |
|
||||
| 3 | `POST /api/admin/tokens` with cookie, label `"Recovered via CLI YYYY-MM-DDTHH:mm"`. |
|
||||
| 4 | Persist plaintext to `meta.json` via `writeMeta`. |
|
||||
| 5 | Print the token banner and next-steps hints (e.g. `mosaic gateway status`). |
|
||||
| 6 | Exit `0`. |
|
||||
|
||||
Key property: this command is **runnable with nothing but email+password in hand**.
|
||||
It assumes the gateway is up but assumes no prior CLI session state.
|
||||
|
||||
### 3.5 File touch list (for CU-03-02..05 execution)
|
||||
|
||||
| File | Change |
|
||||
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| `packages/mosaic/src/commands/gateway.ts` | Register `login`, `config recover-token`, `config rotate-token` subcommands under `gw`. |
|
||||
| `packages/mosaic/src/commands/gateway/config.ts` | Add `runRecoverToken`, `runRotateToken` handlers; export from module. |
|
||||
| `packages/mosaic/src/commands/gateway/login.ts` (new) | Thin wrapper calling shared `runLogin` helper with meta-derived default URL. |
|
||||
| `packages/mosaic/src/auth.ts` | No change expected. Possibly export a `requireSession(gatewayUrl)` helper (reuse pattern). |
|
||||
| `packages/mosaic/src/commands/gateway/install.ts` | `bootstrapFirstUser` branch: "user exists, no token" -> offer recovery (see Section 4). |
|
||||
|
||||
## 4. Installer Fix (CU-03-06 preview)
|
||||
|
||||
Current stranding point is `install.ts:388-395`. The fix:
|
||||
|
||||
```
|
||||
if (!status.needsSetup) {
|
||||
if (meta.adminToken) {
|
||||
// unchanged — happy path
|
||||
} else {
|
||||
// NEW: prompt "Admin exists but no token on file. Recover now? [Y/n]"
|
||||
// If yes -> call runRecoverToken(gatewayUrl) inline (interactive):
|
||||
// - prompt email + password
|
||||
// - signIn -> saveSession
|
||||
// - POST /api/admin/tokens
|
||||
// - writeMeta(meta) with returned plaintext
|
||||
// - print banner
|
||||
// If no -> print the current stranded message but include:
|
||||
// "Run `mosaic gateway config recover-token` when ready."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Shape notes (actual code lands in CU-03-06):
|
||||
|
||||
- Extract the recovery body so it can be called **both** from the standalone
|
||||
command and from `bootstrapFirstUser` without duplicating prompts.
|
||||
- Reuse the same `rl` readline interface already open in `bootstrapFirstUser`
|
||||
for the inline prompts.
|
||||
- Preserve non-interactive behavior: if `process.stdin.isTTY` is false, skip the
|
||||
prompt and emit the "run recover-token" hint only.
|
||||
|
||||
## 5. Test Strategy (CU-03-07 scope)
|
||||
|
||||
### 5.1 Happy paths
|
||||
|
||||
| Command | Scenario | Expected |
|
||||
| ------------------------------------- | ------------------------------------------------ | -------------------------------------------------------- |
|
||||
| `mosaic gateway login` | Valid creds | `session.json` written, 7-day expiry, exit 0 |
|
||||
| `mosaic gateway config rotate-token` | Valid session, server reachable | `meta.json` updated, banner printed, new token usable |
|
||||
| `mosaic gateway config recover-token` | No session, valid creds, server reachable | Prompts for creds, writes session + meta, exit 0 |
|
||||
| Installer inline recovery | Re-run after `meta.json` wipe, operator says yes | Meta restored, banner printed, no manual CLI step needed |
|
||||
|
||||
### 5.2 Error paths (must all produce actionable messages and non-zero exit)
|
||||
|
||||
| Failure | Expected handling |
|
||||
| --------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| Invalid email/password | BetterAuth 401 surfaced as "Sign-in failed: <server message>", exit 2 |
|
||||
| Expired stored session | Recover command silently re-prompts; rotate command exits 2 with "run login" hint |
|
||||
| Gateway down / connection refused | "Could not reach gateway at <url>" exit 1 |
|
||||
| Server rejects token creation | Print status + body excerpt, exit 3 |
|
||||
| Meta file missing (recover) | Fall back to `--gateway` flag or default; warn that meta will be created |
|
||||
| Non-admin user | `AdminGuard` 403 surfaced as "User is not an admin", exit 2 |
|
||||
|
||||
### 5.3 Integration test (recommended)
|
||||
|
||||
Spin up gateway in test harness, create admin user via `/api/bootstrap/setup`,
|
||||
wipe `meta.json`, invoke `mosaic gateway config recover-token` programmatically,
|
||||
assert new `meta.adminToken` works against `GET /api/admin/tokens`.
|
||||
|
||||
## 6. Risks & Open Questions
|
||||
|
||||
| # | Item | Severity | Mitigation |
|
||||
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | `AdminGuard.validateSession` calls `getSession` with `fromNodeHeaders(request.raw.headers)`. CLI sends `Cookie:` header only. Confirm BetterAuth reads from `Cookie`, not `Set-Cookie`. | Low | Confirmed — `mosaic login` + `mosaic tui` already use this flow successfully (`cli.ts:137-181`). |
|
||||
| 2 | Session cookie local expiry (7d) vs BetterAuth server-side expiry may drift. | Low | `validateSession` hits `get-session`; handle 401 by re-prompting. |
|
||||
| 3 | Label collision / unbounded token growth if operators run `recover-token` repeatedly. | Low | Include ISO timestamp in label. Optional `--revoke-old` in CU-03-02. Add `tokens list/prune` later. |
|
||||
| 4 | `mosaic login` exists at top level and `mosaic gateway login` is a wrapper — risk of confusion. | Low | Document that `gateway login` is the preferred entry for gateway operators; top-level stays for compatibility. |
|
||||
| 5 | `meta.json` write is not atomic. Crash between token creation and `writeMeta` leaves an orphan token server-side with no plaintext on disk. | Medium | Accept for now — re-running `recover-token` mints a fresh token. Document as known limitation. |
|
||||
| 6 | Non-TTY installer runs (CI, headless provisioners) cannot prompt for creds interactively. | Medium | Installer inline recovery must skip prompt when `!process.stdin.isTTY`; emit the recover-token hint. |
|
||||
| 7 | If `BETTER_AUTH_SECRET` rotates between login and recover, the session cookie is invalid — user must re-login. Acceptable but surface a clear error. | Low | Error handler maps 401 on recover -> "Session invalid; re-run `mosaic gateway login`". |
|
||||
| 8 | No MFA today. When MFA lands, BetterAuth sign-in will return a challenge, not a cookie — recovery UX will need a second prompt step. | Future | Out of scope for this mission. Flag for future CLI work. |
|
||||
|
||||
## 7. Downstream Task Hooks
|
||||
|
||||
| Task | Scope |
|
||||
| -------- | -------------------------------------------------------------------------- |
|
||||
| CU-03-02 | Implement `mosaic gateway login` wrapper + shared `runLogin` extraction. |
|
||||
| CU-03-03 | Implement `mosaic gateway config rotate-token`. |
|
||||
| CU-03-04 | Implement `mosaic gateway config recover-token`. |
|
||||
| CU-03-05 | Wire commands into `gateway.ts` registration, update `--help` copy. |
|
||||
| CU-03-06 | Installer inline recovery hook in `bootstrapFirstUser`. |
|
||||
| CU-03-07 | Tests per Section 5. |
|
||||
| CU-03-08 | Docs: update gateway install README + operator runbook with recovery flow. |
|
||||
@@ -0,0 +1,60 @@
|
||||
# Task Queue Unification — @mosaicstack/queue as Unified Orchestration Layer
|
||||
|
||||
> **Status:** Stub — deferred. Referenced from `2026-03-15-agent-platform-architecture.md` (Task Queue & Orchestration section).
|
||||
> Implement after Workspaces (P8-015) is complete. Requires workspace file structure to be in place.
|
||||
|
||||
**Date:** 2026-03-15
|
||||
**Packages:** `packages/queue`, `packages/coord`, `packages/db`, `apps/gateway`
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Two disconnected task systems exist:
|
||||
|
||||
1. **`@mosaicstack/coord`** — file-based missions (`mission.json`, `TASKS.md`), file locks, subprocess spawning. Single-machine orchestrator pattern.
|
||||
2. **PG tables** (`tasks`, `mission_tasks`, `missions`) — DB-backed CRUD, REST API, Brain repos.
|
||||
|
||||
An agent using `coord_mission_status` gets file data. The dashboard shows DB data. They are never in sync.
|
||||
|
||||
---
|
||||
|
||||
## Vision
|
||||
|
||||
`@mosaicstack/queue` becomes the unified task orchestration service bridging PG, workspace files, and Valkey:
|
||||
|
||||
- DB is source of truth for structured state (status, assignees, timestamps)
|
||||
- Workspace files (`TASKS.md`, PRDs) are working copies for agent interaction
|
||||
- Valkey handles real-time assignment queues and agent claim locks
|
||||
- Flatfile fallback for no-DB single-machine deployments (preserves `@mosaicstack/coord` pattern)
|
||||
|
||||
---
|
||||
|
||||
## Scope (To Be Designed)
|
||||
|
||||
- [ ] `@mosaicstack/queue` refactor — elevate from ioredis primitive to task orchestration service
|
||||
- [ ] DB ↔ file sync layer — writes to PG propagate to `TASKS.md`; file edits by agents sync back
|
||||
- [ ] Task assignment queue — Valkey-backed RPUSH/BLPOP for agent task claiming
|
||||
- [ ] Agent claim locks — `mosaic:queue:project:{id}:lock:{taskId}` with TTL
|
||||
- [ ] `@mosaicstack/coord` consolidation — file-based ops ported into queue service; `@mosaicstack/coord` becomes thin adapter or deprecated
|
||||
- [ ] Flatfile fallback — queue service writes JSON manifests when PG unavailable
|
||||
- [ ] Status pub/sub — real-time task status updates via Valkey pub/sub
|
||||
- [ ] Dependency resolution — block task assignment until dependencies are met
|
||||
- [ ] Orchestrator monitor — gateway process watches task queue, assigns next based on dependency graph
|
||||
- [ ] API surface — queue service exposes typed interface used by agents, gateway, and CLI
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Workspaces (P8-015) — file sync targets the workspace directory structure
|
||||
- Teams architecture (P8-007) — project ownership determines queue namespacing
|
||||
- DB schema stable — task/mission tables must not change mid-unification
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Original design context: `docs/plans/2026-03-15-agent-platform-architecture.md` → "Task Queue & Orchestration" section
|
||||
- Current `@mosaicstack/coord` implementation: `packages/coord/src/`
|
||||
- Current `@mosaicstack/queue` implementation: `packages/queue/src/`
|
||||
@@ -0,0 +1,10 @@
|
||||
# Archived Matrix/MACP proposals
|
||||
|
||||
> **Status:** Historical draft proposals. These records moved byte-identically from migration quarantine on 2026-08-10 and have no implementation or operational authority.
|
||||
|
||||
- [RFC-001: MACP Matrix-native communications](rfc-001-macp-matrix-native.md)
|
||||
- [RFC-002: install, configuration, and topology](rfc-002-install-config-topology.md)
|
||||
|
||||
Current source and focused tests do not establish the proposed Matrix adapter, identity mapping, persistence, homeserver/appservice topology, or federation operations. See the canonical [channel protocol](../../../DEVELOPER-GUIDE/architecture/channel-protocol.md) for the implemented Discord boundary and explicit Matrix limitations.
|
||||
|
||||
Do not use these archived RFCs as deployment instructions or as evidence that Matrix/MACP functionality shipped.
|
||||
@@ -0,0 +1,449 @@
|
||||
# RFC-001 — MACP: A Mosaic-Native, Matrix-Native Comms Layer
|
||||
|
||||
- **Status:** DRAFT — for Team Lead → Orchestrator staffing
|
||||
- **Author:** MS-LEAD (reviewer identity `ms-lead-reviewer`)
|
||||
- **Sponsor / veto:** Jason (human lead)
|
||||
- **Date:** 2026-07-24
|
||||
- **Program:** Mosaic Stack comms-evolution
|
||||
- **Supersedes backbone:** the Hermes MCP chat bridge (strangler-retired, see §9)
|
||||
- **Audience:** Team Leads, the Mosaic orchestrator, infra, and any harness maintainer (Claude Code / Codex / Pi / Goose)
|
||||
|
||||
> This is a **design document**. No code ships from this RFC. It exists to be decomposed into missions (P1→P5, §10) with per-phase acceptance criteria. Where a claim is uncertain or needs live validation, it is flagged **[VERIFY]**.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
|
||||
We are building a **Mosaic-native comms layer on Matrix**. We self-host a **Synapse** homeserver and register a privileged **Mosaic Application Service** (the "appservice") that the orchestrator controls. The appservice bulk-provisions one Matrix identity per agent-spin, creates and manages rooms, posts agent introductions, and tracks presence/liveness. Agents talk to it through a thin `packages/comms` client SDK. **tmux stays the P0 same-host fast path**; Matrix is the durable, presence-aware, federated layer above it, and **MACP** (the Mosaic Agent Comms Protocol) is the standard that says which path to use when. Federation is **per-site Synapse homeservers federated over TLS we control** — a direct answer to "the homelab agent went dark and took comms with it." We adopt **Buzz's patterns** (auto-detect/enroll, signed identity, unified event log, humans-and-agents on one surface) without adopting Buzz/Nostr as transport. We migrate off Hermes by the **strangler** pattern: stand native alongside, move channels as proven, retire at parity.
|
||||
|
||||
The **first standalone shippable slice is presence** (P1).
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals / Non-Goals
|
||||
|
||||
### 1.1 Goals
|
||||
|
||||
- **G1 — Presence & liveness first.** A Team Lead must be able to answer "is my coordinator online, away, or dead?" in seconds, not by polling for 13 hours. Presence is the P1 slice and ships before anything else.
|
||||
- **G2 — A native backbone we own.** Replace _Hermes-as-backbone_ with a self-hosted Synapse + a Mosaic-controlled appservice. External chat bridging becomes an optional edge, not the spine.
|
||||
- **G3 — Turnkey harness enrollment.** `mosaic enroll` auto-detects the harness and self-registers the agent via the appservice on spin. No hand-rolled per-bot identity juggling.
|
||||
- **G4 — A real protocol (MACP v1).** Structured, versioned event schema over Matrix custom event types; a documented routing contract for tmux vs Matrix; a documented escalation policy.
|
||||
- **G5 — No central SPOF.** Per-site homeservers federated over TLS/DNS we already control, so one site going dark cannot take the fleet's comms with it.
|
||||
- **G6 — Gate-action integrity.** Reviews / merges / approvals carry **signed authorship** (Buzz pattern) so a gate-critical action is cryptographically attributable, retiring the fragile "distinct bot identity" juggling.
|
||||
|
||||
### 1.2 Non-Goals
|
||||
|
||||
- **NG1 — Do NOT rip out working comms mid-MVP.** tmux fast-path and the existing `mos-comms` git-branch channel keep working until their replacement is proven at parity. This RFC is strangler, not big-bang.
|
||||
- **NG2 — tmux is NOT being replaced.** tmux inter-agent comms remains **P0**. Matrix is _above_ it, not instead of it. MACP defines the boundary; it does not move it.
|
||||
- **NG3 — Not adopting Buzz/Nostr as transport.** We adopt Buzz's _patterns_; the wire is Matrix.
|
||||
- **NG4 — Not building a new chat client in P1–P4.** HIL uses an existing Matrix client (Element or equivalent) until/unless a custom client is justified (open question, §11).
|
||||
- **NG5 — Not federating to the public Matrix network.** Federation is Mosaic-site-to-Mosaic-site over infrastructure we control. Public `matrix.org` federation is out of scope (and should likely be firewalled off).
|
||||
- **NG6 — Not a Hermes feature-clone.** We reach _parity on the channels that matter_ (§9 checklist), not bug-for-bug Hermes compatibility.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### 2.1 Layer diagram
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────┐
|
||||
│ HUMAN (Jason / HIL) │
|
||||
│ Element (or custom client) — §11 │
|
||||
└───────────────────────┬───────────────────────┘
|
||||
│ (same Matrix surface as agents)
|
||||
│
|
||||
┌──────────────────────────────────────────────▼──────────────────────────────────────────────┐
|
||||
│ SYNAPSE HOMESERVER (self-hosted, ours) │
|
||||
│ - Client-Server API (agents + humans send/receive events) │
|
||||
│ - Application Service API (privileged AS hooks: transactions, user/room namespaces) │
|
||||
│ - Presence EDUs, receipts, typing │
|
||||
│ - Federation API (S2S) over TLS ── to peer site homeservers (§6, P4) │
|
||||
└───────▲───────────────────────────────────▲──────────────────────────────────────▲───────────┘
|
||||
│ AS API (hs_token / as_token) │ C-S API (per-agent access_token) │ S2S
|
||||
│ │ │
|
||||
┌───────┴───────────────────────┐ ┌────────┴─────────────────┐ ┌─────────┴──────────┐
|
||||
│ MOSAIC APPSERVICE │ │ packages/comms (SDK) │ │ PEER SITE Synapse │
|
||||
│ (apps/matrix-appservice) │ │ used by every harness │ │ (site-B, site-C…) │
|
||||
│ THE "native layer" │ │ - login/whoami │ │ own appservice │
|
||||
│ replacing Hermes-backbone │ │ - send MACP events │ │ own agents │
|
||||
│ │ │ - subscribe/sync │ └────────────────────┘
|
||||
│ - bulk-provision MXIDs │◄───┤ - presence heartbeat │
|
||||
│ (@mosaic_<agent>:site) │ │ - signed-authorship │
|
||||
│ - create/manage rooms │ │ envelope (gate acts) │
|
||||
│ - post introductions │ └────────────┬─────────────┘
|
||||
│ - track presence/liveness │ │ in-process / IPC
|
||||
│ - enforce room taxonomy │ ┌─────────▼──────────────────────────────────────────┐
|
||||
│ - escalation watchdog (§5) │ │ AGENT HARNESS │
|
||||
│ - controlled by ORCHESTRATOR │ │ Claude Code / Codex / Pi / Goose │
|
||||
└───────▲───────────────────────┘ │ `mosaic enroll` runs on spin (§4.1) │
|
||||
│ orchestrator drives AS └─────────┬───────────────────────────────────────────┘
|
||||
┌───────┴───────────────────────┐ │
|
||||
│ MOSAIC ORCHESTRATOR │ │ P0 FAST PATH (same host, low-latency)
|
||||
│ (~/.config/mosaic) │ ┌─────────▼──────────┐ tmux send-keys / pane I/O
|
||||
│ spins agents, owns rooms, │◄──────►│ tmux (P0) │◄─►│ peer agent on same host │
|
||||
│ sets escalation policy │ MACP └────────────────────┘ └─────────────────────────┘
|
||||
└───────────────────────────────┘ routing rules decide tmux vs Matrix per message (§4.6)
|
||||
```
|
||||
|
||||
Key idea: **the appservice is the backbone.** It is a long-lived privileged process registered with Synapse via an appservice registration file (`hs_token`/`as_token`, namespaces). It is the thing that used to be "Hermes-as-backbone," except we own it, it is inside the orchestrator's control plane, and it speaks native Matrix.
|
||||
|
||||
### 2.2 Message flow: agent spin-up → auto-enroll → room join → introduction → presence-online
|
||||
|
||||
```
|
||||
Orchestrator Harness (mosaic enroll) Mosaic Appservice Synapse
|
||||
│ │ │ │
|
||||
1. spin agent ─────────────────► │ │ │
|
||||
│ │ 2. auto-detect harness │ │
|
||||
│ │ (Claude/Codex/Pi/Goose) │ │
|
||||
│ │ 3. POST /enroll {agent meta} ─► │
|
||||
│ │ │ 4. provision MXID │
|
||||
│ │ │ @mosaic_<agent>:site │
|
||||
│ │ │ via AS API register ─► (201, in namespace)
|
||||
│ │ │ 5. mint access_token │
|
||||
│ │ 6. ◄── {mxid, token, rooms}──┤ (or as_token masq) │
|
||||
│ │ │ 7. invite+join rooms ─► (mission/team/fleet)
|
||||
│ │ 8. /sync (via packages/comms)─────────────────────────► (joined state)
|
||||
│ │ │ 9. post introduction ─► m.room.message +
|
||||
│ │ │ (mosaic.introduction) custom event → rooms
|
||||
│ │ 10. set presence ONLINE ─────────────────────────────► presence EDU
|
||||
│ │ 11. start heartbeat loop │ │
|
||||
│ │ (mosaic.presence ping) │ │
|
||||
│ 12. appservice reports agent │ │ │
|
||||
│ ◄──── live in fleet room ────┤ (watchdog now tracks liveness) │
|
||||
```
|
||||
|
||||
Notes on the steps that matter:
|
||||
|
||||
- **Step 4/5** use the **Application Service API**: the appservice can register users inside its namespace (`@mosaic_*:site`) and act on their behalf. Two viable modes: (a) mint a real per-agent `access_token` via appservice login, or (b) have the appservice **masquerade** using `user_id` query param on C-S calls with the `as_token`. **Recommendation: mint per-agent tokens** for P2 so the agent process holds only its own credential (blast-radius containment, §8); reserve masquerade for bulk/bootstrap operations the appservice itself performs. **[VERIFY]** exact token-lifetime and refresh behavior against the running Synapse version.
|
||||
- **Step 9** — the introduction is both a human-readable `m.room.message` _and_ a structured `mosaic.introduction` custom event (so other agents can machine-parse capabilities without scraping prose).
|
||||
- **Step 10/11** — presence goes online immediately, then a **heartbeat** keeps liveness fresh. Native Matrix presence auto-decays to `unavailable`/`offline`, but we do **not** rely solely on it (Synapse presence timeouts are coarse and federation presence is lossy **[VERIFY]**); MACP adds an explicit `mosaic.presence` heartbeat event for deterministic liveness (§4.5, §5).
|
||||
|
||||
---
|
||||
|
||||
## 3. Repo-home decision (RESOLVED — recommendation)
|
||||
|
||||
The core tension: **product monorepo** (`mosaicstack/stack`, this checkout `/src/mosaic-stack`) vs **framework** (`~/.config/mosaic`, the agent/harness runtime that every agent shares regardless of product). The boundary rule I am ratifying:
|
||||
|
||||
> **Product-monorepo owns the deployed _services and libraries_. Framework owns the _agent/harness contract_ — anything an agent needs the moment it spins, before any product code is checked out.**
|
||||
|
||||
Applying that rule:
|
||||
|
||||
| Piece | Home | Rationale |
|
||||
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Synapse deployment** (compose/helm, config, TLS, `.well-known`, pipelines) | **Product monorepo** → `infra/matrix/` + CI in the monorepo's pipeline dir | It is deployed infrastructure with the same lifecycle/observability as Gateway/Web/DB. Lives beside other `infra/` and Woodpecker pipelines. |
|
||||
| **Mosaic Appservice** | **Product monorepo** → `apps/matrix-appservice` | It is a first-class deployed service (NestJS-style app, same conventions: ESM, `@Inject()`, DTOs, OTEL-before-bootstrap). It talks to Synapse, holds `hs_token`/`as_token`, and is operated like any other app. It is **controlled by** the orchestrator but **is not** the orchestrator. |
|
||||
| **Client SDK** | **Product monorepo** → `packages/comms` | A workspace library consumed by product apps _and_ by harnesses. Published/linked like `packages/queue`, `packages/db`. Versioned with the product. |
|
||||
| **MACP spec** (the standard doc + JSON schemas) | **Framework** → `~/.config/mosaic/spec/macp/` (mirrored/vendored into `packages/comms` at build) | The protocol is an agent-level contract that must exist independent of any one product checkout. Framework is the source of truth; `packages/comms` vendors a pinned copy so the SDK and spec cannot drift silently. |
|
||||
| **`mosaic enroll` harness glue** (auto-detect, spin hook) | **Framework** → `~/.config/mosaic/tools/enroll/` | Agents/harnesses are framework-level. Enrollment must run _before_ the agent has product context; it cannot depend on `/src/mosaic-stack` being present. This is exactly parallel to the existing `~/.config/mosaic/tools/*` wrappers. |
|
||||
|
||||
**Boundary summary:** _the wire and the services are product; the contract and the spin-time glue are framework._ The one deliberate coupling is **MACP**: framework is authoritative, but `packages/comms` pins a vendored copy and CI fails if they diverge, so an agent enrolling via framework and a service validating via `packages/comms` agree on the schema by construction.
|
||||
|
||||
**Rejected alternative:** putting the appservice in the framework. Rejected because the appservice is a stateful, deployed, secret-holding network service that needs the product's CI/observability/secret plumbing; burying it in `~/.config/mosaic` would split its operational story from every other Mosaic service.
|
||||
|
||||
---
|
||||
|
||||
## 4. MACP v1 — the standard
|
||||
|
||||
MACP (Mosaic Agent Comms Protocol) v1 is a **profile of Matrix**: it does not invent a transport, it constrains how Mosaic agents use Matrix so that behavior is uniform across harnesses. Versioned via a `macp_version` field on every custom event; v1 is frozen at ratification (P3).
|
||||
|
||||
### 4.1 Enrollment contract
|
||||
|
||||
`mosaic enroll` MUST, on agent spin, in order:
|
||||
|
||||
1. **Auto-detect harness.** Detection order + signal:
|
||||
- Claude Code — presence of the Claude Code runtime/env (e.g. `CLAUDE_CODE_*` env, `~/.claude`) **[VERIFY exact signal per harness]**
|
||||
- Codex — Codex runtime markers
|
||||
- Pi — Pi SDK runtime (`packages/agent` / `packages/mosaic` context)
|
||||
- Goose — Goose runtime markers
|
||||
- Fallback: explicit `--harness` flag; if undetectable, enroll as `generic` and warn.
|
||||
2. **Provision identity** — call appservice `POST /enroll` with `{agent_slug, harness, host, mission_id?, team_id?, capabilities[]}`. Appservice returns `{mxid, access_token, homeserver, rooms[]}` (§2.2 step 4–6).
|
||||
3. **Join rooms** — accept invites / join the returned room set per taxonomy (§4.6).
|
||||
4. **Introduce** — post `mosaic.introduction` (+ human-readable `m.room.message`) to each joined room.
|
||||
5. **Go present** — set Matrix presence `online` and start the `mosaic.presence` heartbeat loop.
|
||||
|
||||
Enrollment is **idempotent**: re-running `mosaic enroll` for an existing agent slug rebinds to the same MXID (re-mints token if needed) rather than creating a duplicate identity. This is what retires the "distinct bot identity juggling."
|
||||
|
||||
### 4.2 Structured event schema (Matrix custom event types)
|
||||
|
||||
All MACP events carry a common envelope in `content`:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"macp_version": "1.0",
|
||||
"macp_type": "<see below>",
|
||||
"agent": { "mxid": "@mosaic_teamlead-3:site-a", "slug": "teamlead-3", "harness": "claude-code" },
|
||||
"ts": 1753300000000,
|
||||
"mission_id": "KBN-101", // optional
|
||||
"signature": { ... } // present ONLY for gate actions, §4.4
|
||||
// ...type-specific fields...
|
||||
}
|
||||
```
|
||||
|
||||
Event types (Matrix `type` shown; timeline events use `m.room.message` with a custom `msgtype` where a human-visible fallback is desirable, state events use a dotted custom `type`):
|
||||
|
||||
| MACP type | Matrix carrier | Purpose | Notable fields |
|
||||
| --------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| **message** | `m.room.message`, `msgtype: "mosaic.message"` (+ `body` fallback) | ordinary agent/human chat | `body`, `thread?` |
|
||||
| **presence** | `m.room.message` msgtype `mosaic.presence` in fleet room **or** custom EDU-backed state | heartbeat/liveness ping | `status: online\|away\|offline`, `seq`, `interval_ms` |
|
||||
| **workflow-step** | state event `mosaic.workflow.step` (state_key = step id) | durable record of a workflow/mission step | `step`, `status: started\|blocked\|done\|failed`, `detail` |
|
||||
| **review / approval** | `m.room.message` msgtype `mosaic.review` | a review verdict or merge/approval | `subject` (PR/commit ref), `verdict: approve\|reject\|request-changes`, `signature` (REQUIRED) |
|
||||
| **escalation** | `m.room.message` msgtype `mosaic.escalation` | raise a stuck/dark condition to HIL/fallback | `reason`, `target`, `severity`, `since_ts` |
|
||||
|
||||
Rationale for the carrier split: **timeline events** (`m.room.message` variants) are the durable, receipted, replayable log (this _is_ Buzz's "unified event log," §7). **State events** (`mosaic.workflow.step`, presence-as-state) give last-writer-wins current status that a newly-joined agent reads instantly from room state without replaying history.
|
||||
|
||||
Every custom event is validated against a JSON Schema shipped in the MACP spec (§3). Unknown/newer `macp_version` → consumers MUST degrade gracefully (render `body`, ignore unknown fields).
|
||||
|
||||
### 4.3 Agent identity model
|
||||
|
||||
- **MXID:** appservice-namespaced `@mosaic_<slug>:<site-domain>`. The `mosaic_` prefix is the AS **user namespace** declared in the registration file (`namespaces.users` regex `@mosaic_.*`), so Synapse routes those users to our appservice and no human can squat the namespace.
|
||||
- **Provisioning:** exclusively via the appservice (AS API register). Never hand-created. One MXID per agent-spin; idempotent rebind on re-enroll (§4.1).
|
||||
- **Signed authorship** overlays identity for gate actions only (§4.4). MXID answers "who is this account"; signature answers "did the real key-holder authorize this gate action."
|
||||
|
||||
### 4.4 Signed-authorship for gate-critical actions (Buzz pattern, scoped)
|
||||
|
||||
Gate-critical = **reviews, merges, approvals** — anything that can move code to `main` or unblock a mission gate.
|
||||
|
||||
- Each enrolled agent is issued (or generates) an **Ed25519 keypair**; the **public** key is registered with the appservice at enrollment and published as agent profile state (`mosaic.identity.pubkey`). Private key custody per §8.
|
||||
- A gate action event carries `content.signature = { alg: "ed25519", key_id, sig }` over a canonical serialization of the envelope (canonical-JSON of `{macp_type, agent.mxid, subject, verdict, ts, mission_id}`).
|
||||
- Verifiers (the appservice gate-watcher, and any agent acting on a verdict) MUST reject an unsigned or bad-signature gate event. Non-gate events are unsigned (keeps the hot path cheap).
|
||||
|
||||
This is deliberately **narrow**: we do not sign every chat line (Buzz signs everything; we take the pattern only where forgery has teeth). Scope may widen post-P5 if warranted.
|
||||
|
||||
### 4.5 Presence & liveness model
|
||||
|
||||
Three visible states plus an explicit heartbeat:
|
||||
|
||||
- **online** — agent set presence online AND last `mosaic.presence` heartbeat within `heartbeat_interval` (default **30s [VERIFY tuning]**) × miss-tolerance (default 2).
|
||||
- **away** — presence `unavailable`, or heartbeats late but < dark threshold.
|
||||
- **offline / dark** — no heartbeat for `dark_threshold` (default **N minutes**, policy value, §5/§11) OR presence `offline`.
|
||||
|
||||
Why not rely on native Matrix presence alone: Synapse presence is (a) coarse-grained, (b) can be disabled for load reasons, and (c) **degrades across federation** **[VERIFY]**. So MACP layers an explicit heartbeat carried as a lightweight timeline/state event in the **fleet presence room**, giving a deterministic, federation-safe liveness signal the escalation watchdog (§5) can reason about. Native presence EDUs are still emitted (they make Element show the right dot for humans) but the _authoritative_ liveness source is the heartbeat.
|
||||
|
||||
### 4.6 Room / channel taxonomy (orchestrator-owned)
|
||||
|
||||
The **orchestrator** (via the appservice) owns room lifecycle. Agents never create backbone rooms ad hoc.
|
||||
|
||||
| Room | Scope | Membership | Purpose |
|
||||
| ----------------------- | ------------------------------------------ | ---------------------------------------- | ----------------------------------------------------------------------- |
|
||||
| **Fleet presence room** | one per site (federated view across sites) | every enrolled agent + HIL | heartbeats, the single "who's alive" board. This is the P1 deliverable. |
|
||||
| **Per-mission room** | one per mission (e.g. `#mission-KBN-101`) | agents on that mission + Team Lead + HIL | workflow-steps, mission chat, reviews for that mission |
|
||||
| **Per-team room** | one per team | team members + Team Lead | intra-team coordination |
|
||||
| **HIL room** | one (or one per site) | humans + escalation-privileged agents | where escalations land; Jason's pane on the fleet |
|
||||
|
||||
Rooms are created with orchestrator-controlled power levels: appservice = admin (PL100), Team Leads elevated, worker agents default. Room aliases (`#mission-KBN-101:site-a`) are stable handles.
|
||||
|
||||
### 4.7 tmux ↔ Matrix routing rules (the fast-path/durable boundary)
|
||||
|
||||
MACP mandates this decision per message. **Default bias: if it must survive the agent, be seen by an offline party, cross a host, or be audited — Matrix. If it is same-host, synchronous, and ephemeral — tmux.**
|
||||
|
||||
| Signal | Route | Why |
|
||||
| ---------------------------------------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- |
|
||||
| Same-host, live pane-to-pane prompt/nudge, sub-second need | **tmux (P0)** | lowest latency, no server round-trip; this is the working fast path we keep |
|
||||
| Recipient may be offline / on another host | **Matrix** | durability + store-and-forward; tmux can't reach a dark or remote pane |
|
||||
| Presence / heartbeat / liveness | **Matrix** | must be observable fleet-wide, including by the watchdog and HIL |
|
||||
| Workflow-step, review/approval, escalation | **Matrix** | must be durable, receipted, auditable, signed (gate acts) |
|
||||
| Cross-site anything | **Matrix (federated)** | tmux is same-host only |
|
||||
| Bulk log spew / high-frequency scratch between co-located agents | **tmux**, with periodic **Matrix** checkpoints | avoid flooding the durable log; keep an audit checkpoint |
|
||||
|
||||
Rule of thumb encoded in the SDK: `comms.send()` takes a `durability` hint; `ephemeral+same-host` short-circuits to tmux, everything else goes Matrix. A message can be **dual-routed** (tmux for immediacy + a Matrix checkpoint) when both speed and durability matter.
|
||||
|
||||
---
|
||||
|
||||
## 5. Coordinator-availability + HIL escalation (the 13h-stall / homelab-dark fix)
|
||||
|
||||
**The failure we are killing:** a Team Lead blocked ~13h, polling every 15 min, unable to distinguish "coordinator offline" from "coordinator busy"; and a homelab agent that goes dark taking comms with it.
|
||||
|
||||
**The fix — presence-driven, policy-encoded escalation:**
|
||||
|
||||
1. **Deterministic liveness (§4.5).** Every agent heartbeats into the fleet presence room. The appservice **escalation watchdog** subscribes and maintains `last_seen` per agent. No polling by the Team Lead — it _subscribes_ (Matrix `/sync` long-poll) and is pushed state changes.
|
||||
|
||||
2. **Encoded policy in MACP:** a machine-readable escalation policy attached to each agent/role, e.g.:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"role": "team-lead",
|
||||
"coordinator": "@mosaic_coordinator-1:site-a",
|
||||
"dark_threshold_min": 10, // OPEN QUESTION §11 — Jason/Mos to set N
|
||||
"on_coordinator_dark": {
|
||||
"action": "escalate",
|
||||
"fallback": "@mosaic_coordinator-2:site-b", // cross-site fallback
|
||||
"then": "notify-HIL",
|
||||
"hil_room": "#hil:site-a",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
3. **Detection & action by the Team Lead:** when the watchdog (or the Team Lead's own SDK) observes `coordinator.last_seen > dark_threshold`, it:
|
||||
- emits a `mosaic.escalation` event into the mission room and the HIL room (`reason: "coordinator dark", since_ts, severity`),
|
||||
- **re-routes** to the declared fallback coordinator (possibly on another site — this is why federation matters),
|
||||
- if no fallback answers within a second threshold, pages **HIL** (Jason) in the HIL room.
|
||||
The Team Lead **never sits blocked polling**; a dark coordinator is a _pushed event_, and the fallback/HIL path is automatic.
|
||||
|
||||
4. **Homelab-dark specifically:** because heartbeats are federated into a cross-site fleet room, a whole _site_ going dark is visible from other sites — the watchdog on site-B sees site-A's coordinator stop heartbeating and triggers the same escalation. A dark homelab can no longer silently strand its agents, because the liveness signal and the fallback live _off that host_.
|
||||
|
||||
**Design invariant:** liveness authority and fallback targets must never be co-located with the thing they monitor. The watchdog for site-A's coordinator should also run (or be mirrored) on site-B.
|
||||
|
||||
---
|
||||
|
||||
## 6. Federation (P4)
|
||||
|
||||
**Model:** each site runs its **own Synapse homeserver** with its **own Mosaic appservice** and its own agents. Sites **federate** with each other over the standard Matrix server-to-server (S2S) API, restricted to Mosaic sites.
|
||||
|
||||
**Why per-site, not one central server:**
|
||||
|
||||
- **No SPOF.** The homelab going dark is the founding trauma of this program. A single central homeserver would recreate exactly that risk at fleet scale. Per-site means a site outage is contained: its agents drop, but every other site's comms and the cross-site fleet room survive.
|
||||
- **Locality.** Same-site agents get low-latency local homeserver traffic; only cross-site events pay the federation cost.
|
||||
- **Blast radius.** A compromised or misbehaving site can be defederated without touching the rest.
|
||||
|
||||
**How federation is wired (real Matrix mechanics):**
|
||||
|
||||
- **Server discovery** via `https://<domain>/.well-known/matrix/server` returning `{"m.server": "matrix.<domain>:443"}`, and/or an `_matrix._tcp` **SRV** record. We control the DNS/domains, so we control the federation graph. **[VERIFY]** current `.well-known` vs SRV precedence for the deployed Synapse version.
|
||||
- **TLS:** federation requires valid TLS on the federation endpoint; we terminate with certs from our own CA/ACME on domains we own.
|
||||
- **Allowlist:** use Synapse `federation_domain_whitelist` to restrict federation to the set of Mosaic site domains — **no public-network federation** (NG5). This is a hard security boundary.
|
||||
- **Cross-site rooms:** the fleet presence room and any cross-mission rooms are federated rooms whose membership spans site homeservers. Room state replicates via S2S; presence heartbeats propagate as events (not relying on lossy presence EDUs across federation, §4.5).
|
||||
|
||||
**Cross-site identity:** an agent on site-B is `@mosaic_<slug>:site-b`. The signed-authorship pubkey travels in profile state, so a site-A verifier can validate a site-B agent's gate action without trusting site-B's homeserver blindly (signature ≠ homeserver trust).
|
||||
|
||||
---
|
||||
|
||||
## 7. Buzz-pattern adoption map
|
||||
|
||||
We adopt Buzz's **ideas**, on Matrix rails, phased:
|
||||
|
||||
| Buzz idea | Adopt? | How, on Matrix | Phase |
|
||||
| --------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
|
||||
| **Harness auto-detect / enroll** | **Yes** | `mosaic enroll` detects Claude/Codex/Pi/Goose and self-registers via the appservice (§4.1) | P2 (enroll v1); scan-machine-and-offer-integrate is **v2** |
|
||||
| **Keypair signed identity** | **Yes, scoped** | Ed25519 signed-authorship on **gate actions only** (reviews/merges/approvals), pubkey in profile state (§4.4) | P5 (hardening); keys issued at enroll from P2 |
|
||||
| **Unified event log (humans + agents, one log)** | **Yes** | Matrix room timeline _is_ the unified, receipted, replayable event log; MACP custom events are first-class entries (§4.2) | P2→P3 |
|
||||
| **Humans and agents on the same surface** | **Yes** | HIL uses the same Matrix rooms via Element/custom client; escalations land where Jason already is (§4.6, §5) | P1 (fleet room) → P2 |
|
||||
| **Scan machine, offer to integrate existing tools** | **Deferred (v2)** | enroll v1 auto-detects the _harness_; scanning a host for other integratable tools is explicitly **enroll v2**, post-P5 | v2 |
|
||||
| **Buzz/Nostr as transport** | **No** | transport is Matrix; only patterns are borrowed (NG3) | — |
|
||||
|
||||
---
|
||||
|
||||
## 8. Security
|
||||
|
||||
- **Homeserver hardening.** Disable open registration (`enable_registration: false`); agents come only via the appservice. `federation_domain_whitelist` to Mosaic sites only (§6). Rate-limiting on. Media repo locked down or disabled if unused. Admin API bound to localhost / behind auth. Run Synapse behind our reverse proxy with TLS termination we control. **[VERIFY]** current recommended hardening flags for the deployed Synapse version.
|
||||
- **Appservice token custody (`hs_token` / `as_token`).** These are the crown jewels — `as_token` lets the holder act as any user in the namespace; `hs_token` authenticates the homeserver → appservice callbacks. They live **only** in the appservice's secret store (see Vault interaction below), never in an agent process, never in the SDK, never in git. Registration file (which contains them) is a secret artifact, mounted at deploy, not committed. Rotate on suspected exposure; rotation requires updating both Synapse's registration and the appservice.
|
||||
- **Per-agent credentials (the hard part — fleet key management).**
|
||||
- Agents hold **only their own** per-agent `access_token` (minted by the appservice at enroll), not the `as_token`. Compromise of one agent ≠ compromise of the namespace.
|
||||
- **Issuance:** at `mosaic enroll`, the appservice mints the token and returns it over the enroll channel (which must itself be authenticated — enroll requests carry a spin-time bootstrap secret / orchestrator-signed nonce **[VERIFY design in P2]**, so a random process can't enroll a rogue agent).
|
||||
- **Signed-authorship keys:** Ed25519 private keys should be generated agent-side and only the public key leaves the agent (best custody: private key never transits the network). Where agents are ephemeral, keys are minted per-spin and discarded on teardown; the pubkey-in-profile record remains for audit.
|
||||
- **Rotation:** tokens are short-lived where the Synapse version supports refresh **[VERIFY]**; otherwise the appservice supports explicit re-issue on re-enroll. A rotation runbook is a P2 deliverable.
|
||||
- **Federation TLS.** Valid certs on federation endpoints; whitelist-only federation; monitor for cert expiry (a cert lapse silently defederates a site — add to observability).
|
||||
- **Interaction with existing Vault/secrets.** The appservice's `hs_token`/`as_token` and the enroll bootstrap secret are stored in the existing secret manager (Vault or the project's chosen store — **open question §11**) and injected at deploy, consistent with how Gateway/DB secrets are handled today. No new bespoke secret store. Per-agent tokens are _transient runtime_ secrets, not persisted to Vault (they're re-mintable). **[VERIFY]** current Mosaic secret-management choice — the CLAUDE.md notes secrets/KBN work in flight, so align with whatever KBN-101 lands.
|
||||
|
||||
---
|
||||
|
||||
## 9. Migration — strangler off Hermes
|
||||
|
||||
**Principle:** stand the native layer up _alongside_ Hermes, move channels over **as each is proven at parity**, retire Hermes only when the parity checklist is green. Never a flag-day cutover (NG1).
|
||||
|
||||
**Sequence:**
|
||||
|
||||
1. **Stand alongside.** Native Synapse + appservice + fleet presence room live in parallel; Hermes still carries everything it carries today. Presence (P1) is _additive_ — it gives us something Hermes never had, at zero risk to existing flows.
|
||||
2. **Move channels as proven.** Per channel (fleet presence → mission coordination → reviews/approvals → external-chat edge), cut traffic to Matrix, keep Hermes as hot fallback until the channel meets parity for a soak period.
|
||||
3. **Retire at parity.** When every checklist item is green and soaked, decommission the Hermes MCP bridge as backbone. (External chat platforms, if still needed, can be re-attached as a _bridge at the edge_ of Matrix rather than the spine.)
|
||||
|
||||
**Parity checklist (must all be green before Hermes retires):**
|
||||
|
||||
- [ ] Every message class Hermes carries today has a MACP equivalent (message, review/approval, escalation, workflow-step).
|
||||
- [ ] Presence/liveness is strictly better than today (it is: today = none).
|
||||
- [ ] Cross-site delivery works over federation with no central SPOF.
|
||||
- [ ] HIL (Jason) can see and act on escalations on the Matrix surface.
|
||||
- [ ] Signed-authorship enforced on gate actions (no unsigned merge/approve accepted).
|
||||
- [ ] `mosaic enroll` auto-onboards all four harnesses (Claude/Codex/Pi/Goose) with zero manual identity setup.
|
||||
- [ ] Delivery receipts / durability demonstrably ≥ Hermes (no lost messages over a soak window).
|
||||
- [ ] Runbooks exist: appservice token rotation, site defederation, dark-site escalation, homeserver restore.
|
||||
- [ ] Observability: appservice + Synapse traced into OTEL/Jaeger like the rest of the stack.
|
||||
- [ ] Rollback path documented (re-enable Hermes channel) for the soak period.
|
||||
|
||||
---
|
||||
|
||||
## 10. Phased delivery plan (P1 → P5) with acceptance criteria
|
||||
|
||||
> This section is the decomposition surface: each phase → one or more missions for the orchestrator.
|
||||
|
||||
### P1 — Presence / availability on a minimal single-site Synapse _(the first standalone shippable slice)_
|
||||
|
||||
**Scope:** one Synapse homeserver, a **minimal** appservice (or even a scripted provisioner) whose only job is: register a handful of agent MXIDs, create the **fleet presence room**, and carry heartbeats; a minimal `packages/comms` slice that sets presence and heartbeats; native presence surfaced to a human via Element.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- A1. Single-site Synapse deployed (`infra/matrix/`), reachable over TLS, open registration OFF.
|
||||
- A2. ≥3 agents enroll (even if semi-manually) and appear in a **fleet presence room** with a live online/away/offline indicator.
|
||||
- A3. `mosaic.presence` heartbeat implemented; an agent killed hard flips to **offline/dark** within `dark_threshold` deterministically (not dependent on native presence timeout alone).
|
||||
- A4. A human (Jason) can open Element, join the fleet room, and see fleet liveness at a glance.
|
||||
- A5. Zero impact to existing tmux + `mos-comms` flows (they still work untouched).
|
||||
|
||||
### P2 — Native appservice + orchestrator auto-enroll / room-provisioning
|
||||
|
||||
**Scope:** full **`apps/matrix-appservice`** registered with Synapse (`hs_token`/`as_token`, namespaces); `mosaic enroll` harness auto-detect; orchestrator-owned room taxonomy; per-agent token minting; introductions.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- B1. Appservice registered with Synapse via registration file; owns `@mosaic_*` user namespace and room-alias namespace.
|
||||
- B2. `mosaic enroll` auto-detects all four harnesses (Claude/Codex/Pi/Goose) and self-registers on spin, idempotently.
|
||||
- B3. On spin, an agent is provisioned an MXID, minted its **own** access token, joined to the correct mission/team/fleet rooms, and posts a `mosaic.introduction`.
|
||||
- B4. Orchestrator can create/destroy mission & team rooms with correct power levels via the appservice.
|
||||
- B5. Enroll bootstrap is authenticated (a rogue local process cannot enroll a rogue agent).
|
||||
- B6. Appservice + Synapse traced into OTEL/Jaeger.
|
||||
|
||||
### P3 — MACP v1 spec ratified
|
||||
|
||||
**Scope:** freeze the standard (§4): envelope, event types + JSON Schemas, identity model, presence model, room taxonomy, tmux↔Matrix routing rules. Spec lives in framework (`~/.config/mosaic/spec/macp`), vendored+pinned into `packages/comms` with CI drift-check.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- C1. MACP v1 document ratified (MS-LEAD sign-off, Jason veto window closed).
|
||||
- C2. JSON Schemas for all five event types published; `packages/comms` validates outbound/inbound against them.
|
||||
- C3. CI fails if framework spec and vendored `packages/comms` copy diverge.
|
||||
- C4. Routing-rule conformance test: SDK provably sends ephemeral+same-host over tmux, everything else over Matrix.
|
||||
- C5. Unknown-`macp_version` graceful-degrade behavior tested.
|
||||
|
||||
### P4 — Federation
|
||||
|
||||
**Scope:** a second site homeserver + appservice; S2S federation over our DNS/TLS; cross-site fleet room; whitelist-only federation; cross-site escalation.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- D1. Two sites federate via `.well-known`/SRV over TLS we control; `federation_domain_whitelist` restricts to Mosaic sites (no public federation).
|
||||
- D2. A cross-site fleet presence room shows agents from both sites; heartbeats propagate as events across federation.
|
||||
- D3. **Homelab-dark test:** killing site-A's coordinator is observed from site-B within `dark_threshold`, and the escalation/fallback fires cross-site (§5).
|
||||
- D4. Cross-site gate action: a site-B agent's signed review is verified by a site-A verifier.
|
||||
- D5. Defederation runbook proven (a site can be cut off cleanly).
|
||||
|
||||
### P5 — Buzz-hardening + signed-authorship + Hermes retired
|
||||
|
||||
**Scope:** Ed25519 signed-authorship enforced on gate actions; security hardening pass; complete the strangler and retire Hermes at parity.
|
||||
**Acceptance criteria:**
|
||||
|
||||
- E1. Every merge/approve/review gate action is signed; unsigned or bad-sig gate events are rejected by the appservice watcher and by consuming agents.
|
||||
- E2. Token/key rotation runbooks executed at least once in anger (rotate `as_token`, rotate a per-agent key).
|
||||
- E3. Security review complete: homeserver hardening flags, token custody, federation TLS/whitelist all verified.
|
||||
- E4. **Parity checklist (§9) fully green + soaked.**
|
||||
- E5. Hermes MCP bridge retired as backbone (optionally re-attached as an edge bridge only).
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions for Jason / Mos (need a human/coordinator ruling)
|
||||
|
||||
1. **DNS / domains per site.** What domain(s) do we own and want to use per site for homeserver names and `.well-known` (e.g. `site-a.mosaicstack.dev`)? Federation identity is permanent-ish once agents mint MXIDs against it — this needs a ruling **before P1 hardens** because MXIDs bake in the domain.
|
||||
2. **Secret-management choice.** Is it Vault, or whatever KBN-101 lands? The appservice `hs_token`/`as_token` and enroll bootstrap secret custody depend on this (§8). CLAUDE.md signals secrets work is in flight — need the authoritative target.
|
||||
3. **N-minute escalation threshold.** What is `dark_threshold_min` for a coordinator, and the second threshold before HIL is paged (§5)? Default proposed: 10 min → fallback, +5 min → HIL. Jason/Mos to confirm per role.
|
||||
4. **HIL client: Element vs custom.** Do humans use off-the-shelf **Element** (fast, free, P1-ready) or do we invest in a custom HIL client? Proposed: **Element for P1–P4**, revisit custom only if HIL ergonomics demand it.
|
||||
5. **Ephemeral vs persistent agent keys.** For signed-authorship, do we mint Ed25519 keys per-spin (simplest custody, no long-term private key at rest) or issue durable per-agent keys (stable identity across spins, but key-at-rest custody problem)? Proposed: **per-spin**, pubkey retained for audit.
|
||||
6. **Federation topology / trust.** Full mesh between all sites, or hub-and-spoke-with-redundancy? Full mesh maximizes no-SPOF but grows O(n²); needs a call once site count is known.
|
||||
7. **Fallback-coordinator assignment authority.** Who assigns each Team Lead's fallback coordinator, and is it always cross-site? (Design invariant §5 wants the fallback off the monitored host — confirm this is acceptable operationally.)
|
||||
8. **Retention / compliance.** How long do we retain the Matrix event log (the unified audit trail)? Affects Synapse storage sizing and any purge policy.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Real Matrix concepts this RFC leans on (quick reference)
|
||||
|
||||
- **Application Service (AS) API** — a privileged service registered with the homeserver via a registration YAML declaring `id`, `url`, `as_token`, `hs_token`, and `namespaces` (users/aliases/rooms regexes). The homeserver pushes events to the AS in transactions; the AS can register/act-as users in its namespace. _(This is our appservice backbone.)_
|
||||
- **`hs_token` / `as_token`** — `hs_token`: homeserver→AS authentication on pushed transactions; `as_token`: AS→homeserver authentication, grants acting-as any namespaced user. Both are high-value secrets (§8).
|
||||
- **Masquerade (`user_id` query param)** — the AS may act as a namespaced user on C-S calls using `as_token` + `?user_id=`. We prefer per-agent tokens for blast-radius; masquerade for AS-internal bulk ops.
|
||||
- **Custom event types** — timeline events via `m.room.message` with a custom `msgtype` (keeps a human-visible `body` fallback) and/or fully custom `type` (dotted, e.g. `mosaic.workflow.step`); **state events** for last-writer-wins current status readable from room state without history replay.
|
||||
- **Presence EDUs** — native online/unavailable/offline signals; coarse and lossy over federation, so MACP adds an explicit heartbeat event as the authoritative liveness source (§4.5).
|
||||
- **Federation (S2S API)** — server-to-server over TLS; discovery via `/.well-known/matrix/server` and/or `_matrix._tcp` SRV; restrictable with `federation_domain_whitelist`.
|
||||
- **Synapse config knobs cited** — `enable_registration`, `federation_domain_whitelist`, appservice registration file, rate-limiting, admin API binding. **[VERIFY]** exact flags/paths against the deployed Synapse version at implementation time.
|
||||
|
||||
_All Matrix mechanics above are cited from architecture knowledge and MUST be re-verified against the actual deployed Synapse version during P1 — every **[VERIFY]** in this document is a checkpoint, not an assumption._
|
||||
@@ -0,0 +1,622 @@
|
||||
# RFC-002 — Install, Configuration & Topology for the Mosaic Matrix/MACP Comms System
|
||||
|
||||
- **Status:** DRAFT — for Team Lead → Orchestrator staffing
|
||||
- **Author:** MS-LEAD (reviewer identity `ms-lead-reviewer`)
|
||||
- **Sponsor / veto:** Jason (human lead)
|
||||
- **Date:** 2026-07-24
|
||||
- **Program:** Mosaic Stack comms-evolution
|
||||
- **Companion to:** RFC-001 — _MACP: A Mosaic-Native, Matrix-Native Comms Layer_. RFC-001 is the architecture (self-hosted Synapse + Mosaic appservice backbone + `packages/comms` SDK + MACP standard + per-site federation). **RFC-002 is the config substrate the whole thing installs and runs on.**
|
||||
- **Audience:** Team Leads, the Mosaic orchestrator, infra, harness maintainers, and — critically — **strangers who install this open-source product on hardware we will never see.**
|
||||
|
||||
> This is a **design document**. No code ships from this RFC. It is written to be decomposed into missions with per-phase acceptance criteria, and it slots under RFC-001's P1→P5. Every uncertain or must-live-validate claim is flagged **[VERIFY]**.
|
||||
|
||||
> **The one framing that governs every decision below:** this is an **open-source product**. Someone we have never met will `git clone` it and run it on their own domains, their own DNS, their own certs, their own hardware. **NOTHING may hardcode our fleet's topology.** There is no `woltje.com` in the code, no assumption that DNS exists, no assumption that a second site exists. Every topology fact is **user-supplied config**. Where this doc uses `mosaic.woltje.com` / `mosaic.uscllc.com`, those are **illustrative operator values** (Jason's real installs), never defaults and never literals in the product.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
|
||||
The comms system installs against a **user-supplied topology**, never a baked-in one. At install the operator declares exactly one of **three topology modes**: **(A) split-domain** (identity `server_name` ≠ homeserver host, wired via Matrix delegation — this is Jason's `mosaic.woltje.com` identity + `matrix.woltje.com` host setup), **(B) single-domain** (`server_name` == homeserver host), or **(C) IP-only standalone** (no DNS, no federation, fully supported for local/airgapped). **The PRIMARY/home instance is ALWAYS configured; federation is OPTIONAL.** A single standalone instance MUST work with zero federation.
|
||||
|
||||
**Federation is a hard-gated capability: it REQUIRES DNS + valid certificates. IP-only federation is not possible and is not supported.** IP-only means standalone-only, forever, until the operator acquires DNS + certs.
|
||||
|
||||
Certificates are **one ACME integration** with a user-chosen **directory URL**: either **step-ca** (self-hosted private ACME CA, for total control and private/internal domains public CAs can't issue for) or **Let's Encrypt** (public ACME, ease-of-use). The operator also picks a challenge type (HTTP-01 / DNS-01 / TLS-ALPN-01); **DNS-01 is the answer for private/split-horizon domains.**
|
||||
|
||||
Secrets go through a **pluggable `SecretBackend` interface** — no forced paid dependency. Ships with a **Vault** implementation and a **Vaultwarden** implementation; the operator picks at install. The Vaultwarden model (org + orchestrator enrolled as authority + per-agent scoped access) is designed-for, with an honest **[VERIFY]** on how far Vaultwarden's machine-account coverage has matured.
|
||||
|
||||
Config is **DB-backed with sane defaults and install-time overrides**. Precedence: **install-time → DB override → default.** Config is split into **install-time-immutable** (e.g. `server_name`, which is baked into every MXID and cannot change without re-homing every identity) and **runtime-tunable** (e.g. dark-threshold).
|
||||
|
||||
RFC-002 is the substrate; **RFC-001's P1 (presence) needs only Mode A/B single-instance clean-domain and does NOT require federation, IP-only, or the secret-backend rotation story resolved.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals / Non-Goals
|
||||
|
||||
### 1.1 Goals
|
||||
|
||||
- **G1 — Installable by a stranger.** A person with no relationship to our fleet can install, configure, and run the comms system from published artifacts and a guided installer, on their own hardware and domains, with no edits to product code.
|
||||
- **G2 — Zero hardcoded topology.** Every topology fact — `server_name`, homeserver host/IP, delegation method, federation peers, cert mode, secret backend — is **user-supplied config**, validated at install, stored in the product DB. No fleet-specific literal ships in the product.
|
||||
- **G3 — Standalone MUST work.** The PRIMARY instance is always fully functional with **zero federation**, including with **no DNS at all** (Mode C, IP-only). Presence, rooms, MACP, HIL-via-Element all work single-instance.
|
||||
- **G4 — Federation is optional but honestly gated.** Federation is opt-in and, when opted into, **requires DNS + valid certificates as a hard precondition.** The installer must refuse to _claim_ federation is working when the DNS/cert preconditions aren't met.
|
||||
- **G5 — One ACME integration, two CA choices.** Build a single ACME cert-provisioning path; the operator selects step-ca or Let's Encrypt by supplying an **ACME directory URL** plus a challenge type. No second, bespoke cert path.
|
||||
- **G6 — No forced paid dependency for secrets.** A pluggable `SecretBackend` with at least Vault and Vaultwarden implementations, chosen at install. Open-source ethos: the free/self-hostable path must be first-class.
|
||||
- **G7 — Defaults that just work, overrides where they matter.** DB-backed config with sane defaults so most operators change little; install-time overrides for the topology-critical values; a clear immutable-vs-tunable boundary so operators can't foot-gun `server_name`.
|
||||
- **G8 — A clean upgrade path.** An operator who starts standalone can later turn on federation with a documented, honest procedure (including the real cost if they started IP-only and must now acquire a stable `server_name`).
|
||||
|
||||
### 1.2 Non-Goals
|
||||
|
||||
- **NG1 — Not hosting a managed service.** This RFC is about _self-install_. We are not building multi-tenant SaaS provisioning; each operator runs their own instance(s).
|
||||
- **NG2 — Not a new cert stack.** We do not write our own CA, our own ACME client protocol, or a non-ACME cert path. We integrate ACME and let the operator point it at step-ca or Let's Encrypt. (We _may_ bundle/recommend step-ca as the self-hosted CA, but via its standard ACME provisioner, not a fork.)
|
||||
- **NG3 — Not a new secret manager.** We define an interface and ship adapters. We do not build a secret store; we do not force one.
|
||||
- **NG4 — Not public-network Matrix federation.** Consistent with RFC-001 NG5: federation is Mosaic-site-to-Mosaic-site over infrastructure the operator controls, allowlisted. No `matrix.org` federation.
|
||||
- **NG5 — Not making IP-only federate.** We will not ship a hack (self-signed S2S trust bundles, `/etc/hosts` federation) that pretends IP-only can federate. IP-only is standalone. This is a deliberate, honest boundary (§2.4, §7).
|
||||
- **NG6 — Not re-homing identities silently.** We will not offer a "just change your `server_name`" button that quietly orphans every MXID. Any path that changes `server_name` is a flagged, gated, documented identity re-home (§5.3, §7).
|
||||
|
||||
---
|
||||
|
||||
## 2. The topology model
|
||||
|
||||
### 2.1 The core split: `server_name` vs homeserver host
|
||||
|
||||
Matrix has exactly the split Jason described, natively:
|
||||
|
||||
- **`server_name`** — the Synapse config value that is the server's **identity domain**. It is the part after the colon in every MXID (`@mosaic_agent:mosaic.woltje.com`) and every room alias (`#mission-KBN-101:mosaic.woltje.com`). It is **baked into every identity the moment that identity is minted.** Changing it re-homes everything. This is `server_name` in Synapse's `homeserver.yaml`.
|
||||
- **Homeserver host** — the actual network location (hostname:port or IP:port) where the Synapse process answers federation and (optionally proxied) client traffic. It **can differ** from `server_name`. Matrix reconciles the difference through **delegation**: `https://<server_name>/.well-known/matrix/server` returning `{"m.server": "matrix.woltje.com:443"}`, and/or a `_matrix._tcp.<server_name>` **SRV** record. **[VERIFY]** `.well-known` vs SRV precedence on the deployed Synapse version (RFC-001 §6 flags the same).
|
||||
|
||||
So Jason's "mosaic._ app-domain + matrix._ homeserver-domain" split maps precisely: **`server_name = mosaic.woltje.com` (identity, in MXIDs), homeserver runs at `matrix.woltje.com` (discovered via delegation).** That is **Mode A**.
|
||||
|
||||
### 2.2 The topology config schema
|
||||
|
||||
One canonical config object, stored in the product DB (§5), populated at install (§6). Illustrative shape (field names decomposition-ready, not frozen):
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"topology": {
|
||||
"mode": "split-domain | single-domain | ip-only-standalone", // A | B | C — install-time-immutable
|
||||
|
||||
"identity": {
|
||||
"server_name": "mosaic.woltje.com", // INSTALL-TIME-IMMUTABLE. In MXIDs. Never change without re-home (§5.3).
|
||||
"server_name_kind": "domain | ip", // "ip" only legal in Mode C
|
||||
},
|
||||
|
||||
"homeserver": {
|
||||
"host": "matrix.woltje.com", // where Synapse actually listens (Mode A: differs from server_name)
|
||||
"port": 8448, // federation port (default 8448) or 443 if proxied — [VERIFY] per deploy
|
||||
"client_bind": "https://matrix.woltje.com", // C-S API public URL (proxied)
|
||||
"bind_ip": null, // Mode C: e.g. "192.168.1.50" ; Modes A/B: null (DNS-resolved)
|
||||
},
|
||||
|
||||
"delegation": {
|
||||
"method": "well-known | srv | none", // Mode A: well-known or srv ; Mode B/C: none
|
||||
"well_known_server": { "m.server": "matrix.woltje.com:443" }, // if method=well-known
|
||||
"srv_record": "_matrix._tcp.mosaic.woltje.com. 3600 IN SRV 10 0 443 matrix.woltje.com.", // if method=srv (documented, operator provisions)
|
||||
},
|
||||
|
||||
"federation": {
|
||||
"enabled": true, // OPTIONAL. Mode C forces false.
|
||||
"domain_whitelist": [
|
||||
// Synapse federation_domain_whitelist — allowlist ONLY
|
||||
"mosaic.woltje.com",
|
||||
"mosaic.uscllc.com",
|
||||
],
|
||||
"peers": [
|
||||
// operator-declared peer sites (for room/fleet wiring)
|
||||
{ "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true },
|
||||
],
|
||||
},
|
||||
|
||||
"tls": {
|
||||
"acme": {
|
||||
"directory_url": "https://acme.mosaic.woltje.com/acme/acme/directory", // step-ca OR https://acme-v02.api.letsencrypt.org/directory
|
||||
"ca_kind": "step-ca | letsencrypt", // informational label; the directory_url is the real switch
|
||||
"challenge": "dns-01 | http-01 | tls-alpn-01",
|
||||
"account_email": "[email protected]", // ACME account contact
|
||||
"eab": { "kid": null, "hmac_key_ref": null }, // External Account Binding if the CA requires it (some step-ca provisioners) — secret via SecretBackend
|
||||
},
|
||||
"client_tls_mode": "acme | self-signed", // Mode C may use self-signed for local C-S TLS (weaker trust, §8)
|
||||
},
|
||||
|
||||
"secrets": {
|
||||
"backend": "vault | vaultwarden", // pluggable, install-time choice (§4)
|
||||
"connection": {
|
||||
"address": "https://vault.woltje.com:8200", // or Vaultwarden/Bitwarden server URL
|
||||
"auth_ref": "…", // how the appservice authenticates to the backend (bootstrap, §4/§8)
|
||||
"namespace_or_org": "mosaic-fleet", // Vault namespace / mount, OR Vaultwarden org id
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 The three modes, concretely
|
||||
|
||||
Exactly three supported modes (Jason's ruling — no others):
|
||||
|
||||
**Mode A — split-domain (identity ≠ host, delegated).** _Jason's PRIMARY._ Federation-capable. This is the recommended production shape because it lets identity live on a clean app-domain while the homeserver runs on a separate operational host.
|
||||
|
||||
```jsonc
|
||||
// Mode A — mosaic.woltje.com identity, matrix.woltje.com host, federated with a second site
|
||||
{
|
||||
"topology": {
|
||||
"mode": "split-domain",
|
||||
"identity": { "server_name": "mosaic.woltje.com", "server_name_kind": "domain" },
|
||||
"homeserver": {
|
||||
"host": "matrix.woltje.com",
|
||||
"port": 443,
|
||||
"client_bind": "https://matrix.woltje.com",
|
||||
"bind_ip": null,
|
||||
},
|
||||
"delegation": {
|
||||
"method": "well-known",
|
||||
"well_known_server": { "m.server": "matrix.woltje.com:443" },
|
||||
},
|
||||
"federation": {
|
||||
"enabled": true,
|
||||
"domain_whitelist": ["mosaic.woltje.com", "mosaic.uscllc.com"],
|
||||
"peers": [{ "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true }],
|
||||
},
|
||||
"tls": {
|
||||
"acme": {
|
||||
"directory_url": "https://acme-v02.api.letsencrypt.org/directory", // public LE, or a step-ca directory
|
||||
"ca_kind": "letsencrypt",
|
||||
"challenge": "dns-01",
|
||||
"account_email": "[email protected]",
|
||||
},
|
||||
"client_tls_mode": "acme",
|
||||
},
|
||||
"secrets": {
|
||||
"backend": "vaultwarden",
|
||||
"connection": { "address": "https://vw.woltje.com", "namespace_or_org": "mosaic-fleet" },
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
MXIDs on this instance: `@mosaic_coordinator-1:mosaic.woltje.com`. A human/agent's homeserver is discovered by resolving `.well-known/matrix/server` on `mosaic.woltje.com` → `matrix.woltje.com:443`.
|
||||
|
||||
**Mode B — single-domain (identity == host).** Simpler; the `server_name` _is_ the host. No delegation needed. Federation-capable (still needs DNS + cert on that one domain).
|
||||
|
||||
```jsonc
|
||||
// Mode B — one domain does everything
|
||||
{
|
||||
"topology": {
|
||||
"mode": "single-domain",
|
||||
"identity": { "server_name": "matrix.example.org", "server_name_kind": "domain" },
|
||||
"homeserver": {
|
||||
"host": "matrix.example.org",
|
||||
"port": 8448,
|
||||
"client_bind": "https://matrix.example.org",
|
||||
"bind_ip": null,
|
||||
},
|
||||
"delegation": { "method": "none" },
|
||||
"federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // optional — off here
|
||||
"tls": {
|
||||
"acme": {
|
||||
"directory_url": "https://acme-v02.api.letsencrypt.org/directory",
|
||||
"ca_kind": "letsencrypt",
|
||||
"challenge": "http-01",
|
||||
"account_email": "[email protected]",
|
||||
},
|
||||
"client_tls_mode": "acme",
|
||||
},
|
||||
"secrets": {
|
||||
"backend": "vault",
|
||||
"connection": { "address": "https://vault.example.org:8200", "namespace_or_org": "mosaic" },
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
MXIDs: `@mosaic_coordinator-1:matrix.example.org`. Here `server_name == host`, so `@:matrix.example.org` is both the identity domain and where Synapse actually answers.
|
||||
|
||||
**Mode C — IP-only standalone (no DNS, no federation).** Fully supported for local/airgapped/homelab-without-DNS. `server_name` is an IP:port literal. **Cannot federate — ever — in this mode** (federation requires a real domain + valid CA cert; §2.4). Client TLS may be self-signed (weaker trust, §8).
|
||||
|
||||
```jsonc
|
||||
// Mode C — airgapped / local, no DNS, standalone only
|
||||
{
|
||||
"topology": {
|
||||
"mode": "ip-only-standalone",
|
||||
"identity": { "server_name": "192.168.1.50:8448", "server_name_kind": "ip" },
|
||||
"homeserver": {
|
||||
"host": "192.168.1.50",
|
||||
"port": 8448,
|
||||
"client_bind": "https://192.168.1.50:8448",
|
||||
"bind_ip": "192.168.1.50",
|
||||
},
|
||||
"delegation": { "method": "none" },
|
||||
"federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // FORCED false in Mode C
|
||||
"tls": { "acme": null, "client_tls_mode": "self-signed" }, // may use a private step-ca or self-signed for C-S TLS
|
||||
"secrets": {
|
||||
"backend": "vaultwarden",
|
||||
"connection": { "address": "http://192.168.1.51:8080", "namespace_or_org": "mosaic-local" },
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
MXIDs: `@mosaic_coordinator-1:192.168.1.50:8448`. **Warning surfaced at install:** this `server_name` is an IP literal; if the operator ever wants federation they must move to a domain, which is an **identity re-home** (§5.3, §7).
|
||||
|
||||
> **[VERIFY]** Synapse accepts an `ip:port` `server_name` and mints usable MXIDs against it for local/standalone use. This is believed workable for non-federated operation but must be validated against the deployed Synapse version; some Synapse versions/tools assume a DNS-resolvable `server_name`. If an IP literal is rejected, Mode C falls back to a **fabricated local domain** (e.g. `mosaic.local`) resolved via `/etc/hosts` or a local resolver — still standalone-only, same re-home caveat.
|
||||
|
||||
### 2.4 The hard federation gate (Jason's HARD STOP)
|
||||
|
||||
**Federation REQUIRES DNS + valid certificates. This is a hard stop, enforced by the installer and by the config validator, not a suggestion.**
|
||||
|
||||
| Precondition | Why | Enforced where |
|
||||
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| `server_name` resolves in public/peer DNS (or delegated target does) | S2S discovery uses `.well-known`/SRV over DNS; peers must resolve you | installer reachability check (§6); config validator rejects `federation.enabled=true` with `server_name_kind=ip` |
|
||||
| Valid TLS cert on the federation endpoint, chained to a CA the peer trusts | S2S is TLS; a peer validates your cert. Self-signed/untrusted ⇒ peer refuses ⇒ you are defederated | installer cert probe (§6); cert monitor (§3.4) |
|
||||
| Federation `domain_whitelist` non-empty and mutually consistent with peers | allowlist-only federation (RFC-001 NG5/§6) | config validator |
|
||||
|
||||
**IP-only ⇒ federation is impossible.** There is no valid public/peer CA cert for a bare IP in our trust model (and we will not ship a self-signed S2S trust hack — NG5). Therefore **Mode C is standalone-only by construction**, and the config validator makes `mode=ip-only-standalone ∧ federation.enabled=true` an **illegal state that cannot be persisted.**
|
||||
|
||||
This is the honest, load-bearing boundary of the whole topology model:
|
||||
|
||||
```
|
||||
DNS + valid cert?
|
||||
┌─────────────┴─────────────┐
|
||||
YES NO
|
||||
│ │
|
||||
Mode A or B Mode C (IP-only)
|
||||
federation OPTIONAL STANDALONE ONLY
|
||||
(opt-in, allowlisted) (federation impossible)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Certificate provisioning — one ACME integration
|
||||
|
||||
### 3.1 Single integration, directory-URL as the switch
|
||||
|
||||
We build **exactly one** cert-provisioning integration: an **ACME client integration**. Both supported CAs are ACME CAs. The operator does **not** choose between two code paths; they choose an **ACME directory URL** and a **challenge type**. That is the entire surface.
|
||||
|
||||
| CA choice | What it is | ACME directory URL (illustrative) | Why an operator picks it |
|
||||
| ----------------------- | ------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **step-ca (Smallstep)** | self-hosted **private** ACME CA | `https://acme.<internal-domain>/acme/<provisioner>/directory` | **Total control**; issues for **private/internal/split-horizon domains** a public CA can't (e.g. `mosaic.internal`, RFC-1918 split-horizon); airgap-friendly; you own the root |
|
||||
| **Let's Encrypt** | public ACME CA | `https://acme-v02.api.letsencrypt.org/directory` (staging: `.../acme-staging-v02...`) | **Ease of use**; universally trusted chain (ISRG Root X1); zero CA to operate; ideal for public domains |
|
||||
|
||||
Because both speak ACME, the same client (account key, order, authorization, challenge, finalize, cert-fetch, renew) drives either. The `ca_kind` label in config is informational for UX; the **`directory_url` is the real determinant**. **[VERIFY]** whether the chosen ACME library requires per-CA quirks (LE rate limits, staging switch; step-ca **External Account Binding** on some provisioners — if EAB is required the `kid`/`hmac_key` come from the SecretBackend, §4).
|
||||
|
||||
### 3.2 Challenge-type matrix (which challenge for which topology)
|
||||
|
||||
The operator picks one challenge type per the domains they're covering. This is the crux for **public vs private/split-horizon**:
|
||||
|
||||
| Challenge | How it proves control | Best for | Cannot / caveat |
|
||||
| --------------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **HTTP-01** | CA GETs `http://<domain>/.well-known/acme-challenge/<token>` on port 80 | **Public, single hostname**, port 80 reachable from CA (Mode B, or Mode A's homeserver host) | Needs inbound :80 from the CA; **cannot** do wildcards; **useless for private domains** a public CA can't reach |
|
||||
| **DNS-01** | CA checks a `_acme-challenge.<domain>` **TXT** record you publish | **Private / split-horizon / internal domains**, **wildcards**, and any domain where inbound HTTP from the CA is impossible | Requires **programmatic DNS API** access to publish TXT (or manual for step-ca where you own the resolver). **This is the answer for step-ca on private domains** and for Mode A when the homeserver host isn't publicly HTTP-reachable |
|
||||
| **TLS-ALPN-01** | CA connects TLS on :443 with ALPN `acme-tls/1` | Public host where **:443 is free** but :80 is blocked | Needs the ACME client to own the :443 TLS handshake briefly; awkward behind some reverse proxies — **[VERIFY]** against our proxy (RFC-001 terminates TLS at a reverse proxy) |
|
||||
|
||||
**Guidance baked into the installer:**
|
||||
|
||||
- **Let's Encrypt + public domain, port 80 open →** HTTP-01 (simplest). Wildcard or no inbound :80 → DNS-01.
|
||||
- **step-ca + private/internal/split-horizon domain →** **DNS-01** (the private CA can validate against a resolver you control; public HTTP reachability is irrelevant). This is the combination that lets a private homelab domain get real certs.
|
||||
- **:443-only public host →** TLS-ALPN-01.
|
||||
|
||||
### 3.3 Delegation setup for split-domain (Mode A)
|
||||
|
||||
For Mode A, the cert and the delegation must agree. Concretely, at install for `server_name=mosaic.woltje.com`, host `matrix.woltje.com`:
|
||||
|
||||
1. **Cert(s):** obtain a valid cert for **`matrix.woltje.com`** (the federation/host endpoint — this is where the TLS handshake actually lands). If serving `.well-known` over HTTPS on `mosaic.woltje.com`, that origin **also** needs a valid cert for `mosaic.woltje.com`. So Mode A typically provisions **two** SANs/certs: identity-domain (`mosaic.woltje.com`, serves `.well-known`) and host (`matrix.woltje.com`, serves S2S+C-S). **[VERIFY]** whether a single multi-SAN cert is preferable operationally.
|
||||
2. **Delegation record**, one of:
|
||||
- **`.well-known`:** serve `https://mosaic.woltje.com/.well-known/matrix/server` → `{"m.server":"matrix.woltje.com:443"}` (and `.well-known/matrix/client` for C-S discovery so agents/Element find the host).
|
||||
- **SRV:** `_matrix._tcp.mosaic.woltje.com. IN SRV 10 0 443 matrix.woltje.com.` The installer **documents and validates** the record but the operator provisions it in their DNS (we don't run their DNS). **[VERIFY]** `.well-known` vs SRV precedence on the deployed Synapse.
|
||||
3. **Validate:** installer fetches the operator's own `.well-known`/SRV and confirms it points at the configured host, and that the host presents a valid cert (§6). Only then does it declare Mode A "federation-ready."
|
||||
|
||||
### 3.4 Renewal & monitoring — a lapsed federation cert silently defederates
|
||||
|
||||
**This is the operational trap and it must alarm.** ACME certs are short-lived (LE = 90 days; step-ca often shorter by policy). A federation cert that lapses does **not** throw a loud error — peers simply **stop trusting the S2S handshake and the site silently drops out of federation.** From inside, everything looks fine; from peers, the site went dark. That is exactly the "homelab went dark and took comms with it" trauma (RFC-001 §5), but caused by a cert, not a host.
|
||||
|
||||
Requirements:
|
||||
|
||||
- **Auto-renew** on the standard ACME schedule (renew at ~⅓ lifetime remaining; LE guidance ~30 days before expiry). The ACME integration owns this loop.
|
||||
- **Expiry monitoring as a first-class alarm.** Emit cert-days-remaining into OTEL/Jaeger metrics (consistent with RFC-001 §8's "monitor for cert expiry — a cert lapse silently defederates"). Alarm thresholds (e.g. warn <14d, critical <3d) are **runtime-tunable** config (§5).
|
||||
- **Federation-health probe:** periodically resolve our own delegation and validate our own cert _as a peer would_ (external vantage where possible), so a broken renewal is caught as "we would fail a peer's validation" before a peer notices.
|
||||
- **Escalation tie-in:** a critical cert-expiry or federation-health failure raises a `mosaic.escalation` (RFC-001 §4.2/§5) into the HIL room. A cert lapse is a fleet-visibility incident, not a silent config drift.
|
||||
|
||||
---
|
||||
|
||||
## 4. Secret backend interface
|
||||
|
||||
### 4.1 The `SecretBackend` contract
|
||||
|
||||
A single pluggable interface. The appservice and orchestrator depend on the **interface**, never on Vault or Vaultwarden directly. Chosen at install; swappable without touching callers. Illustrative contract (decomposition-ready, not frozen):
|
||||
|
||||
```ts
|
||||
interface SecretBackend {
|
||||
// --- static secret CRUD (appservice tokens, ACME EAB, DB creds) ---
|
||||
get(ref: SecretRef): Promise<SecretValue>;
|
||||
put(ref: SecretRef, value: SecretValue, opts?: { immutable?: boolean }): Promise<void>;
|
||||
rotate(
|
||||
ref: SecretRef,
|
||||
next: SecretValue,
|
||||
): Promise<{ previous: SecretVersion; current: SecretVersion }>;
|
||||
list(prefix: SecretRef): Promise<SecretRef[]>;
|
||||
delete(ref: SecretRef): Promise<void>;
|
||||
|
||||
// --- agent-credential lifecycle (the fleet-identity part) ---
|
||||
enrollAgent(input: {
|
||||
agentSlug: string;
|
||||
scope: CredentialScope; // which rooms/secrets this agent may read
|
||||
ttl?: Duration; // ephemeral-by-default per RFC-001 §8
|
||||
}): Promise<AgentCredentialHandle>; // wraps the per-agent access_token + optional pubkey record
|
||||
|
||||
revokeAgent(agentSlug: string): Promise<void>; // must be authoritative & immediate
|
||||
|
||||
// --- health / bootstrap ---
|
||||
health(): Promise<BackendHealth>;
|
||||
authenticateSelf(bootstrap: BootstrapAuth): Promise<void>; // how the appservice/orchestrator logs into the backend
|
||||
}
|
||||
```
|
||||
|
||||
Design intent: **`get/put/rotate`** cover the static crown-jewel secrets (appservice `hs_token`/`as_token`, ACME account/EAB keys, DB DSN). **`enrollAgent/revokeAgent`** cover the _fleet-identity_ lifecycle — this is where RFC-001's "mint per-agent token at enroll, discard on teardown" (RFC-001 §4.1, §8) actually lands.
|
||||
|
||||
### 4.2 How appservice / agent tokens map onto it
|
||||
|
||||
RFC-001 defines three tiers of Matrix secret. They map cleanly:
|
||||
|
||||
| RFC-001 secret | Sensitivity | `SecretBackend` treatment |
|
||||
| ------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **`as_token`** (AS→HS, acts as any namespaced user) | crown jewel | `put(immutable-ish)` + `rotate`; **only** the appservice may `get` it; stored under a fleet-admin scope; never handed to an agent |
|
||||
| **`hs_token`** (HS→AS callback auth) | crown jewel | same as `as_token`; both live only in appservice scope (RFC-001 §8) |
|
||||
| **enroll bootstrap secret / orchestrator-signed nonce** | high | `get` by orchestrator + appservice only; used to authenticate `POST /enroll` so a rogue process can't enroll a rogue agent (RFC-001 §8, B5) |
|
||||
| **per-agent `access_token`** | transient runtime | minted via `enrollAgent`, scoped to that agent, **short-TTL / re-mintable**, discarded on teardown; **not** durably persisted (RFC-001 §8 says per-agent tokens are transient) — the backend may hold a short-lived handle or a personal-vault entry for the agent's own lifetime |
|
||||
| **Ed25519 signed-authorship keypair** | high (private key) | private key generated **agent-side**, only pubkey leaves the agent (RFC-001 §4.4/§8); the SecretBackend stores the **pubkey record** for audit; per-spin keys need no at-rest custody |
|
||||
|
||||
The key blast-radius property (RFC-001 §8) is preserved: agents receive **only their own** credential via `enrollAgent`; the `as_token` never leaves appservice scope.
|
||||
|
||||
### 4.3 Vault implementation
|
||||
|
||||
Vault maps naturally:
|
||||
|
||||
- Static secrets → **KV v2** at a mount/namespace (`mosaic-fleet/`), with versioning giving `rotate` semantics for free.
|
||||
- **`enrollAgent`** → issue a scoped, TTL'd token or use **AppRole** / a scoped policy per agent; Vault's native TTL + revocation is exactly the transient per-agent model. `revokeAgent` → Vault token/lease revoke (authoritative, immediate).
|
||||
- **`authenticateSelf`** → the appservice authenticates to Vault via AppRole (role_id from config, secret_id injected at deploy) or a platform auth method; consistent with how Gateway/DB secrets are handled today (RFC-001 §8). **[VERIFY]** align with whatever KBN-101 lands for Mosaic secret management (CLAUDE.md flags secrets work in flight).
|
||||
- **Trade-off (honesty):** Vault is the most capable backend but is **not** the free-and-simple default for a hobbyist stranger; hence it must not be _forced_ (G6).
|
||||
|
||||
### 4.4 Vaultwarden implementation + the org/enroll/revoke agent-account model
|
||||
|
||||
Vaultwarden (self-hostable Bitwarden-compatible server) is the **open-source-ethos default candidate** — free, self-hostable, familiar. Jason's model, mapped onto Bitwarden/Vaultwarden's org primitives:
|
||||
|
||||
1. **Operator creates one or more Bitwarden orgs** at install (e.g. `mosaic-fleet`).
|
||||
2. **The orchestrator is enrolled into the org and granted authority** to enroll/revoke agent sessions — it is the org's automation principal (admin/manager over an agents **collection**).
|
||||
3. **Agents get scoped credential access:** each agent gets access to a **collection** (or a personal vault provisioned for its spin) holding exactly the secrets its scope allows. `enrollAgent` = grant the agent principal access to its collection + provision its per-agent Matrix token entry; `revokeAgent` = remove the agent principal / revoke its access, immediately.
|
||||
4. **User + agents share scoped access:** the human operator and the agents both hold credentials in the same org, scoped by collection — humans and agents on one secret surface, mirroring RFC-001's "humans and agents on one comms surface" pattern.
|
||||
|
||||
Mapping to the interface:
|
||||
|
||||
| Interface op | Vaultwarden mechanism |
|
||||
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `get/put/rotate` (static) | items in an org **collection**; rotate = new item version / replace + old-version audit |
|
||||
| `enrollAgent` | create/attach agent principal to its **collection**; provision per-agent token item, TTL enforced by our teardown (Vaultwarden itself is not TTL-native — see [VERIFY]) |
|
||||
| `revokeAgent` | revoke the agent principal's org membership / collection access |
|
||||
| `authenticateSelf` | orchestrator authenticates as the org automation principal |
|
||||
|
||||
**HONEST MATURITY FLAG — [VERIFY]:** Bitwarden's clean **machine-account / service-account** primitive lives in **Bitwarden Secrets Manager**, and **Vaultwarden's coverage of Secrets Manager / machine accounts is partial and evolving.** What is known to work today on Vaultwarden: **orgs, collections, per-user (incl. a per-agent "user") vaults, and collection-scoped sharing.** What **may not** be fully there: the polished **machine-account API**, native short-TTL service credentials, and fine-grained programmatic access-token issuance equivalent to hosted Bitwarden Secrets Manager. **[VERIFY]** current Vaultwarden version's Secrets Manager / machine-account support before P-level commitment.
|
||||
|
||||
**Why this is not a blocker:** the interface is designed so **either backend is viable**. If Vaultwarden's machine-account API isn't ready, the Vaultwarden adapter implements `enrollAgent` via the **personal-vault-per-agent + org-collection** model that works _today_ (create an agent principal, share the scoped collection, we enforce TTL via orchestrator teardown rather than backend-native TTL). If an operator needs backend-native short-TTL machine credentials now, they choose the **Vault** adapter. **We are not blocked on Vaultwarden maturing**, because the `SecretBackend` abstraction lets the polished-machine-account behavior land later without changing any caller.
|
||||
|
||||
---
|
||||
|
||||
## 5. Config system
|
||||
|
||||
### 5.1 Storage & precedence
|
||||
|
||||
Config is **DB-backed** (Postgres, per the stack — CLAUDE.md/RFC-001), with sane defaults compiled into the product and install-time overrides. **Precedence, highest wins:**
|
||||
|
||||
```
|
||||
install-time value > DB override (runtime) > compiled default
|
||||
```
|
||||
|
||||
- **Compiled default** — ships in the product; what a stranger gets with zero config for every non-topology-critical key.
|
||||
- **Install-time value** — captured by the installer (§6), written to DB, and for **install-time-immutable** keys, **locked** (marked non-overridable).
|
||||
- **DB override** — runtime tuning via admin surface, allowed **only** for keys classified runtime-tunable.
|
||||
|
||||
> Nuance: "install-time > DB override" applies to **immutable** keys — the install-time value is frozen and a DB override of it is rejected. For **tunable** keys, the DB override is the live value and the install-time value is just the initial seed. The classification (§5.3) is what makes the precedence unambiguous per key.
|
||||
|
||||
### 5.2 DB schema shape
|
||||
|
||||
Illustrative (Drizzle/Postgres, per stack conventions):
|
||||
|
||||
```sql
|
||||
-- one row per config key
|
||||
CREATE TABLE comms_config (
|
||||
key text PRIMARY KEY, -- e.g. 'topology.identity.server_name'
|
||||
value jsonb NOT NULL, -- current effective value
|
||||
source text NOT NULL, -- 'install' | 'db-override' | 'default'
|
||||
mutability text NOT NULL, -- 'install-immutable' | 'runtime-tunable'
|
||||
set_by text, -- operator/agent/system that set it
|
||||
set_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT immutable_not_overridable
|
||||
CHECK (NOT (mutability = 'install-immutable' AND source = 'db-override'))
|
||||
);
|
||||
|
||||
-- append-only audit of every change (esp. attempted immutable changes)
|
||||
CREATE TABLE comms_config_audit (
|
||||
id bigserial PRIMARY KEY,
|
||||
key text NOT NULL,
|
||||
old_value jsonb,
|
||||
new_value jsonb,
|
||||
actor text NOT NULL,
|
||||
action text NOT NULL, -- 'set' | 'override' | 'rejected-immutable'
|
||||
at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
The DB `CHECK` is a belt-and-braces backstop; the application-layer config service enforces mutability and records rejected immutable-change attempts in the audit table. Secrets are **referenced** here (a `SecretRef`), never stored inline — actual secret values live in the `SecretBackend` (§4).
|
||||
|
||||
### 5.3 Install-time-immutable vs runtime-tunable — the key table
|
||||
|
||||
The single most important classification: **what can never change after install** vs **what an operator tunes anytime.** Getting `server_name` on the wrong side of this line is a foot-gun that orphans every identity.
|
||||
|
||||
| Config key | Mutability | Rationale / cost of change |
|
||||
| ------------------------------------------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `topology.mode` (A/B/C) | **install-immutable** | changing mode changes identity/federation semantics; a mode change is effectively a reinstall/re-home |
|
||||
| `topology.identity.server_name` | **install-immutable** | **baked into every MXID and room alias.** Changing it re-homes every identity — see re-home note below. **This is THE immutable value.** |
|
||||
| `topology.identity.server_name_kind` | **install-immutable** | domain↔ip change is a re-home (Mode C→A/B) |
|
||||
| `topology.homeserver.host` | install-immutable (**delegation-tunable**) | in Mode A you _can_ move the host if you update delegation to match — the identity is unchanged; treat as immutable-with-migration, not free |
|
||||
| `topology.homeserver.bind_ip` / `port` | runtime-tunable (ops) | operational network binding; no identity impact |
|
||||
| `topology.delegation.method` / records | tunable-with-care | can switch well-known↔SRV as long as both still resolve to the same host; validated on change |
|
||||
| `topology.federation.enabled` | **runtime-tunable (gated)** | can flip **on** only if DNS+cert preconditions pass (§2.4); flipping standalone→federated is the §7 upgrade path |
|
||||
| `topology.federation.domain_whitelist` / `peers` | runtime-tunable | add/remove peers over time; each add re-validated |
|
||||
| `tls.acme.directory_url` / `ca_kind` | runtime-tunable | can switch CA (e.g. LE→step-ca); triggers re-issue; monitor for trust-chain change |
|
||||
| `tls.acme.challenge` | runtime-tunable | switch challenge type if DNS/HTTP reachability changes |
|
||||
| `tls.client_tls_mode` | tunable-with-care | self-signed→acme is fine; acme→self-signed weakens trust (§8) |
|
||||
| `secrets.backend` | install-immutable (**migration-only**) | switching Vault↔Vaultwarden requires a secret migration; not a live flip |
|
||||
| `secrets.connection.*` | runtime-tunable | rotate backend address/auth without changing which backend |
|
||||
| `presence.heartbeat_interval_ms` | runtime-tunable | RFC-001 §4.5 default ~30s; pure tuning |
|
||||
| `presence.miss_tolerance` | runtime-tunable | RFC-001 §4.5 default 2 |
|
||||
| `escalation.dark_threshold_min` | runtime-tunable | RFC-001 §5/§11 — default 10min → fallback |
|
||||
| `escalation.hil_threshold_min` | runtime-tunable | RFC-001 §5 — default +5min → HIL |
|
||||
| `cert.expiry_warn_days` / `expiry_critical_days` | runtime-tunable | §3.4 alarm thresholds (default 14 / 3) |
|
||||
| `federation.health_probe_interval` | runtime-tunable | §3.4 |
|
||||
|
||||
**Identity re-home note (the cost of changing `server_name`):** because every MXID (`@mosaic_*:server_name`) and room alias (`#…:server_name`) embeds `server_name`, changing it means: every agent gets a **new identity**, all rooms must be **recreated/re-aliased**, signed-authorship pubkey records re-published, and federation peers re-pointed. There is **no in-place rename** in Matrix. Hence `server_name` is install-immutable and the installer gates it behind an explicit warning (§6.7). Changing it is a **migration/reinstall**, honestly (§7).
|
||||
|
||||
### 5.4 How the appservice / homeserver read config
|
||||
|
||||
- **Synapse (homeserver)** reads a _rendered_ `homeserver.yaml`. The config service **renders** Synapse's config (server_name, listeners, `federation_domain_whitelist`, `enable_registration: false`, appservice registration path, TLS/delegation) from the DB-backed config at deploy/reconfigure time. Synapse itself is not DB-config-aware; the source of truth is the product DB, and Synapse config is a **generated artifact**. Certain Synapse values (notably `server_name`) require a **Synapse restart** and are exactly the immutable ones — reinforcing §5.3.
|
||||
- **The appservice** reads config **live** from the DB config service for runtime-tunable values (thresholds, whitelist changes, cert alarm thresholds) and from the `SecretBackend` for secrets. Immutable topology values are read once at boot (they can't change under it).
|
||||
- **`packages/comms` SDK** receives the values it needs (homeserver client URL, presence intervals) from the appservice at enroll (RFC-001 §4.1 returns `{mxid, access_token, homeserver, rooms[]}`), so agents never read the config DB directly.
|
||||
- **Delegation artifacts** (`.well-known/matrix/server`, `.well-known/matrix/client`) are likewise **rendered** from config and served by the reverse proxy / homeserver.
|
||||
|
||||
---
|
||||
|
||||
## 6. Installer UX flow
|
||||
|
||||
A guided installer (`mosaic comms install` or equivalent) that captures topology, provisions certs, wires secrets, and **validates before declaring success.** It must never report success it hasn't proven. Steps:
|
||||
|
||||
**6.1 — Preflight & detection.** Detect existing DNS records for a candidate domain, existing certs, an existing reachable Synapse, an existing Vault/Vaultwarden. Offer detected values as suggestions (never as silent defaults). Detect whether the host has public inbound :80/:443 (informs challenge-type guidance, §3.2).
|
||||
|
||||
**6.2 — Primary instance (ALWAYS).** The PRIMARY/home instance is always configured — there is no "skip primary." Prompt for its identity. This is non-optional and is what makes standalone work out of the box.
|
||||
|
||||
**6.3 — Pick topology mode (A/B/C).** Ask the shape:
|
||||
|
||||
- Do you have a domain? **No →** Mode C (IP-only standalone); warn federation is impossible here (§2.4) and that `server_name` will be an IP (re-home cost if they later want federation).
|
||||
- Yes, and identity domain differs from the homeserver host? **Yes →** Mode A (split-domain); capture `server_name` + host + delegation method.
|
||||
- Yes, one domain does everything? **→** Mode B (single-domain).
|
||||
|
||||
**6.4 — Pick CA (ACME directory).** step-ca vs Let's Encrypt → capture `directory_url`, account email, and challenge type with the §3.2 guidance surfaced (e.g. "private/internal domain? → DNS-01"). Capture EAB if the CA requires it (→ SecretBackend). For step-ca, offer to point at an existing step-ca or document standing one up.
|
||||
|
||||
**6.5 — Pick secret backend.** Vault vs Vaultwarden → capture connection (address, org/namespace, bootstrap auth). If Vaultwarden, walk the org/collection setup (§4.4) and **surface the machine-account maturity [VERIFY]** honestly so the operator chooses eyes-open.
|
||||
|
||||
**6.6 — Federation (OPTIONAL).** Only offered if Mode A/B. Ask whether to enable federation now; if yes, capture peer `server_name`s and build the `domain_whitelist`. If Mode C, federation is not offered (greyed out with the explanation). Federation-off is a first-class, fully-supported outcome.
|
||||
|
||||
**6.7 — The "what can't be changed later" warning gate.** Before writing immutable config, present an explicit confirmation:
|
||||
|
||||
```
|
||||
⚠ IMMUTABLE CHOICES — read before confirming
|
||||
server_name = "mosaic.woltje.com"
|
||||
This becomes part of every agent identity (e.g. @mosaic_coordinator-1:mosaic.woltje.com)
|
||||
and every room alias. It CANNOT be changed later without re-homing every identity
|
||||
(new MXIDs for all agents, recreating all rooms). There is no in-place rename in Matrix.
|
||||
topology.mode = "split-domain" — changing modes later is a reinstall.
|
||||
secrets.backend = "vaultwarden" — switching backends later requires a secret migration.
|
||||
Type the server_name to confirm you understand it is permanent: ____________
|
||||
```
|
||||
|
||||
The operator must **retype `server_name`** to proceed — a deliberate friction gate on the one truly permanent value.
|
||||
|
||||
**6.8 — Provision & validate (no success claim until proven).** The installer then:
|
||||
|
||||
1. Renders Synapse config + delegation artifacts; brings up Synapse with `enable_registration: false`.
|
||||
2. Runs the ACME flow; obtains cert(s); verifies they're valid and installed.
|
||||
3. Authenticates to the SecretBackend; stores `hs_token`/`as_token`, enroll bootstrap; runs `health()`.
|
||||
4. **Reachability & cert validation** appropriate to mode:
|
||||
- Mode A: fetch our own `.well-known`/SRV, confirm it points at host; TLS-probe host cert as a peer would; confirm C-S discovery.
|
||||
- Mode B: TLS-probe the single domain; confirm C-S + (if federation) S2S.
|
||||
- Mode C: confirm local C-S reachability over the bind IP; confirm (self-signed or private-CA) client TLS; **explicitly report "standalone — federation not available."**
|
||||
- If federation enabled: validate each peer resolves + presents a peer-trusted cert; confirm `domain_whitelist` mutual consistency. If any peer fails, **federation is reported NOT-ready** — the primary still succeeds standalone.
|
||||
5. **Only now** declare success, with a per-capability report: `PRIMARY: ✅ | CERT: ✅ (expires in 90d, auto-renew on) | SECRETS: ✅ (vaultwarden) | FEDERATION: ✅ 1 peer / ⚠ not-ready / ⛔ n-a (Mode C)`.
|
||||
|
||||
**6.9 — Post-install.** Emit the cert-expiry monitor + federation-health probe (§3.4) into OTEL; write config to DB with correct mutability flags; print the immutable-values summary again for the record.
|
||||
|
||||
---
|
||||
|
||||
## 7. Standalone → federated upgrade path
|
||||
|
||||
An operator who started standalone later wants federation. The path depends on **whether they started with a domain**:
|
||||
|
||||
**Case 1 — started Mode A or B (had a domain), federation was just off.** _Cheap, no re-home._ `server_name` is already a real domain and identities are already minted against it. To federate:
|
||||
|
||||
1. Ensure DNS resolves for peers (their `server_name`s and yours) — likely already true.
|
||||
2. Ensure a **valid, peer-trusted cert** on the federation endpoint (if they were running client-only self-signed, they now need a real ACME cert; if already ACME, done).
|
||||
3. Set `federation.enabled = true`, populate `domain_whitelist` + `peers` (all runtime-tunable, §5.3).
|
||||
4. Re-run the installer's **federation validation** (§6.8 step 4) against each peer. On green, federation is live. **No identity change** — existing MXIDs simply become reachable cross-site. This is the intended, low-friction upgrade.
|
||||
|
||||
**Case 2 — started Mode C (IP-only), now wants federation.** _Expensive — an identity re-home, and we say so plainly._ Federation requires DNS + a valid cert (§2.4), which an IP `server_name` can never satisfy. So the operator must:
|
||||
|
||||
1. **Acquire a domain** and DNS, and provision a **valid ACME cert** (LE public, or step-ca if the domain is private — but note a _private_ domain can only federate with peers who trust that private CA root, §8).
|
||||
2. **Change `server_name` from the IP literal to the domain** — this is the **install-immutable value**, so this is a **re-home, not a config tweak**:
|
||||
- Every agent identity `@mosaic_*:192.168.1.50:8448` becomes `@mosaic_*:newdomain` — **all new MXIDs.**
|
||||
- Every room + alias must be **recreated** under the new `server_name`.
|
||||
- Signed-authorship pubkey records re-published under the new identities.
|
||||
- Any durable references to old MXIDs (escalation policies, fallback-coordinator targets, RFC-001 §5) must be re-pointed.
|
||||
3. Effectively: **treat it as a fresh install in Mode A/B with a data migration of rooms/history**, not an in-place flip. Matrix has **no in-place `server_name` rename**; this cost is intrinsic to Matrix, not to our design.
|
||||
|
||||
**Honest guidance the installer gives Mode C operators up front (§6.3):** "If there is _any_ chance you'll want to federate later, start with a domain (Mode A/B) even if you keep federation off — flipping federation on later is free, but changing an IP `server_name` to a domain later is a full identity re-home." This lets an informed operator avoid the expensive path by choosing Mode B-with-federation-off instead of Mode C.
|
||||
|
||||
---
|
||||
|
||||
## 8. Security
|
||||
|
||||
**8.1 — Cert trust model per CA choice.**
|
||||
|
||||
- **Let's Encrypt (public):** chains to a universally-trusted root (ISRG). Peers, humans' browsers, and Element trust it with no extra distribution. Best for public domains; nothing to distribute.
|
||||
- **step-ca (private):** chains to a **root you operate**. Nothing trusts it by default. Therefore the **step-ca root must be distributed** to everyone who validates certs: peer homeservers (so cross-site S2S validates — a peer must add your root to its federation trust store, **[VERIFY]** Synapse's mechanism for trusting a custom federation CA), agent hosts, and any human client. This is the price of "total control" and airgap capability. For **federation between two private-CA sites**, both sites must trust each other's roots (or a shared root). Getting this wrong reproduces the silent-defederation failure (§3.4) — a peer that doesn't trust your root silently refuses your S2S.
|
||||
- **Mode C self-signed client TLS:** weakest — see 8.4.
|
||||
|
||||
**8.2 — Federation whitelist.** `federation_domain_whitelist` is a **hard allowlist** (RFC-001 §6/NG5): only listed Mosaic site domains may federate; no public-network federation. The installer/config validator keeps the whitelist consistent with the declared peer list. Adding a peer is an explicit, audited config change.
|
||||
|
||||
**8.3 — Secret-backend auth.** The appservice/orchestrator authenticate to the `SecretBackend` via a **bootstrap credential injected at deploy** (Vault AppRole secret_id, or Vaultwarden org automation principal), never committed, consistent with existing Gateway/DB secret handling (RFC-001 §8). The `as_token`/`hs_token` live **only** in backend + appservice memory; agents get only their own scoped, re-mintable token (§4.2). Enroll is authenticated (RFC-001 B5) so a rogue local process can't enroll a rogue agent. Backend access is scoped: an agent's credential can read only its collection/policy, never the fleet-admin scope holding the crown jewels.
|
||||
|
||||
**8.4 — Honest note: IP-only standalone with self-signed client TLS is a weaker-trust local mode.** In Mode C, client TLS may be self-signed (or a local private CA). This means: no third party vouches for the endpoint; clients must be told to trust the self-signed cert (TOFU or manual root import); there is no external validation of who's on the other end. This is **acceptable and supported for local/airgapped/homelab** use where the network is already trusted, but it is **explicitly a weaker trust posture** than a real CA. The installer states this plainly at install (§6.8 Mode C). It is one more reason Mode C cannot federate: we will not extend this weaker-trust local posture across sites (NG5).
|
||||
|
||||
**8.5 — Homeserver hardening** (inherited from RFC-001 §8, config-rendered here): `enable_registration: false` always (agents come only via the appservice), rate-limiting on, admin API bound to localhost/behind auth, media repo locked/disabled if unused, TLS terminated at our controlled proxy. These are **rendered from config** (§5.4) so a stranger gets them by default, not by remembering to set them. **[VERIFY]** current recommended Synapse hardening flags at implementation.
|
||||
|
||||
---
|
||||
|
||||
## 9. How RFC-002 integrates with RFC-001's P1–P5
|
||||
|
||||
RFC-002 is the **substrate**. Each RFC-001 phase consumes a subset of it. Critically, **P1 does not need the hard parts** — presence ships on a single clean-domain instance with no federation, no IP-only, and no secret-rotation story resolved.
|
||||
|
||||
| RFC-001 phase | RFC-002 pieces it NEEDS | RFC-002 pieces it does NOT need yet |
|
||||
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **P1 — Presence (first shippable slice)** | **Mode A or B, single-instance, clean domain** (§2.3). One ACME cert (LE or step-ca) via the single integration (§3). Config system minimal: `server_name` immutable + presence thresholds tunable (§5.3). Installer path 6.1–6.3, 6.4 (cert), 6.7 (immutable gate), 6.8 (validate single-instance). A **minimal** SecretBackend just holding the appservice token. | **No federation** (§2.4 gate irrelevant — single site). **No IP-only** needed for P1 (P1 wants a clean domain so Element/humans connect cleanly). **No agent-credential rotation** maturity. **No peer/whitelist** config. Vaultwarden machine-account [VERIFY] does **not** block P1. |
|
||||
| **P2 — Appservice + auto-enroll** | Full `SecretBackend` **`enrollAgent`/`revokeAgent`** (§4.1), `hs_token`/`as_token` custody (§4.2), enroll-bootstrap secret. Config: room taxonomy, per-agent token classification. Chosen backend (Vault or Vaultwarden) real. | Federation, cross-site, IP-only. |
|
||||
| **P3 — MACP v1 spec** | Config keys for MACP versioning/thresholds as runtime-tunable (§5.3); nothing new topology-wise. | Federation, secret rotation-in-anger. |
|
||||
| **P4 — Federation** | **The whole federation half of RFC-002:** Mode A delegation (§3.3), the hard DNS+cert gate (§2.4), `domain_whitelist`+peers config, cert monitoring/silent-defederation alarm (§3.4), per-CA trust distribution for cross-site (§8.1). This is where standalone→federated (§7 Case 1) and Jason's `mosaic.woltje.com`↔`mosaic.uscllc.com` shape land. | IP-only (federation excludes it by construction). |
|
||||
| **P5 — Hardening + signed-authorship + Hermes retired** | Secret **rotation runbooks** executed in anger (§4, RFC-001 E2), pubkey-record custody for Ed25519 (§4.2), cert-rotation runbook (§3.4), full homeserver hardening validated (§8.5), backend auth review (§8.3). | — |
|
||||
|
||||
**One-line integration statement:** P1 rides on the _smallest_ slice of RFC-002 (single clean-domain instance + one cert + minimal config + minimal secret storage); the federation/IP-only/backend-maturity complexity is deferred to exactly the phases that need it (mostly P4/P5). RFC-002 therefore does not gate P1.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions
|
||||
|
||||
Deliberately few — most topology/cert/secret decisions are resolved by Jason's rulings and baked in above.
|
||||
|
||||
1. **[VERIFY] IP-only `server_name` acceptance.** Does the deployed Synapse version accept an `ip:port` `server_name` and mint usable MXIDs for standalone (§2.3)? If not, Mode C uses a fabricated local domain (`mosaic.local` via local resolver) — confirm which, since it affects the re-home wording for Mode C→A/B (§7).
|
||||
2. **[VERIFY] Vaultwarden machine-account maturity.** Confirm the current Vaultwarden version's Secrets Manager / machine-account coverage (§4.4). Determines whether the Vaultwarden adapter's `enrollAgent` uses native machine accounts or the personal-vault-per-agent + collection model. Does **not** block (interface absorbs either), but sets P2 expectations.
|
||||
3. **[VERIFY] step-ca root distribution for cross-site federation.** Confirm Synapse's supported mechanism for trusting a **custom federation CA root** (§8.1) so two private-CA sites can federate. If Synapse won't easily trust a private federation CA, private-domain federation may in practice require public certs (LE) on the federation SANs even when internal traffic uses step-ca.
|
||||
4. **Default secret backend for the published installer.** Given the open-source ethos (G6), should the installer _default-suggest_ Vaultwarden (free, self-hostable) while clearly offering Vault, or present them neutrally? Recommendation: suggest Vaultwarden as the zero-cost path with the maturity caveat surfaced, Vault as the "I need native short-TTL machine creds now" path. Jason to confirm the framing.
|
||||
5. **Single multi-SAN cert vs two certs in Mode A** (§3.3) — operational preference for identity-domain + host coverage. Minor; validate during P4.
|
||||
6. **Reconfigure-time Synapse restart policy.** Which rendered-config changes (§5.4) require a Synapse restart vs hot-reload on the deployed version, so the config service knows when a tunable change needs a bounce. **[VERIFY]** at implementation.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Real mechanics this RFC leans on (quick reference)
|
||||
|
||||
- **`server_name`** — Synapse identity domain; the `:suffix` of every MXID/alias; install-immutable; changing it = re-home (no in-place rename). Distinct from where Synapse _listens_.
|
||||
- **Delegation** — `https://<server_name>/.well-known/matrix/server` → `{"m.server":"host:port"}` and/or `_matrix._tcp.<server_name>` **SRV**; how identity-domain ≠ host is reconciled (Mode A). **[VERIFY]** precedence on deployed Synapse.
|
||||
- **`.well-known/matrix/client`** — C-S discovery so agents/Element find the homeserver host from the identity domain.
|
||||
- **`federation_domain_whitelist`** — Synapse allowlist; only listed domains federate; our hard no-public-federation boundary.
|
||||
- **ACME** — single provisioning protocol for both CAs; operator supplies **directory URL** (step-ca vs Let's Encrypt) + **challenge type**.
|
||||
- **HTTP-01 / DNS-01 / TLS-ALPN-01** — challenge types; **DNS-01 is the one for private/split-horizon/wildcard**; HTTP-01 for public single host with :80; TLS-ALPN-01 for :443-only public.
|
||||
- **step-ca ACME provisioner** — Smallstep's self-hosted CA exposing an ACME directory; enables private/internal-domain certs and total control; may require **EAB**; root must be distributed to validators.
|
||||
- **Let's Encrypt** — public ACME CA; universally-trusted chain; 90-day certs; staging endpoint for testing.
|
||||
- **Bitwarden/Vaultwarden org + collection + machine/service-account** — org holds collections; collections scope access; machine/service accounts (Bitwarden **Secrets Manager**) are the clean automation primitive but **Vaultwarden coverage is partial/evolving [VERIFY]**; personal-vault-per-agent + org-collection works today.
|
||||
- **Vault KV v2 / AppRole / lease-TTL / revoke** — the capable backend; native versioning=rotate, TTL+revoke=transient per-agent creds.
|
||||
- **Silent defederation** — a lapsed/renewal-failed federation cert causes peers to stop trusting S2S with no local error; must be monitored + alarmed (§3.4).
|
||||
|
||||
_All Matrix/ACME/secret-backend mechanics above are cited from architecture knowledge and MUST be re-verified against the actually deployed versions during implementation — every **[VERIFY]** is a checkpoint, not an assumption. Every illustrative domain (`mosaic.woltje.com`, `mosaic.uscllc.com`, `matrix.woltje.com`) is an operator-supplied example, never a product default or literal._
|
||||
@@ -0,0 +1,19 @@
|
||||
# Monorepo consolidation planning bundle
|
||||
|
||||
> **Status:** Historical planning evidence. The five source records were moved byte-identically from migration quarantine on 2026-08-10.
|
||||
|
||||
This bundle records the decision and proposed work packages for consolidating prior Forge, MACP, and OpenClaw framework work into this monorepo.
|
||||
|
||||
## Records
|
||||
|
||||
- [Consolidation brief](brief.md) — original scope, target layout, constraints, and success criteria.
|
||||
- [Board review](board-review.md) — historical deliberation and conditional approval.
|
||||
- [WP1: Forge package](wp1-forge-package.md) — proposed TypeScript Forge implementation.
|
||||
- [WP2: MACP package](wp2-macp-package.md) — proposed protocol, gate, credential, and event implementation.
|
||||
- [WP3: Mosaic framework plugin](wp3-mosaic-framework-plugin.md) — proposed OpenClaw framework plugin port.
|
||||
|
||||
## Current boundary
|
||||
|
||||
`packages/forge`, `packages/macp`, and `plugins/mosaic-framework` exist in the current checkout. That existence is sufficient to classify this bundle as historical planning, but it does **not** prove that every stated success criterion, integration, coverage target, or behavior remains satisfied.
|
||||
|
||||
Use current package source, manifests, and tests for implementation truth. Do not use this bundle as an active task ledger or as authority to change package behavior.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,231 @@
|
||||
# Brief: Monorepo Consolidation — mosaic/stack → mosaic/mosaic-stack
|
||||
|
||||
## Source
|
||||
|
||||
Architecture consolidation — merge the mosaic/stack repo (Forge pipeline, MACP protocol, framework tools) into mosaic/mosaic-stack (Harness Foundation platform). Two repos doing related work that need to converge.
|
||||
|
||||
## Context
|
||||
|
||||
**mosaic/stack** (OLD) contains:
|
||||
|
||||
- Forge progressive refinement pipeline (stages, agents, personas, rails, debate protocol, brief classification)
|
||||
- MACP protocol (JSON schemas, deterministic Python controller, dispatcher, event system, gate runner)
|
||||
- Credential resolver (Python — OC config, mosaic files, ambient env, JSON5 parser)
|
||||
- OC framework plugin (injects Mosaic rails into all agent sessions)
|
||||
- Profiles (runtime-neutral context packs for tech stacks and domains)
|
||||
- Stage adapter (Forge→MACP bridge)
|
||||
- Board tasks (multi-agent board evaluation)
|
||||
- OpenBrain specialist memory (learning capture/recall)
|
||||
- 17 guides, 5 universal skills
|
||||
|
||||
**mosaic/mosaic-stack** (NEW) contains:
|
||||
|
||||
- Harness Foundation platform (NestJS gateway, Next.js web, Drizzle ORM, Pi SDK runtime)
|
||||
- 5 provider adapters, task classifier, routing rules, model capability matrix
|
||||
- MACP OC plugin (ACP runtime backend with Pi bridge)
|
||||
- TS coord package (mission runner, tasks file manager, status tracker — 1635 lines)
|
||||
- BullMQ job queue, OTEL telemetry, channel plugins (Discord, Telegram)
|
||||
- CLI with TUI, 65/65 tasks done, v0.2.0
|
||||
|
||||
**Decision:** NEW repo is the base. All unique work from OLD gets ported into NEW as packages.
|
||||
|
||||
## Scope
|
||||
|
||||
### Work Package 1: Forge Pipeline Package (`packages/forge`)
|
||||
|
||||
Port the entire Forge progressive refinement pipeline as a TypeScript package.
|
||||
|
||||
**From OLD:**
|
||||
|
||||
- `forge/pipeline/stages/*.md` — 11 stage definitions
|
||||
- `forge/pipeline/agents/{board,generalists,specialists,cross-cutting}/*.md` — all persona definitions
|
||||
- `forge/pipeline/rails/*.md` — debate protocol, dynamic composition, worker rails
|
||||
- `forge/pipeline/gates/` — gate reviewer definitions
|
||||
- `forge/pipeline/orchestrator/run-structure.md` — file-based observability spec
|
||||
- `forge/templates/` — brief and PRD templates
|
||||
- `forge/pipeline/orchestrator/board_tasks.py` → rewrite in TS
|
||||
- `forge/pipeline/orchestrator/stage_adapter.py` → rewrite in TS
|
||||
- `forge/pipeline/orchestrator/pipeline_runner.py` → rewrite in TS
|
||||
- `forge/forge` CLI (Python) → rewrite in TS, integrate with `packages/cli`
|
||||
|
||||
**Package structure:**
|
||||
|
||||
```
|
||||
packages/forge/
|
||||
├── src/
|
||||
│ ├── index.ts # Public API
|
||||
│ ├── pipeline-runner.ts # Orchestrates full pipeline run
|
||||
│ ├── stage-adapter.ts # Maps stages to MACP/coord tasks
|
||||
│ ├── board-tasks.ts # Multi-agent board evaluation task generator
|
||||
│ ├── brief-classifier.ts # strategic/technical/hotfix classification
|
||||
│ ├── types.ts # Stage specs, run manifest, gate results
|
||||
│ └── constants.ts # Stage sequence, timeouts, labels
|
||||
├── pipeline/
|
||||
│ ├── stages/ # .md stage definitions (copied)
|
||||
│ ├── agents/ # .md persona definitions (copied)
|
||||
│ │ ├── board/
|
||||
│ │ ├── cross-cutting/
|
||||
│ │ ├── generalists/
|
||||
│ │ └── specialists/
|
||||
│ │ ├── language/
|
||||
│ │ └── domain/
|
||||
│ ├── rails/ # .md rails (copied)
|
||||
│ ├── gates/ # .md gate definitions (copied)
|
||||
│ └── templates/ # brief + PRD templates (copied)
|
||||
└── package.json
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
|
||||
- Pipeline markdown assets are runtime data, not compiled — ship as-is in the package
|
||||
- `pipeline-runner.ts` calls into `packages/coord` for task execution (not a separate controller)
|
||||
- Stage adapter generates coord-compatible tasks, not MACP JSON directly
|
||||
- Board tasks use `depends_on_policy: "all_terminal"` for synthesis
|
||||
- Per-stage timeouts from `STAGE_TIMEOUTS` map
|
||||
- Brief classifier supports CLI flag, YAML frontmatter, and keyword auto-detection
|
||||
- Run output goes to project-scoped `.forge/runs/{run-id}/` (not inside the Forge package)
|
||||
|
||||
**Persona override system (new):**
|
||||
|
||||
- Base personas ship with the package (read-only)
|
||||
- Project-level overrides in `.forge/personas/{role}.md` extend (not replace) base personas
|
||||
- Board composition configurable via `.forge/config.yaml`:
|
||||
```yaml
|
||||
board:
|
||||
additional_members:
|
||||
- compliance-officer.md
|
||||
skip_members: []
|
||||
specialists:
|
||||
always_include:
|
||||
- proxmox-expert
|
||||
```
|
||||
- OpenBrain integration for cross-run specialist memory (when enabled)
|
||||
|
||||
### Work Package 2: MACP Protocol Package (`packages/macp`)
|
||||
|
||||
Port the MACP protocol layer, event system, and gate runner as a TypeScript package.
|
||||
|
||||
**From OLD:**
|
||||
|
||||
- `tools/macp/protocol/task.schema.json` — task JSON schema
|
||||
- `tools/macp/protocol/` — event schemas
|
||||
- `tools/macp/controller/gate_runner.py` → rewrite in TS as `gate-runner.ts`
|
||||
- `tools/macp/events/` — event watcher, webhook adapter, Discord formatter → rewrite in TS
|
||||
- `tools/macp/dispatcher/credential_resolver.py` → rewrite in TS as `credential-resolver.ts`
|
||||
- `tools/macp/memory/learning_capture.py` + `learning_recall.py` → rewrite in TS
|
||||
|
||||
**Package structure:**
|
||||
|
||||
```
|
||||
packages/macp/
|
||||
├── src/
|
||||
│ ├── index.ts # Public API
|
||||
│ ├── types.ts # Task, event, result, gate types
|
||||
│ ├── schemas/ # JSON schemas (copied)
|
||||
│ ├── gate-runner.ts # Mechanical + AI review quality gates
|
||||
│ ├── credential-resolver.ts # Provider credential resolution (mosaic files, OC config, ambient)
|
||||
│ ├── event-emitter.ts # Append events to ndjson, structured event types
|
||||
│ ├── event-watcher.ts # Poll events.ndjson with cursor persistence
|
||||
│ ├── webhook-adapter.ts # POST events to configurable URL
|
||||
│ ├── discord-formatter.ts # Human-readable event messages
|
||||
│ └── learning.ts # OpenBrain capture + recall
|
||||
└── package.json
|
||||
```
|
||||
|
||||
**Integration with existing packages:**
|
||||
|
||||
- `packages/coord` uses `packages/macp` for event emission, gate running, and credential resolution
|
||||
- `plugins/macp` uses `packages/macp` for protocol types and credential resolution
|
||||
- `packages/forge` uses `packages/macp` gate types for stage gates
|
||||
|
||||
### Work Package 3: OC Framework Plugin (`plugins/mosaic-framework`)
|
||||
|
||||
Port the OC framework plugin that injects Mosaic rails into all agent sessions.
|
||||
|
||||
**From OLD:**
|
||||
|
||||
- `oc-plugins/mosaic-framework/index.ts` — `before_agent_start` + `subagent_spawning` hooks
|
||||
- `oc-plugins/mosaic-framework/openclaw.plugin.json`
|
||||
|
||||
**Structure:**
|
||||
|
||||
```
|
||||
plugins/mosaic-framework/
|
||||
├── src/
|
||||
│ └── index.ts # Plugin hooks
|
||||
└── package.json
|
||||
```
|
||||
|
||||
**This is separate from `plugins/macp`:**
|
||||
|
||||
- `mosaic-framework` = injects Mosaic rails/contracts into every OC session (passive enforcement)
|
||||
- `macp` = provides an ACP runtime backend for MACP task execution (active runtime)
|
||||
|
||||
### Work Package 4: Profiles + Guides + Skills
|
||||
|
||||
Port reference content as a documentation/config package or top-level directories.
|
||||
|
||||
**From OLD:**
|
||||
|
||||
- `profiles/domains/*.json` — HIPAA, fintech, crypto context packs
|
||||
- `profiles/tech-stacks/*.json` — NestJS, Next.js, FastAPI, React conventions
|
||||
- `profiles/workflows/*.json` — API development, frontend component, testing workflows
|
||||
- `guides/*.md` — 17 guides (auth, backend, QA, orchestrator, PRD, etc.)
|
||||
- `skills-universal/` — jarvis, macp, mosaic-standards, prd, setup-cicd skills
|
||||
|
||||
**Destination:**
|
||||
|
||||
```
|
||||
profiles/ # Top-level (same as OLD)
|
||||
guides/ # Top-level (same as OLD)
|
||||
skills/ # Top-level (renamed from skills-universal)
|
||||
```
|
||||
|
||||
These are runtime-neutral assets consumed by any agent or profile loader — they don't belong in a compiled package.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Rewriting the NestJS orchestrator app from OLD (`apps/orchestrator/`) — its functionality is subsumed by `packages/coord` + `apps/gateway`
|
||||
- Porting the FastAPI coordinator from OLD (`apps/coordinator/`) — its functionality (webhook receiver, issue parser, quality orchestrator) is handled by `packages/coord` + `apps/gateway` in the new architecture
|
||||
- Porting the Prisma schema or OLD's `apps/api` — Drizzle migration is complete
|
||||
- Old Docker Compose configs (Traefik, Matrix, OpenBao) — NEW has its own infra setup
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. `packages/forge` exists with all 11 stage definitions, all persona markdowns, all rails, and TS implementations of pipeline-runner, stage-adapter, board-tasks, and brief-classifier
|
||||
2. `packages/macp` exists with gate-runner, credential-resolver, event system, and learning capture/recall — all in TypeScript
|
||||
3. `plugins/mosaic-framework` exists and registers OC hooks for rails injection
|
||||
4. Profiles, guides, and skills are present at top-level
|
||||
5. `packages/forge` integrates with `packages/coord` for task execution
|
||||
6. `packages/macp` credential-resolver is used by `plugins/macp` Pi bridge
|
||||
7. All existing tests pass (no regressions)
|
||||
8. New packages have test coverage ≥85%
|
||||
9. `pnpm lint && pnpm typecheck && pnpm build` passes
|
||||
10. `.forge/runs/` project-scoped output directory works for at least one test run
|
||||
|
||||
## Technical Constraints
|
||||
|
||||
- All new code is ESM with NodeNext module resolution
|
||||
- No Python in the new repo — everything rewrites to TypeScript
|
||||
- Pipeline markdown assets (stages, personas, rails) are shipped as package data, not compiled
|
||||
- Credential resolver must support: mosaic credential files, OC config (JSON5), ambient environment — same resolution order as the Python version
|
||||
- Must preserve `depends_on_policy` semantics (all, any, all_terminal)
|
||||
- Per-stage timeouts must be preserved
|
||||
- JSON5 stripping must use the placeholder-extraction approach (not naive regex on string content)
|
||||
|
||||
## Estimated Complexity
|
||||
|
||||
High — crosses 4 work packages with protocol porting, TS rewrites, and integration wiring. Each work package is independently shippable.
|
||||
|
||||
**Suggested execution order:**
|
||||
|
||||
1. WP4 (profiles/guides/skills) — pure copy, no code, fast win
|
||||
2. WP2 (packages/macp) — protocol foundation, needed by WP1 and WP3
|
||||
3. WP1 (packages/forge) — the big one, depends on WP2
|
||||
4. WP3 (plugins/mosaic-framework) — OC integration, can parallel with WP1
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `packages/coord` must be stable (it is — WP1 integrates with it)
|
||||
- `plugins/macp` must be stable (it is — WP2 provides types/credentials to it)
|
||||
- Pi SDK (`@mariozechner/pi-agent-core`) already in the dependency tree
|
||||
@@ -0,0 +1,265 @@
|
||||
# WP1: packages/forge — Forge Pipeline Package
|
||||
|
||||
## Context
|
||||
|
||||
Port the Forge progressive refinement pipeline from Python (~/src/mosaic-stack/forge/) to TypeScript as `packages/forge` in this monorepo. The pipeline markdown assets (stages, agents, personas, rails, gates, templates) are already copied to `packages/forge/pipeline/`. This task is the TypeScript implementation layer.
|
||||
|
||||
**Board decisions that constrain this work:**
|
||||
|
||||
- Abstract TaskExecutor interface — packages/forge must NOT hard-import packages/coord. Define an abstract interface; coord satisfies it.
|
||||
- Clean index.ts exports, no internal path leakage, no hardcoded paths
|
||||
- 85% test coverage on TS implementation files (markdown assets excluded)
|
||||
- Test strategy for non-deterministic AI orchestration: fixture-based integration tests
|
||||
- OpenBrain is OUT OF SCOPE
|
||||
- ESM only, zero Python
|
||||
|
||||
**Dependencies available:**
|
||||
|
||||
- `@mosaicstack/macp` (packages/macp) is built and provides: GateEntry, GateResult, Task types, credential resolution, gate running, event emission
|
||||
|
||||
## Source Files (Python → TypeScript)
|
||||
|
||||
### 1. types.ts
|
||||
|
||||
Define all Forge-specific types:
|
||||
|
||||
```typescript
|
||||
// Stage specification
|
||||
interface StageSpec {
|
||||
number: string;
|
||||
title: string;
|
||||
dispatch: 'exec' | 'yolo' | 'pi';
|
||||
type: 'research' | 'review' | 'coding' | 'deploy';
|
||||
gate: string;
|
||||
promptFile: string;
|
||||
qualityGates: (string | GateEntry)[];
|
||||
}
|
||||
|
||||
// Brief classification
|
||||
type BriefClass = 'strategic' | 'technical' | 'hotfix';
|
||||
type ClassSource = 'cli' | 'frontmatter' | 'auto';
|
||||
|
||||
// Run manifest (persisted to disk)
|
||||
interface RunManifest {
|
||||
runId: string;
|
||||
brief: string;
|
||||
codebase: string;
|
||||
briefClass: BriefClass;
|
||||
classSource: ClassSource;
|
||||
forceBoard: boolean;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
currentStage: string;
|
||||
status: 'in_progress' | 'completed' | 'failed' | 'interrupted' | 'rejected';
|
||||
stages: Record<string, StageStatus>;
|
||||
}
|
||||
|
||||
// Abstract task executor (decouples from packages/coord)
|
||||
interface TaskExecutor {
|
||||
submitTask(task: ForgeTask): Promise<void>;
|
||||
waitForCompletion(taskId: string, timeoutMs: number): Promise<TaskResult>;
|
||||
}
|
||||
|
||||
// Persona override config
|
||||
interface ForgeConfig {
|
||||
board?: {
|
||||
additionalMembers?: string[];
|
||||
skipMembers?: string[];
|
||||
};
|
||||
specialists?: {
|
||||
alwaysInclude?: string[];
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### 2. constants.ts
|
||||
|
||||
**Source:** Top of `~/src/mosaic-stack/forge/lib` (ALL_STAGES, LABELS, STAGE_SPECS equivalent) + `~/src/mosaic-stack/forge/pipeline/orchestrator/stage_adapter.py` (STAGE_TIMEOUTS)
|
||||
|
||||
```typescript
|
||||
export const STAGE_SEQUENCE = [
|
||||
'00-intake',
|
||||
'00b-discovery',
|
||||
'01-board',
|
||||
'01b-brief-analyzer',
|
||||
'02-planning-1',
|
||||
'03-planning-2',
|
||||
'04-planning-3',
|
||||
'05-coding',
|
||||
'06-review',
|
||||
'07-remediate',
|
||||
'08-test',
|
||||
'09-deploy',
|
||||
];
|
||||
|
||||
export const STAGE_TIMEOUTS: Record<string, number> = {
|
||||
'00-intake': 120,
|
||||
'00b-discovery': 300,
|
||||
'01-board': 120,
|
||||
'02-planning-1': 600,
|
||||
// ... etc
|
||||
};
|
||||
|
||||
export const STAGE_LABELS: Record<string, string> = {
|
||||
'00-intake': 'INTAKE',
|
||||
// ... etc
|
||||
};
|
||||
```
|
||||
|
||||
Also: STRATEGIC_KEYWORDS, TECHNICAL_KEYWORDS for brief classification.
|
||||
|
||||
### 3. brief-classifier.ts
|
||||
|
||||
**Source:** `classify_brief()`, `parse_brief_frontmatter()`, `stages_for_class()` from `~/src/mosaic-stack/forge/lib`
|
||||
|
||||
- Auto-classify brief by keyword analysis (strategic vs technical)
|
||||
- Parse YAML frontmatter for explicit `class:` field
|
||||
- CLI flag override
|
||||
- Return stage list based on classification (strategic = full pipeline, technical = skip board, hotfix = skip board + brief analyzer)
|
||||
|
||||
### 4. stage-adapter.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/forge/pipeline/orchestrator/stage_adapter.py`
|
||||
|
||||
- `mapStageToTask()`: Convert a Forge stage into a task compatible with TaskExecutor
|
||||
- Stage briefs written to `{runDir}/{stageName}/brief.md`
|
||||
- Result paths at `{runDir}/{stageName}/result.json`
|
||||
- Previous results read from disk at runtime (not baked into brief)
|
||||
- Per-stage timeouts from STAGE_TIMEOUTS
|
||||
- depends_on chain built from stage sequence
|
||||
|
||||
### 5. board-tasks.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/forge/pipeline/orchestrator/board_tasks.py`
|
||||
|
||||
- `loadBoardPersonas()`: Read all .md files from `pipeline/agents/board/`
|
||||
- `generateBoardTasks()`: One task per persona + synthesis task
|
||||
- Synthesis depends on all persona tasks with `depends_on_policy: 'all_terminal'`
|
||||
- Persona briefs include role description + brief under review
|
||||
- Synthesis script merges independent reviews into board memo
|
||||
|
||||
### 6. pipeline-runner.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/forge/pipeline/orchestrator/pipeline_runner.py` + `~/src/mosaic-stack/forge/lib` (cmd_run, cmd_resume, cmd_status)
|
||||
|
||||
- `runPipeline(briefPath, projectRoot, options)`: Full pipeline execution
|
||||
- Creates run directory at `{projectRoot}/.forge/runs/{runId}/`
|
||||
- Generates tasks for all stages, submits to TaskExecutor
|
||||
- Tracks manifest.json with stage statuses
|
||||
- `resumePipeline(runDir)`: Pick up from last incomplete stage
|
||||
- `getPipelineStatus(runDir)`: Read manifest and report
|
||||
|
||||
**Key difference from Python:** Run output goes to PROJECT-scoped `.forge/runs/`, not inside the Forge package.
|
||||
|
||||
### 7. Persona Override System (NEW — not in Python)
|
||||
|
||||
- Base personas read from `packages/forge/pipeline/agents/`
|
||||
- Project overrides read from `{projectRoot}/.forge/personas/{role}.md`
|
||||
- Merge strategy: project persona content APPENDED to base persona (not replaced)
|
||||
- Board composition configurable via `{projectRoot}/.forge/config.yaml`
|
||||
- If no project config exists, use defaults (all base personas, no overrides)
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
packages/forge/
|
||||
├── src/
|
||||
│ ├── index.ts
|
||||
│ ├── types.ts
|
||||
│ ├── constants.ts
|
||||
│ ├── brief-classifier.ts
|
||||
│ ├── stage-adapter.ts
|
||||
│ ├── board-tasks.ts
|
||||
│ ├── pipeline-runner.ts
|
||||
│ └── persona-loader.ts
|
||||
├── pipeline/ # Already copied (WP4) — markdown assets
|
||||
│ ├── stages/
|
||||
│ ├── agents/
|
||||
│ ├── rails/
|
||||
│ ├── gates/
|
||||
│ └── templates/
|
||||
├── __tests__/
|
||||
│ ├── brief-classifier.test.ts
|
||||
│ ├── stage-adapter.test.ts
|
||||
│ ├── board-tasks.test.ts
|
||||
│ ├── pipeline-runner.test.ts
|
||||
│ └── persona-loader.test.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── vitest.config.ts
|
||||
```
|
||||
|
||||
## Package.json
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@mosaicstack/forge",
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./src/index.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@mosaicstack/macp": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vitest": "workspace:*",
|
||||
"typescript": "workspace:*"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Only dependency: @mosaicstack/macp (for gate types, event emission).
|
||||
|
||||
## Test Strategy (Board requirement)
|
||||
|
||||
**Deterministic code (brief-classifier, stage-adapter, board-tasks, persona-loader, constants):**
|
||||
|
||||
- Standard unit tests with known inputs/outputs
|
||||
- 100% of classification logic, stage mapping, persona loading covered
|
||||
|
||||
**Non-deterministic code (pipeline-runner):**
|
||||
|
||||
- Fixture-based integration tests using a mock TaskExecutor
|
||||
- Mock executor returns pre-recorded results for each stage
|
||||
- Tests verify: manifest progression, stage ordering, dependency enforcement, resume behavior, error handling
|
||||
- NO real AI calls in tests
|
||||
|
||||
**Markdown assets:** Excluded from coverage measurement (configure vitest to exclude `pipeline/` directory).
|
||||
|
||||
## ESM Requirements
|
||||
|
||||
- `"type": "module"` in package.json
|
||||
- NodeNext module resolution in tsconfig
|
||||
- `.js` extensions in all imports
|
||||
- No CommonJS
|
||||
|
||||
## Key Design: Abstract TaskExecutor
|
||||
|
||||
```typescript
|
||||
// In packages/forge/src/types.ts
|
||||
export interface TaskExecutor {
|
||||
submitTask(task: ForgeTask): Promise<void>;
|
||||
waitForCompletion(taskId: string, timeoutMs: number): Promise<TaskResult>;
|
||||
getTaskStatus(taskId: string): Promise<TaskStatus>;
|
||||
}
|
||||
|
||||
// In packages/coord (or wherever the concrete impl lives)
|
||||
export class CoordTaskExecutor implements TaskExecutor {
|
||||
// ... uses packages/coord runner
|
||||
}
|
||||
```
|
||||
|
||||
This means packages/forge can be tested with a mock executor and deployed with any backend.
|
||||
|
||||
## Asset Resolution
|
||||
|
||||
Pipeline markdown assets (stages, personas, rails) must be resolved relative to the package installation, NOT hardcoded paths:
|
||||
|
||||
```typescript
|
||||
// Use import.meta.url to find package root
|
||||
const PACKAGE_ROOT = new URL('..', import.meta.url).pathname;
|
||||
const PIPELINE_DIR = path.join(PACKAGE_ROOT, 'pipeline');
|
||||
```
|
||||
|
||||
Project-level overrides resolved relative to projectRoot parameter.
|
||||
@@ -0,0 +1,150 @@
|
||||
# WP2: packages/macp — MACP Protocol Package
|
||||
|
||||
## Context
|
||||
|
||||
Port the MACP protocol layer from Python (in ~/src/mosaic-stack/tools/macp/) to TypeScript as `packages/macp` in this monorepo. This package provides the foundational protocol types, quality gate execution, credential resolution, and event system that `packages/coord` and `plugins/macp` depend on.
|
||||
|
||||
**Board decisions that constrain this work:**
|
||||
|
||||
- No Python in the new repo — everything rewrites to TypeScript
|
||||
- OpenBrain learning capture/recall is OUT OF SCOPE (deferred to future brief)
|
||||
- 85% test coverage on TS implementation files
|
||||
- Credential resolver behavior must be captured as test fixtures BEFORE rewrite
|
||||
- Clean index.ts exports, no internal path leakage
|
||||
|
||||
## Source Files (Python → TypeScript)
|
||||
|
||||
### 1. credential-resolver.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/tools/macp/dispatcher/credential_resolver.py`
|
||||
|
||||
Resolution order (MUST preserve exactly):
|
||||
|
||||
1. Mosaic credential files (`~/.config/mosaic/credentials/{provider}.env`)
|
||||
2. OpenClaw config (`~/.openclaw/openclaw.json`) — env block + models.providers.{provider}.apiKey
|
||||
3. Ambient environment variables
|
||||
4. CredentialError (failure)
|
||||
|
||||
Key behaviors to preserve:
|
||||
|
||||
- Provider registry: anthropic, openai, zai → env var names + credential file paths + OC config paths
|
||||
- Dotenv parser: handles single/double quotes, comments, blank lines
|
||||
- JSON5 stripping: placeholder-extraction approach (NOT naive regex) — protects URLs and timestamps inside string values
|
||||
- OC config permission check: warn on world-readable, skip if wrong owner
|
||||
- Redacted marker detection: `__OPENCLAW_REDACTED__` values skipped
|
||||
- Task-level override via `credentials.provider_key_env`
|
||||
|
||||
### 2. gate-runner.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/tools/macp/controller/gate_runner.py`
|
||||
|
||||
Three gate types:
|
||||
|
||||
- `mechanical`: shell command, pass = exit code 0
|
||||
- `ai-review`: shell command producing JSON, parse findings, fail on blockers
|
||||
- `ci-pipeline`: placeholder (always passes for now)
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- `normalize_gate()`: accepts string or dict, normalizes to gate entry
|
||||
- `run_gate()`: executes single gate, returns result with pass/fail
|
||||
- `run_gates()`: executes all gates, emits events, returns (all_passed, results)
|
||||
- AI review parsing: `_count_ai_findings()` reads stats.blockers or findings[].severity
|
||||
- `fail_on` modes: "blocker" (default) or "any"
|
||||
|
||||
### 3. event-emitter.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/tools/macp/controller/gate_runner.py` (emit_event, append_event functions) + `~/src/mosaic-stack/tools/macp/events/`
|
||||
|
||||
- Append structured events to ndjson file
|
||||
- Event types: task.assigned, task.started, task.completed, task.failed, task.escalated, task.gated, task.retry.scheduled, rail.check.started, rail.check.passed, rail.check.failed
|
||||
- Each event: event_id (uuid), event_type, task_id, status, timestamp, source, message, metadata
|
||||
|
||||
### 4. types.ts
|
||||
|
||||
**Source:** `~/src/mosaic-stack/tools/macp/protocol/task.schema.json`
|
||||
|
||||
TypeScript types for:
|
||||
|
||||
- Task (id, title, status, dispatch, runtime, depends_on, depends_on_policy, quality_gates, timeout_seconds, metadata, etc.)
|
||||
- Event (event_id, event_type, task_id, status, timestamp, source, message, metadata)
|
||||
- GateResult (command, exit_code, type, passed, output, findings, blockers)
|
||||
- TaskResult (task_id, status, completed_at, exit_code, gate_results, files_changed, etc.)
|
||||
- CredentialError, ProviderRegistry
|
||||
|
||||
### 5. schemas/ (copy)
|
||||
|
||||
Copy `~/src/mosaic-stack/tools/macp/protocol/task.schema.json` as-is.
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
packages/macp/
|
||||
├── src/
|
||||
│ ├── index.ts
|
||||
│ ├── types.ts
|
||||
│ ├── credential-resolver.ts
|
||||
│ ├── gate-runner.ts
|
||||
│ ├── event-emitter.ts
|
||||
│ └── schemas/
|
||||
│ └── task.schema.json
|
||||
├── __tests__/
|
||||
│ ├── credential-resolver.test.ts
|
||||
│ ├── gate-runner.test.ts
|
||||
│ └── event-emitter.test.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── vitest.config.ts
|
||||
```
|
||||
|
||||
## Package.json
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@mosaicstack/macp",
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./src/index.ts"
|
||||
},
|
||||
"dependencies": {},
|
||||
"devDependencies": {
|
||||
"vitest": "workspace:*",
|
||||
"typescript": "workspace:*"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Zero external dependencies. Uses node:fs, node:path, node:child_process, node:crypto only.
|
||||
|
||||
## Test Requirements
|
||||
|
||||
Port ALL existing Python tests as TypeScript equivalents:
|
||||
|
||||
- `test_resolve_from_file` → credential file resolution
|
||||
- `test_resolve_from_ambient` → ambient env resolution
|
||||
- `test_resolve_from_oc_config_env_block` → OC config env block
|
||||
- `test_resolve_from_oc_config_provider_apikey` → OC config provider
|
||||
- `test_oc_config_precedence` → mosaic file wins over OC config
|
||||
- `test_oc_config_missing_file` → graceful fallback
|
||||
- `test_json5_strip` → structural transforms
|
||||
- `test_json5_strip_urls_and_timestamps` → URLs/timestamps survive
|
||||
- `test_redacted_values_skipped` → redacted marker detection
|
||||
- `test_oc_config_permission_warning` → file permission check
|
||||
- `test_resolve_missing_raises` → CredentialError thrown
|
||||
- Gate runner: mechanical pass/fail, AI review parsing, ci-pipeline placeholder
|
||||
- Event emitter: append to ndjson, event structure validation
|
||||
|
||||
## ESM Requirements
|
||||
|
||||
- `"type": "module"` in package.json
|
||||
- NodeNext module resolution in tsconfig
|
||||
- `.js` extensions in all imports
|
||||
- No CommonJS (`require`, `module.exports`)
|
||||
|
||||
## Integration Points
|
||||
|
||||
After this package is built:
|
||||
|
||||
- `packages/coord` should import `@mosaicstack/macp` for event emission and gate types
|
||||
- `plugins/macp` should import `@mosaicstack/macp` for credential resolution and protocol types
|
||||
@@ -0,0 +1,63 @@
|
||||
# WP3: plugins/mosaic-framework — OC Rails Injection Plugin
|
||||
|
||||
## Context
|
||||
|
||||
Port the OpenClaw framework plugin from ~/src/mosaic-stack/oc-plugins/mosaic-framework/ to `plugins/mosaic-framework` in this monorepo. This plugin injects Mosaic framework contracts (rails, completion gates, worktree requirements) into every OpenClaw agent session.
|
||||
|
||||
**This is SEPARATE from plugins/macp:**
|
||||
|
||||
- `mosaic-framework` = passive enforcement — injects rails into all OC sessions
|
||||
- `macp` = active runtime — provides ACP backend for MACP task execution
|
||||
|
||||
## Source Files
|
||||
|
||||
**Source:** `~/src/mosaic-stack/oc-plugins/mosaic-framework/`
|
||||
|
||||
- `index.ts` — plugin hooks (before_agent_start, subagent_spawning)
|
||||
- `openclaw.plugin.json` — plugin manifest
|
||||
- `package.json`
|
||||
|
||||
## What It Does
|
||||
|
||||
### For OC native agents (before_agent_start hook):
|
||||
|
||||
- Injects Mosaic global hard rules via `appendSystemContext`
|
||||
- Completion gates: code review ✓ | security review ✓ | tests GREEN ✓ | CI green ✓
|
||||
- Worker completion protocol: open PR → fire system event → EXIT — never merge
|
||||
- Worktree requirement: `~/src/{repo}-worktrees/{task-slug}`, never `/tmp`
|
||||
- Injects dynamic mission state via `prependContext` (reads from project's `.mosaic/orchestrator/mission.json`)
|
||||
|
||||
### For ACP coding workers (subagent_spawning hook):
|
||||
|
||||
- Writes `~/.codex/instructions.md` or `~/.claude/CLAUDE.md` BEFORE the process starts
|
||||
- Full runtime contract: mandatory load order, hard gates, mode declaration
|
||||
- Global framework rules + worktree + completion gate requirements
|
||||
|
||||
## Implementation
|
||||
|
||||
Port the TypeScript source, updating hardcoded paths to be configurable. The OC plugin SDK imports should reference the installed OpenClaw location dynamically (not hardcoded `/home/jarvis/` paths like the OLD version).
|
||||
|
||||
**Structure:**
|
||||
|
||||
```
|
||||
plugins/mosaic-framework/
|
||||
├── src/
|
||||
│ └── index.ts
|
||||
├── openclaw.plugin.json
|
||||
├── package.json
|
||||
└── tsconfig.json
|
||||
```
|
||||
|
||||
## Key Constraint
|
||||
|
||||
The plugin SDK imports in the OLD version use absolute paths:
|
||||
|
||||
```typescript
|
||||
import type { OpenClawPluginApi } from '/home/jarvis/.npm-global/lib/node_modules/openclaw/dist/plugin-sdk/index.js';
|
||||
```
|
||||
|
||||
This must be resolved dynamically or via a peer dependency. Check how `plugins/macp` handles this in the new repo and follow the same pattern.
|
||||
|
||||
## Tests
|
||||
|
||||
Minimal — plugin hooks are integration-tested against OC runtime. Unit test the context string builders and config resolution.
|
||||
Reference in New Issue
Block a user