chore(orchestrator): Bootstrap storage abstraction retrofit
Mission: Decouple gateway from hardcoded Postgres/Valkey backends. 20 tasks across 5 phases. Estimated total: ~214K tokens. Phase 1: Interface extraction (4 tasks) Phase 2: Wrap existing backends as adapters (5 tasks) Phase 3: Local tier implementation (4 tasks) Phase 4: Config + CLI commands (4 tasks) Phase 5: Migration + docs (3 tasks)
This commit is contained in:
Jarvis
@@ -1,73 +1,30 @@
|
|||||||
# Tasks — Harness Foundation
|
# Tasks — Storage Abstraction Retrofit
|
||||||
|
|
||||||
> Single-writer: orchestrator only. Workers read but never modify.
|
> Single-writer: orchestrator only. Workers read but never modify.
|
||||||
>
|
>
|
||||||
|
> **Mission:** Decouple gateway from hardcoded Postgres/Valkey backends. Introduce interface-driven middleware so the gateway is backend-agnostic. Default to local tier (SQLite + JSON) for zero-dependency installs.
|
||||||
|
>
|
||||||
> **`agent` column values:** `codex` | `sonnet` | `haiku` | `glm-5` | `opus` | `—` (auto/default)
|
> **`agent` column values:** `codex` | `sonnet` | `haiku` | `glm-5` | `opus` | `—` (auto/default)
|
||||||
|
|
||||||
| id | status | agent | milestone | description | pr | notes |
|
| id | status | description | issue | repo | branch | depends_on | blocks | agent | started_at | completed_at | estimate | used | notes |
|
||||||
| ------ | ------ | ------ | ------------------ | ------------------------------------------------------------------ | ---- | ----------- |
|
| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | ----- | -------------------- | ------------------------ | ----------------------------- | --------- | ----- | ---------- | ------------ | -------- | ---- | ----- |
|
||||||
| M1-001 | done | sonnet | M1: Persistence | Wire ChatGateway → ConversationsRepo for user messages | #292 | #224 closed |
|
| SA-P1-001 | not-started | Define QueueAdapter interface in packages/queue/src/types.ts — enqueue, dequeue, length, publish, subscribe, close | | queue | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | |
|
||||||
| M1-002 | done | sonnet | M1: Persistence | Wire agent event relay → ConversationsRepo for assistant responses | #292 | #225 closed |
|
| SA-P1-002 | not-started | Define StorageAdapter interface in packages/storage/src/types.ts — CRUD, query, find, transaction, close | | storage | feat/storage-abstraction | | SA-P1-004 | codex | | | 8K | | |
|
||||||
| M1-003 | done | sonnet | M1: Persistence | Store message metadata: model, provider, tokens, tool calls | #292 | #226 closed |
|
| SA-P1-003 | not-started | Define MemoryAdapter interface in packages/memory/src/types.ts — preferences, insights, search, embedder support | | memory | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | |
|
||||||
| M1-004 | done | sonnet | M1: Persistence | Load message history into Pi session on resume | #301 | #227 closed |
|
| SA-P1-004 | not-started | Create adapter factory pattern + config types: createQueue(config), createStorage(config), createMemory(config) | | queue,storage,memory | feat/storage-abstraction | SA-P1-001,SA-P1-002,SA-P1-003 | SA-P2-001 | codex | | | 8K | | |
|
||||||
| M1-005 | done | sonnet | M1: Persistence | Context window management: summarize when >80% | #301 | #228 closed |
|
| SA-P2-001 | not-started | Refactor @mosaic/queue: move queue.ts → adapters/bullmq.ts, implement QueueAdapter interface, export factory | | queue | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | |
|
||||||
| M1-006 | done | sonnet | M1: Persistence | Conversation search endpoint | #299 | #229 closed |
|
| SA-P2-002 | not-started | Create @mosaic/storage package: move @mosaic/db Drizzle logic → adapters/postgres.ts, implement StorageAdapter interface | | storage | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 15K | | |
|
||||||
| M1-007 | done | sonnet | M1: Persistence | TUI /history command | #297 | #230 closed |
|
| SA-P2-003 | not-started | Refactor @mosaic/memory: extract pgvector logic → adapters/pgvector.ts, implement MemoryAdapter interface | | memory | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | |
|
||||||
| M1-008 | done | sonnet | M1: Persistence | Verify persistence — 20 tests | #304 | #231 closed |
|
| SA-P2-004 | not-started | Update gateway database.module.ts, queue.module.ts, memory.module.ts to use factories + NestJS DI tokens | | gateway | feat/storage-abstraction | SA-P2-001,SA-P2-002,SA-P2-003 | SA-P2-005 | codex | | | 15K | | |
|
||||||
| M2-001 | done | sonnet | M2: Security | InsightsRepo userId on searchByEmbedding | #290 | #232 closed |
|
| SA-P2-005 | not-started | Verify Phase 2: existing Postgres/Valkey behavior unchanged — all existing tests pass, typecheck clean | | gateway | feat/storage-abstraction | SA-P2-004 | SA-P3-001 | codex | | | 10K | | |
|
||||||
| M2-002 | done | sonnet | M2: Security | InsightsRepo userId on findByUser/decay | #290 | #233 closed |
|
| SA-P3-001 | not-started | Implement local queue adapter: in-process Map + JSON file persistence in adapters/local.ts | | queue | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 10K | | |
|
||||||
| M2-003 | done | sonnet | M2: Security | PreferencesRepo userId verified | #294 | #234 closed |
|
| SA-P3-002 | not-started | Implement SQLite storage adapter: better-sqlite3 with schema mirroring Drizzle tables, in adapters/sqlite.ts | | storage | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 20K | | |
|
||||||
| M2-004 | done | sonnet | M2: Security | Memory tools userId injection fixed | #294 | #235 closed |
|
| SA-P3-003 | not-started | Implement keyword memory adapter: TF-IDF or simple keyword matching, no vector dependency, in adapters/keyword.ts | | memory | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 12K | | |
|
||||||
| M2-005 | done | sonnet | M2: Security | ConversationsRepo ownership checks | #293 | #236 closed |
|
| SA-P3-004 | not-started | Verify Phase 3: gateway starts with local config (no PG/Valkey), basic CRUD works, all adapter tests pass | | gateway | feat/storage-abstraction | SA-P3-001,SA-P3-002,SA-P3-003 | SA-P4-001 | codex | | | 15K | | |
|
||||||
| M2-006 | done | sonnet | M2: Security | AgentsRepo findAccessible scoped | #293 | #237 closed |
|
| SA-P4-001 | not-started | Add mosaic.config.ts schema + loader: storage tier selection (local/team), backend config, defaults to local | | config | feat/storage-abstraction | SA-P3-004 | SA-P4-002 | codex | | | 10K | | |
|
||||||
| M2-007 | done | sonnet | M2: Security | Cross-user isolation — 28 tests | #305 | #238 closed |
|
| SA-P4-002 | not-started | CLI: mosaic gateway init — interactive wizard to generate mosaic.config.ts with tier selection | | cli | feat/storage-abstraction | SA-P4-001 | SA-P4-003 | codex | | | 12K | | |
|
||||||
| M2-008 | done | sonnet | M2: Security | Valkey SCAN + /gc admin-only | #298 | #239 closed |
|
| SA-P4-003 | not-started | CLI: mosaic gateway start/stop/status — lifecycle management for the gateway daemon | | cli | feat/storage-abstraction | SA-P4-002 | SA-P4-004 | codex | | | 12K | | |
|
||||||
| M3-001 | done | sonnet | M3: Providers | IProviderAdapter + OllamaAdapter | #306 | #240 closed |
|
| SA-P4-004 | not-started | Verify Phase 4: fresh install with `mosaic gateway init && mosaic gateway start` works end-to-end | | cli,gateway | feat/storage-abstraction | SA-P4-003 | SA-P5-001 | codex | | | 10K | | |
|
||||||
| M3-002 | done | sonnet | M3: Providers | AnthropicAdapter | #309 | #241 closed |
|
| SA-P5-001 | not-started | Migration tooling: mosaic storage export/import for local↔postgres tier migration | | cli,storage | feat/storage-abstraction | SA-P4-004 | SA-P5-002 | codex | | | 15K | | |
|
||||||
| M3-003 | done | sonnet | M3: Providers | OpenAIAdapter | #310 | #242 closed |
|
| SA-P5-002 | not-started | Docker Compose profiles: local (gateway only) vs team (gateway+pg+valkey), update docker-compose.yml | | infra | feat/storage-abstraction | SA-P5-001 | SA-P5-003 | codex | | | 8K | | |
|
||||||
| M3-004 | done | sonnet | M3: Providers | OpenRouterAdapter | #311 | #243 closed |
|
| SA-P5-003 | not-started | Final verification + docs: README update, architecture diagram, configuration guide | | docs | feat/storage-abstraction | SA-P5-002 | | codex | | | 10K | | |
|
||||||
| M3-005 | done | sonnet | M3: Providers | ZaiAdapter (GLM-5) | #314 | #244 closed |
|
|
||||||
| M3-006 | done | sonnet | M3: Providers | Ollama embedding support | #311 | #245 closed |
|
|
||||||
| M3-007 | done | sonnet | M3: Providers | Provider health checks | #308 | #246 closed |
|
|
||||||
| M3-008 | done | sonnet | M3: Providers | Model capability matrix | #303 | #247 closed |
|
|
||||||
| M3-009 | done | sonnet | M3: Providers | EmbeddingService → Ollama default | #308 | #248 closed |
|
|
||||||
| M3-010 | done | sonnet | M3: Providers | OAuth token storage (AES-256-GCM) | #317 | #249 closed |
|
|
||||||
| M3-011 | done | sonnet | M3: Providers | Provider credentials CRUD | #317 | #250 closed |
|
|
||||||
| M3-012 | done | sonnet | M3: Providers | Verify providers — 40 tests | #319 | #251 closed |
|
|
||||||
| M4-001 | done | sonnet | M4: Routing | routing_rules DB schema | #315 | #252 closed |
|
|
||||||
| M4-002 | done | sonnet | M4: Routing | Condition types | #315 | #253 closed |
|
|
||||||
| M4-003 | done | sonnet | M4: Routing | Action types | #315 | #254 closed |
|
|
||||||
| M4-004 | done | sonnet | M4: Routing | Default routing rules (11 seeds) | #316 | #255 closed |
|
|
||||||
| M4-005 | done | sonnet | M4: Routing | Task classifier (60+ tests) | #316 | #256 closed |
|
|
||||||
| M4-006 | done | sonnet | M4: Routing | Routing decision pipeline | #318 | #257 closed |
|
|
||||||
| M4-007 | done | sonnet | M4: Routing | /model override | #323 | #258 closed |
|
|
||||||
| M4-008 | done | sonnet | M4: Routing | Routing transparency in session:info | #323 | #259 closed |
|
|
||||||
| M4-009 | done | sonnet | M4: Routing | Routing rules CRUD API | #320 | #260 closed |
|
|
||||||
| M4-010 | done | sonnet | M4: Routing | Per-user routing overrides | #320 | #261 closed |
|
|
||||||
| M4-011 | done | sonnet | M4: Routing | Agent specialization capabilities | #320 | #262 closed |
|
|
||||||
| M4-012 | done | sonnet | M4: Routing | Routing wired into ChatGateway | #323 | #263 closed |
|
|
||||||
| M4-013 | done | sonnet | M4: Routing | Verify routing — 9 E2E tests | #323 | #264 closed |
|
|
||||||
| M5-001 | done | sonnet | M5: Sessions | Agent config loaded on session create | #323 | #265 closed |
|
|
||||||
| M5-002 | done | sonnet | M5: Sessions | /model command end-to-end | #323 | #266 closed |
|
|
||||||
| M5-003 | done | sonnet | M5: Sessions | /agent command mid-session | #323 | #267 closed |
|
|
||||||
| M5-004 | done | sonnet | M5: Sessions | Session ↔ conversation binding | #321 | #268 closed |
|
|
||||||
| M5-005 | done | sonnet | M5: Sessions | Session info broadcast | #321 | #269 closed |
|
|
||||||
| M5-006 | done | sonnet | M5: Sessions | /agent new from TUI | #321 | #270 closed |
|
|
||||||
| M5-007 | done | sonnet | M5: Sessions | Session metrics | #321 | #271 closed |
|
|
||||||
| M5-008 | done | sonnet | M5: Sessions | Verify sessions — 28 tests | #324 | #272 closed |
|
|
||||||
| M6-001 | done | sonnet | M6: Jobs | BullMQ + Valkey config | #324 | #273 closed |
|
|
||||||
| M6-002 | done | sonnet | M6: Jobs | Queue service with typed jobs | #324 | #274 closed |
|
|
||||||
| M6-003 | done | sonnet | M6: Jobs | Summarization → BullMQ | #324 | #275 closed |
|
|
||||||
| M6-004 | done | sonnet | M6: Jobs | GC → BullMQ | #324 | #276 closed |
|
|
||||||
| M6-005 | done | sonnet | M6: Jobs | Tier management → BullMQ | #324 | #277 closed |
|
|
||||||
| M6-006 | done | sonnet | M6: Jobs | Admin jobs API | #325 | #278 closed |
|
|
||||||
| M6-007 | done | sonnet | M6: Jobs | Job event logging | #325 | #279 closed |
|
|
||||||
| M6-008 | done | sonnet | M6: Jobs | Verify jobs | #324 | #280 closed |
|
|
||||||
| M7-001 | done | sonnet | M7: Channel Design | IChannelAdapter interface | #325 | #281 closed |
|
|
||||||
| M7-002 | done | sonnet | M7: Channel Design | Channel message protocol | #325 | #282 closed |
|
|
||||||
| M7-003 | done | sonnet | M7: Channel Design | Matrix integration design | #326 | #283 closed |
|
|
||||||
| M7-004 | done | sonnet | M7: Channel Design | Conversation multiplexing | #326 | #284 closed |
|
|
||||||
| M7-005 | done | sonnet | M7: Channel Design | Remote auth bridging | #326 | #285 closed |
|
|
||||||
| M7-006 | done | sonnet | M7: Channel Design | Agent-to-agent via Matrix | #326 | #286 closed |
|
|
||||||
| M7-007 | done | sonnet | M7: Channel Design | Multi-user isolation in Matrix | #326 | #287 closed |
|
|
||||||
| M7-008 | done | sonnet | M7: Channel Design | channel-protocol.md published | #326 | #288 closed |
|
|
||||||
|
|||||||
555
docs/design/storage-abstraction-middleware.md
Normal file
555
docs/design/storage-abstraction-middleware.md
Normal file
@@ -0,0 +1,555 @@
|
|||||||
|
# Storage & Queue Abstraction — Middleware Architecture
|
||||||
|
|
||||||
|
Design
|
||||||
|
Status: Design (retrofit required)
|
||||||
|
date: 2026-04-02
|
||||||
|
context: Agents coupled directly to infrastructure backends, bypassing intended middleware layer
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Problem
|
||||||
|
|
||||||
|
Current packages are **direct adapters**, not **middleware**:
|
||||||
|
| Package | Current State | Intended Design |
|
||||||
|
|---------|---------------|-----------------|
|
||||||
|
| `@mosaic/queue` | `ioredis` hardcoded | Interface → BullMQ OR local-files |
|
||||||
|
| `@mosaic/db` | Drizzle + Postgres hardcoded | Interface → Postgres OR SQLite OR JSON/MD |
|
||||||
|
| `@mosaic/memory` | pgvector required | Interface → pgvector OR sqlite-vec OR keyword-search |
|
||||||
|
|
||||||
|
## The gateway and TUI import these packages directly, which means they they're coupled to specific infrastructure. Users cannot run Mosaic Stack without Postgres + Valkey.
|
||||||
|
|
||||||
|
## The Intended Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Gateway / TUI / CLI │
|
||||||
|
│ (agnostic of storage backend, talks to middleware) │
|
||||||
|
└───────────────────────────┬─────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
┌───────────────────┼───────────────────┐
|
||||||
|
│ │ │
|
||||||
|
▼─────────────────┴─────────────────┴─────────────────┘
|
||||||
|
| | | |
|
||||||
|
▼─────────────────┴───────────────────┴─────────────────┘
|
||||||
|
| | | |
|
||||||
|
Queue Storage Memory
|
||||||
|
| | | |
|
||||||
|
┌─────────┬─────────┬─────────┬─────────────────────────────────┐
|
||||||
|
| BullMQ | | Local | | Postgres | SQLite | JSON/MD | pgvector | sqlite-vec | keyword |
|
||||||
|
|(Valkey)| |(files) | | | | | |
|
||||||
|
└─────────┴─────────┴─────────┴─────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
The gateway imports the interface, not the backend. At startup it reads config and instantiates the correct adapter.
|
||||||
|
|
||||||
|
## The Drift
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// What should have happened:
|
||||||
|
gateway/queue.service.ts → @mosaic/queue (interface) → queue.adapter.ts
|
||||||
|
|
||||||
|
// What actually happened:
|
||||||
|
gateway/queue.service.ts → @mosaic/queue → ioredis (hardcoded)
|
||||||
|
```
|
||||||
|
|
||||||
|
## The Current State Analysis
|
||||||
|
|
||||||
|
### `@mosaic/queue` (packages/queue/src/queue.ts)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import Redis from 'ioredis'; // ← Direct import of backend
|
||||||
|
|
||||||
|
export function createQueue(config?: QueueConfig): QueueHandle {
|
||||||
|
const url = config?.url ?? process.env['VALKEY_URL'] ?? DEFAULT_VALKEY_URL;
|
||||||
|
const redis = new Redis(url, { maxRetriesPerRequest: 3 });
|
||||||
|
// ...queue ops directly on redis...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Problem:** `ioredis` is imported in the package, not the adapter interface. Consumers cannot swap backends.
|
||||||
|
|
||||||
|
### `@mosaic/db` (packages/db/src/client.ts)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { drizzle, type PostgresJsDatabase } from 'drizzle-orm/postgres-js';
|
||||||
|
import postgres from 'postgres';
|
||||||
|
|
||||||
|
export function createDb(url?: string): DbHandle {
|
||||||
|
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
|
||||||
|
const sql = postgres(connectionString, { max: 20, idle_timeout: 30, connect_timeout: 5 });
|
||||||
|
const db = drizzle(sql, { schema });
|
||||||
|
// ...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Problem:** Drizzle + Postgres is hardcoded. No SQLite, JSON, or file-based options.
|
||||||
|
|
||||||
|
### `@mosaic/memory` (packages/memory/src/memory.ts)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import type { Db } from '@mosaic/db'; // ← Depends on Drizzle/PG
|
||||||
|
|
||||||
|
export function createMemory(db: Db): Memory {
|
||||||
|
return {
|
||||||
|
preferences: createPreferencesRepo(db),
|
||||||
|
insights: createInsightsRepo(db),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Problem:** Memory package is tightly coupled to `@mosaic/db` (which is Postgres-only). No alternative storage backends.
|
||||||
|
|
||||||
|
## The Target Interfaces
|
||||||
|
|
||||||
|
### Queue Interface
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// packages/queue/src/types.ts
|
||||||
|
export interface QueueAdapter {
|
||||||
|
readonly name: string;
|
||||||
|
|
||||||
|
enqueue(queueName: string, payload: TaskPayload): Promise<void>;
|
||||||
|
dequeue(queueName: string): Promise<TaskPayload | null>;
|
||||||
|
length(queueName: string): Promise<number>;
|
||||||
|
publish(channel: string, message: string): Promise<void>;
|
||||||
|
subscribe(channel: string, handler: (message: string) => void): () => void;
|
||||||
|
close(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface TaskPayload {
|
||||||
|
id: string;
|
||||||
|
type: string;
|
||||||
|
data: Record<string, unknown>;
|
||||||
|
createdAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface QueueConfig {
|
||||||
|
type: 'bullmq' | 'local';
|
||||||
|
url?: string; // For bullmq: Valkey/Redis URL
|
||||||
|
dataDir?: string; // For local: directory for JSON persistence
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Storage Interface
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// packages/storage/src/types.ts
|
||||||
|
export interface StorageAdapter {
|
||||||
|
readonly name: string;
|
||||||
|
|
||||||
|
// Entity CRUD
|
||||||
|
create<T>(collection: string, data: O): Promise<T>;
|
||||||
|
read<T>(collection: string, id: string): Promise<T | null>;
|
||||||
|
update<T>(collection: string, id: string, data: Partial<O>): Promise<T | null>;
|
||||||
|
delete(collection: string, id: string): Promise<boolean>;
|
||||||
|
|
||||||
|
// Queries
|
||||||
|
find<T>(collection: string, filter: Record<string, unknown>): Promise<T[]>;
|
||||||
|
findOne<T>(collection: string, filter: Record<string, unknown): Promise<T | null>;
|
||||||
|
|
||||||
|
// Bulk operations
|
||||||
|
createMany<T>(collection: string, items: O[]): Promise<T[]>;
|
||||||
|
updateMany<T>(collection: string, ids: string[], data: Partial<O>): Promise<number>;
|
||||||
|
deleteMany(collection: string, ids: string[]): Promise<number>;
|
||||||
|
|
||||||
|
// Raw queries (for complex queries)
|
||||||
|
query<T>(collection: string, query: string, params?: unknown[]): Promise<T[]>;
|
||||||
|
|
||||||
|
// Transaction support
|
||||||
|
transaction<T>(fn: (tx: StorageTransaction) => Promise<T>): Promise<T>;
|
||||||
|
|
||||||
|
close(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StorageTransaction {
|
||||||
|
commit(): Promise<void>;
|
||||||
|
rollback(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StorageConfig {
|
||||||
|
type: 'postgres' | 'sqlite' | 'files';
|
||||||
|
url?: string; // For postgres
|
||||||
|
path?: string; // For sqlite/files
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Interface (Vector + Preferences)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// packages/memory/src/types.ts
|
||||||
|
export interface MemoryAdapter {
|
||||||
|
readonly name: string;
|
||||||
|
|
||||||
|
// Preferences (key-value storage)
|
||||||
|
getPreference(userId: string, key: string): Promise<unknown | null>;
|
||||||
|
setPreference(userId: string, key: string, value: unknown): Promise<void>;
|
||||||
|
deletePreference(userId: string, key: string): Promise<boolean>;
|
||||||
|
listPreferences(
|
||||||
|
userId: string,
|
||||||
|
category?: string,
|
||||||
|
): Promise<Array<{ key: string; value: unknown }>>;
|
||||||
|
|
||||||
|
// Insights (with optional vector search)
|
||||||
|
storeInsight(insight: NewInsight): Promise<Insight>;
|
||||||
|
getInsight(id: string): Promise<Insight | null>;
|
||||||
|
searchInsights(query: string, limit?: number, filter?: InsightFilter): Promise<SearchResult[]>;
|
||||||
|
deleteInsight(id: string): Promise<boolean>;
|
||||||
|
|
||||||
|
// Embedding provider (optional, null = no vector search)
|
||||||
|
readonly embedder?: EmbeddingProvider | null;
|
||||||
|
|
||||||
|
close(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface NewInsight {
|
||||||
|
id: string;
|
||||||
|
userId: string;
|
||||||
|
content: string;
|
||||||
|
embedding?: number[]; // If embedder is available
|
||||||
|
source: 'agent' | 'user' | 'summarization' | 'system';
|
||||||
|
category: 'decision' | 'learning' | 'preference' | 'fact' | 'pattern' | 'general';
|
||||||
|
relevanceScore: number;
|
||||||
|
metadata?: Record<string, unknown>;
|
||||||
|
createdAt: Date;
|
||||||
|
decayedAt?: Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface InsightFilter {
|
||||||
|
userId?: string;
|
||||||
|
category?: string;
|
||||||
|
source?: string;
|
||||||
|
minRelevance?: number;
|
||||||
|
fromDate?: Date;
|
||||||
|
toDate?: Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SearchResult {
|
||||||
|
documentId: string;
|
||||||
|
content: string;
|
||||||
|
distance: number;
|
||||||
|
metadata?: Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MemoryConfig {
|
||||||
|
type: 'pgvector' | 'sqlite-vec' | 'keyword';
|
||||||
|
storage: StorageAdapter;
|
||||||
|
embedder?: EmbeddingProvider;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EmbeddingProvider {
|
||||||
|
embed(text: string): Promise<number[]>;
|
||||||
|
embedBatch(texts: string[]): Promise<number[][]>;
|
||||||
|
readonly dimensions: number;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Three Tiers
|
||||||
|
|
||||||
|
### Tier 1: Local (Zero Dependencies)
|
||||||
|
|
||||||
|
**Target:** Single user, single machine, no external services
|
||||||
|
|
||||||
|
| Component | Backend | Storage |
|
||||||
|
| --------- | --------------------------------------------- | ------------ |
|
||||||
|
| Queue | In-process + JSON files in `~/.mosaic/queue/` |
|
||||||
|
| Storage | SQLite (better-sqlite3) `~/.mosaic/data.db` |
|
||||||
|
| Memory | Keyword search | SQLite table |
|
||||||
|
| Vector | None | N/A |
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- `better-sqlite3` (bundled)
|
||||||
|
- No Postgres, No Valkey, No pgvector
|
||||||
|
|
||||||
|
**Upgrade path:**
|
||||||
|
|
||||||
|
1. Run `mosaic gateway configure` → select "local" tier
|
||||||
|
2. Gateway starts with SQLite database
|
||||||
|
3. Optional: run `mosaic gateway upgrade --tier team` to migrate to Postgres
|
||||||
|
|
||||||
|
### Tier 2: Team (Postgres + Valkey)
|
||||||
|
|
||||||
|
**Target:** Multiple users, shared server, CI/CD environments
|
||||||
|
|
||||||
|
| Component | Backend | Storage |
|
||||||
|
| --------- | -------------- | ------------------------------ |
|
||||||
|
| Queue | BullMQ | Valkey |
|
||||||
|
| Storage | Postgres | Shared PG instance |
|
||||||
|
| Memory | pgvector | Postgres with vector extension |
|
||||||
|
| Vector | LLM embeddings | Configured provider |
|
||||||
|
|
||||||
|
**Dependencies:**
|
||||||
|
|
||||||
|
- PostgreSQL 17+ with pgvector extension
|
||||||
|
- Valkey (Redis-compatible)
|
||||||
|
- LLM provider for embeddings
|
||||||
|
|
||||||
|
**Migration from Local → Team:**
|
||||||
|
|
||||||
|
1. `mosaic gateway backup` → creates dump of SQLite database
|
||||||
|
2. `mosaic gateway upgrade --tier team` → restores to Postgres
|
||||||
|
3. Queue replays from BullMQ (may need manual reconciliation for in-flight jobs)
|
||||||
|
4. Memory embeddings regenerated if vector search was new
|
||||||
|
|
||||||
|
### Tier 3: Enterprise (Clustered)
|
||||||
|
|
||||||
|
**Target:** Large teams, multi-region, high availability
|
||||||
|
|
||||||
|
| Component | Backend | Storage |
|
||||||
|
| --------- | --------------------------- | ----------------------------- |
|
||||||
|
| Queue | BullMQ cluster | Multiple Valkey nodes |
|
||||||
|
| Storage | Postgres cluster | Primary + replicas |
|
||||||
|
| Memory | Dedicated vector DB | Qdrant, Pinecone, or pgvector |
|
||||||
|
| Vector | Dedicated embedding service | Separate microservice |
|
||||||
|
|
||||||
|
## MarkdownDB Integration
|
||||||
|
|
||||||
|
For file-based storage, we use [MarkdownDB](https://markdowndb.com) to parse MD files into queryable data.
|
||||||
|
|
||||||
|
**What it provides:**
|
||||||
|
|
||||||
|
- Parses frontmatter (YAML/JSON/TOML)
|
||||||
|
- Extracts links, tags, metadata
|
||||||
|
- Builds index in JSON or SQLite
|
||||||
|
- Queryable via SQL-like interface
|
||||||
|
|
||||||
|
**Usage in Mosaic:**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Local tier with MD files for documents
|
||||||
|
const storage = createStorageAdapter({
|
||||||
|
type: 'files',
|
||||||
|
path: path.join(mosaicHome, 'docs'),
|
||||||
|
markdowndb: {
|
||||||
|
parseFrontmatter: true,
|
||||||
|
extractLinks: true,
|
||||||
|
indexFile: 'index.json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dream Mode — Memory Consolidation
|
||||||
|
|
||||||
|
Automated equivalent to Claude Code's "Dream: Memory Consolidation" cycle
|
||||||
|
|
||||||
|
**Trigger:** Every 24 hours (if 5+ sessions active)
|
||||||
|
|
||||||
|
**Phases:**
|
||||||
|
|
||||||
|
1. **Orient** — What happened, what's the current state
|
||||||
|
- Scan recent session logs
|
||||||
|
- Identify active tasks, missions, conversations
|
||||||
|
- Calculate time window (last 24h)
|
||||||
|
|
||||||
|
2. **Gather** — Pull in relevant context
|
||||||
|
- Load conversations, decisions, agent logs
|
||||||
|
- Extract key interactions and outcomes
|
||||||
|
- Identify patterns and learnings
|
||||||
|
|
||||||
|
3. **Consolidate** — Summarize and compress
|
||||||
|
- Generate summary of the last 24h
|
||||||
|
- Extract key decisions and their rationale
|
||||||
|
- Identify recurring patterns
|
||||||
|
- Compress verbose logs into concise insights
|
||||||
|
|
||||||
|
4. **Prune** — Archive and cleanup
|
||||||
|
- Archive raw session files to dated folders
|
||||||
|
- Delete redundant/temporary data
|
||||||
|
- Update MEMORY.md with consolidated content
|
||||||
|
- Update insight relevance scores
|
||||||
|
|
||||||
|
**Implementation:**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// In @mosaic/dream (new package)
|
||||||
|
export async function runDreamCycle(config: DreamConfig): Promise<DreamResult> {
|
||||||
|
const memory = await loadMemoryAdapter(config.storage);
|
||||||
|
|
||||||
|
// Orient
|
||||||
|
const sessions = await memory.getRecentSessions(24 * 60 * 60 * 1000);
|
||||||
|
if (sessions.length < 5) return { skipped: true, reason: 'insufficient_sessions' };
|
||||||
|
|
||||||
|
// Gather
|
||||||
|
const context = await gatherContext(memory, sessions);
|
||||||
|
|
||||||
|
// Consolidate
|
||||||
|
const consolidated = await consolidateWithLLM(context, config.llm);
|
||||||
|
|
||||||
|
// Prune
|
||||||
|
await pruneArchivedData(memory, config.retention);
|
||||||
|
|
||||||
|
// Store consolidated insights
|
||||||
|
await memory.storeInsights(consolidated.insights);
|
||||||
|
|
||||||
|
return {
|
||||||
|
sessionsProcessed: sessions.length,
|
||||||
|
insightsCreated: consolidated.insights.length,
|
||||||
|
bytesPruned: consolidated.bytesRemoved,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Retrofit Plan
|
||||||
|
|
||||||
|
### Phase 1: Interface Extraction (2-3 days)
|
||||||
|
|
||||||
|
**Goal:** Define interfaces without changing existing behavior
|
||||||
|
|
||||||
|
1. Create `packages/queue/src/types.ts` with `QueueAdapter` interface
|
||||||
|
2. Create `packages/storage/src/types.ts` with `StorageAdapter` interface
|
||||||
|
3. Create `packages/memory/src/types.ts` with `MemoryAdapter` interface (refactor existing)
|
||||||
|
4. Add adapter registry pattern to each package
|
||||||
|
5. No breaking changes — existing code continues to work
|
||||||
|
|
||||||
|
### Phase 2: Refactor Existing to Adapters (3-5 days)
|
||||||
|
|
||||||
|
**Goal:** Move existing implementations behind adapters
|
||||||
|
|
||||||
|
#### 2.1 Queue Refactor
|
||||||
|
|
||||||
|
1. Rename `packages/queue/src/queue.ts` → `packages/queue/src/adapters/bullmq.ts`
|
||||||
|
2. Create `packages/queue/src/index.ts` to export factory function
|
||||||
|
3. Factory function reads config, instantiates correct adapter
|
||||||
|
4. Update gateway imports to use factory
|
||||||
|
|
||||||
|
#### 2.2 Storage Refactor
|
||||||
|
|
||||||
|
1. Create `packages/storage/` (new package)
|
||||||
|
2. Move Drizzle logic to `packages/storage/src/adapters/postgres.ts`
|
||||||
|
3. Create SQLite adapter in `packages/storage/src/adapters/sqlite.ts`
|
||||||
|
4. Update gateway to use storage factory
|
||||||
|
5. Deprecate direct `@mosaic/db` imports
|
||||||
|
|
||||||
|
#### 2.3 Memory Refactor
|
||||||
|
|
||||||
|
1. Extract existing logic to `packages/memory/src/adapters/pgvector.ts`
|
||||||
|
2. Create keyword adapter in `packages/memory/src/adapters/keyword.ts`
|
||||||
|
3. Update vector-store.ts to be adapter-agnostic
|
||||||
|
|
||||||
|
### Phase 3: Local Tier Implementation (2-3 days)
|
||||||
|
|
||||||
|
**Goal:** Zero-dependency baseline
|
||||||
|
|
||||||
|
1. Implement `packages/queue/src/adapters/local.ts` (in-process + JSON persistence)
|
||||||
|
2. Implement `packages/storage/src/adapters/files.ts` (JSON + MD via MarkdownDB)
|
||||||
|
3. Implement `packages/memory/src/adapters/keyword.ts` (TF-IDF search)
|
||||||
|
4. Add `packages/dream/` for consolidation cycle
|
||||||
|
5. Wire up local tier in gateway startup
|
||||||
|
|
||||||
|
### Phase 4: Configuration System (1-2 days)
|
||||||
|
|
||||||
|
**Goal:** Runtime backend selection
|
||||||
|
|
||||||
|
1. Create `packages/config/src/storage.ts` for storage configuration
|
||||||
|
2. Add `mosaic.config.ts` schema with storage tier settings
|
||||||
|
3. Update gateway to read config on startup
|
||||||
|
4. Add `mosaic gateway configure` CLI command
|
||||||
|
5. Add tier migration commands (`mosaic gateway upgrade`)
|
||||||
|
|
||||||
|
### Phase 5: Testing & Documentation (2-3 days)
|
||||||
|
|
||||||
|
1. Unit tests for each adapter
|
||||||
|
2. Integration tests for factory pattern
|
||||||
|
3. Migration tests (local → team)
|
||||||
|
4. Update README and architecture docs
|
||||||
|
5. Add configuration guide
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Changes Summary
|
||||||
|
|
||||||
|
### New Files
|
||||||
|
|
||||||
|
```
|
||||||
|
packages/
|
||||||
|
├── config/
|
||||||
|
│ └── src/
|
||||||
|
│ ├── storage.ts # Storage config schema
|
||||||
|
│ └── index.ts
|
||||||
|
├── dream/ # NEW: Dream mode consolidation
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── index.ts
|
||||||
|
│ │ ├── orient.ts
|
||||||
|
│ │ ├── gather.ts
|
||||||
|
│ │ ├── consolidate.ts
|
||||||
|
│ │ └── prune.ts
|
||||||
|
│ └── package.json
|
||||||
|
├── queue/
|
||||||
|
│ └── src/
|
||||||
|
│ ├── types.ts # NEW: QueueAdapter interface
|
||||||
|
│ ├── index.ts # NEW: Factory function
|
||||||
|
│ └── adapters/
|
||||||
|
│ ├── bullmq.ts # MOVED from queue.ts
|
||||||
|
│ └── local.ts # NEW: In-process adapter
|
||||||
|
├── storage/ # NEW: Storage abstraction
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── types.ts # StorageAdapter interface
|
||||||
|
│ │ ├── index.ts # Factory function
|
||||||
|
│ │ └── adapters/
|
||||||
|
│ │ ├── postgres.ts # MOVED from @mosaic/db
|
||||||
|
│ │ ├── sqlite.ts # NEW: SQLite adapter
|
||||||
|
│ │ └── files.ts # NEW: JSON/MD adapter
|
||||||
|
│ └── package.json
|
||||||
|
└── memory/
|
||||||
|
└── src/
|
||||||
|
├── types.ts # UPDATED: MemoryAdapter interface
|
||||||
|
├── index.ts # UPDATED: Factory function
|
||||||
|
└── adapters/
|
||||||
|
├── pgvector.ts # EXTRACTED from existing code
|
||||||
|
├── sqlite-vec.ts # NEW: SQLite with vectors
|
||||||
|
└── keyword.ts # NEW: TF-IDF search
|
||||||
|
```
|
||||||
|
|
||||||
|
### Modified Files
|
||||||
|
|
||||||
|
```
|
||||||
|
packages/
|
||||||
|
├── db/ # DEPRECATED: Logic moved to storage adapters
|
||||||
|
├── queue/
|
||||||
|
│ └── src/
|
||||||
|
│ └── queue.ts # → adapters/bullmq.ts
|
||||||
|
├── memory/
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── memory.ts # → use factory
|
||||||
|
│ │ ├── insights.ts # → use factory
|
||||||
|
│ │ └── preferences.ts # → use factory
|
||||||
|
│ └── package.json # Remove pgvector from dependencies
|
||||||
|
└── gateway/
|
||||||
|
└── src/
|
||||||
|
├── database/
|
||||||
|
│ └── database.module.ts # Update to use storage factory
|
||||||
|
├── memory/
|
||||||
|
│ └── memory.module.ts # Update to use memory factory
|
||||||
|
└── queue/
|
||||||
|
└── queue.module.ts # Update to use queue factory
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Breaking Changes
|
||||||
|
|
||||||
|
1. **`@mosaic/db`** → **`@mosaic/storage`** (with migration guide)
|
||||||
|
2. Direct `ioredis` imports → Use `@mosaic/queue` factory
|
||||||
|
3. Direct `pgvector` queries → Use `@mosaic/memory` factory
|
||||||
|
4. Gateway startup now requires storage config (defaults to local)
|
||||||
|
|
||||||
|
## Non-Breaking Migration Path
|
||||||
|
|
||||||
|
1. Existing deployments with Postgres/Valkey continue to work (default config)
|
||||||
|
2. New deployments can choose local tier
|
||||||
|
3. Migration commands available when ready to upgrade
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- [ ] Local tier runs with zero external dependencies
|
||||||
|
- [ ] All three tiers (local, team, enterprise) work correctly
|
||||||
|
- [ ] Factory pattern correctly selects backend at runtime
|
||||||
|
- [ ] Migration from local → team preserves all data
|
||||||
|
- [ ] Dream mode consolidates 24h of sessions
|
||||||
|
- [ ] Documentation covers all three tiers and migration paths
|
||||||
|
- [ ] All existing tests pass
|
||||||
|
- [ ] New adapters have >80% coverage
|
||||||
Reference in New Issue
Block a user