diff --git a/docs/TASKS.md b/docs/TASKS.md index b246f9e..8525140 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -1,73 +1,30 @@ -# Tasks — Harness Foundation +# Tasks — Storage Abstraction Retrofit > Single-writer: orchestrator only. Workers read but never modify. > +> **Mission:** Decouple gateway from hardcoded Postgres/Valkey backends. Introduce interface-driven middleware so the gateway is backend-agnostic. Default to local tier (SQLite + JSON) for zero-dependency installs. +> > **`agent` column values:** `codex` | `sonnet` | `haiku` | `glm-5` | `opus` | `—` (auto/default) -| id | status | agent | milestone | description | pr | notes | -| ------ | ------ | ------ | ------------------ | ------------------------------------------------------------------ | ---- | ----------- | -| M1-001 | done | sonnet | M1: Persistence | Wire ChatGateway → ConversationsRepo for user messages | #292 | #224 closed | -| M1-002 | done | sonnet | M1: Persistence | Wire agent event relay → ConversationsRepo for assistant responses | #292 | #225 closed | -| M1-003 | done | sonnet | M1: Persistence | Store message metadata: model, provider, tokens, tool calls | #292 | #226 closed | -| M1-004 | done | sonnet | M1: Persistence | Load message history into Pi session on resume | #301 | #227 closed | -| M1-005 | done | sonnet | M1: Persistence | Context window management: summarize when >80% | #301 | #228 closed | -| M1-006 | done | sonnet | M1: Persistence | Conversation search endpoint | #299 | #229 closed | -| M1-007 | done | sonnet | M1: Persistence | TUI /history command | #297 | #230 closed | -| M1-008 | done | sonnet | M1: Persistence | Verify persistence — 20 tests | #304 | #231 closed | -| M2-001 | done | sonnet | M2: Security | InsightsRepo userId on searchByEmbedding | #290 | #232 closed | -| M2-002 | done | sonnet | M2: Security | InsightsRepo userId on findByUser/decay | #290 | #233 closed | -| M2-003 | done | sonnet | M2: Security | PreferencesRepo userId verified | #294 | #234 closed | -| M2-004 | done | sonnet | M2: Security | Memory tools userId injection fixed | #294 | #235 closed | -| M2-005 | done | sonnet | M2: Security | ConversationsRepo ownership checks | #293 | #236 closed | -| M2-006 | done | sonnet | M2: Security | AgentsRepo findAccessible scoped | #293 | #237 closed | -| M2-007 | done | sonnet | M2: Security | Cross-user isolation — 28 tests | #305 | #238 closed | -| M2-008 | done | sonnet | M2: Security | Valkey SCAN + /gc admin-only | #298 | #239 closed | -| M3-001 | done | sonnet | M3: Providers | IProviderAdapter + OllamaAdapter | #306 | #240 closed | -| M3-002 | done | sonnet | M3: Providers | AnthropicAdapter | #309 | #241 closed | -| M3-003 | done | sonnet | M3: Providers | OpenAIAdapter | #310 | #242 closed | -| M3-004 | done | sonnet | M3: Providers | OpenRouterAdapter | #311 | #243 closed | -| M3-005 | done | sonnet | M3: Providers | ZaiAdapter (GLM-5) | #314 | #244 closed | -| M3-006 | done | sonnet | M3: Providers | Ollama embedding support | #311 | #245 closed | -| M3-007 | done | sonnet | M3: Providers | Provider health checks | #308 | #246 closed | -| M3-008 | done | sonnet | M3: Providers | Model capability matrix | #303 | #247 closed | -| M3-009 | done | sonnet | M3: Providers | EmbeddingService → Ollama default | #308 | #248 closed | -| M3-010 | done | sonnet | M3: Providers | OAuth token storage (AES-256-GCM) | #317 | #249 closed | -| M3-011 | done | sonnet | M3: Providers | Provider credentials CRUD | #317 | #250 closed | -| M3-012 | done | sonnet | M3: Providers | Verify providers — 40 tests | #319 | #251 closed | -| M4-001 | done | sonnet | M4: Routing | routing_rules DB schema | #315 | #252 closed | -| M4-002 | done | sonnet | M4: Routing | Condition types | #315 | #253 closed | -| M4-003 | done | sonnet | M4: Routing | Action types | #315 | #254 closed | -| M4-004 | done | sonnet | M4: Routing | Default routing rules (11 seeds) | #316 | #255 closed | -| M4-005 | done | sonnet | M4: Routing | Task classifier (60+ tests) | #316 | #256 closed | -| M4-006 | done | sonnet | M4: Routing | Routing decision pipeline | #318 | #257 closed | -| M4-007 | done | sonnet | M4: Routing | /model override | #323 | #258 closed | -| M4-008 | done | sonnet | M4: Routing | Routing transparency in session:info | #323 | #259 closed | -| M4-009 | done | sonnet | M4: Routing | Routing rules CRUD API | #320 | #260 closed | -| M4-010 | done | sonnet | M4: Routing | Per-user routing overrides | #320 | #261 closed | -| M4-011 | done | sonnet | M4: Routing | Agent specialization capabilities | #320 | #262 closed | -| M4-012 | done | sonnet | M4: Routing | Routing wired into ChatGateway | #323 | #263 closed | -| M4-013 | done | sonnet | M4: Routing | Verify routing — 9 E2E tests | #323 | #264 closed | -| M5-001 | done | sonnet | M5: Sessions | Agent config loaded on session create | #323 | #265 closed | -| M5-002 | done | sonnet | M5: Sessions | /model command end-to-end | #323 | #266 closed | -| M5-003 | done | sonnet | M5: Sessions | /agent command mid-session | #323 | #267 closed | -| M5-004 | done | sonnet | M5: Sessions | Session ↔ conversation binding | #321 | #268 closed | -| M5-005 | done | sonnet | M5: Sessions | Session info broadcast | #321 | #269 closed | -| M5-006 | done | sonnet | M5: Sessions | /agent new from TUI | #321 | #270 closed | -| M5-007 | done | sonnet | M5: Sessions | Session metrics | #321 | #271 closed | -| M5-008 | done | sonnet | M5: Sessions | Verify sessions — 28 tests | #324 | #272 closed | -| M6-001 | done | sonnet | M6: Jobs | BullMQ + Valkey config | #324 | #273 closed | -| M6-002 | done | sonnet | M6: Jobs | Queue service with typed jobs | #324 | #274 closed | -| M6-003 | done | sonnet | M6: Jobs | Summarization → BullMQ | #324 | #275 closed | -| M6-004 | done | sonnet | M6: Jobs | GC → BullMQ | #324 | #276 closed | -| M6-005 | done | sonnet | M6: Jobs | Tier management → BullMQ | #324 | #277 closed | -| M6-006 | done | sonnet | M6: Jobs | Admin jobs API | #325 | #278 closed | -| M6-007 | done | sonnet | M6: Jobs | Job event logging | #325 | #279 closed | -| M6-008 | done | sonnet | M6: Jobs | Verify jobs | #324 | #280 closed | -| M7-001 | done | sonnet | M7: Channel Design | IChannelAdapter interface | #325 | #281 closed | -| M7-002 | done | sonnet | M7: Channel Design | Channel message protocol | #325 | #282 closed | -| M7-003 | done | sonnet | M7: Channel Design | Matrix integration design | #326 | #283 closed | -| M7-004 | done | sonnet | M7: Channel Design | Conversation multiplexing | #326 | #284 closed | -| M7-005 | done | sonnet | M7: Channel Design | Remote auth bridging | #326 | #285 closed | -| M7-006 | done | sonnet | M7: Channel Design | Agent-to-agent via Matrix | #326 | #286 closed | -| M7-007 | done | sonnet | M7: Channel Design | Multi-user isolation in Matrix | #326 | #287 closed | -| M7-008 | done | sonnet | M7: Channel Design | channel-protocol.md published | #326 | #288 closed | +| id | status | description | issue | repo | branch | depends_on | blocks | agent | started_at | completed_at | estimate | used | notes | +| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | ----- | -------------------- | ------------------------ | ----------------------------- | --------- | ----- | ---------- | ------------ | -------- | ---- | ----- | +| SA-P1-001 | not-started | Define QueueAdapter interface in packages/queue/src/types.ts — enqueue, dequeue, length, publish, subscribe, close | | queue | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | | +| SA-P1-002 | not-started | Define StorageAdapter interface in packages/storage/src/types.ts — CRUD, query, find, transaction, close | | storage | feat/storage-abstraction | | SA-P1-004 | codex | | | 8K | | | +| SA-P1-003 | not-started | Define MemoryAdapter interface in packages/memory/src/types.ts — preferences, insights, search, embedder support | | memory | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | | +| SA-P1-004 | not-started | Create adapter factory pattern + config types: createQueue(config), createStorage(config), createMemory(config) | | queue,storage,memory | feat/storage-abstraction | SA-P1-001,SA-P1-002,SA-P1-003 | SA-P2-001 | codex | | | 8K | | | +| SA-P2-001 | not-started | Refactor @mosaic/queue: move queue.ts → adapters/bullmq.ts, implement QueueAdapter interface, export factory | | queue | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | | +| SA-P2-002 | not-started | Create @mosaic/storage package: move @mosaic/db Drizzle logic → adapters/postgres.ts, implement StorageAdapter interface | | storage | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 15K | | | +| SA-P2-003 | not-started | Refactor @mosaic/memory: extract pgvector logic → adapters/pgvector.ts, implement MemoryAdapter interface | | memory | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | | +| SA-P2-004 | not-started | Update gateway database.module.ts, queue.module.ts, memory.module.ts to use factories + NestJS DI tokens | | gateway | feat/storage-abstraction | SA-P2-001,SA-P2-002,SA-P2-003 | SA-P2-005 | codex | | | 15K | | | +| SA-P2-005 | not-started | Verify Phase 2: existing Postgres/Valkey behavior unchanged — all existing tests pass, typecheck clean | | gateway | feat/storage-abstraction | SA-P2-004 | SA-P3-001 | codex | | | 10K | | | +| SA-P3-001 | not-started | Implement local queue adapter: in-process Map + JSON file persistence in adapters/local.ts | | queue | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 10K | | | +| SA-P3-002 | not-started | Implement SQLite storage adapter: better-sqlite3 with schema mirroring Drizzle tables, in adapters/sqlite.ts | | storage | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 20K | | | +| SA-P3-003 | not-started | Implement keyword memory adapter: TF-IDF or simple keyword matching, no vector dependency, in adapters/keyword.ts | | memory | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 12K | | | +| SA-P3-004 | not-started | Verify Phase 3: gateway starts with local config (no PG/Valkey), basic CRUD works, all adapter tests pass | | gateway | feat/storage-abstraction | SA-P3-001,SA-P3-002,SA-P3-003 | SA-P4-001 | codex | | | 15K | | | +| SA-P4-001 | not-started | Add mosaic.config.ts schema + loader: storage tier selection (local/team), backend config, defaults to local | | config | feat/storage-abstraction | SA-P3-004 | SA-P4-002 | codex | | | 10K | | | +| SA-P4-002 | not-started | CLI: mosaic gateway init — interactive wizard to generate mosaic.config.ts with tier selection | | cli | feat/storage-abstraction | SA-P4-001 | SA-P4-003 | codex | | | 12K | | | +| SA-P4-003 | not-started | CLI: mosaic gateway start/stop/status — lifecycle management for the gateway daemon | | cli | feat/storage-abstraction | SA-P4-002 | SA-P4-004 | codex | | | 12K | | | +| SA-P4-004 | not-started | Verify Phase 4: fresh install with `mosaic gateway init && mosaic gateway start` works end-to-end | | cli,gateway | feat/storage-abstraction | SA-P4-003 | SA-P5-001 | codex | | | 10K | | | +| SA-P5-001 | not-started | Migration tooling: mosaic storage export/import for local↔postgres tier migration | | cli,storage | feat/storage-abstraction | SA-P4-004 | SA-P5-002 | codex | | | 15K | | | +| SA-P5-002 | not-started | Docker Compose profiles: local (gateway only) vs team (gateway+pg+valkey), update docker-compose.yml | | infra | feat/storage-abstraction | SA-P5-001 | SA-P5-003 | codex | | | 8K | | | +| SA-P5-003 | not-started | Final verification + docs: README update, architecture diagram, configuration guide | | docs | feat/storage-abstraction | SA-P5-002 | | codex | | | 10K | | | diff --git a/docs/design/storage-abstraction-middleware.md b/docs/design/storage-abstraction-middleware.md new file mode 100644 index 0000000..82db7fc --- /dev/null +++ b/docs/design/storage-abstraction-middleware.md @@ -0,0 +1,555 @@ +# Storage & Queue Abstraction — Middleware Architecture + +Design +Status: Design (retrofit required) +date: 2026-04-02 +context: Agents coupled directly to infrastructure backends, bypassing intended middleware layer + +--- + +## The Problem + +Current packages are **direct adapters**, not **middleware**: +| Package | Current State | Intended Design | +|---------|---------------|-----------------| +| `@mosaic/queue` | `ioredis` hardcoded | Interface → BullMQ OR local-files | +| `@mosaic/db` | Drizzle + Postgres hardcoded | Interface → Postgres OR SQLite OR JSON/MD | +| `@mosaic/memory` | pgvector required | Interface → pgvector OR sqlite-vec OR keyword-search | + +## The gateway and TUI import these packages directly, which means they they're coupled to specific infrastructure. Users cannot run Mosaic Stack without Postgres + Valkey. + +## The Intended Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Gateway / TUI / CLI │ +│ (agnostic of storage backend, talks to middleware) │ +└───────────────────────────┬─────────────────────────────────────┘ + │ + ┌───────────────────┼───────────────────┐ + │ │ │ + ▼─────────────────┴─────────────────┴─────────────────┘ + | | | | + ▼─────────────────┴───────────────────┴─────────────────┘ + | | | | + Queue Storage Memory + | | | | + ┌─────────┬─────────┬─────────┬─────────────────────────────────┐ + | BullMQ | | Local | | Postgres | SQLite | JSON/MD | pgvector | sqlite-vec | keyword | + |(Valkey)| |(files) | | | | | | + └─────────┴─────────┴─────────┴─────────────────────────────────┘ +``` + +The gateway imports the interface, not the backend. At startup it reads config and instantiates the correct adapter. + +## The Drift + +```typescript +// What should have happened: +gateway/queue.service.ts → @mosaic/queue (interface) → queue.adapter.ts + +// What actually happened: +gateway/queue.service.ts → @mosaic/queue → ioredis (hardcoded) +``` + +## The Current State Analysis + +### `@mosaic/queue` (packages/queue/src/queue.ts) + +```typescript +import Redis from 'ioredis'; // ← Direct import of backend + +export function createQueue(config?: QueueConfig): QueueHandle { + const url = config?.url ?? process.env['VALKEY_URL'] ?? DEFAULT_VALKEY_URL; + const redis = new Redis(url, { maxRetriesPerRequest: 3 }); + // ...queue ops directly on redis... +} +``` + +**Problem:** `ioredis` is imported in the package, not the adapter interface. Consumers cannot swap backends. + +### `@mosaic/db` (packages/db/src/client.ts) + +```typescript +import { drizzle, type PostgresJsDatabase } from 'drizzle-orm/postgres-js'; +import postgres from 'postgres'; + +export function createDb(url?: string): DbHandle { + const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL; + const sql = postgres(connectionString, { max: 20, idle_timeout: 30, connect_timeout: 5 }); + const db = drizzle(sql, { schema }); + // ... +} +``` + +**Problem:** Drizzle + Postgres is hardcoded. No SQLite, JSON, or file-based options. + +### `@mosaic/memory` (packages/memory/src/memory.ts) + +```typescript +import type { Db } from '@mosaic/db'; // ← Depends on Drizzle/PG + +export function createMemory(db: Db): Memory { + return { + preferences: createPreferencesRepo(db), + insights: createInsightsRepo(db), + }; +} +``` + +**Problem:** Memory package is tightly coupled to `@mosaic/db` (which is Postgres-only). No alternative storage backends. + +## The Target Interfaces + +### Queue Interface + +```typescript +// packages/queue/src/types.ts +export interface QueueAdapter { + readonly name: string; + + enqueue(queueName: string, payload: TaskPayload): Promise; + dequeue(queueName: string): Promise; + length(queueName: string): Promise; + publish(channel: string, message: string): Promise; + subscribe(channel: string, handler: (message: string) => void): () => void; + close(): Promise; +} + +export interface TaskPayload { + id: string; + type: string; + data: Record; + createdAt: string; +} + +export interface QueueConfig { + type: 'bullmq' | 'local'; + url?: string; // For bullmq: Valkey/Redis URL + dataDir?: string; // For local: directory for JSON persistence +} +``` + +### Storage Interface + +```typescript +// packages/storage/src/types.ts +export interface StorageAdapter { + readonly name: string; + + // Entity CRUD + create(collection: string, data: O): Promise; + read(collection: string, id: string): Promise; + update(collection: string, id: string, data: Partial): Promise; + delete(collection: string, id: string): Promise; + + // Queries + find(collection: string, filter: Record): Promise; + findOne(collection: string, filter: Record; + + // Bulk operations + createMany(collection: string, items: O[]): Promise; + updateMany(collection: string, ids: string[], data: Partial): Promise; + deleteMany(collection: string, ids: string[]): Promise; + + // Raw queries (for complex queries) + query(collection: string, query: string, params?: unknown[]): Promise; + + // Transaction support + transaction(fn: (tx: StorageTransaction) => Promise): Promise; + + close(): Promise; +} + +export interface StorageTransaction { + commit(): Promise; + rollback(): Promise; +} + +export interface StorageConfig { + type: 'postgres' | 'sqlite' | 'files'; + url?: string; // For postgres + path?: string; // For sqlite/files +} +``` + +### Memory Interface (Vector + Preferences) + +```typescript +// packages/memory/src/types.ts +export interface MemoryAdapter { + readonly name: string; + + // Preferences (key-value storage) + getPreference(userId: string, key: string): Promise; + setPreference(userId: string, key: string, value: unknown): Promise; + deletePreference(userId: string, key: string): Promise; + listPreferences( + userId: string, + category?: string, + ): Promise>; + + // Insights (with optional vector search) + storeInsight(insight: NewInsight): Promise; + getInsight(id: string): Promise; + searchInsights(query: string, limit?: number, filter?: InsightFilter): Promise; + deleteInsight(id: string): Promise; + + // Embedding provider (optional, null = no vector search) + readonly embedder?: EmbeddingProvider | null; + + close(): Promise; +} + +export interface NewInsight { + id: string; + userId: string; + content: string; + embedding?: number[]; // If embedder is available + source: 'agent' | 'user' | 'summarization' | 'system'; + category: 'decision' | 'learning' | 'preference' | 'fact' | 'pattern' | 'general'; + relevanceScore: number; + metadata?: Record; + createdAt: Date; + decayedAt?: Date; +} + +export interface InsightFilter { + userId?: string; + category?: string; + source?: string; + minRelevance?: number; + fromDate?: Date; + toDate?: Date; +} + +export interface SearchResult { + documentId: string; + content: string; + distance: number; + metadata?: Record; +} + +export interface MemoryConfig { + type: 'pgvector' | 'sqlite-vec' | 'keyword'; + storage: StorageAdapter; + embedder?: EmbeddingProvider; +} + +export interface EmbeddingProvider { + embed(text: string): Promise; + embedBatch(texts: string[]): Promise; + readonly dimensions: number; +} +``` + +## Three Tiers + +### Tier 1: Local (Zero Dependencies) + +**Target:** Single user, single machine, no external services + +| Component | Backend | Storage | +| --------- | --------------------------------------------- | ------------ | +| Queue | In-process + JSON files in `~/.mosaic/queue/` | +| Storage | SQLite (better-sqlite3) `~/.mosaic/data.db` | +| Memory | Keyword search | SQLite table | +| Vector | None | N/A | + +**Dependencies:** + +- `better-sqlite3` (bundled) +- No Postgres, No Valkey, No pgvector + +**Upgrade path:** + +1. Run `mosaic gateway configure` → select "local" tier +2. Gateway starts with SQLite database +3. Optional: run `mosaic gateway upgrade --tier team` to migrate to Postgres + +### Tier 2: Team (Postgres + Valkey) + +**Target:** Multiple users, shared server, CI/CD environments + +| Component | Backend | Storage | +| --------- | -------------- | ------------------------------ | +| Queue | BullMQ | Valkey | +| Storage | Postgres | Shared PG instance | +| Memory | pgvector | Postgres with vector extension | +| Vector | LLM embeddings | Configured provider | + +**Dependencies:** + +- PostgreSQL 17+ with pgvector extension +- Valkey (Redis-compatible) +- LLM provider for embeddings + +**Migration from Local → Team:** + +1. `mosaic gateway backup` → creates dump of SQLite database +2. `mosaic gateway upgrade --tier team` → restores to Postgres +3. Queue replays from BullMQ (may need manual reconciliation for in-flight jobs) +4. Memory embeddings regenerated if vector search was new + +### Tier 3: Enterprise (Clustered) + +**Target:** Large teams, multi-region, high availability + +| Component | Backend | Storage | +| --------- | --------------------------- | ----------------------------- | +| Queue | BullMQ cluster | Multiple Valkey nodes | +| Storage | Postgres cluster | Primary + replicas | +| Memory | Dedicated vector DB | Qdrant, Pinecone, or pgvector | +| Vector | Dedicated embedding service | Separate microservice | + +## MarkdownDB Integration + +For file-based storage, we use [MarkdownDB](https://markdowndb.com) to parse MD files into queryable data. + +**What it provides:** + +- Parses frontmatter (YAML/JSON/TOML) +- Extracts links, tags, metadata +- Builds index in JSON or SQLite +- Queryable via SQL-like interface + +**Usage in Mosaic:** + +```typescript +// Local tier with MD files for documents +const storage = createStorageAdapter({ + type: 'files', + path: path.join(mosaicHome, 'docs'), + markdowndb: { + parseFrontmatter: true, + extractLinks: true, + indexFile: 'index.json', + }, +}); +``` + +## Dream Mode — Memory Consolidation + +Automated equivalent to Claude Code's "Dream: Memory Consolidation" cycle + +**Trigger:** Every 24 hours (if 5+ sessions active) + +**Phases:** + +1. **Orient** — What happened, what's the current state + - Scan recent session logs + - Identify active tasks, missions, conversations + - Calculate time window (last 24h) + +2. **Gather** — Pull in relevant context + - Load conversations, decisions, agent logs + - Extract key interactions and outcomes + - Identify patterns and learnings + +3. **Consolidate** — Summarize and compress + - Generate summary of the last 24h + - Extract key decisions and their rationale + - Identify recurring patterns + - Compress verbose logs into concise insights + +4. **Prune** — Archive and cleanup + - Archive raw session files to dated folders + - Delete redundant/temporary data + - Update MEMORY.md with consolidated content + - Update insight relevance scores + +**Implementation:** + +```typescript +// In @mosaic/dream (new package) +export async function runDreamCycle(config: DreamConfig): Promise { + const memory = await loadMemoryAdapter(config.storage); + + // Orient + const sessions = await memory.getRecentSessions(24 * 60 * 60 * 1000); + if (sessions.length < 5) return { skipped: true, reason: 'insufficient_sessions' }; + + // Gather + const context = await gatherContext(memory, sessions); + + // Consolidate + const consolidated = await consolidateWithLLM(context, config.llm); + + // Prune + await pruneArchivedData(memory, config.retention); + + // Store consolidated insights + await memory.storeInsights(consolidated.insights); + + return { + sessionsProcessed: sessions.length, + insightsCreated: consolidated.insights.length, + bytesPruned: consolidated.bytesRemoved, + }; +} +``` + +--- + +## Retrofit Plan + +### Phase 1: Interface Extraction (2-3 days) + +**Goal:** Define interfaces without changing existing behavior + +1. Create `packages/queue/src/types.ts` with `QueueAdapter` interface +2. Create `packages/storage/src/types.ts` with `StorageAdapter` interface +3. Create `packages/memory/src/types.ts` with `MemoryAdapter` interface (refactor existing) +4. Add adapter registry pattern to each package +5. No breaking changes — existing code continues to work + +### Phase 2: Refactor Existing to Adapters (3-5 days) + +**Goal:** Move existing implementations behind adapters + +#### 2.1 Queue Refactor + +1. Rename `packages/queue/src/queue.ts` → `packages/queue/src/adapters/bullmq.ts` +2. Create `packages/queue/src/index.ts` to export factory function +3. Factory function reads config, instantiates correct adapter +4. Update gateway imports to use factory + +#### 2.2 Storage Refactor + +1. Create `packages/storage/` (new package) +2. Move Drizzle logic to `packages/storage/src/adapters/postgres.ts` +3. Create SQLite adapter in `packages/storage/src/adapters/sqlite.ts` +4. Update gateway to use storage factory +5. Deprecate direct `@mosaic/db` imports + +#### 2.3 Memory Refactor + +1. Extract existing logic to `packages/memory/src/adapters/pgvector.ts` +2. Create keyword adapter in `packages/memory/src/adapters/keyword.ts` +3. Update vector-store.ts to be adapter-agnostic + +### Phase 3: Local Tier Implementation (2-3 days) + +**Goal:** Zero-dependency baseline + +1. Implement `packages/queue/src/adapters/local.ts` (in-process + JSON persistence) +2. Implement `packages/storage/src/adapters/files.ts` (JSON + MD via MarkdownDB) +3. Implement `packages/memory/src/adapters/keyword.ts` (TF-IDF search) +4. Add `packages/dream/` for consolidation cycle +5. Wire up local tier in gateway startup + +### Phase 4: Configuration System (1-2 days) + +**Goal:** Runtime backend selection + +1. Create `packages/config/src/storage.ts` for storage configuration +2. Add `mosaic.config.ts` schema with storage tier settings +3. Update gateway to read config on startup +4. Add `mosaic gateway configure` CLI command +5. Add tier migration commands (`mosaic gateway upgrade`) + +### Phase 5: Testing & Documentation (2-3 days) + +1. Unit tests for each adapter +2. Integration tests for factory pattern +3. Migration tests (local → team) +4. Update README and architecture docs +5. Add configuration guide + +--- + +## File Changes Summary + +### New Files + +``` +packages/ +├── config/ +│ └── src/ +│ ├── storage.ts # Storage config schema +│ └── index.ts +├── dream/ # NEW: Dream mode consolidation +│ ├── src/ +│ │ ├── index.ts +│ │ ├── orient.ts +│ │ ├── gather.ts +│ │ ├── consolidate.ts +│ │ └── prune.ts +│ └── package.json +├── queue/ +│ └── src/ +│ ├── types.ts # NEW: QueueAdapter interface +│ ├── index.ts # NEW: Factory function +│ └── adapters/ +│ ├── bullmq.ts # MOVED from queue.ts +│ └── local.ts # NEW: In-process adapter +├── storage/ # NEW: Storage abstraction +│ ├── src/ +│ │ ├── types.ts # StorageAdapter interface +│ │ ├── index.ts # Factory function +│ │ └── adapters/ +│ │ ├── postgres.ts # MOVED from @mosaic/db +│ │ ├── sqlite.ts # NEW: SQLite adapter +│ │ └── files.ts # NEW: JSON/MD adapter +│ └── package.json +└── memory/ + └── src/ + ├── types.ts # UPDATED: MemoryAdapter interface + ├── index.ts # UPDATED: Factory function + └── adapters/ + ├── pgvector.ts # EXTRACTED from existing code + ├── sqlite-vec.ts # NEW: SQLite with vectors + └── keyword.ts # NEW: TF-IDF search +``` + +### Modified Files + +``` +packages/ +├── db/ # DEPRECATED: Logic moved to storage adapters +├── queue/ +│ └── src/ +│ └── queue.ts # → adapters/bullmq.ts +├── memory/ +│ ├── src/ +│ │ ├── memory.ts # → use factory +│ │ ├── insights.ts # → use factory +│ │ └── preferences.ts # → use factory +│ └── package.json # Remove pgvector from dependencies +└── gateway/ + └── src/ + ├── database/ + │ └── database.module.ts # Update to use storage factory + ├── memory/ + │ └── memory.module.ts # Update to use memory factory + └── queue/ + └── queue.module.ts # Update to use queue factory +``` + +--- + +## Breaking Changes + +1. **`@mosaic/db`** → **`@mosaic/storage`** (with migration guide) +2. Direct `ioredis` imports → Use `@mosaic/queue` factory +3. Direct `pgvector` queries → Use `@mosaic/memory` factory +4. Gateway startup now requires storage config (defaults to local) + +## Non-Breaking Migration Path + +1. Existing deployments with Postgres/Valkey continue to work (default config) +2. New deployments can choose local tier +3. Migration commands available when ready to upgrade + +--- + +## Success Criteria + +- [ ] Local tier runs with zero external dependencies +- [ ] All three tiers (local, team, enterprise) work correctly +- [ ] Factory pattern correctly selects backend at runtime +- [ ] Migration from local → team preserves all data +- [ ] Dream mode consolidates 24h of sessions +- [ ] Documentation covers all three tiers and migration paths +- [ ] All existing tests pass +- [ ] New adapters have >80% coverage