mosaicstack/stack
1
0
Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases 11 Wiki Activity

chore(orchestrator): Bootstrap storage abstraction retrofit

Browse Source 
Mission: Decouple gateway from hardcoded Postgres/Valkey backends.
20 tasks across 5 phases. Estimated total: ~214K tokens.

Phase 1: Interface extraction (4 tasks)
Phase 2: Wrap existing backends as adapters (5 tasks)
Phase 3: Local tier implementation (4 tasks)
Phase 4: Config + CLI commands (4 tasks)
Phase 5: Migration + docs (3 tasks)
This commit is contained in:
Jarvis
2026-04-02 20:15:57 -05:00
parent 0be9729e40
commit 73043773d8
2 changed files with 580 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
docs/TASKS.md
View File

@@ -1,73 +1,30 @@
# Tasks — Harness Foundation
# Tasks — Storage Abstraction Retrofit
> Single-writer: orchestrator only. Workers read but never modify.
>
> **Mission:** Decouple gateway from hardcoded Postgres/Valkey backends. Introduce interface-driven middleware so the gateway is backend-agnostic. Default to local tier (SQLite + JSON) for zero-dependency installs.
>
> **`agent` column values:** `codex` | `sonnet` | `haiku` | `glm-5` | `opus` | `—` (auto/default)
| id | status | agent | milestone | description | pr | notes |
| ------ | ------ | ------ | ------------------ | ------------------------------------------------------------------ | ---- | ----------- |
| M1-001 | done | sonnet | M1: Persistence | Wire ChatGateway → ConversationsRepo for user messages | #292 | #224 closed |
| M1-002 | done | sonnet | M1: Persistence | Wire agent event relay → ConversationsRepo for assistant responses | #292 | #225 closed |
| M1-003 | done | sonnet | M1: Persistence | Store message metadata: model, provider, tokens, tool calls | #292 | #226 closed |
| M1-004 | done | sonnet | M1: Persistence | Load message history into Pi session on resume | #301 | #227 closed |
| M1-005 | done | sonnet | M1: Persistence | Context window management: summarize when >80% | #301 | #228 closed |
| M1-006 | done | sonnet | M1: Persistence | Conversation search endpoint | #299 | #229 closed |
| M1-007 | done | sonnet | M1: Persistence | TUI /history command | #297 | #230 closed |
| M1-008 | done | sonnet | M1: Persistence | Verify persistence — 20 tests | #304 | #231 closed |
| M2-001 | done | sonnet | M2: Security | InsightsRepo userId on searchByEmbedding | #290 | #232 closed |
| M2-002 | done | sonnet | M2: Security | InsightsRepo userId on findByUser/decay | #290 | #233 closed |
| M2-003 | done | sonnet | M2: Security | PreferencesRepo userId verified | #294 | #234 closed |
| M2-004 | done | sonnet | M2: Security | Memory tools userId injection fixed | #294 | #235 closed |
| M2-005 | done | sonnet | M2: Security | ConversationsRepo ownership checks | #293 | #236 closed |
| M2-006 | done | sonnet | M2: Security | AgentsRepo findAccessible scoped | #293 | #237 closed |
| M2-007 | done | sonnet | M2: Security | Cross-user isolation — 28 tests | #305 | #238 closed |
| M2-008 | done | sonnet | M2: Security | Valkey SCAN + /gc admin-only | #298 | #239 closed |
| M3-001 | done | sonnet | M3: Providers | IProviderAdapter + OllamaAdapter | #306 | #240 closed |
| M3-002 | done | sonnet | M3: Providers | AnthropicAdapter | #309 | #241 closed |
| M3-003 | done | sonnet | M3: Providers | OpenAIAdapter | #310 | #242 closed |
| M3-004 | done | sonnet | M3: Providers | OpenRouterAdapter | #311 | #243 closed |
| M3-005 | done | sonnet | M3: Providers | ZaiAdapter (GLM-5) | #314 | #244 closed |
| M3-006 | done | sonnet | M3: Providers | Ollama embedding support | #311 | #245 closed |
| M3-007 | done | sonnet | M3: Providers | Provider health checks | #308 | #246 closed |
| M3-008 | done | sonnet | M3: Providers | Model capability matrix | #303 | #247 closed |
| M3-009 | done | sonnet | M3: Providers | EmbeddingService → Ollama default | #308 | #248 closed |
| M3-010 | done | sonnet | M3: Providers | OAuth token storage (AES-256-GCM) | #317 | #249 closed |
| M3-011 | done | sonnet | M3: Providers | Provider credentials CRUD | #317 | #250 closed |
| M3-012 | done | sonnet | M3: Providers | Verify providers — 40 tests | #319 | #251 closed |
| M4-001 | done | sonnet | M4: Routing | routing_rules DB schema | #315 | #252 closed |
| M4-002 | done | sonnet | M4: Routing | Condition types | #315 | #253 closed |
| M4-003 | done | sonnet | M4: Routing | Action types | #315 | #254 closed |
| M4-004 | done | sonnet | M4: Routing | Default routing rules (11 seeds) | #316 | #255 closed |
| M4-005 | done | sonnet | M4: Routing | Task classifier (60+ tests) | #316 | #256 closed |
| M4-006 | done | sonnet | M4: Routing | Routing decision pipeline | #318 | #257 closed |
| M4-007 | done | sonnet | M4: Routing | /model override | #323 | #258 closed |
| M4-008 | done | sonnet | M4: Routing | Routing transparency in session:info | #323 | #259 closed |
| M4-009 | done | sonnet | M4: Routing | Routing rules CRUD API | #320 | #260 closed |
| M4-010 | done | sonnet | M4: Routing | Per-user routing overrides | #320 | #261 closed |
| M4-011 | done | sonnet | M4: Routing | Agent specialization capabilities | #320 | #262 closed |
| M4-012 | done | sonnet | M4: Routing | Routing wired into ChatGateway | #323 | #263 closed |
| M4-013 | done | sonnet | M4: Routing | Verify routing — 9 E2E tests | #323 | #264 closed |
| M5-001 | done | sonnet | M5: Sessions | Agent config loaded on session create | #323 | #265 closed |
| M5-002 | done | sonnet | M5: Sessions | /model command end-to-end | #323 | #266 closed |
| M5-003 | done | sonnet | M5: Sessions | /agent command mid-session | #323 | #267 closed |
| M5-004 | done | sonnet | M5: Sessions | Session ↔ conversation binding | #321 | #268 closed |
| M5-005 | done | sonnet | M5: Sessions | Session info broadcast | #321 | #269 closed |
| M5-006 | done | sonnet | M5: Sessions | /agent new from TUI | #321 | #270 closed |
| M5-007 | done | sonnet | M5: Sessions | Session metrics | #321 | #271 closed |
| M5-008 | done | sonnet | M5: Sessions | Verify sessions — 28 tests | #324 | #272 closed |
| M6-001 | done | sonnet | M6: Jobs | BullMQ + Valkey config | #324 | #273 closed |
| M6-002 | done | sonnet | M6: Jobs | Queue service with typed jobs | #324 | #274 closed |
| M6-003 | done | sonnet | M6: Jobs | Summarization → BullMQ | #324 | #275 closed |
| M6-004 | done | sonnet | M6: Jobs | GC → BullMQ | #324 | #276 closed |
| M6-005 | done | sonnet | M6: Jobs | Tier management → BullMQ | #324 | #277 closed |
| M6-006 | done | sonnet | M6: Jobs | Admin jobs API | #325 | #278 closed |
| M6-007 | done | sonnet | M6: Jobs | Job event logging | #325 | #279 closed |
| M6-008 | done | sonnet | M6: Jobs | Verify jobs | #324 | #280 closed |
| M7-001 | done | sonnet | M7: Channel Design | IChannelAdapter interface | #325 | #281 closed |
| M7-002 | done | sonnet | M7: Channel Design | Channel message protocol | #325 | #282 closed |
| M7-003 | done | sonnet | M7: Channel Design | Matrix integration design | #326 | #283 closed |
| M7-004 | done | sonnet | M7: Channel Design | Conversation multiplexing | #326 | #284 closed |
| M7-005 | done | sonnet | M7: Channel Design | Remote auth bridging | #326 | #285 closed |
| M7-006 | done | sonnet | M7: Channel Design | Agent-to-agent via Matrix | #326 | #286 closed |
| M7-007 | done | sonnet | M7: Channel Design | Multi-user isolation in Matrix | #326 | #287 closed |
| M7-008 | done | sonnet | M7: Channel Design | channel-protocol.md published | #326 | #288 closed |
| id | status | description | issue | repo | branch | depends_on | blocks | agent | started_at | completed_at | estimate | used | notes |
| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ | ----- | -------------------- | ------------------------ | ----------------------------- | --------- | ----- | ---------- | ------------ | -------- | ---- | ----- |
| SA-P1-001 | not-started | Define QueueAdapter interface in packages/queue/src/types.ts — enqueue, dequeue, length, publish, subscribe, close | | queue | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | |
| SA-P1-002 | not-started | Define StorageAdapter interface in packages/storage/src/types.ts — CRUD, query, find, transaction, close | | storage | feat/storage-abstraction | | SA-P1-004 | codex | | | 8K | | |
| SA-P1-003 | not-started | Define MemoryAdapter interface in packages/memory/src/types.ts — preferences, insights, search, embedder support | | memory | feat/storage-abstraction | | SA-P1-004 | codex | | | 5K | | |
| SA-P1-004 | not-started | Create adapter factory pattern + config types: createQueue(config), createStorage(config), createMemory(config) | | queue,storage,memory | feat/storage-abstraction | SA-P1-001,SA-P1-002,SA-P1-003 | SA-P2-001 | codex | | | 8K | | |
| SA-P2-001 | not-started | Refactor @mosaic/queue: move queue.ts → adapters/bullmq.ts, implement QueueAdapter interface, export factory | | queue | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | |
| SA-P2-002 | not-started | Create @mosaic/storage package: move @mosaic/db Drizzle logic → adapters/postgres.ts, implement StorageAdapter interface | | storage | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 15K | | |
| SA-P2-003 | not-started | Refactor @mosaic/memory: extract pgvector logic → adapters/pgvector.ts, implement MemoryAdapter interface | | memory | feat/storage-abstraction | SA-P1-004 | SA-P2-004 | codex | | | 12K | | |
| SA-P2-004 | not-started | Update gateway database.module.ts, queue.module.ts, memory.module.ts to use factories + NestJS DI tokens | | gateway | feat/storage-abstraction | SA-P2-001,SA-P2-002,SA-P2-003 | SA-P2-005 | codex | | | 15K | | |
| SA-P2-005 | not-started | Verify Phase 2: existing Postgres/Valkey behavior unchanged — all existing tests pass, typecheck clean | | gateway | feat/storage-abstraction | SA-P2-004 | SA-P3-001 | codex | | | 10K | | |
| SA-P3-001 | not-started | Implement local queue adapter: in-process Map + JSON file persistence in adapters/local.ts | | queue | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 10K | | |
| SA-P3-002 | not-started | Implement SQLite storage adapter: better-sqlite3 with schema mirroring Drizzle tables, in adapters/sqlite.ts | | storage | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 20K | | |
| SA-P3-003 | not-started | Implement keyword memory adapter: TF-IDF or simple keyword matching, no vector dependency, in adapters/keyword.ts | | memory | feat/storage-abstraction | SA-P2-005 | SA-P3-004 | codex | | | 12K | | |
| SA-P3-004 | not-started | Verify Phase 3: gateway starts with local config (no PG/Valkey), basic CRUD works, all adapter tests pass | | gateway | feat/storage-abstraction | SA-P3-001,SA-P3-002,SA-P3-003 | SA-P4-001 | codex | | | 15K | | |
| SA-P4-001 | not-started | Add mosaic.config.ts schema + loader: storage tier selection (local/team), backend config, defaults to local | | config | feat/storage-abstraction | SA-P3-004 | SA-P4-002 | codex | | | 10K | | |
| SA-P4-002 | not-started | CLI: mosaic gateway init — interactive wizard to generate mosaic.config.ts with tier selection | | cli | feat/storage-abstraction | SA-P4-001 | SA-P4-003 | codex | | | 12K | | |
| SA-P4-003 | not-started | CLI: mosaic gateway start/stop/status — lifecycle management for the gateway daemon | | cli | feat/storage-abstraction | SA-P4-002 | SA-P4-004 | codex | | | 12K | | |
| SA-P4-004 | not-started | Verify Phase 4: fresh install with `mosaic gateway init && mosaic gateway start` works end-to-end | | cli,gateway | feat/storage-abstraction | SA-P4-003 | SA-P5-001 | codex | | | 10K | | |
| SA-P5-001 | not-started | Migration tooling: mosaic storage export/import for local↔postgres tier migration | | cli,storage | feat/storage-abstraction | SA-P4-004 | SA-P5-002 | codex | | | 15K | | |
| SA-P5-002 | not-started | Docker Compose profiles: local (gateway only) vs team (gateway+pg+valkey), update docker-compose.yml | | infra | feat/storage-abstraction | SA-P5-001 | SA-P5-003 | codex | | | 8K | | |
| SA-P5-003 | not-started | Final verification + docs: README update, architecture diagram, configuration guide | | docs | feat/storage-abstraction | SA-P5-002 | | codex | | | 10K | | |

555
docs/design/storage-abstraction-middleware.md Normal file
View File

@@ -0,0 +1,555 @@
# Storage & Queue Abstraction — Middleware Architecture
Design
Status: Design (retrofit required)
date: 2026-04-02
context: Agents coupled directly to infrastructure backends, bypassing intended middleware layer
---
## The Problem
Current packages are **direct adapters**, not **middleware**:
| Package | Current State | Intended Design |
|---------|---------------|-----------------|
| `@mosaic/queue` | `ioredis` hardcoded | Interface → BullMQ OR local-files |
| `@mosaic/db` | Drizzle + Postgres hardcoded | Interface → Postgres OR SQLite OR JSON/MD |
| `@mosaic/memory` | pgvector required | Interface → pgvector OR sqlite-vec OR keyword-search |
## The gateway and TUI import these packages directly, which means they they're coupled to specific infrastructure. Users cannot run Mosaic Stack without Postgres + Valkey.
## The Intended Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Gateway / TUI / CLI │
│ (agnostic of storage backend, talks to middleware) │
└───────────────────────────┬─────────────────────────────────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
▼─────────────────┴─────────────────┴─────────────────┘
| | | |
▼─────────────────┴───────────────────┴─────────────────┘
| | | |
Queue Storage Memory
| | | |
┌─────────┬─────────┬─────────┬─────────────────────────────────┐
| BullMQ | | Local | | Postgres | SQLite | JSON/MD | pgvector | sqlite-vec | keyword |
|(Valkey)| |(files) | | | | | |
└─────────┴─────────┴─────────┴─────────────────────────────────┘
```
The gateway imports the interface, not the backend. At startup it reads config and instantiates the correct adapter.
## The Drift
```typescript
// What should have happened:
gateway/queue.service.ts → @mosaic/queue (interface) → queue.adapter.ts
// What actually happened:
gateway/queue.service.ts → @mosaic/queue → ioredis (hardcoded)
```
## The Current State Analysis
### `@mosaic/queue` (packages/queue/src/queue.ts)
```typescript
import Redis from 'ioredis'; // ← Direct import of backend
export function createQueue(config?: QueueConfig): QueueHandle {
const url = config?.url ?? process.env['VALKEY_URL'] ?? DEFAULT_VALKEY_URL;
const redis = new Redis(url, { maxRetriesPerRequest: 3 });
// ...queue ops directly on redis...
}
```
**Problem:** `ioredis` is imported in the package, not the adapter interface. Consumers cannot swap backends.
### `@mosaic/db` (packages/db/src/client.ts)
```typescript
import { drizzle, type PostgresJsDatabase } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
export function createDb(url?: string): DbHandle {
const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
const sql = postgres(connectionString, { max: 20, idle_timeout: 30, connect_timeout: 5 });
const db = drizzle(sql, { schema });
// ...
}
```
**Problem:** Drizzle + Postgres is hardcoded. No SQLite, JSON, or file-based options.
### `@mosaic/memory` (packages/memory/src/memory.ts)
```typescript
import type { Db } from '@mosaic/db'; // ← Depends on Drizzle/PG
export function createMemory(db: Db): Memory {
return {
preferences: createPreferencesRepo(db),
insights: createInsightsRepo(db),
};
}
```
**Problem:** Memory package is tightly coupled to `@mosaic/db` (which is Postgres-only). No alternative storage backends.
## The Target Interfaces
### Queue Interface
```typescript
// packages/queue/src/types.ts
export interface QueueAdapter {
readonly name: string;
enqueue(queueName: string, payload: TaskPayload): Promise<void>;
dequeue(queueName: string): Promise<TaskPayload | null>;
length(queueName: string): Promise<number>;
publish(channel: string, message: string): Promise<void>;
subscribe(channel: string, handler: (message: string) => void): () => void;
close(): Promise<void>;
}
export interface TaskPayload {
id: string;
type: string;
data: Record<string, unknown>;
createdAt: string;
}
export interface QueueConfig {
type: 'bullmq' | 'local';
url?: string; // For bullmq: Valkey/Redis URL
dataDir?: string; // For local: directory for JSON persistence
}
```
### Storage Interface
```typescript
// packages/storage/src/types.ts
export interface StorageAdapter {
readonly name: string;
// Entity CRUD
create<T>(collection: string, data: O): Promise<T>;
read<T>(collection: string, id: string): Promise<T | null>;
update<T>(collection: string, id: string, data: Partial<O>): Promise<T | null>;
delete(collection: string, id: string): Promise<boolean>;
// Queries
find<T>(collection: string, filter: Record<string, unknown>): Promise<T[]>;
findOne<T>(collection: string, filter: Record<string, unknown): Promise<T | null>;
// Bulk operations
createMany<T>(collection: string, items: O[]): Promise<T[]>;
updateMany<T>(collection: string, ids: string[], data: Partial<O>): Promise<number>;
deleteMany(collection: string, ids: string[]): Promise<number>;
// Raw queries (for complex queries)
query<T>(collection: string, query: string, params?: unknown[]): Promise<T[]>;
// Transaction support
transaction<T>(fn: (tx: StorageTransaction) => Promise<T>): Promise<T>;
close(): Promise<void>;
}
export interface StorageTransaction {
commit(): Promise<void>;
rollback(): Promise<void>;
}
export interface StorageConfig {
type: 'postgres' | 'sqlite' | 'files';
url?: string; // For postgres
path?: string; // For sqlite/files
}
```
### Memory Interface (Vector + Preferences)
```typescript
// packages/memory/src/types.ts
export interface MemoryAdapter {
readonly name: string;
// Preferences (key-value storage)
getPreference(userId: string, key: string): Promise<unknown | null>;
setPreference(userId: string, key: string, value: unknown): Promise<void>;
deletePreference(userId: string, key: string): Promise<boolean>;
listPreferences(
userId: string,
category?: string,
): Promise<Array<{ key: string; value: unknown }>>;
// Insights (with optional vector search)
storeInsight(insight: NewInsight): Promise<Insight>;
getInsight(id: string): Promise<Insight | null>;
searchInsights(query: string, limit?: number, filter?: InsightFilter): Promise<SearchResult[]>;
deleteInsight(id: string): Promise<boolean>;
// Embedding provider (optional, null = no vector search)
readonly embedder?: EmbeddingProvider | null;
close(): Promise<void>;
}
export interface NewInsight {
id: string;
userId: string;
content: string;
embedding?: number[]; // If embedder is available
source: 'agent' | 'user' | 'summarization' | 'system';
category: 'decision' | 'learning' | 'preference' | 'fact' | 'pattern' | 'general';
relevanceScore: number;
metadata?: Record<string, unknown>;
createdAt: Date;
decayedAt?: Date;
}
export interface InsightFilter {
userId?: string;
category?: string;
source?: string;
minRelevance?: number;
fromDate?: Date;
toDate?: Date;
}
export interface SearchResult {
documentId: string;
content: string;
distance: number;
metadata?: Record<string, unknown>;
}
export interface MemoryConfig {
type: 'pgvector' | 'sqlite-vec' | 'keyword';
storage: StorageAdapter;
embedder?: EmbeddingProvider;
}
export interface EmbeddingProvider {
embed(text: string): Promise<number[]>;
embedBatch(texts: string[]): Promise<number[][]>;
readonly dimensions: number;
}
```
## Three Tiers
### Tier 1: Local (Zero Dependencies)
**Target:** Single user, single machine, no external services
| Component | Backend | Storage |
| --------- | --------------------------------------------- | ------------ |
| Queue | In-process + JSON files in `~/.mosaic/queue/` |
| Storage | SQLite (better-sqlite3) `~/.mosaic/data.db` |
| Memory | Keyword search | SQLite table |
| Vector | None | N/A |
**Dependencies:**
- `better-sqlite3` (bundled)
- No Postgres, No Valkey, No pgvector
**Upgrade path:**
1. Run `mosaic gateway configure` → select "local" tier
2. Gateway starts with SQLite database
3. Optional: run `mosaic gateway upgrade --tier team` to migrate to Postgres
### Tier 2: Team (Postgres + Valkey)
**Target:** Multiple users, shared server, CI/CD environments
| Component | Backend | Storage |
| --------- | -------------- | ------------------------------ |
| Queue | BullMQ | Valkey |
| Storage | Postgres | Shared PG instance |
| Memory | pgvector | Postgres with vector extension |
| Vector | LLM embeddings | Configured provider |
**Dependencies:**
- PostgreSQL 17+ with pgvector extension
- Valkey (Redis-compatible)
- LLM provider for embeddings
**Migration from Local → Team:**
1. `mosaic gateway backup` → creates dump of SQLite database
2. `mosaic gateway upgrade --tier team` → restores to Postgres
3. Queue replays from BullMQ (may need manual reconciliation for in-flight jobs)
4. Memory embeddings regenerated if vector search was new
### Tier 3: Enterprise (Clustered)
**Target:** Large teams, multi-region, high availability
| Component | Backend | Storage |
| --------- | --------------------------- | ----------------------------- |
| Queue | BullMQ cluster | Multiple Valkey nodes |
| Storage | Postgres cluster | Primary + replicas |
| Memory | Dedicated vector DB | Qdrant, Pinecone, or pgvector |
| Vector | Dedicated embedding service | Separate microservice |
## MarkdownDB Integration
For file-based storage, we use [MarkdownDB](https://markdowndb.com) to parse MD files into queryable data.
**What it provides:**
- Parses frontmatter (YAML/JSON/TOML)
- Extracts links, tags, metadata
- Builds index in JSON or SQLite
- Queryable via SQL-like interface
**Usage in Mosaic:**
```typescript
// Local tier with MD files for documents
const storage = createStorageAdapter({
type: 'files',
path: path.join(mosaicHome, 'docs'),
markdowndb: {
parseFrontmatter: true,
extractLinks: true,
indexFile: 'index.json',
},
});
```
## Dream Mode — Memory Consolidation
Automated equivalent to Claude Code's "Dream: Memory Consolidation" cycle
**Trigger:** Every 24 hours (if 5+ sessions active)
**Phases:**
1. **Orient** — What happened, what's the current state
- Scan recent session logs
- Identify active tasks, missions, conversations
- Calculate time window (last 24h)
2. **Gather** — Pull in relevant context
- Load conversations, decisions, agent logs
- Extract key interactions and outcomes
- Identify patterns and learnings
3. **Consolidate** — Summarize and compress
- Generate summary of the last 24h
- Extract key decisions and their rationale
- Identify recurring patterns
- Compress verbose logs into concise insights
4. **Prune** — Archive and cleanup
- Archive raw session files to dated folders
- Delete redundant/temporary data
- Update MEMORY.md with consolidated content
- Update insight relevance scores
**Implementation:**
```typescript
// In @mosaic/dream (new package)
export async function runDreamCycle(config: DreamConfig): Promise<DreamResult> {
const memory = await loadMemoryAdapter(config.storage);
// Orient
const sessions = await memory.getRecentSessions(24 * 60 * 60 * 1000);
if (sessions.length < 5) return { skipped: true, reason: 'insufficient_sessions' };
// Gather
const context = await gatherContext(memory, sessions);
// Consolidate
const consolidated = await consolidateWithLLM(context, config.llm);
// Prune
await pruneArchivedData(memory, config.retention);
// Store consolidated insights
await memory.storeInsights(consolidated.insights);
return {
sessionsProcessed: sessions.length,
insightsCreated: consolidated.insights.length,
bytesPruned: consolidated.bytesRemoved,
};
}
```
---
## Retrofit Plan
### Phase 1: Interface Extraction (2-3 days)
**Goal:** Define interfaces without changing existing behavior
1. Create `packages/queue/src/types.ts` with `QueueAdapter` interface
2. Create `packages/storage/src/types.ts` with `StorageAdapter` interface
3. Create `packages/memory/src/types.ts` with `MemoryAdapter` interface (refactor existing)
4. Add adapter registry pattern to each package
5. No breaking changes — existing code continues to work
### Phase 2: Refactor Existing to Adapters (3-5 days)
**Goal:** Move existing implementations behind adapters
#### 2.1 Queue Refactor
1. Rename `packages/queue/src/queue.ts` → `packages/queue/src/adapters/bullmq.ts`
2. Create `packages/queue/src/index.ts` to export factory function
3. Factory function reads config, instantiates correct adapter
4. Update gateway imports to use factory
#### 2.2 Storage Refactor
1. Create `packages/storage/` (new package)
2. Move Drizzle logic to `packages/storage/src/adapters/postgres.ts`
3. Create SQLite adapter in `packages/storage/src/adapters/sqlite.ts`
4. Update gateway to use storage factory
5. Deprecate direct `@mosaic/db` imports
#### 2.3 Memory Refactor
1. Extract existing logic to `packages/memory/src/adapters/pgvector.ts`
2. Create keyword adapter in `packages/memory/src/adapters/keyword.ts`
3. Update vector-store.ts to be adapter-agnostic
### Phase 3: Local Tier Implementation (2-3 days)
**Goal:** Zero-dependency baseline
1. Implement `packages/queue/src/adapters/local.ts` (in-process + JSON persistence)
2. Implement `packages/storage/src/adapters/files.ts` (JSON + MD via MarkdownDB)
3. Implement `packages/memory/src/adapters/keyword.ts` (TF-IDF search)
4. Add `packages/dream/` for consolidation cycle
5. Wire up local tier in gateway startup
### Phase 4: Configuration System (1-2 days)
**Goal:** Runtime backend selection
1. Create `packages/config/src/storage.ts` for storage configuration
2. Add `mosaic.config.ts` schema with storage tier settings
3. Update gateway to read config on startup
4. Add `mosaic gateway configure` CLI command
5. Add tier migration commands (`mosaic gateway upgrade`)
### Phase 5: Testing & Documentation (2-3 days)
1. Unit tests for each adapter
2. Integration tests for factory pattern
3. Migration tests (local → team)
4. Update README and architecture docs
5. Add configuration guide
---
## File Changes Summary
### New Files
```
packages/
├── config/
│ └── src/
│ ├── storage.ts # Storage config schema
│ └── index.ts
├── dream/ # NEW: Dream mode consolidation
│ ├── src/
│ │ ├── index.ts
│ │ ├── orient.ts
│ │ ├── gather.ts
│ │ ├── consolidate.ts
│ │ └── prune.ts
│ └── package.json
├── queue/
│ └── src/
│ ├── types.ts # NEW: QueueAdapter interface
│ ├── index.ts # NEW: Factory function
│ └── adapters/
│ ├── bullmq.ts # MOVED from queue.ts
│ └── local.ts # NEW: In-process adapter
├── storage/ # NEW: Storage abstraction
│ ├── src/
│ │ ├── types.ts # StorageAdapter interface
│ │ ├── index.ts # Factory function
│ │ └── adapters/
│ │ ├── postgres.ts # MOVED from @mosaic/db
│ │ ├── sqlite.ts # NEW: SQLite adapter
│ │ └── files.ts # NEW: JSON/MD adapter
│ └── package.json
└── memory/
└── src/
├── types.ts # UPDATED: MemoryAdapter interface
├── index.ts # UPDATED: Factory function
└── adapters/
├── pgvector.ts # EXTRACTED from existing code
├── sqlite-vec.ts # NEW: SQLite with vectors
└── keyword.ts # NEW: TF-IDF search
```
### Modified Files
```
packages/
├── db/ # DEPRECATED: Logic moved to storage adapters
├── queue/
│ └── src/
│ └── queue.ts # → adapters/bullmq.ts
├── memory/
│ ├── src/
│ │ ├── memory.ts # → use factory
│ │ ├── insights.ts # → use factory
│ │ └── preferences.ts # → use factory
│ └── package.json # Remove pgvector from dependencies
└── gateway/
└── src/
├── database/
│ └── database.module.ts # Update to use storage factory
├── memory/
│ └── memory.module.ts # Update to use memory factory
└── queue/
└── queue.module.ts # Update to use queue factory
```
---
## Breaking Changes
1. **`@mosaic/db`** → **`@mosaic/storage`** (with migration guide)
2. Direct `ioredis` imports → Use `@mosaic/queue` factory
3. Direct `pgvector` queries → Use `@mosaic/memory` factory
4. Gateway startup now requires storage config (defaults to local)
## Non-Breaking Migration Path
1. Existing deployments with Postgres/Valkey continue to work (default config)
2. New deployments can choose local tier
3. Migration commands available when ready to upgrade
---
## Success Criteria
- [ ] Local tier runs with zero external dependencies
- [ ] All three tiers (local, team, enterprise) work correctly
- [ ] Factory pattern correctly selects backend at runtime
- [ ] Migration from local → team preserves all data
- [ ] Dream mode consolidates 24h of sessions
- [ ] Documentation covers all three tiers and migration paths
- [ ] All existing tests pass
- [ ] New adapters have >80% coverage