docs(design): add Knowledge Module design and implementation plan
- Full design document with architecture, data model, API specs - 28 implementation issues across 5 phases (~127h total) - Wiki-link syntax, semantic search, graph visualization - Integration points for agent access Ref: memory/2025-01-29-agent-orchestration.md
This commit is contained in:
Jason Woltje
73
docs/design/README.md
Normal file
73
docs/design/README.md
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
# Design Documents
|
||||||
|
|
||||||
|
Technical design documents for major Mosaic Stack features.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Design documents serve as:
|
||||||
|
- **Blueprints** for implementation
|
||||||
|
- **Reference** for architectural decisions
|
||||||
|
- **Communication** between team members
|
||||||
|
- **Historical record** of design evolution
|
||||||
|
|
||||||
|
## Document Structure
|
||||||
|
|
||||||
|
Each design document should include:
|
||||||
|
|
||||||
|
1. **Problem Statement** — What are we solving?
|
||||||
|
2. **Architecture Overview** — High-level design with diagrams
|
||||||
|
3. **Database Schema** — Tables, indexes, relationships
|
||||||
|
4. **API Specifications** — Endpoints, request/response formats
|
||||||
|
5. **Implementation Plan** — Phased rollout with milestones
|
||||||
|
6. **Security & Performance** — Considerations and constraints
|
||||||
|
|
||||||
|
## Documents
|
||||||
|
|
||||||
|
### [Agent Orchestration Layer](./agent-orchestration.md)
|
||||||
|
|
||||||
|
**Status:** Design Phase
|
||||||
|
**Version:** 1.0
|
||||||
|
**Date:** 2025-01-29
|
||||||
|
|
||||||
|
Infrastructure for persistent task management and autonomous agent coordination. Enables long-running background work independent of user sessions.
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
- Task queue with priority scheduling
|
||||||
|
- Agent health monitoring and automatic recovery
|
||||||
|
- Checkpoint-based resumption for interrupted work
|
||||||
|
- Multi-workspace coordination with row-level security
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### [Knowledge Module](./knowledge-module.md)
|
||||||
|
|
||||||
|
**Status:** Design Phase
|
||||||
|
**Version:** 1.0
|
||||||
|
**Date:** 2025-01-29
|
||||||
|
**Issues:** [Implementation Tracker](./knowledge-module-issues.md)
|
||||||
|
|
||||||
|
Native knowledge management with wiki-style linking, semantic search, and graph visualization. Enables teams and agents to capture, connect, and query organizational knowledge.
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
- Wiki-style `[[links]]` between entries
|
||||||
|
- Full-text and semantic (vector) search
|
||||||
|
- Interactive knowledge graph visualization
|
||||||
|
- Version history with diff view
|
||||||
|
- Tag-based organization
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
When creating a new design document:
|
||||||
|
|
||||||
|
1. Copy the structure from an existing document
|
||||||
|
2. Use ASCII diagrams for architecture (keep them simple)
|
||||||
|
3. Include code examples in TypeScript
|
||||||
|
4. Specify database schema in SQL (PostgreSQL dialect)
|
||||||
|
5. Add implementation phases with clear milestones
|
||||||
|
6. Update this README with a summary
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated:** 2025-01-29
|
||||||
670
docs/design/knowledge-module-issues.md
Normal file
670
docs/design/knowledge-module-issues.md
Normal file
@@ -0,0 +1,670 @@
|
|||||||
|
# Knowledge Module - Implementation Issues
|
||||||
|
|
||||||
|
> **Epic:** Knowledge Module
|
||||||
|
> **Design Doc:** [knowledge-module.md](./knowledge-module.md)
|
||||||
|
> **Target:** 6 weeks
|
||||||
|
> **Created:** 2025-01-29
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Epic Overview
|
||||||
|
|
||||||
|
Build a native knowledge management module for Mosaic Stack with wiki-style linking, semantic search, and graph visualization.
|
||||||
|
|
||||||
|
**Labels:** `epic`, `feature`, `knowledge-module`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Foundation (Week 1-2)
|
||||||
|
|
||||||
|
### KNOW-001: Database Schema for Knowledge Module
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `database`, `schema`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Create Prisma schema and migrations for the Knowledge module.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `KnowledgeEntry` model with all fields
|
||||||
|
- [ ] `KnowledgeEntryVersion` model for history
|
||||||
|
- [ ] `KnowledgeLink` model for wiki-links
|
||||||
|
- [ ] `KnowledgeTag` and `KnowledgeEntryTag` models
|
||||||
|
- [ ] `KnowledgeEmbedding` model (pgvector ready)
|
||||||
|
- [ ] All indexes defined
|
||||||
|
- [ ] Migration runs cleanly
|
||||||
|
- [ ] Seed data for testing
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Reference design doc for full schema
|
||||||
|
- Ensure `@@unique([workspaceId, slug])` constraint
|
||||||
|
- Add `search_vector` column for full-text search
|
||||||
|
- pgvector extension may need separate setup
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-002: Entry CRUD API Endpoints
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `api`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Implement RESTful API for knowledge entry management.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `POST /api/knowledge/entries` - Create entry
|
||||||
|
- [ ] `GET /api/knowledge/entries` - List entries (paginated, filterable)
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug` - Get single entry
|
||||||
|
- [ ] `PUT /api/knowledge/entries/:slug` - Update entry
|
||||||
|
- [ ] `DELETE /api/knowledge/entries/:slug` - Archive entry (soft delete)
|
||||||
|
- [ ] Workspace isolation enforced
|
||||||
|
- [ ] Input validation with class-validator
|
||||||
|
- [ ] OpenAPI/Swagger documentation
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Follow existing Mosaic API patterns
|
||||||
|
- Use `@WorkspaceGuard()` for tenant isolation
|
||||||
|
- Slug generation from title with collision handling
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-003: Tag Management API
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 3h
|
||||||
|
**Labels:** `api`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Implement tag CRUD and entry-tag associations.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `GET /api/knowledge/tags` - List workspace tags
|
||||||
|
- [ ] `POST /api/knowledge/tags` - Create tag
|
||||||
|
- [ ] `PUT /api/knowledge/tags/:slug` - Update tag
|
||||||
|
- [ ] `DELETE /api/knowledge/tags/:slug` - Delete tag
|
||||||
|
- [ ] `GET /api/knowledge/tags/:slug/entries` - Entries with tag
|
||||||
|
- [ ] Entry creation/update accepts tag slugs
|
||||||
|
- [ ] Auto-create tags if `autoCreateTags: true`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-004: Basic Markdown Rendering
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `rendering`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Render markdown content to HTML with caching.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Markdown-to-HTML conversion on entry save
|
||||||
|
- [ ] Support GFM (tables, task lists, strikethrough)
|
||||||
|
- [ ] Code syntax highlighting (highlight.js or Shiki)
|
||||||
|
- [ ] Sanitize HTML output (XSS prevention)
|
||||||
|
- [ ] Cache rendered HTML in `contentHtml` field
|
||||||
|
- [ ] Invalidate cache on content update
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Use `marked` or `remark` for parsing
|
||||||
|
- Wiki-links (`[[...]]`) parsed but not resolved yet (Phase 2)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-005: Entry List Page UI
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `frontend`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Build the knowledge entry list page in the web UI.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] List view with title, summary, tags, updated date
|
||||||
|
- [ ] Filter by status (draft/published/archived)
|
||||||
|
- [ ] Filter by tag
|
||||||
|
- [ ] Sort by updated/created/title
|
||||||
|
- [ ] Pagination
|
||||||
|
- [ ] Quick search (client-side filter)
|
||||||
|
- [ ] Create new entry button → editor
|
||||||
|
- [ ] Responsive design
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-006: Entry Detail/Editor Page UI
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 8h
|
||||||
|
**Labels:** `frontend`, `editor`, `phase-1`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Build the entry view and edit page.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] View mode with rendered markdown
|
||||||
|
- [ ] Edit mode with markdown editor
|
||||||
|
- [ ] Split view option (edit + preview)
|
||||||
|
- [ ] Title editing
|
||||||
|
- [ ] Tag selection/creation
|
||||||
|
- [ ] Status dropdown
|
||||||
|
- [ ] Save/cancel/delete actions
|
||||||
|
- [ ] Unsaved changes warning
|
||||||
|
- [ ] Keyboard shortcuts (Cmd+S to save)
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Consider CodeMirror or Monaco for editor
|
||||||
|
- May use existing rich-text patterns from Mosaic
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Linking (Week 2-3)
|
||||||
|
|
||||||
|
### KNOW-007: Wiki-Link Parser
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `parsing`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Parse `[[wiki-link]]` syntax from markdown content.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Extract all `[[...]]` patterns from content
|
||||||
|
- [ ] Support `[[slug]]` basic syntax
|
||||||
|
- [ ] Support `[[slug|display text]]` aliased links
|
||||||
|
- [ ] Support `[[slug#header]]` section links
|
||||||
|
- [ ] Return structured link data with positions
|
||||||
|
- [ ] Handle edge cases (nested brackets, escaping)
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
```typescript
|
||||||
|
interface ParsedLink {
|
||||||
|
raw: string; // "[[design|Design Doc]]"
|
||||||
|
target: string; // "design"
|
||||||
|
display: string; // "Design Doc"
|
||||||
|
section?: string; // "header" if [[design#header]]
|
||||||
|
position: { start: number; end: number };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-008: Link Resolution Service
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Resolve parsed wiki-links to actual entries.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Resolve by exact slug match
|
||||||
|
- [ ] Resolve by title match (case-insensitive)
|
||||||
|
- [ ] Fuzzy match fallback (optional)
|
||||||
|
- [ ] Mark unresolved links as broken
|
||||||
|
- [ ] Extract surrounding context for link record
|
||||||
|
- [ ] Batch resolution for efficiency
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-009: Link Storage and Sync
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `database`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Store links in database and keep in sync with content.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] On entry save: parse → resolve → store links
|
||||||
|
- [ ] Remove stale links on update
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug/links/outgoing`
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug/links/incoming` (backlinks)
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug/links/broken`
|
||||||
|
- [ ] Broken link report for workspace
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-010: Backlinks Display UI
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 3h
|
||||||
|
**Labels:** `frontend`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Show incoming links (backlinks) on entry pages.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Backlinks section on entry detail page
|
||||||
|
- [ ] Show linking entry title + context snippet
|
||||||
|
- [ ] Click to navigate to linking entry
|
||||||
|
- [ ] Count badge in sidebar/header
|
||||||
|
- [ ] Empty state when no backlinks
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-011: Link Autocomplete in Editor
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `frontend`, `editor`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Autocomplete suggestions when typing `[[`.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Trigger on `[[` typed in editor
|
||||||
|
- [ ] Show dropdown with matching entries
|
||||||
|
- [ ] Search by title and slug
|
||||||
|
- [ ] Show recent entries first, then alphabetical
|
||||||
|
- [ ] Insert selected entry as `[[slug]]` or `[[slug|title]]`
|
||||||
|
- [ ] Keyboard navigation (arrows, enter, escape)
|
||||||
|
- [ ] Debounced API calls
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-012: Render Links in View Mode
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 3h
|
||||||
|
**Labels:** `frontend`, `rendering`, `phase-2`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Render wiki-links as clickable links in entry view.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `[[slug]]` renders as link to `/knowledge/slug`
|
||||||
|
- [ ] `[[slug|text]]` shows custom text
|
||||||
|
- [ ] Broken links styled differently (red, dashed underline)
|
||||||
|
- [ ] Hover preview (optional, stretch goal)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: Search (Week 3-4)
|
||||||
|
|
||||||
|
### KNOW-013: Full-Text Search Setup
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `database`, `search`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Set up PostgreSQL full-text search for entries.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Add `tsvector` column to entries table
|
||||||
|
- [ ] Create GIN index on search vector
|
||||||
|
- [ ] Weight title (A), summary (B), content (C)
|
||||||
|
- [ ] Trigger to update vector on insert/update
|
||||||
|
- [ ] Verify search performance with test data
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-014: Search API Endpoint
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `search`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Implement search API with full-text search.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `GET /api/knowledge/search?q=...`
|
||||||
|
- [ ] Return ranked results with snippets
|
||||||
|
- [ ] Highlight matching terms in snippets
|
||||||
|
- [ ] Filter by tags, status
|
||||||
|
- [ ] Pagination
|
||||||
|
- [ ] Response time < 200ms
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-015: Search UI
|
||||||
|
|
||||||
|
**Priority:** P0
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `frontend`, `search`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Build search interface in web UI.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Search input in knowledge module header
|
||||||
|
- [ ] Search results page
|
||||||
|
- [ ] Highlighted snippets
|
||||||
|
- [ ] Filter sidebar (tags, status)
|
||||||
|
- [ ] "No results" state with suggestions
|
||||||
|
- [ ] Search as you type (debounced)
|
||||||
|
- [ ] Keyboard shortcut (Cmd+K) to focus search
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-016: pgvector Setup
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `database`, `vector`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Set up pgvector extension for semantic search.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Enable pgvector extension in PostgreSQL
|
||||||
|
- [ ] Create embeddings table with vector column
|
||||||
|
- [ ] HNSW index for fast similarity search
|
||||||
|
- [ ] Verify extension works in dev and prod
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- May need PostgreSQL 15+ for best pgvector support
|
||||||
|
- Consider managed options (Supabase, Neon) if self-hosting is complex
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-017: Embedding Generation Pipeline
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `api`, `vector`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Generate embeddings for entries using OpenAI or local model.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Service to generate embeddings from text
|
||||||
|
- [ ] On entry create/update: queue embedding job
|
||||||
|
- [ ] Background worker processes queue
|
||||||
|
- [ ] Store embedding in database
|
||||||
|
- [ ] Handle API rate limits and errors
|
||||||
|
- [ ] Config for embedding model selection
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Start with OpenAI `text-embedding-ada-002`
|
||||||
|
- Consider local options (sentence-transformers) for cost/privacy
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-018: Semantic Search API
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `search`, `vector`, `phase-3`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Implement semantic (vector) search endpoint.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `POST /api/knowledge/search/semantic`
|
||||||
|
- [ ] Accept natural language query
|
||||||
|
- [ ] Generate query embedding
|
||||||
|
- [ ] Find similar entries by cosine similarity
|
||||||
|
- [ ] Return results with similarity scores
|
||||||
|
- [ ] Configurable similarity threshold
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4: Graph (Week 4-5)
|
||||||
|
|
||||||
|
### KNOW-019: Graph Data API
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `graph`, `phase-4`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
API to retrieve knowledge graph data.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] `GET /api/knowledge/graph` - Full graph (nodes + edges)
|
||||||
|
- [ ] `GET /api/knowledge/graph/:slug` - Subgraph centered on entry
|
||||||
|
- [ ] `GET /api/knowledge/graph/stats` - Graph statistics
|
||||||
|
- [ ] Include orphan detection
|
||||||
|
- [ ] Filter by tag, status
|
||||||
|
- [ ] Limit node count option
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-020: Graph Visualization Component
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 8h
|
||||||
|
**Labels:** `frontend`, `graph`, `phase-4`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Interactive knowledge graph visualization.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Force-directed graph layout
|
||||||
|
- [ ] Nodes sized by connection count
|
||||||
|
- [ ] Nodes colored by status
|
||||||
|
- [ ] Click node to navigate or show details
|
||||||
|
- [ ] Zoom and pan controls
|
||||||
|
- [ ] Layout toggle (force, hierarchical, radial)
|
||||||
|
- [ ] Performance OK with 500+ nodes
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Use D3.js or Cytoscape.js
|
||||||
|
- Consider WebGL renderer for large graphs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-021: Entry-Centered Graph View
|
||||||
|
|
||||||
|
**Priority:** P2
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `frontend`, `graph`, `phase-4`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Show mini-graph on entry detail page.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Small graph showing entry + direct connections
|
||||||
|
- [ ] 1-2 hop neighbors
|
||||||
|
- [ ] Click to expand or navigate
|
||||||
|
- [ ] Toggle to show/hide on entry page
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-022: Graph Statistics Dashboard
|
||||||
|
|
||||||
|
**Priority:** P2
|
||||||
|
**Estimate:** 3h
|
||||||
|
**Labels:** `frontend`, `graph`, `phase-4`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Dashboard showing knowledge base health.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Total entries, links, tags
|
||||||
|
- [ ] Orphan entry count (no links)
|
||||||
|
- [ ] Broken link count
|
||||||
|
- [ ] Average connections per entry
|
||||||
|
- [ ] Most connected entries
|
||||||
|
- [ ] Recently updated
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5: Polish (Week 5-6)
|
||||||
|
|
||||||
|
### KNOW-023: Version History API
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `versioning`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
API for entry version history.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Create version on each save
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug/versions`
|
||||||
|
- [ ] `GET /api/knowledge/entries/:slug/versions/:v`
|
||||||
|
- [ ] `POST /api/knowledge/entries/:slug/restore/:v`
|
||||||
|
- [ ] Version limit per entry (configurable, default 50)
|
||||||
|
- [ ] Prune old versions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-024: Version History UI
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 6h
|
||||||
|
**Labels:** `frontend`, `versioning`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
UI to browse and restore versions.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Version list sidebar/panel
|
||||||
|
- [ ] Show version date, author, change note
|
||||||
|
- [ ] Click to view historical version
|
||||||
|
- [ ] Diff view between versions
|
||||||
|
- [ ] Restore button with confirmation
|
||||||
|
- [ ] Compare any two versions
|
||||||
|
|
||||||
|
**Technical Notes:**
|
||||||
|
- Use diff library for content comparison
|
||||||
|
- Highlight additions/deletions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-025: Markdown Import
|
||||||
|
|
||||||
|
**Priority:** P2
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `import`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Import existing markdown files into knowledge base.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Upload `.md` file(s)
|
||||||
|
- [ ] Parse frontmatter for metadata
|
||||||
|
- [ ] Generate slug from filename or title
|
||||||
|
- [ ] Resolve wiki-links if target entries exist
|
||||||
|
- [ ] Report on import results
|
||||||
|
- [ ] Bulk import from folder/zip
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-026: Export Functionality
|
||||||
|
|
||||||
|
**Priority:** P2
|
||||||
|
**Estimate:** 3h
|
||||||
|
**Labels:** `api`, `export`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Export entries to markdown/PDF.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Export single entry as markdown
|
||||||
|
- [ ] Export single entry as PDF
|
||||||
|
- [ ] Bulk export (all or filtered)
|
||||||
|
- [ ] Include frontmatter in markdown export
|
||||||
|
- [ ] Preserve wiki-links in export
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-027: Caching Layer
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `api`, `performance`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Implement Valkey caching for knowledge module.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] Cache entry JSON
|
||||||
|
- [ ] Cache rendered HTML
|
||||||
|
- [ ] Cache graph data
|
||||||
|
- [ ] Cache search results (short TTL)
|
||||||
|
- [ ] Proper invalidation on updates
|
||||||
|
- [ ] Cache hit metrics
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-028: Documentation
|
||||||
|
|
||||||
|
**Priority:** P1
|
||||||
|
**Estimate:** 4h
|
||||||
|
**Labels:** `docs`, `phase-5`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Document the knowledge module.
|
||||||
|
|
||||||
|
**Acceptance Criteria:**
|
||||||
|
- [ ] User guide for knowledge module
|
||||||
|
- [ ] API reference (OpenAPI already in place)
|
||||||
|
- [ ] Wiki-link syntax reference
|
||||||
|
- [ ] Admin/config documentation
|
||||||
|
- [ ] Architecture overview for developers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stretch Goals / Future
|
||||||
|
|
||||||
|
### KNOW-029: Real-Time Collaboration
|
||||||
|
|
||||||
|
**Priority:** P3
|
||||||
|
**Labels:** `future`, `collaboration`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Multiple users editing same entry simultaneously.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
- Would require CRDT or OT implementation
|
||||||
|
- Significant complexity
|
||||||
|
- Evaluate need before committing
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-030: Entry Templates
|
||||||
|
|
||||||
|
**Priority:** P3
|
||||||
|
**Labels:** `future`, `templates`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Pre-defined templates for common entry types.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
- ADR template
|
||||||
|
- Design doc template
|
||||||
|
- Meeting notes template
|
||||||
|
- Custom templates per workspace
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### KNOW-031: Attachments
|
||||||
|
|
||||||
|
**Priority:** P3
|
||||||
|
**Labels:** `future`, `attachments`
|
||||||
|
|
||||||
|
**Description:**
|
||||||
|
Upload and embed images/files in entries.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
- S3/compatible storage backend
|
||||||
|
- Image optimization
|
||||||
|
- Paste images into editor
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Phase | Issues | Est. Hours | Focus |
|
||||||
|
|-------|--------|------------|-------|
|
||||||
|
| 1 | KNOW-001 to KNOW-006 | 31h | CRUD + Basic UI |
|
||||||
|
| 2 | KNOW-007 to KNOW-012 | 24h | Wiki-links |
|
||||||
|
| 3 | KNOW-013 to KNOW-018 | 28h | Search |
|
||||||
|
| 4 | KNOW-019 to KNOW-022 | 19h | Graph |
|
||||||
|
| 5 | KNOW-023 to KNOW-028 | 25h | Polish |
|
||||||
|
| **Total** | 28 issues | ~127h | ~3-4 dev weeks |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Generated by Jarvis • 2025-01-29*
|
||||||
758
docs/design/knowledge-module.md
Normal file
758
docs/design/knowledge-module.md
Normal file
@@ -0,0 +1,758 @@
|
|||||||
|
# Knowledge Module - Design Document
|
||||||
|
|
||||||
|
> **Status:** Draft
|
||||||
|
> **Author:** Agent (Jarvis)
|
||||||
|
> **Created:** 2025-01-29
|
||||||
|
> **Related:** [Agent Orchestration](./agent-orchestration.md)
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
Development teams and AI agents working on complex projects need a way to:
|
||||||
|
|
||||||
|
1. **Capture decisions** — Why was X chosen over Y?
|
||||||
|
2. **Track connections** — How does component A relate to concept B?
|
||||||
|
3. **Search contextually** — Find relevant context without knowing exact keywords
|
||||||
|
4. **Evolve understanding** — Knowledge changes; track that evolution
|
||||||
|
5. **Share across boundaries** — Human and agent access to the same knowledge base
|
||||||
|
|
||||||
|
### Current Pain Points
|
||||||
|
|
||||||
|
- **Scattered documentation** — README, comments, Slack threads, memory files
|
||||||
|
- **No explicit linking** — Connections exist but aren't captured
|
||||||
|
- **Agent amnesia** — Each session starts fresh, relies on file search
|
||||||
|
- **No decision archaeology** — Hard to find *why* something was decided
|
||||||
|
- **Human/agent mismatch** — Humans browse, agents grep
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Functional Requirements
|
||||||
|
|
||||||
|
| ID | Requirement | Priority |
|
||||||
|
|----|-------------|----------|
|
||||||
|
| FR1 | Create, read, update, delete knowledge entries | P0 |
|
||||||
|
| FR2 | Wiki-style linking between entries (`[[link]]` syntax) | P0 |
|
||||||
|
| FR3 | Tagging and categorization | P0 |
|
||||||
|
| FR4 | Full-text search | P0 |
|
||||||
|
| FR5 | Semantic/vector search for agents | P1 |
|
||||||
|
| FR6 | Graph visualization of connections | P1 |
|
||||||
|
| FR7 | Version history and diff view | P1 |
|
||||||
|
| FR8 | Timeline view of changes | P2 |
|
||||||
|
| FR9 | Import from markdown files | P2 |
|
||||||
|
| FR10 | Export to markdown/PDF | P2 |
|
||||||
|
|
||||||
|
### Non-Functional Requirements
|
||||||
|
|
||||||
|
| ID | Requirement | Target |
|
||||||
|
|----|-------------|--------|
|
||||||
|
| NFR1 | Search response time | < 200ms |
|
||||||
|
| NFR2 | Entry render time | < 100ms |
|
||||||
|
| NFR3 | Graph render (< 1000 nodes) | < 500ms |
|
||||||
|
| NFR4 | Multi-tenant isolation | Complete |
|
||||||
|
| NFR5 | API-first design | All features via API |
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Mosaic Web UI │
|
||||||
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Knowledge Browser │ Graph View │ Search │ Timeline │
|
||||||
|
└─────────┬───────────┴──────┬───────┴────┬─────┴────┬────────────┘
|
||||||
|
│ │ │ │
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Knowledge API (NestJS) │
|
||||||
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
|
│ EntryController │ SearchController │ GraphController │
|
||||||
|
│ TagController │ LinkController │ VersionController │
|
||||||
|
└─────────┬─────────┴─────────┬──────────┴──────────┬─────────────┘
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────┐
|
||||||
|
│ PostgreSQL │ │ Valkey │ │ Vector Store │
|
||||||
|
│ │ │ │ │ (pgvector) │
|
||||||
|
│ - entries │ │ - search cache │ │ │
|
||||||
|
│ - entry_versions │ │ - graph cache │ │ - embeddings │
|
||||||
|
│ - entry_links │ │ - hot entries │ │ - semantic index │
|
||||||
|
│ - tags │ │ │ │ │
|
||||||
|
└──────────────────┘ └──────────────────┘ └──────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### Core Entities
|
||||||
|
|
||||||
|
```prisma
|
||||||
|
// Entry - A single knowledge entry (document/page)
|
||||||
|
model KnowledgeEntry {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
workspaceId String
|
||||||
|
workspace Workspace @relation(fields: [workspaceId], references: [id])
|
||||||
|
|
||||||
|
slug String // URL-friendly identifier
|
||||||
|
title String
|
||||||
|
content String @db.Text // Markdown content
|
||||||
|
contentHtml String? @db.Text // Rendered HTML (cached)
|
||||||
|
summary String? // Auto-generated or manual summary
|
||||||
|
|
||||||
|
status EntryStatus @default(DRAFT)
|
||||||
|
visibility Visibility @default(PRIVATE)
|
||||||
|
|
||||||
|
// Metadata
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
updatedAt DateTime @updatedAt
|
||||||
|
createdBy String
|
||||||
|
updatedBy String
|
||||||
|
|
||||||
|
// Relations
|
||||||
|
tags KnowledgeEntryTag[]
|
||||||
|
outgoingLinks KnowledgeLink[] @relation("SourceEntry")
|
||||||
|
incomingLinks KnowledgeLink[] @relation("TargetEntry")
|
||||||
|
versions KnowledgeEntryVersion[]
|
||||||
|
embedding KnowledgeEmbedding?
|
||||||
|
|
||||||
|
@@unique([workspaceId, slug])
|
||||||
|
@@index([workspaceId, status])
|
||||||
|
@@index([workspaceId, updatedAt])
|
||||||
|
}
|
||||||
|
|
||||||
|
enum EntryStatus {
|
||||||
|
DRAFT
|
||||||
|
PUBLISHED
|
||||||
|
ARCHIVED
|
||||||
|
}
|
||||||
|
|
||||||
|
enum Visibility {
|
||||||
|
PRIVATE // Only creator
|
||||||
|
WORKSPACE // All workspace members
|
||||||
|
PUBLIC // Anyone with link
|
||||||
|
}
|
||||||
|
|
||||||
|
// Version history
|
||||||
|
model KnowledgeEntryVersion {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
entryId String
|
||||||
|
entry KnowledgeEntry @relation(fields: [entryId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
version Int
|
||||||
|
title String
|
||||||
|
content String @db.Text
|
||||||
|
summary String?
|
||||||
|
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
createdBy String
|
||||||
|
changeNote String? // Optional commit message
|
||||||
|
|
||||||
|
@@unique([entryId, version])
|
||||||
|
@@index([entryId, version])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wiki-style links between entries
|
||||||
|
model KnowledgeLink {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
|
||||||
|
sourceId String
|
||||||
|
source KnowledgeEntry @relation("SourceEntry", fields: [sourceId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
targetId String
|
||||||
|
target KnowledgeEntry @relation("TargetEntry", fields: [targetId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
// Link metadata
|
||||||
|
linkText String // The text used in [[link|display text]]
|
||||||
|
context String? // Surrounding text for context
|
||||||
|
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
|
||||||
|
@@unique([sourceId, targetId])
|
||||||
|
@@index([sourceId])
|
||||||
|
@@index([targetId])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tags for categorization
|
||||||
|
model KnowledgeTag {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
workspaceId String
|
||||||
|
workspace Workspace @relation(fields: [workspaceId], references: [id])
|
||||||
|
|
||||||
|
name String
|
||||||
|
slug String
|
||||||
|
color String? // Hex color for UI
|
||||||
|
description String?
|
||||||
|
|
||||||
|
entries KnowledgeEntryTag[]
|
||||||
|
|
||||||
|
@@unique([workspaceId, slug])
|
||||||
|
}
|
||||||
|
|
||||||
|
model KnowledgeEntryTag {
|
||||||
|
entryId String
|
||||||
|
entry KnowledgeEntry @relation(fields: [entryId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
tagId String
|
||||||
|
tag KnowledgeTag @relation(fields: [tagId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
@@id([entryId, tagId])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vector embeddings for semantic search
|
||||||
|
model KnowledgeEmbedding {
|
||||||
|
id String @id @default(cuid())
|
||||||
|
entryId String @unique
|
||||||
|
entry KnowledgeEntry @relation(fields: [entryId], references: [id], onDelete: Cascade)
|
||||||
|
|
||||||
|
embedding Unsupported("vector(1536)") // OpenAI ada-002 dimension
|
||||||
|
model String // Which model generated this
|
||||||
|
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
updatedAt DateTime @updatedAt
|
||||||
|
|
||||||
|
@@index([embedding], type: Hnsw(ops: VectorCosineOps))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontmatter Schema
|
||||||
|
|
||||||
|
Entries support YAML frontmatter for structured metadata:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
title: Agent Orchestration Design
|
||||||
|
status: published
|
||||||
|
tags: [architecture, agents, orchestration]
|
||||||
|
created: 2025-01-29
|
||||||
|
updated: 2025-01-29
|
||||||
|
author: jarvis
|
||||||
|
related:
|
||||||
|
- "[[task-queues]]"
|
||||||
|
- "[[valkey-patterns]]"
|
||||||
|
decision:
|
||||||
|
status: accepted
|
||||||
|
date: 2025-01-29
|
||||||
|
participants: [jason, jarvis]
|
||||||
|
supersedes: null
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### Entry Management
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/knowledge/entries Create entry
|
||||||
|
GET /api/knowledge/entries List entries (paginated)
|
||||||
|
GET /api/knowledge/entries/:slug Get entry by slug
|
||||||
|
PUT /api/knowledge/entries/:slug Update entry
|
||||||
|
DELETE /api/knowledge/entries/:slug Delete entry (soft delete → archive)
|
||||||
|
|
||||||
|
GET /api/knowledge/entries/:slug/versions List versions
|
||||||
|
GET /api/knowledge/entries/:slug/versions/:v Get specific version
|
||||||
|
POST /api/knowledge/entries/:slug/restore/:v Restore to version
|
||||||
|
```
|
||||||
|
|
||||||
|
### Search
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/knowledge/search?q=... Full-text search
|
||||||
|
POST /api/knowledge/search/semantic Semantic search (vector)
|
||||||
|
GET /api/knowledge/search/suggestions Autocomplete suggestions
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/knowledge/graph Full graph (nodes + edges)
|
||||||
|
GET /api/knowledge/graph/:slug Subgraph centered on entry
|
||||||
|
GET /api/knowledge/graph/stats Graph statistics
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tags
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/knowledge/tags List all tags
|
||||||
|
POST /api/knowledge/tags Create tag
|
||||||
|
PUT /api/knowledge/tags/:slug Update tag
|
||||||
|
DELETE /api/knowledge/tags/:slug Delete tag
|
||||||
|
GET /api/knowledge/tags/:slug/entries Entries with tag
|
||||||
|
```
|
||||||
|
|
||||||
|
### Links
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/knowledge/entries/:slug/links/outgoing Outgoing links
|
||||||
|
GET /api/knowledge/entries/:slug/links/incoming Incoming links (backlinks)
|
||||||
|
GET /api/knowledge/entries/:slug/links/broken Broken links
|
||||||
|
POST /api/knowledge/links/resolve Resolve [[link]] to entry
|
||||||
|
```
|
||||||
|
|
||||||
|
## Link Processing
|
||||||
|
|
||||||
|
### Wiki-Link Syntax
|
||||||
|
|
||||||
|
The module supports Obsidian-compatible wiki-link syntax:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
Basic link: [[entry-slug]]
|
||||||
|
Display text: [[entry-slug|Custom Display Text]]
|
||||||
|
Header link: [[entry-slug#section-header]]
|
||||||
|
Block link: [[entry-slug#^block-id]]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Link Resolution Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐
|
||||||
|
│ Entry Content │
|
||||||
|
│ "See [[design]] │
|
||||||
|
│ for details" │
|
||||||
|
└────────┬────────┘
|
||||||
|
│ Parse
|
||||||
|
▼
|
||||||
|
┌─────────────────┐
|
||||||
|
│ Extract Links │
|
||||||
|
│ [[design]] │
|
||||||
|
└────────┬────────┘
|
||||||
|
│ Resolve
|
||||||
|
▼
|
||||||
|
┌─────────────────┐
|
||||||
|
│ Find Target │
|
||||||
|
│ slug: "design" │
|
||||||
|
│ OR title match │
|
||||||
|
│ OR fuzzy match │
|
||||||
|
└────────┬────────┘
|
||||||
|
│
|
||||||
|
┌────┴────┐
|
||||||
|
▼ ▼
|
||||||
|
┌───────┐ ┌───────────┐
|
||||||
|
│ Found │ │ Not Found │
|
||||||
|
│ │ │ (broken) │
|
||||||
|
└───┬───┘ └─────┬─────┘
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌───────────────────────┐
|
||||||
|
│ Create/Update Link │
|
||||||
|
│ Record in entry_links │
|
||||||
|
│ Mark broken if needed │
|
||||||
|
└───────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Automatic Link Detection
|
||||||
|
|
||||||
|
On entry save:
|
||||||
|
1. Parse content for `[[...]]` patterns
|
||||||
|
2. Resolve each link to target entry
|
||||||
|
3. Update `KnowledgeLink` records
|
||||||
|
4. Flag broken links for UI warning
|
||||||
|
|
||||||
|
## Search Implementation
|
||||||
|
|
||||||
|
### Full-Text Search (PostgreSQL)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Create search index
|
||||||
|
ALTER TABLE knowledge_entries
|
||||||
|
ADD COLUMN search_vector tsvector
|
||||||
|
GENERATED ALWAYS AS (
|
||||||
|
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
|
||||||
|
setweight(to_tsvector('english', coalesce(summary, '')), 'B') ||
|
||||||
|
setweight(to_tsvector('english', coalesce(content, '')), 'C')
|
||||||
|
) STORED;
|
||||||
|
|
||||||
|
CREATE INDEX idx_knowledge_search ON knowledge_entries USING GIN(search_vector);
|
||||||
|
|
||||||
|
-- Search query
|
||||||
|
SELECT id, slug, title,
|
||||||
|
ts_rank(search_vector, query) as rank,
|
||||||
|
ts_headline('english', content, query) as snippet
|
||||||
|
FROM knowledge_entries, plainto_tsquery('english', $1) query
|
||||||
|
WHERE search_vector @@ query
|
||||||
|
AND workspace_id = $2
|
||||||
|
ORDER BY rank DESC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Semantic Search (pgvector)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Semantic search query
|
||||||
|
SELECT e.id, e.slug, e.title, e.summary,
|
||||||
|
1 - (emb.embedding <=> $1::vector) as similarity
|
||||||
|
FROM knowledge_entries e
|
||||||
|
JOIN knowledge_embeddings emb ON e.id = emb.entry_id
|
||||||
|
WHERE e.workspace_id = $2
|
||||||
|
AND 1 - (emb.embedding <=> $1::vector) > 0.7 -- similarity threshold
|
||||||
|
ORDER BY emb.embedding <=> $1::vector
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Embedding Generation
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function generateEmbedding(entry: KnowledgeEntry): Promise<number[]> {
|
||||||
|
const text = `${entry.title}\n\n${entry.summary || ''}\n\n${entry.content}`;
|
||||||
|
|
||||||
|
// Use OpenAI or local model
|
||||||
|
const response = await openai.embeddings.create({
|
||||||
|
model: 'text-embedding-ada-002',
|
||||||
|
input: text.slice(0, 8000), // Token limit
|
||||||
|
});
|
||||||
|
|
||||||
|
return response.data[0].embedding;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Graph Visualization
|
||||||
|
|
||||||
|
### Data Structure
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface KnowledgeGraph {
|
||||||
|
nodes: GraphNode[];
|
||||||
|
edges: GraphEdge[];
|
||||||
|
stats: GraphStats;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface GraphNode {
|
||||||
|
id: string;
|
||||||
|
slug: string;
|
||||||
|
title: string;
|
||||||
|
type: 'entry' | 'tag' | 'external';
|
||||||
|
status: EntryStatus;
|
||||||
|
linkCount: number; // in + out
|
||||||
|
tags: string[];
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface GraphEdge {
|
||||||
|
id: string;
|
||||||
|
source: string; // node id
|
||||||
|
target: string; // node id
|
||||||
|
type: 'link' | 'tag';
|
||||||
|
label?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface GraphStats {
|
||||||
|
nodeCount: number;
|
||||||
|
edgeCount: number;
|
||||||
|
orphanCount: number; // entries with no links
|
||||||
|
brokenLinkCount: number;
|
||||||
|
avgConnections: number;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph Query
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Get full graph for workspace
|
||||||
|
WITH nodes AS (
|
||||||
|
SELECT
|
||||||
|
id, slug, title, 'entry' as type, status,
|
||||||
|
(SELECT COUNT(*) FROM knowledge_links WHERE source_id = e.id OR target_id = e.id) as link_count,
|
||||||
|
updated_at
|
||||||
|
FROM knowledge_entries e
|
||||||
|
WHERE workspace_id = $1 AND status != 'ARCHIVED'
|
||||||
|
),
|
||||||
|
edges AS (
|
||||||
|
SELECT
|
||||||
|
l.id, l.source_id as source, l.target_id as target, 'link' as type, l.link_text as label
|
||||||
|
FROM knowledge_links l
|
||||||
|
JOIN knowledge_entries e ON l.source_id = e.id
|
||||||
|
WHERE e.workspace_id = $1
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
json_build_object(
|
||||||
|
'nodes', (SELECT json_agg(nodes) FROM nodes),
|
||||||
|
'edges', (SELECT json_agg(edges) FROM edges)
|
||||||
|
) as graph;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontend Rendering
|
||||||
|
|
||||||
|
Use D3.js force-directed graph or Cytoscape.js:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Graph component configuration
|
||||||
|
const graphConfig = {
|
||||||
|
layout: 'force-directed',
|
||||||
|
physics: {
|
||||||
|
repulsion: 100,
|
||||||
|
springLength: 150,
|
||||||
|
springStrength: 0.05,
|
||||||
|
},
|
||||||
|
nodeSize: (node) => Math.sqrt(node.linkCount) * 10 + 20,
|
||||||
|
nodeColor: (node) => {
|
||||||
|
switch (node.status) {
|
||||||
|
case 'PUBLISHED': return '#22c55e';
|
||||||
|
case 'DRAFT': return '#f59e0b';
|
||||||
|
case 'ARCHIVED': return '#6b7280';
|
||||||
|
}
|
||||||
|
},
|
||||||
|
edgeStyle: {
|
||||||
|
color: '#94a3b8',
|
||||||
|
width: 1,
|
||||||
|
arrows: 'to',
|
||||||
|
},
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
## Caching Strategy
|
||||||
|
|
||||||
|
### Valkey Key Patterns
|
||||||
|
|
||||||
|
```
|
||||||
|
knowledge:{workspaceId}:entry:{slug} Entry cache (JSON)
|
||||||
|
knowledge:{workspaceId}:entry:{slug}:html Rendered HTML cache
|
||||||
|
knowledge:{workspaceId}:graph Full graph cache
|
||||||
|
knowledge:{workspaceId}:graph:{slug} Subgraph cache
|
||||||
|
knowledge:{workspaceId}:search:{hash} Search result cache
|
||||||
|
knowledge:{workspaceId}:tags Tag list cache
|
||||||
|
knowledge:{workspaceId}:recent Recent entries list
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cache Invalidation
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function invalidateEntryCache(workspaceId: string, slug: string) {
|
||||||
|
const keys = [
|
||||||
|
`knowledge:${workspaceId}:entry:${slug}`,
|
||||||
|
`knowledge:${workspaceId}:entry:${slug}:html`,
|
||||||
|
`knowledge:${workspaceId}:graph`, // Full graph affected
|
||||||
|
`knowledge:${workspaceId}:graph:${slug}`,
|
||||||
|
`knowledge:${workspaceId}:recent`,
|
||||||
|
];
|
||||||
|
|
||||||
|
// Also invalidate subgraphs for linked entries
|
||||||
|
const linkedSlugs = await getLinkedEntrySlugs(workspaceId, slug);
|
||||||
|
for (const linked of linkedSlugs) {
|
||||||
|
keys.push(`knowledge:${workspaceId}:graph:${linked}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
await valkey.del(...keys);
|
||||||
|
|
||||||
|
// Invalidate search caches (pattern delete)
|
||||||
|
const searchKeys = await valkey.keys(`knowledge:${workspaceId}:search:*`);
|
||||||
|
if (searchKeys.length) await valkey.del(...searchKeys);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## UI Components
|
||||||
|
|
||||||
|
### Entry Editor
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────────────────────────┐
|
||||||
|
│ [📄] Agent Orchestration Design [Save] [···]│
|
||||||
|
├────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Status: [Published ▼] Tags: [architecture] [agents] [+] │
|
||||||
|
├────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ # Problem Statement │
|
||||||
|
│ │
|
||||||
|
│ Development teams and AI agents working on complex projects │
|
||||||
|
│ need a way to [[capture-decisions|capture decisions]]... │
|
||||||
|
│ │
|
||||||
|
│ See also: [[task-queues]] and [[valkey-patterns]] │
|
||||||
|
│ │
|
||||||
|
│ ─────────────────────────────────────────────────────────────│
|
||||||
|
│ Backlinks (3): │
|
||||||
|
│ • [[mosaic-roadmap]] - "...implements agent orchestration..." │
|
||||||
|
│ • [[design-index]] - "Core designs: [[agent-orchestration]]" │
|
||||||
|
│ • [[jarvis-memory]] - "Created orchestration design..." │
|
||||||
|
│ │
|
||||||
|
└────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph View
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Knowledge Graph [Filter ▼] [Layout ▼] │
|
||||||
|
├────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌─────────┐ │
|
||||||
|
│ │ valkey │ │
|
||||||
|
│ │patterns │ │
|
||||||
|
│ └────┬────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌────────────┼────────────┐ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ ▼ ▼ ▼ │
|
||||||
|
│ ┌──────┐ ┌────────┐ ┌────────┐ │
|
||||||
|
│ │cache │ │ task │ │ agent │◄─────┐ │
|
||||||
|
│ │layer │ │ queues │ │ orch │ │ │
|
||||||
|
│ └──────┘ └────────┘ └───┬────┘ │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ ▼ │ │
|
||||||
|
│ ┌────────┐ ┌────┴───┐ │
|
||||||
|
│ │recovery│ │ mosaic │ │
|
||||||
|
│ │patterns│ │roadmap │ │
|
||||||
|
│ └────────┘ └────────┘ │
|
||||||
|
│ │
|
||||||
|
│ 🟢 Published (6) 🟡 Draft (2) ⚪ Orphan (0) │
|
||||||
|
└────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Search Results
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────────────────────────┐
|
||||||
|
│ 🔍 [agent recovery ] [Search]│
|
||||||
|
├────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ 📄 Agent Orchestration - Recovery Patterns │
|
||||||
|
│ ...automatic **recovery** when an **agent** fails or the... │
|
||||||
|
│ Tags: architecture, agents • Updated 2 hours ago │
|
||||||
|
│ │
|
||||||
|
│ 📄 Agent Health Monitoring │
|
||||||
|
│ ...heartbeat monitoring enables **recovery** of stale... │
|
||||||
|
│ Tags: agents, monitoring • Updated 1 day ago │
|
||||||
|
│ │
|
||||||
|
│ 📄 Task Queue Design │
|
||||||
|
│ ...retry logic with exponential backoff for **agent**... │
|
||||||
|
│ Tags: architecture, queues • Updated 3 days ago │
|
||||||
|
│ │
|
||||||
|
│ ───────────────────────────────────────────────────────────── │
|
||||||
|
│ Also try: Semantic search for conceptually related entries │
|
||||||
|
│ │
|
||||||
|
└────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
### Phase 1: Foundation (Week 1-2)
|
||||||
|
|
||||||
|
**Goal:** Basic CRUD + storage working
|
||||||
|
|
||||||
|
- [ ] Database schema + migrations
|
||||||
|
- [ ] Entry CRUD API endpoints
|
||||||
|
- [ ] Basic markdown rendering
|
||||||
|
- [ ] Tag management
|
||||||
|
- [ ] Entry list/detail pages
|
||||||
|
|
||||||
|
**Deliverables:**
|
||||||
|
- Can create, edit, view, delete entries
|
||||||
|
- Tags work
|
||||||
|
- Basic search (title/slug match)
|
||||||
|
|
||||||
|
### Phase 2: Linking (Week 2-3)
|
||||||
|
|
||||||
|
**Goal:** Wiki-link functionality
|
||||||
|
|
||||||
|
- [ ] Link parser (`[[...]]` syntax)
|
||||||
|
- [ ] Link resolution logic
|
||||||
|
- [ ] Broken link detection
|
||||||
|
- [ ] Backlinks display
|
||||||
|
- [ ] Link autocomplete in editor
|
||||||
|
|
||||||
|
**Deliverables:**
|
||||||
|
- Links between entries work
|
||||||
|
- Backlinks show on entry pages
|
||||||
|
- Editor suggests links as you type
|
||||||
|
|
||||||
|
### Phase 3: Search (Week 3-4)
|
||||||
|
|
||||||
|
**Goal:** Full-text + semantic search
|
||||||
|
|
||||||
|
- [ ] PostgreSQL full-text search setup
|
||||||
|
- [ ] Search API endpoint
|
||||||
|
- [ ] Search UI with highlighting
|
||||||
|
- [ ] pgvector extension setup
|
||||||
|
- [ ] Embedding generation pipeline
|
||||||
|
- [ ] Semantic search API
|
||||||
|
|
||||||
|
**Deliverables:**
|
||||||
|
- Fast full-text search
|
||||||
|
- Semantic search for "fuzzy" queries
|
||||||
|
- Search results with snippets
|
||||||
|
|
||||||
|
### Phase 4: Graph (Week 4-5)
|
||||||
|
|
||||||
|
**Goal:** Visual knowledge graph
|
||||||
|
|
||||||
|
- [ ] Graph data API
|
||||||
|
- [ ] D3.js/Cytoscape integration
|
||||||
|
- [ ] Interactive graph view
|
||||||
|
- [ ] Subgraph (entry-centered) view
|
||||||
|
- [ ] Graph statistics
|
||||||
|
|
||||||
|
**Deliverables:**
|
||||||
|
- Can view full knowledge graph
|
||||||
|
- Can explore from any entry
|
||||||
|
- Visual indicators for status/orphans
|
||||||
|
|
||||||
|
### Phase 5: Polish (Week 5-6)
|
||||||
|
|
||||||
|
**Goal:** Production-ready
|
||||||
|
|
||||||
|
- [ ] Version history UI
|
||||||
|
- [ ] Diff view between versions
|
||||||
|
- [ ] Import from markdown files
|
||||||
|
- [ ] Export functionality
|
||||||
|
- [ ] Performance optimization
|
||||||
|
- [ ] Caching implementation
|
||||||
|
- [ ] Documentation
|
||||||
|
|
||||||
|
**Deliverables:**
|
||||||
|
- Version history works
|
||||||
|
- Can import existing docs
|
||||||
|
- Performance is acceptable
|
||||||
|
- Module is documented
|
||||||
|
|
||||||
|
## Integration Points
|
||||||
|
|
||||||
|
### Agent Access
|
||||||
|
|
||||||
|
The Knowledge module should be accessible to agents via API:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Agent tool for knowledge access
|
||||||
|
interface KnowledgeTools {
|
||||||
|
// Search
|
||||||
|
searchKnowledge(query: string): Promise<SearchResult[]>;
|
||||||
|
semanticSearch(query: string): Promise<SearchResult[]>;
|
||||||
|
|
||||||
|
// CRUD
|
||||||
|
getEntry(slug: string): Promise<KnowledgeEntry>;
|
||||||
|
createEntry(data: CreateEntryInput): Promise<KnowledgeEntry>;
|
||||||
|
updateEntry(slug: string, data: UpdateEntryInput): Promise<KnowledgeEntry>;
|
||||||
|
|
||||||
|
// Graph
|
||||||
|
getRelatedEntries(slug: string): Promise<KnowledgeEntry[]>;
|
||||||
|
getBacklinks(slug: string): Promise<KnowledgeEntry[]>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Clawdbot Integration
|
||||||
|
|
||||||
|
For Clawdbot specifically, the Knowledge module could:
|
||||||
|
|
||||||
|
1. Sync with `memory/*.md` files
|
||||||
|
2. Provide semantic search for `memory_search` tool
|
||||||
|
3. Generate embeddings for memory entries
|
||||||
|
4. Visualize agent memory as a knowledge graph
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
| Metric | Target | Measurement |
|
||||||
|
|--------|--------|-------------|
|
||||||
|
| Entry creation time | < 200ms | API response time |
|
||||||
|
| Search latency (full-text) | < 100ms | p95 response time |
|
||||||
|
| Search latency (semantic) | < 300ms | p95 response time |
|
||||||
|
| Graph render (100 nodes) | < 200ms | Client-side time |
|
||||||
|
| Graph render (1000 nodes) | < 1s | Client-side time |
|
||||||
|
| Adoption | 50+ entries/workspace | After 1 month |
|
||||||
|
| Link density | > 2 links/entry avg | Graph statistics |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Embedding model** — Use OpenAI embeddings or self-hosted? (Cost vs privacy)
|
||||||
|
2. **Real-time collab** — Do we need multiplayer editing? (CRDT complexity)
|
||||||
|
3. **Permissions** — Entry-level permissions or workspace-level only?
|
||||||
|
4. **Templates** — Support entry templates (ADR, design doc, etc.)?
|
||||||
|
5. **Attachments** — Allow images/files in entries?
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [Obsidian](https://obsidian.md/) — Wiki-link syntax inspiration
|
||||||
|
- [Roam Research](https://roamresearch.com/) — Block-level linking
|
||||||
|
- [pgvector](https://github.com/pgvector/pgvector) — PostgreSQL vector extension
|
||||||
|
- [Mosaic Agent Orchestration](./agent-orchestration.md) — Related design
|
||||||
Reference in New Issue
Block a user