feat: add semantic search with pgvector (closes #68, #69, #70)
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed

Issues resolved:
- #68: pgvector Setup
  * Added pgvector vector index migration for knowledge_embeddings
  * Vector index uses HNSW algorithm with cosine distance
  * Optimized for 1536-dimension OpenAI embeddings

- #69: Embedding Generation Pipeline
  * Created EmbeddingService with OpenAI integration
  * Automatic embedding generation on entry create/update
  * Batch processing endpoint for existing entries
  * Async generation to avoid blocking API responses
  * Content preparation with title weighting

- #70: Semantic Search API
  * POST /api/knowledge/search/semantic - pure vector search
  * POST /api/knowledge/search/hybrid - RRF combined search
  * POST /api/knowledge/embeddings/batch - batch generation
  * Comprehensive test coverage
  * Full documentation in docs/SEMANTIC_SEARCH.md

Technical details:
- Uses OpenAI text-embedding-3-small model (1536 dims)
- HNSW index for O(log n) similarity search
- Reciprocal Rank Fusion for hybrid search
- Graceful degradation when OpenAI not configured
- Async embedding generation for performance

Configuration:
- Added OPENAI_API_KEY to .env.example
- Optional feature - disabled if API key not set
- Falls back to keyword search in hybrid mode
This commit is contained in:
Jason Woltje
2026-01-30 00:24:41 -06:00
parent 22cd68811d
commit 3ec2059470
14 changed files with 1408 additions and 5 deletions

View File

@@ -2,7 +2,11 @@ import { Module } from "@nestjs/common";
import { PrismaModule } from "../prisma/prisma.module";
import { AuthModule } from "../auth/auth.module";
import { KnowledgeService } from "./knowledge.service";
import { KnowledgeController, KnowledgeCacheController } from "./knowledge.controller";
import {
KnowledgeController,
KnowledgeCacheController,
KnowledgeEmbeddingsController,
} from "./knowledge.controller";
import { SearchController } from "./search.controller";
import { KnowledgeStatsController } from "./stats.controller";
import {
@@ -12,6 +16,7 @@ import {
GraphService,
StatsService,
KnowledgeCacheService,
EmbeddingService,
} from "./services";
@Module({
@@ -19,6 +24,7 @@ import {
controllers: [
KnowledgeController,
KnowledgeCacheController,
KnowledgeEmbeddingsController,
SearchController,
KnowledgeStatsController,
],
@@ -30,7 +36,8 @@ import {
GraphService,
StatsService,
KnowledgeCacheService,
EmbeddingService,
],
exports: [KnowledgeService, LinkResolutionService, SearchService],
exports: [KnowledgeService, LinkResolutionService, SearchService, EmbeddingService],
})
export class KnowledgeModule {}