stack/docs/scratchpads/126-llm-manager-service.md

Issue #126: Create LLM Manager Service

Objective

Implement LLM Manager for provider instance management and selection with hot reload capabilities.

Approach

1. Service Responsibilities

The LlmManagerService will:

• Load provider instances from the database (LlmProviderInstance model)
• Initialize provider instances on module startup
• Maintain a registry of active provider instances
• Support dynamic provider registration/unregistration (hot reload)
• Provide instance selection logic (system-level vs user-level)
• Handle default provider selection
• Cache provider instances for performance
• Health check all registered providers

2. Architecture

Provider Registry Structure:

Map<string, LlmProviderInterface>; // instanceId -> provider instance

Selection Logic:

1. User-level providers take precedence over system-level
2. Default provider (isDefault=true) is fallback
3. Filter by enabled status (isEnabled=true)

Hot Reload Strategy:

• Expose registerProvider() and unregisterProvider() methods
• No restart required - just reload from database
• Update in-memory registry dynamically

3. Implementation Plan

Files to create:

• /home/jwoltje/src/mosaic-stack/apps/api/src/llm/llm-manager.service.ts
• /home/jwoltje/src/mosaic-stack/apps/api/src/llm/llm-manager.service.spec.ts

TDD Steps:

1. Write tests for provider registration
2. Write tests for provider selection logic
3. Write tests for hot reload functionality
4. Write tests for health checks
5. Write tests for error handling
6. Implement service to pass all tests

4. Database Integration

• Use PrismaService to query LlmProviderInstance
• Load on module init: findMany({ where: { isEnabled: true } })
• Parse config JSON field to instantiate providers
• Support provider types: "ollama", "claude", "openai"

5. Provider Factory

Need a factory function to instantiate providers based on type:

private createProvider(instance: LlmProviderInstance): LlmProviderInterface {
  switch (instance.providerType) {
    case 'ollama':
      return new OllamaProvider(instance.config);
    // Future: case 'claude', 'openai'
    default:
      throw new Error(`Unknown provider type: ${instance.providerType}`);
  }
}

Progress

• Created scratchpad
• Analyzed existing code structure
• Designed service architecture
• Write tests (RED phase) - 31 comprehensive tests
• Implement service (GREEN phase) - All tests passing
• Refactor and optimize (REFACTOR phase) - Code quality improvements
• Verify 85%+ coverage - 93.05% coverage achieved
• Pass all quality gates (typecheck, lint, tests)
• Updated LlmModule to export LlmManagerService
• Stage files for commit

Testing Strategy

Unit Tests Required:

1. Provider Loading

   • Load all enabled providers from database
   • Skip disabled providers
   • Handle empty database

2. Provider Registration

   • Register new provider dynamically
   • Update existing provider
   • Unregister provider

3. Provider Selection

   • Get provider by ID
   • Get all active providers
   • Get default provider (system-level)
   • Get user-specific provider (user-level takes precedence)

4. Health Checks

   • Health check all providers
   • Handle individual provider failures
   • Return aggregated health status

5. Error Handling

   • Invalid provider type
   • Provider initialization failure
   • Database connection failure
   • Provider not found

Coverage Goal: ≥85%

Dependencies

• PrismaService (for database access)
• OllamaProvider (for instantiation)
• LlmProviderInterface (type checking)

Notes

• Module initialization uses NestJS OnModuleInit lifecycle hook
• Provider instances are cached in memory for performance
• Hot reload allows runtime updates without restart
• User-level providers override system-level providers
• Default provider serves as fallback when no specific provider requested

Implementation Summary

Files Created

1. llm-manager.service.ts (303 lines)

   • Comprehensive provider management service
   • 93.05% test coverage
   • Full JSDoc documentation
   • Type-safe with proper error handling

2. llm-manager.service.spec.ts (492 lines)

   • 31 comprehensive unit tests
   • Tests all major functionality
   • Tests error scenarios
   • Uses Vitest with mocked OllamaProvider

Key Features Implemented

• ✅ Load providers from database on startup
• ✅ Register/unregister providers dynamically (hot reload)
• ✅ Get provider by ID
• ✅ Get all active providers
• ✅ Get default system provider
• ✅ Get user-specific provider with fallback
• ✅ Health check all providers
• ✅ Reload from database without restart
• ✅ Provider factory pattern for type-based instantiation
• ✅ Graceful error handling with detailed logging

Test Results

• Total Tests: 31
• Passing: 31 (100%)
• Coverage: 93.05% (exceeds 85% requirement)
• Quality Gates: All passing (typecheck, lint, tests)

Integration Points

• PrismaService: Queries LlmProviderInstance table
• OllamaProvider: Factory instantiates based on config
• LlmModule: Service registered and exported
• Type Safety: Full TypeScript type checking with proper casting