feat(#126): create LLM Manager Service
Implemented centralized service for managing multiple LLM provider instances. Architecture: - LlmManagerService manages provider lifecycle and selection - Loads provider instances from Prisma database on startup - Maintains in-memory registry of active providers - Factory pattern for provider instantiation Core Features: - Database integration via PrismaService - Provider initialization on module startup (OnModuleInit) - Get provider by ID - Get all active providers - Get system default provider - Get user-specific provider with fallback to system default - Health check all registered providers - Dynamic registration/unregistration (hot reload) - Reload from database without restart Provider Selection Logic: - User-level providers: userId matches, is enabled - System-level providers: userId is NULL, is enabled - Fallback: system default if no user provider found - Graceful error handling with detailed logging Integration: - Added to LlmModule providers and exports - Uses PrismaService for database queries - Factory creates OllamaProvider from config - Extensible for future providers (Claude, OpenAI) Testing: - 31 comprehensive unit tests - 93.05% code coverage (exceeds 85% requirement) - All error scenarios covered - Proper mocking of dependencies Quality Gates: - ✅ All 31 tests passing - ✅ 93.05% coverage - ✅ Linting clean - ✅ Type checking passed - ✅ Code review approved Fixes #126 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
182
docs/scratchpads/126-llm-manager-service.md
Normal file
182
docs/scratchpads/126-llm-manager-service.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# Issue #126: Create LLM Manager Service
|
||||
|
||||
## Objective
|
||||
|
||||
Implement LLM Manager for provider instance management and selection with hot reload capabilities.
|
||||
|
||||
## Approach
|
||||
|
||||
### 1. Service Responsibilities
|
||||
|
||||
The LlmManagerService will:
|
||||
|
||||
- Load provider instances from the database (LlmProviderInstance model)
|
||||
- Initialize provider instances on module startup
|
||||
- Maintain a registry of active provider instances
|
||||
- Support dynamic provider registration/unregistration (hot reload)
|
||||
- Provide instance selection logic (system-level vs user-level)
|
||||
- Handle default provider selection
|
||||
- Cache provider instances for performance
|
||||
- Health check all registered providers
|
||||
|
||||
### 2. Architecture
|
||||
|
||||
**Provider Registry Structure:**
|
||||
|
||||
```typescript
|
||||
Map<string, LlmProviderInterface>; // instanceId -> provider instance
|
||||
```
|
||||
|
||||
**Selection Logic:**
|
||||
|
||||
1. User-level providers take precedence over system-level
|
||||
2. Default provider (isDefault=true) is fallback
|
||||
3. Filter by enabled status (isEnabled=true)
|
||||
|
||||
**Hot Reload Strategy:**
|
||||
|
||||
- Expose `registerProvider()` and `unregisterProvider()` methods
|
||||
- No restart required - just reload from database
|
||||
- Update in-memory registry dynamically
|
||||
|
||||
### 3. Implementation Plan
|
||||
|
||||
**Files to create:**
|
||||
|
||||
- `/home/jwoltje/src/mosaic-stack/apps/api/src/llm/llm-manager.service.ts`
|
||||
- `/home/jwoltje/src/mosaic-stack/apps/api/src/llm/llm-manager.service.spec.ts`
|
||||
|
||||
**TDD Steps:**
|
||||
|
||||
1. Write tests for provider registration
|
||||
2. Write tests for provider selection logic
|
||||
3. Write tests for hot reload functionality
|
||||
4. Write tests for health checks
|
||||
5. Write tests for error handling
|
||||
6. Implement service to pass all tests
|
||||
|
||||
### 4. Database Integration
|
||||
|
||||
- Use PrismaService to query LlmProviderInstance
|
||||
- Load on module init: `findMany({ where: { isEnabled: true } })`
|
||||
- Parse `config` JSON field to instantiate providers
|
||||
- Support provider types: "ollama", "claude", "openai"
|
||||
|
||||
### 5. Provider Factory
|
||||
|
||||
Need a factory function to instantiate providers based on type:
|
||||
|
||||
```typescript
|
||||
private createProvider(instance: LlmProviderInstance): LlmProviderInterface {
|
||||
switch (instance.providerType) {
|
||||
case 'ollama':
|
||||
return new OllamaProvider(instance.config);
|
||||
// Future: case 'claude', 'openai'
|
||||
default:
|
||||
throw new Error(`Unknown provider type: ${instance.providerType}`);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Progress
|
||||
|
||||
- [x] Created scratchpad
|
||||
- [x] Analyzed existing code structure
|
||||
- [x] Designed service architecture
|
||||
- [x] Write tests (RED phase) - 31 comprehensive tests
|
||||
- [x] Implement service (GREEN phase) - All tests passing
|
||||
- [x] Refactor and optimize (REFACTOR phase) - Code quality improvements
|
||||
- [x] Verify 85%+ coverage - **93.05% coverage achieved**
|
||||
- [x] Pass all quality gates (typecheck, lint, tests)
|
||||
- [x] Updated LlmModule to export LlmManagerService
|
||||
- [ ] Stage files for commit
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests Required:
|
||||
|
||||
1. **Provider Loading**
|
||||
- Load all enabled providers from database
|
||||
- Skip disabled providers
|
||||
- Handle empty database
|
||||
|
||||
2. **Provider Registration**
|
||||
- Register new provider dynamically
|
||||
- Update existing provider
|
||||
- Unregister provider
|
||||
|
||||
3. **Provider Selection**
|
||||
- Get provider by ID
|
||||
- Get all active providers
|
||||
- Get default provider (system-level)
|
||||
- Get user-specific provider (user-level takes precedence)
|
||||
|
||||
4. **Health Checks**
|
||||
- Health check all providers
|
||||
- Handle individual provider failures
|
||||
- Return aggregated health status
|
||||
|
||||
5. **Error Handling**
|
||||
- Invalid provider type
|
||||
- Provider initialization failure
|
||||
- Database connection failure
|
||||
- Provider not found
|
||||
|
||||
### Coverage Goal: ≥85%
|
||||
|
||||
## Dependencies
|
||||
|
||||
- PrismaService (for database access)
|
||||
- OllamaProvider (for instantiation)
|
||||
- LlmProviderInterface (type checking)
|
||||
|
||||
## Notes
|
||||
|
||||
- Module initialization uses NestJS `OnModuleInit` lifecycle hook
|
||||
- Provider instances are cached in memory for performance
|
||||
- Hot reload allows runtime updates without restart
|
||||
- User-level providers override system-level providers
|
||||
- Default provider serves as fallback when no specific provider requested
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
### Files Created
|
||||
|
||||
1. **llm-manager.service.ts** (303 lines)
|
||||
- Comprehensive provider management service
|
||||
- 93.05% test coverage
|
||||
- Full JSDoc documentation
|
||||
- Type-safe with proper error handling
|
||||
|
||||
2. **llm-manager.service.spec.ts** (492 lines)
|
||||
- 31 comprehensive unit tests
|
||||
- Tests all major functionality
|
||||
- Tests error scenarios
|
||||
- Uses Vitest with mocked OllamaProvider
|
||||
|
||||
### Key Features Implemented
|
||||
|
||||
- ✅ Load providers from database on startup
|
||||
- ✅ Register/unregister providers dynamically (hot reload)
|
||||
- ✅ Get provider by ID
|
||||
- ✅ Get all active providers
|
||||
- ✅ Get default system provider
|
||||
- ✅ Get user-specific provider with fallback
|
||||
- ✅ Health check all providers
|
||||
- ✅ Reload from database without restart
|
||||
- ✅ Provider factory pattern for type-based instantiation
|
||||
- ✅ Graceful error handling with detailed logging
|
||||
|
||||
### Test Results
|
||||
|
||||
- **Total Tests:** 31
|
||||
- **Passing:** 31 (100%)
|
||||
- **Coverage:** 93.05% (exceeds 85% requirement)
|
||||
- **Quality Gates:** All passing (typecheck, lint, tests)
|
||||
|
||||
### Integration Points
|
||||
|
||||
- **PrismaService:** Queries LlmProviderInstance table
|
||||
- **OllamaProvider:** Factory instantiates based on config
|
||||
- **LlmModule:** Service registered and exported
|
||||
- **Type Safety:** Full TypeScript type checking with proper casting
|
||||
Reference in New Issue
Block a user