Keep both Mosaic Telemetry section (from develop) and Matrix Dev
Environment section (from feature branch) in .env.example.
Regenerate pnpm-lock.yaml with both dependency trees merged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- llm-cost-table.ts: Add undefined guard for MODEL_COSTS lookup
- llm-telemetry-tracker.service.ts: Allow undefined in callingContext
for exactOptionalPropertyTypes compatibility
Refs #371
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create PredictionService for pre-task cost/token estimates
- Refresh common predictions on startup
- Integrate predictions into LLM telemetry tracker
- Add GET /api/telemetry/estimate endpoint
- Graceful degradation when no prediction data available
- Add unit tests for prediction service
Refs #373
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create LlmTelemetryTrackerService for non-blocking event emission
- Normalize token usage across Anthropic, OpenAI, Ollama providers
- Add cost table with per-token pricing in microdollars
- Instrument chat, chatStream, and embed methods
- Infer task type from calling context
- Aggregate streaming tokens after stream ends with fallback estimation
- Add 69 unit tests for tracker service, cost table, and LLM service
Refs #371
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements comprehensive LLM usage tracking with analytics endpoints.
Implementation:
- Added LlmUsageLog model to Prisma schema
- Created llm-usage module with service, controller, and DTOs
- Added tracking for token usage, costs, and durations
- Implemented analytics aggregation by provider, model, and task type
- Added filtering by workspace, provider, model, user, and date range
Testing:
- 20 unit tests with 90.8% coverage (exceeds 85% requirement)
- Tests for service and controller with full error handling
- Tests use Vitest following project conventions
API Endpoints:
- GET /api/llm-usage/analytics - Aggregated usage analytics
- GET /api/llm-usage/by-workspace/:workspaceId - Workspace usage logs
- GET /api/llm-usage/by-workspace/:workspaceId/provider/:provider - Provider logs
- GET /api/llm-usage/by-workspace/:workspaceId/model/:model - Model logs
Database:
- LlmUsageLog table with indexes for efficient queries
- Relations to User, Workspace, and LlmProviderInstance
- Ready for migration with: pnpm prisma migrate dev
Refs #309
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implement REST API endpoints for managing LLM provider instances.
Changes:
- Created DTOs for provider CRUD operations (CreateLlmProviderDto, UpdateLlmProviderDto, LlmProviderResponseDto)
- Implemented LlmProviderAdminController with full CRUD endpoints:
- GET /llm/admin/providers - List all providers
- GET /llm/admin/providers/:id - Get provider details
- POST /llm/admin/providers - Create new provider
- PATCH /llm/admin/providers/:id - Update provider
- DELETE /llm/admin/providers/:id - Delete provider
- POST /llm/admin/providers/:id/test - Test connection
- POST /llm/admin/reload - Reload from database
- Updated llm-manager.service.ts to support OpenAI and Claude providers
- Added comprehensive test suite with 97.95% coverage
- Proper validation, error handling, and type safety
All tests pass. Pre-commit hooks pass.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement Anthropic Claude provider for Claude Opus, Sonnet, and Haiku models.
Implementation details:
- Created ClaudeProvider class implementing LlmProviderInterface
- Added @anthropic-ai/sdk npm package integration
- Implemented chat completion with streaming support
- Claude-specific message format (system prompt separate from messages)
- Static model list (Claude API doesn't provide list models endpoint)
- Embeddings throw error as Claude doesn't support native embeddings
- Added OpenTelemetry tracing with @TraceLlmCall decorator
- 100% statement, function, and line coverage (79% branch coverage)
Tests:
- Created comprehensive test suite with 20 tests
- All tests follow TDD pattern (written before implementation)
- Tests cover initialization, health checks, chat, streaming, and error handling
- Mocked Anthropic SDK client for isolated unit testing
Quality checks:
- All tests pass (1131 total tests across project)
- ESLint passes with no errors
- TypeScript type checking passes
- Follows existing code patterns from OpenAI and Ollama providers
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement OpenAI provider for GPT-4, GPT-3.5, and other OpenAI models.
Implementation includes:
- OpenAI SDK integration with API key authentication
- Chat completion with streaming support
- Embeddings generation
- Health checks and model listing
- OpenTelemetry tracing
- Comprehensive test suite with 97% coverage
Follows TDD methodology:
- Written tests first (RED phase)
- Implemented minimal code to pass tests (GREEN phase)
- Code passes typecheck, linter, and all quality gates
Test coverage: 97.18% statements, 97.05% lines
All 22 tests passing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Refactor LlmService to delegate to LlmManagerService instead of using
Ollama directly. This enables multiple provider support and user-specific
provider configuration.
Changes:
- Remove direct Ollama client from LlmService
- Delegate all LLM operations to provider via LlmManagerService
- Update health status to use provider-agnostic interface
- Add PrismaModule to LlmModule for manager service
- Maintain backward compatibility with existing API
- Achieve 89.74% test coverage
Fixes#127
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implemented centralized service for managing multiple LLM provider instances.
Architecture:
- LlmManagerService manages provider lifecycle and selection
- Loads provider instances from Prisma database on startup
- Maintains in-memory registry of active providers
- Factory pattern for provider instantiation
Core Features:
- Database integration via PrismaService
- Provider initialization on module startup (OnModuleInit)
- Get provider by ID
- Get all active providers
- Get system default provider
- Get user-specific provider with fallback to system default
- Health check all registered providers
- Dynamic registration/unregistration (hot reload)
- Reload from database without restart
Provider Selection Logic:
- User-level providers: userId matches, is enabled
- System-level providers: userId is NULL, is enabled
- Fallback: system default if no user provider found
- Graceful error handling with detailed logging
Integration:
- Added to LlmModule providers and exports
- Uses PrismaService for database queries
- Factory creates OllamaProvider from config
- Extensible for future providers (Claude, OpenAI)
Testing:
- 31 comprehensive unit tests
- 93.05% code coverage (exceeds 85% requirement)
- All error scenarios covered
- Proper mocking of dependencies
Quality Gates:
- ✅ All 31 tests passing
- ✅ 93.05% coverage
- ✅ Linting clean
- ✅ Type checking passed
- ✅ Code review approved
Fixes#126
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implemented first concrete LLM provider following the provider interface pattern.
Implementation:
- OllamaProvider class implementing LlmProviderInterface
- All required methods: initialize(), checkHealth(), listModels(), chat(), chatStream(), embed(), getConfig()
- OllamaProviderConfig extending LlmProviderConfig
- Proper error handling with NestJS Logger
- Configuration immutability protection
Features:
- System prompt injection support
- Temperature and max tokens configuration
- Embedding with truncation control (defaults to enabled)
- Streaming and non-streaming chat completions
- Health check with model listing
Testing:
- 21 comprehensive test cases (TDD approach)
- 100% statement, function, and line coverage
- 86.36% branch coverage (exceeds 85% requirement)
- All error scenarios tested
- Mock-based unit tests
Code Review Fixes:
- Fixed truncate logic to match original LlmService behavior (defaults to true)
- Added test for system prompt deduplication
- Increased branch coverage from 77% to 86%
Quality Gates:
- ✅ All 21 tests passing
- ✅ Linting clean
- ✅ Type checking passed
- ✅ Code review approved
Fixes#123
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add Ollama client library (ollama npm package)
- Create LlmService for chat completion and embeddings
- Support streaming responses via Server-Sent Events
- Add configuration via env vars (OLLAMA_HOST, OLLAMA_TIMEOUT)
- Create endpoints: GET /llm/health, GET /llm/models, POST /llm/chat, POST /llm/embed
- Replace old OllamaModule with new LlmModule
- Add comprehensive tests with >85% coverage
Closes#21
