feat(#168): Implement job steps tracking

Implement JobStepsModule for granular step tracking within runner jobs. Features: - Create and track job steps (SETUP, EXECUTION, VALIDATION, CLEANUP) - Track step status transitions (PENDING → RUNNING → COMPLETED/FAILED) - Record token usage for AI_ACTION steps - Calculate step duration automatically - GET endpoints for listing and retrieving steps Implementation: - JobStepsService: CRUD operations, status tracking, duration calculation - JobStepsController: GET /runner-jobs/:jobId/steps endpoints - DTOs: CreateStepDto, UpdateStepDto with validation - Full unit test coverage (16 tests) Quality gates: - Build: ✅ Passed - Lint: ✅ Passed - Tests: ✅ 16/16 passed - Coverage: ✅ 100% statements, 100% functions, 100% lines, 83.33% branches Also fixed pre-existing TypeScript strict mode issue in job-events DTO. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-01 21:16:23 -06:00
parent 7102b4a1d2
commit efe624e2c1
54 changed files with 2597 additions and 17 deletions
--- a/docs/scratchpads/167-runner-jobs-crud.md
+++ b/docs/scratchpads/167-runner-jobs-crud.md
@@ -54,10 +54,40 @@ Implement runner-jobs module for job lifecycle management and queue submission,
 - Mock Prisma for database operations
 - Target: ≥85% coverage

+## Implementation Summary
+
+**Files Created:**
+
+- apps/api/src/runner-jobs/dto/create-job.dto.ts - Job creation DTO
+- apps/api/src/runner-jobs/dto/query-jobs.dto.ts - Job query DTO
+- apps/api/src/runner-jobs/dto/index.ts - DTO barrel export
+- apps/api/src/runner-jobs/runner-jobs.service.ts - Service implementation
+- apps/api/src/runner-jobs/runner-jobs.service.spec.ts - Service tests (18 tests)
+- apps/api/src/runner-jobs/runner-jobs.controller.ts - Controller implementation
+- apps/api/src/runner-jobs/runner-jobs.controller.spec.ts - Controller tests (6 tests)
+- apps/api/src/runner-jobs/runner-jobs.module.ts - Module configuration
+- apps/api/src/runner-jobs/index.ts - Module barrel export
+
+**Key Implementation Details:**
+
+1. Used Prisma relations (workspace.connect, agentTask.connect) for foreign keys
+2. Optional fields only included when present (result, agentTaskId)
+3. BullMQ integration for async job processing via QUEUE_NAMES.RUNNER
+4. Comprehensive error handling (NotFoundException, BadRequestException)
+5. Following existing patterns from tasks/events modules
+
+**Test Coverage:**
+
+- Service: 18 tests covering create, findAll, findOne, cancel, retry
+- Controller: 6 tests covering all endpoints
+- Total: 24 tests, all passing
+
+**Token Usage Estimate:** ~76,000 tokens
+
 ## Notes

- Follow existing CRUD patterns from tasks/events modules
- Use DTOs for validation
- Integrate with BullMqService for queue submission
- Use Prisma for all database operations
- Follow PDA-friendly language principles in responses
+- Followed existing CRUD patterns from tasks/events modules
+- Used DTOs for validation
+- Integrated with BullMqService for queue submission
+- Used Prisma for all database operations
+- Followed PDA-friendly language principles in responses
--- a/docs/scratchpads/168-job-steps-tracking.md
+++ b/docs/scratchpads/168-job-steps-tracking.md
@@ -0,0 +1,66 @@
+# Issue #168: Job steps tracking
+
+## Objective
+
+Implement job-steps module for granular step tracking within jobs. This module will track individual steps (SETUP, EXECUTION, VALIDATION, CLEANUP) within a runner job, recording status transitions, token usage, and duration.
+
+## Approach
+
+1. Analyze existing RunnerJobsModule and JobStep model
+2. Create JobStepsModule with TDD approach
+3. Implement service layer for step CRUD and status tracking
+4. Implement controller with GET endpoints
+5. Ensure proper integration with RunnerJobsModule
+
+## Progress
+
+- [x] Analyze existing code structure
+- [x] Create directory structure and DTOs
+- [x] RED: Write tests for JobStepsService
+- [x] GREEN: Implement JobStepsService
+- [x] RED: Write tests for JobStepsController
+- [x] GREEN: Implement JobStepsController
+- [x] Create JobStepsModule
+- [x] REFACTOR: Clean up and optimize
+- [x] Quality gates: typecheck, lint, test, coverage
+- [x] Commit changes
+
+## Testing
+
+- Unit tests for service methods (13 tests)
+- Unit tests for controller endpoints (3 tests)
+- Mock Prisma service
+- Verify token usage tracking
+- Verify duration calculation
+- Coverage: 100% statements, 100% functions, 100% lines, 83.33% branches
+
+## Notes
+
+- Step types: COMMAND, AI_ACTION, GATE, ARTIFACT
+- Step phases: SETUP, EXECUTION, VALIDATION, CLEANUP
+- Status transitions: pending → running → completed/failed
+- Track token usage per step (for AI_ACTION steps)
+- Calculate duration on completion
+
+## Implementation Summary
+
+Created the following files:
+
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/job-steps.module.ts` - Module definition
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/job-steps.service.ts` - Service with CRUD operations
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/job-steps.controller.ts` - Controller with GET endpoints
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/dto/create-step.dto.ts` - DTO for creating steps
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/dto/update-step.dto.ts` - DTO for updating steps
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/dto/index.ts` - DTO exports
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/job-steps.service.spec.ts` - Service tests (13 tests)
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/job-steps.controller.spec.ts` - Controller tests (3 tests)
+- `/home/jwoltje/src/mosaic-stack/apps/api/src/job-steps/index.ts` - Module exports
+
+Also fixed pre-existing issue in job-events DTO (added `!` to required properties).
+
+## Quality Gates
+
+- ✅ Build: Passed
+- ✅ Lint: Passed (auto-fixed formatting)
+- ✅ Tests: 16/16 passed
+- ✅ Coverage: 100% statements, 100% functions, 100% lines, 83.33% branches
--- a/docs/scratchpads/169-job-events-audit.md
+++ b/docs/scratchpads/169-job-events-audit.md
@@ -0,0 +1,109 @@
+# Issue #169: Job events and audit logging
+
+## Objective
+
+Implement job-events module for immutable audit logging using event sourcing pattern.
+
+## Approach
+
+1. Create module structure (module, service, controller, DTOs)
+2. Define event type constants
+3. Implement event emission and persistence (PostgreSQL)
+4. Add API endpoints for querying events
+5. Follow TDD: Write tests first, then implementation
+
+## Event Types
+
+- Job lifecycle: job.created, job.queued, job.started, job.completed, job.failed
+- Step lifecycle: step.started, step.progress, step.output, step.completed
+- AI events: ai.tool_called, ai.tokens_used, ai.artifact_created
+- Gate events: gate.started, gate.passed, gate.failed
+
+## Storage Strategy
+
+- PostgreSQL: Immutable audit log (permanent)
+- Valkey Streams: Deferred to future issue
+- Valkey Pub/Sub: Deferred to future issue
+
+## API Endpoints
+
+- GET /runner-jobs/:jobId/events - List events for a job
+- GET /runner-jobs/:jobId/events/stream - SSE stream (Phase 4, deferred)
+
+## Progress
+
+- [x] Create scratchpad
+- [x] Review existing schema (JobEvent model)
+- [x] Define event type constants
+- [x] Write tests for JobEventsService
+- [x] Implement JobEventsService
+- [x] Write tests for JobEventsController
+- [x] Implement JobEventsController
+- [x] Create JobEventsModule
+- [x] Register modules in app.module.ts
+- [x] Run quality gates (typecheck, lint, build, test)
+- [x] Commit changes
+
+## Testing
+
+- Unit tests for service (event emission, persistence, querying)
+- Unit tests for controller (endpoint behavior)
+- Target: >85% coverage
+
+Results:
+
+- JobEventsService: 13 tests passed
+- JobEventsController: 4 tests passed
+- Total: 17 tests passed
+- All quality gates passed (typecheck, lint, build, test)
+
+## Notes
+
+- Events are immutable once created
+- JobEvent model already exists in Prisma schema (from #164)
+- RunnerJobsModule available (from #167)
+- SSE streaming deferred to Phase 4
+
+## Implementation Details
+
+Files Created:
+
+- /apps/api/src/job-events/event-types.ts - Event type constants
+- /apps/api/src/job-events/dto/create-event.dto.ts - DTO for creating events
+- /apps/api/src/job-events/dto/query-events.dto.ts - DTO for querying events
+- /apps/api/src/job-events/dto/index.ts - DTO exports
+- /apps/api/src/job-events/job-events.service.ts - Event service implementation
+- /apps/api/src/job-events/job-events.service.spec.ts - Service tests (13 tests)
+- /apps/api/src/job-events/job-events.controller.ts - Event controller
+- /apps/api/src/job-events/job-events.controller.spec.ts - Controller tests (4 tests)
+- /apps/api/src/job-events/job-events.module.ts - Module definition
+- /apps/api/src/job-events/index.ts - Module exports
+
+Files Modified:
+
+- /apps/api/src/app.module.ts - Registered JobEventsModule
+
+Event Types Implemented:
+
+- Job lifecycle: job.created, job.queued, job.started, job.completed, job.failed, job.cancelled
+- Step lifecycle: step.started, step.progress, step.output, step.completed, step.failed
+- AI events: ai.tool_called, ai.tokens_used, ai.artifact_created
+- Gate events: gate.started, gate.passed, gate.failed
+
+API Endpoints:
+
+- GET /api/runner-jobs/:jobId/events - List events for a job (with pagination and filtering)
+
+Service Methods:
+
+- emitEvent() - Generic event emission
+- getEventsByJobId() - Query events with filters
+- Convenience methods: emitJobCreated(), emitJobStarted(), emitStepStarted(), emitAiTokensUsed(), etc.
+
+Quality Gates:
+
+- Typecheck: PASSED
+- Lint: PASSED
+- Build: PASSED
+- Tests: PASSED (17/17 tests)
+- Full test suite: PASSED (1327 tests)