fix(#196): fix race condition in job status updates

Implemented optimistic locking with version field and SELECT FOR UPDATE
transactions to prevent data corruption from concurrent job status updates.

Changes:
- Added version field to RunnerJob schema for optimistic locking
- Created migration 20260202_add_runner_job_version_for_concurrency
- Implemented ConcurrentUpdateException for conflict detection
- Updated RunnerJobsService methods with optimistic locking:
  * updateStatus() - with version checking and retry logic
  * updateProgress() - with version checking and retry logic
  * cancel() - with version checking and retry logic
- Updated CoordinatorIntegrationService with SELECT FOR UPDATE:
  * updateJobStatus() - transaction with row locking
  * completeJob() - transaction with row locking
  * failJob() - transaction with row locking
  * updateJobProgress() - optimistic locking
- Added retry mechanism (3 attempts) with exponential backoff
- Added comprehensive concurrency tests (10 tests, all passing)
- Updated existing test mocks to support updateMany

Test Results:
- All 10 concurrency tests passing ✓
- Tests cover concurrent status updates, progress updates, completions,
  cancellations, retry logic, and exponential backoff

This fix prevents race conditions that could cause:
- Lost job results (double completion)
- Lost progress updates
- Invalid status transitions
- Data corruption under concurrent access

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Jason Woltje
2026-02-02 12:51:17 -06:00
parent a3b48dd631
commit ef25167c24
251 changed files with 7045 additions and 261 deletions

View File

@@ -0,0 +1,23 @@
import { ConflictException } from "@nestjs/common";
/**
* Exception thrown when a concurrent update conflict is detected
* This occurs when optimistic locking detects that a record has been
* modified by another process between read and write operations
*/
export class ConcurrentUpdateException extends ConflictException {
constructor(resourceType: string, resourceId: string, currentVersion?: number) {
const message = currentVersion
? `Concurrent update detected for ${resourceType} ${resourceId} at version ${currentVersion}. The record was modified by another process.`
: `Concurrent update detected for ${resourceType} ${resourceId}. The record was modified by another process.`;
super({
message,
error: "Concurrent Update Conflict",
resourceType,
resourceId,
currentVersion,
retryable: true,
});
}
}