fix(#196): fix race condition in job status updates
Implemented optimistic locking with version field and SELECT FOR UPDATE transactions to prevent data corruption from concurrent job status updates. Changes: - Added version field to RunnerJob schema for optimistic locking - Created migration 20260202_add_runner_job_version_for_concurrency - Implemented ConcurrentUpdateException for conflict detection - Updated RunnerJobsService methods with optimistic locking: * updateStatus() - with version checking and retry logic * updateProgress() - with version checking and retry logic * cancel() - with version checking and retry logic - Updated CoordinatorIntegrationService with SELECT FOR UPDATE: * updateJobStatus() - transaction with row locking * completeJob() - transaction with row locking * failJob() - transaction with row locking * updateJobProgress() - optimistic locking - Added retry mechanism (3 attempts) with exponential backoff - Added comprehensive concurrency tests (10 tests, all passing) - Updated existing test mocks to support updateMany Test Results: - All 10 concurrency tests passing ✓ - Tests cover concurrent status updates, progress updates, completions, cancellations, retry logic, and exponential backoff This fix prevents race conditions that could cause: - Lost job results (double completion) - Lost progress updates - Invalid status transitions - Data corruption under concurrent access Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,23 @@
|
||||
import { ConflictException } from "@nestjs/common";
|
||||
|
||||
/**
|
||||
* Exception thrown when a concurrent update conflict is detected
|
||||
* This occurs when optimistic locking detects that a record has been
|
||||
* modified by another process between read and write operations
|
||||
*/
|
||||
export class ConcurrentUpdateException extends ConflictException {
|
||||
constructor(resourceType: string, resourceId: string, currentVersion?: number) {
|
||||
const message = currentVersion
|
||||
? `Concurrent update detected for ${resourceType} ${resourceId} at version ${currentVersion}. The record was modified by another process.`
|
||||
: `Concurrent update detected for ${resourceType} ${resourceId}. The record was modified by another process.`;
|
||||
|
||||
super({
|
||||
message,
|
||||
error: "Concurrent Update Conflict",
|
||||
resourceType,
|
||||
resourceId,
|
||||
currentVersion,
|
||||
retryable: true,
|
||||
});
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user