mosaic/stack
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

[CRITICAL] Fix silent Valkey event parsing failures #263

New Issue
Closed
opened 2026-02-02 23:16:17 +00:00 by jason.woltje · 1 comment
jason.woltje commented 2026-02-02 23:16:17 +00:00
Owner
Copy Link

Priority: CRITICAL - Silent failures

Problem:
Malformed JSON events in Valkey pub/sub are silently dropped. Critical orchestration events may be lost without any notification.

File: apps/orchestrator/src/valkey/valkey.client.ts:206-213

Hidden Errors:

  • Malformed JSON in event messages
  • Event schema changes causing parsing failures
  • Corrupted data in pub/sub channel
  • Type mismatches between sender and receiver

Impact:

  • Agents may miss killswitch commands
  • State transitions may not propagate
  • Task assignments may be dropped
  • System state becomes inconsistent

Acceptance Criteria:

  • JSON parse errors logged at ERROR level (not console.error)
  • Error events emitted to notify subscribers of parsing failures
  • Event handler receives notification of malformed events
  • Tests verify error event emission
  • Original message preserved in error event for debugging

Recommended Fix:

this.subscriber.on('message', async (channel: string, message: string) => {
  try {
    const event = JSON.parse(message) as OrchestratorEvent;
    await handler(event);
  } catch (error) {
    const errorMsg = `Failed to parse event: ${error.message}`;
    this.logger.error(errorMsg, { channel, messagePreview: message.substring(0, 100) });
    
    // Emit error event to notify subscribers
    const errorEvent: OrchestratorEvent = {
      type: 'event.parse_error',
      timestamp: new Date().toISOString(),
      error: errorMsg,
      data: { channel, messagePreview: message.substring(0, 100) }
    };
    await handler(errorEvent);
  }
});

Code Review Confidence: 88%
Found by: pr-review-toolkit:silent-failure-hunter

**Priority:** CRITICAL - Silent failures **Problem:** Malformed JSON events in Valkey pub/sub are silently dropped. Critical orchestration events may be lost without any notification. **File:** `apps/orchestrator/src/valkey/valkey.client.ts:206-213` **Hidden Errors:** - Malformed JSON in event messages - Event schema changes causing parsing failures - Corrupted data in pub/sub channel - Type mismatches between sender and receiver **Impact:** - Agents may miss killswitch commands - State transitions may not propagate - Task assignments may be dropped - System state becomes inconsistent **Acceptance Criteria:** - [ ] JSON parse errors logged at ERROR level (not console.error) - [ ] Error events emitted to notify subscribers of parsing failures - [ ] Event handler receives notification of malformed events - [ ] Tests verify error event emission - [ ] Original message preserved in error event for debugging **Recommended Fix:** ```typescript this.subscriber.on('message', async (channel: string, message: string) => { try { const event = JSON.parse(message) as OrchestratorEvent; await handler(event); } catch (error) { const errorMsg = `Failed to parse event: ${error.message}`; this.logger.error(errorMsg, { channel, messagePreview: message.substring(0, 100) }); // Emit error event to notify subscribers const errorEvent: OrchestratorEvent = { type: 'event.parse_error', timestamp: new Date().toISOString(), error: errorMsg, data: { channel, messagePreview: message.substring(0, 100) } }; await handler(errorEvent); } }); ``` **Code Review Confidence:** 88% **Found by:** pr-review-toolkit:silent-failure-hunter
jason.woltje added this to the M6-AgentOrchestration (0.0.6) milestone 2026-02-02 23:16:17 +00:00
jason.woltje added the orchestrator label 2026-02-02 23:16:17 +00:00
jason.woltje commented 2026-02-03 17:37:37 +00:00
Author
Owner
Copy Link

✅ Fixed: Added EventErrorHandler type and proper error handling for event parsing failures in ValkeyClient.

✅ Fixed: Added EventErrorHandler type and proper error handling for event parsing failures in ValkeyClient.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ai
AI/LLM integration
 api
Backend/API work
 api
auth
Authentication
 database
database
Database/schema work
 devops
Docker/deployment
 docs
Documentation
 frontend
graph
knowledge-module
migration
Data migration
 orchestrator
Orchestrator service (apps/orchestrator/)
 p0
Critical priority
 p1
High priority
 p2
Medium priority
 p3
Low priority
 performance
Performance work
 phase-1
phase-2
phase-3
phase-4
phase-5
plugin
MoltBot plugin work
 search
security
Security-related
 setup
Initial setup
 testing
Testing/QA
 web
Frontend work
No Label orchestrator
Milestone
No items
No Milestone M6-AgentOrchestration (0.0.6)
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaic/stack#263
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?