mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

wake/digest quarantine dead-letter alarm: route via WAKE_ALARM_SINK_CMD with per-wake_id dedup (not journal-local — G2a silent-miss) #924

New Issue
Closed
opened 2026-07-26 08:49:19 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-07-26 08:49:19 +00:00
Owner
Copy Link

Non-blocking follow-up from the dragon-lin pilot's cure-verification of finding #6 / PR #922 (cure otherwise verified 3/3 on the finding bench). Normal cadence with #913/#917/#923.

Defect (G2a silent-degradation shape). PR #920's per-entry drain QUARANTINE dead-letters a render-refused entry and raises the alarm to stderr/journal-LOCAL only. But a dead-lettered obligation is store-accounted — the reconciler will therefore never re-flag it (it counts as accounted). So journal-local visibility means an unattended operator can permanently miss a real obligation: the entry is quarantined (not delivered), store-accounted (not re-flagged), and only visible in a local journal nobody is watching. That is exactly the G2a silent-degradation the canon exists to prevent.

RULING (Mos). The dead-letter quarantine alarm must route through WAKE_ALARM_SINK_CMD (the same pluggable off-host alarm-sink adapter W6/beacon uses — operator target resolved by-name, fail-closed) instead of (or in addition to) stderr, so a dead-lettered obligation reaches an OFF-HOST alarm an operator actually monitors. With per-wake_id DEDUP: alarm ONCE per dead-lettered entry (keyed by wake_id / observed_seq), never per re-render — the digest re-renders the cumulative unacked set each tick, so a still-dead-lettered entry must not re-alarm every drain. The dedup directly answers the repeat-alarm concern that motivated the original local-only choice.

Doctrine line (rides the fix, into the design/guide): any STORE-ACCOUNTED terminal disposition (dead-letter, quarantine) MUST alarm via the off-host sink with dedup, never journal-local — because store-accounting suppresses reconciler re-flagging, so a journal-local alarm is a permanent silent-miss (G2a).

Red-first: a dead-lettered entry routes exactly one alarm to WAKE_ALARM_SINK_CMD (fail-closed if unconfigured, consistent with W6); re-draining the same still-dead-lettered entry N times routes ZERO additional alarms (dedup by wake_id); a NEW distinct dead-lettered entry routes its own one alarm. Preserve the stderr diagnostic too (local + off-host, not either/or). Refs: #892, #920 (quarantine), W6/#910 (alarm-sink adapter), dragon-lin finding #6.

Non-blocking follow-up from the dragon-lin pilot's cure-verification of finding #6 / PR #922 (cure otherwise verified 3/3 on the finding bench). Normal cadence with #913/#917/#923. **Defect (G2a silent-degradation shape).** PR #920's per-entry drain QUARANTINE dead-letters a render-refused entry and raises the alarm to **stderr/journal-LOCAL only**. But a dead-lettered obligation is **store-accounted** — the reconciler will therefore **never re-flag it** (it counts as accounted). So journal-local visibility means an **unattended operator can permanently miss a real obligation**: the entry is quarantined (not delivered), store-accounted (not re-flagged), and only visible in a local journal nobody is watching. That is exactly the G2a silent-degradation the canon exists to prevent. **RULING (Mos).** The dead-letter quarantine alarm must route through **`WAKE_ALARM_SINK_CMD`** (the same pluggable off-host alarm-sink adapter W6/beacon uses — operator target resolved by-name, fail-closed) instead of (or in addition to) stderr, so a dead-lettered obligation reaches an OFF-HOST alarm an operator actually monitors. With **per-`wake_id` DEDUP**: alarm ONCE per dead-lettered entry (keyed by wake_id / observed_seq), **never per re-render** — the digest re-renders the cumulative unacked set each tick, so a still-dead-lettered entry must not re-alarm every drain. The dedup directly answers the repeat-alarm concern that motivated the original local-only choice. **Doctrine line (rides the fix, into the design/guide):** any STORE-ACCOUNTED terminal disposition (dead-letter, quarantine) MUST alarm via the off-host sink with dedup, never journal-local — because store-accounting suppresses reconciler re-flagging, so a journal-local alarm is a permanent silent-miss (G2a). **Red-first:** a dead-lettered entry routes exactly one alarm to WAKE_ALARM_SINK_CMD (fail-closed if unconfigured, consistent with W6); re-draining the same still-dead-lettered entry N times routes ZERO additional alarms (dedup by wake_id); a NEW distinct dead-lettered entry routes its own one alarm. Preserve the stderr diagnostic too (local + off-host, not either/or). Refs: #892, #920 (quarantine), W6/#910 (alarm-sink adapter), dragon-lin finding #6.
Mos closed this issue 2026-07-26 11:45:10 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#924
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?