mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

fix(wake): #924 route dead-letter quarantine alarm via WAKE_ALARM_SINK_CMD with per-observed_seq dedup (G2a) #929

Merged
Mos merged 1 commits from fix/wake-924-deadletter-alarm into main 2026-07-26 11:45:09 +00:00
Conversation 1 Commits 1 Files Changed 3 +317 -16
jason.woltje commented 2026-07-26 11:35:10 +00:00
Owner
Copy Link

Closes #924
Part of #892

Defect (G2a silent-degradation)

PR #920's per-entry drain QUARANTINE dead-letters a render-refused digest entry and raises its fail-loud alarm to stderr/journal-LOCAL only. A dead-lettered entry is STORE-ACCOUNTED (§2.3), so the reconciler never re-flags it — journal-local visibility means an unattended operator can permanently miss a real obligation: the entry is quarantined (not delivered), store-accounted (not re-flagged), and only visible in a local journal nobody is watching.

Fix

digest.sh's _quarantine_entry now ALSO routes the same per-entry alarm through WAKE_ALARM_SINK_CMD — reusing beacon.sh's (W6/#910) exact pluggable off-host alarm-sink adapter contract verbatim: sh -c "$WAKE_ALARM_SINK_CMD" with the alarm payload (JSON, naming observed_seq) on stdin. The operator's adapter command resolves its target by-name internally; digest.sh inlines no endpoint/secret. This is in addition to, never instead of, the existing stderr diagnostic (kept unchanged).

Fail-closed (mirrors beacon.sh's _fire_alarm): an unconfigured (unset) or unreachable (non-zero exit) WAKE_ALARM_SINK_CMD produces a LOUD stderr diagnostic naming the silent-no-alarm-host hazard — never a silent no-alarm. This failure is intentionally a per-entry diagnostic, not a whole-render exit-code failure: one bad/missing alarm target must never wedge delivery of the rest of the cumulative-state drain (#920's core no-head-of-line-blocking property is preserved).

Dedup

Entries carry no per-entry wake_id (that field is the per-render ack copy-run id minted by sign.sh); the entry's durable identity is its store-allocated observed_seq (#908's sole monotonic allocator — stable for the life of the entry). A durable alarmed-set file under STATE_DIR (dead-letter-alarmed.set, one observed_seq per line, atomic-written via the existing _atomic_write primitive) records which observed_seqs have already been routed off-host, so:

  • a still-dead-lettered entry re-drained every timer tick — or across a process restart, since the marker is durable state, not an in-process cache — is alarmed off-host exactly once;
  • a new distinct dead-lettered entry (a new observed_seq) is not yet in the set, so it still routes its own one alarm.

Red-first

Extends test-wake-digest-quarantine.sh with groups Q6-Q9 (kept alongside the pre-existing #920 Q1-Q5 invariants, which remain green throughout):

  • Q6 (a): a dead-lettered entry routes EXACTLY ONE alarm to a captured WAKE_ALARM_SINK_CMD (payload names observed_seq), AND the stderr diagnostic still fires.
  • Q7 (b): re-draining the SAME still-dead-lettered entry 4x routes ZERO additional off-host alarms (dedup).
  • Q8 (c): a NEW distinct dead-lettered entry routes its OWN one alarm.
  • Q9 (d): WAKE_ALARM_SINK_CMD unconfigured OR unreachable → FAIL LOUD diagnostic naming the hazard, never silent no-alarm; render still exits 0 (per-entry, not whole-drain).

Proven RED against the pre-fix (stderr-only) digest.sh via git stash (Q6-Q9 all fail: 0 alarms ever captured, no off-host fail-closed diagnostic exists), then GREEN after the fix, with Q1-Q5 unaffected throughout.

Gates

  • shellcheck 0.11: clean on both changed .sh files.
  • verify-sanitized.sh: PASS (no operator identity/host leakage).
  • Full pnpm run test:framework-shell: rc=0 normal AND with openssl masked off PATH (command -v openssl empty) — the digest-quarantine harness has no openssl dependency.
  • No ts/json files touched (prettier gate N/A).
  • Bumped the wake component manifest.txt to 0.6.6 + inventory blurb.

Files changed

  • packages/mosaic/framework/tools/wake/digest.sh — the fix (only file with behavior changes; beacon.sh, store.sh, reconcile.sh unchanged, adapter contract reused not modified).
  • packages/mosaic/framework/tools/wake/test-wake-digest-quarantine.sh — Q6-Q9.
  • packages/mosaic/framework/tools/wake/manifest.txt — version bump + changelog + inventory.
Closes #924 Part of #892 ## Defect (G2a silent-degradation) PR #920's per-entry drain QUARANTINE dead-letters a render-refused digest entry and raises its fail-loud alarm to **stderr/journal-LOCAL only**. A dead-lettered entry is **STORE-ACCOUNTED** (§2.3), so the reconciler never re-flags it — journal-local visibility means an unattended operator can **permanently miss a real obligation**: the entry is quarantined (not delivered), store-accounted (not re-flagged), and only visible in a local journal nobody is watching. ## Fix `digest.sh`'s `_quarantine_entry` now ALSO routes the same per-entry alarm through **`WAKE_ALARM_SINK_CMD`** — reusing beacon.sh's (W6/#910) exact pluggable off-host alarm-sink adapter contract **verbatim**: `sh -c "$WAKE_ALARM_SINK_CMD"` with the alarm payload (JSON, naming `observed_seq`) on stdin. The operator's adapter command resolves its target by-name internally; digest.sh inlines no endpoint/secret. This is **in addition to**, never instead of, the existing stderr diagnostic (kept unchanged). **Fail-closed** (mirrors beacon.sh's `_fire_alarm`): an unconfigured (unset) or unreachable (non-zero exit) `WAKE_ALARM_SINK_CMD` produces a LOUD stderr diagnostic naming the silent-no-alarm-host hazard — never a silent no-alarm. This failure is intentionally a per-entry diagnostic, not a whole-render exit-code failure: one bad/missing alarm target must never wedge delivery of the rest of the cumulative-state drain (#920's core no-head-of-line-blocking property is preserved). ## Dedup Entries carry no per-entry `wake_id` (that field is the per-render ack copy-run id minted by `sign.sh`); the entry's durable identity is its store-allocated `observed_seq` (#908's sole monotonic allocator — stable for the life of the entry). A durable alarmed-set file under `STATE_DIR` (`dead-letter-alarmed.set`, one `observed_seq` per line, atomic-written via the existing `_atomic_write` primitive) records which `observed_seq`s have already been routed off-host, so: - a still-dead-lettered entry re-drained every timer tick — or across a process restart, since the marker is durable state, not an in-process cache — is alarmed off-host **exactly once**; - a **new distinct** dead-lettered entry (a new `observed_seq`) is not yet in the set, so it still routes its **own** one alarm. ## Red-first Extends `test-wake-digest-quarantine.sh` with groups Q6-Q9 (kept alongside the pre-existing #920 Q1-Q5 invariants, which remain green throughout): - **Q6 (a)**: a dead-lettered entry routes EXACTLY ONE alarm to a captured `WAKE_ALARM_SINK_CMD` (payload names `observed_seq`), AND the stderr diagnostic still fires. - **Q7 (b)**: re-draining the SAME still-dead-lettered entry 4x routes ZERO additional off-host alarms (dedup). - **Q8 (c)**: a NEW distinct dead-lettered entry routes its OWN one alarm. - **Q9 (d)**: `WAKE_ALARM_SINK_CMD` unconfigured OR unreachable → FAIL LOUD diagnostic naming the hazard, never silent no-alarm; render still exits 0 (per-entry, not whole-drain). Proven RED against the pre-fix (stderr-only) `digest.sh` via `git stash` (Q6-Q9 all fail: 0 alarms ever captured, no off-host fail-closed diagnostic exists), then GREEN after the fix, with Q1-Q5 unaffected throughout. ## Gates - shellcheck 0.11: clean on both changed `.sh` files. - `verify-sanitized.sh`: PASS (no operator identity/host leakage). - Full `pnpm run test:framework-shell`: rc=0 normal AND with `openssl` masked off PATH (`command -v openssl` empty) — the digest-quarantine harness has no openssl dependency. - No ts/json files touched (prettier gate N/A). - Bumped the wake component `manifest.txt` to `0.6.6` + inventory blurb. ## Files changed - `packages/mosaic/framework/tools/wake/digest.sh` — the fix (only file with behavior changes; `beacon.sh`, `store.sh`, `reconcile.sh` unchanged, adapter contract reused not modified). - `packages/mosaic/framework/tools/wake/test-wake-digest-quarantine.sh` — Q6-Q9. - `packages/mosaic/framework/tools/wake/manifest.txt` — version bump + changelog + inventory.
jason.woltje added 1 commit 2026-07-26 11:35:11 +00:00
fix(wake): #924 route dead-letter quarantine alarm via WAKE_ALARM_SINK_CMD with per-wake_id dedup (G2a)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Details
74da62b336 
The #920 per-entry drain quarantine dead-letters a render-refused digest
entry and raises its fail-loud alarm to stderr/journal-LOCAL only. A
dead-lettered entry is STORE-ACCOUNTED (§2.3), so the reconciler never
re-flags it: journal-local-only visibility means an unattended operator
can PERMANENTLY MISS a real obligation (G2a silent-degradation).

digest.sh now ALSO routes the SAME per-entry quarantine alarm through
WAKE_ALARM_SINK_CMD, reusing beacon.sh's (W6/#910) exact pluggable
off-host alarm-sink adapter contract verbatim (operator target resolved
by-name inside the adapter, fail-closed) -- in addition to, never
instead of, the existing stderr diagnostic.

Dedup: entries carry no per-entry wake_id (that field is the per-render
ack copy-run id); the entry's durable identity is its store-allocated
observed_seq (#908's sole monotonic allocator). A durable alarmed-set
file under STATE_DIR (dead-letter-alarmed.set, atomic-written) records
which observed_seqs have already been routed off-host, so a
still-dead-lettered entry re-drained every timer tick -- or across a
process restart -- is alarmed off-host EXACTLY ONCE. A new distinct
dead-lettered entry (a new observed_seq) still routes its own one alarm.

An unconfigured or unreachable WAKE_ALARM_SINK_CMD is a LOUD per-entry
stderr diagnostic (mirrors beacon.sh's fail-closed wording) but does not
itself fail the whole render -- per-entry fail-loud, never a whole-drain
wedge, preserving #920's core no-head-of-line-blocking property.

RED-FIRST: extends test-wake-digest-quarantine.sh with Q6-Q9, proven RED
against the pre-fix stderr-only code (0 alarms ever captured; no
off-host fail-closed diagnostic exists) and GREEN after the fix, while
the pre-existing Q1-Q5 (#920) invariants remain green throughout.

Bumps the wake component manifest to 0.6.6.

Closes #924
Part of #892

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
jason.woltje changed title from fix(wake): #924 route dead-letter quarantine alarm via WAKE_ALARM_SINK_CMD with per-wake_id dedup (G2a) to fix(wake): #924 route dead-letter quarantine alarm via WAKE_ALARM_SINK_CMD with per-observed_seq dedup (G2a) 2026-07-26 11:44:14 +00:00
jason.woltje commented 2026-07-26 11:44:49 +00:00
Author
Owner
Copy Link

Record of Review — PR #929 (issue #924): dead-letter quarantine alarm → off-host sink + dedup (G2a) — FINAL, CI-GREEN

VERDICT: APPROVE — GO. REVIEWED-HEAD (full-40): 74da62b336f0fb42e06e64147ff9230206649964. CI: TERMINAL-GREEN (pipeline 2077).

Review basis

Independent reviewer aae3edad (opus, ≠ builder mosaic-coder) — APPROVE, all 6 criteria PASS, criteria 2/3/4 independently REPRODUCED:

  1. Scope/provenance — head==74da62b3…; mergeable; id2 ≠ commit-author mosaic-coder; body Closes #924 + Part of #892; delta = only digest.sh, manifest.txt, test-wake-digest-quarantine.sh; manifest 0.6.5→0.6.6.
  2. ★ Off-host routing + fail-closed (reproduced) — _dlq_route_alarm uses beacon.shs exact WAKE_ALARM_SINK_CMD adapter (payload on stdin, target by-name, no framework-inlined endpoint — firewall clean); kept IN ADDITION to stderr. Reviewer repro: sink received 1 alarm naming observed_seq + stderr fired; unconfigured→FAIL LOUD (silent-no-alarm named), failing→FAIL LOUD (UNREACHABLE).
  3. ★ Fail-closed stays PER-ENTRY — no #920 HOL regression (reproduced) — _dlq_* use return NOT exit (deliberate correct adaptation vs beacon.shs exit). Mixed-feed repro (bad + 2 clean, sink=false): render rc=0, both clean siblings delivered, bad excluded+dead-lettered, FAIL LOUD fired, alarmed.set left UNMARKED (failed route retries, not falsely deduped). #920 no-head-of-line-block property holds.
  4. ★ Dedup correctness = observed_seq (reproduced) — durable $STATE_DIR/dead-letter-alarmed.set (atomic, one seq/line, mode 0600). Repro: 6 fresh-process drains (=restarts) of the same dead-lettered entry → EXACTLY 1 off-host alarm (stderr fires each — local leg not deduped); new distinct seq → its own 1 alarm. observed_seq = #908 sole-allocator unique+durable per-obligation id → sound key.
  5. Contract preserved — #920 Q1-Q5 green; only _quarantine_entry + new _dlq_* helpers touched; scrub/exit-4/two-tier unchanged; alarmed.set separate from dead-letter.jsonl.
  6. Red-first + gates — Q6-Q9 RED against pre-fix (10 assertions), GREEN after; shellcheck 0.11 clean; verify-sanitized PASS; test:framework-shell rc=0 normal AND openssl-masked; firewall 0.

Non-blocking follow-ups

(1) PR title said "wake_id" (code correctly uses observed_seq) — CORRECTED in-place. (2) _dlq_alarm_offhost is check-then-act without a lock → two concurrent renders on one STATE_DIR could double-route; UNREACHABLE in the per-host single-instance deployment (digest drains single-instance via the detector daemon flock) — non-blocking, noted for if concurrent renders ever become possible.

6-check — GO

  1. open/mergeable base main ✓ · 2. CI TERMINAL-GREEN @ full-40 74da62b336f0fb42e06e64147ff9230206649964 ✓ · 3. reviewer(aae3edad)≠builder durable RoR ✓ · 4. reviewed==CI==merge-head all 74da62b3 ✓ · 5. body Closes #924 + Part of #892 ✓ · 6. queue-guard at merge.
    GO for id-11 + squash-merge at FULL-40 74da62b336f0fb42e06e64147ff9230206649964. Closes #924 (G2a: dead-letter obligations now reach an off-host alarm, deduped per observed_seq). Non-executor.
# Record of Review — PR #929 (issue #924): dead-letter quarantine alarm → off-host sink + dedup (G2a) — FINAL, CI-GREEN **VERDICT: APPROVE — GO.** **REVIEWED-HEAD (full-40):** `74da62b336f0fb42e06e64147ff9230206649964`. **CI: TERMINAL-GREEN** (pipeline 2077). ## Review basis Independent reviewer `aae3edad` (opus, ≠ builder `mosaic-coder`) — **APPROVE, all 6 criteria PASS**, criteria 2/3/4 independently REPRODUCED: 1. **Scope/provenance** — head==`74da62b3…`; mergeable; id2 ≠ commit-author `mosaic-coder`; body `Closes #924` + `Part of #892`; delta = only digest.sh, manifest.txt, test-wake-digest-quarantine.sh; manifest 0.6.5→0.6.6. 2. **★ Off-host routing + fail-closed (reproduced)** — `_dlq_route_alarm` uses beacon.shs exact `WAKE_ALARM_SINK_CMD` adapter (payload on stdin, target by-name, **no framework-inlined endpoint** — firewall clean); kept IN ADDITION to stderr. Reviewer repro: sink received 1 alarm naming observed_seq + stderr fired; unconfigured→FAIL LOUD (silent-no-alarm named), failing→FAIL LOUD (UNREACHABLE). 3. **★ Fail-closed stays PER-ENTRY — no #920 HOL regression (reproduced)** — `_dlq_*` use `return` NOT `exit` (deliberate correct adaptation vs beacon.shs exit). Mixed-feed repro (bad + 2 clean, sink=false): render rc=0, both clean siblings delivered, bad excluded+dead-lettered, FAIL LOUD fired, alarmed.set left UNMARKED (failed route retries, not falsely deduped). #920 no-head-of-line-block property holds. 4. **★ Dedup correctness = observed_seq (reproduced)** — durable `$STATE_DIR/dead-letter-alarmed.set` (atomic, one seq/line, mode 0600). Repro: 6 fresh-process drains (=restarts) of the same dead-lettered entry → EXACTLY 1 off-host alarm (stderr fires each — local leg not deduped); new distinct seq → its own 1 alarm. observed_seq = #908 sole-allocator unique+durable per-obligation id → sound key. 5. **Contract preserved** — #920 Q1-Q5 green; only `_quarantine_entry` + new `_dlq_*` helpers touched; scrub/exit-4/two-tier unchanged; alarmed.set separate from dead-letter.jsonl. 6. **Red-first + gates** — Q6-Q9 RED against pre-fix (10 assertions), GREEN after; shellcheck 0.11 clean; verify-sanitized PASS; test:framework-shell rc=0 normal AND openssl-masked; firewall 0. ## Non-blocking follow-ups (1) PR title said "wake_id" (code correctly uses observed_seq) — CORRECTED in-place. (2) `_dlq_alarm_offhost` is check-then-act without a lock → two concurrent renders on one STATE_DIR could double-route; UNREACHABLE in the per-host single-instance deployment (digest drains single-instance via the detector daemon flock) — non-blocking, noted for if concurrent renders ever become possible. ## 6-check — GO 1. open/mergeable base main ✓ · 2. **CI TERMINAL-GREEN @ full-40 `74da62b336f0fb42e06e64147ff9230206649964`** ✓ · 3. reviewer(`aae3edad`)≠builder durable RoR ✓ · 4. reviewed==CI==merge-head all `74da62b3` ✓ · 5. body `Closes #924` + `Part of #892` ✓ · 6. queue-guard at merge. **GO for id-11 + squash-merge at FULL-40 `74da62b336f0fb42e06e64147ff9230206649964`. Closes #924 (G2a: dead-letter obligations now reach an off-host alarm, deduped per observed_seq). Non-executor.**
Mos approved these changes 2026-07-26 11:45:08 +00:00
Mos left a comment
First-time contributor
Copy Link

GO — Gate-16 id-11 stamp, pinned to exact FULL head 74da62b336.
Basis: verbatim FINAL RoR (comment 19213) — independent review APPROVE 6/6 with the load-bearing interaction reproduced: the sink-routing fail-closes PER-ENTRY (return-not-exit) so #920's no-head-of-line guarantee survives even an alarm-path failure; dedup keys on observed_seq (the durable store identity — MS-LEAD's in-place title correction was right) and held exactly-once across six drains and restarts. Closes #924 — the G2a silent-miss is fixed: a dead-lettered obligation now reaches an off-host alarm exactly once, never vanishing into a journal. Part of #892. Named executor: Mos (id-11), squash pinned.

GO — Gate-16 id-11 stamp, pinned to exact FULL head 74da62b336f0fb42e06e64147ff9230206649964. Basis: verbatim FINAL RoR (comment 19213) — independent review APPROVE 6/6 with the load-bearing interaction reproduced: the sink-routing fail-closes PER-ENTRY (return-not-exit) so #920's no-head-of-line guarantee survives even an alarm-path failure; dedup keys on observed_seq (the durable store identity — MS-LEAD's in-place title correction was right) and held exactly-once across six drains and restarts. Closes #924 — the G2a silent-miss is fixed: a dead-lettered obligation now reaches an off-host alarm exactly once, never vanishing into a journal. Part of #892. Named executor: Mos (id-11), squash pinned.
Mos merged commit 347c1d57c1 into main 2026-07-26 11:45:09 +00:00
Mos deleted branch fix/wake-924-deadletter-alarm 2026-07-26 11:45:10 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Mos
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#929
Delete Branch "fix/wake-924-deadletter-alarm"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?