mosaicstack/stack
1
0
Fork 1
Code Issues 111 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity
Files
63c5d2056fddc02005538630d996dd1f739d64c5
stack/packages/mosaic/framework/tools/wake/test-wake-digest-quarantine.sh
T
mos-dt-0andMos 6a7fce34bb
ci/woodpecker/push/publish Pipeline was successful
Details
ci/woodpecker/push/ci Pipeline was successful
Details
fix(wake): #946 digest ack watermark clamped at quarantined seqs — disclose AND clamp (#951) 
Closes #946.

The digest omitted a quarantined entry's claim from disclosure while still advancing the ack watermark it instructed the consumer to run — converting a fail-safe HOLD into a silent DISCARD, through the documented normal path. Measured: the burial instruction was re-issued FIVE times, four fresh digests plus one system-initiated redelivery fired purely because the entry had gone unconsumed for 1826s. That redelivery is the proof of the 'indefinitely' half: the mechanism re-asserted itself with no new information.

SCOPE — this was NOT a missing check in the consume path. Measured before the fix: dead-letter occurrences were digest.sh 27, store.sh 0, ack.sh 0, detector.sh 0, reconcile.sh 0. Quarantine was owned ENTIRELY by the renderer; the store that advances the watermark had zero knowledge the ledger existed, so an entry could be quarantined by one subsystem and consumed by another with no possible interaction. The fix is therefore a deliberate cross-module decision — option (b), quarantine recorded into a store-owned file, preserving the existing direction of dependency — pre-registered by the consumer before any diff existed.

VERIFICATION
- Pipeline 2107 terminal SUCCESS at e11bc6622, read clone-inclusive from the provider API rather than through `pipeline-status.sh` (which filters `.type != "clone"` per workflow and would hide a clone failure behind an all-green table): 9/9 children success, exit 0 each, no non-success member. Its `test` step runs all nine wake harnesses via turbo -> packages/mosaic `test` -> `test:framework-shell`.
- Independent review by the consumer on the affected lane: eleven pre-registered acceptance checks, authored and delivered BEFORE the diff was read — the file list deliberately unlooked-at, because a filename alone would have disclosed which option was chosen. All eleven resolved, no blocker.
- The check that decides it: a RAW `ack.sh consumed --upto N` with no digest involved must refuse to advance past a quarantined seq — the case an agent hits when a digest is MISSED, and the one that would have sunk a disclosure-only fix. Covered at the head as a named assertion (T13 ordinary-path bypass), written independently of the reviewer's list.
- Mutation: one asserted site disabled -> TWELVE assertions die, every one BEHAVIOURAL, ZERO count assertions, including one killing across the module boundary the fix spans.
- RED control at base a6b5f6a: 34 and 10 assertions fail, matching the body exactly.
- Coordinator re-verify by a different instrument than the reviewer used: static reference counts across the base/head boundary — store.sh 0 -> 49, ack.sh 0 -> 6, `--agent` unchanged at 4 (so #949 correctly stayed out). A fix present-but-inert passes the count and fails the mutation; a fix behaviourally correct but smuggling #949 passes the mutation and fails the count. Neither result is reachable by repeating the other.

KNOWN RESIDUALS
- The `consumed-hashes` repair criterion is met only for keys that RE-EMIT. A corrupted row whose key never recurs stays false indefinitely; the sweep covers those, and the known-false row named in the acceptance criteria had already self-healed by re-emission rather than by design — safe by population, not by design.
- The audit's clean-sweep message names one unprovable class; a second exists (a surviving dead-letter row with an empty `observed_hash` cannot be convicted either). Wording, not logic. Filed separately.
- The audit's provability bound makes dead-letter RETENTION load-bearing for auditability. Nothing prunes it today, so this is latent — but any future rotation or size cap silently converts provable rows into unprovable ones with no signal at either end. This is not a defect; it is a property that BECAME load-bearing and is recorded nowhere. Filed separately.
- `test-wake-detector.sh` D4 fails at this head AND identically at base, with an empty diff over detector files — pre-existing, tracked, not introduced here.

Authored by pepper (sb-it-1-dt); reviewed independently by mos-dt (sb-it-1-dt). The mos-dt-0 commit and fork identity does not identify the author — attribution collapse tracked separately.

Co-authored-by: mos-dt-0 <[email protected]>
2026-07-30 15:41:03 +00:00

585 lines
37 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
# test-wake-digest-quarantine.sh — RED-FIRST invariant harness for #920 (EPIC
# #892): per-entry QUARANTINE of a render-refused digest entry (no head-of-line
# blocking) + reconciler ENUMERATIONS render ORIENTATION-tier.
#
# Each test asserts ONE invariant and goes RED against the pre-#920 digest.sh:
# Q1 (a) QUARANTINE + REST-DELIVERS: a malformed `actionable` carrying NO hard
# locator ({kind,id,observed_hash} — a content hash but no ADDRESS, no
# reconciled marker) is DEAD-LETTERED + alarmed AND EXCLUDED, while a
# clean valid sibling in the SAME drain still renders (exit 0).
# RED baseline: the old whole-digest exit-4 delivered NOTHING (the live
# head-of-line-blocking wedge). (#920 FIX1)
# [#944 AMENDMENT: at #920 this fixture pinned the live pilot's shape
# {kind,id,path,observed_hash} as the malformed case — ratifying a gate
# the detector's own locators could never satisfy (every actionable
# heartbeat-planning delta dead-lettered; live seqs 63/68). #944 amends
# that ruling: `path` is now a hard-locator arm — and ONLY path;
# bare snapshot_sha is deliberately NOT an arm, Q11(d) asserts it
# still quarantines — so the quarantine fixtures here and in Q6-Q9
# use genuinely ADDRESS-FREE shapes instead.]
# Q2 (b) ENUM-AS-ORIENTATION: a reconciler enumeration (locators.reconciled==
# true) renders as an ORIENTATION-tier pointer and does NOT exit-4 / is
# NOT quarantined. RED baseline: reconciled `actionable` + soft locators
# {kind,id,path,observed_hash} -> exit-4. (#920 FIX2)
# Q3 (c) TWO DISTINCT ENUMERATIONS BOTH SURVIVE: two distinct enumerations both
# render as SEPARATE orientation pointers (neither coalesced away) — the
# anti-collapse proof pinning the amended ruling (the REJECTED class=digest
# would have collapsed them to one; store class stays non-coalescing). (#920 FIX2)
# Q4 PRESERVED ACTIONABLE FAIL-LOUD: a genuine malformed ACTIONABLE claim
# (no reconciled marker, no hard locator) is STILL loudly surfaced
# (dead-letter + alarm) and NEVER delivered as if valid — fail-loud is
# preserved, now per-entry. (§2.1)
# Q5 CLAIM-PRECEDENCE GATED: a non-actionable-class entry that carries a
# `claim` (a consequential fact) with no hard locator is STILL gated
# (quarantined), and the reconciled exemption does not leak to it. (§2.1)
#
# #924 (G2a fix — the #920 alarm was stderr/journal-LOCAL only, a permanent
# silent-miss hazard since a dead-lettered entry is store-accounted and the
# reconciler never re-flags it): the SAME per-entry alarm now ALSO routes
# off-host via WAKE_ALARM_SINK_CMD (beacon.sh's W6/#910 pluggable adapter,
# REUSED verbatim), deduped by observed_seq (the entry's durable identity).
# Q6 (a) ONE off-host alarm + stderr diagnostic STILL fires: a dead-lettered
# entry routes EXACTLY ONE alarm to a captured WAKE_ALARM_SINK_CMD
# (payload names observed_seq), AND the #920 stderr diagnostic still
# fires (local + off-host, never either/or).
# Q7 (b) RE-DRAIN DEDUP: re-draining the SAME still-dead-lettered entry N
# times routes ZERO additional off-host alarms (durable dedup by
# observed_seq survives across separate digest.sh invocations, i.e.
# across drains/restarts, since each invocation is a fresh process).
# Q8 (c) NEW ENTRY OWN ALARM: a NEW distinct dead-lettered entry (a new
# observed_seq) routes its OWN one alarm — dedup is per-entry, not a
# global "alarm already fired at all" latch.
# Q9 (d) FAIL-CLOSED: WAKE_ALARM_SINK_CMD unconfigured OR unreachable (exit
# non-zero) is a LOUD stderr diagnostic (mirrors beacon.sh's
# fail-closed wording) — never a silent no-alarm host. Per-entry, not
# whole-drain: render still exits 0 (#920's no-head-of-line-block
# property is preserved even when the off-host leg itself fails).
#
# RED-FIRST: Q6-Q9 all go RED against the pre-#924 (stderr-only) digest.sh —
# it never references WAKE_ALARM_SINK_CMD at all, so no payload is EVER routed
# (Q6/Q7/Q8 all see 0 captured alarms) and no "FAIL LOUD ... alarm sink"
# diagnostic exists to fire (Q9).
#
# #944 (unsatisfiable-gate fix): _has_hard_locator gains the `path` arm — and
# ONLY that arm — covering the detector-built board_file vocabulary.
# Q11 POSITIVE CONTROL + RETAINED NEGATIVES, one drain: (a) the live
# seq-68 entry VERBATIM (the exact production entry that dead-lettered
# under the unsatisfiable gate: kind/id/observed_hash + path +
# snapshot_sha + snapshot_ts, class=actionable, as detector.sh
# actually emits it — NOT a hand-built dict) must RENDER as a
# CLAIM@seq with the one-call `git show <snapshot_sha>:<path>`
# re-verify hint — the assertion is the rendered claim, not merely
# the predicate returning true; (b) a path-only sibling (no snapshot
# attestation — a pre-#940 adapter or a dropped attestation) must
# ALSO render, with the "re-read <path>" hint; (c) an address-free
# sibling ({kind,id,observed_hash}) must STILL quarantine + route its
# own alarm — observed_hash is a content hash, not an address; (d)
# bare path-less snapshot_sha siblings must ALSO still quarantine —
# the widened gate must not widen PAST the board_file vocabulary
# (review-adopted criterion) — asserted at THREE lengths (7-char
# abbreviation, 40-hex, 64-char sha-256) spanning the detector's
# actual attestation validation ^[0-9a-f]{7,64}$ (detector.sh), so
# the assertion distinguishes "no snapshot_sha arm" from "an arm
# present but length-gated". RED baseline: pre-#944
# digest.sh dead-letters (a) and (b) — an assertion nobody has seen
# succeed is as unproven as one nobody has seen fail.
#
# #946 (ack watermark passes quarantined entries): the rendered digest embedded
# `ack.sh consumed --upto <observed>` even when entries in (consumed, observed]
# were quarantined — the copy-run line itself instructed the consumer to record
# deliveries that never happened (live: five successive digests each stepping
# the consumer past buried seq 68). Fix = DISCLOSE + CLAMP + force-only-past:
# Q12 disclosure (by seq — content stays EXCLUDED per Q1/Q4) + the
# embedded ack CLAMPED below the lowest quarantined seq, hermetic
# --from-file; a foreign-data render must NOT write the store's
# quarantined.set.
# Q13 nothing quarantined -> unclamped ack at the observed cursor; no
# disclosure section, no clamp note.
# Q14 store-mode render SYNCS quarantined.set (REPLACE) -> the store's
# ordinary consume path refuses past the held seq END-TO-END.
# Q15 gate-fix RECOVERY: a stale quarantined.set is REPLACED (cleared) by
# a clean store-mode render — the clamp self-heals (#944 recovery
# invariant; a cumulative-forever set would keep blocking acks on
# entries a fixed gate now renders).
# Q16 (guard, green-by-design) Q2's ENUM-B fixture must STAY address-free
# so the reconciled exemption remains load-bearing at the gate
# (#944 F1); goes RED only if the fixture regresses.
#
# Hermetic: feeds controlled JSONL via `digest.sh render --from-file` — NO store,
# NO network, NO openssl (so it runs identically under the CI openssl-mask).
# (Q14/Q15 are the intentional exception: the #946 store sync is store-mode-only
# behavior, so they drive store.sh enqueue/consume against a temp state home.)
#
# Each test runs in its own (..) subshell for env isolation; the per-subshell
# WAKE_STATE_HOME export is intentional (mirrors test-wake-reconcile.sh).
# shellcheck disable=SC2030,SC2031
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DIGEST="$SCRIPT_DIR/digest.sh"
command -v jq >/dev/null 2>&1 || {
echo "SKIP: jq not available" >&2
exit 0
}
TMP_ROOT="$(mktemp -d)"
trap 'rm -rf "$TMP_ROOT"' EXIT
FAILFILE="$TMP_ROOT/failures"
: >"$FAILFILE"
pass=0
fail_msg() {
echo " FAIL: $*" >&2
echo "x" >>"$FAILFILE"
}
ok() { pass=$((pass + 1)); }
# A 40-hex sha = a valid §2.1 hard locator.
SHA40="abcdef0123456789abcdef0123456789abcdef01"
# capture-alarm (#924): a REACHABLE off-host alarm route that APPENDS the
# routed payload (one JSON line per invocation) to $ALARM_OUT, so a test can
# count exactly how many off-host alarms fired across one or more digest.sh
# invocations. Mirrors test-wake-beacon.sh's capture-alarm idiom.
CAPTURE_ALARM="$TMP_ROOT/capture-alarm.sh"
cat >"$CAPTURE_ALARM" <<'EOF'
#!/usr/bin/env bash
cat >>"$ALARM_OUT"
EOF
chmod +x "$CAPTURE_ALARM"
# fresh_home NAME — a fresh WAKE_STATE_HOME dir, echoed.
fresh_home() {
local d="$TMP_ROOT/$1"
rm -rf "$d"
mkdir -p "$d"
printf '%s' "$d"
}
# dlq HOME — the dead-letter path for the default agent under HOME.
dlq() { printf '%s/default/dead-letter.jsonl' "$1"; }
echo "== Q1 (a): malformed address-free {kind,id,observed_hash} QUARANTINES; clean sibling STILL delivers =="
(
home="$(fresh_home q1)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q1.jsonl"
# An ADDRESS-FREE malformed shape (no reconciled marker) + a clean sibling.
# [#944: the original fixture carried `path`, which is now a hard-locator arm.]
{
printf '%s\n' '{"observed_seq":1,"class":"actionable","locators":{"kind":"repo","id":"MALFORMED-Q","observed_hash":"deadbeef"},"emit_ts":1}'
printf '{"observed_seq":2,"class":"actionable","locators":{"sha":"%s","file":"src/a.ts"},"emit_ts":1}\n' "$SHA40"
} >"$f"
err="$TMP_ROOT/q1.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q1: render must EXIT 0 (per-entry quarantine, not whole-digest exit-4), got rc=$rc"
printf '%s' "$out" | grep -q "$SHA40" || fail_msg "Q1: the clean sibling (sha $SHA40) must STILL be delivered in the same drain [head-of-line block]"
printf '%s' "$out" | grep -q 'MALFORMED-Q' && fail_msg "Q1: the quarantined entry must be EXCLUDED from the rendered digest"
[ -f "$(dlq "$home")" ] || fail_msg "Q1: a durable dead-letter file must be written"
grep -q 'MALFORMED-Q' "$(dlq "$home")" 2>/dev/null || fail_msg "Q1: the malformed entry must be DEAD-LETTERED (accounted-for, not silently dropped)"
grep -qi 'QUARANTINE' "$err" || fail_msg "Q1: a LOUD per-entry alarm must fire on stderr"
grep -q 'observed_seq=1' "$err" || fail_msg "Q1: the alarm must identify the offending entry (observed_seq=1)"
) && ok
echo "== Q2 (b): reconciler enumeration (reconciled:true) renders ORIENTATION-tier, NO exit-4 =="
(
home="$(fresh_home q2)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q2.jsonl"
# A reconciler enumeration: store class actionable (unchanged) + reconciled marker.
# ADDRESS-FREE on purpose (#944 F1): no path/file/sha/repo+issue — the reconciled
# exemption must be the ONLY thing keeping this entry out of quarantine, so the
# exemption is proven load-bearing AT THE GATE (mutation-killable), not merely at
# the tier label. (Q3's ENUM-C* stay path-bearing: reconciled + valid-locator mix.)
printf '%s\n' '{"observed_seq":5,"class":"actionable","locators":{"kind":"repo","id":"ENUM-B","observed_hash":"cafe1234","reconciled":true},"emit_ts":1}' >"$f"
err="$TMP_ROOT/q2.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q2: a reconciler enumeration must NOT exit-4 (it is ORIENTATION-tier), got rc=$rc"
printf '%s' "$out" | grep -q 'id=ENUM-B' || fail_msg "Q2: the enumeration must render as an ORIENTATION pointer (id=ENUM-B via _locator_line)"
printf '%s' "$out" | grep -q 'CLAIM@seq' && fail_msg "Q2: an enumeration must NOT render as an ACTIONABLE CLAIM@seq"
[ -s "$(dlq "$home")" ] && fail_msg "Q2: an ORIENTATION-tier enumeration must NOT be quarantined/dead-lettered"
true
) && ok
echo "== Q3 (c): TWO DISTINCT enumerations BOTH survive as SEPARATE orientation pointers (anti-collapse) =="
(
home="$(fresh_home q3)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q3.jsonl"
{
printf '%s\n' '{"observed_seq":6,"class":"actionable","locators":{"kind":"repo","id":"ENUM-C1","path":"a.md","observed_hash":"1111","reconciled":true},"emit_ts":1}'
printf '%s\n' '{"observed_seq":7,"class":"actionable","locators":{"kind":"repo","id":"ENUM-C2","path":"b.md","observed_hash":"2222","reconciled":true},"emit_ts":1}'
} >"$f"
err="$TMP_ROOT/q3.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q3: two enumerations must render (exit 0), got rc=$rc"
n="$(printf '%s\n' "$out" | grep -c 'id=ENUM-C[12]' || true)"
[ "$n" = "2" ] || fail_msg "Q3: BOTH distinct enumerations must survive as SEPARATE orientation pointers (expected 2, got $n) — neither coalesced away"
printf '%s' "$out" | grep -q 'id=ENUM-C1' || fail_msg "Q3: enumeration ENUM-C1 must be present"
printf '%s' "$out" | grep -q 'id=ENUM-C2' || fail_msg "Q3: enumeration ENUM-C2 must be present"
[ -s "$(dlq "$home")" ] && fail_msg "Q3: enumerations must NOT be quarantined"
true
) && ok
echo "== Q4: PRESERVED — a genuine malformed ACTIONABLE claim is STILL loud (dead-letter+alarm), never delivered as valid =="
(
home="$(fresh_home q4)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q4.jsonl"
printf '%s\n' '{"observed_seq":9,"class":"actionable","locators":{"kind":"board_file","id":"CLAIM-KEEP","observed_hash":"beef"},"emit_ts":1}' >"$f"
err="$TMP_ROOT/q4.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q4: render must exit 0 (per-entry quarantine), got rc=$rc"
printf '%s' "$out" | grep -q 'CLAIM-KEEP' && fail_msg "Q4: a malformed actionable must NOT be delivered as if valid"
printf '%s' "$out" | grep -q '(none) — no consequential claims pending' || fail_msg "Q4: with the only claim quarantined, the ACTIONABLE section must show (none)"
grep -q 'CLAIM-KEEP' "$(dlq "$home")" 2>/dev/null || fail_msg "Q4: the malformed actionable must be DEAD-LETTERED (loudly surfaced, not silent)"
grep -qi 'QUARANTINE' "$err" || fail_msg "Q4: the malformed actionable must raise a LOUD alarm"
) && ok
echo "== Q5: claim-precedence gated — a non-actionable-class entry carrying a claim (no hard locator) is STILL quarantined =="
(
home="$(fresh_home q5)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q5.jsonl"
# class=digest but locators carry a consequential `claim` -> actionable-tier by
# precedence; no hard locator + no reconciled marker -> must be quarantined.
printf '%s\n' '{"observed_seq":11,"class":"digest","locators":{"claim":"CI is green","kind":"repo","id":"CLAIMY"},"emit_ts":1}' >"$f"
err="$TMP_ROOT/q5.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q5: render must exit 0, got rc=$rc"
grep -q 'CLAIMY' "$(dlq "$home")" 2>/dev/null || fail_msg "Q5: a claim-carrying entry with no hard locator must be quarantined (claim precedence, §2.1)"
printf '%s' "$out" | grep -q 'CI is green' && fail_msg "Q5: the un-verifiable claim must NOT be delivered"
true
) && ok
echo "== Q6 (a): dead-lettered entry routes EXACTLY ONE off-host alarm (payload names observed_seq) + stderr diagnostic STILL fires =="
(
home="$(fresh_home q6)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q6.jsonl"
printf '%s\n' '{"observed_seq":21,"class":"actionable","locators":{"kind":"repo","id":"DLQ-Q6","observed_hash":"aaaa"},"emit_ts":1}' >"$f"
ALARM_OUT="$TMP_ROOT/q6.alarm.jsonl"
export ALARM_OUT
: >"$ALARM_OUT"
export WAKE_ALARM_SINK_CMD="$CAPTURE_ALARM"
err="$TMP_ROOT/q6.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q6: render must still EXIT 0 (per-entry quarantine, not whole-drain wedge), got rc=$rc"
n="$(grep -c . "$ALARM_OUT" 2>/dev/null || true)"
[ "$n" = "1" ] || fail_msg "Q6: EXACTLY ONE off-host alarm must route for the dead-lettered entry (got $n) [$(cat "$ALARM_OUT" 2>/dev/null)]"
grep -q '"observed_seq":21' "$ALARM_OUT" 2>/dev/null || fail_msg "Q6: the routed alarm payload must name the entry's observed_seq (21)"
grep -qi 'QUARANTINE' "$err" || fail_msg "Q6: the existing #920 stderr diagnostic must STILL fire (local + off-host, not either/or)"
printf '%s' "$out" | grep -q 'DLQ-Q6' && fail_msg "Q6: the quarantined entry must still be EXCLUDED from the rendered digest"
true
) && ok
echo "== Q7 (b): re-draining the SAME still-dead-lettered entry N times routes ZERO additional off-host alarms (dedup by observed_seq) =="
(
home="$(fresh_home q7)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q7.jsonl"
printf '%s\n' '{"observed_seq":22,"class":"actionable","locators":{"kind":"repo","id":"DLQ-Q7","observed_hash":"bbbb"},"emit_ts":1}' >"$f"
ALARM_OUT="$TMP_ROOT/q7.alarm.jsonl"
export ALARM_OUT
: >"$ALARM_OUT"
export WAKE_ALARM_SINK_CMD="$CAPTURE_ALARM"
for i in 1 2 3 4; do
"$DIGEST" render --from-file "$f" --agent default >/dev/null 2>"$TMP_ROOT/q7.err.$i"
done
n="$(grep -c . "$ALARM_OUT" 2>/dev/null || true)"
[ "$n" = "1" ] || fail_msg "Q7: re-draining the SAME dead-lettered entry 4x must route ONLY ONE off-host alarm total (durable dedup by observed_seq); got $n [$(cat "$ALARM_OUT" 2>/dev/null)]"
grep -qi 'QUARANTINE' "$TMP_ROOT/q7.err.4" || fail_msg "Q7: the #920 stderr diagnostic must STILL fire on every re-drain (only the off-host route is deduped)"
) && ok
echo "== Q8 (c): a NEW distinct dead-lettered entry routes its OWN one alarm (dedup is per-entry, not a global latch) =="
(
home="$(fresh_home q8)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f1="$TMP_ROOT/q8a.jsonl"
f2="$TMP_ROOT/q8b.jsonl"
printf '%s\n' '{"observed_seq":31,"class":"actionable","locators":{"kind":"repo","id":"DLQ-Q8A","observed_hash":"c1"},"emit_ts":1}' >"$f1"
printf '%s\n' '{"observed_seq":32,"class":"actionable","locators":{"kind":"repo","id":"DLQ-Q8B","observed_hash":"c2"},"emit_ts":1}' >"$f2"
ALARM_OUT="$TMP_ROOT/q8.alarm.jsonl"
export ALARM_OUT
: >"$ALARM_OUT"
export WAKE_ALARM_SINK_CMD="$CAPTURE_ALARM"
"$DIGEST" render --from-file "$f1" --agent default >/dev/null 2>/dev/null
"$DIGEST" render --from-file "$f1" --agent default >/dev/null 2>/dev/null # re-drain seq 31 -> must NOT re-alarm
"$DIGEST" render --from-file "$f2" --agent default >/dev/null 2>/dev/null # NEW distinct seq 32 -> its own alarm
n="$(grep -c . "$ALARM_OUT" 2>/dev/null || true)"
[ "$n" = "2" ] || fail_msg "Q8: two DISTINCT dead-lettered entries must together route exactly 2 off-host alarms total (got $n) [$(cat "$ALARM_OUT" 2>/dev/null)]"
grep -q '"observed_seq":31' "$ALARM_OUT" 2>/dev/null || fail_msg "Q8: seq 31's alarm must be present"
grep -q '"observed_seq":32' "$ALARM_OUT" 2>/dev/null || fail_msg "Q8: seq 32's (the new distinct entry's) OWN alarm must be present"
) && ok
echo "== Q9 (d): WAKE_ALARM_SINK_CMD unconfigured OR unreachable -> FAIL LOUD (never silent no-alarm); per-entry, render still exits 0 =="
(
home="$(fresh_home q9a)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q9.jsonl"
printf '%s\n' '{"observed_seq":41,"class":"actionable","locators":{"kind":"repo","id":"DLQ-Q9","observed_hash":"dddd"},"emit_ts":1}' >"$f"
# (a) UNCONFIGURED alarm sink.
unset WAKE_ALARM_SINK_CMD
err_a="$TMP_ROOT/q9a.err"
out_a="$("$DIGEST" render --from-file "$f" --agent default 2>"$err_a")"
rc_a=$?
[ "$rc_a" -eq 0 ] || fail_msg "Q9a: per-entry quarantine must still exit 0 even when the off-host alarm sink is unconfigured (no whole-drain wedge), got rc=$rc_a"
grep -qi 'FAIL LOUD' "$err_a" || fail_msg "Q9a: an unconfigured off-host alarm target must FAIL LOUD on stderr [$(cat "$err_a")]"
grep -Eqi 'silent no-alarm|silent-miss|PERMANENTLY miss|permanent silent miss' "$err_a" || fail_msg "Q9a: the diagnostic must name the silent-miss hazard (G2a), mirroring beacon.sh's fail-closed wording [$(cat "$err_a")]"
printf '%s' "$out_a" | grep -q 'DLQ-Q9' && fail_msg "Q9a: the quarantined entry must still be EXCLUDED from the rendered digest"
true
) && ok
(
home2="$(fresh_home q9b)"
export WAKE_STATE_HOME="$home2"
unset WAKE_AGENT
f="$TMP_ROOT/q9.jsonl"
export WAKE_ALARM_SINK_CMD="false"
err_b="$TMP_ROOT/q9b.err"
out_b="$("$DIGEST" render --from-file "$f" --agent default 2>"$err_b")"
rc_b=$?
[ "$rc_b" -eq 0 ] || fail_msg "Q9b: per-entry quarantine must still exit 0 even when the off-host alarm sink is unreachable, got rc=$rc_b"
grep -qi 'FAIL LOUD' "$err_b" || fail_msg "Q9b: an unreachable off-host alarm target must FAIL LOUD on stderr [$(cat "$err_b")]"
grep -qi 'UNREACHABLE' "$err_b" || fail_msg "Q9b: the diagnostic must name the unreachable target [$(cat "$err_b")]"
printf '%s' "$out_b" | grep -q 'DLQ-Q9' && fail_msg "Q9b: the quarantined entry must still be EXCLUDED from the rendered digest"
true
) && ok
echo "== Q10: snapshot metadata (#940) — snapshot_sha/snapshot_ts render on the locator line and feed the re-verify hint =="
(
home="$(fresh_home q10)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q10.jsonl"
{
# (a) a digest-class ORIENTATION pointer WITH adapter-attested snapshot
# metadata — the fields must render on the pointer line (the tier where
# the snapshot-lag class actually bit; re-verify hints are actionable-
# tier-only by design and are asserted via (c) below).
printf '%s\n' '{"observed_seq":21,"class":"digest","locators":{"kind":"board_file","id":"SNAP-A","path":"BOARD.md","observed_hash":"aaaa1111","snapshot_sha":"0123abc4567890def0123abc4567890def012345","snapshot_ts":1753850000},"emit_ts":2}'
# (b) a sibling WITHOUT the fields — must render no snapshot vestige.
printf '%s\n' '{"observed_seq":22,"class":"digest","locators":{"kind":"board_file","id":"SNAP-B","path":"OTHER.md","observed_hash":"bbbb2222"},"emit_ts":2}'
# (c) an ACTIONABLE entry (hard locator: repo+issue) carrying path +
# snapshot_sha — its re-verify hint must upgrade to the one-call
# `git show <snapshot_sha>:<path>` (most-specific-first).
printf '%s\n' '{"observed_seq":23,"class":"actionable","locators":{"repo":"example/repo","issue":7,"path":"BOARD.md","observed_hash":"cccc3333","snapshot_sha":"0123abc4567890def0123abc4567890def012345","snapshot_ts":1753850000},"emit_ts":2}'
} >"$f"
out="$("$DIGEST" render --from-file "$f" --agent default 2>/dev/null)"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q10: render must succeed (rc=$rc)"
snapa_line="$(printf '%s\n' "$out" | grep 'id=SNAP-A' || true)"
printf '%s' "$snapa_line" | grep -q 'snapshot_sha=0123abc4567890def0123abc4567890def012345' \
|| fail_msg "Q10: snapshot_sha must render on the ORIENTATION pointer line [$snapa_line]"
printf '%s' "$snapa_line" | grep -q 'snapshot_ts=1753850000' \
|| fail_msg "Q10: snapshot_ts must render on the ORIENTATION pointer line (age = emit_ts - snapshot_ts, local arithmetic) [$snapa_line]"
printf '%s' "$out" | grep -q 'git show 0123abc4567890def0123abc4567890def012345:BOARD.md' \
|| fail_msg "Q10: snapshot_sha+path must upgrade the actionable re-verify hint to a one-call git show"
# The sibling without metadata must not grow empty snapshot_ fields.
snapb_line="$(printf '%s\n' "$out" | grep 'id=SNAP-B' || true)"
printf '%s' "$snapb_line" | grep -q 'snapshot_' \
&& fail_msg "Q10: an entry without metadata must render NO snapshot_ fields [$snapb_line]"
true
) && ok
echo "== Q11 (#944): REAL detector-shape actionable RENDERS as CLAIM@seq; address-free + bare-snapshot_sha siblings STILL quarantine =="
(
home="$(fresh_home q11)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
f="$TMP_ROOT/q11.jsonl"
{
# (a) the LIVE seq-68 entry VERBATIM — the exact production entry that
# dead-lettered on the mos-dt lane under the unsatisfiable gate
# (dragon-lin dead-letter.jsonl, 2026-07-30). Detector-emitted shape,
# not a hand-built dict.
printf '%s\n' '{"observed_seq":68,"locators":{"kind":"board_file","id":"heartbeat-planning","observed_hash":"2e85f2474001961e920976a91981eca5bb86a5ff0df044e082a1e1f2dc7493b5","snapshot_sha":"55d4909569d2b5fbccfec49ec9ca83db5049f3ce","snapshot_ts":1785414523,"path":"docs/scratchpads/heartbeat-planning"},"class":"actionable","emit_ts":1785415057,"hmac":""}'
# (b) same vocabulary WITHOUT snapshot attestation (pre-#940 adapter or
# dropped-as-malformed attestation) — must pass via the path arm alone.
printf '%s\n' '{"observed_seq":69,"class":"actionable","locators":{"kind":"board_file","id":"PILOT-LOCAL","observed_hash":"abcd1234","path":"BOARD.md"},"emit_ts":2}'
# (c) ADDRESS-FREE — the retained negative: a content hash is not an address.
printf '%s\n' '{"observed_seq":70,"class":"actionable","locators":{"kind":"board_file","id":"ADDR-FREE","observed_hash":"ffff0000"},"emit_ts":2}'
# (d) bare path-less snapshot_sha — must NOT pass: the widened gate must
# not widen past the board_file vocabulary. Asserted at all three
# lengths the detector's attestation validation ^[0-9a-f]{7,64}$
# admits: a 40-hex sha-1, a 7-char abbreviation, a 64-char sha-256.
# One length alone cannot distinguish "no arm" from "arm present but
# length-gated" (enumeration finding E1).
printf '%s\n' '{"observed_seq":71,"class":"actionable","locators":{"kind":"board_file","id":"SNAP-ONLY-40","observed_hash":"eeee1111","snapshot_sha":"55d4909569d2b5fbccfec49ec9ca83db5049f3ce"},"emit_ts":2}'
printf '%s\n' '{"observed_seq":72,"class":"actionable","locators":{"kind":"board_file","id":"SNAP-ONLY-7","observed_hash":"eeee2222","snapshot_sha":"55d4909"},"emit_ts":2}'
printf '%s\n' '{"observed_seq":73,"class":"actionable","locators":{"kind":"board_file","id":"SNAP-ONLY-64","observed_hash":"eeee3333","snapshot_sha":"55d4909569d2b5fbccfec49ec9ca83db5049f3ce55d4909569d2b5fbccfec49e"},"emit_ts":2}'
} >"$f"
ALARM_OUT="$TMP_ROOT/q11.alarm.jsonl"
export ALARM_OUT
: >"$ALARM_OUT"
export WAKE_ALARM_SINK_CMD="$CAPTURE_ALARM"
err="$TMP_ROOT/q11.err"
out="$("$DIGEST" render --from-file "$f" --agent default 2>"$err")"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q11: render must exit 0, got rc=$rc"
# (a) POSITIVE: the live entry RENDERS as an actionable claim (not merely
# passes the predicate) with the one-call git-show re-verify hint.
printf '%s' "$out" | grep -q 'seq 68 — CLAIM@seq' || fail_msg "Q11a: the live seq-68 detector-shape entry must RENDER as CLAIM@seq (positive control)"
printf '%s' "$out" | grep -q 'git show 55d4909569d2b5fbccfec49ec9ca83db5049f3ce:docs/scratchpads/heartbeat-planning' \
|| fail_msg "Q11a: the rendered claim must carry the one-call re-verify hint git show <snapshot_sha>:<path>"
# (b) POSITIVE: path arm alone suffices; hint degrades to one-call re-read.
printf '%s' "$out" | grep -q 'seq 69 — CLAIM@seq' || fail_msg "Q11b: a path-only detector-shape entry must RENDER as CLAIM@seq (path arm)"
printf '%s' "$out" | grep -q 're-read BOARD.md' || fail_msg "Q11b: the path-only claim must carry the one-call re-read <path> hint"
n_claims="$(printf '%s\n' "$out" | grep -c 'CLAIM@seq' || true)"
[ "$n_claims" = "2" ] || fail_msg "Q11: EXACTLY the two valid entries must render as CLAIM@seq (got $n_claims)"
# (c)+(d) NEGATIVES retained: both quarantine, each with its OWN alarm.
printf '%s' "$out" | grep -q 'ADDR-FREE' && fail_msg "Q11c: the address-free entry must be EXCLUDED from the digest"
printf '%s' "$out" | grep -q 'SNAP-ONLY' && fail_msg "Q11d: no bare path-less snapshot_sha entry may appear in the digest (gate must not widen past board_file)"
grep -q 'ADDR-FREE' "$(dlq "$home")" 2>/dev/null || fail_msg "Q11c: the address-free entry must be DEAD-LETTERED"
for snap_id in SNAP-ONLY-40 SNAP-ONLY-7 SNAP-ONLY-64; do
grep -q "$snap_id" "$(dlq "$home")" 2>/dev/null || fail_msg "Q11d: the bare snapshot_sha entry ($snap_id) must be DEAD-LETTERED"
done
grep -q 'heartbeat-planning' "$(dlq "$home")" 2>/dev/null && fail_msg "Q11a: the valid live entry must NOT be dead-lettered"
grep -q 'PILOT-LOCAL' "$(dlq "$home")" 2>/dev/null && fail_msg "Q11b: the valid path-only entry must NOT be dead-lettered"
n_alarms="$(grep -c . "$ALARM_OUT" 2>/dev/null || true)"
[ "$n_alarms" = "4" ] || fail_msg "Q11: exactly the four invalid entries must alarm (got $n_alarms) [$(cat "$ALARM_OUT" 2>/dev/null)]"
grep -q '"observed_seq":70' "$ALARM_OUT" 2>/dev/null || fail_msg "Q11c: seq 70's own alarm must be present"
for snap_seq in 71 72 73; do
grep -q "\"observed_seq\":$snap_seq" "$ALARM_OUT" 2>/dev/null || fail_msg "Q11d: seq $snap_seq's own alarm must be present"
done
true
) && ok
echo "== Q12 (#946): quarantined entries are DISCLOSED (by seq, content withheld) and the embedded ack is CLAMPED below them =="
(
home="$(fresh_home q12)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
mkdir -p "$home/default"
printf '2' >"$home/default/observed_seq"
f="$TMP_ROOT/q12.jsonl"
{
printf '{"observed_seq":1,"class":"actionable","locators":{"sha":"%s","file":"src/a.ts"},"emit_ts":1}\n' "$SHA40"
printf '%s\n' '{"observed_seq":2,"class":"actionable","locators":{"kind":"board_file","id":"ADDR-Q12","observed_hash":"qq12"},"emit_ts":1}'
} >"$f"
out="$("$DIGEST" render --from-file "$f" --agent default 2>/dev/null)"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q12: render must exit 0, got rc=$rc"
# DISCLOSURE: a held entry must be VISIBLE in the digest it was held from —
# a silent hold is how five successive digests each stepped past seq 68...
printf '%s' "$out" | grep -q 'QUARANTINED' || fail_msg "Q12: the digest must carry a QUARANTINED disclosure section (no silent hold)"
printf '%s' "$out" | grep -q 'seq 2 .*HELD' || fail_msg "Q12: the disclosure must name the held seq (2) as HELD"
# ...but WITHOUT re-injecting the refused content: disclosure is by seq only;
# the Q1/Q4/Q5/Q6/Q9/Q11 exclusion property stands.
printf '%s' "$out" | grep -q 'ADDR-Q12' && fail_msg "Q12: the quarantined entry's content/locators must stay EXCLUDED from the digest"
# CLAMP: the embedded ack stops BELOW the quarantined seq, and says so loudly.
printf '%s' "$out" | grep -Eq 'consumed --upto 1$' || fail_msg "Q12: the embedded ack must be CLAMPED to --upto 1 (below quarantined seq 2)"
printf '%s' "$out" | grep -Eq 'consumed --upto 2( |$)' && fail_msg "Q12: the raw observed cursor (2) must NOT be embedded while seq 2 is quarantined"
printf '%s' "$out" | grep -q 'ACK CLAMPED' || fail_msg "Q12: the clamp must be LOUDLY disclosed in the ACK section"
# A foreign-data render must NOT rewrite the lane's quarantine truth.
[ -e "$home/default/quarantined.set" ] && fail_msg "Q12: a --from-file render must NOT write the store's quarantined.set (lane truth is store-mode only)"
true
) && ok
echo "== Q13 (#946): nothing quarantined -> ack UNCLAMPED at the observed cursor; no disclosure section, no clamp note =="
(
home="$(fresh_home q13)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
mkdir -p "$home/default"
printf '1' >"$home/default/observed_seq"
f="$TMP_ROOT/q13.jsonl"
printf '{"observed_seq":1,"class":"actionable","locators":{"sha":"%s","file":"src/a.ts"},"emit_ts":1}\n' "$SHA40" >"$f"
out="$("$DIGEST" render --from-file "$f" --agent default 2>/dev/null)"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q13: render must exit 0, got rc=$rc"
printf '%s' "$out" | grep -Eq 'consumed --upto 1$' || fail_msg "Q13: with nothing quarantined the ack must embed the observed cursor (1) unchanged"
printf '%s' "$out" | grep -q 'QUARANTINED' && fail_msg "Q13: no disclosure section when nothing is quarantined"
printf '%s' "$out" | grep -q 'ACK CLAMPED' && fail_msg "Q13: no clamp note when nothing is quarantined"
true
) && ok
echo "== Q14 (#946): store-mode render SYNCS quarantine truth into the store — the clamp is enforced END-TO-END at consume =="
(
home="$(fresh_home q14)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
STORE="$SCRIPT_DIR/store.sh"
"$STORE" enqueue --class actionable --locators "{\"sha\":\"$SHA40\",\"file\":\"src/a.ts\"}" >/dev/null
"$STORE" enqueue --class actionable --locators '{"kind":"board_file","id":"ADDR-Q14","observed_hash":"qq14"}' >/dev/null
out="$("$DIGEST" render --from-store --agent default 2>/dev/null)"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q14: render must exit 0, got rc=$rc"
printf '%s' "$out" | grep -q 'QUARANTINED' || fail_msg "Q14: the store-mode digest must disclose the held entry"
printf '%s' "$out" | grep -Eq 'consumed --upto 1$' || fail_msg "Q14: the embedded ack must clamp to 1 (below quarantined seq 2)"
qf="$home/default/quarantined.set"
[ "$(cat "$qf" 2>/dev/null)" = "2" ] || fail_msg "Q14: a store-mode render must sync quarantined.set to exactly {2}, got [$(cat "$qf" 2>/dev/null)]"
# END-TO-END: even a hand-typed upto past the held seq is refused at the
# store — the copy-run defect (#946) cannot re-land via a different path.
if "$STORE" consume --upto 2 >/dev/null 2>&1; then
fail_msg "Q14: store consume --upto 2 must be REFUSED after the render synced the quarantine"
fi
"$STORE" consume --upto 1 >/dev/null 2>&1 || fail_msg "Q14: consume --upto 1 (the clamped value) must succeed"
) && ok
echo "== Q15 (#946): gate-fix RECOVERY — a stale quarantined.set is REPLACED by a clean store-mode render; the clamp self-heals =="
(
home="$(fresh_home q15)"
export WAKE_STATE_HOME="$home"
unset WAKE_AGENT
STORE="$SCRIPT_DIR/store.sh"
"$STORE" enqueue --class actionable --locators "{\"sha\":\"$SHA40\",\"file\":\"src/a.ts\"}" >/dev/null
"$STORE" enqueue --class actionable --locators '{"kind":"board_file","id":"OK-NOW","observed_hash":"h","path":"BOARD.md"}' >/dev/null
# A stale set from a broken-gate era: both seqs wrongly quarantined.
printf '1\n2\n' >"$home/default/quarantined.set"
out="$("$DIGEST" render --from-store --agent default 2>/dev/null)"
rc=$?
[ "$rc" -eq 0 ] || fail_msg "Q15: render must exit 0, got rc=$rc"
printf '%s' "$out" | grep -q 'QUARANTINED' && fail_msg "Q15: nothing quarantines under the fixed gate — no disclosure section"
printf '%s' "$out" | grep -Eq 'consumed --upto 2$' || fail_msg "Q15: the ack must embed the full observed cursor (2) once the gate is fixed"
[ -s "$home/default/quarantined.set" ] && fail_msg "Q15: the clean render must REPLACE (clear) the stale quarantined.set — a cumulative-forever set would block acks on entries that now render, got [$(cat "$home/default/quarantined.set")]"
"$STORE" consume --upto 2 >/dev/null 2>&1 || fail_msg "Q15: the ordinary consume must succeed after the clamp self-heals"
) && ok
echo "== Q16 (guard): Q2's ENUM-B fixture must STAY address-free — the reconciled exemption must remain load-bearing at the gate (#944 F1) =="
(
self="$SCRIPT_DIR/test-wake-digest-quarantine.sh"
# Token concatenated so THIS guard's own source lines never contain the
# literal fixture id and cannot self-match.
enum_id='ENUM''-B'
fixture_line="$(grep -F "\"id\":\"$enum_id\"" "$self" | grep -F '"observed_seq":5' | head -n1)"
[ -n "$fixture_line" ] || fail_msg "Q16: could not locate Q2's $enum_id fixture line (renamed/renumbered? update this guard)"
fixture_json="$(printf '%s' "$fixture_line" | sed "s/.*'\({.*}\)'.*/\1/")"
# Positive controls FIRST (blind-instrument rule): the extraction must yield
# the real fixture, and the predicate must be able to detect a hard locator.
printf '%s' "$fixture_json" | jq -e . >/dev/null 2>&1 || fail_msg "Q16: extracted fixture is not valid JSON [$fixture_json]"
printf '%s' "$fixture_json" | jq -e '.locators.reconciled == true' >/dev/null 2>&1 || fail_msg "Q16: fixture must carry reconciled:true (wrong line extracted?) [$fixture_json]"
hard_arms='.locators | ((.repo // "") != "" and (((.issue // "") | tostring) != "")) or (((.sha // "") | tostring) | test("^[0-9a-f]{40}$")) or ((.file // "") != "") or ((.path // "") != "")'
printf '%s' '{"locators":{"path":"BOARD.md"}}' | jq -e "$hard_arms" >/dev/null 2>&1 || fail_msg "Q16: positive control failed — the inline hard-locator predicate did not detect a path arm (instrument broken; re-sync it with digest.sh _has_hard_locator)"
# THE GUARD: the fixture must remain ADDRESS-FREE. If it ever gains a hard
# locator, Q2 passes the gate for the wrong reason and the reconciled
# exemption stops being exercised (#944 F1: an exemption nobody exercises is
# as unproven as a gate nobody has seen refuse).
if printf '%s' "$fixture_json" | jq -e "$hard_arms" >/dev/null 2>&1; then
fail_msg "Q16: Q2's $enum_id fixture has grown a hard-locator arm — restore an address-free fixture so the reconciled exemption stays load-bearing [$fixture_json]"
fi
true
) && ok
echo
if [ -s "$FAILFILE" ]; then
echo "wake digest-quarantine harness: FAILED ($(grep -c . "$FAILFILE") assertion(s))" >&2
exit 1
fi
echo "wake digest-quarantine harness: all invariants passed ($pass groups)"
Reference in New Issue View Git Blame Copy Permalink