fix(wake): #946 digest ack watermark clamped at quarantined seqs — disclose AND clamp

The embedded ack suggestion covered dead-lettered entries: quarantine is a
render-time filter (0.6.14), so `ack.sh consumed --upto <observed_seq>`
stepped the cursor past quarantined seqs and _record_last_consumed wrote
consumed-hash witness rows for deliveries that never happened (live: mos-dt
seq 68 buried under five successive digests; Finding A: a false
9d0f639f…@63 witness row).

Fix, per the ruled disposition on #946:

- digest.sh: rendered digest gains a QUARANTINED section (seq + class +
  HELD only — ids and locator values stay withheld, preserving the
  exclusion property); embedded ack clamped to
  min(observed_seq, min quarantined seq - 1) with a loud
  "# ACK CLAMPED (#946)" note; render --from-store syncs the store-owned
  quarantined.set via `store.sh quarantine-sync` (full replace — a gate
  fix self-heals stale quarantine); --from-file/--stdin never touch the
  set.

- store.sh: consume REFUSES to cross an unconsumed quarantined seq;
  `--force-past-quarantine` is the ONLY way past, loud per-seq on stderr,
  and even the forced path never writes a consumed-hash witness for a
  quarantined seq; crossed seqs are pruned from the set after the cursor
  moves. New `quarantine-sync` (stdin seqs, full replace, invalid input is
  a loud no-op) and `quarantine-audit [--repair]` (sweeps consumed-hashes
  for rows provably contradicted by the dead-letter ledger; report exits
  1; --repair removes only provably-false rows; the ledger is history and
  never modified; rows whose dead-letter evidence was pruned are
  unprovable and untouched).

- ack.sh: plumbs --force-past-quarantine through to store consume; forced-
  path loudness is re-emitted on stderr while `CONSUMED <n>` stays clean.

Tests: test-wake-store-ack.sh T13-T16 (34 assertions RED at base),
test-wake-digest-quarantine.sh Q12-Q16 (10 RED at base; Q13/Q16
green-by-design controls). All nine wake suites green. version=0.6.15.

Closes #946

Written-by: pepper (sb-it-1-dt)
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NsKce8iZuSuRnu3gVMCBKB

packages/mosaic/framework/tools/wake/store.sh

@@ -77,11 +77,42 @@ Commands:
                                          the actual paste is out of scope.
                                          If --require-idle-cmd is given and it
                                          exits non-zero, emit nothing (not idle).
-  consume --upto N                       Advance consumed_seq over the contiguous
+  consume --upto N [--force-past-quarantine]
+                                         Advance consumed_seq over the contiguous
                                          gapless prefix <=N; drop consumed
                                          entries. Rejects a gap (cannot ack N
                                          while N-1 is unconsumed). Cumulative &
-                                         idempotent.
+                                         idempotent. REFUSES to advance past a
+                                         QUARANTINED seq (#946): a quarantined
+                                         entry was dead-lettered at render and
+                                         never delivered in any digest, so an
+                                         ordinary ack may not record it consumed.
+                                         --force-past-quarantine is the ONLY way
+                                         past — loud per stepped-over seq, and
+                                         even then NO consumed-hash witness is
+                                         recorded for the quarantined entry.
+  quarantine-sync                        REPLACE the store-owned quarantined.set
+                                         with the observed_seqs read from stdin
+                                         (one per line; empty input CLEARS).
+                                         Called by digest.sh after each
+                                         authoritative store render — the set is
+                                         re-DERIVED per render, never accumulated,
+                                         so a fixed locator gate self-heals the
+                                         consume clamp (#944 recovery).
+  quarantine-audit [--repair]            Report consumed-hashes rows that are
+                                         PROVABLY FALSE: the row matches a
+                                         dead-letter ledger entry on
+                                         (kind,id,observed_seq,observed_hash) at
+                                         or below consumed_seq — i.e. the recorded
+                                         consumption was of a quarantined,
+                                         never-delivered entry (#946 Finding A).
+                                         Report-only by default (exit 1 when any
+                                         found); --repair removes exactly those
+                                         rows (atomic, loud). The dead-letter
+                                         ledger itself is NEVER modified (it is
+                                         history). Rows whose dead-letter evidence
+                                         was pruned are NOT provable and are
+                                         never touched.
   cursors                                Print observed_seq / consumed_seq / depth.
 
 Environment:
@@ -354,10 +385,20 @@ cmd_drain() {
 # (the reconciler re-enumerates the consumed state once — no lost obligation),
 # never a failed consume.
 _record_last_consumed() {
-  local upto="$1" existing new_records merged
+  local upto="$1" existing new_records merged qjson
   [ -f "$STATE_DIR/pending.jsonl" ] || return 0
-  new_records="$(jq -c --argjson upto "$upto" '
-    select((.observed_seq // -1) <= $upto)
+  # #946: seqs in quarantined.set are EXCLUDED from the record — the trust
+  # boundary above says "existence implies durably enqueued+CONSUMED", but a
+  # quarantined entry was dead-lettered at render and NEVER delivered, so a row
+  # for it would witness a delivery that never happened. This holds even on the
+  # FORCED step-over path: the reconciler re-enumerating the state once is
+  # safe-but-noisy; a false witness silences it forever.
+  qjson="$(jq -nR -c '[inputs | select(length > 0) | tonumber? // empty]' "$STATE_DIR/quarantined.set" 2>/dev/null || true)"
+  [ -n "$qjson" ] || qjson='[]'
+  new_records="$(jq -c --argjson upto "$upto" --argjson quarantined "$qjson" '
+    (.observed_seq // -1) as $seq
+    | select($seq <= $upto)
+    | select(($quarantined | index($seq)) == null)
     | select((.locators.kind // "") != "" and (.locators.id // "") != "" and (.locators.observed_hash // "") != "")
     | {kind:.locators.kind, id:.locators.id, observed_hash:.locators.observed_hash, observed_seq:.observed_seq}
   ' "$STATE_DIR/pending.jsonl" 2>/dev/null || true)"
@@ -372,13 +413,17 @@ _record_last_consumed() {
 }
 
 cmd_consume() {
-  local upto=''
+  local upto='' force=0
   while [ $# -gt 0 ]; do
     case "$1" in
     --upto)
       upto="${2:-}"
       shift 2
       ;;
+    --force-past-quarantine)
+      force=1
+      shift
+      ;;
     *)
       echo "store.sh consume: unknown option '$1'" >&2
       exit 2
@@ -426,6 +471,33 @@ cmd_consume() {
     k=$((k + 1))
   done
 
+  # --- #946 quarantine CLAMP -------------------------------------------------
+  # A quarantined seq was DEAD-LETTERED at render (no §2.1 hard locator): it was
+  # never delivered in any digest, so advancing consumed_seq past it would record
+  # consumption of an entry the consumer has never seen. The ordinary path
+  # REFUSES; --force-past-quarantine is the ONLY way past, and it is loud per
+  # stepped-over seq. digest.sh re-derives the set at each authoritative store
+  # render (quarantine-sync REPLACE), so a fixed locator gate self-heals this
+  # clamp without operator action.
+  local qfile="$STATE_DIR/quarantined.set" blocked='' q
+  if [ -s "$qfile" ]; then
+    while IFS= read -r q; do
+      case "$q" in '' | *[!0-9]*) continue ;; esac
+      if [ "$q" -gt "$consumed" ] && [ "$q" -le "$upto" ]; then
+        blocked="$blocked $q"
+      fi
+    done <"$qfile"
+  fi
+  if [ -n "$blocked" ]; then
+    if [ "$force" -eq 0 ]; then
+      echo "store.sh consume: REFUSED (#946 quarantine clamp) — quarantined seq(s):$blocked inside (consumed_seq=$consumed, upto=$upto] were dead-lettered at render and NEVER delivered in any digest. Advancing past them would record consumption of entries the consumer has never seen. Disposition them (see $STATE_DIR/dead-letter.jsonl) or step over EXPLICITLY with --force-past-quarantine." >&2
+      exit 1
+    fi
+    for q in $blocked; do
+      echo "store.sh consume: FORCED PAST QUARANTINE (#946) — stepping consumed_seq over quarantined seq $q (dead-lettered, NEVER delivered). No consumed-hash witness is recorded for it; the obligation stays visible ONLY in $STATE_DIR/dead-letter.jsonl." >&2
+    done
+  fi
+
   # #932: record the last-consumed observed_hash per (kind,id) BEFORE the pending
   # prefix is dropped (this reads the entries about to be truncated), so the
   # reconciler can recognise an already-consumed state as ACCOUNTED instead of
@@ -439,9 +511,137 @@ cmd_consume() {
   awk -v c="$upto" 'NF && $1+0 > c' "$STATE_DIR/observed.set" 2>/dev/null |
     _atomic_write "$STATE_DIR/observed.set" || true
   printf '%s' "$upto" | _atomic_write "$STATE_DIR/consumed_seq"
+  # #946: on a forced step-over, PRUNE the stepped-over seqs from quarantined.set
+  # (they are inside the consumed prefix now; a stale entry would re-refuse the
+  # next consume forever). Mirrors the observed.set prune idiom above.
+  if [ -n "$blocked" ]; then
+    awk -v c="$upto" 'NF && $1+0 > c' "$qfile" 2>/dev/null |
+      _atomic_write "$qfile" || true
+  fi
   echo "$upto"
 }
 
+# cmd_quarantine_sync — #946: REPLACE the store-owned quarantined.set with the
+# observed_seqs read from stdin (one per line). Called by digest.sh after each
+# AUTHORITATIVE full-set render (src=store): the set is re-DERIVED per render,
+# never accumulated, so a fixed locator gate self-heals the consume clamp (the
+# #944 recovery case — a cumulative-forever set would keep refusing acks on
+# entries that now render). Empty input CLEARS the set. Foreign-data renders
+# (--from-file/--stdin) never call this. Atomic write; invalid input is refused
+# loudly with the set left untouched.
+cmd_quarantine_sync() {
+  [ $# -eq 0 ] || {
+    echo "store.sh quarantine-sync: takes no options (observed_seqs on stdin, one per line)" >&2
+    exit 2
+  }
+  local seq list=''
+  while IFS= read -r seq; do
+    [ -n "$seq" ] || continue
+    case "$seq" in
+    *[!0-9]*)
+      echo "store.sh quarantine-sync: invalid observed_seq '$seq' — one non-negative integer per line; set left untouched" >&2
+      exit 2
+      ;;
+    esac
+    list="$list$seq"$'\n'
+  done
+  _wake_init_dir "$STATE_DIR"
+  if ! { printf '%s' "$list" | grep -v '^[[:space:]]*$' || true; } | sort -n | uniq | _atomic_write "$STATE_DIR/quarantined.set"; then
+    echo "store.sh quarantine-sync: quarantined.set write FAILED — the consume clamp may be stale for this lane" >&2
+    exit 1
+  fi
+}
+
+# cmd_quarantine_audit — #946 Finding A: the pre-#946 consume recorded
+# consumed-hash rows for QUARANTINED (never-delivered) entries — false
+# witnesses that silence the reconciler for keys whose only "consumption" was a
+# dead-lettered entry (live: safe by population only where the key re-emitted
+# and a later seq won the per-key max_by merge; keys that never recurred keep
+# the false row forever).
+#
+# A row is PROVABLY FALSE iff the dead-letter ledger contains an entry matching
+# it on (kind, id, observed_seq, observed_hash) AND row.observed_seq <=
+# consumed_seq: the per-key max_by merge means the surviving row's provenance IS
+# that quarantined entry (a healed row differs in seq/hash and never matches).
+# PROVABILITY BOUND: a row whose dead-letter evidence was pruned/rotated away is
+# NOT provable and is never touched — this audit only ever removes what the
+# ledger can convict. The dead-letter ledger itself is history and is NEVER
+# modified here.
+cmd_quarantine_audit() {
+  local repair=0
+  while [ $# -gt 0 ]; do
+    case "$1" in
+    --repair)
+      repair=1
+      shift
+      ;;
+    *)
+      echo "store.sh quarantine-audit: unknown option '$1'" >&2
+      exit 2
+      ;;
+    esac
+  done
+  _need_jq
+  _wake_init_dir "$STATE_DIR"
+  local rec="$STATE_DIR/consumed-hashes.jsonl" dlf="$STATE_DIR/dead-letter.jsonl"
+  if [ ! -s "$rec" ]; then
+    echo "store.sh quarantine-audit: OK — no consumed-hashes record to audit"
+    return 0
+  fi
+  local dl
+  dl="$(jq -s -c '[.[] | {kind: (.locators.kind // ""), id: ((.locators.id // "") | tostring), observed_seq: (.observed_seq // -1), observed_hash: (.locators.observed_hash // "")}]' "$dlf" 2>/dev/null || true)"
+  [ -n "$dl" ] || dl='[]'
+  local consumed
+  consumed="$(_wake_read_int "$STATE_DIR/consumed_seq" 0)"
+  local false_rows
+  false_rows="$(jq -c --argjson dl "$dl" --argjson consumed "$consumed" '
+    . as $row
+    | select(($row.observed_seq // -1) <= $consumed)
+    | select(($dl | map(select(
+        .kind == ($row.kind // "")
+        and .id == (($row.id // "") | tostring)
+        and .observed_seq == ($row.observed_seq // -1)
+        and .observed_hash == ($row.observed_hash // "")
+      )) | length) > 0)
+  ' "$rec" 2>/dev/null || true)"
+  if [ -z "$false_rows" ]; then
+    echo "store.sh quarantine-audit: OK — no provably-false consumed-hash rows (rows without surviving dead-letter evidence are not provable and were not judged)"
+    return 0
+  fi
+  local n row
+  n="$(printf '%s\n' "$false_rows" | grep -c . || true)"
+  while IFS= read -r row; do
+    [ -n "$row" ] || continue
+    if [ "$repair" -eq 1 ]; then
+      echo "store.sh quarantine-audit: REPAIR — removing FALSE WITNESS row $row (matches a dead-lettered, never-delivered entry at/below consumed_seq=$consumed)" >&2
+    else
+      echo "FALSE WITNESS — consumed-hashes row $row matches a dead-lettered, never-delivered entry at/below consumed_seq=$consumed (the recorded consumption never happened)"
+    fi
+  done <<<"$false_rows"
+  if [ "$repair" -eq 0 ]; then
+    echo "store.sh quarantine-audit: $n provably-false row(s) found — run with --repair to remove exactly these rows"
+    return 1
+  fi
+  local kept
+  kept="$(jq -c --argjson dl "$dl" --argjson consumed "$consumed" '
+    . as $row
+    | select(
+        (($row.observed_seq // -1) > $consumed)
+        or (($dl | map(select(
+          .kind == ($row.kind // "")
+          and .id == (($row.id // "") | tostring)
+          and .observed_seq == ($row.observed_seq // -1)
+          and .observed_hash == ($row.observed_hash // "")
+        )) | length) == 0)
+      )
+  ' "$rec" 2>/dev/null || true)"
+  if ! { printf '%s\n' "$kept" | grep -v '^[[:space:]]*$' || true; } | _atomic_write "$rec"; then
+    echo "store.sh quarantine-audit: consumed-hashes rewrite FAILED — record left untouched" >&2
+    exit 1
+  fi
+  echo "store.sh quarantine-audit: repaired — removed $n provably-false row(s); dead-letter ledger untouched (history)"
+}
+
 cmd_cursors() {
   _wake_init_dir "$STATE_DIR"
   local observed consumed depth
@@ -463,6 +663,8 @@ main() {
   enqueue) cmd_enqueue "$@" ;;
   drain) cmd_drain "$@" ;;
   consume) cmd_consume "$@" ;;
+  quarantine-sync) cmd_quarantine_sync "$@" ;;
+  quarantine-audit) cmd_quarantine_audit "$@" ;;
   cursors) cmd_cursors "$@" ;;
   -h | --help | help) usage ;;
   *)