Closes #946. The digest omitted a quarantined entry's claim from disclosure while still advancing the ack watermark it instructed the consumer to run — converting a fail-safe HOLD into a silent DISCARD, through the documented normal path. Measured: the burial instruction was re-issued FIVE times, four fresh digests plus one system-initiated redelivery fired purely because the entry had gone unconsumed for 1826s. That redelivery is the proof of the 'indefinitely' half: the mechanism re-asserted itself with no new information. SCOPE — this was NOT a missing check in the consume path. Measured before the fix: dead-letter occurrences were digest.sh 27, store.sh 0, ack.sh 0, detector.sh 0, reconcile.sh 0. Quarantine was owned ENTIRELY by the renderer; the store that advances the watermark had zero knowledge the ledger existed, so an entry could be quarantined by one subsystem and consumed by another with no possible interaction. The fix is therefore a deliberate cross-module decision — option (b), quarantine recorded into a store-owned file, preserving the existing direction of dependency — pre-registered by the consumer before any diff existed. VERIFICATION - Pipeline 2107 terminal SUCCESS ate11bc6622, read clone-inclusive from the provider API rather than through `pipeline-status.sh` (which filters `.type != "clone"` per workflow and would hide a clone failure behind an all-green table): 9/9 children success, exit 0 each, no non-success member. Its `test` step runs all nine wake harnesses via turbo -> packages/mosaic `test` -> `test:framework-shell`. - Independent review by the consumer on the affected lane: eleven pre-registered acceptance checks, authored and delivered BEFORE the diff was read — the file list deliberately unlooked-at, because a filename alone would have disclosed which option was chosen. All eleven resolved, no blocker. - The check that decides it: a RAW `ack.sh consumed --upto N` with no digest involved must refuse to advance past a quarantined seq — the case an agent hits when a digest is MISSED, and the one that would have sunk a disclosure-only fix. Covered at the head as a named assertion (T13 ordinary-path bypass), written independently of the reviewer's list. - Mutation: one asserted site disabled -> TWELVE assertions die, every one BEHAVIOURAL, ZERO count assertions, including one killing across the module boundary the fix spans. - RED control at basea6b5f6a: 34 and 10 assertions fail, matching the body exactly. - Coordinator re-verify by a different instrument than the reviewer used: static reference counts across the base/head boundary — store.sh 0 -> 49, ack.sh 0 -> 6, `--agent` unchanged at 4 (so #949 correctly stayed out). A fix present-but-inert passes the count and fails the mutation; a fix behaviourally correct but smuggling #949 passes the mutation and fails the count. Neither result is reachable by repeating the other. KNOWN RESIDUALS - The `consumed-hashes` repair criterion is met only for keys that RE-EMIT. A corrupted row whose key never recurs stays false indefinitely; the sweep covers those, and the known-false row named in the acceptance criteria had already self-healed by re-emission rather than by design — safe by population, not by design. - The audit's clean-sweep message names one unprovable class; a second exists (a surviving dead-letter row with an empty `observed_hash` cannot be convicted either). Wording, not logic. Filed separately. - The audit's provability bound makes dead-letter RETENTION load-bearing for auditability. Nothing prunes it today, so this is latent — but any future rotation or size cap silently converts provable rows into unprovable ones with no signal at either end. This is not a defect; it is a property that BECAME load-bearing and is recorded nowhere. Filed separately. - `test-wake-detector.sh` D4 fails at this head AND identically at base, with an empty diff over detector files — pre-existing, tracked, not introduced here. Authored by pepper (sb-it-1-dt); reviewed independently by mos-dt (sb-it-1-dt). The mos-dt-0 commit and fork identity does not identify the author — attribution collapse tracked separately. Co-authored-by: mos-dt-0 <[email protected]>
This commit was merged in pull request #951.
This commit is contained in:
@@ -19,6 +19,15 @@
|
||||
# T10 concurrency: two concurrent enqueues get DISTINCT seqs (lock) (#908)
|
||||
# T12 consume records the last-consumed observed_hash per (kind,id) into the
|
||||
# store-owned record (additive; monotonic last-seq wins; lazily created) (#932)
|
||||
# T13 #946 quarantine CLAMP: ordinary consume (store + ack wrapper) REFUSES to
|
||||
# advance past a quarantined seq; the refusal names the seq + the force flag
|
||||
# T14 #946 forced step-over: --force-past-quarantine advances LOUDLY, prunes the
|
||||
# set, and NEVER fabricates a consumed-hash row for the quarantined entry
|
||||
# T15 #946 quarantine-sync: full REPLACE semantics (sorted/deduped; empty input
|
||||
# CLEARS — the clamp self-heals once the gate is fixed; invalid input refused)
|
||||
# T16 #946 quarantine-audit: consumed-hashes rows provably false against the
|
||||
# dead-letter ledger are reported (exit 1) and removed only under --repair;
|
||||
# healed rows and the ledger itself are untouched
|
||||
#
|
||||
# Isolated: every test runs against a fresh WAKE_STATE_HOME temp dir.
|
||||
set -uo pipefail
|
||||
@@ -521,6 +530,156 @@ echo "== T12: #932 — consume records the last-consumed observed_hash per (kind
|
||||
"$STORE" cursors | grep -q 'consumed_seq=3' || fail_msg "T12: consumed_seq must be 3 after CONSUMED 3"
|
||||
) && ok
|
||||
|
||||
echo "== T13: #946 — ordinary consume REFUSES to advance past a quarantined seq (store + ack paths) =="
|
||||
(
|
||||
WAKE_STATE_HOME="$(fresh_state t13)"
|
||||
export WAKE_STATE_HOME
|
||||
unset WAKE_AGENT
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"a","observed_hash":"HA"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"b","observed_hash":"HB"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"c","observed_hash":"HC"}' >/dev/null
|
||||
printf '2\n' | "$STORE" quarantine-sync || fail_msg "T13: quarantine-sync must accept a valid seq list"
|
||||
# A quarantined seq was dead-lettered at render and NEVER delivered in any
|
||||
# digest; the ordinary path must REFUSE to record it consumed (#946: a force
|
||||
# flag the ordinary path can bypass is decoration).
|
||||
if "$STORE" consume --upto 3 >/dev/null 2>&1; then
|
||||
fail_msg "T13: ordinary consume --upto 3 must be REFUSED while seq 2 is quarantined"
|
||||
fi
|
||||
err="$("$STORE" consume --upto 3 2>&1 >/dev/null || true)"
|
||||
echo "$err" | grep -q 'quarantined seq(s): 2' || fail_msg "T13: the refusal must NAME the quarantined seq [$err]"
|
||||
echo "$err" | grep -q -- '--force-past-quarantine' || fail_msg "T13: the refusal must NAME the force flag [$err]"
|
||||
cur="$("$STORE" cursors)"
|
||||
echo "$cur" | grep -q 'consumed_seq=0' || fail_msg "T13: a refused consume must NOT advance the cursor [$cur]"
|
||||
# BELOW the quarantined seq the ordinary path is unaffected.
|
||||
"$STORE" consume --upto 1 >/dev/null 2>&1 || fail_msg "T13: consume --upto 1 (below the quarantined seq) must succeed"
|
||||
# The ack wrapper propagates the refusal — no ordinary-path bypass exists.
|
||||
if "$ACK" consumed --upto 3 --no-sync >/dev/null 2>&1; then
|
||||
fail_msg "T13: ack.sh consumed --upto 3 must be REFUSED while seq 2 is quarantined (ordinary-path bypass)"
|
||||
fi
|
||||
cur="$("$STORE" cursors)"
|
||||
echo "$cur" | grep -q 'consumed_seq=1' || fail_msg "T13: cursor must still be 1 after the refused ack [$cur]"
|
||||
) && ok
|
||||
|
||||
echo "== T14: #946 — FORCED step-over is LOUD, prunes the set, and NEVER fabricates a consumed-hash row for the quarantined entry =="
|
||||
(
|
||||
WAKE_STATE_HOME="$(fresh_state t14)"
|
||||
export WAKE_STATE_HOME
|
||||
unset WAKE_AGENT
|
||||
rec="$WAKE_STATE_HOME/default/consumed-hashes.jsonl"
|
||||
qf="$WAKE_STATE_HOME/default/quarantined.set"
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"a","observed_hash":"HA"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"b","observed_hash":"HB"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"c","observed_hash":"HC"}' >/dev/null
|
||||
printf '2\n' | "$STORE" quarantine-sync || fail_msg "T14: quarantine-sync failed"
|
||||
errf="$TMP_ROOT/t14.err"
|
||||
out="$("$STORE" consume --upto 3 --force-past-quarantine 2>"$errf")"
|
||||
rc=$?
|
||||
[ "$rc" -eq 0 ] || fail_msg "T14: forced consume must succeed (rc=$rc) [$(cat "$errf")]"
|
||||
[ "$out" = "3" ] || fail_msg "T14: forced consume must print the new cursor 3, got '$out'"
|
||||
grep -q 'FORCED PAST QUARANTINE' "$errf" || fail_msg "T14: the forced path must be LOUD on stderr [$(cat "$errf")]"
|
||||
grep -q 'seq 2' "$errf" || fail_msg "T14: the forced-path diagnostic must name the stepped-over seq 2 [$(cat "$errf")]"
|
||||
"$STORE" cursors | grep -q 'consumed_seq=3' || fail_msg "T14: forced consume must advance the cursor to 3"
|
||||
# NO FALSE WITNESS: the quarantined entry (repo/b) was NEVER delivered, so no
|
||||
# consumed-hash row may exist for it — even on the forced path (the reconciler
|
||||
# re-enumerating it once is safe-but-noisy; a false witness silences it
|
||||
# forever). Its delivered siblings' rows must exist.
|
||||
jq_any "$rec" '.kind=="repo" and .id=="a" and .observed_hash=="HA"' || fail_msg "T14: the delivered sibling repo/a must have its consumed-hash row"
|
||||
jq_any "$rec" '.kind=="repo" and .id=="c" and .observed_hash=="HC"' || fail_msg "T14: the delivered sibling repo/c must have its consumed-hash row"
|
||||
jq_any "$rec" '.kind=="repo" and .id=="b"' && fail_msg "T14: the quarantined entry repo/b must have NO consumed-hash row (a row would witness a delivery that never happened)"
|
||||
# The stepped-over seq is PRUNED from the set (it is consumed now; a stale
|
||||
# entry would re-refuse forever).
|
||||
grep -qxF '2' "$qf" 2>/dev/null && fail_msg "T14: seq 2 must be PRUNED from quarantined.set after the forced step-over"
|
||||
# The ack wrapper's force flag passes through, stays LOUD on stderr, and
|
||||
# still reports a CLEAN cursor line on stdout.
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"d","observed_hash":"HD"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"e","observed_hash":"HE"}' >/dev/null
|
||||
printf '5\n' | "$STORE" quarantine-sync || fail_msg "T14: quarantine-sync (2nd) failed"
|
||||
errf2="$TMP_ROOT/t14b.err"
|
||||
out2="$("$ACK" consumed --upto 5 --no-sync --force-past-quarantine 2>"$errf2")"
|
||||
rc2=$?
|
||||
[ "$rc2" -eq 0 ] || fail_msg "T14: forced ack must succeed (rc=$rc2) [$(cat "$errf2")]"
|
||||
echo "$out2" | grep -q '^CONSUMED 5$' || fail_msg "T14: forced ack must report a CLEAN cursor line 'CONSUMED 5', got '$out2'"
|
||||
grep -q 'FORCED PAST QUARANTINE' "$errf2" || fail_msg "T14: the forced-path loudness must survive the ack wrapper (stderr) [$(cat "$errf2")]"
|
||||
jq_any "$rec" '.kind=="repo" and .id=="e"' && fail_msg "T14: the quarantined repo/e must have NO consumed-hash row via the forced ack path either"
|
||||
true
|
||||
) && ok
|
||||
|
||||
echo "== T15: #946 — quarantine-sync is a full REPLACE (sorted, deduped; empty input CLEARS; invalid input REFUSED) =="
|
||||
(
|
||||
WAKE_STATE_HOME="$(fresh_state t15)"
|
||||
export WAKE_STATE_HOME
|
||||
unset WAKE_AGENT
|
||||
qf="$WAKE_STATE_HOME/default/quarantined.set"
|
||||
printf '3\n1\n3\n' | "$STORE" quarantine-sync || fail_msg "T15: sync of a valid list must succeed"
|
||||
[ "$(cat "$qf" 2>/dev/null)" = "$(printf '1\n3')" ] || fail_msg "T15: set must be sorted+deduped {1,3}, got [$(cat "$qf" 2>/dev/null)]"
|
||||
printf '2\n' | "$STORE" quarantine-sync || fail_msg "T15: re-sync must succeed"
|
||||
[ "$(cat "$qf" 2>/dev/null)" = "2" ] || fail_msg "T15: sync must REPLACE, not merge — expected {2}, got [$(cat "$qf" 2>/dev/null)]"
|
||||
# Empty input CLEARS the set: the set is re-DERIVED per authoritative render,
|
||||
# never accumulated, so a fixed locator gate self-heals the clamp.
|
||||
: | "$STORE" quarantine-sync || fail_msg "T15: empty sync (clear) must succeed"
|
||||
[ ! -s "$qf" ] || fail_msg "T15: empty sync must CLEAR the set, got [$(cat "$qf")]"
|
||||
# Invalid input is refused loudly and must not corrupt the set.
|
||||
printf '1\n' | "$STORE" quarantine-sync || fail_msg "T15: re-seed failed"
|
||||
if printf 'abc\n' | "$STORE" quarantine-sync >/dev/null 2>&1; then
|
||||
fail_msg "T15: a non-integer line must be REFUSED"
|
||||
fi
|
||||
[ "$(cat "$qf" 2>/dev/null)" = "1" ] || fail_msg "T15: a refused sync must leave the set untouched, got [$(cat "$qf" 2>/dev/null)]"
|
||||
# End-to-end: a cleared set stops clamping (the #944 recovery case).
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"x","observed_hash":"H1"}' >/dev/null
|
||||
if "$STORE" consume --upto 1 >/dev/null 2>&1; then
|
||||
fail_msg "T15: consume --upto 1 must be refused while seq 1 is quarantined"
|
||||
fi
|
||||
: | "$STORE" quarantine-sync || fail_msg "T15: clear failed"
|
||||
"$STORE" consume --upto 1 >/dev/null 2>&1 || fail_msg "T15: after the set is cleared (gate fixed), the ordinary consume must succeed — the clamp must self-heal"
|
||||
) && ok
|
||||
|
||||
echo "== T16: #946 — quarantine-audit: a consumed-hash row matching a dead-letter entry on (kind,id,seq,hash) at/below consumed_seq is PROVABLY FALSE; --repair removes ONLY those rows =="
|
||||
(
|
||||
WAKE_STATE_HOME="$(fresh_state t16)"
|
||||
export WAKE_STATE_HOME
|
||||
unset WAKE_AGENT
|
||||
STATE_DIR="$WAKE_STATE_HOME/default"
|
||||
rec="$STATE_DIR/consumed-hashes.jsonl"
|
||||
dl="$STATE_DIR/dead-letter.jsonl"
|
||||
# Rebuild the historical false-witness state via the REAL flow the defect
|
||||
# used: X@1 was quarantined (dead-lettered) yet consumed under pre-#946 code;
|
||||
# Y@2 is clean; Z re-emitted (dead-lettered at seq 3, healed by seq 4 winning
|
||||
# the per-key max_by merge — Finding A's live-canary shape).
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"X","observed_hash":"HX"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"Y","observed_hash":"HY"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"Z","observed_hash":"HZ-OLD"}' >/dev/null
|
||||
"$STORE" enqueue --class actionable --locators '{"kind":"repo","id":"Z","observed_hash":"HZ-NEW"}' >/dev/null
|
||||
{
|
||||
printf '%s\n' '{"observed_seq":1,"locators":{"kind":"repo","id":"X","observed_hash":"HX"},"class":"actionable","emit_ts":1,"hmac":""}'
|
||||
printf '%s\n' '{"observed_seq":3,"locators":{"kind":"repo","id":"Z","observed_hash":"HZ-OLD"},"class":"actionable","emit_ts":1,"hmac":""}'
|
||||
} >"$dl"
|
||||
# Pre-#946-shaped consume: NO quarantined.set exists, so this consume writes
|
||||
# the false witness for X@1 exactly as the live defect did.
|
||||
"$STORE" consume --upto 4 >/dev/null 2>&1 || fail_msg "T16: baseline consume failed"
|
||||
jq_any "$rec" '.id=="X" and .observed_seq==1' || fail_msg "T16: fixture broken — the false X@1 row was not written"
|
||||
# REPORT: exactly the X row is provably false; non-zero exit signals findings.
|
||||
rep="$TMP_ROOT/t16.rep"
|
||||
if "$STORE" quarantine-audit >"$rep" 2>&1; then
|
||||
fail_msg "T16: report-mode audit must exit NON-ZERO when false rows exist"
|
||||
fi
|
||||
grep -q 'FALSE WITNESS' "$rep" || fail_msg "T16: the audit must name the false row loudly [$(cat "$rep")]"
|
||||
grep -q '"id":"X"' "$rep" || fail_msg "T16: the audit must identify the false row (repo/X@1) [$(cat "$rep")]"
|
||||
grep -q '"id":"Y"' "$rep" && fail_msg "T16: the clean row repo/Y must NOT be flagged"
|
||||
grep -q '"id":"Z"' "$rep" && fail_msg "T16: the HEALED row repo/Z@4 must NOT be flagged (its dead-letter evidence is seq 3 with a different hash)"
|
||||
# Report mode modifies nothing.
|
||||
jq_any "$rec" '.id=="X"' || fail_msg "T16: report mode must not modify the record"
|
||||
# REPAIR: exactly the false row is removed; the dead-letter LEDGER is history
|
||||
# and must never be modified.
|
||||
"$STORE" quarantine-audit --repair >"$TMP_ROOT/t16.fix" 2>&1 || fail_msg "T16: --repair must succeed [$(cat "$TMP_ROOT/t16.fix")]"
|
||||
jq_any "$rec" '.id=="X"' && fail_msg "T16: --repair must REMOVE the provably-false X row"
|
||||
jq_any "$rec" '.id=="Y" and .observed_hash=="HY"' || fail_msg "T16: --repair must keep the clean Y row"
|
||||
jq_any "$rec" '.id=="Z" and .observed_hash=="HZ-NEW" and .observed_seq==4' || fail_msg "T16: --repair must keep the healed Z@4 row"
|
||||
[ "$(grep -c . "$dl")" = "2" ] || fail_msg "T16: the dead-letter LEDGER must be untouched by --repair"
|
||||
# Clean re-audit: OK, exit 0.
|
||||
"$STORE" quarantine-audit >"$TMP_ROOT/t16.ok" 2>&1 || fail_msg "T16: a clean audit must exit 0 [$(cat "$TMP_ROOT/t16.ok")]"
|
||||
grep -qi 'OK' "$TMP_ROOT/t16.ok" || fail_msg "T16: a clean audit must say OK [$(cat "$TMP_ROOT/t16.ok")]"
|
||||
) && ok
|
||||
|
||||
echo
|
||||
if [ -s "$FAILFILE" ]; then
|
||||
echo "wake store/ack harness: FAILED ($(grep -c . "$FAILFILE") assertion(s))" >&2
|
||||
|
||||
Reference in New Issue
Block a user