fix(wake): #932 stop reconciler re-enumerating already-CONSUMED detector state
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Wake-pilot finding #7 (safe-but-noisy G2a alarm-hygiene): after consume-truncation a consumed detector-observed state matched NO accounting record — the inbox was truncated, the reconciler's seen-ledger only covers its OWN enumerations, and the detector hash-file is correctly DISTRUSTED — so the reconciler treated it as UNACCOUNTED and re-enumerated it: one DUPLICATE orientation wake + one SPURIOUS rc=1 CRITICAL per detector-active window per cycle (functionally safe, no lost obligation, but cry-wolf erosion at fleet scale). Fix (Mos ruling, built exactly): 1. store.sh records the last-consumed observed_hash per (kind,id) at consume-truncation into a NEW store-owned durable record consumed-hashes.jsonl (atomic write; ADDITIVE — existing on-disk format unchanged/read-compatible; #908 allocator untouched). 2. reconcile.sh adds a THIRD accounting source alongside the inbox and its seen-ledger: a detector-observed state whose observed_hash MATCHES the store's recorded last-consumed hash for that (kind,id) is ACCOUNTED — not re-enumerated (no dup wake, no spurious CRITICAL). 3. Trust boundary (load-bearing): the 3rd check consults ONLY the store-written record. Its existence implies the state was durably enqueued+consumed, so it structurally cannot exhibit the §5 hash-advance-without-enqueue swallow signature. Trusting DETECTOR hash-files STAYS REJECTED. 4. G3 not weakened: only states the store RECORDED as consumed are suppressed. A genuinely-unaccounted state (enqueued-but-unconsumed, still in the inbox, OR a real gap) still re-enumerates + alarms. Red-first tests: - store-ack T12: consume writes the store-owned last-consumed record (per-(kind,id), monotonic last-seq wins, lazily created). - reconcile R10: a consumed state is ACCOUNTED (rc=0, UNACCOUNTED=0, no re-enumeration). - reconcile R11: pilot repro — consumed state SUPPRESSED while a distinct unconsumed/gap state STILL re-enumerates (G3 teeth intact; no over-suppression). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
This commit is contained in:
@@ -326,6 +326,51 @@ cmd_drain() {
|
||||
grep -v '^[[:space:]]*$' "$STATE_DIR/pending.jsonl" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# _record_last_consumed UPTO — #932: at consume-truncation, record the
|
||||
# last-consumed observed_hash per (kind,id) into a STORE-OWNED durable record
|
||||
# (consumed-hashes.jsonl). This is the record the reconciler consults as its
|
||||
# THIRD accounting source: after consume truncates the pending prefix, a consumed
|
||||
# state matches NEITHER the inbox (truncated) NOR the reconciler's seen-ledger
|
||||
# (which only covers the reconciler's OWN enumerations), so the reconciler used
|
||||
# to treat it as UNACCOUNTED and re-enumerate it — one duplicate orientation wake
|
||||
# + one spurious rc=1 CRITICAL per detector-active window per cycle (safe but
|
||||
# noisy; G2a alarm-hygiene at fleet scale).
|
||||
#
|
||||
# Called BEFORE the pending prefix is dropped (it reads the entries being
|
||||
# consumed). For each entry with observed_seq <= UPTO that carries full source
|
||||
# locators (kind,id,observed_hash), keep the HIGHEST-seq hash per (kind,id) as
|
||||
# that source's last-consumed state; merge with any prior record (monotonic —
|
||||
# consume only advances, so a newer consume's seq always wins its key).
|
||||
#
|
||||
# TRUST BOUNDARY (load-bearing): this record is written ONLY here, at CONSUME of a
|
||||
# durably-enqueued entry, so its existence implies the state WAS durably
|
||||
# enqueued+consumed. It therefore structurally cannot exhibit the §5 swallow-hole
|
||||
# signature (hash-advance-WITHOUT-enqueue). It is a STORE record — the detector's
|
||||
# own hash-file stays DISTRUSTED and is never consulted for accounting.
|
||||
#
|
||||
# ADDITIVE: a new file consumed-hashes.jsonl; the existing on-disk format is
|
||||
# unchanged and read-compatible (older code ignores this file). Atomic write.
|
||||
# Best-effort: a failure here degrades to the pre-#932 safe-but-noisy behaviour
|
||||
# (the reconciler re-enumerates the consumed state once — no lost obligation),
|
||||
# never a failed consume.
|
||||
_record_last_consumed() {
|
||||
local upto="$1" existing new_records merged
|
||||
[ -f "$STATE_DIR/pending.jsonl" ] || return 0
|
||||
new_records="$(jq -c --argjson upto "$upto" '
|
||||
select((.observed_seq // -1) <= $upto)
|
||||
| select((.locators.kind // "") != "" and (.locators.id // "") != "" and (.locators.observed_hash // "") != "")
|
||||
| {kind:.locators.kind, id:.locators.id, observed_hash:.locators.observed_hash, observed_seq:.observed_seq}
|
||||
' "$STATE_DIR/pending.jsonl" 2>/dev/null || true)"
|
||||
[ -n "$new_records" ] || return 0
|
||||
existing="$(cat "$STATE_DIR/consumed-hashes.jsonl" 2>/dev/null || true)"
|
||||
merged="$(printf '%s\n%s\n' "$existing" "$new_records" | grep -v '^[[:space:]]*$' |
|
||||
jq -s -c 'group_by([.kind, .id]) | .[] | max_by(.observed_seq)' 2>/dev/null || true)"
|
||||
[ -n "$merged" ] || return 0
|
||||
if ! printf '%s\n' "$merged" | grep -v '^[[:space:]]*$' | _atomic_write "$STATE_DIR/consumed-hashes.jsonl"; then
|
||||
echo "store.sh consume: WARN — could not record last-consumed hashes (#932); the reconciler may re-enumerate the consumed state once (safe-but-noisy), no lost obligation." >&2
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_consume() {
|
||||
local upto=''
|
||||
while [ $# -gt 0 ]; do
|
||||
@@ -381,6 +426,12 @@ cmd_consume() {
|
||||
k=$((k + 1))
|
||||
done
|
||||
|
||||
# #932: record the last-consumed observed_hash per (kind,id) BEFORE the pending
|
||||
# prefix is dropped (this reads the entries about to be truncated), so the
|
||||
# reconciler can recognise an already-consumed state as ACCOUNTED instead of
|
||||
# re-enumerating it. Additive, store-owned, best-effort (never fails consume).
|
||||
_record_last_consumed "$upto"
|
||||
|
||||
# Advance the consumer cursor and drop the now-consumed prefix from the
|
||||
# durable store (retain-until-CONSUMED is satisfied).
|
||||
jq -c "select(.observed_seq > $upto)" "$STATE_DIR/pending.jsonl" 2>/dev/null |
|
||||
|
||||
Reference in New Issue
Block a user