mos-dt-0andMos
8d1d6e5e76
Closes #958.
The preimage definition (source-adapter.sh) is the single most consequential file in the wake pipeline — every observed_hash is a sha256 of what it emits — and it was UNVERSIONED: no git history, no backup. When it was edited at 07:27 on 2026-07-30, attribution was recoverable only because an agent transcript happened to still be on disk. A11 gives that file durable provenance (option (2) of #958: recorded content-addressed, not in-band).
DESIGN, per the pre-registration:
- Provenance is OUT-OF-BAND (never on the adapter's stdout) — an in-band record would advance observed_hash for every source at once and manufacture the re-baseline it exists to explain (B3).
- The obligation never depends on the provenance path: a missing/corrupt store cannot halt the detector or swallow a wake (#940 advisory-fields precedent; B4).
- DESC_FMT=d1 is NOT the provenance record — the tag versions the descriptor FORMAT; a behaviour change that keeps descriptor shape re-baselines every hash and leaves the tag unchanged (B6).
CREDENTIAL HARD GATE (rebuilt after the first verdict FAILED it): byte capture is now RECORD-ONLY BY DEFAULT (extras opt in via WAKE_PREIMAGE_CAPTURE), not allow-by-default-refuse-on-shape — because a shape list can only refuse the secrets someone already enumerated, and the tool's own usage text recommended adding detector.env (where HMAC material lives). Deny is evaluated on BOTH raw and resolved path forms with resolved anchors, ordered before allow — closing the realpath-before-deny ordering defect that let a renamed symlink target through.
VERIFICATION (reviewer, mos-dt, independent of the author's claims):
- Seven decoy cases by planted-marker-then-grep-whole-state-dir: known cred path / same-name symlink / RENAMED-target symlink / prefixed secret / prefixless secret / opted-in-symlink-to-DENIED-target all REFUSED; opted-in-symlink-to-ALLOWED-target CAPTURED (positive control that the harness can capture at all, and that C was not closed by breaking every symlink).
- Polarity-completeness self-test RE-RUN with the shape list stubbed always-allow AND both deny lists stubbed — case D still safe: the flip is complete, the shape list is not load-bearing. Each stub proven live first (a stub that silently fails to apply reports the dangerous state as safe).
- B11: rm-then-change fails LOUD (rc=1), refuses to re-baseline, leaves the ledger absent; absent-with-emptied-objects still first-installs cleanly (absent-is-not-corrupt not paid for by breaking first install).
- RED-first reproduced exactly P13-P16 pre-fix; each refusal corroborated three ways (loud stderr, ledger row captured:false WITH a hash so attribution survives refusal, objects/ holding only the adapter).
KNOWN RESIDUAL (filed #969, non-gating): the deny check is both-forms but the suite needles only the resolved form — a deny reduced to resolved-only survives 17/17 green and would leak a renamed-symlink case. No reachable leak at this head (shipped code correct on all seven decoys); it constrains a FUTURE edit. Doctrine: a both-forms fix needs a needle per form; a fixture that satisfies its assertion through a DIFFERENT rule is testing the rule it did not mean to test.
Authored by pepper (sb-it-1-dt); independently reviewed by mos-dt (sb-it-1-dt) under diff-blind pre-registration (7242688b1, predating first read) — NOT CLEAR on the first verdict (B2/B11 failed by decoy), CLEAR at 8aff7d8 after the polarity rebuild. Manifest version 0.7.0.
Co-authored-by: mos-dt-0 <[email protected]>
584 lines
24 KiB
Bash
Executable File
584 lines
24 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# reconcile.sh — A7 of the wake canon (EPIC #892, W5): the source-parity
|
|
# reconciler.
|
|
#
|
|
# CONTRACT ANCHORS (docs/scratchpads/heartbeat-planning/CONVERGED-DESIGN.md):
|
|
# §4/G3 Source parity / reconciliation — the GO condition has TWO ordered
|
|
# parts:
|
|
# (i) a source-coverage PARITY INVENTORY FIRST: a lane-by-lane
|
|
# DECLARED inventory of every operational source the lane
|
|
# depends on, so an OMITTED source cannot make the §4 vector pass
|
|
# VACUOUSLY. An incomplete / undeclared inventory FLAGS (it must
|
|
# NOT silently pass).
|
|
# (ii) then a periodic full RECONCILE: enumerate declared+configured
|
|
# source state vs observed_seq/inbox => 0 UNACCOUNTED. Any source
|
|
# state not reflected in observed_seq/inbox is a gap => FLAG.
|
|
# §4 vector { source-parity inventory complete AND reconcile = 0 unaccounted }
|
|
# §7-res5 FN-oracle + reconciliation are a real, NON-OPTIONAL operating cost.
|
|
# W4 review division: the detector's first-seen-baseline does NOT wake; the
|
|
# RECONCILER is what enumerates PRE-EXISTING / startup source state into
|
|
# the durable store (at startup and periodically). Without it, state
|
|
# that already existed when the detector first saw it (baselined
|
|
# silently) would never enter the inbox and would be lost.
|
|
#
|
|
# SCOPE: calls the PUBLIC APIs of store.sh (cursors/drain/enqueue) only. It does
|
|
# NOT reimplement or modify the store, ack, digest, signer, or detector. It
|
|
# OBSERVES current source state through the SAME operator adapter contract the
|
|
# detector uses (WAKE_DETECTOR_SOURCE_CMD <kind> <id>, def on stdin), with the
|
|
# SAME fail-loud (G2a) discipline: a source error / ambiguous-empty is never
|
|
# silently "no state".
|
|
#
|
|
# CONTRACT NOTES (flagged, not silently guessed — see PR body):
|
|
# * observed_seq DUAL-ALLOCATOR HAZARD — RESOLVED by #908 (single store-side
|
|
# allocator). HISTORY: observed_seq once had TWO INDEPENDENT allocators — the
|
|
# detector's PRIVATE counter (W4) and the STORE cursor (observed_seq+offset)
|
|
# this reconciler used to enumerate from. Because the detector's next seq was
|
|
# (private_counter+1), NOT (store_observed_seq+1), the two were decoupled and
|
|
# even serializing them did NOT prevent collision: the reconciler enumerating
|
|
# alpha->seq1, beta->seq2 while the detector's private counter was still 0
|
|
# meant the detector's next delta re-allocated seq1, ALIASING alpha, and a
|
|
# consumer acking CONSUMED 1 silently dropped a distinct obligation.
|
|
# #908 DISSOLVES this at the root: store.sh enqueue is now the SOLE allocator
|
|
# (reads its own observed_seq cursor, next=+1, under an exclusive lock). This
|
|
# reconciler now enumerates by calling store.sh enqueue WITHOUT --seq — the
|
|
# SAME single allocator the detector uses — so co-feeding one store is SAFE:
|
|
# both paths draw distinct, contiguous seqs from the one cursor. The former
|
|
# fail-closed refusal (and the reconciler-SOLE-FEEDER assertion) are therefore
|
|
# RETIRED; --allow-enumerate / WAKE_RECONCILE_ALLOW_ENUMERATE remain accepted
|
|
# as deprecated no-ops for caller compatibility. The G3 accounting invariants
|
|
# (0-UNACCOUNTED, vacuous-pass prevention) are unaffected and still enforced.
|
|
# * "accounted" is judged against THREE sources: observed_seq/inbox (§4/G3
|
|
# wording), the reconciler's OWN durable reconciled-state ledger, and (#932)
|
|
# the STORE's last-consumed record (consumed-hashes.jsonl, store-written at
|
|
# consume-truncation) — NOT the detector's hash-file. Judging by the detector
|
|
# baseline would let a silent first-seen baseline swallow a pre-existing
|
|
# obligation (the exact hole G3 closes). The 3rd source suppresses ONLY states
|
|
# the store RECORDED as consumed (structurally cannot be hash-advance-without-
|
|
# enqueue), so a truly-unaccounted state still re-enumerates (G3 teeth intact).
|
|
# * enumeration uses the non-coalescible fail-safe class `actionable` (§2.3)
|
|
# so two distinct pre-existing obligations can never coalesce into one.
|
|
#
|
|
# Operator-agnostic: all state via XDG/env; no operator paths/names/secrets.
|
|
set -uo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck source=./_wake-common.sh disable=SC1091
|
|
. "$SCRIPT_DIR/_wake-common.sh"
|
|
|
|
STORE_SH="$SCRIPT_DIR/store.sh"
|
|
MANIFEST="$SCRIPT_DIR/manifest.txt"
|
|
|
|
STATE_DIR="$(wake_state_dir)"
|
|
# Reconciler-local state (its durable reconciled-state ledger) lives in its own
|
|
# subdir under the store's STATE_DIR so it never collides with store/detector.
|
|
RECON_DIR="$STATE_DIR/reconciler"
|
|
|
|
_need_jq() {
|
|
command -v jq >/dev/null 2>&1 || {
|
|
echo "reconcile.sh: jq is required" >&2
|
|
exit 3
|
|
}
|
|
}
|
|
|
|
_hash() {
|
|
if command -v sha256sum >/dev/null 2>&1; then
|
|
sha256sum | awk '{print $1}'
|
|
elif command -v shasum >/dev/null 2>&1; then
|
|
shasum -a 256 | awk '{print $1}'
|
|
elif command -v openssl >/dev/null 2>&1; then
|
|
openssl dgst -sha256 | awk '{print $NF}'
|
|
else
|
|
echo "reconcile.sh: no sha256 tool (sha256sum/shasum/openssl) available" >&2
|
|
exit 3
|
|
fi
|
|
}
|
|
|
|
usage() {
|
|
cat >&2 <<'EOF'
|
|
Usage: reconcile.sh <command> [options]
|
|
|
|
Commands:
|
|
inventory
|
|
Part (i) — source-coverage PARITY INVENTORY. Prints a lane-by-lane
|
|
declared inventory and FLAGS any incompleteness that could let the §4
|
|
vector pass VACUOUSLY:
|
|
* a declared operational source (repos/board_files/lane_anchors) that
|
|
NO watch covers (an OMITTED source);
|
|
* a watch that references a source with no declared definition (dangling);
|
|
* a lane with zero sources; an empty watch-list;
|
|
* if a lane declares `required_sources`, any id missing from its sources.
|
|
Exit 0 IFF the inventory is complete; non-zero (loud) on any flag.
|
|
|
|
reconcile [--no-enumerate] [--allow-enumerate]
|
|
Part (ii) — periodic full reconcile. Enumerates each covered source's
|
|
CURRENT state and compares it against observed_seq/inbox (+ the
|
|
reconciler's reconciled-state ledger). Any source state NOT reflected =>
|
|
UNACCOUNTED => FLAG. Unless --no-enumerate, each unaccounted source is
|
|
ENUMERATED into the durable store (the startup/pre-existing obligation
|
|
path). Exit 0 IFF 0 unaccounted were FOUND this pass.
|
|
|
|
SINGLE STORE-SIDE ALLOCATOR (#908, resolved): enumeration allocates
|
|
observed_seq by calling store.sh enqueue WITHOUT --seq — the SAME sole
|
|
allocator the detector uses (store cursor +1 under an exclusive lock). A
|
|
detector co-feeding this store is therefore SAFE: both draw distinct,
|
|
contiguous seqs from the one cursor, so no aliasing is possible. The old
|
|
fail-closed refusal is retired; --allow-enumerate (and
|
|
WAKE_RECONCILE_ALLOW_ENUMERATE=1) are accepted as deprecated no-ops.
|
|
--no-enumerate (and `check`) never write, so they remain side-effect-free.
|
|
|
|
check
|
|
The full §4/G3 gate, side-effect-free: `inventory` AND
|
|
`reconcile --no-enumerate`. Exit 0 IFF the inventory is complete AND 0
|
|
unaccounted. Use this to evaluate the vector's G3 clause without writing.
|
|
|
|
Environment:
|
|
WAKE_WATCH_LIST Operator watch-list JSON (required).
|
|
WAKE_DETECTOR_SOURCE_CMD Source adapter (required for reconcile/check; same
|
|
contract as the detector: <cmd> <kind> <id>, def on
|
|
stdin; exit!=0 or empty => FAIL LOUD).
|
|
WAKE_STATE_HOME/WAKE_AGENT store namespace (see store.sh).
|
|
WAKE_RECONCILE_ALLOW_ENUMERATE DEPRECATED no-op (#908 made co-feeding safe via
|
|
the single store-side allocator); accepted for
|
|
backward compatibility, no longer required.
|
|
EOF
|
|
}
|
|
|
|
_manifest_val() {
|
|
local key="$1"
|
|
[ -f "$MANIFEST" ] || return 0
|
|
sed -n "s/^${key}=//p" "$MANIFEST" | head -n1 | tr -d '[:space:]'
|
|
}
|
|
|
|
# _load_watchlist — validate path + JSON + shape + Gate B schema range (mirrors
|
|
# the detector's fail-loud validation so the reconciler certifies the SAME
|
|
# watch-list the detector runs). Echoes validated JSON on stdout.
|
|
_load_watchlist() {
|
|
_need_jq
|
|
local wl="${WAKE_WATCH_LIST:-}"
|
|
if [ -z "$wl" ]; then
|
|
echo "reconcile.sh: WAKE_WATCH_LIST is not set (no watch-list to reconcile)" >&2
|
|
return 2
|
|
fi
|
|
if [ ! -f "$wl" ]; then
|
|
echo "reconcile.sh: watch-list not found: $wl" >&2
|
|
return 2
|
|
fi
|
|
local json
|
|
if ! json="$(jq -e . "$wl" 2>/dev/null)"; then
|
|
echo "reconcile.sh: watch-list is not valid JSON: $wl" >&2
|
|
return 2
|
|
fi
|
|
if ! printf '%s' "$json" | jq -e 'has("schema_version") and has("watches")' >/dev/null 2>&1; then
|
|
echo "reconcile.sh: watch-list missing required 'schema_version' or 'watches'" >&2
|
|
return 2
|
|
fi
|
|
local ver smin smax
|
|
ver="$(printf '%s' "$json" | jq -r '.schema_version')"
|
|
smin="$(_manifest_val schema_min)"
|
|
smax="$(_manifest_val schema_max)"
|
|
case "$ver" in
|
|
'' | *[!0-9]*)
|
|
echo "reconcile.sh: watch-list schema_version must be an integer (got '$ver')" >&2
|
|
return 2
|
|
;;
|
|
esac
|
|
if [ -z "$smin" ] || [ -z "$smax" ]; then
|
|
echo "reconcile.sh: manifest is missing schema_min/schema_max (cannot validate Gate B)" >&2
|
|
return 2
|
|
fi
|
|
if [ "$ver" -lt "$smin" ] || [ "$ver" -gt "$smax" ]; then
|
|
echo "reconcile.sh: FAIL LOUD (Gate B) — watch-list schema_version $ver is OUTSIDE the supported range [$smin, $smax]" >&2
|
|
return 2
|
|
fi
|
|
printf '%s' "$json"
|
|
}
|
|
|
|
# _coll_for_kind KIND — the top-level collection a source kind is defined in.
|
|
_coll_for_kind() {
|
|
case "$1" in
|
|
repo) printf 'repos' ;;
|
|
board_file) printf 'board_files' ;;
|
|
lane_anchor) printf 'lane_anchors' ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
|
|
# --- part (i): source-coverage parity inventory ----------------------------
|
|
|
|
cmd_inventory() {
|
|
local json
|
|
json="$(_load_watchlist)" || exit $?
|
|
|
|
local flags=0
|
|
|
|
# An empty watch-list can never certify coverage of anything.
|
|
local nwatch
|
|
nwatch="$(printf '%s' "$json" | jq -r '.watches | length')"
|
|
if [ "${nwatch:-0}" -eq 0 ]; then
|
|
echo "reconcile.sh: FLAG (G3 parity) — watch-list declares NO watches; coverage is vacuous." >&2
|
|
flags=$((flags + 1))
|
|
fi
|
|
|
|
echo "# source-coverage parity inventory (lane-by-lane)"
|
|
|
|
# Per-lane declared inventory + empty-lane + required_sources checks.
|
|
local nlanes li
|
|
nlanes="$(printf '%s' "$json" | jq -r '.watches | length')"
|
|
li=0
|
|
while [ "$li" -lt "${nlanes:-0}" ]; do
|
|
local lane nsrc
|
|
lane="$(printf '%s' "$json" | jq -r --argjson i "$li" '.watches[$i].lane // "(unnamed)"')"
|
|
nsrc="$(printf '%s' "$json" | jq -r --argjson i "$li" '.watches[$i].sources | length')"
|
|
printf ' lane %s: %s declared source(s)\n' "$lane" "${nsrc:-0}"
|
|
|
|
if [ "${nsrc:-0}" -eq 0 ]; then
|
|
echo "reconcile.sh: FLAG (G3 parity) — lane '$lane' declares ZERO sources (empty coverage)." >&2
|
|
flags=$((flags + 1))
|
|
fi
|
|
|
|
# required_sources (optional, additive; ignored by the detector). If the
|
|
# operator declares the lane's true dependency set here, every id MUST be
|
|
# covered by the lane's sources — an OMITTED dependency FLAGS.
|
|
local missing
|
|
missing="$(printf '%s' "$json" | jq -r --argjson i "$li" '
|
|
(.watches[$i].required_sources // []) as $req
|
|
| ($req - [ .watches[$i].sources[].id ]) | .[]')"
|
|
if [ -n "$missing" ]; then
|
|
local m
|
|
while IFS= read -r m; do
|
|
[ -n "$m" ] || continue
|
|
echo "reconcile.sh: FLAG (G3 parity) — lane '$lane' requires source '$m' but it is OMITTED from the lane's watched sources (vacuous-pass prevented)." >&2
|
|
flags=$((flags + 1))
|
|
done <<EOF
|
|
$missing
|
|
EOF
|
|
fi
|
|
li=$((li + 1))
|
|
done
|
|
|
|
# Dangling: every watched (kind,id) MUST have a declared definition.
|
|
local pairs kind id
|
|
pairs="$(printf '%s' "$json" | jq -r '[ .watches[].sources[] | "\(.kind)\t\(.id)" ] | unique | .[]')"
|
|
while IFS=$'\t' read -r kind id; do
|
|
[ -n "$kind" ] || continue
|
|
local coll def
|
|
if ! coll="$(_coll_for_kind "$kind")"; then
|
|
echo "reconcile.sh: FLAG (G3 parity) — unknown source kind '$kind' in a watch." >&2
|
|
flags=$((flags + 1))
|
|
continue
|
|
fi
|
|
def="$(printf '%s' "$json" | jq -c --arg c "$coll" --arg id "$id" '(.[$c] // []) | map(select(.id == $id)) | .[0] // empty')"
|
|
if [ -z "$def" ]; then
|
|
echo "reconcile.sh: FLAG (G3 parity) — watch references '$kind/$id' but no such entry is declared in '$coll' (dangling)." >&2
|
|
flags=$((flags + 1))
|
|
fi
|
|
done <<EOF
|
|
$pairs
|
|
EOF
|
|
|
|
# Omitted-from-coverage: every DECLARED operational source must be covered by
|
|
# at least one watch. A declared-but-unwatched source is exactly the OMITTED
|
|
# source that would let the vector pass VACUOUSLY (the detector never checks it).
|
|
local coll kinds
|
|
kinds="repos:repo board_files:board_file lane_anchors:lane_anchor"
|
|
for coll in $kinds; do
|
|
local cname ckind ids
|
|
cname="${coll%%:*}"
|
|
ckind="${coll##*:}"
|
|
ids="$(printf '%s' "$json" | jq -r --arg c "$cname" '(.[$c] // [])[].id')"
|
|
local sid covered
|
|
while IFS= read -r sid; do
|
|
[ -n "$sid" ] || continue
|
|
covered="$(printf '%s' "$json" | jq -r --arg k "$ckind" --arg id "$sid" \
|
|
'[ .watches[].sources[] | select(.kind==$k and .id==$id) ] | length')"
|
|
if [ "${covered:-0}" -eq 0 ]; then
|
|
echo "reconcile.sh: FLAG (G3 parity) — declared source '$ckind/$sid' is NOT covered by any watch (OMITTED from coverage; vacuous-pass prevented)." >&2
|
|
flags=$((flags + 1))
|
|
fi
|
|
done <<EOF
|
|
$ids
|
|
EOF
|
|
done
|
|
|
|
if [ "$flags" -ne 0 ]; then
|
|
echo "reconcile.sh: source-parity inventory INCOMPLETE — $flags flag(s). The §4 vector's G3 clause does NOT pass." >&2
|
|
return 1
|
|
fi
|
|
echo "reconcile.sh: source-parity inventory COMPLETE (every declared source is covered; no dangling/empty/omitted)."
|
|
return 0
|
|
}
|
|
|
|
# --- part (ii): full reconcile (0 unaccounted) -----------------------------
|
|
|
|
# _scope_anchor ANCHOR (content on stdin) — mirror of the detector's anchor
|
|
# scoping so the reconciler hashes the SAME region the detector would. Extract
|
|
# from the first line containing the anchor up to the next heading (or EOF).
|
|
_scope_anchor() {
|
|
local anchor="$1"
|
|
awk -v a="$anchor" '
|
|
index($0, a) > 0 && !inzone { inzone=1; print; next }
|
|
inzone && /^#/ { exit }
|
|
inzone { print }
|
|
'
|
|
}
|
|
|
|
# _recon_key KIND ID — stable per-source ledger key.
|
|
_recon_key() {
|
|
printf '%s\037%s' "$1" "$2" | _hash
|
|
}
|
|
|
|
# _observe_source KIND ID DEF — observe current scoped state; echo the hash on
|
|
# success (rc 0). rc 1 = FAIL LOUD (source error / ambiguous-empty), same G2a
|
|
# discipline as the detector: an error is never silently "no state".
|
|
_observe_source() {
|
|
local kind="$1" id="$2" def="$3"
|
|
local anchor raw rc deftmp
|
|
anchor="$(printf '%s' "$def" | jq -r '.anchor // empty')"
|
|
mkdir -p "$RECON_DIR"
|
|
# shellcheck disable=SC2154
|
|
deftmp="$(mktemp "$RECON_DIR/${_wake_tmp_prefix}defXXXXXX")" || return 1
|
|
printf '%s' "$def" >"$deftmp"
|
|
raw="$("$WAKE_DETECTOR_SOURCE_CMD" "$kind" "$id" <"$deftmp" 2>/dev/null)"
|
|
rc=$?
|
|
rm -f "$deftmp"
|
|
if [ "$rc" -ne 0 ]; then
|
|
echo "reconcile.sh: FAIL LOUD (G2a) — source '$kind/$id' errored (adapter exit $rc). Cannot certify parity; not treated as 'no state'." >&2
|
|
return 1
|
|
fi
|
|
if [ -z "$raw" ]; then
|
|
echo "reconcile.sh: FAIL LOUD (G2a) — source '$kind/$id' returned AMBIGUOUS-EMPTY (empty-that-might-mean-hidden is never 'no state')." >&2
|
|
return 1
|
|
fi
|
|
local scoped
|
|
if [ -n "$anchor" ]; then
|
|
scoped="$(printf '%s' "$raw" | _scope_anchor "$anchor")"
|
|
if [ -z "$scoped" ]; then
|
|
echo "reconcile.sh: FAIL LOUD (G2a) — source '$kind/$id' anchor '$anchor' not present (ambiguous section state)." >&2
|
|
return 1
|
|
fi
|
|
else
|
|
scoped="$raw"
|
|
fi
|
|
printf '%s' "$scoped" | _hash
|
|
}
|
|
|
|
# _inbox_has KIND ID HASH — true iff a pending inbox entry reflects this exact
|
|
# source state (observed_seq/inbox is the §4/G3 accounting universe).
|
|
_inbox_has() {
|
|
local kind="$1" id="$2" h="$3" n
|
|
n="$("$STORE_SH" drain 2>/dev/null | jq -s --arg k "$kind" --arg id "$id" --arg h "$h" \
|
|
'[ .[] | select(.locators.kind==$k and .locators.id==$id and .locators.observed_hash==$h) ] | length' 2>/dev/null || echo 0)"
|
|
[ "${n:-0}" -gt 0 ]
|
|
}
|
|
|
|
# _store_consumed_has KIND ID HASH — #932 THIRD accounting source. True iff the
|
|
# STORE's last-consumed record (consumed-hashes.jsonl, written by store.sh at
|
|
# consume-truncation) records HASH as the last-consumed observed_hash for
|
|
# (kind,id). After consume truncates the pending prefix, a consumed state matches
|
|
# NEITHER _inbox_has (inbox truncated) NOR the reconciler's own seen-ledger (it
|
|
# only covers the reconciler's OWN enumerations) — so without this check the
|
|
# reconciler re-enumerates the just-consumed state: one DUPLICATE orientation
|
|
# wake + one SPURIOUS rc=1 CRITICAL per detector-active window per cycle (#932).
|
|
#
|
|
# TRUST BOUNDARY (load-bearing — do NOT violate): this consults ONLY the
|
|
# store-written record under STATE_DIR — NEVER a detector-owned hash-file. The
|
|
# store record is trustworthy BY CONSTRUCTION: it is written only at CONSUME of a
|
|
# durably-enqueued entry, so its existence implies the state WAS durably enqueued,
|
|
# and it structurally cannot exhibit the §5 swallow-hole signature (hash-advance-
|
|
# WITHOUT-enqueue). Trusting DETECTOR hash-files STAYS REJECTED (it reopens §5).
|
|
#
|
|
# G3 is NOT weakened: this suppresses ONLY states the store has RECORDED as
|
|
# consumed. A genuinely-unaccounted state (durably enqueued but not yet consumed
|
|
# — still in the inbox — OR a real gap) has no matching consumed record, so it is
|
|
# still UNACCOUNTED and still re-enumerates + alarms.
|
|
_store_consumed_has() {
|
|
local kind="$1" id="$2" h="$3" n
|
|
[ -f "$STATE_DIR/consumed-hashes.jsonl" ] || return 1
|
|
n="$(jq -s --arg k "$kind" --arg id "$id" --arg h "$h" \
|
|
'[ .[] | select(.kind==$k and .id==$id and .observed_hash==$h) ] | length' \
|
|
"$STATE_DIR/consumed-hashes.jsonl" 2>/dev/null || echo 0)"
|
|
[ "${n:-0}" -gt 0 ]
|
|
}
|
|
|
|
cmd_reconcile() {
|
|
local enumerate=1 allow_enumerate=0
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--no-enumerate)
|
|
enumerate=0
|
|
shift
|
|
;;
|
|
--allow-enumerate)
|
|
allow_enumerate=1
|
|
shift
|
|
;;
|
|
*)
|
|
echo "reconcile.sh reconcile: unknown option '$1'" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
[ "${WAKE_RECONCILE_ALLOW_ENUMERATE:-0}" = "1" ] && allow_enumerate=1
|
|
|
|
local json
|
|
json="$(_load_watchlist)" || exit $?
|
|
if [ -z "${WAKE_DETECTOR_SOURCE_CMD:-}" ]; then
|
|
echo "reconcile.sh: WAKE_DETECTOR_SOURCE_CMD is not set (no adapter to observe source state)" >&2
|
|
exit 2
|
|
fi
|
|
|
|
"$STORE_SH" init >/dev/null 2>&1 || true
|
|
mkdir -p "$RECON_DIR"
|
|
|
|
# --- SINGLE STORE-SIDE ALLOCATOR (#908) -----------------------------------
|
|
# Enumeration allocates observed_seq by calling store.sh enqueue WITHOUT --seq
|
|
# (the sole allocator the detector also uses). Co-feeding one store is safe —
|
|
# both paths draw distinct, contiguous seqs from the one store cursor under its
|
|
# exclusive lock — so the former dual-allocator fail-closed refusal is retired.
|
|
# allow_enumerate is now a deprecated no-op, retained only so old callers that
|
|
# pass --allow-enumerate / WAKE_RECONCILE_ALLOW_ENUMERATE=1 still work.
|
|
: "$allow_enumerate"
|
|
|
|
local pairs kind id failed=0 unaccounted=0 enumerated=0
|
|
|
|
# #958 preimage provenance: same pre-step as the detector's poll tick, for
|
|
# the path where the preimage changed while the detector was down — without
|
|
# it, a changed adapter surfaces here only as N UNACCOUNTED enumerations
|
|
# with no first-class cause line. Run BEFORE observing any source so the
|
|
# cause entry's observed_seq precedes the enumerations it explains. Loud
|
|
# infrastructure failure marks the reconcile failed but does not stop it.
|
|
if ! "$SCRIPT_DIR/preimage.sh" check --enqueue; then
|
|
echo "reconcile.sh: FAIL LOUD — preimage provenance check failed (see preimage.sh above); reconcile continues but exits non-zero." >&2
|
|
failed=1
|
|
fi
|
|
pairs="$(printf '%s' "$json" | jq -r '[ .watches[].sources[] | "\(.kind)\t\(.id)" ] | unique | .[]')"
|
|
if [ -z "$pairs" ]; then
|
|
echo "reconcile.sh: watch-list declares no sources under watches[].sources[]" >&2
|
|
exit 2
|
|
fi
|
|
|
|
echo "# full reconcile (declared+configured source state vs observed_seq/inbox)"
|
|
while IFS=$'\t' read -r kind id; do
|
|
[ -n "$kind" ] || continue
|
|
local coll def
|
|
if ! coll="$(_coll_for_kind "$kind")"; then
|
|
echo "reconcile.sh: FAIL LOUD — unknown source kind '$kind' in watch-list" >&2
|
|
failed=1
|
|
continue
|
|
fi
|
|
def="$(printf '%s' "$json" | jq -c --arg c "$coll" --arg id "$id" '(.[$c] // []) | map(select(.id == $id)) | .[0] // empty')"
|
|
if [ -z "$def" ]; then
|
|
echo "reconcile.sh: FAIL LOUD (G3 parity) — watch references '$kind/$id' but no such entry is declared in '$coll'" >&2
|
|
failed=1
|
|
continue
|
|
fi
|
|
|
|
local curhash key ledger accounted
|
|
if ! curhash="$(_observe_source "$kind" "$id" "$def")"; then
|
|
failed=1
|
|
continue
|
|
fi
|
|
key="$(_recon_key "$kind" "$id")"
|
|
ledger="$RECON_DIR/seen-$key"
|
|
|
|
accounted=0
|
|
# Reflected in the durable inbox (delivered, awaiting consume)?
|
|
if _inbox_has "$kind" "$id" "$curhash"; then
|
|
accounted=1
|
|
# Or already reconciled to this exact state before (durable ledger)?
|
|
elif [ -f "$ledger" ] && [ "$(tr -d '[:space:]' <"$ledger")" = "$curhash" ]; then
|
|
accounted=1
|
|
# Or the STORE recorded this exact state as ALREADY CONSUMED (#932 — the THIRD
|
|
# accounting source; a consume-truncated state matches neither the inbox nor
|
|
# the seen-ledger, so without this it would spuriously re-enumerate). Consults
|
|
# ONLY the store-written record, never a detector hash-file (§5 stays closed).
|
|
elif _store_consumed_has "$kind" "$id" "$curhash"; then
|
|
accounted=1
|
|
fi
|
|
|
|
if [ "$accounted" -eq 1 ]; then
|
|
printf ' %s/%s: ACCOUNTED\n' "$kind" "$id"
|
|
continue
|
|
fi
|
|
|
|
# UNACCOUNTED: a source state not reflected in observed_seq/inbox => a gap.
|
|
unaccounted=$((unaccounted + 1))
|
|
printf ' %s/%s: UNACCOUNTED (current state not reflected in observed_seq/inbox)\n' "$kind" "$id" >&2
|
|
|
|
if [ "$enumerate" -eq 1 ]; then
|
|
# ENUMERATE the pre-existing/startup obligation into the durable store via
|
|
# the SINGLE store-side allocator (#908): store.sh enqueue WITHOUT --seq
|
|
# allocates the next observed_seq under the store's lock and PRINTS it. Both
|
|
# the detector and this reconciler draw from that one cursor, so a co-fed
|
|
# store cannot alias. Class = non-coalescible fail-safe `actionable` (§2.3)
|
|
# so distinct pre-existing obligations never collapse together.
|
|
local seq locators
|
|
locators="$(jq -cn --arg kind "$kind" --arg id "$id" --arg hash "$curhash" \
|
|
--argjson sdef "$def" \
|
|
'{kind:$kind, id:$id, observed_hash:$hash, reconciled:true}
|
|
+ ( $sdef | {repo, path, anchor, remote, branches} | with_entries(select(.value != null)) )')"
|
|
if seq="$("$STORE_SH" enqueue --class actionable --locators "$locators")"; then
|
|
printf '%s' "$curhash" | _atomic_write "$ledger"
|
|
enumerated=$((enumerated + 1))
|
|
printf ' -> enumerated into store as observed_seq=%s (class=actionable)\n' "$seq"
|
|
else
|
|
echo "reconcile.sh: enumerate FAILED for '$kind/$id'" >&2
|
|
failed=1
|
|
fi
|
|
fi
|
|
done <<EOF
|
|
$pairs
|
|
EOF
|
|
|
|
echo "reconcile.sh: UNACCOUNTED=$unaccounted ENUMERATED=$enumerated"
|
|
if [ "$failed" -ne 0 ]; then
|
|
echo "reconcile.sh: reconcile FAILED (a source could not be observed/enumerated)." >&2
|
|
return 1
|
|
fi
|
|
if [ "$unaccounted" -ne 0 ]; then
|
|
if [ "$enumerate" -eq 1 ]; then
|
|
echo "reconcile.sh: FLAG — $unaccounted unaccounted source-state(s) found and enumerated. Re-run to confirm 0." >&2
|
|
else
|
|
echo "reconcile.sh: FLAG — $unaccounted unaccounted source-state(s) (audit mode; nothing enumerated)." >&2
|
|
fi
|
|
return 1
|
|
fi
|
|
echo "reconcile.sh: reconcile CLEAN — 0 unaccounted (every declared source state is reflected in observed_seq/inbox)."
|
|
return 0
|
|
}
|
|
|
|
cmd_check() {
|
|
# The full §4/G3 gate, side-effect-free.
|
|
local rc=0
|
|
cmd_inventory || rc=1
|
|
cmd_reconcile --no-enumerate || rc=1
|
|
if [ "$rc" -eq 0 ]; then
|
|
echo "reconcile.sh: G3 GATE PASS — inventory complete AND 0 unaccounted."
|
|
else
|
|
echo "reconcile.sh: G3 GATE FAIL — inventory incomplete OR unaccounted state present." >&2
|
|
fi
|
|
return "$rc"
|
|
}
|
|
|
|
main() {
|
|
[ $# -ge 1 ] || {
|
|
usage
|
|
exit 2
|
|
}
|
|
local cmd="$1"
|
|
shift
|
|
case "$cmd" in
|
|
inventory) cmd_inventory "$@" ;;
|
|
reconcile) cmd_reconcile "$@" ;;
|
|
check) cmd_check "$@" ;;
|
|
-h | --help | help) usage ;;
|
|
*)
|
|
echo "reconcile.sh: unknown command '$cmd'" >&2
|
|
usage
|
|
exit 2
|
|
;;
|
|
esac
|
|
}
|
|
|
|
main "$@"
|