forked from mosaicstack/stack
test(wake): #973 validation harness — trace/inventory/arm evidence chain
validate-973.sh produces the complete evidence chain in one run:
0 instrument self-test (microtest, now C1-C11) before any evidence
1 expected set: 261 coordinates from the frozen denominator artifact
(+3 header shift), count declared BEFORE any suite runs
2 static inventory re-derived from SOURCE TEXT == expected set — the
ledger is an execution trace, not an inventory, so the inventory leg
comes from the text (mos-dt amendment ONE)
3 green instrumented run: ten suites, per-suite pinned sentinels
(preimage's divergent format included), exit codes reported per file
4 trace arithmetic on coordinate SETS: trace-minus-expected must be
empty; expected-minus-trace (converted-but-never-executed) is
enumerated and every entry must match a row in the committed
unexecuted-sites-dispositions.txt, with stale dispositions rejected
symmetrically
5 21 forced-error arms (19 denominator canaries + E-form store-ack:736,
the A6 $(count_lines) shape + F-form quarantine:563, the multi-grep
pipeline capture): each must emit ARMED, abort naming the site, exit
non-zero, emit NO sentinel, and have its ledger row already landed
6 residual sweep with the denominator's own classifier: zero unconverted
verdict greps, six per-form plants found in the same run
The nine unexecuted sites are the FAILED-branch summary lines, structurally
unreachable in a green run; microtest C11 executes the exact template on a
deterministically red mini-suite so their dispositions cite a measured
execution, not an inference.
First full run: OK — 252 executed + 9 dispositioned = 261, all arms fired,
sweep clean.
Written-by: pepper (sb-it-1-dt)
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NsKce8iZuSuRnu3gVMCBKB
This commit is contained in:
Jason Woltje
co-authored by
Claude Fable 5
Claude Fable 5
parent
8cb5866a3d
commit
bd57ef0c74
@@ -0,0 +1,280 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check-973.py — computation backend for the #973 validation harness.
|
||||
|
||||
Everything here derives from exactly two inputs: the frozen denominator
|
||||
artifact (denominator-089615f.json) and the SOURCE TEXT of the ten converted
|
||||
suites at the current tree. It never reads the ledger — set arithmetic against
|
||||
the runtime trace belongs to validate-973.sh, so the two legs of the
|
||||
comparison come from independent code paths.
|
||||
|
||||
Subcommands (all print sorted, stable output; non-zero exit on any failure):
|
||||
|
||||
expected The expected coordinate set from the ARTIFACT: one
|
||||
"<helper> <file>:<line+3>" row per denominator row (+3 = the
|
||||
uniform header shift the converter applied; converter-verified).
|
||||
Multi-grep lines stay ONE coordinate.
|
||||
|
||||
static The converted-site inventory from the SOURCE TEXT at the current
|
||||
tree: every non-comment line bearing a has_match/count_lines
|
||||
token, as "<helper> <file>:<line>". Independent of the artifact
|
||||
row list, so `expected == static` is a real check on the
|
||||
conversion, not a tautology. (Amendment ONE, leg 1: the ledger is
|
||||
an execution trace, not an inventory — the inventory must come
|
||||
from the text.)
|
||||
|
||||
arms The forced-error arm list: the 19 denominator canaries plus one
|
||||
E-form arm (store-ack:733→736, a $(count_lines) capture compared
|
||||
afterward — the A6 shape) plus one F-form arm (quarantine:560→563,
|
||||
the multi-grep pipeline capture), as "<helper> <file>:<line+3>
|
||||
<form>". Both extras are asserted to exist in the artifact with
|
||||
the expected form — a renumber that moved them fails here, not
|
||||
silently downstream.
|
||||
|
||||
sweep Residual sweep: the denominator's own classifier (ported from the
|
||||
frozen derivation) over the ten suites at the current tree must
|
||||
find ZERO unconverted verdict-form grep sites; and, IN THE SAME
|
||||
RUN, six per-form specimens planted into a temp copy of a real
|
||||
suite must ALL be found with their correct forms — an instrument
|
||||
that reports zero must first be seen finding what it claims to
|
||||
find (A5).
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
WAKE = HERE.parent
|
||||
ART = HERE / "denominator-089615f.json"
|
||||
|
||||
HEADER_SHIFT = 3 # converter inserted 3 header lines after SCRIPT_DIR in every suite
|
||||
|
||||
# The two hand-picked extra arms (base coordinates; forms asserted at load).
|
||||
EXTRA_ARMS = [
|
||||
("test-wake-store-ack.sh", 733, "E-count-capture"),
|
||||
("test-wake-digest-quarantine.sh", 560, "F-extract-capture"),
|
||||
]
|
||||
|
||||
RX_HELPER = re.compile(r"(^|[^A-Za-z0-9_.-])(has_match|count_lines)([^A-Za-z0-9_.-]|$)")
|
||||
|
||||
# ---- classifier, ported verbatim in logic from the frozen denominator
|
||||
# ---- derivation (docs/journal/fleet/drift-derive-089615f__pepper.py)
|
||||
RX_FAIL_SAME = re.compile(r"(\|\||&&)\s*fail")
|
||||
RX_COUNT_SUB = re.compile(r"\$\(.*grep\s+[^)]*-c|\$\(\s*grep\s+-c")
|
||||
RX_ASSIGN_SUB = re.compile(r'=\s*"?\$\(.*grep')
|
||||
RX_IF = re.compile(r"^\s*(el)?if\s+.*grep")
|
||||
RX_GREP = re.compile(r"(^|[^A-Za-z0-9_.-])grep([^A-Za-z0-9_.-]|$)")
|
||||
|
||||
|
||||
def polarity(line):
|
||||
m = RX_FAIL_SAME.search(line)
|
||||
return "OR" if m.group(1) == "||" else "AND"
|
||||
|
||||
|
||||
def classify(lines):
|
||||
"""Return (sites, dispo). Every line containing the word grep gets a row."""
|
||||
sites, dispo = [], []
|
||||
n = len(lines)
|
||||
for i, raw in enumerate(lines):
|
||||
line = raw
|
||||
ln = i + 1
|
||||
if not RX_GREP.search(line):
|
||||
continue
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("#"):
|
||||
dispo.append((ln, "comment", stripped))
|
||||
continue
|
||||
nxt = ""
|
||||
for j in range(i + 1, min(i + 3, n)):
|
||||
if lines[j].strip():
|
||||
nxt = lines[j].strip()
|
||||
break
|
||||
if "$(" in line and RX_COUNT_SUB.search(line):
|
||||
sites.append((ln, "E-count-capture", stripped))
|
||||
continue
|
||||
if RX_ASSIGN_SUB.search(line) and "grep -c" not in line:
|
||||
sites.append((ln, "F-extract-capture", stripped))
|
||||
continue
|
||||
if RX_FAIL_SAME.search(line):
|
||||
sites.append((ln, "A-same-line", stripped))
|
||||
continue
|
||||
if stripped.endswith("\\"):
|
||||
k = i + 1
|
||||
joined = stripped[:-1]
|
||||
while k < n:
|
||||
cont = lines[k].strip()
|
||||
joined += " " + (cont[:-1] if cont.endswith("\\") else cont)
|
||||
if not cont.endswith("\\"):
|
||||
break
|
||||
k += 1
|
||||
if re.search(r"(\|\||&&)\s*fail", joined) or (
|
||||
joined.rstrip().endswith(("||", "&&"))
|
||||
and k + 1 < n
|
||||
and lines[k + 1].strip().startswith("fail")
|
||||
):
|
||||
sites.append((ln, "C-cont-backslash", stripped))
|
||||
continue
|
||||
dispo.append((ln, "backslash-no-fail-continuation", stripped))
|
||||
continue
|
||||
if stripped.endswith(("||", "&&")) and nxt.startswith("fail"):
|
||||
sites.append((ln, "B-cont-operator", stripped))
|
||||
continue
|
||||
if RX_IF.search(line):
|
||||
window = " ".join(lines[j] for j in range(i, min(i + 5, n)))
|
||||
if "fail" in window:
|
||||
sites.append((ln, "D-if-form", stripped))
|
||||
continue
|
||||
dispo.append((ln, "if-grep-no-fail-window", stripped))
|
||||
continue
|
||||
win = " ".join(lines[j] for j in range(max(0, i - 2), min(i + 3, n)))
|
||||
if re.search(r"fail", win, re.I):
|
||||
dispo.append((ln, "BACKSTOP-HAND-REVIEW", stripped))
|
||||
else:
|
||||
dispo.append((ln, "no-verdict-context", stripped))
|
||||
return sites, dispo
|
||||
|
||||
|
||||
def load_art():
|
||||
art = json.loads(ART.read_text())
|
||||
assert art["total"] == 261 == len(art["rows"]), "artifact self-consistency"
|
||||
return art
|
||||
|
||||
|
||||
def helper_for(row):
|
||||
return "count_lines" if row["form"].startswith("E") else "has_match"
|
||||
|
||||
|
||||
def suite_files(art):
|
||||
return sorted({r["file"] for r in art["rows"]})
|
||||
|
||||
|
||||
def cmd_expected():
|
||||
art = load_art()
|
||||
out = sorted(
|
||||
f"{helper_for(r)} {r['file']}:{r['line'] + HEADER_SHIFT}" for r in art["rows"]
|
||||
)
|
||||
assert len(out) == len(set(out)) == 261, "expected set must be 261 distinct rows"
|
||||
print("\n".join(out))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_static():
|
||||
art = load_art()
|
||||
rows = []
|
||||
for f in suite_files(art):
|
||||
for i, line in enumerate((WAKE / f).read_text().split("\n"), start=1):
|
||||
if line.strip().startswith("#"):
|
||||
continue
|
||||
m = RX_HELPER.search(line)
|
||||
if not m:
|
||||
continue
|
||||
helper = (
|
||||
"count_lines"
|
||||
if RX_HELPER.search(line).group(2) == "count_lines"
|
||||
else "has_match"
|
||||
)
|
||||
rows.append(f"{helper} {f}:{i}")
|
||||
print("\n".join(sorted(rows)))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_arms():
|
||||
art = load_art()
|
||||
by_key = {(r["file"], r["line"]): r for r in art["rows"]}
|
||||
rows = []
|
||||
canaries = [r for r in art["rows"] if r.get("canary")]
|
||||
assert len(canaries) == 19, f"expected 19 canaries, artifact has {len(canaries)}"
|
||||
for r in canaries:
|
||||
rows.append(f"{helper_for(r)} {r['file']}:{r['line'] + HEADER_SHIFT} {r['form']}")
|
||||
for f, ln, want_form in EXTRA_ARMS:
|
||||
r = by_key.get((f, ln))
|
||||
assert r is not None, f"extra arm {f}:{ln} not in artifact — renumbered?"
|
||||
assert r["form"] == want_form, f"extra arm {f}:{ln} form {r['form']} != {want_form}"
|
||||
rows.append(f"{helper_for(r)} {f}:{ln + HEADER_SHIFT} {r['form']}")
|
||||
assert len(rows) == 21
|
||||
print("\n".join(rows))
|
||||
return 0
|
||||
|
||||
|
||||
PLANTS = [
|
||||
("A-same-line", ['grep -q needle haystack || fail "plant-A"']),
|
||||
("B-cont-operator", ["grep -q needle haystack ||", ' fail "plant-B"']),
|
||||
("C-cont-backslash", ["grep -q needle \\", ' haystack || fail "plant-C"']),
|
||||
("D-if-form", ["if ! grep -q needle haystack; then", ' fail "plant-D"', "fi"]),
|
||||
("E-count-capture", ['[ "$(grep -c needle haystack)" = "1" ] || fail "plant-E"']),
|
||||
("F-extract-capture", ['val="$(grep needle haystack)"']),
|
||||
]
|
||||
|
||||
|
||||
def cmd_sweep():
|
||||
art = load_art()
|
||||
bad = 0
|
||||
|
||||
# leg 1: real suites at the current tree must be residual-free
|
||||
for f in suite_files(art):
|
||||
lines = (WAKE / f).read_text().split("\n")
|
||||
sites, _dispo = classify(lines)
|
||||
residual = []
|
||||
for ln, form, text in sites:
|
||||
if RX_HELPER.search(lines[ln - 1]):
|
||||
# converted line whose PATTERN argument contains the word grep:
|
||||
# not an unconverted site, but never silently absorbed either
|
||||
print(f"SWEEP-NOTE {f}:{ln} converted line matches grep-token ({form}): {text[:80]}")
|
||||
continue
|
||||
residual.append((ln, form, text))
|
||||
for ln, form, text in residual:
|
||||
print(f"SWEEP-RESIDUAL {f}:{ln} {form}: {text[:100]}")
|
||||
bad += 1
|
||||
print(f"SWEEP {f}: {len(residual)} residual verdict site(s)")
|
||||
|
||||
# leg 2, SAME RUN: the instrument must find six per-form plants
|
||||
donor = suite_files(art)[0]
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
planted = Path(td) / donor
|
||||
shutil.copy(WAKE / donor, planted)
|
||||
base_lines = planted.read_text().split("\n")
|
||||
offset = len(base_lines)
|
||||
expect = {}
|
||||
for form, snippet in PLANTS:
|
||||
expect[offset + 1] = form # first physical line of each plant
|
||||
base_lines.extend(snippet)
|
||||
offset = len(base_lines)
|
||||
planted.write_text("\n".join(base_lines))
|
||||
sites, _ = classify(planted.read_text().split("\n"))
|
||||
found = {ln: form for ln, form, _t in sites if ln in expect}
|
||||
unexpected = [(ln, form) for ln, form, _t in sites if ln not in expect]
|
||||
hits = sum(1 for ln, form in expect.items() if found.get(ln) == form)
|
||||
print(f"SWEEP-PLANTS found={hits}/6 in planted copy of {donor}")
|
||||
if hits != 6:
|
||||
for ln, form in sorted(expect.items()):
|
||||
got = found.get(ln, "<missed>")
|
||||
if got != form:
|
||||
print(f"SWEEP-PLANT-MISS line {ln}: expected {form}, got {got}")
|
||||
bad += 1
|
||||
if unexpected:
|
||||
# the donor is a converted suite: any non-plant site the sweep finds
|
||||
# in the copy contradicts the zero it just reported on the original
|
||||
for ln, form in unexpected:
|
||||
print(f"SWEEP-PLANT-UNEXPECTED {donor}(copy):{ln} {form}")
|
||||
bad += 1
|
||||
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
def main():
|
||||
cmds = {
|
||||
"expected": cmd_expected,
|
||||
"static": cmd_static,
|
||||
"arms": cmd_arms,
|
||||
"sweep": cmd_sweep,
|
||||
}
|
||||
if len(sys.argv) != 2 or sys.argv[1] not in cmds:
|
||||
sys.exit(f"usage: check-973.py {{{'|'.join(cmds)}}}")
|
||||
sys.exit(cmds[sys.argv[1]]())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -34,6 +34,14 @@
|
||||
# misreports the continuation line, wake_assert_init aborts loudly and
|
||||
# nothing past init executes — a pin whose failure arm was never seen
|
||||
# firing is an undertaking, not a control.
|
||||
# C11 the FAILED-summary template executes on the red path: a mini-suite
|
||||
# driven deterministically red emits the exact converted summary shape
|
||||
# (`FAILED ($(count_lines . "$FAILFILE") assertion(s))`) with the right
|
||||
# count, exits 1, and the summary site's ledger row lands. The nine
|
||||
# real-suite summary sites are structurally unreachable in a green run
|
||||
# (guarded by [ -s "$FAILFILE" ]); their dispositions cite THIS check as
|
||||
# the measured execution of the same template, so "unexecuted in the
|
||||
# green run" never silently means "never executed anywhere".
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -247,6 +255,42 @@ else
|
||||
check C10 1 "rc=$rc out=$(printf '%s' "$out" | tail -n 2 | tr '\n' ' ')"
|
||||
fi
|
||||
|
||||
# --- C11: red path executes the converted FAILED-summary template -----------
|
||||
cat >"$TMP/mini-c.sh" <<'MINI_C'
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
. "$WAKE_COMMON"
|
||||
wake_assert_init
|
||||
TMP="$1"
|
||||
FAILFILE="$TMP/failures-c"
|
||||
: >"$FAILFILE"
|
||||
fail_msg() { echo " FAIL: $*" >&2; echo x >>"$FAILFILE"; }
|
||||
ok() { :; }
|
||||
(
|
||||
has_match -q alpha "$TMP/data.txt" && fail_msg "alpha present" # SITE:c-inverted (deterministically red: alpha IS in the fixture)
|
||||
) && ok
|
||||
echo
|
||||
if [ -s "$FAILFILE" ]; then
|
||||
echo "wake mini-c harness: FAILED ($(count_lines . "$FAILFILE") assertion(s))" >&2 # SITE:c-summary
|
||||
exit 1
|
||||
fi
|
||||
echo "wake mini-c harness: all invariants passed (1 group)"
|
||||
MINI_C
|
||||
chmod +x "$TMP/mini-c.sh"
|
||||
LEDGER="$TMP/ledger-c11"
|
||||
: >"$LEDGER"
|
||||
out="$(WAKE_ASSERT_LEDGER="$LEDGER" bash "$TMP/mini-c.sh" "$TMP" 2>&1)"
|
||||
rc=$?
|
||||
summary_ln="$(site_line "$TMP/mini-c.sh" c-summary)"
|
||||
if [ "$rc" -eq 1 ] &&
|
||||
printf '%s' "$out" | grep -q 'wake mini-c harness: FAILED (1 assertion(s))' &&
|
||||
! printf '%s' "$out" | grep -q 'all invariants passed' &&
|
||||
grep -q "^count_lines mini-c.sh:${summary_ln}\$" "$LEDGER"; then
|
||||
check C11 0 ""
|
||||
else
|
||||
check C11 1 "rc=$rc summary_ln=$summary_ln ledger=$(tr '\n' ' ' <"$LEDGER") out=$(printf '%s' "$out" | tail -n 2 | tr '\n' ' ')"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo "microtest-wake-assert: FAILED ($fails check(s))" >&2
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# unexecuted-sites-dispositions.txt — #973 validation, amendment ONE leg 3.
|
||||
#
|
||||
# The ledger is an execution trace, not an inventory: a green instrumented run
|
||||
# cannot execute a site that only lives on a suite's red path. Every converted
|
||||
# site that did NOT appear in the green-run trace is enumerated here with an
|
||||
# individual disposition; validate-973.sh fails if any unexecuted site lacks a
|
||||
# row here, and ALSO fails if a row here names a site that DID execute (stale
|
||||
# disposition). Key = first two whitespace-separated fields; text after "—" is
|
||||
# the adjudication.
|
||||
#
|
||||
# All nine sites below are the same structural shape, adjudicated one by one
|
||||
# from source text: the suite's FAILED-branch summary line,
|
||||
# echo "wake <name> harness: FAILED ($(count_lines . "$FAILFILE") assertion(s))" >&2
|
||||
# guarded by `if [ -s "$FAILFILE" ]` — structurally unreachable while every
|
||||
# assertion passes, which is precisely the state a green validation run is
|
||||
# required to be in. (The tenth suite, test-wake-preimage.sh, uses its own
|
||||
# X/Y summary format with no grep in the red branch, so it has no row here.)
|
||||
#
|
||||
# The disposition is NOT "it would work": the exact template is EXECUTED red
|
||||
# in microtest C11 (deterministically failed mini-suite, same
|
||||
# count_lines-in-substitution summary shape → right count, exit 1, ledger row
|
||||
# at the summary coordinate), and the E-in-substitution abort path is proven
|
||||
# by microtest C5 plus the forced-error arm at test-wake-store-ack.sh:736.
|
||||
# Each site's conversion text is independently verified by the static
|
||||
# inventory (check-973.py static == expected, all 261 rows).
|
||||
#
|
||||
# Verified guard per site (line numbers at branch tip, +3 header shift):
|
||||
|
||||
count_lines test-wake-beacon.sh:350 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 349; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-detector.sh:702 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 701; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-digest-hmac.sh:434 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 433; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-digest-quarantine.sh:584 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 583; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-fn-oracle.sh:132 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 131; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-install.sh:434 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 433; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-reconcile.sh:389 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 388; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-store-ack.sh:741 — red-path summary; guard `[ -s "$FAILFILE" ]` at line 740; template execution measured by microtest C11; text verified by static inventory
|
||||
count_lines test-wake-store-enqueue-race.sh:208 — red-path summary (with "#927 TOCTOU reproduced (RED)" tail); guard `[ -s "$FAILFILE" ]` at line 207; template execution measured by microtest C11; text verified by static inventory
|
||||
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env bash
|
||||
# validate-973.sh — #973 validation driver. One run produces the complete
|
||||
# evidence chain for the 261-site conversion:
|
||||
#
|
||||
# 0. instrument self-test (microtest) — no validate evidence is trusted
|
||||
# before the instrument itself has been proven, including its abort arms.
|
||||
# 1. expected set: 261 coordinates from the FROZEN artifact (+3 header
|
||||
# shift), count asserted against the number declared below BEFORE any
|
||||
# suite runs.
|
||||
# 2. static inventory: converted call sites re-derived from SOURCE TEXT,
|
||||
# must equal the expected set exactly (amendment ONE, leg 1 — the
|
||||
# inventory comes from the text, never from the ledger).
|
||||
# 3. green instrumented run: all ten suites with WAKE_ASSERT_LEDGER; each
|
||||
# must exit 0 AND emit its own sentinel (per-suite formats differ and are
|
||||
# pinned here — a suite that died early must never pass on another
|
||||
# suite's output).
|
||||
# 4. trace arithmetic on coordinate SETS (loops re-execute sites and the
|
||||
# multi-grep lines append twice per pass, so counts are meaningless;
|
||||
# sets are not):
|
||||
# trace − expected MUST be empty (a helper ran at a coordinate the
|
||||
# denominator never measured);
|
||||
# expected − trace = converted-but-never-executed: enumerated, and
|
||||
# every entry must carry a disposition in the
|
||||
# committed unexecuted-sites-dispositions.txt, with
|
||||
# no stale dispositions the other way (amendment
|
||||
# ONE, legs 2+3 — the ledger is an execution trace,
|
||||
# not an inventory; the difference is enumerated and
|
||||
# individually dispositioned, never silently absent).
|
||||
# 5. forced-error arms: the 19 denominator canaries plus one E-form and one
|
||||
# F-form site, each run with WAKE_ASSERT_FORCE_GREP_ERROR_AT: the suite
|
||||
# must emit the ARMED line (the arm proved it fired), the ABORT line
|
||||
# naming the site, exit non-zero, emit NO sentinel, and the aborting
|
||||
# site's ledger row must already be present (the append lands before the
|
||||
# grep).
|
||||
# 6. residual sweep: the denominator's own classifier finds zero unconverted
|
||||
# verdict greps in the suites — and six per-form plants in the same run.
|
||||
#
|
||||
# Output discipline (A10): every line that reports on a suite names the file
|
||||
# under test; exit codes are reported before failure counts.
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WAKE="$(cd "$HERE/.." && pwd)"
|
||||
CHECK="$HERE/check-973.py"
|
||||
DISPO="$HERE/unexecuted-sites-dispositions.txt"
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# Declared BEFORE any suite runs (A2): the run must produce THESE numbers,
|
||||
# not be described by whatever numbers it produced.
|
||||
EXPECTED_SUITES=10
|
||||
EXPECTED_SITES=261
|
||||
EXPECTED_ARMS=21
|
||||
|
||||
fails=0
|
||||
flag() {
|
||||
printf 'FAIL %s\n' "$*"
|
||||
fails=$((fails + 1))
|
||||
}
|
||||
|
||||
SUITES=(
|
||||
test-wake-beacon.sh
|
||||
test-wake-detector.sh
|
||||
test-wake-digest-hmac.sh
|
||||
test-wake-digest-quarantine.sh
|
||||
test-wake-fn-oracle.sh
|
||||
test-wake-install.sh
|
||||
test-wake-preimage.sh
|
||||
test-wake-reconcile.sh
|
||||
test-wake-store-ack.sh
|
||||
test-wake-store-enqueue-race.sh
|
||||
)
|
||||
[ "${#SUITES[@]}" -eq "$EXPECTED_SUITES" ] ||
|
||||
flag "suite list has ${#SUITES[@]} entries, declared $EXPECTED_SUITES"
|
||||
|
||||
# Per-suite sentinel patterns, pinned: nine suites share the harness template
|
||||
# (enqueue-race appends a tail after it); preimage uses its own format.
|
||||
sentinel_for() {
|
||||
case "$1" in
|
||||
test-wake-preimage.sh) printf '%s' '^== test-wake-preimage: 17/17 passed ==$' ;;
|
||||
*) printf '%s' 'harness: all invariants passed' ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- 0: instrument self-test ------------------------------------------------
|
||||
if bash "$HERE/microtest-wake-assert.sh" >"$TMP/microtest.out" 2>&1; then
|
||||
echo "MICROTEST microtest-wake-assert.sh exit=0 (instrument proven)"
|
||||
else
|
||||
rc=$?
|
||||
echo "MICROTEST microtest-wake-assert.sh exit=$rc"
|
||||
sed 's/^/ /' "$TMP/microtest.out" | tail -n 15
|
||||
flag "instrument self-test failed — no validate evidence below is trustworthy"
|
||||
fi
|
||||
|
||||
# --- 1+2: expected set (artifact) vs static inventory (source text) ---------
|
||||
python3 "$CHECK" expected | sort >"$TMP/expected.txt" ||
|
||||
flag "check-973.py expected failed"
|
||||
n_expected="$(grep -c . "$TMP/expected.txt")"
|
||||
echo "EXPECTED-SET $n_expected coordinates (declared: $EXPECTED_SITES)"
|
||||
[ "$n_expected" -eq "$EXPECTED_SITES" ] ||
|
||||
flag "expected set has $n_expected coordinates, declared $EXPECTED_SITES"
|
||||
|
||||
python3 "$CHECK" static | sort >"$TMP/static.txt" ||
|
||||
flag "check-973.py static failed"
|
||||
if cmp -s "$TMP/expected.txt" "$TMP/static.txt"; then
|
||||
echo "STATIC-INVENTORY equals expected set ($(grep -c . "$TMP/static.txt") rows from source text)"
|
||||
else
|
||||
flag "static inventory (source text) differs from expected set (artifact):"
|
||||
diff "$TMP/expected.txt" "$TMP/static.txt" | head -n 20 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# --- 3: green instrumented run ----------------------------------------------
|
||||
LEDGER="$TMP/ledger"
|
||||
: >"$LEDGER"
|
||||
for s in "${SUITES[@]}"; do
|
||||
out="$(WAKE_ASSERT_LEDGER="$LEDGER" bash "$WAKE/$s" 2>&1)"
|
||||
rc=$?
|
||||
if printf '%s\n' "$out" | grep -Eq "$(sentinel_for "$s")"; then
|
||||
sent="present"
|
||||
else
|
||||
sent="ABSENT"
|
||||
fi
|
||||
echo "SUITE $s exit=$rc sentinel=$sent"
|
||||
[ "$rc" -eq 0 ] || flag "$s exited $rc in the green instrumented run"
|
||||
[ "$sent" = "present" ] || flag "$s did not emit its sentinel"
|
||||
done
|
||||
|
||||
# --- 4: trace arithmetic on coordinate sets ---------------------------------
|
||||
sort -u "$LEDGER" >"$TMP/trace.txt"
|
||||
echo "TRACE $(grep -c . "$TMP/trace.txt") distinct coordinates from $(grep -c . "$LEDGER") ledger rows"
|
||||
|
||||
comm -13 "$TMP/expected.txt" "$TMP/trace.txt" >"$TMP/rogue.txt"
|
||||
if [ -s "$TMP/rogue.txt" ]; then
|
||||
flag "trace contains coordinates OUTSIDE the frozen denominator:"
|
||||
sed 's/^/ ROGUE /' "$TMP/rogue.txt"
|
||||
else
|
||||
echo "TRACE-MINUS-EXPECTED empty (no helper ran at an unmeasured coordinate)"
|
||||
fi
|
||||
|
||||
comm -23 "$TMP/expected.txt" "$TMP/trace.txt" >"$TMP/unexec.txt"
|
||||
n_unexec="$(grep -c . "$TMP/unexec.txt" || true)"
|
||||
echo "UNEXECUTED $n_unexec of $EXPECTED_SITES converted sites did not execute in the green run"
|
||||
if [ ! -f "$DISPO" ]; then
|
||||
flag "disposition file missing: $DISPO — every unexecuted site must be individually dispositioned"
|
||||
sed 's/^/ UNDISPOSITIONED /' "$TMP/unexec.txt"
|
||||
else
|
||||
awk '!/^#/ && NF >= 2 {print $1, $2}' "$DISPO" | sort -u >"$TMP/dispo-keys.txt"
|
||||
comm -23 "$TMP/unexec.txt" "$TMP/dispo-keys.txt" >"$TMP/undispo.txt"
|
||||
comm -13 "$TMP/unexec.txt" "$TMP/dispo-keys.txt" >"$TMP/stale-dispo.txt"
|
||||
if [ -s "$TMP/undispo.txt" ]; then
|
||||
flag "unexecuted sites WITHOUT a disposition:"
|
||||
sed 's/^/ UNDISPOSITIONED /' "$TMP/undispo.txt"
|
||||
fi
|
||||
if [ -s "$TMP/stale-dispo.txt" ]; then
|
||||
flag "dispositions for sites that DID execute (stale — the file no longer matches the run):"
|
||||
sed 's/^/ STALE-DISPO /' "$TMP/stale-dispo.txt"
|
||||
fi
|
||||
if [ ! -s "$TMP/undispo.txt" ] && [ ! -s "$TMP/stale-dispo.txt" ]; then
|
||||
echo "DISPOSITIONS all $n_unexec unexecuted sites individually dispositioned, none stale"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- 5: forced-error arms ---------------------------------------------------
|
||||
python3 "$CHECK" arms >"$TMP/arms.txt" || flag "check-973.py arms failed"
|
||||
n_arms="$(grep -c . "$TMP/arms.txt")"
|
||||
echo "ARMS $n_arms forced-error arms (declared: $EXPECTED_ARMS)"
|
||||
[ "$n_arms" -eq "$EXPECTED_ARMS" ] ||
|
||||
flag "arm list has $n_arms entries, declared $EXPECTED_ARMS"
|
||||
|
||||
while read -r helper site form; do
|
||||
f="${site%%:*}"
|
||||
aled="$TMP/ledger-arm"
|
||||
: >"$aled"
|
||||
out="$(WAKE_ASSERT_LEDGER="$aled" WAKE_ASSERT_FORCE_GREP_ERROR_AT="$site" \
|
||||
bash "$WAKE/$f" 2>&1)"
|
||||
rc=$?
|
||||
bad=""
|
||||
[ "$rc" -ne 0 ] || bad="$bad exit=0"
|
||||
printf '%s\n' "$out" | grep -q "WAKE-ASSERT ARMED: forcing real grep error at $site" ||
|
||||
bad="$bad no-ARMED-line"
|
||||
printf '%s\n' "$out" | grep -q "WAKE-ASSERT ABORT: ${helper} at ${site}: grep exit" ||
|
||||
bad="$bad no-ABORT-line"
|
||||
printf '%s\n' "$out" | grep -Eq "$(sentinel_for "$f")" &&
|
||||
bad="$bad sentinel-emitted"
|
||||
grep -q "^${helper} ${site}\$" "$aled" ||
|
||||
bad="$bad no-ledger-row"
|
||||
if [ -z "$bad" ]; then
|
||||
echo "ARM $site ($form) exit=$rc armed+abort+no-sentinel+ledger-row"
|
||||
else
|
||||
echo "ARM $site ($form) exit=$rc DEFECTS:$bad"
|
||||
flag "arm $site ($form) failed:$bad"
|
||||
fi
|
||||
done <"$TMP/arms.txt"
|
||||
|
||||
# --- 6: residual sweep ------------------------------------------------------
|
||||
if python3 "$CHECK" sweep >"$TMP/sweep.out" 2>&1; then
|
||||
echo "SWEEP exit=0"
|
||||
else
|
||||
echo "SWEEP exit=$?"
|
||||
flag "residual sweep failed"
|
||||
fi
|
||||
sed 's/^/ /' "$TMP/sweep.out"
|
||||
|
||||
# --- summary (exit codes above, failure count last — A10) --------------------
|
||||
echo
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo "validate-973: FAILED ($fails failure(s))" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "validate-973: OK — $EXPECTED_SUITES suites, $EXPECTED_SITES sites, $EXPECTED_ARMS arms, sweep clean"
|
||||
Reference in New Issue
Block a user