Compare commits

...
Author SHA1 Message Date
code-be-01 80752188b1 test(git-tools): classify tool failure at every needle site (#1380-FF)
ci/woodpecker/pr/ci Pipeline was successful
Pipeline 2637 red on a SECOND needle arm (home_case symlink message)
proved the flake shape is not one site: every 'if ! ... | grep -Fq'
assertion in the harness printed the identical FAIL line for a transient
grep failure (rc>=2, fork/alloc under node pressure) and a real mismatch.
The retry/classify helper is hoisted to one shared function and applied
at all three needle sites (fixture loop, home_case, standalone case):
retry tool errors 3x, rc=1 is the real mismatch, persistent tool failure
reports TOOL-ERROR (fail stays 1). 292/292 x2 + busybox.
2026-08-24 16:17:54 -05:00
code-be-01 b5792537b0 test(git-tools): wrapper-guard harness distinguishes tool failure from drift (#1380-FF)
ci/woodpecker/pr/ci Pipeline failed
Pipeline 2635 (merge-verify of 8eb7e635) failed one advice assertion
('a PR is an issue where labels live...') while pipeline 2634 (same
content, PR event) and 2636 (same tree + one commit) both passed, and the
fixture is stable under 200 direct iterations, 10 suite runs, four
locales, and 60 suite runs inside alpine/busybox 1.37 (the CI userspace).
Mechanism: the harness's 'if ! ... | grep -Fq' shape treats ANY nonzero
from the pipeline as an advice mismatch — including a transient grep
failure (fork/alloc error on a loaded k8s node), which prints the
identical FAIL line without any guard defect. The guard's rc-2 sites were
audited: every classification branch that can fire on this payload names
issue-edit.sh, so no code path produces the observed output.

Fix: the advice assertion now retries grep on rc>=2 (tool error) up to
3x, treats rc=1 as the real mismatch, and on persistent tool failure
prints a distinct TOOL-ERROR line (still fail=1 — never green on
infra noise, but the line names the class so a re-run can be judged
instead of debugged as guard drift). Advice-mismatch direction proven
unchanged by mutation (bogus wrapper name still FAILs both labels arms).
Suite 292/292 x2 locally and under busybox.
2026-08-24 16:08:43 -05:00
@@ -515,12 +515,40 @@ FIXTURES="$TMP/fixtures.tsv"
printf '0\t{"tool_input":{"command":"curl -s https://git.example.invalid/api/v1/repos/a/b/issues?q=a%%20b"}}\ta percent-escape in a READ is not this hook'"'"'s business\n'
} > "$FIXTURES"
# Flake containment (#1380-FF, pipelines 2635/2637): a transient failure of the
# ASSERTION TOOLING (grep rc>=2 — fork/alloc error under node pressure) is not a
# contract drift, but the `if ! ... | grep -Fq` shape printed the identical
# FAIL line for both, failing merge verifies on correct guard output. Classify
# instead: retry tool errors 3x; rc=1 is the real mismatch; persistent tool
# failure reports TOOL-ERROR (fail stays 1 — never green on infra noise — but
# the line names the class so a re-run can be judged, not debugged).
assert_out_contains() { # needle why
local attempt rc1
for attempt in 1 2 3; do
printf '%s' "$out" | grep -Fq -- "$1" && return 0
rc1=$?
[ "$rc1" -eq 1 ] && return 1 # grep answered NO — real mismatch
sleep 0.2 # rc>=2: grep itself errored — retry
done
printf 'TOOL-ERROR %s (assertion grep failed 3x — infra/tooling, not guard drift)\n' "$2" >&2
return 2
}
fail=0 n=0
while IFS=$'\t' read -r want payload why remedy; do
[ -n "${want:-}" ] || continue
n=$((n + 1))
out="$(printf '%s' "$payload" | "$GUARD" 2>&1)"
got=$?
# Flake containment (stack#1380-FF, pipeline 2635): a transient failure of
# the ASSERTION TOOLING (grep/fork/alloc error under node pressure) is not a
# contract drift, but this loop's `if !` shape made it indistinguishable
# from one — the harness printed the advice-mismatch FAIL and failed a merge
# verify on a run whose guard output was correct. Distinguish the two: an
# assertion tool that itself fails is retried a bounded number of times, and
# if it never succeeds the case reports a TOOL-ERROR line (fail=1 stays, so
# the run is never green-on-infra-noise, but the line names the real class
# and a re-run can be judged instead of debugged as a guard defect).
if [ "$got" != "$want" ]; then
printf 'FAIL %s (want exit %s, got %s)\n' "$why" "$want" "$got"
fail=1
@@ -530,10 +558,15 @@ while IFS=$'\t' read -r want payload why remedy; do
# now it was invisible here: the harness read the exit code and nothing else,
# so /issues/1/labels blocking with "use issue-create.sh" passed every run for
# six rounds. Where a fixture states the remediation it expects, assert it.
if [ -n "${remedy:-}" ] && ! printf '%s' "$out" | grep -Fq -- "$remedy"; then
printf 'FAIL %s (blocked, but the advice does not name %s)\n' "$why" "$remedy"
fail=1
continue
if [ -n "${remedy:-}" ]; then
assert_out_contains "$remedy" "$why"
arcret=$?
if [ "$arcret" -eq 1 ]; then
printf 'FAIL %s (blocked, but the advice does not name %s)\n' "$why" "$remedy"
fail=1
continue
fi
[ "$arcret" -eq 0 ] || { fail=1; continue; }
fi
printf 'ok %s\n' "$why"
done < "$FIXTURES"
@@ -572,10 +605,15 @@ home_case() {
fail=1
return
fi
if [ -n "$needle" ] && ! printf '%s' "$out" | grep -Fq -- "$needle"; then
printf 'FAIL %s (exit %s, but the message does not say %s)\n' "$why" "$got" "$needle"
fail=1
return
if [ -n "$needle" ]; then
assert_out_contains "$needle" "$why"
arcret=$?
if [ "$arcret" -eq 1 ]; then
printf 'FAIL %s (exit %s, but the message does not say %s)\n' "$why" "$got" "$needle"
fail=1
return
fi
[ "$arcret" -eq 0 ] || { fail=1; return; }
fi
printf 'ok %s\n' "$why"
}
@@ -665,10 +703,15 @@ lone_case() {
fail=1
return
fi
if [ -n "$needle" ] && ! printf '%s' "$out" | grep -Fq -- "$needle"; then
printf 'FAIL %s [standalone] (exit %s, but the message does not say %s)\n' "$why" "$got" "$needle"
fail=1
return
if [ -n "$needle" ]; then
assert_out_contains "$needle" "$why"
arcret=$?
if [ "$arcret" -eq 1 ]; then
printf 'FAIL %s [standalone] (exit %s, but the message does not say %s)\n' "$why" "$got" "$needle"
fail=1
return
fi
[ "$arcret" -eq 0 ] || { fail=1; return; }
fi
printf 'ok %s [standalone]\n' "$why"
}