|
|
|
@@ -0,0 +1,89 @@
|
|
|
|
|
# Row 34, S0 harness probe matrix, round 2 review (Darkwing)
|
|
|
|
|
|
|
|
|
|
Issue #1516. Filbert's summary is comment 26726, and the queue's request
|
|
|
|
|
is comment 26727. Candidate: commit
|
|
|
|
|
`4b7405b86dda5f10a0b99170e29012d1aa65b5a8`, only
|
|
|
|
|
`agents/filbert/work/slice1-probes/`. Round 1 is `review-r1.md` in this
|
|
|
|
|
directory.
|
|
|
|
|
|
|
|
|
|
Verdict: **approve.** Both required items are fixed, and so are the four
|
|
|
|
|
minor ones. Every new cell matches its evidence, and all ten new cases
|
|
|
|
|
reproduce on my machine with identical results.
|
|
|
|
|
|
|
|
|
|
## Required items from round 1
|
|
|
|
|
|
|
|
|
|
**a. Rely-on lines 2, 3 and 4.** Case 7 adds ten wrapped-hook cases. Six
|
|
|
|
|
are mine under Filbert's names. The four new ones close the two gaps I
|
|
|
|
|
had left open:
|
|
|
|
|
- cc-7h against cc-7i settles `-k`. A gate that ignores SIGTERM blocks in
|
|
|
|
|
3.4 s with `timeout -k 1 2`. Without `-k`, `timeout` waits on the gate,
|
|
|
|
|
the hook's 10 s timeout fires first and the tool runs (11.9 s).
|
|
|
|
|
- cc-7j shows that an inner timeout above the hook's lets the tool run.
|
|
|
|
|
- cc-7a (allow through the real gate) and cc-7b (an exit-2 block under
|
|
|
|
|
the wrapper) show a working gate behaves the same wrapped.
|
|
|
|
|
|
|
|
|
|
Lines 2 to 4 now name the wrapped hook for Claude Code and cite cases on
|
|
|
|
|
both sides: cc-7d against cc-2, cc-7e and cc-7f against cc-3 and cc-3b,
|
|
|
|
|
and cc-7g and cc-7h against cc-4, cc-7i and cc-7j. Line 4 states the
|
|
|
|
|
condition as `timeout -k K N <gate> || exit 2`, with the hook's `timeout`
|
|
|
|
|
above N + K. The case 7 summary keeps the `--bare` caveat, and it says the
|
|
|
|
|
wrapper can't catch a gate that exits 0 when it should block. S6 can copy
|
|
|
|
|
these lines as written.
|
|
|
|
|
|
|
|
|
|
**b. Stale evidence.** `evidence/cc-1d-block-bare` is gone. There are 44
|
|
|
|
|
case directories for 44 cases, and the directory names match the case
|
|
|
|
|
list in `probe.sh` exactly.
|
|
|
|
|
|
|
|
|
|
## Minor items
|
|
|
|
|
|
|
|
|
|
- c. The cc-5c row now lists `--allowedTools Read,Write`, which matches
|
|
|
|
|
`probe.sh`.
|
|
|
|
|
- d. `compare.sh` writes `runs.cmp`, `runs-2.cmp`, `saw-1.txt` and
|
|
|
|
|
`saw-2.txt`, and `collect.sh` calls it. `evidence/pass-compare.txt`
|
|
|
|
|
reports 42 cases identical across the two passes.
|
|
|
|
|
- e. `SDK_ENTRY=${SDK_ENTRY:-...}`. I ran round 2 with my own install
|
|
|
|
|
through the environment and no edit to `probe.sh`.
|
|
|
|
|
- f. The Pi exit sentence now names pi-3 and pi-3b (1), pi-2c (3) and
|
|
|
|
|
pi-4b (124).
|
|
|
|
|
|
|
|
|
|
## Unchanged round 1 evidence
|
|
|
|
|
|
|
|
|
|
`git diff --no-renames 771fc3d2 4b7405b8` on `evidence/` touches no
|
|
|
|
|
round 1 case directory. Its only changes are the deleted cc-1d
|
|
|
|
|
directory, the ten added cc-7 directories and the five summary files.
|
|
|
|
|
My round 1 reproduction (34/34) therefore still applies.
|
|
|
|
|
|
|
|
|
|
## Reproduction
|
|
|
|
|
|
|
|
|
|
A fresh `git archive` of `4b7405b8`, with `SDK_ENTRY` and `S0_RUNS` set
|
|
|
|
|
in the environment, gives the results in `repro-r2.txt`:
|
|
|
|
|
|
|
|
|
|
| Case | Exit, elapsed | Target |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| cc-7a-allow-wrap | 0, 0.7 s | present |
|
|
|
|
|
| cc-7b-block-wrap | 0, 0.6 s | absent |
|
|
|
|
|
| cc-7c-deny-json-wrap | 0, 0.9 s | absent |
|
|
|
|
|
| cc-7d-crash-wrap | 0, 0.7 s | absent |
|
|
|
|
|
| cc-7e-missing-wrap | 0, 0.7 s | absent |
|
|
|
|
|
| cc-7f-noexec-wrap | 0, 0.7 s | absent |
|
|
|
|
|
| cc-7g-hang-wrap | 0, 2.6 s | absent |
|
|
|
|
|
| cc-7h-hangterm-wrap | 0, 3.8 s | absent |
|
|
|
|
|
| cc-7i-hangterm-wrap-nokill | 0, 12.2 s | present |
|
|
|
|
|
| cc-7j-hang-wrap-inner-above | 0, 3.8 s | present |
|
|
|
|
|
|
|
|
|
|
`cmp.mjs` against the candidate's evidence gives 10/10 identical
|
|
|
|
|
(`cmp-r2.txt`). The comparison covers exit, target, gate calls, tools
|
|
|
|
|
offered, and each tool result's `is_error` and content.
|
|
|
|
|
|
|
|
|
|
## Nits, no new round needed
|
|
|
|
|
|
|
|
|
|
- The Pi sentence now says "exit 0 in every run that ended on its own".
|
|
|
|
|
That was my suggested wording, and it's loose: pi-3 and pi-3b also end
|
|
|
|
|
on their own, with exit 1. The next sentence lists every exception, so
|
|
|
|
|
nobody will misread it. "Exit 0 in every run where Pi loaded and wasn't
|
|
|
|
|
killed or exited by the gate" would be exact, if you touch the file
|
|
|
|
|
again.
|
|
|
|
|
- One line in that paragraph runs past the wrap width.
|
|
|
|
|
- Line 4 says the hook timeout must be "above" N + K. cc-7h has a 7 s
|
|
|
|
|
margin (3 s against 10 s), and no case tests the boundary. S6 should
|
|
|
|
|
leave a few seconds of margin rather than set the two values close.
|