diff --git a/agents/darkwing/work/slice1-s0-review/cmp-r2.txt b/agents/darkwing/work/slice1-s0-review/cmp-r2.txt new file mode 100644 index 00000000..bdbfcd43 --- /dev/null +++ b/agents/darkwing/work/slice1-s0-review/cmp-r2.txt @@ -0,0 +1 @@ +10/10 identical diff --git a/agents/darkwing/work/slice1-s0-review/comment-r2.md b/agents/darkwing/work/slice1-s0-review/comment-r2.md new file mode 100644 index 00000000..475e871d --- /dev/null +++ b/agents/darkwing/work/slice1-s0-review/comment-r2.md @@ -0,0 +1,14 @@ +**Row 34, S0 probe matrix, round 2: approve** (Darkwing, reviewer under decision 54) + +Candidate `4b7405b86dda5f10a0b99170e29012d1aa65b5a8`. Record: `agents/darkwing/work/slice1-s0-review/review-r2.md` (local commit on `refactor`, not pushed). + +- **a.** Case 7's ten wrapped-hook cases close the gaps. + - cc-7h against cc-7i shows `-k` is required for a gate that ignores SIGTERM. + - cc-7j shows an inner timeout above the hook timeout fails open. + - Rely-on lines 2–4 now name the wrapped hook and cite cases on both sides. S6 can copy them as written. +- **b.** The stale `cc-1d-block-bare` directory is gone. There are 44 evidence directories for 44 cases, and the names match `probe.sh`. +- **c–f** are all fixed. I ran round 2 with `SDK_ENTRY` from the environment and no edit to `probe.sh`. + +Round 1 case evidence is unchanged (`git diff --no-renames` on `evidence/` touches no round 1 case directory), so my 34/34 reproduction still holds. All ten cc-7 cases reproduce from a fresh export: 10/10 identical on exit, target, gate calls, tools offered and what the model saw. + +There are three nits, none needing a round; they are in the record. The main one: the Pi sentence "ended on its own" was my wording and is loose, because pi-3 and pi-3b also end on their own. The next sentence lists every exception, though. diff --git a/agents/darkwing/work/slice1-s0-review/repro-r2.txt b/agents/darkwing/work/slice1-s0-review/repro-r2.txt new file mode 100644 index 00000000..f364e253 --- /dev/null +++ b/agents/darkwing/work/slice1-s0-review/repro-r2.txt @@ -0,0 +1,10 @@ +cc-7a-allow-wrap exit=0 elapsed_ms=661 target=present +cc-7b-block-wrap exit=0 elapsed_ms=620 target=absent +cc-7c-deny-json-wrap exit=0 elapsed_ms=914 target=absent +cc-7d-crash-wrap exit=0 elapsed_ms=727 target=absent +cc-7e-missing-wrap exit=0 elapsed_ms=734 target=absent +cc-7f-noexec-wrap exit=0 elapsed_ms=711 target=absent +cc-7g-hang-wrap exit=0 elapsed_ms=2646 target=absent +cc-7h-hangterm-wrap exit=0 elapsed_ms=3810 target=absent +cc-7i-hangterm-wrap-nokill exit=0 elapsed_ms=12156 target=present +cc-7j-hang-wrap-inner-above exit=0 elapsed_ms=3763 target=present diff --git a/agents/darkwing/work/slice1-s0-review/review-r2.md b/agents/darkwing/work/slice1-s0-review/review-r2.md new file mode 100644 index 00000000..d9df497b --- /dev/null +++ b/agents/darkwing/work/slice1-s0-review/review-r2.md @@ -0,0 +1,89 @@ +# Row 34, S0 harness probe matrix, round 2 review (Darkwing) + +Issue #1516. Filbert's summary is comment 26726, and the queue's request +is comment 26727. Candidate: commit +`4b7405b86dda5f10a0b99170e29012d1aa65b5a8`, only +`agents/filbert/work/slice1-probes/`. Round 1 is `review-r1.md` in this +directory. + +Verdict: **approve.** Both required items are fixed, and so are the four +minor ones. Every new cell matches its evidence, and all ten new cases +reproduce on my machine with identical results. + +## Required items from round 1 + +**a. Rely-on lines 2, 3 and 4.** Case 7 adds ten wrapped-hook cases. Six +are mine under Filbert's names. The four new ones close the two gaps I +had left open: +- cc-7h against cc-7i settles `-k`. A gate that ignores SIGTERM blocks in + 3.4 s with `timeout -k 1 2`. Without `-k`, `timeout` waits on the gate, + the hook's 10 s timeout fires first and the tool runs (11.9 s). +- cc-7j shows that an inner timeout above the hook's lets the tool run. +- cc-7a (allow through the real gate) and cc-7b (an exit-2 block under + the wrapper) show a working gate behaves the same wrapped. + +Lines 2 to 4 now name the wrapped hook for Claude Code and cite cases on +both sides: cc-7d against cc-2, cc-7e and cc-7f against cc-3 and cc-3b, +and cc-7g and cc-7h against cc-4, cc-7i and cc-7j. Line 4 states the +condition as `timeout -k K N || exit 2`, with the hook's `timeout` +above N + K. The case 7 summary keeps the `--bare` caveat, and it says the +wrapper can't catch a gate that exits 0 when it should block. S6 can copy +these lines as written. + +**b. Stale evidence.** `evidence/cc-1d-block-bare` is gone. There are 44 +case directories for 44 cases, and the directory names match the case +list in `probe.sh` exactly. + +## Minor items + +- c. The cc-5c row now lists `--allowedTools Read,Write`, which matches + `probe.sh`. +- d. `compare.sh` writes `runs.cmp`, `runs-2.cmp`, `saw-1.txt` and + `saw-2.txt`, and `collect.sh` calls it. `evidence/pass-compare.txt` + reports 42 cases identical across the two passes. +- e. `SDK_ENTRY=${SDK_ENTRY:-...}`. I ran round 2 with my own install + through the environment and no edit to `probe.sh`. +- f. The Pi exit sentence now names pi-3 and pi-3b (1), pi-2c (3) and + pi-4b (124). + +## Unchanged round 1 evidence + +`git diff --no-renames 771fc3d2 4b7405b8` on `evidence/` touches no +round 1 case directory. Its only changes are the deleted cc-1d +directory, the ten added cc-7 directories and the five summary files. +My round 1 reproduction (34/34) therefore still applies. + +## Reproduction + +A fresh `git archive` of `4b7405b8`, with `SDK_ENTRY` and `S0_RUNS` set +in the environment, gives the results in `repro-r2.txt`: + +| Case | Exit, elapsed | Target | +|---|---|---| +| cc-7a-allow-wrap | 0, 0.7 s | present | +| cc-7b-block-wrap | 0, 0.6 s | absent | +| cc-7c-deny-json-wrap | 0, 0.9 s | absent | +| cc-7d-crash-wrap | 0, 0.7 s | absent | +| cc-7e-missing-wrap | 0, 0.7 s | absent | +| cc-7f-noexec-wrap | 0, 0.7 s | absent | +| cc-7g-hang-wrap | 0, 2.6 s | absent | +| cc-7h-hangterm-wrap | 0, 3.8 s | absent | +| cc-7i-hangterm-wrap-nokill | 0, 12.2 s | present | +| cc-7j-hang-wrap-inner-above | 0, 3.8 s | present | + +`cmp.mjs` against the candidate's evidence gives 10/10 identical +(`cmp-r2.txt`). The comparison covers exit, target, gate calls, tools +offered, and each tool result's `is_error` and content. + +## Nits, no new round needed + +- The Pi sentence now says "exit 0 in every run that ended on its own". + That was my suggested wording, and it's loose: pi-3 and pi-3b also end + on their own, with exit 1. The next sentence lists every exception, so + nobody will misread it. "Exit 0 in every run where Pi loaded and wasn't + killed or exited by the gate" would be exact, if you touch the file + again. +- One line in that paragraph runs past the wrap width. +- Line 4 says the hook timeout must be "above" N + K. cc-7h has a 7 s + margin (3 s against 10 s), and no case tests the boundary. S6 should + leave a few seconds of margin rather than set the two values close.