jason.woltjeandClaude Opus 5.5
1abebe5321
Record for #1516 comment 26725: 34/34 cases reproduce, and a wrapped Claude command hook fails closed on crash, missing path, noexec and hang. Co-Authored-By: Claude Opus 5.5 <[email protected]>
22 lines
1.8 KiB
Markdown
22 lines
1.8 KiB
Markdown
**Row 34, S0 probe matrix, round 1: changes** (Darkwing, reviewer under decision 54)
|
|
|
|
Candidate `771fc3d2700bbadacc76b31534674ca342eb99ff`. The full record is `agents/darkwing/work/slice1-s0-review/review-r1.md` (local commit on `refactor`, not pushed).
|
|
|
|
The matrix is accurate. Every cell matches `evidence/<case>/` (exit, target, gate calls, tools offered, and the `is_error` and content the model saw). On my machine all 34 cases reproduce with identical results: 32/32 short cases and 2/2 of the 600 s cases. The scripted mock in a loopback-only namespace is a sound method for this matrix.
|
|
|
|
Required:
|
|
- **a.** Rely-on lines 2, 3 and 4 tell S6 to fall back on the tool limit for Claude Code command hooks that crash, sit at a missing path or hang. A wrapped hook fails closed in all three cases. I ran six extra cases on the cc-2 setup:
|
|
- `<gate> crash || exit 2` blocks.
|
|
- A missing gate and a non-executable gate, each with `|| exit 2`, block.
|
|
- `timeout 2 <gate> hang || exit 2`, with hook `timeout` 10, blocks in 2.4 s.
|
|
- Allow (`true || exit 2`) and the JSON deny still behave as before.
|
|
|
|
Please add wrapper cases and amend the three lines. The record has suggested text and the caveats: the inner timeout must sit below the hook timeout; use `timeout -k`; `--bare` still disables hooks.
|
|
- **b.** `evidence/cc-1d-block-bare` has no case in `probe.sh` and no row in the table (35 directories for 34 cases). Remove it or add the case.
|
|
|
|
Minor, can go in the same round:
|
|
- **c.** The cc-5c row omits `--allowedTools Read,Write`.
|
|
- **d.** Nothing committed writes the `runs.cmp`, `runs-2.cmp` and `saw-*.txt` files that `collect.sh` reads.
|
|
- **e.** `SDK_ENTRY` can't be overridden from the environment.
|
|
- **f.** "exit 0 in every run that reached a model" doesn't hold for pi-2c (3) and pi-4b (124).
|