stack/agents/filbert/work/slice1-probes/MATRIX.md

# Slice 1 S0: harness probe matrix (Filbert)

Row 34, issue #1516, reviewer Darkwing. Brief:
`docs/plans/2026-10-04_slice-1.md`, section "Slice 1 S0". Survey:
`agents/filbert/work/meta-harness-survey-2026-10-04.md`, section 9.

Versions probed: Pi 0.85.1 (`node_modules/@earendil-works/pi-coding-agent`,
`--version` prints 0.85.1) and Claude Code 2.1.289
(`~/.local/share/claude/versions/2.1.289`, sha256 `a186b99e…8d1348`).
Claude Agent SDK 0.3.289, whose `claudeCodeVersion` is 2.1.289, run against
that same binary. Node 26.8.1, kernel 7.2.2.

In slice 1, agents run as Jason's OS user. Every "blocks" below means the
harness refused the call; nothing here is a wall against a hostile process.
A hook is a rule that a well-behaved agent's harness applies.

## Method

- **The action.** Every case tries to create `work/target.txt`. The
  evidence is whether the file exists afterwards. Each gate appends a line
  to `gate.log` when it is called, so "the gate never ran" and "the gate
  ran and failed" are told apart.
- **A scripted model, not a real one (my choice).** `mock-anthropic.mjs`
  serves the Anthropic Messages API on loopback. On its first request it
  returns one scripted tool call; once that call has a tool result, it
  answers with text and ends the turn. It logs every request, including
  each `tool_result` the harness sent back. That logged content is "the
  message the model saw" in the tables. Tradeoff: the real provider path
  isn't exercised. The cases are deterministic, cost nothing and need no
  credentials. Hooks run inside the harness before any tool result is
  sent, so the provider doesn't change what they do.
- **Isolation.** `probe.sh` runs each case in a fresh
  `~/filbert-scratch/s0/runs/<case>/` with its own `home/` and `work/`.
  Each case gets:
  - `env -i` with only `PATH`, `LANG`, `HOME` and the probe variables, so
    no service tokens;
  - a network namespace with only `lo` up, so no network at all, not even
    the model providers;
  - uid 1000 inside the namespace, because Claude Code treats uid 0
    specially;
  - `CLAUDE_CONFIG_DIR` and `PI_CODING_AGENT_DIR` pointed into the run's
    `home/`, plus a dummy API key. The real `~/.claude` and `~/.pi` are
    never read.

  `inner.sh` records `uid=1000 net=lo` in every `outcome.txt`.
- **The SDK install.** The SDK isn't in this repository. I installed
  `@anthropic-ai/[email protected]` from the npm registry into
  `~/filbert-scratch/s0/sdk` with `--ignore-scripts --omit=optional
  --save-exact`. Its integrity matches the registry's (`sha512-fQRvZPhf…`).
  `pathToClaudeCodeExecutable` points it at the pinned 2.1.289 binary.
- **Runs.** 44 cases: 34 from round 1, plus the 10 wrapped-hook cases
  (`cc-7*`) added in round 2 after Darkwing's review. 42 of them ran twice,
  into `runs/` and `runs-2/`. Exit code, `target.txt` and the tool results
  the model saw are identical across the two passes. The two long
  default-timeout cases (`cc-4d`, `sdk-6c`) ran once.
- **Reproduce.** Run `probe.sh <case>…` or `probe.sh all` (set `SDK_ENTRY`
  to your SDK install's `sdk.mjs`, and `S0_RUNS` for the run directory),
  then `node summarize.mjs ~/filbert-scratch/s0/runs`. `compare.sh` writes
  and diffs the per-pass files (`runs.cmp`, `runs-2.cmp`, `saw-1.txt`,
  `saw-2.txt`), and `collect.sh` copies pass 1 into `evidence/`. Evidence
  for each case is in `evidence/<case>/`: `command.txt`, `env.txt`,
  `settings.json`, `outcome.txt`, `gate.log`, `mock.jsonl`, `stdout.txt` and
  `stderr.txt`.

### Base commands

Pi (`<gate>` is `-e <path>`; `GATE_MODE` picks the gate's behaviour):

```
node …/pi-coding-agent/dist/bundle/cli.js -p --mode json --provider probe --model probe-model \
  --no-session --no-extensions --no-skills --no-context-files --offline --tools read,write,bash <gate> probe
```

Claude Code (`settings.json` holds one `PreToolUse` hook, matcher `Write`
unless a case says otherwise):

```
~/.local/share/claude/versions/2.1.289 -p probe --output-format stream-json --verbose \
  --model claude-sonnet-4-5 --settings <run>/settings.json --strict-mcp-config \
  --no-session-persistence --tools Read,Write,Bash --allowedTools Read,Write,Bash
```

SDK: `node sdk-probe.mjs`. It calls `query()` with `tools` and
`allowedTools` set to `Read, Write, Bash`, `settingSources: []`, and one
`PreToolUse` callback on `Write`. `HOOK_MODE` picks the callback's
behaviour and `HOOK_TIMEOUT` sets the matcher's `timeout`.

Controls: `pi-0-nogate` and `cc-0-nogate`, run with no gate, both create
`target.txt`. The model saw `Successfully wrote to target.txt` (Pi) and
`File created successfully at: …` (Claude).

## The matrix

"Ran" means `target.txt` exists. "Fails closed" means a gate failure left
the action not done. Claude Code appends a `<system-reminder>` block (a
token count) to every tool result it sends; the cells leave it out, and
`evidence/<case>/mock.jsonl` has the full text. Paths in the evidence have
`$HOME` replaced by `~`.

### Case 1: a gate that blocks

| Case | Gate | Exit | Ran | Model saw (`is_error`, content) | Fails closed |
|---|---|---|---|---|---|
| pi-1-block | `tool_call` returns `{block: true, reason}` for `write` | 0 | no | true, `PROBE-GATE: write is blocked by policy` | n/a, the gate worked |
| cc-1-block | command hook, exit 2 with stderr | 0 | no | true, `PreToolUse:Write hook error: [<hook command>]: PROBE-GATE: write is blocked by policy (exit 2)` | n/a |
| cc-1b-deny-json | command hook, exit 0 with JSON `permissionDecision: "deny"` | 0 | no | true, `PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (JSON)` | n/a |
| cc-1c-block-bypass | as cc-1, under `--permission-mode bypassPermissions` | 0 | no | same as cc-1 | n/a |
| cc-1d-bash-hook | as cc-1, matcher `Bash`, model calls `Bash` | 0 | no | true, `PreToolUse:Bash hook error: […]` | n/a |
| cc-1e-bash-hook-bare | as cc-1d, plus `--bare` | 0 | **yes** | false, `(Bash completed with no output)`; `gate.log` empty | **no: `--bare` skips settings hooks** |
| sdk-6a-deny | SDK callback returns `permissionDecision: "deny"` | 0 | no | true, `PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (SDK callback)` | n/a |

Notes:
- A block from an exit-2 command hook shows the model the hook's full
  command line. The JSON deny shows only the reason.
- `--bare` also removed `Write` from the offered tools (cc-1e offered
  `Bash, Read`), which is why that case gates `Bash`. The help text says
  `--safe-mode` disables hooks too; I didn't probe it.

### Case 2: a gate that crashes

| Case | Gate | Exit | Ran | Model saw | Fails closed |
|---|---|---|---|---|---|
| pi-2-throw | handler throws `Error` | 0 | no | true, `PROBE-GATE crashed (throw)` | yes |
| pi-2b-throw-string | handler throws a string | 0 | no | true, `Extension failed, blocking execution: PROBE-GATE crashed (non-Error throw)` | yes |
| pi-2c-exit | handler calls `process.exit(3)` | 3 | no | nothing: Pi died after `tool_execution_start` and sent no second request | yes (the whole run dies) |
| cc-2-crash | command hook exits 1 | 0 | **yes** | false, `File created successfully at: …` | **no** |
| cc-2b-crash3 | command hook exits 3 | 0 | **yes** | false, `File created successfully at: …` | **no** |
| cc-2c-crash-events | as cc-2, plus `--include-hook-events` | 0 | **yes** | same as cc-2 | **no**; the stream gains `hook_response` with `exit_code: 1, outcome: "error"` |
| sdk-6-throw | see case 6 | | | | |

Without `--include-hook-events`, cc-2 and cc-2b leave no trace in the
stream-json output: the event types are `system:init`, `assistant`, `user`
and `result:success`. Stderr is empty. Pi's source matches its behaviour:
`_installAgentToolHooks` in the bundle chunk `chunk-JVUZSMYM.js` rethrows
any handler error, wrapping a non-`Error` value as "Extension failed,
blocking execution" (survey section 9 cites the same path in source).

### Case 3: a gate at a missing path

| Case | Gate | Exit | Ran | Model saw | Fails closed |
|---|---|---|---|---|---|
| pi-3-missing | `-e` names a file that doesn't exist | 1 | no | no request at all | yes: Pi refuses to start. Stderr: `Failed to load extension "…/no-such-gate.ts": Extension path does not exist` |
| pi-3b-broken | `-e` names a file with a syntax error | 1 | no | no request at all | yes. Stderr: `Failed to load extension … ParseError: Missing semicolon. …pi-gate-broken.ts:5:7` |
| cc-3-missing | hook `command` names a file that doesn't exist | 0 | **yes** | false, `File created successfully at: …` | **no, and silent**: no stderr, no stream event |
| cc-3b-noexec | hook `command` names a file without the execute bit | 0 | **yes** | false, `File created successfully at: …` | **no, and silent** |

This confirms the survey's claim, from the hooks page, that "a mistyped
path … leaves the gate silently disabled".

### Case 4: a gate that times out

| Case | Gate | Exit | Ran | Model saw | Fails closed |
|---|---|---|---|---|---|
| pi-4-hang | handler returns a promise that never settles and holds no timer or I/O | **0** after 0.35 s | no | nothing; the stream ends at `tool_execution_start`, with no `agent_end` | yes for the action, but **exit 0 reports success for a turn that never finished**. Node's event loop emptied and the process exited |
| pi-4b-slow | handler waits on a one-hour timer, as a gate blocked on I/O would | 124 (my 60 s wall clock) | no | nothing | yes, by stalling. **Pi has no timeout on a `tool_call` handler**: the run waited until the external limit killed it |
| cc-4-hang | command hook sleeps 60 s, `timeout: 3` | 0 after 3.4 s | **yes** | false, `File created successfully at: …` | **no** |
| cc-4b-hang-events | as cc-4, plus `--include-hook-events` | 0 | **yes** | same as cc-4 | **no**; `hook_response` shows `exit_code: 1, outcome: "cancelled"` |
| cc-4c-hang-default | command hook sleeps 60 s, then exits 2; no `timeout` set | 0 after 60.4 s | no | true, `…: PROBE-GATE: blocked after sleeping 60 s` | n/a: the default timeout is above 60 s |
| cc-4d-hang700-default | command hook sleeps 700 s, then exits 2; no `timeout` set | 0 after 600.5 s | **yes** | false, `File created successfully at: …` | **no**: the default timeout is 600 s, and when it expires the tool runs |
| sdk-6b-hang | SDK callback never settles, matcher `timeout: 3` | 0 after 3.4 s | no | true, `PreToolUse hook did not respond before its timeout (host client may be unreachable). The tool call was not executed; other configured hooks may not have completed.` | **yes** |
| sdk-6c-hang-default | SDK callback never settles; no `timeout` set | 0 after 600.6 s | no | the same timeout message as sdk-6b | **yes**: the default timeout is 600 s, and the tool isn't run |

### Case 5: a `bash` route to the same action

| Case | Setup | Exit | Ran | Model saw | Blocked |
|---|---|---|---|---|---|
| pi-5-bash | the case-1 gate (blocks `write`); model calls `bash` with `echo probe > target.txt` | 0 | **yes** | false, `(no output)` | **no**. The gate was called (`tool=bash`) and let it through, because it is keyed on the tool name |
| pi-5b-toollimit | no gate; `--tools read,write`; model calls `bash` | 0 | no | true, `Tool bash not found` | yes; `bash` wasn't offered (offered: `read, write`) |
| pi-5c-toollimit-write | no gate; `--tools read,bash`; model calls `write` | 0 | no | true, `Tool write not found` | yes |
| cc-5-bash | the case-1 hook (matcher `Write`); model calls `Bash` | 0 | **yes** | false, `(Bash completed with no output)` | **no**; the `Write` hook was never called |
| cc-5b-toollimit | no hook; `--tools Read,Write`; model calls `Bash` | 0 | no | true, `<tool_use_error>Error: No such tool available: Bash. Bash is disabled for this session, in subagents as well as here.</tool_use_error>` | yes; `Bash` wasn't offered |
| cc-5c-disallowed | no hook; `--tools Read,Write,Bash --allowedTools Read,Write --disallowedTools Bash` | 0 | no | same as cc-5b | yes; `Bash` wasn't offered |

### Case 6: an Agent SDK callback hook that throws (Claude Code only)

| Case | Callback | Exit | Ran | Model saw | Fails closed |
|---|---|---|---|---|---|
| sdk-6-throw | PreToolUse callback on `Write` throws `Error` | 0 | **yes** | false, `File created successfully at: …` | **no**. The CLI's stderr shows `Error in hook callback hook_0: … PROBE-GATE crashed (SDK callback throw)`; the query ends normally |
| sdk-6a-deny | control: the callback returns deny | 0 | no | see case 1 | n/a |
| sdk-6b-hang | the callback never settles, `timeout: 3` | 0 | no | see case 4 | yes |

This was the survey's "not established" item. It fails open.

### Case 7: wrapped command hooks (round 2)

Darkwing's review of round 1 ran six of these (`dw-*`, in
`agents/darkwing/work/slice1-s0-review/`). I reran them under my own
names and added four: an allow through a real gate, an exit-2 block, a gate
that ignores SIGTERM, and an inner timeout above the hook timeout.

Claude Code runs a command hook through `/bin/sh -c`. Exit 2 blocks and any
other non-zero exit lets the tool run (cases 2 to 4). Adding `|| exit 2`
turns every failure of the gate into exit 2. All cases use the case-1 setup:
hook matcher `Write`, and the model calls `Write`. `<gate>` is
`claude-gate.sh`; the `hangterm` mode runs `trap '' TERM` and then sleeps
60 s.

| Case | Hook command (hook `timeout`) | Exit | Ran | Model saw | Fails closed |
|---|---|---|---|---|---|
| cc-7a-allow-wrap | `<gate> allow \|\| exit 2` | 0 | yes | false, `File created successfully at: …` | n/a: an allowing gate still allows |
| cc-7b-block-wrap | `<gate> block \|\| exit 2` | 0 | no | true, `PreToolUse:Write hook error: [<hook command>]: PROBE-GATE: write is blocked by policy (exit 2)` | n/a: the block keeps its message |
| cc-7c-deny-json-wrap | `<gate> deny-json \|\| exit 2` | 0 | no | true, `PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (JSON)` | n/a |
| cc-7d-crash-wrap | `<gate> crash \|\| exit 2` | 0 | no | true, `…: PROBE-GATE crashed (exit 1)` | **yes** (cc-2: no) |
| cc-7e-missing-wrap | `<missing path> block \|\| exit 2` | 0 | no | true, `…: /bin/sh: line 1: …/no-such-gate.sh: No such file or directory` | **yes** (cc-3: no) |
| cc-7f-noexec-wrap | `<noexec gate> block \|\| exit 2` | 0 | no | true, `…: /bin/sh: line 1: …/claude-gate-noexec.sh: Permission denied` | **yes** (cc-3b: no) |
| cc-7g-hang-wrap | `timeout -k 1 2 <gate> hang \|\| exit 2` (10) | 0 after 2.7 s | no | true, `…: No stderr output` | **yes** (cc-4: no) |
| cc-7h-hangterm-wrap | `timeout -k 1 2 <gate> hangterm \|\| exit 2` (10) | 0 after 3.4 s | no | true, `…: No stderr output` | **yes**: SIGKILL one second after the ignored SIGTERM |
| cc-7i-hangterm-wrap-nokill | `timeout 2 <gate> hangterm \|\| exit 2` (10) | 0 after 11.9 s | **yes** | false, `File created successfully at: …` | **no**: without `-k`, `timeout` waits for the gate, the hook's 10 s timeout expires and the tool runs |
| cc-7j-hang-wrap-inner-above | `timeout -k 1 20 <gate> hang \|\| exit 2` (3) | 0 after 3.9 s | **yes** | false, `File created successfully at: …` | **no**: the hook's timeout expired before the inner one |

So the wrapper fails closed for a crash, a missing or non-executable
command and a hang, under three conditions:
- `timeout -k`, so a gate that ignores SIGTERM is still killed (cc-7h
  against cc-7i).
- The inner timeout plus its kill delay below the hook's `timeout`
  (cc-7j).
- No `--bare`, which skips settings hooks whatever their command (cc-1e).

The wrapper can't catch a gate that exits 0 when it should have blocked.

## Exit codes

- **Pi `--mode json`** (the survey's "inferred" item): exit 0 in every run
  that ended on its own, whether the tool ran, was blocked or the turn
  never finished (pi-4-hang). The other exit codes were 1 for an extension
  that fails to load (pi-3, pi-3b), the gate's own code after
  `process.exit` (3, pi-2c) and 124 from my wall clock (pi-4b). pi-2c and
  pi-4b both reached the model before they ended. A launcher can't read success from exit 0. A completed
  turn has an `agent_end` event in the stream.
- **Claude Code `-p`**: exit 0 and `result:success` in every run here,
  including blocked and fail-open runs.

## What slice 1 can rely on

One line per case. "Tool limit" means Pi `--tools` and Claude `--tools` or
`--disallowedTools`. These are rules the harness applies, not walls
(agents run as Jason's OS user).

1. **A gate that blocks**: the hook, on both harnesses: a Pi `tool_call`
   block, a Claude command hook (exit 2 or JSON deny, which holds under
   `bypassPermissions`), or an SDK callback deny. The launcher must not
   pass `--bare`, which turns Claude settings hooks off.
2. **A gate that crashes**: on Pi, the hook, because a throw blocks and a
   `process.exit` ends the run. On Claude Code, a command hook wrapped as
   `<gate> || exit 2`, because the wrapper turns a crash into a block
   (cc-7d); an unwrapped hook fails open (cc-2). An SDK callback fails
   open; see line 6.
3. **A gate at a missing path**: on Pi, the hook, because a missing or
   unloadable `-e` refuses to start. On Claude Code, a command hook wrapped
   as `<gate> || exit 2`, because `/bin/sh` fails on a missing or
   non-executable command and the wrapper turns that into a block (cc-7e,
   cc-7f); an unwrapped one is silently skipped (cc-3, cc-3b).
4. **A gate that times out**:
   - On Pi, the hook, which never lets the tool run, but only with an
     external wall clock and an `agent_end` check. Pi has no handler
     timeout, and a gate whose pending promise holds nothing open lets Pi
     exit 0 mid-turn.
   - On Claude Code, a command hook wrapped as `timeout -k K N <gate> ||
     exit 2`, with the hook's `timeout` above N + K (cc-7g, cc-7h). An
     unwrapped hook whose timeout expires (explicit, or the 600 s default)
     lets the tool run, and so does a wrapped one without `-k` whose gate
     ignores SIGTERM (cc-7i) or whose inner timeout is too long (cc-7j).
   - For an SDK callback, the hook, because an expired timeout (explicit,
     or the 600 s default) means the tool is not run.
5. **A `bash` route to the same action**: the tool limit, on both
   harnesses. A hook keyed on a tool name doesn't stop the same effect
   through `bash`.
6. **An Agent SDK callback that throws**: the tool limit; the callback
   fails open. A callback is usable as a hook only if it catches its own
   errors and returns deny, which is what sdk-6a shows.

## Files

- `mock-anthropic.mjs`: the scripted model.
- `probe.sh`: the case list and launcher.
- `compare.sh`: pass 1 against pass 2. `collect.sh`: copies pass 1 into
  `evidence/` and runs `compare.sh`.
- `inner.sh`: the in-namespace runner.
- `summarize.mjs`: per-case summary from the run directories.
- `pi-gate.ts`, `pi-gate-broken.ts`: Pi gates.
- `claude-gate.sh`, `claude-gate-noexec.sh`: Claude command hooks.
- `sdk-probe.mjs`: the SDK case.
- `evidence/`: pass 1 run records per case, plus `summary-pass1.txt`,
  `summary-pass2.txt` and `pass-compare.txt`.