stack/agents/filbert/work/slice1-probes/MATRIX.md

Slice 1 S0: harness probe matrix (Filbert)

Row 34, issue #1516, reviewer Darkwing. Brief: docs/plans/2026-10-04_slice-1.md, section "Slice 1 S0". Survey: agents/filbert/work/meta-harness-survey-2026-10-04.md, section 9.

Versions probed: Pi 0.85.1 (node_modules/@earendil-works/pi-coding-agent, --version prints 0.85.1) and Claude Code 2.1.289 (~/.local/share/claude/versions/2.1.289, sha256 a186b99e…8d1348). Claude Agent SDK 0.3.289, whose claudeCodeVersion is 2.1.289, run against that same binary. Node 26.8.1, kernel 7.2.2.

In slice 1, agents run as Jason's OS user. Every "blocks" below means the harness refused the call; nothing here is a wall against a hostile process. A hook is a rule that a well-behaved agent's harness applies.

Method

• The action. Every case tries to create work/target.txt. The evidence is whether the file exists afterwards. Each gate appends a line to gate.log when it is called, so "the gate never ran" and "the gate ran and failed" are told apart.

• A scripted model, not a real one (my choice). mock-anthropic.mjs serves the Anthropic Messages API on loopback. On its first request it returns one scripted tool call; once that call has a tool result, it answers with text and ends the turn. It logs every request, including each tool_result the harness sent back. That logged content is "the message the model saw" in the tables. Tradeoff: the real provider path isn't exercised. The cases are deterministic, cost nothing and need no credentials. Hooks run inside the harness before any tool result is sent, so the provider doesn't change what they do.

• Isolation. probe.sh runs each case in a fresh ~/filbert-scratch/s0/runs/<case>/ with its own home/ and work/. Each case gets:

  • env -i with only PATH, LANG, HOME and the probe variables, so no service tokens;
  • a network namespace with only lo up, so no network at all, not even the model providers;
  • uid 1000 inside the namespace, because Claude Code treats uid 0 specially;
  • CLAUDE_CONFIG_DIR and PI_CODING_AGENT_DIR pointed into the run's home/, plus a dummy API key. The real ~/.claude and ~/.pi are never read.

  inner.sh records uid=1000 net=lo in every outcome.txt.

• The SDK install. The SDK isn't in this repository. I installed @anthropic-ai/[email protected] from the npm registry into ~/filbert-scratch/s0/sdk with --ignore-scripts --omit=optional --save-exact. Its integrity matches the registry's (sha512-fQRvZPhf…). pathToClaudeCodeExecutable points it at the pinned 2.1.289 binary.

• Runs. 44 cases: 34 from round 1, plus the 10 wrapped-hook cases (cc-7*) added in round 2 after Darkwing's review. 42 of them ran twice, into runs/ and runs-2/. Exit code, target.txt and the tool results the model saw are identical across the two passes. The two long default-timeout cases (cc-4d, sdk-6c) ran once.

• Reproduce. Run probe.sh <case>… or probe.sh all (set SDK_ENTRY to your SDK install's sdk.mjs, and S0_RUNS for the run directory), then node summarize.mjs ~/filbert-scratch/s0/runs. compare.sh writes and diffs the per-pass files (runs.cmp, runs-2.cmp, saw-1.txt, saw-2.txt), and collect.sh copies pass 1 into evidence/. Evidence for each case is in evidence/<case>/: command.txt, env.txt, settings.json, outcome.txt, gate.log, mock.jsonl, stdout.txt and stderr.txt.

Base commands

Pi (<gate> is -e <path>; GATE_MODE picks the gate's behaviour):

node …/pi-coding-agent/dist/bundle/cli.js -p --mode json --provider probe --model probe-model \
  --no-session --no-extensions --no-skills --no-context-files --offline --tools read,write,bash <gate> probe

Claude Code (settings.json holds one PreToolUse hook, matcher Write unless a case says otherwise):

~/.local/share/claude/versions/2.1.289 -p probe --output-format stream-json --verbose \
  --model claude-sonnet-4-5 --settings <run>/settings.json --strict-mcp-config \
  --no-session-persistence --tools Read,Write,Bash --allowedTools Read,Write,Bash

SDK: node sdk-probe.mjs. It calls query() with tools and allowedTools set to Read, Write, Bash, settingSources: [], and one PreToolUse callback on Write. HOOK_MODE picks the callback's behaviour and HOOK_TIMEOUT sets the matcher's timeout.

Controls: pi-0-nogate and cc-0-nogate, run with no gate, both create target.txt. The model saw Successfully wrote to target.txt (Pi) and File created successfully at: … (Claude).

The matrix

"Ran" means target.txt exists. "Fails closed" means a gate failure left the action not done. Claude Code appends a <system-reminder> block (a token count) to every tool result it sends; the cells leave it out, and evidence/<case>/mock.jsonl has the full text. Paths in the evidence have $HOME replaced by ~.

Case 1: a gate that blocks

Case	Gate	Exit	Ran	Model saw (is_error, content)	Fails closed
pi-1-block	tool_call returns {block: true, reason} for write	0	no	true, PROBE-GATE: write is blocked by policy	n/a, the gate worked
cc-1-block	command hook, exit 2 with stderr	0	no	true, PreToolUse:Write hook error: [<hook command>]: PROBE-GATE: write is blocked by policy (exit 2)	n/a
cc-1b-deny-json	command hook, exit 0 with JSON permissionDecision: "deny"	0	no	true, PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (JSON)	n/a
cc-1c-block-bypass	as cc-1, under --permission-mode bypassPermissions	0	no	same as cc-1	n/a
cc-1d-bash-hook	as cc-1, matcher Bash, model calls Bash	0	no	true, PreToolUse:Bash hook error: […]	n/a
cc-1e-bash-hook-bare	as cc-1d, plus --bare	0	yes	false, (Bash completed with no output); gate.log empty	no: --bare skips settings hooks
sdk-6a-deny	SDK callback returns permissionDecision: "deny"	0	no	true, PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (SDK callback)	n/a

Notes:

• A block from an exit-2 command hook shows the model the hook's full command line. The JSON deny shows only the reason.
• --bare also removed Write from the offered tools (cc-1e offered Bash, Read), which is why that case gates Bash. The help text says --safe-mode disables hooks too; I didn't probe it.

Case 2: a gate that crashes

Case	Gate	Exit	Ran	Model saw	Fails closed
pi-2-throw	handler throws Error	0	no	true, PROBE-GATE crashed (throw)	yes
pi-2b-throw-string	handler throws a string	0	no	true, Extension failed, blocking execution: PROBE-GATE crashed (non-Error throw)	yes
pi-2c-exit	handler calls process.exit(3)	3	no	nothing: Pi died after tool_execution_start and sent no second request	yes (the whole run dies)
cc-2-crash	command hook exits 1	0	yes	false, File created successfully at: …	no
cc-2b-crash3	command hook exits 3	0	yes	false, File created successfully at: …	no
cc-2c-crash-events	as cc-2, plus --include-hook-events	0	yes	same as cc-2	no; the stream gains hook_response with exit_code: 1, outcome: "error"
sdk-6-throw	see case 6

Without --include-hook-events, cc-2 and cc-2b leave no trace in the stream-json output: the event types are system:init, assistant, user and result:success. Stderr is empty. Pi's source matches its behaviour: _installAgentToolHooks in the bundle chunk chunk-JVUZSMYM.js rethrows any handler error, wrapping a non-Error value as "Extension failed, blocking execution" (survey section 9 cites the same path in source).

Case 3: a gate at a missing path

Case	Gate	Exit	Ran	Model saw	Fails closed
pi-3-missing	-e names a file that doesn't exist	1	no	no request at all	yes: Pi refuses to start. Stderr: Failed to load extension "…/no-such-gate.ts": Extension path does not exist
pi-3b-broken	-e names a file with a syntax error	1	no	no request at all	yes. Stderr: Failed to load extension … ParseError: Missing semicolon. …pi-gate-broken.ts:5:7
cc-3-missing	hook command names a file that doesn't exist	0	yes	false, File created successfully at: …	no, and silent: no stderr, no stream event
cc-3b-noexec	hook command names a file without the execute bit	0	yes	false, File created successfully at: …	no, and silent

This confirms the survey's claim, from the hooks page, that "a mistyped path … leaves the gate silently disabled".

Case 4: a gate that times out

Case	Gate	Exit	Ran	Model saw	Fails closed
pi-4-hang	handler returns a promise that never settles and holds no timer or I/O	0 after 0.35 s	no	nothing; the stream ends at tool_execution_start, with no agent_end	yes for the action, but exit 0 reports success for a turn that never finished. Node's event loop emptied and the process exited
pi-4b-slow	handler waits on a one-hour timer, as a gate blocked on I/O would	124 (my 60 s wall clock)	no	nothing	yes, by stalling. Pi has no timeout on a tool_call handler: the run waited until the external limit killed it
cc-4-hang	command hook sleeps 60 s, timeout: 3	0 after 3.4 s	yes	false, File created successfully at: …	no
cc-4b-hang-events	as cc-4, plus --include-hook-events	0	yes	same as cc-4	no; hook_response shows exit_code: 1, outcome: "cancelled"
cc-4c-hang-default	command hook sleeps 60 s, then exits 2; no timeout set	0 after 60.4 s	no	true, …: PROBE-GATE: blocked after sleeping 60 s	n/a: the default timeout is above 60 s
cc-4d-hang700-default	command hook sleeps 700 s, then exits 2; no timeout set	0 after 600.5 s	yes	false, File created successfully at: …	no: the default timeout is 600 s, and when it expires the tool runs
sdk-6b-hang	SDK callback never settles, matcher timeout: 3	0 after 3.4 s	no	true, PreToolUse hook did not respond before its timeout (host client may be unreachable). The tool call was not executed; other configured hooks may not have completed.	yes
sdk-6c-hang-default	SDK callback never settles; no timeout set	0 after 600.6 s	no	the same timeout message as sdk-6b	yes: the default timeout is 600 s, and the tool isn't run

Case 5: a bash route to the same action

Case	Setup	Exit	Ran	Model saw	Blocked
pi-5-bash	the case-1 gate (blocks write); model calls bash with echo probe > target.txt	0	yes	false, (no output)	no. The gate was called (tool=bash) and let it through, because it is keyed on the tool name
pi-5b-toollimit	no gate; --tools read,write; model calls bash	0	no	true, Tool bash not found	yes; bash wasn't offered (offered: read, write)
pi-5c-toollimit-write	no gate; --tools read,bash; model calls write	0	no	true, Tool write not found	yes
cc-5-bash	the case-1 hook (matcher Write); model calls Bash	0	yes	false, (Bash completed with no output)	no; the Write hook was never called
cc-5b-toollimit	no hook; --tools Read,Write; model calls Bash	0	no	true, <tool_use_error>Error: No such tool available: Bash. Bash is disabled for this session, in subagents as well as here.</tool_use_error>	yes; Bash wasn't offered
cc-5c-disallowed	no hook; --tools Read,Write,Bash --allowedTools Read,Write --disallowedTools Bash	0	no	same as cc-5b	yes; Bash wasn't offered

Case 6: an Agent SDK callback hook that throws (Claude Code only)

Case	Callback	Exit	Ran	Model saw	Fails closed
sdk-6-throw	PreToolUse callback on Write throws Error	0	yes	false, File created successfully at: …	no. The CLI's stderr shows Error in hook callback hook_0: … PROBE-GATE crashed (SDK callback throw); the query ends normally
sdk-6a-deny	control: the callback returns deny	0	no	see case 1	n/a
sdk-6b-hang	the callback never settles, timeout: 3	0	no	see case 4	yes

This was the survey's "not established" item. It fails open.

Case 7: wrapped command hooks (round 2)

Darkwing's review of round 1 ran six of these (dw-*, in agents/darkwing/work/slice1-s0-review/). I reran them under my own names and added four: an allow through a real gate, an exit-2 block, a gate that ignores SIGTERM, and an inner timeout above the hook timeout.

Claude Code runs a command hook through /bin/sh -c. Exit 2 blocks and any other non-zero exit lets the tool run (cases 2 to 4). Adding || exit 2 turns every failure of the gate into exit 2. All cases use the case-1 setup: hook matcher Write, and the model calls Write. <gate> is claude-gate.sh; the hangterm mode runs trap '' TERM and then sleeps 60 s.

Case	Hook command (hook timeout)	Exit	Ran	Model saw	Fails closed
cc-7a-allow-wrap	<gate> allow || exit 2	0	yes	false, File created successfully at: …	n/a: an allowing gate still allows
cc-7b-block-wrap	<gate> block || exit 2	0	no	true, PreToolUse:Write hook error: [<hook command>]: PROBE-GATE: write is blocked by policy (exit 2)	n/a: the block keeps its message
cc-7c-deny-json-wrap	<gate> deny-json || exit 2	0	no	true, PreToolUse:Write hook error: PROBE-GATE: write is denied by policy (JSON)	n/a
cc-7d-crash-wrap	<gate> crash || exit 2	0	no	true, …: PROBE-GATE crashed (exit 1)	yes (cc-2: no)
cc-7e-missing-wrap	<missing path> block || exit 2	0	no	true, …: /bin/sh: line 1: …/no-such-gate.sh: No such file or directory	yes (cc-3: no)
cc-7f-noexec-wrap	<noexec gate> block || exit 2	0	no	true, …: /bin/sh: line 1: …/claude-gate-noexec.sh: Permission denied	yes (cc-3b: no)
cc-7g-hang-wrap	timeout -k 1 2 <gate> hang || exit 2 (10)	0 after 2.7 s	no	true, …: No stderr output	yes (cc-4: no)
cc-7h-hangterm-wrap	timeout -k 1 2 <gate> hangterm || exit 2 (10)	0 after 3.4 s	no	true, …: No stderr output	yes: SIGKILL one second after the ignored SIGTERM
cc-7i-hangterm-wrap-nokill	timeout 2 <gate> hangterm || exit 2 (10)	0 after 11.9 s	yes	false, File created successfully at: …	no: without -k, timeout waits for the gate, the hook's 10 s timeout expires and the tool runs
cc-7j-hang-wrap-inner-above	timeout -k 1 20 <gate> hang || exit 2 (3)	0 after 3.9 s	yes	false, File created successfully at: …	no: the hook's timeout expired before the inner one

So the wrapper fails closed for a crash, a missing or non-executable command and a hang, under three conditions:

• timeout -k, so a gate that ignores SIGTERM is still killed (cc-7h against cc-7i).
• The inner timeout plus its kill delay below the hook's timeout (cc-7j).
• No --bare, which skips settings hooks whatever their command (cc-1e).

The wrapper can't catch a gate that exits 0 when it should have blocked.

Exit codes

• Pi --mode json (the survey's "inferred" item): exit 0 in every run that ended on its own, whether the tool ran, was blocked or the turn never finished (pi-4-hang). The other exit codes were 1 for an extension that fails to load (pi-3, pi-3b), the gate's own code after process.exit (3, pi-2c) and 124 from my wall clock (pi-4b). pi-2c and pi-4b both reached the model before they ended. A launcher can't read success from exit 0. A completed turn has an agent_end event in the stream.
• Claude Code -p: exit 0 and result:success in every run here, including blocked and fail-open runs.

What slice 1 can rely on

One line per case. "Tool limit" means Pi --tools and Claude --tools or --disallowedTools. These are rules the harness applies, not walls (agents run as Jason's OS user).

1. A gate that blocks: the hook, on both harnesses: a Pi tool_call block, a Claude command hook (exit 2 or JSON deny, which holds under bypassPermissions), or an SDK callback deny. The launcher must not pass --bare, which turns Claude settings hooks off.
2. A gate that crashes: on Pi, the hook, because a throw blocks and a process.exit ends the run. On Claude Code, a command hook wrapped as <gate> || exit 2, because the wrapper turns a crash into a block (cc-7d); an unwrapped hook fails open (cc-2). An SDK callback fails open; see line 6.
3. A gate at a missing path: on Pi, the hook, because a missing or unloadable -e refuses to start. On Claude Code, a command hook wrapped as <gate> || exit 2, because /bin/sh fails on a missing or non-executable command and the wrapper turns that into a block (cc-7e, cc-7f); an unwrapped one is silently skipped (cc-3, cc-3b).
4. A gate that times out:
   • On Pi, the hook, which never lets the tool run, but only with an external wall clock and an agent_end check. Pi has no handler timeout, and a gate whose pending promise holds nothing open lets Pi exit 0 mid-turn.
   • On Claude Code, a command hook wrapped as timeout -k K N <gate> || exit 2, with the hook's timeout above N + K (cc-7g, cc-7h). An unwrapped hook whose timeout expires (explicit, or the 600 s default) lets the tool run, and so does a wrapped one without -k whose gate ignores SIGTERM (cc-7i) or whose inner timeout is too long (cc-7j).
   • For an SDK callback, the hook, because an expired timeout (explicit, or the 600 s default) means the tool is not run.
5. A bash route to the same action: the tool limit, on both harnesses. A hook keyed on a tool name doesn't stop the same effect through bash.
6. An Agent SDK callback that throws: the tool limit; the callback fails open. A callback is usable as a hook only if it catches its own errors and returns deny, which is what sdk-6a shows.

Files

• mock-anthropic.mjs: the scripted model.
• probe.sh: the case list and launcher.
• compare.sh: pass 1 against pass 2. collect.sh: copies pass 1 into evidence/ and runs compare.sh.
• inner.sh: the in-namespace runner.
• summarize.mjs: per-case summary from the run directories.
• pi-gate.ts, pi-gate-broken.ts: Pi gates.
• claude-gate.sh, claude-gate-noexec.sh: Claude command hooks.
• sdk-probe.mjs: the SDK case.
• evidence/: pass 1 run records per case, plus summary-pass1.txt, summary-pass2.txt and pass-compare.txt.