docs: measure pi tool registry

2026-08-07 12:27:23 -05:00
parent 87ef1c1439
commit 289425ed2c
1 changed files with 93 additions and 0 deletions
@@ -0,0 +1,93 @@
+# W-B — Measure Pi's real tool registry
+
+- **Task / internal ref:** W-B from the lease-remediation orchestrator brief (no matching `docs/TASKS.md` row; workers do not modify that file)
+- **Objective:** identify the exact tool names emitted as `event.toolName` by the installed Pi runtime and compare them with the broker's Pi read-only carve-out.
+- **Scope:** measurement and report only; no broker or runtime source changes. W-C is out of scope.
+- **Budget:** no explicit token cap; constrained to this scratchpad and one local commit.
+- **Installed runtime:** `@earendil-works/pi-coding-agent` / `pi` `0.84.1`.
+
+## Method
+
+I created a throwaway extension at `/tmp/measure-pi-tool-registry.ts` (not in the worktree). On `session_start` it recorded `pi.getAllTools()` and `pi.getActiveTools()`; on every `tool_call` it appended the exact `event.toolName`. I then launched an isolated, ephemeral Pi session with all built-ins explicitly selected:
+
+```text
+PI_OFFLINE=1 pi --mode print --no-session --no-approve \
+  --no-context-files --no-skills --no-prompt-templates --no-extensions \
+  -e /tmp/measure-pi-tool-registry.ts \
+  --tools read,bash,edit,write,grep,find,ls <deterministic probe prompt>
+```
+
+The prompt exercised file read, content search, file search, directory listing, shell execution, file write, and file edit. Pi exited `0`; every selected tool produced one `tool_call`. The write/edit control artifact ended with exact content `after`, proving the mutating calls executed in order.
+
+This runtime observation was cross-checked against the installed distribution's canonical registry at `dist/core/tools/index.js:17`, which declares the same seven names. The gate consumes the measured field directly at `packages/mosaic/framework/runtime/pi/mosaic-extension.ts:368`.
+
+## Exact distinct built-in set
+
+The installed Pi built-in registry is exactly:
+
+```text
+{bash, edit, find, grep, ls, read, write}
+```
+
+| Tool | Runtime registry observation | `tool_call` observation | Installed definition |
+| --- | --- | --- | --- |
+| `read` | `<builtin:read>` | observed once | `dist/core/tools/read.js:138` |
+| `bash` | `<builtin:bash>` | observed once | `dist/core/tools/bash.js:231` |
+| `edit` | `<builtin:edit>` | observed once | `dist/core/tools/edit.js:170` |
+| `write` | `<builtin:write>` | observed once | `dist/core/tools/write.js:138` |
+| `grep` | `<builtin:grep>` | observed once | `dist/core/tools/grep.js:79` |
+| `find` | `<builtin:find>` | observed once | `dist/core/tools/find.js:79` |
+| `ls` | `<builtin:ls>` | observed once | `dist/core/tools/ls.js:61` |
+
+The raw distinct `event.toolName` result was:
+
+```json
+["bash", "edit", "find", "grep", "ls", "read", "write"]
+```
+
+Pi registers all seven, but its default active set is only `read`, `bash`, `edit`, and `write` (`dist/core/sdk.js:132`). The probe explicitly activated all seven so the three search/list tools could be observed at the hook.
+
+## Positive control
+
+The known `read` tool was the control. The method surfaced it twice:
+
+1. `pi.getAllTools()` returned `read` with source path `<builtin:read>`.
+2. Reading `/tmp/pi-registry-probe/seed.txt`, which contained `CONTROL_TOKEN`, produced one hook record with `event.toolName === "read"`.
+
+The control was therefore positive; the seven-name result is measured, not an empty-probe inference.
+
+## Carve-out comparison and collision result
+
+The broker currently declares `{"read", "grep", "find", "ls"}` at `packages/mosaic/framework/tools/lease-broker/daemon.py:54`.
+
+- `read`: real built-in.
+- `grep`: real built-in.
+- `find`: real built-in.
+- `ls`: real built-in.
+
+All four carve-out names are exact, case-sensitive Pi tool names.
+
+The general execution/writing tool names are `bash`, `edit`, and `write`. Their intersection with the carve-out is empty:
+
+```text
+{bash, edit, write} ∩ {read, grep, find, ls} = ∅
+```
+
+Therefore no general shell-exec or file-mutating Pi tool shares a name with a carve-out entry. `grep` and `find` may invoke constrained search helpers internally, but neither exposes an arbitrary command interface; the arbitrary command tool is distinctly named `bash`.
+
+The Mosaic extension separately registers the non-built-in custom tool `mosaic_context_recover` at `packages/mosaic/framework/runtime/pi/mosaic-extension.ts:379`; the broker handles that identity through its dedicated recovery exemption rather than the read-only set (`daemon.py:722`). Unknown or third-party custom tools are not part of Pi's built-in seven-name registry and remain outside the carve-out.
+
+## Verification evidence
+
+- `pi --version` → `0.84.1`.
+- Isolated probe exit → `0`.
+- Runtime `getAllTools()` count → `7`, all with `sourceInfo.source === "builtin"`.
+- Distinct hook names → `bash`, `edit`, `find`, `grep`, `ls`, `read`, `write`.
+- Hook counts → exactly one call for each of the seven names.
+- Mutation artifact after `write` then `edit` → exact content `after`.
+- Installed registry source → `allToolNames = new Set(["read", "bash", "edit", "write", "grep", "find", "ls"])`.
+
+## Risks / limitations
+
+- The probe deliberately disabled all other extensions, so extension-defined third-party tools were excluded from the built-in registry measurement. The production gate still receives those names and treats names outside the broker carve-out as mutating/fail-closed.
+- Explicit `--tools` activation was required to exercise `grep`, `find`, and `ls`; this does not imply they are active in Pi's default four-tool configuration.