stack/agents/researcher/work/i1-proxy-test/REPORT.md

Row 96 (#1590): stand-in-key test of CLIProxyAPI for role sessions

Author: Researcher seat. Brief: docs/plans/2026-10-12_local-model-proxy.md § "Stand-in-key test of CLIProxyAPI for role sessions". Reviewer: Dewey. Evidence for every claim is in out/. The numbers below come from the final run (out/run.log, started 2026-10-13T20:57:40Z).

Verdict in short

• A second CLIProxyAPI process carries both pinned clients (Claude Code 2.1.296, pi 0.85.1) through a streamed turn and a tool round trip, using only stand-in keys.
• Claude Code works with ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN as the session allowlist stands. Pi does not: it ignores ANTHROPIC_BASE_URL. It needs a models.json in the session's HOME (details in "Pi finding").
• Per-agent keys work from the command line: issue = 1 PUT, revoke = 1 DELETE, no restart, effective on the next request. Keys survive a proxy restart.
• The upstream sees only the proxy's own key. The client key never leaves the proxy.
• What does not hold or is weak: attribution by client key (queue-only, 60 s window, plaintext keys), the revoke call's key in the access log, silent multi-minute retry in Claude Code on 401, and every management write rewriting the whole config file. See "What doesn't hold".

Bounds kept

• No real credential, no spend, no request off the machine. Everything runs inside unshare -rn: a private network namespace with only lo. The test processes cannot reach the live proxy on 127.0.0.1:8317 or anything else. out/run.log line 3 lists the listeners visible inside the namespace (18317, 18400 only).
• The running proxy was not called, restarted or configured. Its management key and config content were never read. run-all.sh records before/after: pid 113963, start time, argv, the 8317 listener, and stat (mtime, size) of its config. All identical (out/host-guard.before.txt, .after.txt, .diff empty, host-guard.txt).
• Files under the live proxy's data directory (core-start-output.log, usage.db) did change during the run. Those belong to the live proxy serving other clients (2 to 6 established connections to 8317 from other host processes at the time). The test cannot touch them from inside the namespace. They are listed as informational in out/host-activity.txt, not in the verdict, because they would change whatever the test does.
• Nothing left behind: cleanup.sh ran, out/host-guard.txt says "left behind outside out/: 0".

Proxy version, config fields, endpoints

cli-proxy-api -h and the startup line: CLIProxyAPI 8.0.27, commit bf934162, built 2026-10-12T16:37:54Z (out/run.log line 4; flags in docs/proxy-help.txt). Started as cli-proxy-api -config state/config.yaml -local-model under env -i HOME=… PATH=….

Config fields used, from docs/config.example.v8.0.27.yaml (the example config shipped with 8.0.27; the line numbers are in that file):

Field	Line	Used for
config-version	10	8
server.host / port	24	127.0.0.1:18317
management.secret-key	89	made-up key; an empty value disables management
management.allow-remote	84	false
management.disable-control-panel	92	true (no WebUI)
access.api-keys	106, 108	client keys the proxy accepts
routing.retry.request-retry	149, 194	0 (no hidden proxy retries)
oauth.auth-dir	884	inside the packet, so request logs land there
observability.logs.debug / logging-to-file / request-log / logs-max-total-size-mb	1114 to 1131	per-request logs; size cap 200 MB
observability.usage.usage-statistics-enabled	1133, 1135	true
observability.usage.redis-usage-queue-retention-seconds	1140	default 60 (not set)
api-keys.claude[] (name, base-url, keys[].api-key)	~650 to 720	the proxy's own upstream: the stand-in server and its made-up key

Management endpoints used, from docs/management-api-v8.0.27.md. Auth is Authorization: Bearer <management key> (or X-Management-Key):

Endpoint	Used for
GET/PUT /v8/management/config/access/api-keys	list, replace the client-key list
DELETE /v0/management/api-keys?value=<key>	remove one client key
GET /v8/management/observability/usage/api-keys	usage counters
GET /v8/management/observability/usage/queue	per-request usage events
GET /v8/management/observability/logs	log lines

Also noted in the doc and not used: PATCH on the access/api-keys path, and /observability/logs/requests/<id>. I did not test whether a v8 DELETE exists; the v0 one worked.

Stand-in upstream

stand-in-upstream.mjs: HTTP on 127.0.0.1:18400, accepts only standin-upstream-key-0001 (Authorization: Bearer or x-api-key), records every request as JSONL (out/upstream.all.jsonl), answers like the Messages API: SSE stream in three chunks, a Read tool call when the prompt contains TOOL-TEST, and ROUNDTRIP-OK: <tool result> when the history ends in the matching tool_result. It also answers count_tokens and /v1/models.

Client runs

Both clients ran through the repo's real adapters (adapters/claude/adapter.sh, adapters/pi/adapter.sh) under env -i with only the session allowlist (packages/seat/src/session.mjs ENV_ALLOW: HOME, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, MOSAIC_CLAUDE_BIN, …) plus the adapter's MOSAIC_* inputs. Claude ran with </dev/null under timeout -k 2.

Run	Exit	Result
S2-claude-A (Claude Code, key A)	0	ROUNDTRIP-OK: 1 probe-file-content: STAND-IN-PROBE-7f3a 2; 2 upstream requests (tool call, then tool result)
S2-pi-A (pi, key A, models.json override)	0	ROUNDTRIP-OK: probe-file-content: STAND-IN-PROBE-7f3a; 2 upstream requests
S2-pi-env-A (pi, key A, no override)	1	Connection error.; 0 upstream requests
S3-claude-B	0	streamed answer intact: STAND-IN-ANSWER-11 (streamed in three chunks)

Per request, from out/analysis.txt (proxy request log = what the client sent; upstream log = what arrived):

Check	Claude Code	pi
Key the proxy saw	key A, Authorization: Bearer	key A, x-api-key
Key the upstream saw	standin-upstream-key-0001 via Authorization: Bearer, accepted	same
Client key present anywhere in upstream headers	no	no
anthropic-version	2023-06-01 → 2023-06-01	same
anthropic-beta	9 flags → same 9, identical	1 flag → same, identical
cache_control blocks	3 → 3	3 → 3
Body	equal as JSON	equal as JSON
Headers dropped	none	sec-fetch-mode, accept-language
Headers changed	x-stainless-package-version 0.128.0 → 0.112.1	none

The proxy accepts both header styles for the client key. What it changed is limited to the SDK version header and two browser-style headers; nothing the Anthropic API uses to select behaviour was dropped. Caveat: the stand-in does not model what the real API does with these headers, so this shows pass-through, not API acceptance.

Pi finding

Pinned pi 0.85.1 ignores ANTHROPIC_BASE_URL. The Anthropic SDK reads that variable only as a default; pi-ai passes the built-in model's hardcoded baseUrl (https://api.anthropic.com), so a bare variable sends the request to api.anthropic.com (here the namespace has no route, hence Connection error; on the host it would go out). Pi's own documentation (models.md § "Overriding Built-in Providers") gives the supported route: ~/.pi/agent/models.json with {"providers":{"anthropic":{"baseUrl":"<proxy URL>"}}}, no /v1 suffix. HOME is on the allowlist, so a per-session HOME containing that file works with ANTHROPIC_API_KEY carrying the client key. inner.sh kind pi does exactly this (state/home-<label>/.pi/agent/models.json, 1 file).

Change needed for sessions: the session builder (or the pi adapter) must write that models.json into the session's HOME when the proxy is in use. An env-only change is not enough. Pi also prints Warning: Model "claude-sonnet-5-5" not found for provider "anthropic". Using custom model id. on stderr; harmless here.

Per-agent keys from the command line

Step	Call	Result
Issue A and B	PUT /v8/management/config/access/api-keys with [bootstrap, A, B] (plus 1 GET before, 1 GET after to verify)	200; A and B work on the very next request (S1-A-immediately, S1-B-immediately: HTTP 200). No restart.
Revoke A	DELETE /v0/management/api-keys?value=A (plus 1 GET to verify)	200; A's next request: HTTP 401 {"error":"Invalid API key"}. B: still 200, Claude B run exit 0.
After restart	proxy killed and started again with the same config file	B 200, A 401 (S4b-*). Keys persist.

Step count: issuing N keys is 1 write; revoking one is 1 write. A bootstrap key must stay in the list so the proxy never falls into open mode (see below). No WebUI was used (disable-control-panel: true).

Attribution, as observed:

• GET …/usage/queue has one event per request: api_key (the full plaintext client key), access_provider: config-inline, source (upstream key), request_id, failed, fail.status_code, user agent. Claude A run: 2 events with key A; pi A run: 2 events with key A; Claude B: 1 event with key B. A revoked-key request leaves no event.
• Reads drain: a plain GET returns one event and removes it; ?count=100 returned all 3 pending events; the next read returned 0 (out/queue-read-semantics.txt). Undrained events expire after 60 s (redis-usage-queue-retention-seconds, default 60). Evidence: the upstream-down Claude run made about 8 failed requests but only the last was still in the queue when read, while the usage counter says 13 failures in total.
• GET …/usage/api-keys is keyed by upstream credential (http://127.0.0.1:18400|standin-upstream-key-0001: success/failed), not by client key. It cannot answer "what did agent A do".
• Request logs (out/proxy-logs/v1-messages-*.log) show the client key masked (first 4 and last 4: sk-s...df78), so A and B can be told apart in them when the tails differ. A refused request produces a log with no upstream call.

Unknown key, missing key, upstream down

Case	Proxy reply	Claude Code prints	pi prints
Revoked key	401 {"error":"Invalid API key"}	nothing for 100 s, then killed by the harness timeout (exit 124)	401 {"error":"Invalid API key"}, exit 1, immediate
Unknown key	same	nothing for about 3 minutes (181 s), then Failed to authenticate. API Error: 401 {"error":"Invalid API key"}, exit 1	same as revoked
Missing key	401 {"error":"Missing API key"}	Not logged in · Please run /login (checked client-side; the proxy logged no request from it), exit 1	No API key found for anthropic. Use /login …, exit 1
Upstream down	500 {"type":"error","error":{"type":"api_error","message":"Post \"…\": dial tcp 127.0.0.1:18400: connect: connection refused"}}	silent retry, exit 124 at 100 s	500 {…same body…}, exit 1, after about 16 s of its own retries

The 401 body is {"error":"Invalid API key"}, not Anthropic-shaped ({"type":"error","error":{…}}). Claude Code treats it as retryable and backs off, which is what makes a revoked key look like a hang. The 500 body is Anthropic-shaped.

What holds

• Both clients work through the proxy with only allowlisted variables, one needing a models.json.
• Streaming, tool calls and tool results round-trip intact. cache_control blocks and the anthropic-version/anthropic-beta headers pass through unchanged.
• The upstream sees the proxy's own key, never the client key.
• Per-agent keys: issue and revoke are single management calls, immediate, persistent, B unaffected by A's revocation.

What doesn't hold, or needs a decision

1. Plaintext keys. Client keys are stored in plaintext in config.yaml (out/config.after-issue.yaml), appear in full in the usage-queue events, and the v0 DELETE …?value=<key> writes the full key into the access log (out/proxy-logs/main.log line 354: the revoked key A in clear). The PUT/GET paths and the request logs mask or omit it.
2. Every management write rewrites the whole config. The file comes back reordered, 4-space indented, with runtime defaults added and comments moved; the management secret-key is replaced by a bcrypt hash at startup (out/config.as-written.yaml vs out/config.after-start.yaml). A human-edited config file and API-issued keys fight over one file.
3. No per-client-key accounting. The usage counters are per upstream credential. The per-request events carry the client key but live 60 s and are drained on read. Per-agent usage needs a collector that polls faster than every 60 s, or the request logs.
4. Revoked or wrong keys look like a hang in Claude Code (100 s to 3 min with no output). The bus should not rely on the client to report a revocation; it should check the key before starting the session or on a short timeout.
5. Empty key list: unknown. I kept a bootstrap key in access.api-keys throughout as a precaution and never tested an empty list. The shipped example config says nothing about what an empty list does (it may accept any caller). The design row should test it on a stand-in before relying on "no keys means closed".
6. Management key hashing at startup. The proxy hashes management.secret-key in place on first start. The generating tool must not rely on the plaintext file afterwards.
7. Pi override. See "Pi finding"; this is a code change in session setup.
8. Stand-in limits. The stand-in does not model the real Anthropic API's reaction to the forwarded headers, nor rate limits, token counting or model routing. Routing a seat to the real upstream is a separate step with real credentials, outside this row.

What the #237 design row must settle

• Where issued keys live. The proxy keeps them in its own config (plaintext, rewritten on each write). The stack needs its own record of key id → seat/session, written under the data root, and a decision on whether the proxy config is treated as derived state that the stack regenerates, or as authoritative.
• How the bus hands a key to a session. Today ENV_ALLOW carries one ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN. The key must arrive per session, runtime-only, not in argv or on disk beyond the session's HOME. Pi additionally needs the models.json file in that HOME (so the session HOME becomes part of the handoff).
• Rotation. Mechanically: PUT the new list, start sessions with the new key, DELETE the old one (3 calls, no restart). Open points: running sessions hold the old key until they end; the 60 s event window; whether a rotation window needs two keys per seat at once.
• Revocation feedback. Because Claude Code stalls on 401, decide who detects and reports a revoked key (bus-side preflight against the proxy, or session timeout).
• Accounting. Whether per-seat usage comes from a queue poller, the request logs, or the bus counting its own turns.
• Who may call the management API. The management key is outside this row's reach. The design row decides which component holds it and how; no agent reads it.

Rerun

cd agents/researcher/work/i1-proxy-test
./run-all.sh > out/run-all.console.txt 2>&1     # about 7 minutes
cat out/host-guard.txt out/analysis.txt
bash cleanup.sh                                  # idempotent; run-all.sh already calls it

run-all.sh creates state/, starts the namespace, runs inner.sh (all scenarios S1 to S7), analyses, calls cleanup.sh, and writes the host-guard verdict. state/ is the only runtime directory; cleanup.sh stops only processes whose cwd or command line is under this packet and then deletes state/. Ports need no cleanup: the namespace disappears with its last process. Requirements: unshare with unprivileged user namespaces, node, curl, ss, the proxy binary at ~/.local/opt/EasyCLIProxyAPI/cpa-core/cli-proxy-api, the repo's pinned pi and ~/.local/bin/claude (or MOSAIC_CLAUDE_BIN).

Files

File	Purpose
run-all.sh, inner.sh, cleanup.sh	harness, scenarios, cleanup
stand-in-upstream.mjs, config.template.yaml	the stand-in server and the made-up proxy config
analyze.mjs	pairs proxy logs with upstream records
docs/	cited sources: example config, management API doc, -h output
out/	evidence of the final run (all keys in it are made up)
candidate-manifest.sha256	sha256 of the packet files, for the gate

Not done

• Switching any live seat to a proxy; real credentials; the subscription-terms question; #237 design or build; containerized sessions (I1).
• An empty key list, key rotation under a running session, PATCH on the key list, and the v8 equivalent of the DELETE were not exercised.
• The run did not take the test slot: it runs no suite, mutant or Docker, only a private namespace on its own ports. The seat judged that without asking first and reports it in the hand-off to Sage.