Row 332 R2 (#1830): record R2 evidence in fethry BUILD.md

Append-only R2 section: correction to R1's evidence, the two causes of the 53 cli failures, items 1 to 6 with their tests, RED, hermetic and GREEN tables, five killed mutants on 39eadf8ba, and the F1, F4 and F2 docs follow-ups.
2026-11-02 15:24:56 -06:00
1 parent 39eadf8ba8
commit 6e9f3d7231
@@ -365,3 +365,279 @@ Native helper join1961/mutations/run.py records exact one-match source anchors a
 M1 narrows filter bug to passing only Claude OAuth; proxy-removal assertion catches it while nonproxy control still passes. M2 mutates only final models.json write to replace environment reference with own fake key (after original in-memory safety check), so actual on-disk literal-key defect is exercised; single-Codex-provider reference and privacy assertions kill it. M3 changes actual serialized harnessEnv to bundle.env; six managed native/explicit-route/Claude privacy assertions fail on credential value in session.json, while unsupported-provider refusal control still passes. These are genuine assertion REDs, not syntax/fixture/timeouts;0cancelled/0skips every phase. All6 production hashes match original pinned candidate after each restore. No assertion/deadline weakened.

 All3 named mutants now KILLED; prior pending notes remain historical. Receipts include each baseline/full mutant/full restored log and source-anchor/SHA details; cleanup.json verifies0remaining owned-root processes. Product-source git diff vsd6 is empty. Restored positive controls2/2,7/7,7/7. No further fullgate/global/package runs, shareddragon fallback,secret read,borrowed token,dependency install or realprovider/authturn. No push. Three kills plus117 targeted tests do NOT claim full acceptance; independent review/fullgate/installation/live behavior remain pending/UNVERIFIED.
+
+## R2, the pi credential preflight and the whole cli suite, 2026-11-02
+
+Row 332 R2, #1830. Written by Claude Opus 5.5 as the implementation child of the Mosaic Stack lead (T3 c5c1e8ba),
+on the lead's brief /mnt/storage/scratch/mosaic-lead/briefs/row332-r2-fix.md. Checkout
+/home/jwoltje/.t3/worktrees/mosaic-stack/t3-dd0db440, branch `t3/row332-r2-fix`, fast-forwarded from 09a847eda to
+eabeb1aa8 before any change. Earlier sections are unchanged. Times are UTC. Each file or suite ran alone with
+`timeout 1800`, absolute nice 19 and TMPDIR from `mktemp -d /mnt/storage/scratch/tmp/XXXXXX` (31 bytes). The nice
+column is what the run read at start; ananicy reniced one RED run to -4, recorded as seen.
+
+### Correction to R1's evidence
+
+R1's evidence ran touched files only, not the whole cli suite. The 117/0/0 table under "Mos1961 actual joined
+source/fixture result" is 11 whole files, and the named mutants above ran one file each. None of the 13 files that
+later failed was among them. The lead's landing gate ran pkg-cli whole and got 1044 pass, 53 fail
+(/mnt/storage/scratch/b2gate/out-r2/pkg-cli.log). R1 did list full package runs as pending; the review approved on
+the touched files.
+
+### Cause
+
+The brief named two causes. They explain all 53 failures, and I found no third.
+
+1. P3's pi preflight (packages/plugins/harness/pi/index.mjs:39-46) refuses `no-session-credential` unless the
+   launch has a session proxy key or one of ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY or ZAI_API_KEY.
+   Twelve cli files copy the shell's environment, delete only the SESSION_CREDENTIALS names
+   (packages/seat/src/session.mjs:253) or nothing at all, and launch pi. This host's shell exports
+   ANTHROPIC_AUTH_TOKEN and ANTHROPIC_BASE_URL and none of pi's four names (checked by name only), so every pi
+   launch refused. That is 52 failures. Most carry `no-session-credential` in the log. The rest, in
+   compactions-launch, session-clear, idle-stop-race and revoke-window, fail with "timed out waiting", "no RESULT"
+   or an empty state list, because the session they wait on never started. On a host that exports
+   OPENAI_API_KEY the same tests pass on a real key, so the suite's result depended on the host.
+2. packages/cli/tests/stack-copy.test.mjs:51 calls `bundle()` with no argument on every harness. P3 made pi's
+   bundle destructure its argument with no default, so it threw "Cannot destructure property 'dir' of
+   'undefined'". That is 1 failure.
+
+The proof is the hermetic run below. With only the test change for cause 1 and no product change, the 12 files ran
+87 tests with 0 failures. With the product change for cause 2, stack-copy passes 19/19.
+
+### Item 1, the refusal stays
+
+No change to the preflight. pi's own credential is still exactly those four names or a proxy key, the preflight
+still runs outside proxy mode, and no existing assertion was weakened. The one changed assertion is stronger
+(item 2).
+
+### Item 2, hermetic cli tests
+
+packages/cli/tests/helpers.mjs gains three exports.
+
+- `HOST_CREDENTIALS`, the SESSION_CREDENTIALS names plus OPENAI_API_KEY, OPENROUTER_API_KEY and ZAI_API_KEY. It is
+  built from the seat's list, so a new session login name lands in it too.
+- `FAKE_PI_KEY = "fx332-fake-pi"`.
+- `withoutHostCredentials(env, { piKey })`, which deletes every HOST_CREDENTIALS name and, for a pi launch, sets
+  ZAI_API_KEY to the fake key.
+
+How each file changed and how its env assertions keep their meaning:
+
+- session-launch, idle-stop-race, revoke-window, compactions-launch, session-clear, thread-launch, lifecycle and
+  thread-revoke. `for (const k of SESSION_CREDENTIALS.flat()) delete f.env[k];` became
+  `withoutHostCredentials(f.env, { piKey: true });`. All eight launch pi only and assert nothing about which
+  credential variables the session env holds. The only change is that the host no longer decides the outcome.
+- launch-limit-checks and launch-limits. The call is added after `fixture()`. These files deleted nothing before,
+  so they ran on the host's whole environment.
+- launch-routes. The call is added in setup before `Object.assign(f.env, more)`. The metered-route test (line 78)
+  still sets the session login names through `more`, which runs after the delete. The fake key is ZAI_API_KEY,
+  which is no session login, so the account checks that test asserts on see only what it set.
+- launch-account. setup calls `withoutHostCredentials(f.env, { piKey: harness === 'pi' })` before it sets `set`.
+  "REQ-LIMIT-2 a pi launch carries account null and skips both account checks" still checks that no session login,
+  and two at once, both reach mayLaunch with account null. Its comment now says pi runs on the fake key, which is
+  no session login. The account checks read only SESSION_CREDENTIALS names, so ZAI_API_KEY can't change them.
+- launcher.test.mjs. prepare() replaces its delete loop and its inline `ZAI_API_KEY = "fx332-pi"` with the helper.
+  "a pi launch with no own credential refuses no-session-credential" asserted that the SESSION_CREDENTIALS names
+  were absent. It meant "this env holds no credential pi could use", and the old list missed three of pi's names.
+  It now asserts every HOST_CREDENTIALS name is absent, which holds on any host.
+
+Host-dependence check, 21:06:35-21:06:36, nice 19, with a made-up `OPENAI_API_KEY=fx332-host-openai` in the shell
+and `--test-name-pattern="a pi launch with no own credential"`. The eabeb1aa8 copy of launcher.test.mjs, run as a
+temporary file and removed after, fails 0/1 with "Missing expected rejection". The changed file passes 1/0.
+
+Other package suites. The bus launcher tests (model-historical-ipc, model-historical-cleanup, model-restart,
+model-review-fixtures, model-hire-ownership-fixture) build the launch env from an empty object, and the plugins
+tests pass an explicit env, so neither reads host credentials. The other copies of process.env in package tests
+feed git or CLI subprocesses, not launches. Nothing outside cli needed the change.
+
+### Item 3, pi bundle default
+
+pi/index.mjs:70 is now `bundle({ dir, models, proxy = null } = {})`. With no argument it returns
+`{ files: {}, digests: [], sources: { piExtension, ...UNSLOP } }`, the value 09a847eda's `bundle = () => (...)`
+returned. The proxy path is unchanged. Test: packages/plugins/tests/proxy-harness.test.mjs "row 332 R2 pi bundle()
+with no argument returns the non-proxy bundle it returned before P3". stack-copy.test.mjs covers it as well.
+
+### Item 4, review F2
+
+pi.bundle now skips a launch.models entry when proxyRouteFor throws `proxy-provider-unsupported`. Any other route
+error, `proxy-route-missing` or `proxy-route-unknown`, still throws, so a malformed proxy entry still fails closed.
+sessionModel routes the session's own selection before the bundle runs, so the selected entry can't be skipped
+without a refusal.
+
+I chose to skip unroutable entries rather than write only the selected primary and fallback, because the
+amendment's tests need more than the selection. "P3 models file is private..." and "P3 every launch
+regenerates..." in packages/harness/tests/proxy-bundle.test.mjs select `a` from declared `{a, z}` and expect both
+the anthropic and mosaic-proxy-glm providers in models.json. Writing only the selection breaks them.
+
+Tests:
+
+- packages/harness/tests/proxy-bundle.test.mjs "row 332 R2 an unroutable model table entry is left out of
+  models.json and the launch goes on". The review's table (opus on anthropic, glm on zai, router on openrouter with
+  model x), selecting opus. Expects providers anthropic and mosaic-proxy-glm, no model x and PI_MODEL
+  claude-fixture. An added `{provider: 'proxy'}` entry with no route still throws `proxy-route-missing`.
+- packages/cli/tests/proxy-context-join.test.mjs "row 332 R2 joined pi launch goes live when launch.models also
+  holds an unroutable entry, which models.json leaves out". The review's failure scenario on the joined launcher
+  with P2 admission. The session goes live with one successful preflight and one run directory, models.json has
+  claude-opus-5-5 and no x, and the private-artifact check finds no key.
+
+### Item 5, review F3
+
+The launcher branches at packages/cli/src/launcher.mjs:435-439 are unchanged. packages/cli/tests/launch-account.test.mjs
+setup() gains `proxy` (the launcher's static proxy context) and `edit` options, and the new tests add a model
+`proxy-sonnet` with provider proxy and route anthropic. One test per branch:
+
+- "row 332 R2 a proxy session refuses account-credential-missing when launch.account names a host login, not proxy".
+- "row 332 R2 launch.account credential proxy with no proxy session refuses account-credential-missing".
+- "row 332 R2 account-is-credential: launch.account.name is the session proxy key; another name passes the account
+  checks". A positive control with the name jason-max reaches mayLaunch with account jason-max.
+
+Each checks the refusal code and why, that the launch log holds no value or key, and that mayLaunch was not called.
+
+The BundleError at packages/harness/src/bundle.mjs:332 now has the code `invalid-proxy-context`. Test:
+proxy-bundle.test.mjs "row 332 R2 a session proxy context without a key or loopback URL refuses
+invalid-proxy-context", with an empty key, a localhost URL and a missing key. The code has no `proxy-` prefix on
+purpose. The launcher catch at launcher.mjs:686 turns only `proxy-*` codes into a Refusal with that code and
+reports everything else as launch-failed, so this error still ends as launch-failed, as it did under the old
+default code `bundle-refused`. No code or test at eabeb1aa8 named this error by message or code. The comment at
+bundle.mjs:50 still describes the default for other errors and is unchanged.
+
+### Item 6, open follow-ups, unchanged in R2
+
+- F1, medium. packages/cli/src/launcher.mjs:401-402, together with 524-545. In proxy mode the first check()
+  returns early, so the capacity, limit, sandbox and mayLaunch refusals run only after ensureCheckouts and P2
+  modelPreflight. It fails closed but wastes a key issuance, git work and an authorization record.
+- F4, info. packages/cli/src/launcher.mjs:224 and 403-411. The createLauncher static `proxy` option bypasses P2
+  admission. Production passes none. The new launch-account tests use this option for their proxy context, as
+  launcher.test.mjs:152 does, so a later rename or removal touches them too.
+- F2 docs. packages/plugins/README.md describes pi's models.json and could say that unroutable entries are left
+  out. The brief asked me to avoid that file, so it is unedited.
+
+### RED on eabeb1aa8, the 13 failing cli files, before any change
+
+| File | Tests | Pass | Fail | Time | Nice |
+| --- | --- | --- | --- | --- | --- |
+| lifecycle | 24 | 13 | 11 | 20:50:30-20:50:38 | 19 |
+| thread-launch | 15 | 3 | 12 | 20:50:38-20:53:10 | -4 |
+| session-launch | 8 | 0 | 8 | 20:53:10-20:54:12 | 19 |
+| compactions-launch | 5 | 0 | 5 | 20:54:12-20:55:56 | 19 |
+| session-clear | 5 | 0 | 5 | 20:55:56-20:57:47 | 19 |
+| launch-limit-checks | 5 | 2 | 3 | 20:57:47-20:57:47 | 19 |
+| launch-routes | 9 | 6 | 3 | 20:57:47-20:57:47 | 19 |
+| idle-stop-race | 1 | 0 | 1 | 20:57:47-20:58:07 | 19 |
+| launch-account | 7 | 6 | 1 | 20:58:07-20:58:08 | 19 |
+| launch-limits | 1 | 0 | 1 | 20:58:08-20:58:08 | 19 |
+| revoke-window | 1 | 0 | 1 | 20:58:08-20:58:28 | 19 |
+| stack-copy | 19 | 18 | 1 | 20:58:28-20:58:46 | 19 |
+| thread-revoke | 3 | 2 | 1 | 20:58:46-20:59:07 | 19 |
+
+53 failures, the gate's count, file for file. 0 skipped, 0 cancelled.
+
+### Hermetic test change only, no product change
+
+| File | Tests | Pass | Fail | Time |
+| --- | --- | --- | --- | --- |
+| lifecycle | 24 | 24 | 0 | 21:03:23-21:03:36 |
+| thread-launch | 15 | 15 | 0 | 21:03:36-21:03:55 |
+| session-launch | 8 | 8 | 0 | 21:03:55-21:04:09 |
+| compactions-launch | 5 | 5 | 0 | 21:04:09-21:04:35 |
+| session-clear | 5 | 5 | 0 | 21:04:35-21:04:47 |
+| launch-limit-checks | 5 | 5 | 0 | 21:04:47-21:04:47 |
+| launch-routes | 9 | 9 | 0 | 21:04:47-21:04:47 |
+| idle-stop-race | 1 | 1 | 0 | 21:04:47-21:04:49 |
+| launch-account | 10 | 10 | 0 | 21:04:49-21:04:49 |
+| launch-limits | 1 | 1 | 0 | 21:04:49-21:04:49 |
+| revoke-window | 1 | 1 | 0 | 21:04:49-21:04:50 |
+| thread-revoke | 3 | 3 | 0 | 21:04:50-21:04:51 |
+
+87 tests, 0 fail, all nice 19. launch-account counts the three F3 tests, which need no product change.
+
+### New tests fail without their product fix
+
+Run with the new tests in place and pi/index.mjs and bundle.mjs still at eabeb1aa8, then again after the fix.
+
+| File | Without fix p/f | Time | With fix p/f | Time | What failed |
+| --- | --- | --- | --- | --- | --- |
+| harness proxy-bundle | 7/2 | 21:05:03-21:05:04 | 9/0 | 21:05:55 | F2 test throws proxy-provider-unsupported; invalid-context test gets bundle-refused |
+| plugins proxy-harness | 5/1 | 21:05:04 | 6/0 | 21:05:55 | TypeError, cannot destructure 'dir' |
+| cli proxy-context-join | 7/1 | 21:05:04-21:05:11 | 8/0 | 21:05:55-21:06:04 | launch refused proxy-provider-unsupported after admission |
+| cli stack-copy | 18/1 | 21:05:11-21:05:28 | 19/0 | 21:06:04-21:06:22 | TypeError, cannot destructure 'dir' |
+
+The F3 launcher tests and the changed launcher.test.mjs assertion fail without their fix as shown by the mutants
+and the host-dependence check.
+
+### GREEN, every package suite but queue, whole
+
+`node --test tests/*.test.mjs` from each package, one at a time, on the final bytes.
+
+| Package | Tests | Pass | Fail | Skip | Time | Seconds |
+| --- | --- | --- | --- | --- | --- | --- |
+| business | 289 | 289 | 0 | 0 | 21:07:03-21:07:06 | 3 |
+| control-board | 130 | 130 | 0 | 0 | 21:07:06-21:07:06 | 0 |
+| conversation | 187 | 187 | 0 | 0 | 21:07:06-21:07:42 | 36 |
+| discord | 200 | 200 | 0 | 0 | 21:07:42-21:07:45 | 3 |
+| harness | 205 | 205 | 0 | 0 | 21:07:45-21:07:57 | 12 |
+| ledger | 89 | 89 | 0 | 0 | 21:07:57-21:08:11 | 14 |
+| library | 168 | 168 | 0 | 0 | 21:08:11-21:08:17 | 6 |
+| model-proxy | 100 | 100 | 0 | 0 | 21:08:17-21:08:19 | 2 |
+| mosaic | 69 | 69 | 0 | 0 | 21:08:19-21:08:21 | 2 |
+| plugins | 126 | 126 | 0 | 0 | 21:08:21-21:08:21 | 0 |
+| processing-control | 62 | 62 | 0 | 0 | 21:08:21-21:08:22 | 1 |
+| project | 41 | 41 | 0 | 0 | 21:08:22-21:08:24 | 2 |
+| runs | 43 | 43 | 0 | 0 | 21:08:24-21:08:24 | 0 |
+| seat | 223 | 223 | 0 | 0 | 21:08:25-21:08:31 | 6 |
+| supervisor | 45 | 45 | 0 | 0 | 21:08:31-21:08:31 | 0 |
+| tasks | 100 | 100 | 0 | 0 | 21:08:31-21:08:33 | 2 |
+| webui | 490 | 490 | 0 | 0 | 21:08:33-21:10:13 | 100 |
+| bus | 1349 | 1349 | 0 | 0 | 21:10:13-21:11:01 | 48 |
+| cli | 1057 | 1056 | 0 | 1 | 21:11:01-21:21:37 | 636 |
+| harness, rerun | 205 | 205 | 0 | 0 | 21:21:55-21:22:08 | 13 |
+
+19 suites, 4973 tests, 4972 pass, 0 fail, 1 skipped, 0 cancelled, every run nice 19. The skip is "FED R7
+scratch two-instance acceptance", which skips by design (it runs from scripts/test-federation-acceptance.sh) and
+was skipped at the lead's gate too. The bus flake at model-final-completion.test.mjs:69 did not occur, so there was
+nothing to rerun.
+
+The first 19 runs used the working tree before commit. After the harness run I removed one blank line at the end
+of proxy-bundle.test.mjs that `git diff --check` flagged, then reran harness whole (last row). No other byte
+changed before the product commit 39eadf8ba, and `git diff --stat HEAD` was empty after it.
+
+The gate's batch had 1098 cli tests, because it carried other rows. This branch is 09a847eda plus row 332 plus R2,
+with 1057 cli tests, four of them new in R2.
+
+### Mutants
+
+Mutants ran on the product commit 39eadf8ba, one at a time. A scratch script made each one an exact one-match
+replace and printed the file's sha256, first 16 hex, before and after. Each killing test ran once, then
+`git checkout -- <file>` and `git status --short`. The commit came first so that the checkout restores committed
+bytes.
+
+| Mutant | Change | Killing test | p/f/s; exit | sha16 committed, mutant, restored | Time |
+| --- | --- | --- | --- | --- | --- |
+| pi-preflight-open | pi/index.mjs, `return null;` as the first line of preflight | plugins tests/proxy-harness.test.mjs | 4/2/0; 1 | b37d81c5b552db21, c35c3cdbbcc10ccf, b37d81c5b552db21 | 21:22:22 |
+| | | cli tests/launcher.test.mjs, name pattern "a pi launch with no own credential" | 0/1/0; 1 | | 21:22:22-21:22:23 |
+| bundle-all-entries | pi/index.mjs, the R2 skip replaced by base's `const route = proxyRouteFor(entry);` | harness tests/proxy-bundle.test.mjs | 8/1/0; 1 | b37d81c5b552db21, e901d619074e7729, b37d81c5b552db21 | 21:22:28 |
+| | | cli tests/proxy-context-join.test.mjs | 7/1/0; 1 | | 21:22:28-21:22:36 |
+| proxy-account-unchecked | launcher.mjs:438-439, the proxy credential check and its throw, deleted | cli tests/launch-account.test.mjs | 8/2/0; 1 | 1001674131006a7e, 3d52f49ff1a1b8c8, 1001674131006a7e | 21:22:44-21:22:45 |
+| session-proxy-key-unchecked, extra | launcher.mjs:435, the "session proxy key" hold, deleted | cli tests/launch-account.test.mjs | 9/1/0; 1 | 1001674131006a7e, 8cfd4d43e3f189a0, 1001674131006a7e | 21:22:49-21:22:50 |
+| M1, re-run | session.mjs:262 `!(proxy && isProviderVariable(k))` to `!(proxy && isProviderVariable(k) && k !== "CLAUDE_CODE_OAUTH_TOKEN")` | seat tests/proxy-session-env.test.mjs | 1/1/0; 1 | 488e593811adf369, b37acf6ba05ee6e6, 488e593811adf369 | 21:22:52 |
+
+5 of 5 killed, each by an assertion. There were no syntax errors, timeouts, skips or cancellations. After every
+restore, `git status --short` printed nothing and the restored hash matched the committed one.
+
+- pi-preflight-open. "P3 pi refuses no-session-credential with no provider key" and "...with only a Claude OAuth
+  token" fail, and the launcher test fails "Missing expected rejection".
+- bundle-all-entries. Both F2 tests fail with "provider is unsupported by the supplied model proxy context" from
+  proxyRouteFor at bundle.mjs:86.
+- proxy-account-unchecked. The brief names 437-439. Line 437 is the outer `if` whose `else` holds the host-login
+  checks, so deleting it as well breaks the syntax. I deleted 438-439, which removes the refusal and leaves the
+  outer `if` with an empty body. Both branch tests get `fixture-policy` where they expect
+  `account-credential-missing`.
+- session-proxy-key-unchecked. Not required by the brief. It covers the third F3 branch. The account-is-credential
+  test gets `fixture-policy` where it expects `account-is-credential`.
+- M1. "P3 proxy sessionEnv removes every instance provider credential without widening ENV_ALLOW" fails on
+  CLAUDE_CODE_OAUTH_TOKEN, as in R1.
+
+### Cleanup
+
+No test process of mine is left running, checked through the /proc cwd links of my own processes. After this
+commit the node_modules links (`node_modules` and `packages/webui/node_modules`) and the TMPDIR come out, as the
+brief asks. The logs behind these tables lived in that TMPDIR and are not kept. The counts and times here are
+copied from them.