Files
jason.woltje e69260a3ce row 244: gate runs mount a 0600 fixture pi auth file, test-release activates on the mock adapter, and real-model test-task cases sit behind MOSAIC_LIVE_PROVIDER=1 (#1757)
Candidate morgana/row244 94ab4c2fe on a36523713, manifest 6e7e3e3c (8 paths).
Webby APPROVE r1, rev 3008 (webby-0084, REVIEW-r1.md at webby/reviews
ebb3d83bf; request #1757 c30154). Gate on Sage's slot at 81e037d20
(sage-2147): the five gate-lane files whole 51/0, test-task.sh 91/0,
test-release.sh 14/0, mutants M-G, M-L, M-A, M-T and M-F killed.
94ab4c2fe adds only BUILD.md results. Webby's drift check: a36523713..d37210288
changes none of the 8 paths and no path either suite reads; merge-tree clean.
Rulings A and C (sage-1914, sage-2061). The dropped real-model coverage goes
to DEFERRED in the follow-up doc commit.
2026-10-21 16:44:37 -05:00

16 KiB

Row 244: gate runs use a fixture pi auth file

Brief: docs/plans/2026-10-21_gate-env-rows.md § "Gate runs use a fixture pi auth file" (Sage, sage-1909). Reviewer Webby. DEFERRED entry "Gate runs mount the host's pi auth file" (059577886). Design notes written before row 197 lands, read against origin/herb/row197-r6 (df0c59608). Recheck every line cite against 197's landing before code starts.

Where Compose is reached

compose.yaml:43 mounts ${PI_AUTH_FILE:-/home/jwoltje/.pi/agent/auth.json} read-only. The default is a literal path, not $HOME, so keeping HOME in the lane env doesn't change it.

Gate steps come from discoverSteps (gate-lane.mjs:32): one pkg-<name> step per package with tests (not queue) and one script-<name> step per scripts/test-*.sh. A grep of those for run-task.sh, docker compose, hello.sh, verify.sh, agent.sh and release.sh activate finds two Compose paths. The other hits (cohort.test.mjs "hello" shim requests, test-auth.sh) don't reach Compose.

Step Path to Compose Adapter
test-task.sh :150-385 scripts/run-task.sh run → mosaic-task.mjs:396 docker compose run with spawnEnv = {...process.env} mock (mock_config)
test-release.sh :69, :85, :93 release.sh activate → health_check (:41-55) → run-task.sh run tasks/hello-marker.json pi (config :61 sets no adapter; provider zai, model glm-5.3-flash)

mosaic-task.mjs passes process.env through, so a PI_AUTH_FILE exported by a suite reaches Compose.

The conflict: test-release's healthy activation is a live provider call

"healthy activation succeeds" (:85) and "repeat activation succeeds" (:93) need health_check to pass. That runs the real pi adapter, asks glm-5.3-flash for MOSAIC_HELLO_OK, and authenticates through the mounted auth file. The lane's cleanEnv drops ZAI_API_KEY, so the mount is the only credential. With a fixture auth file the provider call fails, and four checks fail: :85, the pointer check after it, :93 and "log is append-only". So every gate run today spends a real provider call on the host credential.

Options put to Sage (morgana-0037):

  • A (recommended). test-release's config sets "adapter":"mock", and the two healthy activations run with MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK. The fault injection still refuses, because its task expects MOSAIC_FAULT_<random>. The suite keeps proving the release state machine, the image and Compose path, and the pointer and log. It stops proving that the pi image can reach the provider. Real release.sh activate runs still do that, with the real credential. No credential and no spend in the gate.
  • B. Keep the live call. The lane passes the real auth file explicitly. That contradicts the brief and DEFERRED, so I don't take it without a ruling.

Ruling: A (sage-1914, 09:51Z). B keeps a live billed call on every gate, which is what this row stops. Sage's conditions:

  1. test-release.sh's suite config sets "adapter":"mock", with a comment beside it saying why: no credential and no provider call in the gate (row 244). The healthy activations run with MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK.
  2. Fault injection still refuses. It keeps the random MOSAIC_FAULT_<n> expectation, and the mock's fixed reply can't satisfy it. Mutant M-F forces the fault task's expected value to MOSAIC_HELLO_OK. It must fail the suite.
  3. The fake-docker test also asserts that test-release's activations ran with the mock adapter. The fake docker records MOSAIC_ADAPTER, which mosaic-task.mjs sets in spawnEnv from the resolved config. Then a suite drifting back to pi fails the test.
  4. Coverage dropped: test-release.sh no longer proves that the pi image reaches the provider. A real release.sh activate still proves it, with the real credential. Sage adds a DEFERRED line at landing if Webby wants one.

guardDocker's new refusal also gets a mutant (M-G, below).

Change

  1. scripts/test-task.sh and scripts/test-release.sh: right after SANDBOX is made, write $SANDBOX/pi/auth.json ({}, mode set by chmod 600 after the write, so it doesn't depend on the umask, row 243) and export PI_AUTH_FILE. It has to be a regular file that exists, because a missing bind source makes Docker create a directory on the host. The suites set it themselves, so the legacy gate (outside the lane) is covered too.
  2. scripts/test-release.sh: option A above.
  3. scripts/testing/gate-lane.mjs cleanEnv: set PI_AUTH_FILE to join(o.tempRoot,'pi-auth.json'), and run writes that file ({}, flag:'wx', then chmod 0600) next to where it writes bin/docker. Any later step that reaches Compose without its own fixture then gets the lane's fixture.
  4. scripts/testing/lane-docker.mjs guardDocker: compose run also refuses unless PI_AUTH_FILE is set, resolves inside run.tempRoot and isn't the host default (refusal text Compose auth mount not a lane fixture). The guard already sees every Docker call a lane step makes, so this is where "no gate step mounts the host file" is enforced at run time. The CLI entry passes process.env.
  5. compose.yaml is unchanged (brief: not in scope).
  6. At landing: the DEFERRED entry moves to Done with the landing commit.

Tests

In packages/cli/tests/gate-lane.test.mjs (row 197's file; the exact file follows 197's landing):

  • guardDocker refuses compose run with PI_AUTH_FILE unset, set to the host default, or set outside tempRoot; it accepts the lane fixture and a suite fixture under tempRoot. Other verbs are unchanged.
  • cleanEnv returns a PI_AUTH_FILE inside tempRoot that isn't the host default.
  • The suites, read through their Compose environment: run test-task.sh and test-release.sh with a fake docker first on PATH. It answers info and image inspect with 0, and for compose run it appends PI_AUTH_FILE and whether that file is a regular 0600 file to a record. Then it prints the mock response. The test ignores the suites' own exit code and asserts that the record has at least one compose run from each suite and that every entry names an existing fixture file, never /home/jwoltje/.pi/agent/auth.json. The test never reads the host file. If a full suite run with the fake docker turns out too slow or brittle, I'll say so in the review request, with timings, and fall back to the guard and cleanEnv tests plus a static check that each suite exports PI_AUTH_FILE before its first Compose call. I won't drop it silently.

Mutants (each must fail a test; run on Sage's slot):

  • M-F: the fault task's expectExact forced to MOSAIC_HELLO_OK in release.sh health_check. Must fail test-release.sh ("fault-injected activation refuses").
  • M-G: the guardDocker PI_AUTH_FILE check removed. Must fail the guard test.
  • M-A: test-release's config back to no adapter (pi). Must fail the fake-docker test (condition 3).
  • M-L: cleanEnv stops setting PI_AUTH_FILE. Must fail the cleanEnv test.

Gate: those tests, test-task.sh and test-release.sh whole, the 197 lane tests, and the four mutants, all on Sage's slot.

Files (expected)

scripts/test-task.sh, scripts/test-release.sh, scripts/testing/gate-lane.mjs, scripts/testing/lane-docker.mjs, packages/cli/tests/gate-lane.test.mjs (or a new gate-lane-auth.test.mjs), docs/plans/DEFERRED.md (at landing), scripts/testing/README.md if it lists the guard's rules.

Correction (morgana-0040, 18:50Z)

The table above says test-task.sh uses only the mock adapter. That was wrong. On a36523713 two sections run the pi adapter through $CONFIG (provider zai) and authenticate through the host mount: :443-501 (live hello, result.json written and correct, wrong expectExact, distinct run dirs, list, fork base, fork child, fork recall, ancestor untouched, child branch file) and :518-530 (user recall run, recalled the user name). That's 13 checks and about 6 billed provider calls per gate run. I missed them because I read the mock section and stopped there.

Ruling C (sage-2061, 18:50Z). D (keep them live) refused.

  1. The run-record checks (hello, result.json, wrong expectExact, distinct run dirs, list) move to the mock adapter. Every assertion is kept. Only the response source changes.
  2. Fork recall and user recall run only under MOSAIC_LIVE_PROVIDER=1. Otherwise the suite prints skip live provider cases (row 244: no provider call in the gate), which the lane's requiredSkips regex doesn't match, on purpose. A test asserts the line prints when the variable is unset and that the live checks run, and fail under a fixture, when it's 1 (M-T).
  3. The dropped coverage and the DEFERRED wording are below. Sage writes the entry at landing.

Build (on a36523713, sage-2056)

Rechecked against 197's landing: lane-docker.mjs:30 guardDocker, gate-lane.mjs:109 cleanEnv and :181 (bin/docker written), compose.yaml:43, release.sh:41-55 health_check, mosaic-task.mjs:321-322 spawnEnv and MOSAIC_ADAPTER, test-release.sh:61 config. All hold.

  • scripts/testing/lane-docker.mjs: guardDocker(argv,run,env=process.env). After the option check, a compose run calls laneAuthFile. It refuses (Compose auth mount not a lane fixture) unless PI_AUTH_FILE is absolute, isn't the host default, resolves (realpath) inside the realpath of run.tempRoot, and is a regular file. Symlinks out of the root and relative paths refuse. info and image inspect need no auth file. The CLI entry passes process.env.
  • scripts/testing/gate-lane.mjs: cleanEnv sets PI_AUTH_FILE=join(o.tempRoot,'pi-auth.json'). runGate writes it ({}, flag wx, then chmodSync 0600) right after bin/docker. MOSAIC_LIVE_PROVIDER isn't on cleanEnv's keep list, so the lane can't request live cases.
  • scripts/test-release.sh: exports a $SANDBOX/pi/auth.json fixture ({}, chmod 600 after the write, exits 1 if the write fails). The config adds "adapter":"mock" with the ruling A comment. The fault, healthy and repeat activations all run with MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK, so the fault activation refuses only because its MOSAIC_FAULT_<n> expectation is wrong. (Correction, before any run: the first build gave the fault activation no mock response. The mock then answered empty, which fails any expectation, so M-F would have survived. Fixed in the follow-up commit.)
  • scripts/test-task.sh: same fixture, exported only when MOSAIC_LIVE_PROVIDER isn't 1, so an operator's live run keeps their own PI_AUTH_FILE (or Compose's default). Ruling C (a) and (b) as above. The section header changes from "live: real runs" to "run records through the mock adapter", the check name "live hello task..." becomes "hello task...", and the docker-less branch of the new split prints skip run-record cases (docker unavailable).
  • packages/cli/tests/gate-lane-image.test.mjs (:21, :106, :119): the compose-run calls pass run.env, the composeFixture env, which already holds a 0600 PI_AUTH_FILE inside tempRoot (accepted by sage-2061).
  • scripts/testing/README.md: one sentence on the lane fixture and the refusal.
  • New packages/cli/tests/gate-lane-auth.test.mjs, five tests:
    1. guardDocker accepts the lane fixture and a suite fixture under tempRoot. It refuses unset, empty, the host default, a file outside, a symlink inside that points outside, a missing path, a directory and a relative path. The option check still comes first. info and image inspect pass with no env.
    2. cleanEnv sets the lane fixture path and drops MOSAIC_LIVE_PROVIDER and the caller's PI_AUTH_FILE.
    3. test-release.sh in a private copy of the checkout, with a fake docker first on PATH. The fake answers every verb with 0. For compose run it records PI_AUTH_FILE, stat -c '%F %a' of it (never for the host path) and MOSAIC_ADAPTER, and it prints MOSAIC_MOCK_RESPONSE when the adapter is mock. Asserts: at least 3 compose runs, every one on a regular 0600 file under the suite's TMPDIR, and every adapter mock (condition 3). The healthy and fault lines print OK.
    4. test-task.sh, same harness: every compose run on its fixture and the mock adapter, the skip line printed exactly twice, and the hello check OK.
    5. M-T: test-task.sh with MOSAIC_LIVE_PROVIDER=1 and an operator-style fixture. No skip line, FAIL fork base: teach succeeds and FAIL recalled user name, at least one pi run, and every run on the operator's file. The copy leaves out .git, node_modules (symlinked), v1, agents, docs, BUILD-LOG.md and scripts/foundation, about 14 MB. Copying is needed because test-release.sh rewrites RELEASE while it runs, which would race other cli tests that read it. The suites ignore their own exit codes here: the fake doesn't emit the mock adapter's env lines, so some seam checks fail by design.
  • Readers: gate-lane.mjs is imported by gate-lane.test, gate-lane-state.test, gate-lane-fixtures.test, gate-lane-image.test and fixtures/gate-lane-child.mjs (unchanged exports). lane-docker.mjs by gate-lane-image.test. No test hash-pins any changed file. vars.test.mjs names scripts/test-task.sh only as a string value.

Coverage dropped from the gate (for Webby, and for the DEFERRED entry):

  • test-release.sh no longer proves that the pi image reaches the provider (ruling A).
  • test-task.sh no longer proves session fork recall or onboarding user recall against a real model (ruling C).

DEFERRED wording for Sage (sage-2061 condition 3): "Real-model suite cases run outside the gate. Row 244 moved test-task.sh fork recall (M11) and user recall (M16) behind MOSAIC_LIVE_PROVIDER=1, and test-release.sh activations to the mock adapter. Before any re-pin whose tree changes packages/harness or the session fork code (scripts/mosaic-task.mjs session and sessionForkFrom handling, the pi adapter), an operator runs MOSAIC_LIVE_PROVIDER=1 bash scripts/test-task.sh with the real pi credential and a real release.sh activate, and records both results on the re-pin row."

Mutants (each must fail a named test; on Sage's slot):

  • M-F: release.sh health_check fault task expectExact forced to MOSAIC_HELLO_OK. Fails test-release.sh "fault-injected activation refuses", and test 3's OK match. This needs the fault activation's MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK (see the correction above).
  • M-G: the laneAuthFile(env,run) call removed from guardDocker. Fails test 1.
  • M-A: ,"adapter":"mock" removed from test-release's config. Fails test 3 (adapter set is pi).
  • M-L: the PI_AUTH_FILE key removed from cleanEnv. Fails test 2.
  • M-T: test 5 itself (sage-2061). As a mutant check too: the MOSAIC_LIVE_PROVIDER gate in test-task forced on fails test 4 (skip line count and pi adapter).

Results, Sage's slot (sage-2147), candidate 81e037d20, 2026-10-21 20:46:16Z to 20:46:59Z. Busy check CLEAR before and after. Image mosaic-poc-agent:0.85.1-r0.0.12 present, so nothing was built. No unexpected failure. Singles ran under umask 022 with TMPDIR under /mnt/storage/scratch/tmp.

Run rc Result Time
gate-lane-auth.test.mjs (new) 0 5 pass, 0 fail 14 s
gate-lane-image.test.mjs 0 11 pass, 0 fail 0 s
gate-lane.test.mjs 0 13 pass, 0 fail 0 s
gate-lane-state.test.mjs 0 14 pass, 0 fail 2 s
gate-lane-fixtures.test.mjs 0 8 pass, 0 fail 0 s
test-task.sh 0 OK 91, FAIL 0, the two live-skip lines 12 s
test-release.sh 0 OK 14, FAIL 0, Docker section on the mock adapter 2 s

Mutants, each run on its named gate-lane-auth test and restored after:

Mutant Target Result
M-G test 1 killed (0 pass, 1 fail)
M-L test 2 killed (0 pass, 1 fail)
M-A test 3 killed (0 pass, 1 fail)
M-T test 4 killed (0 pass, 1 fail), 6 s
M-F test 3 killed (0 pass, 1 fail)
M-F real test-release.sh killed: rc 1, OK 11, FAIL 3, the first fault-injected activation refuses (exit 0, expected 1)