Candidate morgana/row24494ab4c2feona36523713, manifest 6e7e3e3c (8 paths). Webby APPROVE r1, rev 3008 (webby-0084, REVIEW-r1.md at webby/reviews ebb3d83bf; request #1757 c30154). Gate on Sage's slot at81e037d20(sage-2147): the five gate-lane files whole 51/0, test-task.sh 91/0, test-release.sh 14/0, mutants M-G, M-L, M-A, M-T and M-F killed.94ab4c2feadds only BUILD.md results. Webby's drift check: a36523713..d37210288 changes none of the 8 paths and no path either suite reads; merge-tree clean. Rulings A and C (sage-1914, sage-2061). The dropped real-model coverage goes to DEFERRED in the follow-up doc commit.
231 lines
16 KiB
Markdown
231 lines
16 KiB
Markdown
# Row 244: gate runs use a fixture pi auth file
|
|
|
|
Brief: `docs/plans/2026-10-21_gate-env-rows.md` § "Gate runs use a fixture pi auth file" (Sage, sage-1909).
|
|
Reviewer Webby. DEFERRED entry "Gate runs mount the host's pi auth file" (059577886). Design notes
|
|
written before row 197 lands, read against `origin/herb/row197-r6` (df0c59608). Recheck every line cite
|
|
against 197's landing before code starts.
|
|
|
|
## Where Compose is reached
|
|
|
|
`compose.yaml:43` mounts `${PI_AUTH_FILE:-/home/jwoltje/.pi/agent/auth.json}` read-only. The default is
|
|
a literal path, not `$HOME`, so keeping `HOME` in the lane env doesn't change it.
|
|
|
|
Gate steps come from `discoverSteps` (`gate-lane.mjs:32`): one `pkg-<name>` step per package with tests
|
|
(not queue) and one `script-<name>` step per `scripts/test-*.sh`. A grep of those for `run-task.sh`,
|
|
`docker compose`, `hello.sh`, `verify.sh`, `agent.sh` and `release.sh activate` finds two Compose
|
|
paths. The other hits (`cohort.test.mjs` "hello" shim requests, `test-auth.sh`) don't reach Compose.
|
|
|
|
| Step | Path to Compose | Adapter |
|
|
|---|---|---|
|
|
| `test-task.sh` :150-385 | `scripts/run-task.sh run` → `mosaic-task.mjs:396` `docker compose run` with `spawnEnv = {...process.env}` | mock (`mock_config`) |
|
|
| `test-release.sh` :69, :85, :93 | `release.sh activate` → `health_check` (:41-55) → `run-task.sh run tasks/hello-marker.json` | **pi** (config :61 sets no adapter; provider zai, model glm-5.3-flash) |
|
|
|
|
`mosaic-task.mjs` passes `process.env` through, so a `PI_AUTH_FILE` exported by a suite reaches Compose.
|
|
|
|
## The conflict: test-release's healthy activation is a live provider call
|
|
|
|
"healthy activation succeeds" (:85) and "repeat activation succeeds" (:93) need `health_check` to
|
|
pass. That runs the real pi adapter, asks glm-5.3-flash for `MOSAIC_HELLO_OK`, and authenticates through
|
|
the mounted auth file. The lane's `cleanEnv` drops `ZAI_API_KEY`, so the mount is the only credential.
|
|
With a fixture auth file the provider call fails, and four checks fail: :85, the pointer check after it,
|
|
:93 and "log is append-only". So every gate run today spends a real provider call on the host credential.
|
|
|
|
Options put to Sage (morgana-0037):
|
|
- **A (recommended).** test-release's config sets `"adapter":"mock"`, and the two healthy activations
|
|
run with `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`. The fault injection still refuses, because its task
|
|
expects `MOSAIC_FAULT_<random>`. The suite keeps proving the release state machine, the image and
|
|
Compose path, and the pointer and log. It stops proving that the pi image can reach the provider.
|
|
Real `release.sh activate` runs still do that, with the real credential. No credential and no spend
|
|
in the gate.
|
|
- **B.** Keep the live call. The lane passes the real auth file explicitly. That contradicts the brief
|
|
and DEFERRED, so I don't take it without a ruling.
|
|
|
|
**Ruling: A (sage-1914, 09:51Z).** B keeps a live billed call on every gate, which is what this row
|
|
stops. Sage's conditions:
|
|
1. test-release.sh's suite config sets `"adapter":"mock"`, with a comment beside it saying why: no
|
|
credential and no provider call in the gate (row 244). The healthy activations run with
|
|
`MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`.
|
|
2. Fault injection still refuses. It keeps the random `MOSAIC_FAULT_<n>` expectation, and the mock's
|
|
fixed reply can't satisfy it. Mutant M-F forces the fault task's expected value to `MOSAIC_HELLO_OK`.
|
|
It must fail the suite.
|
|
3. The fake-docker test also asserts that test-release's activations ran with the mock adapter. The
|
|
fake docker records `MOSAIC_ADAPTER`, which `mosaic-task.mjs` sets in `spawnEnv` from the resolved
|
|
config. Then a suite drifting back to pi fails the test.
|
|
4. Coverage dropped: test-release.sh no longer proves that the pi image reaches the provider. A real
|
|
`release.sh activate` still proves it, with the real credential. Sage adds a DEFERRED line at landing
|
|
if Webby wants one.
|
|
|
|
guardDocker's new refusal also gets a mutant (M-G, below).
|
|
|
|
## Change
|
|
|
|
1. `scripts/test-task.sh` and `scripts/test-release.sh`: right after `SANDBOX` is made, write
|
|
`$SANDBOX/pi/auth.json` (`{}`, mode set by `chmod 600` after the write, so it doesn't depend on the
|
|
umask, row 243) and `export PI_AUTH_FILE`. It has to be a regular file that exists, because a missing
|
|
bind source makes Docker create a directory on the host. The suites set it themselves, so the legacy
|
|
gate (outside the lane) is covered too.
|
|
2. `scripts/test-release.sh`: option A above.
|
|
3. `scripts/testing/gate-lane.mjs` `cleanEnv`: set `PI_AUTH_FILE` to `join(o.tempRoot,'pi-auth.json')`,
|
|
and `run` writes that file (`{}`, `flag:'wx'`, then chmod 0600) next to where it writes `bin/docker`.
|
|
Any later step that reaches Compose without its own fixture then gets the lane's fixture.
|
|
4. `scripts/testing/lane-docker.mjs` `guardDocker`: `compose run` also refuses unless `PI_AUTH_FILE` is
|
|
set, resolves inside `run.tempRoot` and isn't the host default (refusal text
|
|
`Compose auth mount not a lane fixture`). The guard already sees every Docker call a lane step makes,
|
|
so this is where "no gate step mounts the host file" is enforced at run time. The CLI entry passes
|
|
`process.env`.
|
|
5. `compose.yaml` is unchanged (brief: not in scope).
|
|
6. At landing: the DEFERRED entry moves to Done with the landing commit.
|
|
|
|
## Tests
|
|
|
|
In `packages/cli/tests/gate-lane.test.mjs` (row 197's file; the exact file follows 197's landing):
|
|
|
|
- `guardDocker` refuses `compose run` with `PI_AUTH_FILE` unset, set to the host default, or set outside
|
|
`tempRoot`; it accepts the lane fixture and a suite fixture under `tempRoot`. Other verbs are unchanged.
|
|
- `cleanEnv` returns a `PI_AUTH_FILE` inside `tempRoot` that isn't the host default.
|
|
- The suites, read through their Compose environment: run `test-task.sh` and `test-release.sh` with a
|
|
fake `docker` first on `PATH`. It answers `info` and `image inspect` with 0, and for `compose run` it
|
|
appends `PI_AUTH_FILE` and whether that file is a regular 0600 file to a record. Then it prints the
|
|
mock response. The test ignores the suites' own exit code and asserts that the record has at least one
|
|
`compose run` from each suite and that every entry names an existing fixture file, never
|
|
`/home/jwoltje/.pi/agent/auth.json`. The test never reads the host file. If a full suite run with the
|
|
fake docker turns out too slow or brittle, I'll say so in the review request, with timings, and fall
|
|
back to the guard and `cleanEnv` tests plus a static check that each suite exports `PI_AUTH_FILE`
|
|
before its first Compose call. I won't drop it silently.
|
|
|
|
Mutants (each must fail a test; run on Sage's slot):
|
|
- M-F: the fault task's `expectExact` forced to `MOSAIC_HELLO_OK` in `release.sh` `health_check`. Must
|
|
fail `test-release.sh` ("fault-injected activation refuses").
|
|
- M-G: the `guardDocker` `PI_AUTH_FILE` check removed. Must fail the guard test.
|
|
- M-A: test-release's config back to no adapter (pi). Must fail the fake-docker test (condition 3).
|
|
- M-L: `cleanEnv` stops setting `PI_AUTH_FILE`. Must fail the `cleanEnv` test.
|
|
|
|
Gate: those tests, `test-task.sh` and `test-release.sh` whole, the 197 lane tests, and the four
|
|
mutants, all on Sage's slot.
|
|
|
|
## Files (expected)
|
|
|
|
`scripts/test-task.sh`, `scripts/test-release.sh`, `scripts/testing/gate-lane.mjs`,
|
|
`scripts/testing/lane-docker.mjs`, `packages/cli/tests/gate-lane.test.mjs` (or a new
|
|
`gate-lane-auth.test.mjs`), `docs/plans/DEFERRED.md` (at landing), `scripts/testing/README.md` if it
|
|
lists the guard's rules.
|
|
|
|
## Correction (morgana-0040, 18:50Z)
|
|
|
|
The table above says `test-task.sh` uses only the mock adapter. That was wrong. On a36523713 two
|
|
sections run the pi adapter through `$CONFIG` (provider zai) and authenticate through the host mount:
|
|
`:443-501` (live hello, result.json written and correct, wrong expectExact, distinct run dirs, list,
|
|
fork base, fork child, fork recall, ancestor untouched, child branch file) and `:518-530` (user recall
|
|
run, recalled the user name). That's 13 checks and about 6 billed provider calls per gate run. I missed
|
|
them because I read the mock section and stopped there.
|
|
|
|
**Ruling C (sage-2061, 18:50Z).** D (keep them live) refused.
|
|
1. The run-record checks (hello, result.json, wrong expectExact, distinct run dirs, list) move to the
|
|
mock adapter. Every assertion is kept. Only the response source changes.
|
|
2. Fork recall and user recall run only under `MOSAIC_LIVE_PROVIDER=1`. Otherwise the suite prints
|
|
`skip live provider cases (row 244: no provider call in the gate)`, which the lane's requiredSkips
|
|
regex doesn't match, on purpose. A test asserts the line prints when the variable is unset and that
|
|
the live checks run, and fail under a fixture, when it's 1 (M-T).
|
|
3. The dropped coverage and the DEFERRED wording are below. Sage writes the entry at landing.
|
|
|
|
## Build (on a36523713, sage-2056)
|
|
|
|
Rechecked against 197's landing: `lane-docker.mjs:30` guardDocker, `gate-lane.mjs:109` cleanEnv and
|
|
`:181` (bin/docker written), `compose.yaml:43`, `release.sh:41-55` health_check, `mosaic-task.mjs:321-322`
|
|
spawnEnv and MOSAIC_ADAPTER, `test-release.sh:61` config. All hold.
|
|
|
|
- `scripts/testing/lane-docker.mjs`: `guardDocker(argv,run,env=process.env)`. After the option check, a
|
|
`compose run` calls `laneAuthFile`. It refuses (`Compose auth mount not a lane fixture`) unless
|
|
`PI_AUTH_FILE` is absolute, isn't the host default, resolves (realpath) inside the realpath of
|
|
`run.tempRoot`, and is a regular file. Symlinks out of the root and relative paths refuse. `info` and
|
|
`image inspect` need no auth file. The CLI entry passes `process.env`.
|
|
- `scripts/testing/gate-lane.mjs`: `cleanEnv` sets `PI_AUTH_FILE=join(o.tempRoot,'pi-auth.json')`.
|
|
`runGate` writes it (`{}`, flag `wx`, then `chmodSync` 0600) right after `bin/docker`.
|
|
`MOSAIC_LIVE_PROVIDER` isn't on cleanEnv's keep list, so the lane can't request live cases.
|
|
- `scripts/test-release.sh`: exports a `$SANDBOX/pi/auth.json` fixture (`{}`, `chmod 600` after the
|
|
write, exits 1 if the write fails). The config adds `"adapter":"mock"` with the ruling A comment. The
|
|
fault, healthy and repeat activations all run with `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`, so the
|
|
fault activation refuses only because its `MOSAIC_FAULT_<n>` expectation is wrong. (Correction,
|
|
before any run: the first build gave the fault activation no mock response. The mock then answered
|
|
empty, which fails any expectation, so M-F would have survived. Fixed in the follow-up commit.)
|
|
- `scripts/test-task.sh`: same fixture, exported only when `MOSAIC_LIVE_PROVIDER` isn't 1, so an
|
|
operator's live run keeps their own `PI_AUTH_FILE` (or Compose's default). Ruling C (a) and (b) as
|
|
above. The section header changes from "live: real runs" to "run records through the mock adapter",
|
|
the check name "live hello task..." becomes "hello task...", and the docker-less branch of the new
|
|
split prints `skip run-record cases (docker unavailable)`.
|
|
- `packages/cli/tests/gate-lane-image.test.mjs` (:21, :106, :119): the compose-run calls pass
|
|
`run.env`, the composeFixture env, which already holds a 0600 `PI_AUTH_FILE` inside tempRoot (accepted
|
|
by sage-2061).
|
|
- `scripts/testing/README.md`: one sentence on the lane fixture and the refusal.
|
|
- New `packages/cli/tests/gate-lane-auth.test.mjs`, five tests:
|
|
1. guardDocker accepts the lane fixture and a suite fixture under tempRoot. It refuses unset, empty,
|
|
the host default, a file outside, a symlink inside that points outside, a missing path, a directory
|
|
and a relative path. The option check still comes first. `info` and `image inspect` pass with no env.
|
|
2. cleanEnv sets the lane fixture path and drops `MOSAIC_LIVE_PROVIDER` and the caller's
|
|
`PI_AUTH_FILE`.
|
|
3. test-release.sh in a private copy of the checkout, with a fake `docker` first on PATH. The fake
|
|
answers every verb with 0. For `compose run` it records `PI_AUTH_FILE`, `stat -c '%F %a'` of it
|
|
(never for the host path) and `MOSAIC_ADAPTER`, and it prints `MOSAIC_MOCK_RESPONSE` when the
|
|
adapter is mock. Asserts: at least 3 compose runs, every one on a regular 0600 file under the
|
|
suite's TMPDIR, and every adapter mock (condition 3). The healthy and fault lines print OK.
|
|
4. test-task.sh, same harness: every compose run on its fixture and the mock adapter, the skip line
|
|
printed exactly twice, and the hello check OK.
|
|
5. M-T: test-task.sh with `MOSAIC_LIVE_PROVIDER=1` and an operator-style fixture. No skip line,
|
|
`FAIL fork base: teach succeeds` and `FAIL recalled user name`, at least one pi run, and every run
|
|
on the operator's file.
|
|
The copy leaves out `.git`, `node_modules` (symlinked), `v1`, `agents`, `docs`, `BUILD-LOG.md` and
|
|
`scripts/foundation`, about 14 MB. Copying is needed because `test-release.sh` rewrites `RELEASE`
|
|
while it runs, which would race other cli tests that read it. The suites ignore their own exit codes
|
|
here: the fake doesn't emit the mock adapter's env lines, so some seam checks fail by design.
|
|
- Readers: `gate-lane.mjs` is imported by gate-lane.test, gate-lane-state.test,
|
|
gate-lane-fixtures.test, gate-lane-image.test and fixtures/gate-lane-child.mjs (unchanged exports).
|
|
`lane-docker.mjs` by gate-lane-image.test. No test hash-pins any changed file. vars.test.mjs names
|
|
`scripts/test-task.sh` only as a string value.
|
|
|
|
Coverage dropped from the gate (for Webby, and for the DEFERRED entry):
|
|
- `test-release.sh` no longer proves that the pi image reaches the provider (ruling A).
|
|
- `test-task.sh` no longer proves session fork recall or onboarding user recall against a real model
|
|
(ruling C).
|
|
|
|
DEFERRED wording for Sage (sage-2061 condition 3): "**Real-model suite cases run outside the gate.**
|
|
Row 244 moved `test-task.sh` fork recall (M11) and user recall (M16) behind `MOSAIC_LIVE_PROVIDER=1`,
|
|
and `test-release.sh` activations to the mock adapter. Before any re-pin whose tree changes
|
|
`packages/harness` or the session fork code (`scripts/mosaic-task.mjs` session and `sessionForkFrom`
|
|
handling, the pi adapter), an operator runs `MOSAIC_LIVE_PROVIDER=1 bash scripts/test-task.sh` with the
|
|
real pi credential and a real `release.sh activate`, and records both results on the re-pin row."
|
|
|
|
Mutants (each must fail a named test; on Sage's slot):
|
|
- M-F: `release.sh` health_check fault task `expectExact` forced to `MOSAIC_HELLO_OK`. Fails
|
|
test-release.sh "fault-injected activation refuses", and test 3's OK match. This needs the fault
|
|
activation's `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK` (see the correction above).
|
|
- M-G: the `laneAuthFile(env,run)` call removed from guardDocker. Fails test 1.
|
|
- M-A: `,"adapter":"mock"` removed from test-release's config. Fails test 3 (adapter set is pi).
|
|
- M-L: the `PI_AUTH_FILE` key removed from cleanEnv. Fails test 2.
|
|
- M-T: test 5 itself (sage-2061). As a mutant check too: the `MOSAIC_LIVE_PROVIDER` gate in test-task
|
|
forced on fails test 4 (skip line count and pi adapter).
|
|
|
|
Results, Sage's slot (sage-2147), candidate 81e037d20, 2026-10-21 20:46:16Z to 20:46:59Z. Busy check
|
|
CLEAR before and after. Image `mosaic-poc-agent:0.85.1-r0.0.12` present, so nothing was built. No
|
|
unexpected failure. Singles ran under umask 022 with TMPDIR under /mnt/storage/scratch/tmp.
|
|
|
|
| Run | rc | Result | Time |
|
|
|---|---|---|---|
|
|
| gate-lane-auth.test.mjs (new) | 0 | 5 pass, 0 fail | 14 s |
|
|
| gate-lane-image.test.mjs | 0 | 11 pass, 0 fail | 0 s |
|
|
| gate-lane.test.mjs | 0 | 13 pass, 0 fail | 0 s |
|
|
| gate-lane-state.test.mjs | 0 | 14 pass, 0 fail | 2 s |
|
|
| gate-lane-fixtures.test.mjs | 0 | 8 pass, 0 fail | 0 s |
|
|
| test-task.sh | 0 | OK 91, FAIL 0, the two live-skip lines | 12 s |
|
|
| test-release.sh | 0 | OK 14, FAIL 0, Docker section on the mock adapter | 2 s |
|
|
|
|
Mutants, each run on its named gate-lane-auth test and restored after:
|
|
|
|
| Mutant | Target | Result |
|
|
|---|---|---|
|
|
| M-G | test 1 | killed (0 pass, 1 fail) |
|
|
| M-L | test 2 | killed (0 pass, 1 fail) |
|
|
| M-A | test 3 | killed (0 pass, 1 fail) |
|
|
| M-T | test 4 | killed (0 pass, 1 fail), 6 s |
|
|
| M-F | test 3 | killed (0 pass, 1 fail) |
|
|
| M-F | real test-release.sh | killed: rc 1, OK 11, FAIL 3, the first `fault-injected activation refuses (exit 0, expected 1)` |
|