row 244: gate runs mount a 0600 fixture pi auth file, test-release activates on the mock adapter, and real-model test-task cases sit behind MOSAIC_LIVE_PROVIDER=1 (#1757)
Candidate morgana/row24494ab4c2feona36523713, manifest 6e7e3e3c (8 paths). Webby APPROVE r1, rev 3008 (webby-0084, REVIEW-r1.md at webby/reviews ebb3d83bf; request #1757 c30154). Gate on Sage's slot at81e037d20(sage-2147): the five gate-lane files whole 51/0, test-task.sh 91/0, test-release.sh 14/0, mutants M-G, M-L, M-A, M-T and M-F killed.94ab4c2feadds only BUILD.md results. Webby's drift check: a36523713..d37210288 changes none of the 8 paths and no path either suite reads; merge-tree clean. Rulings A and C (sage-1914, sage-2061). The dropped real-model coverage goes to DEFERRED in the follow-up doc commit.
This commit is contained in:
1 parent
3a853f35dc
commit
e69260a3ce
8 files changed
+400
-23
No files matched your search
@@ -0,0 +1,230 @@
|
||||
# Row 244: gate runs use a fixture pi auth file
|
||||
|
||||
Brief: `docs/plans/2026-10-21_gate-env-rows.md` § "Gate runs use a fixture pi auth file" (Sage, sage-1909).
|
||||
Reviewer Webby. DEFERRED entry "Gate runs mount the host's pi auth file" (059577886). Design notes
|
||||
written before row 197 lands, read against `origin/herb/row197-r6` (df0c59608). Recheck every line cite
|
||||
against 197's landing before code starts.
|
||||
|
||||
## Where Compose is reached
|
||||
|
||||
`compose.yaml:43` mounts `${PI_AUTH_FILE:-/home/jwoltje/.pi/agent/auth.json}` read-only. The default is
|
||||
a literal path, not `$HOME`, so keeping `HOME` in the lane env doesn't change it.
|
||||
|
||||
Gate steps come from `discoverSteps` (`gate-lane.mjs:32`): one `pkg-<name>` step per package with tests
|
||||
(not queue) and one `script-<name>` step per `scripts/test-*.sh`. A grep of those for `run-task.sh`,
|
||||
`docker compose`, `hello.sh`, `verify.sh`, `agent.sh` and `release.sh activate` finds two Compose
|
||||
paths. The other hits (`cohort.test.mjs` "hello" shim requests, `test-auth.sh`) don't reach Compose.
|
||||
|
||||
| Step | Path to Compose | Adapter |
|
||||
|---|---|---|
|
||||
| `test-task.sh` :150-385 | `scripts/run-task.sh run` → `mosaic-task.mjs:396` `docker compose run` with `spawnEnv = {...process.env}` | mock (`mock_config`) |
|
||||
| `test-release.sh` :69, :85, :93 | `release.sh activate` → `health_check` (:41-55) → `run-task.sh run tasks/hello-marker.json` | **pi** (config :61 sets no adapter; provider zai, model glm-5.3-flash) |
|
||||
|
||||
`mosaic-task.mjs` passes `process.env` through, so a `PI_AUTH_FILE` exported by a suite reaches Compose.
|
||||
|
||||
## The conflict: test-release's healthy activation is a live provider call
|
||||
|
||||
"healthy activation succeeds" (:85) and "repeat activation succeeds" (:93) need `health_check` to
|
||||
pass. That runs the real pi adapter, asks glm-5.3-flash for `MOSAIC_HELLO_OK`, and authenticates through
|
||||
the mounted auth file. The lane's `cleanEnv` drops `ZAI_API_KEY`, so the mount is the only credential.
|
||||
With a fixture auth file the provider call fails, and four checks fail: :85, the pointer check after it,
|
||||
:93 and "log is append-only". So every gate run today spends a real provider call on the host credential.
|
||||
|
||||
Options put to Sage (morgana-0037):
|
||||
- **A (recommended).** test-release's config sets `"adapter":"mock"`, and the two healthy activations
|
||||
run with `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`. The fault injection still refuses, because its task
|
||||
expects `MOSAIC_FAULT_<random>`. The suite keeps proving the release state machine, the image and
|
||||
Compose path, and the pointer and log. It stops proving that the pi image can reach the provider.
|
||||
Real `release.sh activate` runs still do that, with the real credential. No credential and no spend
|
||||
in the gate.
|
||||
- **B.** Keep the live call. The lane passes the real auth file explicitly. That contradicts the brief
|
||||
and DEFERRED, so I don't take it without a ruling.
|
||||
|
||||
**Ruling: A (sage-1914, 09:51Z).** B keeps a live billed call on every gate, which is what this row
|
||||
stops. Sage's conditions:
|
||||
1. test-release.sh's suite config sets `"adapter":"mock"`, with a comment beside it saying why: no
|
||||
credential and no provider call in the gate (row 244). The healthy activations run with
|
||||
`MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`.
|
||||
2. Fault injection still refuses. It keeps the random `MOSAIC_FAULT_<n>` expectation, and the mock's
|
||||
fixed reply can't satisfy it. Mutant M-F forces the fault task's expected value to `MOSAIC_HELLO_OK`.
|
||||
It must fail the suite.
|
||||
3. The fake-docker test also asserts that test-release's activations ran with the mock adapter. The
|
||||
fake docker records `MOSAIC_ADAPTER`, which `mosaic-task.mjs` sets in `spawnEnv` from the resolved
|
||||
config. Then a suite drifting back to pi fails the test.
|
||||
4. Coverage dropped: test-release.sh no longer proves that the pi image reaches the provider. A real
|
||||
`release.sh activate` still proves it, with the real credential. Sage adds a DEFERRED line at landing
|
||||
if Webby wants one.
|
||||
|
||||
guardDocker's new refusal also gets a mutant (M-G, below).
|
||||
|
||||
## Change
|
||||
|
||||
1. `scripts/test-task.sh` and `scripts/test-release.sh`: right after `SANDBOX` is made, write
|
||||
`$SANDBOX/pi/auth.json` (`{}`, mode set by `chmod 600` after the write, so it doesn't depend on the
|
||||
umask, row 243) and `export PI_AUTH_FILE`. It has to be a regular file that exists, because a missing
|
||||
bind source makes Docker create a directory on the host. The suites set it themselves, so the legacy
|
||||
gate (outside the lane) is covered too.
|
||||
2. `scripts/test-release.sh`: option A above.
|
||||
3. `scripts/testing/gate-lane.mjs` `cleanEnv`: set `PI_AUTH_FILE` to `join(o.tempRoot,'pi-auth.json')`,
|
||||
and `run` writes that file (`{}`, `flag:'wx'`, then chmod 0600) next to where it writes `bin/docker`.
|
||||
Any later step that reaches Compose without its own fixture then gets the lane's fixture.
|
||||
4. `scripts/testing/lane-docker.mjs` `guardDocker`: `compose run` also refuses unless `PI_AUTH_FILE` is
|
||||
set, resolves inside `run.tempRoot` and isn't the host default (refusal text
|
||||
`Compose auth mount not a lane fixture`). The guard already sees every Docker call a lane step makes,
|
||||
so this is where "no gate step mounts the host file" is enforced at run time. The CLI entry passes
|
||||
`process.env`.
|
||||
5. `compose.yaml` is unchanged (brief: not in scope).
|
||||
6. At landing: the DEFERRED entry moves to Done with the landing commit.
|
||||
|
||||
## Tests
|
||||
|
||||
In `packages/cli/tests/gate-lane.test.mjs` (row 197's file; the exact file follows 197's landing):
|
||||
|
||||
- `guardDocker` refuses `compose run` with `PI_AUTH_FILE` unset, set to the host default, or set outside
|
||||
`tempRoot`; it accepts the lane fixture and a suite fixture under `tempRoot`. Other verbs are unchanged.
|
||||
- `cleanEnv` returns a `PI_AUTH_FILE` inside `tempRoot` that isn't the host default.
|
||||
- The suites, read through their Compose environment: run `test-task.sh` and `test-release.sh` with a
|
||||
fake `docker` first on `PATH`. It answers `info` and `image inspect` with 0, and for `compose run` it
|
||||
appends `PI_AUTH_FILE` and whether that file is a regular 0600 file to a record. Then it prints the
|
||||
mock response. The test ignores the suites' own exit code and asserts that the record has at least one
|
||||
`compose run` from each suite and that every entry names an existing fixture file, never
|
||||
`/home/jwoltje/.pi/agent/auth.json`. The test never reads the host file. If a full suite run with the
|
||||
fake docker turns out too slow or brittle, I'll say so in the review request, with timings, and fall
|
||||
back to the guard and `cleanEnv` tests plus a static check that each suite exports `PI_AUTH_FILE`
|
||||
before its first Compose call. I won't drop it silently.
|
||||
|
||||
Mutants (each must fail a test; run on Sage's slot):
|
||||
- M-F: the fault task's `expectExact` forced to `MOSAIC_HELLO_OK` in `release.sh` `health_check`. Must
|
||||
fail `test-release.sh` ("fault-injected activation refuses").
|
||||
- M-G: the `guardDocker` `PI_AUTH_FILE` check removed. Must fail the guard test.
|
||||
- M-A: test-release's config back to no adapter (pi). Must fail the fake-docker test (condition 3).
|
||||
- M-L: `cleanEnv` stops setting `PI_AUTH_FILE`. Must fail the `cleanEnv` test.
|
||||
|
||||
Gate: those tests, `test-task.sh` and `test-release.sh` whole, the 197 lane tests, and the four
|
||||
mutants, all on Sage's slot.
|
||||
|
||||
## Files (expected)
|
||||
|
||||
`scripts/test-task.sh`, `scripts/test-release.sh`, `scripts/testing/gate-lane.mjs`,
|
||||
`scripts/testing/lane-docker.mjs`, `packages/cli/tests/gate-lane.test.mjs` (or a new
|
||||
`gate-lane-auth.test.mjs`), `docs/plans/DEFERRED.md` (at landing), `scripts/testing/README.md` if it
|
||||
lists the guard's rules.
|
||||
|
||||
## Correction (morgana-0040, 18:50Z)
|
||||
|
||||
The table above says `test-task.sh` uses only the mock adapter. That was wrong. On a36523713 two
|
||||
sections run the pi adapter through `$CONFIG` (provider zai) and authenticate through the host mount:
|
||||
`:443-501` (live hello, result.json written and correct, wrong expectExact, distinct run dirs, list,
|
||||
fork base, fork child, fork recall, ancestor untouched, child branch file) and `:518-530` (user recall
|
||||
run, recalled the user name). That's 13 checks and about 6 billed provider calls per gate run. I missed
|
||||
them because I read the mock section and stopped there.
|
||||
|
||||
**Ruling C (sage-2061, 18:50Z).** D (keep them live) refused.
|
||||
1. The run-record checks (hello, result.json, wrong expectExact, distinct run dirs, list) move to the
|
||||
mock adapter. Every assertion is kept. Only the response source changes.
|
||||
2. Fork recall and user recall run only under `MOSAIC_LIVE_PROVIDER=1`. Otherwise the suite prints
|
||||
`skip live provider cases (row 244: no provider call in the gate)`, which the lane's requiredSkips
|
||||
regex doesn't match, on purpose. A test asserts the line prints when the variable is unset and that
|
||||
the live checks run, and fail under a fixture, when it's 1 (M-T).
|
||||
3. The dropped coverage and the DEFERRED wording are below. Sage writes the entry at landing.
|
||||
|
||||
## Build (on a36523713, sage-2056)
|
||||
|
||||
Rechecked against 197's landing: `lane-docker.mjs:30` guardDocker, `gate-lane.mjs:109` cleanEnv and
|
||||
`:181` (bin/docker written), `compose.yaml:43`, `release.sh:41-55` health_check, `mosaic-task.mjs:321-322`
|
||||
spawnEnv and MOSAIC_ADAPTER, `test-release.sh:61` config. All hold.
|
||||
|
||||
- `scripts/testing/lane-docker.mjs`: `guardDocker(argv,run,env=process.env)`. After the option check, a
|
||||
`compose run` calls `laneAuthFile`. It refuses (`Compose auth mount not a lane fixture`) unless
|
||||
`PI_AUTH_FILE` is absolute, isn't the host default, resolves (realpath) inside the realpath of
|
||||
`run.tempRoot`, and is a regular file. Symlinks out of the root and relative paths refuse. `info` and
|
||||
`image inspect` need no auth file. The CLI entry passes `process.env`.
|
||||
- `scripts/testing/gate-lane.mjs`: `cleanEnv` sets `PI_AUTH_FILE=join(o.tempRoot,'pi-auth.json')`.
|
||||
`runGate` writes it (`{}`, flag `wx`, then `chmodSync` 0600) right after `bin/docker`.
|
||||
`MOSAIC_LIVE_PROVIDER` isn't on cleanEnv's keep list, so the lane can't request live cases.
|
||||
- `scripts/test-release.sh`: exports a `$SANDBOX/pi/auth.json` fixture (`{}`, `chmod 600` after the
|
||||
write, exits 1 if the write fails). The config adds `"adapter":"mock"` with the ruling A comment. The
|
||||
fault, healthy and repeat activations all run with `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK`, so the
|
||||
fault activation refuses only because its `MOSAIC_FAULT_<n>` expectation is wrong. (Correction,
|
||||
before any run: the first build gave the fault activation no mock response. The mock then answered
|
||||
empty, which fails any expectation, so M-F would have survived. Fixed in the follow-up commit.)
|
||||
- `scripts/test-task.sh`: same fixture, exported only when `MOSAIC_LIVE_PROVIDER` isn't 1, so an
|
||||
operator's live run keeps their own `PI_AUTH_FILE` (or Compose's default). Ruling C (a) and (b) as
|
||||
above. The section header changes from "live: real runs" to "run records through the mock adapter",
|
||||
the check name "live hello task..." becomes "hello task...", and the docker-less branch of the new
|
||||
split prints `skip run-record cases (docker unavailable)`.
|
||||
- `packages/cli/tests/gate-lane-image.test.mjs` (:21, :106, :119): the compose-run calls pass
|
||||
`run.env`, the composeFixture env, which already holds a 0600 `PI_AUTH_FILE` inside tempRoot (accepted
|
||||
by sage-2061).
|
||||
- `scripts/testing/README.md`: one sentence on the lane fixture and the refusal.
|
||||
- New `packages/cli/tests/gate-lane-auth.test.mjs`, five tests:
|
||||
1. guardDocker accepts the lane fixture and a suite fixture under tempRoot. It refuses unset, empty,
|
||||
the host default, a file outside, a symlink inside that points outside, a missing path, a directory
|
||||
and a relative path. The option check still comes first. `info` and `image inspect` pass with no env.
|
||||
2. cleanEnv sets the lane fixture path and drops `MOSAIC_LIVE_PROVIDER` and the caller's
|
||||
`PI_AUTH_FILE`.
|
||||
3. test-release.sh in a private copy of the checkout, with a fake `docker` first on PATH. The fake
|
||||
answers every verb with 0. For `compose run` it records `PI_AUTH_FILE`, `stat -c '%F %a'` of it
|
||||
(never for the host path) and `MOSAIC_ADAPTER`, and it prints `MOSAIC_MOCK_RESPONSE` when the
|
||||
adapter is mock. Asserts: at least 3 compose runs, every one on a regular 0600 file under the
|
||||
suite's TMPDIR, and every adapter mock (condition 3). The healthy and fault lines print OK.
|
||||
4. test-task.sh, same harness: every compose run on its fixture and the mock adapter, the skip line
|
||||
printed exactly twice, and the hello check OK.
|
||||
5. M-T: test-task.sh with `MOSAIC_LIVE_PROVIDER=1` and an operator-style fixture. No skip line,
|
||||
`FAIL fork base: teach succeeds` and `FAIL recalled user name`, at least one pi run, and every run
|
||||
on the operator's file.
|
||||
The copy leaves out `.git`, `node_modules` (symlinked), `v1`, `agents`, `docs`, `BUILD-LOG.md` and
|
||||
`scripts/foundation`, about 14 MB. Copying is needed because `test-release.sh` rewrites `RELEASE`
|
||||
while it runs, which would race other cli tests that read it. The suites ignore their own exit codes
|
||||
here: the fake doesn't emit the mock adapter's env lines, so some seam checks fail by design.
|
||||
- Readers: `gate-lane.mjs` is imported by gate-lane.test, gate-lane-state.test,
|
||||
gate-lane-fixtures.test, gate-lane-image.test and fixtures/gate-lane-child.mjs (unchanged exports).
|
||||
`lane-docker.mjs` by gate-lane-image.test. No test hash-pins any changed file. vars.test.mjs names
|
||||
`scripts/test-task.sh` only as a string value.
|
||||
|
||||
Coverage dropped from the gate (for Webby, and for the DEFERRED entry):
|
||||
- `test-release.sh` no longer proves that the pi image reaches the provider (ruling A).
|
||||
- `test-task.sh` no longer proves session fork recall or onboarding user recall against a real model
|
||||
(ruling C).
|
||||
|
||||
DEFERRED wording for Sage (sage-2061 condition 3): "**Real-model suite cases run outside the gate.**
|
||||
Row 244 moved `test-task.sh` fork recall (M11) and user recall (M16) behind `MOSAIC_LIVE_PROVIDER=1`,
|
||||
and `test-release.sh` activations to the mock adapter. Before any re-pin whose tree changes
|
||||
`packages/harness` or the session fork code (`scripts/mosaic-task.mjs` session and `sessionForkFrom`
|
||||
handling, the pi adapter), an operator runs `MOSAIC_LIVE_PROVIDER=1 bash scripts/test-task.sh` with the
|
||||
real pi credential and a real `release.sh activate`, and records both results on the re-pin row."
|
||||
|
||||
Mutants (each must fail a named test; on Sage's slot):
|
||||
- M-F: `release.sh` health_check fault task `expectExact` forced to `MOSAIC_HELLO_OK`. Fails
|
||||
test-release.sh "fault-injected activation refuses", and test 3's OK match. This needs the fault
|
||||
activation's `MOSAIC_MOCK_RESPONSE=MOSAIC_HELLO_OK` (see the correction above).
|
||||
- M-G: the `laneAuthFile(env,run)` call removed from guardDocker. Fails test 1.
|
||||
- M-A: `,"adapter":"mock"` removed from test-release's config. Fails test 3 (adapter set is pi).
|
||||
- M-L: the `PI_AUTH_FILE` key removed from cleanEnv. Fails test 2.
|
||||
- M-T: test 5 itself (sage-2061). As a mutant check too: the `MOSAIC_LIVE_PROVIDER` gate in test-task
|
||||
forced on fails test 4 (skip line count and pi adapter).
|
||||
|
||||
Results, Sage's slot (sage-2147), candidate 81e037d20, 2026-10-21 20:46:16Z to 20:46:59Z. Busy check
|
||||
CLEAR before and after. Image `mosaic-poc-agent:0.85.1-r0.0.12` present, so nothing was built. No
|
||||
unexpected failure. Singles ran under umask 022 with TMPDIR under /mnt/storage/scratch/tmp.
|
||||
|
||||
| Run | rc | Result | Time |
|
||||
|---|---|---|---|
|
||||
| gate-lane-auth.test.mjs (new) | 0 | 5 pass, 0 fail | 14 s |
|
||||
| gate-lane-image.test.mjs | 0 | 11 pass, 0 fail | 0 s |
|
||||
| gate-lane.test.mjs | 0 | 13 pass, 0 fail | 0 s |
|
||||
| gate-lane-state.test.mjs | 0 | 14 pass, 0 fail | 2 s |
|
||||
| gate-lane-fixtures.test.mjs | 0 | 8 pass, 0 fail | 0 s |
|
||||
| test-task.sh | 0 | OK 91, FAIL 0, the two live-skip lines | 12 s |
|
||||
| test-release.sh | 0 | OK 14, FAIL 0, Docker section on the mock adapter | 2 s |
|
||||
|
||||
Mutants, each run on its named gate-lane-auth test and restored after:
|
||||
|
||||
| Mutant | Target | Result |
|
||||
|---|---|---|
|
||||
| M-G | test 1 | killed (0 pass, 1 fail) |
|
||||
| M-L | test 2 | killed (0 pass, 1 fail) |
|
||||
| M-A | test 3 | killed (0 pass, 1 fail) |
|
||||
| M-T | test 4 | killed (0 pass, 1 fail), 6 s |
|
||||
| M-F | test 3 | killed (0 pass, 1 fail) |
|
||||
| M-F | real test-release.sh | killed: rc 1, OK 11, FAIL 3, the first `fault-injected activation refuses (exit 0, expected 1)` |
|
||||
Reference in new issue
Block a user