feat(harness): the pi fit check runs pi once against a scripted loopback model and caches the verdict for the digest (#1873) #2019

Merged
jason.woltje merged 2 commits from pi-unpin-s6 into refactor 2026-11-20 17:59:39 +00:00
Owner

What changes

Lane pi-unpin-s6, slice S6 of the pi unpin design: the fit check runs pi once against a scripted model, so the behaviours Mosaic needs are proved on the pi that runs.

  • The scripted run. New packages/harness/src/pi-fit-model.mjs is a loopback model that runs as the first process in the sandbox, with the network unshared, under a stand-in provider and a model named fit-model. The script reads a file the gate allows, a file outside the project, the same file under each of the four read spellings pi 0.85.1 tries, and a file that a second fit-only extension (src/pi-fit-throw.mjs) throws for.
  • Verdicts. A pi that runs a blocked tool, swallows a handler's throw, drops the --system-prompt file or never emits agent_end with the model's final text is refused with the harness-feature-missing line. A run that times out, whose model gets no request, or that exits non-zero is undecided. A pi that leaks a blocked read and then hangs is refused, not undecided.
  • Cache. The verdict is cached with the pass for the digest, so the run happens on first sight of a digest and not per launch.
  • Tests. tests/helpers.mjs reuses the scripted model. pi-stand-in.mjs gains the faults the new tests need. pi-session.test.mjs takes the expected pi from the pi module's own lookup, so it follows MOSAIC_PI_BIN.

No new verb, refusal code, grant, role, schema or config format. The codes harness-feature-missing and engine-feature-missing were approved on 2026-11-19.

Speed

Median of 5 cold runs of the real pi 0.85.1 check, base and tip alternating: 2151 ms at base, 2161 ms at the tip. About 10 ms added.

Tests

  • Lane proof on df3ff6499: 211 files under harness, plugins, conversation, discord and cli, 2830 pass, 2 fail, 33 skipped. The 2 fails are flows.test.mjs and reader.test.mjs, which import Python's jsonschema. This host lacks it and both are red at base.
  • Rebased onto 4aa008e9f with no shared files. range-diff shows both commits unchanged.
  • Lead check in a clean environment on the rebased tip 9869200a0: 303 files, 3447 pass, 0 fail, 2 skipped.

Review

Two rounds by fresh claude-opus-5-5 reviewers. Round one found six minor issues, mostly checks that no test held, all fixed. The re-check approved with no new findings.

Left for later

  • pi 1.5.2 by hand is undecided: "it didn't answer get_state". A pi outside the repository gets only its own package directory mounted (S4), so its hoisted jiti dependency is missing and no extension loads. With the whole node_modules mounted it was still undecided on a different control. Both need a follow-up before the lock moves to 1.5.2.
  • S0 line 5 (bash) is not checked at start.
  • A read spelling pi adds after 0.85.1 is not scripted.
  • A pi built to forge its own record and marker files on the run's writable mount is out of scope, as the design says.
  • The agent_end check needs a string stopReason but does not check its value.
## What changes Lane pi-unpin-s6, slice S6 of the pi unpin design: the fit check runs pi once against a scripted model, so the behaviours Mosaic needs are proved on the pi that runs. - **The scripted run.** New `packages/harness/src/pi-fit-model.mjs` is a loopback model that runs as the first process in the sandbox, with the network unshared, under a stand-in provider and a model named fit-model. The script reads a file the gate allows, a file outside the project, the same file under each of the four read spellings pi 0.85.1 tries, and a file that a second fit-only extension (`src/pi-fit-throw.mjs`) throws for. - **Verdicts.** A pi that runs a blocked tool, swallows a handler's throw, drops the `--system-prompt` file or never emits `agent_end` with the model's final text is refused with the `harness-feature-missing` line. A run that times out, whose model gets no request, or that exits non-zero is undecided. A pi that leaks a blocked read and then hangs is refused, not undecided. - **Cache.** The verdict is cached with the pass for the digest, so the run happens on first sight of a digest and not per launch. - **Tests.** `tests/helpers.mjs` reuses the scripted model. `pi-stand-in.mjs` gains the faults the new tests need. `pi-session.test.mjs` takes the expected pi from the pi module's own lookup, so it follows `MOSAIC_PI_BIN`. No new verb, refusal code, grant, role, schema or config format. The codes `harness-feature-missing` and `engine-feature-missing` were approved on 2026-11-19. ## Speed Median of 5 cold runs of the real pi 0.85.1 check, base and tip alternating: 2151 ms at base, 2161 ms at the tip. About 10 ms added. ## Tests - Lane proof on df3ff6499: 211 files under harness, plugins, conversation, discord and cli, 2830 pass, 2 fail, 33 skipped. The 2 fails are `flows.test.mjs` and `reader.test.mjs`, which import Python's `jsonschema`. This host lacks it and both are red at base. - Rebased onto 4aa008e9f with no shared files. `range-diff` shows both commits unchanged. - Lead check in a clean environment on the rebased tip 9869200a0: 303 files, 3447 pass, 0 fail, 2 skipped. ## Review Two rounds by fresh claude-opus-5-5 reviewers. Round one found six minor issues, mostly checks that no test held, all fixed. The re-check approved with no new findings. ## Left for later - pi 1.5.2 by hand is undecided: "it didn't answer get_state". A pi outside the repository gets only its own package directory mounted (S4), so its hoisted `jiti` dependency is missing and no extension loads. With the whole node_modules mounted it was still undecided on a different control. Both need a follow-up before the lock moves to 1.5.2. - S0 line 5 (bash) is not checked at start. - A read spelling pi adds after 0.85.1 is not scripted. - A pi built to forge its own record and marker files on the run's writable mount is out of scope, as the design says. - The `agent_end` check needs a string stopReason but does not check its value.
jason.woltje added 2 commits 2026-11-20 17:59:36 +00:00
jason.woltje merged commit 9869200a03 into refactor 2026-11-20 17:59:39 +00:00
Sign in to join this conversation.