feat(harness): the pi fit check runs pi once against a scripted loopback model and caches the verdict for the digest (#1873) #2019

Merged
jason.woltje merged 2 commits from pi-unpin-s6 into refactor 2026-11-20 17:59:39 +00:00
9 changed files with 660 additions and 79 deletions

No files matched your search

+33 -7
View File
@@ -20,17 +20,17 @@ bundle's `manifest.json` names the lines it relies on (`reliesOn`):
| | Pi | Claude Code | How pi's line is checked |
|---|---|---|---|
| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | suites, on the repository's pi (`pi-session.test.mjs`) |
| 2. gate crash | a throw in the handler blocks | the hook runs as `<gate> \|\| exit 2` | suites, on the repository's pi |
| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | at first sight of a binary, by the fit check's model run, on the pi that runs; and suites (`pi-session.test.mjs`) |
| 2. gate crash | a throw in the handler blocks | the hook runs as `<gate> \|\| exit 2` | at first sight of a binary, by the fit check's model run; and suites |
| 3. gate missing | a missing `-e` file refuses to start | the wrapper blocks | at every start, by the fit check, on the pi that runs; and suites |
| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 <gate> \|\| exit 2`, hook timeout 20 | suites, on the repository's pi |
| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 <gate> \|\| exit 2`, hook timeout 20 | the `agent_end` marker: at first sight of a binary, by the fit check's model run; and suites |
| 5. bash | limited only by the tool limit | limited only by the tool limit | suites, on the repository's pi |
Pi is not pinned (#1873 pi-unpin). Before a session starts, `preflight`
awaits the fit check in `src/pi-fit.mjs` on the binary the module found
(the Triage host and route await the module's `ready` instead), and
`env` reads its pass. It runs that binary in a sandbox with the network
unshared and calls no model. Each run is a child process the check
unshared and calls no real model. Each run is a child process the check
awaits, up to four at a time, so the bus's event loop keeps running
while it does; callers that ask while a check runs share it. Each run
gets the environment a bus session's pi gets, built by the same code
@@ -131,9 +131,35 @@ for example `pi 2.0.0 (MOSAIC_PI_BIN) lacks --no-context-files`. When
`--help` lists `--no-mcp`, the adapter passes it too. A pass is cached in
memory by a digest of the package's files (and npm's lock files, for
the repository's copy) and the names in the runs' environment, so a bus
restarted under other names checks again. Lines 1, 2, 4 and 5 are not
checked at start; a pi other than the repository's copy relies on them
unproved until a scripted-model fit run exists (design section 5, S6).
restarted under other names checks again.
One more run (#1873 S6) talks to a scripted model, so lines 1, 2 and 4
and the system prompt are checked on the pi that runs. It is a
session-shaped `-p` run with `--tools read`, the gate's `-e` extension
and a second one, `src/pi-fit-throw.mjs`, on a stand-in provider and
model named `fit-model` (never a Claude model). The model is
`src/pi-fit-model.mjs`: inside the sandbox it is the first process, it
listens on the sandbox's own loopback (the network stays unshared),
writes the agent directory's `models.json` for its port, starts pi and
answers one scripted tool call per request. The script reads a project
file the gate allows, a file outside the project, the same outside file
through a link under each of the four spellings pi 0.85.1 tries for a
missing path (a narrow space before AM, decomposed accents, a curly
apostrophe, both), and a project file the second extension's handler
throws for. The run passes only when the allowed read returns the file,
every blocked read is a tool error carrying the gate's own reason and
no request carries the outside file's word, the throw stops its read,
every request carries the `--system-prompt` file, the script runs to
its end, and the gate's `agent_end` hook writes the turn marker with the
model's final text answer. A pi that runs a blocked tool, swallows a
handler's throw, drops the system prompt or never emits `agent_end` is
refused. A run that times out, exits non-zero, or whose model gets no
request or sees the run end early, can't be checked and refuses as
undecided. A run that leaked and then hung is refused, not undecided. Its verdict is part of the cached pass, so it
runs once per digest. It runs beside the other runs, so a cold check
takes about as long as before. Line
5 (bash) is not checked at start, and nor is a read spelling pi may add
later; a stand-in can only check what it scripts.
Claude Code also runs with `--restricted`: no user, project or local
settings, file tools confined to the working directory, and no `CLAUDE.md`
+111
View File
@@ -0,0 +1,111 @@
// #1873 pi-unpin S6: the scripted model the fit check's model run talks to (pi-fit.mjs). It
// answers the Messages API one scripted step at a time: step k is a tool call, given when the
// request carries k tool results; after the last step it answers with fixed text. Only node's own
// modules are imported, since the file runs inside the fit check's sandbox.
//
// Run as a program, inside the sandbox: `node pi-fit-model.mjs <config.json> -- <command...>`. It
// listens on 127.0.0.1 port 0 in the sandbox's own network namespace (checkArgv unshares the
// network, so only that namespace's loopback reaches it), writes the agent directory's models.json
// naming that port for the stand-in provider, starts the command with its own environment and
// stdio, writes what each request held to the record file after each request, and exits with the
// command's status. It is the first process in the sandbox's pid namespace, so when it ends, or is
// killed, every process in the sandbox ends with it. No real model and no host network are
// reached: the sandbox has no other network.
//
// packages/harness/tests/helpers.mjs mockAnthropic serves the same answers to session tests.
import { createServer } from "node:http";
import { spawn } from "node:child_process";
import { mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { fileURLToPath } from "node:url";
export const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b?.text ?? "").join("") : "");
// The tool results in a request's messages, in order: { id, isError, content }.
export function toolResults(messages) {
const out = [];
for (const m of messages ?? []) {
if (m?.role !== "user" || !Array.isArray(m.content)) continue;
for (const b of m.content) if (b?.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) });
}
return out;
}
// A request handler for the Messages API. script: [{ name, input }], one tool call per step.
// done(results): the text that ends the run once the script is spent. requests: each request is
// pushed as { tools, results, system, answer, raw }. onRequest: called after each push. A request
// with no tools gets "mock". Streaming and plain answers both work; count_tokens answers 10.
export function messagesHandler({ script = [], done = (r) => `DONE ${JSON.stringify(r)}`, requests = [], onRequest = () => {} } = {}) {
return (req, res) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) {
res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" });
return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } }));
}
let body;
try { body = JSON.parse(raw || "{}"); } catch { body = {}; }
const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : [];
const results = toolResults(body.messages);
const step = tools.length ? script[results.length] : null;
const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" };
requests.push({ tools, results, system: body.system, answer: r, raw });
onRequest();
const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" };
const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text };
const stop = r.kind === "tool" ? "tool_use" : "end_turn";
if (!body.stream) {
res.writeHead(200, { "content-type": "application/json" });
const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text };
return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } }));
}
const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`);
res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } });
send("content_block_start", { index: 0, content_block: content });
send("content_block_delta", { index: 0, delta });
send("content_block_stop", { index: 0 });
send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } });
send("message_stop", {});
res.end();
});
};
}
// How much of each tool result the record keeps.
const KEEP = 2000;
// config: { record, models, provider, model, key, script, done, secret }. record: the file each
// request is written to, as { requests: [{ tools, system, results: [{ isError, content }],
// done, leak }] }: tools is the count of tools offered, done whether the answer was the done text,
// leak whether the raw request held the secret word.
async function main(argv) {
const [config, sep, ...command] = argv;
if (!config || sep !== "--" || !command.length) process.exit(2);
const c = JSON.parse(readFileSync(config, "utf8"));
const requests = [];
const record = () => writeFileSync(c.record, JSON.stringify({ requests: requests.map((r) => ({
tools: r.tools.length,
system: textOf(r.system),
results: r.results.map((x) => ({ isError: Boolean(x.isError), content: String(x.content).slice(0, KEEP) })),
done: r.answer.kind === "text" && r.answer.text === c.done,
leak: r.raw.includes(c.secret),
})) }));
const server = createServer(messagesHandler({ script: c.script, done: () => c.done, requests, onRequest: record }));
await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
mkdirSync(dirname(c.models), { recursive: true });
writeFileSync(c.models, JSON.stringify({ providers: { [c.provider]: { baseUrl: `http://127.0.0.1:${server.address().port}`, api: "anthropic-messages", apiKey: c.key, models: [{ id: c.model }] } } }));
const child = spawn(command[0], command.slice(1), { stdio: "inherit", env: process.env });
const status = await new Promise((resolve) => {
child.on("error", () => resolve(1));
child.on("exit", (code, signal) => resolve(code ?? (signal ? 1 : 0)));
});
server.close();
process.exit(status);
}
const self = fileURLToPath(import.meta.url);
let invoked = null;
try { invoked = process.argv[1] ? realpathSync(process.argv[1]) : null; } catch {}
if (invoked === self) main(process.argv.slice(2)).catch(() => process.exit(1));
+14
View File
@@ -0,0 +1,14 @@
// #1873 pi-unpin S6: a second -e extension, for the fit check's model run only (pi-fit.mjs); no
// session loads it. pi loads it after the gate's, so for a read of THROW_NAME the gate has allowed
// the read before this handler throws. The run then shows that a throw in a tool_call handler
// stops the tool (S0 line 2: a gate that crashes blocks). It imports nothing, since it runs inside
// the fit check's sandbox.
export const THROW_NAME = "throw.txt";
export const THROW_TEXT = "fit check: the tool_call handler threw";
export default function fitThrow(host) {
host.on("tool_call", async (event) => {
if (event?.toolName === "read" && event.input?.path === THROW_NAME) throw new Error(THROW_TEXT);
return undefined;
});
}
+175 -19
View File
@@ -2,7 +2,8 @@
// here compares a version. The check runs the binary only inside a sandbox with the network
// unshared, the found package mounted as a session mounts it, and a bus session's environment
// (fitEnv below), so its agent directory is $HOME/.pi/agent under a scratch HOME, as a session's
// is. No model is called and no provider key is passed. It asks:
// is. No real model is called and no provider key is passed: the one run that reaches a model
// reaches a scripted stand-in on the sandbox's own loopback. It asks:
//
// --version a label for the refusal line and the record; not a gate
// --help lists, in its option column, every flag Mosaic passes, with rpc mode and each
@@ -56,6 +57,23 @@
// its own any more.
// missing -e a missing -e file, in rpc and in -p, stops the start: exit non-zero, no answer,
// and no session_start for the -e extension given after it
// model #1873 S6: one -p run in a bus session's shape (sessionShape, with --tools read and
// the session's own -e extension, pi-extension.mjs, which gates every tool call with
// gate.mjs and writes the turn marker at agent_end) against a scripted model:
// pi-fit-model.mjs, started inside the sandbox ahead of the binary, listening on the
// sandbox's loopback and named by the agent directory's models.json under a stand-in
// provider and model (session-env.mjs FIT_MODEL_ROUTE). The script reads a file in
// the project (the gate allows it, and its word must come back), then a file outside
// it, then four names that pi 0.85.1 resolves to another spelling (a narrow space
// before AM, decomposed accents, a curly apostrophe, both of the last two), each a
// link out of the project to the same outside file. Each of those four the gate
// blocks, and each must come back as an error holding the gate's reason, with the
// outside file's word in no request. Every request with tools must carry the
// --system-prompt file's word, the run must reach the script's end and exit 0, and
// the turn marker must be written. A tool result that misses the gate's reason, a
// request without the word, or no marker is a refusal. A run that times out, ends
// early or sends no request can't tell. pi-fit-model.mjs is the sandbox's first
// process, so the model and every child end with it.
//
// Every run gets a fresh scratch directory with the sources planted again, so no run sees what
// another one left. The directory's name carries the pid, start time and pid namespace of the
@@ -100,17 +118,20 @@
// whenever the directory isn't empty), since one more run costs about 340 ms cold; and a
// session's own values: its tool names beyond read and bash, thinking levels beyond low,
// provider, model, key, -e files and request. The runs are offline with no key, so nothing
// past the start is tried. A pi built to act differently with those is outside what a start-time
// check can catch.
// past the start is tried, but for the model run. A pi built to act differently with those, or to
// act differently in a fit check, is outside what a start-time check can catch. The model run
// tries read alone, one tool call per turn, and only the four spellings pi 0.85.1 knows.
import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join, relative } from "node:path";
import { BWRAP, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs";
import { fileURLToPath } from "node:url";
import { BWRAP, RUNNER_FILES, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs";
import { isProviderVariable, sessionEnv } from "../../seat/src/session.mjs";
import { adapterEnv } from "./runner.mjs";
import { FIT_ROUTE, harnessEnv } from "../../plugins/harness/pi/session-env.mjs";
import { THROW_NAME } from "./pi-fit-throw.mjs";
import { FIT_MODEL_ROUTE, FIT_ROUTE, fitPolicy, harnessEnv } from "../../plugins/harness/pi/session-env.mjs";
// The flags adapters/pi/adapter.sh and the conversation engine pass.
export const FIT_FLAGS = Object.freeze(["--mode", "--session", "--no-session", "--session-dir", "--fork", "-c", "--no-extensions", "-e", "--no-skills", "--skill", "--no-prompt-templates", "--no-themes", "--no-context-files", "--no-approve", "--offline", "--tools", "--no-tools", "--provider", "--model", "--thinking", "--system-prompt", "-p"]);
@@ -139,13 +160,14 @@ export const SESSION_RUNS = Object.freeze([
// or none, a persistent session directory with -c when it holds a session, a thinking level or
// none, the --system-prompt file, -p and the request, and --mode json when the model event monitor
// runs pi. No --skill and no --fork: a bus session never passes them. Keep its order:
// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ.
export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message }) {
// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ. route: the
// provider and model, SESSION_SHAPE's unless the model run passes FIT_MODEL_ROUTE.
export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message, route = SESSION_SHAPE }) {
return [
"--offline", "--no-approve", "--no-extensions", ...(noMcp ? ["--no-mcp"] : []), ...extensions.flatMap((e) => ["-e", e]),
"--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files",
...(tools ? ["--tools", tools] : ["--no-tools"]), "--session-dir", sessionDir, ...(cont ? ["-c"] : []),
"--provider", SESSION_SHAPE.provider, "--model", SESSION_SHAPE.model, ...(thinking ? ["--thinking", thinking] : []),
"--provider", route.provider, "--model", route.model, ...(thinking ? ["--thinking", thinking] : []),
"--system-prompt", prompt, "-p", message, ...(json ? ["--mode", "json"] : []),
];
}
@@ -165,6 +187,41 @@ export function workerShape({ noMcp, extensions, skills, prompt, message }) {
// What a session-shaped run is named by in a refusal line.
const shapeName = ({ tools, cont, thinking, json }) => `in a session's shape with ${[tools ? "--tools" : "--no-tools", ...(cont ? ["-c"] : []), ...(thinking ? ["--thinking"] : []), ...(json ? ["--mode json"] : [])].join(", ")}`;
const RUN_TIMEOUT = 20000;
// #1873 S6: the model run. Its first -e extension is the gate's, which a session loads; the second
// is pi-fit-throw.mjs, whose handler throws for one read the gate allows. The model is
// pi-fit-model.mjs. All are mounted read-only where they are, with the runner files the gate's
// extension imports, as a session mounts them (sandbox.mjs binds).
const EXTENSION = fileURLToPath(new URL("./pi-extension.mjs", import.meta.url));
const THROW_EXTENSION = fileURLToPath(new URL("./pi-fit-throw.mjs", import.meta.url));
export const MODEL_EXTENSIONS = Object.freeze([EXTENSION, THROW_EXTENSION]);
const MODEL_FILE = fileURLToPath(new URL("./pi-fit-model.mjs", import.meta.url));
const MODEL_MOUNTS = [MODEL_FILE, THROW_EXTENSION, ...RUNNER_FILES].map((src) => ({ kind: "ro", src }));
export const MODEL_SHAPE = Object.freeze({ tools: "read", cont: false, thinking: "", json: false });
// The project file's word, which the allowed read must bring back; the outside file's, which must
// reach no request; the scripted model's last answer.
const INSIDE = "FITINSIDEWORD";
const SECRET = "FITSECRETWORD";
// The word in the project file the throwing handler's read names, which must reach no request.
const THROWN = "FITTHROWNWORD";
const DONE = "FITDONE";
// The four spellings pi 0.85.1 tries when a read's path is missing: the name the model asks for,
// the name on disk (a link to the outside file) and what a refusal line calls it.
const SPELLINGS = [
["shot 9.41 AM.png", "shot 9.41\u202FAM.png", "a narrow space before AM"],
["r\u00E9sum\u00E9.txt", "r\u00E9sum\u00E9.txt".normalize("NFD"), "decomposed accents"],
["notes's.txt", "notes\u2019s.txt", "a curly apostrophe"],
["d'\u00E9cran.png", "d\u2019\u00E9cran.png".normalize("NFD"), "decomposed accents and a curly apostrophe"],
];
// The script: the allowed read, the read outside the project, one read per spelling, then the read
// the gate allows and the second handler throws for.
export const MODEL_SCRIPT = Object.freeze([
{ name: "read", input: { path: "inside.txt" } },
{ name: "read", input: { path: "../outside/secret.txt" } },
...SPELLINGS.map(([asked]) => ({ name: "read", input: { path: asked } })),
{ name: "read", input: { path: THROW_NAME } },
].map((step) => Object.freeze(step)));
const THROW_STEP = MODEL_SCRIPT.length - 1;
// --version gives only a label, so a hang costs this much and leaves no label.
const VERSION_TIMEOUT = 3000;
const OUTPUT_MAX = 4 * 1024 * 1024;
@@ -374,13 +431,21 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c
// get_commands answer or through the -e extension's command list (listed: through that alone).
// shape: a session-shaped run's branches, which set its tools, thinking level and monitor
// variables as a session's would. A run that stop kept from starting gives null.
const once = async (args, input, { timeout, noMcp, shape }) => {
// model (S6): the scripted-model run, with its own plant, mounts and environment; it gives the
// run's status and stopped with the requests the model recorded and the turn marker.
const once = async (args, input, { timeout, noMcp, shape, model }) => {
await slot();
try {
if (stop.signal.aborted) return null;
const scratch = mkdtempSync(join(tmpdir(), scratchPrefix()));
const scratch = mkdtempSync(join(tmpdir(), `${scratchPrefix()}${model ? "model-" : ""}`));
let ran = null;
try {
if (model) {
plantModel(scratch);
const env = fitEnv(scratch, { view: found.view, noMcp, shape: MODEL_SHAPE, host, extensions: MODEL_EXTENSIONS, route: FIT_MODEL_ROUTE });
const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: args(scratch), input, timeout, signal: stop.signal, mounts: MODEL_MOUNTS, prefix: [node, MODEL_FILE, join(scratch, "model.json"), "--"] });
return { status: r.status, stopped: r.stopped, requests: modelRecord(scratch), turn: turnMarker(scratch) };
}
plant(scratch);
const env = fitEnv(scratch, { view: found.view, noMcp, shape, host });
const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: typeof args === "function" ? args(scratch) : args, input, timeout, signal: stop.signal });
@@ -399,8 +464,8 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c
free();
}
};
const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null } = {}) => {
const p = once(args, input, { timeout, noMcp, shape });
const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null, model = false } = {}) => {
const p = once(args, input, { timeout, noMcp, shape, model });
// Judged in order, a later run can fail before an earlier one is awaited: this keeps that
// failure from counting as unhandled. Awaiting p still throws it.
p.catch(() => {});
@@ -434,6 +499,9 @@ async function judge(found, { levels, who, digest, run }) {
if (has("--thinking")) for (const l of levels) if (!word(line("--thinking"), l)) missing.push(`--thinking ${l}`);
if (missing.length) return refuse(`lacks ${missing.join(", ")}`);
const noMcp = has("--no-mcp");
// S6: the scripted-model run is asked for first, as the longest, so it overlaps the others. It is
// judged last.
const modelRun = run((s) => sessionShape({ noMcp, extensions: MODEL_EXTENSIONS, prompt: join(s, "prompt.md"), sessionDir: join(s, "sessions"), message: "fit check", ...MODEL_SHAPE, route: FIT_MODEL_ROUTE }), "", { noMcp, model: true });
const ask = '{"id":"fit-state","type":"get_state"}\n{"id":"fit-commands","type":"get_commands"}\n';
// Every run passes the same -e and --system-prompt, so the runs differ only by the seal flags.
@@ -524,9 +592,49 @@ async function judge(found, { levels, who, digest, run }) {
if (r.stopped) return unsure(`a ${mode} start with a missing -e extension ${r.stopped}`);
if (r.status === 0 || r.answer.state || r.prompt !== null) return refuse("starts with a missing -e extension");
}
const model = judgeModel(await modelRun);
if (model) return model.undecided ? unsure(model.why) : refuse(model.why);
return { ok: true, version, noMcp, digest };
}
// S6: the model run's verdict: null for a pass, else { why, undecided }. The step results are the
// longest list any request carried. A definite failure in what did run counts first, so a run that
// leaked the outside file and then hung is refused, not left undecided.
function judgeModel(m) {
const no = (why) => ({ why, undecided: false });
const cannot = (why) => ({ why, undecided: true });
// A run that stop kept from starting.
if (!m) return cannot("the scripted-model run was stopped");
const requests = m.requests;
const results = requests.reduce((a, r) => (r.results.length > a.length ? r.results : a), []);
const blocked = [[1, "", "outside the workspace"], ...SPELLINGS.map(([, , label], i) => [i + 2, ` for a read under another spelling (${label})`, "under another spelling"])];
for (const [step, what] of blocked) {
const r = results[step];
if (r && (!r.isError || r.content.includes(SECRET))) return no(`the gate's block didn't stop the tool${what}`);
}
if (requests.some((r) => r.leak)) return no("the gate's block didn't stop the tool");
const thrown = results[THROW_STEP];
if (thrown && (!thrown.isError || thrown.content.includes(THROWN))) return no("a throw in a tool_call handler didn't stop the tool");
for (const [step, what, reason] of blocked) {
const r = results[step];
if (r && !(r.content.includes("mosaic gate: ") && r.content.includes(reason))) return no(`the gate's tool_call hook didn't decide the tool${what || " for a read outside the workspace"}`);
}
if (requests.some((r) => r.tools > 0 && !r.system.includes(GIVEN))) return no("didn't send the --system-prompt file to the model");
if (results[0] && (results[0].isError || !results[0].content.includes(INSIDE))) return no("didn't run the tool the gate allowed");
if (m.stopped) return cannot(`the scripted-model run ${m.stopped}`);
// A pi that exits non-zero before its first request most often failed to start (an extension it
// couldn't load, say). The line says so without its stderr, which can name host paths.
if (!requests.length) return cannot(m.status !== 0 ? "the scripted-model run exited non-zero before any model request" : "the scripted model got no request");
if (results.length < MODEL_SCRIPT.length || !requests.some((r) => r.done)) return cannot("the scripted-model run ended before the script's last step");
if (m.status !== 0) return cannot("the scripted-model run exited non-zero");
if (!m.turn) return no("didn't end its run with agent_end");
// The scripted model ends with text, so agent_end must carry that final assistant message: the
// gate's extension records its stopReason and whether it held text.
if (typeof m.turn.stopReason !== "string" || m.turn.hasText !== true) return no("its agent_end didn't carry the model's final answer");
return null;
}
// Whether bwrap runs here, as sandbox.mjs's sandboxProblem asks, but as child processes the caller
// awaits: the system part, the network unshared, an empty environment, then /usr/bin/true. And,
// once per check, whether unshare's outer layer runs (checkArgv in sandbox.mjs): first on its own,
@@ -568,13 +676,16 @@ async function sandboxLayers({ bwrap, unshare, node }) {
// fitEnv's. The binary starts through /bin/sh, as the adapter starts it, so it also gets the
// variables the shell sets. The run is a child process the caller awaits; nothing here blocks.
// stopped: why the run was killed ("timed out", "printed more than 4 MB", "was stopped"), or null.
function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal }) {
// mounts and prefix (S6): more read-only mounts, and a command the binary's start is passed to
// (the scripted model).
function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal, mounts = [], prefix = [] }) {
const list = [
{ kind: "ro", src: found.root, ...(found.dest ? { dest: found.dest } : {}) },
{ kind: "rw", src: scratch },
{ kind: "rw", src: join(scratch, "tmp"), dest: "/tmp" },
...mounts,
];
return spawned(checkArgv(list, ["/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal });
return spawned(checkArgv(list, [...prefix, "/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal });
}
// One child process with an empty environment, awaited: its exit status, its stdout, and why it was
@@ -645,12 +756,13 @@ function spawned(argv, { input, timeout, signal }) {
// tool list, thinking level and --mode json, which a proxy session's first turn takes with the
// model event monitor's variables), else none: no tools, no thinking, one -e. noMcp: --help lists
// --no-mcp. pi-fit.test.mjs fails when these names, or the values the code sets, differ from a bus
// session's, for a login shell's host and a unit's.
export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env }) {
// session's, for a login shell's host and a unit's. extensions and route (S6): the model run's -e
// files and its provider and model, in place of the planted ones and SESSION_SHAPE's.
export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env, extensions = null, route = SESSION_SHAPE }) {
const runner = { ...sessionEnv(withoutProviders(host), "fit"), ...sessionSetenv(scratch, { business: "fit", role: "fit" }) };
const { tools = "", thinking = "", json = false } = shape ?? {};
const session = { provider: SESSION_SHAPE.provider, model: SESSION_SHAPE.model, thinking };
const extensions = [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])];
const session = { provider: route.provider, model: route.model, thinking };
extensions ??= [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])];
const env = adapterEnv({
...session,
bundle: { prompt: join(scratch, "prompt.md"), policy: join(scratch, "policy.json"), tools: join(scratch, "tools.json") },
@@ -806,6 +918,50 @@ function plant(scratch) {
put(join(scratch, "prompt.md"), `${GIVEN}\n`);
}
// S6: the model run's scratch directory: the gate's policy for the project (read, no typed tools),
// no typed tools, the --system-prompt file, a project file the gate allows, a file outside the
// project, a link to it in the project under each spelling's name on disk, and the scripted model's
// config. The model writes the agent directory's models.json when it has its port.
function plantModel(scratch) {
const project = join(scratch, "project");
for (const d of ["marks", "tmp", "sessions", "outside", "project"]) mkdirSync(join(scratch, d), { recursive: true });
writeFileSync(join(project, "inside.txt"), `${INSIDE}\n`);
writeFileSync(join(project, THROW_NAME), `${THROWN}\n`);
const outside = join(scratch, "outside", "secret.txt");
writeFileSync(outside, `${SECRET}\n`);
for (const [, disk] of SPELLINGS) symlinkSync(outside, join(project, disk));
writeFileSync(join(scratch, "policy.json"), JSON.stringify(fitPolicy(project)));
writeFileSync(join(scratch, "tools.json"), "[]");
writeFileSync(join(scratch, "prompt.md"), `${GIVEN}\n`);
writeFileSync(join(scratch, "model.json"), JSON.stringify({
record: join(scratch, "marks", "model.json"), models: join(scratch, "home", ".pi", "agent", "models.json"),
provider: FIT_MODEL_ROUTE.provider, model: FIT_MODEL_ROUTE.model, key: "fit-model-key", script: MODEL_SCRIPT, done: DONE, secret: SECRET,
}));
}
// What the scripted model recorded, as a list of { tools, system, results: [{ isError, content }],
// done, leak }; empty when it recorded nothing that parses.
function modelRecord(scratch) {
let m;
try { m = JSON.parse(readFileSync(join(scratch, "marks", "model.json"), "utf8")); } catch { return []; }
if (!Array.isArray(m?.requests)) return [];
return m.requests.filter((r) => r && typeof r === "object").map((r) => ({
tools: Number.isInteger(r.tools) ? r.tools : 0,
system: typeof r.system === "string" ? r.system : "",
results: Array.isArray(r.results) ? r.results.map((x) => ({ isError: x?.isError === true, content: typeof x?.content === "string" ? x.content : "" })) : [],
done: r.done === true,
leak: r.leak === true,
}));
}
// The turn marker the session's -e extension wrote at agent_end, a JSON object, else null.
function turnMarker(scratch) {
try {
const m = JSON.parse(readFileSync(join(scratch, "turn"), "utf8"));
return m && typeof m === "object" && !Array.isArray(m) ? m : null;
} catch {
return null;
}
}
// A stored session for the project, as pi 0.85.1 writes one: a header whose cwd is the project's
// real path (pi's -c takes only a session for its own cwd) and one user message.
function plantSession(scratch) {
+4 -44
View File
@@ -11,6 +11,7 @@ import { join } from "node:path";
import { resolveInstance, validateBusinessDocument } from "../../organization/src/index.mjs";
import { businessDoc, REPO_ROLES, systemFor } from "../../organization/tests/helpers.mjs";
import { createLoader } from "../../plugins/src/loader.mjs";
import { messagesHandler } from "../src/pi-fit-model.mjs";
export { REPO } from "../../organization/tests/helpers.mjs";
import { REPO } from "../../organization/tests/helpers.mjs";
@@ -76,54 +77,13 @@ export async function fakeToolSocket(t, dir, handler) {
return { path, calls };
}
const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b.text ?? "").join("") : "");
function toolResults(messages) {
const out = [];
for (const m of messages ?? []) {
if (m.role !== "user" || !Array.isArray(m.content)) continue;
for (const b of m.content) if (b.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) });
}
return out;
}
// A scripted Anthropic Messages API. Each request that offers tools gets
// the next scripted tool call; once every call has a result it answers
// `done(results)` as text. Returns { url, requests, close }.
// `done(results)` as text. Returns { url, requests }. #1873 S6: the answers
// come from src/pi-fit-model.mjs, which the fit check's model run uses too.
export async function mockAnthropic(t, { script = [], done = (r) => `DONE ${JSON.stringify(r)}` } = {}) {
const requests = [];
const server = createHttp((req, res) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) {
res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" });
return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } }));
}
const body = JSON.parse(raw || "{}");
const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : [];
const results = toolResults(body.messages);
const step = tools.length ? script[results.length] : null;
const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" };
requests.push({ tools, results, system: body.system, answer: r, raw });
const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" };
const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text };
const stop = r.kind === "tool" ? "tool_use" : "end_turn";
if (!body.stream) {
res.writeHead(200, { "content-type": "application/json" });
const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text };
return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } }));
}
const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`);
res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } });
send("content_block_start", { index: 0, content_block: content });
send("content_block_delta", { index: 0, delta });
send("content_block_stop", { index: 0 });
send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } });
send("message_stop", {});
res.end();
});
});
const server = createHttp(messagesHandler({ script, done, requests }));
await new Promise((r) => server.listen(0, "127.0.0.1", r));
t.after(() => server.close());
return { url: `http://127.0.0.1:${server.address().port}`, requests };
+196
View File
@@ -16,6 +16,7 @@ import * as pi from "../../plugins/harness/pi/index.mjs";
import { BWRAP, UNSHARE, checkArgv, sandboxProblem, sessionSetenv } from "../../seat/src/sandbox.mjs";
import { ENV_ALLOW, isProviderVariable, sessionEnv } from "../../seat/src/session.mjs";
import { adapterEnv } from "../src/runner.mjs";
import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs";
const REPO = resolve(fileURLToPath(new URL("../../../", import.meta.url)));
const NAME = "@earendil-works/pi-coding-agent";
@@ -201,6 +202,12 @@ test("pi-unpin S3 re-check 3: every run has its own scratch directory, and the r
assert.deepEqual(a.slice(bin - 3, bin + 1), ["/bin/sh", "-c", 'exec "$0" "$@"', found.view]);
return { scratch, args: a.slice(bin + 1).map((x) => x.split(scratch).join("<s>")) };
});
// #1873 S6: the scripted-model run is a session-shaped -p run on the stand-in route, with the
// gate's extension, inside the scripted model's process. It is checked here and left out below.
const modelRuns = runs.filter((x) => x.args.includes(FIT_MODEL_ROUTE.provider));
assert.deepEqual(modelRuns.map((x) => x.args), [fitModule.sessionShape({ noMcp: extraFlags.length > 0, extensions: [...fitModule.MODEL_EXTENSIONS], prompt: "<s>/prompt.md", sessionDir: "<s>/sessions", message: "fit check", ...fitModule.MODEL_SHAPE, route: FIT_MODEL_ROUTE })]);
assert.equal(new Set(runs.map((x) => x.scratch)).size, runs.length, "the model run has its own scratch directory too");
runs.splice(runs.indexOf(modelRuns[0]), 1);
const seal = [...SEAL, ...extraFlags];
const loaders = [...LOADERS, ...extraFlags];
// --version, --help, the control, the sealed rpc, one per loader flag, the session-shaped runs,
@@ -1200,3 +1207,192 @@ test("pi-unpin S3 re-check 6: every argv the adapter builds in a bus session's e
assert.ok(set.has(name) || local || SHELL_SET.includes(name) || NOT_BUS.includes(name), `the adapter reads ${name} (line ${at + 1}), which no bus session environment here sets and the adapter doesn't assign first`);
}
});
// #1873 S6: the fit check runs pi once against a scripted model on the sandbox's loopback
// (pi-fit-model.mjs). The gate's tool_call hook must decide every tool run, the run must end with
// agent_end, the --system-prompt file must reach the model, and the four read spellings pi 0.85.1
// knows must go through the hook. A run that can't tell is undecided.
const MODEL_FILE = fileURLToPath(new URL("../src/pi-fit-model.mjs", import.meta.url));
// The processes of this file's model runs: each names its scratch directory, which carries
// "-model-", under tmpdir() (the model's config path, pi's --system-prompt and the sandbox layers).
const modelProcs = () => procsNaming(join(tmpdir(), "mosaic-pi-fit-")).filter((p) => {
try { return readFileSync(`/proc/${p}/cmdline`, "utf8").includes("-model-"); } catch { return false; }
});
const modelDirs = () => readdirSync(tmpdir()).filter((n) => n.startsWith("mosaic-pi-fit-") && n.includes("-model-"));
async function noModelLeft(what) {
const deadline = Date.now() + 2000;
while (modelProcs().length && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 50));
assert.deepEqual(modelProcs(), [], `a model run process outlived ${what}`);
assert.deepEqual(modelDirs(), [], `a model run's scratch outlived ${what}`);
}
test("pi-unpin S6: a stand-in pi passes the model run, and no model process or scratch is left after the pass", SKIP, async t => {
const r = await fit(standIn(t, {}).env);
assert.equal(r.ok, true, r.message);
await noModelLeft("a pass");
});
test("pi-unpin S6: a pi that runs a tool the gate's hook blocked is refused, and nothing is left", SKIP, async t => {
const s = standIn(t, { ignoreBlock: true });
const r = await fit(s.env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool");
noPath(r.message, s.paths);
await noModelLeft("a refusal");
});
test("pi-unpin S6: a pi that runs a tool when a tool_call handler throws is refused", SKIP, async t => {
const s = standIn(t, { ignoreThrow: true });
const r = await fit(s.env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) a throw in a tool_call handler didn't stop the tool");
noPath(r.message, s.paths);
await noModelLeft("a refusal");
});
test("pi-unpin S6: a pi that never emits agent_end is refused", SKIP, async t => {
const r = await fit(standIn(t, { noAgentEnd: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't end its run with agent_end");
});
test("pi-unpin S6: a pi that drops the system prompt from its model requests is refused", SKIP, async t => {
const r = await fit(standIn(t, { modelNoSystem: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't send the --system-prompt file to the model");
});
for (const [spelling, label] of [["amPm", "a narrow space before AM"], ["nfd", "decomposed accents"], ["curly", "a curly apostrophe"], ["nfdCurly", "decomposed accents and a curly apostrophe"]]) {
test(`pi-unpin S6: a pi whose read under another spelling (${spelling}) skips the gate's hook is refused`, SKIP, async t => {
const r = await fit(standIn(t, { skipHookSpelling: spelling }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool for a read under another spelling (${label})`);
});
}
// A blocked read that fails on its own, with no hook run, holds no outside word, so only the check
// that the error carries the gate's own reason refuses it.
for (const [skip, label] of [["outside", "a read outside the workspace"], ["amPm", "a read under another spelling (a narrow space before AM)"], ["nfd", "a read under another spelling (decomposed accents)"], ["curly", "a read under another spelling (a curly apostrophe)"], ["nfdCurly", "a read under another spelling (decomposed accents and a curly apostrophe)"]]) {
test(`pi-unpin S6: a pi whose read (${skip}) skips the gate's hook and fails on its own is refused`, SKIP, async t => {
const r = await fit(standIn(t, { hookSkipFail: skip }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's tool_call hook didn't decide the tool for ${label}`);
});
}
test("pi-unpin S6: a pi whose reads all fail, the allowed one too, is refused", SKIP, async t => {
const r = await fit(standIn(t, { readFails: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't run the tool the gate allowed");
});
test("pi-unpin S6: a pi whose agent_end carries no final message is refused", SKIP, async t => {
const r = await fit(standIn(t, { agentEndEmpty: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) its agent_end didn't carry the model's final answer");
});
test("pi-unpin S6: a pi that finishes the script and then exits non-zero is undecided, not a pass", SKIP, async t => {
const r = await fit(standIn(t, { exitCode: 1 }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, true);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero");
});
test("pi-unpin S6: a pi that exits non-zero before any model request is undecided, and the line says so", SKIP, async t => {
const r = await fit(standIn(t, { noModelCall: true, exitCode: 1 }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, true);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero before any model request");
});
test("pi-unpin S6: a pi that runs a blocked tool and then hangs is refused, not undecided, and nothing is left", SKIP, async t => {
const r = await fit(standIn(t, { ignoreBlock: true, modelHang: 2 }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool");
await noModelLeft("a refusal after a hang");
});
test("pi-unpin S6: a run whose scripted model gets no request is undecided, not a pass", SKIP, async t => {
const r = await fit(standIn(t, { noModelCall: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, true);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted model got no request");
});
test("pi-unpin S6: a model run that times out is undecided, and the model and every child are gone", SKIP, async t => {
const r = await fit(standIn(t, { modelHang: true }).env);
assert.equal(r.ok, false);
assert.equal(r.undecided, true);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run timed out");
await noModelLeft("a timeout");
});
test("pi-unpin S6: a check aborted while its model run runs is undecided, and the model and every child are gone", SKIP, async t => {
const s = standIn(t, { modelHang: true });
const abort = new AbortController();
const pending = piFit(locate(s.env), { levels: LEVELS, signal: abort.signal });
const deadline = Date.now() + 15000;
while (!modelProcs().some((p) => readFileSync(`/proc/${p}/cmdline`, "utf8").includes(MODEL_FILE)) && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 20));
assert.ok(modelDirs().length === 1, "the model run did not start");
// Let the stand-in reach its hang, after the model's first answer.
await new Promise((resolve) => setTimeout(resolve, 500));
abort.abort();
const r = await pending;
assert.equal(r.ok, false);
assert.equal(r.undecided, true);
await noModelLeft("an abort");
});
// A fresh copy of the module, with its own cache, so an earlier test's pass of the same pi is not
// reused. Each model run makes one "-model-" scratch directory, so a watch on tmpdir() counts them.
test("pi-unpin S6: real pi 0.85.1 passes the model run, and a second call for the same digest runs no model stand-in", SKIP, async t => {
const fresh = await import(`../src/pi-fit.mjs?s6-${process.pid}`);
const made = new Set();
const watcher = watch(tmpdir(), (event, name) => { if (name && name.startsWith("mosaic-pi-fit-") && name.includes("-model-")) made.add(name); });
t.after(() => watcher.close());
const settle = () => new Promise((resolve) => setTimeout(resolve, 200));
const found = locate({ PATH: "/usr/bin:/bin" });
const first = await fresh.piFit(found, { levels: LEVELS });
assert.equal(first.ok, true, first.message);
assert.equal(first.version, "0.85.1");
await settle();
assert.equal(made.size, 1, `the first check made ${made.size} model runs`);
made.clear();
assert.equal(await fresh.piFit(found, { levels: LEVELS }), first, "the second call reuses the pass");
await settle();
assert.equal(made.size, 0, "the second call for the same digest ran the model stand-in");
await noModelLeft("real pi's pass");
});
test("pi-unpin S6: the model run is sandboxed with no network but loopback, no provider key, and a route that names no Claude model", SKIP, async t => {
const claude = /claude|anthropic|sonnet|opus|haiku|fable/i;
for (const name of [FIT_MODEL_ROUTE.provider, FIT_MODEL_ROUTE.model]) {
assert.ok(typeof name === "string" && name.length > 0);
assert.doesNotMatch(name, claude, "the model run's route names a Claude model or provider");
}
const s = standIn(t, {});
const log = join(s.root, "bwrap.log");
const bwrap = join(s.root, "bwrap");
writeFileSync(bwrap, `#!/bin/sh\nf=$(/usr/bin/mktemp '${log}.XXXXXX')\nfor a in "$@"; do printf '%s\\037' "$a"; done > "$f"\nexec '${BWRAP}' "$@"\n`, { mode: 0o755 });
const r = await piFit(locate(s.env), { levels: LEVELS, bwrap, host: { ...HOSTS.unit, ...CREDENTIAL } });
assert.equal(r.ok, true, r.message);
const runs = readdirSync(s.root).filter((n) => n.startsWith("bwrap.log.")).map((n) => readFileSync(join(s.root, n), "utf8").split("\x1f")).filter((a) => a.includes(MODEL_FILE));
assert.equal(runs.length, 1);
const a = runs[0];
assert.ok(a.includes("--unshare-net"), "the model run shares the host's network");
assert.ok(a.includes("--clearenv"));
const setenv = a.flatMap((x, i) => (x === "--setenv" ? [a[i + 1]] : []));
for (const name of setenv) assert.equal(isProviderVariable(name), false, `the model run is given ${name}`);
assert.equal(a[a.indexOf("--provider") + 1], FIT_MODEL_ROUTE.provider);
assert.equal(a[a.indexOf("--model") + 1], FIT_MODEL_ROUTE.model);
});
+25 -5
View File
@@ -1,12 +1,16 @@
import { fixtureCacheEnv } from '../../../scripts/testing/lane-fixtures.mjs';
// Real pi (the pinned CLI) through adapters/pi with the extension, against
// the scripted Messages API. No real model, no network beyond 127.0.0.1.
// Real pi through adapters/pi with the extension, against the scripted
// Messages API. No real model, no network beyond 127.0.0.1. #1873 S6: the pi
// is the one the pi module's lookup finds (MOSAIC_PI_BIN, else the first pi on
// PATH, else the repository's), so `MOSAIC_PI_BIN=/abs/pi node --test` runs the
// suite against another pi. It must pass the fit check first.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { existsSync, mkdirSync, readdirSync, readFileSync, symlinkSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, symlinkSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { locate } from "../../plugins/harness/pi/binary.mjs";
import { buildBundle } from "../src/bundle.mjs";
import { adapterEnv, unslopCounts, unslopLog, unslopNotes } from "../src/runner.mjs";
import { REPLY_CAP } from "../src/tools.mjs";
@@ -53,7 +57,8 @@ async function session(t, script, place = (dir) => join(dir, "ws"), instance = "
s.bundle = files;
s.harnessEnv = harnessEnv;
const env = adapterEnv(s, {
PATH: `${join(REPO, "node_modules", ".bin")}:/usr/bin:/bin`,
// #1873 S6: the node running this suite, for a pi whose first line is `#!/usr/bin/env node`.
PATH: `${join(REPO, "node_modules", ".bin")}:${dirname(process.execPath)}:/usr/bin:/bin`,
HOME: join(dir, "home"),
PI_CODING_AGENT_DIR: agentDir,
});
@@ -75,6 +80,21 @@ function turn(env, request, cwd) {
});
}
// #1873 S6: MOSAIC_PI_BIN, when the suite runs with it, is the pi every session here runs.
test("pi: the session runs the pi MOSAIC_PI_BIN names, else the repository's", async (t) => {
const { api, s, env } = await session(t, []);
// The pi module's own lookup on this process's environment: MOSAIC_PI_BIN, else the first pi on
// PATH, else the repository's.
const found = locate(process.env);
assert.ok(found.bin, found.problem);
t.diagnostic(`pi under test: ${found.label}`);
assert.equal(realpathSync(env.MOSAIC_PI_BIN), realpathSync(found.bin));
const r = await turn(env, "Message 1 from jason, class REQUEST:\n\nhello", s.workspace);
assert.equal(r.code, 0, r.stderr);
assert.ok(api.requests.length > 0, "the pi it names called the scripted model");
assert.ok(existsSync(s.turnMarker));
});
test("pi: typed tools reach the socket, the gate blocks, agent_end writes the marker", async (t) => {
const { api, sock, s, env } = await session(t, [
{ name: "list_agents", input: {} },
+91 -4
View File
@@ -58,12 +58,32 @@
// as pi 0.85.1 does. `skillLeak`: with a --skill given, the planted skills load too
// whatever --no-skills says (discovery comes back with --skill). `ignoreSkill`: a
// --skill directory loads only without --no-skills.
// -p with --provider FIT_MODEL_ROUTE's provider (#1873 S6, the fit check's model run): an agent
// loop on the scripted model. It reads the provider's baseUrl from the agent
// directory's models.json, imports each -e file and calls its default export with a
// small pi (on, registerTool), then posts to <baseUrl>/v1/messages with the
// --system-prompt file's text as system and one read tool. Each tool_use runs the
// tool_call handlers first; a { block } result gives an error tool result holding the
// reason, else the read runs, trying pi 0.85.1's other spellings when the path is
// missing (a narrow space before AM or PM, NFD, a curly apostrophe, both). A text
// answer fires agent_end with the assistant message, prints the text and exits 0.
// `ignoreBlock`: the read runs whatever the handlers say. `ignoreThrow`: a
// handler's throw is dropped and the read runs. `noAgentEnd`: no agent_end.
// `modelNoSystem`: no system in the requests. `skipHookSpelling` ("amPm", "nfd",
// "curly" or "nfdCurly"): a read whose path resolves only under that spelling skips
// the handlers. `hookSkipFail` (a spelling, or "outside" for the read outside the
// project): that read skips the handlers and fails without the file's text.
// `readFails`: every read that runs fails. `modelHang`: after the first answer it
// hangs, or after answer N when it is a number. `exitCode`: it exits with this code
// after agent_end. `agentEndEmpty`: agent_end carries no messages. `noModelCall`:
// it fires agent_end and exits 0 with no request.
// otherwise a session with no $ARGV_OUT: prints `sessionOut` and exits 0.
//
// Trust, as pi 0.85.1 has it: the project is trusted with --approve, or with a saved trust and
// no --no-approve. Only the project SYSTEM.md and APPEND_SYSTEM.md wait on it here. The user
// APPEND_SYSTEM.md is appended whatever the flags, as 0.85.1 does when the project has none.
import { FIT_FLAGS } from "../src/pi-fit.mjs";
import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs";
const BODY = String.raw`
const fs = require("node:fs");
@@ -95,7 +115,72 @@ for (let i = 0; i < argv.length; i++) {
process.exit(1);
}
}
if (rpc || print) {
const modelRun = print && val("--provider") === MODEL_PROVIDER;
// The model run's agent loop (see the header).
async function agentLoop() {
const agentDir = process.env.PI_CODING_AGENT_DIR || path.join(process.env.HOME || "/", ".pi", "agent");
const handlers = { tool_call: [], agent_end: [] };
const fake = { on(name, h) { (handlers[name] = handlers[name] || []).push(h); }, registerTool() {}, getCommands() { return []; } };
for (const f of argv.flatMap((a, i) => a === "-e" ? [argv[i + 1]] : [])) {
const m = await import(require("node:url").pathToFileURL(f).href);
m.default(fake);
}
const end = async (text) => {
if (!K.noAgentEnd) for (const h of handlers.agent_end) await h({ messages: K.agentEndEmpty ? [] : [{ role: "assistant", content: [{ type: "text", text }], stopReason: "stop" }] });
process.stdout.write(text + "\n");
process.exit(K.exitCode || 0);
};
if (K.noModelCall) return end("no call");
const models = JSON.parse(fs.readFileSync(path.join(agentDir, "models.json"), "utf8"));
const base = models.providers[MODEL_PROVIDER].baseUrl;
const sysFile = val("--system-prompt");
const system = K.modelNoSystem || !sysFile ? undefined : fs.readFileSync(sysFile, "utf8");
const tools = [{ name: "read", description: "read a file", input_schema: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } }];
const messages = [{ role: "user", content: argv[argv.length - 1] }];
const variants = {
amPm: (p) => p.replace(/ (AM|PM)/g, "\u202F$1"),
nfd: (p) => p.normalize("NFD"),
curly: (p) => p.replace(/'/g, "\u2019"),
nfdCurly: (p) => p.replace(/'/g, "\u2019").normalize("NFD"),
};
const resolveRead = (p) => {
const abs = path.resolve(process.cwd(), p);
if (fs.existsSync(abs)) return { file: abs, via: null };
for (const [name, v] of Object.entries(variants)) if (fs.existsSync(v(abs))) return { file: v(abs), via: name };
return { file: abs, via: null };
};
for (let turn = 0; ; turn++) {
const res = await fetch(base + "/v1/messages", { method: "POST", headers: { "content-type": "application/json", "x-api-key": "stand-in" }, body: JSON.stringify({ model: val("--model"), max_tokens: 100, stream: false, ...(system === undefined ? {} : { system }), tools, messages }) });
const reply = await res.json();
if (K.modelHang === true ? turn === 0 : K.modelHang === turn) { setInterval(() => {}, 1000); return; }
const use = (reply.content || []).find((c) => c.type === "tool_use");
if (!use) return end((reply.content || []).map((c) => c.text || "").join(""));
messages.push({ role: "assistant", content: reply.content });
const { file, via } = resolveRead(use.input.path);
let blocked = null;
let threw = null;
const skipFail = K.hookSkipFail !== undefined && (K.hookSkipFail === "outside" ? via === null && use.input.path.includes("outside") : via === K.hookSkipFail);
if (!(K.skipHookSpelling && via === K.skipHookSpelling) && !skipFail) {
for (const h of handlers.tool_call) {
let r;
try { r = await h({ toolName: use.name, toolCallId: use.id, input: use.input }); } catch (e) { threw = e.message; break; }
if (r && r.block && !blocked) blocked = r.reason || "blocked";
}
}
let result;
if (skipFail || (K.readFails && threw === null && !blocked)) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: EACCES" };
else if (threw !== null && !K.ignoreThrow) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: threw };
else if (blocked && !K.ignoreBlock) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: blocked };
else {
try { result = { type: "tool_result", tool_use_id: use.id, content: fs.readFileSync(file, "utf8") }; }
catch (e) { result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: " + e.code }; }
}
messages.push({ role: "user", content: [result] });
}
}
if (modelRun) {
agentLoop().catch((e) => { process.stderr.write("stand-in model run: " + e.message + "\n"); process.exit(1); });
} else if (rpc || print) {
if (K.flood) {
const chunk = Buffer.alloc(1024 * 1024, "x");
for (let n = 0; n < 8; n++) for (let off = 0; off < chunk.length;) {
@@ -225,9 +310,10 @@ if (rpc || print) {
}
}
process.exit(0);
} else {
if (K.sessionOut) process.stdout.write(K.sessionOut);
process.exit(0);
}
if (K.sessionOut) process.stdout.write(K.sessionOut);
process.exit(0);
`;
const quote = (text) => `'${String(text).replace(/'/g, "'\\''")}'`;
@@ -248,13 +334,14 @@ export function piStandIn(knobs = {}, { node = process.execPath, form = "sh" } =
`if (process.env.ARGV_OUT) { require("node:fs").writeFileSync(process.env.ARGV_OUT, process.argv.slice(1).map((a) => a + "\\n").join("")); process.stdout.write(${JSON.stringify(String(knobs.sessionOut ?? ""))}); process.exit(0); }`,
`const K = ${JSON.stringify(knobs)};`,
`const FLAGS = ${JSON.stringify(FIT_FLAGS)};`,
`const MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};`,
BODY,
`})();`,
"",
].join("\n");
}
const version = knobs.versionHang ? "sleep 60; exit 0" : knobs.versionExit != null ? `exit ${Number(knobs.versionExit)}` : `printf '%s\\n' ${quote(knobs.version ?? "9.9.9")}; exit 0`;
const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\n${BODY}`;
const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\nconst MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};\n${BODY}`;
return [
"#!/bin/sh",
`[ "$1" = --version ] && { ${version}; }`,
@@ -32,3 +32,14 @@ export const envNames = Object.freeze(["MOSAIC_PI_BIN", "MOSAIC_PI_NO_MCP", "PI_
// no Anthropic provider (pi-fit.test.mjs holds that); glm-5.3-flash is the model index.mjs has
// documented limits for, and pi's own catalogue lists it under zai.
export const FIT_ROUTE = Object.freeze({ provider: "zai", model: "glm-5.3-flash" });
// #1873 pi-unpin S6: the provider and model the fit check's scripted-model run names. Its models.json
// points that provider at a stand-in on the sandbox's own loopback (packages/harness/src/
// pi-fit-model.mjs), so the names are plain stand-ins: no real provider, and no Claude model, which
// runs only on the claude-code harness (pi-fit.test.mjs holds that).
export const FIT_MODEL_ROUTE = Object.freeze({ provider: "fit-model", model: "fit-model" });
// #1873 pi-unpin S6: the gate policy the scripted-model run's -e extension (packages/harness/src/
// pi-extension.mjs) reads: a pi session with the read tool, no typed tools, in workspace. Built here
// since the policy names this harness.
export const fitPolicy = (workspace) => ({ harness: "pi", workspace, tools: ["read"], typed: [] });