diff --git a/packages/harness/README.md b/packages/harness/README.md index 6ca7e0291..ff7656534 100644 --- a/packages/harness/README.md +++ b/packages/harness/README.md @@ -20,17 +20,17 @@ bundle's `manifest.json` names the lines it relies on (`reliesOn`): | | Pi | Claude Code | How pi's line is checked | |---|---|---|---| -| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | suites, on the repository's pi (`pi-session.test.mjs`) | -| 2. gate crash | a throw in the handler blocks | the hook runs as ` \|\| exit 2` | suites, on the repository's pi | +| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | at first sight of a binary, by the fit check's model run, on the pi that runs; and suites (`pi-session.test.mjs`) | +| 2. gate crash | a throw in the handler blocks | the hook runs as ` \|\| exit 2` | at first sight of a binary, by the fit check's model run; and suites | | 3. gate missing | a missing `-e` file refuses to start | the wrapper blocks | at every start, by the fit check, on the pi that runs; and suites | -| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 \|\| exit 2`, hook timeout 20 | suites, on the repository's pi | +| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 \|\| exit 2`, hook timeout 20 | the `agent_end` marker: at first sight of a binary, by the fit check's model run; and suites | | 5. bash | limited only by the tool limit | limited only by the tool limit | suites, on the repository's pi | Pi is not pinned (#1873 pi-unpin). Before a session starts, `preflight` awaits the fit check in `src/pi-fit.mjs` on the binary the module found (the Triage host and route await the module's `ready` instead), and `env` reads its pass. It runs that binary in a sandbox with the network -unshared and calls no model. Each run is a child process the check +unshared and calls no real model. Each run is a child process the check awaits, up to four at a time, so the bus's event loop keeps running while it does; callers that ask while a check runs share it. Each run gets the environment a bus session's pi gets, built by the same code @@ -131,9 +131,35 @@ for example `pi 2.0.0 (MOSAIC_PI_BIN) lacks --no-context-files`. When `--help` lists `--no-mcp`, the adapter passes it too. A pass is cached in memory by a digest of the package's files (and npm's lock files, for the repository's copy) and the names in the runs' environment, so a bus -restarted under other names checks again. Lines 1, 2, 4 and 5 are not -checked at start; a pi other than the repository's copy relies on them -unproved until a scripted-model fit run exists (design section 5, S6). +restarted under other names checks again. + +One more run (#1873 S6) talks to a scripted model, so lines 1, 2 and 4 +and the system prompt are checked on the pi that runs. It is a +session-shaped `-p` run with `--tools read`, the gate's `-e` extension +and a second one, `src/pi-fit-throw.mjs`, on a stand-in provider and +model named `fit-model` (never a Claude model). The model is +`src/pi-fit-model.mjs`: inside the sandbox it is the first process, it +listens on the sandbox's own loopback (the network stays unshared), +writes the agent directory's `models.json` for its port, starts pi and +answers one scripted tool call per request. The script reads a project +file the gate allows, a file outside the project, the same outside file +through a link under each of the four spellings pi 0.85.1 tries for a +missing path (a narrow space before AM, decomposed accents, a curly +apostrophe, both), and a project file the second extension's handler +throws for. The run passes only when the allowed read returns the file, +every blocked read is a tool error carrying the gate's own reason and +no request carries the outside file's word, the throw stops its read, +every request carries the `--system-prompt` file, the script runs to +its end, and the gate's `agent_end` hook writes the turn marker with the +model's final text answer. A pi that runs a blocked tool, swallows a +handler's throw, drops the system prompt or never emits `agent_end` is +refused. A run that times out, exits non-zero, or whose model gets no +request or sees the run end early, can't be checked and refuses as +undecided. A run that leaked and then hung is refused, not undecided. Its verdict is part of the cached pass, so it +runs once per digest. It runs beside the other runs, so a cold check +takes about as long as before. Line +5 (bash) is not checked at start, and nor is a read spelling pi may add +later; a stand-in can only check what it scripts. Claude Code also runs with `--restricted`: no user, project or local settings, file tools confined to the working directory, and no `CLAUDE.md` diff --git a/packages/harness/src/pi-fit-model.mjs b/packages/harness/src/pi-fit-model.mjs new file mode 100644 index 000000000..15eb57384 --- /dev/null +++ b/packages/harness/src/pi-fit-model.mjs @@ -0,0 +1,111 @@ +// #1873 pi-unpin S6: the scripted model the fit check's model run talks to (pi-fit.mjs). It +// answers the Messages API one scripted step at a time: step k is a tool call, given when the +// request carries k tool results; after the last step it answers with fixed text. Only node's own +// modules are imported, since the file runs inside the fit check's sandbox. +// +// Run as a program, inside the sandbox: `node pi-fit-model.mjs -- `. It +// listens on 127.0.0.1 port 0 in the sandbox's own network namespace (checkArgv unshares the +// network, so only that namespace's loopback reaches it), writes the agent directory's models.json +// naming that port for the stand-in provider, starts the command with its own environment and +// stdio, writes what each request held to the record file after each request, and exits with the +// command's status. It is the first process in the sandbox's pid namespace, so when it ends, or is +// killed, every process in the sandbox ends with it. No real model and no host network are +// reached: the sandbox has no other network. +// +// packages/harness/tests/helpers.mjs mockAnthropic serves the same answers to session tests. +import { createServer } from "node:http"; +import { spawn } from "node:child_process"; +import { mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +export const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b?.text ?? "").join("") : ""); + +// The tool results in a request's messages, in order: { id, isError, content }. +export function toolResults(messages) { + const out = []; + for (const m of messages ?? []) { + if (m?.role !== "user" || !Array.isArray(m.content)) continue; + for (const b of m.content) if (b?.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) }); + } + return out; +} + +// A request handler for the Messages API. script: [{ name, input }], one tool call per step. +// done(results): the text that ends the run once the script is spent. requests: each request is +// pushed as { tools, results, system, answer, raw }. onRequest: called after each push. A request +// with no tools gets "mock". Streaming and plain answers both work; count_tokens answers 10. +export function messagesHandler({ script = [], done = (r) => `DONE ${JSON.stringify(r)}`, requests = [], onRequest = () => {} } = {}) { + return (req, res) => { + let raw = ""; + req.on("data", (c) => (raw += c)); + req.on("end", () => { + if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) { + res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" }); + return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } })); + } + let body; + try { body = JSON.parse(raw || "{}"); } catch { body = {}; } + const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : []; + const results = toolResults(body.messages); + const step = tools.length ? script[results.length] : null; + const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" }; + requests.push({ tools, results, system: body.system, answer: r, raw }); + onRequest(); + const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" }; + const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text }; + const stop = r.kind === "tool" ? "tool_use" : "end_turn"; + if (!body.stream) { + res.writeHead(200, { "content-type": "application/json" }); + const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text }; + return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } })); + } + const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`); + res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" }); + send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } }); + send("content_block_start", { index: 0, content_block: content }); + send("content_block_delta", { index: 0, delta }); + send("content_block_stop", { index: 0 }); + send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } }); + send("message_stop", {}); + res.end(); + }); + }; +} + +// How much of each tool result the record keeps. +const KEEP = 2000; + +// config: { record, models, provider, model, key, script, done, secret }. record: the file each +// request is written to, as { requests: [{ tools, system, results: [{ isError, content }], +// done, leak }] }: tools is the count of tools offered, done whether the answer was the done text, +// leak whether the raw request held the secret word. +async function main(argv) { + const [config, sep, ...command] = argv; + if (!config || sep !== "--" || !command.length) process.exit(2); + const c = JSON.parse(readFileSync(config, "utf8")); + const requests = []; + const record = () => writeFileSync(c.record, JSON.stringify({ requests: requests.map((r) => ({ + tools: r.tools.length, + system: textOf(r.system), + results: r.results.map((x) => ({ isError: Boolean(x.isError), content: String(x.content).slice(0, KEEP) })), + done: r.answer.kind === "text" && r.answer.text === c.done, + leak: r.raw.includes(c.secret), + })) })); + const server = createServer(messagesHandler({ script: c.script, done: () => c.done, requests, onRequest: record })); + await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); }); + mkdirSync(dirname(c.models), { recursive: true }); + writeFileSync(c.models, JSON.stringify({ providers: { [c.provider]: { baseUrl: `http://127.0.0.1:${server.address().port}`, api: "anthropic-messages", apiKey: c.key, models: [{ id: c.model }] } } })); + const child = spawn(command[0], command.slice(1), { stdio: "inherit", env: process.env }); + const status = await new Promise((resolve) => { + child.on("error", () => resolve(1)); + child.on("exit", (code, signal) => resolve(code ?? (signal ? 1 : 0))); + }); + server.close(); + process.exit(status); +} + +const self = fileURLToPath(import.meta.url); +let invoked = null; +try { invoked = process.argv[1] ? realpathSync(process.argv[1]) : null; } catch {} +if (invoked === self) main(process.argv.slice(2)).catch(() => process.exit(1)); diff --git a/packages/harness/src/pi-fit-throw.mjs b/packages/harness/src/pi-fit-throw.mjs new file mode 100644 index 000000000..94b823fb8 --- /dev/null +++ b/packages/harness/src/pi-fit-throw.mjs @@ -0,0 +1,14 @@ +// #1873 pi-unpin S6: a second -e extension, for the fit check's model run only (pi-fit.mjs); no +// session loads it. pi loads it after the gate's, so for a read of THROW_NAME the gate has allowed +// the read before this handler throws. The run then shows that a throw in a tool_call handler +// stops the tool (S0 line 2: a gate that crashes blocks). It imports nothing, since it runs inside +// the fit check's sandbox. +export const THROW_NAME = "throw.txt"; +export const THROW_TEXT = "fit check: the tool_call handler threw"; + +export default function fitThrow(host) { + host.on("tool_call", async (event) => { + if (event?.toolName === "read" && event.input?.path === THROW_NAME) throw new Error(THROW_TEXT); + return undefined; + }); +} diff --git a/packages/harness/src/pi-fit.mjs b/packages/harness/src/pi-fit.mjs index a4f32c22c..4990d1023 100644 --- a/packages/harness/src/pi-fit.mjs +++ b/packages/harness/src/pi-fit.mjs @@ -2,7 +2,8 @@ // here compares a version. The check runs the binary only inside a sandbox with the network // unshared, the found package mounted as a session mounts it, and a bus session's environment // (fitEnv below), so its agent directory is $HOME/.pi/agent under a scratch HOME, as a session's -// is. No model is called and no provider key is passed. It asks: +// is. No real model is called and no provider key is passed: the one run that reaches a model +// reaches a scripted stand-in on the sandbox's own loopback. It asks: // // --version a label for the refusal line and the record; not a gate // --help lists, in its option column, every flag Mosaic passes, with rpc mode and each @@ -56,6 +57,23 @@ // its own any more. // missing -e a missing -e file, in rpc and in -p, stops the start: exit non-zero, no answer, // and no session_start for the -e extension given after it +// model #1873 S6: one -p run in a bus session's shape (sessionShape, with --tools read and +// the session's own -e extension, pi-extension.mjs, which gates every tool call with +// gate.mjs and writes the turn marker at agent_end) against a scripted model: +// pi-fit-model.mjs, started inside the sandbox ahead of the binary, listening on the +// sandbox's loopback and named by the agent directory's models.json under a stand-in +// provider and model (session-env.mjs FIT_MODEL_ROUTE). The script reads a file in +// the project (the gate allows it, and its word must come back), then a file outside +// it, then four names that pi 0.85.1 resolves to another spelling (a narrow space +// before AM, decomposed accents, a curly apostrophe, both of the last two), each a +// link out of the project to the same outside file. Each of those four the gate +// blocks, and each must come back as an error holding the gate's reason, with the +// outside file's word in no request. Every request with tools must carry the +// --system-prompt file's word, the run must reach the script's end and exit 0, and +// the turn marker must be written. A tool result that misses the gate's reason, a +// request without the word, or no marker is a refusal. A run that times out, ends +// early or sends no request can't tell. pi-fit-model.mjs is the sandbox's first +// process, so the model and every child end with it. // // Every run gets a fresh scratch directory with the sources planted again, so no run sees what // another one left. The directory's name carries the pid, start time and pid namespace of the @@ -100,17 +118,20 @@ // whenever the directory isn't empty), since one more run costs about 340 ms cold; and a // session's own values: its tool names beyond read and bash, thinking levels beyond low, // provider, model, key, -e files and request. The runs are offline with no key, so nothing -// past the start is tried. A pi built to act differently with those is outside what a start-time -// check can catch. +// past the start is tried, but for the model run. A pi built to act differently with those, or to +// act differently in a fit check, is outside what a start-time check can catch. The model run +// tries read alone, one tool call per turn, and only the four spellings pi 0.85.1 knows. import { spawn } from "node:child_process"; import { createHash } from "node:crypto"; -import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join, relative } from "node:path"; -import { BWRAP, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs"; +import { fileURLToPath } from "node:url"; +import { BWRAP, RUNNER_FILES, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs"; import { isProviderVariable, sessionEnv } from "../../seat/src/session.mjs"; import { adapterEnv } from "./runner.mjs"; -import { FIT_ROUTE, harnessEnv } from "../../plugins/harness/pi/session-env.mjs"; +import { THROW_NAME } from "./pi-fit-throw.mjs"; +import { FIT_MODEL_ROUTE, FIT_ROUTE, fitPolicy, harnessEnv } from "../../plugins/harness/pi/session-env.mjs"; // The flags adapters/pi/adapter.sh and the conversation engine pass. export const FIT_FLAGS = Object.freeze(["--mode", "--session", "--no-session", "--session-dir", "--fork", "-c", "--no-extensions", "-e", "--no-skills", "--skill", "--no-prompt-templates", "--no-themes", "--no-context-files", "--no-approve", "--offline", "--tools", "--no-tools", "--provider", "--model", "--thinking", "--system-prompt", "-p"]); @@ -139,13 +160,14 @@ export const SESSION_RUNS = Object.freeze([ // or none, a persistent session directory with -c when it holds a session, a thinking level or // none, the --system-prompt file, -p and the request, and --mode json when the model event monitor // runs pi. No --skill and no --fork: a bus session never passes them. Keep its order: -// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ. -export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message }) { +// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ. route: the +// provider and model, SESSION_SHAPE's unless the model run passes FIT_MODEL_ROUTE. +export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message, route = SESSION_SHAPE }) { return [ "--offline", "--no-approve", "--no-extensions", ...(noMcp ? ["--no-mcp"] : []), ...extensions.flatMap((e) => ["-e", e]), "--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files", ...(tools ? ["--tools", tools] : ["--no-tools"]), "--session-dir", sessionDir, ...(cont ? ["-c"] : []), - "--provider", SESSION_SHAPE.provider, "--model", SESSION_SHAPE.model, ...(thinking ? ["--thinking", thinking] : []), + "--provider", route.provider, "--model", route.model, ...(thinking ? ["--thinking", thinking] : []), "--system-prompt", prompt, "-p", message, ...(json ? ["--mode", "json"] : []), ]; } @@ -165,6 +187,41 @@ export function workerShape({ noMcp, extensions, skills, prompt, message }) { // What a session-shaped run is named by in a refusal line. const shapeName = ({ tools, cont, thinking, json }) => `in a session's shape with ${[tools ? "--tools" : "--no-tools", ...(cont ? ["-c"] : []), ...(thinking ? ["--thinking"] : []), ...(json ? ["--mode json"] : [])].join(", ")}`; const RUN_TIMEOUT = 20000; + +// #1873 S6: the model run. Its first -e extension is the gate's, which a session loads; the second +// is pi-fit-throw.mjs, whose handler throws for one read the gate allows. The model is +// pi-fit-model.mjs. All are mounted read-only where they are, with the runner files the gate's +// extension imports, as a session mounts them (sandbox.mjs binds). +const EXTENSION = fileURLToPath(new URL("./pi-extension.mjs", import.meta.url)); +const THROW_EXTENSION = fileURLToPath(new URL("./pi-fit-throw.mjs", import.meta.url)); +export const MODEL_EXTENSIONS = Object.freeze([EXTENSION, THROW_EXTENSION]); +const MODEL_FILE = fileURLToPath(new URL("./pi-fit-model.mjs", import.meta.url)); +const MODEL_MOUNTS = [MODEL_FILE, THROW_EXTENSION, ...RUNNER_FILES].map((src) => ({ kind: "ro", src })); +export const MODEL_SHAPE = Object.freeze({ tools: "read", cont: false, thinking: "", json: false }); +// The project file's word, which the allowed read must bring back; the outside file's, which must +// reach no request; the scripted model's last answer. +const INSIDE = "FITINSIDEWORD"; +const SECRET = "FITSECRETWORD"; +// The word in the project file the throwing handler's read names, which must reach no request. +const THROWN = "FITTHROWNWORD"; +const DONE = "FITDONE"; +// The four spellings pi 0.85.1 tries when a read's path is missing: the name the model asks for, +// the name on disk (a link to the outside file) and what a refusal line calls it. +const SPELLINGS = [ + ["shot 9.41 AM.png", "shot 9.41\u202FAM.png", "a narrow space before AM"], + ["r\u00E9sum\u00E9.txt", "r\u00E9sum\u00E9.txt".normalize("NFD"), "decomposed accents"], + ["notes's.txt", "notes\u2019s.txt", "a curly apostrophe"], + ["d'\u00E9cran.png", "d\u2019\u00E9cran.png".normalize("NFD"), "decomposed accents and a curly apostrophe"], +]; +// The script: the allowed read, the read outside the project, one read per spelling, then the read +// the gate allows and the second handler throws for. +export const MODEL_SCRIPT = Object.freeze([ + { name: "read", input: { path: "inside.txt" } }, + { name: "read", input: { path: "../outside/secret.txt" } }, + ...SPELLINGS.map(([asked]) => ({ name: "read", input: { path: asked } })), + { name: "read", input: { path: THROW_NAME } }, +].map((step) => Object.freeze(step))); +const THROW_STEP = MODEL_SCRIPT.length - 1; // --version gives only a label, so a hang costs this much and leaves no label. const VERSION_TIMEOUT = 3000; const OUTPUT_MAX = 4 * 1024 * 1024; @@ -374,13 +431,21 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c // get_commands answer or through the -e extension's command list (listed: through that alone). // shape: a session-shaped run's branches, which set its tools, thinking level and monitor // variables as a session's would. A run that stop kept from starting gives null. - const once = async (args, input, { timeout, noMcp, shape }) => { + // model (S6): the scripted-model run, with its own plant, mounts and environment; it gives the + // run's status and stopped with the requests the model recorded and the turn marker. + const once = async (args, input, { timeout, noMcp, shape, model }) => { await slot(); try { if (stop.signal.aborted) return null; - const scratch = mkdtempSync(join(tmpdir(), scratchPrefix())); + const scratch = mkdtempSync(join(tmpdir(), `${scratchPrefix()}${model ? "model-" : ""}`)); let ran = null; try { + if (model) { + plantModel(scratch); + const env = fitEnv(scratch, { view: found.view, noMcp, shape: MODEL_SHAPE, host, extensions: MODEL_EXTENSIONS, route: FIT_MODEL_ROUTE }); + const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: args(scratch), input, timeout, signal: stop.signal, mounts: MODEL_MOUNTS, prefix: [node, MODEL_FILE, join(scratch, "model.json"), "--"] }); + return { status: r.status, stopped: r.stopped, requests: modelRecord(scratch), turn: turnMarker(scratch) }; + } plant(scratch); const env = fitEnv(scratch, { view: found.view, noMcp, shape, host }); const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: typeof args === "function" ? args(scratch) : args, input, timeout, signal: stop.signal }); @@ -399,8 +464,8 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c free(); } }; - const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null } = {}) => { - const p = once(args, input, { timeout, noMcp, shape }); + const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null, model = false } = {}) => { + const p = once(args, input, { timeout, noMcp, shape, model }); // Judged in order, a later run can fail before an earlier one is awaited: this keeps that // failure from counting as unhandled. Awaiting p still throws it. p.catch(() => {}); @@ -434,6 +499,9 @@ async function judge(found, { levels, who, digest, run }) { if (has("--thinking")) for (const l of levels) if (!word(line("--thinking"), l)) missing.push(`--thinking ${l}`); if (missing.length) return refuse(`lacks ${missing.join(", ")}`); const noMcp = has("--no-mcp"); + // S6: the scripted-model run is asked for first, as the longest, so it overlaps the others. It is + // judged last. + const modelRun = run((s) => sessionShape({ noMcp, extensions: MODEL_EXTENSIONS, prompt: join(s, "prompt.md"), sessionDir: join(s, "sessions"), message: "fit check", ...MODEL_SHAPE, route: FIT_MODEL_ROUTE }), "", { noMcp, model: true }); const ask = '{"id":"fit-state","type":"get_state"}\n{"id":"fit-commands","type":"get_commands"}\n'; // Every run passes the same -e and --system-prompt, so the runs differ only by the seal flags. @@ -524,9 +592,49 @@ async function judge(found, { levels, who, digest, run }) { if (r.stopped) return unsure(`a ${mode} start with a missing -e extension ${r.stopped}`); if (r.status === 0 || r.answer.state || r.prompt !== null) return refuse("starts with a missing -e extension"); } + + const model = judgeModel(await modelRun); + if (model) return model.undecided ? unsure(model.why) : refuse(model.why); return { ok: true, version, noMcp, digest }; } +// S6: the model run's verdict: null for a pass, else { why, undecided }. The step results are the +// longest list any request carried. A definite failure in what did run counts first, so a run that +// leaked the outside file and then hung is refused, not left undecided. +function judgeModel(m) { + const no = (why) => ({ why, undecided: false }); + const cannot = (why) => ({ why, undecided: true }); + // A run that stop kept from starting. + if (!m) return cannot("the scripted-model run was stopped"); + const requests = m.requests; + const results = requests.reduce((a, r) => (r.results.length > a.length ? r.results : a), []); + const blocked = [[1, "", "outside the workspace"], ...SPELLINGS.map(([, , label], i) => [i + 2, ` for a read under another spelling (${label})`, "under another spelling"])]; + for (const [step, what] of blocked) { + const r = results[step]; + if (r && (!r.isError || r.content.includes(SECRET))) return no(`the gate's block didn't stop the tool${what}`); + } + if (requests.some((r) => r.leak)) return no("the gate's block didn't stop the tool"); + const thrown = results[THROW_STEP]; + if (thrown && (!thrown.isError || thrown.content.includes(THROWN))) return no("a throw in a tool_call handler didn't stop the tool"); + for (const [step, what, reason] of blocked) { + const r = results[step]; + if (r && !(r.content.includes("mosaic gate: ") && r.content.includes(reason))) return no(`the gate's tool_call hook didn't decide the tool${what || " for a read outside the workspace"}`); + } + if (requests.some((r) => r.tools > 0 && !r.system.includes(GIVEN))) return no("didn't send the --system-prompt file to the model"); + if (results[0] && (results[0].isError || !results[0].content.includes(INSIDE))) return no("didn't run the tool the gate allowed"); + if (m.stopped) return cannot(`the scripted-model run ${m.stopped}`); + // A pi that exits non-zero before its first request most often failed to start (an extension it + // couldn't load, say). The line says so without its stderr, which can name host paths. + if (!requests.length) return cannot(m.status !== 0 ? "the scripted-model run exited non-zero before any model request" : "the scripted model got no request"); + if (results.length < MODEL_SCRIPT.length || !requests.some((r) => r.done)) return cannot("the scripted-model run ended before the script's last step"); + if (m.status !== 0) return cannot("the scripted-model run exited non-zero"); + if (!m.turn) return no("didn't end its run with agent_end"); + // The scripted model ends with text, so agent_end must carry that final assistant message: the + // gate's extension records its stopReason and whether it held text. + if (typeof m.turn.stopReason !== "string" || m.turn.hasText !== true) return no("its agent_end didn't carry the model's final answer"); + return null; +} + // Whether bwrap runs here, as sandbox.mjs's sandboxProblem asks, but as child processes the caller // awaits: the system part, the network unshared, an empty environment, then /usr/bin/true. And, // once per check, whether unshare's outer layer runs (checkArgv in sandbox.mjs): first on its own, @@ -568,13 +676,16 @@ async function sandboxLayers({ bwrap, unshare, node }) { // fitEnv's. The binary starts through /bin/sh, as the adapter starts it, so it also gets the // variables the shell sets. The run is a child process the caller awaits; nothing here blocks. // stopped: why the run was killed ("timed out", "printed more than 4 MB", "was stopped"), or null. -function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal }) { +// mounts and prefix (S6): more read-only mounts, and a command the binary's start is passed to +// (the scripted model). +function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal, mounts = [], prefix = [] }) { const list = [ { kind: "ro", src: found.root, ...(found.dest ? { dest: found.dest } : {}) }, { kind: "rw", src: scratch }, { kind: "rw", src: join(scratch, "tmp"), dest: "/tmp" }, + ...mounts, ]; - return spawned(checkArgv(list, ["/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal }); + return spawned(checkArgv(list, [...prefix, "/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal }); } // One child process with an empty environment, awaited: its exit status, its stdout, and why it was @@ -645,12 +756,13 @@ function spawned(argv, { input, timeout, signal }) { // tool list, thinking level and --mode json, which a proxy session's first turn takes with the // model event monitor's variables), else none: no tools, no thinking, one -e. noMcp: --help lists // --no-mcp. pi-fit.test.mjs fails when these names, or the values the code sets, differ from a bus -// session's, for a login shell's host and a unit's. -export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env }) { +// session's, for a login shell's host and a unit's. extensions and route (S6): the model run's -e +// files and its provider and model, in place of the planted ones and SESSION_SHAPE's. +export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env, extensions = null, route = SESSION_SHAPE }) { const runner = { ...sessionEnv(withoutProviders(host), "fit"), ...sessionSetenv(scratch, { business: "fit", role: "fit" }) }; const { tools = "", thinking = "", json = false } = shape ?? {}; - const session = { provider: SESSION_SHAPE.provider, model: SESSION_SHAPE.model, thinking }; - const extensions = [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])]; + const session = { provider: route.provider, model: route.model, thinking }; + extensions ??= [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])]; const env = adapterEnv({ ...session, bundle: { prompt: join(scratch, "prompt.md"), policy: join(scratch, "policy.json"), tools: join(scratch, "tools.json") }, @@ -806,6 +918,50 @@ function plant(scratch) { put(join(scratch, "prompt.md"), `${GIVEN}\n`); } +// S6: the model run's scratch directory: the gate's policy for the project (read, no typed tools), +// no typed tools, the --system-prompt file, a project file the gate allows, a file outside the +// project, a link to it in the project under each spelling's name on disk, and the scripted model's +// config. The model writes the agent directory's models.json when it has its port. +function plantModel(scratch) { + const project = join(scratch, "project"); + for (const d of ["marks", "tmp", "sessions", "outside", "project"]) mkdirSync(join(scratch, d), { recursive: true }); + writeFileSync(join(project, "inside.txt"), `${INSIDE}\n`); + writeFileSync(join(project, THROW_NAME), `${THROWN}\n`); + const outside = join(scratch, "outside", "secret.txt"); + writeFileSync(outside, `${SECRET}\n`); + for (const [, disk] of SPELLINGS) symlinkSync(outside, join(project, disk)); + writeFileSync(join(scratch, "policy.json"), JSON.stringify(fitPolicy(project))); + writeFileSync(join(scratch, "tools.json"), "[]"); + writeFileSync(join(scratch, "prompt.md"), `${GIVEN}\n`); + writeFileSync(join(scratch, "model.json"), JSON.stringify({ + record: join(scratch, "marks", "model.json"), models: join(scratch, "home", ".pi", "agent", "models.json"), + provider: FIT_MODEL_ROUTE.provider, model: FIT_MODEL_ROUTE.model, key: "fit-model-key", script: MODEL_SCRIPT, done: DONE, secret: SECRET, + })); +} +// What the scripted model recorded, as a list of { tools, system, results: [{ isError, content }], +// done, leak }; empty when it recorded nothing that parses. +function modelRecord(scratch) { + let m; + try { m = JSON.parse(readFileSync(join(scratch, "marks", "model.json"), "utf8")); } catch { return []; } + if (!Array.isArray(m?.requests)) return []; + return m.requests.filter((r) => r && typeof r === "object").map((r) => ({ + tools: Number.isInteger(r.tools) ? r.tools : 0, + system: typeof r.system === "string" ? r.system : "", + results: Array.isArray(r.results) ? r.results.map((x) => ({ isError: x?.isError === true, content: typeof x?.content === "string" ? x.content : "" })) : [], + done: r.done === true, + leak: r.leak === true, + })); +} +// The turn marker the session's -e extension wrote at agent_end, a JSON object, else null. +function turnMarker(scratch) { + try { + const m = JSON.parse(readFileSync(join(scratch, "turn"), "utf8")); + return m && typeof m === "object" && !Array.isArray(m) ? m : null; + } catch { + return null; + } +} + // A stored session for the project, as pi 0.85.1 writes one: a header whose cwd is the project's // real path (pi's -c takes only a session for its own cwd) and one user message. function plantSession(scratch) { diff --git a/packages/harness/tests/helpers.mjs b/packages/harness/tests/helpers.mjs index c34b1b1f9..ba78242f7 100644 --- a/packages/harness/tests/helpers.mjs +++ b/packages/harness/tests/helpers.mjs @@ -11,6 +11,7 @@ import { join } from "node:path"; import { resolveInstance, validateBusinessDocument } from "../../organization/src/index.mjs"; import { businessDoc, REPO_ROLES, systemFor } from "../../organization/tests/helpers.mjs"; import { createLoader } from "../../plugins/src/loader.mjs"; +import { messagesHandler } from "../src/pi-fit-model.mjs"; export { REPO } from "../../organization/tests/helpers.mjs"; import { REPO } from "../../organization/tests/helpers.mjs"; @@ -76,54 +77,13 @@ export async function fakeToolSocket(t, dir, handler) { return { path, calls }; } -const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b.text ?? "").join("") : ""); -function toolResults(messages) { - const out = []; - for (const m of messages ?? []) { - if (m.role !== "user" || !Array.isArray(m.content)) continue; - for (const b of m.content) if (b.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) }); - } - return out; -} - // A scripted Anthropic Messages API. Each request that offers tools gets // the next scripted tool call; once every call has a result it answers -// `done(results)` as text. Returns { url, requests, close }. +// `done(results)` as text. Returns { url, requests }. #1873 S6: the answers +// come from src/pi-fit-model.mjs, which the fit check's model run uses too. export async function mockAnthropic(t, { script = [], done = (r) => `DONE ${JSON.stringify(r)}` } = {}) { const requests = []; - const server = createHttp((req, res) => { - let raw = ""; - req.on("data", (c) => (raw += c)); - req.on("end", () => { - if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) { - res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" }); - return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } })); - } - const body = JSON.parse(raw || "{}"); - const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : []; - const results = toolResults(body.messages); - const step = tools.length ? script[results.length] : null; - const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" }; - requests.push({ tools, results, system: body.system, answer: r, raw }); - const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" }; - const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text }; - const stop = r.kind === "tool" ? "tool_use" : "end_turn"; - if (!body.stream) { - res.writeHead(200, { "content-type": "application/json" }); - const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text }; - return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } })); - } - const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`); - res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" }); - send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } }); - send("content_block_start", { index: 0, content_block: content }); - send("content_block_delta", { index: 0, delta }); - send("content_block_stop", { index: 0 }); - send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } }); - send("message_stop", {}); - res.end(); - }); - }); + const server = createHttp(messagesHandler({ script, done, requests })); await new Promise((r) => server.listen(0, "127.0.0.1", r)); t.after(() => server.close()); return { url: `http://127.0.0.1:${server.address().port}`, requests }; diff --git a/packages/harness/tests/pi-fit.test.mjs b/packages/harness/tests/pi-fit.test.mjs index 9bb45e8b4..7ce8f9cc5 100644 --- a/packages/harness/tests/pi-fit.test.mjs +++ b/packages/harness/tests/pi-fit.test.mjs @@ -16,6 +16,7 @@ import * as pi from "../../plugins/harness/pi/index.mjs"; import { BWRAP, UNSHARE, checkArgv, sandboxProblem, sessionSetenv } from "../../seat/src/sandbox.mjs"; import { ENV_ALLOW, isProviderVariable, sessionEnv } from "../../seat/src/session.mjs"; import { adapterEnv } from "../src/runner.mjs"; +import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs"; const REPO = resolve(fileURLToPath(new URL("../../../", import.meta.url))); const NAME = "@earendil-works/pi-coding-agent"; @@ -201,6 +202,12 @@ test("pi-unpin S3 re-check 3: every run has its own scratch directory, and the r assert.deepEqual(a.slice(bin - 3, bin + 1), ["/bin/sh", "-c", 'exec "$0" "$@"', found.view]); return { scratch, args: a.slice(bin + 1).map((x) => x.split(scratch).join("")) }; }); + // #1873 S6: the scripted-model run is a session-shaped -p run on the stand-in route, with the + // gate's extension, inside the scripted model's process. It is checked here and left out below. + const modelRuns = runs.filter((x) => x.args.includes(FIT_MODEL_ROUTE.provider)); + assert.deepEqual(modelRuns.map((x) => x.args), [fitModule.sessionShape({ noMcp: extraFlags.length > 0, extensions: [...fitModule.MODEL_EXTENSIONS], prompt: "/prompt.md", sessionDir: "/sessions", message: "fit check", ...fitModule.MODEL_SHAPE, route: FIT_MODEL_ROUTE })]); + assert.equal(new Set(runs.map((x) => x.scratch)).size, runs.length, "the model run has its own scratch directory too"); + runs.splice(runs.indexOf(modelRuns[0]), 1); const seal = [...SEAL, ...extraFlags]; const loaders = [...LOADERS, ...extraFlags]; // --version, --help, the control, the sealed rpc, one per loader flag, the session-shaped runs, @@ -1200,3 +1207,192 @@ test("pi-unpin S3 re-check 6: every argv the adapter builds in a bus session's e assert.ok(set.has(name) || local || SHELL_SET.includes(name) || NOT_BUS.includes(name), `the adapter reads ${name} (line ${at + 1}), which no bus session environment here sets and the adapter doesn't assign first`); } }); + +// #1873 S6: the fit check runs pi once against a scripted model on the sandbox's loopback +// (pi-fit-model.mjs). The gate's tool_call hook must decide every tool run, the run must end with +// agent_end, the --system-prompt file must reach the model, and the four read spellings pi 0.85.1 +// knows must go through the hook. A run that can't tell is undecided. +const MODEL_FILE = fileURLToPath(new URL("../src/pi-fit-model.mjs", import.meta.url)); +// The processes of this file's model runs: each names its scratch directory, which carries +// "-model-", under tmpdir() (the model's config path, pi's --system-prompt and the sandbox layers). +const modelProcs = () => procsNaming(join(tmpdir(), "mosaic-pi-fit-")).filter((p) => { + try { return readFileSync(`/proc/${p}/cmdline`, "utf8").includes("-model-"); } catch { return false; } +}); +const modelDirs = () => readdirSync(tmpdir()).filter((n) => n.startsWith("mosaic-pi-fit-") && n.includes("-model-")); +async function noModelLeft(what) { + const deadline = Date.now() + 2000; + while (modelProcs().length && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 50)); + assert.deepEqual(modelProcs(), [], `a model run process outlived ${what}`); + assert.deepEqual(modelDirs(), [], `a model run's scratch outlived ${what}`); +} + +test("pi-unpin S6: a stand-in pi passes the model run, and no model process or scratch is left after the pass", SKIP, async t => { + const r = await fit(standIn(t, {}).env); + assert.equal(r.ok, true, r.message); + await noModelLeft("a pass"); +}); + +test("pi-unpin S6: a pi that runs a tool the gate's hook blocked is refused, and nothing is left", SKIP, async t => { + const s = standIn(t, { ignoreBlock: true }); + const r = await fit(s.env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool"); + noPath(r.message, s.paths); + await noModelLeft("a refusal"); +}); + +test("pi-unpin S6: a pi that runs a tool when a tool_call handler throws is refused", SKIP, async t => { + const s = standIn(t, { ignoreThrow: true }); + const r = await fit(s.env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) a throw in a tool_call handler didn't stop the tool"); + noPath(r.message, s.paths); + await noModelLeft("a refusal"); +}); + +test("pi-unpin S6: a pi that never emits agent_end is refused", SKIP, async t => { + const r = await fit(standIn(t, { noAgentEnd: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't end its run with agent_end"); +}); + +test("pi-unpin S6: a pi that drops the system prompt from its model requests is refused", SKIP, async t => { + const r = await fit(standIn(t, { modelNoSystem: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't send the --system-prompt file to the model"); +}); + +for (const [spelling, label] of [["amPm", "a narrow space before AM"], ["nfd", "decomposed accents"], ["curly", "a curly apostrophe"], ["nfdCurly", "decomposed accents and a curly apostrophe"]]) { + test(`pi-unpin S6: a pi whose read under another spelling (${spelling}) skips the gate's hook is refused`, SKIP, async t => { + const r = await fit(standIn(t, { skipHookSpelling: spelling }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool for a read under another spelling (${label})`); + }); +} + +// A blocked read that fails on its own, with no hook run, holds no outside word, so only the check +// that the error carries the gate's own reason refuses it. +for (const [skip, label] of [["outside", "a read outside the workspace"], ["amPm", "a read under another spelling (a narrow space before AM)"], ["nfd", "a read under another spelling (decomposed accents)"], ["curly", "a read under another spelling (a curly apostrophe)"], ["nfdCurly", "a read under another spelling (decomposed accents and a curly apostrophe)"]]) { + test(`pi-unpin S6: a pi whose read (${skip}) skips the gate's hook and fails on its own is refused`, SKIP, async t => { + const r = await fit(standIn(t, { hookSkipFail: skip }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's tool_call hook didn't decide the tool for ${label}`); + }); +} + +test("pi-unpin S6: a pi whose reads all fail, the allowed one too, is refused", SKIP, async t => { + const r = await fit(standIn(t, { readFails: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't run the tool the gate allowed"); +}); + +test("pi-unpin S6: a pi whose agent_end carries no final message is refused", SKIP, async t => { + const r = await fit(standIn(t, { agentEndEmpty: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) its agent_end didn't carry the model's final answer"); +}); + +test("pi-unpin S6: a pi that finishes the script and then exits non-zero is undecided, not a pass", SKIP, async t => { + const r = await fit(standIn(t, { exitCode: 1 }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, true); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero"); +}); + +test("pi-unpin S6: a pi that exits non-zero before any model request is undecided, and the line says so", SKIP, async t => { + const r = await fit(standIn(t, { noModelCall: true, exitCode: 1 }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, true); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero before any model request"); +}); + +test("pi-unpin S6: a pi that runs a blocked tool and then hangs is refused, not undecided, and nothing is left", SKIP, async t => { + const r = await fit(standIn(t, { ignoreBlock: true, modelHang: 2 }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, false); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool"); + await noModelLeft("a refusal after a hang"); +}); + +test("pi-unpin S6: a run whose scripted model gets no request is undecided, not a pass", SKIP, async t => { + const r = await fit(standIn(t, { noModelCall: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, true); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted model got no request"); +}); + +test("pi-unpin S6: a model run that times out is undecided, and the model and every child are gone", SKIP, async t => { + const r = await fit(standIn(t, { modelHang: true }).env); + assert.equal(r.ok, false); + assert.equal(r.undecided, true); + assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run timed out"); + await noModelLeft("a timeout"); +}); + +test("pi-unpin S6: a check aborted while its model run runs is undecided, and the model and every child are gone", SKIP, async t => { + const s = standIn(t, { modelHang: true }); + const abort = new AbortController(); + const pending = piFit(locate(s.env), { levels: LEVELS, signal: abort.signal }); + const deadline = Date.now() + 15000; + while (!modelProcs().some((p) => readFileSync(`/proc/${p}/cmdline`, "utf8").includes(MODEL_FILE)) && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 20)); + assert.ok(modelDirs().length === 1, "the model run did not start"); + // Let the stand-in reach its hang, after the model's first answer. + await new Promise((resolve) => setTimeout(resolve, 500)); + abort.abort(); + const r = await pending; + assert.equal(r.ok, false); + assert.equal(r.undecided, true); + await noModelLeft("an abort"); +}); + +// A fresh copy of the module, with its own cache, so an earlier test's pass of the same pi is not +// reused. Each model run makes one "-model-" scratch directory, so a watch on tmpdir() counts them. +test("pi-unpin S6: real pi 0.85.1 passes the model run, and a second call for the same digest runs no model stand-in", SKIP, async t => { + const fresh = await import(`../src/pi-fit.mjs?s6-${process.pid}`); + const made = new Set(); + const watcher = watch(tmpdir(), (event, name) => { if (name && name.startsWith("mosaic-pi-fit-") && name.includes("-model-")) made.add(name); }); + t.after(() => watcher.close()); + const settle = () => new Promise((resolve) => setTimeout(resolve, 200)); + const found = locate({ PATH: "/usr/bin:/bin" }); + const first = await fresh.piFit(found, { levels: LEVELS }); + assert.equal(first.ok, true, first.message); + assert.equal(first.version, "0.85.1"); + await settle(); + assert.equal(made.size, 1, `the first check made ${made.size} model runs`); + made.clear(); + assert.equal(await fresh.piFit(found, { levels: LEVELS }), first, "the second call reuses the pass"); + await settle(); + assert.equal(made.size, 0, "the second call for the same digest ran the model stand-in"); + await noModelLeft("real pi's pass"); +}); + +test("pi-unpin S6: the model run is sandboxed with no network but loopback, no provider key, and a route that names no Claude model", SKIP, async t => { + const claude = /claude|anthropic|sonnet|opus|haiku|fable/i; + for (const name of [FIT_MODEL_ROUTE.provider, FIT_MODEL_ROUTE.model]) { + assert.ok(typeof name === "string" && name.length > 0); + assert.doesNotMatch(name, claude, "the model run's route names a Claude model or provider"); + } + const s = standIn(t, {}); + const log = join(s.root, "bwrap.log"); + const bwrap = join(s.root, "bwrap"); + writeFileSync(bwrap, `#!/bin/sh\nf=$(/usr/bin/mktemp '${log}.XXXXXX')\nfor a in "$@"; do printf '%s\\037' "$a"; done > "$f"\nexec '${BWRAP}' "$@"\n`, { mode: 0o755 }); + const r = await piFit(locate(s.env), { levels: LEVELS, bwrap, host: { ...HOSTS.unit, ...CREDENTIAL } }); + assert.equal(r.ok, true, r.message); + const runs = readdirSync(s.root).filter((n) => n.startsWith("bwrap.log.")).map((n) => readFileSync(join(s.root, n), "utf8").split("\x1f")).filter((a) => a.includes(MODEL_FILE)); + assert.equal(runs.length, 1); + const a = runs[0]; + assert.ok(a.includes("--unshare-net"), "the model run shares the host's network"); + assert.ok(a.includes("--clearenv")); + const setenv = a.flatMap((x, i) => (x === "--setenv" ? [a[i + 1]] : [])); + for (const name of setenv) assert.equal(isProviderVariable(name), false, `the model run is given ${name}`); + assert.equal(a[a.indexOf("--provider") + 1], FIT_MODEL_ROUTE.provider); + assert.equal(a[a.indexOf("--model") + 1], FIT_MODEL_ROUTE.model); +}); diff --git a/packages/harness/tests/pi-session.test.mjs b/packages/harness/tests/pi-session.test.mjs index 427240865..4f689cfea 100644 --- a/packages/harness/tests/pi-session.test.mjs +++ b/packages/harness/tests/pi-session.test.mjs @@ -1,12 +1,16 @@ import { fixtureCacheEnv } from '../../../scripts/testing/lane-fixtures.mjs'; -// Real pi (the pinned CLI) through adapters/pi with the extension, against -// the scripted Messages API. No real model, no network beyond 127.0.0.1. +// Real pi through adapters/pi with the extension, against the scripted +// Messages API. No real model, no network beyond 127.0.0.1. #1873 S6: the pi +// is the one the pi module's lookup finds (MOSAIC_PI_BIN, else the first pi on +// PATH, else the repository's), so `MOSAIC_PI_BIN=/abs/pi node --test` runs the +// suite against another pi. It must pass the fit check first. import { test } from "node:test"; import assert from "node:assert/strict"; import { spawn } from "node:child_process"; -import { existsSync, mkdirSync, readdirSync, readFileSync, symlinkSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; +import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, symlinkSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { locate } from "../../plugins/harness/pi/binary.mjs"; import { buildBundle } from "../src/bundle.mjs"; import { adapterEnv, unslopCounts, unslopLog, unslopNotes } from "../src/runner.mjs"; import { REPLY_CAP } from "../src/tools.mjs"; @@ -53,7 +57,8 @@ async function session(t, script, place = (dir) => join(dir, "ws"), instance = " s.bundle = files; s.harnessEnv = harnessEnv; const env = adapterEnv(s, { - PATH: `${join(REPO, "node_modules", ".bin")}:/usr/bin:/bin`, + // #1873 S6: the node running this suite, for a pi whose first line is `#!/usr/bin/env node`. + PATH: `${join(REPO, "node_modules", ".bin")}:${dirname(process.execPath)}:/usr/bin:/bin`, HOME: join(dir, "home"), PI_CODING_AGENT_DIR: agentDir, }); @@ -75,6 +80,21 @@ function turn(env, request, cwd) { }); } +// #1873 S6: MOSAIC_PI_BIN, when the suite runs with it, is the pi every session here runs. +test("pi: the session runs the pi MOSAIC_PI_BIN names, else the repository's", async (t) => { + const { api, s, env } = await session(t, []); + // The pi module's own lookup on this process's environment: MOSAIC_PI_BIN, else the first pi on + // PATH, else the repository's. + const found = locate(process.env); + assert.ok(found.bin, found.problem); + t.diagnostic(`pi under test: ${found.label}`); + assert.equal(realpathSync(env.MOSAIC_PI_BIN), realpathSync(found.bin)); + const r = await turn(env, "Message 1 from jason, class REQUEST:\n\nhello", s.workspace); + assert.equal(r.code, 0, r.stderr); + assert.ok(api.requests.length > 0, "the pi it names called the scripted model"); + assert.ok(existsSync(s.turnMarker)); +}); + test("pi: typed tools reach the socket, the gate blocks, agent_end writes the marker", async (t) => { const { api, sock, s, env } = await session(t, [ { name: "list_agents", input: {} }, diff --git a/packages/harness/tests/pi-stand-in.mjs b/packages/harness/tests/pi-stand-in.mjs index 5d06668b2..ca086c500 100644 --- a/packages/harness/tests/pi-stand-in.mjs +++ b/packages/harness/tests/pi-stand-in.mjs @@ -58,12 +58,32 @@ // as pi 0.85.1 does. `skillLeak`: with a --skill given, the planted skills load too // whatever --no-skills says (discovery comes back with --skill). `ignoreSkill`: a // --skill directory loads only without --no-skills. +// -p with --provider FIT_MODEL_ROUTE's provider (#1873 S6, the fit check's model run): an agent +// loop on the scripted model. It reads the provider's baseUrl from the agent +// directory's models.json, imports each -e file and calls its default export with a +// small pi (on, registerTool), then posts to /v1/messages with the +// --system-prompt file's text as system and one read tool. Each tool_use runs the +// tool_call handlers first; a { block } result gives an error tool result holding the +// reason, else the read runs, trying pi 0.85.1's other spellings when the path is +// missing (a narrow space before AM or PM, NFD, a curly apostrophe, both). A text +// answer fires agent_end with the assistant message, prints the text and exits 0. +// `ignoreBlock`: the read runs whatever the handlers say. `ignoreThrow`: a +// handler's throw is dropped and the read runs. `noAgentEnd`: no agent_end. +// `modelNoSystem`: no system in the requests. `skipHookSpelling` ("amPm", "nfd", +// "curly" or "nfdCurly"): a read whose path resolves only under that spelling skips +// the handlers. `hookSkipFail` (a spelling, or "outside" for the read outside the +// project): that read skips the handlers and fails without the file's text. +// `readFails`: every read that runs fails. `modelHang`: after the first answer it +// hangs, or after answer N when it is a number. `exitCode`: it exits with this code +// after agent_end. `agentEndEmpty`: agent_end carries no messages. `noModelCall`: +// it fires agent_end and exits 0 with no request. // otherwise a session with no $ARGV_OUT: prints `sessionOut` and exits 0. // // Trust, as pi 0.85.1 has it: the project is trusted with --approve, or with a saved trust and // no --no-approve. Only the project SYSTEM.md and APPEND_SYSTEM.md wait on it here. The user // APPEND_SYSTEM.md is appended whatever the flags, as 0.85.1 does when the project has none. import { FIT_FLAGS } from "../src/pi-fit.mjs"; +import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs"; const BODY = String.raw` const fs = require("node:fs"); @@ -95,7 +115,72 @@ for (let i = 0; i < argv.length; i++) { process.exit(1); } } -if (rpc || print) { +const modelRun = print && val("--provider") === MODEL_PROVIDER; +// The model run's agent loop (see the header). +async function agentLoop() { + const agentDir = process.env.PI_CODING_AGENT_DIR || path.join(process.env.HOME || "/", ".pi", "agent"); + const handlers = { tool_call: [], agent_end: [] }; + const fake = { on(name, h) { (handlers[name] = handlers[name] || []).push(h); }, registerTool() {}, getCommands() { return []; } }; + for (const f of argv.flatMap((a, i) => a === "-e" ? [argv[i + 1]] : [])) { + const m = await import(require("node:url").pathToFileURL(f).href); + m.default(fake); + } + const end = async (text) => { + if (!K.noAgentEnd) for (const h of handlers.agent_end) await h({ messages: K.agentEndEmpty ? [] : [{ role: "assistant", content: [{ type: "text", text }], stopReason: "stop" }] }); + process.stdout.write(text + "\n"); + process.exit(K.exitCode || 0); + }; + if (K.noModelCall) return end("no call"); + const models = JSON.parse(fs.readFileSync(path.join(agentDir, "models.json"), "utf8")); + const base = models.providers[MODEL_PROVIDER].baseUrl; + const sysFile = val("--system-prompt"); + const system = K.modelNoSystem || !sysFile ? undefined : fs.readFileSync(sysFile, "utf8"); + const tools = [{ name: "read", description: "read a file", input_schema: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } }]; + const messages = [{ role: "user", content: argv[argv.length - 1] }]; + const variants = { + amPm: (p) => p.replace(/ (AM|PM)/g, "\u202F$1"), + nfd: (p) => p.normalize("NFD"), + curly: (p) => p.replace(/'/g, "\u2019"), + nfdCurly: (p) => p.replace(/'/g, "\u2019").normalize("NFD"), + }; + const resolveRead = (p) => { + const abs = path.resolve(process.cwd(), p); + if (fs.existsSync(abs)) return { file: abs, via: null }; + for (const [name, v] of Object.entries(variants)) if (fs.existsSync(v(abs))) return { file: v(abs), via: name }; + return { file: abs, via: null }; + }; + for (let turn = 0; ; turn++) { + const res = await fetch(base + "/v1/messages", { method: "POST", headers: { "content-type": "application/json", "x-api-key": "stand-in" }, body: JSON.stringify({ model: val("--model"), max_tokens: 100, stream: false, ...(system === undefined ? {} : { system }), tools, messages }) }); + const reply = await res.json(); + if (K.modelHang === true ? turn === 0 : K.modelHang === turn) { setInterval(() => {}, 1000); return; } + const use = (reply.content || []).find((c) => c.type === "tool_use"); + if (!use) return end((reply.content || []).map((c) => c.text || "").join("")); + messages.push({ role: "assistant", content: reply.content }); + const { file, via } = resolveRead(use.input.path); + let blocked = null; + let threw = null; + const skipFail = K.hookSkipFail !== undefined && (K.hookSkipFail === "outside" ? via === null && use.input.path.includes("outside") : via === K.hookSkipFail); + if (!(K.skipHookSpelling && via === K.skipHookSpelling) && !skipFail) { + for (const h of handlers.tool_call) { + let r; + try { r = await h({ toolName: use.name, toolCallId: use.id, input: use.input }); } catch (e) { threw = e.message; break; } + if (r && r.block && !blocked) blocked = r.reason || "blocked"; + } + } + let result; + if (skipFail || (K.readFails && threw === null && !blocked)) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: EACCES" }; + else if (threw !== null && !K.ignoreThrow) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: threw }; + else if (blocked && !K.ignoreBlock) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: blocked }; + else { + try { result = { type: "tool_result", tool_use_id: use.id, content: fs.readFileSync(file, "utf8") }; } + catch (e) { result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: " + e.code }; } + } + messages.push({ role: "user", content: [result] }); + } +} +if (modelRun) { + agentLoop().catch((e) => { process.stderr.write("stand-in model run: " + e.message + "\n"); process.exit(1); }); +} else if (rpc || print) { if (K.flood) { const chunk = Buffer.alloc(1024 * 1024, "x"); for (let n = 0; n < 8; n++) for (let off = 0; off < chunk.length;) { @@ -225,9 +310,10 @@ if (rpc || print) { } } process.exit(0); +} else { + if (K.sessionOut) process.stdout.write(K.sessionOut); + process.exit(0); } -if (K.sessionOut) process.stdout.write(K.sessionOut); -process.exit(0); `; const quote = (text) => `'${String(text).replace(/'/g, "'\\''")}'`; @@ -248,13 +334,14 @@ export function piStandIn(knobs = {}, { node = process.execPath, form = "sh" } = `if (process.env.ARGV_OUT) { require("node:fs").writeFileSync(process.env.ARGV_OUT, process.argv.slice(1).map((a) => a + "\\n").join("")); process.stdout.write(${JSON.stringify(String(knobs.sessionOut ?? ""))}); process.exit(0); }`, `const K = ${JSON.stringify(knobs)};`, `const FLAGS = ${JSON.stringify(FIT_FLAGS)};`, + `const MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};`, BODY, `})();`, "", ].join("\n"); } const version = knobs.versionHang ? "sleep 60; exit 0" : knobs.versionExit != null ? `exit ${Number(knobs.versionExit)}` : `printf '%s\\n' ${quote(knobs.version ?? "9.9.9")}; exit 0`; - const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\n${BODY}`; + const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\nconst MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};\n${BODY}`; return [ "#!/bin/sh", `[ "$1" = --version ] && { ${version}; }`, diff --git a/packages/plugins/harness/pi/session-env.mjs b/packages/plugins/harness/pi/session-env.mjs index 518a5235f..688f5ca72 100644 --- a/packages/plugins/harness/pi/session-env.mjs +++ b/packages/plugins/harness/pi/session-env.mjs @@ -32,3 +32,14 @@ export const envNames = Object.freeze(["MOSAIC_PI_BIN", "MOSAIC_PI_NO_MCP", "PI_ // no Anthropic provider (pi-fit.test.mjs holds that); glm-5.3-flash is the model index.mjs has // documented limits for, and pi's own catalogue lists it under zai. export const FIT_ROUTE = Object.freeze({ provider: "zai", model: "glm-5.3-flash" }); + +// #1873 pi-unpin S6: the provider and model the fit check's scripted-model run names. Its models.json +// points that provider at a stand-in on the sandbox's own loopback (packages/harness/src/ +// pi-fit-model.mjs), so the names are plain stand-ins: no real provider, and no Claude model, which +// runs only on the claude-code harness (pi-fit.test.mjs holds that). +export const FIT_MODEL_ROUTE = Object.freeze({ provider: "fit-model", model: "fit-model" }); + +// #1873 pi-unpin S6: the gate policy the scripted-model run's -e extension (packages/harness/src/ +// pi-extension.mjs) reads: a pi session with the read tool, no typed tools, in workspace. Built here +// since the policy names this harness. +export const fitPolicy = (workspace) => ({ harness: "pi", workspace, tools: ["read"], typed: [] });