feat(harness): the pi fit check runs pi once against a scripted loopback model and caches the verdict for the digest (#1873) #2019
No files matched your search
@@ -20,17 +20,17 @@ bundle's `manifest.json` names the lines it relies on (`reliesOn`):
|
||||
|
||||
| | Pi | Claude Code | How pi's line is checked |
|
||||
|---|---|---|---|
|
||||
| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | suites, on the repository's pi (`pi-session.test.mjs`) |
|
||||
| 2. gate crash | a throw in the handler blocks | the hook runs as `<gate> \|\| exit 2` | suites, on the repository's pi |
|
||||
| 1. block a tool | `tool_call` handler returns a block | PreToolUse command hook exits 2; `--bare` is never passed | at first sight of a binary, by the fit check's model run, on the pi that runs; and suites (`pi-session.test.mjs`) |
|
||||
| 2. gate crash | a throw in the handler blocks | the hook runs as `<gate> \|\| exit 2` | at first sight of a binary, by the fit check's model run; and suites |
|
||||
| 3. gate missing | a missing `-e` file refuses to start | the wrapper blocks | at every start, by the fit check, on the pi that runs; and suites |
|
||||
| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 <gate> \|\| exit 2`, hook timeout 20 | suites, on the repository's pi |
|
||||
| 4. gate hang | the runner's wall clock plus the `agent_end` marker | `timeout -k 2 10 <gate> \|\| exit 2`, hook timeout 20 | the `agent_end` marker: at first sight of a binary, by the fit check's model run; and suites |
|
||||
| 5. bash | limited only by the tool limit | limited only by the tool limit | suites, on the repository's pi |
|
||||
|
||||
Pi is not pinned (#1873 pi-unpin). Before a session starts, `preflight`
|
||||
awaits the fit check in `src/pi-fit.mjs` on the binary the module found
|
||||
(the Triage host and route await the module's `ready` instead), and
|
||||
`env` reads its pass. It runs that binary in a sandbox with the network
|
||||
unshared and calls no model. Each run is a child process the check
|
||||
unshared and calls no real model. Each run is a child process the check
|
||||
awaits, up to four at a time, so the bus's event loop keeps running
|
||||
while it does; callers that ask while a check runs share it. Each run
|
||||
gets the environment a bus session's pi gets, built by the same code
|
||||
@@ -131,9 +131,35 @@ for example `pi 2.0.0 (MOSAIC_PI_BIN) lacks --no-context-files`. When
|
||||
`--help` lists `--no-mcp`, the adapter passes it too. A pass is cached in
|
||||
memory by a digest of the package's files (and npm's lock files, for
|
||||
the repository's copy) and the names in the runs' environment, so a bus
|
||||
restarted under other names checks again. Lines 1, 2, 4 and 5 are not
|
||||
checked at start; a pi other than the repository's copy relies on them
|
||||
unproved until a scripted-model fit run exists (design section 5, S6).
|
||||
restarted under other names checks again.
|
||||
|
||||
One more run (#1873 S6) talks to a scripted model, so lines 1, 2 and 4
|
||||
and the system prompt are checked on the pi that runs. It is a
|
||||
session-shaped `-p` run with `--tools read`, the gate's `-e` extension
|
||||
and a second one, `src/pi-fit-throw.mjs`, on a stand-in provider and
|
||||
model named `fit-model` (never a Claude model). The model is
|
||||
`src/pi-fit-model.mjs`: inside the sandbox it is the first process, it
|
||||
listens on the sandbox's own loopback (the network stays unshared),
|
||||
writes the agent directory's `models.json` for its port, starts pi and
|
||||
answers one scripted tool call per request. The script reads a project
|
||||
file the gate allows, a file outside the project, the same outside file
|
||||
through a link under each of the four spellings pi 0.85.1 tries for a
|
||||
missing path (a narrow space before AM, decomposed accents, a curly
|
||||
apostrophe, both), and a project file the second extension's handler
|
||||
throws for. The run passes only when the allowed read returns the file,
|
||||
every blocked read is a tool error carrying the gate's own reason and
|
||||
no request carries the outside file's word, the throw stops its read,
|
||||
every request carries the `--system-prompt` file, the script runs to
|
||||
its end, and the gate's `agent_end` hook writes the turn marker with the
|
||||
model's final text answer. A pi that runs a blocked tool, swallows a
|
||||
handler's throw, drops the system prompt or never emits `agent_end` is
|
||||
refused. A run that times out, exits non-zero, or whose model gets no
|
||||
request or sees the run end early, can't be checked and refuses as
|
||||
undecided. A run that leaked and then hung is refused, not undecided. Its verdict is part of the cached pass, so it
|
||||
runs once per digest. It runs beside the other runs, so a cold check
|
||||
takes about as long as before. Line
|
||||
5 (bash) is not checked at start, and nor is a read spelling pi may add
|
||||
later; a stand-in can only check what it scripts.
|
||||
|
||||
Claude Code also runs with `--restricted`: no user, project or local
|
||||
settings, file tools confined to the working directory, and no `CLAUDE.md`
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
// #1873 pi-unpin S6: the scripted model the fit check's model run talks to (pi-fit.mjs). It
|
||||
// answers the Messages API one scripted step at a time: step k is a tool call, given when the
|
||||
// request carries k tool results; after the last step it answers with fixed text. Only node's own
|
||||
// modules are imported, since the file runs inside the fit check's sandbox.
|
||||
//
|
||||
// Run as a program, inside the sandbox: `node pi-fit-model.mjs <config.json> -- <command...>`. It
|
||||
// listens on 127.0.0.1 port 0 in the sandbox's own network namespace (checkArgv unshares the
|
||||
// network, so only that namespace's loopback reaches it), writes the agent directory's models.json
|
||||
// naming that port for the stand-in provider, starts the command with its own environment and
|
||||
// stdio, writes what each request held to the record file after each request, and exits with the
|
||||
// command's status. It is the first process in the sandbox's pid namespace, so when it ends, or is
|
||||
// killed, every process in the sandbox ends with it. No real model and no host network are
|
||||
// reached: the sandbox has no other network.
|
||||
//
|
||||
// packages/harness/tests/helpers.mjs mockAnthropic serves the same answers to session tests.
|
||||
import { createServer } from "node:http";
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b?.text ?? "").join("") : "");
|
||||
|
||||
// The tool results in a request's messages, in order: { id, isError, content }.
|
||||
export function toolResults(messages) {
|
||||
const out = [];
|
||||
for (const m of messages ?? []) {
|
||||
if (m?.role !== "user" || !Array.isArray(m.content)) continue;
|
||||
for (const b of m.content) if (b?.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// A request handler for the Messages API. script: [{ name, input }], one tool call per step.
|
||||
// done(results): the text that ends the run once the script is spent. requests: each request is
|
||||
// pushed as { tools, results, system, answer, raw }. onRequest: called after each push. A request
|
||||
// with no tools gets "mock". Streaming and plain answers both work; count_tokens answers 10.
|
||||
export function messagesHandler({ script = [], done = (r) => `DONE ${JSON.stringify(r)}`, requests = [], onRequest = () => {} } = {}) {
|
||||
return (req, res) => {
|
||||
let raw = "";
|
||||
req.on("data", (c) => (raw += c));
|
||||
req.on("end", () => {
|
||||
if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) {
|
||||
res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" });
|
||||
return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } }));
|
||||
}
|
||||
let body;
|
||||
try { body = JSON.parse(raw || "{}"); } catch { body = {}; }
|
||||
const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : [];
|
||||
const results = toolResults(body.messages);
|
||||
const step = tools.length ? script[results.length] : null;
|
||||
const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" };
|
||||
requests.push({ tools, results, system: body.system, answer: r, raw });
|
||||
onRequest();
|
||||
const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" };
|
||||
const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text };
|
||||
const stop = r.kind === "tool" ? "tool_use" : "end_turn";
|
||||
if (!body.stream) {
|
||||
res.writeHead(200, { "content-type": "application/json" });
|
||||
const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text };
|
||||
return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } }));
|
||||
}
|
||||
const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`);
|
||||
res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
|
||||
send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } });
|
||||
send("content_block_start", { index: 0, content_block: content });
|
||||
send("content_block_delta", { index: 0, delta });
|
||||
send("content_block_stop", { index: 0 });
|
||||
send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } });
|
||||
send("message_stop", {});
|
||||
res.end();
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
// How much of each tool result the record keeps.
|
||||
const KEEP = 2000;
|
||||
|
||||
// config: { record, models, provider, model, key, script, done, secret }. record: the file each
|
||||
// request is written to, as { requests: [{ tools, system, results: [{ isError, content }],
|
||||
// done, leak }] }: tools is the count of tools offered, done whether the answer was the done text,
|
||||
// leak whether the raw request held the secret word.
|
||||
async function main(argv) {
|
||||
const [config, sep, ...command] = argv;
|
||||
if (!config || sep !== "--" || !command.length) process.exit(2);
|
||||
const c = JSON.parse(readFileSync(config, "utf8"));
|
||||
const requests = [];
|
||||
const record = () => writeFileSync(c.record, JSON.stringify({ requests: requests.map((r) => ({
|
||||
tools: r.tools.length,
|
||||
system: textOf(r.system),
|
||||
results: r.results.map((x) => ({ isError: Boolean(x.isError), content: String(x.content).slice(0, KEEP) })),
|
||||
done: r.answer.kind === "text" && r.answer.text === c.done,
|
||||
leak: r.raw.includes(c.secret),
|
||||
})) }));
|
||||
const server = createServer(messagesHandler({ script: c.script, done: () => c.done, requests, onRequest: record }));
|
||||
await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
|
||||
mkdirSync(dirname(c.models), { recursive: true });
|
||||
writeFileSync(c.models, JSON.stringify({ providers: { [c.provider]: { baseUrl: `http://127.0.0.1:${server.address().port}`, api: "anthropic-messages", apiKey: c.key, models: [{ id: c.model }] } } }));
|
||||
const child = spawn(command[0], command.slice(1), { stdio: "inherit", env: process.env });
|
||||
const status = await new Promise((resolve) => {
|
||||
child.on("error", () => resolve(1));
|
||||
child.on("exit", (code, signal) => resolve(code ?? (signal ? 1 : 0)));
|
||||
});
|
||||
server.close();
|
||||
process.exit(status);
|
||||
}
|
||||
|
||||
const self = fileURLToPath(import.meta.url);
|
||||
let invoked = null;
|
||||
try { invoked = process.argv[1] ? realpathSync(process.argv[1]) : null; } catch {}
|
||||
if (invoked === self) main(process.argv.slice(2)).catch(() => process.exit(1));
|
||||
@@ -0,0 +1,14 @@
|
||||
// #1873 pi-unpin S6: a second -e extension, for the fit check's model run only (pi-fit.mjs); no
|
||||
// session loads it. pi loads it after the gate's, so for a read of THROW_NAME the gate has allowed
|
||||
// the read before this handler throws. The run then shows that a throw in a tool_call handler
|
||||
// stops the tool (S0 line 2: a gate that crashes blocks). It imports nothing, since it runs inside
|
||||
// the fit check's sandbox.
|
||||
export const THROW_NAME = "throw.txt";
|
||||
export const THROW_TEXT = "fit check: the tool_call handler threw";
|
||||
|
||||
export default function fitThrow(host) {
|
||||
host.on("tool_call", async (event) => {
|
||||
if (event?.toolName === "read" && event.input?.path === THROW_NAME) throw new Error(THROW_TEXT);
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
+175
-19
@@ -2,7 +2,8 @@
|
||||
// here compares a version. The check runs the binary only inside a sandbox with the network
|
||||
// unshared, the found package mounted as a session mounts it, and a bus session's environment
|
||||
// (fitEnv below), so its agent directory is $HOME/.pi/agent under a scratch HOME, as a session's
|
||||
// is. No model is called and no provider key is passed. It asks:
|
||||
// is. No real model is called and no provider key is passed: the one run that reaches a model
|
||||
// reaches a scripted stand-in on the sandbox's own loopback. It asks:
|
||||
//
|
||||
// --version a label for the refusal line and the record; not a gate
|
||||
// --help lists, in its option column, every flag Mosaic passes, with rpc mode and each
|
||||
@@ -56,6 +57,23 @@
|
||||
// its own any more.
|
||||
// missing -e a missing -e file, in rpc and in -p, stops the start: exit non-zero, no answer,
|
||||
// and no session_start for the -e extension given after it
|
||||
// model #1873 S6: one -p run in a bus session's shape (sessionShape, with --tools read and
|
||||
// the session's own -e extension, pi-extension.mjs, which gates every tool call with
|
||||
// gate.mjs and writes the turn marker at agent_end) against a scripted model:
|
||||
// pi-fit-model.mjs, started inside the sandbox ahead of the binary, listening on the
|
||||
// sandbox's loopback and named by the agent directory's models.json under a stand-in
|
||||
// provider and model (session-env.mjs FIT_MODEL_ROUTE). The script reads a file in
|
||||
// the project (the gate allows it, and its word must come back), then a file outside
|
||||
// it, then four names that pi 0.85.1 resolves to another spelling (a narrow space
|
||||
// before AM, decomposed accents, a curly apostrophe, both of the last two), each a
|
||||
// link out of the project to the same outside file. Each of those four the gate
|
||||
// blocks, and each must come back as an error holding the gate's reason, with the
|
||||
// outside file's word in no request. Every request with tools must carry the
|
||||
// --system-prompt file's word, the run must reach the script's end and exit 0, and
|
||||
// the turn marker must be written. A tool result that misses the gate's reason, a
|
||||
// request without the word, or no marker is a refusal. A run that times out, ends
|
||||
// early or sends no request can't tell. pi-fit-model.mjs is the sandbox's first
|
||||
// process, so the model and every child end with it.
|
||||
//
|
||||
// Every run gets a fresh scratch directory with the sources planted again, so no run sees what
|
||||
// another one left. The directory's name carries the pid, start time and pid namespace of the
|
||||
@@ -100,17 +118,20 @@
|
||||
// whenever the directory isn't empty), since one more run costs about 340 ms cold; and a
|
||||
// session's own values: its tool names beyond read and bash, thinking levels beyond low,
|
||||
// provider, model, key, -e files and request. The runs are offline with no key, so nothing
|
||||
// past the start is tried. A pi built to act differently with those is outside what a start-time
|
||||
// check can catch.
|
||||
// past the start is tried, but for the model run. A pi built to act differently with those, or to
|
||||
// act differently in a fit check, is outside what a start-time check can catch. The model run
|
||||
// tries read alone, one tool call per turn, and only the four spellings pi 0.85.1 knows.
|
||||
import { spawn } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, readlinkSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative } from "node:path";
|
||||
import { BWRAP, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { BWRAP, RUNNER_FILES, SETPRIV, UNSHARE, checkArgv, outerArgv, sessionSetenv } from "../../seat/src/sandbox.mjs";
|
||||
import { isProviderVariable, sessionEnv } from "../../seat/src/session.mjs";
|
||||
import { adapterEnv } from "./runner.mjs";
|
||||
import { FIT_ROUTE, harnessEnv } from "../../plugins/harness/pi/session-env.mjs";
|
||||
import { THROW_NAME } from "./pi-fit-throw.mjs";
|
||||
import { FIT_MODEL_ROUTE, FIT_ROUTE, fitPolicy, harnessEnv } from "../../plugins/harness/pi/session-env.mjs";
|
||||
|
||||
// The flags adapters/pi/adapter.sh and the conversation engine pass.
|
||||
export const FIT_FLAGS = Object.freeze(["--mode", "--session", "--no-session", "--session-dir", "--fork", "-c", "--no-extensions", "-e", "--no-skills", "--skill", "--no-prompt-templates", "--no-themes", "--no-context-files", "--no-approve", "--offline", "--tools", "--no-tools", "--provider", "--model", "--thinking", "--system-prompt", "-p"]);
|
||||
@@ -139,13 +160,14 @@ export const SESSION_RUNS = Object.freeze([
|
||||
// or none, a persistent session directory with -c when it holds a session, a thinking level or
|
||||
// none, the --system-prompt file, -p and the request, and --mode json when the model event monitor
|
||||
// runs pi. No --skill and no --fork: a bus session never passes them. Keep its order:
|
||||
// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ.
|
||||
export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message }) {
|
||||
// pi-fit.test.mjs runs the adapter through every branch and fails when the two differ. route: the
|
||||
// provider and model, SESSION_SHAPE's unless the model run passes FIT_MODEL_ROUTE.
|
||||
export function sessionShape({ noMcp, extensions, prompt, sessionDir, tools, cont, thinking, json, message, route = SESSION_SHAPE }) {
|
||||
return [
|
||||
"--offline", "--no-approve", "--no-extensions", ...(noMcp ? ["--no-mcp"] : []), ...extensions.flatMap((e) => ["-e", e]),
|
||||
"--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files",
|
||||
...(tools ? ["--tools", tools] : ["--no-tools"]), "--session-dir", sessionDir, ...(cont ? ["-c"] : []),
|
||||
"--provider", SESSION_SHAPE.provider, "--model", SESSION_SHAPE.model, ...(thinking ? ["--thinking", thinking] : []),
|
||||
"--provider", route.provider, "--model", route.model, ...(thinking ? ["--thinking", thinking] : []),
|
||||
"--system-prompt", prompt, "-p", message, ...(json ? ["--mode", "json"] : []),
|
||||
];
|
||||
}
|
||||
@@ -165,6 +187,41 @@ export function workerShape({ noMcp, extensions, skills, prompt, message }) {
|
||||
// What a session-shaped run is named by in a refusal line.
|
||||
const shapeName = ({ tools, cont, thinking, json }) => `in a session's shape with ${[tools ? "--tools" : "--no-tools", ...(cont ? ["-c"] : []), ...(thinking ? ["--thinking"] : []), ...(json ? ["--mode json"] : [])].join(", ")}`;
|
||||
const RUN_TIMEOUT = 20000;
|
||||
|
||||
// #1873 S6: the model run. Its first -e extension is the gate's, which a session loads; the second
|
||||
// is pi-fit-throw.mjs, whose handler throws for one read the gate allows. The model is
|
||||
// pi-fit-model.mjs. All are mounted read-only where they are, with the runner files the gate's
|
||||
// extension imports, as a session mounts them (sandbox.mjs binds).
|
||||
const EXTENSION = fileURLToPath(new URL("./pi-extension.mjs", import.meta.url));
|
||||
const THROW_EXTENSION = fileURLToPath(new URL("./pi-fit-throw.mjs", import.meta.url));
|
||||
export const MODEL_EXTENSIONS = Object.freeze([EXTENSION, THROW_EXTENSION]);
|
||||
const MODEL_FILE = fileURLToPath(new URL("./pi-fit-model.mjs", import.meta.url));
|
||||
const MODEL_MOUNTS = [MODEL_FILE, THROW_EXTENSION, ...RUNNER_FILES].map((src) => ({ kind: "ro", src }));
|
||||
export const MODEL_SHAPE = Object.freeze({ tools: "read", cont: false, thinking: "", json: false });
|
||||
// The project file's word, which the allowed read must bring back; the outside file's, which must
|
||||
// reach no request; the scripted model's last answer.
|
||||
const INSIDE = "FITINSIDEWORD";
|
||||
const SECRET = "FITSECRETWORD";
|
||||
// The word in the project file the throwing handler's read names, which must reach no request.
|
||||
const THROWN = "FITTHROWNWORD";
|
||||
const DONE = "FITDONE";
|
||||
// The four spellings pi 0.85.1 tries when a read's path is missing: the name the model asks for,
|
||||
// the name on disk (a link to the outside file) and what a refusal line calls it.
|
||||
const SPELLINGS = [
|
||||
["shot 9.41 AM.png", "shot 9.41\u202FAM.png", "a narrow space before AM"],
|
||||
["r\u00E9sum\u00E9.txt", "r\u00E9sum\u00E9.txt".normalize("NFD"), "decomposed accents"],
|
||||
["notes's.txt", "notes\u2019s.txt", "a curly apostrophe"],
|
||||
["d'\u00E9cran.png", "d\u2019\u00E9cran.png".normalize("NFD"), "decomposed accents and a curly apostrophe"],
|
||||
];
|
||||
// The script: the allowed read, the read outside the project, one read per spelling, then the read
|
||||
// the gate allows and the second handler throws for.
|
||||
export const MODEL_SCRIPT = Object.freeze([
|
||||
{ name: "read", input: { path: "inside.txt" } },
|
||||
{ name: "read", input: { path: "../outside/secret.txt" } },
|
||||
...SPELLINGS.map(([asked]) => ({ name: "read", input: { path: asked } })),
|
||||
{ name: "read", input: { path: THROW_NAME } },
|
||||
].map((step) => Object.freeze(step)));
|
||||
const THROW_STEP = MODEL_SCRIPT.length - 1;
|
||||
// --version gives only a label, so a hang costs this much and leaves no label.
|
||||
const VERSION_TIMEOUT = 3000;
|
||||
const OUTPUT_MAX = 4 * 1024 * 1024;
|
||||
@@ -374,13 +431,21 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c
|
||||
// get_commands answer or through the -e extension's command list (listed: through that alone).
|
||||
// shape: a session-shaped run's branches, which set its tools, thinking level and monitor
|
||||
// variables as a session's would. A run that stop kept from starting gives null.
|
||||
const once = async (args, input, { timeout, noMcp, shape }) => {
|
||||
// model (S6): the scripted-model run, with its own plant, mounts and environment; it gives the
|
||||
// run's status and stopped with the requests the model recorded and the turn marker.
|
||||
const once = async (args, input, { timeout, noMcp, shape, model }) => {
|
||||
await slot();
|
||||
try {
|
||||
if (stop.signal.aborted) return null;
|
||||
const scratch = mkdtempSync(join(tmpdir(), scratchPrefix()));
|
||||
const scratch = mkdtempSync(join(tmpdir(), `${scratchPrefix()}${model ? "model-" : ""}`));
|
||||
let ran = null;
|
||||
try {
|
||||
if (model) {
|
||||
plantModel(scratch);
|
||||
const env = fitEnv(scratch, { view: found.view, noMcp, shape: MODEL_SHAPE, host, extensions: MODEL_EXTENSIONS, route: FIT_MODEL_ROUTE });
|
||||
const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: args(scratch), input, timeout, signal: stop.signal, mounts: MODEL_MOUNTS, prefix: [node, MODEL_FILE, join(scratch, "model.json"), "--"] });
|
||||
return { status: r.status, stopped: r.stopped, requests: modelRecord(scratch), turn: turnMarker(scratch) };
|
||||
}
|
||||
plant(scratch);
|
||||
const env = fitEnv(scratch, { view: found.view, noMcp, shape, host });
|
||||
const r = ran = await runIn(found, { bwrap, unshare, node, scratch, env, args: typeof args === "function" ? args(scratch) : args, input, timeout, signal: stop.signal });
|
||||
@@ -399,8 +464,8 @@ async function check(found, { levels, bwrap, unshare, node, who, digest, host, c
|
||||
free();
|
||||
}
|
||||
};
|
||||
const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null } = {}) => {
|
||||
const p = once(args, input, { timeout, noMcp, shape });
|
||||
const run = (args, input = "", { timeout = RUN_TIMEOUT, noMcp = false, shape = null, model = false } = {}) => {
|
||||
const p = once(args, input, { timeout, noMcp, shape, model });
|
||||
// Judged in order, a later run can fail before an earlier one is awaited: this keeps that
|
||||
// failure from counting as unhandled. Awaiting p still throws it.
|
||||
p.catch(() => {});
|
||||
@@ -434,6 +499,9 @@ async function judge(found, { levels, who, digest, run }) {
|
||||
if (has("--thinking")) for (const l of levels) if (!word(line("--thinking"), l)) missing.push(`--thinking ${l}`);
|
||||
if (missing.length) return refuse(`lacks ${missing.join(", ")}`);
|
||||
const noMcp = has("--no-mcp");
|
||||
// S6: the scripted-model run is asked for first, as the longest, so it overlaps the others. It is
|
||||
// judged last.
|
||||
const modelRun = run((s) => sessionShape({ noMcp, extensions: MODEL_EXTENSIONS, prompt: join(s, "prompt.md"), sessionDir: join(s, "sessions"), message: "fit check", ...MODEL_SHAPE, route: FIT_MODEL_ROUTE }), "", { noMcp, model: true });
|
||||
|
||||
const ask = '{"id":"fit-state","type":"get_state"}\n{"id":"fit-commands","type":"get_commands"}\n';
|
||||
// Every run passes the same -e and --system-prompt, so the runs differ only by the seal flags.
|
||||
@@ -524,9 +592,49 @@ async function judge(found, { levels, who, digest, run }) {
|
||||
if (r.stopped) return unsure(`a ${mode} start with a missing -e extension ${r.stopped}`);
|
||||
if (r.status === 0 || r.answer.state || r.prompt !== null) return refuse("starts with a missing -e extension");
|
||||
}
|
||||
|
||||
const model = judgeModel(await modelRun);
|
||||
if (model) return model.undecided ? unsure(model.why) : refuse(model.why);
|
||||
return { ok: true, version, noMcp, digest };
|
||||
}
|
||||
|
||||
// S6: the model run's verdict: null for a pass, else { why, undecided }. The step results are the
|
||||
// longest list any request carried. A definite failure in what did run counts first, so a run that
|
||||
// leaked the outside file and then hung is refused, not left undecided.
|
||||
function judgeModel(m) {
|
||||
const no = (why) => ({ why, undecided: false });
|
||||
const cannot = (why) => ({ why, undecided: true });
|
||||
// A run that stop kept from starting.
|
||||
if (!m) return cannot("the scripted-model run was stopped");
|
||||
const requests = m.requests;
|
||||
const results = requests.reduce((a, r) => (r.results.length > a.length ? r.results : a), []);
|
||||
const blocked = [[1, "", "outside the workspace"], ...SPELLINGS.map(([, , label], i) => [i + 2, ` for a read under another spelling (${label})`, "under another spelling"])];
|
||||
for (const [step, what] of blocked) {
|
||||
const r = results[step];
|
||||
if (r && (!r.isError || r.content.includes(SECRET))) return no(`the gate's block didn't stop the tool${what}`);
|
||||
}
|
||||
if (requests.some((r) => r.leak)) return no("the gate's block didn't stop the tool");
|
||||
const thrown = results[THROW_STEP];
|
||||
if (thrown && (!thrown.isError || thrown.content.includes(THROWN))) return no("a throw in a tool_call handler didn't stop the tool");
|
||||
for (const [step, what, reason] of blocked) {
|
||||
const r = results[step];
|
||||
if (r && !(r.content.includes("mosaic gate: ") && r.content.includes(reason))) return no(`the gate's tool_call hook didn't decide the tool${what || " for a read outside the workspace"}`);
|
||||
}
|
||||
if (requests.some((r) => r.tools > 0 && !r.system.includes(GIVEN))) return no("didn't send the --system-prompt file to the model");
|
||||
if (results[0] && (results[0].isError || !results[0].content.includes(INSIDE))) return no("didn't run the tool the gate allowed");
|
||||
if (m.stopped) return cannot(`the scripted-model run ${m.stopped}`);
|
||||
// A pi that exits non-zero before its first request most often failed to start (an extension it
|
||||
// couldn't load, say). The line says so without its stderr, which can name host paths.
|
||||
if (!requests.length) return cannot(m.status !== 0 ? "the scripted-model run exited non-zero before any model request" : "the scripted model got no request");
|
||||
if (results.length < MODEL_SCRIPT.length || !requests.some((r) => r.done)) return cannot("the scripted-model run ended before the script's last step");
|
||||
if (m.status !== 0) return cannot("the scripted-model run exited non-zero");
|
||||
if (!m.turn) return no("didn't end its run with agent_end");
|
||||
// The scripted model ends with text, so agent_end must carry that final assistant message: the
|
||||
// gate's extension records its stopReason and whether it held text.
|
||||
if (typeof m.turn.stopReason !== "string" || m.turn.hasText !== true) return no("its agent_end didn't carry the model's final answer");
|
||||
return null;
|
||||
}
|
||||
|
||||
// Whether bwrap runs here, as sandbox.mjs's sandboxProblem asks, but as child processes the caller
|
||||
// awaits: the system part, the network unshared, an empty environment, then /usr/bin/true. And,
|
||||
// once per check, whether unshare's outer layer runs (checkArgv in sandbox.mjs): first on its own,
|
||||
@@ -568,13 +676,16 @@ async function sandboxLayers({ bwrap, unshare, node }) {
|
||||
// fitEnv's. The binary starts through /bin/sh, as the adapter starts it, so it also gets the
|
||||
// variables the shell sets. The run is a child process the caller awaits; nothing here blocks.
|
||||
// stopped: why the run was killed ("timed out", "printed more than 4 MB", "was stopped"), or null.
|
||||
function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal }) {
|
||||
// mounts and prefix (S6): more read-only mounts, and a command the binary's start is passed to
|
||||
// (the scripted model).
|
||||
function runIn(found, { bwrap, unshare, node, scratch, env, args, input, timeout, signal, mounts = [], prefix = [] }) {
|
||||
const list = [
|
||||
{ kind: "ro", src: found.root, ...(found.dest ? { dest: found.dest } : {}) },
|
||||
{ kind: "rw", src: scratch },
|
||||
{ kind: "rw", src: join(scratch, "tmp"), dest: "/tmp" },
|
||||
...mounts,
|
||||
];
|
||||
return spawned(checkArgv(list, ["/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal });
|
||||
return spawned(checkArgv(list, [...prefix, "/bin/sh", "-c", 'exec "$0" "$@"', found.view, ...args], { env, chdir: join(scratch, "project"), node, bwrap, unshare }), { input, timeout, signal });
|
||||
}
|
||||
|
||||
// One child process with an empty environment, awaited: its exit status, its stdout, and why it was
|
||||
@@ -645,12 +756,13 @@ function spawned(argv, { input, timeout, signal }) {
|
||||
// tool list, thinking level and --mode json, which a proxy session's first turn takes with the
|
||||
// model event monitor's variables), else none: no tools, no thinking, one -e. noMcp: --help lists
|
||||
// --no-mcp. pi-fit.test.mjs fails when these names, or the values the code sets, differ from a bus
|
||||
// session's, for a login shell's host and a unit's.
|
||||
export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env }) {
|
||||
// session's, for a login shell's host and a unit's. extensions and route (S6): the model run's -e
|
||||
// files and its provider and model, in place of the planted ones and SESSION_SHAPE's.
|
||||
export function fitEnv(scratch, { view, noMcp = false, shape = null, host = process.env, extensions = null, route = SESSION_SHAPE }) {
|
||||
const runner = { ...sessionEnv(withoutProviders(host), "fit"), ...sessionSetenv(scratch, { business: "fit", role: "fit" }) };
|
||||
const { tools = "", thinking = "", json = false } = shape ?? {};
|
||||
const session = { provider: SESSION_SHAPE.provider, model: SESSION_SHAPE.model, thinking };
|
||||
const extensions = [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])];
|
||||
const session = { provider: route.provider, model: route.model, thinking };
|
||||
extensions ??= [join(scratch, "explicit", "fit.ts"), ...(shape ? [join(scratch, "explicit", "second.ts")] : [])];
|
||||
const env = adapterEnv({
|
||||
...session,
|
||||
bundle: { prompt: join(scratch, "prompt.md"), policy: join(scratch, "policy.json"), tools: join(scratch, "tools.json") },
|
||||
@@ -806,6 +918,50 @@ function plant(scratch) {
|
||||
put(join(scratch, "prompt.md"), `${GIVEN}\n`);
|
||||
}
|
||||
|
||||
// S6: the model run's scratch directory: the gate's policy for the project (read, no typed tools),
|
||||
// no typed tools, the --system-prompt file, a project file the gate allows, a file outside the
|
||||
// project, a link to it in the project under each spelling's name on disk, and the scripted model's
|
||||
// config. The model writes the agent directory's models.json when it has its port.
|
||||
function plantModel(scratch) {
|
||||
const project = join(scratch, "project");
|
||||
for (const d of ["marks", "tmp", "sessions", "outside", "project"]) mkdirSync(join(scratch, d), { recursive: true });
|
||||
writeFileSync(join(project, "inside.txt"), `${INSIDE}\n`);
|
||||
writeFileSync(join(project, THROW_NAME), `${THROWN}\n`);
|
||||
const outside = join(scratch, "outside", "secret.txt");
|
||||
writeFileSync(outside, `${SECRET}\n`);
|
||||
for (const [, disk] of SPELLINGS) symlinkSync(outside, join(project, disk));
|
||||
writeFileSync(join(scratch, "policy.json"), JSON.stringify(fitPolicy(project)));
|
||||
writeFileSync(join(scratch, "tools.json"), "[]");
|
||||
writeFileSync(join(scratch, "prompt.md"), `${GIVEN}\n`);
|
||||
writeFileSync(join(scratch, "model.json"), JSON.stringify({
|
||||
record: join(scratch, "marks", "model.json"), models: join(scratch, "home", ".pi", "agent", "models.json"),
|
||||
provider: FIT_MODEL_ROUTE.provider, model: FIT_MODEL_ROUTE.model, key: "fit-model-key", script: MODEL_SCRIPT, done: DONE, secret: SECRET,
|
||||
}));
|
||||
}
|
||||
// What the scripted model recorded, as a list of { tools, system, results: [{ isError, content }],
|
||||
// done, leak }; empty when it recorded nothing that parses.
|
||||
function modelRecord(scratch) {
|
||||
let m;
|
||||
try { m = JSON.parse(readFileSync(join(scratch, "marks", "model.json"), "utf8")); } catch { return []; }
|
||||
if (!Array.isArray(m?.requests)) return [];
|
||||
return m.requests.filter((r) => r && typeof r === "object").map((r) => ({
|
||||
tools: Number.isInteger(r.tools) ? r.tools : 0,
|
||||
system: typeof r.system === "string" ? r.system : "",
|
||||
results: Array.isArray(r.results) ? r.results.map((x) => ({ isError: x?.isError === true, content: typeof x?.content === "string" ? x.content : "" })) : [],
|
||||
done: r.done === true,
|
||||
leak: r.leak === true,
|
||||
}));
|
||||
}
|
||||
// The turn marker the session's -e extension wrote at agent_end, a JSON object, else null.
|
||||
function turnMarker(scratch) {
|
||||
try {
|
||||
const m = JSON.parse(readFileSync(join(scratch, "turn"), "utf8"));
|
||||
return m && typeof m === "object" && !Array.isArray(m) ? m : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// A stored session for the project, as pi 0.85.1 writes one: a header whose cwd is the project's
|
||||
// real path (pi's -c takes only a session for its own cwd) and one user message.
|
||||
function plantSession(scratch) {
|
||||
|
||||
@@ -11,6 +11,7 @@ import { join } from "node:path";
|
||||
import { resolveInstance, validateBusinessDocument } from "../../organization/src/index.mjs";
|
||||
import { businessDoc, REPO_ROLES, systemFor } from "../../organization/tests/helpers.mjs";
|
||||
import { createLoader } from "../../plugins/src/loader.mjs";
|
||||
import { messagesHandler } from "../src/pi-fit-model.mjs";
|
||||
|
||||
export { REPO } from "../../organization/tests/helpers.mjs";
|
||||
import { REPO } from "../../organization/tests/helpers.mjs";
|
||||
@@ -76,54 +77,13 @@ export async function fakeToolSocket(t, dir, handler) {
|
||||
return { path, calls };
|
||||
}
|
||||
|
||||
const textOf = (c) => (typeof c === "string" ? c : Array.isArray(c) ? c.map((b) => b.text ?? "").join("") : "");
|
||||
function toolResults(messages) {
|
||||
const out = [];
|
||||
for (const m of messages ?? []) {
|
||||
if (m.role !== "user" || !Array.isArray(m.content)) continue;
|
||||
for (const b of m.content) if (b.type === "tool_result") out.push({ id: b.tool_use_id, isError: b.is_error ?? false, content: textOf(b.content) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// A scripted Anthropic Messages API. Each request that offers tools gets
|
||||
// the next scripted tool call; once every call has a result it answers
|
||||
// `done(results)` as text. Returns { url, requests, close }.
|
||||
// `done(results)` as text. Returns { url, requests }. #1873 S6: the answers
|
||||
// come from src/pi-fit-model.mjs, which the fit check's model run uses too.
|
||||
export async function mockAnthropic(t, { script = [], done = (r) => `DONE ${JSON.stringify(r)}` } = {}) {
|
||||
const requests = [];
|
||||
const server = createHttp((req, res) => {
|
||||
let raw = "";
|
||||
req.on("data", (c) => (raw += c));
|
||||
req.on("end", () => {
|
||||
if (req.method !== "POST" || !/\/messages(\?|$)/.test(req.url)) {
|
||||
res.writeHead(req.url.includes("count_tokens") ? 200 : 404, { "content-type": "application/json" });
|
||||
return res.end(JSON.stringify(req.url.includes("count_tokens") ? { input_tokens: 10 } : { type: "error", error: { type: "not_found_error", message: "mock" } }));
|
||||
}
|
||||
const body = JSON.parse(raw || "{}");
|
||||
const tools = Array.isArray(body.tools) ? body.tools.map((x) => x.name) : [];
|
||||
const results = toolResults(body.messages);
|
||||
const step = tools.length ? script[results.length] : null;
|
||||
const r = step ? { kind: "tool", id: `toolu_${results.length + 1}`, name: step.name, input: step.input } : { kind: "text", text: tools.length ? done(results) : "mock" };
|
||||
requests.push({ tools, results, system: body.system, answer: r, raw });
|
||||
const content = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: {} } : { type: "text", text: "" };
|
||||
const delta = r.kind === "tool" ? { type: "input_json_delta", partial_json: JSON.stringify(r.input) } : { type: "text_delta", text: r.text };
|
||||
const stop = r.kind === "tool" ? "tool_use" : "end_turn";
|
||||
if (!body.stream) {
|
||||
res.writeHead(200, { "content-type": "application/json" });
|
||||
const c = r.kind === "tool" ? { type: "tool_use", id: r.id, name: r.name, input: r.input } : { type: "text", text: r.text };
|
||||
return res.end(JSON.stringify({ id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [c], stop_reason: stop, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 5 } }));
|
||||
}
|
||||
const send = (event, data) => res.write(`event: ${event}\ndata: ${JSON.stringify({ type: event, ...data })}\n\n`);
|
||||
res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
|
||||
send("message_start", { message: { id: "msg_mock", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 10, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 } } });
|
||||
send("content_block_start", { index: 0, content_block: content });
|
||||
send("content_block_delta", { index: 0, delta });
|
||||
send("content_block_stop", { index: 0 });
|
||||
send("message_delta", { delta: { stop_reason: stop, stop_sequence: null }, usage: { output_tokens: 5 } });
|
||||
send("message_stop", {});
|
||||
res.end();
|
||||
});
|
||||
});
|
||||
const server = createHttp(messagesHandler({ script, done, requests }));
|
||||
await new Promise((r) => server.listen(0, "127.0.0.1", r));
|
||||
t.after(() => server.close());
|
||||
return { url: `http://127.0.0.1:${server.address().port}`, requests };
|
||||
|
||||
@@ -16,6 +16,7 @@ import * as pi from "../../plugins/harness/pi/index.mjs";
|
||||
import { BWRAP, UNSHARE, checkArgv, sandboxProblem, sessionSetenv } from "../../seat/src/sandbox.mjs";
|
||||
import { ENV_ALLOW, isProviderVariable, sessionEnv } from "../../seat/src/session.mjs";
|
||||
import { adapterEnv } from "../src/runner.mjs";
|
||||
import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs";
|
||||
|
||||
const REPO = resolve(fileURLToPath(new URL("../../../", import.meta.url)));
|
||||
const NAME = "@earendil-works/pi-coding-agent";
|
||||
@@ -201,6 +202,12 @@ test("pi-unpin S3 re-check 3: every run has its own scratch directory, and the r
|
||||
assert.deepEqual(a.slice(bin - 3, bin + 1), ["/bin/sh", "-c", 'exec "$0" "$@"', found.view]);
|
||||
return { scratch, args: a.slice(bin + 1).map((x) => x.split(scratch).join("<s>")) };
|
||||
});
|
||||
// #1873 S6: the scripted-model run is a session-shaped -p run on the stand-in route, with the
|
||||
// gate's extension, inside the scripted model's process. It is checked here and left out below.
|
||||
const modelRuns = runs.filter((x) => x.args.includes(FIT_MODEL_ROUTE.provider));
|
||||
assert.deepEqual(modelRuns.map((x) => x.args), [fitModule.sessionShape({ noMcp: extraFlags.length > 0, extensions: [...fitModule.MODEL_EXTENSIONS], prompt: "<s>/prompt.md", sessionDir: "<s>/sessions", message: "fit check", ...fitModule.MODEL_SHAPE, route: FIT_MODEL_ROUTE })]);
|
||||
assert.equal(new Set(runs.map((x) => x.scratch)).size, runs.length, "the model run has its own scratch directory too");
|
||||
runs.splice(runs.indexOf(modelRuns[0]), 1);
|
||||
const seal = [...SEAL, ...extraFlags];
|
||||
const loaders = [...LOADERS, ...extraFlags];
|
||||
// --version, --help, the control, the sealed rpc, one per loader flag, the session-shaped runs,
|
||||
@@ -1200,3 +1207,192 @@ test("pi-unpin S3 re-check 6: every argv the adapter builds in a bus session's e
|
||||
assert.ok(set.has(name) || local || SHELL_SET.includes(name) || NOT_BUS.includes(name), `the adapter reads ${name} (line ${at + 1}), which no bus session environment here sets and the adapter doesn't assign first`);
|
||||
}
|
||||
});
|
||||
|
||||
// #1873 S6: the fit check runs pi once against a scripted model on the sandbox's loopback
|
||||
// (pi-fit-model.mjs). The gate's tool_call hook must decide every tool run, the run must end with
|
||||
// agent_end, the --system-prompt file must reach the model, and the four read spellings pi 0.85.1
|
||||
// knows must go through the hook. A run that can't tell is undecided.
|
||||
const MODEL_FILE = fileURLToPath(new URL("../src/pi-fit-model.mjs", import.meta.url));
|
||||
// The processes of this file's model runs: each names its scratch directory, which carries
|
||||
// "-model-", under tmpdir() (the model's config path, pi's --system-prompt and the sandbox layers).
|
||||
const modelProcs = () => procsNaming(join(tmpdir(), "mosaic-pi-fit-")).filter((p) => {
|
||||
try { return readFileSync(`/proc/${p}/cmdline`, "utf8").includes("-model-"); } catch { return false; }
|
||||
});
|
||||
const modelDirs = () => readdirSync(tmpdir()).filter((n) => n.startsWith("mosaic-pi-fit-") && n.includes("-model-"));
|
||||
async function noModelLeft(what) {
|
||||
const deadline = Date.now() + 2000;
|
||||
while (modelProcs().length && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
assert.deepEqual(modelProcs(), [], `a model run process outlived ${what}`);
|
||||
assert.deepEqual(modelDirs(), [], `a model run's scratch outlived ${what}`);
|
||||
}
|
||||
|
||||
test("pi-unpin S6: a stand-in pi passes the model run, and no model process or scratch is left after the pass", SKIP, async t => {
|
||||
const r = await fit(standIn(t, {}).env);
|
||||
assert.equal(r.ok, true, r.message);
|
||||
await noModelLeft("a pass");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that runs a tool the gate's hook blocked is refused, and nothing is left", SKIP, async t => {
|
||||
const s = standIn(t, { ignoreBlock: true });
|
||||
const r = await fit(s.env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool");
|
||||
noPath(r.message, s.paths);
|
||||
await noModelLeft("a refusal");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that runs a tool when a tool_call handler throws is refused", SKIP, async t => {
|
||||
const s = standIn(t, { ignoreThrow: true });
|
||||
const r = await fit(s.env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) a throw in a tool_call handler didn't stop the tool");
|
||||
noPath(r.message, s.paths);
|
||||
await noModelLeft("a refusal");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that never emits agent_end is refused", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { noAgentEnd: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't end its run with agent_end");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that drops the system prompt from its model requests is refused", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { modelNoSystem: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't send the --system-prompt file to the model");
|
||||
});
|
||||
|
||||
for (const [spelling, label] of [["amPm", "a narrow space before AM"], ["nfd", "decomposed accents"], ["curly", "a curly apostrophe"], ["nfdCurly", "decomposed accents and a curly apostrophe"]]) {
|
||||
test(`pi-unpin S6: a pi whose read under another spelling (${spelling}) skips the gate's hook is refused`, SKIP, async t => {
|
||||
const r = await fit(standIn(t, { skipHookSpelling: spelling }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool for a read under another spelling (${label})`);
|
||||
});
|
||||
}
|
||||
|
||||
// A blocked read that fails on its own, with no hook run, holds no outside word, so only the check
|
||||
// that the error carries the gate's own reason refuses it.
|
||||
for (const [skip, label] of [["outside", "a read outside the workspace"], ["amPm", "a read under another spelling (a narrow space before AM)"], ["nfd", "a read under another spelling (decomposed accents)"], ["curly", "a read under another spelling (a curly apostrophe)"], ["nfdCurly", "a read under another spelling (decomposed accents and a curly apostrophe)"]]) {
|
||||
test(`pi-unpin S6: a pi whose read (${skip}) skips the gate's hook and fails on its own is refused`, SKIP, async t => {
|
||||
const r = await fit(standIn(t, { hookSkipFail: skip }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, `pi 2.0.0 (MOSAIC_PI_BIN) the gate's tool_call hook didn't decide the tool for ${label}`);
|
||||
});
|
||||
}
|
||||
|
||||
test("pi-unpin S6: a pi whose reads all fail, the allowed one too, is refused", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { readFails: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't run the tool the gate allowed");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi whose agent_end carries no final message is refused", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { agentEndEmpty: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) its agent_end didn't carry the model's final answer");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that finishes the script and then exits non-zero is undecided, not a pass", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { exitCode: 1 }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, true);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that exits non-zero before any model request is undecided, and the line says so", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { noModelCall: true, exitCode: 1 }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, true);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run exited non-zero before any model request");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a pi that runs a blocked tool and then hangs is refused, not undecided, and nothing is left", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { ignoreBlock: true, modelHang: 2 }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, false);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) the gate's block didn't stop the tool");
|
||||
await noModelLeft("a refusal after a hang");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a run whose scripted model gets no request is undecided, not a pass", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { noModelCall: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, true);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted model got no request");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a model run that times out is undecided, and the model and every child are gone", SKIP, async t => {
|
||||
const r = await fit(standIn(t, { modelHang: true }).env);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, true);
|
||||
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: the scripted-model run timed out");
|
||||
await noModelLeft("a timeout");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: a check aborted while its model run runs is undecided, and the model and every child are gone", SKIP, async t => {
|
||||
const s = standIn(t, { modelHang: true });
|
||||
const abort = new AbortController();
|
||||
const pending = piFit(locate(s.env), { levels: LEVELS, signal: abort.signal });
|
||||
const deadline = Date.now() + 15000;
|
||||
while (!modelProcs().some((p) => readFileSync(`/proc/${p}/cmdline`, "utf8").includes(MODEL_FILE)) && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 20));
|
||||
assert.ok(modelDirs().length === 1, "the model run did not start");
|
||||
// Let the stand-in reach its hang, after the model's first answer.
|
||||
await new Promise((resolve) => setTimeout(resolve, 500));
|
||||
abort.abort();
|
||||
const r = await pending;
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.undecided, true);
|
||||
await noModelLeft("an abort");
|
||||
});
|
||||
|
||||
// A fresh copy of the module, with its own cache, so an earlier test's pass of the same pi is not
|
||||
// reused. Each model run makes one "-model-" scratch directory, so a watch on tmpdir() counts them.
|
||||
test("pi-unpin S6: real pi 0.85.1 passes the model run, and a second call for the same digest runs no model stand-in", SKIP, async t => {
|
||||
const fresh = await import(`../src/pi-fit.mjs?s6-${process.pid}`);
|
||||
const made = new Set();
|
||||
const watcher = watch(tmpdir(), (event, name) => { if (name && name.startsWith("mosaic-pi-fit-") && name.includes("-model-")) made.add(name); });
|
||||
t.after(() => watcher.close());
|
||||
const settle = () => new Promise((resolve) => setTimeout(resolve, 200));
|
||||
const found = locate({ PATH: "/usr/bin:/bin" });
|
||||
const first = await fresh.piFit(found, { levels: LEVELS });
|
||||
assert.equal(first.ok, true, first.message);
|
||||
assert.equal(first.version, "0.85.1");
|
||||
await settle();
|
||||
assert.equal(made.size, 1, `the first check made ${made.size} model runs`);
|
||||
made.clear();
|
||||
assert.equal(await fresh.piFit(found, { levels: LEVELS }), first, "the second call reuses the pass");
|
||||
await settle();
|
||||
assert.equal(made.size, 0, "the second call for the same digest ran the model stand-in");
|
||||
await noModelLeft("real pi's pass");
|
||||
});
|
||||
|
||||
test("pi-unpin S6: the model run is sandboxed with no network but loopback, no provider key, and a route that names no Claude model", SKIP, async t => {
|
||||
const claude = /claude|anthropic|sonnet|opus|haiku|fable/i;
|
||||
for (const name of [FIT_MODEL_ROUTE.provider, FIT_MODEL_ROUTE.model]) {
|
||||
assert.ok(typeof name === "string" && name.length > 0);
|
||||
assert.doesNotMatch(name, claude, "the model run's route names a Claude model or provider");
|
||||
}
|
||||
const s = standIn(t, {});
|
||||
const log = join(s.root, "bwrap.log");
|
||||
const bwrap = join(s.root, "bwrap");
|
||||
writeFileSync(bwrap, `#!/bin/sh\nf=$(/usr/bin/mktemp '${log}.XXXXXX')\nfor a in "$@"; do printf '%s\\037' "$a"; done > "$f"\nexec '${BWRAP}' "$@"\n`, { mode: 0o755 });
|
||||
const r = await piFit(locate(s.env), { levels: LEVELS, bwrap, host: { ...HOSTS.unit, ...CREDENTIAL } });
|
||||
assert.equal(r.ok, true, r.message);
|
||||
const runs = readdirSync(s.root).filter((n) => n.startsWith("bwrap.log.")).map((n) => readFileSync(join(s.root, n), "utf8").split("\x1f")).filter((a) => a.includes(MODEL_FILE));
|
||||
assert.equal(runs.length, 1);
|
||||
const a = runs[0];
|
||||
assert.ok(a.includes("--unshare-net"), "the model run shares the host's network");
|
||||
assert.ok(a.includes("--clearenv"));
|
||||
const setenv = a.flatMap((x, i) => (x === "--setenv" ? [a[i + 1]] : []));
|
||||
for (const name of setenv) assert.equal(isProviderVariable(name), false, `the model run is given ${name}`);
|
||||
assert.equal(a[a.indexOf("--provider") + 1], FIT_MODEL_ROUTE.provider);
|
||||
assert.equal(a[a.indexOf("--model") + 1], FIT_MODEL_ROUTE.model);
|
||||
});
|
||||
@@ -1,12 +1,16 @@
|
||||
import { fixtureCacheEnv } from '../../../scripts/testing/lane-fixtures.mjs';
|
||||
// Real pi (the pinned CLI) through adapters/pi with the extension, against
|
||||
// the scripted Messages API. No real model, no network beyond 127.0.0.1.
|
||||
// Real pi through adapters/pi with the extension, against the scripted
|
||||
// Messages API. No real model, no network beyond 127.0.0.1. #1873 S6: the pi
|
||||
// is the one the pi module's lookup finds (MOSAIC_PI_BIN, else the first pi on
|
||||
// PATH, else the repository's), so `MOSAIC_PI_BIN=/abs/pi node --test` runs the
|
||||
// suite against another pi. It must pass the fit check first.
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import { existsSync, mkdirSync, readdirSync, readFileSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { locate } from "../../plugins/harness/pi/binary.mjs";
|
||||
import { buildBundle } from "../src/bundle.mjs";
|
||||
import { adapterEnv, unslopCounts, unslopLog, unslopNotes } from "../src/runner.mjs";
|
||||
import { REPLY_CAP } from "../src/tools.mjs";
|
||||
@@ -53,7 +57,8 @@ async function session(t, script, place = (dir) => join(dir, "ws"), instance = "
|
||||
s.bundle = files;
|
||||
s.harnessEnv = harnessEnv;
|
||||
const env = adapterEnv(s, {
|
||||
PATH: `${join(REPO, "node_modules", ".bin")}:/usr/bin:/bin`,
|
||||
// #1873 S6: the node running this suite, for a pi whose first line is `#!/usr/bin/env node`.
|
||||
PATH: `${join(REPO, "node_modules", ".bin")}:${dirname(process.execPath)}:/usr/bin:/bin`,
|
||||
HOME: join(dir, "home"),
|
||||
PI_CODING_AGENT_DIR: agentDir,
|
||||
});
|
||||
@@ -75,6 +80,21 @@ function turn(env, request, cwd) {
|
||||
});
|
||||
}
|
||||
|
||||
// #1873 S6: MOSAIC_PI_BIN, when the suite runs with it, is the pi every session here runs.
|
||||
test("pi: the session runs the pi MOSAIC_PI_BIN names, else the repository's", async (t) => {
|
||||
const { api, s, env } = await session(t, []);
|
||||
// The pi module's own lookup on this process's environment: MOSAIC_PI_BIN, else the first pi on
|
||||
// PATH, else the repository's.
|
||||
const found = locate(process.env);
|
||||
assert.ok(found.bin, found.problem);
|
||||
t.diagnostic(`pi under test: ${found.label}`);
|
||||
assert.equal(realpathSync(env.MOSAIC_PI_BIN), realpathSync(found.bin));
|
||||
const r = await turn(env, "Message 1 from jason, class REQUEST:\n\nhello", s.workspace);
|
||||
assert.equal(r.code, 0, r.stderr);
|
||||
assert.ok(api.requests.length > 0, "the pi it names called the scripted model");
|
||||
assert.ok(existsSync(s.turnMarker));
|
||||
});
|
||||
|
||||
test("pi: typed tools reach the socket, the gate blocks, agent_end writes the marker", async (t) => {
|
||||
const { api, sock, s, env } = await session(t, [
|
||||
{ name: "list_agents", input: {} },
|
||||
|
||||
@@ -58,12 +58,32 @@
|
||||
// as pi 0.85.1 does. `skillLeak`: with a --skill given, the planted skills load too
|
||||
// whatever --no-skills says (discovery comes back with --skill). `ignoreSkill`: a
|
||||
// --skill directory loads only without --no-skills.
|
||||
// -p with --provider FIT_MODEL_ROUTE's provider (#1873 S6, the fit check's model run): an agent
|
||||
// loop on the scripted model. It reads the provider's baseUrl from the agent
|
||||
// directory's models.json, imports each -e file and calls its default export with a
|
||||
// small pi (on, registerTool), then posts to <baseUrl>/v1/messages with the
|
||||
// --system-prompt file's text as system and one read tool. Each tool_use runs the
|
||||
// tool_call handlers first; a { block } result gives an error tool result holding the
|
||||
// reason, else the read runs, trying pi 0.85.1's other spellings when the path is
|
||||
// missing (a narrow space before AM or PM, NFD, a curly apostrophe, both). A text
|
||||
// answer fires agent_end with the assistant message, prints the text and exits 0.
|
||||
// `ignoreBlock`: the read runs whatever the handlers say. `ignoreThrow`: a
|
||||
// handler's throw is dropped and the read runs. `noAgentEnd`: no agent_end.
|
||||
// `modelNoSystem`: no system in the requests. `skipHookSpelling` ("amPm", "nfd",
|
||||
// "curly" or "nfdCurly"): a read whose path resolves only under that spelling skips
|
||||
// the handlers. `hookSkipFail` (a spelling, or "outside" for the read outside the
|
||||
// project): that read skips the handlers and fails without the file's text.
|
||||
// `readFails`: every read that runs fails. `modelHang`: after the first answer it
|
||||
// hangs, or after answer N when it is a number. `exitCode`: it exits with this code
|
||||
// after agent_end. `agentEndEmpty`: agent_end carries no messages. `noModelCall`:
|
||||
// it fires agent_end and exits 0 with no request.
|
||||
// otherwise a session with no $ARGV_OUT: prints `sessionOut` and exits 0.
|
||||
//
|
||||
// Trust, as pi 0.85.1 has it: the project is trusted with --approve, or with a saved trust and
|
||||
// no --no-approve. Only the project SYSTEM.md and APPEND_SYSTEM.md wait on it here. The user
|
||||
// APPEND_SYSTEM.md is appended whatever the flags, as 0.85.1 does when the project has none.
|
||||
import { FIT_FLAGS } from "../src/pi-fit.mjs";
|
||||
import { FIT_MODEL_ROUTE } from "../../plugins/harness/pi/session-env.mjs";
|
||||
|
||||
const BODY = String.raw`
|
||||
const fs = require("node:fs");
|
||||
@@ -95,7 +115,72 @@ for (let i = 0; i < argv.length; i++) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
if (rpc || print) {
|
||||
const modelRun = print && val("--provider") === MODEL_PROVIDER;
|
||||
// The model run's agent loop (see the header).
|
||||
async function agentLoop() {
|
||||
const agentDir = process.env.PI_CODING_AGENT_DIR || path.join(process.env.HOME || "/", ".pi", "agent");
|
||||
const handlers = { tool_call: [], agent_end: [] };
|
||||
const fake = { on(name, h) { (handlers[name] = handlers[name] || []).push(h); }, registerTool() {}, getCommands() { return []; } };
|
||||
for (const f of argv.flatMap((a, i) => a === "-e" ? [argv[i + 1]] : [])) {
|
||||
const m = await import(require("node:url").pathToFileURL(f).href);
|
||||
m.default(fake);
|
||||
}
|
||||
const end = async (text) => {
|
||||
if (!K.noAgentEnd) for (const h of handlers.agent_end) await h({ messages: K.agentEndEmpty ? [] : [{ role: "assistant", content: [{ type: "text", text }], stopReason: "stop" }] });
|
||||
process.stdout.write(text + "\n");
|
||||
process.exit(K.exitCode || 0);
|
||||
};
|
||||
if (K.noModelCall) return end("no call");
|
||||
const models = JSON.parse(fs.readFileSync(path.join(agentDir, "models.json"), "utf8"));
|
||||
const base = models.providers[MODEL_PROVIDER].baseUrl;
|
||||
const sysFile = val("--system-prompt");
|
||||
const system = K.modelNoSystem || !sysFile ? undefined : fs.readFileSync(sysFile, "utf8");
|
||||
const tools = [{ name: "read", description: "read a file", input_schema: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } }];
|
||||
const messages = [{ role: "user", content: argv[argv.length - 1] }];
|
||||
const variants = {
|
||||
amPm: (p) => p.replace(/ (AM|PM)/g, "\u202F$1"),
|
||||
nfd: (p) => p.normalize("NFD"),
|
||||
curly: (p) => p.replace(/'/g, "\u2019"),
|
||||
nfdCurly: (p) => p.replace(/'/g, "\u2019").normalize("NFD"),
|
||||
};
|
||||
const resolveRead = (p) => {
|
||||
const abs = path.resolve(process.cwd(), p);
|
||||
if (fs.existsSync(abs)) return { file: abs, via: null };
|
||||
for (const [name, v] of Object.entries(variants)) if (fs.existsSync(v(abs))) return { file: v(abs), via: name };
|
||||
return { file: abs, via: null };
|
||||
};
|
||||
for (let turn = 0; ; turn++) {
|
||||
const res = await fetch(base + "/v1/messages", { method: "POST", headers: { "content-type": "application/json", "x-api-key": "stand-in" }, body: JSON.stringify({ model: val("--model"), max_tokens: 100, stream: false, ...(system === undefined ? {} : { system }), tools, messages }) });
|
||||
const reply = await res.json();
|
||||
if (K.modelHang === true ? turn === 0 : K.modelHang === turn) { setInterval(() => {}, 1000); return; }
|
||||
const use = (reply.content || []).find((c) => c.type === "tool_use");
|
||||
if (!use) return end((reply.content || []).map((c) => c.text || "").join(""));
|
||||
messages.push({ role: "assistant", content: reply.content });
|
||||
const { file, via } = resolveRead(use.input.path);
|
||||
let blocked = null;
|
||||
let threw = null;
|
||||
const skipFail = K.hookSkipFail !== undefined && (K.hookSkipFail === "outside" ? via === null && use.input.path.includes("outside") : via === K.hookSkipFail);
|
||||
if (!(K.skipHookSpelling && via === K.skipHookSpelling) && !skipFail) {
|
||||
for (const h of handlers.tool_call) {
|
||||
let r;
|
||||
try { r = await h({ toolName: use.name, toolCallId: use.id, input: use.input }); } catch (e) { threw = e.message; break; }
|
||||
if (r && r.block && !blocked) blocked = r.reason || "blocked";
|
||||
}
|
||||
}
|
||||
let result;
|
||||
if (skipFail || (K.readFails && threw === null && !blocked)) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: EACCES" };
|
||||
else if (threw !== null && !K.ignoreThrow) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: threw };
|
||||
else if (blocked && !K.ignoreBlock) result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: blocked };
|
||||
else {
|
||||
try { result = { type: "tool_result", tool_use_id: use.id, content: fs.readFileSync(file, "utf8") }; }
|
||||
catch (e) { result = { type: "tool_result", tool_use_id: use.id, is_error: true, content: "read failed: " + e.code }; }
|
||||
}
|
||||
messages.push({ role: "user", content: [result] });
|
||||
}
|
||||
}
|
||||
if (modelRun) {
|
||||
agentLoop().catch((e) => { process.stderr.write("stand-in model run: " + e.message + "\n"); process.exit(1); });
|
||||
} else if (rpc || print) {
|
||||
if (K.flood) {
|
||||
const chunk = Buffer.alloc(1024 * 1024, "x");
|
||||
for (let n = 0; n < 8; n++) for (let off = 0; off < chunk.length;) {
|
||||
@@ -225,9 +310,10 @@ if (rpc || print) {
|
||||
}
|
||||
}
|
||||
process.exit(0);
|
||||
} else {
|
||||
if (K.sessionOut) process.stdout.write(K.sessionOut);
|
||||
process.exit(0);
|
||||
}
|
||||
if (K.sessionOut) process.stdout.write(K.sessionOut);
|
||||
process.exit(0);
|
||||
`;
|
||||
|
||||
const quote = (text) => `'${String(text).replace(/'/g, "'\\''")}'`;
|
||||
@@ -248,13 +334,14 @@ export function piStandIn(knobs = {}, { node = process.execPath, form = "sh" } =
|
||||
`if (process.env.ARGV_OUT) { require("node:fs").writeFileSync(process.env.ARGV_OUT, process.argv.slice(1).map((a) => a + "\\n").join("")); process.stdout.write(${JSON.stringify(String(knobs.sessionOut ?? ""))}); process.exit(0); }`,
|
||||
`const K = ${JSON.stringify(knobs)};`,
|
||||
`const FLAGS = ${JSON.stringify(FIT_FLAGS)};`,
|
||||
`const MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};`,
|
||||
BODY,
|
||||
`})();`,
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
const version = knobs.versionHang ? "sleep 60; exit 0" : knobs.versionExit != null ? `exit ${Number(knobs.versionExit)}` : `printf '%s\\n' ${quote(knobs.version ?? "9.9.9")}; exit 0`;
|
||||
const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\n${BODY}`;
|
||||
const program = `"use strict";\nconst K = ${JSON.stringify(knobs)};\nconst FLAGS = ${JSON.stringify(FIT_FLAGS)};\nconst MODEL_PROVIDER = ${JSON.stringify(FIT_MODEL_ROUTE.provider)};\n${BODY}`;
|
||||
return [
|
||||
"#!/bin/sh",
|
||||
`[ "$1" = --version ] && { ${version}; }`,
|
||||
|
||||
@@ -32,3 +32,14 @@ export const envNames = Object.freeze(["MOSAIC_PI_BIN", "MOSAIC_PI_NO_MCP", "PI_
|
||||
// no Anthropic provider (pi-fit.test.mjs holds that); glm-5.3-flash is the model index.mjs has
|
||||
// documented limits for, and pi's own catalogue lists it under zai.
|
||||
export const FIT_ROUTE = Object.freeze({ provider: "zai", model: "glm-5.3-flash" });
|
||||
|
||||
// #1873 pi-unpin S6: the provider and model the fit check's scripted-model run names. Its models.json
|
||||
// points that provider at a stand-in on the sandbox's own loopback (packages/harness/src/
|
||||
// pi-fit-model.mjs), so the names are plain stand-ins: no real provider, and no Claude model, which
|
||||
// runs only on the claude-code harness (pi-fit.test.mjs holds that).
|
||||
export const FIT_MODEL_ROUTE = Object.freeze({ provider: "fit-model", model: "fit-model" });
|
||||
|
||||
// #1873 pi-unpin S6: the gate policy the scripted-model run's -e extension (packages/harness/src/
|
||||
// pi-extension.mjs) reads: a pi session with the read tool, no typed tools, in workspace. Built here
|
||||
// since the policy names this harness.
|
||||
export const fitPolicy = (workspace) => ({ harness: "pi", workspace, tools: ["read"], typed: [] });
|
||||
Reference in new issue
Block a user