From df2608530364a9a0ddc019124cc42abb8ea8cb33 Mon Sep 17 00:00:00 2001 From: Jason Woltje Date: Thu, 19 Nov 2026 15:51:29 -0600 Subject: [PATCH] fix(pi): pi-unpin S3 review: the control must show each planted kind and answer get_commands, the -e extension reads the system prompt so context and trusted project files are checked, settings and package sources are planted, help flags come from the option column, missing -e and the seal run in -p too, and the digest covers npm's lock files (#1873) --- packages/harness/README.md | 25 ++- packages/harness/src/pi-fit.mjs | 225 ++++++++++++++++++------- packages/harness/tests/pi-fit.test.mjs | 59 ++++++- packages/harness/tests/pi-stand-in.mjs | 92 +++++++--- packages/plugins/README.md | 5 +- packages/plugins/harness/pi/index.mjs | 6 +- 6 files changed, 310 insertions(+), 102 deletions(-) diff --git a/packages/harness/README.md b/packages/harness/README.md index ac9d590f5..dc7787413 100644 --- a/packages/harness/README.md +++ b/packages/harness/README.md @@ -30,16 +30,27 @@ Pi is not pinned (#1873 pi-unpin). Before a session starts, `preflight` and `env` run the fit check in `src/pi-fit.mjs` on the binary the module found. It runs that binary in a sandbox with the network unshared, an empty environment and a scratch home, and calls no model. It needs -`--help` to list every flag the adapter passes (with rpc mode and each -thinking level), a missing `-e` file to stop the start, and the planted -ambient sources (extensions, settings lists, `mcp.json` files, skills, -prompt templates, context files) to stay unloaded with the seal flags on. -A control run without the seal flags must load at least one of them, or -the check can't tell and refuses. Any failure refuses the launch with +`--help` to list, in its option column, every flag the adapter passes +(with rpc mode and each thinking level), and a missing `-e` file to stop +the start in rpc and in `-p`. The planted ambient sources must stay +unloaded with the seal flags on, in rpc and in `-p`: extensions, MCP +servers, skills and prompt templates (found in their directories, listed +in a `settings.json`, or in a local package a `settings.json` lists), +context files (`AGENTS.md`, `CLAUDE.md`, `SYSTEM.md`) and a trusted +project's `.pi/SYSTEM.md` and `.pi/APPEND_SYSTEM.md`. The project has a +saved trust, so only `--no-approve` keeps those two out. An `-e` +extension writes the system prompt it sees at `session_start`; the +sealed runs must show the `--system-prompt` file and none of the planted +files. A control run without the seal flags must show each kind of +source (an MCP server only when `--help` lists `--no-mcp`) and answer +`get_state` and `get_commands`, or the check can't tell and refuses. Pi +0.85.1 appends the agent directory's `APPEND_SYSTEM.md` whatever the +flags, so the check doesn't plant one. Any failure refuses the launch with `harness-feature-missing` and one line that names the binary by its label, for example `pi 2.0.0 (MOSAIC_PI_BIN) lacks --no-context-files`. When `--help` lists `--no-mcp`, the adapter passes it too. A pass is cached in -memory by a digest of the package's files. Lines 1, 2, 4 and 5 are not +memory by a digest of the package's files (and npm's lock files, for +the repository's copy). Lines 1, 2, 4 and 5 are not checked at start; a pi other than the repository's copy relies on them unproved until a scripted-model fit run exists (design section 5, S6). diff --git a/packages/harness/src/pi-fit.mjs b/packages/harness/src/pi-fit.mjs index 7022048ff..73cc700fc 100644 --- a/packages/harness/src/pi-fit.mjs +++ b/packages/harness/src/pi-fit.mjs @@ -1,23 +1,35 @@ // #1873 pi-unpin S3: the start-time fit check for a pi binary the harness module found. Nothing // here compares a version. The check runs the binary only inside a sandbox with the network // unshared, an empty environment, a scratch HOME and agent directory, and the found package -// mounted as a session mounts it. No model is called. It asks: +// mounted as a session mounts it. No model is called and no provider key is passed. It asks: // // --version a label for the refusal line and the record; not a gate -// --help lists every flag Mosaic passes, with rpc mode and each thinking level -// -e /missing a missing extension stops the start (exit non-zero) before any answer -// control with planted ambient sources and no seal flags, the binary answers get_state and -// loads at least one planted source, so the sealed run can tell -// sealed with the adapter's seal flags (and --no-mcp when --help lists it) and one -e -// extension, the binary answers get_state, loads that extension (a session's gate -// comes in that way) and loads none of the planted sources +// --help lists, in its option column, every flag Mosaic passes, with rpc mode and each +// thinking level +// control rpc, planted ambient sources, a saved trust for the project, no seal flags and +// one -e extension that writes the system prompt it sees at session_start. The +// binary answers get_state and get_commands, and shows each kind of planted source: +// an extension, a skill, a prompt template, a context file, a trusted project file +// (the trust --no-approve must undo), and an MCP server when --help lists --no-mcp. +// A kind it doesn't show could not be told apart from a sealed one. +// sealed rpc the adapter's seal flags (and --no-mcp when --help lists it), the -e extension and +// --system-prompt. The binary answers both, loads that extension (a session's gate +// comes in that way), uses the --system-prompt file, and shows no planted source. +// sealed -p the same in -p, the mode the adapter runs. Its -e extension must reach +// session_start, and no planted source may show. +// missing -e a missing -e file, in rpc and in -p, stops the start: exit non-zero, no answer, +// and no session_start for the -e extension given after it // -// A check that cannot tell refuses: a sandbox that fails, a run that times out, an answer that -// doesn't parse. A pass is cached in memory by a digest of the files that run (the package's -// package.json, every file under dist/bundle/, and every file in the directory tree that holds -// the binary's real file) and the mount layout. +// A check that cannot tell refuses: a sandbox that fails, a run that times out, an answer that is +// missing or doesn't parse, a kind of source the control doesn't show. A pass is cached in memory +// by a digest of the files that run (the package's package.json, every file under dist/bundle/, +// every file in the directory tree that holds the binary's real file, and, for a copy mounted +// with the node_modules it sits in, npm's lock files there) and the mount layout. // A refusal line names the found binary by its label (MOSAIC_PI_BIN, the pi on PATH, the // repository's pi), never a host path. +// +// Known limit: pi 0.85.1 appends the agent directory's APPEND_SYSTEM.md whatever the flags. The +// check doesn't plant one; a session's agent directory is the run's own. import { spawnSync } from "node:child_process"; import { createHash } from "node:crypto"; import { lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs"; @@ -62,40 +74,66 @@ export function piFit(found, { levels = [], bwrap = BWRAP, node = process.execPa function check(found, { levels, bwrap, node, scratch, who, digest }) { const run = (args, input = "") => runIn(found, { bwrap, node, scratch, args, input }); + const refuse = (why) => ({ ok: false, message: `${who(version)} ${why}` }); plant(scratch); const v = run(["--version"]); const version = v.status === 0 ? versionLabel(v.stdout) : null; const help = run(["--help"]); - if (help.status !== 0) return { ok: false, message: `${who(version)} can't be checked: --help failed` }; + if (help.status !== 0) return refuse("can't be checked: --help failed"); const lines = help.stdout.split("\n"); - const has = (flag) => lines.some((l) => flagIn(l, flag)); - const line = (flag) => lines.find((l) => flagIn(l, flag)) ?? ""; + const has = (flag) => lines.some((l) => optionsIn(l).includes(flag)); + const line = (flag) => lines.find((l) => optionsIn(l).includes(flag)) ?? ""; const missing = FIT_FLAGS.filter((f) => !has(f)); if (has("--mode") && !word(line("--mode"), "rpc")) missing.push("--mode rpc"); if (has("--thinking")) for (const l of levels) if (!word(line("--thinking"), l)) missing.push(`--thinking ${l}`); - if (missing.length) return { ok: false, message: `${who(version)} lacks ${missing.join(", ")}` }; + if (missing.length) return refuse(`lacks ${missing.join(", ")}`); const noMcp = has("--no-mcp"); - const approve = has("--approve"); const ask = '{"id":"fit-state","type":"get_state"}\n{"id":"fit-commands","type":"get_commands"}\n'; - plant(scratch); - const missingExt = run(["-e", join(scratch, "missing", "extension.ts"), "--mode", "rpc", ...SEAL], ask); - if (missingExt.timedOut) return { ok: false, message: `${who(version)} can't be checked: a start with a missing -e extension timed out` }; - if (missingExt.status === 0 || answered(missingExt.stdout).state) return { ok: false, message: `${who(version)} starts with a missing -e extension` }; + const explicit = ["-e", join(scratch, "explicit", "fit.ts")]; + const sealed = [...SEAL, ...(noMcp ? ["--no-mcp"] : []), ...explicit, "--system-prompt", join(scratch, "prompt.md")]; + // -p takes a message; offline with no key, pi stops before any model call. + const print = ["-p", ...sealed, "fit check"]; plant(scratch); - const control = run(["--mode", "rpc", "--no-session", "--offline", ...(approve ? ["--approve"] : [])], ask); + const control = run(["--mode", "rpc", "--no-session", "--offline", ...explicit], ask); + if (control.timedOut) return refuse("can't be checked: the start without the seal flags timed out"); const controlAnswer = answered(control.stdout); - if (!controlAnswer.state) return { ok: false, message: `${who(version)} can't be checked: it didn't answer get_state` }; - if (seen(scratch, controlAnswer).length === 0) return { ok: false, message: `${who(version)} can't be checked: it loaded none of the planted sources without the seal flags` }; + if (!controlAnswer.state) return refuse("can't be checked: it didn't answer get_state"); + if (!controlAnswer.commands) return refuse("can't be checked: it didn't answer get_commands"); + if (!marked(scratch, EXPLICIT) || prompted(scratch) === null) return refuse("can't be checked: its -e extension didn't see the system prompt without the seal flags"); + const shown = new Set(seen(scratch, controlAnswer).map((s) => s.kind)); + const unshown = KINDS.filter((k) => (k !== "MCP server" || noMcp) && !shown.has(k)); + if (unshown.length) return refuse(`can't be checked: it didn't load a planted ${unshown.join(", ")} without the seal flags`); plant(scratch); - const sealed = run(["--mode", "rpc", ...SEAL, ...(noMcp ? ["--no-mcp"] : []), "-e", join(scratch, "explicit", "fit.ts"), "--system-prompt", join(scratch, "prompt.md")], ask); - const sealedAnswer = answered(sealed.stdout); - if (!sealedAnswer.state) return { ok: false, message: `${who(version)} can't be checked: it didn't answer get_state with the seal flags on` }; - if (!marked(scratch, EXPLICIT)) return { ok: false, message: `${who(version)} didn't load an -e extension with the seal flags on` }; - const leaked = seen(scratch, sealedAnswer); - if (leaked.length) return { ok: false, message: `${who(version)} loaded a planted ${leaked.join(", ")} with the seal flags on` }; + const rpc = run(["--mode", "rpc", ...sealed], ask); + if (rpc.timedOut) return refuse("can't be checked: the start with the seal flags timed out"); + const rpcAnswer = answered(rpc.stdout); + if (!rpcAnswer.state) return refuse("can't be checked: it didn't answer get_state with the seal flags on"); + if (!rpcAnswer.commands) return refuse("can't be checked: it didn't answer get_commands with the seal flags on"); + if (!marked(scratch, EXPLICIT)) return refuse("didn't load an -e extension with the seal flags on"); + const rpcPrompt = prompted(scratch); + if (rpcPrompt === null) return refuse("can't be checked: its -e extension didn't see the system prompt with the seal flags on"); + if (!rpcPrompt.includes(GIVEN)) return refuse("didn't use the --system-prompt file with the seal flags on"); + let leaked = seen(scratch, rpcAnswer); + if (leaked.length) return refuse(`loaded a planted ${leaked.map((s) => s.source).join(", ")} with the seal flags on`); + + plant(scratch); + const p = run(print); + if (p.timedOut) return refuse("can't be checked: a -p start with the seal flags timed out"); + if (prompted(scratch) === null) return refuse("can't be checked: a -p start with the seal flags didn't run its -e extension"); + leaked = seen(scratch, { commands: [] }); + if (leaked.length) return refuse(`loaded a planted ${leaked.map((s) => s.source).join(", ")} with the seal flags on in -p`); + + // A missing -e file stops the start. pi 0.85.1 loads the -e files in order and still loads the + // one after the missing one, then exits 1 before session_start, so only session_start tells. + for (const [mode, args, input] of [["rpc", ["--mode", "rpc"], ask], ["-p", ["-p"], ""]]) { + plant(scratch); + const r = run([...args, ...SEAL, "-e", join(scratch, "missing", "extension.ts"), ...explicit, "--system-prompt", join(scratch, "prompt.md"), ...(mode === "-p" ? ["fit check"] : [])], input); + if (r.timedOut) return refuse(`can't be checked: a ${mode} start with a missing -e extension timed out`); + if (r.status === 0 || answered(r.stdout).state || prompted(scratch) !== null) return refuse("starts with a missing -e extension"); + } return { ok: true, version, noMcp, digest }; } @@ -113,13 +151,14 @@ function runIn(found, { bwrap, node, scratch, args, input }) { return { status: r.status, stdout: typeof r.stdout === "string" ? r.stdout : "", timedOut: r.error?.code === "ETIMEDOUT" || r.signal === "SIGKILL" }; } -// One flag in one help line: the flag as its own token, at the line's start or after a space or -// a comma, and followed by the end, a space, a comma, "=" or "<". -function flagIn(line, flag) { - const at = (i) => i === 0 || line[i - 1] === " " || line[i - 1] === "," || line[i - 1] === "\t"; - const after = (i) => i === line.length || " ,=<\t".includes(line[i]); - for (let i = line.indexOf(flag); i !== -1; i = line.indexOf(flag, i + 1)) if (at(i) && after(i + flag.length)) return true; - return false; +// The options one help line lists in its option column: a line that starts with "-" (after its +// indent), up to the first run of two spaces or a tab, split at commas; the first word of each +// part. " --extension, -e Load an extension" lists --extension and -e. A flag named in a +// sentence is not listed. +function optionsIn(line) { + const t = line.trim(); + if (!t.startsWith("-")) return []; + return t.split(/ {2,}|\t/)[0].split(",").map((part) => part.trim().split(/[ =<[]/)[0]).filter((o) => /^--?[A-Za-z0-9]/.test(o)); } const word = (line, w) => line.split(/[^A-Za-z0-9-]+/).includes(w); @@ -131,83 +170,133 @@ function versionLabel(stdout) { } // The rpc answers in the output: whether get_state succeeded, and the command names get_commands -// listed (an empty list when it failed). +// listed, or null when it didn't answer with a { commands: [...] } list. function answered(stdout) { let state = false; - const commands = []; + let commands = null; for (const l of String(stdout).split("\n")) { let m; try { m = JSON.parse(l); } catch { continue; } if (!m || m.type !== "response" || m.success !== true) continue; if (m.id === "fit-state" && m.command === "get_state" && m.data && typeof m.data === "object") state = true; if (m.id === "fit-commands" && m.command === "get_commands" && Array.isArray(m.data?.commands)) { - for (const c of m.data.commands) if (typeof c?.name === "string") commands.push(c.name); + commands = m.data.commands.filter((c) => typeof c?.name === "string").map((c) => c.name); } } return { state, commands }; } -// The planted sources. Each extension and each MCP server touches its own marker; each skill and -// prompt template has a name get_commands would list. Context and system prompt files are planted -// too, so nothing ambient that a session could meet is missing from the tree. -// "default agent ..." sits in $HOME/.pi/agent, where pi looks when PI_CODING_AGENT_DIR is unset. -const MARKED = ["user extension", "user settings extension", "default agent extension", "project extension", "project settings extension", "user mcp.json", "default agent mcp.json", "project mcp.json", "project .mcp.json"]; -const NAMED = { "fit-user-skill": "user skill", "fit-default-skill": "default agent skill", "fit-project-skill": "project skill", "fit-agents-skill": "project .agents skill", "fit-home-skill": "home .agents skill", "fit-user-prompt": "user prompt template", "fit-default-prompt": "default agent prompt template", "fit-project-prompt": "project prompt template" }; -// The -e extension the sealed run passes, which must load. +// The planted sources, each with the kind the control must show. Each extension and each MCP +// server touches its own marker; each skill and prompt template has a name get_commands lists; +// each context and system prompt file holds a word the -e extension's view of the system prompt +// shows. "default agent ..." sits in $HOME/.pi/agent, where pi looks when PI_CODING_AGENT_DIR is +// unset. "settings" sources are listed by a settings.json; "package" sources come from a local +// package a settings.json lists. "parent" is a directory above the project. +const KINDS = ["extension", "MCP server", "skill", "prompt template", "context file", "trusted project file"]; +const MARKED = { + "user extension": "extension", "user settings extension": "extension", "user package extension": "extension", "default agent extension": "extension", + "project extension": "extension", "project settings extension": "extension", "project package extension": "extension", + "user mcp.json": "MCP server", "default agent mcp.json": "MCP server", "project mcp.json": "MCP server", "project .mcp.json": "MCP server", +}; +const NAMED = { + "fit-user-skill": ["user skill", "skill"], "fit-user-set-skill": ["user settings skill", "skill"], "fit-user-pkg-skill": ["user package skill", "skill"], + "fit-default-skill": ["default agent skill", "skill"], "fit-project-skill": ["project skill", "skill"], "fit-project-set-skill": ["project settings skill", "skill"], + "fit-project-pkg-skill": ["project package skill", "skill"], "fit-agents-skill": ["project .agents skill", "skill"], "fit-home-skill": ["home .agents skill", "skill"], + "fit-user-prompt": ["user prompt template", "prompt template"], "fit-user-set-prompt": ["user settings prompt template", "prompt template"], + "fit-user-pkg-prompt": ["user package prompt template", "prompt template"], "fit-default-prompt": ["default agent prompt template", "prompt template"], + "fit-project-prompt": ["project prompt template", "prompt template"], "fit-project-set-prompt": ["project settings prompt template", "prompt template"], + "fit-project-pkg-prompt": ["project package prompt template", "prompt template"], +}; +// Planted file (under the scratch directory), the word it holds, the source, the kind. A project +// SYSTEM.md or APPEND_SYSTEM.md loads only in a trusted project: the saved trust makes it trusted, +// and only --no-approve undoes that with the seal flags on. +const TEXTS = [ + ["agent/SYSTEM.md", "FITUSERSYSTEM", "user SYSTEM.md", "context file"], + ["agent/AGENTS.md", "FITUSERAGENTS", "user AGENTS.md", "context file"], + ["home/.pi/agent/AGENTS.md", "FITDEFAULTAGENTS", "default agent AGENTS.md", "context file"], + ["project/.pi/SYSTEM.md", "FITPROJECTSYSTEM", "project SYSTEM.md", "trusted project file"], + ["project/.pi/APPEND_SYSTEM.md", "FITPROJECTAPPEND", "project APPEND_SYSTEM.md", "trusted project file"], + ["project/AGENTS.md", "FITPROJECTAGENTS", "project AGENTS.md", "context file"], + ["project/CLAUDE.md", "FITPROJECTCLAUDE", "project CLAUDE.md", "context file"], + ["CLAUDE.md", "FITPARENTCLAUDE", "parent CLAUDE.md", "context file"], +]; +// The --system-prompt file's word, which the sealed runs must show. +const GIVEN = "FITGIVENPROMPT"; +// The -e extension the control and sealed runs pass, which must load. const EXPLICIT = "explicit extension"; const markerName = (source) => source.replace(/[^a-z]/g, "-"); const marked = (scratch, source) => { try { return readdirSync(join(scratch, "marks")).includes(markerName(source)); } catch { return false; } }; +// The system prompt the -e extension saw at session_start, or null when it wrote none. +const prompted = (scratch) => { try { return readFileSync(join(scratch, "marks", "system-prompt"), "utf8"); } catch { return null; } }; function plant(scratch) { - for (const d of ["home", "agent", "project", "marks", "tmp", "missing", "explicit"]) rmSync(join(scratch, d), { recursive: true, force: true }); + for (const d of ["home", "agent", "project", "marks", "tmp", "missing", "explicit", "listed"]) rmSync(join(scratch, d), { recursive: true, force: true }); const marks = join(scratch, "marks"); const put = (path, text) => { mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, text); }; const ext = (source) => `import { writeFileSync } from "node:fs";\nexport default function (): void { writeFileSync(${JSON.stringify(join(marks, markerName(source)))}, ""); }\n`; const mcp = (source) => JSON.stringify({ mcpServers: { fit: { command: "/bin/sh", args: ["-c", `: > ${join(marks, markerName(source))}`] } } }); const skill = (name) => `---\nname: ${name}\ndescription: a planted skill\n---\nA planted skill.\n`; + const prompt = "A planted prompt template.\n"; const agent = join(scratch, "agent"); const project = join(scratch, "project"); mkdirSync(marks, { recursive: true }); mkdirSync(join(scratch, "tmp"), { recursive: true }); + mkdirSync(project, { recursive: true }); + // Each settings.json lists an extension, a skill directory, a prompt template and a local + // package holding one of each. + const listed = (who) => { + const dir = join(scratch, "listed", who); + put(join(dir, "fit.ts"), ext(`${who} settings extension`)); + put(join(dir, `fit-${who}-set-skill`, "SKILL.md"), skill(`fit-${who}-set-skill`)); + put(join(dir, `fit-${who}-set-prompt.md`), prompt); + const pkg = join(dir, "package"); + put(join(pkg, "package.json"), JSON.stringify({ name: `fit-${who}-package`, version: "1.0.0" })); + put(join(pkg, "extensions", "fit.ts"), ext(`${who} package extension`)); + put(join(pkg, "skills", `fit-${who}-pkg-skill`, "SKILL.md"), skill(`fit-${who}-pkg-skill`)); + put(join(pkg, "prompts", `fit-${who}-pkg-prompt.md`), prompt); + return { extensions: [join(dir, "fit.ts")], skills: [join(dir, `fit-${who}-set-skill`)], prompts: [join(dir, `fit-${who}-set-prompt.md`)], packages: [pkg] }; + }; + // A saved trust for the project, so the control loads its trust-gated files without --approve. + put(join(agent, "settings.json"), JSON.stringify({ ...listed("user"), defaultProjectTrust: "always" })); + put(join(agent, "trust.json"), JSON.stringify({ [realpathSync(project)]: true })); put(join(agent, "extensions", "fit.ts"), ext("user extension")); - put(join(agent, "listed", "fit.ts"), ext("user settings extension")); - put(join(agent, "settings.json"), JSON.stringify({ extensions: [join(agent, "listed", "fit.ts")] })); put(join(agent, "mcp.json"), mcp("user mcp.json")); put(join(agent, "skills", "fit-user-skill", "SKILL.md"), skill("fit-user-skill")); - put(join(agent, "prompts", "fit-user-prompt.md"), "A planted prompt template.\n"); - for (const f of ["SYSTEM.md", "APPEND_SYSTEM.md", "AGENTS.md"]) put(join(agent, f), "A planted file.\n"); + put(join(agent, "prompts", "fit-user-prompt.md"), prompt); + put(join(project, ".pi", "settings.json"), JSON.stringify(listed("project"))); put(join(project, ".pi", "extensions", "fit.ts"), ext("project extension")); - put(join(project, ".pi", "listed", "fit.ts"), ext("project settings extension")); - put(join(project, ".pi", "settings.json"), JSON.stringify({ extensions: [join(project, ".pi", "listed", "fit.ts")] })); put(join(project, ".pi", "mcp.json"), mcp("project mcp.json")); put(join(project, ".mcp.json"), mcp("project .mcp.json")); put(join(project, ".pi", "skills", "fit-project-skill", "SKILL.md"), skill("fit-project-skill")); put(join(project, ".agents", "skills", "fit-agents-skill", "SKILL.md"), skill("fit-agents-skill")); - put(join(project, ".pi", "prompts", "fit-project-prompt.md"), "A planted prompt template.\n"); - for (const f of [".pi/SYSTEM.md", ".pi/APPEND_SYSTEM.md", "AGENTS.md", "CLAUDE.md"]) put(join(project, f), "A planted file.\n"); + put(join(project, ".pi", "prompts", "fit-project-prompt.md"), prompt); put(join(scratch, "home", ".agents", "skills", "fit-home-skill", "SKILL.md"), skill("fit-home-skill")); const fallback = join(scratch, "home", ".pi", "agent"); put(join(fallback, "extensions", "fit.ts"), ext("default agent extension")); put(join(fallback, "mcp.json"), mcp("default agent mcp.json")); put(join(fallback, "skills", "fit-default-skill", "SKILL.md"), skill("fit-default-skill")); - put(join(fallback, "prompts", "fit-default-prompt.md"), "A planted prompt template.\n"); - put(join(scratch, "explicit", "fit.ts"), ext(EXPLICIT)); - put(join(scratch, "prompt.md"), "A fit check.\n"); + put(join(fallback, "prompts", "fit-default-prompt.md"), prompt); + for (const [file, text] of TEXTS) put(join(scratch, file), `${text}\n`); + put(join(scratch, "explicit", "fit.ts"), `import { writeFileSync } from "node:fs";\nexport default function (pi: any): void {\n writeFileSync(${JSON.stringify(join(marks, markerName(EXPLICIT)))}, "");\n pi.on("session_start", (_event: any, ctx: any) => { writeFileSync(${JSON.stringify(join(marks, "system-prompt"))}, ctx.getSystemPrompt()); });\n}\n`); + put(join(scratch, "prompt.md"), `${GIVEN}\n`); } -// The planted sources a run loaded: markers present, and planted names get_commands listed. +// The planted sources a run loaded, as { source, kind }: markers present, planted names +// get_commands listed, and planted words in the system prompt the -e extension saw. function seen(scratch, answer) { let marks = []; try { marks = readdirSync(join(scratch, "marks")); } catch { marks = []; } - const out = MARKED.filter((s) => marks.includes(markerName(s))); - for (const [name, source] of Object.entries(NAMED)) if (answer.commands.some((c) => c === name || c.endsWith(`:${name}`))) out.push(source); + const out = Object.entries(MARKED).filter(([s]) => marks.includes(markerName(s))).map(([source, kind]) => ({ source, kind })); + for (const [name, [source, kind]] of Object.entries(NAMED)) if ((answer.commands ?? []).some((c) => c === name || c.endsWith(`:${name}`))) out.push({ source, kind }); + const text = prompted(scratch) ?? ""; + for (const [, word, source, kind] of TEXTS) if (text.includes(word)) out.push({ source, kind }); return out; } // sha256 over the sorted (path, sha256) list of the package's package.json, every regular file -// under dist/bundle/ (design section 4), and every regular file in the directory tree that holds -// the binary's real file, so a bin outside dist/bundle/ can't import a file the digest skips. A -// symlink there is read through. +// under dist/bundle/ (design section 4), every regular file in the directory tree that holds the +// binary's real file, so a bin outside dist/bundle/ can't import a file the digest skips, and npm's +// lock files for a copy mounted with its node_modules. A symlink there is read through. function filesDigest(found) { const pkg = found.pkg; const bin = realpathSync(found.bin); @@ -225,6 +314,12 @@ function filesDigest(found) { walk(join(pkg, "dist", "bundle")); const binDir = dirname(bin); if (binDir !== pkg && binDir.startsWith(`${pkg}/`)) walk(binDir); + // A copy mounted with the node_modules it sits in imports packages beside it: npm's lock files + // there stand for them. + if (found.root !== pkg) for (const f of [join(found.root, ".package-lock.json"), join(dirname(found.root), "package-lock.json")]) { + try { lstatSync(f); } catch (e) { if (e.code === "ENOENT") continue; throw e; } + files.add(f); + } const h = createHash("sha256"); for (const f of [...files].sort()) { h.update(relative(pkg, f)).update("\0").update(createHash("sha256").update(readFileSync(f)).digest("hex")).update("\n"); diff --git a/packages/harness/tests/pi-fit.test.mjs b/packages/harness/tests/pi-fit.test.mjs index b6ec8321e..2059ba9ec 100644 --- a/packages/harness/tests/pi-fit.test.mjs +++ b/packages/harness/tests/pi-fit.test.mjs @@ -73,6 +73,8 @@ test("pi-unpin S3: a stand-in that exits 0 on a missing -e is refused", SKIP, t const problem = pi.preflight({ env: { ...CREDENTIAL, ...s.env } }); assert.deepEqual(problem, { code: "harness-feature-missing", message: "pi 2.0.0 (MOSAIC_PI_BIN) starts with a missing -e extension" }); noPath(problem.message, s.paths); + // Review finding 5: the adapter runs -p, so a pi that starts anyway in either mode is refused. + for (const mode of ["print", "rpc"]) assert.equal(fit(standIn(t, { missingExtOk: mode }).env).message, "pi 2.0.0 (MOSAIC_PI_BIN) starts with a missing -e extension", mode); }); test("pi-unpin S3: a stand-in that touches the planted MCP marker under the seal flags is refused", SKIP, t => { @@ -82,9 +84,13 @@ test("pi-unpin S3: a stand-in that touches the planted MCP marker under the seal noPath(problem.message, s.paths); // Each other kind of planted source is caught the same way. const lines = { - extensions: "loaded a planted user extension, user settings extension, project extension, project settings extension with the seal flags on", - skills: "loaded a planted user skill, project skill, project .agents skill, home .agents skill with the seal flags on", - prompts: "loaded a planted user prompt template, project prompt template with the seal flags on", + extensions: "loaded a planted user extension, user settings extension, user package extension, project extension, project settings extension, project package extension with the seal flags on", + skills: "loaded a planted user skill, user settings skill, user package skill, project skill, project settings skill, project package skill, project .agents skill, home .agents skill with the seal flags on", + prompts: "loaded a planted user prompt template, user settings prompt template, user package prompt template, project prompt template, project settings prompt template, project package prompt template with the seal flags on", + // Review finding 2: the context files reach the system prompt an -e extension sees. + context: "loaded a planted user AGENTS.md, project AGENTS.md, parent CLAUDE.md with the seal flags on", + // A saved trust makes the project trusted, so only --no-approve keeps its APPEND_SYSTEM.md out. + approve: "loaded a planted project APPEND_SYSTEM.md with the seal flags on", }; for (const [kind, line] of Object.entries(lines)) assert.equal(fit(standIn(t, { sealLeak: [kind] }).env).message, `pi 2.0.0 (MOSAIC_PI_BIN) ${line}`, kind); }); @@ -92,10 +98,36 @@ test("pi-unpin S3: a stand-in that touches the planted MCP marker under the seal test("pi-unpin S3: a stand-in that ignores the planted sources even without the seal flags fails the positive control and is refused as can't tell", SKIP, t => { const s = standIn(t, { ignoreAll: true }); const problem = pi.preflight({ env: { ...CREDENTIAL, ...s.env } }); - assert.deepEqual(problem, { code: "harness-feature-missing", message: "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it loaded none of the planted sources without the seal flags" }); + assert.deepEqual(problem, { code: "harness-feature-missing", message: "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't load a planted extension, skill, prompt template, context file, trusted project file without the seal flags" }); noPath(problem.message, s.paths); }); +// Review finding 1: one planted source seen in the control does not prove the others can be seen. +test("pi-unpin S3 review: a get_commands answer that fails or changes shape is refused as can't tell, even when skills and prompts leak", SKIP, t => { + for (const commands of ["fail", "list"]) { + const s = standIn(t, { commands, sealLeak: ["skills", "prompts"] }); + assert.equal(fit(s.env).message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't answer get_commands", commands); + } + // MCP is a kind the control must show only when --help lists --no-mcp. + assert.equal(fit(standIn(t, { extraFlags: ["--no-mcp"], ignoreAll: true }).env).message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't load a planted extension, MCP server, skill, prompt template, context file, trusted project file without the seal flags"); +}); + +// Review finding 2: the sealed runs read the system prompt through the -e extension, so a pi that +// drops the --system-prompt file, or never runs session_start in -p, can't pass. +test("pi-unpin S3 review: a stand-in that drops the --system-prompt file, or runs no session_start in -p, is refused", SKIP, t => { + assert.equal(fit(standIn(t, { ignoreSystemPrompt: true }).env).message, "pi 2.0.0 (MOSAIC_PI_BIN) didn't use the --system-prompt file with the seal flags on"); + assert.equal(fit(standIn(t, { printNoStart: true }).env).message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: a -p start with the seal flags didn't run its -e extension"); +}); + +// Review finding 3: only the option column counts, not a flag named in a sentence. +test("pi-unpin S3 review: a flag named only in help prose is missing", SKIP, t => { + const s = standIn(t, { dropFlags: ["--no-context-files", "--no-approve"], helpProse: ["Removed: --no-context-files is gone in this release", "Note: --no-approve was removed", " -x, --no-context-files-old a different flag"] }); + assert.equal(fit(s.env).message, "pi 2.0.0 (MOSAIC_PI_BIN) lacks --no-context-files, --no-approve"); + // An alias column counts: "-e" listed as "--extension, -e ". + const aliased = standIn(t, { dropFlags: ["-e"], helpProse: [" --extension, -e Load an extension file"] }); + assert.equal(fit(aliased.env).ok, true, fit(aliased.env).message); +}); + test("pi-unpin S3: a stand-in that ignores an -e extension with the seal flags on is refused, since a session's gate comes in that way", SKIP, t => { const s = standIn(t, { dropExplicit: true }); const problem = pi.preflight({ env: { ...CREDENTIAL, ...s.env } }); @@ -123,6 +155,25 @@ test("pi-unpin S3: a pass is cached by the files that run; a changed file is che assert.equal(fit(s.env).ok, true, "the refusal was not kept"); }); +// Review finding 7: a copy mounted with the node_modules it sits in (the repository's copy) is +// checked again when npm's lock files there change, since it imports packages beside it. +test("pi-unpin S3 review: for a copy mounted with its node_modules, a changed lock file is checked again", SKIP, t => { + const s = standIn(t, {}); + const nodeModules = join(s.root, "prefix", "lib", "node_modules"); + const found = { ...locate(s.env), root: nodeModules, dest: null }; + const lock = join(nodeModules, ".package-lock.json"); + writeFileSync(lock, "{}"); + const first = piFit(found, { levels: LEVELS }); + assert.equal(first.ok, true, first.message); + assert.equal(piFit(found, { levels: LEVELS }), first); + writeFileSync(lock, '{"packages":{}}'); + const again = piFit(found, { levels: LEVELS }); + assert.equal(again.ok, true, again.message); + assert.notEqual(again, first, "a changed .package-lock.json is checked again"); + writeFileSync(join(s.root, "prefix", "lib", "package-lock.json"), "{}"); + assert.notEqual(piFit(found, { levels: LEVELS }), again, "a changed package-lock.json beside node_modules is checked again"); +}); + test("pi-unpin S3: when --help lists --no-mcp the session env asks the adapter to pass it", SKIP, t => { const s = standIn(t, { extraFlags: ["--no-mcp"] }); assert.equal(fit(s.env).noMcp, true); diff --git a/packages/harness/tests/pi-stand-in.mjs b/packages/harness/tests/pi-stand-in.mjs index 50d928038..374132140 100644 --- a/packages/harness/tests/pi-stand-in.mjs +++ b/packages/harness/tests/pi-stand-in.mjs @@ -7,14 +7,26 @@ // $ARGV_OUT set: a session. Writes its argv there, prints `sessionOut`, exits 0. This part is // sh, so it runs in a sandbox that holds no node. // otherwise runs node (the absolute path, since the fit check gives it no PATH) on the rest: -// --help lists every flag the fit check wants, less `dropFlags`, plus `extraFlags` -// -e a file that doesn't exist stops it with exit 1, unless `missingExtOk`; one that -// exists is loaded even with --no-extensions, unless `dropExplicit` -// --mode rpc loads the planted sources the way pi finds them, honouring the seal flags, -// then answers get_state and get_commands. `ignoreAll` loads nothing. -// `sealLeak` lists kinds (extensions, mcp, skills, prompts) loaded even when -// their seal flag is on. +// --help lists every flag the fit check wants, less `dropFlags`, plus `extraFlags`, then +// the `helpProse` lines as they are +// -e a file that doesn't exist stops it with exit 1, unless `missingExtOk` is true or +// names the mode ("rpc" or "print") it starts in anyway +// --mode rpc loads the planted sources the way pi 0.85.1 finds them, honouring the seal flags, +// or -p runs each -e extension's session_start (one that writes ctx.getSystemPrompt() +// gets the prompt this stand-in builds), then answers get_state and get_commands +// (rpc), or prints `sessionOut` and exits 0 (-p). +// An -e file that exists loads even with --no-extensions, unless `dropExplicit` +// (then it loads only without --no-extensions). `printNoStart`: -p runs no +// session_start. +// `ignoreAll` loads no planted source. `sealLeak` lists kinds (extensions, mcp, +// skills, prompts, context, approve) loaded even when their seal flag is on. +// `ignoreSystemPrompt` drops the --system-prompt file. `commands` "fail" answers +// get_commands with success false; "list" answers with a bare list. // otherwise a session with no $ARGV_OUT: prints `sessionOut` and exits 0. +// +// Trust, as pi 0.85.1 has it: the project is trusted with --approve, or with a saved trust and +// no --no-approve. Only the project SYSTEM.md and APPEND_SYSTEM.md wait on it here. The user +// APPEND_SYSTEM.md is appended whatever the flags, as 0.85.1 does when the project has none. import { FIT_FLAGS } from "../src/pi-fit.mjs"; const BODY = String.raw` @@ -34,34 +46,49 @@ if (argv[0] === "--help") { else lines.push(" " + f); } for (const f of K.extraFlags || []) lines.push(" " + f); + for (const l of K.helpProse || []) lines.push(l); process.stdout.write(lines.join("\n") + "\n"); process.exit(0); } +const rpc = val("--mode") === "rpc"; +const print = !rpc && has("-p"); +const startsAnyway = K.missingExtOk === true || (rpc && K.missingExtOk === "rpc") || (print && K.missingExtOk === "print"); for (let i = 0; i < argv.length; i++) { - if (argv[i] === "-e" && !fs.existsSync(argv[i + 1]) && !K.missingExtOk) { + if (argv[i] === "-e" && !fs.existsSync(argv[i + 1]) && !startsAnyway) { process.stderr.write("Failed to load extension: it does not exist\n"); process.exit(1); } } -if (val("--mode") === "rpc") { +if (rpc || print) { const leak = new Set(K.sealLeak || []); const on = (kind, flag) => !K.ignoreAll && (!has(flag) || leak.has(kind)); const agent = process.env.PI_CODING_AGENT_DIR || path.join(process.env.HOME || "/", ".pi", "agent"); const cwd = process.cwd(); const list = (dir, test) => { try { return fs.readdirSync(dir).filter(test).map((n) => path.join(dir, n)); } catch { return []; } }; const json = (file) => { try { return JSON.parse(fs.readFileSync(file, "utf8")); } catch { return null; } }; + const read = (file) => { try { return fs.readFileSync(file, "utf8"); } catch { return null; } }; + const settings = [json(path.join(agent, "settings.json")), json(path.join(cwd, ".pi", "settings.json"))].filter(Boolean); + const trust = json(path.join(agent, "trust.json")) || {}; + const saved = settings.some((s) => s.defaultProjectTrust === "always") || trust[cwd] === true; + const trusted = !K.ignoreAll && (has("--approve") || (saved && (!has("--no-approve") || leak.has("approve")))); const commands = []; - // Each planted extension writes one marker; do what it would do on load. + // A planted extension writes one marker on load; an -e extension may also write the system + // prompt at session_start. Do what each would do. + const starts = []; const load = (f) => { - let text = ""; - try { text = fs.readFileSync(f, "utf8"); } catch { return; } - const m = text.match(/writeFileSync\(("(?:[^"\\]|\\.)*")/); - if (m) fs.writeFileSync(JSON.parse(m[1]), ""); + const text = read(f); + if (text === null) return; + for (const m of text.matchAll(/writeFileSync\(("(?:[^"\\]|\\.)*")(, ctx\.getSystemPrompt\(\))?/g)) { + if (m[2]) starts.push(JSON.parse(m[1])); + else fs.writeFileSync(JSON.parse(m[1]), ""); + } }; - if (!K.dropExplicit) for (let i = 0; i < argv.length; i++) if (argv[i] === "-e") load(argv[i + 1]); + if (!(K.dropExplicit && has("--no-extensions"))) for (let i = 0; i < argv.length; i++) if (argv[i] === "-e") load(argv[i + 1]); + const packages = settings.flatMap((s) => (s.packages || []).map((p) => typeof p === "string" ? p : p.source)); if (on("extensions", "--no-extensions")) { const files = [...list(path.join(agent, "extensions"), (n) => n.endsWith(".ts")), ...list(path.join(cwd, ".pi", "extensions"), (n) => n.endsWith(".ts"))]; - for (const s of [path.join(agent, "settings.json"), path.join(cwd, ".pi", "settings.json")]) files.push(...(json(s)?.extensions || [])); + for (const s of settings) files.push(...(s.extensions || [])); + for (const p of packages) files.push(...list(path.join(p, "extensions"), (n) => n.endsWith(".ts"))); for (const f of files) load(f); } if (on("mcp", "--no-extensions") && (!has("--no-mcp") || leak.has("mcp"))) { @@ -71,14 +98,30 @@ if (val("--mode") === "rpc") { } if (on("skills", "--no-skills")) { const home = process.env.HOME || "/"; - for (const d of [path.join(agent, "skills"), path.join(cwd, ".pi", "skills"), path.join(cwd, ".agents", "skills"), path.join(home, ".agents", "skills")]) { - for (const p of list(d, () => true)) commands.push({ name: "skill:" + path.basename(p), source: "skill" }); - } + const dirs = [path.join(agent, "skills"), path.join(cwd, ".pi", "skills"), path.join(cwd, ".agents", "skills"), path.join(home, ".agents", "skills"), ...packages.map((p) => path.join(p, "skills"))]; + for (const d of dirs) for (const p of list(d, () => true)) commands.push({ name: "skill:" + path.basename(p), source: "skill" }); + for (const s of settings) for (const p of s.skills || []) commands.push({ name: "skill:" + path.basename(p), source: "skill" }); } if (on("prompts", "--no-prompt-templates")) { - for (const d of [path.join(agent, "prompts"), path.join(cwd, ".pi", "prompts")]) { - for (const p of list(d, (n) => n.endsWith(".md"))) commands.push({ name: path.basename(p, ".md"), source: "prompt" }); - } + const dirs = [path.join(agent, "prompts"), path.join(cwd, ".pi", "prompts"), ...packages.map((p) => path.join(p, "prompts"))]; + for (const d of dirs) for (const p of list(d, (n) => n.endsWith(".md"))) commands.push({ name: path.basename(p, ".md"), source: "prompt" }); + for (const s of settings) for (const p of s.prompts || []) commands.push({ name: path.basename(p, ".md"), source: "prompt" }); + } + // The system prompt: the --system-prompt file, else SYSTEM.md; then APPEND_SYSTEM.md; then the + // context files, from the agent directory and from / down to the working directory. + const given = K.ignoreSystemPrompt ? null : val("--system-prompt"); + const own = (name) => (trusted && read(path.join(cwd, ".pi", name))) || (!K.ignoreAll && read(path.join(agent, name))) || ""; + const parts = [given ? (read(given) || "") : own("SYSTEM.md"), own("APPEND_SYSTEM.md")]; + if (!K.ignoreAll && (!has("--no-context-files") || leak.has("context"))) { + const dirs = []; + for (let d = cwd; ; d = path.dirname(d)) { dirs.unshift(d); if (path.dirname(d) === d) break; } + parts.push(read(path.join(agent, "AGENTS.md")) || ""); + for (const d of dirs) parts.push(read(path.join(d, "AGENTS.md")) || read(path.join(d, "CLAUDE.md")) || ""); + } + if (!(print && K.printNoStart)) for (const f of starts) fs.writeFileSync(f, parts.join("\n")); + if (print) { + if (K.sessionOut) process.stdout.write(K.sessionOut); + process.exit(0); } let input = ""; try { input = fs.readFileSync(0, "utf8"); } catch { input = ""; } @@ -86,7 +129,10 @@ if (val("--mode") === "rpc") { let m; try { m = JSON.parse(line); } catch { continue; } if (m.type === "get_state") process.stdout.write(JSON.stringify({ id: m.id, type: "response", command: "get_state", success: true, data: { isStreaming: false } }) + "\n"); - if (m.type === "get_commands") process.stdout.write(JSON.stringify({ id: m.id, type: "response", command: "get_commands", success: true, data: { commands } }) + "\n"); + if (m.type === "get_commands") { + const answer = K.commands === "fail" ? { success: false, error: "no" } : { success: true, data: K.commands === "list" ? commands : { commands } }; + process.stdout.write(JSON.stringify({ id: m.id, type: "response", command: "get_commands", ...answer }) + "\n"); + } } process.exit(0); } diff --git a/packages/plugins/README.md b/packages/plugins/README.md index a2d1358e8..c8da9ad44 100644 --- a/packages/plugins/README.md +++ b/packages/plugins/README.md @@ -253,7 +253,8 @@ export returns: which must be an absolute path to an executable file; else the first `pi` on PATH, skipping the repository's own `node_modules/.bin`; else the repository's copy. The binary must belong to the - `@earendil-works/pi-coding-agent` package. With none, the message is + `@earendil-works/pi-coding-agent` package: it is the bin that package's + `package.json` names. With none, the message is "no pi found: set MOSAIC_PI_BIN, put pi on PATH, or run npm ci". No version is pinned. Pi's `mounts` and `installRoots` hold the found package's directory (the repository's whole `node_modules` for the @@ -264,7 +265,7 @@ export returns: the fit check (#1873 S3, `packages/harness/src/pi-fit.mjs`), which refuses `harness-feature-missing` when the binary lacks a flag the adapter passes, starts with a missing `-e` file, loads a planted ambient - source with the seal flags on, or can't be checked. When its `--help` + source or context file with the seal flags on, or can't be checked. When its `--help` lists `--no-mcp`, `env` sets `MOSAIC_PI_NO_MCP=1` and the adapter passes `--no-mcp`; otherwise it sets `MOSAIC_PI_NO_MCP=0`. - `version({env})` is what the manifest records as `harnessVersion`. Pi's diff --git a/packages/plugins/harness/pi/index.mjs b/packages/plugins/harness/pi/index.mjs index f18642607..43517da50 100644 --- a/packages/plugins/harness/pi/index.mjs +++ b/packages/plugins/harness/pi/index.mjs @@ -118,7 +118,11 @@ export function bundle({ dir, models, proxy = null, transports = {} } = {}) { // The tool limit is the built-ins plus the typed tools, which the extension serves. #1873 S2: // MOSAIC_PI_BIN is the found binary's path inside the session; the bus's own environment names it -// (env defaults to it, since the launcher and the Triage host call env() without one). +// (env defaults to it, since buildBundle and the Triage host call env() without one). preflight, +// version and mounts get the launcher's env instead. Both find the same binary: the session env +// copies PATH and MOSAIC_PI_BIN from the bus's, and locate() skips the repository's .bin on PATH. +// Should they ever differ, env() still runs the fit check on what it found and throws, so the +// launch fails closed (as harness-failed, not harness-feature-missing). export function env({ policy, tools, session, proxy = null, env: hostEnv = process.env }) { const at = found(hostEnv); // #1873 S3: a session runs only a binary that passed the fit check (cached from preflight).