fix(pi): pi-unpin S3 re-check 4: one sealed run in a bus session's shape, built by sessionShape, which a test holds to the adapter's argv; a one-out run must show only its own kind; the README and header list what a session passes that no fit run does (#1873)

This commit is contained in:
jason.woltje committed 2026-11-20 01:17:33 -06:00
1 parent fa5ce3ae65
commit 749d6d98f5
4 files changed
+151 -29

No files matched your search

+21 -8
View File
@@ -52,14 +52,27 @@ and answer `get_state` and `get_commands`, or the check can't tell and
refuses. For every kind but the trusted project files, only a source
outside the project counts there, since a project source can wait on
trust and `--no-approve` would then hide a seal flag the binary ignores.
Then one sealed rpc run per loader flag leaves out only that flag, and
its kind must show. So each loader flag is shown to keep its own kind
out, and no other seal flag, nor their order, can stand in for it. Each
run gets a fresh scratch directory, so no run sees what another left.
The fit runs do not pass the flags production adds (`--provider`,
`--model`, `--mode json`, `--session-dir`); a pi built to act one way
without them and another way with them is outside what a start-time
check can catch. A run killed for taking
Then one sealed rpc run per loader flag leaves out only that flag: its
kind must show, and no other kind (an MCP server may show beside an
extension when `--help` lists no `--no-mcp`, since pi then ties MCP to
`--no-extensions`). So each loader flag is shown to keep its own kind
out, and no other seal flag, nor their order, can stand in for it. One
more sealed rpc run has a bus session's shape: the arguments the adapter
passes for a session with tools and a fresh persistent session, in its
order, from `sessionShape` in `src/pi-fit.mjs`, which a test holds to
the adapter's real argv. That is `--tools read` in place of
`--no-tools`, `--session-dir` in place of `--no-session`, a fixed
`--provider` and `--model`, and `-e` right after `--no-extensions`. It
must show no planted source. Each run gets a fresh scratch directory, so
no run sees what another left. What a session can pass that no fit run
does: `--mode json` (the model event monitor) or `-p` with a request in
the session's shape (that run is rpc), `--thinking`, `-c` (a session
directory that already holds a session), `--fork`, `--skill` with
`--no-skills` dropped (worker runs only), more than one `-e`, a tool
list other than `read`, its own provider and model, and a provider key.
The runs are offline with no key, so nothing past the start is tried. A
pi built to act one way in the fit runs and another way with those is
outside what a start-time check can catch. A run killed for taking
over 20 s or printing over 4 MB can't be checked and says which. A
`--version` that fails or hangs past 3 s only leaves no label. Pi
0.85.1 appends the agent directory's `APPEND_SYSTEM.md` whatever the
+52 -8
View File
@@ -21,8 +21,16 @@
// --system-prompt. The binary answers both, loads that extension (a session's gate
// comes in that way), uses the --system-prompt file, and shows no planted source.
// one out one more sealed rpc run per loader flag, with every seal flag but that one. The
// kind it keeps out must show. So each loader flag is shown to keep its own kind out,
// and no other seal flag, nor the order they come in, can stand in for it.
// kind it keeps out must show, and no other kind (pi without --no-mcp ties MCP to
// --no-extensions, so an MCP server may show beside an extension). So each loader flag
// is shown to keep its own kind out, and no other seal flag, nor the order they come
// in, can stand in for it.
// session one more sealed rpc run in a bus session's shape, with the arguments
// adapters/pi/adapter.sh passes for a session with tools and a fresh persistent
// session, in its order (sessionShape below; a test holds it to the adapter's argv):
// --tools read in place of --no-tools, --session-dir in place of --no-session, a fixed
// --provider and --model, and -e right after --no-extensions. It answers both, loads the
// -e extension, uses the --system-prompt file, and shows no planted source.
// sealed -p the same in -p, the mode the adapter runs. Its -e extension must reach
// session_start, and no planted extension, MCP server, context file or trusted
// project file may show. -p answers no get_commands, so skills and prompt templates
@@ -31,12 +39,13 @@
// and no session_start for the -e extension given after it
//
// Every run gets a fresh scratch directory with the sources planted again, so no run sees what
// another one left. Each run passes the same -e and --system-prompt; runs differ only by their seal
// flags, their mode, the -p message and the missing -e file.
// another one left. Each run passes the same -e and --system-prompt; the other runs differ only by
// their seal flags, their mode, the -p message and the missing -e file.
//
// A check that cannot tell refuses: a sandbox that fails, a run that times out or prints more
// than 4 MB, an answer that is missing or doesn't parse, a kind of source the control doesn't
// show, or the kind a one-out run should show. A --version that fails or hangs past 3 s only leaves no label. A pass is cached in memory
// show, or the kind a one-out run should show. A one-out run that shows another kind, or a
// session-shaped run that shows any, is a leak. A --version that fails or hangs past 3 s only leaves no label. A pass is cached in memory
// by a digest of the files that run (the package's package.json, every file under dist/bundle/,
// every file in the directory tree that holds the binary's real file, and, for a copy mounted
// with the node_modules it sits in, npm's lock files there) and the mount layout.
@@ -44,9 +53,13 @@
// repository's pi), never a host path.
//
// Known limit: pi 0.85.1 appends the agent directory's APPEND_SYSTEM.md whatever the flags. The
// check doesn't plant one; a session's agent directory is the run's own. The runs don't pass the
// flags a session adds (--provider, --model, --mode json, --session-dir), so a pi built to act
// differently with them is outside what this check can catch.
// check doesn't plant one; a session's agent directory is the run's own. What a session can pass
// that no run does: --mode json (the model event monitor) or -p with a request in the session's
// shape (that run is rpc), --thinking, -c (a session directory that already holds a session),
// --fork, --skill with --no-skills dropped (worker runs only), more than one -e, a tool list
// other than read, its own provider and model, and a provider key. The runs are offline with no
// key, so nothing past the start is tried. A pi built to act differently with those is outside
// what a start-time check can catch.
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
@@ -60,6 +73,22 @@ export const FIT_FLAGS = Object.freeze(["--mode", "--session", "--no-session", "
const SEAL = ["--offline", "--no-approve", "--no-extensions", "--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files", "--no-tools", "--no-session"];
// The loader flags: each keeps one kind of planted source out. --no-mcp counts when --help lists it.
const LOADERS = [["--no-approve", "trusted project file"], ["--no-extensions", "extension"], ["--no-skills", "skill"], ["--no-prompt-templates", "prompt template"], ["--no-context-files", "context file"]];
// The session-shaped run's fixed tools, provider and model. Offline with no key, pi stops before
// any model call.
export const SESSION_SHAPE = Object.freeze({ tools: "read", provider: "anthropic", model: "claude-sonnet-4-5" });
// The arguments adapters/pi/adapter.sh builds for a bus session with tools (SESSION_SHAPE.tools),
// a fresh persistent session directory, the given -e files and --system-prompt file, no --thinking
// and no --skill, less the -p and request it appends. Keep its order: pi-fit.test.mjs runs the
// adapter and fails when the two differ.
export function sessionShape({ noMcp, extensions, prompt, sessionDir }) {
return [
"--offline", "--no-approve", "--no-extensions", ...(noMcp ? ["--no-mcp"] : []), ...extensions.flatMap((e) => ["-e", e]),
"--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files",
"--tools", SESSION_SHAPE.tools, "--session-dir", sessionDir,
"--provider", SESSION_SHAPE.provider, "--model", SESSION_SHAPE.model,
"--system-prompt", prompt,
];
}
const RUN_TIMEOUT = 20000;
// --version gives only a label, so a hang costs this much and leaves no label.
const VERSION_TIMEOUT = 3000;
@@ -157,8 +186,21 @@ function check(found, { levels, bwrap, node, who, digest }) {
if (!r.answer.state || !r.answer.commands) return refuse(`can't be checked: it didn't answer get_state and get_commands ${but}`);
if (r.prompt === null) return refuse(`can't be checked: its -e extension didn't see the system prompt ${but}`);
if (!r.seen.some((s) => s.kind === kind)) return refuse(`can't be checked: it didn't load a planted ${kind} ${but}`);
const others = r.seen.filter((s) => s.kind !== kind && !(flag === "--no-extensions" && !noMcp && s.kind === "MCP server"));
if (others.length) return refuse(`loaded a planted ${others.map((s) => s.source).join(", ")} ${but}`);
}
// A bus session's shape: every session passes --tools, and a persistent one --session-dir, so a
// seal that holds only with --no-tools or --no-session, or only in the order above, shows here.
const shaped = run((s) => [...sessionShape({ noMcp, extensions: [join(s, "explicit", "fit.ts")], prompt: join(s, "prompt.md"), sessionDir: join(s, "sessions") }), "--mode", "rpc"], ask);
const shape = "in a session's shape";
if (shaped.stopped) return refuse(`can't be checked: the start ${shape} ${shaped.stopped}`);
if (!shaped.answer.state || !shaped.answer.commands) return refuse(`can't be checked: it didn't answer get_state and get_commands ${shape}`);
if (!shaped.explicit) return refuse(`didn't load an -e extension ${shape}`);
if (shaped.prompt === null) return refuse(`can't be checked: its -e extension didn't see the system prompt ${shape}`);
if (!shaped.prompt.includes(GIVEN)) return refuse(`didn't use the --system-prompt file ${shape}`);
if (shaped.seen.length) return refuse(`loaded a planted ${shaped.seen.map((s) => s.source).join(", ")} with the seal flags ${shape}`);
// -p takes a message; offline with no key, pi stops before any model call.
const p = run((s) => ["-p", ...seal, ...given(s), "fit check"], "", { commands: false });
if (p.stopped) return refuse(`can't be checked: a -p start with the seal flags ${p.stopped}`);
@@ -281,6 +323,8 @@ function plant(scratch) {
mkdirSync(marks, { recursive: true });
mkdirSync(join(scratch, "tmp"), { recursive: true });
mkdirSync(project, { recursive: true });
// The session-shaped run's --session-dir, made as the adapter makes it.
mkdirSync(join(scratch, "sessions"), { recursive: true });
// Each settings.json lists an extension, a skill directory, a prompt template and a local
// package holding one of each.
const listed = (who) => {
+71 -10
View File
@@ -9,7 +9,7 @@ import { spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { piFit } from "../src/pi-fit.mjs";
import * as fitModule from "../src/pi-fit.mjs";
import { piStandIn } from "./pi-stand-in.mjs";
import { locate } from "../../plugins/harness/pi/binary.mjs";
import * as pi from "../../plugins/harness/pi/index.mjs";
@@ -38,6 +38,7 @@ function standIn(t, knobs, version = "2.0.0") {
return { env, root, cli: join(pkg, "dist", "bundle", "cli.js"), paths: [root, realpathSync(root), tmpdir(), REPO] };
}
const { piFit } = fitModule;
const fit = (env) => piFit(locate(env), { levels: LEVELS });
// A refusal names no host path: none of the stand-in's, the scratch's or the repository's, and no
// "/" at all, since every expected line is plain words and flag names.
@@ -144,11 +145,16 @@ test("pi-unpin S3 re-check 3: every run has its own scratch directory, and the r
});
const seal = [...SEAL, ...extraFlags];
const loaders = [...LOADERS, ...extraFlags];
// --version, --help, the control, the sealed rpc, one per loader flag, -p, two missing -e.
assert.equal(runs.length, 2 + 2 + loaders.length + 1 + 2);
// --version, --help, the control, the sealed rpc, one per loader flag, the session-shaped run,
// -p, two missing -e.
assert.equal(runs.length, 2 + 2 + loaders.length + 1 + 1 + 2);
assert.equal(new Set(runs.map((x) => x.scratch)).size, runs.length, "each run has its own scratch directory");
// Re-check 4 item 1: the session-shaped run comes after the one-out runs, built by the same
// function the adapter test below holds to the adapter's argv, with rpc at the end.
const at = 2 + 2 + loaders.length;
assert.deepEqual(runs[at].args, [...fitModule.sessionShape({ noMcp: extraFlags.length > 0, extensions: ["<s>/explicit/fit.ts"], prompt: "<s>/prompt.md", sessionDir: "<s>/sessions" }), "--mode", "rpc"]);
const flagLists = [];
for (const { args } of runs.slice(2)) {
for (const { args } of runs.slice(2).filter((_, i) => i + 2 !== at)) {
const flags = args.filter((a) => seal.includes(a));
assert.deepEqual(flags, seal.filter((f) => flags.includes(f)), "the seal flags keep their order");
const rest = [];
@@ -173,16 +179,44 @@ test("pi-unpin S3 re-check 3: --no-tools or --no-themes can't stand in for the l
test("pi-unpin S3 re-check 3: each loader flag is shown on its own, so no other seal flag can stand in for it", SKIP, t => {
const but = (kind, flag) => `pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't load a planted ${kind} with every seal flag but ${flag}`;
assert.equal(fit(standIn(t, { standIn: ["--no-approve", "--no-extensions"] }).env).message, but("trusted project file", "--no-approve"));
for (const [flag, kind] of [["--no-extensions", "extension"], ["--no-skills", "skill"], ["--no-prompt-templates", "prompt template"], ["--no-context-files", "context file"]]) {
assert.equal(fit(standIn(t, { standIn: [flag, "--no-approve"] }).env).message, but(kind, flag), flag);
}
assert.equal(fit(standIn(t, { extraFlags: ["--no-mcp"], standIn: ["--no-mcp", "--no-approve"] }).env).message, but("MCP server", "--no-mcp"));
const leaked = (sources, flag) => `pi 2.0.0 (MOSAIC_PI_BIN) loaded a planted ${sources} with every seal flag but ${flag}`;
// Each stand-in ignores its flag and keeps its kind out with a loader flag whose run comes later
// (re-check 4 item 2 refuses one that leans on an earlier one in that earlier run, below), so
// each is refused in its own flag's run.
for (const [flag, other, kind, extraFlags] of [
["--no-approve", "--no-extensions", "trusted project file", []],
["--no-extensions", "--no-skills", "extension", []],
["--no-skills", "--no-prompt-templates", "skill", []],
["--no-prompt-templates", "--no-context-files", "prompt template", []],
["--no-context-files", "--no-mcp", "context file", ["--no-mcp"]],
]) assert.equal(fit(standIn(t, { standIn: [flag, other], extraFlags }).env).message, but(kind, flag), flag);
// Re-check 4 item 2: a stand-in that leans on an earlier loader flag shows its kind in that
// flag's run, which refuses it for showing a kind not its own. --no-mcp's run is the last.
assert.equal(fit(standIn(t, { standIn: ["--no-extensions", "--no-approve"] }).env).message, leaked("user extension, user settings extension, user package extension, project extension, project settings extension, project package extension, user mcp.json, project mcp.json, project .mcp.json", "--no-approve"));
assert.equal(fit(standIn(t, { extraFlags: ["--no-mcp"], standIn: ["--no-mcp", "--no-approve"] }).env).message, leaked("user mcp.json, project mcp.json, project .mcp.json", "--no-approve"));
// Re-check 3's dispatchers: a pi that hides everything whenever one always-passed loader flag is on.
assert.equal(fit(standIn(t, { hideWhen: "--no-approve" }).env).message, but("extension", "--no-extensions"));
// It now leaks every other kind in the --no-approve run, where --no-approve is left out.
assert.match(fit(standIn(t, { hideWhen: "--no-approve" }).env).message, /^pi 2\.0\.0 \(MOSAIC_PI_BIN\) loaded a planted user extension, .* user skill, .* user prompt template, .* user AGENTS\.md, project AGENTS\.md, parent CLAUDE\.md with every seal flag but --no-approve$/);
assert.equal(fit(standIn(t, { hideWhen: "--no-extensions" }).env).message, but("trusted project file", "--no-approve"));
});
// Re-check 4 item 2: a one-out run must show its own kind and no other.
test("pi-unpin S3 re-check 4: a pi that loads other kinds when one loader flag is left out is refused at that run", SKIP, t => {
for (const [leakWithout, flag] of [[["--no-skills"], "--no-skills"], [LOADERS, "--no-approve"]]) {
const m = fit(standIn(t, { leakWithout }).env).message;
assert.match(m, new RegExp(`^pi 2\\.0\\.0 \\(MOSAIC_PI_BIN\\) loaded a planted user extension, .* with every seal flag but ${flag}$`), String(leakWithout));
if (flag === "--no-skills") assert.ok(!/skill/.test(m.replace(/but --no-skills$/, "")), m);
}
});
// Re-check 4 item 1: every bus session runs pi with --tools and most with --session-dir, so one
// sealed run has a session's shape: no --no-tools, no --no-session.
test("pi-unpin S3 re-check 4: a pi whose loader flags work only with --no-tools or --no-session is refused in a session's shape", SKIP, t => {
for (const extraFlags of [[], ["--no-mcp"]]) for (const needs of ["--no-tools", "--no-session"]) {
assert.match(fit(standIn(t, { needs, extraFlags }).env).message ?? "passed", /^pi 2\.0\.0 \(MOSAIC_PI_BIN\) loaded a planted user extension, .* with the seal flags in a session's shape$/, `${needs} ${extraFlags}`);
}
});
test("pi-unpin S3 re-check 3: a pi that remembers the control in its scratch directory and hides later is refused", SKIP, t => {
assert.match(fit(standIn(t, { remember: true }).env).message, /^pi 2\.0\.0 \(MOSAIC_PI_BIN\) loaded a planted user extension, .* with the seal flags on$/);
});
@@ -332,3 +366,30 @@ test("pi-unpin S3: the adapter passes --no-mcp only when the module asks for it"
assert.equal(run(standIn(t, { extraFlags: ["--no-mcp"] })).filter((a) => a === "--no-mcp").length, 1);
assert.equal(run(standIn(t, {})).includes("--no-mcp"), false);
});
// Re-check 4 item 1: the session-shaped fit run's arguments are the adapter's for a bus session
// with tools and a fresh persistent session, so the two can't drift. The adapter runs as a session
// runs it, from pi.env(), with the fit run's tools, provider, model and files put in.
test("pi-unpin S3 re-check 4: the session-shaped fit run passes exactly what the adapter passes", SKIP, t => {
assert.equal(typeof fitModule.sessionShape, "function", "pi-fit.mjs exports sessionShape");
const session = { provider: "fixture", model: "m", thinking: "low", toolSocket: "/run/tools.sock" };
for (const extraFlags of [[], ["--no-mcp"]]) {
const s = standIn(t, { extraFlags });
const out = join(s.root, "argv");
const prompt = join(s.root, "prompt.md");
const ext = join(s.root, "fit.ts");
const sessions = join(s.root, "sessions");
writeFileSync(prompt, "p");
writeFileSync(ext, "");
const given = pi.env({ policy: { tools: ["read"] }, tools: [], session, env: s.env });
assert.ok(given.MOSAIC_TOOLS, "a bus session always has tools");
const shape = fitModule.SESSION_SHAPE;
const r = spawnSync("/bin/sh", [join(REPO, "adapters", "pi", "adapter.sh")], {
encoding: "utf8",
env: { ...given, MOSAIC_TOOLS: shape.tools, PI_PROVIDER: shape.provider, PI_MODEL: shape.model, MOSAIC_THINKING: "", MOSAIC_EXTENSIONS: ext, PATH: "/usr/bin:/bin", ARGV_OUT: out, MOSAIC_SYSTEM_PROMPT_FILE: prompt, MOSAIC_REQUEST: "x", MOSAIC_WORKSPACE: join(s.root, "ws"), MOSAIC_SESSION_DIR: sessions },
});
assert.equal(r.status, 0, r.stderr);
const argv = readFileSync(out, "utf8").split("\n").slice(0, -1);
assert.deepEqual(argv, [...fitModule.sessionShape({ noMcp: extraFlags.length > 0, extensions: [ext], prompt, sessionDir: sessions }), "-p", "x"]);
}
});
+7 -3
View File
@@ -33,7 +33,9 @@
// `standIn`: [flag, other]: it ignores flag, and other keeps flag's kind out instead.
// `remember`: the first run in a scratch directory (it leaves a marker beside the
// -e file's directory) loads every kind whatever the seal flags; a later run loads
// none. `commands` "fail" answers get_commands with success false; "list" answers
// none. `needs`: a flag the loader flags work only with; without it they do
// nothing. `leakWithout`: flags; when any of them is absent, every kind loads.
// `commands` "fail" answers get_commands with success false; "list" answers
// with a bare list.
// otherwise a session with no $ARGV_OUT: prints `sessionOut` and exits 0.
//
@@ -92,8 +94,10 @@ if (rpc || print) {
if (fs.existsSync(marker)) ignoreAll = true;
else { fs.writeFileSync(marker, ""); leak = new Set(ALL); }
}
// Whether a seal flag is on: with standIn, its flag reads as the other flag.
const sealed = (flag) => has(K.standIn && K.standIn[0] === flag ? K.standIn[1] : flag);
if ((K.leakWithout || []).some((f) => !has(f))) leak = new Set(ALL);
// Whether a seal flag is on: with standIn, its flag reads as the other flag; with needs, only
// when that flag is given too.
const sealed = (flag) => has(K.standIn && K.standIn[0] === flag ? K.standIn[1] : flag) && (!K.needs || has(K.needs));
const on = (kind, flag) => !ignoreAll && (!sealed(flag) || leak.has(kind));
const agent = process.env.PI_CODING_AGENT_DIR || path.join(process.env.HOME || "/", ".pi", "agent");
const cwd = process.cwd();