mosaicstack/stack
25
0
Fork 1
Code Issues 365 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

seat, cli, harness: role sessions confined to their own files (#1586)

Browse Source 
Row 92, round 2. Darkwing built it. A role session runs in a bwrap
sandbox that sees only its own workspace, a private /proc, /dev and IPC
namespace, and the variables ENV_ALLOW passes. The launcher checks the
session credential before a Claude launch, records a runner killed from
the host as killed rather than failed, and the bus unit template carries
the credential file (decision 119). The harness gate and the cli, seat
and harness READMEs say what the sandbox does and does not hide.

Candidate manifest 8bf4141cbe69fc7bed188555bcc88dcab0b1ea2149912cb6f970ba8c2629dd97
(13 paths), base ac37cd17, packet agents/darkwing/work/queue-92/r2/.
Approvals: Dewey 27978, Filbert 27984 (round 1: Dewey 27942, Filbert
asked for changes in 27951). Sage's gate rerun 22:27:37Z-22:30:35Z on
1b7f3781 plus the patch, manifest 13/13: seat 33, cli 120, harness 65,
bus 108, webui 86, control-board 124, and every scripts/test-*.sh exited 0
(test-release 4, Docker cases skipped at a dead socket).

Live at the next bus restart and the next role session launch (decision
109). Installing the new unit and restarting is Mos's call.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jason.woltje
2026-10-12 17:30:53 -05:00
co-authored by Claude Opus 5.5
parent 4835a8e282
commit 57948a81f3
13 changed files with 799 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/cli/README.md
+36 -3
View File
@@ -220,6 +220,8 @@ is `{"ok": true, "result": {run, instance, harness, model}}` or
| the instance isn't running | `instance-running` | | the instance isn't running | `instance-running` |
| the instance resolves to `pi` or `claude-code` and a model | `unsupported-harness` | | the instance resolves to `pi` or `claude-code` and a model | `unsupported-harness` |
| its role has a contract file | `no-role-contract` | | its role has a contract file | `no-role-contract` |
| a Claude Code session has `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_API_KEY`, or `ANTHROPIC_AUTH_TOKEN` with `ANTHROPIC_BASE_URL` to log in with | `no-session-credential` |
| bwrap runs here | `sandbox-unavailable` |
| its model names exactly one `launch.max` family | `unknown-model-family` | | its model names exactly one `launch.max` family | `unknown-model-family` |
| that family has room; every running session counts, the PM's too | `capacity-full` | | that family has room; every running session counts, the PM's too | `capacity-full` |
| the broker authorizes `role.launch` for the caller | the broker's code (`launch-revoked`, `decision-required`, …) | | the broker authorizes `role.launch` for the caller | the broker's code (`launch-revoked`, `decision-required`, …) |
@@ -229,13 +231,31 @@ Every refusal is an `action.refused` event (the broker's `refuse`, or its
own refusal for the last broker check) and a line in the launch log. Then own refusal for the last broker check) and a line in the launch log. Then
the launcher reads the instance's credential status from the broker the launcher reads the instance's credential status from the broker
(metadata only), builds the bundle in the run directory, spawns the runner (metadata only), builds the bundle in the run directory, spawns the runner
under `unshare`, binds the outer process's pid and start time, adds the in its bwrap sandbox (`packages/seat/README.md`, "The sandbox"), binds the
outer bwrap process's pid and start time, adds the
session to `sessions.json`, and writes the capability as one line on the session to `sessions.json`, and writes the capability as one line on the
runner's stdin. The capability is never on disk, in argv or in the runner's stdin. The capability is never on disk, in argv or in the
environment. When the runner exits, the launcher sends `endLaunch` with the environment. When the runner exits, the launcher sends `endLaunch` with the
end reason from its exit code (`packages/harness/README.md`), which writes end reason from its exit code (`packages/harness/README.md`), which writes
`session.ended`, releases the role and kills the run's capabilities. `session.ended`, releases the role and kills the run's capabilities.
A Claude Code session logs in with `CLAUDE_CODE_OAUTH_TOKEN`,
`ANTHROPIC_API_KEY`, or `ANTHROPIC_AUTH_TOKEN` with `ANTHROPIC_BASE_URL`
(a proxy's key), from the host's environment, which the allowlist passes
on. The bus unit loads it ("Unit" below). Nothing from the host's
`~/.claude` reaches a session, whose `HOME` is its own. With none of them
set, a Claude Code launch refuses `no-session-credential` before anything
starts. A Pi session gets the same variables through the allowlist. A
shell the harness opens, Claude Code's `Bash` or Pi's `bash`, can read
them.
A Pi session logs in only through the variables `ENV_ALLOW` passes. Its
`HOME` is the run's own, so Pi never reads the operator's `~/.pi/agent`
(`auth.json`, `models.json`, `settings.json`). A Pi launch has no
credential check: a Pi role on a provider whose login lives only in those
files, or whose key isn't an `ENV_ALLOW` variable, launches and then fails
at its first turn.
The manifest records the Pi version from the repository's pin (the The manifest records the Pi version from the repository's pin (the
`@earendil-works/pi-coding-agent` dependency in the root `package.json`) `@earendil-works/pi-coding-agent` dependency in the root `package.json`)
and, for Claude Code, `--version` from the binary sessions run: and, for Claude Code, `--version` from the binary sessions run:
@@ -257,7 +277,7 @@ A host that starts reads the `sessions.json` the last host left. An entry
for this business means that host died without ending the launch, and its for this business means that host died without ending the launch, and its
run would keep holding the role. Before the launch socket opens, the run would keep holding the role. Before the launch socket opens, the
launcher SIGKILLs each such session that still runs (same start time, launcher SIGKILLs each such session that still runs (same start time,
`unshare` on its command line), binds the run again with the recorded bwrap on its command line), binds the run again with the recorded
identity and ends it as `host-lost`, which releases the role. A identity and ends it as `host-lost`, which releases the role. A
`sessions.json` that can't be read refuses the host start with exit 3; move `sessions.json` that can't be read refuses the host start with exit 3; move
it aside after checking which sessions still run. it aside after checking which sessions still run.
@@ -308,6 +328,7 @@ The unit settings:
first when it isn't one of those first when it isn't one of those
- `Environment=MOSAIC_CLAUDE_BIN=`: the Claude Code binary that managed - `Environment=MOSAIC_CLAUDE_BIN=`: the Claude Code binary that managed
sessions and the manifest's version run sessions and the manifest's version run
- `EnvironmentFile=`: the Claude Code session credential (below)
`MOSAIC_CLAUDE_BIN` comes from `--claude PATH`, or else `command -v claude` `MOSAIC_CLAUDE_BIN` comes from `--claude PATH`, or else `command -v claude`
in the installing shell. The path is kept as given, so the native install's in the installing shell. The path is kept as given, so the native install's
@@ -316,9 +337,21 @@ following its updates; reinstall to change it. It must be absolute, an
executable file, and only `A-Z a-z 0-9 / . _ + -`, because systemd expands executable file, and only `A-Z a-z 0-9 / . _ + -`, because systemd expands
`%` and splits `Environment=` on spaces; anything else refuses with exit 1 `%` and splits `Environment=` on spaces; anything else refuses with exit 1
and writes nothing. With no claude found, the unit gets a comment line and writes nothing. With no claude found, the unit gets a comment line
instead and sessions run `claude` from the unit's `PATH`. A running host instead and sessions run `claude` from the unit's `PATH`, which in the
sandbox finds only a `claude` under `/usr`. A running host
keeps the environment it started with until it restarts. keeps the environment it started with until it restarts.
The session credential comes from
`EnvironmentFile=-%h/.config/mosaic-dev/secrets/claude-session.env`, the
one place the path is set. The file holds one line, for example
`CLAUDE_CODE_OAUTH_TOKEN=<token>`, mode 0600. The operator writes it;
nothing in the repository creates or reads it, and no session sees the
file itself, only the variable. The leading `-` lets the unit start
without it, so Pi sessions still launch and a Claude Code launch refuses
`no-session-credential`. To change the credential, edit or swap the file
(or this line and this paragraph), then restart the unit; a running host
keeps the environment it started with.
The unit has no `network-online.target` ordering, since a user unit cannot The unit has no `network-online.target` ordering, since a user unit cannot
order on that system target. The notifier needs no network at start: a order on that system target. The notifier needs no network at start: a
send that fails is journaled and retried. send that fails is journaled and retried.
packages/cli/src/launcher.mjs
+39 -16
View File
@@ -13,17 +13,21 @@
// 3. the instance isn't already running (instance-running) // 3. the instance isn't already running (instance-running)
// 4. it resolves, has a role contract (no-role-contract) and a supported // 4. it resolves, has a role contract (no-role-contract) and a supported
// harness and model (unsupported-harness) // harness and model (unsupported-harness)
// 5. its model is in exactly one launch.max family (unknown-model-family) // 5. a Claude Code session has a credential to log in with
// 6. that family has room; every running session counts, the PM's too // (no-session-credential)
// 6. bwrap runs here (sandbox-unavailable)
// 7. its model is in exactly one launch.max family (unknown-model-family)
// 8. that family has room; every running session counts, the PM's too
// (capacity-full) // (capacity-full)
// 7. the broker authorizes role.launch for the caller (its own codes: // 9. the broker authorizes role.launch for the caller (its own codes:
// launch-revoked, gated, ...) // launch-revoked, gated, ...)
// //
// Steps 2 to 6 refuse through the broker's refuse(), so every refusal is an // Steps 2 to 8 refuse through the broker's refuse(), so every refusal is an
// action.refused event; step 7's refusals the broker records itself. Then // action.refused event; step 9's refusals the broker records itself. Then
// the host builds the bundle, spawns the runner, binds the run's process // the host builds the bundle, spawns the runner in its sandbox
// identity (the outer unshare pid) and writes the new capability on the // (packages/seat/src/sandbox.mjs) with the paths it may see, binds the run's
// runner's stdin. When the runner exits, the host ends the launch. // process identity (the outer bwrap pid) and writes the new capability on
// the runner's stdin. When the runner exits, the host ends the launch.
// //
// `bus start <business> --pm` launches launch.by the same way with // `bus start <business> --pm` launches launch.by the same way with
// launchedBy = the business's human and no caller capability. // launchedBy = the business's human and no caller capability.
@@ -39,17 +43,20 @@ import {
appendLaunchLog, appendLaunchLog,
endReason, endReason,
family, family,
hasSessionCredential,
launchSocketPath, launchSocketPath,
newRun, newRun,
readSessions, readSessions,
Registry, Registry,
runnerOf, runnerOf,
SESSION_CREDENTIALS,
sessionEnv, sessionEnv,
sessionExit,
spawnSession, spawnSession,
UNSHARE,
workspaceDir, workspaceDir,
writeSessionFile, writeSessionFile,
} from "../../seat/src/session.mjs"; } from "../../seat/src/session.mjs";
import { BWRAP, sandboxProblem } from "../../seat/src/sandbox.mjs";
import { cmdlineOf, startTimeOf } from "../../seat/src/proc.mjs"; import { cmdlineOf, startTimeOf } from "../../seat/src/proc.mjs";
import { REPO, rolesDir } from "./config.mjs"; import { REPO, rolesDir } from "./config.mjs";
import { CliError } from "./errors.mjs"; import { CliError } from "./errors.mjs";
@@ -84,7 +91,7 @@ export function claudeVersion(env = process.env) {
} }
// host: startHost()'s handle. tracker: true when the broker has task verbs for // host: startHost()'s handle. tracker: true when the broker has task verbs for
// the business. adapters, namespace, sessionDefaults, versions and // the business. adapters, bwrap, readOnly, sessionDefaults, versions and
// stopTimeoutMs exist for the tests; the command line never sets them. // stopTimeoutMs exist for the tests; the command line never sets them.
export function createLauncher({ export function createLauncher({
host, host,
@@ -92,7 +99,8 @@ export function createLauncher({
env = process.env, env = process.env,
tracker = false, tracker = false,
adapters = ADAPTERS, adapters = ADAPTERS,
namespace = true, bwrap = BWRAP,
readOnly = [],
sessionDefaults = {}, sessionDefaults = {},
versions = null, versions = null,
stopTimeoutMs = STOP_TIMEOUT_MS, stopTimeoutMs = STOP_TIMEOUT_MS,
@@ -132,7 +140,7 @@ export function createLauncher({
return run; return run;
} }
// Steps 2 to 6. Returns what the launch needs. Only `--pm` may name // Steps 2 to 8. Returns what the launch needs. Only `--pm` may name
// launch.by itself. // launch.by itself.
function check(instance, { pm = false } = {}) { function check(instance, { pm = false } = {}) {
if (!b.launch) throw new Refusal("launch-not-configured", `business ${b.id} has no launch block`); if (!b.launch) throw new Refusal("launch-not-configured", `business ${b.id} has no launch block`);
@@ -146,6 +154,12 @@ export function createLauncher({
throw new Refusal("unsupported-harness", e.message); throw new Refusal("unsupported-harness", e.message);
} }
if (!resolved.contract) throw new Refusal("no-role-contract", `role ${resolved.definition} has no contract file`); if (!resolved.contract) throw new Refusal("no-role-contract", `role ${resolved.definition} has no contract file`);
if (model.harness === "claude-code" && !hasSessionCredential(env)) {
const names = SESSION_CREDENTIALS.map((set) => set.join(" with ")).join(", ");
throw new Refusal("no-session-credential", `the bus host has none of ${names}; the bus unit's EnvironmentFile sets one when the bus starts`);
}
const unavailable = sandboxProblem({ bwrap });
if (unavailable) throw new Refusal("sandbox-unavailable", unavailable);
const fam = family(model.model, b.launch.max); const fam = family(model.model, b.launch.max);
if (!fam) throw new Refusal("unknown-model-family", `model ${model.model} is in none, or more than one, of ${Object.keys(b.launch.max).join(", ")}`); if (!fam) throw new Refusal("unknown-model-family", `model ${model.model} is in none, or more than one, of ${Object.keys(b.launch.max).join(", ")}`);
if (registry.count(b.id, fam) >= b.launch.max[fam]) throw new Refusal("capacity-full", `${fam} sessions: ${registry.count(b.id, fam)} of ${b.launch.max[fam]}`); if (registry.count(b.id, fam) >= b.launch.max[fam]) throw new Refusal("capacity-full", `${fam} sessions: ${registry.count(b.id, fam)} of ${b.launch.max[fam]}`);
@@ -194,16 +208,25 @@ export function createLauncher({
turnMarker, turnMarker,
}); });
const child = spawnSession({ runDir, env: sessionEnv(env, run), namespace }); const paths = {
workspace,
adapter: adapters[model.harness],
harness: model.harness,
brokerSocket: host.path,
launchSocket: resolved.launch ? launchSocketPath(dataRoot) : null,
claudeBin: env.MOSAIC_CLAUDE_BIN || null,
readOnly,
};
const child = spawnSession({ runDir, env: sessionEnv(env, run), paths, bwrap });
let bound = false; let bound = false;
let entry = null; let entry = null;
const exited = new Promise((resolve) => child.once("close", (code, signal) => resolve({ code, signal }))); const exited = new Promise((resolve) => child.once("close", (code, signal) => resolve(sessionExit(code, signal))));
child.on("error", (e) => log(`run ${run}: ${e.message}`)); child.on("error", (e) => log(`run ${run}: ${e.message}`));
child.stdin.on("error", () => {}); child.stdin.on("error", () => {});
try { try {
if (child.pid === undefined) throw new Error("the session process did not start"); if (child.pid === undefined) throw new Error("the session process did not start");
const outerStart = startTimeOf(child.pid); const outerStart = startTimeOf(child.pid);
const runner = await runnerOf(child, { namespace }); const runner = await runnerOf(child);
if (!outerStart) throw new Error("the session process exited at once"); if (!outerStart) throw new Error("the session process exited at once");
const launch = await host.bindLaunch({ business: b.id, role: instance, run, harness: model.harness, pid: child.pid, startTime: outerStart }); const launch = await host.bindLaunch({ business: b.id, role: instance, run, harness: model.harness, pid: child.pid, startTime: outerStart });
bound = true; bound = true;
@@ -308,7 +331,7 @@ export function createLauncher({
// its identical launch record, which the broker accepts. // its identical launch record, which the broker accepts.
async function recover() { async function recover() {
for (const s of leftover) { for (const s of leftover) {
if (startTimeOf(s.pid) === s.startTime && (cmdlineOf(s.pid) ?? [])[0] === UNSHARE[0]) { if (startTimeOf(s.pid) === s.startTime && (cmdlineOf(s.pid) ?? [])[0] === bwrap) {
try { try {
process.kill(s.pid, "SIGKILL"); process.kill(s.pid, "SIGKILL");
} catch {} } catch {}
packages/cli/systemd/mosaic-bus.service.in
+5
View File
@@ -17,6 +17,11 @@ Environment=PATH=@PATH@
# claudeVersion run, pinned at install (`--claude PATH`, default # claudeVersion run, pinned at install (`--claude PATH`, default
# `command -v claude`). Without it they run `claude` from PATH above. # `command -v claude`). Without it they run `claude` from PATH above.
@CLAUDE_ENV@ @CLAUDE_ENV@
# Claude Code sessions log in with what this 0600 file sets, which the
# operator writes: CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, or a proxy's
# ANTHROPIC_AUTH_TOKEN with ANTHROPIC_BASE_URL. The `-` lets the unit start
# without it, and a Claude Code launch then refuses.
EnvironmentFile=-%h/.config/mosaic-dev/secrets/claude-session.env
# The host reads the system config and the business file, boots the broker # The host reads the system config and the business file, boots the broker
# (packages/bus/src/process.mjs) and starts the notifier. Config problems and # (packages/bus/src/process.mjs) and starts the notifier. Config problems and
# refusals exit 3, usage 4, invalid input 2: none of those is retried. A # refusals exit 3, usage 4, invalid input 2: none of those is retried. A
packages/cli/tests/fixtures/launch-host.mjs
Vendored
+4 -2
View File
@@ -1,12 +1,13 @@
// A bus host with its launcher in a process of its own, so a test can kill // A bus host with its launcher in a process of its own, so a test can kill
// it hard. It launches the PM, prints the launch result as one JSON line and // it hard. It launches the PM, prints the launch result as one JSON line and
// runs until it is killed. argv: the adapter to use for both harnesses. // runs until it is killed. argv: the adapter to use for both harnesses, then
// the files the sandbox shows the session read-only (the fake adapter).
import { bootConfig, loadSystem } from "../../src/config.mjs"; import { bootConfig, loadSystem } from "../../src/config.mjs";
import { startHost } from "../../src/host.mjs"; import { startHost } from "../../src/host.mjs";
import { createLauncher } from "../../src/launcher.mjs"; import { createLauncher } from "../../src/launcher.mjs";
const [adapter] = process.argv.slice(2); const [adapter, ...readOnly] = process.argv.slice(2);
const system = loadSystem({ env: process.env }); const system = loadSystem({ env: process.env });
const boot = bootConfig({ system, businessId: "acme", env: process.env }); const boot = bootConfig({ system, businessId: "acme", env: process.env });
const host = await startHost({ boot, business: "acme", log: () => {} }); const host = await startHost({ boot, business: "acme", log: () => {} });
@@ -17,6 +18,7 @@ const launcher = createLauncher({
adapters: { pi: adapter, "claude-code": adapter }, adapters: { pi: adapter, "claude-code": adapter },
sessionDefaults: { pollInterval: 100 }, sessionDefaults: { pollInterval: 100 },
versions: { pi: "0.85.1", claude: null }, versions: { pi: "0.85.1", claude: null },
readOnly,
log: () => {}, log: () => {},
}); });
await launcher.listen(); await launcher.listen();
packages/cli/tests/host.test.mjs
+6
View File
@@ -482,6 +482,12 @@ test("bus-service.sh renders the unit and installs it into a given directory", (
assert.equal(render.status, 0, render.stderr); assert.equal(render.status, 0, render.stderr);
assert.match(render.stdout, new RegExp(`ExecStart=${REPO.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/scripts/mosaic bus start %i`)); assert.match(render.stdout, new RegExp(`ExecStart=${REPO.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/scripts/mosaic bus start %i`));
assert.match(render.stdout, /RestartPreventExitStatus=2 3 4/); assert.match(render.stdout, /RestartPreventExitStatus=2 3 4/);
// One session credential file, optional so Pi sessions start without it.
// The README's paragraph names the same file; nothing else does.
const files = render.stdout.match(/^EnvironmentFile=.*$/gm);
assert.equal(files.length, 1, files.join("\n"));
assert.match(files[0], /^EnvironmentFile=-%h\/\S+$/);
assert.ok(readFileSync(join(REPO, "packages", "cli", "README.md"), "utf8").includes(`\`${files[0]}\``), "the README names the unit's file");
assert.doesNotMatch(render.stdout, /@REPO@|@PATH@|@CLAUDE_ENV@/); assert.doesNotMatch(render.stdout, /@REPO@|@PATH@|@CLAUDE_ENV@/);
// A user unit cannot order on a system target (Darkwing F3). // A user unit cannot order on a system target (Darkwing F3).
assert.doesNotMatch(render.stdout, /network-online/); assert.doesNotMatch(render.stdout, /network-online/);
packages/cli/tests/launcher.test.mjs
+138 -8
View File
@@ -1,5 +1,5 @@
// The S6 launcher inside a real bus host: real broker child, real runners // The S6 launcher inside a real bus host: real broker child, real runners
// under unshare, the fake adapter from packages/harness standing in for the // in the bwrap sandbox, the fake adapter from packages/harness standing in for the
// harness. The PM launches a coder through its `launch` tool; every host // harness. The PM launches a coder through its `launch` tool; every host
// refusal comes back over the launch socket and lands in the launch log. // refusal comes back over the launch socket and lands in the launch log.
@@ -9,20 +9,21 @@ import { spawn, spawnSync } from "node:child_process";
import { once } from "node:events"; import { once } from "node:events";
import { createInterface } from "node:readline"; import { createInterface } from "node:readline";
import { connect } from "node:net"; import { connect } from "node:net";
import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from "node:fs"; import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from "node:fs";
import { join } from "node:path"; import { join } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { Client } from "../../bus/src/client.mjs"; import { Client } from "../../bus/src/client.mjs";
import { launchLogFile, launchSocketPath, readSessions, Registry, sessionsFile, UNSHARE } from "../../seat/src/session.mjs"; import { launchLogFile, launchSocketPath, readSessions, Registry, SESSION_CREDENTIALS, sessionsFile } from "../../seat/src/session.mjs";
import { sandboxProblem } from "../../seat/src/sandbox.mjs";
import { bootConfig, loadSystem } from "../src/config.mjs"; import { bootConfig, loadSystem } from "../src/config.mjs";
import { startHost, startTimeOf } from "../src/host.mjs"; import { startHost, startTimeOf } from "../src/host.mjs";
import { claudeVersion, createLauncher } from "../src/launcher.mjs"; import { claudeVersion, createLauncher } from "../src/launcher.mjs";
import { mockAnthropic, REPO } from "../../harness/tests/helpers.mjs";
import { fixture, tmp } from "./helpers.mjs"; import { fixture, tmp } from "./helpers.mjs";
const FAKE = fileURLToPath(new URL("../../harness/tests/fixtures/fake-adapter.mjs", import.meta.url)); const FAKE = fileURLToPath(new URL("../../harness/tests/fixtures/fake-adapter.mjs", import.meta.url));
const HOST = fileURLToPath(new URL("./fixtures/launch-host.mjs", import.meta.url)); const HOST = fileURLToPath(new URL("./fixtures/launch-host.mjs", import.meta.url));
const NAMESPACES = spawnSync(UNSHARE[0], [...UNSHARE.slice(1), "true"]).status === 0; const skip = sandboxProblem() ?? false;
const skip = NAMESPACES ? false : "unprivileged user and PID namespaces are unavailable here";
// pm opus, coder and reviewer sonnet, one of each family; cto is listed but // pm opus, coder and reviewer sonnet, one of each family; cto is listed but
// runs the system model, which is in no family. // runs the system model, which is in no family.
@@ -44,9 +45,13 @@ async function until(fn, ms = 15000) {
} }
// A config and data root with the fake adapter; setup() starts a host on it. // A config and data root with the fake adapter; setup() starts a host on it.
// The session credential is a stand-in, and the fake adapter never uses it.
// Every credential variable of the shell running the tests is dropped.
function prepare(t, more = (doc) => doc) { function prepare(t, more = (doc) => doc) {
const root = tmp(t, "mosaic-launch-"); const root = tmp(t, "mosaic-launch-");
const f = fixture(root, "acme", (doc) => more(edit(doc))); const f = fixture(root, "acme", (doc) => more(edit(doc)));
for (const k of SESSION_CREDENTIALS.flat()) delete f.env[k];
f.env.CLAUDE_CODE_OAUTH_TOKEN = "stand-in";
mkdirSync(f.dataRoot, { mode: 0o700 }); mkdirSync(f.dataRoot, { mode: 0o700 });
const adapter = join(root, "adapter.sh"); const adapter = join(root, "adapter.sh");
writeFileSync(adapter, `exec '${process.execPath}' '${FAKE}'\n`, { mode: 0o644 }); writeFileSync(adapter, `exec '${process.execPath}' '${FAKE}'\n`, { mode: 0o644 });
@@ -67,6 +72,7 @@ async function setup(t, more = (doc) => doc, { root, f, adapter } = prepare(t, m
adapters: { pi: adapter, "claude-code": adapter }, adapters: { pi: adapter, "claude-code": adapter },
sessionDefaults: { pollInterval: 100 }, sessionDefaults: { pollInterval: 100 },
versions: { pi: "0.85.1", claude: null }, versions: { pi: "0.85.1", claude: null },
readOnly: [FAKE],
log: (l) => logs.push(l), log: (l) => logs.push(l),
...options, ...options,
}); });
@@ -117,7 +123,7 @@ test("bus start --pm: the PM runs under its own PID namespace, registered, with
assert.equal(s.live, true); assert.equal(s.live, true);
assert.equal(s.family, "opus"); assert.equal(s.family, "opus");
assert.equal(s.launchedBy, "jason"); assert.equal(s.launchedBy, "jason");
assert.notEqual(s.pid, s.runnerPid, "the runner is the unshare process's child"); assert.notEqual(s.pid, s.runnerPid, "the runner is the bwrap process's child");
assert.equal(statSync(sessionsFile(f.dataRoot)).mode & 0o777, 0o600); assert.equal(statSync(sessionsFile(f.dataRoot)).mode & 0o777, 0o600);
const runDir = s.runDir; const runDir = s.runDir;
@@ -190,6 +196,10 @@ test("the PM launches a coder through its launch tool; the coder answers; refusa
assert.equal(coder.run, result.run); assert.equal(coder.run, result.run);
assert.equal(coder.launchedBy, "pm"); assert.equal(coder.launchedBy, "pm");
assert.equal(coder.live, true); assert.equal(coder.live, true);
// Only the role that may launch has the launch socket in its sandbox.
const bound = (pid) => readFileSync(`/proc/${pid}/cmdline`, "utf8").split("\0").includes(sock);
assert.equal(bound(readSessions(f.dataRoot).find((s) => s.instance === "pm").pid), true);
assert.equal(bound(coder.pid), false);
// The coder claimed its role and answers its own messages. // The coder claimed its role and answers its own messages.
const hello = await client.call("message.send", { to: "coder", class: "REQUEST", body: "hello" }); const hello = await client.call("message.send", { to: "coder", class: "REQUEST", body: "hello" });
@@ -197,7 +207,7 @@ test("the PM launches a coder through its launch tool; the coder answers; refusa
assert.equal(echo.from_role, "coder"); assert.equal(echo.from_role, "coder");
assert.match(echo.body, /^echo: Message .* from cto, class REQUEST:\n\nhello/); assert.match(echo.body, /^echo: Message .* from cto, class REQUEST:\n\nhello/);
// Host refusals, asked with the cto capability (steps 2 to 6 come before // Host refusals, asked with the cto capability (steps 2 to 8 come before
// the broker's role.launch check, which cto would fail). // the broker's role.launch check, which cto would fail).
const refusals = [ const refusals = [
["pm", "instance-not-listed"], ["pm", "instance-not-listed"],
@@ -249,11 +259,131 @@ test("a runner that stops at once ends its launch with the runner's reason", { s
assert.match(readFileSync(join(f.dataRoot, "launches", "acme", pm.run, "runner.log"), "utf8"), /vikunja/); assert.match(readFileSync(join(f.dataRoot, "launches", "acme", pm.run, "runner.log"), "utf8"), /vikunja/);
}); });
test("a Claude Code launch with no session credential refuses before it starts", { timeout: 30000 }, async (t) => {
const given = prepare(t, (doc) => {
doc.roles.pm.vars.harness = "claude-code";
return doc;
});
delete given.f.env.CLAUDE_CODE_OAUTH_TOKEN;
given.f.env.ANTHROPIC_AUTH_TOKEN = "stand-in"; // a proxy key with no proxy
const { f, launcher } = await setup(t, undefined, given);
const names = "CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN with ANTHROPIC_BASE_URL";
await assert.rejects(launcher.launchPm(), (e) => e.exitCode === 3 && e.message === `PM launch refused: no-session-credential (the bus host has none of ${names}; the bus unit's EnvironmentFile sets one when the bus starts)`);
assert.deepEqual(log(f).map((e) => [e.event, e.code]), [["refused", "no-session-credential"]]);
assert.deepEqual(readSessions(f.dataRoot), []);
assert.deepEqual(readdirSync(join(f.dataRoot, "launches")), ["acme.jsonl"], "no run was made");
});
test("a pi launch needs no session credential: pi logs in through its own ENV_ALLOW variables", { skip, timeout: 30000 }, async (t) => {
const given = prepare(t);
delete given.f.env.CLAUDE_CODE_OAUTH_TOKEN;
assert.deepEqual(SESSION_CREDENTIALS.flat().filter((k) => k in given.f.env), []);
const { f, launcher } = await setup(t, undefined, given);
const pm = await launcher.launchPm();
assert.equal(pm.harness, "pi");
assert.deepEqual(log(f).map((e) => [e.event, e.instance]), [["launch", "pm"]]);
});
// The runner is pid 1 in its namespace and ignores a SIGKILL sent from
// inside it, so the host kills it by its host pid. bwrap then exits 137,
// which the launcher reads back as the signal.
test("a runner killed from the host ends its launch as killed, not failed", { skip, timeout: 30000 }, async (t) => {
const { f, launcher } = await setup(t);
const pm = await launcher.launchPm();
const runnerLog = join(f.dataRoot, "launches", "acme", pm.run, "runner.log");
await until(() => existsSync(runnerLog) && readFileSync(runnerLog, "utf8").includes("claimed by run"));
const [s] = readSessions(f.dataRoot);
process.kill(s.runnerPid, "SIGKILL");
const end = await until(() => log(f).find((e) => e.event === "end" && e.run === pm.run));
assert.deepEqual([end.reason, end.exitCode, end.signal], ["killed", null, "SIGKILL"]);
assert.deepEqual(readSessions(f.dataRoot), []);
});
test("a missing or failing bwrap refuses sandbox-unavailable before anything starts; there is no unconfined fallback",{ timeout: 30000 }, async (t) => {
const given = prepare(t);
const bad = join(given.root, "bwrap");
writeFileSync(bad, "#!/bin/sh\necho 'bwrap: setting up uid map: Permission denied' >&2\nexit 1\n", { mode: 0o755 });
const { f, launcher, cto } = await setup(t, undefined, given, { bwrap: bad });
await assert.rejects(launcher.launchPm(), (e) => e.message === `PM launch refused: sandbox-unavailable (${bad} failed: bwrap: setting up uid map: Permission denied)`);
rmSync(bad);
await assert.rejects(launcher.launchPm(), (e) => e.message === `PM launch refused: sandbox-unavailable (${bad} is missing)`);
// A session's launch request gets the same code, through refuse().
assert.deepEqual(await launcher.fromSession(cto.cap, "coder"), { ok: false, error: "sandbox-unavailable" });
assert.deepEqual(log(f).map((e) => e.code), ["sandbox-unavailable", "sandbox-unavailable", "sandbox-unavailable"]);
assert.deepEqual(readSessions(f.dataRoot), []);
assert.deepEqual(readdirSync(join(f.dataRoot, "launches")), ["acme.jsonl"], "no run was made");
});
// The brief's one real Claude Code launch: the host's claude in a coder's
// sandbox, logged in with a stand-in proxy key against the scripted Messages
// API, runs one Bash call that tries the founder's stand-ins, another run
// and the checkout. No real credential and no real model.
const which = spawnSync("sh", ["-c", "command -v claude"], { encoding: "utf8" });
const CLAUDE = which.status === 0 ? which.stdout.trim() : null;
test("a real Claude Code coder's shell can't read a stand-in secret or anything else its sandbox doesn't hold", { skip: skip || (CLAUDE ? false : "claude is not on PATH"), timeout: 180000 }, async (t) => {
const script = [];
const api = await mockAnthropic(t, { script, done: (r) => `ANSWER ${JSON.stringify(r.map((x) => [x.isError, x.content.slice(0, 1500)]))}` });
const given = prepare(t, (doc) => {
doc.roles.coder.vars.harness = "claude-code";
doc.roles.coder.vars["limits.tools"] = ["bash"];
return doc;
});
delete given.f.env.CLAUDE_CODE_OAUTH_TOKEN;
Object.assign(given.f.env, { ANTHROPIC_AUTH_TOKEN: "stand-in", ANTHROPIC_BASE_URL: api.url, MOSAIC_CLAUDE_BIN: CLAUDE });
const secret = join(given.root, "home", ".config", "mosaic-dev", "secrets", "stand-in.env");
const mosaic = join(given.root, "home", ".mosaic", "fleet", "token");
for (const p of [secret, mosaic]) {
mkdirSync(join(p, ".."), { recursive: true, mode: 0o700 });
writeFileSync(p, "FAKE_TOKEN=stand-in-value\n", { mode: 0o600 });
}
const claudeAdapter = join(REPO, "adapters", "claude", "adapter.sh");
const { f, launcher, client } = await setup(t, undefined, given, { adapters: { pi: given.adapter, "claude-code": claudeAdapter } });
const pm = await launcher.launchPm();
const pmRun = readSessions(f.dataRoot).find((s) => s.run === pm.run).runDir;
// [what, shell command, may the session do it]
const probes = [
["read the stand-in secret", `cat '${secret}'`, false],
["read the stand-in ~/.mosaic", `cat '${mosaic}'`, false],
["read the PM's run directory", `cat '${pmRun}/session.json'`, false],
["read the data root's sessions file", `cat '${sessionsFile(f.dataRoot)}'`, false],
["read the checkout", `cat '${join(REPO, "AGENTS.md")}'`, false],
["write outside", `touch '${given.root}/escaped'`, false],
["write and read its workspace", "echo ws > probe && cat probe", true],
["write and read /tmp", "echo tmp > /tmp/probe && cat /tmp/probe", true],
["HOME is the run's home/", `case "$HOME" in */home) test -w "$HOME";; *) false;; esac`, true],
];
const command = probes.map(([what, cmd], i) => `if { ${cmd}; } >/dev/null 2>&1; then echo "${i} yes"; else echo "${i} no"; fi`).join("\n");
script.push({ name: "Bash", input: { command, description: "probe the sandbox" } });
const sent = await client.call("message.send", { to: "pm", class: "REQUEST", body: 'TOOL launch {"instance":"coder"}' });
const launched = await until(async () => (await client.call("message.receive")).find((m) => m.in_reply_to === sent.id));
assert.equal(JSON.parse(launched.body).instance, "coder", launched.body);
const coder = readSessions(f.dataRoot).find((s) => s.instance === "coder");
assert.equal(coder.harness, "claude-code");
const hello = await client.call("message.send", { to: "coder", class: "REQUEST", body: "probe" });
const reply = await until(async () => (await client.call("message.receive")).find((m) => m.in_reply_to === hello.id), 150000);
assert.equal(reply.from_role, "coder");
const answer = reply.body.match(/ANSWER (\[.*\])/s);
assert.ok(answer, reply.body);
const [[isError, out]] = JSON.parse(answer[1]);
assert.equal(isError, false, out);
const got = Object.fromEntries(out.trim().split("\n").map((l) => l.split(" ")).map(([i, v]) => [probes[i][0], v === "yes"]));
assert.deepEqual(got, Object.fromEntries(probes.map(([what, , may]) => [what, may])));
assert.ok(!existsSync(join(given.root, "escaped")));
assert.ok(!reply.body.includes("stand-in-value"), "the secret's value never comes back");
// The session logged in with the stand-in proxy key, against the mock.
assert.ok(api.requests.length > 0);
assert.ok(api.requests.some((r) => r.tools.includes("Bash")), api.requests.map((r) => r.tools.join(",")).join(" | "));
});
for (const exited of [false, true]) { for (const exited of [false, true]) {
test(`a host that died hard leaves its sessions to the next host, which ${exited ? "finds one already exited (23)" : "kills them"} and ends their runs so the role can be launched again`, { skip, timeout: 30000 }, async (t) => { test(`a host that died hard leaves its sessions to the next host, which ${exited ? "finds one already exited (23)" : "kills them"} and ends their runs so the role can be launched again`, { skip, timeout: 30000 }, async (t) => {
const given = prepare(t); const given = prepare(t);
const { f } = given; const { f } = given;
const old = spawn(process.execPath, [HOST, given.adapter], { env: f.env, stdio: ["ignore", "pipe", "inherit"] }); const old = spawn(process.execPath, [HOST, given.adapter, FAKE], { env: f.env, stdio: ["ignore", "pipe", "inherit"] });
t.after(() => old.kill("SIGKILL")); t.after(() => old.kill("SIGKILL"));
const [line] = await once(createInterface({ input: old.stdout }), "line"); const [line] = await once(createInterface({ input: old.stdout }), "line");
const first = JSON.parse(line); const first = JSON.parse(line);
packages/harness/README.md
+35 -22
View File
@@ -176,28 +176,41 @@ Each turn's stderr goes to `turns/NNNN.stderr` (0600) in the run directory.
These are limits, not bugs, and nothing in this package claims otherwise. These are limits, not bugs, and nothing in this package claims otherwise.
- **The gate is not a sandbox.** It gates the harness's own tool calls. - **The gate is not the sandbox.** It gates the harness's own tool calls.
`bash`, where the role's tool limit grants it, can reach anything the OS `bash`, where the role's tool limit grants it (S0 line 5), reaches what
user can (S0 line 5). the session's sandbox holds (`packages/seat/README.md`, "The sandbox"):
- **Same UID.** Every session runs as the host's user. Founder files in the system directories, the runner's own repository files, its
`HOME` (`~/.git-credentials`, the tea config, `~/.ssh`) are readable workspace and its run directory. The data root, other workspaces and
through `bash`; only the environment is filtered. `/tmp` is shared. The runs, `~/.config`, `~/.mosaic`, `~/.ssh`, `~/.claude` and the rest of
PID namespace (below) hides other processes, so a session can't signal the checkout aren't there. `HOME` and `/tmp` are the run's own. The
or `ptrace` another one by pid. session's credential variable is in its environment, so `bash` (or
- **The broker socket is shared.** A session that can run `bash` can Claude Code's `Bash`) can read it (`packages/cli/README.md`,
"Sessions").
- **Same UID.** The sandbox's user namespace maps the session to the
host's user. It can't reach the host's other files, but anything outside the
sandbox that it can talk to (below) acts with that user's full rights.
- **Network and abstract sockets aren't confined.** The session shares
the host's network namespace: it reaches the network, every service on
`127.0.0.1`, and every abstract Unix socket (a name starting with a NUL
byte, which no mount hides). A service listening there, an X server or a
session bus that offers one, acts for the session outside the sandbox.
The shared namespace also shows up in `/proc/net/unix`, which lists the
host's Unix sockets by name, path ones included; a name gives no access
to a path socket the mounts hide.
The policy's `network` field is recorded, not enforced.
- **The broker socket is bound in.** A session that can run `bash` can
connect to it; it still needs a capability to do anything there. The connect to it; it still needs a capability to do anything there. The
tool socket is reachable through `bash` as well, and acts with the tool socket is in the run directory and acts with the session's own
session's own capability only. capability only. The launch socket is bound in only for the role whose
- **Network.** Abstract Unix sockets and the network are not confined; the launch tool calls it.
policy's `network` field is recorded, not enforced.
- **PID namespace.** Each session's runner is pid 1 of its own namespace - **PID namespace.** Each session's runner is pid 1 of its own namespace
(`unshare --user --map-current-user --pid --fork --kill-child (bwrap `--unshare-pid --as-pid-1`), with its own IPC namespace. A
--mount-proc`). It doesn't reap zombies; turns are reaped by the runner's session sees no other process, so it can't signal or `ptrace` one by
own waits, and the namespace dies with it. A same-UID process can still pid. The runner doesn't reap zombies; turns are reaped by the runner's
ask something outside its tree (a systemd user manager, an existing tmux own waits, and the namespace dies with it.
server) to run a command. - **`mosaic stop`** is same-UID process control from outside the sandbox:
- **`mosaic stop`** is same-UID process control: it checks the registry's it checks the registry's start time and that the process is a runner
start time and that the process is a runner before sending SIGTERM. before sending SIGTERM.
- **If the host dies,** sessions keep running until their broker polls fail - **If the host dies,** sessions keep running until their broker polls fail
`brokerRetries` times, then exit 23. The next host kills what still runs `brokerRetries` times, then exit 23. The next host kills what still runs
and ends those launches as `host-lost` (`packages/cli/README.md`, and ends those launches as `host-lost` (`packages/cli/README.md`,
@@ -224,7 +237,7 @@ These are limits, not bugs, and nothing in this package claims otherwise.
another process of the same user) isn't seen. That is the same reach as another process of the same user) isn't seen. That is the same reach as
`bash` itself. `bash` itself.
- **The system prompt is in argv** for both adapters, so the same UID can - **The system prompt is in argv** for both adapters, so the same UID can
read it in `/proc/<pid>/cmdline`; the PID namespace hides it from other read it in `/proc/<pid>/cmdline` outside the sandbox; the PID namespace
sessions. It holds no secret. hides it from other sessions. It holds no secret.
- **Pi adapter paths** with spaces break its unquoted `-e` and skill - **Pi adapter paths** with spaces break its unquoted `-e` and skill
splitting; the launcher's paths don't contain spaces. splitting; the launcher's paths don't contain spaces.
packages/harness/src/gate.mjs
+4 -4
View File
@@ -3,10 +3,10 @@
// everything else is blocked, and every path argument of a file tool must // everything else is blocked, and every path argument of a file tool must
// resolve inside the workspace. // resolve inside the workspace.
// //
// This is a gate on the harness's own tool calls, not a sandbox. `bash` // This is a gate on the harness's own tool calls, not the sandbox. `bash`
// isn't confined (the role's tool limit decides whether it exists at all), // isn't gated (the role's tool limit decides whether it exists at all):
// and anything bash can reach, the session can reach. See the README's // what it can reach is what the session's bubblewrap sandbox holds
// limits. // (packages/seat/src/sandbox.mjs). See the README's limits.
import { lstatSync, realpathSync } from "node:fs"; import { lstatSync, realpathSync } from "node:fs";
import { homedir } from "node:os"; import { homedir } from "node:os";
packages/seat/README.md
+57 -16
View File
@@ -165,10 +165,10 @@ launch flow and the on-disk layout.
(`ENV_ALLOW`), the repository's `node_modules/.bin` on `PATH`, and (`ENV_ALLOW`), the repository's `node_modules/.bin` on `PATH`, and
`MOSAIC_RUN_ID`. Founder credential variables never pass. `MOSAIC_RUN_ID`. Founder credential variables never pass.
`MOSAIC_CLAUDE_BIN`, the host's pinned Claude Code binary, does. `MOSAIC_CLAUDE_BIN`, the host's pinned Claude Code binary, does.
- `spawnSession` starts `packages/harness/src/runner.mjs` under - `spawnSession` starts `packages/harness/src/runner.mjs` in its sandbox
`unshare --user --map-current-user --pid --fork --kill-child --mount-proc`, (`src/sandbox.mjs`, below), detached, with stdin open for the capability
detached, with stdin open for the capability line and output to line and output to `runner.log` (0600). `runnerOf` finds the runner, the
`runner.log` (0600). `runnerOf` finds the runner inside it. outer bwrap process's only child.
- `Registry` is the host's list of running sessions, mirrored to - `Registry` is the host's list of running sessions, mirrored to
`bus-host/sessions.json` (0600); `readSessions` reads it and marks each `bus-host/sessions.json` (0600); `readSessions` reads it and marks each
entry live or stale by the runner's start time, and refuses a file it entry live or stale by the runner's start time, and refuses a file it
@@ -176,22 +176,63 @@ launch flow and the on-disk layout.
- `stopSession` is `mosaic stop`: SIGTERM to a live runner, resent every - `stopSession` is `mosaic stop`: SIGTERM to a live runner, resent every
second until it exits. second until it exits.
Limits of the PID namespace, which are limits and not a sandbox: ### The sandbox (#1586)
- The runner is pid 1 of its namespace. A detached child is reparented to `src/sandbox.mjs` builds the bubblewrap command line a session runs under.
it, not to the user's init, so it stays under the launch's pid. Pid 1 The mount namespace starts from an empty root and gets only these binds,
ignores a signal it has no handler for; the runner installs its handlers each at its own host path; `binds()` returns them with the reason for each,
first, and the senders resend SIGTERM to cover Node's startup. and the file's header comment has the table.
- The namespace isolates pids only. With its own `/proc`, a session sees
no process outside it, so it can't signal or `ptrace` another session or - Read-only: `/usr`, `/etc` and the host's `/bin`, `/lib`, `/lib64`,
the host by pid (checked on this host, Yama `ptrace_scope` 1). It still `/sbin` links into `/usr`; the ten repository files the runner loads
runs as the host's user, with that user's files, `/tmp`, network and (`RUNNER_FILES`); the adapter; for pi the repository's `node_modules`,
sockets. for Claude Code the `MOSAIC_CLAUDE_BIN` binary when it lives outside
- A session can still ask a process outside its namespace (a systemd user `/usr`; node when it lives outside `/usr`; the broker socket, and the
manager, an existing tmux server) to run a command for it. launch socket for the role whose launch tool calls it.
- Read-write: the instance's workspace and the run directory. The run's
`bundle/` and `session.json` are bound read-only again on top.
- Private: `/tmp` is the run's `tmp/`, `HOME` the run's empty `home/`, and
`/dev` and `/proc` are the namespace's own.
The root is remounted read-only, so a write anywhere else fails. Nothing
else of the host's filesystem exists inside: no data root, no other
workspace or run, no `~/.config`, `~/.mosaic`, `~/.ssh` or `~/.claude`, no
checkout beyond the runner's files. Claude Code logs in with a
credential variable the environment passes (`SESSION_CREDENTIALS`;
`packages/cli/README.md`, "Sessions"), never with a file from the host's
`~/.claude`. Pi likewise logs in only through `ENV_ALLOW`'s variables,
never with the host's `~/.pi/agent`.
There is no unconfined fallback. `sandboxProblem` runs bwrap once with the
system binds; when bwrap is missing or fails, the launcher refuses the
launch with `sandbox-unavailable` and the reason.
The namespaces and how a session dies:
- bwrap unshares the user, PID and IPC namespaces. The runner is pid 1 of
its PID namespace (`--as-pid-1`). A detached child is reparented to it,
so it stays under the launch's pid. Pid 1 ignores a signal it has no
handler for; the runner installs its handlers first, and the senders
resend SIGTERM to cover Node's startup.
- The runner dies when the outer bwrap process does: bwrap starts it
through `setpriv --pdeathsig KILL`. bwrap's own `--die-with-parent`
isn't used, because it also ties bwrap to the host, and a session has to
outlive a host that died hard so the next host can recover it. The
parent-death signal is set inside the namespace just before the runner's
exec; if bwrap died in that instant, the runner would keep going until
its broker polls fail.
- With its own `/proc`, a session sees no process outside its namespace,
so it can't signal or `ptrace` another session or the host by pid.
- The outer bwrap exits with 128 plus the signal number when the runner is
killed; `sessionExit` turns that back into the signal for the launch
log.
- The runner doesn't reap zombies other than its own turns; they go when - The runner doesn't reap zombies other than its own turns; they go when
the namespace does. the namespace does.
The sandbox doesn't confine the network, abstract Unix sockets, or the
user: a session runs as the host's UID. `packages/harness/README.md`
("Limits") says what that leaves open.
## Exit codes ## Exit codes
`launch` exits with the launch script's own code once the script runs. `launch` exits with the launch script's own code once the script runs.
packages/seat/src/sandbox.mjs
+143
View File
@@ -0,0 +1,143 @@
// The confinement a managed role session runs in (#1586). The runner and
// everything it starts, the harness and every shell the harness opens, run
// inside a bubblewrap mount namespace built from an empty root. The session
// sees only the binds below, each at its own host path so session.json's
// paths mean the same inside and out:
//
// read-only /usr, /etc and the host's /bin, /lib, /lib64, /sbin links;
// the repository files the runner loads; the adapter; for pi
// the repository's node_modules, for Claude Code its pinned
// binary; node when it lives outside /usr; the broker socket,
// and the launch socket for the role that may launch
// read-write the instance's workspace and the run directory, except the
// run's bundle/ and session.json, which stay read-only
// private /tmp (the run's tmp/), HOME (the run's home/), /dev, /proc
//
// Nothing else exists inside: no data root, no other workspace or run
// directory, no ~/.config, ~/.mosaic or ~/.ssh, no checkout beyond the
// runner's files. The root is remounted read-only, so a write anywhere but
// a read-write bind fails. The session keeps a user and PID namespace (the
// runner is pid 1, as before) and an IPC namespace. The runner dies with the
// bwrap process outside, as it did with unshare's --kill-child: setpriv
// gives it SIGKILL as its parent-death signal. bwrap's own --die-with-parent
// isn't used, because it also kills bwrap when the host dies, and a session
// outlives a host that died hard (the next host recovers it). The network,
// abstract Unix sockets and the host UID are not confined
// (packages/harness/README.md, "Limits").
//
// There is no unconfined fallback: a bwrap that is missing or fails refuses
// the launch (sandbox-unavailable).
import { spawnSync } from "node:child_process";
import { existsSync, lstatSync, mkdirSync, readlinkSync, realpathSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
export const REPO = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
export const BWRAP = "/usr/bin/bwrap";
const KILL_WITH_PARENT = ["/usr/bin/setpriv", "--pdeathsig", "KILL", "--"];
// What the runner imports, and the three files the harness starts from the
// bundle's settings (claude-gate, mcp-server, pi-extension) with theirs.
// packages/seat/tests/sandbox.test.mjs walks the imports and fails when one
// is missing here.
export const RUNNER_FILES = Object.freeze([
"packages/harness/src/runner.mjs",
"packages/harness/src/bundle.mjs",
"packages/harness/src/gate.mjs",
"packages/harness/src/tools.mjs",
"packages/harness/src/claude-gate.mjs",
"packages/harness/src/mcp-server.mjs",
"packages/harness/src/pi-extension.mjs",
"packages/bus/src/client.mjs",
"packages/bus/src/broker.mjs",
"packages/business/src/vocabulary.mjs",
].map((p) => join(REPO, p)));
const SYSTEM_LINKS = ["/bin", "/lib", "/lib64", "/sbin"];
const NAMESPACES = ["--unshare-user", "--unshare-pid", "--unshare-ipc", "--as-pid-1"];
// The system part every session gets, and the probe runs: /usr, the host's
// links into it, /etc, a private /dev and /proc, and node when it lives
// outside /usr.
function systemBinds(node = process.execPath) {
const out = [{ kind: "ro", src: "/usr", why: "the system's programs and libraries: sh, node, git, coreutils" }];
for (const p of SYSTEM_LINKS) {
let st;
try {
st = lstatSync(p);
} catch {
continue;
}
if (st.isSymbolicLink()) out.push({ kind: "symlink", src: readlinkSync(p), dest: p, why: `the host's ${p} link into /usr` });
else out.push({ kind: "ro", src: p, why: `the host's ${p}, which isn't a link into /usr` });
}
out.push(
{ kind: "ro", src: "/etc", why: "name resolution, CA certificates, passwd for the user's name" },
{ kind: "dev", dest: "/dev", why: "a private /dev with null, zero, urandom and a pty" },
{ kind: "proc", dest: "/proc", why: "the session's own PID namespace; node and the shell read /proc/self" },
);
const real = realpathSync(node);
if (!real.startsWith("/usr/")) out.push({ kind: "ro", src: real, dest: node, why: "node, which runs the runner and the gate, lives outside /usr" });
return out;
}
// The binds for one session, in mount order; each says why the session
// needs it. paths: the run directory, the instance's workspace, the adapter,
// the harness ("pi" or "claude-code"), the broker socket, the launch socket
// (null for a role that may not launch), the Claude Code binary
// (MOSAIC_CLAUDE_BIN, or null) and node. `readOnly` adds files, for tests only.
export function binds({ runDir, workspace, adapter, harness, brokerSocket, launchSocket = null, claudeBin = null, node = process.execPath, readOnly = [] }) {
const out = systemBinds(node);
for (const f of RUNNER_FILES) out.push({ kind: "ro", src: f, why: "a repository file the runner loads" });
out.push({ kind: "ro", src: adapter, why: "the harness adapter the runner starts each turn" });
if (harness === "pi") out.push({ kind: "ro", src: realpathSync(join(REPO, "node_modules")), dest: join(REPO, "node_modules"), why: "pi and its packages, on the session's PATH" });
if (harness === "claude-code" && claudeBin) {
const bin = realpathSync(claudeBin);
if (!bin.startsWith("/usr/")) out.push({ kind: "ro", src: bin, dest: claudeBin, why: "the pinned Claude Code binary (MOSAIC_CLAUDE_BIN)" });
}
out.push({ kind: "ro", src: brokerSocket, why: "the broker socket the runner claims and calls through" });
if (launchSocket) out.push({ kind: "ro", src: launchSocket, why: "the launch socket, for the role whose launch tool calls it" });
for (const f of readOnly) out.push({ kind: "ro", src: f, why: "a test fixture" });
out.push(
{ kind: "rw", src: workspace, why: "the instance's workspace, where the harness works" },
{ kind: "rw", src: runDir, why: "the run directory: turns/, session/, tools.sock, turn.done, home/, tmp/" },
{ kind: "ro", src: join(runDir, "bundle"), why: "the run's bundle stays as the host wrote it" },
{ kind: "ro", src: join(runDir, "session.json"), why: "the session file stays as the host wrote it" },
{ kind: "rw", src: join(runDir, "tmp"), dest: "/tmp", why: "a private /tmp, on disk in the run directory" },
);
return out;
}
function argsOf(list) {
const args = [];
for (const b of list) {
if (b.kind === "ro") args.push("--ro-bind", b.src, b.dest ?? b.src);
else if (b.kind === "rw") args.push("--bind", b.src, b.dest ?? b.src);
else if (b.kind === "symlink") args.push("--symlink", b.src, b.dest);
else if (b.kind === "dev") args.push("--dev", b.dest);
else if (b.kind === "proc") args.push("--proc", b.dest);
}
return args;
}
// The run's home/ and tmp/, made before the spawn: binds need their sources.
export function prepareRun(runDir) {
for (const d of ["home", "tmp"]) mkdirSync(join(runDir, d), { recursive: true, mode: 0o700 });
}
// The full command line for `command` inside the sandbox. HOME is the run's
// home/, empty at the start; the harness makes what it needs there.
export function sandboxArgv(list, runDir, command, { bwrap = BWRAP } = {}) {
return [bwrap, ...NAMESPACES, ...argsOf(list), "--remount-ro", "/", "--setenv", "HOME", join(runDir, "home"), "--setenv", "TMPDIR", "/tmp", "--chdir", runDir, "--", ...KILL_WITH_PARENT, ...command];
}
// Whether bwrap runs here at all: the namespaces and the system binds, then
// setpriv and `true`. Returns null, or why not.
export function sandboxProblem({ bwrap = BWRAP } = {}) {
if (!existsSync(bwrap)) return `${bwrap} is missing`;
const r = spawnSync(bwrap, [...NAMESPACES, ...argsOf(systemBinds()), "--remount-ro", "/", "--", ...KILL_WITH_PARENT, "/usr/bin/true"], { encoding: "utf8", timeout: 10000 });
if (r.status === 0) return null;
const why = (r.stderr ?? "").trim().split("\n")[0] || (r.error ? r.error.code : `exit ${r.status ?? r.signal}`);
return `${bwrap} failed: ${why}`;
}
packages/seat/src/session.mjs
+42 -24
View File
@@ -4,12 +4,13 @@
// decides authority and capacity first, and this module only does what it // decides authority and capacity first, and this module only does what it
// is told. // is told.
// //
// A session is `node packages/harness/src/runner.mjs <run dir>` inside // A session is `node packages/harness/src/runner.mjs <run dir>` inside the
// `unshare --user --map-current-user --pid --fork --kill-child --mount-proc`. // bubblewrap sandbox of sandbox.mjs, which sees only its own workspace, its
// The runner is pid 1 of its own PID namespace, so a child that detaches // run directory and the files it runs from. The runner is pid 1 of its own
// (`setsid -f`) is reparented to the runner and stays under the launch's // PID namespace, so a child that detaches (`setsid -f`) is reparented to the
// pid, which the human CLI's ancestry check refuses. unshare itself ignores // runner and stays under the launch's pid, which the human CLI's ancestry
// SIGTERM and passes the runner's exit status on; `stop` signals the runner. // check refuses. bwrap passes the runner's exit status on, a death by signal
// as 128 + the signal; `stop` signals the runner.
// //
// On disk, under the data root: // On disk, under the data root:
// launches/<business>/<run>/ the run directory (0700): session.json, // launches/<business>/<run>/ the run directory (0700): session.json,
@@ -25,18 +26,18 @@
import { spawn } from "node:child_process"; import { spawn } from "node:child_process";
import { randomBytes } from "node:crypto"; import { randomBytes } from "node:crypto";
import { appendFileSync, closeSync, existsSync, mkdirSync, openSync, readFileSync, renameSync, writeFileSync } from "node:fs"; import { appendFileSync, closeSync, existsSync, mkdirSync, openSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path"; import { constants } from "node:os";
import { fileURLToPath } from "node:url"; import { dirname, join } from "node:path";
import { childrenOf, cmdlineOf, startTimeOf } from "./proc.mjs"; import { childrenOf, cmdlineOf, startTimeOf } from "./proc.mjs";
import { binds, prepareRun, REPO, sandboxArgv } from "./sandbox.mjs";
import { SeatError } from "./seat.mjs"; import { SeatError } from "./seat.mjs";
export const REPO = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", ".."); export { REPO };
export const RUNNER = join(REPO, "packages", "harness", "src", "runner.mjs"); export const RUNNER = join(REPO, "packages", "harness", "src", "runner.mjs");
export const ADAPTERS = Object.freeze({ export const ADAPTERS = Object.freeze({
pi: join(REPO, "adapters", "pi", "adapter.sh"), pi: join(REPO, "adapters", "pi", "adapter.sh"),
"claude-code": join(REPO, "adapters", "claude", "adapter.sh"), "claude-code": join(REPO, "adapters", "claude", "adapter.sh"),
}); });
export const UNSHARE = Object.freeze(["unshare", "--user", "--map-current-user", "--pid", "--fork", "--kill-child", "--mount-proc"]);
export const SESSIONS_VERSION = 1; export const SESSIONS_VERSION = 1;
export const sessionsFile = (dataRoot) => join(dataRoot, "bus-host", "sessions.json"); export const sessionsFile = (dataRoot) => join(dataRoot, "bus-host", "sessions.json");
@@ -57,7 +58,8 @@ export function family(model, max) {
// environment passes, so a founder credential variable (GITEA_TOKEN, // environment passes, so a founder credential variable (GITEA_TOKEN,
// GH_TOKEN, SSH_AUTH_SOCK, ...) never reaches a session (REQ-CRED-2). // GH_TOKEN, SSH_AUTH_SOCK, ...) never reaches a session (REQ-CRED-2).
// Provider variables pass because the harness needs them to reach a model. // Provider variables pass because the harness needs them to reach a model.
// MOSAIC_CLAUDE_BIN, the host's pinned Claude Code binary, is a path. // MOSAIC_CLAUDE_BIN, the host's pinned Claude Code binary, is a path. HOME
// passes to bwrap only; inside, HOME is the run's own home/.
export const ENV_ALLOW = Object.freeze([ export const ENV_ALLOW = Object.freeze([
"HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TERM", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TERM",
"ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_BASE_URL", "CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_BASE_URL", "CLAUDE_CODE_OAUTH_TOKEN",
@@ -65,6 +67,13 @@ export const ENV_ALLOW = Object.freeze([
"MOSAIC_CLAUDE_BIN", "MOSAIC_CLAUDE_BIN",
]); ]);
// The ways a Claude Code session can log in, each a set of variables that
// must all be set; ENV_ALLOW passes them on. The bus unit loads them
// (packages/cli/README.md, "Unit"). The session gets nothing from ~/.claude,
// which the sandbox doesn't show.
export const SESSION_CREDENTIALS = Object.freeze([["CLAUDE_CODE_OAUTH_TOKEN"], ["ANTHROPIC_API_KEY"], ["ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_BASE_URL"]].map((set) => Object.freeze(set)));
export const hasSessionCredential = (env) => SESSION_CREDENTIALS.some((set) => set.every((k) => Boolean(env[k])));
export function sessionEnv(env, run) { export function sessionEnv(env, run) {
const out = {}; const out = {};
for (const k of ENV_ALLOW) if (env[k] !== undefined) out[k] = env[k]; for (const k of ENV_ALLOW) if (env[k] !== undefined) out[k] = env[k];
@@ -103,13 +112,15 @@ export function appendLaunchLog(dataRoot, business, entry) {
appendFileSync(file, `${JSON.stringify({ at: new Date().toISOString(), ...entry })}\n`, { mode: 0o600 }); appendFileSync(file, `${JSON.stringify({ at: new Date().toISOString(), ...entry })}\n`, { mode: 0o600 });
} }
// Start the runner. stdin stays open for the capability line the host writes // Start the runner in its sandbox. stdin stays open for the capability line
// once the broker has bound the launch; stdout and stderr go to runner.log. // the host writes once the broker has bound the launch; stdout and stderr go
// `namespace: false` exists for hosts without unprivileged user namespaces // to runner.log. paths: what sandbox.mjs's binds() takes besides the run
// in the tests only; the host always passes true. // directory (workspace, adapter, harness, brokerSocket, launchSocket,
export function spawnSession({ runDir, env, namespace = true, node = process.execPath }) { // claudeBin; readOnly for tests). `bwrap` is for the tests.
export function spawnSession({ runDir, env, paths, node = process.execPath, bwrap }) {
prepareRun(runDir);
const argv = sandboxArgv(binds({ ...paths, runDir, node }), runDir, [node, RUNNER, runDir], { bwrap });
const out = openSync(join(runDir, "runner.log"), "a", 0o600); const out = openSync(join(runDir, "runner.log"), "a", 0o600);
const argv = [...(namespace ? UNSHARE : []), node, RUNNER, runDir];
try { try {
return spawn(argv[0], argv.slice(1), { cwd: runDir, env, stdio: ["pipe", out, out], detached: true }); return spawn(argv[0], argv.slice(1), { cwd: runDir, env, stdio: ["pipe", out, out], detached: true });
} finally { } finally {
@@ -118,18 +129,17 @@ export function spawnSession({ runDir, env, namespace = true, node = process.exe
} }
// A runner's command line is `<node> <RUNNER> <run dir>`. Matching argv[1] // A runner's command line is `<node> <RUNNER> <run dir>`. Matching argv[1]
// rather than any argument skips unshare's forked child before its exec, // rather than any argument skips bwrap's forked child before its exec, whose
// whose argv still names the runner. // argv still names the runner.
export const isRunner = (pid) => (cmdlineOf(pid) ?? [])[1] === RUNNER; export const isRunner = (pid) => (cmdlineOf(pid) ?? [])[1] === RUNNER;
// The runner's pid and start time as seen from here. Under unshare the // The runner's pid and start time as seen from here: the outer bwrap
// runner is the outer process's only child; without a namespace it is the // process's only child.
// process itself. export async function runnerOf(child, { timeoutMs = 5000 } = {}) {
export async function runnerOf(child, { namespace = true, timeoutMs = 5000 } = {}) {
const deadline = Date.now() + timeoutMs; const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) { while (Date.now() < deadline) {
if (child.exitCode !== null || child.signalCode !== null) break; if (child.exitCode !== null || child.signalCode !== null) break;
const pid = namespace ? childrenOf(child.pid)[0] : child.pid; const pid = childrenOf(child.pid)[0];
const start = pid ? startTimeOf(pid) : null; const start = pid ? startTimeOf(pid) : null;
if (start && isRunner(pid)) return { pid, startTime: start }; if (start && isRunner(pid)) return { pid, startTime: start };
await new Promise((r) => setTimeout(r, 20)); await new Promise((r) => setTimeout(r, 20));
@@ -137,6 +147,14 @@ export async function runnerOf(child, { namespace = true, timeoutMs = 5000 } = {
throw new SeatError(`runner for ${child.pid} did not start`, 1); throw new SeatError(`runner for ${child.pid} did not start`, 1);
} }
// The outer bwrap's exit as the runner's: bwrap exits 128 + n when the
// runner dies of signal n, and the runner's own codes are all below 128.
const SIGNALS = Object.fromEntries(Object.entries(constants.signals).map(([name, n]) => [n, name]));
export function sessionExit(code, signal) {
if (!signal && code > 128 && SIGNALS[code - 128]) return { code: null, signal: SIGNALS[code - 128] };
return { code, signal };
}
// The launch end reason the broker records for a runner's exit // The launch end reason the broker records for a runner's exit
// (packages/harness/README.md, "The runner"). // (packages/harness/README.md, "The runner").
const REASONS = { 0: "stopped", 2: "failed", 20: "founder-credentials", 21: "claim-refused", 22: "session-invalid", 23: "broker-unreachable" }; const REASONS = { 0: "stopped", 2: "failed", 20: "founder-credentials", 21: "claim-refused", 22: "session-invalid", 23: "broker-unreachable" };
packages/seat/tests/sandbox.test.mjs
+259
View File
@@ -0,0 +1,259 @@
// The sandbox a role session runs in (src/sandbox.mjs, #1586). A real
// session, the runner in its bwrap sandbox against an in-process broker,
// answers each message by running the body in /bin/sh; the tests read the
// answers. Stand-ins for the founder's files sit in the test's own
// directory, plus a file in the host's /dev/shm and a SysV shared memory
// segment, each checked to exist on the host first, so a refusal means the
// sandbox hid it.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { basename, dirname, join, resolve } from "node:path";
import { Broker } from "../../bus/src/broker.mjs";
import { serve } from "../../bus/src/server.mjs";
import { Store } from "../../bus/src/store.mjs";
import { SOURCES } from "../../harness/src/bundle.mjs";
import { startTimeOf } from "../src/proc.mjs";
import { newRun, RUNNER, runnerOf, sessionEnv, spawnSession, writeSessionFile } from "../src/session.mjs";
import { binds, prepareRun, REPO, RUNNER_FILES, sandboxArgv, sandboxProblem } from "../src/sandbox.mjs";
const FAKE = new URL("../../harness/tests/fixtures/fake-adapter.mjs", import.meta.url).pathname;
const skip = sandboxProblem() ?? false;
function scratch(t) {
const root = realpathSync(mkdtempSync(join(tmpdir(), "mosaic-sandbox-")));
t.after(() => rmSync(root, { recursive: true, force: true }));
return root;
}
test("RUNNER_FILES is exactly what the runner and the harness sources it starts import", () => {
const seen = new Set();
const walk = (file) => {
if (seen.has(file)) return;
seen.add(file);
for (const [, rel] of readFileSync(file, "utf8").matchAll(/(?:from|import\()\s*["'](\.\.?\/[^"']+)["']/g)) walk(resolve(dirname(file), rel));
};
for (const file of [RUNNER, ...Object.values(SOURCES)]) walk(file);
assert.deepEqual([...seen].sort(), [...RUNNER_FILES].sort());
});
test("sandboxProblem names a missing bwrap and a bwrap that fails", (t) => {
const root = scratch(t);
assert.equal(sandboxProblem({ bwrap: join(root, "nope") }), `${join(root, "nope")} is missing`);
const bad = join(root, "bwrap");
writeFileSync(bad, "#!/bin/sh\necho 'bwrap: No permissions to create new namespace' >&2\nexit 1\n");
chmodSync(bad, 0o755);
assert.equal(sandboxProblem({ bwrap: bad }), `${bad} failed: bwrap: No permissions to create new namespace`);
if (!skip) assert.equal(sandboxProblem(), null);
});
test("every bind says why; the bundle and session file stay read-only over the run directory", (t) => {
const root = scratch(t);
const runDir = join(root, "run");
const list = binds({ runDir, workspace: join(root, "ws"), adapter: join(root, "a.sh"), harness: "claude-code", brokerSocket: join(root, "bus.sock") });
assert.ok(list.every((b) => b.why));
const at = (src) => list.findIndex((b) => b.src === src);
assert.equal(list[at(runDir)].kind, "rw");
for (const f of ["bundle", "session.json"]) {
assert.equal(list[at(join(runDir, f))].kind, "ro");
assert.ok(at(join(runDir, f)) > at(runDir), `${f} is mounted over the run directory`);
}
assert.deepEqual(list.at(-1), { kind: "rw", src: join(runDir, "tmp"), dest: "/tmp", why: list.at(-1).why });
assert.equal(at(join(root, "launch.sock")), -1, "no launch socket for a role that may not launch");
assert.ok(!list.some((b) => b.src === realpathSync(join(REPO, "node_modules"))), "Claude Code gets no node_modules");
});
// The founder's files and the other sessions' state, as stand-ins. The
// /dev/shm file and the SysV segment are the host's, made for this test and
// removed after it.
function standIns(t, root, dataRoot) {
const s = {
secret: join(root, "home", ".config", "mosaic-dev", "secrets", "stand-in.env"),
mosaic: join(root, "home", ".mosaic", "fleet", "token"),
otherWs: join(root, "ws-other"),
shm: join("/dev/shm", basename(root)),
};
writeFileSync(s.shm, "stand-in\n", { mode: 0o600 });
t.after(() => rmSync(s.shm, { force: true }));
const ipc = spawnSync("/usr/bin/ipcmk", ["-M", "4096"], { encoding: "utf8" });
assert.equal(ipc.status, 0, ipc.stderr);
s.segment = Number(ipc.stdout.trim().split(" ").at(-1));
t.after(() => spawnSync("/usr/bin/ipcrm", ["-m", String(s.segment)]));
for (const f of [s.secret, s.mosaic]) {
mkdirSync(dirname(f), { recursive: true, mode: 0o700 });
writeFileSync(f, "stand-in\n", { mode: 0o600 });
}
mkdirSync(s.otherWs);
writeFileSync(join(s.otherWs, "notes.md"), "another instance's work\n");
const other = newRun(dataRoot, "acme");
writeFileSync(join(other.runDir, "session.json"), "{}\n", { mode: 0o600 });
s.otherRun = other.runDir;
return s;
}
// [what, shell command, may the session do it, the output it must give].
function checks({ s, ws, runDir, run, dataRoot, store }) {
return [
["HOME is the run's home/, empty", `printf %s "$HOME:"; ls -A "$HOME"`, true, `${join(runDir, "home")}:`],
["TMPDIR is the private /tmp", `printf %s "$TMPDIR"`, true, "/tmp"],
["read a stand-in ~/.config/mosaic-dev/secrets", `cat '${s.secret}'`, false],
["read a stand-in ~/.mosaic", `cat '${s.mosaic}'`, false],
["read another instance's workspace", `cat '${s.otherWs}/notes.md'`, false],
["read another run directory", `cat '${s.otherRun}/session.json'`, false],
["read the data root's store", `cat '${store}' >/dev/null`, false],
["the data root holds only this run", `ls -A '${dataRoot}'; ls -A '${dataRoot}/launches/acme'`, true, `launches\n${run}`],
["read the checkout outside the runner's files", `cat '${join(REPO, "AGENTS.md")}' >/dev/null`, false],
["read the seat's own source", `cat '${join(REPO, "packages", "seat", "src", "session.mjs")}' >/dev/null`, false],
["read a stand-in in the host's /dev/shm", `cat '${s.shm}'`, false],
["see the test's own pid in /proc", `test -e /proc/${process.pid}`, false],
["read its own IPC namespace's segment list", "test -r /proc/sysvipc/shm", true],
["see a SysV segment the host made", `awk '$2 == ${s.segment} { f = 1 } END { exit !f }' /proc/sysvipc/shm`, false],
["write to /", "touch /probe", false],
["write beside the workspace", `touch '${dirname(ws)}/probe'`, false],
["write in the data root", `touch '${dataRoot}/probe'`, false],
["write a runner file", `test -w '${RUNNER}'`, false],
["write the run's bundle", `test -w '${runDir}/bundle/prompt.md' || touch '${runDir}/bundle/probe'`, false],
["write the session file", `test -w '${runDir}/session.json'`, false],
["write and read its workspace", `echo ws > '${ws}/probe' && cat '${ws}/probe'`, true, "ws"],
["write and read its run directory", `echo run > '${runDir}/probe' && cat '${runDir}/probe'`, true, "run"],
["write and read /tmp", "echo tmp > /tmp/probe && cat /tmp/probe", true, "tmp"],
["run node", `'${process.execPath}' -e 'console.log(6 * 7)'`, true, "42"],
["run pi, the session's harness", `'${join(REPO, "node_modules", ".bin", "pi")}' --version >/dev/null`, true],
];
}
// The checks whose answer isn't the expected one, by name.
async function wrong(table, sh) {
const out = [];
for (const [what, cmd, may, expect] of table) {
const r = await sh(cmd);
if ((r.rc === 0) !== may || (may && expect !== undefined && r.out !== expect)) out.push(what);
}
return out;
}
// A real session for coder, its adapter a shell that runs each message's
// body, or the fake adapter for a TOOL message.
// expose(standIns) names extra read-only binds, for the test that breaks
// the sandbox on purpose.
async function session(t, { expose = () => [] } = {}) {
const root = scratch(t);
const dataRoot = join(root, "data");
mkdirSync(dataRoot, { mode: 0o700 });
const store = new Store(dataRoot);
const broker = new Broker({
store,
businesses: {
acme: {
id: "acme",
human: "jason",
arbiters: { technical: "pm", delivery: "pm" },
roles: {
pm: { authority: { withinRole: ["message.send"], crossRole: [] } },
coder: { authority: { withinRole: ["message.send"], crossRole: [] } },
},
},
},
});
const brokerSocket = join(root, "bus.sock");
const server = await serve({ broker, path: brokerSocket });
t.after(async () => {
await server.close().catch(() => {});
store.close();
});
const { run, runDir } = newRun(dataRoot, "acme");
const s = standIns(t, root, dataRoot);
const bundle = join(runDir, "bundle");
const ws = join(root, "ws");
mkdirSync(bundle, { mode: 0o700 });
mkdirSync(ws);
writeFileSync(join(bundle, "prompt.md"), "# test\n");
writeFileSync(join(bundle, "policy.json"), JSON.stringify({ harness: "pi", workspace: ws, tools: [], typed: [], network: "none", needs: [], credentials: [] }));
writeFileSync(join(bundle, "tools.json"), "[]");
const adapter = join(root, "adapter.sh");
writeFileSync(adapter, [
`body=$(printf '%s\\n' "$MOSAIC_REQUEST" | sed '1,/^$/d')`,
`case $body in TOOL\\ *) exec '${process.execPath}' '${FAKE}' ;; esac`,
`out=$(cd "$MOSAIC_WORKSPACE" && /bin/sh -c "$body" 2>&1); rc=$?`,
`printf 'rc=%s\\n%s' "$rc" "$out"`,
`echo done > "$MOSAIC_TURN_MARKER"`,
"",
].join("\n"), { mode: 0o644 });
writeSessionFile(runDir, {
business: "acme", instance: "coder", run, launchedBy: "pm", harness: "pi", provider: "fake", model: "fake-model",
brokerSocket, launchSocket: join(root, "launch.sock"), workspace: ws, sessionDir: join(runDir, "session"), adapter,
bundle: { prompt: join(bundle, "prompt.md"), policy: join(bundle, "policy.json"), tools: join(bundle, "tools.json") },
toolSocket: join(runDir, "tools.sock"), turnMarker: join(runDir, "turn.done"), pollInterval: 50,
});
const paths = { workspace: ws, adapter, harness: "pi", brokerSocket, readOnly: [FAKE, ...expose(s)] };
const child = spawnSession({ runDir, env: sessionEnv({ HOME: process.env.HOME, PATH: process.env.PATH }, run), paths });
const exited = new Promise((r) => child.once("close", (code, signal) => r({ code, signal })));
t.after(async () => {
try {
process.kill(child.pid, "SIGTERM");
} catch {}
await exited;
});
await runnerOf(child);
const cap = broker.bindLaunch({ business: "acme", role: "coder", run, harness: "pi", pid: child.pid, startTime: startTimeOf(child.pid) });
const pm = broker.bindLaunch({ business: "acme", role: "pm", run: "pm-run", harness: "pi" });
broker.request(pm, { verb: "role.claim" });
child.stdin.end(`${JSON.stringify({ cap })}\n`);
const ask = async (body) => {
const sent = broker.request(pm, { verb: "message.send", args: { to: "coder", body, class: "REQUEST" } });
const end = Date.now() + 15000;
while (Date.now() < end) {
const reply = broker.request(pm, { verb: "message.receive" }).find((m) => m.in_reply_to === sent.id);
if (reply) return reply.body;
await new Promise((r) => setTimeout(r, 25));
}
throw new Error(`no answer to ${body}`);
};
const sh = async (cmd) => {
const m = (await ask(cmd)).match(/^rc=(\d+)(?:\n([\s\S]*))?$/);
assert.ok(m, `the session's shell answered ${cmd}`);
return { rc: Number(m[1]), out: m[2] ?? "" };
};
const facts = { s, ws, runDir, run, dataRoot, store: store.path, root, adapter, brokerSocket };
for (const f of [s.secret, s.mosaic, s.shm, join(s.otherWs, "notes.md"), join(s.otherRun, "session.json"), store.path, join(REPO, "AGENTS.md"), `/proc/${process.pid}`]) {
assert.ok(existsSync(f), `${f} exists on the host`);
}
const listed = spawnSync("awk", [`$2 == ${s.segment} { f = 1 } END { exit !f }`, "/proc/sysvipc/shm"]);
assert.equal(listed.status, 0, `segment ${s.segment} is listed on the host`);
return { ...facts, ask, sh };
}
test("a confined session's shell reaches its own workspace and run directory and nothing of the founder's or another session's", { skip, timeout: 60000 }, async (t) => {
const x = await session(t);
assert.deepEqual(await wrong(checks(x), x.sh), []);
// The private /tmp is the run's tmp/ on the host, not the host's /tmp.
assert.equal(readFileSync(join(x.runDir, "tmp", "probe"), "utf8"), "tmp\n");
assert.equal(readFileSync(join(x.ws, "probe"), "utf8"), "ws\n");
// The harness reaches its tool socket: the refusal is the tool server's,
// not a failed connect.
assert.equal(await x.ask("TOOL no_such_tool {}"), "refused: unknown-tool");
});
test("a bind that exposes a stand-in, or a missing workspace bind, fails the matching check", { skip, timeout: 60000 }, async (t) => {
const x = await session(t, { expose: (s) => [s.secret, s.mosaic] });
assert.deepEqual(await wrong(checks(x), x.sh), ["read a stand-in ~/.config/mosaic-dev/secrets", "read a stand-in ~/.mosaic"]);
// The same checks in a sandbox started directly from binds(), without the
// session: all pass with every bind, and the workspace check fails
// without the workspace's.
const all = binds({ runDir: x.runDir, workspace: x.ws, adapter: x.adapter, harness: "pi", brokerSocket: x.brokerSocket });
const direct = (list) => async (cmd) => {
const argv = sandboxArgv(list, x.runDir, ["/bin/sh", "-c", cmd]);
const r = spawnSync(argv[0], argv.slice(1), { encoding: "utf8", timeout: 10000 });
return { rc: r.status, out: (r.stdout + r.stderr).replace(/\n$/, "") };
};
prepareRun(x.runDir);
assert.deepEqual(await wrong(checks(x), direct(all)), []);
assert.deepEqual(await wrong(checks(x), direct(all.filter((b) => b.src !== x.ws))), ["write and read its workspace"]);
});
packages/seat/tests/session.test.mjs
+31 -10
View File
@@ -1,11 +1,10 @@
// Managed role sessions on disk and as processes (src/session.mjs). The // Managed role sessions on disk and as processes (src/session.mjs). The
// runner is the real one, under the real unshare, against an in-process // runner is the real one, in the real bwrap sandbox, against an in-process
// broker on a real socket; the harness turn is packages/harness's fake // broker on a real socket; the harness turn is packages/harness's fake
// adapter. The launcher's checks are tested in packages/cli. // adapter. The launcher's checks are tested in packages/cli.
import { test } from "node:test"; import { test } from "node:test";
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
@@ -19,6 +18,8 @@ import {
endReason, endReason,
ENV_ALLOW, ENV_ALLOW,
family, family,
hasSessionCredential,
SESSION_CREDENTIALS,
isRunner, isRunner,
launchLogFile, launchLogFile,
newRun, newRun,
@@ -28,14 +29,14 @@ import {
sessionEnv, sessionEnv,
sessionsFile, sessionsFile,
spawnSession, spawnSession,
sessionExit,
stopSession, stopSession,
UNSHARE,
writeSessionFile, writeSessionFile,
} from "../src/session.mjs"; } from "../src/session.mjs";
import { BWRAP, sandboxProblem } from "../src/sandbox.mjs";
const FAKE = new URL("../../harness/tests/fixtures/fake-adapter.mjs", import.meta.url).pathname; const FAKE = new URL("../../harness/tests/fixtures/fake-adapter.mjs", import.meta.url).pathname;
const NAMESPACES = spawnSync(UNSHARE[0], [...UNSHARE.slice(1), "true"]).status === 0; const skip = sandboxProblem() ?? false;
const skip = NAMESPACES ? false : "unprivileged user and PID namespaces are unavailable here";
function scratch(t) { function scratch(t) {
const dir = realpathSync(mkdtempSync(join(tmpdir(), "mosaic-session-"))); const dir = realpathSync(mkdtempSync(join(tmpdir(), "mosaic-session-")));
@@ -63,6 +64,24 @@ test("sessionEnv passes only the allowlist, the repo's bin on PATH, and the run
assert.equal(sessionEnv({ MOSAIC_CLAUDE_BIN: "/opt/claude" }, "r0123456789ab").MOSAIC_CLAUDE_BIN, "/opt/claude"); assert.equal(sessionEnv({ MOSAIC_CLAUDE_BIN: "/opt/claude" }, "r0123456789ab").MOSAIC_CLAUDE_BIN, "/opt/claude");
}); });
test("sessionExit gives a runner's signal death back as the signal; a session credential is one whole set of nonempty variables", () => {
assert.deepEqual(sessionExit(0, null), { code: 0, signal: null });
assert.deepEqual(sessionExit(23, null), { code: 23, signal: null });
assert.deepEqual(sessionExit(128, null), { code: 128, signal: null });
assert.deepEqual(sessionExit(137, null), { code: null, signal: "SIGKILL" });
assert.deepEqual(sessionExit(143, null), { code: null, signal: "SIGTERM" });
assert.deepEqual(sessionExit(null, "SIGTERM"), { code: null, signal: "SIGTERM" }, "bwrap itself killed");
assert.equal(endReason(...Object.values(sessionExit(137, null))), "killed");
assert.equal(hasSessionCredential({ CLAUDE_CODE_OAUTH_TOKEN: "t" }), true);
assert.equal(hasSessionCredential({ ANTHROPIC_API_KEY: "k" }), true);
assert.equal(hasSessionCredential({ ANTHROPIC_AUTH_TOKEN: "t", ANTHROPIC_BASE_URL: "http://127.0.0.1:1" }), true, "a proxy's key");
assert.equal(hasSessionCredential({ ANTHROPIC_AUTH_TOKEN: "t" }), false, "a key with no proxy to send it to");
assert.equal(hasSessionCredential({ ANTHROPIC_BASE_URL: "http://127.0.0.1:1" }), false);
assert.equal(hasSessionCredential({ CLAUDE_CODE_OAUTH_TOKEN: "", ANTHROPIC_API_KEY: "" }), false);
assert.equal(hasSessionCredential({}), false);
for (const k of SESSION_CREDENTIALS.flat()) assert.ok(ENV_ALLOW.includes(k), k);
});
test("newRun: short ids, 0700 directories, and a refusal when the socket path won't fit", (t) => { test("newRun: short ids, 0700 directories, and a refusal when the socket path won't fit", (t) => {
const root = scratch(t); const root = scratch(t);
const a = newRun(join(root, "data"), "acme"); const a = newRun(join(root, "data"), "acme");
@@ -130,7 +149,7 @@ test("stopSession refuses an unknown run, reports a stale one, and won't signal
await assert.rejects(stopSession(root, "me"), /is not a session runner; refusing to signal it/); await assert.rejects(stopSession(root, "me"), /is not a session runner; refusing to signal it/);
}); });
test("a session runs under unshare as pid 1 of its namespace, claims, answers, and stops on mosaic stop", { skip }, async (t) => { test("a session runs in its sandbox as pid 1 of its namespace, claims, answers, and stops on mosaic stop", { skip }, async (t) => {
const root = scratch(t); const root = scratch(t);
const store = new Store(root); const store = new Store(root);
const broker = new Broker({ const broker = new Broker({
@@ -172,7 +191,8 @@ test("a session runs under unshare as pid 1 of its namespace, claims, answers, a
toolSocket: join(runDir, "tools.sock"), turnMarker: join(runDir, "turn.done"), pollInterval: 50, toolSocket: join(runDir, "tools.sock"), turnMarker: join(runDir, "turn.done"), pollInterval: 50,
}); });
const child = spawnSession({ runDir, env: sessionEnv({ HOME: process.env.HOME, PATH: process.env.PATH, GITEA_TOKEN: "x" }, run) }); const paths = { workspace: ws, adapter, harness: "pi", brokerSocket, readOnly: [FAKE] };
const child = spawnSession({ runDir, env: sessionEnv({ HOME: process.env.HOME, PATH: process.env.PATH, GITEA_TOKEN: "x" }, run), paths });
t.after(() => { t.after(() => {
try { try {
process.kill(child.pid, "SIGKILL"); process.kill(child.pid, "SIGKILL");
@@ -181,15 +201,16 @@ test("a session runs under unshare as pid 1 of its namespace, claims, answers, a
const exited = new Promise((r) => child.once("close", (code, signal) => r({ code, signal }))); const exited = new Promise((r) => child.once("close", (code, signal) => r({ code, signal })));
const runner = await runnerOf(child); const runner = await runnerOf(child);
assert.notEqual(runner.pid, child.pid); assert.notEqual(runner.pid, child.pid);
assert.deepEqual(cmdlineOf(child.pid).slice(0, UNSHARE.length), UNSHARE); assert.equal(cmdlineOf(child.pid)[0], BWRAP);
assert.ok(isRunner(runner.pid)); assert.ok(isRunner(runner.pid));
assert.equal(isRunner(child.pid), false, "unshare's argv names the runner too"); assert.equal(isRunner(child.pid), false, "bwrap's argv names the runner too");
// Inside its namespace the runner is pid 1. // Inside its namespace the runner is pid 1.
const nspid = readFileSync(`/proc/${runner.pid}/status`, "utf8").match(/^NSpid:\s*(.*)$/m)[1].trim().split(/\s+/); const nspid = readFileSync(`/proc/${runner.pid}/status`, "utf8").match(/^NSpid:\s*(.*)$/m)[1].trim().split(/\s+/);
assert.equal(nspid.at(-1), "1"); assert.equal(nspid.at(-1), "1");
const environ = readFileSync(`/proc/${runner.pid}/environ`, "utf8").split("\0"); const environ = readFileSync(`/proc/${runner.pid}/environ`, "utf8").split("\0");
assert.ok(environ.includes(`MOSAIC_RUN_ID=${run}`)); assert.ok(environ.includes(`MOSAIC_RUN_ID=${run}`));
assert.ok(!environ.some((v) => v.startsWith("GITEA_TOKEN="))); assert.ok(!environ.some((v) => v.startsWith("GITEA_TOKEN=")));
assert.ok(environ.includes(`HOME=${join(runDir, "home")}`), "HOME is the run's own");
const cap = broker.bindLaunch({ business: "acme", role: "coder", run, harness: "pi", pid: child.pid, startTime: startTimeOf(child.pid) }); const cap = broker.bindLaunch({ business: "acme", role: "coder", run, harness: "pi", pid: child.pid, startTime: startTimeOf(child.pid) });
const pm = broker.bindLaunch({ business: "acme", role: "pm", run: "pm-run", harness: "pi" }); const pm = broker.bindLaunch({ business: "acme", role: "pm", run: "pm-run", harness: "pi" });
@@ -209,7 +230,7 @@ test("a session runs under unshare as pid 1 of its namespace, claims, answers, a
registry.add({ business: "acme", instance: "coder", run, pid: child.pid, startTime: startTimeOf(child.pid), runnerPid: runner.pid, runnerStartTime: runner.startTime }); registry.add({ business: "acme", instance: "coder", run, pid: child.pid, startTime: startTimeOf(child.pid), runnerPid: runner.pid, runnerStartTime: runner.startTime });
const stopped = await stopSession(dataRoot, run, { timeoutMs: 10000 }); const stopped = await stopSession(dataRoot, run, { timeoutMs: 10000 });
assert.deepEqual(stopped, { stopped: true, business: "acme", instance: "coder", run }); assert.deepEqual(stopped, { stopped: true, business: "acme", instance: "coder", run });
assert.deepEqual(await exited, { code: 0, signal: null }, "unshare passes the runner's exit 0 on"); assert.deepEqual(await exited, { code: 0, signal: null }, "bwrap passes the runner's exit 0 on");
assert.equal(store.get("SELECT op FROM role_claims WHERE role='coder' ORDER BY seq DESC LIMIT 1").op, "release"); assert.equal(store.get("SELECT op FROM role_claims WHERE role='coder' ORDER BY seq DESC LIMIT 1").op, "release");
assert.equal(existsSync(join(runDir, "tools.sock")), false); assert.equal(existsSync(join(runDir, "tools.sock")), false);
assert.equal(mode(join(runDir, "runner.log")), 0o600); assert.equal(mode(join(runDir, "runner.log")), 0o600);