1551 lines
83 KiB
JavaScript
1551 lines
83 KiB
JavaScript
// The S6 launcher, inside the trusted bus host: it starts managed role
|
|
// sessions (packages/seat/src/session.mjs) and serves the launch socket the
|
|
// PM's `launch` tool calls. One launch at a time.
|
|
//
|
|
// A session asks over `<dataRoot>/bus-host/launch.sock` (0600) with one line,
|
|
// `{"cap": "<its capability>", "instance": "<role instance>"}`, and gets one
|
|
// line back: `{"ok": true, "result": {run, instance, harness, model}}` or
|
|
// `{"ok": false, "error": "<code>"}`. The host checks, in order:
|
|
//
|
|
// 1. identity: the capability names a role of this business
|
|
// 2. the business file's launch.instances lists the instance
|
|
// (instance-not-listed)
|
|
// 3. the instance isn't already running (instance-running)
|
|
// 4. it resolves, has a role contract (no-role-contract), sets the
|
|
// harness, model and thinking vars (no-route names the missing one)
|
|
// and its model is a launch.models key (unsupported-harness)
|
|
// 5. the route's harness (row 65): plugins.harness switches it on, and a
|
|
// stated fallback's too (harness-off); the plug-in loader loads it
|
|
// (harness-refused names the module and the loader's reason); it takes
|
|
// the thinking level (thinking-unsupported); its preflight passes
|
|
// (the module's code, e.g. no-session-credential or
|
|
// harness-binary-missing). A failed primary preflight takes the
|
|
// stated fallback through the same checks; there is no other retry.
|
|
// A harness whose launches count against launch.account gets the
|
|
// row 126 account checks.
|
|
// 6. bwrap runs here (sandbox-unavailable)
|
|
// 7. its model names an exact provider/model table entry
|
|
// 8. total, provider and provider/model have room, including the launcher
|
|
// (capacity-full)
|
|
// 9. every switched-on limit check allows it (limit-reached,
|
|
// limit-check-failed), asked through the bus's plug-in loader only
|
|
// 10. the broker authorizes role.launch for the caller (its own codes:
|
|
// launch-revoked, gated, ...)
|
|
//
|
|
// Steps 2 to 9 refuse through the broker's refuse(), so every refusal is an
|
|
// action.refused event; step 10's refusals the broker records itself. Then
|
|
// the host makes sure the workspace has a clone of each project
|
|
// (packages/seat/src/checkout.mjs): one it can't make refuses
|
|
// checkout-failed, one it fetches but doesn't fast-forward is a
|
|
// checkout-fetched record in the launch log, and one it leaves alone is a
|
|
// checkout-kept record. Then it builds the bundle, spawns the runner in its
|
|
// sandbox (packages/seat/src/sandbox.mjs) with the paths it may see, binds
|
|
// the run's process identity (the outer bwrap pid) and writes the new
|
|
// capability on the runner's stdin. When the runner exits, the host ends the launch.
|
|
//
|
|
// `bus start <business> --pm` launches launch.by the same way with
|
|
// launchedBy = the business's human and no caller capability.
|
|
//
|
|
// listen() also serves the git socket (src/git-socket.mjs, row 93), which
|
|
// the sandbox binds for a role that holds git.push.working,
|
|
// review.verdict or git.push.remote (row 199), or a merge decision to land
|
|
// (landHeld, row 137);
|
|
// session.json names it as gitSocket.
|
|
//
|
|
// Thread sessions (row 121, #1615). A thread instance (an instance whose
|
|
// role definition is in THREAD_DEFINITIONS) runs once per thread, never on its own:
|
|
// a launch from a session or --pm refuses thread-required. Every tick
|
|
// (threadTickMs) the host asks the broker for the business's threads
|
|
// (launch op threadsState) and:
|
|
//
|
|
// - starts a session for a thread with no running session that has an
|
|
// undelivered message to its instance, or an open newer than its last
|
|
// claim (REQ-LAUNCH-1). The start goes through check(), so it refuses
|
|
// as any launch does, and the next tick tries again;
|
|
// - stops a thread session once the thread has had no message or open
|
|
// for launch.threadIdleSeconds, the way close() stops a run. The broker
|
|
// decides (launch op threadIdle): it releases the thread only when the
|
|
// run is between turns, with nothing delivered and unread and nothing
|
|
// waiting, and the SIGTERM goes only after it releases. A message sent
|
|
// after the snapshot gets a turn from the same run or waits for the
|
|
// next one, instead of going to the stopping run. An idle thread holds
|
|
// no session and counts against no limit.
|
|
// - FED R4 (#1801): never starts a thread the broker marks revoked, and
|
|
// SIGTERMs its running session at once (the broker already ended the
|
|
// claim). bindLaunch refuses a new run of one with thread-revoked.
|
|
//
|
|
// A thread run's sessionDir is the thread's store (session.mjs
|
|
// threadStore()), so the next run for the thread resumes the harness
|
|
// session (REQ-THREAD-2).
|
|
//
|
|
// Row 269 (264b, #1774): threadsState has one row per session of each
|
|
// thread, and everything above is per session. Each session has its own
|
|
// store (session.mjs sessionStore()) and its own workspace under
|
|
// session-workspaces/ (sessionWorkspace()), which a lock beside it keeps to
|
|
// one writer from before the clones until the run's proven exit. A legacy
|
|
// session (op legacy-3l) keeps the thread's store, and its new workspace
|
|
// shows inside the sandbox at the old instance path, the cwd its history
|
|
// recorded. The broker's half of the fence is its per-session claim.
|
|
//
|
|
// Landing runs (row 156, #1651). Every launch of a non-thread instance
|
|
// reads the broker's landHolders (launch op): the approved, unspent merge
|
|
// decisions each non-thread instance raised. The instance's entries go into
|
|
// session.json as `landing: [{decision, task_ref}]`, and the same read
|
|
// decides land_work, so a landing turn always comes with the tool. The
|
|
// runner takes one turn per entry before its receive loop. After the
|
|
// thread pass, every tick also:
|
|
//
|
|
// - starts a holder that launch.instances lists and that isn't running,
|
|
// through check() as any launch, once per decision. A run that ended
|
|
// without landing isn't started again, and a refusal is logged once per
|
|
// code and isn't a start;
|
|
// - stops a running holder whose landing doesn't name the decision (it
|
|
// started before the resolution, so it has no land_work for it). The
|
|
// broker decides (launch op landRelease): it releases the role only
|
|
// between turns, with nothing delivered and unread, and leaves waiting
|
|
// messages undelivered for the next run. The SIGTERM goes only after it
|
|
// releases, and the next tick starts the instance.
|
|
//
|
|
// Nothing here is persisted. After a restart recover() ends the earlier
|
|
// host's runs, so the first tick starts each holder.
|
|
//
|
|
// Row 194 (#1692, D1.1): the lifecycle pass, on the same tick, carries out the
|
|
// human's and the tester's requests the broker recorded (launch, end,
|
|
// regenerate, delete). The broker never starts a process or touches a
|
|
// workspace. Under serialize, each open request is taken (host op
|
|
// lifecycleTake, which re-checks it and closes it with a code when it may not
|
|
// run), then acted on, then closed with lifecycleDone. A taken request never
|
|
// runs again, whatever happens after the take. A done that throws leaves it
|
|
// taken with no done, which the CLI reports as outcome-unknown.
|
|
// - launch: check() and start() with `human`, so an unlisted instance runs.
|
|
// - end: SIGTERM to each named run's runner after the start-time check, and
|
|
// to any run of the instance in the registry that spawned by the verb's
|
|
// time (N6, DESIGN.md:118-120). A run spawned after it is the next end's.
|
|
// The exit handler records the end. No SIGKILL for a terminate.
|
|
// - a fire's end (kill: true, S6): the same runs, each through row 191's
|
|
// stopRun with reason fire, which escalates to SIGKILL after the grace.
|
|
// A run it killed or left alive goes back in lifecycleDone's stops.
|
|
// - regenerate: SIGTERM to every run of the instance, then a new run once
|
|
// the old one has left the registry. Not gone by the request's 60 s mark
|
|
// closes it regenerate-timeout and starts nothing.
|
|
// While row 191's switch is on the tick runs only this pass (D6): a
|
|
// terminate still ends runs, and the take closes a launch or regenerate
|
|
// processing-halted.
|
|
// - delete: instance-running against the registry, then lifecycleDelete
|
|
// (the writer's commit), then the rename of workspaces/<b>/I/ to
|
|
// workspaces/<b>/.retired/I-<transaction>/, and of
|
|
// session-workspaces/<b>/I/ to session-workspaces/<b>/.retired/I-<transaction>/
|
|
// (row 269). A failed rename closes it workspace-rename-failed with the
|
|
// transaction; a later delete retries it.
|
|
|
|
import { createServer } from "node:net";
|
|
import { createHash } from 'node:crypto';
|
|
import { watchModelOutput } from './model-stall.mjs';
|
|
import { chmodSync, existsSync, lstatSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
import { dirname, join, relative } from "node:path";
|
|
import { configDir, loadBusiness, resolveInstance, systemVars } from "../../business/src/index.mjs";
|
|
import { organizationWriter } from "../../business/src/organization.mjs";
|
|
import { businessFilePath, validateBusinessDocument } from "../../business/src/business.mjs";
|
|
import { selectModelProvider } from "../../plugins/src/model-provider-selection.mjs";
|
|
import { THREAD_DEFINITIONS } from "../../business/src/vocabulary.mjs";
|
|
import { limitHit } from "../../business/src/launch.mjs";
|
|
import { BundleError, buildBundle, hasFallback, readSkills, sessionModel } from "../../harness/src/bundle.mjs";
|
|
import { createLoader } from "../../plugins/src/loader.mjs";
|
|
import {
|
|
appendLaunchLog,
|
|
endReason,
|
|
gitSocketPath,
|
|
launchSocketPath,
|
|
newRun,
|
|
readSessions,
|
|
Registry,
|
|
runnerOf,
|
|
SESSION_CREDENTIALS,
|
|
sessionEnv,
|
|
isProviderVariable,
|
|
sessionExit,
|
|
sessionStore,
|
|
sessionWorkspace,
|
|
lockSessionWorkspace,
|
|
spawnSession,
|
|
workspaceDir,
|
|
writeSessionFile,
|
|
} from "../../seat/src/session.mjs";
|
|
import { BWRAP, sandboxProblem } from "../../seat/src/sandbox.mjs";
|
|
import { collectCompactions, READ_BYTES, readBudget, readStreams, STREAMS } from "../../seat/src/compactions.mjs";
|
|
import { CheckoutError, ensureCheckouts } from "../../seat/src/checkout.mjs";
|
|
import { createGitSocket } from "./git-socket.mjs";
|
|
import { cmdlineOf, startTimeOf } from "../../seat/src/proc.mjs";
|
|
import { rolesDir } from "./config.mjs";
|
|
import { CliError } from "./errors.mjs";
|
|
import { assertProcessing, processingStatus, ProcessingError } from "../../processing-control/src/state.mjs";
|
|
import { registerRuntime, updateRuntime, runtime, unregisterRuntime, processIdentity, stopRun } from "../../processing-control/src/runtimes.mjs";
|
|
import { withProcessing } from '../../processing-control/src/effects.mjs';
|
|
|
|
const STOP_TIMEOUT_MS = 30000;
|
|
const LINE_MAX = 4096;
|
|
// How often the host asks for the threads' state; a poll period, not a limit.
|
|
const THREAD_TICK_MS = 2000;
|
|
// Row 194: the broker's LIFECYCLE_REQUEST_MS. A regenerate's old run must be gone by then.
|
|
const LIFECYCLE_REQUEST_MS = 60000;
|
|
// #774 R2: the session proxy key the first check of a proxy launch gives the harness preflights in
|
|
// place of the key P2 hasn't admitted yet. A fixed marker, never a credential; it never leaves check().
|
|
export const PENDING_ADMISSION_KEY = "pending-admission";
|
|
|
|
// A refusal the host decided; `code` goes back to the caller and to the broker.
|
|
class Refusal extends Error {
|
|
constructor(code, message = code) {
|
|
super(message);
|
|
this.code = code;
|
|
}
|
|
}
|
|
|
|
// A harness module runs with the host's authority, and anything it throws refuses the launch
|
|
// and names the module rather than escaping the host.
|
|
function call(name, what, fn) {
|
|
try {
|
|
return fn();
|
|
} catch (e) {
|
|
throw new Refusal("harness-failed", `harness ${name} ${what}: ${e.message}`);
|
|
}
|
|
}
|
|
|
|
// host: startHost()'s handle. tracker: true when the broker has task verbs for
|
|
// the business. launches: false for `bus start --no-launch` (row 163, the D2
|
|
// rehearsal): start() refuses launches-disabled, the ticker runs only the
|
|
// lifecycle pass (row 194), which closes each request launches-disabled, a
|
|
// session's launch request is refused, recover() kills and ends nothing, and
|
|
// sessions.json stays as it was.
|
|
// makeLoader (the plug-in loader's factory, which tests point at a
|
|
// scratch tree), bwrap, readOnly, sessionDefaults, harnessVersion,
|
|
// stopTimeoutMs, threadTickMs, lifecycleRequestMs, git (createGitSocket
|
|
// options) and stopOptions (row 191's stopRun options, for a fire's stop)
|
|
// exist for the tests; the command line never sets them.
|
|
export function createLauncher({
|
|
host,
|
|
system,
|
|
env = process.env,
|
|
// Tests only (#774 F4): a static session proxy context {baseUrl,key}, or a
|
|
// factory giving one per check, for a provider "proxy" route while
|
|
// system.proxy is unset. It skips P2 admission, so createLauncher refuses it
|
|
// together with system.proxy. P3 never reads a key store or calls the proxy.
|
|
testProxy = null,
|
|
tracker = false,
|
|
launches = true,
|
|
makeLoader = createLoader,
|
|
bwrap = BWRAP,
|
|
readOnly = [],
|
|
sessionDefaults = {},
|
|
harnessVersion = undefined,
|
|
stopTimeoutMs = STOP_TIMEOUT_MS,
|
|
threadTickMs = THREAD_TICK_MS,
|
|
lifecycleRequestMs = LIFECYCLE_REQUEST_MS,
|
|
git = {},
|
|
stopOptions = {},
|
|
log = (l) => process.stderr.write(`mosaic-bus: ${l}\n`),
|
|
}) {
|
|
if (testProxy !== null && system.proxy)
|
|
throw Object.assign(new Error("testProxy skips P2 admission, so it is refused with system.proxy set"), { code: "test-proxy-refused" });
|
|
const dataRoot = system.dataRoot;
|
|
function admission(generation = null) {
|
|
try { return assertProcessing(dataRoot, generation); }
|
|
catch (e) {
|
|
if (e instanceof ProcessingError) throw new Refusal(e.code);
|
|
throw e;
|
|
}
|
|
}
|
|
// Row 64 (#1557): where a role's skills are read from; scripts/skill.sh activate fills it.
|
|
const skillsDir = join(system.instanceRoot ?? dataRoot, "skills-enabled");
|
|
let b = loadBusiness(host.business, { dir: configDir(env), rolesDir: rolesDir(env) });
|
|
const vars = systemVars(system);
|
|
|
|
// Sessions an earlier host left in sessions.json: it died without ending
|
|
// them, so their runs still hold their roles. listen() ends them. A file
|
|
// that can't be read refuses the host rather than drop them unseen.
|
|
let leftover;
|
|
try {
|
|
leftover = readSessions(dataRoot).filter((s) => s.business === b.id);
|
|
} catch (e) {
|
|
throw new CliError(`${e.message}; check which sessions still run, then move the file aside`, 3);
|
|
}
|
|
const registry = new Registry(dataRoot, { readOnly: !launches });
|
|
// Row 269: earlier-host runs still live after recover(). check() sees each while it lives.
|
|
const survivors = [];
|
|
const liveRun = (s) => startTimeOf(s.pid) === s.startTime || startTimeOf(s.runnerPid) === s.runnerStartTime;
|
|
const children = new Map();
|
|
let chain = Promise.resolve();
|
|
let closed = false;
|
|
let server = null;
|
|
let ticker = null;
|
|
let ticking = false;
|
|
const stopping = new Set();
|
|
const refusedThreads = new Map();
|
|
const startedFor = new Map();
|
|
// Row 271 (264d): each clear's stop time (by op), and each clear refusal logged once (by op and code).
|
|
const clearStops = new Map();
|
|
const clearLogged = new Map();
|
|
// Row 194 N6: run -> when its process was spawned, so an end signals only the runs spawned by its verb.
|
|
const spawnedAt = new Map();
|
|
// Row 156: run -> the decisions its landing names; decision -> started; decision -> last
|
|
// refusal code; run -> last landRelease refusal code; and landHolders' last refusal.
|
|
const landingOf = new Map();
|
|
const landStarted = new Set();
|
|
const refusedLand = new Map();
|
|
const refusedRelease = new Map();
|
|
let refusedHolders = null;
|
|
let refusedRequests = null;
|
|
const requestDrains = new Map();
|
|
const sockets = new Set();
|
|
const gitSocket = createGitSocket({ host, business: b, dataRoot, env, log, ...git });
|
|
|
|
const record = (entry) => {
|
|
try {
|
|
appendLaunchLog(dataRoot, b.id, entry);
|
|
} catch (e) {
|
|
log(`launch log: ${e.message}`);
|
|
}
|
|
};
|
|
|
|
// Row 270 (264c, D6): count a thread session's completed compactions into its
|
|
// store's ledger, at run start (what earlier runs and a resume left) and at
|
|
// run end. The caller holds the session's workspace lock. A failure is
|
|
// logged and recorded, and never stops or fails the run: the count then
|
|
// reads unknown with its reason. Row 270 r2 (B1, mos-1352): each collection
|
|
// reads at most the system config's compactionReadBytes, a human-only
|
|
// setting; a missing or bad value takes the default.
|
|
const budget = readBudget(system?.compactionReadBytes);
|
|
if (system?.compactionReadBytes !== undefined && budget !== system.compactionReadBytes) {
|
|
log(`system config: compactionReadBytes ${JSON.stringify(system.compactionReadBytes)} isn't a whole number of bytes from 1048576 to 1073741824; compaction collections read up to the default ${READ_BYTES}`);
|
|
}
|
|
// Row 270 r3 (B4, sage-3038): and opens at most compactionStreams stream
|
|
// files, the same kind of setting.
|
|
const streams = readStreams(system?.compactionStreams);
|
|
if (system?.compactionStreams !== undefined && streams !== system.compactionStreams) {
|
|
log(`system config: compactionStreams ${JSON.stringify(system.compactionStreams)} isn't a whole number from 32 to 65536; compaction collections open up to the default ${STREAMS} streams`);
|
|
}
|
|
// Row 307 (#1814): the route's harness module reads its own store through the kind's
|
|
// `compactions`. The call skips call(): call() turns a throw into a launch refusal, and a
|
|
// collection that fails must never stop or fail the run, so this catch records it instead.
|
|
const compactions = (store, name, harness, { instance, thread, session, run }) => {
|
|
let c;
|
|
try {
|
|
c = collectCompactions(store, { name, compactions: harness.compactions }, { budget, streams });
|
|
} catch (e) {
|
|
log(`run ${run}: compactions: ${e.message}`);
|
|
c = { count: null, known: false, reason: "store-unreadable" };
|
|
}
|
|
record({ event: "compactions", instance, thread, session, run, count: c.count, known: c.known, reason: c.reason });
|
|
};
|
|
|
|
function serialize(fn) {
|
|
const run = chain.then(fn);
|
|
chain = run.catch(() => {});
|
|
return run;
|
|
}
|
|
|
|
// Steps 2 to 9. Returns what the launch needs. Only `--pm` may name
|
|
// launch.by itself. A thread instance launches only for a thread, and
|
|
// nothing else launches for one. Row 194 (#1692, V2): `human` is the
|
|
// human's launch request, which may name any declared instance. The listing
|
|
// limits the chief, not the human. launch.by still takes the `pm` branch.
|
|
async function check(instance, { pm = false, thread = null, session = null, human = false, proxyContext = null, proxyReady = false } = {}) {
|
|
const admissionGeneration = admission().generation;
|
|
let businessRevision;
|
|
try {
|
|
const snapshot = organizationWriter({dir:configDir(env),rolesDir:rolesDir(env)}).read(host.business);
|
|
businessRevision = snapshot.revision;
|
|
b = validateBusinessDocument(snapshot.document, businessFilePath(host.business, configDir(env)), {rolesDir:rolesDir(env)});
|
|
} catch(e){throw new Refusal("organization-refused",e.message);}
|
|
if (!b.launch) throw new Refusal("launch-not-configured", `business ${b.id} has no launch block`);
|
|
if (!(pm ? instance === b.launch.by : b.launch.instances.includes(instance) || (human && Object.hasOwn(b.roles, instance)))) throw new Refusal("instance-not-listed");
|
|
const threaded = THREAD_DEFINITIONS.includes(b.roles[instance]?.definition);
|
|
if (threaded && !thread) throw new Refusal("thread-required", `${instance} runs only for a thread`);
|
|
if (!threaded && thread) throw new Refusal("thread-not-allowed", `${instance} isn't a thread instance`);
|
|
if (threaded && !session) throw new Refusal("session-required", `${instance} runs for one session of a thread`);
|
|
if (registry.running(b.id, instance, thread, session)) throw new Refusal("instance-running");
|
|
// Row 269 (Q2): a run an earlier host started with no session works in the instance's old
|
|
// workspace and the thread's store, so no session of the instance starts while one lives.
|
|
if (threaded && [...registry.sessions, ...leftover, ...survivors.filter(liveRun)].some((s) => s.business === b.id && s.instance === instance && !s.session))
|
|
throw new Refusal("legacy-run-live", `a run of ${instance} with no session is still live`);
|
|
let resolved;
|
|
try {
|
|
resolved = resolveInstance({ system: vars, business: b, instance });
|
|
} catch (e) {
|
|
throw new Refusal("unsupported-harness", e.message);
|
|
}
|
|
// Resolve the business's current provider modules with a fresh loader; a refused
|
|
// load never inherits a previous business's authority.
|
|
const providerNames = b.plugins?.['model-provider'] ?? [];
|
|
const providerModules = new Map();
|
|
if (providerNames.length) {
|
|
const loader = makeLoader({ log: entry => record({ ...entry, instance }) });
|
|
const loaded = await loader.load({ 'model-provider': providerNames });
|
|
if (loaded.refused.length) throw new Refusal('model-provider-unavailable', 'model provider module refused');
|
|
for (const { name, module } of loader.snapshot('model-provider')) providerModules.set(name, module);
|
|
}
|
|
const resolver = providerModules.size ? ({ selection, harness }) => {
|
|
const selected = selectModelProvider([...providerModules].map(([name,module]) => ({name,module})), selection);
|
|
if (!selected) return { code: 'model-provider-unavailable' };
|
|
return selected.module.route({ selection, harness });
|
|
} : undefined;
|
|
// Row 65 (#1558): the route's harness, model and thinking come from the instance's vars only.
|
|
const routeOf = (which) => {
|
|
try {
|
|
return sessionModel(resolved, b.launch.models, which, { proxy: Boolean(system.proxy || proxyContext), resolver });
|
|
} catch (e) {
|
|
const code = e instanceof BundleError && (e.code === "no-route" || e.code.startsWith("proxy-") || e.code.startsWith("model-")) ? e.code : "unsupported-harness";
|
|
throw new Refusal(code, e.message);
|
|
}
|
|
};
|
|
const primary = routeOf("primary");
|
|
const fallback = hasFallback(resolved) ? routeOf("fallback") : null;
|
|
if (!resolved.contract) throw new Refusal("no-role-contract", `role ${resolved.definition} has no contract file`);
|
|
// Row 64 (#1557): every skill on the role's list is enabled and readable before a run exists.
|
|
try {
|
|
readSkills(resolved.skills, skillsDir);
|
|
} catch (e) {
|
|
if (e instanceof BundleError) throw new Refusal(e.code === "skill-not-enabled" ? "skill-not-enabled" : "skill-refused", e.message);
|
|
throw e;
|
|
}
|
|
// Both of a launch's harnesses must be switched on, whichever it takes.
|
|
const on = Array.isArray(b.plugins?.harness) ? b.plugins.harness : [];
|
|
for (const m of [primary, fallback].filter(Boolean)) {
|
|
if (!on.includes(m.harness)) throw new Refusal("harness-off", `harness ${m.harness} isn't switched on in plugins.harness`);
|
|
}
|
|
// One harness through a fresh loader, so a refusal never leaves an earlier load's module in use.
|
|
const harnessOf = async (m) => {
|
|
const loader = makeLoader({ log: (entry) => record({ ...entry, instance }) });
|
|
const { refused } = await loader.load({ harness: [m.harness] });
|
|
if (refused.length) throw new Refusal("harness-refused", `harness ${m.harness}: ${refused.map((r) => `${r.code}: ${r.message}`).join("; ")}`);
|
|
const module = loader.modules("harness")[0];
|
|
const levels = call(m.harness, "levels", () => module.levels());
|
|
if (!levels.includes(m.thinking)) throw new Refusal("thinking-unsupported", `harness ${m.harness} takes thinking ${levels.join(", ")}, not ${m.thinking}`);
|
|
return module;
|
|
};
|
|
let model = primary;
|
|
let harness = await harnessOf(primary);
|
|
let route = "primary";
|
|
let routeReason = null;
|
|
// A launch is checked twice. Without system.proxy both are full checks: the first before the
|
|
// authorization, workspace and checkout work, the final one in startLocked before newRun. With
|
|
// system.proxy the first check comes before P2 admits a key in startLocked, so it predicts the
|
|
// route: it runs the harness preflights on a stand-in key and picks the route as the final check
|
|
// does (#774 R2). The prediction holds because a harness preflight tests only that a proxy key
|
|
// is present, never its value. On the predicted route it runs the account declaration
|
|
// (credential "proxy" only), the sandbox, capacity, the limit checks and admission, so a launch
|
|
// that can never start costs no authorization record, checkout or key (#774 F1). It skips only
|
|
// the match of the session proxy key against launch.account.name. The final check is always
|
|
// full, on the admitted key, and it picks the route again: if a harness binary appears or goes
|
|
// between the two checks, its pick decides. Never reject proxy-only login using upstream host env.
|
|
const early = Boolean(system.proxy) && !proxyReady;
|
|
// The checks that follow the route's preflight, on the route the caller picked. In the first
|
|
// check of a proxy launch sessionProxy is null, and the account must then name credential "proxy".
|
|
const gate = async (model, harness, sessionProxy) => {
|
|
const proxied = early || Boolean(sessionProxy);
|
|
// Row 126 (#1620): only a launch on a harness whose module counts the account spends the named
|
|
// account. The bus environment must hold the named set and no other credential variable, so the
|
|
// session can't log in some other way. Messages name variables, never values.
|
|
const account = call(model.harness, "countsAccount", () => harness.countsAccount()) === true ? b.launch.account?.name ?? null : null;
|
|
if (account !== null) {
|
|
// The usage records keep this name (bus schema 3i), so it must never be a credential's value.
|
|
const holds = [...new Set(SESSION_CREDENTIALS.flat())].filter((k) => env[k] === account);
|
|
if (sessionProxy?.key === account) holds.push("session proxy key");
|
|
if (holds.length) throw new Refusal("account-is-credential", `launch.account.name is the value of ${holds.join(", ")}; it names the account, never its credential`);
|
|
if (proxied || b.launch.account.credential === "proxy") {
|
|
if (!proxied || b.launch.account.credential !== "proxy")
|
|
throw new Refusal("account-credential-missing", `account ${account} needs the declared proxy credential`);
|
|
} else {
|
|
const named = b.launch.account.credential.split("+");
|
|
if (!SESSION_CREDENTIALS.some((set) => set.join("+") === named.join("+")) || !named.every((k) => Boolean(env[k])))
|
|
throw new Refusal("account-credential-missing", `account ${account} names ${named.join(" with ")}, which the bus host doesn't hold as a session login`);
|
|
const other = [...new Set(SESSION_CREDENTIALS.flat())].filter((k) => !named.includes(k) && Boolean(env[k]));
|
|
if (other.length) throw new Refusal("account-credential-ambiguous", `account ${account} names ${named.join(" with ")}, but the bus host also sets ${other.join(", ")}`);
|
|
}
|
|
}
|
|
const unavailable = sandboxProblem({ bwrap });
|
|
if (unavailable) throw new Refusal("sandbox-unavailable", unavailable);
|
|
const hit=limitHit(registry.sessions,b.id,b.launch.limits,model);
|
|
if(hit){const error=new Refusal("capacity-full",JSON.stringify(hit));error.limit=hit;throw error;}
|
|
// Row 125 (#1619, REQ-LIMIT-4): the limit checks. An answer that isn't an allow refuses,
|
|
// and so does a bus that can't answer (fail closed, lead decision 150).
|
|
let may;
|
|
try {
|
|
may = await host.op({ op: "mayLaunch", business: b.id, instance, provider: model.provider, model: model.model, account });
|
|
} catch (e) {
|
|
throw new Refusal("limit-check-failed", `bus: ${e.code ?? "launch-op-refused"}`);
|
|
}
|
|
// Row 333: under a daily key quota the bus refuses key-quota-no-model-key for an agent P2
|
|
// hasn't issued a key yet. Only the first check of a proxy launch lets that one answer pass:
|
|
// startLocked's modelPreflight issues the key, and the final check asks again with it.
|
|
const deferredKey = early && may?.allow === false && may.check === null && may.code === "key-quota-no-model-key";
|
|
if (may?.allow !== true && !deferredKey) {
|
|
if (may?.hit) { const error = new Refusal("limit-reached", JSON.stringify(may.hit)); error.limit = { check: may.check, ...may.hit }; throw error; }
|
|
throw new Refusal("limit-check-failed", `${may?.check ?? "bus"}: ${may?.code ?? "malformed-answer"}`);
|
|
}
|
|
const selected = b.launch.models[model.modelKey];
|
|
if (selected?.module || (proxied && providerModules.size)) {
|
|
let checked;
|
|
try { checked = await host.op({ op: 'modelAdmission', business: b.id, instance, modelKey: model.modelKey, harness: model.harness,
|
|
revision: businessRevision, entry: selected, provider: model.provider, model: model.model, account, route, transport: model.proxyTransport }); }
|
|
catch { throw new Refusal('model-catalogue-unavailable', 'model catalogue admission unavailable'); }
|
|
if (checked?.allow !== true) {
|
|
const code = ['model-provider-unavailable', 'model-catalogue-unavailable', 'model-harness-incompatible'].includes(checked?.code) ? checked.code : 'model-provider-unavailable';
|
|
throw new Refusal(code, `model ${selected?.module ?? 'legacy'}/${selected?.upstream ?? selected?.provider}/${selected?.model} unavailable (${code})`);
|
|
}
|
|
if (!proxied) throw new Refusal('model-provider-unavailable', 'module-backed model requires a proxy');
|
|
}
|
|
return account;
|
|
};
|
|
const proxyFor = async (selected) => {
|
|
if (!system.proxy && selected.provider !== "proxy") return null;
|
|
const context = system.proxy ? proxyContext : typeof testProxy === "function" ? await testProxy({ business: b.id, instance }) : testProxy;
|
|
if (!context || typeof context.key !== "string" || !context.key
|
|
|| typeof context.baseUrl !== "string" || !/^http:\/\/127\.0\.0\.1:[0-9]+$/.test(context.baseUrl)) {
|
|
throw new Refusal("no-session-credential", "proxy model needs its own session proxy context");
|
|
}
|
|
// Copy only the agreed seam; never keep a store or management handle.
|
|
return { baseUrl: context.baseUrl, key: context.key };
|
|
};
|
|
// The route both checks pick (#774 R2): the primary when its preflight passes, else the fallback
|
|
// through its own preflight. proxyOf gives the session proxy for a route; the preflights see it,
|
|
// and the helper returns the last one they saw.
|
|
const pickRoute = async (proxyOf) => {
|
|
let sessionProxy = await proxyOf(primary);
|
|
const failed = call(primary.harness, "preflight", () => harness.preflight({ env: sessionProxy ? sessionEnv(env, "preflight", sessionProxy) : env, proxy: sessionProxy }));
|
|
if (failed) {
|
|
if (!fallback) throw new Refusal(failed.code, failed.message);
|
|
// The fallback only on a failed primary preflight, and only at launch (row 65). Any refusal of
|
|
// the fallback names the primary's failure too, so the reason it was tried stays visible.
|
|
const second = await harnessOf(fallback).catch((e) => {
|
|
if (e instanceof Refusal) e.message += ` (primary ${primary.harness}: ${failed.message})`;
|
|
throw e;
|
|
});
|
|
sessionProxy = await proxyOf(fallback);
|
|
const also = call(fallback.harness, "preflight", () => second.preflight({ env: sessionProxy ? sessionEnv(env, "preflight", sessionProxy) : env, proxy: sessionProxy }));
|
|
if (also) throw new Refusal(also.code, `fallback ${fallback.harness}: ${also.message} (primary ${primary.harness}: ${failed.message})`);
|
|
[model, harness, route, routeReason] = [fallback, second, "fallback", failed.code];
|
|
}
|
|
return sessionProxy;
|
|
};
|
|
if (early) {
|
|
// The first check predicts the final check's route. A harness preflight tests only that a proxy
|
|
// key is present, so a stand-in key gets the answer the admitted key will get, and the route
|
|
// picked here is the one the final check picks unless a harness binary appears or goes in
|
|
// between; then the final check's own pick decides. The stand-in reaches only the preflights:
|
|
// pickRoute's answer is dropped, gate() gets null, and nothing records or returns it. The gate
|
|
// then runs on the predicted route only, with the codes, messages and limits the final check uses.
|
|
const standIn = { baseUrl: Number.isInteger(system.proxy.port) ? `http://127.0.0.1:${system.proxy.port}` : "http://127.0.0.1:0", key: PENDING_ADMISSION_KEY };
|
|
await pickRoute(() => standIn);
|
|
await gate(model, harness, null);
|
|
admission(admissionGeneration);
|
|
return { resolved, model, account: null, harness, route, routeReason, admissionGeneration, resolver };
|
|
}
|
|
const sessionProxy = await pickRoute(proxyFor);
|
|
const account = await gate(model, harness, sessionProxy);
|
|
admission(admissionGeneration);
|
|
return { resolved, model, account, harness, route, routeReason, admissionGeneration, sessionProxy, resolver };
|
|
}
|
|
|
|
// Reread host state on every new-run path, including opens/clears queued before archive.
|
|
async function threadAdmission(thread, session) {
|
|
if (!thread) return;
|
|
const rows = await host.op({ op: "threadsState", business: b.id });
|
|
const row = rows.find(r => r.thread === thread && r.session === session);
|
|
if (!row) throw new Refusal("session-not-found");
|
|
if (row.archived) throw new Refusal("project-archived");
|
|
}
|
|
|
|
// Bundle, spawn, bind, register, hand over the capability.
|
|
async function start({ instance, launchedBy, resolved, model, account = null, thread = null, session = null, legacy = false, legacyThread = legacy, human = false, admissionGeneration }) {
|
|
admission(admissionGeneration);
|
|
// Every launch path ends here, so a --no-launch host spawns nothing whichever path asked.
|
|
if (!launches) throw new Refusal("launches-disabled", "this host was started with --no-launch");
|
|
await threadAdmission(thread, session);
|
|
// A thread session's store is checked first, so a bad id or an unsafe
|
|
// directory launches nothing.
|
|
let store = null;
|
|
if (thread) {
|
|
try {
|
|
store = sessionStore(dataRoot, b.id, thread, session, { legacy, legacyThread });
|
|
} catch (e) {
|
|
throw new Refusal(e.code === "legacy-store-unmapped" ? e.code : "thread-store-refused", e.message);
|
|
}
|
|
}
|
|
const credentials = await host.op({ op: "credentialStatus", business: b.id, role: instance });
|
|
admission(admissionGeneration);
|
|
// The clones come before the run, so one that can't be made leaves no
|
|
// run behind. The host's git treats each clone as hostile, and its
|
|
// checks hold only while no session writes the workspace. A non-thread
|
|
// workspace is one instance's, check() refused instance-running, and
|
|
// launches are serialized. A thread session's workspace is its own (row
|
|
// 269), and its lock refuses a second writer, whichever host asks.
|
|
let workspace;
|
|
let workspaceDest = null;
|
|
let lock = null;
|
|
if (thread) {
|
|
try {
|
|
const live = registry.sessions.filter((s) => s.business === b.id && s.workspace).map((s) => s.workspace);
|
|
workspace = sessionWorkspace(dataRoot, b.id, instance, session, { live });
|
|
lock = lockSessionWorkspace(workspace, { hostPid: process.pid, hostStartTime: startTimeOf(process.pid), business: b.id, instance, thread, session });
|
|
} catch (e) {
|
|
if (["session-workspace-unsafe", "session-workspace-overlap", "session-workspace-busy"].includes(e.code)) throw new Refusal(e.code, e.message);
|
|
throw e;
|
|
}
|
|
if (legacy) workspaceDest = workspaceDir(dataRoot, b.id, instance);
|
|
} else {
|
|
workspace = workspaceDir(dataRoot, b.id, instance);
|
|
mkdirSync(workspace, { recursive: true, mode: 0o700 });
|
|
}
|
|
try {
|
|
return await startLocked({ instance, launchedBy, resolved, model, account, thread, session, human, admissionGeneration, store, credentials, workspace, workspaceDest, lock });
|
|
} catch (e) {
|
|
// Every failure path in startLocked has proven the run's exit, or never spawned one.
|
|
lock?.release();
|
|
throw e;
|
|
}
|
|
}
|
|
|
|
// The rest of start(): clones, bundle, spawn, bind, register. The lock, when there is one, is
|
|
// released at the run's close.
|
|
async function startLocked({ instance, launchedBy, resolved, model, account, thread, session, human, admissionGeneration, store, credentials, workspace, workspaceDest, lock }) {
|
|
// What the session sees: a legacy session's workspace at the old instance path.
|
|
const inSandbox = workspaceDest ?? workspace;
|
|
// Lifecycle Task 2 (#797): a product the request lifecycle built under <dataRoot>/products is cloned only
|
|
// once the bus has admitted it. The bus answers from its receipts, so a reserved or half-built product
|
|
// (a mapping published before the seed and records finished) is left out and the launch goes on without
|
|
// it. Ordinary mappings also recheck archive state through the same host admission.
|
|
const codeProjects = {};
|
|
for (const [project, entry] of Object.entries(b.projects)) {
|
|
if (entry.kind === 'library') continue;
|
|
let answer = null;
|
|
try { answer = await host.op({ op: 'request.admission', business: b.id, project }); } catch (e) { answer = { ready: false, code: e.code ?? 'admission-unavailable' }; }
|
|
if (answer?.ready !== true) { record({ event: "checkout-skipped", instance, project, reason: answer?.code ?? 'project-pending' }); continue; }
|
|
codeProjects[project] = entry;
|
|
}
|
|
await threadAdmission(thread, session);
|
|
let checkouts;
|
|
try {
|
|
checkouts = await withProcessing(dataRoot, () => {
|
|
admission(admissionGeneration);
|
|
return ensureCheckouts({ workspace, projects: codeProjects, env });
|
|
});
|
|
} catch (e) {
|
|
if (e instanceof CheckoutError) throw new Refusal("checkout-failed", `${e.project}: ${e.message}`);
|
|
throw e;
|
|
}
|
|
for (const c of checkouts) {
|
|
if (c.outcome === "kept") record({ event: "checkout-kept", instance, project: c.project, reason: c.reason });
|
|
else if (c.outcome === "fetched") record({ event: "checkout-fetched", instance, project: c.project, reason: c.reason });
|
|
}
|
|
// Retain P2's real admission/ownership operation; do not substitute the
|
|
// old stand-in option or acquire again in each metadata/preflight check.
|
|
let admittedProxy = null;
|
|
if (system.proxy) {
|
|
try { admittedProxy = await host.op({ op: 'modelPreflight', business: b.id, role: instance }); }
|
|
catch (error) { throw new Refusal(error.code ?? 'proxy-down', error.code ?? 'proxy-down'); }
|
|
if (!admittedProxy || typeof admittedProxy.key !== 'string' || !admittedProxy.key || admittedProxy.baseUrl !== `http://127.0.0.1:${system.proxy.port}`) throw new Refusal('proxy-down', 'proxy-down');
|
|
}
|
|
let harness, route, routeReason, sessionProxy, resolver;
|
|
({resolved,model,account,harness,route,routeReason,sessionProxy,resolver}=await check(instance,{pm:instance===b.launch.by,thread,session,human,proxyContext:admittedProxy,proxyReady:true}));
|
|
admission(admissionGeneration);
|
|
const { run, runDir } = newRun(dataRoot, b.id);
|
|
let modelKeyId;
|
|
if (admittedProxy) {
|
|
modelKeyId = createHash('sha256').update(sessionProxy.key).digest('hex').slice(0, 12);
|
|
// Internal opt-in only: no session.json, user environment or config field.
|
|
try { writeFileSync(join(runDir, 'proxy-first-turn-enabled'), 'enabled\n', { flag: 'wx', mode: 0o600 }); }
|
|
catch { throw new Refusal('launch-failed', 'first-turn protocol setup failed'); }
|
|
}
|
|
const sessionDir = store ? store.session : join(runDir, "session");
|
|
if (!store) mkdirSync(sessionDir, { mode: 0o700 });
|
|
if (store) compactions(store, model.harness, harness, { instance, thread, session, run });
|
|
const toolSocket = join(runDir, "tools.sock");
|
|
const turnMarker = join(runDir, "turn.done");
|
|
const version = harnessVersion !== undefined ? harnessVersion : call(model.harness, "version", () => harness.version({ env }));
|
|
// Row 137 (#1631): a role and thread holding an approved, unspent merge decision get land_work.
|
|
// The broker checks the decision again on every land; this only decides what the bundle shows.
|
|
// Row 156 (#1651): a non-thread instance's land and landing come from one landHolders read, so a
|
|
// decision resolved between two reads can't give a landing turn without land_work.
|
|
const holders = thread ? null : await host.op({ op: "landHolders", business: b.id });
|
|
if (holders !== null && !Array.isArray(holders)) throw new Refusal("launch-failed", "landHolders: malformed answer");
|
|
const holding = holders?.filter((h) => h.instance === instance) ?? null;
|
|
const land = thread ? await host.op({ op: "landHeld", record: { business: b.id, role: instance, thread } }) : holding.length > 0;
|
|
const bundle = buildBundle({
|
|
dir: join(runDir, "bundle"),
|
|
resolved,
|
|
models: b.launch.models,
|
|
proxy: sessionProxy,
|
|
resolver,
|
|
contract: readFileSync(resolved.contract, "utf8"),
|
|
arbiter: Object.values(b.arbiters).includes(instance),
|
|
tracker,
|
|
library: Object.values(b.projects ?? {}).some(project => project.kind === "library"),
|
|
credentials,
|
|
session: { run, launchedBy, workspace: inSandbox, toolSocket, turnMarker, checkouts: inSandbox === workspace ? checkouts : checkouts.map((c) => ({ ...c, path: join(inSandbox, relative(workspace, c.path)) })) },
|
|
harness,
|
|
route,
|
|
routeReason,
|
|
harnessVersion: version,
|
|
land,
|
|
skillsDir,
|
|
personaDir: join(configDir(env), "personas"),
|
|
});
|
|
const { manifest, ...files } = bundle.files;
|
|
const adapter = call(model.harness, "adapter", () => harness.adapter());
|
|
const turnEnd = call(model.harness, "turnEnd", () => harness.turnEnd());
|
|
const outputPolicy = call(model.harness, "outputPolicy", () => harness.outputPolicy({ proxy: Boolean(sessionProxy) }));
|
|
if (!["adapter", "non-newline-stdout", "stdout"].includes(outputPolicy)) throw new Refusal("harness-refused", "invalid harness output policy");
|
|
const transcript = call(model.harness, "transcript", () => harness.transcript({ runDir, store }));
|
|
const harnessMounts = call(model.harness, "mounts", () => harness.mounts({ env }));
|
|
// Row 93: the git socket, for a role that pushes work or reviews it. Row 137: or lands it.
|
|
// Row 199 (D11): or publishes it, once a reviewed commit routes git.push.remote within-role.
|
|
const held = [...resolved.limits.authority.withinRole, ...resolved.limits.authority.crossRole];
|
|
const git = land || ["git.push.working", "review.verdict", "git.push.remote"].some((a) => held.includes(a)) ? gitSocketPath(dataRoot) : null;
|
|
// Secrets go only to spawn, never session.json. Keep this split at the
|
|
// actual serialization boundary, even if a harness adds another key.
|
|
const harnessEnv = {}, transientEnv = {};
|
|
for (const [name, value] of Object.entries(bundle.env)) {
|
|
(sessionProxy && isProviderVariable(name) ? transientEnv : harnessEnv)[name] = value;
|
|
}
|
|
writeSessionFile(runDir, {
|
|
...sessionDefaults,
|
|
business: b.id,
|
|
instance,
|
|
...(thread ? { thread } : {}),
|
|
run,
|
|
launchedBy,
|
|
harness: model.harness,
|
|
provider: model.provider,
|
|
model: model.model,
|
|
brokerSocket: host.path,
|
|
launchSocket: launchSocketPath(dataRoot),
|
|
gitSocket: git,
|
|
workspace: inSandbox,
|
|
sessionDir,
|
|
adapter,
|
|
bundle: files,
|
|
harnessEnv,
|
|
turnEnd,
|
|
outputPolicy,
|
|
outputPolicyVersion: 1,
|
|
toolSocket,
|
|
turnMarker,
|
|
...(holding?.length ? { landing: holding.map(({ decision, task_ref }) => ({ decision, task_ref })) } : {}),
|
|
});
|
|
|
|
const paths = {
|
|
workspace,
|
|
workspaceDest,
|
|
adapter,
|
|
harnessMounts,
|
|
brokerSocket: host.path,
|
|
launchSocket: resolved.launch ? launchSocketPath(dataRoot) : null,
|
|
gitSocket: git,
|
|
threadStore: store,
|
|
readOnly,
|
|
};
|
|
admission(admissionGeneration);
|
|
await threadAdmission(thread, session);
|
|
// Archive can change while the checkout/bundle is prepared. Never spawn with a stale manifest.
|
|
for (const project of Object.keys(codeProjects)) {
|
|
const ready = await host.op({ op: "request.admission", business: b.id, project });
|
|
if (ready?.ready !== true) throw new Refusal(ready?.code ?? "project-pending");
|
|
}
|
|
// One synchronous broker snapshot covers the thread and the entire manifest after every
|
|
// asynchronous preparation read. Admission is the launch boundary; bindLaunch remains a fence.
|
|
const finalAdmission = await host.op({ op: "request.admission", business: b.id, projects: Object.keys(codeProjects), thread, session });
|
|
if (finalAdmission?.ready !== true) throw new Refusal(finalAdmission?.code ?? "project-pending");
|
|
const managed = registerRuntime(dataRoot, { kind: 'session', business: b.id, run });
|
|
let child;
|
|
try {
|
|
admission(admissionGeneration);
|
|
spawnedAt.set(run, Date.now());
|
|
child = spawnSession({ runDir, env: { ...sessionEnv(env, run, sessionProxy), ...transientEnv }, paths, business: b.id, role: instance, bwrap });
|
|
} catch (e) {
|
|
spawnedAt.delete(run);
|
|
unregisterRuntime(dataRoot, managed.id);
|
|
throw e;
|
|
}
|
|
let bound = false;
|
|
let entry = null;
|
|
const exited = new Promise((resolve) => child.once("close", (code, signal) => resolve(sessionExit(code, signal))));
|
|
child.on("error", (e) => log(`run ${run}: ${e.message}`));
|
|
child.stdin.on("error", () => {});
|
|
try {
|
|
if (child.pid === undefined) throw new Error("the session process did not start");
|
|
const outerStart = startTimeOf(child.pid);
|
|
const identity = processIdentity(child.pid);
|
|
if (!identity) throw new Error("the session process exited at once");
|
|
updateRuntime(dataRoot, managed.id, { ...identity, group: true, phase: 'running' });
|
|
const runner = await runnerOf(child);
|
|
updateRuntime(dataRoot, managed.id, { runnerPid: runner.pid, runnerStartTime: runner.startTime });
|
|
admission(admissionGeneration);
|
|
if (!outerStart) throw new Error("the session process exited at once");
|
|
lock?.update({ run, runPid: child.pid, runStartTime: outerStart });
|
|
const launch = await host.bindLaunch({ business: b.id, role: instance, run, harness: model.harness, provider: model.provider, model: model.model, account, transcript, pid: child.pid, startTime: outerStart, ...(thread ? { thread, session } : {}), ...(admittedProxy ? { modelKeyId } : {}) });
|
|
bound = true;
|
|
admission(admissionGeneration);
|
|
entry = {
|
|
business: b.id, instance, ...(thread ? { thread, session, workspace } : {}), run, launchedBy, harness: model.harness, model: model.model, provider: model.provider, modelKey: model.modelKey,
|
|
pid: child.pid, startTime: outerStart, runnerPid: runner.pid, runnerStartTime: runner.startTime, runDir, startedAt: new Date().toISOString(),
|
|
...(admittedProxy ? { modelKeyId } : {}),
|
|
};
|
|
registry.add(entry);
|
|
// Row 156: one start per decision, whichever launch wrote it into landing.
|
|
landingOf.set(run, new Set(holding?.map((h) => h.decision) ?? []));
|
|
for (const h of holding ?? []) landStarted.add(h.decision);
|
|
child.stdin.end(`${JSON.stringify({ cap: launch.cap })}\n`);
|
|
} catch (e) {
|
|
child.kill("SIGKILL");
|
|
await exited;
|
|
if (bound) await endRun(run, "failed", null);
|
|
spawnedAt.delete(run);
|
|
unregisterRuntime(dataRoot, managed.id);
|
|
if (e instanceof Refusal) throw e;
|
|
if (e.code?.startsWith('proxy-')) throw new Refusal(e.code, e.code);
|
|
throw new Refusal("launch-failed", e.message);
|
|
}
|
|
children.set(run, exited);
|
|
const outputWatch = admittedProxy ? watchModelOutput({ runDir, armAtFirstTurn: true, keyId: modelKeyId, seconds: system.proxy.firstOutputSeconds ?? 120,
|
|
stop: () => host.op({ op: 'modelStall', business: b.id, role: instance, run, keyId: modelKeyId }), receipt: record }) : null;
|
|
exited.then(async ({ code, signal }) => {
|
|
outputWatch?.cancel();
|
|
let receipt;
|
|
try { receipt = runtime(dataRoot, managed.id); }
|
|
catch { log(`run ${run}: processing receipt unavailable at exit`); }
|
|
const state = processingStatus(dataRoot);
|
|
const haltGeneration = receipt?.haltGeneration ?? (state.halted && !state.invalid ? state.generation : null);
|
|
const reason = haltGeneration ? 'kill-switch' : endReason(code, signal);
|
|
try {
|
|
const ended = await endRun(run, reason, signal ? null : code);
|
|
// Keep the exact launch identity for the next host if the broker
|
|
// refused or did not confirm the end.
|
|
if (typeof ended?.released === "boolean") registry.remove(b.id, run);
|
|
// Row 125 (#1619, REQ-LIMIT-4): a run a limit check stopped names the limit, from the claim row.
|
|
record({ event: "end", instance, ...(thread ? { thread, session } : {}), run, reason, exitCode: code, signal, ...(haltGeneration ? { haltGeneration } : {}), ...(ended?.limit ? { limit: ended.limit } : {}) });
|
|
log(`run ${run} (${instance}) ended: ${reason}`);
|
|
} catch (e) {
|
|
log(`run ${run}: ${e.message}`);
|
|
} finally {
|
|
try { unregisterRuntime(dataRoot, managed.id); }
|
|
catch { log(`run ${run}: processing receipt cleanup refused`); }
|
|
if (store) compactions(store, model.harness, harness, { instance, thread, session, run });
|
|
try { lock?.release(); }
|
|
catch (e) { log(`run ${run}: workspace lock release: ${e.message}`); }
|
|
children.delete(run);
|
|
stopping.delete(run);
|
|
spawnedAt.delete(run);
|
|
landingOf.delete(run);
|
|
refusedRelease.delete(run);
|
|
}
|
|
});
|
|
record({ event: "launch", instance, ...(thread ? { thread, session } : {}), run, launchedBy, harness: model.harness, route, routeReason, model: model.model, provider: model.provider, modelKey: model.modelKey, account, pid: child.pid, runnerPid: entry.runnerPid, manifest });
|
|
return { run, instance, harness: model.harness, model: model.model };
|
|
}
|
|
|
|
// Row 121: one tick of thread starts and idle stops, then (row 156) the
|
|
// landing pass. A tick that is still running when the next is due skips it.
|
|
// Row 194 (D6): while row 191's switch is on, only the lifecycle pass runs.
|
|
async function tick() {
|
|
if (ticking || closed) return;
|
|
let halted = false;
|
|
try { admission(); } catch { halted = true; }
|
|
ticking = true;
|
|
try {
|
|
// A --no-launch host runs only the lifecycle pass, which closes each request launches-disabled.
|
|
if (launches && !halted) {
|
|
await threadPass();
|
|
await landPass();
|
|
}
|
|
await requestPass(halted);
|
|
await lifecyclePass();
|
|
} finally {
|
|
ticking = false;
|
|
}
|
|
}
|
|
|
|
// Request recovery starts only the configured chief through the existing --pm path.
|
|
// Workers always come back through that chief's ordinary launch tool/capability.
|
|
async function requestPass(halted = false) {
|
|
for (const [run, request] of launches && !halted ? requestDrains : []) {
|
|
const session = registry.sessions.find(s => s.run === run);
|
|
if (!session) { requestDrains.delete(run); continue; }
|
|
try {
|
|
const out = await host.op({ op: 'request.release', cap: request.cap, record: { business: b.id, run, instance: session.instance, project: request.project } });
|
|
if (out?.released) { terminate([session]); requestDrains.delete(run); }
|
|
} catch (e) {
|
|
// A revoked chief capability cannot keep retiring workers.
|
|
requestDrains.delete(run);
|
|
await host.op({ op: 'request.launchResult', record: { business: b.id, instance: session.instance, code: e.code ?? 'launch-failed', run: null } });
|
|
}
|
|
}
|
|
const pending = await host.op({ op: 'request.launches', business: b.id });
|
|
if (!Array.isArray(pending) || !pending.length) return;
|
|
if (!launches || halted) {
|
|
await host.op({ op: 'request.launchResult', record: { business: b.id, instance: b.launch.by, code: !launches ? 'launches-disabled' : 'processing-halted', run: null } });
|
|
return;
|
|
}
|
|
for (const i of pending) {
|
|
const session = registry.running(b.id, i.target, null);
|
|
if (!session || !i.project || stopping.has(session.run)) continue;
|
|
let projects = [];
|
|
try { projects = JSON.parse(readFileSync(join(session.runDir, 'bundle/manifest.json'), 'utf8')).projects ?? []; } catch {}
|
|
if (!projects.includes(i.project)) {
|
|
const ready = await host.op({ op: 'request.admission', business: b.id, project: i.project });
|
|
if (ready?.ready === true) await host.op({ op: 'request.manifest', record: { business: b.id, instance: i.target, project: i.project, run: session.run } });
|
|
}
|
|
}
|
|
const chief = b.launch?.by;
|
|
if (!chief || registry.running(b.id, chief, null) || pending.some(i => i.chief_run)) return;
|
|
const next = pending.find(i => i.chief === chief && i.chief_attempt !== i.prerequisite);
|
|
if (!next) return;
|
|
try {
|
|
await host.op({ op: 'request.chief', business: b.id });
|
|
const out = await launchPm();
|
|
await host.op({ op: 'request.launchResult', record: { business: b.id, instance: chief, code: null, run: out.run } });
|
|
} catch (e) {
|
|
await host.op({ op: 'request.launchResult', record: { business: b.id, instance: chief, code: e.code ?? 'launch-failed', run: null } });
|
|
}
|
|
}
|
|
|
|
async function threadPass() {
|
|
let threads;
|
|
try {
|
|
threads = await host.op({ op: "threadsState", business: b.id });
|
|
} catch (e) {
|
|
log(`threadsState refused: ${e.code ?? e.message}`);
|
|
return;
|
|
}
|
|
const now = Date.now();
|
|
const ms = (at) => (at ? Date.parse(at) : 0);
|
|
// Row 269: a thread with a legacy session keeps its old store for that session only.
|
|
const legacyThreads = new Set(threads.filter((t) => t.legacy).map((t) => t.thread));
|
|
// Row 271 (264d): a session in a clear is the clear pass's, and nothing below starts or idles it.
|
|
const clearing = await clearPass(now, legacyThreads);
|
|
for (const t of threads) {
|
|
if (closed) return;
|
|
const key = `${t.thread}/${t.session}`;
|
|
if (clearing.has(t.session)) continue;
|
|
const s = registry.running(b.id, t.instance, t.thread, t.session);
|
|
// FED R4 (#1801, design 3.7): the broker marks a thread whose user owner lost the grant, and has
|
|
// ended its claim before it returns the row, whether the file changed or a grant expired. A
|
|
// revoked thread never starts, and a running one gets the idle stop's start-time-checked
|
|
// SIGTERM at once, with no threadIdle: there is no claim left to release.
|
|
if (t.revoked) {
|
|
if (s && !stopping.has(s.run)) {
|
|
stopping.add(s.run);
|
|
record({ event: "thread-revoked", instance: t.instance, thread: t.thread, session: t.session, run: s.run });
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
}
|
|
continue;
|
|
}
|
|
if (s) {
|
|
const last = Math.max(ms(s.startedAt), ms(t.lastMessage), ms(t.lastOpen));
|
|
if (!stopping.has(s.run) && now - last >= b.launch.threadIdleSeconds * 1000) {
|
|
let idle;
|
|
try {
|
|
idle = await host.op({ op: "threadIdle", record: { business: b.id, thread: t.thread, run: s.run } });
|
|
} catch (e) {
|
|
log(`run ${s.run}: threadIdle refused: ${e.code ?? e.message}`);
|
|
continue;
|
|
}
|
|
// A turn in progress, or a message since the snapshot: the run keeps going.
|
|
if (!idle.released) continue;
|
|
stopping.add(s.run);
|
|
record({ event: "thread-idle", instance: t.instance, thread: t.thread, session: t.session, run: s.run });
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
}
|
|
continue;
|
|
}
|
|
const opened = ms(t.lastOpen) > ms(t.lastClaim);
|
|
if (t.holder || !(t.pending > 0 || opened)) continue;
|
|
// One start per message or open: a session that ended without taking
|
|
// the work (it died before its claim, say) waits for the next one
|
|
// rather than start again every tick. A refused start isn't a start.
|
|
const signature = `${t.lastMessage}|${t.lastOpen}`;
|
|
if (startedFor.get(key) === signature) continue;
|
|
await serialize(async () => {
|
|
if (closed || registry.running(b.id, t.instance, t.thread, t.session)) return;
|
|
try {
|
|
const where = { thread: t.thread, session: t.session };
|
|
await start({ instance: t.instance, launchedBy: b.human, ...where, legacy: t.legacy, legacyThread: legacyThreads.has(t.thread), ...(await check(t.instance, where)) });
|
|
startedFor.set(key, signature);
|
|
refusedThreads.delete(key);
|
|
} catch (e) {
|
|
const code = e instanceof Refusal ? e.code : "launch-failed";
|
|
// Logged once per code, not once per tick.
|
|
if (refusedThreads.get(key) === code) return;
|
|
refusedThreads.set(key, code);
|
|
log(`thread ${t.thread} session ${t.session} (${t.instance}): start refused: ${code}`);
|
|
record({ event: "refused", by: b.human, instance: t.instance, thread: t.thread, session: t.session, code, why: e.message, ...(e.limit?{limit:e.limit}:{}) });
|
|
}
|
|
});
|
|
}
|
|
}
|
|
|
|
// Row 271 (264d, #1776, D7 steps 3-5): one step of each clear the broker lists. The drain comes
|
|
// from the run itself (its receive, or its end), so a requested clear waits here. A drained run
|
|
// gets the idle stop's start-time-checked SIGTERM once, and 30 s without its exit proof records
|
|
// uncertain once; nothing is signalled again or forced. The proof is the run's own close handler,
|
|
// whose endRun the broker checks, or for no run, the session's workspace lock. Then retirement,
|
|
// and the successor's start through the same check()/start() as any session. Returns the
|
|
// sessions those steps own this tick.
|
|
async function clearPass(now, legacyThreads) {
|
|
const owned = new Set();
|
|
let clears;
|
|
try {
|
|
clears = await host.op({ op: "sessionClears", business: b.id });
|
|
} catch (e) {
|
|
clearLog("sessionClears", e.code ?? e.message, `sessionClears refused: ${e.code ?? e.message}`);
|
|
return owned;
|
|
}
|
|
for (const c of clears) {
|
|
if (closed) break;
|
|
const start = ["retired", "start-held", "start-refused"].includes(c.stage);
|
|
owned.add(start ? c.successor : c.session);
|
|
try {
|
|
await (start ? clearStart(c, legacyThreads) : clearStop(c, now));
|
|
} catch (e) {
|
|
clearLog(c.op, e.code ?? e.message, `clear ${c.op} (session ${c.session}): ${e.code ?? e.message}`);
|
|
}
|
|
}
|
|
return owned;
|
|
}
|
|
|
|
function clearLog(op, code, line) {
|
|
if (clearLogged.get(op) === code) return;
|
|
clearLogged.set(op, code);
|
|
log(line);
|
|
}
|
|
|
|
async function clearStop(c, now) {
|
|
const where = { business: b.id, thread: c.thread, op: c.op };
|
|
let stage = c.stage;
|
|
if (stage === "drained" || stage === "uncertain") {
|
|
const s = registry.running(b.id, c.instance, c.thread, c.session);
|
|
if (s && s.run === c.run) {
|
|
if (!clearStops.has(c.op)) {
|
|
// An uncertain clear was signalled before this launcher started, and isn't signalled again.
|
|
if (stage === "uncertain") return void clearStops.set(c.op, now);
|
|
// A cancel since the snapshot leaves the run alone; one after this read is a residual.
|
|
const fresh = (await host.op({ op: "sessionClears", business: b.id })).find((x) => x.thread === c.thread && x.op === c.op);
|
|
if (fresh?.stage !== "drained") return;
|
|
clearStops.set(c.op, now);
|
|
stopping.add(s.run);
|
|
record({ event: "session-clear-stop", instance: c.instance, thread: c.thread, session: c.session, run: s.run, op: c.op });
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
} else if (stage === "drained" && now - clearStops.get(c.op) >= stopTimeoutMs) {
|
|
await host.op({ op: "sessionClearUncertain", record: where });
|
|
record({ event: "session-clear-uncertain", instance: c.instance, thread: c.thread, session: c.session, run: s.run, op: c.op });
|
|
}
|
|
return;
|
|
}
|
|
// Some other run of the session (one that never claimed) exits on its refused claim first.
|
|
if (s) return;
|
|
// An earlier host's run that outlived recover()'s kill was ended host-lost, but it still runs:
|
|
// its exit, not that end, is the proof (row 269 Q2). It isn't signalled.
|
|
if (survivors.some((x) => x.business === b.id && x.run === c.run && liveRun(x))) return;
|
|
if (c.run === null) {
|
|
// No run: the proof is the session's workspace lock, which a spawned run holds until its close.
|
|
const live = registry.sessions.filter((x) => x.business === b.id && x.workspace).map((x) => x.workspace);
|
|
const workspace = sessionWorkspace(dataRoot, b.id, c.instance, c.session, { live });
|
|
lockSessionWorkspace(workspace, { hostPid: process.pid, hostStartTime: startTimeOf(process.pid), business: b.id, instance: c.instance, thread: c.thread, session: c.session }).release();
|
|
}
|
|
await host.op({ op: "sessionClearExited", record: { ...where, run: c.run } });
|
|
stage = "exited";
|
|
}
|
|
if (stage !== "exited") return;
|
|
const out = await host.op({ op: "sessionClearRetire", record: where });
|
|
clearStops.delete(c.op);
|
|
record({ event: out.state === "retired" ? "session-clear-retired" : "session-clear-failed", instance: c.instance, thread: c.thread, session: c.session, successor: c.successor, op: c.op, ...(out.refusal ? { code: out.refusal } : {}) });
|
|
}
|
|
|
|
// D8: capacity and limits hold a start, which spends no work attempt. Any other refusal is one.
|
|
const CLEAR_HELD = new Set(["capacity-full", "limit-reached"]);
|
|
|
|
async function clearStart(c, legacyThreads) {
|
|
const where = { business: b.id, thread: c.thread, op: c.op };
|
|
await serialize(async () => {
|
|
if (closed) return;
|
|
if (!registry.running(b.id, c.instance, c.thread, c.successor)) {
|
|
const session = { thread: c.thread, session: c.successor };
|
|
try {
|
|
await start({ instance: c.instance, launchedBy: b.human, ...session, legacy: false, legacyThread: legacyThreads.has(c.thread), ...(await check(c.instance, session)) });
|
|
} catch (e) {
|
|
const code = e instanceof Refusal ? e.code : "launch-failed";
|
|
// Reported and logged once per code, not once per tick.
|
|
if (clearLogged.get(c.op) === code) return;
|
|
clearLogged.set(c.op, code);
|
|
log(`clear ${c.op}: successor ${c.successor} (${c.instance}): start ${CLEAR_HELD.has(code) ? "held" : "refused"}: ${code}`);
|
|
record({ event: "refused", by: b.human, instance: c.instance, thread: c.thread, session: c.successor, code, why: e.message, ...(e.limit ? { limit: e.limit } : {}) });
|
|
await host.op({ op: "sessionClearStart", record: { ...where, start: CLEAR_HELD.has(code) ? "held" : "refused", refusal: code } });
|
|
return;
|
|
}
|
|
}
|
|
clearLogged.delete(c.op);
|
|
await host.op({ op: "sessionClearStart", record: { ...where, start: "started" } });
|
|
});
|
|
}
|
|
|
|
// Row 156 (#1651): start each listed merge decision holder once, and stop a
|
|
// running holder that started before its decision's resolution.
|
|
async function landPass() {
|
|
if (!b.launch) return;
|
|
let holders;
|
|
try {
|
|
holders = await host.op({ op: "landHolders", business: b.id });
|
|
if (!Array.isArray(holders)) throw new Refusal("malformed-answer");
|
|
refusedHolders = null;
|
|
} catch (e) {
|
|
const code = e.code ?? e.message;
|
|
if (refusedHolders !== code) log(`landHolders refused: ${code}`);
|
|
refusedHolders = code;
|
|
return;
|
|
}
|
|
for (const h of holders) {
|
|
if (closed) return;
|
|
if (!b.launch.instances.includes(h.instance)) continue;
|
|
const s = registry.running(b.id, h.instance, null);
|
|
if (s) {
|
|
if (stopping.has(s.run) || landingOf.get(s.run)?.has(h.decision)) continue;
|
|
let r;
|
|
try {
|
|
r = await host.op({ op: "landRelease", record: { business: b.id, run: s.run, decision: h.decision } });
|
|
} catch (e) {
|
|
const code = e.code ?? e.message;
|
|
if (refusedRelease.get(s.run) !== code) log(`run ${s.run}: landRelease refused: ${code}`);
|
|
refusedRelease.set(s.run, code);
|
|
continue;
|
|
}
|
|
// A landing turn or a message turn in progress: a later tick.
|
|
if (!r?.released) continue;
|
|
stopping.add(s.run);
|
|
record({ event: "land-relaunch", instance: h.instance, run: s.run, decision: h.decision });
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
continue;
|
|
}
|
|
if (landStarted.has(h.decision)) continue;
|
|
await serialize(async () => {
|
|
if (closed || registry.running(b.id, h.instance, null)) return;
|
|
try {
|
|
await start({ instance: h.instance, launchedBy: b.human, ...(await check(h.instance)) });
|
|
refusedLand.delete(h.decision);
|
|
} catch (e) {
|
|
const code = e instanceof Refusal ? e.code : "launch-failed";
|
|
if (refusedLand.get(h.decision) === code) return;
|
|
refusedLand.set(h.decision, code);
|
|
log(`decision ${h.decision} (${h.instance}): landing start refused: ${code}`);
|
|
record({ event: "refused", by: b.human, instance: h.instance, decision: h.decision, code, why: e.message, ...(e.limit ? { limit: e.limit } : {}) });
|
|
}
|
|
});
|
|
}
|
|
}
|
|
|
|
// Row 194 (#1692, D1.1): the human's and the tester's lifecycle requests, oldest first.
|
|
async function lifecyclePass() {
|
|
let requests;
|
|
try {
|
|
requests = await host.op({ op: "lifecycleRequests", business: b.id });
|
|
if (!Array.isArray(requests)) throw new Refusal("malformed-answer");
|
|
refusedRequests = null;
|
|
} catch (e) {
|
|
const code = e.code ?? e.message;
|
|
if (refusedRequests !== code) log(`lifecycleRequests refused: ${code}`);
|
|
refusedRequests = code;
|
|
return;
|
|
}
|
|
for (const r of requests) {
|
|
if (closed) return;
|
|
// A taken request is past this pass: it ran, or its done failed.
|
|
if (r.state !== "open") continue;
|
|
await serialize(() => carry(r));
|
|
}
|
|
}
|
|
|
|
// One request: the take, the act, the done. Runs under serialize, so no start() runs alongside it.
|
|
async function carry(r) {
|
|
if (closed) return;
|
|
let take;
|
|
try {
|
|
take = await host.op({ op: "lifecycleTake", record: { business: b.id, request: r.request } });
|
|
} catch (e) {
|
|
log(`request ${r.request} (${r.instance}): lifecycleTake refused: ${e.code ?? e.message}`);
|
|
return;
|
|
}
|
|
if (take?.go !== true) {
|
|
if (take?.refusal && take.refusal !== "request-closed") record({ event: "refused", by: r.by, instance: r.instance, request: r.request, code: take.refusal });
|
|
return;
|
|
}
|
|
let done;
|
|
try {
|
|
done = await act(take);
|
|
} catch (e) {
|
|
done = { refusal: e instanceof Refusal ? e.code : "launch-failed", why: e.message, limit: e.limit };
|
|
}
|
|
const { why, limit, ...closing } = done;
|
|
if (closing.refusal) record({ event: "refused", by: r.by, instance: r.instance, request: r.request, code: closing.refusal, ...(why ? { why } : {}), ...(limit ? { limit } : {}) });
|
|
try {
|
|
await host.op({ op: "lifecycleDone", record: { business: b.id, request: r.request, ...closing } });
|
|
} catch (e) {
|
|
log(`request ${r.request} (${r.instance}): lifecycleDone refused: ${e.code ?? e.message}${closing.run ? `; run ${closing.run} started` : ""}`);
|
|
}
|
|
}
|
|
|
|
// What the request did: { run } for a start, { transaction } for a delete, {} for an end, or { refusal }.
|
|
async function act(r) {
|
|
// Row 163: a --no-launch host starts, signals and renames nothing.
|
|
if (!launches) return { refusal: "launches-disabled" };
|
|
const launchedBy = r.by === "tester" ? "tester" : b.human;
|
|
if (r.action === "lifecycle.launch") return launch(r.instance, launchedBy);
|
|
if (r.action === "lifecycle.end") {
|
|
const ends = ending(r);
|
|
if (r.kill !== true) {
|
|
terminate(ends);
|
|
return {};
|
|
}
|
|
return { stops: await fireStop(r, ends) };
|
|
}
|
|
if (r.action === "lifecycle.regenerate") {
|
|
terminate(registry.sessions.filter((s) => s.business === b.id && s.instance === r.instance));
|
|
const deadline = Date.parse(r.at) + lifecycleRequestMs;
|
|
while (registry.sessions.some((s) => s.business === b.id && s.instance === r.instance)) {
|
|
if (closed) return { refusal: "host-closing" };
|
|
if (Date.now() >= deadline) return { refusal: "regenerate-timeout" };
|
|
await new Promise((resolve) => setTimeout(resolve, 50));
|
|
}
|
|
return launch(r.instance, launchedBy);
|
|
}
|
|
if (r.action === "lifecycle.delete") return remove(r);
|
|
return { refusal: "invalid-request" };
|
|
}
|
|
|
|
async function launch(instance, launchedBy) {
|
|
const pm = instance === b.launch?.by;
|
|
const { run } = await start({ instance, launchedBy, human: true, ...(await check(instance, { pm, human: true })) });
|
|
return { run };
|
|
}
|
|
|
|
// The registry runs an end stops (N6, DESIGN.md:118-120): the runs its verb named, and the runs of
|
|
// the instance spawned by the verb's time that the verb didn't name, which hadn't bound then. A
|
|
// run spawned after the verb is the next end's. A thread instance's end names only the runs it
|
|
// revoked (the tester's own). A run this host didn't spawn counts from its registry start.
|
|
function ending(r) {
|
|
const runs = r.runs ?? [];
|
|
const at = Date.parse(r.at);
|
|
const threaded = THREAD_DEFINITIONS.includes(b.roles[r.instance]?.definition);
|
|
return registry.sessions.filter((s) => s.business === b.id && (runs.includes(s.run) ||
|
|
(!threaded && s.instance === r.instance && (spawnedAt.get(s.run) ?? Date.parse(s.startedAt)) <= at)));
|
|
}
|
|
|
|
// SIGTERM, after the start-time check, to each run. No SIGKILL: only a fire escalates.
|
|
function terminate(sessions) {
|
|
for (const s of sessions) {
|
|
stopping.add(s.run);
|
|
record({ event: "lifecycle-end", instance: s.instance, run: s.run });
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
}
|
|
}
|
|
|
|
// D2.2 (S6): a fire's end stops each run through row 191's stopRun, owned by the end request (fire
|
|
// takes no decision), so a halt sweep that meets the run joins this stop. stopRun checks the
|
|
// identity (start ticks and boot id) before each signal and escalates to SIGKILL after its
|
|
// grace. The runs it killed or left alive go back to the broker as lifecycle.end.killed and
|
|
// lifecycle.end.alive.
|
|
async function fireStop(r, sessions) {
|
|
const results = await Promise.all(sessions.map(async (s) => {
|
|
stopping.add(s.run);
|
|
record({ event: "lifecycle-end", instance: s.instance, run: s.run, kill: true });
|
|
let out;
|
|
try {
|
|
out = await stopRun({ root: dataRoot, business: b.id, run: s.run, reason: "fire", owner: { decision: r.request } }, stopOptions);
|
|
} catch (e) {
|
|
log(`run ${s.run}: fire stop failed: ${e.message}`);
|
|
out = { outcome: "unknown", pid: null, signals: [] };
|
|
}
|
|
record({ event: "fire-stop", instance: s.instance, run: s.run, request: r.request, outcome: out.outcome, pid: out.pid, signals: out.signals });
|
|
return { run: s.run, pid: out.pid, outcome: out.outcome };
|
|
}));
|
|
// sage-2289: a stop that can't confirm the kill (unknown) reads as alive with reason unknown,
|
|
// never as killed, so the Console flags it.
|
|
return results.flatMap((x) => x.outcome === "unknown" ? [{ ...x, outcome: "alive", reason: "unknown" }]
|
|
: x.outcome === "killed" || x.outcome === "alive" ? [x] : []);
|
|
}
|
|
|
|
// V5 steps 1 to 3. The file goes first: once committed, check() refuses the name, so no launch
|
|
// opens the old workspace even if the rename fails.
|
|
async function remove(r) {
|
|
if (registry.sessions.some((s) => s.business === b.id && s.instance === r.instance)) return { refusal: "instance-running" };
|
|
let out;
|
|
try {
|
|
out = await host.op({ op: "lifecycleDelete", record: { business: b.id, request: r.request } });
|
|
} catch (e) {
|
|
return { refusal: e.code ?? "outcome-unknown" };
|
|
}
|
|
const transaction = out?.transaction ?? null;
|
|
if (transaction === null) return { refusal: "workspace-rename-failed" };
|
|
const bot = out.bot ? { bot: out.bot } : {};
|
|
// The old instance workspace, then (row 269) the instance's session workspaces, each moved
|
|
// whole. A retry moves whichever is still in place.
|
|
for (const from of [workspaceDir(dataRoot, b.id, r.instance), join(dataRoot, "session-workspaces", b.id, r.instance)]) {
|
|
if (!existsSync(from)) continue;
|
|
const to = join(dirname(from), ".retired", `${r.instance}-${transaction}`);
|
|
try {
|
|
mkdirSync(dirname(to), { recursive: true, mode: 0o700 });
|
|
chmodSync(dirname(to), 0o700);
|
|
if (existsSync(to)) throw new Error(`${to} exists`);
|
|
renameSync(from, to);
|
|
chmodSync(to, 0o700);
|
|
} catch (e) {
|
|
log(`request ${r.request} (${r.instance}): workspace rename failed: ${e.message}; left at ${from}`);
|
|
return { transaction, ...bot, refusal: "workspace-rename-failed", why: `${e.message}; left at ${from}` };
|
|
}
|
|
record({ event: "workspace-retired", instance: r.instance, request: r.request, transaction, path: to });
|
|
}
|
|
return { transaction, ...bot };
|
|
}
|
|
|
|
// endLaunch's answer ({ released, limit? }), or null when it refused.
|
|
async function endRun(run, reason, exitCode) {
|
|
try {
|
|
return await host.op({ op: "endLaunch", record: { business: b.id, run, reason, exitCode } });
|
|
} catch (e) {
|
|
log(`run ${run}: endLaunch refused: ${e.code ?? e.message}`);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// A launch a session asked for. Resolves to the reply line's object.
|
|
function fromSession(cap, instance) {
|
|
return serialize(async () => {
|
|
if (closed) return { ok: false, error: "host-closing" };
|
|
let who;
|
|
try {
|
|
who = await host.op({ op: "identity", cap });
|
|
} catch (e) {
|
|
return { ok: false, error: e.code ?? "unauthenticated" };
|
|
}
|
|
if (!who.role || who.business !== b.id) {
|
|
await host.op({ op: "refuse", cap, code: "not-a-role" }).catch(() => {});
|
|
return { ok: false, error: "not-a-role" };
|
|
}
|
|
if (!launches) {
|
|
await host.op({ op: "refuse", cap, code: "launches-disabled" }).catch(() => {});
|
|
record({ event: "refused", by: who.role, instance, code: "launches-disabled" });
|
|
return { ok: false, error: "launches-disabled" };
|
|
}
|
|
const reply = async out => {
|
|
if (typeof instance === 'string') await host.op({ op: 'request.launchResult', record: { business: b.id, instance, code: out.ok ? null : out.error, run: out.ok ? out.result.run : null } });
|
|
return out;
|
|
};
|
|
try { b = loadBusiness(host.business, { dir: configDir(env), rolesDir: rolesDir(env) }); }
|
|
catch { return reply({ ok: false, error: 'organization-refused' }); }
|
|
// Product enums and clones are frozen in a session's manifest. A legitimate chief
|
|
// launch may request a drain, never interrupt the turn or start a privileged worker.
|
|
const running = registry.running(b.id, instance, null);
|
|
if (running && who.role === b.launch?.by) {
|
|
const pending = await host.op({ op: 'request.launches', business: b.id });
|
|
const needed = pending.filter(i => i.target === instance && i.project);
|
|
let projects = [];
|
|
try { projects = JSON.parse(readFileSync(join(running.runDir, 'bundle/manifest.json'), 'utf8')).projects ?? []; } catch {}
|
|
const missing = needed.find(i => !projects.includes(i.project));
|
|
if (missing) {
|
|
if (!b.launch?.instances.includes(instance)) return reply({ ok: false, error: 'instance-not-listed' });
|
|
try {
|
|
await host.op({ op: 'authorizeLaunch', cap, instance });
|
|
const ready = await host.op({ op: 'request.admission', business: b.id, project: missing.project });
|
|
if (ready?.ready !== true) return reply({ ok: false, error: ready?.code ?? 'project-pending' });
|
|
const out = await host.op({ op: 'request.release', cap, record: { business: b.id, run: running.run, instance, project: missing.project } });
|
|
if (out?.released) terminate([running]);
|
|
else requestDrains.set(running.run, { cap, project: missing.project });
|
|
return reply({ ok: false, error: 'request-manifest-draining' });
|
|
} catch (e) { return reply({ ok: false, error: e.code ?? 'launch-failed' }); }
|
|
}
|
|
}
|
|
let checked;
|
|
try {
|
|
if (typeof instance !== "string") throw new Refusal("invalid-request");
|
|
checked = await check(instance);
|
|
} catch (e) {
|
|
if (!(e instanceof Refusal)) throw e;
|
|
await host.op({ op: "refuse", cap, code: e.code }).catch(() => {});
|
|
record({ event: "refused", by: who.role, instance, code: e.code, why: e.message, ...(e.limit?{limit:e.limit}:{}) });
|
|
return reply({ ok: false, error: e.code });
|
|
}
|
|
try {
|
|
await host.op({ op: "authorizeLaunch", cap, instance });
|
|
} catch (e) {
|
|
record({ event: "refused", by: who.role, instance, code: e.code ?? "not-authorized" });
|
|
return reply({ ok: false, error: e.code ?? "not-authorized" });
|
|
}
|
|
try {
|
|
return reply({ ok: true, result: await start({ instance, launchedBy: who.role, ...checked }) });
|
|
} catch (e) {
|
|
const code = e instanceof Refusal ? e.code : "launch-failed";
|
|
log(`launch of ${instance} for ${who.role} failed: ${e.message}`);
|
|
await host.op({ op: "refuse", cap, code }).catch(() => {});
|
|
record({ event: "refused", by: who.role, instance, code, why: e.message, ...(e.limit?{limit:e.limit}:{}) });
|
|
return reply({ ok: false, error: code });
|
|
}
|
|
});
|
|
}
|
|
|
|
// `bus start --pm`: the business's launcher instance, launched by the human.
|
|
function launchPm() {
|
|
return serialize(async () => {
|
|
if (!b.launch) throw new CliError(`business ${b.id} has no launch block, so --pm has nothing to launch`, 3);
|
|
try {
|
|
return await start({ instance: b.launch.by, launchedBy: b.human, ...(await check(b.launch.by, { pm: true })) });
|
|
} catch (e) {
|
|
if (!(e instanceof Refusal)) throw e;
|
|
record({ event: "refused", by: b.human, instance: b.launch.by, code: e.code, why: e.message, ...(e.limit?{limit:e.limit}:{}) });
|
|
const error = new CliError(`PM launch refused: ${e.code}${e.message !== e.code ? ` (${e.message})` : ""}`, 3);
|
|
error.code = e.code;
|
|
throw error;
|
|
}
|
|
});
|
|
}
|
|
|
|
// The earlier host's sessions. Their capabilities died with its broker, so
|
|
// one that still runs can do nothing: kill it (the namespace takes its
|
|
// turn with it). Then end each run in the bus, which releases its role.
|
|
// This broker never bound those runs, so each is bound again first with
|
|
// its identical launch record, which the broker accepts.
|
|
async function recover() {
|
|
// Row 163 (mos-0961): a --no-launch host never signals or ends an earlier host's session. Its
|
|
// data root may hold a sessions.json that names live processes, so it only says what it left.
|
|
if (!launches) {
|
|
for (const s of leftover) log(`run ${s.run} (${s.instance}) from an earlier host: left as it is, this host was started with --no-launch`);
|
|
leftover = [];
|
|
return;
|
|
}
|
|
for (const s of leftover) {
|
|
if (startTimeOf(s.pid) === s.startTime && (cmdlineOf(s.pid) ?? [])[0] === bwrap) {
|
|
try {
|
|
process.kill(s.pid, "SIGKILL");
|
|
} catch {}
|
|
const end = Date.now() + 5000;
|
|
while (liveRun(s) && Date.now() < end) await new Promise((r) => setTimeout(r, 20));
|
|
}
|
|
// Row 269 (Q2): the session or its runner outlived the kill, or wasn't a session to kill.
|
|
if (liveRun(s)) {
|
|
survivors.push(s);
|
|
log(`run ${s.run} (${s.instance}) from an earlier host still runs (pid ${s.pid}, runner ${s.runnerPid}): no session of ${s.instance} starts until it exits`);
|
|
}
|
|
try {
|
|
await host.op({ op: 'recoverLaunch', record: { business: b.id, role: s.instance, run: s.run, harness: s.harness, pid: s.pid, startTime: s.startTime, ...(s.thread ? { thread: s.thread } : {}), ...(s.session ? { session: s.session } : {}) } });
|
|
} catch (e) {
|
|
// A host that died after ending the run but before updating sessions.json.
|
|
if (e.code === "run-ended") log(`run ${s.run} (${s.instance}) from an earlier host was already ended`);
|
|
else log(`run ${s.run} (${s.instance}) from an earlier host: not ended, historical cleanup refused: ${e.code ?? e.message}`);
|
|
continue;
|
|
}
|
|
record({ event: "end", instance: s.instance, ...(s.thread ? { thread: s.thread } : {}), ...(s.session ? { session: s.session } : {}), run: s.run, reason: "host-lost", exitCode: null, signal: null });
|
|
log(`run ${s.run} (${s.instance}) from an earlier host ended: host-lost`);
|
|
}
|
|
leftover = [];
|
|
}
|
|
|
|
async function listen() {
|
|
await recover();
|
|
const path = launchSocketPath(dataRoot);
|
|
if (existsSync(path)) {
|
|
if (!lstatSync(path).isSocket()) throw new CliError(`${path} exists and isn't a socket`, 3);
|
|
rmSync(path);
|
|
}
|
|
server = createServer((socket) => {
|
|
sockets.add(socket);
|
|
socket.once("close", () => sockets.delete(socket));
|
|
let buf = "";
|
|
let answered = false;
|
|
const reply = (obj) => {
|
|
if (answered) return;
|
|
answered = true;
|
|
socket.end(`${JSON.stringify(obj)}\n`);
|
|
};
|
|
socket.setTimeout(10000, () => reply({ ok: false, error: "invalid-request" }));
|
|
socket.on("error", () => {});
|
|
socket.on("data", (chunk) => {
|
|
if (answered) return;
|
|
buf += chunk;
|
|
if (buf.length > LINE_MAX) return reply({ ok: false, error: "invalid-request" });
|
|
const nl = buf.indexOf("\n");
|
|
if (nl < 0) return;
|
|
socket.setTimeout(0);
|
|
let req;
|
|
try {
|
|
req = JSON.parse(buf.slice(0, nl));
|
|
} catch {
|
|
return reply({ ok: false, error: "invalid-request" });
|
|
}
|
|
if (typeof req?.cap !== "string" || !/^[0-9a-f]{64}$/.test(req.cap)) return reply({ ok: false, error: "unauthenticated" });
|
|
fromSession(req.cap, req.instance).then(reply, (e) => {
|
|
log(`launch socket: ${e.message}`);
|
|
reply({ ok: false, error: "launch-failed" });
|
|
});
|
|
});
|
|
});
|
|
// bus-host/ is 0700, so the moment before the chmod exposes nothing.
|
|
await new Promise((resolve, reject) => {
|
|
server.once("error", reject);
|
|
server.listen(path, resolve);
|
|
});
|
|
chmodSync(path, 0o600);
|
|
await gitSocket.listen();
|
|
if (b.launch) ticker = setInterval(() => tick().catch((e) => log(`thread tick: ${e.message}`)), threadTickMs);
|
|
}
|
|
|
|
// SIGTERM every runner, wait, then end what's left. Runs before the host
|
|
// stops the broker, so endLaunch still works.
|
|
async function close() {
|
|
closed = true;
|
|
if (ticker) clearInterval(ticker);
|
|
if (server) {
|
|
// server.close waits for every connection, and a client that never
|
|
// ends its side (or never sends) would hold it; so they go first.
|
|
const stopped = new Promise((r) => server.close(r));
|
|
for (const socket of sockets) socket.destroy();
|
|
await stopped;
|
|
rmSync(launchSocketPath(dataRoot), { force: true });
|
|
}
|
|
await gitSocket.close();
|
|
await chain;
|
|
// Again every second: a runner still starting has no handler yet, and as
|
|
// pid 1 of its namespace it never sees a signal sent before it has one.
|
|
const term = () => {
|
|
for (const s of registry.sessions) {
|
|
try {
|
|
if (startTimeOf(s.runnerPid) === s.runnerStartTime) process.kill(s.runnerPid, "SIGTERM");
|
|
} catch {}
|
|
}
|
|
};
|
|
term();
|
|
const again = setInterval(term, 1000);
|
|
const pending = [...children.values()];
|
|
const timer = setTimeout(() => {
|
|
for (const s of registry.sessions) {
|
|
try {
|
|
if (startTimeOf(s.pid) === s.startTime) process.kill(s.pid, "SIGKILL");
|
|
} catch {}
|
|
}
|
|
}, stopTimeoutMs);
|
|
await Promise.all(pending);
|
|
// The exit handlers run on the same tick as the close events; let them finish.
|
|
while (children.size) await new Promise((r) => setTimeout(r, 20));
|
|
clearInterval(again);
|
|
clearTimeout(timer);
|
|
}
|
|
|
|
host.beforeClose(close);
|
|
return { business: b, registry, listen, launchPm, fromSession, tick, close };
|
|
}
|