fix(harness): back off refused launches (#797)

BUILD-LOG.md

@@ -6210,3 +6210,5 @@ After: personally authored three-file skills/ms-ux-improvement coordinator packa
 2026-11-06 UTC | #797 stall-guard implementation started | Pinned 5df42478b6c1ef5e22e2eb6c3fa8605722ab5f77 on feat/ui797-stall-guard. Before: both pi print mode and Claude text mode emit the turn reply only at completion, so a live first turn can be stopped despite model output; a chief can also immediately repeat an instance-running launch refusal. Scope is event-based first output and bounded launch refusal handling with focused and package gates; no configuration, role, schema, proxy initialization or live-instance change. The queue command refused in the linked worktree; no canonical queue access.
 
 2026-11-06 UTC | #797 F6 streamed first output | Managed pi JSON events and Claude stream-json partial events pass through one harness monitor; substantive text, thinking and tool-use events mark a fixed output token while stdout remains the final answer. Six real confined runner/adapter controls use a six-second guard: both streaming turns survived and both headers-only and keepalive-only cases stopped; both streaming tests failed on the pinned base and all four stall controls passed there. Harness suite 250 passed; CLI main partition 909 passed, one failed and one skipped, with its 53-test file rerun green; the failure also reproduced on the base. No guard threshold, proxy initialization, configuration, schema or role changes. Release and lane partitions are retained under /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-evidence/.
+
+2026-11-06 UTC | #797 F7 launch refusal guard | The typed launch path now reports instance-running as an existing run to wait for, serializes calls per instance and delays a new call for 30 seconds after a refusal; a disconnected caller cannot start a queued launch later. There is no automatic retry, an unknown outcome remains distinct, other instances are independent, and success clears the delay. Five new tests failed on the F6 base and passed after the fix; the complete harness suite passed 255/255. The backoff lives for the requesting runner's lifetime and resets on its restart. No policy, role, schema, configuration or live change.

docs/SESSIONS.md

@@ -2071,3 +2071,4 @@ are never rewritten or removed; corrections are new entries.
 2026-11-06 UTC | ui797-support-fix-r4 implementation sub-agent | #797 review corrections on feat/ui797-support above 905d313cd | Minimal browser mount/environment boundary; independent config-byte proof and retained binding keep halt on sidecar damage; normal-path campaign hash/epoch agreement; unconditional namespaces and HTTPS 17443/oidc docs. Baseline failures retained; six controls corrected; whole files 112 pass, 0 fail/skip/cancel; five retained probes exit 0, zero forbidden traffic, no owned survivors/listeners, TMPDIR removed. Five support paths, one appended BUILD entry and this line; prefix bytes verified. Queue refused linked worktree; no queue/forge/push/stash/model/secrets/.53/live or unrelated-session action. Local author candidate for independent review. Report: /mnt/storage/scratch/mosaic-lead/mvp3/ui797-support-fix-r4-report.md.
 2026-11-06 UTC | MVP3 lead #797 (Opus) | #797 support R4 P1 fix on feat/ui797-support above fce831c05 | Conflicting ledger targets now need owner corroboration or are replaced by the owner-verified pin, else refused; 4 regressions fail on r4 and pass after; whole files 116 pass/0 fail; reviewer probe 0 wrong halts. Local commit; independent r5 review next. No push/stash/.53/live action.
 2026-11-06 UTC | MVP3 lead #797 (Opus) | #797 support R5 P1 fix on feat/ui797-support above 85e1ec5d1 | Pin must name its own campaign's and the owner's target; serve freezes startup paths and retains the ledger path; running service corroborates by its unchanged startup ledger. Owner business/config retarget refuses without retained evidence (tradeoff recorded). 4 regressions (3 fail on r5); whole files 120 pass/0 fail; reviewer probes 0 wrong halts. Local commit; independent r6 review next. No push/stash/.53/live action.
+2026-11-06 UTC | ui797 stall-guard implementation sub-agent | #797 two fixes on feat/ui797-stall-guard from 5df42478b | Adapter event monitoring for first model output and typed launch refusal backoff, with base/fix evidence and package gates under /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-evidence/; linked-worktree queue refused. Local commits authorized; independent review pending. No push, stash, protected host, live-instance, role, policy, schema or config change. Report: /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-report.md.

packages/harness/src/runner.mjs

@@ -33,6 +33,7 @@
 // stayed unreachable, 2 a bad run directory.
 
 import { spawn } from "node:child_process";
+import { setTimeout as delay } from "node:timers/promises";
 import { chmodSync, existsSync, mkdirSync, openSync, readFileSync, rmSync, closeSync } from "node:fs";
 import { createServer, connect } from "node:net";
 import { join } from "node:path";
@@ -286,6 +287,44 @@ function launchCall(path, request, timeout = 120_000) {
   });
 }
 
+// Serialize launches per instance, and slow repeated refusals. A launch is
+// never retried here: only a new tool call can reach the host again.
+export function launchBackoff(send, { now = Date.now, wait = (ms, signal) => delay(ms, undefined, { signal }) } = {}) {
+  const instances = new Map();
+  return async (args, { signal } = {}) => {
+    const key = args.instance;
+    for (const [name, state] of instances) {
+      if (!state.queued && state.until <= now()) instances.delete(name);
+    }
+    let state = instances.get(key);
+    if (!state) { state = { tail: Promise.resolve(), queued: 0, until: 0 }; instances.set(key, state); }
+    const previous = state.tail;
+    let release;
+    state.tail = new Promise(resolve => { release = resolve; });
+    state.queued++;
+    try {
+      await previous;
+      if (signal?.aborted) throw Object.assign(new Error('cancelled'), { code: 'cancelled' });
+      const remaining = state.until - now();
+      if (remaining > 0) await wait(remaining, signal);
+      if (signal?.aborted) throw Object.assign(new Error('cancelled'), { code: 'cancelled' });
+      try {
+        const result = await send(args);
+        state.until = 0;
+        return result;
+      } catch (e) {
+        // An unknown outcome is not a refusal; do not describe it as one.
+        if (e.code && e.code !== 'outcome-unknown') state.until = now() + 30_000;
+        throw e;
+      }
+    } finally {
+      state.queued--;
+      release();
+      if (!state.queued && !state.until) instances.delete(key);
+    }
+  };
+}
+
 // Serve the typed tools on `path`. Returns the server. `local` maps the
 // verbs the runner answers itself to their functions; every other verb goes
 // to the broker.
@@ -294,13 +333,18 @@ export function toolServer({ path, tools, client, local }) {
   rmSync(path, { force: true });
   const server = createServer((socket) => {
     let input = "";
+    let received = false;
+    const controller = new AbortController();
+    socket.on("close", () => controller.abort());
     const answer = (r) => socket.end(JSON.stringify(r) + "\n");
     socket.setTimeout(150_000, () => socket.destroy());
     socket.on("error", () => {});
     socket.on("data", async (b) => {
+      if (received) return;
       input += b.toString("utf8");
       if (input.length > 65536) return answer({ ok: false, error: "request-too-large" });
       if (!input.includes("\n")) return;
+      received = true;
       let request;
       try {
         request = JSON.parse(input);
@@ -311,7 +355,7 @@ export function toolServer({ path, tools, client, local }) {
       if (!tool) return answer({ ok: false, error: "unknown-tool" });
       const args = request.args && typeof request.args === "object" && !Array.isArray(request.args) ? request.args : {};
       try {
-        const result = Object.hasOwn(local, tool.verb) ? await local[tool.verb](args) : await client.call(tool.verb, args);
+        const result = Object.hasOwn(local, tool.verb) ? await local[tool.verb](args, { signal: controller.signal }) : await client.call(tool.verb, args);
         answer({ ok: true, result });
       } catch (e) {
         answer({ ok: false, error: /^[a-z-]{1,64}$/.test(e.code ?? "") ? e.code : "tool-failed" });
@@ -497,7 +541,7 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
     tools,
     client,
     local: {
-      launch: (args) => launchCall(session.launchSocket, { cap, instance: args.instance }),
+      launch: launchBackoff((args) => launchCall(session.launchSocket, { cap, instance: args.instance })),
       "working.push": (args) => pushWorking({ path: session.gitSocket, cap, workspace: session.workspace, task_ref: args.task_ref, project: args.project, branch: args.branch }),
       "working.fetch": (args) => fetchReview({ path: session.gitSocket, cap, workspace: session.workspace, task_ref: args.task_ref, request: args.request }),
       "working.land": (args) => landWork({ path: session.gitSocket, cap, task_ref: args.task_ref, decision: args.decision }),

packages/harness/src/tools.mjs

@@ -164,7 +164,7 @@ export function typedTools({ resolved, arbiter = false, tracker = false, project
   // Row 194 (#1692, D3 item 5): the instance is a name, not an enum fixed when the bundle was built,
   // so the chief launches its own hire without a restart. The host checks the listing at each launch.
   if (resolved.launch) {
-    add("launch", "launch", "role.launch", `Launch a managed session for a role instance the organization lists for you to launch. When this session started they were: ${resolved.launch.instances.join(", ")}. An instance you hired since is listed too. The host refuses an instance that isn't listed, one that is already running, a full model family, or a launch while launches are switched off.`, obj({
+    add("launch", "launch", "role.launch", `Launch a managed session for a role instance the organization lists for you to launch. When this session started they were: ${resolved.launch.instances.join(", ")}. An instance you hired since is listed too. The host refuses an instance that isn't listed, one that is already running, a full model family, or a launch while launches are switched off. If instance-running is returned, the instance is already running; wait for its existing run instead of retrying.`, obj({
       instance: str("Role instance to launch", { pattern: ID_PATTERN.source }),
     }, ["instance"]));
   }
@@ -429,7 +429,7 @@ export function callTool(path, tool, args = {}, { timeout = 60_000 } = {}) {
       if (!input.includes("\n")) return;
       try {
         const r = JSON.parse(input);
-        r.ok ? end(null, r.result) : end(new Error(tool === "intake_question" && r.error === "outcome-unknown" ? "outcome-unknown" : `refused: ${r.error}`));
+        r.ok ? end(null, r.result) : end(new Error(tool === "intake_question" && r.error === "outcome-unknown" ? "outcome-unknown" : tool === "launch" && r.error === "instance-running" ? "refused: instance-running: already running; wait for the existing run" : `refused: ${r.error}`));
       } catch {
         end(unknown("tool reply isn't JSON"));
       }

packages/harness/tests/launch-backoff.test.mjs

@@ -0,0 +1,96 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
+import { connect } from 'node:net';
+import * as runner from '../src/runner.mjs';
+import { callTool } from '../src/tools.mjs';
+
+const turn = () => new Promise(resolve => setImmediate(resolve));
+test('a running instance is a wait: repeated launch calls are delayed, independently per instance', async t => {
+  const guard = runner.launchBackoff ?? (send => send);
+  const dir = fs.mkdtempSync(join(tmpdir(),'backoff797-'));
+  t.after(() => fs.rmSync(dir, {recursive:true,force:true}));
+  const calls=[], delays=[]; let now=0;
+  const launch = guard(async ({instance}) => {
+    calls.push(instance);
+    if (instance === 'pm') throw Object.assign(new Error('instance-running'), {code:'instance-running'});
+    return {run:'other-run'};
+  }, {now:()=>now, wait:ms=>new Promise(resolve=>delays.push({ms,resolve}))});
+  const path=join(dir,'tools.sock');
+  const server=await runner.toolServer({path,tools:[{name:'launch',verb:'launch'}],client:{},local:{launch}});
+  t.after(()=>new Promise(resolve=>server.close(resolve)));
+  await assert.rejects(callTool(path,'launch',{instance:'pm'}),/instance-running.*already running.*wait/i);
+  const second=callTool(path,'launch',{instance:'pm'});
+  for(let n=0;n<100 && !delays.length;n++) await new Promise(resolve=>setTimeout(resolve,2));
+  assert.deepEqual(calls,['pm']);
+  assert.equal(delays.length,1); assert.ok(delays[0].ms >= 30000);
+  assert.deepEqual(await callTool(path,'launch',{instance:'coder'}),{run:'other-run'});
+  now += delays[0].ms; delays[0].resolve();
+  await assert.rejects(second,/instance-running.*already running.*wait/i);
+  assert.deepEqual(calls,['pm','coder','pm']);
+});
+
+test('other repeated launch refusals back off too, and a success clears the delay', async () => {
+  const guard = runner.launchBackoff ?? (send => send);
+  let now=0, calls=0, refusal=true; const waits=[];
+  const launch=guard(async () => {calls++; if(refusal) throw Object.assign(new Error('capacity-full'),{code:'capacity-full'}); return {run:'r1'};},
+    {now:()=>now,wait:async ms=>{waits.push(ms);now+=ms;}});
+  await assert.rejects(launch({instance:'coder'}),{code:'capacity-full'});
+  refusal=false;
+  assert.deepEqual(await launch({instance:'coder'}),{run:'r1'});
+  assert.deepEqual(waits,[30000]);
+  assert.deepEqual(await launch({instance:'coder'}),{run:'r1'});
+  assert.deepEqual(waits,[30000]); assert.equal(calls,3);
+});
+
+test('disconnecting a delayed launch cancels it before it can reach the host', async () => {
+  const guard = runner.launchBackoff ?? (send => send);
+  let calls=0; const controller=new AbortController();
+  const launch=guard(async () => {calls++; throw Object.assign(new Error('instance-running'),{code:'instance-running'});},
+    {now:()=>0,wait:(ms,signal)=>new Promise((resolve,reject)=>signal.addEventListener('abort',()=>reject(Object.assign(new Error('cancelled'),{code:'cancelled'})),{once:true}))});
+  await assert.rejects(launch({instance:'pm'}),{code:'instance-running'});
+  const waiting=launch({instance:'pm'},{signal:controller.signal}).catch(e=>e);
+  await turn(); assert.equal(calls,1,'no second host call before the waiting period');
+  controller.abort(); await waiting;
+  assert.equal(calls,1,'a disconnected caller cannot trigger a later launch');
+});
+
+test('simultaneous calls for one instance cannot bypass a refusal', async () => {
+  const guard = runner.launchBackoff ?? (send => send);
+  let calls=0, finish; const waits=[];
+  const launch=guard(async () => {
+    calls++;
+    if (calls === 1) await new Promise(resolve => { finish=resolve; });
+    throw Object.assign(new Error('instance-running'),{code:'instance-running'});
+  }, {now:()=>0,wait:ms=>new Promise(resolve=>waits.push({ms,resolve}))});
+  const first=launch({instance:'pm'}).catch(e=>e);
+  await turn();
+  const second=launch({instance:'pm'}).catch(e=>e);
+  await turn(); assert.equal(calls,1);
+  finish(); await first; await turn();
+  assert.equal(calls,1); assert.equal(waits.length,1);
+  waits[0].resolve(); await second; assert.equal(calls,2);
+});
+
+test('a disconnected tool socket cancels a launch waiting for backoff', async t => {
+  const guard = runner.launchBackoff ?? (send => send);
+  const dir=fs.mkdtempSync(join(tmpdir(),'backoff797-socket-'));
+  t.after(()=>fs.rmSync(dir,{recursive:true,force:true}));
+  let calls=0, waiting;
+  const launch=guard(async ()=>{calls++;throw Object.assign(new Error('instance-running'),{code:'instance-running'});},
+    {now:()=>0,wait:(ms,signal)=>new Promise((resolve,reject)=>{waiting={resolve,signal};signal.addEventListener('abort',()=>reject(new Error('cancelled')),{once:true});})});
+  const path=join(dir,'tools.sock');
+  const server=await runner.toolServer({path,tools:[{name:'launch',verb:'launch'}],client:{},local:{launch}});
+  await assert.rejects(callTool(path,'launch',{instance:'pm'}));
+  const socket=connect(path);
+  t.after(()=>{ socket.destroy(); return new Promise(resolve=>server.close(resolve)); });
+  socket.on('error',()=>{});
+  socket.write(JSON.stringify({tool:'launch',args:{instance:'pm'}})+'\n');
+  for(let n=0;n<100 && !waiting;n++) await new Promise(resolve=>setTimeout(resolve,2));
+  assert.ok(waiting); socket.destroy();
+  for(let n=0;n<100 && !waiting.signal.aborted;n++) await new Promise(resolve=>setTimeout(resolve,2));
+  assert.equal(waiting.signal.aborted,true);
+  waiting.resolve(); await turn(); assert.equal(calls,1);
+});