fix(harness): treat a length stop with no answer as a failed turn (#1852)
Pi could spend its output limit on thinking and the runner relayed an empty answer as a result. Preserve final stop metadata and allow two bounded continuations, accounting for the whole message in one transcript usage delta. Record exhaustion on the message trail and notify the chief without an automatic reply cycle. Mount the shared notice prefix in the sandbox.
This commit is contained in:
1 parent
3598839e63
commit
3c689efa31
12 files changed
+248
-8
No files matched your search
@@ -6320,3 +6320,6 @@ R5 quote fallback room (#797 D3, 2026-11-06, MVP3 lead): r4 review P2 found that
|
||||
2026-11-06 UTC | #797 round A fix r3 (Sol r2 P2) | Before (16384bb12): a mosaic-request-state for an id advanced the thread's epoch only when a remembered miss was deleted, so a 404 still in flight from before the observation could record the miss afterwards and timer refreshes kept fabricating 404s for an open request. After: each observed state advances that id's own generation on every thread that retains it; an in-flight 404 records a miss only if both the thread epoch and the id generation are unchanged. Other ids' slow misses still settle. Generations are pruned to retained rows with the misses. New held-404 test fails on 16384bb12 chat.js at recovery; webui chat-sessions-access, chat and browser files 41/0/0 (Node 26 serial). Lead ruling: no third review round. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/roundA-fix-r3-evidence/.
|
||||
2026-11-06 UTC | #797 round A fix r4 (Lead push gate red on 9d3eb9e11) | Before (9d3eb9e11): the chat.js fetch wrapper recorded a request-not-found from /api/requests/<id>/workspace as a missing request, then served a synthetic 404 to every later read of that id, including request panel rereads; request-browser.test.mjs failed 11/24 (its stub has no workspace route), base 6b84fe136 24/0. After: only the request's own read records a miss; a remembered miss still answers its workspace read in the page. New regression test fails on 9d3eb9e11 chat.js and passes on the fix. The r3 held-404 test raced Files' cached repaint under concurrency 3 (678/1; alone 5/5 pass); it now waits, bounded, for the refresh's reads to reach the server and still fails on 16384bb12. request-browser 24/0; whole webui suite except project-files 679/0/0/0 (Node 26, concurrency 3). Correction: the r1-r3 author gate ran only chat-sessions-access, chat and browser and omitted request-browser.test.mjs, which also exercises chat.js; that omission let this regression reach the push gate. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/roundA-fix-push-gate/r4-*.
|
||||
2026-11-06 UTC | #1847/#1848 fix row (Round B halt, key roundb-2-halt) | Before (1186cf7b1): the runner auto-replied a RESULT to every non-RESULT message, so chief INFO/RESULT replies to assistant RESULTs looped (thread 40df75c5, 54 messages, 1,036,184 assistant tokens); continuation notices keyed on business-wide launch/end/claim seqs, so every unrelated lifecycle event, including kill-switch ends during a halt, re-noticed the chief and drove 13 instance-running refusals (1,198,607 tokens). After: the broker adds in_reply_to_class to agent inbox rows and the runner sends no automatic reply to an agent's non-REQUEST reply to a RESULT (REQUEST, ordinary INFO and human/tester/user messages unchanged); continuation versions use only the target's own lifecycle events and claims, no notice while processing is halted or while the target's latest launch has not ended; the instance-running refusal tells the chief not to relaunch or relay. Builder 4048e0f60 (Codex Terra xhigh); Sol xhigh review r1 CHANGES: a halted broker restart failed on the new retry refusal (P1) and an older unmatched launch hid later ends, worsening #1851 (P2). Lead fix c04529c7b: startup request recovery waits while processing is halted or invalid, as the periodic pass does, and resume reschedules it; only the target's latest lifecycle event gates notices. Each regression test fails on its base. Gate on c04529c7b (Node 26.12.1): touched 123/0, bus except project-files 1648/0, harness 318/0, webui 678/1 then 679/0 on rerun (decide-browser I1 timed out under full-suite load; file alone 9/9 twice; no webui diff vs base). project-files.test.mjs hangs on base and candidate alike. Residual: a latest launch whose end is lost still suppresses notices until host recovery (#1851). Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/fix-1847-1848-gate/.
|
||||
|
||||
2026-11-06 UTC | #1852 length-stop implementation | Before: an empty Pi length stop could become an empty RESULT, and the GLM registration inherited a 16,384 output cap. Candidate: marker metadata identifies the failed turn, bounded continuations share one usage transcript delta, exhaustion writes a message-trail event and a chief notice, and GLM limits are a separate plugin change. The lead accepted the larger limit's Pi compact-and-retry effect and the later usage limit check. The first bus gate exposed a missing sandbox binding for the shared notice constant; the binding was added, and the three affected launch cases passed in their file. A Claude row-66 assertion failed intermittently in candidate runs, while an untouched base full harness run passed; no base reproduction is claimed. The existing library citation retry's :retry usage calls remain a follow-up. Local work only; no push or deployment. Evidence: /mnt/storage/scratch/tmp/mDnSad/.
|
||||
2026-11-06 UTC | #1852 verification | Node 26 final harness 332/0, bus excluding project-files 1648/0, launcher and sandbox 69/0, and additional grep-selected tests 252/0. An overlapping bus run had library-sync-busy and a 30-second timeout; both affected files then passed 27/0 and the full bus rerun passed. The candidate's Claude row-66 test failed in an earlier full and isolated run, then the candidate and exact-base full harness suites passed 332/0 and 318/0 respectively; the base failure was not reproduced. Evidence: /mnt/storage/scratch/tmp/mDnSad/.
|
||||
@@ -2109,3 +2109,4 @@ are never rewritten or removed; corrections are new entries.
|
||||
2026-11-06 UTC | mvp3-lead-797 (Claude, #797 lane) | roundA-fix r3 on 16384bb12 for Sol r2 P2 | A request state observed before an in-flight 404 settles now fences that miss via a per-id generation. Negative fails on 16384bb12, webui gate 41/0/0. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/roundA-fix-r3-evidence/.
|
||||
2026-11-06 UTC | mvp3-lead-797 (Claude, #797 lane) | roundA-fix r4 on 9d3eb9e11 for the Lead's red webui push gate | Workspace 404s no longer record a request miss; the r3 held-404 test waits for its refresh reads. request-browser 24/0, whole webui 679/0/0/0; r1-r3 gate omission of request-browser recorded in BUILD-LOG. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/roundA-fix-push-gate/r4-*.
|
||||
2026-11-06 UTC | mvp3-lead-797 (Claude, #797 lane) | #1847/#1848 fix row: Terra build 4048e0f60, Sol r1 CHANGES, lead fix c04529c7b | Echo auto-replies and duplicate continuation notices stopped; halted-restart and stale-launch review findings fixed with regression tests. Gate touched 123/0, bus 1648/0 (project-files excluded, hangs on base), harness 318/0, webui 679/0 on rerun. Not pushed; Lead pushes. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/fix-1847-1848-gate/.
|
||||
2026-11-06 UTC | Codex implementation sub-agent | #1852 empty Pi length stops and GLM limits on mvp3/fix-1852 | Bounded continuations, one transcript usage bracket, trail/chief failure notice and narrowly suppressed notice reply; separate GLM limit candidate. Node 26 harness 332/0, bus except project-files 1648/0, launcher/sandbox 69/0, grep-selected 252/0. Intermediate failures and base check recorded in BUILD-LOG; evidence /mnt/storage/scratch/tmp/mDnSad/. Local candidate for review; no push or deployment.
|
||||
@@ -0,0 +1,2 @@
|
||||
// Shared by the sender and recipient's reply-cycle check.
|
||||
export const LENGTH_FAILURE_NOTICE_PREFIX = 'Mosaic turn failure: ';
|
||||
@@ -51,7 +51,8 @@ export function modelEventStream({ harness, runDir }) {
|
||||
if (event.type === 'agent_end') {
|
||||
const last = event.messages?.at(-1);
|
||||
if (last?.role === 'assistant') {
|
||||
failed = ['error', 'aborted'].includes(last.stopReason);
|
||||
failed = ['error', 'aborted'].includes(last.stopReason) || (last.stopReason === 'length' &&
|
||||
!(last.content ?? []).some(part => (part.type === 'text' && typeof part.text === 'string' && part.text.trim()) || part.type === 'toolCall'));
|
||||
answer = (last.content ?? []).filter(part => part.type === 'text').map(part => part.text + '\n').join('');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,8 +40,14 @@ export default function (pi) {
|
||||
const verdict = decide(policy, event.toolName, event.input);
|
||||
return verdict.allow ? undefined : { block: true, reason: verdict.reason };
|
||||
});
|
||||
pi.on("agent_end", async () => {
|
||||
writeFileSync(marker, `${new Date().toISOString()}\n`);
|
||||
pi.on("agent_end", async (event) => {
|
||||
const last = event.messages?.at(-1);
|
||||
const content = last?.role === "assistant" ? last.content ?? [] : [];
|
||||
writeFileSync(marker, JSON.stringify({ at: new Date().toISOString(),
|
||||
stopReason: last?.role === "assistant" ? last.stopReason : undefined,
|
||||
hasText: content.some(part => part.type === "text" && typeof part.text === "string" && part.text.trim().length > 0),
|
||||
hasToolCall: content.some(part => part.type === "toolCall"),
|
||||
}) + "\n");
|
||||
});
|
||||
|
||||
for (const t of tools) {
|
||||
|
||||
@@ -40,10 +40,16 @@ import { createServer, connect } from "node:net";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { Client } from "../../bus/src/client.mjs";
|
||||
import { LENGTH_FAILURE_NOTICE_PREFIX } from "./failure-notice.mjs";
|
||||
import { ID_PATTERN } from "../../business/src/vocabulary.mjs";
|
||||
import { firstOutputMarker, firstTurnMarker } from './first-output.mjs';
|
||||
import { GUARD_CODES, FIXED_TEXTS } from "../../library/src/reply-protocol.mjs";
|
||||
import { fetchReview, landWork, pushWorking } from "./git-tools.mjs";
|
||||
|
||||
export const isLengthFailureNotice = m => m.class === "INFO" && typeof m.from_role === "string"
|
||||
&& ID_PATTERN.test(m.from_role) && !["human", "tester", "user"].includes(m.from_role)
|
||||
&& typeof m.body === "string" && m.body.startsWith(LENGTH_FAILURE_NOTICE_PREFIX);
|
||||
|
||||
export const EXIT = Object.freeze({ stopped: 0, founder: 20, claim: 21, session: 22, broker: 23, usage: 2 });
|
||||
|
||||
// Environment variables that carry a founder (human) credential for a
|
||||
@@ -441,6 +447,14 @@ function runTurn({ session, request, n, env, onChild }) {
|
||||
onChild(null);
|
||||
if (markerFailed) return resolve({ ok: false, markerFailed: true, reason: "first-turn marker publication failed" });
|
||||
if (timedOut) return resolve({ ok: false, reason: `turn passed its ${session.turnTimeout} s wall clock` });
|
||||
// The managed event monitor exits 1 for this same length stop. Never mask
|
||||
// a signal, timeout, marker publication failure, or other adapter exit.
|
||||
if (session.turnEnd === "marker" && !signal && (code === 0 || (session.proxyFirstTurn && code === 1))) {
|
||||
let status;
|
||||
try { status = JSON.parse(readFileSync(marker, "utf8")); } catch {}
|
||||
if (status?.stopReason === "length" && status.hasText === false && status.hasToolCall === false)
|
||||
return resolve({ ok: false, emptyLength: true, reason: "output limit reached while thinking before any answer or tool call" });
|
||||
}
|
||||
if (code !== 0) return resolve({ ok: false, reason: signal ? `adapter killed by ${signal}` : `adapter exited ${code}` });
|
||||
// A "marker" harness (row 65: the module's turnEnd()) writes the marker when its turn ends.
|
||||
if (session.turnEnd === "marker" && !existsSync(marker)) return resolve({ ok: false, reason: `${session.harness} exited before the turn ended (no turn marker)` });
|
||||
@@ -598,6 +612,34 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
log("runner: first-turn marker publication failed");
|
||||
return true;
|
||||
};
|
||||
const continuationRequest = "Your last turn hit the output limit while thinking, before any answer or tool call. Now answer or call a tool, and keep your reasoning short.";
|
||||
// State checks the broker's stop switch before another process can start.
|
||||
// The surrounding usage bracket captures every attempt in the transcript.
|
||||
const continueLength = async (initial, counter) => {
|
||||
let turn = initial;
|
||||
while (turn.emptyLength && counter.count < 2 && !stopping) {
|
||||
try { await client.call("state"); }
|
||||
catch (error) { return { ok: false, brokerError: error, reason: "could not read the current state" }; }
|
||||
if (stopping) break;
|
||||
++counter.count;
|
||||
++n;
|
||||
turn = await runTurn({ session, request: continuationRequest, n, env: turnEnv, onChild: (c) => (child = c) });
|
||||
unslop.read();
|
||||
}
|
||||
return turn;
|
||||
};
|
||||
const recordLengthFailure = async m => {
|
||||
const detail = "turn failed: output limit reached while thinking before any answer or tool call after two continuations";
|
||||
log(`runner: message ${m.id} ${detail}`);
|
||||
// The existing routine event is visible to the chief via trail and to the
|
||||
// Console on this message's trail; it does not send an automatic reply.
|
||||
await client.call("event.emit", { kind: "action.allowed", body: { action: "routine", target: detail }, subject: m.id });
|
||||
if (m.from_role === "chief") { log(`runner: chief notice for ${m.id} skipped: chief is the sender`); return; }
|
||||
try {
|
||||
await client.call("message.send", { to: "chief", class: "INFO", in_reply_to: m.id,
|
||||
body: `${LENGTH_FAILURE_NOTICE_PREFIX}Turn for message ${m.id} from ${m.from_role} failed: the output limit was reached while thinking, before any answer or tool call, on the turn and 2 continuations. No answer was sent.` });
|
||||
} catch (e) { log(`runner: chief notice for ${m.id} failed: ${e.code ?? e.message}`); }
|
||||
};
|
||||
for (const e of session.landing ?? []) {
|
||||
if (stopping) break;
|
||||
let block;
|
||||
@@ -655,18 +697,27 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
try { await client.call("usage.begin", { turn: m.id }); }
|
||||
catch (e) { log(`runner: usage baseline unavailable: ${e.code ?? "failed"}`); }
|
||||
}
|
||||
const turn = unread
|
||||
let turn = unread
|
||||
? { ok: false, reason: unread }
|
||||
: await runTurn({ session, request: `${block}\n${turnRequest(m)}`, n, env: turnEnv, onChild: (c) => (child = c) });
|
||||
unslop.read();
|
||||
// A failed/stopped harness can still have used tokens. Missing data remains unknown.
|
||||
const continuationCount = { count: 0 };
|
||||
if (!turn.markerFailed && !stopping) turn = await continueLength(turn, continuationCount);
|
||||
// One transcript delta covers the original attempt and all continuations,
|
||||
// even if the runner is stopped or a state check fails.
|
||||
try { await client.call("usage.end", { turn: m.id }); }
|
||||
catch (e) { log(`runner: usage outcome unavailable: ${e.code ?? "failed"}`); }
|
||||
if (failedMarker(turn)) break loop;
|
||||
if (stopping) break loop;
|
||||
if (turn.brokerError && ended(turn.brokerError)) break loop;
|
||||
if (turn.emptyLength) {
|
||||
try { await recordLengthFailure(m); await client.call("message.read", { id: m.id }); }
|
||||
catch (e) { log(`runner: failure record for ${m.id} failed: ${e.code ?? e.message}`); if (ended(e, false)) break loop; }
|
||||
continue;
|
||||
}
|
||||
if (!turn.ok) log(`runner: turn ${n} for message ${m.id} failed: ${turn.reason}`);
|
||||
try {
|
||||
if (m.class !== "RESULT" && (m.class === "REQUEST" || ["human", "tester", "user"].includes(m.from_role) || m.in_reply_to_class !== "RESULT")) {
|
||||
if (!isLengthFailureNotice(m) && m.class !== "RESULT" && (m.class === "REQUEST" || ["human", "tester", "user"].includes(m.from_role) || m.in_reply_to_class !== "RESULT")) {
|
||||
if (!library) {
|
||||
const body = turn.ok ? turn.text || "(the session gave no answer)" : `The turn failed: ${turn.reason}`;
|
||||
await client.call("message.send", { to: m.from_role, body: clip(body) + unslop.note(), class: "RESULT", in_reply_to: m.id });
|
||||
@@ -707,12 +758,21 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
++n;
|
||||
try { await client.call("usage.begin", { turn: `${m.id}:retry` }); }
|
||||
catch (e) { log(`runner: usage baseline unavailable: ${e.code ?? "failed"}`); }
|
||||
const retry = await runTurn({ session, request: `${block ?? ""}\n${turnRequest(m)}\n\nYour answer was refused with ${error.code}: ${reason}\nReply again in this same session.`, n, env: turnEnv, onChild: (c) => (child = c) });
|
||||
let retry = await runTurn({ session, request: `${block ?? ""}\n${turnRequest(m)}\n\nYour answer was refused with ${error.code}: ${reason}\nReply again in this same session.`, n, env: turnEnv, onChild: (c) => (child = c) });
|
||||
unslop.read();
|
||||
try { await client.call("usage.end", { turn: `${m.id}:retry` }); }
|
||||
catch (e) { log(`runner: usage outcome unavailable: ${e.code ?? "failed"}`); }
|
||||
if (failedMarker(retry)) break loop;
|
||||
if (stopping) break loop;
|
||||
retry = await continueLength(retry, continuationCount);
|
||||
if (failedMarker(retry)) break loop;
|
||||
if (stopping) break loop;
|
||||
if (retry.brokerError && ended(retry.brokerError)) break loop;
|
||||
if (retry.emptyLength) {
|
||||
await recordLengthFailure(m);
|
||||
await client.call("message.read", { id: m.id });
|
||||
continue;
|
||||
}
|
||||
if (!retry.ok) log(`runner: retry turn ${n} for message ${m.id} failed: ${retry.reason}`);
|
||||
try { await send(retry); }
|
||||
catch (again) {
|
||||
|
||||
@@ -142,3 +142,40 @@ test('C2-06 the retry turn delivers citation-unavailable reason to the agent', a
|
||||
const requests = readFileSync(join(f.runDir, 'fixture-requests.jsonl'), 'utf8').trim().split('\n').map(JSON.parse);
|
||||
assert.equal(requests[1].includes(`citation-unavailable: ${reason}`), true);
|
||||
});
|
||||
|
||||
for (const library of [false, true]) {
|
||||
test(`three empty length stops produce no sender reply and one visible failure, library=${library}`, async t => {
|
||||
const f = await setup(t, () => Array(3).fill({marker:{stopReason:'length',hasText:false,hasToolCall:false}}), {library});
|
||||
const sent = await f.call(f.human, 'message.send', {to:'coder',body:'Fixture question.',class:'REQUEST'});
|
||||
await until(() => f.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'", sent.id));
|
||||
assert.equal(f.count(),3);
|
||||
assert.equal(f.store.get("SELECT count(*) n FROM messages WHERE from_role='coder' AND to_role='human'").n,0);
|
||||
const events = f.store.all("SELECT * FROM events WHERE subject=? AND kind='action.allowed'", sent.id);
|
||||
assert.equal(events.length,1); assert.match(events[0].body,/length|output limit/i);
|
||||
assert.match(f.log(),/length|output limit/i);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
for (const library of [false, true]) {
|
||||
test(`normal stop with text and empty text preserves replies, library=${library}`, async t => {
|
||||
const f = await setup(t, () => [
|
||||
{text:'answer',marker:{stopReason:'stop',hasText:true,hasToolCall:false}},
|
||||
{text:'',marker:{stopReason:'stop',hasText:false,hasToolCall:false}},
|
||||
], {library,requireCitation:false});
|
||||
assert.equal((await f.ask()).body,'answer');
|
||||
assert.equal((await f.ask()).body,library ? FIXED_TEXTS.empty : '(the session gave no answer)');
|
||||
assert.equal(f.count(),2);
|
||||
});
|
||||
}
|
||||
|
||||
test('a library guard retry that exhausts empty length continuations sends no fixed RESULT', async t => {
|
||||
const f = await setup(t, () => [
|
||||
{text:'uncited'}, ...Array(3).fill({marker:{stopReason:'length',hasText:false,hasToolCall:false}}),
|
||||
]);
|
||||
const sent = await f.call(f.human,'message.send',{to:'coder',body:'Fixture question.',class:'REQUEST'});
|
||||
await until(() => f.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'",sent.id));
|
||||
assert.equal(f.count(),4);
|
||||
assert.equal(f.store.get("SELECT count(*) n FROM messages WHERE from_role='coder' AND to_role='human'").n,0);
|
||||
assert.equal(f.store.get("SELECT count(*) n FROM events WHERE subject=? AND kind='action.allowed'",sent.id).n,1);
|
||||
});
|
||||
+3
-1
@@ -16,6 +16,8 @@ for (const args of step.tools ?? []) {
|
||||
catch (e) { result = { ok: false, error: e.message }; }
|
||||
appendFileSync(join(root, 'fixture-tools.jsonl'), JSON.stringify(result) + '\n');
|
||||
}
|
||||
if (step.sleep) await new Promise(() => setInterval(() => {}, 1000));
|
||||
if (step.fail) process.exit(3);
|
||||
process.stdout.write(step.text ?? '');
|
||||
writeFileSync(process.env.MOSAIC_TURN_MARKER, 'done\n');
|
||||
writeFileSync(process.env.MOSAIC_TURN_MARKER, step.marker ? JSON.stringify(step.marker) : 'done\n');
|
||||
if (step.exitCode) process.exit(step.exitCode);
|
||||
@@ -275,3 +275,20 @@ test('Claude adapter preserves text-mode bytes for a tool-only final turn', asyn
|
||||
assert.deepEqual(streamed.stdout,text.stdout); assert.deepEqual(streamed.stdout,Buffer.from('\n'));
|
||||
assert.equal(readFileSync(join(f.runDir,'proxy-first-output'),'utf8'),'output\n');
|
||||
});
|
||||
|
||||
test('Pi turn marker records final assistant stop reason and output presence', async t => {
|
||||
const runDir = scratch(t);
|
||||
writeFileSync(join(runDir, 'policy'), JSON.stringify({harness:'pi', tools:[], typed:[]}));
|
||||
writeFileSync(join(runDir, 'tools'), '[]');
|
||||
const values = { MOSAIC_POLICY_FILE:join(runDir,'policy'), MOSAIC_TOOLS_FILE:join(runDir,'tools'), MOSAIC_TOOL_SOCKET:join(runDir,'sock'), MOSAIC_TURN_MARKER:join(runDir,'done') };
|
||||
const prior = Object.fromEntries(Object.keys(values).map(k => [k, process.env[k]]));
|
||||
Object.assign(process.env, values);
|
||||
t.after(() => { for (const [k,v] of Object.entries(prior)) { if (v === undefined) delete process.env[k]; else process.env[k] = v; } });
|
||||
const handlers = new Map();
|
||||
piExtension({on:(name,fn) => handlers.set(name,fn), registerTool(){}});
|
||||
for (const [content, hasText, hasToolCall] of [[[{type:'thinking',thinking:'x'}], false, false], [[{type:'text',text:'answer'}], true, false], [[{type:'toolCall',name:'read'}], false, true]]) {
|
||||
await handlers.get('agent_end')({messages:[{role:'assistant',stopReason:'length',content}]});
|
||||
const marker = JSON.parse(readFileSync(join(runDir,'done'),'utf8'));
|
||||
assert.equal(marker.stopReason, 'length'); assert.equal(marker.hasText, hasText); assert.equal(marker.hasToolCall, hasToolCall);
|
||||
}
|
||||
});
|
||||
@@ -111,3 +111,18 @@ for (const [name, events, marked] of [
|
||||
if (marked) { assert.equal(fs.readFileSync(join(root,'proxy-first-output'),'utf8'), 'output\n'); assert.equal(Buffer.concat(chunks).toString(), 'final answer\n'); }
|
||||
});
|
||||
}
|
||||
|
||||
test('Pi rejects an empty length stop, but accepts text or a tool call', t => {
|
||||
const root = fs.mkdtempSync(join(tmpdir(), 'length-'));
|
||||
t.after(() => fs.rmSync(root, { recursive: true, force: true }));
|
||||
for (const [content, fails] of [
|
||||
[[{type:'thinking', thinking:'private'}, {type:'text', text:' '}], true],
|
||||
[[{type:'text', text:'answer'}], false],
|
||||
[[{type:'toolCall', id:'t', name:'read'}], false],
|
||||
]) {
|
||||
const stream = output.modelEventStream({ harness:'pi', runDir:root });
|
||||
stream.push(JSON.stringify({type:'agent_end',messages:[{role:'assistant',stopReason:'length',content}]}) + '\n');
|
||||
if (fails) assert.throws(() => stream.finish(), /model stream did not complete/);
|
||||
else assert.doesNotThrow(() => stream.finish());
|
||||
}
|
||||
});
|
||||
@@ -1066,3 +1066,98 @@ test("row 66 a refused send keeps the note: it rides the next RESULT the session
|
||||
assert.equal(next.body, 'env: fake-model\n\n[unslop: 1 reply of this session flagged since the last note: banned word "delve" x1. The session gets a correction notice on its next turn.]');
|
||||
assert.equal(ctx.inbox.length, 1);
|
||||
});
|
||||
|
||||
test('empty length continues in the same session, records usage and relays the answer once', async t => {
|
||||
const calls = [];
|
||||
let ctx;
|
||||
ctx = await setup(t, { usage: { collect: async (_session, phase, turn) => { const file = join(ctx.runDir, "fixture-count"); calls.push({phase, turn, attempts: existsSync(file) ? Number(readFileSync(file, "utf8")) : 0}); return {recorded:true}; } } });
|
||||
const fixture = new URL('./fixtures/citation-adapter.mjs', import.meta.url).pathname;
|
||||
writeFileSync(join(ctx.dir, 'adapter.sh'), `exec '${process.execPath}' '${fixture}'\n`);
|
||||
writeFileSync(join(ctx.runDir, 'fixture-steps.json'), JSON.stringify([
|
||||
{marker:{stopReason:'length',hasText:false,hasToolCall:false}}, {text:'recovered',marker:{stopReason:'stop',hasText:true,hasToolCall:false}},
|
||||
]));
|
||||
const r = startRunner(ctx.runDir, ctx.cap); t.after(() => r.child.kill('SIGKILL'));
|
||||
await until(() => ctx.store.get("SELECT 1 FROM role_claims WHERE role='coder' AND op='claim'"));
|
||||
const sent = ctx.call(ctx.pm, 'message.send', {to:'coder',body:'work',class:'REQUEST'});
|
||||
await until(() => ctx.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'", sent.id));
|
||||
const rows = ctx.store.all("SELECT * FROM messages WHERE from_role='coder' AND to_role='pm'");
|
||||
assert.equal(rows.length,1); assert.equal(rows[0].class,'RESULT'); assert.equal(rows[0].body,'recovered');
|
||||
assert.equal(Number(readFileSync(join(ctx.runDir,'fixture-count'),'utf8')),2);
|
||||
const requests = readFileSync(join(ctx.runDir,'fixture-requests.jsonl'),'utf8').trim().split('\n').map(JSON.parse);
|
||||
assert.match(requests[1], /output limit.*thinking/i); assert.match(requests[1], /answer or call a tool/i);
|
||||
assert.deepEqual(calls, [{phase:'begin',turn:sent.id,attempts:0},{phase:'end',turn:sent.id,attempts:2}]);
|
||||
});
|
||||
|
||||
|
||||
test('empty length exhaustion records one usage bracket, one chief notice and one trail event', async t => {
|
||||
const calls = []; let ctx;
|
||||
ctx = await setup(t, { usage: { collect: async (_s, phase, turn) => { const file = join(ctx.runDir, 'fixture-count'); calls.push({phase,turn,attempts:existsSync(file) ? Number(readFileSync(file,'utf8')) : 0}); return {recorded:true}; } } });
|
||||
const fixture = new URL('./fixtures/citation-adapter.mjs', import.meta.url).pathname;
|
||||
writeFileSync(join(ctx.dir,'adapter.sh'), `exec '${process.execPath}' '${fixture}'\n`);
|
||||
writeFileSync(join(ctx.runDir,'fixture-steps.json'), JSON.stringify(Array(3).fill({marker:{stopReason:'length',hasText:false,hasToolCall:false}})));
|
||||
const r = startRunner(ctx.runDir, ctx.cap); t.after(() => r.child.kill('SIGKILL'));
|
||||
await until(() => ctx.store.get("SELECT 1 FROM role_claims WHERE role='coder' AND op='claim'"));
|
||||
const sent = ctx.call(ctx.pm,'message.send',{to:'coder',body:'work',class:'REQUEST'});
|
||||
await until(() => ctx.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'",sent.id));
|
||||
assert.deepEqual(calls,[{phase:'begin',turn:sent.id,attempts:0},{phase:'end',turn:sent.id,attempts:3}]);
|
||||
assert.equal(ctx.store.get("SELECT count(*) n FROM messages WHERE from_role='coder' AND to_role='pm'").n,0);
|
||||
const notice = ctx.store.all("SELECT * FROM messages WHERE from_role='coder' AND to_role='chief'");
|
||||
assert.equal(notice.length,1); assert.equal(notice[0].class,'INFO'); assert.equal(notice[0].in_reply_to,sent.id);
|
||||
assert.match(notice[0].body,/output limit.*thinking/i);
|
||||
assert.equal(ctx.store.get("SELECT count(*) n FROM events WHERE subject=? AND kind='action.allowed'",sent.id).n,1);
|
||||
});
|
||||
|
||||
test('chief does not reply to the failure notice, but replies to ordinary agent and prefixed human INFO', async t => {
|
||||
const { LENGTH_FAILURE_NOTICE_PREFIX } = await import('../src/failure-notice.mjs');
|
||||
const ctx = await setup(t,{as:'chief'});
|
||||
const coder = ctx.broker.bindLaunch({business:'demo',role:'coder',run:'coder-notifier',harness:'pi'});
|
||||
ctx.call(coder,'role.claim');
|
||||
const r = startRunner(ctx.runDir,ctx.cap); t.after(() => r.child.kill('SIGKILL'));
|
||||
await until(() => ctx.store.get("SELECT 1 FROM role_claims WHERE role='chief' AND op='claim'"));
|
||||
const seed = ctx.call(ctx.pm,'message.send',{to:'coder',body:'work',class:'REQUEST'});
|
||||
const notice = ctx.call(coder,'message.send',{to:'chief',body:LENGTH_FAILURE_NOTICE_PREFIX+'fixture',class:'INFO',in_reply_to:seed.id});
|
||||
const ordinary = ctx.call(coder,'message.send',{to:'chief',body:'ordinary information',class:'INFO',in_reply_to:seed.id});
|
||||
const human = ctx.call(ctx.human,'message.send',{to:'chief',body:LENGTH_FAILURE_NOTICE_PREFIX+'human information',class:'INFO',in_reply_to:seed.id});
|
||||
await until(() => ctx.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'",human.id));
|
||||
assert.equal(ctx.store.get("SELECT count(*) n FROM messages WHERE from_role='chief' AND in_reply_to=?",notice.id).n,0);
|
||||
for (const id of [ordinary.id,human.id]) assert.equal(ctx.store.get("SELECT count(*) n FROM messages WHERE from_role='chief' AND class='RESULT' AND in_reply_to=?",id).n,1);
|
||||
});
|
||||
|
||||
|
||||
test('stop interrupts a length continuation and still closes usage', async t => {
|
||||
const calls = []; let ctx;
|
||||
ctx = await setup(t,{usage:{collect:async (_s,phase,turn) => { calls.push({phase,turn}); return {recorded:true}; }}});
|
||||
const fixture = new URL('./fixtures/citation-adapter.mjs',import.meta.url).pathname;
|
||||
writeFileSync(join(ctx.dir,'adapter.sh'),`exec '${process.execPath}' '${fixture}'\n`);
|
||||
writeFileSync(join(ctx.runDir,'fixture-steps.json'),JSON.stringify([{marker:{stopReason:'length',hasText:false,hasToolCall:false}},{sleep:true}]));
|
||||
const r = startRunner(ctx.runDir,ctx.cap); t.after(() => r.child.kill('SIGKILL'));
|
||||
await until(() => ctx.store.get("SELECT 1 FROM role_claims WHERE role='coder' AND op='claim'"));
|
||||
const sent = ctx.call(ctx.pm,'message.send',{to:'coder',body:'work',class:'REQUEST'});
|
||||
await until(() => existsSync(join(ctx.runDir,'fixture-count')) && readFileSync(join(ctx.runDir,'fixture-count'),'utf8') === '2');
|
||||
r.child.kill('SIGTERM'); assert.equal((await r.exited).code,EXIT.stopped);
|
||||
assert.deepEqual(calls,[{phase:'begin',turn:sent.id},{phase:'end',turn:sent.id}]);
|
||||
assert.equal(ctx.store.get("SELECT count(*) n FROM messages WHERE from_role='coder'").n,0);
|
||||
});
|
||||
|
||||
test('managed event monitor exit 1 with empty length marker continues', async t => {
|
||||
const ctx = await setup(t);
|
||||
const fixture = new URL('./fixtures/citation-adapter.mjs',import.meta.url).pathname;
|
||||
writeFileSync(join(ctx.dir,'adapter.sh'),`exec '${process.execPath}' '${fixture}'\n`);
|
||||
writeFileSync(join(ctx.runDir,'proxy-first-turn-enabled'),'enabled\n');
|
||||
writeFileSync(join(ctx.runDir,'fixture-steps.json'),JSON.stringify([{marker:{stopReason:'length',hasText:false,hasToolCall:false},exitCode:1},{text:'recovered'}]));
|
||||
const r = startRunner(ctx.runDir,ctx.cap); t.after(() => r.child.kill('SIGKILL'));
|
||||
await until(() => ctx.store.get("SELECT 1 FROM role_claims WHERE role='coder' AND op='claim'"));
|
||||
const sent = ctx.call(ctx.pm,'message.send',{to:'coder',body:'work',class:'REQUEST'});
|
||||
const reply = await until(() => ctx.store.get("SELECT * FROM messages WHERE in_reply_to=? AND class='RESULT'",sent.id));
|
||||
assert.equal(reply.body,'recovered'); assert.equal(readFileSync(join(ctx.runDir,'fixture-count'),'utf8'),'2');
|
||||
});
|
||||
|
||||
test('failure notice recognition excludes bus notices and non-agent senders', async () => {
|
||||
const { LENGTH_FAILURE_NOTICE_PREFIX } = await import('../src/failure-notice.mjs');
|
||||
const { isLengthFailureNotice } = await import('../src/runner.mjs');
|
||||
const notice = {class:'INFO',from_role:'coder',body:LENGTH_FAILURE_NOTICE_PREFIX+'fixture'};
|
||||
assert.equal(isLengthFailureNotice(notice),true);
|
||||
for (const from_role of ['human','tester','user','_bus']) assert.equal(isLengthFailureNotice({...notice,from_role}),false,from_role);
|
||||
assert.equal(isLengthFailureNotice({...notice,class:'REQUEST'}),false);
|
||||
assert.equal(isLengthFailureNotice({...notice,body:'ordinary information'}),false);
|
||||
});
|
||||
@@ -49,6 +49,7 @@ const KILL_WITH_PARENT = ["/usr/bin/setpriv", "--pdeathsig", "KILL", "--"];
|
||||
// is missing here.
|
||||
export const RUNNER_FILES = Object.freeze([
|
||||
"packages/harness/src/runner.mjs",
|
||||
"packages/harness/src/failure-notice.mjs",
|
||||
// Row 331: fixed output-exists marker; writes only within the existing run directory.
|
||||
"packages/harness/src/first-output.mjs",
|
||||
"packages/harness/src/gate.mjs",
|
||||
|
||||
Reference in new issue
Block a user