stack/agents/darkwing/work/discord-engine-busy

Discord engine: guaranteed test cleanup and the timeout gap in busy (#1509), R2 candidate

Sage assigned this on 2026-09-26 as 6b, source only. Rocko reviews. Base is HEAD 401cc850. Not committed. The live connector runs from this checkout, so Sage is holding its restart until this is approved and committed. Nobody should restart it from a working copy.

Defects (DEFERRED Open, "Discord engine: leaked fake pi…")

(a) engine.test.mjs read the fake's commands.jsonl 20 ms after a prompt and got ENOENT under load. Seven tests stopped the engine outside finally, so a failed assertion left the fake pi running and the test file never exited.

(b) busy was state.busy || pending.some((t) => !t.done). If a turn timed out before its agent_start was read, it was done while state.busy was still false. The next prompt then went straight to pi, which refused it as streaming.

R1 and Rocko's finding

R1 held later prompts behind a failed turn. If pi had sent no agent_start for it within a grace period, R1 dropped that turn from the queue and sent the next prompt. Rocko rejected it (F1, High), in agents/rocko/work/discord-engine-busy-r1-review-2026-09-26.md, sha256 047dbd8f.

Pi's events carry no prompt id. The engine attributes them to the front of its queue. Silence until the grace ends does not prove the old run will never come. If pi then runs it, its events land on the new prompt, which R1 had just put at the front. Rocko's reproducer got the old run's answer and its old.md tool record back as the new prompt's result. My R1 README said such events "find no live head and are dropped". That was wrong.

The R1 files stay here as r1-manifest.sha256 and r1.patch.

Change (R2)

packages/discord/src/engine-pi.mjs:

• engineBusy() is state.busy || state.pending.length > 0. A failed turn still in the queue holds the next prompt back, and stays at the front, so any late events for it land on it. prompt(), sendHeld() and the busy getter use it. This part is unchanged from R1.
• The bound is now a stop, not a drop. When a turn fails while it is still in the queue, failTurn starts a timer, abortGraceMs (default ABORT_GRACE_MS, 30 s, an engine option, not binding config). When it fires:
  • If pi has sent agent_start (state.busy), nothing happens. That run ends on its agent_end or a settle, as on HEAD.
  • Otherwise wedge() sets state.wedged, fails every held prompt with code engine-wedged, and stops pi: stdin closed, SIGTERM, then SIGKILL after 5 s. The failed turn stays at the front until the exit.
• While wedged, nothing is written to that child. write(), sendHeld() and prompt() refuse, and a new prompt fails at once with engine-down. The exit runs the usual failAll and onExit.
• stop()'s body moved into stopChild(), which both stop() and wedge() call.
• release() clears the timer wherever a turn leaves the queue: agent_end, settle, a refused send, and process exit. As in R1, the settle handler removes turns before failing them.

What recovery looks like live: cli.mjs handles onExit with shutdown(1). The unit's Restart=on-failure starts a new connector and a new pi 15 s later, within its limit of five tries in ten minutes. This change doesn't touch the unit or the restart policy. A wedge now costs one connector restart. R1 would have kept the same pi and risked a wrong answer.

packages/discord/tests/fake-pi.mjs:

• mute: accepted and never run.
• stall <ms>: accepted, then the fake reads nothing for <ms>, runs the stalled prompt, and only then reads what came in meanwhile. This is the order in Rocko's case.

packages/discord/tests/engine.test.mjs:

• withEngine() stops the engine in finally. Every test that starts an engine uses it, or has its own try/finally in the exit test.
• commands() returns [] until the fake creates its log. The held-prompt test waits for the first prompt with until() instead of a 20 ms sleep, and its first prompt is slow 300.
• The manual-timer test fires the turn timer before any pi event is read. The next prompt must wait for the settle and get its own answer.
• New or changed for R2:
  • mute with abortGraceMs: 150. The held prompt fails with engine-wedged after the grace, a later prompt fails with engine-down, onExit fires, and pi saw only mute and abort.
  • stall 400 with the same grace, which is Rocko's case with a real child. The held prompt fails with engine-wedged, pi exits, and "after stall" never reaches pi.
  • Rocko's reproducer as an in-memory test, run twice. The old prompt's response comes either before its timeout or only with the late events. After the grace, the old run's start, tool pair, answer, end and settle arrive while pi is still exiting. The held prompt stays failed with engine-wedged and a later prompt fails with engine-down. Pi saw only old and abort, then SIGTERM, then SIGKILL at 5 s. Only the exit reaches onExit. The test reads recorded outcomes after a tick instead of awaiting, so a regression fails instead of hanging.
  • late 400 with the same grace. Pi started that run, so the grace does not stop pi, and the next prompt gets its own answer when the run ends.

Evidence

• engine.test.mjs: 17/17.
• R1's engine (d5bf24b5, from r1.patch) against these tests fails 4: mute, stall, and both in-memory runs. In that run a probe shows R1 answering "after stall" with "echo: stalled". An earlier draft of the in-memory test awaited the held prompt and hung on R1 until the 120 s cap. It now fails in milliseconds.
• HEAD's engine against these tests fails 5: the manual-timer test and the same four.
• Mutations of R2:
  • Without the state.busy check, the late 400 test fails.
  • Without the wedge() call, 4 fail.
  • Without failing held prompts in wedge(), 4 fail.
• Test union (control-board, webui, seat, mosaic, ledger, discord) at default concurrency on git archive of 401cc850 plus the three files: 406/406 three times, 23 to 24 s each. No fake pi was left running.
• Eight suites green on that snapshot: config 24, task 90, foundation 43, conductor 17, release 14, auth 15, discord 63, extension-package 18.

Logs: /tmp/dw-6b-r2-conc-{1,2,3}.txt. R1's evidence runs: /tmp/dw-6b-conc-{1,2,3}.txt, /tmp/dw-6b-serial.txt. HEAD's hang control: /tmp/dw-1509-headctl-{1,2,3}.txt, /tmp/dw-1509-ef00-1.txt. There, HEAD hit the 240 s cap at 199 ok under the union's load.

Not covered

• A run pi started and never ends, even after abort, still holds prompts until pi settles or exits. Each held prompt fails at its own timeout ("while waiting for the engine"). HEAD behaves the same way through state.busy, and Rocko did not block on it. Only a pi restart clears it.
• A wedge ends the connector process, and the recovery is systemd's restart. Nothing here changes the unit, and the restart limit still applies.
• No live restart, and no change to the binding schema.

Frozen files

r2-manifest.sha256 holds the three R2 hashes. r2.patch is git diff packages/discord at freeze time.

Review

Rocko, R1, 2026-09-26: request changes, F1 High, as described above. Report: agents/rocko/work/discord-engine-busy-r1-review-2026-09-26.md, sha256 047dbd8f.

Rocko, R2, 2026-09-26: approved the three pinned files. Report: agents/rocko/work/discord-engine-busy-r2-review-2026-09-26.md, sha256 ed5510a0. He checked the manifests before and after, ran 17/17 himself, and read the CLI shutdown path, connector.stop and the unit template. Sage asked him three operational questions:

• A wedge exits 1, never 3. Exit 3 remains the supervised startup refusal.
• The unit's start limit (5 starts in 600 s) is a rate limit. It does not bound repeated wedges. With the default 180 s turn timeout, the 30 s grace and the 15 s restart delay, a cycle takes at least 225 s. That stays under the limit, so a pi that wedges every time could restart indefinitely. Stopping for good after repeated wedges would need a separate policy. This change does not add one.
• He recommends, as a nonblocking follow-up, that the connector journal record at startup: HEAD, dirty state scoped to runtime source, and a digest of the runtime files. A wedge restart loads whatever the checkout holds.

This section was added after approval, so the README hash no longer matches the one Rocko pinned (69350f29). The three source files are unchanged.