Files
stack/agents/darkwing/work/queue-111/r1/build.md
T

13 KiB

Row 111 (#1605) round 1: bus host start, the steps after the broker's reply and the child error event

Brief: docs/plans/2026-10-13_landing-follow-ups.md, section "Bus host start: release after the broker's reply, and the child error event" (blob 19c939e10 at 02fe2d9d5). BASE: 02fe2d9d55. Candidate: build.patch against BASE. candidate-manifest.sha256 lists the 2 files, and applying the patch to a fresh worktree at BASE reproduces both digests. No commit after BASE touches either path. Per the brief, packages/cli/README.md is unchanged; row 110 owns it next.

What changed

All in packages/cli/src/host.mjs, startHost and its firstReply helper.

  • One failure path after the broker's reply. The reader lookup, the notifier's fork, its start send, the wait for its reply, a refusal, rmSync of an old host.json and writeHostState now all run in one try. Its catch sends the notifier SIGTERM if one was forked and waits up to 5 s for it, sends the broker close if the channel is still up, waits up to CLOSE_TIMEOUT_MS for the broker, calls release() and rethrows. That is the path the notifier's no-reply case already took. At BASE, a throw from the notifier fork, the start send, the reader lookup or writeHostState left host.lock and a live broker.
  • A missing reader. When the broker's reply has no reader for business, the start refuses with broker has no reader for business <id>, exit 3. At BASE that was a TypeError on reader.cap. The check runs with and without a notifier. Without one, BASE started a host anyway; mosaic bus start always boots the broker for the business it names, so the only change in practice is a refusal where BASE threw a TypeError.
  • The notifier's wait starts after its send, as the broker's has since row 106. At BASE the wait started first, so a send that threw would have left a wait whose later rejection nobody handled.
  • A notifier that refuses now gets SIGTERM before the wait, like every other failure. It has already called stop(3) or is about to, stop runs once, and SIGTERM to a child that has exited does nothing, so the refusal and its exit code are unchanged.
  • The child error event. Node emits error on a child when a send with no callback fails, or a kill fails with anything but ESRCH. With no listener, the emitter throws, the process ends, and host.lock and host.json stay. startHost now puts a listener on each child right after its fork:
    • During the start, firstReply also rejects on error, with <broker|notifier> failed before it replied: <message>, exit 1, so the start fails at once instead of at the reply timeout. An error at any other time during the start is kept, and the shared try throws it as <name> failed: <message> just before host.json is written. A broker error while the notifier starts is the case that reaches this check. An error during either failure path is kept and dropped, since the start is failing already.
    • Once the host is up, an error on the broker breaks the channel through breakChannel("broker error: <message>"). That rejects the request waiting on the broker, if any, logs and calls close(1). An error on the notifier logs notifier error (<message>); stopping the host and calls close(1), as a notifier that dies does. Either way done resolves with 1 and close() removes host.json and host.lock.

An error after a successful start

The brief asks what an error after a successful start does and why. At BASE it ended the process with host.lock and host.json left behind. request() sends to the broker with no callback, so a failed send there emits error with nothing listening. The lock was left held, so the brief puts the change in scope. I chose to stop the host with exit 1. The unit restarts it, and that is what a child's death already does (watchChildren). For the broker the error goes through breakChannel: a request whose message never reached the broker would otherwise wait for its timeout and then break the channel anyway. A kill that fails on a running host means a child we can't signal, and stopping the host is the fail-closed answer there too.

The up flag switches the listener from the start behaviour to the running one. It is set in the same synchronous run as host.json's write, after breakChannel, stopping and close exist, so no event can reach the listener in between.

Tests

packages/cli/tests/host.test.mjs: 31 at BASE, 41 in the candidate. The 10 new ones:

Test What it checks
a notifier fork that throws stops the broker ... (loop, fork and send) the throw's own error; the broker has ended; with send, the notifier has ended too; no host.lock, host.json or writer.lock
a boot reply with no reader for the business refuses with exit 3 naming it, ... (loop, with and without a notifier) exit 3 and the message; no notifier forked; the broker has ended; no host.lock, host.json or writer.lock
a host.json that can't be written stops both children and leaves no host.lock EEXIST from writeHostState; both children ended; no host.lock, host.json or writer.lock
`an error event on the <broker notifier> before it replies fails the start at once, ...` (loop)
an error event on the broker while the notifier starts fails the start once the notifier replies broker failed: write EPIPE; both children ended; no host.lock or host.json
`an error event on the <broker notifier> once the host is up stops the host with exit 1, ...` (loop)

How the tests make each failure:

  • Fork and send throws use throwOn. It now takes nth, the fork it targets, and leaves other children alone; the row 106 broker tests pass the default 0. A fork that throws returns no child, so startHost can't stop that notifier; throwOn kills it after the test, and the fork test checks only the broker.
  • The no-reader case starts a host for business other on a boot config for acme.
  • writeHostState writes through host.json.<pid>.tmp with the wx flag. The test makes that file first, so the write throws EEXIST in the test's own process.
  • The error before a reply uses failSends: the boot or start send fails with an EPIPE error, delivered as an error event on the next tick because the send has no callback. The child is left running and never gets the message, so only the error can end the wait in time.
  • The kept broker error and the errors once up are emitted on the child directly, on the next tick in the first case.
  • forksSeen records every child forked during the test, so a test can check each one has ended. It kills them all after the test.

Mutants

mutants.py and gate.sh are the scripts I ran, copied here; they point at my scratch worktree, so change WT/G to rerun them. mutants.py runs each mutant against the named tests only, through --test-name-pattern, and counts a pattern that matches no test as a fault. n00 puts BASE's host.mjs, read from the worktree's HEAD, under the new tests.

Final run 23:32:52 to 23:33:46Z on 2026-10-13. Each file is restored from its .orig, and the candidate's diff hashed 2ef465444b6936ca (first 16 hex) before and after. Every run matched the tests it names: 1 each, 2 for the two loops that share a prefix.

Id Mutant Tests Result and how it fails
n00 BASE's host.mjs under the new tests all 9 patterns killed, all 9: a child still running, Missing expected rejection, or the reader.cap TypeError
n01 firstReply doesn't listen for error error before reply, broker and notifier killed: both hit the 8 s test timeout
n02 no error listener on the broker kept broker error; broker error once up killed: the error throws with nothing listening
n03 no error listener on the notifier notifier error once up killed: the error throws with nothing listening
n04 a broker error once up is dropped broker error once up killed: 15 s test timeout, the host never stops
n05 a notifier error once up only logs notifier error once up killed: 15 s test timeout, the host never stops
n06 no reader check no reader, both killed: Missing expected rejection without a notifier, the reader.cap TypeError with one
n07 the notifier fork runs before the shared try notifier fork throws killed: the broker has not ended
n08 the notifier's wait starts before its send notifier send throws killed: the abandoned wait rejects with notifier exited (null) before it replied and nothing handles it
n09 host.json is written after the shared try host.json write killed: the children have not ended
n10 a kept child error is not thrown kept broker error killed: Missing expected rejection
n11 the failure path leaves the notifier running host.json write; notifier send throws killed: the notifier has not ended
n12 the failure path doesn't wait for the broker no reader, both; host.json write killed: the broker has not ended
n13 the failure path keeps the lock no reader, both; host.json write; notifier fork throws killed: host.lock is still there

Two earlier runs did not count, and their logs are here as mutants-run-hung.log and mutants-run-count.log.

  • The first hung on n00. BASE leaves the broker and notifier running after a failed start, and their IPC channels kept the test file from exiting, so the run sat at mutants.py's 600 s timeout. I stopped it; mutants.py put the candidate's host.mjs back. forksSeen now kills every child it saw after the test, as throwOn and failSends already did, so a start that leaves children running fails the test instead of hanging it. The host tests reran at 41 passed, 0 failed.
  • The second reported n01, n04 and n05 as faults with no tests run. Each of those tests had hit its timeout, which node counts as cancelled, not failed, and mutants.py counted only passes and fails. It now counts cancelled tests too. That run had every other mutant killed, and n00 killed on all 9 patterns.

Gate

gate.sh, on BASE 02fe2d9d5 with the final candidate, 23:33:50 to 23:39:29Z on 2026-10-13, DOCKER_HOST unset, Docker 29.7.2.

Step Result Time (UTC)
pkg-bus 128 passed, 0 failed 23:33:50-23:33:53
pkg-business 67 passed, 0 failed 23:33:53-23:33:55
pkg-cli 178 passed, 0 failed 23:33:55-23:34:16
pkg-control-board 124 passed, 0 failed 23:34:16-23:34:16
pkg-conversation 182 passed, 0 failed 23:34:16-23:35:05
pkg-discord 178 passed, 0 failed 23:35:05-23:35:08
pkg-harness 73 passed, 0 failed 23:35:08-23:35:18
pkg-ledger 78 passed, 0 failed 23:35:18-23:35:38
pkg-mosaic 69 passed, 0 failed 23:35:38-23:35:40
pkg-queue 156 passed, 0 failed 23:35:40-23:36:11
pkg-runs 43 passed, 0 failed 23:36:11-23:36:12
pkg-seat 119 passed, 0 failed 23:36:12-23:36:17
pkg-tasks 92 passed, 0 failed 23:36:17-23:36:22
pkg-webui 108 passed, 0 failed 23:36:22-23:37:18
script-auth 15 passed, 0 failed 23:37:18-23:37:19
script-conductor 17 passed, 0 failed 23:37:19-23:37:25
script-config 24 passed, 0 failed 23:37:25-23:37:26
script-discord 66 passed, 0 failed 23:37:26-23:37:34
script-extension-package 18 passed, 0 failed 23:37:34-23:37:35
script-foundation 44 passed, 0 failed 23:37:35-23:38:15
script-queue 28 passed, 0 failed (node 156 passed) 23:38:15-23:38:48
script-release 14 passed, 0 failed 23:38:48-23:38:58
script-task 98 passed, 0 failed 23:38:58-23:39:29

The gate printed GATE_PASSED at 23:39:29Z. The Docker steps left a wt_default network, created at 23:38:48Z with 0 containers; I removed it. An earlier gate run on the candidate before the forksSeen change also passed (23:13:16 to 23:19:01Z, gate-run-r0.log); its network, created at 23:18:19Z with 0 containers, is removed too.

Choices and limits

  • The broker's own refusal (ready.ok not true) keeps its BASE path. It runs before the shared try, and the broker exits on its own after refusing.
  • The shared catch does not remove host.json. writeHostState is the last step in the try, so a host.json this start wrote is never followed by a throw. The rmSync before it removes a stale one only, as at BASE.
  • If renameSync in writeHostState throws, host.json.<pid>.tmp stays. The start still fails and gives the lock back. writeHostState is outside this row's scope, and the name is per pid.
  • The third-racer window, stopHost, hostStatus, host.json's format and the broker are untouched, as the brief says.

Going live (decision 109)

The host change is live only with a new copy, a re-pin and a bus restart, all under Mos's go. Until then the live bus, Console and CLI run host.mjs from the pinned copy at 465bbdfbb. The tests change nothing that runs.