stack/agents/darkwing/work/queue-103/r1/build.md

Row 103 (#1597) round 1: git kill reach, close bound and the last refusal tests

Brief: docs/plans/2026-10-13_post-retest-follow-ups.md, section "Row 100 follow-up: git kill reach, close bound and the last refusal tests" (commit ec6e39881, blob 7741f014). BASE: 79e126be04 (canonical HEAD when the gate ran; 0d71b9ddd..79e126be0 touches only docs/SESSIONS.md, none of the manifest paths). The row was briefed at 0d71b9ddd. Candidate: build.patch against BASE; candidate-manifest.sha256 lists the files.

What changed

• packages/seat/src/checkout.mjs. Every host git now goes through one exported runGit(argv, env, limits). It spawns git detached, so git leads its own process group. Past limits.timeoutMs (GIT_TIMEOUT_MS, 100 s) the group gets SIGTERM, and SIGKILL limits.graceMs (GIT_GRACE_MS, 5 s) later. Once a git that was signalled has closed, the group gets SIGKILL again, for a child that ignored SIGTERM and let go of git's output. A git that ends on its own gets no signal. Clone timeouts used to be a bare SIGKILL to git alone. ensureCheckouts takes limits for the tests.
• packages/seat/src/working.mjs. Its own SIGTERM/SIGKILL code is gone; push and review run git through runGit, so they get the group kill too. Comments updated.
• packages/harness/src/git-tools.mjs. The runner's git() uses the same sequence (TIMEOUT_MS 300 s, GRACE_MS 5 s), in its own copy so the runner doesn't load the host's module. It used to be a bare SIGKILL at 300 s. git is exported for the tests, which shorten limits. The have comment now states the order is per clone (see "Choices").
• packages/cli/src/git-socket.mjs.
  • close() waits at most CLOSE_WAIT_MS (2000 ms) for its host-closing answers to leave, then destroys those sockets. The wait lives in an exported letGo(sockets, ms).
  • send()'s comment states the EPIPE assumption: a reader that leaves while bytes are still queued in this process fails the write that holds them (EPIPE or ECONNRESET), which sets socket.errored. Bytes the kernel already took count as sent; the reviewer's own size check catches a short bundle.
  • readBody takes the idle limit as a parameter, and createGitSocket has a bodyIdle option for the tests. The default is still BODY_IDLE_MS (30 s).
• packages/cli/src/host.mjs (startHost's check only). The read-then-write check is now a claim on a new file, bus-host/host.lock, in claimHostLock(). The brief asks for "exclusive create or a lock"; this is both: the state goes to a file of its own (wx), then linkSync puts it in as host.lock. A link fails if host.lock exists, so of two hosts racing, one takes it, and the lock is never half written. On EEXIST:
  • a live holder refuses a bus host already runs for <business> (pid <pid>), exit 3, as before;
  • a lock whose holder no longer runs is taken over whatever its age. There is no age threshold. It is renamed aside, compared with the file that was read (pid, start time, startedAt), and removed; if it differs, another host's claim was moved aside in the race, so it is linked back;
  • after 3 tries the start refuses <file> kept changing; no bus host started, exit 3. Before the claim, startHost still refuses when host.json names a live host, for a host started before host.lock existed (the live bus today). The claim comes before the broker fork. Each of the 4 boot-failure throws, and close(), first removes host.lock if it still holds this host's state (release()). host.json is written where it was, once the broker and notifier are up, with the same state. readHostState now calls a readState(file) helper, which the claim also uses. stopHost and hostStatus didn't change; they read host.json as before.
• roles/reviewer.md: one line pointing to the nothing-to-fetch text in packages/cli/README.md.
• packages/seat/README.md and packages/cli/README.md: the group kill and its timings, host.lock in the bus start steps (a new step 3; host.json stays at its old place, now step 6) and in the stop paragraph, and the 2 s close cap. working.mjs's lock comment names host.lock.

Tests

Brief item	Test
checkout SIGTERM, grace, SIGKILL	checkout.test: a clone past its time gets SIGTERM; one that ignores SIGTERM gets SIGKILL after the grace period
checkout group kill	checkout.test: a sleep git started dies with the timeout; a child that ignores SIGTERM dies at git's close, well inside the grace period
git-tools SIGTERM, grace, SIGKILL, group	git-tools.test: SIGTERM reaches the runner git's group and SIGKILL follows after the grace; once the signalled git closes, the group gets SIGKILL
close() cap	git-socket.test: letGo against a stand-in socket that never finishes and one that does, cap 300 ms
send()'s post-loop errored arm	git-socket.test: a FIFO as the file, so the test holds the loop past its last chunk; a close there is served, and a failed write there is fetch-aborted
idle refusal	git-socket.test: a bundle with no bytes for bodyIdle (300 ms) refuses body-timeout long before its 60 s deadline
have order over HAVE_MAX	git-tools.test: working refs newest first within each clone, one clone after another
two hosts race (Sage)	host.test: two bus start acme at once; one runs, the other exits 3 naming the winner's pid; while the winner runs bus-host holds host.json and host.lock and nothing else; after bus stop it holds neither
SIGKILL leaves a stale lock (Sage)	host.test: bus start, SIGKILL once it's up, host.lock still names it; once the orphaned broker lets go of writer.lock, a second bus start comes up and the lock and host.json name it
the four boot failures (Sage)	host.test: broker misses its boot wait (bootTimeoutMs: 1, new test); broker refuses (the bus start writer.lock test); notifier refuses; notifier dies before it replies. Each leaves no host.lock and no host.json
a reader during boot (Sage)	host.test: a live host.lock and no host.json; mosaic inbox refuses "no bus host runs here, so name the business with --business", bus stop says "no host runs", readHostState is null and hostStatus's host is null
an old live lock stays (Sage)	host.test: a lock naming this live process, mtime 2000-01-01; claimHostLock refuses naming it, and the file's content and mtime are unchanged
takeover and put-back	host.test: claimHostLock over a dead host's lock, then with a live claim written between the read and the move
a host that holds only host.json	host.test: a live host.json and no lock; startHost refuses naming it and leaves no host.lock

Mutants

32 mutants, mutants.sh in the packet (DRY=1 checks each matches exactly one site). Each ran its named test files in its own process group, killed after 240 s. 31 killed, 1 survived: close-skips-letgo, explained under "Choices and limits". A "fail 0" kill is a run that hit its 240 s limit because a test waited for a SIGKILL the mutant took away. The mutants ran once, on 2b0867bee; the source files are the same on 79e126be0.

co-sigkill-first: killed (ℹ fail 4, ✖ a clone past its time gets SIGTERM and refuses the launch, with nothing left in the work)
co-no-escalation: killed (ℹ fail 0, ✖ a clone that ignores SIGTERM gets SIGKILL once the grace period is over (20100.974295ms))
co-not-detached: killed (ℹ fail 1, ✖ a clone past its time gets SIGTERM and refuses the launch, with nothing left in the work)
co-pid-not-group: killed (ℹ fail 2, ✖ a clone that ignores SIGTERM gets SIGKILL once the grace period is over (20101.011655ms))
co-no-sweep: killed (ℹ fail 1, ✖ once a signalled git has closed, its group gets SIGKILL, so a process that ignored SIGTE)
gt-sigkill-first: killed (ℹ fail 2, ✖ a runner git past its time: SIGTERM reaches its process group, and SIGKILL follows after)
gt-no-escalation: killed (ℹ fail 0, ✖ a runner git past its time: SIGTERM reaches its process group, and SIGKILL follows after)
gt-not-detached: killed (ℹ fail 1, ✖ a runner git past its time: SIGTERM reaches its process group, and SIGKILL follows after)
gt-pid-not-group: killed (ℹ fail 2, ✖ a runner git past its time: SIGTERM reaches its process group, and SIGKILL follows after)
gt-no-sweep: killed (ℹ fail 1, ✖ once a signalled runner git has closed, its group gets SIGKILL, so a process that ignore)
have-origin-first: killed (ℹ fail 1, ✖ fetch_review's have lists the working refs before origin's, and no more than HAVE_MAX (1)
have-oldest-first: killed (ℹ fail 1, ✖ fetch_review's have orders the working refs newest first within each clone, one clone af)
have-uncapped: killed (ℹ fail 2, ✖ fetch_review's have lists the working refs before origin's, and no more than HAVE_MAX (1)
letgo-no-cap: killed (ℹ fail 1, ✖ letGo waits for an answer to leave for at most its cap, then destroys the socket (3001.2)
letgo-no-destroy: killed (ℹ fail 1, ✖ letGo waits for an answer to leave for at most its cap, then destroys the socket (304.15)
close-skips-letgo: survived
postloop-always-served: killed (ℹ fail 1, ✖ send: a close once the loop is past its last chunk is served, and a failed write there i)
postloop-always-aborted: killed (ℹ fail 2, ✖ a reviewer that hangs up the moment it has the whole bundle was served, not aborted (570)
errored-ignored: killed (ℹ fail 2, ✖ send: a close after every byte left is served; one that fails a queued write is fetch-ab)
no-idle: killed (ℹ fail 1, ✖ a bundle with no bytes for bodyIdle refuses body-timeout long before its deadline (5215.)
claim-rename: killed (ℹ fail 3, ✖ two bus starts racing for one data root leave one host running; the other refuses that a)
claim-no-live-check: killed (ℹ fail 3, ✖ two bus starts racing for one data root leave one host running; the other refuses that a)
claim-dead-blocks: killed (ℹ fail 2, ✖ a host takes over a host.lock its dead host left, and puts back a live claim that replac)
claim-no-identity-check: killed (ℹ fail 1, ✖ a host takes over a host.lock its dead host left, and puts back a live claim that replac)
claim-no-release: killed (ℹ fail 6, ✖ a notifier that dies takes the host down with exit 1, so the unit restarts the pair (167)
release-boot-timeout: killed (ℹ fail 1, ✖ a broker that misses its boot wait leaves no host.lock and no host.json (28.121309ms))
release-broker-refused: killed (ℹ fail 1, ✖ bus start refuses with exit 3 and the code when the broker refuses to boot; bus status n)
release-notifier-dies: killed (ℹ fail 1, ✖ a notifier that dies before it replies, after the broker died, still refuses, without a )
release-notifier-refused: killed (ℹ fail 1, ✖ a notifier that refuses stops the broker and the host refuses with exit 3 (152.454618ms))
release-on-close: killed (ℹ fail 2, ✖ a notifier that dies takes the host down with exit 1, so the unit restarts the pair (167)
lock-is-host-json: killed (ℹ fail 4, ✖ two bus starts racing for one data root leave one host running; the other refuses that a)
no-host-json-check: killed (ℹ fail 1, ✖ a host started before host.lock existed, holding only host.json, still refuses a second )

lock-is-host-json (the lock written as host.json) fails the reader-during-boot test among its four, so that test pins the boot window, not just the race.

Choices and limits

• have order. The brief offered a sort across clones or a comment. I chose the comment: per clone, each clone's refs newest first, one clone after another in directory order. Past HAVE_MAX a later clone's newer refs can lose to an earlier clone's older ones. A have is a hint for a smaller bundle, so the cost of a worse pick is bytes, not a wrong result.

• Ctrl-C. A detached git is no longer in the terminal's foreground group, so a Ctrl-C at the terminal of a host run by hand doesn't reach it. A systemd stop still does, since it kills the unit's cgroup. On the runner, the sandbox runs with its own pid namespace (--unshare-pid --as-pid-1), so its end still takes every group in it.

• The close cap's test uses a stand-in Duplex. On a real unix socket a peer that never reads can't hold finish for a few bytes of host-closing: the kernel takes them, and finish follows. So the mutant that drops letGo from close() survives the suite. The cap guards a case the kernel buffer already covers for answers this small; the test pins letGo itself.

• A third racer. With three hosts on one stale lock, one host's put-back can lose to another's claim between the rename and the link. Then two brokers can boot for one dataRoot, and the broker's writer lock refuses the second. The claim's comment says so.

• Why a separate lock file. My first version claimed host.json itself. Sage asked for an audit of host.json's readers, and it showed that version broke them: host.json named a booting host before its broker socket existed. The human CLI then got outcome-unknown (exit 1), the Console a 503, and a Console send read "sent: unknown" with nothing sent; bus stop could SIGTERM a host whose handler wasn't installed yet. Sage approved moving the claim to host.lock. The table below is for this candidate.

  Reader	Where	No host.json (a host booting, or none)
  human verbs' default business	cli.mjs businessFor	usage refusal "no bus host runs here, so name the business with --business"
  the app (no verb)	cli.mjs app()	opens with no business
  Console reads	webui reads.mjs businessId	no-bus-host ("no bus host is running and no --business was given")
  Console bus calls and sends	webui bus.mjs busCaller	no-bus-host, which sendOutcome reports as sent: false
  bus stop	host.mjs stopHost	"no host runs", exit 0, no signal
  bus status	host.mjs hostStatus	host: none; socket and writer.lock as they are
  the host's own close	host.mjs close()	removes host.json only if it names this host

  All of these are what each did before this row, since BASE also wrote host.json after the notifier came up. None of them reads host.lock. A --business given explicitly skips the file in every case. The test row "a reader during boot" pins the CLI path and stopHost/hostStatus.

• An existing window, not new. host.json is written before the launcher listens and the PM launches, and cli.mjs installs its SIGTERM handler after those. A bus stop in that gap still kills the host without close(). That was so at BASE. With this row the lock then stays behind, and the next start takes it over because its holder is dead.

• startedAt is now taken at claim time, before the broker boots, not after. It is a start stamp for display and the identity triple; nothing compares it to a clock.

• release() reads the lock it is about to remove. If that read fails (a lock rewritten by hand into something malformed), release() leaves it alone rather than mask the boot error it runs under.

• Scope. host.mjs changes stay in startHost's check, its helpers (readState, same, claimHostLock, writeHostState untouched in body), a release() before each boot-failure throw and one in close(). No reindent, so Rocko's row 85 sees a small diff. No change to cli.mjs or webui. I told Sage before widening from host.json to host.lock, and they approved it.

• packages/harness/README.md isn't mine and still says 300 s, which stays true.

Gate

gate.sh in the packet: every package's own test script from its directory, then every scripts/test-*.sh, one after another, TMPDIR in my scratch. packages/ledger has tests but no package.json, so the script skips it; I ran node --test packages/ledger/tests/*.test.mjs after it (last line). It ran on 2b0867bee, green, then again on 79e126be0 when HEAD moved by one SESSIONS.md line. This is the second run:

bus rc=0 21:06:33-21:06:36 ℹ pass 127 ℹ fail 0 ℹ skipped 0  0 marks
business rc=0 21:06:36-21:06:38 ℹ pass 67 ℹ fail 0 ℹ skipped 0  0 marks
cli rc=0 21:06:38-21:06:54 ℹ pass 156 ℹ fail 0 ℹ skipped 0  0 marks
control-board rc=0 21:06:54-21:06:54 ℹ pass 124 ℹ fail 0 ℹ skipped 0  0 marks
conversation rc=0 21:06:54-21:07:42 ℹ pass 182 ℹ fail 0 ℹ skipped 0  0 marks
discord rc=0 21:07:42-21:07:45 ℹ pass 178 ℹ fail 0 ℹ skipped 0  0 marks
harness rc=0 21:07:45-21:07:55 ℹ pass 73 ℹ fail 0 ℹ skipped 0  0 marks
ledger: no test script
mosaic rc=0 21:07:55-21:07:57 ℹ pass 69 ℹ fail 0 ℹ skipped 0  0 marks
queue rc=0 21:07:57-21:08:26 ℹ pass 150 ℹ fail 0 ℹ skipped 0  0 marks
runs rc=0 21:08:26-21:08:27 ℹ pass 43 ℹ fail 0 ℹ skipped 0  0 marks
seat rc=0 21:08:27-21:08:32 ℹ pass 113 ℹ fail 0 ℹ skipped 0  0 marks
tasks rc=0 21:08:32-21:08:38 ℹ pass 92 ℹ fail 0 ℹ skipped 0  0 marks
webui rc=0 21:08:38-21:09:33 ℹ pass 108 ℹ fail 0 ℹ skipped 0  0 marks
test-auth rc=0 21:09:33-21:09:34 selftest: 15 passed, 0 failed
test-conductor rc=0 21:09:34-21:09:40 selftest: 17 passed, 0 failed
test-config rc=0 21:09:40-21:09:41 selftest: 24 passed, 0 failed
test-discord rc=0 21:09:41-21:09:48 discord suite: 66 passed, 0 failed
test-extension-package rc=0 21:09:48-21:09:49 extension package selftest: 18 passed, 0 failed
test-foundation rc=0 21:09:49-21:10:25 selftest: 44 passed, 0 failed
test-queue rc=0 21:10:25-21:10:57 queue suite: 27 passed, 0 failed
test-release rc=0 21:10:57-21:11:07 selftest: 14 passed, 0 failed
test-task rc=0 21:11:07-21:11:09 selftest: 26 passed, 0 failed
ledger rc=0 ℹ pass 78 ℹ fail 0 ℹ skipped 0

test-task.sh ran with DOCKER_HOST=unix:///nonexistent, set by line 19 of gate.sh and carried over from my earlier gate scripts. Under that value docker info fails, so test-task.sh skipped its adapter-seam, workspace/capability, live-task and recall cases: 26 passed, where a run with a working Docker daemon gives 98. The daemon on this host was up (docker info exits 0 without the override), so this was the script's choice, not an outage. The row 103 code in cli, seat and harness doesn't reach those cases, and Sage's landing gate runs them with Docker. Nothing was rerun. Every other suite ran with DOCKER_HOST unset.

Before the gate, the race test's "no claim file left over" check listed the whole bus-host directory and failed on the host's own launch.sock, git.sock and sessions.json; it now lists only host.* names. The Docker network the gate created (wt_default) was removed after each run, and no process of mine was left.