Files
stack/agents/darkwing/work/cohort-k1/build.md
T
jason.woltjeandClaude Opus 5.5 7478ee2c02 docs(conversation): row 46 cohort K1/K3/K10 diagnosis, receipts and candidate packet (darkwing)
The ignoreTerm tool child was TERMed before Node installed its handler
when the claim store sits on a fast disk. Candidate fixes the fixture;
the patch itself is not committed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-09 08:11:48 -05:00

7.2 KiB

Row 46: K1, K3 and K10 on the scope fixtures

Darkwing, 2026-10-09. Issue #1528, reviewer Dewey. Brief: docs/plans/2026-10-09_s4-follow-up-and-cohort.md, section "Conversation cohort: K1, K3 and K10 fail on the scope fixtures". Ruling: lead decision 72. The candidate is not committed, staged or pushed.

Cause

It's a race in the test fixture. Neither the host's systemd setup nor cohort.mjs is at fault.

spawnChild in packages/conversation/tests/fake-pi.mjs returns the child's pid as soon as spawn() returns. The child is node -e, and it installs its SIGTERM handler only after Node has booted. That takes 16 to 22 ms on an idle host (trace-pass.jsonl) and 43 to 80 ms under 48 CPU burners (trace-load.jsonl). A TERM that arrives before the handler gets the default action, and the child dies of SIGTERM. Each failing assertion is that death seen from a different place:

  • K1, "the escaped child is a listed member". The child died during the TERM grace, so the freeze-phase enumeration doesn't list it.
  • K3, "the member ignored TERM". alive(child) is false at the kill phase.
  • K10, "the member is alive across the crash". Same as K3, before the restart.

The time between spawn and TERM depends on the disk under TMPDIR. The fixture puts the claim store there. Before it sends TERM, the controller publishes claim revisions, each with an fsync on the file and one on the directory (packages/conversation/src/claim.mjs:159 and :176).

TMPDIR Disk write+fsync median (fsync.txt) spawn to TERM Unfixed K1/K3/K10
/mnt/storage/scratch/tmp nvme0, ext4 0.65 ms 12 to 13 ms (trace-fail.jsonl) 0/3 (runs/canon-scratchtmp.txt)
/tmp nvme1, ext4 4.63 ms not traced 3/3 (runs/canon-tmp.txt)
~/darkwing-scratch/tmp nvme1, ext4 4.69 ms 53 to 110 ms (trace-pass.jsonl) 3/3 (runs/canon-hometmp.txt)

On the fast disk, TERM lands about 12 ms after spawn, before Node is up. In trace-fail.jsonl the children never log ready.

Why it started failing

This host's seats now get TMPDIR=/mnt/storage/scratch/tmp. My shell had it by default, and so did Sage's gate runs. It comes from Vikunja task 59. ~/.config/systemd/user/t3code.service.d/tmpdir.conf was written 2026-10-04 20:33Z. Its own comment says it takes effect only when t3code.service restarts, which hasn't happened (active since 2026-09-23), and that until then seats get TMPDIR from their harness config. I didn't establish what TMPDIR the row 44 gate ran with on 2026-10-05, so the "since when" is likely but unproven. The change that exposed the race is a seat's TMPDIR. It isn't a systemd or user manager setting, and I changed nothing on the host.

The scope is not a factor

Outside any scope, race.mjs gives the same split: a TERM 0, 10 or 20 ms after spawn kills 10/10, and at 30 or 60 ms 10/10 survive (race.txt). Sage's outside-scope probe found the child surviving. I haven't seen that probe, but it most likely sent TERM after the handler was in place.

Receipts in a scope

scope-receipt.mjs launches a delegated scope per run, with the same systemd-run flags ScopeLauncher uses. The scope's main process spawns the fixture's ignoreTerm child and records systemctl --user show on the scope, then cgroup.procs before and after TERM, then the child's exit. Output: receipts.jsonl.

Mode TERM after spawn In cgroup.procs before After Child exit
race 12 20 to 25 ms 5/5 1/5 4/5 signal: "SIGTERM", 1 alive
race 60 69 to 71 ms 5/5 5/5 5/5 alive 500 ms after TERM
ready 0 28 to 38 ms (ready at 22 to 30) 5/5 5/5 5/5 alive 500 ms after TERM

Each line also carries the scope's Id, LoadState=loaded, ActiveState=active, InvocationID, ControlGroup and Delegate=yes. The first line, for example: ControlGroup=/user.slice/user-1000.slice/[email protected]/app.slice/dw-r46-race-12-3662746-0.scope, before [3662760, 3662788], after [3662760], child exit {"code":null,"signal":"SIGTERM","atMs":24}.

trace.patch is the scratch-only instrumentation behind the two traces. It logs spawn, ready and the shim's term per pid to $DW_TRACE. It is not part of the candidate.

The fix

build.patch changes one file, packages/conversation/tests/fake-pi.mjs:

  • The child writes one byte to stdout as its first action after installing its TERM handler. A child without ignoreTerm writes it once its code starts.
  • spawnChild returns a promise. It resolves with the pid on that byte, then closes its end of the pipe. It rejects if the child exits or errors first, or if 10 s pass.
  • The child control op returns that promise. The op dispatcher now answers ok: false when an op's promise rejects. Before, a rejected op promise went unhandled, and under Node's default that ends fake-pi. The change covers every op that returns a promise, not only child. childOf in cohort.test.mjs already asserts r.ok.

No assertion changed, and cohort.mjs is untouched. The three cases now test what their names say: TERM reaches a child that is already ignoring TERM.

build-manifest.sha256 (sha256 9228414352a56e0465e896b81643cdce000bcedba70f86adac07fc50cfadad2d) pins fake-pi.mjs after the patch. build.patch sha256 is 04234ce1d886e36dfd06cbd81153b38cbb86909b6ee933151306b8cdedd629d9. In a fresh worktree at 521597bb the patch applies and the manifest checks 1/1.

Mutants

Both ran with TMPDIR=/mnt/storage/scratch/tmp, the condition that fails.

Mutant K1/K3/K10 File
M1: resolve at spawn, no wait (the old behavior) 0/3 runs/mutant-m1-nowait.txt
M2: wait for ready, but the child has no TERM handler 0/3, the same three assertions runs/mutant-m2-noignore.txt

M1 shows the wait is what fixes it. M2 shows the assertions still catch a child that dies on TERM.

Runs

The patch was applied in a scratch worktree at 521597bb. Node v26.8.1. Start time and load are in runs/start.txt.

Run TMPDIR Result File
node --test "packages/conversation/tests/*.test.mjs" /mnt/storage/scratch/tmp 152/152 runs/node-conversation.txt
node --test "packages/webui/tests/*.test.mjs" /mnt/storage/scratch/tmp 14/14 runs/node-webui.txt
K1/K3/K10 isolated, 3 consecutive /mnt/storage/scratch/tmp 3/3, 3/3, 3/3 runs/k-iso-{1,2,3}.txt
K1/K3/K10 isolated /tmp 3/3 runs/k-tmp.txt
K1/K3/K10 isolated ~/darkwing-scratch/tmp 3/3 runs/k-home.txt
K1/K3/K10 isolated under 48 CPU burners, 3 runs (load 13.6 to 24.9) /mnt/storage/scratch/tmp 3/3, 3/3, 3/3 runs/k-load48-{1,2,3}.txt

For the gate rerun, keep the default TMPDIR=/mnt/storage/scratch/tmp. That's the condition that failed, and a slower TMPDIR would pass with or without the patch.

Files

  • build.md, this file.
  • build.patch, build-manifest.sha256: the candidate.
  • scope-receipt.mjs, receipts.jsonl: the scope receipts.
  • race.mjs, race.txt: TERM timing outside a scope.
  • fsync.mjs, fsync.txt: write+fsync latency per TMPDIR.
  • trace.patch, trace-fail.jsonl, trace-pass.jsonl, trace-load.jsonl: the traced runs.
  • runs/: suite, isolated, load and mutant outputs, and the unfixed canonical tree under three TMPDIRs.