The ignoreTerm tool child was TERMed before Node installed its handler when the claim store sits on a fast disk. Candidate fixes the fixture; the patch itself is not committed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
7.2 KiB
Row 46: K1, K3 and K10 on the scope fixtures
Darkwing, 2026-10-09. Issue #1528, reviewer Dewey. Brief:
docs/plans/2026-10-09_s4-follow-up-and-cohort.md, section "Conversation
cohort: K1, K3 and K10 fail on the scope fixtures". Ruling: lead decision 72.
The candidate is not committed, staged or pushed.
Cause
It's a race in the test fixture. Neither the host's systemd setup nor
cohort.mjs is at fault.
spawnChild in packages/conversation/tests/fake-pi.mjs returns the
child's pid as soon as spawn() returns. The child is node -e, and it
installs its SIGTERM handler only after Node has booted. That takes 16 to
22 ms on an idle host (trace-pass.jsonl) and 43 to 80 ms under 48 CPU burners (trace-load.jsonl). A
TERM that arrives before the handler gets the default action, and the child
dies of SIGTERM. Each failing assertion is that death seen from a different
place:
- K1, "the escaped child is a listed member". The child died during the TERM grace, so the freeze-phase enumeration doesn't list it.
- K3, "the member ignored TERM".
alive(child)is false at the kill phase. - K10, "the member is alive across the crash". Same as K3, before the restart.
The time between spawn and TERM depends on the disk under TMPDIR. The
fixture puts the claim store there. Before it sends TERM, the controller
publishes claim revisions, each with an fsync on the file and one on the
directory (packages/conversation/src/claim.mjs:159 and :176).
| TMPDIR | Disk | write+fsync median (fsync.txt) |
spawn to TERM | Unfixed K1/K3/K10 |
|---|---|---|---|---|
/mnt/storage/scratch/tmp |
nvme0, ext4 | 0.65 ms | 12 to 13 ms (trace-fail.jsonl) |
0/3 (runs/canon-scratchtmp.txt) |
/tmp |
nvme1, ext4 | 4.63 ms | not traced | 3/3 (runs/canon-tmp.txt) |
~/darkwing-scratch/tmp |
nvme1, ext4 | 4.69 ms | 53 to 110 ms (trace-pass.jsonl) |
3/3 (runs/canon-hometmp.txt) |
On the fast disk, TERM lands about 12 ms after spawn, before Node is up.
In trace-fail.jsonl the children never log ready.
Why it started failing
This host's seats now get TMPDIR=/mnt/storage/scratch/tmp. My shell had
it by default, and so did Sage's gate runs. It comes from Vikunja task 59.
~/.config/systemd/user/t3code.service.d/tmpdir.conf was written
2026-10-04 20:33Z. Its own comment says it takes effect only when
t3code.service restarts, which hasn't happened (active since 2026-09-23),
and that until then seats get TMPDIR from their harness config. I didn't
establish what TMPDIR the row 44 gate ran with on 2026-10-05, so the
"since when" is likely but unproven. The change that exposed the race is a
seat's TMPDIR. It isn't a systemd or user manager setting, and I changed
nothing on the host.
The scope is not a factor
Outside any scope, race.mjs gives the same split: a TERM 0, 10 or 20 ms
after spawn kills 10/10, and at 30 or 60 ms 10/10 survive (race.txt).
Sage's outside-scope probe found the child surviving. I haven't seen that
probe, but it most likely sent TERM after the handler was in place.
Receipts in a scope
scope-receipt.mjs launches a delegated scope per run, with the same
systemd-run flags ScopeLauncher uses. The scope's main process spawns
the fixture's ignoreTerm child and records systemctl --user show on the
scope, then cgroup.procs before and after TERM, then the child's exit.
Output: receipts.jsonl.
| Mode | TERM after spawn | In cgroup.procs before |
After | Child exit |
|---|---|---|---|---|
race 12 |
20 to 25 ms | 5/5 | 1/5 | 4/5 signal: "SIGTERM", 1 alive |
race 60 |
69 to 71 ms | 5/5 | 5/5 | 5/5 alive 500 ms after TERM |
ready 0 |
28 to 38 ms (ready at 22 to 30) | 5/5 | 5/5 | 5/5 alive 500 ms after TERM |
Each line also carries the scope's Id, LoadState=loaded,
ActiveState=active, InvocationID, ControlGroup and Delegate=yes.
The first line, for example:
ControlGroup=/user.slice/user-1000.slice/[email protected]/app.slice/dw-r46-race-12-3662746-0.scope,
before [3662760, 3662788], after [3662760], child exit
{"code":null,"signal":"SIGTERM","atMs":24}.
trace.patch is the scratch-only instrumentation behind the two traces.
It logs spawn, ready and the shim's term per pid to $DW_TRACE. It is
not part of the candidate.
The fix
build.patch changes one file, packages/conversation/tests/fake-pi.mjs:
- The child writes one byte to stdout as its first action after installing
its TERM handler. A child without
ignoreTermwrites it once its code starts. spawnChildreturns a promise. It resolves with the pid on that byte, then closes its end of the pipe. It rejects if the child exits or errors first, or if 10 s pass.- The
childcontrol op returns that promise. The op dispatcher now answersok: falsewhen an op's promise rejects. Before, a rejected op promise went unhandled, and under Node's default that ends fake-pi. The change covers every op that returns a promise, not onlychild.childOfincohort.test.mjsalready assertsr.ok.
No assertion changed, and cohort.mjs is untouched. The three cases now
test what their names say: TERM reaches a child that is already ignoring
TERM.
build-manifest.sha256 (sha256
9228414352a56e0465e896b81643cdce000bcedba70f86adac07fc50cfadad2d) pins
fake-pi.mjs after the patch. build.patch sha256 is
04234ce1d886e36dfd06cbd81153b38cbb86909b6ee933151306b8cdedd629d9. In a
fresh worktree at 521597bb the patch applies and the manifest checks 1/1.
Mutants
Both ran with TMPDIR=/mnt/storage/scratch/tmp, the condition that fails.
| Mutant | K1/K3/K10 | File |
|---|---|---|
| M1: resolve at spawn, no wait (the old behavior) | 0/3 | runs/mutant-m1-nowait.txt |
| M2: wait for ready, but the child has no TERM handler | 0/3, the same three assertions | runs/mutant-m2-noignore.txt |
M1 shows the wait is what fixes it. M2 shows the assertions still catch a child that dies on TERM.
Runs
The patch was applied in a scratch worktree at 521597bb. Node v26.8.1.
Start time and load are in runs/start.txt.
| Run | TMPDIR | Result | File |
|---|---|---|---|
node --test "packages/conversation/tests/*.test.mjs" |
/mnt/storage/scratch/tmp |
152/152 | runs/node-conversation.txt |
node --test "packages/webui/tests/*.test.mjs" |
/mnt/storage/scratch/tmp |
14/14 | runs/node-webui.txt |
| K1/K3/K10 isolated, 3 consecutive | /mnt/storage/scratch/tmp |
3/3, 3/3, 3/3 | runs/k-iso-{1,2,3}.txt |
| K1/K3/K10 isolated | /tmp |
3/3 | runs/k-tmp.txt |
| K1/K3/K10 isolated | ~/darkwing-scratch/tmp |
3/3 | runs/k-home.txt |
| K1/K3/K10 isolated under 48 CPU burners, 3 runs (load 13.6 to 24.9) | /mnt/storage/scratch/tmp |
3/3, 3/3, 3/3 | runs/k-load48-{1,2,3}.txt |
For the gate rerun, keep the default TMPDIR=/mnt/storage/scratch/tmp.
That's the condition that failed, and a slower TMPDIR would pass with or
without the patch.
Files
build.md, this file.build.patch,build-manifest.sha256: the candidate.scope-receipt.mjs,receipts.jsonl: the scope receipts.race.mjs,race.txt: TERM timing outside a scope.fsync.mjs,fsync.txt: write+fsync latency per TMPDIR.trace.patch,trace-fail.jsonl,trace-pass.jsonl,trace-load.jsonl: the traced runs.runs/: suite, isolated, load and mutant outputs, and the unfixed canonical tree under three TMPDIRs.