feat(conversation): release at engine exit through a proven engine-exit stop (#1538, row 52)

The controller now starts a CHAT-01 `engine-exit` stop when the engine
exits on its own. engineExitCohort proves the cohort from the shim's
recorded engine identity (pid, start ticks, boot) and an empty member
list, under a 5 s deadline that frees the escalation slot. A proven stop
releases the scope; anything else ends `uncertain`. close() no longer
signals a stopped binding.

Dewey built it. Filbert (27087) and Darkwing (27088) approved round 1 on
CHAT-01 cc83ee4f and manifest deac7434 (6 files). Sage's gate on
c92cfb8f plus the candidate: conversation 182/0, webui 22/0,
control-board 124/0, every scripts/test-*.sh 0 failed (task 98/0),
chat-01 PASS.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-10 03:31:39 -05:00
co-authored by Claude Opus 5.5
parent c92cfb8faf
commit 0aa5e7b51b
7 changed files with 598 additions and 39 deletions
@@ -0,0 +1,6 @@
774224cf6a105ff80d3e71637bd84bf68cb2da447b3a4f0fec25e021b7401f46 agents/dewey/work/queue-52/evidence.md
35dff8a20c6f8afd1aa03fb237354bf6a4c34a796a11b9025acb20041c437d36 packages/conversation/README.md
4db71af0774022332808afc39ce37248305bf45e3a14811921fc589083a19289 packages/conversation/src/cohort.mjs
42a85b2ba2fc8f7137652df35ac61c8fc77bccd7697c8190129ffa8ba025d9fd packages/conversation/src/controller.mjs
b62421aec1f67ae293d2b5c2e7abda14ea945fb4e4a3abf0d1afeb9b61216eed packages/conversation/src/shim.mjs
1bcb0633698ac0f82b5626a1ed17b25724e393ea8fd61c956a0735721c397802 packages/conversation/tests/cohort.test.mjs
+245
View File
@@ -0,0 +1,245 @@
# Row 52 (#1538): CHAT-01 engine-exit stop and release at engine exit
Dewey, 2026-10-10. Brief: `docs/plans/2026-10-10_cohort-release-follow-ups.md`,
section "CHAT-01 engine-exit stop and release at engine exit" (rev 297).
Base 945440db (row 51 landed). The change has two parts, CHAT-01 first in
its own commit:
- CHAT-01 (commit cc83ee4f, not pushed): the four `docs/plans/chat-01/` files.
- Conversation (candidate, uncommitted): `src/cohort.mjs`,
`src/controller.mjs`, `src/shim.mjs`, `tests/cohort.test.mjs` and the
package README. `tests/ctrl-child.mjs` is unchanged.
## CHAT-01 hashes (each approval names all four)
```
a41fc4e2edb11371c275e3167774c162254e1457fd737121630dedd665431889 docs/plans/chat-01/README.md
ccceaf2653b279f5890748b40d16fae7493f199a33cecee035f82b92e80243af docs/plans/chat-01/check.mjs
941675de949c52c7fe7c4b7bcce3aff48bb26ccc1120e090574c5d25f9d9e22e docs/plans/chat-01/contracts.schema.json
c75c2b9f731bb70d0e033e8aa56f2c28accfa09384ec964fd3ecbd45974b2a7d docs/plans/chat-01/fixtures.json
```
`node docs/plans/chat-01/check.mjs`: 100 shape cases, 76 reference cases,
22 lifecycle sequences, PASS. `node docs/plans/chat-00/check.mjs`: 48 checks
PASS.
## Darkwing's points 1–4
| Point | Model (`check.mjs`) | Conversation |
|---|---|---|
| 1. Never supersede an unfinished force stop | `engine-exit` returns `stop-owned` while the current stop is a force stop (any state) or an engine exit. Sequence `engine-exit-refused-during-force-stop` walks all four force-stop states. | `#engineExit` starts nothing while `escalating` is held or the current stop is a force stop or engine exit: a running force stop (E3, second half) and one that ended `uncertain` before it closed the execution (E5). |
| 2. Binding follows at lines 121, 334, 337 | All three use `ending(mode)` (force stop or engine exit). Sequence `engine-exit-binding-follows-stop`: `stopping` at the fence and on each advance, `uncertain` with the stop, `stopped` only with it. | `#startStop` and `#advanceStop` use the same `ending` (E3 sees `stopping` at `engine-exit-recorded`). |
| 3. The proof names the engine; not `members: []` | `stopped(w, s)` refuses an `engine-exit` proof with no member. The empty-list case is in `engine-exit-live-unreadable-or-empty-cohort-uncertain`. | `engineExitCohort` returns the engine as the one member: PID and start ticks the shim recorded, boot, and the shim's reap time as `terminatedAt`. `#stopped` also refuses an engine-exit proof with no member. E1 checks the members exactly; E6 checks that another PID or start ticks is refused. |
| 4. The proof's deadline frees the slot | The model leaves the slot to the implementation (README says so). | The observation runs under `within(…, exitProof)`. A miss ends the stop `uncertain`, and `finally` frees `escalating`. E4 SIGSTOPs the shim: the stop ends `uncertain` at the deadline, `escalating` is `null`, and a client force stop then proves and releases. |
### Engine identity comes from a shim op
Sage's condition: if the engine identity can't come from a shim op, stop
before relying on `members: []`. It comes from one. The shim reads the
engine's start ticks from `/proc` right after spawn (the exec chain keeps
the PID and start time), and its `exit` handler records `{ code, signal,
at, pid, startTicks, boot }`. `hello` returns that as `engineExit`.
`engineExitCohort` refuses, as `unavailable`, an `engineExit` whose PID or
start ticks differ from the claim's recorded engine, whose start ticks
aren't a positive integer, or whose boot differs from the shim's. Nothing
relies on an empty member list: the force stop's proof (R4, now in E4) is
unchanged and still lists no member when the engine has already gone.
## CHAT-01 change
- Stop mode `engine-exit` (schema enum). The server starts it like a
revocation fence: `request: null`, no confirmation.
- `server(w, 'engine-exit')`: refuses `stop-owned` under a force stop or
another engine exit; otherwise `startStop` (closes admission, recovers
queued drafts, marks pending decisions uncertain, supersedes an
unfinished interrupt or revocation, emits `stopping`). Dispatched or
acknowledged input moves to `delivery-unknown`; `working` input keeps its
state.
- `confirm-stopped` accepts it under the same `stopped(w, s)` check as a
force stop, plus at least one member.
- Recover after it needs its own confirmation bound to the stop (K6, K17,
K18): an unissued one, one issued but not confirmed, and the binding's
stop check are all in `engine-exit-empty-cohort-stopped`.
- Fixtures: `valid-engine-exit-stop`, `refuse-stop-mode-eof`, and five
sequences: `engine-exit-empty-cohort-stopped`,
`engine-exit-live-unreadable-or-empty-cohort-uncertain`,
`engine-exit-stale-confirmation-dispatched-receipt`,
`engine-exit-refused-during-force-stop`, `engine-exit-binding-follows-stop`.
- README: a paragraph under "Control loss, approvals and stopping", a
pointer near the top, and the R3 counts. "SIGTERM, EOF, abort
acknowledgment or idle does not prove death" stays; K15 and K2 are
unchanged.
### Model mutants (`check.mjs` on a scratch copy)
| Mutant | Change | Result |
|---|---|---|
| norefuse | `engine-exit` never refuses `stop-owned` | killed |
| optionB | refuses only while the force stop is unfinished | killed |
| bind121 | `startStop` moves the binding only for a force stop | killed |
| bind334 | `advance-stop` the same | killed |
| bind337 | `confirm-stopped` accepts only a force stop | killed |
| emptyok | no member-count check for an engine exit | killed |
| noreceipt | dispatched input keeps its state | killed |
| nostale | no current-stop check on a confirmation | killed |
## Conversation change
- `#onEnd` (EOF): after `#transport` (which settles in-flight input
`delivery-unknown`/`transport-unknown` as before, H19, N18), starts
`#engineExit`.
- `#engineExit`: only for the current execution with a recorded engine,
no held slot and no current force stop or engine exit. Starts the
`engine-exit` stop, takes `escalating`, pauses at `engine-exit-recorded`
(test barrier), runs `engineExitCohort` under the `exitProof` deadline
(5000 ms; `exitSettle` 2000 ms for EOF before the reap), then goes to
`uncertain` or through `#stopProven`.
- `#endStopUncertain` and `#stopProven` are `#escalate`'s former `fail` and
proof tail, shared. The force-stop path is unchanged: same order, same
claim writes, `resumed` still in its evidence. Evidence and the release
now carry `stop.mode` instead of a literal `force-stop`.
- `engineExitCohort` (cohort.mjs): reads only (`hello`, `events`,
`members`), never freezes or kills. Proven when the shim has reaped the
recorded engine, `engine` reads `populated 0` and no member is listed.
- The claim: no `stopping` claim write for an engine exit. The claim goes
`uncertain` at EOF (`#transport`, as before) and `stopped` only through
`#claimFinish` with the proof. A restart in between finds an ordinary
`uncertain` orphan, never an engine-exit stop to resume.
## Tests (`cohort.test.mjs`, needs a systemd user manager)
R4 is folded into E4; E1–E6 are new.
- E1: the engine exits with no other member. One `engine-exit` stop,
`request: null`, superseding the prior stop, `stopped`; the proof's
members are exactly the engine (PID, boot, start ticks, reap time from
the shim's `hello`); both claim keys stopped on it; the release is
`engine-exit`/`released`/`absent` and the unit is gone. A recover
confirmation issued before the exit is refused; a fresh one is
`recovery-eligible`.
- E2: the engine exits while a tool child runs. The stop ends `uncertain`
(evidence `engine-exit`/`uncertain`), no release, the unit and the child live, no proof. A force-stop
confirmation issued before the exit is refused (H17). A fresh force stop
supersedes the engine-exit stop, kills the child, proves and releases.
- E3: one slot, both orders. The engine-exit stop held at
`engine-exit-recorded`: a client force stop is refused `fenced`, and the
engine exit then proves. A force stop held at `force-stop-recorded`: the
engine's EOF starts no engine-exit stop, and only the force stop is in
the evidence.
- E4: the shim SIGSTOPped. The engine-exit stop misses its deadline, ends
`uncertain` (reason names the deadline), `escalating` is `null`; after
SIGCONT a force stop proves with no member (the R4 assertion) and the
scope goes.
- E5: a force stop whose `stopping` claim write fails once ends `uncertain`
before it closes the execution, so the engine's later EOF still reaches
`#onEnd`. No engine-exit stop starts; the force stop stays current
(point 1 after the slot is free).
- E6: held at `engine-exit-recorded`, `engineExitCohort` with another PID
or other start ticks is `unavailable` ("the shim reaped a process that
isn't the recorded engine"); the recorded identity is `proven`, and the
controller's stop then proves.
Conversation suite on the draft at 945440db: 182/182 (was 177: R4 out,
E1–E6 in).
## Mutation check
Twenty mutants on a fresh copy of the candidate, full conversation suite
each (`~/dewey-scratch/r52/mut-tools`). Run 1 (07:11–07:33Z) was on the
tests before E5, E6 and the E2 evidence check, and left `noowncheck`,
`noident` and `evmode` alive with the five equivalents. I added those
tests and ran all twenty again (run 2, 07:33–07:54Z). Run 2 base:
182/182. No shim, engine or `mosaic-chat-*` unit was left after any run.
| Mutant | Change | Run 2 | Killed by |
|---|---|---|---|
| noexit | `#onEnd` doesn't start `#engineExit` | killed | E1, E2, E3, E4, E6 |
| noslottake | the engine-exit stop doesn't take `escalating` | killed | E3 |
| nofree | `finally` doesn't free the slot | killed | E2, E4 |
| noslotcheck | EOF ignores a held slot | survives, equivalent | – |
| noowncheck | EOF ignores a current force stop or engine exit | killed | E5 |
| nostopcheck | both checks gone | killed | E3, E5 |
| nodeadline | the observation has no deadline | killed | E4 |
| bindstart | `#startStop` moves the binding only for a force stop | killed | E3 |
| bindadvance | `#advanceStop` the same | survives, equivalent | – |
| emptyguard | `#stopped` accepts an engine-exit proof with no member | survives, equivalent alone | – |
| emptyproof | the proof lists no member | killed | E1 |
| emptyboth | both: an empty-list proof that verifies | killed | E1 |
| noident | no PID or integer check on the shim's `engineExit` | killed | E6 |
| nopopulated | a populated `engine` cgroup proves | survives, equivalent | – |
| nomembers | a listed member proves | survives, equivalent | – |
| nolive | neither is read | killed | E2 |
| shimstart | the shim keeps no start ticks | killed | E1, E3, E6 |
| relwhy | the release is recorded as a force stop's | killed | E1 |
| evmode | uncertain evidence names a force stop | killed | E2 |
Why the five survivors are equivalent:
- noslotcheck: `escalating` is set only right after `#startStop` made a
force stop or engine exit the current stop, and the binding is then
`stopping`. Interrupt and revocation need `b.state === "active"`
(controller.mjs `#interruptLocked`, the revocation in the connection
close), so no other stop replaces it while the slot is held. `ending(…)`
covers every state the slot check covers. The check stays as a guard.
- bindadvance: the binding is already `stopping` from `#startStop`;
`#uncertain` keeps `stopping`; `#endStopUncertain` and `#stopProven`
set the binding themselves. The `ending` in `#advanceStop` stays to match
the model's line 334.
- emptyguard: a proven `engineExitCohort` always lists the engine, so
`#stopped`'s guard is reachable only through a second fault. emptyproof
and emptyboth test point 3.
- nopopulated, nomembers: the shim's `members` walks `engine` and its
descendants, so `populated 1` with an empty list (or `populated 0` with
a member) needs a process to start or exit between the two reads. Both
reads stay; each guards the other.
Killing tests are from each mutant's `failing tests:` list. Run 1 and run 2
tables: `~/dewey-scratch/r52/mut-tools/results-run1.txt` and
`results-run2.txt`.
## Gate
Run in a scratch worktree at cc83ee4f with the candidate applied
(`~/dewey-scratch/r52/gate/candidate.diff`), 2026-10-10T07:56:23Z to
08:01:22Z, sequential, each output kept.
| Suite | rc | Result |
|---|---|---|
| `node docs/plans/chat-01/check.mjs` | 0 | PASS: 100 shape, 76 reference, 22 lifecycle |
| conversation | 0 | 182 pass, 0 fail |
| webui | 0 | 22 pass, 0 fail |
| control-board | 0 | 124 pass, 0 fail |
| `test-auth.sh` | 0 | 15 passed, 0 failed |
| `test-conductor.sh` | 0 | 17 passed, 0 failed |
| `test-config.sh` | 0 | 24 passed, 0 failed |
| `test-discord.sh` | 0 | 66 passed, 0 failed |
| `test-extension-package.sh` | 0 | 18 passed, 0 failed |
| `test-foundation.sh` | 0 | 44 passed, 0 failed |
| `test-queue.sh` | 0 | 148 node pass, suite 27 passed, 0 failed; `queue verify` and `render --check` skipped (not the canonical root) |
| `test-release.sh` | 0 | 14 passed, 0 failed |
| `test-task.sh` | 0 | 98 passed, 0 failed |
The `test-queue.sh` skip is that suite's own guard for a non-canonical
checkout; Sage's rerun in the canonical checkout covers it. After the
gate and the mutant runs no `mosaic-chat-*` unit is listed
(`systemctl --user list-units --all 'mosaic-chat-*'`) and `core.hooksPath`
is unset.
## Not tested, follow-ups
Nothing here blocks the row; I'd file them together if Sage wants them.
- The boot comparison in `engineExitCohort` (`exit.boot !== hello.boot`)
is untested: both come from the same shim process, so only a faked
`hello` reaches it.
- An unreadable `engine/cgroup.events` or a failed `members` during the
engine-exit observation (the K15 shapes) is untested for this path. The
tests reach `unavailable` only through the deadline (E4).
- On the process-group fallback a natural exit now starts an
`engine-exit` stop that ends `uncertain` at once ("a pgroup cohort
can't be enumerated completely"). The binding ends `uncertain` as it
did before row 52, but no test runs it.
- The five equivalent mutants above are guards I kept on purpose; no
change proposed.
- Row 51's untested `still listed` retry and the noseat and recafteracq
survivors are #1539, not repeated here.
+27 -4
View File
@@ -378,9 +378,10 @@ can disagree with it.
unavailable, or a proof the verifier rejects) nothing is released: the
scope and its shim are what a later force stop reads, and a collected
scope would be an absent observation, not proof that the cohort ended
(R3). An engine that exits on its own leaves the shim and scope up and
the binding `uncertain`; a proven force stop on the empty cohort then
releases them (R4). A controller killed between recording the stop and
(R3). An engine that exits on its own ends in an `engine-exit` stop
(below), and only a proven one releases the scope (E1). A force stop on
a cohort the engine already left still proves on an empty member list
(E4). A controller killed between recording the stop and
releasing leaves the scope; the restart's `start` finds the claim stopped
on a cohort proof with its unit still listed and releases it (R7, R8). A
release that ended `unavailable` or `still listed` is tried again by the
@@ -393,6 +394,28 @@ can disagree with it.
engine PID, even if the proof stops verifying later: the proof showed
every member ended, so the recorded PID may belong to another process
(R10, R11).
- **Engine exit** (row 52, CHAT-01 `engine-exit`). End of output starts an
`engine-exit` stop: the server starts it with no request and no
confirmation, since nothing is signalled. It closes admission, takes the
one escalation slot (H10) and supersedes the current stop, so a
force-stop confirmation issued before it is refused (H17). It starts
nothing while a force stop holds the slot or is the current stop, even
one that ended `uncertain` (E3, E5), and a client force stop during it is
refused `fenced` (E3). The observation
only reads the scope: `hello`, `events` and `members`, with no freeze
or kill. It is proven when the shim reports that it reaped the recorded
engine (PID, start ticks and boot; E6) and `engine` reads `populated 0`
with no member listed. The proof's one member is the engine, with the reap
time as its death time; a proof with an empty member list never
verifies an `engine-exit` stop. A proven stop goes through the verifier
to `stopped` and releases the scope like a force stop (E1). A live
member ends the stop `uncertain` and releases nothing; a confirmed force
stop then ends the cohort (E2). Any unavailable observation ends it
`uncertain` the same way. The
observation has a deadline (`exitProof`): a shim that doesn't answer
ends the stop `uncertain` and frees the slot for a force stop (E4).
Recover after an `engine-exit` stop needs its own confirmation, bound to
that stop (E1).
- **Confirmations** bind the target, operation and stop, and are consumed on
use (K6). One issued before the stop changed is refused (H17).
- **Recovery.** A confirmed `recover` after a proven stop returns a
@@ -573,7 +596,7 @@ no prompt:
| `claim.test.mjs` | W1–W17, W20, G1–G3: the writer claim, crash barriers, the guard |
| `races.test.mjs` | H1–H4, H9–H23: takeover, Interrupt and force stop, retries, incarnations |
| `turns.test.mjs` | N1–N25, N24b: the turn tracker against the fake engine's Pi behaviors; the engine seal and environment |
| `cohort.test.mjs` | K1–K19, R1–R12: scopes, force stop, proofs, recovery, eligibility, the scope's environment, scope release and its retry after a crash (needs a systemd user manager) |
| `cohort.test.mjs` | K1–K19, R1–R3, R5–R12, E1–E6: scopes, force stop, proofs, recovery, eligibility, the scope's environment, scope release and its retry after a crash, the engine-exit stop (needs a systemd user manager) |
| `flows.test.mjs` | S1–S7, P3, E1–E7, the terminal, and a CHAT-01 schema check of every record produced |
| `smoke.test.mjs` | the pinned Pi binary, as above |
+46
View File
@@ -17,6 +17,10 @@
// membership complete. Anything unavailable ends the stop `uncertain`. The
// scope stays up after the stop either way; `releaseCohort` ends it once the
// controller has recorded a proven stop.
//
// Engine exit (row 52): at EOF the controller reads the cohort without
// signalling it. The shim reports the engine's PID, start ticks and reap time
// from its wait; `engine` empty with that engine reaped is the proof.
import { spawn, spawnSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs";
@@ -204,6 +208,48 @@ export async function forceStopCohort({ kind, unitName, invocationId, shimSocket
};
}
// Observes an engine that exited on its own (row 52, CHAT-01 `engine-exit`).
// It reads only: `hello`, `events` and `members`; nothing is frozen or
// signalled. The cohort is proven ended when the shim has reaped the recorded
// engine (its PID and start ticks) and `engine` reads `populated 0` with no
// member listed. The proof's one member is the engine, dead at the shim's
// wait; an empty list is never the proof. EOF can come before the reap, so it
// reads again until `settleMs` pass. Anything else is `unavailable`. The
// caller bounds the whole observation with its own deadline.
export async function engineExitCohort({ kind, unitName, invocationId, shimSocket, pid, start, settleMs = 2000 }) {
if (kind !== "scope") return { outcome: "unavailable", reason: `a ${kind} cohort can't be enumerated completely` };
const show = systemctlShow(unitName);
if (!show || !invocationId || show.invocationId !== invocationId) {
return { outcome: "unavailable", reason: `invocation ID mismatch or unreadable (recorded ${invocationId}, found ${show?.invocationId ?? "none"})` };
}
const end = Date.now() + settleMs;
for (;;) {
const hello = await shimRequest(shimSocket, "hello");
if (!hello.ok) return { outcome: "unavailable", reason: hello.unavailable };
if (hello.invocationId !== invocationId || hello.scope !== show.controlGroup) return { outcome: "unavailable", reason: "the shim does not answer for the recorded scope" };
const exit = hello.engineExit;
if (exit) {
if (exit.pid !== pid || !Number.isInteger(exit.startTicks) || exit.startTicks < 1 || String(exit.startTicks) !== String(start) || exit.boot !== hello.boot) {
return { outcome: "unavailable", reason: "the shim reaped a process that isn't the recorded engine" };
}
const e = await shimRequest(shimSocket, "events");
if (!e.ok) return { outcome: "unavailable", reason: e.unavailable };
if (e.populated === 0) {
const listed = await shimRequest(shimSocket, "members");
if (!listed.ok) return { outcome: "unavailable", reason: listed.unavailable };
if (listed.members.length === 0) {
return {
outcome: "proven", membershipComplete: true, epoch: invocationId, observedAt: new Date().toISOString(), boot: hello.boot,
members: [{ pid, boot: exit.boot, startTicks: exit.startTicks, terminatedAt: exit.at }],
};
}
}
}
if (Date.now() >= end) return { outcome: "unavailable", reason: exit ? "the engine cgroup still has a member" : "the shim has not reaped the engine" };
await sleep(20);
}
}
// Ends a scope whose cohort is proven stopped (#1536). The shim exits on
// `release` only while `engine` reads `populated 0`, and systemd then
// collects the empty scope. Call it only after a `proven` stop whose claim
+76 -21
View File
@@ -23,7 +23,7 @@ import { fileURLToPath } from "node:url";
import { randomBytes } from "node:crypto";
import { processStart } from "../../discord/src/journal.mjs";
import { ALREADY_ACTIVE, ClaimStore, FOREIGN_HOST, UNSAFE_REPLACEMENT, machineId, seatKey, sessionKey } from "./claim.mjs";
import { AUTHORITY, PgroupLauncher, bootProof, cohortProof, cohortRefOf, effectReport, forceStopCohort, releaseCohort, systemdUnits } from "./cohort.mjs";
import { AUTHORITY, PgroupLauncher, bootProof, cohortProof, cohortRefOf, effectReport, engineExitCohort, forceStopCohort, releaseCohort, systemdUnits } from "./cohort.mjs";
import { EngineLink } from "./engine.mjs";
import { DIALOG_METHODS, KNOWN_UNSHOWN, NOTIFY_METHODS, deltaBlocks, isFoldedUpdate, messageBlocks, partsOf, roleOf, toolResultText } from "./events.mjs";
import { LineSplitter, encodeLine, parseLine } from "./framing.mjs";
@@ -72,11 +72,13 @@ export const VERIFIED_OPERATIONS = Object.freeze(["observe", "prompt", "takeover
export const TEST_ENGINE = Symbol("conversation.test-engine");
const ENGINE_KEYS = new Set(["extraArgs", "cwd", "envKeys"]);
export const TIMEOUTS = Object.freeze({ ack: 5000, state: 5000, start: 5000, clear: 5000, abort: 10000, settle: 10000, grace: 1000, write: 5000, maxRounds: 3 });
export const TIMEOUTS = Object.freeze({ ack: 5000, state: 5000, start: 5000, clear: 5000, abort: 10000, settle: 10000, grace: 1000, write: 5000, maxRounds: 3, exitSettle: 2000, exitProof: 5000 });
const FINAL = new Set(["finished", "failed", "dispatch-refused", "delivery-unknown"]);
const STOP_IN_PROGRESS = new Set(["fenced", "cancelling", "stopping"]);
const STOP_NEXT = { fenced: ["cancelling", "stopping", "uncertain"], cancelling: ["stopping", "uncertain"], stopping: ["uncertain"], uncertain: [] };
// check.mjs `ending`: the binding follows only stops that end the engine.
const ending = (mode) => mode === "force-stop" || mode === "engine-exit";
const SEAL_BASIS = "seal: --no-extensions and no --extension argument (pi-pin.mjs SEAL_FLAGS)";
const DIALOG_REASON = "Pi dialogs are not answered in CHAT-03 (lead decision 30); shown disabled";
@@ -85,6 +87,16 @@ const isGen = (v) => Number.isInteger(v) && v >= 1 && v <= Number.MAX_SAFE_INTEG
const isId = (v) => typeof v === "string" && ID.test(v);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// `p`'s value, or `late` if `ms` pass first.
async function within(p, ms, late) {
let timer;
try {
return await Promise.race([p, new Promise((r) => (timer = setTimeout(() => r(late), ms)))]);
} finally {
clearTimeout(timer);
}
}
// The client request envelope, per operation (CHAT-01 schema `command`).
// Unknown operations pass the shape check and are refused by evaluate.
const SHAPES = {
@@ -1000,6 +1012,7 @@ export class Controller {
#onEnd(exec, link) {
if (this.#stale(exec, link) || exec.closing) return;
this.#transport(exec, "eof");
void this.#engineExit(exec).catch((e) => this.#internal("engine-exit", e));
}
#transport(exec, reason) {
@@ -1233,7 +1246,7 @@ export class Controller {
});
this.stops.set(s.id, s);
b.stop = s.id;
if (mode === "force-stop") b.state = "stopping";
if (ending(mode)) b.state = "stopping";
this.#admission();
if (mode !== "revocation") this.#emit("stopping", { stop: s.id });
this.#push({ kind: "stop", stop: s });
@@ -1244,7 +1257,7 @@ export class Controller {
#advanceStop(stop, state) {
if (!STOP_NEXT[stop.state]?.includes(state)) return false;
stop.state = state;
if (stop.mode === "force-stop") this.b.state = state === "uncertain" ? "uncertain" : "stopping";
if (ending(stop.mode)) this.b.state = state === "uncertain" ? "uncertain" : "stopping";
this.#push({ kind: "stop", stop });
this.#admission();
return true;
@@ -1496,21 +1509,8 @@ export class Controller {
}
async #escalate(stop, { confirmation = null, resumed = false } = {}) {
const b = this.b;
const rec = () => this.claim.record;
const fail = async (reason) => {
if (stop.state !== "superseded") this.#advanceStop(stop, "uncertain");
b.state = "uncertain";
this.closers.add("uncertain");
this.#admission();
this.#emit("uncertain", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: "force-stop", outcome: "uncertain", reason, resumed });
try {
await this.#claimAdvance({ state: "uncertain" });
} catch (err) {
this.#evidenceAdd("uncertain", { reason: "claim", error: String(err.code ?? err.message) });
}
};
const fail = (reason) => this.#endStopUncertain(stop, reason, { resumed });
try {
await this.#claimAdvance((r) => ({ state: "stopping", stop: { id: stop.id, confirmation, phaseStarted: resumed ? r.stop?.phaseStarted ?? null : null, record: stop } }));
} catch (err) {
@@ -1536,6 +1536,59 @@ export class Controller {
if (stop.state === "superseded") return undefined;
this.#advanceStop(stop, "stopping");
if (result.outcome !== "proven") return fail(result.reason ?? "cohort evidence unavailable");
return this.#stopProven(stop, result, { claimStop: rec().stop, fail, detail: { resumed } });
}
// Row 52, CHAT-01 `engine-exit`. EOF starts an observation; it proves
// nothing. The stop takes the one escalation slot (H10), so no force stop
// runs beside it, and it never supersedes a force stop in any state: a
// force stop's kill also ends output. Its observation only reads the
// cohort and has a deadline, so a shim that doesn't answer ends the stop
// `uncertain` and frees the slot for a client force stop. The claim stays
// `uncertain` from the transport loss until a proof is recorded: a restart
// finds an ordinary orphan, never an engine-exit stop to resume.
async #engineExit(exec) {
const b = this.b;
if (exec !== this.exec || !b || !this.claim?.record.engine || this.escalating || ending(this.stops.get(b.stop)?.mode)) return undefined;
const stop = this.#startStop("engine-exit", { requestId: null, connection: null, target: targetOf(b) });
this.escalating = stop.id;
try {
await this.#pause("engine-exit-recorded", { stop: stop.id });
const rec = this.claim.record, engine = rec.engine;
const result = await within(
engineExitCohort({ kind: engine.kind, unitName: rec.unitName, invocationId: rec.invocationId, shimSocket: rec.shim, pid: engine.pid, start: engine.start, settleMs: this.T.exitSettle }),
this.T.exitProof, { outcome: "unavailable", reason: "the engine-exit observation missed its deadline" },
);
const fail = (reason) => this.#endStopUncertain(stop, reason);
this.#advanceStop(stop, "stopping");
if (result.outcome !== "proven") return await fail(result.reason ?? "cohort evidence unavailable");
return await this.#stopProven(stop, result, { claimStop: { id: stop.id, confirmation: null, phaseStarted: null }, fail, detail: {} });
} finally {
if (this.escalating === stop.id) this.escalating = null;
}
}
// Ends a stop `uncertain`: the stop unless superseded, the binding,
// admission and the claim. The scope and its shim stay as evidence.
async #endStopUncertain(stop, reason, detail = {}) {
if (stop.state !== "superseded") this.#advanceStop(stop, "uncertain");
this.b.state = "uncertain";
this.closers.add("uncertain");
this.#admission();
this.#emit("uncertain", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: stop.mode, outcome: "uncertain", reason, ...detail });
try {
await this.#claimAdvance({ state: "uncertain" });
} catch (err) {
this.#evidenceAdd("uncertain", { reason: "claim", error: String(err.code ?? err.message) });
}
}
// A proven observation through the verifier to `stopped`, then the scope's
// release. Every path that can't record the proof ends at `fail`.
async #stopProven(stop, result, { claimStop, fail, detail }) {
const b = this.b;
const rec = () => this.claim.record;
if (!this.verifier) return fail("no verifier");
const tools = this.exec && this.exec.execution === b.execution ? this.exec.tools : this.orphanTools ?? new Map();
const proof = cohortProof({ binding: b, stop: stop.id, result });
@@ -1551,7 +1604,7 @@ export class Controller {
return fail(`session unreadable at proof: ${err.code ?? err.message}`);
}
try {
await this.#claimFinish({ state: "stopped", proof: { kind: "cohortProof", ref: proof.id, effects: effects.id }, leafAtProof: session.leaf, branchAtProof: session.branch, stop: { ...rec().stop, record: stop } });
await this.#claimFinish({ state: "stopped", proof: { kind: "cohortProof", ref: proof.id, effects: effects.id }, leafAtProof: session.leaf, branchAtProof: session.branch, stop: { ...claimStop, record: stop } });
} catch (err) {
return fail(`claim: ${err.code ?? err.message}`);
}
@@ -1565,10 +1618,10 @@ export class Controller {
this.#push({ kind: "stop", stop });
this.#push({ kind: "binding", binding: b });
this.#emit("stopped", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: "force-stop", outcome: "stopped", proofs: { cohort: proof.id, effects: effects.id }, resumed });
this.#evidenceAdd("stop", { stop: stop.id, mode: stop.mode, outcome: "stopped", proofs: { cohort: proof.id, effects: effects.id }, ...detail });
const proven = this.claim;
await this.#pause("scope-release", { stop: stop.id });
await this.#releaseScope("force-stop", proven);
await this.#releaseScope(stop.mode, proven);
return undefined;
}
@@ -1615,6 +1668,8 @@ export class Controller {
const b = this.b, p = this.stoppedProof;
if (!s || s.state !== "stopped" || b.stop !== s.id || !scopeMatch(s.target, targetOf(b)) || !p || !this.verifier) return false;
if (p.proof.id !== s.supervisorEvidence || p.effects.id !== s.effectsEvidence) return false;
// An engine exit is proven on the engine itself, never on an empty list.
if (s.mode === "engine-exit" && p.proof.members.length === 0) return false;
return this.verifier.cohort(p.proof, p.effects, { binding: b, stop: s.id, now: this.now(), epoch: p.epoch });
}
+6 -1
View File
@@ -67,9 +67,14 @@ const child = spawn(
{ stdio: [0, 1, 2] },
);
const enginePid = child.pid;
// Read while the engine lives: /proc has nothing for it once it is reaped.
// The exec chain keeps the PID and the start time.
const engineStart = startOf(enginePid);
let engineExit = null;
// The shim's wait. An engine-exit proof names the engine by these, with the
// reap time as its death time (row 52).
child.on("exit", (code, signal) => {
engineExit = { code, signal, at: new Date().toISOString() };
engineExit = { code, signal, at: new Date().toISOString(), pid: enginePid, startTicks: engineStart, boot: bootId };
});
// The engine holds the controller's pipes; the shim's copies would hide EOF.
closeSync(0);
+192 -13
View File
@@ -4,8 +4,8 @@
// skip when systemd user scopes are unavailable. K2 runs on the process-group
// fallback. K6–K9 and K16–K18 use the in-process fake, whose force stop is a
// fixture stand-in (see FakeLauncher). K19 launches /bin/sleep through
// ScopeLauncher with no controller. Controllers that must die run in
// ctrl-child.mjs.
// ScopeLauncher with no controller. E1–E6 (row 52) are the engine-exit stop
// on the scope fixtures. Controllers that must die run in ctrl-child.mjs.
import { test, after } from "node:test";
import assert from "node:assert/strict";
@@ -14,7 +14,7 @@ import { spawn, spawnSync } from "node:child_process";
import { dirname, join } from "node:path";
import { ClaimStore, FOREIGN_HOST, defaultHost, newClaimId, unitNameFor } from "../src/claim.mjs";
import { ConversationClient } from "../src/client.mjs";
import { AUTHORITY, PgroupLauncher, ScopeLauncher, forceStopCohort, releaseCohort, scopeAvailable, shimRequest, systemctlShow, systemdUnits } from "../src/cohort.mjs";
import { AUTHORITY, PgroupLauncher, ScopeLauncher, engineExitCohort, forceStopCohort, releaseCohort, scopeAvailable, shimRequest, systemctlShow, systemdUnits } from "../src/cohort.mjs";
import { Controller, ELIGIBILITY, TEST_ENGINE } from "../src/controller.mjs";
import { ENGINE_ENV, ENGINE_PIN_MISMATCH, engineEnv } from "../src/pi-pin.mjs";
import { FixtureVerifier, newId } from "../src/records.mjs";
@@ -104,14 +104,14 @@ function gate() {
// A controller in this process on a real engine process: the fake engine
// under ScopeLauncher ("scope") or PgroupLauncher ("pgroup").
async function live({ kind = "scope", barrier = null, verifier = new FixtureVerifier({ authorities: [AUTHORITY] }), launcher = null } = {}) {
async function live({ kind = "scope", barrier = null, verifier = new FixtureVerifier({ authorities: [AUTHORITY] }), launcher = null, timeouts = FAST } = {}) {
const fx = track(fixture());
const control = join(fx.base, "fake.sock");
const ctrl = new Controller({
fixtureRoot: fx.base, claimRoot: fx.claimRoot, socketDir: fx.socketDir, sessionFile: fx.sessionFile, seat: fx.seat,
launcher: launcher ?? (kind === "scope" ? new ScopeLauncher() : new PgroupLauncher()),
engine: { cwd: fx.proj }, [TEST_ENGINE]: { command: process.execPath, preArgs: [FAKE_PI], env: { ...process.env, FAKE_PI_CONTROL: control, FAKE_PI_LOG: join(fx.base, "fake.log") } },
verifier, units: kind === "scope" ? systemdUnits : noUnits, timeouts: FAST, barrier,
verifier, units: kind === "scope" ? systemdUnits : noUnits, timeouts, barrier,
});
await ctrl.start();
assert.equal(ctrl.binding.state, "active", JSON.stringify(ctrl.evidence.uncertain));
@@ -855,28 +855,207 @@ test("R3: no release after an unavailable stop or an unverified proof: the scope
// A normal engine exit leaves the shim and the scope: the shim exits only on
// `release`. The controller doesn't release then: the binding is uncertain
// and the scope is what a force stop reads. The proven stop releases it.
test("R4: after a normal engine exit the scope stays until a proven force stop releases it", NEEDS_SCOPE, async () => {
// ---- engine exit (row 52, CHAT-01 `engine-exit`) ----------------------------
// A confirmation issued and confirmed now, not yet used.
async function confirmation(c, op) {
const issued = await c.request("issue-confirmation", { operationToConfirm: op });
assert.equal(issued.outcome, "confirmation-issued", JSON.stringify(issued));
const answered = await c.request("answer-confirmation", { confirmation: issued.data.confirmation.id, answer: "confirm" });
assert.equal(answered.outcome, "confirmation-confirmed", JSON.stringify(answered));
return issued.data.confirmation.id;
}
const exitStops = (h) => [...h.ctrl.stops.values()].filter((s) => s.mode === "engine-exit");
test("E1: a natural engine exit with no member left: an engine-exit stop, stopped on a proof naming the engine, and the scope released", NEEDS_SCOPE, async () => {
const g = gate();
const h = await live({ barrier: g.barrier });
const unit = h.rec().unitName, shim = h.rec().shim, engine = h.rec().engine;
try {
const before = h.ctrl.binding.stop;
const early = await confirmation(h.c, "recover");
g.hold("scope-release");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("scope-release");
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
const [stop] = exitStops(h);
assert.equal(h.ctrl.binding.stop, stop.id);
assert.deepEqual([stop.state, stop.request, stop.supersedes], ["stopped", null, before]);
// The proof names the engine by the shim's wait: PID, start ticks, boot
// and reap time, read while the shim still answers.
const exit = (await shimRequest(shim, "hello")).engineExit;
assert.equal(exit.code, 0);
assert.deepEqual(h.ctrl.stoppedProof.proof.members, [{ pid: engine.pid, boot: exit.boot, startTicks: Number(engine.start), terminatedAt: exit.at }]);
assert.deepEqual(stoppedOnProof(h.fx).map((r) => [r.stop.id, r.stop.record.mode]), [[stop.id, "engine-exit"], [stop.id, "engine-exit"]]);
g.release("scope-release");
await until(() => h.ctrl.evidence.releases.length > 0, 8000, "the release");
assert.deepEqual(h.ctrl.evidence.releases.map((r) => [r.why, r.outcome, r.unit ?? null]), [["engine-exit", "released", "absent"]]);
assert.ok(unitGone(unit));
// Recover takes its own confirmation, bound to this stop (K6); one issued
// before the engine exited names the old stop.
assert.equal((await h.c.request("recover", { stop: stop.id, confirmation: early })).refusal, "confirmation");
const rec = await h.c.confirmed("recover", { stop: stop.id });
assert.equal(rec.outcome, "recovery-eligible", JSON.stringify(rec));
await h.ctrl.release(rec.data.eligibility);
} finally {
g.release("scope-release");
await h.close();
}
});
test("E2: an engine exit with a tool child alive: the stop ends uncertain and nothing is released; a force stop then ends it", NEEDS_SCOPE, async () => {
const h = await live();
const unit = h.rec().unitName, shim = h.rec().shim;
try {
const child = await childOf(h, {});
const early = await confirmation(h.c, "force-stop");
assert.equal((await h.fake.call("exit")).ok, true);
await until(() => h.ctrl.binding.state === "uncertain", 8000, "the binding to see the exit");
await until(async () => (await shimRequest(shim, "hello")).engineExit !== null, 4000, "the shim to see the exit");
assert.equal((await shimRequest(shim, "hello")).engineExit.code, 0);
await tick(200);
assert.ok(unitActive(unit), "the shim keeps the scope after the engine exits");
await until(() => exitStops(h)[0]?.state === "uncertain", 8000, "the engine-exit stop to end");
assert.equal(h.ctrl.binding.state, "uncertain");
assert.deepEqual(h.ctrl.evidence.stops.map((s) => [s.mode, s.outcome]), [["engine-exit", "uncertain"]]);
assert.deepEqual(h.ctrl.evidence.releases, []);
assert.ok(unitActive(unit));
assert.ok(alive(child));
assert.deepEqual(await memberPids(shim), [child]);
assert.ok(!h.ctrl.stoppedProof);
// A force-stop confirmation issued before the engine exited is stale
// (H17): it names the stop the engine-exit stop superseded.
assert.equal((await h.c.request("force-stop", { confirmation: early })).refusal, "confirmation");
await forceStop(h);
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
assert.deepEqual(h.ctrl.stoppedProof.proof.members, []);
assert.equal(h.ctrl.stops.get(h.ctrl.binding.stop).supersedes, exitStops(h)[0].id);
await until(() => !alive(child), 4000, "the child to die");
await until(() => h.ctrl.evidence.releases.length > 0, 8000, "the release");
assert.equal(h.ctrl.evidence.releases[0].outcome, "released");
assert.deepEqual(h.ctrl.evidence.releases.map((r) => [r.why, r.outcome]), [["force-stop", "released"]]);
await until(() => unitGone(unit), 8000, "the scope to go");
} finally {
await h.close();
}
});
test("E3: an engine exit and a force stop take one escalation slot, in either order", NEEDS_SCOPE, async () => {
{
const g = gate();
const h = await live({ barrier: g.barrier });
try {
g.hold("engine-exit-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("engine-exit-recorded");
assert.equal(h.ctrl.binding.state, "stopping");
assert.equal((await h.c.confirmed("force-stop")).refusal, "fenced");
g.release("engine-exit-recorded");
await until(() => h.ctrl.binding.state === "stopped", 8000, "the engine-exit proof");
assert.deepEqual(h.ctrl.evidence.stops.map((s) => [s.mode, s.outcome]), [["engine-exit", "stopped"]]);
} finally {
g.release("engine-exit-recorded");
await h.close();
}
}
{
const g = gate();
const h = await live({ barrier: g.barrier });
try {
// A force stop already running: the EOF its kill causes starts nothing.
g.hold("force-stop-recorded");
const fs = await h.c.confirmed("force-stop");
assert.equal(fs.outcome, "force-stop-fenced", JSON.stringify(fs));
await g.waitHeld("force-stop-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await tick(300);
assert.deepEqual(exitStops(h), []);
g.release("force-stop-recorded");
await until(() => ["stopped", "uncertain"].includes(h.ctrl.binding.state), 20000, "the force stop to end");
assert.equal(h.ctrl.binding.stop, fs.stop.id);
assert.deepEqual(exitStops(h), []);
assert.deepEqual(h.ctrl.evidence.stops.map((s) => s.mode), ["force-stop"]);
} finally {
g.release("force-stop-recorded");
await h.close();
}
}
});
test("E4: a shim that doesn't answer: the engine-exit stop misses its deadline, ends uncertain and frees the slot", NEEDS_SCOPE, async () => {
const h = await live({ timeouts: { ...FAST, exitSettle: 200, exitProof: 500 } });
const unit = h.rec().unitName, shim = h.rec().shim;
const shimPid = (await shimRequest(shim, "hello")).shimPid;
try {
process.kill(shimPid, "SIGSTOP");
assert.equal((await h.fake.call("exit")).ok, true);
await until(() => exitStops(h)[0]?.state === "uncertain", 8000, "the engine-exit deadline");
assert.equal(h.ctrl.binding.state, "uncertain");
assert.match(h.ctrl.evidence.stops.at(-1).reason, /deadline/);
assert.equal(h.ctrl.escalating, null);
process.kill(shimPid, "SIGCONT");
await forceStop(h);
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
// The force stop's proof is unchanged by row 52: the engine had already
// left the cohort, so it lists no member.
assert.deepEqual(h.ctrl.stoppedProof.proof.members, []);
await until(() => unitGone(unit), 8000, "the scope to go");
} finally {
try {
process.kill(shimPid, "SIGCONT");
} catch {
// gone
}
await h.close();
}
});
test("E5: an engine exit after a force stop ended uncertain starts no engine-exit stop; the force stop stays current", NEEDS_SCOPE, async () => {
const h = await live();
const advance = h.ctrl.store.advance.bind(h.ctrl.store);
let failed = false;
// The force stop's `stopping` claim write fails once, so it ends uncertain
// before it closes the execution and the engine's EOF still arrives.
h.ctrl.store.advance = async (claim, patch) => {
if (!failed && patch?.state === "stopping") {
failed = true;
throw Object.assign(new Error("injected"), { code: "E5-INJECTED" });
}
return advance(claim, patch);
};
try {
const stop = await forceStop(h);
assert.equal(h.ctrl.binding.state, "uncertain");
assert.match(h.ctrl.evidence.stops.at(-1).reason, /E5-INJECTED/);
assert.equal(h.ctrl.escalating, null);
assert.equal((await h.fake.call("exit")).ok, true);
await until(() => h.ctrl.evidence.uncertain.some((u) => u.reason === "eof"), 4000, "the EOF");
await tick(300);
assert.deepEqual(exitStops(h), []);
assert.equal(h.ctrl.binding.stop, stop);
assert.equal(h.ctrl.binding.state, "uncertain");
assert.deepEqual(h.ctrl.evidence.stops.map((s) => s.mode), ["force-stop"]);
} finally {
h.ctrl.store.advance = advance;
await h.close();
}
});
test("E6: the engine-exit observation refuses a reaped process that isn't the recorded engine: another PID or start ticks", NEEDS_SCOPE, async () => {
const g = gate();
const h = await live({ barrier: g.barrier });
try {
g.hold("engine-exit-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("engine-exit-recorded");
const rec = h.rec(), engine = rec.engine;
const observe = (over) => engineExitCohort({ kind: "scope", unitName: rec.unitName, invocationId: rec.invocationId, shimSocket: rec.shim, pid: engine.pid, start: engine.start, settleMs: 200, ...over });
for (const over of [{ pid: engine.pid + 1 }, { start: String(Number(engine.start) + 1) }]) {
const r = await observe(over);
assert.deepEqual([r.outcome, r.reason], ["unavailable", "the shim reaped a process that isn't the recorded engine"], JSON.stringify(over));
}
assert.equal((await observe({})).outcome, "proven");
g.release("engine-exit-recorded");
await until(() => h.ctrl.binding.state === "stopped", 8000, "the engine-exit proof");
} finally {
g.release("engine-exit-recorded");
await h.close();
}
});
test("R5: the harness finds a running shim and reaps it, and never signals a shim outside a mosaic-chat- scope (Filbert N1 and N2 on #1533)", NEEDS_SCOPE, async () => {
const fx = track(fixture());
mkdirSync(fx.socketDir, { recursive: true });