feat(conversation): release at engine exit through a proven engine-exit stop (#1538, row 52)

The controller now starts a CHAT-01 `engine-exit` stop when the engine
exits on its own. engineExitCohort proves the cohort from the shim's
recorded engine identity (pid, start ticks, boot) and an empty member
list, under a 5 s deadline that frees the escalation slot. A proven stop
releases the scope; anything else ends `uncertain`. close() no longer
signals a stopped binding.

Dewey built it. Filbert (27087) and Darkwing (27088) approved round 1 on
CHAT-01 cc83ee4f and manifest deac7434 (6 files). Sage's gate on
c92cfb8f plus the candidate: conversation 182/0, webui 22/0,
control-board 124/0, every scripts/test-*.sh 0 failed (task 98/0),
chat-01 PASS.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-10 03:31:39 -05:00
co-authored by Claude Opus 5.5
parent c92cfb8faf
commit 0aa5e7b51b
7 changed files with 598 additions and 39 deletions
@@ -0,0 +1,6 @@
774224cf6a105ff80d3e71637bd84bf68cb2da447b3a4f0fec25e021b7401f46 agents/dewey/work/queue-52/evidence.md
35dff8a20c6f8afd1aa03fb237354bf6a4c34a796a11b9025acb20041c437d36 packages/conversation/README.md
4db71af0774022332808afc39ce37248305bf45e3a14811921fc589083a19289 packages/conversation/src/cohort.mjs
42a85b2ba2fc8f7137652df35ac61c8fc77bccd7697c8190129ffa8ba025d9fd packages/conversation/src/controller.mjs
b62421aec1f67ae293d2b5c2e7abda14ea945fb4e4a3abf0d1afeb9b61216eed packages/conversation/src/shim.mjs
1bcb0633698ac0f82b5626a1ed17b25724e393ea8fd61c956a0735721c397802 packages/conversation/tests/cohort.test.mjs
+245
View File
@@ -0,0 +1,245 @@
# Row 52 (#1538): CHAT-01 engine-exit stop and release at engine exit
Dewey, 2026-10-10. Brief: `docs/plans/2026-10-10_cohort-release-follow-ups.md`,
section "CHAT-01 engine-exit stop and release at engine exit" (rev 297).
Base 945440db (row 51 landed). The change has two parts, CHAT-01 first in
its own commit:
- CHAT-01 (commit cc83ee4f, not pushed): the four `docs/plans/chat-01/` files.
- Conversation (candidate, uncommitted): `src/cohort.mjs`,
`src/controller.mjs`, `src/shim.mjs`, `tests/cohort.test.mjs` and the
package README. `tests/ctrl-child.mjs` is unchanged.
## CHAT-01 hashes (each approval names all four)
```
a41fc4e2edb11371c275e3167774c162254e1457fd737121630dedd665431889 docs/plans/chat-01/README.md
ccceaf2653b279f5890748b40d16fae7493f199a33cecee035f82b92e80243af docs/plans/chat-01/check.mjs
941675de949c52c7fe7c4b7bcce3aff48bb26ccc1120e090574c5d25f9d9e22e docs/plans/chat-01/contracts.schema.json
c75c2b9f731bb70d0e033e8aa56f2c28accfa09384ec964fd3ecbd45974b2a7d docs/plans/chat-01/fixtures.json
```
`node docs/plans/chat-01/check.mjs`: 100 shape cases, 76 reference cases,
22 lifecycle sequences, PASS. `node docs/plans/chat-00/check.mjs`: 48 checks
PASS.
## Darkwing's points 1–4
| Point | Model (`check.mjs`) | Conversation |
|---|---|---|
| 1. Never supersede an unfinished force stop | `engine-exit` returns `stop-owned` while the current stop is a force stop (any state) or an engine exit. Sequence `engine-exit-refused-during-force-stop` walks all four force-stop states. | `#engineExit` starts nothing while `escalating` is held or the current stop is a force stop or engine exit: a running force stop (E3, second half) and one that ended `uncertain` before it closed the execution (E5). |
| 2. Binding follows at lines 121, 334, 337 | All three use `ending(mode)` (force stop or engine exit). Sequence `engine-exit-binding-follows-stop`: `stopping` at the fence and on each advance, `uncertain` with the stop, `stopped` only with it. | `#startStop` and `#advanceStop` use the same `ending` (E3 sees `stopping` at `engine-exit-recorded`). |
| 3. The proof names the engine; not `members: []` | `stopped(w, s)` refuses an `engine-exit` proof with no member. The empty-list case is in `engine-exit-live-unreadable-or-empty-cohort-uncertain`. | `engineExitCohort` returns the engine as the one member: PID and start ticks the shim recorded, boot, and the shim's reap time as `terminatedAt`. `#stopped` also refuses an engine-exit proof with no member. E1 checks the members exactly; E6 checks that another PID or start ticks is refused. |
| 4. The proof's deadline frees the slot | The model leaves the slot to the implementation (README says so). | The observation runs under `within(…, exitProof)`. A miss ends the stop `uncertain`, and `finally` frees `escalating`. E4 SIGSTOPs the shim: the stop ends `uncertain` at the deadline, `escalating` is `null`, and a client force stop then proves and releases. |
### Engine identity comes from a shim op
Sage's condition: if the engine identity can't come from a shim op, stop
before relying on `members: []`. It comes from one. The shim reads the
engine's start ticks from `/proc` right after spawn (the exec chain keeps
the PID and start time), and its `exit` handler records `{ code, signal,
at, pid, startTicks, boot }`. `hello` returns that as `engineExit`.
`engineExitCohort` refuses, as `unavailable`, an `engineExit` whose PID or
start ticks differ from the claim's recorded engine, whose start ticks
aren't a positive integer, or whose boot differs from the shim's. Nothing
relies on an empty member list: the force stop's proof (R4, now in E4) is
unchanged and still lists no member when the engine has already gone.
## CHAT-01 change
- Stop mode `engine-exit` (schema enum). The server starts it like a
revocation fence: `request: null`, no confirmation.
- `server(w, 'engine-exit')`: refuses `stop-owned` under a force stop or
another engine exit; otherwise `startStop` (closes admission, recovers
queued drafts, marks pending decisions uncertain, supersedes an
unfinished interrupt or revocation, emits `stopping`). Dispatched or
acknowledged input moves to `delivery-unknown`; `working` input keeps its
state.
- `confirm-stopped` accepts it under the same `stopped(w, s)` check as a
force stop, plus at least one member.
- Recover after it needs its own confirmation bound to the stop (K6, K17,
K18): an unissued one, one issued but not confirmed, and the binding's
stop check are all in `engine-exit-empty-cohort-stopped`.
- Fixtures: `valid-engine-exit-stop`, `refuse-stop-mode-eof`, and five
sequences: `engine-exit-empty-cohort-stopped`,
`engine-exit-live-unreadable-or-empty-cohort-uncertain`,
`engine-exit-stale-confirmation-dispatched-receipt`,
`engine-exit-refused-during-force-stop`, `engine-exit-binding-follows-stop`.
- README: a paragraph under "Control loss, approvals and stopping", a
pointer near the top, and the R3 counts. "SIGTERM, EOF, abort
acknowledgment or idle does not prove death" stays; K15 and K2 are
unchanged.
### Model mutants (`check.mjs` on a scratch copy)
| Mutant | Change | Result |
|---|---|---|
| norefuse | `engine-exit` never refuses `stop-owned` | killed |
| optionB | refuses only while the force stop is unfinished | killed |
| bind121 | `startStop` moves the binding only for a force stop | killed |
| bind334 | `advance-stop` the same | killed |
| bind337 | `confirm-stopped` accepts only a force stop | killed |
| emptyok | no member-count check for an engine exit | killed |
| noreceipt | dispatched input keeps its state | killed |
| nostale | no current-stop check on a confirmation | killed |
## Conversation change
- `#onEnd` (EOF): after `#transport` (which settles in-flight input
`delivery-unknown`/`transport-unknown` as before, H19, N18), starts
`#engineExit`.
- `#engineExit`: only for the current execution with a recorded engine,
no held slot and no current force stop or engine exit. Starts the
`engine-exit` stop, takes `escalating`, pauses at `engine-exit-recorded`
(test barrier), runs `engineExitCohort` under the `exitProof` deadline
(5000 ms; `exitSettle` 2000 ms for EOF before the reap), then goes to
`uncertain` or through `#stopProven`.
- `#endStopUncertain` and `#stopProven` are `#escalate`'s former `fail` and
proof tail, shared. The force-stop path is unchanged: same order, same
claim writes, `resumed` still in its evidence. Evidence and the release
now carry `stop.mode` instead of a literal `force-stop`.
- `engineExitCohort` (cohort.mjs): reads only (`hello`, `events`,
`members`), never freezes or kills. Proven when the shim has reaped the
recorded engine, `engine` reads `populated 0` and no member is listed.
- The claim: no `stopping` claim write for an engine exit. The claim goes
`uncertain` at EOF (`#transport`, as before) and `stopped` only through
`#claimFinish` with the proof. A restart in between finds an ordinary
`uncertain` orphan, never an engine-exit stop to resume.
## Tests (`cohort.test.mjs`, needs a systemd user manager)
R4 is folded into E4; E1–E6 are new.
- E1: the engine exits with no other member. One `engine-exit` stop,
`request: null`, superseding the prior stop, `stopped`; the proof's
members are exactly the engine (PID, boot, start ticks, reap time from
the shim's `hello`); both claim keys stopped on it; the release is
`engine-exit`/`released`/`absent` and the unit is gone. A recover
confirmation issued before the exit is refused; a fresh one is
`recovery-eligible`.
- E2: the engine exits while a tool child runs. The stop ends `uncertain`
(evidence `engine-exit`/`uncertain`), no release, the unit and the child live, no proof. A force-stop
confirmation issued before the exit is refused (H17). A fresh force stop
supersedes the engine-exit stop, kills the child, proves and releases.
- E3: one slot, both orders. The engine-exit stop held at
`engine-exit-recorded`: a client force stop is refused `fenced`, and the
engine exit then proves. A force stop held at `force-stop-recorded`: the
engine's EOF starts no engine-exit stop, and only the force stop is in
the evidence.
- E4: the shim SIGSTOPped. The engine-exit stop misses its deadline, ends
`uncertain` (reason names the deadline), `escalating` is `null`; after
SIGCONT a force stop proves with no member (the R4 assertion) and the
scope goes.
- E5: a force stop whose `stopping` claim write fails once ends `uncertain`
before it closes the execution, so the engine's later EOF still reaches
`#onEnd`. No engine-exit stop starts; the force stop stays current
(point 1 after the slot is free).
- E6: held at `engine-exit-recorded`, `engineExitCohort` with another PID
or other start ticks is `unavailable` ("the shim reaped a process that
isn't the recorded engine"); the recorded identity is `proven`, and the
controller's stop then proves.
Conversation suite on the draft at 945440db: 182/182 (was 177: R4 out,
E1–E6 in).
## Mutation check
Twenty mutants on a fresh copy of the candidate, full conversation suite
each (`~/dewey-scratch/r52/mut-tools`). Run 1 (07:11–07:33Z) was on the
tests before E5, E6 and the E2 evidence check, and left `noowncheck`,
`noident` and `evmode` alive with the five equivalents. I added those
tests and ran all twenty again (run 2, 07:33–07:54Z). Run 2 base:
182/182. No shim, engine or `mosaic-chat-*` unit was left after any run.
| Mutant | Change | Run 2 | Killed by |
|---|---|---|---|
| noexit | `#onEnd` doesn't start `#engineExit` | killed | E1, E2, E3, E4, E6 |
| noslottake | the engine-exit stop doesn't take `escalating` | killed | E3 |
| nofree | `finally` doesn't free the slot | killed | E2, E4 |
| noslotcheck | EOF ignores a held slot | survives, equivalent | – |
| noowncheck | EOF ignores a current force stop or engine exit | killed | E5 |
| nostopcheck | both checks gone | killed | E3, E5 |
| nodeadline | the observation has no deadline | killed | E4 |
| bindstart | `#startStop` moves the binding only for a force stop | killed | E3 |
| bindadvance | `#advanceStop` the same | survives, equivalent | – |
| emptyguard | `#stopped` accepts an engine-exit proof with no member | survives, equivalent alone | – |
| emptyproof | the proof lists no member | killed | E1 |
| emptyboth | both: an empty-list proof that verifies | killed | E1 |
| noident | no PID or integer check on the shim's `engineExit` | killed | E6 |
| nopopulated | a populated `engine` cgroup proves | survives, equivalent | – |
| nomembers | a listed member proves | survives, equivalent | – |
| nolive | neither is read | killed | E2 |
| shimstart | the shim keeps no start ticks | killed | E1, E3, E6 |
| relwhy | the release is recorded as a force stop's | killed | E1 |
| evmode | uncertain evidence names a force stop | killed | E2 |
Why the five survivors are equivalent:
- noslotcheck: `escalating` is set only right after `#startStop` made a
force stop or engine exit the current stop, and the binding is then
`stopping`. Interrupt and revocation need `b.state === "active"`
(controller.mjs `#interruptLocked`, the revocation in the connection
close), so no other stop replaces it while the slot is held. `ending(…)`
covers every state the slot check covers. The check stays as a guard.
- bindadvance: the binding is already `stopping` from `#startStop`;
`#uncertain` keeps `stopping`; `#endStopUncertain` and `#stopProven`
set the binding themselves. The `ending` in `#advanceStop` stays to match
the model's line 334.
- emptyguard: a proven `engineExitCohort` always lists the engine, so
`#stopped`'s guard is reachable only through a second fault. emptyproof
and emptyboth test point 3.
- nopopulated, nomembers: the shim's `members` walks `engine` and its
descendants, so `populated 1` with an empty list (or `populated 0` with
a member) needs a process to start or exit between the two reads. Both
reads stay; each guards the other.
Killing tests are from each mutant's `failing tests:` list. Run 1 and run 2
tables: `~/dewey-scratch/r52/mut-tools/results-run1.txt` and
`results-run2.txt`.
## Gate
Run in a scratch worktree at cc83ee4f with the candidate applied
(`~/dewey-scratch/r52/gate/candidate.diff`), 2026-10-10T07:56:23Z to
08:01:22Z, sequential, each output kept.
| Suite | rc | Result |
|---|---|---|
| `node docs/plans/chat-01/check.mjs` | 0 | PASS: 100 shape, 76 reference, 22 lifecycle |
| conversation | 0 | 182 pass, 0 fail |
| webui | 0 | 22 pass, 0 fail |
| control-board | 0 | 124 pass, 0 fail |
| `test-auth.sh` | 0 | 15 passed, 0 failed |
| `test-conductor.sh` | 0 | 17 passed, 0 failed |
| `test-config.sh` | 0 | 24 passed, 0 failed |
| `test-discord.sh` | 0 | 66 passed, 0 failed |
| `test-extension-package.sh` | 0 | 18 passed, 0 failed |
| `test-foundation.sh` | 0 | 44 passed, 0 failed |
| `test-queue.sh` | 0 | 148 node pass, suite 27 passed, 0 failed; `queue verify` and `render --check` skipped (not the canonical root) |
| `test-release.sh` | 0 | 14 passed, 0 failed |
| `test-task.sh` | 0 | 98 passed, 0 failed |
The `test-queue.sh` skip is that suite's own guard for a non-canonical
checkout; Sage's rerun in the canonical checkout covers it. After the
gate and the mutant runs no `mosaic-chat-*` unit is listed
(`systemctl --user list-units --all 'mosaic-chat-*'`) and `core.hooksPath`
is unset.
## Not tested, follow-ups
Nothing here blocks the row; I'd file them together if Sage wants them.
- The boot comparison in `engineExitCohort` (`exit.boot !== hello.boot`)
is untested: both come from the same shim process, so only a faked
`hello` reaches it.
- An unreadable `engine/cgroup.events` or a failed `members` during the
engine-exit observation (the K15 shapes) is untested for this path. The
tests reach `unavailable` only through the deadline (E4).
- On the process-group fallback a natural exit now starts an
`engine-exit` stop that ends `uncertain` at once ("a pgroup cohort
can't be enumerated completely"). The binding ends `uncertain` as it
did before row 52, but no test runs it.
- The five equivalent mutants above are guards I kept on purpose; no
change proposed.
- Row 51's untested `still listed` retry and the noseat and recafteracq
survivors are #1539, not repeated here.
+27 -4
View File
@@ -378,9 +378,10 @@ can disagree with it.
unavailable, or a proof the verifier rejects) nothing is released: the unavailable, or a proof the verifier rejects) nothing is released: the
scope and its shim are what a later force stop reads, and a collected scope and its shim are what a later force stop reads, and a collected
scope would be an absent observation, not proof that the cohort ended scope would be an absent observation, not proof that the cohort ended
(R3). An engine that exits on its own leaves the shim and scope up and (R3). An engine that exits on its own ends in an `engine-exit` stop
the binding `uncertain`; a proven force stop on the empty cohort then (below), and only a proven one releases the scope (E1). A force stop on
releases them (R4). A controller killed between recording the stop and a cohort the engine already left still proves on an empty member list
(E4). A controller killed between recording the stop and
releasing leaves the scope; the restart's `start` finds the claim stopped releasing leaves the scope; the restart's `start` finds the claim stopped
on a cohort proof with its unit still listed and releases it (R7, R8). A on a cohort proof with its unit still listed and releases it (R7, R8). A
release that ended `unavailable` or `still listed` is tried again by the release that ended `unavailable` or `still listed` is tried again by the
@@ -393,6 +394,28 @@ can disagree with it.
engine PID, even if the proof stops verifying later: the proof showed engine PID, even if the proof stops verifying later: the proof showed
every member ended, so the recorded PID may belong to another process every member ended, so the recorded PID may belong to another process
(R10, R11). (R10, R11).
- **Engine exit** (row 52, CHAT-01 `engine-exit`). End of output starts an
`engine-exit` stop: the server starts it with no request and no
confirmation, since nothing is signalled. It closes admission, takes the
one escalation slot (H10) and supersedes the current stop, so a
force-stop confirmation issued before it is refused (H17). It starts
nothing while a force stop holds the slot or is the current stop, even
one that ended `uncertain` (E3, E5), and a client force stop during it is
refused `fenced` (E3). The observation
only reads the scope: `hello`, `events` and `members`, with no freeze
or kill. It is proven when the shim reports that it reaped the recorded
engine (PID, start ticks and boot; E6) and `engine` reads `populated 0`
with no member listed. The proof's one member is the engine, with the reap
time as its death time; a proof with an empty member list never
verifies an `engine-exit` stop. A proven stop goes through the verifier
to `stopped` and releases the scope like a force stop (E1). A live
member ends the stop `uncertain` and releases nothing; a confirmed force
stop then ends the cohort (E2). Any unavailable observation ends it
`uncertain` the same way. The
observation has a deadline (`exitProof`): a shim that doesn't answer
ends the stop `uncertain` and frees the slot for a force stop (E4).
Recover after an `engine-exit` stop needs its own confirmation, bound to
that stop (E1).
- **Confirmations** bind the target, operation and stop, and are consumed on - **Confirmations** bind the target, operation and stop, and are consumed on
use (K6). One issued before the stop changed is refused (H17). use (K6). One issued before the stop changed is refused (H17).
- **Recovery.** A confirmed `recover` after a proven stop returns a - **Recovery.** A confirmed `recover` after a proven stop returns a
@@ -573,7 +596,7 @@ no prompt:
| `claim.test.mjs` | W1–W17, W20, G1–G3: the writer claim, crash barriers, the guard | | `claim.test.mjs` | W1–W17, W20, G1–G3: the writer claim, crash barriers, the guard |
| `races.test.mjs` | H1–H4, H9–H23: takeover, Interrupt and force stop, retries, incarnations | | `races.test.mjs` | H1–H4, H9–H23: takeover, Interrupt and force stop, retries, incarnations |
| `turns.test.mjs` | N1–N25, N24b: the turn tracker against the fake engine's Pi behaviors; the engine seal and environment | | `turns.test.mjs` | N1–N25, N24b: the turn tracker against the fake engine's Pi behaviors; the engine seal and environment |
| `cohort.test.mjs` | K1–K19, R1–R12: scopes, force stop, proofs, recovery, eligibility, the scope's environment, scope release and its retry after a crash (needs a systemd user manager) | | `cohort.test.mjs` | K1–K19, R1–R3, R5–R12, E1–E6: scopes, force stop, proofs, recovery, eligibility, the scope's environment, scope release and its retry after a crash, the engine-exit stop (needs a systemd user manager) |
| `flows.test.mjs` | S1–S7, P3, E1–E7, the terminal, and a CHAT-01 schema check of every record produced | | `flows.test.mjs` | S1–S7, P3, E1–E7, the terminal, and a CHAT-01 schema check of every record produced |
| `smoke.test.mjs` | the pinned Pi binary, as above | | `smoke.test.mjs` | the pinned Pi binary, as above |
+46
View File
@@ -17,6 +17,10 @@
// membership complete. Anything unavailable ends the stop `uncertain`. The // membership complete. Anything unavailable ends the stop `uncertain`. The
// scope stays up after the stop either way; `releaseCohort` ends it once the // scope stays up after the stop either way; `releaseCohort` ends it once the
// controller has recorded a proven stop. // controller has recorded a proven stop.
//
// Engine exit (row 52): at EOF the controller reads the cohort without
// signalling it. The shim reports the engine's PID, start ticks and reap time
// from its wait; `engine` empty with that engine reaped is the proof.
import { spawn, spawnSync } from "node:child_process"; import { spawn, spawnSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs"; import { existsSync, readFileSync } from "node:fs";
@@ -204,6 +208,48 @@ export async function forceStopCohort({ kind, unitName, invocationId, shimSocket
}; };
} }
// Observes an engine that exited on its own (row 52, CHAT-01 `engine-exit`).
// It reads only: `hello`, `events` and `members`; nothing is frozen or
// signalled. The cohort is proven ended when the shim has reaped the recorded
// engine (its PID and start ticks) and `engine` reads `populated 0` with no
// member listed. The proof's one member is the engine, dead at the shim's
// wait; an empty list is never the proof. EOF can come before the reap, so it
// reads again until `settleMs` pass. Anything else is `unavailable`. The
// caller bounds the whole observation with its own deadline.
export async function engineExitCohort({ kind, unitName, invocationId, shimSocket, pid, start, settleMs = 2000 }) {
if (kind !== "scope") return { outcome: "unavailable", reason: `a ${kind} cohort can't be enumerated completely` };
const show = systemctlShow(unitName);
if (!show || !invocationId || show.invocationId !== invocationId) {
return { outcome: "unavailable", reason: `invocation ID mismatch or unreadable (recorded ${invocationId}, found ${show?.invocationId ?? "none"})` };
}
const end = Date.now() + settleMs;
for (;;) {
const hello = await shimRequest(shimSocket, "hello");
if (!hello.ok) return { outcome: "unavailable", reason: hello.unavailable };
if (hello.invocationId !== invocationId || hello.scope !== show.controlGroup) return { outcome: "unavailable", reason: "the shim does not answer for the recorded scope" };
const exit = hello.engineExit;
if (exit) {
if (exit.pid !== pid || !Number.isInteger(exit.startTicks) || exit.startTicks < 1 || String(exit.startTicks) !== String(start) || exit.boot !== hello.boot) {
return { outcome: "unavailable", reason: "the shim reaped a process that isn't the recorded engine" };
}
const e = await shimRequest(shimSocket, "events");
if (!e.ok) return { outcome: "unavailable", reason: e.unavailable };
if (e.populated === 0) {
const listed = await shimRequest(shimSocket, "members");
if (!listed.ok) return { outcome: "unavailable", reason: listed.unavailable };
if (listed.members.length === 0) {
return {
outcome: "proven", membershipComplete: true, epoch: invocationId, observedAt: new Date().toISOString(), boot: hello.boot,
members: [{ pid, boot: exit.boot, startTicks: exit.startTicks, terminatedAt: exit.at }],
};
}
}
}
if (Date.now() >= end) return { outcome: "unavailable", reason: exit ? "the engine cgroup still has a member" : "the shim has not reaped the engine" };
await sleep(20);
}
}
// Ends a scope whose cohort is proven stopped (#1536). The shim exits on // Ends a scope whose cohort is proven stopped (#1536). The shim exits on
// `release` only while `engine` reads `populated 0`, and systemd then // `release` only while `engine` reads `populated 0`, and systemd then
// collects the empty scope. Call it only after a `proven` stop whose claim // collects the empty scope. Call it only after a `proven` stop whose claim
+76 -21
View File
@@ -23,7 +23,7 @@ import { fileURLToPath } from "node:url";
import { randomBytes } from "node:crypto"; import { randomBytes } from "node:crypto";
import { processStart } from "../../discord/src/journal.mjs"; import { processStart } from "../../discord/src/journal.mjs";
import { ALREADY_ACTIVE, ClaimStore, FOREIGN_HOST, UNSAFE_REPLACEMENT, machineId, seatKey, sessionKey } from "./claim.mjs"; import { ALREADY_ACTIVE, ClaimStore, FOREIGN_HOST, UNSAFE_REPLACEMENT, machineId, seatKey, sessionKey } from "./claim.mjs";
import { AUTHORITY, PgroupLauncher, bootProof, cohortProof, cohortRefOf, effectReport, forceStopCohort, releaseCohort, systemdUnits } from "./cohort.mjs"; import { AUTHORITY, PgroupLauncher, bootProof, cohortProof, cohortRefOf, effectReport, engineExitCohort, forceStopCohort, releaseCohort, systemdUnits } from "./cohort.mjs";
import { EngineLink } from "./engine.mjs"; import { EngineLink } from "./engine.mjs";
import { DIALOG_METHODS, KNOWN_UNSHOWN, NOTIFY_METHODS, deltaBlocks, isFoldedUpdate, messageBlocks, partsOf, roleOf, toolResultText } from "./events.mjs"; import { DIALOG_METHODS, KNOWN_UNSHOWN, NOTIFY_METHODS, deltaBlocks, isFoldedUpdate, messageBlocks, partsOf, roleOf, toolResultText } from "./events.mjs";
import { LineSplitter, encodeLine, parseLine } from "./framing.mjs"; import { LineSplitter, encodeLine, parseLine } from "./framing.mjs";
@@ -72,11 +72,13 @@ export const VERIFIED_OPERATIONS = Object.freeze(["observe", "prompt", "takeover
export const TEST_ENGINE = Symbol("conversation.test-engine"); export const TEST_ENGINE = Symbol("conversation.test-engine");
const ENGINE_KEYS = new Set(["extraArgs", "cwd", "envKeys"]); const ENGINE_KEYS = new Set(["extraArgs", "cwd", "envKeys"]);
export const TIMEOUTS = Object.freeze({ ack: 5000, state: 5000, start: 5000, clear: 5000, abort: 10000, settle: 10000, grace: 1000, write: 5000, maxRounds: 3 }); export const TIMEOUTS = Object.freeze({ ack: 5000, state: 5000, start: 5000, clear: 5000, abort: 10000, settle: 10000, grace: 1000, write: 5000, maxRounds: 3, exitSettle: 2000, exitProof: 5000 });
const FINAL = new Set(["finished", "failed", "dispatch-refused", "delivery-unknown"]); const FINAL = new Set(["finished", "failed", "dispatch-refused", "delivery-unknown"]);
const STOP_IN_PROGRESS = new Set(["fenced", "cancelling", "stopping"]); const STOP_IN_PROGRESS = new Set(["fenced", "cancelling", "stopping"]);
const STOP_NEXT = { fenced: ["cancelling", "stopping", "uncertain"], cancelling: ["stopping", "uncertain"], stopping: ["uncertain"], uncertain: [] }; const STOP_NEXT = { fenced: ["cancelling", "stopping", "uncertain"], cancelling: ["stopping", "uncertain"], stopping: ["uncertain"], uncertain: [] };
// check.mjs `ending`: the binding follows only stops that end the engine.
const ending = (mode) => mode === "force-stop" || mode === "engine-exit";
const SEAL_BASIS = "seal: --no-extensions and no --extension argument (pi-pin.mjs SEAL_FLAGS)"; const SEAL_BASIS = "seal: --no-extensions and no --extension argument (pi-pin.mjs SEAL_FLAGS)";
const DIALOG_REASON = "Pi dialogs are not answered in CHAT-03 (lead decision 30); shown disabled"; const DIALOG_REASON = "Pi dialogs are not answered in CHAT-03 (lead decision 30); shown disabled";
@@ -85,6 +87,16 @@ const isGen = (v) => Number.isInteger(v) && v >= 1 && v <= Number.MAX_SAFE_INTEG
const isId = (v) => typeof v === "string" && ID.test(v); const isId = (v) => typeof v === "string" && ID.test(v);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
// `p`'s value, or `late` if `ms` pass first.
async function within(p, ms, late) {
let timer;
try {
return await Promise.race([p, new Promise((r) => (timer = setTimeout(() => r(late), ms)))]);
} finally {
clearTimeout(timer);
}
}
// The client request envelope, per operation (CHAT-01 schema `command`). // The client request envelope, per operation (CHAT-01 schema `command`).
// Unknown operations pass the shape check and are refused by evaluate. // Unknown operations pass the shape check and are refused by evaluate.
const SHAPES = { const SHAPES = {
@@ -1000,6 +1012,7 @@ export class Controller {
#onEnd(exec, link) { #onEnd(exec, link) {
if (this.#stale(exec, link) || exec.closing) return; if (this.#stale(exec, link) || exec.closing) return;
this.#transport(exec, "eof"); this.#transport(exec, "eof");
void this.#engineExit(exec).catch((e) => this.#internal("engine-exit", e));
} }
#transport(exec, reason) { #transport(exec, reason) {
@@ -1233,7 +1246,7 @@ export class Controller {
}); });
this.stops.set(s.id, s); this.stops.set(s.id, s);
b.stop = s.id; b.stop = s.id;
if (mode === "force-stop") b.state = "stopping"; if (ending(mode)) b.state = "stopping";
this.#admission(); this.#admission();
if (mode !== "revocation") this.#emit("stopping", { stop: s.id }); if (mode !== "revocation") this.#emit("stopping", { stop: s.id });
this.#push({ kind: "stop", stop: s }); this.#push({ kind: "stop", stop: s });
@@ -1244,7 +1257,7 @@ export class Controller {
#advanceStop(stop, state) { #advanceStop(stop, state) {
if (!STOP_NEXT[stop.state]?.includes(state)) return false; if (!STOP_NEXT[stop.state]?.includes(state)) return false;
stop.state = state; stop.state = state;
if (stop.mode === "force-stop") this.b.state = state === "uncertain" ? "uncertain" : "stopping"; if (ending(stop.mode)) this.b.state = state === "uncertain" ? "uncertain" : "stopping";
this.#push({ kind: "stop", stop }); this.#push({ kind: "stop", stop });
this.#admission(); this.#admission();
return true; return true;
@@ -1496,21 +1509,8 @@ export class Controller {
} }
async #escalate(stop, { confirmation = null, resumed = false } = {}) { async #escalate(stop, { confirmation = null, resumed = false } = {}) {
const b = this.b;
const rec = () => this.claim.record; const rec = () => this.claim.record;
const fail = async (reason) => { const fail = (reason) => this.#endStopUncertain(stop, reason, { resumed });
if (stop.state !== "superseded") this.#advanceStop(stop, "uncertain");
b.state = "uncertain";
this.closers.add("uncertain");
this.#admission();
this.#emit("uncertain", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: "force-stop", outcome: "uncertain", reason, resumed });
try {
await this.#claimAdvance({ state: "uncertain" });
} catch (err) {
this.#evidenceAdd("uncertain", { reason: "claim", error: String(err.code ?? err.message) });
}
};
try { try {
await this.#claimAdvance((r) => ({ state: "stopping", stop: { id: stop.id, confirmation, phaseStarted: resumed ? r.stop?.phaseStarted ?? null : null, record: stop } })); await this.#claimAdvance((r) => ({ state: "stopping", stop: { id: stop.id, confirmation, phaseStarted: resumed ? r.stop?.phaseStarted ?? null : null, record: stop } }));
} catch (err) { } catch (err) {
@@ -1536,6 +1536,59 @@ export class Controller {
if (stop.state === "superseded") return undefined; if (stop.state === "superseded") return undefined;
this.#advanceStop(stop, "stopping"); this.#advanceStop(stop, "stopping");
if (result.outcome !== "proven") return fail(result.reason ?? "cohort evidence unavailable"); if (result.outcome !== "proven") return fail(result.reason ?? "cohort evidence unavailable");
return this.#stopProven(stop, result, { claimStop: rec().stop, fail, detail: { resumed } });
}
// Row 52, CHAT-01 `engine-exit`. EOF starts an observation; it proves
// nothing. The stop takes the one escalation slot (H10), so no force stop
// runs beside it, and it never supersedes a force stop in any state: a
// force stop's kill also ends output. Its observation only reads the
// cohort and has a deadline, so a shim that doesn't answer ends the stop
// `uncertain` and frees the slot for a client force stop. The claim stays
// `uncertain` from the transport loss until a proof is recorded: a restart
// finds an ordinary orphan, never an engine-exit stop to resume.
async #engineExit(exec) {
const b = this.b;
if (exec !== this.exec || !b || !this.claim?.record.engine || this.escalating || ending(this.stops.get(b.stop)?.mode)) return undefined;
const stop = this.#startStop("engine-exit", { requestId: null, connection: null, target: targetOf(b) });
this.escalating = stop.id;
try {
await this.#pause("engine-exit-recorded", { stop: stop.id });
const rec = this.claim.record, engine = rec.engine;
const result = await within(
engineExitCohort({ kind: engine.kind, unitName: rec.unitName, invocationId: rec.invocationId, shimSocket: rec.shim, pid: engine.pid, start: engine.start, settleMs: this.T.exitSettle }),
this.T.exitProof, { outcome: "unavailable", reason: "the engine-exit observation missed its deadline" },
);
const fail = (reason) => this.#endStopUncertain(stop, reason);
this.#advanceStop(stop, "stopping");
if (result.outcome !== "proven") return await fail(result.reason ?? "cohort evidence unavailable");
return await this.#stopProven(stop, result, { claimStop: { id: stop.id, confirmation: null, phaseStarted: null }, fail, detail: {} });
} finally {
if (this.escalating === stop.id) this.escalating = null;
}
}
// Ends a stop `uncertain`: the stop unless superseded, the binding,
// admission and the claim. The scope and its shim stay as evidence.
async #endStopUncertain(stop, reason, detail = {}) {
if (stop.state !== "superseded") this.#advanceStop(stop, "uncertain");
this.b.state = "uncertain";
this.closers.add("uncertain");
this.#admission();
this.#emit("uncertain", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: stop.mode, outcome: "uncertain", reason, ...detail });
try {
await this.#claimAdvance({ state: "uncertain" });
} catch (err) {
this.#evidenceAdd("uncertain", { reason: "claim", error: String(err.code ?? err.message) });
}
}
// A proven observation through the verifier to `stopped`, then the scope's
// release. Every path that can't record the proof ends at `fail`.
async #stopProven(stop, result, { claimStop, fail, detail }) {
const b = this.b;
const rec = () => this.claim.record;
if (!this.verifier) return fail("no verifier"); if (!this.verifier) return fail("no verifier");
const tools = this.exec && this.exec.execution === b.execution ? this.exec.tools : this.orphanTools ?? new Map(); const tools = this.exec && this.exec.execution === b.execution ? this.exec.tools : this.orphanTools ?? new Map();
const proof = cohortProof({ binding: b, stop: stop.id, result }); const proof = cohortProof({ binding: b, stop: stop.id, result });
@@ -1551,7 +1604,7 @@ export class Controller {
return fail(`session unreadable at proof: ${err.code ?? err.message}`); return fail(`session unreadable at proof: ${err.code ?? err.message}`);
} }
try { try {
await this.#claimFinish({ state: "stopped", proof: { kind: "cohortProof", ref: proof.id, effects: effects.id }, leafAtProof: session.leaf, branchAtProof: session.branch, stop: { ...rec().stop, record: stop } }); await this.#claimFinish({ state: "stopped", proof: { kind: "cohortProof", ref: proof.id, effects: effects.id }, leafAtProof: session.leaf, branchAtProof: session.branch, stop: { ...claimStop, record: stop } });
} catch (err) { } catch (err) {
return fail(`claim: ${err.code ?? err.message}`); return fail(`claim: ${err.code ?? err.message}`);
} }
@@ -1565,10 +1618,10 @@ export class Controller {
this.#push({ kind: "stop", stop }); this.#push({ kind: "stop", stop });
this.#push({ kind: "binding", binding: b }); this.#push({ kind: "binding", binding: b });
this.#emit("stopped", { stop: stop.id }); this.#emit("stopped", { stop: stop.id });
this.#evidenceAdd("stop", { stop: stop.id, mode: "force-stop", outcome: "stopped", proofs: { cohort: proof.id, effects: effects.id }, resumed }); this.#evidenceAdd("stop", { stop: stop.id, mode: stop.mode, outcome: "stopped", proofs: { cohort: proof.id, effects: effects.id }, ...detail });
const proven = this.claim; const proven = this.claim;
await this.#pause("scope-release", { stop: stop.id }); await this.#pause("scope-release", { stop: stop.id });
await this.#releaseScope("force-stop", proven); await this.#releaseScope(stop.mode, proven);
return undefined; return undefined;
} }
@@ -1615,6 +1668,8 @@ export class Controller {
const b = this.b, p = this.stoppedProof; const b = this.b, p = this.stoppedProof;
if (!s || s.state !== "stopped" || b.stop !== s.id || !scopeMatch(s.target, targetOf(b)) || !p || !this.verifier) return false; if (!s || s.state !== "stopped" || b.stop !== s.id || !scopeMatch(s.target, targetOf(b)) || !p || !this.verifier) return false;
if (p.proof.id !== s.supervisorEvidence || p.effects.id !== s.effectsEvidence) return false; if (p.proof.id !== s.supervisorEvidence || p.effects.id !== s.effectsEvidence) return false;
// An engine exit is proven on the engine itself, never on an empty list.
if (s.mode === "engine-exit" && p.proof.members.length === 0) return false;
return this.verifier.cohort(p.proof, p.effects, { binding: b, stop: s.id, now: this.now(), epoch: p.epoch }); return this.verifier.cohort(p.proof, p.effects, { binding: b, stop: s.id, now: this.now(), epoch: p.epoch });
} }
+6 -1
View File
@@ -67,9 +67,14 @@ const child = spawn(
{ stdio: [0, 1, 2] }, { stdio: [0, 1, 2] },
); );
const enginePid = child.pid; const enginePid = child.pid;
// Read while the engine lives: /proc has nothing for it once it is reaped.
// The exec chain keeps the PID and the start time.
const engineStart = startOf(enginePid);
let engineExit = null; let engineExit = null;
// The shim's wait. An engine-exit proof names the engine by these, with the
// reap time as its death time (row 52).
child.on("exit", (code, signal) => { child.on("exit", (code, signal) => {
engineExit = { code, signal, at: new Date().toISOString() }; engineExit = { code, signal, at: new Date().toISOString(), pid: enginePid, startTicks: engineStart, boot: bootId };
}); });
// The engine holds the controller's pipes; the shim's copies would hide EOF. // The engine holds the controller's pipes; the shim's copies would hide EOF.
closeSync(0); closeSync(0);
+192 -13
View File
@@ -4,8 +4,8 @@
// skip when systemd user scopes are unavailable. K2 runs on the process-group // skip when systemd user scopes are unavailable. K2 runs on the process-group
// fallback. K6–K9 and K16–K18 use the in-process fake, whose force stop is a // fallback. K6–K9 and K16–K18 use the in-process fake, whose force stop is a
// fixture stand-in (see FakeLauncher). K19 launches /bin/sleep through // fixture stand-in (see FakeLauncher). K19 launches /bin/sleep through
// ScopeLauncher with no controller. Controllers that must die run in // ScopeLauncher with no controller. E1–E6 (row 52) are the engine-exit stop
// ctrl-child.mjs. // on the scope fixtures. Controllers that must die run in ctrl-child.mjs.
import { test, after } from "node:test"; import { test, after } from "node:test";
import assert from "node:assert/strict"; import assert from "node:assert/strict";
@@ -14,7 +14,7 @@ import { spawn, spawnSync } from "node:child_process";
import { dirname, join } from "node:path"; import { dirname, join } from "node:path";
import { ClaimStore, FOREIGN_HOST, defaultHost, newClaimId, unitNameFor } from "../src/claim.mjs"; import { ClaimStore, FOREIGN_HOST, defaultHost, newClaimId, unitNameFor } from "../src/claim.mjs";
import { ConversationClient } from "../src/client.mjs"; import { ConversationClient } from "../src/client.mjs";
import { AUTHORITY, PgroupLauncher, ScopeLauncher, forceStopCohort, releaseCohort, scopeAvailable, shimRequest, systemctlShow, systemdUnits } from "../src/cohort.mjs"; import { AUTHORITY, PgroupLauncher, ScopeLauncher, engineExitCohort, forceStopCohort, releaseCohort, scopeAvailable, shimRequest, systemctlShow, systemdUnits } from "../src/cohort.mjs";
import { Controller, ELIGIBILITY, TEST_ENGINE } from "../src/controller.mjs"; import { Controller, ELIGIBILITY, TEST_ENGINE } from "../src/controller.mjs";
import { ENGINE_ENV, ENGINE_PIN_MISMATCH, engineEnv } from "../src/pi-pin.mjs"; import { ENGINE_ENV, ENGINE_PIN_MISMATCH, engineEnv } from "../src/pi-pin.mjs";
import { FixtureVerifier, newId } from "../src/records.mjs"; import { FixtureVerifier, newId } from "../src/records.mjs";
@@ -104,14 +104,14 @@ function gate() {
// A controller in this process on a real engine process: the fake engine // A controller in this process on a real engine process: the fake engine
// under ScopeLauncher ("scope") or PgroupLauncher ("pgroup"). // under ScopeLauncher ("scope") or PgroupLauncher ("pgroup").
async function live({ kind = "scope", barrier = null, verifier = new FixtureVerifier({ authorities: [AUTHORITY] }), launcher = null } = {}) { async function live({ kind = "scope", barrier = null, verifier = new FixtureVerifier({ authorities: [AUTHORITY] }), launcher = null, timeouts = FAST } = {}) {
const fx = track(fixture()); const fx = track(fixture());
const control = join(fx.base, "fake.sock"); const control = join(fx.base, "fake.sock");
const ctrl = new Controller({ const ctrl = new Controller({
fixtureRoot: fx.base, claimRoot: fx.claimRoot, socketDir: fx.socketDir, sessionFile: fx.sessionFile, seat: fx.seat, fixtureRoot: fx.base, claimRoot: fx.claimRoot, socketDir: fx.socketDir, sessionFile: fx.sessionFile, seat: fx.seat,
launcher: launcher ?? (kind === "scope" ? new ScopeLauncher() : new PgroupLauncher()), launcher: launcher ?? (kind === "scope" ? new ScopeLauncher() : new PgroupLauncher()),
engine: { cwd: fx.proj }, [TEST_ENGINE]: { command: process.execPath, preArgs: [FAKE_PI], env: { ...process.env, FAKE_PI_CONTROL: control, FAKE_PI_LOG: join(fx.base, "fake.log") } }, engine: { cwd: fx.proj }, [TEST_ENGINE]: { command: process.execPath, preArgs: [FAKE_PI], env: { ...process.env, FAKE_PI_CONTROL: control, FAKE_PI_LOG: join(fx.base, "fake.log") } },
verifier, units: kind === "scope" ? systemdUnits : noUnits, timeouts: FAST, barrier, verifier, units: kind === "scope" ? systemdUnits : noUnits, timeouts, barrier,
}); });
await ctrl.start(); await ctrl.start();
assert.equal(ctrl.binding.state, "active", JSON.stringify(ctrl.evidence.uncertain)); assert.equal(ctrl.binding.state, "active", JSON.stringify(ctrl.evidence.uncertain));
@@ -855,28 +855,207 @@ test("R3: no release after an unavailable stop or an unverified proof: the scope
// A normal engine exit leaves the shim and the scope: the shim exits only on // A normal engine exit leaves the shim and the scope: the shim exits only on
// `release`. The controller doesn't release then: the binding is uncertain // `release`. The controller doesn't release then: the binding is uncertain
// and the scope is what a force stop reads. The proven stop releases it. // and the scope is what a force stop reads. The proven stop releases it.
test("R4: after a normal engine exit the scope stays until a proven force stop releases it", NEEDS_SCOPE, async () => { // ---- engine exit (row 52, CHAT-01 `engine-exit`) ----------------------------
// A confirmation issued and confirmed now, not yet used.
async function confirmation(c, op) {
const issued = await c.request("issue-confirmation", { operationToConfirm: op });
assert.equal(issued.outcome, "confirmation-issued", JSON.stringify(issued));
const answered = await c.request("answer-confirmation", { confirmation: issued.data.confirmation.id, answer: "confirm" });
assert.equal(answered.outcome, "confirmation-confirmed", JSON.stringify(answered));
return issued.data.confirmation.id;
}
const exitStops = (h) => [...h.ctrl.stops.values()].filter((s) => s.mode === "engine-exit");
test("E1: a natural engine exit with no member left: an engine-exit stop, stopped on a proof naming the engine, and the scope released", NEEDS_SCOPE, async () => {
const g = gate();
const h = await live({ barrier: g.barrier });
const unit = h.rec().unitName, shim = h.rec().shim, engine = h.rec().engine;
try {
const before = h.ctrl.binding.stop;
const early = await confirmation(h.c, "recover");
g.hold("scope-release");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("scope-release");
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
const [stop] = exitStops(h);
assert.equal(h.ctrl.binding.stop, stop.id);
assert.deepEqual([stop.state, stop.request, stop.supersedes], ["stopped", null, before]);
// The proof names the engine by the shim's wait: PID, start ticks, boot
// and reap time, read while the shim still answers.
const exit = (await shimRequest(shim, "hello")).engineExit;
assert.equal(exit.code, 0);
assert.deepEqual(h.ctrl.stoppedProof.proof.members, [{ pid: engine.pid, boot: exit.boot, startTicks: Number(engine.start), terminatedAt: exit.at }]);
assert.deepEqual(stoppedOnProof(h.fx).map((r) => [r.stop.id, r.stop.record.mode]), [[stop.id, "engine-exit"], [stop.id, "engine-exit"]]);
g.release("scope-release");
await until(() => h.ctrl.evidence.releases.length > 0, 8000, "the release");
assert.deepEqual(h.ctrl.evidence.releases.map((r) => [r.why, r.outcome, r.unit ?? null]), [["engine-exit", "released", "absent"]]);
assert.ok(unitGone(unit));
// Recover takes its own confirmation, bound to this stop (K6); one issued
// before the engine exited names the old stop.
assert.equal((await h.c.request("recover", { stop: stop.id, confirmation: early })).refusal, "confirmation");
const rec = await h.c.confirmed("recover", { stop: stop.id });
assert.equal(rec.outcome, "recovery-eligible", JSON.stringify(rec));
await h.ctrl.release(rec.data.eligibility);
} finally {
g.release("scope-release");
await h.close();
}
});
test("E2: an engine exit with a tool child alive: the stop ends uncertain and nothing is released; a force stop then ends it", NEEDS_SCOPE, async () => {
const h = await live(); const h = await live();
const unit = h.rec().unitName, shim = h.rec().shim; const unit = h.rec().unitName, shim = h.rec().shim;
try { try {
const child = await childOf(h, {});
const early = await confirmation(h.c, "force-stop");
assert.equal((await h.fake.call("exit")).ok, true); assert.equal((await h.fake.call("exit")).ok, true);
await until(() => h.ctrl.binding.state === "uncertain", 8000, "the binding to see the exit"); await until(() => exitStops(h)[0]?.state === "uncertain", 8000, "the engine-exit stop to end");
await until(async () => (await shimRequest(shim, "hello")).engineExit !== null, 4000, "the shim to see the exit"); assert.equal(h.ctrl.binding.state, "uncertain");
assert.equal((await shimRequest(shim, "hello")).engineExit.code, 0); assert.deepEqual(h.ctrl.evidence.stops.map((s) => [s.mode, s.outcome]), [["engine-exit", "uncertain"]]);
await tick(200);
assert.ok(unitActive(unit), "the shim keeps the scope after the engine exits");
assert.deepEqual(h.ctrl.evidence.releases, []); assert.deepEqual(h.ctrl.evidence.releases, []);
assert.ok(unitActive(unit));
assert.ok(alive(child));
assert.deepEqual(await memberPids(shim), [child]);
assert.ok(!h.ctrl.stoppedProof);
// A force-stop confirmation issued before the engine exited is stale
// (H17): it names the stop the engine-exit stop superseded.
assert.equal((await h.c.request("force-stop", { confirmation: early })).refusal, "confirmation");
await forceStop(h); await forceStop(h);
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops)); assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
assert.deepEqual(h.ctrl.stoppedProof.proof.members, []); assert.equal(h.ctrl.stops.get(h.ctrl.binding.stop).supersedes, exitStops(h)[0].id);
await until(() => !alive(child), 4000, "the child to die");
await until(() => h.ctrl.evidence.releases.length > 0, 8000, "the release"); await until(() => h.ctrl.evidence.releases.length > 0, 8000, "the release");
assert.equal(h.ctrl.evidence.releases[0].outcome, "released"); assert.deepEqual(h.ctrl.evidence.releases.map((r) => [r.why, r.outcome]), [["force-stop", "released"]]);
await until(() => unitGone(unit), 8000, "the scope to go"); await until(() => unitGone(unit), 8000, "the scope to go");
} finally { } finally {
await h.close(); await h.close();
} }
}); });
test("E3: an engine exit and a force stop take one escalation slot, in either order", NEEDS_SCOPE, async () => {
{
const g = gate();
const h = await live({ barrier: g.barrier });
try {
g.hold("engine-exit-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("engine-exit-recorded");
assert.equal(h.ctrl.binding.state, "stopping");
assert.equal((await h.c.confirmed("force-stop")).refusal, "fenced");
g.release("engine-exit-recorded");
await until(() => h.ctrl.binding.state === "stopped", 8000, "the engine-exit proof");
assert.deepEqual(h.ctrl.evidence.stops.map((s) => [s.mode, s.outcome]), [["engine-exit", "stopped"]]);
} finally {
g.release("engine-exit-recorded");
await h.close();
}
}
{
const g = gate();
const h = await live({ barrier: g.barrier });
try {
// A force stop already running: the EOF its kill causes starts nothing.
g.hold("force-stop-recorded");
const fs = await h.c.confirmed("force-stop");
assert.equal(fs.outcome, "force-stop-fenced", JSON.stringify(fs));
await g.waitHeld("force-stop-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await tick(300);
assert.deepEqual(exitStops(h), []);
g.release("force-stop-recorded");
await until(() => ["stopped", "uncertain"].includes(h.ctrl.binding.state), 20000, "the force stop to end");
assert.equal(h.ctrl.binding.stop, fs.stop.id);
assert.deepEqual(exitStops(h), []);
assert.deepEqual(h.ctrl.evidence.stops.map((s) => s.mode), ["force-stop"]);
} finally {
g.release("force-stop-recorded");
await h.close();
}
}
});
test("E4: a shim that doesn't answer: the engine-exit stop misses its deadline, ends uncertain and frees the slot", NEEDS_SCOPE, async () => {
const h = await live({ timeouts: { ...FAST, exitSettle: 200, exitProof: 500 } });
const unit = h.rec().unitName, shim = h.rec().shim;
const shimPid = (await shimRequest(shim, "hello")).shimPid;
try {
process.kill(shimPid, "SIGSTOP");
assert.equal((await h.fake.call("exit")).ok, true);
await until(() => exitStops(h)[0]?.state === "uncertain", 8000, "the engine-exit deadline");
assert.equal(h.ctrl.binding.state, "uncertain");
assert.match(h.ctrl.evidence.stops.at(-1).reason, /deadline/);
assert.equal(h.ctrl.escalating, null);
process.kill(shimPid, "SIGCONT");
await forceStop(h);
assert.equal(h.ctrl.binding.state, "stopped", JSON.stringify(h.ctrl.evidence.stops));
// The force stop's proof is unchanged by row 52: the engine had already
// left the cohort, so it lists no member.
assert.deepEqual(h.ctrl.stoppedProof.proof.members, []);
await until(() => unitGone(unit), 8000, "the scope to go");
} finally {
try {
process.kill(shimPid, "SIGCONT");
} catch {
// gone
}
await h.close();
}
});
test("E5: an engine exit after a force stop ended uncertain starts no engine-exit stop; the force stop stays current", NEEDS_SCOPE, async () => {
const h = await live();
const advance = h.ctrl.store.advance.bind(h.ctrl.store);
let failed = false;
// The force stop's `stopping` claim write fails once, so it ends uncertain
// before it closes the execution and the engine's EOF still arrives.
h.ctrl.store.advance = async (claim, patch) => {
if (!failed && patch?.state === "stopping") {
failed = true;
throw Object.assign(new Error("injected"), { code: "E5-INJECTED" });
}
return advance(claim, patch);
};
try {
const stop = await forceStop(h);
assert.equal(h.ctrl.binding.state, "uncertain");
assert.match(h.ctrl.evidence.stops.at(-1).reason, /E5-INJECTED/);
assert.equal(h.ctrl.escalating, null);
assert.equal((await h.fake.call("exit")).ok, true);
await until(() => h.ctrl.evidence.uncertain.some((u) => u.reason === "eof"), 4000, "the EOF");
await tick(300);
assert.deepEqual(exitStops(h), []);
assert.equal(h.ctrl.binding.stop, stop);
assert.equal(h.ctrl.binding.state, "uncertain");
assert.deepEqual(h.ctrl.evidence.stops.map((s) => s.mode), ["force-stop"]);
} finally {
h.ctrl.store.advance = advance;
await h.close();
}
});
test("E6: the engine-exit observation refuses a reaped process that isn't the recorded engine: another PID or start ticks", NEEDS_SCOPE, async () => {
const g = gate();
const h = await live({ barrier: g.barrier });
try {
g.hold("engine-exit-recorded");
assert.equal((await h.fake.call("exit")).ok, true);
await g.waitHeld("engine-exit-recorded");
const rec = h.rec(), engine = rec.engine;
const observe = (over) => engineExitCohort({ kind: "scope", unitName: rec.unitName, invocationId: rec.invocationId, shimSocket: rec.shim, pid: engine.pid, start: engine.start, settleMs: 200, ...over });
for (const over of [{ pid: engine.pid + 1 }, { start: String(Number(engine.start) + 1) }]) {
const r = await observe(over);
assert.deepEqual([r.outcome, r.reason], ["unavailable", "the shim reaped a process that isn't the recorded engine"], JSON.stringify(over));
}
assert.equal((await observe({})).outcome, "proven");
g.release("engine-exit-recorded");
await until(() => h.ctrl.binding.state === "stopped", 8000, "the engine-exit proof");
} finally {
g.release("engine-exit-recorded");
await h.close();
}
});
test("R5: the harness finds a running shim and reaps it, and never signals a shim outside a mosaic-chat- scope (Filbert N1 and N2 on #1533)", NEEDS_SCOPE, async () => { test("R5: the harness finds a running shim and reaps it, and never signals a shim outside a mosaic-chat- scope (Filbert N1 and N2 on #1533)", NEEDS_SCOPE, async () => {
const fx = track(fixture()); const fx = track(fixture());
mkdirSync(fx.socketDir, { recursive: true }); mkdirSync(fx.socketDir, { recursive: true });