docs: Darkwing row 111 round 1 candidate packet (#1605, bus host start: one failure path after the broker's reply, and the child error event)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,115 @@
|
||||
# Row 111 (#1605) round 1: bus host start, the steps after the broker's reply and the child `error` event
|
||||
|
||||
Brief: docs/plans/2026-10-13_landing-follow-ups.md, section "Bus host start: release after the broker's reply, and the child `error` event" (blob 19c939e10 at 02fe2d9d5).
|
||||
BASE: 02fe2d9d552212049bef5efbca981945bbdb84f5. Candidate: build.patch against BASE. candidate-manifest.sha256 lists the 2 files, and applying the patch to a fresh worktree at BASE reproduces both digests. No commit after BASE touches either path. Per the brief, packages/cli/README.md is unchanged; row 110 owns it next.
|
||||
|
||||
## What changed
|
||||
|
||||
All in packages/cli/src/host.mjs, `startHost` and its `firstReply` helper.
|
||||
|
||||
- One failure path after the broker's reply. The reader lookup, the notifier's fork, its start send, the wait for its reply, a refusal, `rmSync` of an old host.json and `writeHostState` now all run in one `try`. Its `catch` sends the notifier SIGTERM if one was forked and waits up to 5 s for it, sends the broker `close` if the channel is still up, waits up to `CLOSE_TIMEOUT_MS` for the broker, calls `release()` and rethrows. That is the path the notifier's no-reply case already took. At BASE, a throw from the notifier fork, the start send, the reader lookup or `writeHostState` left host.lock and a live broker.
|
||||
- A missing reader. When the broker's reply has no reader for `business`, the start refuses with `broker has no reader for business <id>`, exit 3. At BASE that was a TypeError on `reader.cap`. The check runs with and without a notifier. Without one, BASE started a host anyway; `mosaic bus start` always boots the broker for the business it names, so the only change in practice is a refusal where BASE threw a TypeError.
|
||||
- The notifier's wait starts after its send, as the broker's has since row 106. At BASE the wait started first, so a send that threw would have left a wait whose later rejection nobody handled.
|
||||
- A notifier that refuses now gets SIGTERM before the wait, like every other failure. It has already called `stop(3)` or is about to, `stop` runs once, and SIGTERM to a child that has exited does nothing, so the refusal and its exit code are unchanged.
|
||||
- The child `error` event. Node emits `error` on a child when a send with no callback fails, or a kill fails with anything but ESRCH. With no listener, the emitter throws, the process ends, and host.lock and host.json stay. `startHost` now puts a listener on each child right after its fork:
|
||||
- During the start, `firstReply` also rejects on `error`, with `<broker|notifier> failed before it replied: <message>`, exit 1, so the start fails at once instead of at the reply timeout. An `error` at any other time during the start is kept, and the shared `try` throws it as `<name> failed: <message>` just before host.json is written. A broker `error` while the notifier starts is the case that reaches this check. An `error` during either failure path is kept and dropped, since the start is failing already.
|
||||
- Once the host is up, an `error` on the broker breaks the channel through `breakChannel("broker error: <message>")`. That rejects the request waiting on the broker, if any, logs and calls `close(1)`. An `error` on the notifier logs `notifier error (<message>); stopping the host` and calls `close(1)`, as a notifier that dies does. Either way `done` resolves with 1 and close() removes host.json and host.lock.
|
||||
|
||||
### An `error` after a successful start
|
||||
|
||||
The brief asks what an `error` after a successful start does and why. At BASE it ended the process with host.lock and host.json left behind. `request()` sends to the broker with no callback, so a failed send there emits `error` with nothing listening. The lock was left held, so the brief puts the change in scope. I chose to stop the host with exit 1. The unit restarts it, and that is what a child's death already does (watchChildren). For the broker the error goes through `breakChannel`: a request whose message never reached the broker would otherwise wait for its timeout and then break the channel anyway. A kill that fails on a running host means a child we can't signal, and stopping the host is the fail-closed answer there too.
|
||||
|
||||
The `up` flag switches the listener from the start behaviour to the running one. It is set in the same synchronous run as host.json's write, after `breakChannel`, `stopping` and `close` exist, so no event can reach the listener in between.
|
||||
|
||||
## Tests
|
||||
|
||||
packages/cli/tests/host.test.mjs: 31 at BASE, 41 in the candidate. The 10 new ones:
|
||||
|
||||
| Test | What it checks |
|
||||
|---|---|
|
||||
| `a notifier fork that throws stops the broker ...` (loop, fork and send) | the throw's own error; the broker has ended; with send, the notifier has ended too; no host.lock, host.json or writer.lock |
|
||||
| `a boot reply with no reader for the business refuses with exit 3 naming it, ...` (loop, with and without a notifier) | exit 3 and the message; no notifier forked; the broker has ended; no host.lock, host.json or writer.lock |
|
||||
| `a host.json that can't be written stops both children and leaves no host.lock` | EEXIST from `writeHostState`; both children ended; no host.lock, host.json or writer.lock |
|
||||
| `an error event on the <broker|notifier> before it replies fails the start at once, ...` (loop) | exit 1 and `<name> failed before it replied: write EPIPE`, within an 8 s test timeout; every child ended; no host.lock or host.json |
|
||||
| `an error event on the broker while the notifier starts fails the start once the notifier replies` | `broker failed: write EPIPE`; both children ended; no host.lock or host.json |
|
||||
| `an error event on the <broker|notifier> once the host is up stops the host with exit 1, ...` (loop) | `done` is 1; the log line; both children ended; no host.lock or host.json |
|
||||
|
||||
How the tests make each failure:
|
||||
|
||||
- Fork and send throws use `throwOn`. It now takes `nth`, the fork it targets, and leaves other children alone; the row 106 broker tests pass the default 0. A fork that throws returns no child, so `startHost` can't stop that notifier; `throwOn` kills it after the test, and the fork test checks only the broker.
|
||||
- The no-reader case starts a host for business `other` on a boot config for `acme`.
|
||||
- `writeHostState` writes through `host.json.<pid>.tmp` with the `wx` flag. The test makes that file first, so the write throws EEXIST in the test's own process.
|
||||
- The `error` before a reply uses `failSends`: the boot or start send fails with an EPIPE error, delivered as an `error` event on the next tick because the send has no callback. The child is left running and never gets the message, so only the `error` can end the wait in time.
|
||||
- The kept broker `error` and the errors once up are emitted on the child directly, on the next tick in the first case.
|
||||
- `forksSeen` records every child forked during the test, so a test can check each one has ended. It kills them all after the test.
|
||||
|
||||
## Mutants
|
||||
|
||||
mutants.py and gate.sh are the scripts I ran, copied here; they point at my scratch worktree, so change `WT`/`G` to rerun them. mutants.py runs each mutant against the named tests only, through `--test-name-pattern`, and counts a pattern that matches no test as a fault. n00 puts BASE's host.mjs, read from the worktree's HEAD, under the new tests.
|
||||
|
||||
Final run 23:32:52 to 23:33:46Z on 2026-10-13. Each file is restored from its `.orig`, and the candidate's diff hashed 2ef465444b6936ca (first 16 hex) before and after. Every run matched the tests it names: 1 each, 2 for the two loops that share a prefix.
|
||||
|
||||
| Id | Mutant | Tests | Result and how it fails |
|
||||
|---|---|---|---|
|
||||
| n00 | BASE's host.mjs under the new tests | all 9 patterns | killed, all 9: a child still running, `Missing expected rejection`, or the `reader.cap` TypeError |
|
||||
| n01 | `firstReply` doesn't listen for `error` | error before reply, broker and notifier | killed: both hit the 8 s test timeout |
|
||||
| n02 | no `error` listener on the broker | kept broker error; broker error once up | killed: the `error` throws with nothing listening |
|
||||
| n03 | no `error` listener on the notifier | notifier error once up | killed: the `error` throws with nothing listening |
|
||||
| n04 | a broker `error` once up is dropped | broker error once up | killed: 15 s test timeout, the host never stops |
|
||||
| n05 | a notifier `error` once up only logs | notifier error once up | killed: 15 s test timeout, the host never stops |
|
||||
| n06 | no reader check | no reader, both | killed: `Missing expected rejection` without a notifier, the `reader.cap` TypeError with one |
|
||||
| n07 | the notifier fork runs before the shared `try` | notifier fork throws | killed: the broker has not ended |
|
||||
| n08 | the notifier's wait starts before its send | notifier send throws | killed: the abandoned wait rejects with `notifier exited (null) before it replied` and nothing handles it |
|
||||
| n09 | host.json is written after the shared `try` | host.json write | killed: the children have not ended |
|
||||
| n10 | a kept child `error` is not thrown | kept broker error | killed: `Missing expected rejection` |
|
||||
| n11 | the failure path leaves the notifier running | host.json write; notifier send throws | killed: the notifier has not ended |
|
||||
| n12 | the failure path doesn't wait for the broker | no reader, both; host.json write | killed: the broker has not ended |
|
||||
| n13 | the failure path keeps the lock | no reader, both; host.json write; notifier fork throws | killed: host.lock is still there |
|
||||
|
||||
Two earlier runs did not count, and their logs are here as mutants-run-hung.log and mutants-run-count.log.
|
||||
|
||||
- The first hung on n00. BASE leaves the broker and notifier running after a failed start, and their IPC channels kept the test file from exiting, so the run sat at mutants.py's 600 s timeout. I stopped it; mutants.py put the candidate's host.mjs back. `forksSeen` now kills every child it saw after the test, as `throwOn` and `failSends` already did, so a start that leaves children running fails the test instead of hanging it. The host tests reran at 41 passed, 0 failed.
|
||||
- The second reported n01, n04 and n05 as faults with no tests run. Each of those tests had hit its timeout, which node counts as cancelled, not failed, and mutants.py counted only passes and fails. It now counts cancelled tests too. That run had every other mutant killed, and n00 killed on all 9 patterns.
|
||||
|
||||
## Gate
|
||||
|
||||
gate.sh, on BASE 02fe2d9d5 with the final candidate, 23:33:50 to 23:39:29Z on 2026-10-13, DOCKER_HOST unset, Docker 29.7.2.
|
||||
|
||||
| Step | Result | Time (UTC) |
|
||||
|---|---|---|
|
||||
| pkg-bus | 128 passed, 0 failed | 23:33:50-23:33:53 |
|
||||
| pkg-business | 67 passed, 0 failed | 23:33:53-23:33:55 |
|
||||
| pkg-cli | 178 passed, 0 failed | 23:33:55-23:34:16 |
|
||||
| pkg-control-board | 124 passed, 0 failed | 23:34:16-23:34:16 |
|
||||
| pkg-conversation | 182 passed, 0 failed | 23:34:16-23:35:05 |
|
||||
| pkg-discord | 178 passed, 0 failed | 23:35:05-23:35:08 |
|
||||
| pkg-harness | 73 passed, 0 failed | 23:35:08-23:35:18 |
|
||||
| pkg-ledger | 78 passed, 0 failed | 23:35:18-23:35:38 |
|
||||
| pkg-mosaic | 69 passed, 0 failed | 23:35:38-23:35:40 |
|
||||
| pkg-queue | 156 passed, 0 failed | 23:35:40-23:36:11 |
|
||||
| pkg-runs | 43 passed, 0 failed | 23:36:11-23:36:12 |
|
||||
| pkg-seat | 119 passed, 0 failed | 23:36:12-23:36:17 |
|
||||
| pkg-tasks | 92 passed, 0 failed | 23:36:17-23:36:22 |
|
||||
| pkg-webui | 108 passed, 0 failed | 23:36:22-23:37:18 |
|
||||
| script-auth | 15 passed, 0 failed | 23:37:18-23:37:19 |
|
||||
| script-conductor | 17 passed, 0 failed | 23:37:19-23:37:25 |
|
||||
| script-config | 24 passed, 0 failed | 23:37:25-23:37:26 |
|
||||
| script-discord | 66 passed, 0 failed | 23:37:26-23:37:34 |
|
||||
| script-extension-package | 18 passed, 0 failed | 23:37:34-23:37:35 |
|
||||
| script-foundation | 44 passed, 0 failed | 23:37:35-23:38:15 |
|
||||
| script-queue | 28 passed, 0 failed (node 156 passed) | 23:38:15-23:38:48 |
|
||||
| script-release | 14 passed, 0 failed | 23:38:48-23:38:58 |
|
||||
| script-task | 98 passed, 0 failed | 23:38:58-23:39:29 |
|
||||
|
||||
The gate printed GATE_PASSED at 23:39:29Z. The Docker steps left a `wt_default` network, created at 23:38:48Z with 0 containers; I removed it. An earlier gate run on the candidate before the `forksSeen` change also passed (23:13:16 to 23:19:01Z, gate-run-r0.log); its network, created at 23:18:19Z with 0 containers, is removed too.
|
||||
|
||||
## Choices and limits
|
||||
|
||||
- The broker's own refusal (`ready.ok` not true) keeps its BASE path. It runs before the shared `try`, and the broker exits on its own after refusing.
|
||||
- The shared `catch` does not remove host.json. `writeHostState` is the last step in the `try`, so a host.json this start wrote is never followed by a throw. The `rmSync` before it removes a stale one only, as at BASE.
|
||||
- If `renameSync` in `writeHostState` throws, `host.json.<pid>.tmp` stays. The start still fails and gives the lock back. `writeHostState` is outside this row's scope, and the name is per pid.
|
||||
- The third-racer window, `stopHost`, `hostStatus`, host.json's format and the broker are untouched, as the brief says.
|
||||
|
||||
## Going live (decision 109)
|
||||
|
||||
The host change is live only with a new copy, a re-pin and a bus restart, all under Mos's go. Until then the live bus, Console and CLI run host.mjs from the pinned copy at 465bbdfbb. The tests change nothing that runs.
|
||||
@@ -0,0 +1,300 @@
|
||||
diff --git a/packages/cli/src/host.mjs b/packages/cli/src/host.mjs
|
||||
index 02b15477d..69640f23c 100644
|
||||
--- a/packages/cli/src/host.mjs
|
||||
+++ b/packages/cli/src/host.mjs
|
||||
@@ -124,20 +124,24 @@ function writeHostState(dataRoot, state) {
|
||||
renameSync(tmp, file);
|
||||
}
|
||||
|
||||
-// The first IPC message from child, or a refusal when it exits or the wait runs out.
|
||||
+// The first IPC message from child, or a refusal when it exits, emits
|
||||
+// `error` or the wait runs out.
|
||||
function firstReply(child, timeoutMs, what) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const done = (fn, v) => {
|
||||
clearTimeout(timer);
|
||||
child.off("message", onMessage);
|
||||
child.off("exit", onExit);
|
||||
+ child.off("error", onError);
|
||||
fn(v);
|
||||
};
|
||||
const onMessage = (m) => done(resolve, m);
|
||||
const onExit = (code) => done(reject, new CliError(`${what} exited (${code}) before it replied`, 1));
|
||||
+ const onError = (e) => done(reject, new CliError(`${what} failed before it replied: ${e.message}`, 1));
|
||||
const timer = setTimeout(() => done(reject, new CliError(`${what} did not reply within ${Math.round(timeoutMs / 1000)} s`, 1)), timeoutMs);
|
||||
child.on("message", onMessage);
|
||||
child.on("exit", onExit);
|
||||
+ child.on("error", onError);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -183,6 +187,25 @@ export async function startHost({ boot, business, notifier = null, bootTimeoutMs
|
||||
if (same(held, state)) rmSync(hostLock(dataRoot), { force: true });
|
||||
};
|
||||
|
||||
+ // Node emits `error` on a child for a send with no callback that fails, or
|
||||
+ // a kill that fails, and with no listener that ends this process with
|
||||
+ // host.lock held. Until the host is up, an `error` fails the start: the
|
||||
+ // wait for a reply rejects on it, and one that comes at any other time is
|
||||
+ // kept and thrown before host.json is written. Once the host is up, an
|
||||
+ // `error` stops the host with exit 1, as a child's death does.
|
||||
+ let up = false;
|
||||
+ let childError = null;
|
||||
+ const onChildError = (name) => (e) => {
|
||||
+ if (!up) {
|
||||
+ childError ??= new CliError(`${name} failed: ${e.message}`, 1);
|
||||
+ return;
|
||||
+ }
|
||||
+ if (name === "broker") return breakChannel(`broker error: ${e.message}`);
|
||||
+ if (stopping) return;
|
||||
+ log(`notifier error (${e.message}); stopping the host`);
|
||||
+ close(1);
|
||||
+ };
|
||||
+
|
||||
// A fork or a send that throws gives the lock back too. The wait starts
|
||||
// after the send, so a send that throws leaves no wait behind. The reply
|
||||
// and the exit event come on a later tick, so the wait misses neither.
|
||||
@@ -190,6 +213,7 @@ export async function startHost({ boot, business, notifier = null, bootTimeoutMs
|
||||
let ready;
|
||||
try {
|
||||
broker = fork(BROKER, [], { stdio: ["ignore", "inherit", "inherit", "ipc"] });
|
||||
+ broker.on("error", onChildError("broker"));
|
||||
broker.send({ op: "boot", config: boot });
|
||||
ready = await firstReply(broker, bootTimeoutMs, "broker");
|
||||
} catch (e) {
|
||||
@@ -205,36 +229,36 @@ export async function startHost({ boot, business, notifier = null, bootTimeoutMs
|
||||
release();
|
||||
throw new CliError(`broker refused to start: ${typeof ready?.error === "string" ? ready.error : "startup-refused"}`, 3);
|
||||
}
|
||||
- const reader = ready.readers.find((r) => r.business === business);
|
||||
|
||||
+ // Every step from here to host.json shares one failure path: it stops the
|
||||
+ // notifier if one started, closes the broker, waits for both to end and
|
||||
+ // gives the lock back. The notifier's wait starts after its send, as the
|
||||
+ // broker's does.
|
||||
let notify = null;
|
||||
- if (notifier) {
|
||||
- notify = fork(NOTIFIER, [], { stdio: ["ignore", "inherit", "inherit", "ipc"] });
|
||||
- const started = firstReply(notify, START_TIMEOUT_MS, "notifier");
|
||||
- notify.send({ op: "start", path: ready.path, cap: reader.cap, business, dataRoot, binding: notifier.binding, base: notifier.base, pollMs: notifier.pollMs });
|
||||
- let ok;
|
||||
- try {
|
||||
- ok = await started;
|
||||
- } catch (e) {
|
||||
- notify.kill("SIGTERM");
|
||||
- await ended(notify, 5000);
|
||||
- if (broker.connected) broker.send({ op: "close" }, () => {});
|
||||
- await ended(broker, CLOSE_TIMEOUT_MS);
|
||||
- release();
|
||||
- throw e;
|
||||
+ try {
|
||||
+ const reader = ready.readers?.find((r) => r.business === business);
|
||||
+ if (!reader) throw new CliError(`broker has no reader for business ${business}`, 3);
|
||||
+ if (notifier) {
|
||||
+ notify = fork(NOTIFIER, [], { stdio: ["ignore", "inherit", "inherit", "ipc"] });
|
||||
+ notify.on("error", onChildError("notifier"));
|
||||
+ notify.send({ op: "start", path: ready.path, cap: reader.cap, business, dataRoot, binding: notifier.binding, base: notifier.base, pollMs: notifier.pollMs });
|
||||
+ const ok = await firstReply(notify, START_TIMEOUT_MS, "notifier");
|
||||
+ if (ok?.ok !== true) throw new CliError(`notifier refused to start: ${typeof ok?.error === "string" ? ok.error : "notifier-refused"}`, 3);
|
||||
}
|
||||
- if (ok?.ok !== true) {
|
||||
+ if (childError) throw childError;
|
||||
+ rmSync(hostFile(dataRoot), { force: true });
|
||||
+ writeHostState(dataRoot, state);
|
||||
+ } catch (e) {
|
||||
+ if (notify) {
|
||||
+ notify.kill("SIGTERM");
|
||||
await ended(notify, 5000);
|
||||
- if (broker.connected) broker.send({ op: "close" }, () => {});
|
||||
- await ended(broker, CLOSE_TIMEOUT_MS);
|
||||
- release();
|
||||
- throw new CliError(`notifier refused to start: ${typeof ok?.error === "string" ? ok.error : "notifier-refused"}`, 3);
|
||||
}
|
||||
+ if (broker.connected) broker.send({ op: "close" }, () => {});
|
||||
+ await ended(broker, CLOSE_TIMEOUT_MS);
|
||||
+ release();
|
||||
+ throw e;
|
||||
}
|
||||
|
||||
- rmSync(hostFile(dataRoot), { force: true });
|
||||
- writeHostState(dataRoot, state);
|
||||
-
|
||||
// stopping: close() has started (no new close, child deaths are expected).
|
||||
// closing: the before-close hooks have run, so no more broker requests.
|
||||
let stopping = false;
|
||||
@@ -330,7 +354,9 @@ export async function startHost({ boot, business, notifier = null, bootTimeoutMs
|
||||
|
||||
// A child that dies on its own takes the host down with exit 1: the unit
|
||||
// restarts the pair rather than run a broker without its notifier.
|
||||
- // Runs after `queue` exists, since close() awaits it.
|
||||
+ // Runs after `queue` exists, since close() awaits it. No event comes
|
||||
+ // between host.json and here, so an `error` meets breakChannel and close.
|
||||
+ up = true;
|
||||
watchChildren({ broker, notifier: notify }, (name, code, signal) => {
|
||||
if (stopping) return;
|
||||
log(`${name} exited (${signal ?? code}); stopping the host`);
|
||||
diff --git a/packages/cli/tests/host.test.mjs b/packages/cli/tests/host.test.mjs
|
||||
index 1b55505c6..6a1170357 100644
|
||||
--- a/packages/cli/tests/host.test.mjs
|
||||
+++ b/packages/cli/tests/host.test.mjs
|
||||
@@ -108,14 +108,14 @@ function failSends(t, fake) {
|
||||
});
|
||||
}
|
||||
|
||||
-// Makes the next fork throw (`where` "fork") or the new child's sends throw
|
||||
+// Makes the nth new child's fork throw (`where` "fork") or its sends throw
|
||||
// ("send"), through the `send` that node installs on a new child: a fork
|
||||
-// throws when the install does. Every child it saw is killed after the test,
|
||||
-// the one a throwing fork started included.
|
||||
-function throwOn(t, where) {
|
||||
+// throws when the install does. Other children are left alone. Every child
|
||||
+// it saw is killed after the test, the one a throwing fork started included.
|
||||
+function throwOn(t, where, nth = 0) {
|
||||
const children = [];
|
||||
const onChild = ({ process: child }) => {
|
||||
- children.push(child);
|
||||
+ if (children.push(child) - 1 !== nth) return;
|
||||
let send;
|
||||
Object.defineProperty(child, "send", {
|
||||
configurable: true,
|
||||
@@ -135,6 +135,22 @@ function throwOn(t, where) {
|
||||
});
|
||||
}
|
||||
|
||||
+// Every child this process creates while the test runs, in fork order. Each
|
||||
+// is killed after the test, so a start that leaves a child running fails the
|
||||
+// test rather than hang the file.
|
||||
+function forksSeen(t) {
|
||||
+ const children = [];
|
||||
+ const onChild = ({ process: child }) => children.push(child);
|
||||
+ subscribe("child_process", onChild);
|
||||
+ t.after(() => {
|
||||
+ unsubscribe("child_process", onChild);
|
||||
+ for (const child of children) child.kill("SIGKILL");
|
||||
+ });
|
||||
+ return children;
|
||||
+}
|
||||
+
|
||||
+const gone = (child) => child.exitCode !== null || child.signalCode !== null;
|
||||
+
|
||||
const epipe = (child) => {
|
||||
child.kill("SIGKILL");
|
||||
return Object.assign(new Error("write EPIPE"), { code: "EPIPE", errno: -32, syscall: "write" });
|
||||
@@ -407,6 +423,117 @@ test("a broker that misses its boot wait leaves no host.lock and no host.json",
|
||||
assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false);
|
||||
});
|
||||
|
||||
+for (const where of ["fork", "send"])
|
||||
+ test(`a notifier ${where} that throws stops the broker and leaves no host.lock and no host.json`, async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ throwOn(t, where, 1);
|
||||
+ const started = startHost({ boot, business: "acme", notifier: { binding: "test-seat" }, log: () => {} });
|
||||
+ t.after(async () => (await started.catch(() => null))?.close(0));
|
||||
+ await assert.rejects(started, (e) => e.message === `${where} failed`);
|
||||
+ assert.equal(children.length, 2);
|
||||
+ assert.ok(gone(children[0]), "the broker has ended");
|
||||
+ // A fork that throws returns no child to stop; throwOn kills it.
|
||||
+ if (where === "send") assert.ok(gone(children[1]), "the notifier has ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false);
|
||||
+ });
|
||||
+
|
||||
+for (const withNotifier of [true, false])
|
||||
+ test(`a boot reply with no reader for the business refuses with exit 3 naming it, ${withNotifier ? "before the notifier starts" : "with no notifier"}, and leaves no host.lock and no live broker`, async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ const started = startHost({ boot, business: "other", notifier: withNotifier ? { binding: "test-seat" } : null, log: () => {} });
|
||||
+ t.after(async () => (await started.catch(() => null))?.close(0));
|
||||
+ await assert.rejects(started, (e) => e.exitCode === 3 && e.message === "broker has no reader for business other");
|
||||
+ assert.equal(children.length, 1, "no notifier was started");
|
||||
+ assert.ok(gone(children[0]), "the broker has ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false);
|
||||
+ });
|
||||
+
|
||||
+test("a host.json that can't be written stops both children and leaves no host.lock", async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const discord = await fakeDiscord(t);
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ // The host writes host.json through this name with the `wx` flag.
|
||||
+ mkdirSync(hostDir(f.dataRoot), { recursive: true, mode: 0o700 });
|
||||
+ writeFileSync(`${hostFile(f.dataRoot)}.${process.pid}.tmp`, "");
|
||||
+ const started = startHost({ boot, business: "acme", notifier: { binding: "test-seat", base: discord.base, pollMs: 100 }, log: () => {} });
|
||||
+ t.after(async () => (await started.catch(() => null))?.close(0));
|
||||
+ await assert.rejects(started, (e) => e.code === "EEXIST");
|
||||
+ assert.equal(children.length, 2);
|
||||
+ assert.ok(children.every(gone), "the broker and the notifier have ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false);
|
||||
+});
|
||||
+
|
||||
+for (const [name, op] of [["broker", "boot"], ["notifier", "start"]])
|
||||
+ test(`an error event on the ${name} before it replies fails the start at once, and leaves no host.lock and no live child`, { timeout: 8000 }, async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ // The send fails, and the child, still running, never gets the message.
|
||||
+ failSends(t, (child, m) => (m?.op === op ? Object.assign(new Error("write EPIPE"), { code: "EPIPE" }) : null));
|
||||
+ const started = startHost({ boot, business: "acme", notifier: { binding: "test-seat" }, log: () => {} });
|
||||
+ t.after(async () => (await started.catch(() => null))?.close(0));
|
||||
+ await assert.rejects(started, (e) => e.exitCode === 1 && e.message === `${name} failed before it replied: write EPIPE`);
|
||||
+ assert.ok(children.every(gone), "every child has ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+ });
|
||||
+
|
||||
+test("an error event on the broker while the notifier starts fails the start once the notifier replies", async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const discord = await fakeDiscord(t);
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ spySends(t, (m) => {
|
||||
+ if (m?.op === "start") process.nextTick(() => children[0].emit("error", new Error("write EPIPE")));
|
||||
+ });
|
||||
+ const started = startHost({ boot, business: "acme", notifier: { binding: "test-seat", base: discord.base, pollMs: 100 }, log: () => {} });
|
||||
+ t.after(async () => (await started.catch(() => null))?.close(0));
|
||||
+ await assert.rejects(started, (e) => e.exitCode === 1 && e.message === "broker failed: write EPIPE");
|
||||
+ assert.ok(children.every(gone), "the broker and the notifier have ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+});
|
||||
+
|
||||
+for (const [n, name, line] of [[0, "broker", "broker channel broken (broker error: write EPIPE); stopping the host"], [1, "notifier", "notifier error (write EPIPE); stopping the host"]])
|
||||
+ test(`an error event on the ${name} once the host is up stops the host with exit 1, and leaves no host.lock and no host.json`, { timeout: 15000 }, async (t) => {
|
||||
+ const root = tmp(t);
|
||||
+ const f = fixture(root);
|
||||
+ makeDeployment(root, { dmRecipient: IDS.owner });
|
||||
+ const discord = await fakeDiscord(t);
|
||||
+ const boot = bootConfig({ system: loadSystem({ env: f.env }), businessId: "acme", env: f.env });
|
||||
+ const children = forksSeen(t);
|
||||
+ const logs = [];
|
||||
+ const host = await startHost({ boot, business: "acme", notifier: { binding: "test-seat", base: discord.base, pollMs: 100 }, log: (l) => logs.push(l) });
|
||||
+ t.after(() => host.close(0));
|
||||
+ children[n].emit("error", new Error("write EPIPE"));
|
||||
+ assert.equal(await host.done, 1);
|
||||
+ assert.ok(logs.includes(line), logs.join("\n"));
|
||||
+ assert.ok(children.every(gone), "the broker and the notifier have ended");
|
||||
+ assert.equal(existsSync(hostLock(f.dataRoot)), false);
|
||||
+ assert.equal(existsSync(hostFile(f.dataRoot)), false);
|
||||
+ });
|
||||
+
|
||||
test("a broker reply that misses the wait breaks the channel: the late reply answers nothing, later requests refuse, the host exits 1", { timeout: 30000 }, async (t) => {
|
||||
const root = tmp(t);
|
||||
const f = fixture(root);
|
||||
@@ -0,0 +1,2 @@
|
||||
a0936aee783ff47dd58fc907fefd86f63e571809a3ce5b0d6d931f71e756b15e packages/cli/src/host.mjs
|
||||
816dd386998b154561f9e910db3349534b0bfec5d1df2a8226b37bcedf05d6cc packages/cli/tests/host.test.mjs
|
||||
@@ -0,0 +1,26 @@
|
||||
base 02fe2d9d5 DOCKER_HOST=unset docker=29.7.2 start 23:13:16
|
||||
pkg-bus rc=0 23:13:16-23:13:19 ℹ pass 128 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 3099.289836
|
||||
pkg-business rc=0 23:13:19-23:13:21 ℹ pass 67 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2052.003431
|
||||
pkg-cli rc=0 23:13:21-23:13:42 ℹ pass 178 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 21127.439465
|
||||
pkg-control-board rc=0 23:13:42-23:13:43 ℹ pass 124 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 670.266966
|
||||
pkg-conversation rc=0 23:13:43-23:14:31 ℹ pass 182 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 48128.19695
|
||||
pkg-discord rc=0 23:14:31-23:14:34 ℹ pass 178 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2801.337937
|
||||
pkg-harness rc=0 23:14:34-23:14:44 ℹ pass 73 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 10362.824693
|
||||
pkg-ledger rc=0 23:14:44-23:15:03 ℹ pass 78 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 18802.578409
|
||||
pkg-mosaic rc=0 23:15:03-23:15:05 ℹ pass 69 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2160.094842
|
||||
pkg-queue rc=0 23:15:05-23:15:36 ℹ pass 156 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 30193.909099
|
||||
pkg-runs rc=0 23:15:36-23:15:37 ℹ pass 43 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 943.569598
|
||||
pkg-seat rc=0 23:15:37-23:15:42 ℹ pass 119 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 5037.400537
|
||||
pkg-tasks rc=0 23:15:42-23:15:48 ℹ pass 92 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 6379.444268
|
||||
pkg-webui rc=0 23:15:48-23:16:45 ℹ pass 108 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 56429.288669
|
||||
script-auth rc=0 23:16:45-23:16:45 | selftest: 15 passed, 0 failed
|
||||
script-conductor rc=0 23:16:45-23:16:50 | selftest: 17 passed, 0 failed
|
||||
script-config rc=0 23:16:50-23:16:52 | selftest: 24 passed, 0 failed
|
||||
script-discord rc=0 23:16:52-23:17:00 | discord suite: 66 passed, 0 failed
|
||||
script-extension-package rc=0 23:17:00-23:17:02 | extension package selftest: 18 passed, 0 failed
|
||||
script-foundation rc=0 23:17:02-23:17:45 | selftest: 44 passed, 0 failed
|
||||
script-queue rc=0 23:17:45-23:18:19 ℹ pass 156 ℹ fail 0 | queue suite: 28 passed, 0 failed
|
||||
script-release rc=0 23:18:19-23:18:30 | selftest: 14 passed, 0 failed
|
||||
script-task rc=0 23:18:30-23:19:01 | selftest: 98 passed, 0 failed
|
||||
GATE_PASSED 23:19:01
|
||||
exit 0
|
||||
@@ -0,0 +1,26 @@
|
||||
base 02fe2d9d5 DOCKER_HOST=unset docker=29.7.2 start 23:33:50
|
||||
pkg-bus rc=0 23:33:50-23:33:53 ℹ pass 128 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2953.757653
|
||||
pkg-business rc=0 23:33:53-23:33:55 ℹ pass 67 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 1861.143067
|
||||
pkg-cli rc=0 23:33:55-23:34:16 ℹ pass 178 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 20603.781008
|
||||
pkg-control-board rc=0 23:34:16-23:34:16 ℹ pass 124 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 664.534952
|
||||
pkg-conversation rc=0 23:34:16-23:35:05 ℹ pass 182 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 48917.356554
|
||||
pkg-discord rc=0 23:35:05-23:35:08 ℹ pass 178 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2619.955864
|
||||
pkg-harness rc=0 23:35:08-23:35:18 ℹ pass 73 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 10392.845602
|
||||
pkg-ledger rc=0 23:35:18-23:35:38 ℹ pass 78 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 19810.630651
|
||||
pkg-mosaic rc=0 23:35:38-23:35:40 ℹ pass 69 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 2239.313546
|
||||
pkg-queue rc=0 23:35:40-23:36:11 ℹ pass 156 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 30204.971802
|
||||
pkg-runs rc=0 23:36:11-23:36:12 ℹ pass 43 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 786.33893
|
||||
pkg-seat rc=0 23:36:12-23:36:17 ℹ pass 119 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 5060.666065
|
||||
pkg-tasks rc=0 23:36:17-23:36:22 ℹ pass 92 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 5388.864293
|
||||
pkg-webui rc=0 23:36:22-23:37:18 ℹ pass 108 ℹ fail 0 ℹ skipped 0 | ℹ duration_ms 55882.713431
|
||||
script-auth rc=0 23:37:18-23:37:19 | selftest: 15 passed, 0 failed
|
||||
script-conductor rc=0 23:37:19-23:37:25 | selftest: 17 passed, 0 failed
|
||||
script-config rc=0 23:37:25-23:37:26 | selftest: 24 passed, 0 failed
|
||||
script-discord rc=0 23:37:26-23:37:34 | discord suite: 66 passed, 0 failed
|
||||
script-extension-package rc=0 23:37:34-23:37:35 | extension package selftest: 18 passed, 0 failed
|
||||
script-foundation rc=0 23:37:35-23:38:15 | selftest: 44 passed, 0 failed
|
||||
script-queue rc=0 23:38:15-23:38:48 ℹ pass 156 ℹ fail 0 | queue suite: 28 passed, 0 failed
|
||||
script-release rc=0 23:38:48-23:38:58 | selftest: 14 passed, 0 failed
|
||||
script-task rc=0 23:38:58-23:39:29 | selftest: 98 passed, 0 failed
|
||||
GATE_PASSED 23:39:29
|
||||
exit 0
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
set -o pipefail
|
||||
# Darkwing gate, row 111 (#1605), candidate in the r111 worktree on base 02fe2d9d5 (row 106 landed).
|
||||
# Every package's node suite, then every scripts/test-*.sh, sequential, each
|
||||
# teed to its own log; stops at the first failure.
|
||||
G=$HOME/darkwing-scratch/r111/wt
|
||||
OUT=$HOME/darkwing-scratch/r111/gate
|
||||
export TMPDIR=$HOME/darkwing-scratch/r111/tmp
|
||||
mkdir -p "$OUT" "$TMPDIR"
|
||||
cd "$G" || exit 9
|
||||
[ -e "$G/node_modules" ] || { echo "STOP no node_modules in $G"; exit 8; }
|
||||
echo "base $(git rev-parse --short HEAD) DOCKER_HOST=${DOCKER_HOST:-unset} docker=$(docker info --format '{{.ServerVersion}}' 2>/dev/null || echo none) start $(date -u +%H:%M:%S)"
|
||||
run() { local name="$1" dir="$2" cmd="$3" t0; t0=$(date -u +%H:%M:%S)
|
||||
(cd "$G/$dir" && bash -c "$cmd") 2>&1 | tee "$OUT/$name.log" > /dev/null; local rc=$?
|
||||
echo "$name rc=$rc $t0-$(date -u +%H:%M:%S) $(grep -E '^ℹ (pass|fail|skipped) [0-9]+' "$OUT/$name.log" | tr '\n' ' ') | $(tail -n1 "$OUT/$name.log")"
|
||||
[ $rc -eq 0 ] || { echo "STOP $name"; exit $rc; }; }
|
||||
for d in packages/*/; do p=$(basename "$d"); [ -d "$d/tests" ] || continue
|
||||
run "pkg-$p" "packages/$p" 'node --test tests/*.test.mjs'; done
|
||||
for s in scripts/test-*.sh; do n=$(basename "$s" .sh); run "script-${n#test-}" . "bash $s"; done
|
||||
echo "GATE_PASSED $(date -u +%H:%M:%S)"
|
||||
@@ -0,0 +1,17 @@
|
||||
start diff 2ef465444b6936ca 23:31:43
|
||||
n00 BASE's startHost under the new tests: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n01 the wait for a reply ignores error: FAULT (expected killed) [('host.test.mjs', 'no-tests'), ('host.test.mjs', 'no-tests')]
|
||||
n02 no error listener on the broker: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n03 no error listener on the notifier: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n04 a broker error once up is dropped: FAULT (expected killed) [('host.test.mjs', 'no-tests')]
|
||||
n05 a notifier error once up only logs: FAULT (expected killed) [('host.test.mjs', 'no-tests')]
|
||||
n06 no reader check: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n07 the notifier fork runs before the shared try: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n08 the notifier's wait starts before its send: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n09 host.json is written after the shared try: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n10 a kept child error is not thrown: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n11 the failure path leaves the notifier running: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n12 the failure path does not wait for the broker: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n13 the failure path keeps the lock: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
end diff 2ef465444b6936ca same 23:32:37
|
||||
MUTANTS_BAD ['n01 the wait for a reply ignores error', 'n04 a broker error once up is dropped', 'n05 a notifier error once up only logs']
|
||||
@@ -0,0 +1,15 @@
|
||||
start diff 0b68cb47a6d192e8 23:19:10
|
||||
n00 BASE's startHost under the new tests: FAULT (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 'timeout'), ('host.test.mjs', 'no-tests'), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
Traceback (most recent call last):
|
||||
File "/home/jwoltje/darkwing-scratch/r111/mutants.py", line 104, in <module>
|
||||
rcs = [run(t, p, log) for t, p in tests]
|
||||
^^^^^^^^^^^^^^
|
||||
File "/home/jwoltje/darkwing-scratch/r111/mutants.py", line 71, in run
|
||||
rc = p.wait(timeout=600)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/jwoltje/.pyenv/versions/3.12.8/lib/python3.12/subprocess.py", line 1264, in wait
|
||||
return self._wait(timeout=timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/jwoltje/.pyenv/versions/3.12.8/lib/python3.12/subprocess.py", line 2047, in _wait
|
||||
time.sleep(delay)
|
||||
KeyboardInterrupt
|
||||
@@ -0,0 +1,17 @@
|
||||
start diff 2ef465444b6936ca 23:32:52
|
||||
n00 BASE's startHost under the new tests: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n01 the wait for a reply ignores error: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n02 no error listener on the broker: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n03 no error listener on the notifier: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n04 a broker error once up is dropped: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n05 a notifier error once up only logs: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n06 no reader check: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n07 the notifier fork runs before the shared try: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n08 the notifier's wait starts before its send: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n09 host.json is written after the shared try: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n10 a kept child error is not thrown: killed (expected killed) [('host.test.mjs', 1)]
|
||||
n11 the failure path leaves the notifier running: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n12 the failure path does not wait for the broker: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
n13 the failure path keeps the lock: killed (expected killed) [('host.test.mjs', 1), ('host.test.mjs', 1), ('host.test.mjs', 1)]
|
||||
end diff 2ef465444b6936ca same 23:33:46
|
||||
MUTANTS_OK
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
# Row 111 (#1605) mutants. Each takes out or bends one new check in
|
||||
# startHost; every named (test file, name pattern) run must fail on it. n00
|
||||
# puts BASE's host.mjs back under the new tests. Each file is restored from
|
||||
# its .orig after the run, and the candidate's diff must hash the same at the
|
||||
# end as at the start.
|
||||
import hashlib, os, signal, subprocess, sys, time
|
||||
|
||||
WT = os.path.expanduser("~/darkwing-scratch/r111/wt")
|
||||
OUT = os.path.expanduser("~/darkwing-scratch/r111/mutants")
|
||||
ENV = dict(os.environ, TMPDIR=os.path.expanduser("~/darkwing-scratch/r111/tmp"))
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
os.makedirs(ENV["TMPDIR"], exist_ok=True)
|
||||
|
||||
HO, T = "packages/cli/src/host.mjs", "packages/cli/tests/host.test.mjs"
|
||||
NFORK, NSEND = (T, "^a notifier fork that throws"), (T, "^a notifier send that throws")
|
||||
READER, WRITE = (T, "^a boot reply with no reader"), (T, "^a host.json that can't be written")
|
||||
EB, EN = (T, "^an error event on the broker before it replies"), (T, "^an error event on the notifier before it replies")
|
||||
KEPT = (T, "^an error event on the broker while the notifier starts")
|
||||
UB, UN = (T, "^an error event on the broker once the host is up"), (T, "^an error event on the notifier once the host is up")
|
||||
|
||||
BASE_HO = (HO, None, subprocess.run(["git", "-C", WT, "show", "HEAD:" + HO], capture_output=True, text=True, check=True).stdout)
|
||||
FORK = ' notify = fork(NOTIFIER, [], { stdio: ["ignore", "inherit", "inherit", "ipc"] });\n notify.on("error", onChildError("notifier"));\n'
|
||||
SEND_WAIT = ''' notify.send({ op: "start", path: ready.path, cap: reader.cap, business, dataRoot, binding: notifier.binding, base: notifier.base, pollMs: notifier.pollMs });
|
||||
const ok = await firstReply(notify, START_TIMEOUT_MS, "notifier");'''
|
||||
WRITES = ''' rmSync(hostFile(dataRoot), { force: true });
|
||||
writeHostState(dataRoot, state);
|
||||
'''
|
||||
CATCH_END = ''' if (broker.connected) broker.send({ op: "close" }, () => {});
|
||||
await ended(broker, CLOSE_TIMEOUT_MS);
|
||||
release();
|
||||
throw e;
|
||||
}
|
||||
'''
|
||||
|
||||
# (name, [(path, old, new)], [(test, pattern)], expect) with old None for a whole-file replacement.
|
||||
M = [
|
||||
("n00 BASE's startHost under the new tests", [BASE_HO], [NFORK, NSEND, READER, WRITE, EB, EN, KEPT, UB, UN], "killed"),
|
||||
("n01 the wait for a reply ignores error", [(HO, ' child.on("error", onError);\n', "")], [EB, EN], "killed"),
|
||||
("n02 no error listener on the broker", [(HO, ' broker.on("error", onChildError("broker"));\n', "")], [KEPT, UB], "killed"),
|
||||
("n03 no error listener on the notifier", [(HO, ' notify.on("error", onChildError("notifier"));\n', "")], [UN], "killed"),
|
||||
("n04 a broker error once up is dropped", [(HO, ' if (name === "broker") return breakChannel(`broker error: ${e.message}`);', ' if (name === "broker") return;')], [UB], "killed"),
|
||||
("n05 a notifier error once up only logs", [(HO, " log(`notifier error (${e.message}); stopping the host`);\n close(1);\n", " log(`notifier error (${e.message}); stopping the host`);\n")], [UN], "killed"),
|
||||
("n06 no reader check", [(HO, " if (!reader) throw new CliError(`broker has no reader for business ${business}`, 3);\n", "")], [READER], "killed"),
|
||||
("n07 the notifier fork runs before the shared try", [(HO, FORK, ""), (HO, " let notify = null;\n try {\n", " let notify = null;\n if (notifier) {\n" + FORK.replace(" ", " ") + " }\n try {\n")], [NFORK], "killed"),
|
||||
("n08 the notifier's wait starts before its send", [(HO, SEND_WAIT, ''' const started = firstReply(notify, START_TIMEOUT_MS, "notifier");
|
||||
notify.send({ op: "start", path: ready.path, cap: reader.cap, business, dataRoot, binding: notifier.binding, base: notifier.base, pollMs: notifier.pollMs });
|
||||
const ok = await started;''')], [NSEND], "killed"),
|
||||
("n09 host.json is written after the shared try", [(HO, WRITES, ""), (HO, " throw e;\n }\n\n // stopping:", " throw e;\n }\n" + WRITES.replace(" ", " ") + "\n // stopping:")], [WRITE], "killed"),
|
||||
("n10 a kept child error is not thrown", [(HO, " if (childError) throw childError;\n", "")], [KEPT], "killed"),
|
||||
("n11 the failure path leaves the notifier running", [(HO, ''' if (notify) {
|
||||
notify.kill("SIGTERM");
|
||||
await ended(notify, 5000);
|
||||
}
|
||||
''', "")], [WRITE, NSEND], "killed"),
|
||||
("n12 the failure path does not wait for the broker", [(HO, CATCH_END, CATCH_END.replace(" await ended(broker, CLOSE_TIMEOUT_MS);\n", ""))], [READER, WRITE], "killed"),
|
||||
("n13 the failure path keeps the lock", [(HO, CATCH_END, CATCH_END.replace(" release();\n", ""))], [READER, WRITE, NFORK], "killed"),
|
||||
]
|
||||
|
||||
def diffhash():
|
||||
d = subprocess.run(["git", "-C", WT, "diff", "--", ".", ":!node_modules"], capture_output=True).stdout
|
||||
return hashlib.sha256(d).hexdigest()
|
||||
|
||||
def run(test, pattern, log):
|
||||
pkg = os.path.join(WT, *test.split("/")[:2])
|
||||
with open(log, "ab") as f:
|
||||
f.write(f"== {test} /{pattern}/\n".encode()); f.flush()
|
||||
p = subprocess.Popen(["node", "--test", f"--test-name-pattern={pattern}", os.path.relpath(os.path.join(WT, test), pkg)], cwd=pkg, env=ENV,
|
||||
stdout=f, stderr=subprocess.STDOUT, start_new_session=True)
|
||||
try:
|
||||
rc = p.wait(timeout=600)
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(p.pid, signal.SIGKILL); p.wait(); return "timeout"
|
||||
# A pattern that matched nothing passes with no tests; count that as a fault.
|
||||
# node counts a test that hits its timeout as cancelled, not failed.
|
||||
tail = open(log, encoding="utf8", errors="replace").read().split(f"== {test} /{pattern}/\n")[-1]
|
||||
ran = sum(int(l.split()[2]) for l in tail.splitlines() if l.startswith(("ℹ pass ", "ℹ fail ", "ℹ cancelled ")))
|
||||
return "no-tests" if ran == 0 else rc
|
||||
|
||||
for _, edits, *_ in M:
|
||||
for path, *_ in edits:
|
||||
if os.path.exists(os.path.join(WT, path + ".orig")):
|
||||
sys.exit(f"STOP {path}.orig exists")
|
||||
start = diffhash()
|
||||
print(f"start diff {start[:16]} {time.strftime('%H:%M:%S', time.gmtime())}", flush=True)
|
||||
bad = []
|
||||
only = set(sys.argv[1:])
|
||||
for name, edits, tests, expect in M:
|
||||
if only and name.split()[0] not in only: continue
|
||||
srcs = {}
|
||||
ok = True
|
||||
for path, old, new in edits:
|
||||
full = os.path.join(WT, path)
|
||||
src = srcs.get(path) or open(full).read()
|
||||
if old is not None and src.count(old) != 1:
|
||||
print(f"{name}: ANCHOR {path} {src.count(old)}", flush=True); ok = False; break
|
||||
srcs[path] = new if old is None else src.replace(old, new)
|
||||
if not ok:
|
||||
bad.append(name); continue
|
||||
for path in srcs: os.rename(os.path.join(WT, path), os.path.join(WT, path + ".orig"))
|
||||
try:
|
||||
for path, text in srcs.items(): open(os.path.join(WT, path), "w").write(text)
|
||||
log = os.path.join(OUT, name.split()[0] + ".log")
|
||||
open(log, "w").close()
|
||||
rcs = [run(t, p, log) for t, p in tests]
|
||||
finally:
|
||||
for path in srcs: os.replace(os.path.join(WT, path + ".orig"), os.path.join(WT, path))
|
||||
if any(rc in ("timeout", "no-tests") for rc in rcs): got = "FAULT"
|
||||
elif all(rc != 0 for rc in rcs): got = "killed"
|
||||
elif all(rc == 0 for rc in rcs): got = "survived"
|
||||
else: got = "MIXED"
|
||||
print(f"{name}: {got} (expected {expect}) {list(zip([t.split('/')[-1] for t, _ in tests], rcs))}", flush=True)
|
||||
if got != expect: bad.append(name)
|
||||
end = diffhash()
|
||||
print(f"end diff {end[:16]} {'same' if end == start else 'CHANGED'} {time.strftime('%H:%M:%S', time.gmtime())}", flush=True)
|
||||
print("MUTANTS_OK" if not bad and end == start else f"MUTANTS_BAD {bad}")
|
||||
Reference in New Issue
Block a user