feat(discord): systemd user service with a supervised run; brakes exit 3 and are never retried (#1509)
QUEUE row 17, MVP iteration 2. scripts/discord-service.sh renders and installs mosaic-discord@<binding> from packages/discord/systemd/. The unit's main process is `run --supervised`, which applies the new recover policy first: a lock whose owner is gone is cleared and only the STOP written for that is removed; an operator STOP or a held binding refuses with exit 3, which RestartPreventExitStatus never retries. `recover` is also a CLI verb. First cut used ExecStartPre and looped live, since systemd honours the never-retry status only from the main process; replaced and re-verified before any message traffic. Suite 40/40, 95 node tests. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Claude Fable 5.1
@@ -14,9 +14,11 @@ CHAT-03 conversation controller exists.
|
||||
|
||||
```
|
||||
scripts/discord.sh check <binding>
|
||||
scripts/discord.sh run <binding>
|
||||
scripts/discord.sh run <binding> [--supervised]
|
||||
scripts/discord.sh stop <binding>
|
||||
scripts/discord.sh unlock <binding>
|
||||
scripts/discord.sh recover <binding>
|
||||
scripts/discord-service.sh render | install | uninstall | status <binding>
|
||||
```
|
||||
|
||||
`<binding>` names `<dataRoot>/discord/<binding>.json`. The wrapper passes
|
||||
@@ -29,7 +31,8 @@ scripts/discord.sh unlock <binding>
|
||||
as the message-content intent not being granted in the developer portal.
|
||||
- `run` refuses when `STOP` exists or an unresolved delivery cannot be
|
||||
reconciled. Otherwise it starts pi, connects, and serves turns until
|
||||
SIGTERM, SIGINT or `stop`. Run it in a tmux window; there is no service unit.
|
||||
SIGTERM, SIGINT or `stop`. Run it under the service unit below, or by hand
|
||||
in a tmux window.
|
||||
- `stop` writes `STOP` and sends SIGTERM to the process in `run.lock`, only
|
||||
when that pid is alive and both its start time and the boot id match the
|
||||
recorded ones; a reused pid, a pid from a previous boot, or a pid whose
|
||||
@@ -48,8 +51,54 @@ scripts/discord.sh unlock <binding>
|
||||
and releases itself if it is there, so nothing that starts during an
|
||||
unlock can hold the binding. `run` never reclaims a stale lock on its own;
|
||||
it refuses and names this command. Remove `STOP` to run again.
|
||||
- `recover` is the supervised pre-start; `run --supervised` performs it
|
||||
first, in the same process, and that is the form the service unit uses,
|
||||
because systemd honours a never-retry exit status only from the main
|
||||
process. It refuses, with exit 3 and
|
||||
touching nothing, while `STOP` is present or the binding is held by a live
|
||||
process, an alive pid whose identity cannot be verified, or an unreadable
|
||||
record. A lock whose owner is gone, or that has no record, it clears the
|
||||
way `unlock` does, then it removes the `STOP` it wrote for that so the run
|
||||
that follows can claim. It removes only a `STOP` that consists of the one
|
||||
line it wrote itself; a brake an operator wrote at any point, even during
|
||||
the recovery, stays and the start is refused. Nothing automatic ever
|
||||
removes an operator's `STOP`.
|
||||
|
||||
Exit codes: 0 ok, 1 operation failed, 2 invalid data or configuration, 4 usage.
|
||||
Exit codes: 0 ok, 1 operation failed, 2 invalid data or configuration, 3
|
||||
refused by a brake (`STOP` present or the binding held; a supervisor must
|
||||
not retry), 4 usage.
|
||||
|
||||
## Service unit
|
||||
|
||||
`scripts/discord-service.sh install` renders
|
||||
`packages/discord/systemd/[email protected]` with the repository
|
||||
path and the directory of `node`, writes it to
|
||||
`~/.config/systemd/user/[email protected]` (temp file, then rename;
|
||||
`--dir DIR` for another place, `--no-reload` to skip `daemon-reload`) and
|
||||
prints the commands that follow. One instance per binding:
|
||||
|
||||
```
|
||||
systemctl --user enable --now mosaic-discord@<binding> start now and at login
|
||||
systemctl --user status mosaic-discord@<binding>
|
||||
journalctl --user -u mosaic-discord@<binding> -f the log (stderr of `run`)
|
||||
systemctl --user stop mosaic-discord@<binding> SIGTERM; the turn in flight finishes; restartable
|
||||
scripts/discord.sh stop <binding> the brake: writes STOP; the unit stays down until STOP is removed
|
||||
scripts/discord-service.sh status <binding> unit state, STOP, run.lock
|
||||
```
|
||||
|
||||
What the unit does: `ExecStart` runs `run --supervised`,
|
||||
`Restart=on-failure` with 15 seconds between tries and at most five in ten
|
||||
minutes, and `RestartPreventExitStatus=3` so a brake is never retried. A
|
||||
crash (any other non-zero exit, a signal, an engine that died) restarts,
|
||||
and the supervised run clears the dead lock on its way in. `systemctl --user stop`
|
||||
sends SIGTERM only; the connector exits 0 and no `STOP` is written, so the
|
||||
next start needs no hand. The stop timeout is 3700 seconds, the largest turn
|
||||
timeout a binding may set plus margin; a normal stop takes as long as the
|
||||
turn in flight. The unit never reads the binding or the token; `run` does,
|
||||
at runtime, as before. Surviving logout and reboot needs
|
||||
`loginctl enable-linger`. `render` prints the unit without writing it;
|
||||
`uninstall` refuses while an instance is active. The log goes to journald:
|
||||
`run` writes no message text to stderr, only ids, counts and state.
|
||||
|
||||
## The binding
|
||||
|
||||
@@ -142,5 +191,5 @@ malformed line), restart replay, stop and ceiling.
|
||||
## Not in this piece
|
||||
|
||||
Tools, repository writes, announcements, attachments, slash commands, DMs,
|
||||
per-thread sessions, more than one server or seat, a service unit, a
|
||||
control-board row. Section 8 of the brief keeps the list.
|
||||
per-thread sessions, more than one server or seat, a control-board row.
|
||||
Section 8 of the brief keeps the list.
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
#!/usr/bin/env node
|
||||
// Usage:
|
||||
// mosaic-discord check <binding> [--config PATH] [--repo PATH]
|
||||
// mosaic-discord run <binding> [--config PATH] [--repo PATH]
|
||||
// mosaic-discord run <binding> [--config PATH] [--repo PATH] [--supervised]
|
||||
// mosaic-discord stop <binding> [--config PATH]
|
||||
// mosaic-discord unlock <binding> [--config PATH]
|
||||
// mosaic-discord recover <binding> [--config PATH]
|
||||
//
|
||||
// <binding> names <dataRoot>/discord/<binding>.json. The repository wrapper
|
||||
// is scripts/discord.sh.
|
||||
@@ -23,8 +24,18 @@
|
||||
// with unverifiable identity, or recorded in a file it cannot read. `run` never reclaims on its own, and a
|
||||
// claim that finds STOP after publishing releases itself, so unlock cannot
|
||||
// race a start. Remove STOP to run again.
|
||||
// recover: the supervised pre-start. Refuses while STOP is present or the
|
||||
// binding is held by a live or unverifiable process; clears a lock whose
|
||||
// owner is gone the way unlock does, then removes the STOP it wrote for
|
||||
// that, so the run that follows can claim. Never removes a STOP an operator
|
||||
// wrote. `run --supervised` does the same first thing itself; the service
|
||||
// unit (scripts/discord-service.sh) uses that form, because systemd only
|
||||
// honours a never-retry exit status from the main process, not from a
|
||||
// pre-start command.
|
||||
//
|
||||
// Exit codes: 0 ok; 1 operation failed; 2 invalid data or configuration; 4 usage.
|
||||
// Exit codes: 0 ok; 1 operation failed; 2 invalid data or configuration;
|
||||
// 3 refused by a brake (STOP present or the binding held; a supervisor must
|
||||
// not retry); 4 usage.
|
||||
|
||||
import { existsSync, mkdirSync, mkdtempSync, writeFileSync, statSync, readdirSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
@@ -36,17 +47,18 @@ import { createGateway, CONNECTOR_INTENTS } from "./gateway.mjs";
|
||||
import { createEngine, buildPiArgs } from "./engine-pi.mjs";
|
||||
import { assembleContext } from "./context.mjs";
|
||||
import { createConnector } from "./connector.mjs";
|
||||
import { ensureJournal, requestStop, stopRequested, readPid, stopTarget, writePid, clearPid, unlock } from "./journal.mjs";
|
||||
import { ensureJournal, requestStop, stopRequested, readPid, stopTarget, writePid, clearPid, unlock, recover, BRAKE_EXIT } from "./journal.mjs";
|
||||
|
||||
const USAGE = [
|
||||
"usage: mosaic-discord check <binding> [--config PATH] [--repo PATH]",
|
||||
" mosaic-discord run <binding> [--config PATH] [--repo PATH]",
|
||||
" mosaic-discord run <binding> [--config PATH] [--repo PATH] [--supervised]",
|
||||
" mosaic-discord stop <binding> [--config PATH]",
|
||||
" mosaic-discord unlock <binding> [--config PATH]",
|
||||
" mosaic-discord recover <binding> [--config PATH]",
|
||||
].join("\n");
|
||||
|
||||
function parse(argv) {
|
||||
const opts = { command: null, binding: null, config: defaultConfigPath(), repo: process.cwd() };
|
||||
const opts = { command: null, binding: null, config: defaultConfigPath(), repo: process.cwd(), supervised: false };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--config" || a === "--repo") {
|
||||
@@ -54,14 +66,17 @@ function parse(argv) {
|
||||
opts[a.slice(2)] = resolve(argv[++i]);
|
||||
} else if (a === "--help" || a === "-h") {
|
||||
opts.command = "help";
|
||||
} else if (a === "--supervised") {
|
||||
opts.supervised = true;
|
||||
} else if (a.startsWith("--")) throw new DiscordError(`unknown argument: ${a}\n${USAGE}`, 4);
|
||||
else if (opts.command === null) opts.command = a;
|
||||
else if (opts.binding === null) opts.binding = a;
|
||||
else throw new DiscordError(`unexpected argument: ${a}\n${USAGE}`, 4);
|
||||
}
|
||||
if (opts.command === "help") return opts;
|
||||
if (!["check", "run", "stop", "unlock"].includes(opts.command)) throw new DiscordError(USAGE, 4);
|
||||
if (!["check", "run", "stop", "unlock", "recover"].includes(opts.command)) throw new DiscordError(USAGE, 4);
|
||||
if (opts.binding === null) throw new DiscordError(`${opts.command} needs a binding name\n${USAGE}`, 4);
|
||||
if (opts.supervised && opts.command !== "run") throw new DiscordError(`--supervised applies to run only\n${USAGE}`, 4);
|
||||
return opts;
|
||||
}
|
||||
|
||||
@@ -136,7 +151,11 @@ async function run(opts) {
|
||||
const { binding, contextFiles, pi, journalDir, sessionDir } = prepare(opts);
|
||||
const token = readToken(binding);
|
||||
ensureJournal(journalDir);
|
||||
if (stopRequested(journalDir)) throw new DiscordError(`STOP is present in ${journalDir}; remove it to run`, 1);
|
||||
if (opts.supervised) {
|
||||
const outcome = recover(journalDir);
|
||||
if (outcome === "cleared") warn("supervised start: run.lock left by a process that is gone was removed");
|
||||
}
|
||||
if (stopRequested(journalDir)) throw new DiscordError(`STOP is present in ${journalDir}; remove it to run`, BRAKE_EXIT);
|
||||
writePid(journalDir, process.pid);
|
||||
const cleanupPid = () => clearPid(journalDir, process.pid);
|
||||
try {
|
||||
@@ -242,6 +261,16 @@ function unlockCommand(opts) {
|
||||
say("remove STOP to run again");
|
||||
}
|
||||
|
||||
function recoverCommand(opts) {
|
||||
const dataRoot = loadDataRoot(opts.config);
|
||||
const binding = loadBinding(bindingPath(dataRoot, opts.binding));
|
||||
const journalDir = bindingDataDir(dataRoot, binding.name);
|
||||
ensureJournal(journalDir);
|
||||
const outcome = recover(journalDir);
|
||||
if (outcome === "cleared") say("run.lock left by a process that is gone was removed; STOP is absent; ready to run");
|
||||
else say("no lock and no STOP; ready to run");
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const opts = parse(process.argv.slice(2));
|
||||
if (opts.command === "help") {
|
||||
@@ -251,6 +280,7 @@ async function main() {
|
||||
if (opts.command === "check") await check(opts);
|
||||
else if (opts.command === "run") await run(opts);
|
||||
else if (opts.command === "unlock") unlockCommand(opts);
|
||||
else if (opts.command === "recover") recoverCommand(opts);
|
||||
else stop(opts);
|
||||
return opts.command === "run" ? null : 0;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// One error class for the package. exitCode follows docs/TOOLS.md: 1 operation
|
||||
// failed, 2 invalid data or configuration, 4 usage.
|
||||
// failed, 2 invalid data or configuration, 3 refused by a brake (STOP or a
|
||||
// held binding; a supervisor must not retry), 4 usage.
|
||||
export class DiscordError extends Error {
|
||||
constructor(message, exitCode = 2, details = undefined) {
|
||||
super(message);
|
||||
|
||||
@@ -6,12 +6,15 @@
|
||||
// drops.jsonl one counter line per dropped or refused inbound message
|
||||
// admissions.jsonl one line per turn admitted, before the engine is asked
|
||||
// turns/<id>.json one write-once record per turn
|
||||
// STOP presence refuses new turns
|
||||
// STOP presence refuses new turns; one JSON line per writer
|
||||
// ({at, reason}), appended, so the last line names who
|
||||
// braked; `recover` removes only a STOP it wrote itself
|
||||
// notices.jsonl once-per-day fixed lines already attempted (ceiling)
|
||||
// run.lock/ ownership directory (mkdir is atomic) holding owner.json
|
||||
// {pid, start, boot}; `stop` signals only a live pid whose
|
||||
// start time and boot id match; a stale lock refuses `run`
|
||||
// until `unlock`, which is gated by STOP
|
||||
// until `unlock`, which is gated by STOP; `recover` is the
|
||||
// supervised form for a service unit's pre-start
|
||||
// Directories are 0700, files 0600. Lines are appended, never rewritten.
|
||||
|
||||
import {
|
||||
@@ -185,6 +188,21 @@ export function clearStop(dir) {
|
||||
if (existsSync(path)) unlinkSync(path);
|
||||
}
|
||||
|
||||
// The STOP lines as written, oldest first. A line that does not parse is
|
||||
// kept as {reason: null}: it was not written by this code, so it is never
|
||||
// treated as ours.
|
||||
export function readStop(dir) {
|
||||
if (!stopRequested(dir)) return [];
|
||||
return readFileSync(stopPath(dir), "utf8").split("\n").filter((l) => l.trim() !== "").map((l) => {
|
||||
try {
|
||||
const v = JSON.parse(l);
|
||||
return v && typeof v === "object" && typeof v.reason === "string" ? v : { reason: null };
|
||||
} catch {
|
||||
return { reason: null };
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// --- run lock ---
|
||||
// One directory, <dir>/run.lock, is the ownership primitive: mkdir is atomic,
|
||||
// so two starts cannot both create it. The owner record is published inside
|
||||
@@ -308,17 +326,23 @@ export function ownerAlive(rec, opts) {
|
||||
return ownerState(rec, opts) === "live";
|
||||
}
|
||||
|
||||
// Exit code for a refusal that a supervisor must not retry: STOP is present
|
||||
// or the binding is held. Distinct from 1 (operation failed) so a service
|
||||
// unit can restart after a crash and stay down after a brake.
|
||||
export const BRAKE_EXIT = 3;
|
||||
export const RECOVER_REASON = "recover";
|
||||
|
||||
export const UNLOCK_HINT = "if no connector is running for this binding, run `scripts/discord.sh unlock <binding>`";
|
||||
|
||||
// Explains why an existing lock refuses a new claim. Always a DiscordError.
|
||||
function lockRefusal(dir, opts) {
|
||||
const existing = readPid(dir);
|
||||
const state = ownerState(existing, opts);
|
||||
if (state === "absent") return new DiscordError(`run.lock exists without an owner record: a start is in progress or was interrupted; ${UNLOCK_HINT}`, 1);
|
||||
if (state === "invalid") return new DiscordError(`run.lock has an owner record that cannot be read; refusing. Inspect ${ownerPath(dir)} by hand`, 1);
|
||||
if (state === "live") return new DiscordError(`another connector is running for this binding (pid ${existing.pid})`, 1);
|
||||
if (state === "unknown") return new DiscordError(`run.lock belongs to pid ${existing.pid}, which is alive but whose identity cannot be verified; refusing`, 1);
|
||||
return new DiscordError(`run.lock belongs to pid ${existing.pid}, which is gone or is a different process now; ${UNLOCK_HINT}`, 1);
|
||||
if (state === "absent") return new DiscordError(`run.lock exists without an owner record: a start is in progress or was interrupted; ${UNLOCK_HINT}`, BRAKE_EXIT);
|
||||
if (state === "invalid") return new DiscordError(`run.lock has an owner record that cannot be read; refusing. Inspect ${ownerPath(dir)} by hand`, BRAKE_EXIT);
|
||||
if (state === "live") return new DiscordError(`another connector is running for this binding (pid ${existing.pid})`, BRAKE_EXIT);
|
||||
if (state === "unknown") return new DiscordError(`run.lock belongs to pid ${existing.pid}, which is alive but whose identity cannot be verified; refusing`, BRAKE_EXIT);
|
||||
return new DiscordError(`run.lock belongs to pid ${existing.pid}, which is gone or is a different process now; ${UNLOCK_HINT}`, BRAKE_EXIT);
|
||||
}
|
||||
|
||||
export function writePid(dir, pid, { now = Date.now(), identity = identityOf } = {}) {
|
||||
@@ -338,7 +362,7 @@ export function writePid(dir, pid, { now = Date.now(), identity = identityOf } =
|
||||
// this claim must not stand, however it interleaved with an unlock.
|
||||
if (stopRequested(dir)) {
|
||||
clearPid(dir, pid);
|
||||
throw new DiscordError(`STOP is present in ${dir}; remove it to run`, 1);
|
||||
throw new DiscordError(`STOP is present in ${dir}; remove it to run`, BRAKE_EXIT);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -348,9 +372,10 @@ export function writePid(dir, pid, { now = Date.now(), identity = identityOf } =
|
||||
// Refuses an owner record it cannot read. Otherwise removes the lock.
|
||||
// Returns the record that was cleared (null for a lock without one), or
|
||||
// false when there was no lock. STOP stays in place;
|
||||
// remove it to run again. `beforeRemove` and `identity` are test seams.
|
||||
export function unlock(dir, { beforeRemove = null, identity = identityOf } = {}) {
|
||||
requestStop(dir, "unlock");
|
||||
// remove it to run again. `beforeRemove` and `identity` are test seams;
|
||||
// `reason` is what the STOP line says (`recover` uses its own).
|
||||
export function unlock(dir, { beforeRemove = null, identity = identityOf, reason = "unlock" } = {}) {
|
||||
requestStop(dir, reason);
|
||||
const lock = lockPath(dir);
|
||||
if (!existsSync(lock)) return false;
|
||||
const rec = readPid(dir);
|
||||
@@ -363,6 +388,41 @@ export function unlock(dir, { beforeRemove = null, identity = identityOf } = {})
|
||||
return rec;
|
||||
}
|
||||
|
||||
// Supervised pre-start, for the service unit's ExecStartPre. Lets a run
|
||||
// that crashed (or a reboot) start again without a hand, while every
|
||||
// operator brake still holds. In order:
|
||||
// STOP present refuse with BRAKE_EXIT and touch nothing. `stop`
|
||||
// and `unlock` wrote it; nothing automatic removes it.
|
||||
// The one exception is a STOP made of exactly one
|
||||
// line that an earlier `recover` wrote and then died
|
||||
// before removing: that one is ours and goes.
|
||||
// no run.lock nothing to do.
|
||||
// owner live refuse: another connector holds the binding.
|
||||
// owner unknown, or record invalid refuse, as `unlock` would.
|
||||
// owner dead, or no record `unlock` (STOP written first, so a
|
||||
// claim that publishes meanwhile releases itself),
|
||||
// then that STOP is removed again, but only if it
|
||||
// still consists of the single line `unlock` wrote
|
||||
// for us. Any other line means an operator braked in
|
||||
// the meantime; STOP stays and the start is refused.
|
||||
// Returns "clean" or "cleared". Every refusal is a DiscordError with
|
||||
// BRAKE_EXIT so a supervisor does not retry it. `beforeRemove` and
|
||||
// `identity` are test seams.
|
||||
export function recover(dir, { identity = identityOf, beforeRemove = null } = {}) {
|
||||
const ours = (lines) => lines.length === 1 && lines[0].reason === RECOVER_REASON;
|
||||
if (stopRequested(dir)) {
|
||||
if (!ours(readStop(dir))) throw new DiscordError(`STOP is present in ${dir}; the brake is on. Remove it to run`, BRAKE_EXIT);
|
||||
clearStop(dir);
|
||||
}
|
||||
if (!existsSync(lockPath(dir))) return "clean";
|
||||
const state = ownerState(readPid(dir), { identity });
|
||||
if (state === "live" || state === "unknown" || state === "invalid") throw lockRefusal(dir, { identity });
|
||||
unlock(dir, { identity, reason: RECOVER_REASON, beforeRemove });
|
||||
if (!ours(readStop(dir))) throw new DiscordError(`STOP was written by an operator while recovering ${dir}; the brake stays. Remove it to run`, BRAKE_EXIT);
|
||||
clearStop(dir);
|
||||
return "cleared";
|
||||
}
|
||||
|
||||
// The verified live owner to signal, or null. Never returns a pid whose
|
||||
// identity cannot be proven.
|
||||
export function stopTarget(dir, opts) {
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# Mosaic Discord connector, one instance per binding: mosaic-discord@<binding>.
|
||||
# Rendered by scripts/discord-service.sh from packages/discord/systemd/;
|
||||
# @REPO@ and @PATH@ are filled in at install time. Edit the template and
|
||||
# reinstall; do not edit the installed copy.
|
||||
[Unit]
|
||||
Description=Mosaic Discord connector (%i)
|
||||
Documentation=file://@REPO@/packages/discord/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
StartLimitIntervalSec=600
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=@REPO@
|
||||
Environment=PATH=@PATH@
|
||||
# A supervised run first clears a lock left by a crash or a reboot, and
|
||||
# refuses with exit 3 while STOP is present or the binding is held. Exit 3
|
||||
# is never retried: the brake belongs to the operator. This happens inside
|
||||
# the main process on purpose: systemd honours RestartPreventExitStatus only
|
||||
# for the main process, so a refusing ExecStartPre would loop.
|
||||
ExecStart=@REPO@/scripts/discord.sh run %i --supervised
|
||||
Restart=on-failure
|
||||
RestartSec=15
|
||||
RestartPreventExitStatus=3
|
||||
# `systemctl --user stop` sends SIGTERM; the connector finishes the turn in
|
||||
# flight, then exits 0. The stop timeout covers the largest turn timeout a
|
||||
# binding may set (3600 s) plus margin; a normal stop takes seconds.
|
||||
KillSignal=SIGTERM
|
||||
KillMode=mixed
|
||||
TimeoutStopSec=3700
|
||||
NoNewPrivileges=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -137,6 +137,6 @@ test("cli: run refuses when STOP is present, before any network use", () => {
|
||||
assert.equal(r0.status, 0, r0.stderr);
|
||||
assert.match(r0.stdout, /STOP written/);
|
||||
const r = runCli(["run", "test-seat", "--config", dep.config, "--repo", repo]);
|
||||
assert.equal(r.status, 1, r.stderr);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /STOP is present/);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
// `recover`: the supervised pre-start behind the service unit. It may clear
|
||||
// a lock whose owner is gone and remove only the STOP it wrote for that;
|
||||
// every operator brake, and every held binding, refuses with exit 3.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, existsSync, mkdirSync, readFileSync, unlinkSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
ensureJournal, recover, requestStop, readStop, stopRequested, stopPath, lockPath, ownerPath, writePid, BRAKE_EXIT,
|
||||
RECOVER_REASON,
|
||||
} from "../src/journal.mjs";
|
||||
import { DiscordError } from "../src/errors.mjs";
|
||||
import { makeRoot, makeRepo, makeDeployment } from "./helpers.mjs";
|
||||
|
||||
const cli = fileURLToPath(new URL("../src/cli.mjs", import.meta.url));
|
||||
const worker = fileURLToPath(new URL("../fixtures/legacy-owner-worker.mjs", import.meta.url));
|
||||
const BOOT = "01234567-89ab-cdef-0123-456789abcdef";
|
||||
|
||||
function journal() {
|
||||
const dir = join(makeRoot(), "journal");
|
||||
ensureJournal(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
function deadPid() {
|
||||
const r = spawnSync(process.execPath, ["-e", "process.stdout.write(String(process.pid))"], { encoding: "utf8" });
|
||||
return Number(r.stdout);
|
||||
}
|
||||
|
||||
function publish(dir, rec) {
|
||||
mkdirSync(lockPath(dir), { recursive: true });
|
||||
writeFileSync(ownerPath(dir), JSON.stringify(rec) + "\n", { mode: 0o600 });
|
||||
}
|
||||
|
||||
const brake = (re) => (err) => err instanceof DiscordError && err.exitCode === BRAKE_EXIT && re.test(err.message);
|
||||
|
||||
// A live owner with the given identity arguments, held until `release`.
|
||||
function holdLock(dir, startArg, bootArg) {
|
||||
const done = join(dir, `done-${startArg}-${bootArg}`);
|
||||
const child = spawn(process.execPath, [worker, dir, done, startArg, bootArg], { stdio: ["ignore", "pipe", "inherit"] });
|
||||
const published = new Promise((resolve) => {
|
||||
let out = "";
|
||||
child.stdout.on("data", (d) => { out += d; if (out.includes("legacy-published")) resolve(); });
|
||||
});
|
||||
const closed = new Promise((resolve) => child.on("close", resolve));
|
||||
return { published, release: () => { writeFileSync(done, ""); return closed; } };
|
||||
}
|
||||
|
||||
test("recover: nothing to do is clean; a lock whose owner is gone or that has no record is cleared and STOP ends up absent", () => {
|
||||
const dir = journal();
|
||||
assert.equal(recover(dir), "clean");
|
||||
assert.equal(stopRequested(dir), false);
|
||||
|
||||
publish(dir, { pid: deadPid(), start: "1", boot: BOOT, at: "x" });
|
||||
assert.equal(recover(dir), "cleared");
|
||||
assert.equal(existsSync(lockPath(dir)), false, "dead owner's lock removed");
|
||||
assert.equal(stopRequested(dir), false, "the STOP written for the unlock is gone again");
|
||||
|
||||
mkdirSync(lockPath(dir));
|
||||
assert.equal(recover(dir), "cleared", "a lock without a record (interrupted start) is cleared too");
|
||||
assert.equal(existsSync(lockPath(dir)), false);
|
||||
assert.equal(stopRequested(dir), false);
|
||||
assert.equal(recover(dir), "clean");
|
||||
});
|
||||
|
||||
test("recover: an operator STOP refuses with exit 3 and is never removed, whatever the lock says", () => {
|
||||
const dir = journal();
|
||||
requestStop(dir, "cli stop");
|
||||
assert.throws(() => recover(dir), brake(/STOP is present/));
|
||||
assert.equal(stopRequested(dir), true);
|
||||
publish(dir, { pid: deadPid(), start: "1", boot: BOOT, at: "x" });
|
||||
assert.throws(() => recover(dir), brake(/STOP is present/));
|
||||
assert.equal(existsSync(lockPath(dir)), true, "the lock is not touched behind a brake");
|
||||
assert.equal(readStop(dir).length, 1, "nothing appended to STOP");
|
||||
|
||||
// A STOP that recover itself wrote and then failed to remove is ours, but
|
||||
// only when it is the whole file.
|
||||
const own = journal();
|
||||
requestStop(own, RECOVER_REASON);
|
||||
assert.equal(recover(own), "clean");
|
||||
assert.equal(stopRequested(own), false);
|
||||
requestStop(own, RECOVER_REASON);
|
||||
requestStop(own, "unlock");
|
||||
assert.throws(() => recover(own), brake(/STOP is present/));
|
||||
assert.equal(readStop(own).length, 2);
|
||||
|
||||
// A line this code did not write is never ours.
|
||||
const foreign = journal();
|
||||
writeFileSync(stopPath(foreign), "stop\n", { mode: 0o600 });
|
||||
assert.throws(() => recover(foreign), brake(/STOP is present/));
|
||||
assert.deepEqual(readStop(foreign), [{ reason: null }]);
|
||||
});
|
||||
|
||||
test("recover: a brake written during the unlock wins; STOP stays with both lines and the start is refused", () => {
|
||||
const dir = journal();
|
||||
publish(dir, { pid: deadPid(), start: "1", boot: BOOT, at: "x" });
|
||||
assert.throws(
|
||||
() => recover(dir, { beforeRemove: () => requestStop(dir, "cli stop") }),
|
||||
brake(/written by an operator while recovering/),
|
||||
);
|
||||
assert.equal(existsSync(lockPath(dir)), false, "the dead lock was removed before the brake was seen");
|
||||
assert.deepEqual(readStop(dir).map((l) => l.reason), [RECOVER_REASON, "cli stop"]);
|
||||
assert.throws(() => recover(dir), brake(/STOP is present/), "and it stays refused afterwards");
|
||||
});
|
||||
|
||||
test("recover: a held binding refuses with exit 3 and writes no STOP: live owner, alive pid without verifiable identity, unreadable record", async () => {
|
||||
const live = journal();
|
||||
const held = holdLock(live, "real", "real");
|
||||
await held.published;
|
||||
try {
|
||||
assert.throws(() => recover(live), brake(/another connector is running/));
|
||||
assert.equal(stopRequested(live), false, "recover never brakes a running connector");
|
||||
assert.equal(existsSync(ownerPath(live)), true);
|
||||
} finally {
|
||||
await held.release();
|
||||
}
|
||||
|
||||
const unknown = journal();
|
||||
const legacy = holdLock(unknown, "real", "-");
|
||||
await legacy.published;
|
||||
try {
|
||||
assert.throws(() => recover(unknown), brake(/cannot be verified/));
|
||||
assert.equal(stopRequested(unknown), false);
|
||||
assert.equal(existsSync(ownerPath(unknown)), true);
|
||||
} finally {
|
||||
await legacy.release();
|
||||
}
|
||||
|
||||
const invalid = journal();
|
||||
mkdirSync(lockPath(invalid));
|
||||
writeFileSync(ownerPath(invalid), "{not json", { mode: 0o600 });
|
||||
assert.throws(() => recover(invalid), brake(/cannot be read/));
|
||||
assert.equal(stopRequested(invalid), false);
|
||||
assert.equal(readFileSync(ownerPath(invalid), "utf8"), "{not json");
|
||||
|
||||
// Our own live claim is a held binding too.
|
||||
const mine = journal();
|
||||
writePid(mine, process.pid);
|
||||
assert.throws(() => recover(mine), brake(/another connector is running/));
|
||||
assert.equal(stopRequested(mine), false);
|
||||
});
|
||||
|
||||
test("cli: recover exits 0 when ready, 3 behind a brake or a held binding, and run's own STOP refusal is 3", () => {
|
||||
const root = makeRoot();
|
||||
const repo = makeRepo(root);
|
||||
const dep = makeDeployment(root);
|
||||
const run = (args) => spawnSync(process.execPath, [cli, ...args, "--config", dep.config, "--repo", repo], { encoding: "utf8" });
|
||||
let r = run(["recover", "test-seat"]);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.match(r.stdout, /ready to run/);
|
||||
|
||||
r = run(["stop", "test-seat"]);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
r = run(["recover", "test-seat"]);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /the brake is on/);
|
||||
r = run(["run", "test-seat"]);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
|
||||
const journalDir = join(dep.dataRoot, "discord", "test-seat");
|
||||
unlinkSync(stopPath(journalDir));
|
||||
publish(journalDir, { pid: deadPid(), start: "1", boot: BOOT, at: "x" });
|
||||
r = run(["recover", "test-seat"]);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.match(r.stdout, /was removed; STOP is absent/);
|
||||
assert.equal(existsSync(lockPath(journalDir)), false);
|
||||
assert.equal(stopRequested(journalDir), false);
|
||||
|
||||
r = run(["recover"]);
|
||||
assert.equal(r.status, 4);
|
||||
|
||||
// The service unit's form: the same policy inside the main process, so
|
||||
// the exit status is the one systemd never retries.
|
||||
r = run(["stop", "test-seat"]);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
r = run(["run", "test-seat", "--supervised"]);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /the brake is on/);
|
||||
assert.equal(stopRequested(journalDir), true, "the operator's STOP stays");
|
||||
unlinkSync(stopPath(journalDir));
|
||||
publish(journalDir, { pid: process.pid, at: "x" });
|
||||
r = run(["run", "test-seat", "--supervised"]);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /cannot be verified/);
|
||||
assert.equal(existsSync(ownerPath(journalDir)), true, "a held binding is left alone");
|
||||
assert.equal(stopRequested(journalDir), false);
|
||||
r = run(["stop", "test-seat", "--supervised"]);
|
||||
assert.equal(r.status, 4);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user