seat, cli, harness: git kill reach, close bound and host.lock (#1597)
Row 103, the row 100 follow-up. Built by Darkwing, packet
agents/darkwing/work/queue-103/r1/, candidate manifest
6e5ac8c0782bc595d7c0d711d457738ffb42f1ba5f38253912943dd55e55888a.
Sage approved on #1597, comment 28252; Dewey approved, comment 28260.
Host git runs through runGit: a detached group, SIGTERM at the limit,
SIGKILL after the grace, and one more SIGKILL to the group after a
signalled git closes. git-socket close() waits at most 2 s. startHost
claims bus-host/host.lock by hard link before the broker fork.
packages/cli/README.md and tests/host.test.mjs carry row 85's hunks
too; row 85 lands first.
Sage's gate rerun on 3a1e11b83 with rows 85, 102 and 103 applied,
Docker up: node suites bus 128, business 67, cli 165, control-board
124, conversation 182, discord 178, harness 73, ledger 78, mosaic 69,
queue 155, runs 43, seat 113, tasks 92, webui 108; scripts auth 15,
conductor 17, config 24, discord 66, extension-package 18,
foundation 44, queue 28, release 14, task 98. All pass, 0 fail.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
+15
-6
@@ -279,7 +279,8 @@ so the host never recurses into one: a clone with a submodule is kept, and
|
||||
`status` runs with `--ignore-submodules=all` and every call with
|
||||
`submodule.recurse=false`. The root is read with `GIT_CEILING_DIRECTORIES`
|
||||
at its parent, so a root that isn't a repository is never taken for the
|
||||
one above it. Each git call is killed after 100 s.
|
||||
one above it. Each git call gets SIGTERM after 100 s and SIGKILL 5 s
|
||||
later, as below.
|
||||
|
||||
The checks hold only while nothing else writes the clone. Each workspace
|
||||
is one instance's, and the launcher refuses `instance-running` before it
|
||||
@@ -332,15 +333,23 @@ The temp ref is deleted on every path, and so is the bundle file in
|
||||
`bus-host/`. No other ref moves, and the root is only read.
|
||||
|
||||
The repository's lock is a promise chain inside the host's process, so it
|
||||
assumes one launcher per dataRoot. `startHost` refuses a second bus host
|
||||
while `bus-host/host.json` names a live one, but that check isn't atomic:
|
||||
two launchers started at the same moment could both pass it, and their
|
||||
pushes to one repository would no longer be serialized.
|
||||
assumes one launcher per dataRoot. `startHost` takes `bus-host/host.lock`
|
||||
with a hard link, which fails if the file exists, and refuses while a live
|
||||
host holds it, so two launchers started at the same moment can't both run
|
||||
(`packages/cli/README.md`, `bus start`).
|
||||
|
||||
A host git call that runs past 100 s (`GIT_TIMEOUT_MS`) gets SIGTERM. git
|
||||
removes its lockfiles on SIGTERM. Only a git still running 5 s later
|
||||
(`GIT_GRACE_MS`) gets SIGKILL, which can leave a ref's `.lock` behind; that
|
||||
lock refuses every later push to the ref until someone removes it.
|
||||
lock refuses every later push to the ref until someone removes it. The
|
||||
checkouts (`checkout.mjs`) and the working repositories (`working.mjs`) run
|
||||
git the same way, through `runGit`. Each git is the leader of its own
|
||||
process group, and both signals go to the group, so a hook or helper git
|
||||
started can't outlive it. Once a signalled git has ended, the group gets
|
||||
SIGKILL once more, for a process that ignored SIGTERM. A git that ends on
|
||||
its own isn't signalled. Since git has its own group, a Ctrl-C at the
|
||||
terminal of a host started by hand doesn't reach it; a systemd stop still
|
||||
does, since it kills the unit's cgroup.
|
||||
|
||||
For a review, under the same lock, the host fetches `main` again and
|
||||
writes a bundle of the requested commit with one ref,
|
||||
|
||||
@@ -45,6 +45,9 @@ import { lstatSync, readdirSync, renameSync, rmSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
export const GIT_TIMEOUT_MS = 100_000;
|
||||
// How long a git has after SIGTERM before SIGKILL.
|
||||
export const GIT_GRACE_MS = 5000;
|
||||
export const GIT_LIMITS = Object.freeze({ timeoutMs: GIT_TIMEOUT_MS, graceMs: GIT_GRACE_MS });
|
||||
// A clone's config is a few hundred bytes, a little more per branch.
|
||||
export const CONFIG_MAX_BYTES = 64 * 1024;
|
||||
|
||||
@@ -78,20 +81,54 @@ export function gitEnv(env = process.env) {
|
||||
};
|
||||
}
|
||||
|
||||
// runGit(argv, env, limits) runs git with exactly `argv` and `env`, and
|
||||
// resolves { ok, out, err, why }; working.mjs runs its git through it too.
|
||||
// Async, so a long clone doesn't hold up the rest of the bus host.
|
||||
function git(args, env, extra = {}) {
|
||||
//
|
||||
// git runs as the leader of its own process group (detached), so a timeout
|
||||
// reaches what git started as well: SIGTERM to the group at
|
||||
// limits.timeoutMs, on which git removes its lockfiles, then SIGKILL to the
|
||||
// group limits.graceMs later. Once a git that was signalled has closed, the
|
||||
// group gets SIGKILL again, for a process that ignored SIGTERM and let go of
|
||||
// git's output. A git that ends on its own isn't signalled at all. One
|
||||
// consequence: a Ctrl-C at the terminal of a host run by hand no longer
|
||||
// reaches its git, though a systemd stop still does (it kills the cgroup).
|
||||
export function runGit(argv, env, limits = GIT_LIMITS) {
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn("git", [...SAFE, ...args], { env: { ...gitEnv(env), ...extra }, timeout: GIT_TIMEOUT_MS, killSignal: "SIGKILL", stdio: ["ignore", "pipe", "pipe"] });
|
||||
const child = spawn("git", argv, { env, detached: true, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let out = "";
|
||||
let err = "";
|
||||
let signalled = false;
|
||||
let kill = null;
|
||||
// A git that never started has no pid; an empty group is ESRCH. Neither
|
||||
// leaves anything to signal.
|
||||
const group = (signal) => {
|
||||
try {
|
||||
process.kill(-child.pid, signal);
|
||||
} catch {}
|
||||
};
|
||||
const term = setTimeout(() => {
|
||||
signalled = true;
|
||||
group("SIGTERM");
|
||||
kill = setTimeout(() => group("SIGKILL"), limits.graceMs);
|
||||
}, limits.timeoutMs);
|
||||
const settle = (r) => {
|
||||
clearTimeout(term);
|
||||
clearTimeout(kill);
|
||||
if (signalled) group("SIGKILL");
|
||||
resolve(r);
|
||||
};
|
||||
child.stdout.setEncoding("utf8").on("data", (d) => (out += d));
|
||||
child.stderr.setEncoding("utf8").on("data", (d) => (err += d));
|
||||
// A git that can't start may not emit close, so error settles it too.
|
||||
child.on("error", (e) => resolve({ ok: false, out, why: e.code ?? e.message }));
|
||||
child.on("close", (code, signal) => resolve({ ok: code === 0, out, why: signal ? `killed by ${signal}` : err.trim().split("\n").pop() || `exit ${code}` }));
|
||||
child.on("error", (e) => settle({ ok: false, out, err, why: e.code ?? e.message }));
|
||||
child.on("close", (code, signal) => settle({ ok: code === 0, out, err, why: signal ? `killed by ${signal}` : err.trim().split("\n").pop() || `exit ${code}` }));
|
||||
});
|
||||
}
|
||||
|
||||
// git in a clone or a root, with SAFE first. `limits` is for the tests.
|
||||
const git = (args, env, extra = {}, limits = GIT_LIMITS) => runGit([...SAFE, ...args], { ...gitEnv(env), ...extra }, limits);
|
||||
|
||||
const inClone = (dest) => [`--git-dir=${join(dest, ".git")}`, `--work-tree=${dest}`];
|
||||
|
||||
// The keys `git clone` writes, and what each may hold. branch.<name>.remote
|
||||
@@ -113,7 +150,7 @@ function allowed(key, value, root) {
|
||||
}
|
||||
|
||||
// Why the clone at `dest` isn't safe for the host to run git in, or null.
|
||||
async function unsafe(dest, root, env) {
|
||||
async function unsafe(dest, root, g) {
|
||||
const isDir = (p) => {
|
||||
try {
|
||||
return lstatSync(p).isDirectory();
|
||||
@@ -159,7 +196,7 @@ async function unsafe(dest, root, env) {
|
||||
} catch {
|
||||
return "config-changed";
|
||||
}
|
||||
const list = await git(["config", "--file", join(dest, ".git", "config"), "--list", "--null"], env);
|
||||
const list = await g(["config", "--file", join(dest, ".git", "config"), "--list", "--null"]);
|
||||
if (!list.ok) return "config-changed";
|
||||
const seen = new Set();
|
||||
for (const entry of list.out.split("\0").filter(Boolean)) {
|
||||
@@ -174,23 +211,23 @@ async function unsafe(dest, root, env) {
|
||||
|
||||
// The branch the root has checked out, null when it has none, or an error.
|
||||
// The ceiling stops git finding a repository above a root that isn't one.
|
||||
async function rootBranch(root, env) {
|
||||
async function rootBranch(root, g) {
|
||||
const ceiling = { GIT_CEILING_DIRECTORIES: dirname(root) };
|
||||
const r = await git(["-C", root, "symbolic-ref", "-q", "--short", "HEAD"], env, ceiling);
|
||||
const r = await g(["-C", root, "symbolic-ref", "-q", "--short", "HEAD"], ceiling);
|
||||
if (r.ok) return { branch: r.out.trim() };
|
||||
// symbolic-ref -q exits 1 with no output on a detached HEAD.
|
||||
const head = await git(["-C", root, "rev-parse", "--verify", "-q", "HEAD"], env, ceiling);
|
||||
const head = await g(["-C", root, "rev-parse", "--verify", "-q", "HEAD"], ceiling);
|
||||
return head.ok ? { branch: null } : { error: r.why };
|
||||
}
|
||||
|
||||
async function clone(project, root, workspace, dest, env) {
|
||||
const at = await rootBranch(root, env);
|
||||
async function clone(project, root, workspace, dest, g) {
|
||||
const at = await rootBranch(root, g);
|
||||
if (at.error) throw new CheckoutError(project, `the root ${root} can't be read: ${at.error}`);
|
||||
if (!at.branch) throw new CheckoutError(project, `the root ${root} has no branch checked out`);
|
||||
// A project id never starts with a dot, so the temp name is no project's.
|
||||
const tmp = join(workspace, `.${project}.clone-${randomBytes(4).toString("hex")}`);
|
||||
try {
|
||||
const r = await git(["clone", "-q", "--no-hardlinks", "--template=", "--branch", at.branch, "--", root, tmp], env);
|
||||
const r = await g(["clone", "-q", "--no-hardlinks", "--template=", "--branch", at.branch, "--", root, tmp]);
|
||||
if (!r.ok) throw new CheckoutError(project, `git clone of ${root} failed: ${r.why}`);
|
||||
renameSync(tmp, dest);
|
||||
} catch (e) {
|
||||
@@ -201,33 +238,35 @@ async function clone(project, root, workspace, dest, env) {
|
||||
return { outcome: "cloned", branch: at.branch };
|
||||
}
|
||||
|
||||
async function update(root, dest, env) {
|
||||
const problem = await unsafe(dest, root, env);
|
||||
async function update(root, dest, g) {
|
||||
const problem = await unsafe(dest, root, g);
|
||||
if (problem) return { outcome: "kept", reason: problem };
|
||||
// ls-files reads only the index: no filter, hook or submodule runs.
|
||||
const staged = await git([...inClone(dest), "ls-files", "--stage", "-z"], env);
|
||||
const staged = await g([...inClone(dest), "ls-files", "--stage", "-z"]);
|
||||
if (!staged.ok) return { outcome: "kept", reason: "status-failed" };
|
||||
if (staged.out.split("\0").some((e) => e.startsWith("160000 "))) return { outcome: "kept", reason: "submodule" };
|
||||
const at = await rootBranch(root, env);
|
||||
const at = await rootBranch(root, g);
|
||||
if (at.error) return { outcome: "kept", reason: "root-unreadable" };
|
||||
if (!at.branch) return { outcome: "kept", reason: "root-detached" };
|
||||
const head = await git([...inClone(dest), "symbolic-ref", "-q", "--short", "HEAD"], env);
|
||||
const head = await g([...inClone(dest), "symbolic-ref", "-q", "--short", "HEAD"]);
|
||||
if (!head.ok || head.out.trim() !== at.branch) return { outcome: "kept", reason: "other-branch" };
|
||||
const status = await git(["--no-optional-locks", ...inClone(dest), "status", "--porcelain=v1", "-z", "--untracked-files=all", "--ignore-submodules=all"], env);
|
||||
const status = await g(["--no-optional-locks", ...inClone(dest), "status", "--porcelain=v1", "-z", "--untracked-files=all", "--ignore-submodules=all"]);
|
||||
if (!status.ok) return { outcome: "kept", reason: "status-failed" };
|
||||
if (status.out !== "") return { outcome: "kept", reason: "dirty" };
|
||||
const before = (await git([...inClone(dest), "rev-parse", "HEAD"], env)).out.trim();
|
||||
if (!(await git([...inClone(dest), "fetch", "-q", "--no-tags", "--no-recurse-submodules", "origin"], env)).ok) return { outcome: "kept", reason: "fetch-failed" };
|
||||
if (!(await git([...inClone(dest), "merge", "-q", "--ff-only", "--no-edit", `refs/remotes/origin/${at.branch}`], env)).ok) return { outcome: "kept", reason: "not-fast-forward" };
|
||||
const after = (await git([...inClone(dest), "rev-parse", "HEAD"], env)).out.trim();
|
||||
const before = (await g([...inClone(dest), "rev-parse", "HEAD"])).out.trim();
|
||||
if (!(await g([...inClone(dest), "fetch", "-q", "--no-tags", "--no-recurse-submodules", "origin"])).ok) return { outcome: "kept", reason: "fetch-failed" };
|
||||
if (!(await g([...inClone(dest), "merge", "-q", "--ff-only", "--no-edit", `refs/remotes/origin/${at.branch}`])).ok) return { outcome: "kept", reason: "not-fast-forward" };
|
||||
const after = (await g([...inClone(dest), "rev-parse", "HEAD"])).out.trim();
|
||||
return { outcome: before === after ? "current" : "updated", branch: at.branch };
|
||||
}
|
||||
|
||||
// await ensureCheckouts({ workspace, projects, env }) for the business file's
|
||||
// projects ({ <id>: { root } }). Returns one { project, path, outcome,
|
||||
// reason?, branch? } per project, outcome cloned, updated, current or kept.
|
||||
// Rejects with CheckoutError when a missing clone can't be made.
|
||||
export async function ensureCheckouts({ workspace, projects, env = process.env }) {
|
||||
// await ensureCheckouts({ workspace, projects, env, limits }) for the
|
||||
// business file's projects ({ <id>: { root } }). Returns one { project,
|
||||
// path, outcome, reason?, branch? } per project, outcome cloned, updated,
|
||||
// current or kept. Rejects with CheckoutError when a missing clone can't be
|
||||
// made. `limits` ({ timeoutMs, graceMs }) is for the tests.
|
||||
export async function ensureCheckouts({ workspace, projects, env = process.env, limits = GIT_LIMITS }) {
|
||||
const g = (args, extra) => git(args, env, extra, limits);
|
||||
const out = [];
|
||||
for (const [project, { root }] of Object.entries(projects)) {
|
||||
const dest = join(workspace, project);
|
||||
@@ -238,7 +277,7 @@ export async function ensureCheckouts({ workspace, projects, env = process.env }
|
||||
if (e.code !== "ENOENT") throw new CheckoutError(project, `${dest} can't be read: ${e.code}`);
|
||||
exists = false;
|
||||
}
|
||||
out.push({ project, path: dest, ...(exists ? await update(root, dest, env) : await clone(project, root, workspace, dest, env)) });
|
||||
out.push({ project, path: dest, ...(exists ? await update(root, dest, g) : await clone(project, root, workspace, dest, g)) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -34,22 +34,24 @@
|
||||
// deletes it.
|
||||
//
|
||||
// A git past GIT_TIMEOUT_MS gets SIGTERM, on which git removes its lockfiles,
|
||||
// and SIGKILL only if it still runs GIT_GRACE_MS later. A SIGKILL leaves a
|
||||
// ref's .lock behind, and that lock refuses every later push to the ref.
|
||||
// and SIGKILL only if it still runs GIT_GRACE_MS later (checkout.mjs). Both
|
||||
// go to git's process group, so nothing git started outlives it. A SIGKILL
|
||||
// leaves a ref's .lock behind, and that lock refuses every later push to the
|
||||
// ref.
|
||||
//
|
||||
// The lock is a promise chain in this process, so it holds only while one
|
||||
// launcher serves a dataRoot. startHost refuses a second bus host while
|
||||
// bus-host/host.json names a live one; that check isn't atomic.
|
||||
// launcher serves a dataRoot. startHost refuses a second bus host for the
|
||||
// dataRoot: it takes bus-host/host.lock with a hard link, which fails if
|
||||
// the file exists.
|
||||
//
|
||||
// review, under the same lock: main again from the root, then a bundle of
|
||||
// the requested commit that leaves out the commits the reviewer said it has
|
||||
// (those the repository knows), written to the file the caller names.
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { closeSync, lstatSync, mkdirSync, openSync, readSync, renameSync, rmSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { gitEnv, GIT_TIMEOUT_MS } from "./checkout.mjs";
|
||||
import { gitEnv, GIT_LIMITS, runGit } from "./checkout.mjs";
|
||||
|
||||
// The largest bundle the host takes or sends.
|
||||
export const BUNDLE_MAX = 32 * 1024 * 1024;
|
||||
@@ -59,9 +61,6 @@ const V2 = "# v2 git bundle\n";
|
||||
const OID = /^[0-9a-f]{40}([0-9a-f]{24})?$/;
|
||||
const TASK_REF = /^vikunja:([1-9][0-9]*)\/([1-9][0-9]*)$/;
|
||||
const PROJECT = /^[a-z0-9][a-z0-9._-]{0,63}$/;
|
||||
// How long a git that got SIGTERM has to exit before SIGKILL.
|
||||
export const GIT_GRACE_MS = 5000;
|
||||
const LIMITS = { timeoutMs: GIT_TIMEOUT_MS, graceMs: GIT_GRACE_MS };
|
||||
|
||||
export class WorkingError extends Error {
|
||||
constructor(code, message = code) {
|
||||
@@ -84,28 +83,9 @@ const SAFE = [
|
||||
// Ahead of the fetch from a bundle. Tests replace it to show it matters.
|
||||
export const FSCK = ["-c", "transfer.fsckObjects=true"];
|
||||
|
||||
// `limits` is { timeoutMs, graceMs }; the tests shorten them.
|
||||
function git(args, env, { pre = [], limits = LIMITS } = {}) {
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn("git", [...SAFE, ...pre, ...args], { env: gitEnv(env), stdio: ["ignore", "pipe", "pipe"] });
|
||||
let out = "";
|
||||
let err = "";
|
||||
let kill = null;
|
||||
const term = setTimeout(() => {
|
||||
child.kill("SIGTERM");
|
||||
kill = setTimeout(() => child.kill("SIGKILL"), limits.graceMs);
|
||||
}, limits.timeoutMs);
|
||||
const settle = (r) => {
|
||||
clearTimeout(term);
|
||||
clearTimeout(kill);
|
||||
resolve(r);
|
||||
};
|
||||
child.stdout.setEncoding("utf8").on("data", (d) => (out += d));
|
||||
child.stderr.setEncoding("utf8").on("data", (d) => (err += d));
|
||||
child.on("error", (e) => settle({ ok: false, out, err, why: e.code ?? e.message }));
|
||||
child.on("close", (code, signal) => settle({ ok: code === 0, out, err, why: signal ? `killed by ${signal}` : err.trim().split("\n").pop() || `exit ${code}` }));
|
||||
});
|
||||
}
|
||||
// `limits` is { timeoutMs, graceMs }; the tests shorten them. runGit
|
||||
// (checkout.mjs) runs git in its own process group and signals the group.
|
||||
const git = (args, env, { pre = [], limits = GIT_LIMITS } = {}) => runGit([...SAFE, ...pre, ...args], gitEnv(env), limits);
|
||||
// git with this call's env and limits, for the helpers below.
|
||||
const bound = (env, limits) => (args, opts = {}) => git(args, env, { ...opts, limits });
|
||||
|
||||
@@ -187,7 +167,7 @@ const tip = async (repo, ref, g) => {
|
||||
// Resolves { ref, commit, previous }; rejects with WorkingError. `record`
|
||||
// runs under the lock once the ref moved, and a throw moves it back. `fsck`
|
||||
// and `limits` are for the tests.
|
||||
export function push({ dataRoot, business, project, root, taskRef, branch, bundle, record = async () => {}, env = process.env, fsck = FSCK, limits = LIMITS }) {
|
||||
export function push({ dataRoot, business, project, root, taskRef, branch, bundle, record = async () => {}, env = process.env, fsck = FSCK, limits = GIT_LIMITS }) {
|
||||
if (!PROJECT.test(project ?? "")) refuse("invalid-request", "not a project id");
|
||||
if (!validBranch(branch)) refuse("invalid-request", "not a branch name");
|
||||
const ref = `refs/heads/${workRef(taskRef)}`;
|
||||
@@ -240,7 +220,7 @@ export function push({ dataRoot, business, project, root, taskRef, branch, bundl
|
||||
// Writes a bundle of `commit` to `out`, with one ref, refs/mosaic/review.
|
||||
// `have` lists commits the reviewer's clone holds; those the repository
|
||||
// doesn't know are skipped. Resolves { ref, commit }.
|
||||
export function review({ dataRoot, business, project, root, taskRef, commit, have = [], out, env = process.env, limits = LIMITS }) {
|
||||
export function review({ dataRoot, business, project, root, taskRef, commit, have = [], out, env = process.env, limits = GIT_LIMITS }) {
|
||||
if (!PROJECT.test(project ?? "") || !OID.test(commit ?? "")) refuse("invalid-request");
|
||||
if (!Array.isArray(have) || have.length > HAVE_MAX || have.some((h) => !OID.test(h ?? ""))) refuse("invalid-request", "have must list commit ids");
|
||||
const repo = workingRepo(dataRoot, business, project);
|
||||
|
||||
@@ -331,3 +331,80 @@ test("every project in the business file gets its own clone at its own branch",
|
||||
assert.deepEqual(list.map((c) => [c.project, c.path, c.outcome, c.branch]), [["alpha", join(workspace, "alpha"), "cloned", "main"], ["beta", join(workspace, "beta"), "cloned", "dev"]]);
|
||||
assert.equal(git(join(workspace, "beta"), "symbolic-ref", "--short", "HEAD"), "dev");
|
||||
});
|
||||
|
||||
// A git on PATH that, for `git clone`, runs `body` in place of the real one.
|
||||
// Every other git is the real one.
|
||||
function slowClone(w, body) {
|
||||
const bin = join(w.dir, "bin");
|
||||
mkdirSync(bin);
|
||||
const real = spawnSync("sh", ["-c", "command -v git"], { encoding: "utf8" }).stdout.trim();
|
||||
writeFileSync(join(bin, "git"), `#!/bin/bash\ncase " $* " in *" clone "*) ${body} ;; esac\nexec '${real}' "$@"\n`, { mode: 0o755 });
|
||||
return { PATH: `${bin}:${process.env.PATH}` };
|
||||
}
|
||||
|
||||
// True once `pid` has gone, or is a zombie waiting for init to reap it.
|
||||
async function gone(pid, ms = 3000) {
|
||||
const end = Date.now() + ms;
|
||||
for (;;) {
|
||||
let stat = null;
|
||||
try {
|
||||
stat = readFileSync(`/proc/${pid}/stat`, "utf8");
|
||||
} catch {}
|
||||
if (stat === null || stat.slice(stat.lastIndexOf(")") + 2)[0] === "Z") return true;
|
||||
if (Date.now() > end) return false;
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
}
|
||||
|
||||
// SIGKILL to a sleep a test left behind, if it still runs.
|
||||
const kill = (pid) => {
|
||||
try {
|
||||
process.kill(pid, "SIGKILL");
|
||||
} catch {}
|
||||
};
|
||||
|
||||
// The pid a wrapper wrote to `file`, once it is there.
|
||||
async function pidIn(file) {
|
||||
for (let i = 0; i < 100 && !existsSync(file); i++) await new Promise((r) => setTimeout(r, 20));
|
||||
return Number(readFileSync(file, "utf8").trim());
|
||||
}
|
||||
|
||||
test("a clone past its time gets SIGTERM and refuses the launch, with nothing left in the workspace", async (t) => {
|
||||
const w = world(t);
|
||||
const env = slowClone(w, "exec sleep 30");
|
||||
await assert.rejects(ensureCheckouts({ workspace: w.workspace, projects: { stack: { root: w.root } }, env, limits: { timeoutMs: 500, graceMs: 20000 } }), (e) => e instanceof CheckoutError && /killed by SIGTERM/.test(e.message));
|
||||
assert.deepEqual(readdirSync(w.workspace), []);
|
||||
});
|
||||
|
||||
test("a clone that ignores SIGTERM gets SIGKILL once the grace period is over", { timeout: 20000 }, async (t) => {
|
||||
const w = world(t);
|
||||
const env = slowClone(w, "trap '' TERM; sleep 30");
|
||||
const started = Date.now();
|
||||
await assert.rejects(ensureCheckouts({ workspace: w.workspace, projects: { stack: { root: w.root } }, env, limits: { timeoutMs: 300, graceMs: 700 } }), (e) => e instanceof CheckoutError && /killed by SIGKILL/.test(e.message));
|
||||
assert.ok(Date.now() - started >= 1000, "SIGKILL waited out the grace period");
|
||||
});
|
||||
|
||||
// git is the leader of its own process group, and the group gets the signal:
|
||||
// a process git started (here a sleep) doesn't outlive the kill.
|
||||
test("a timeout signals git's process group, so a process git started dies with it", { timeout: 20000 }, async (t) => {
|
||||
const w = world(t);
|
||||
const file = join(w.dir, "child.pid");
|
||||
const env = slowClone(w, `sleep 30 </dev/null >/dev/null 2>&1 & echo $! > '${file}'; wait`);
|
||||
await assert.rejects(ensureCheckouts({ workspace: w.workspace, projects: { stack: { root: w.root } }, env, limits: { timeoutMs: 500, graceMs: 20000 } }), /killed by SIGTERM/);
|
||||
const pid = await pidIn(file);
|
||||
t.after(() => kill(pid));
|
||||
assert.equal(await gone(pid), true, "the sleep git started is gone");
|
||||
});
|
||||
|
||||
// git dies of SIGTERM, but a process it started ignores SIGTERM and holds
|
||||
// none of git's output, so git's close comes at once. The group gets
|
||||
// SIGKILL then, not after the grace period.
|
||||
test("once a signalled git has closed, its group gets SIGKILL, so a process that ignored SIGTERM dies too", { timeout: 20000 }, async (t) => {
|
||||
const w = world(t);
|
||||
const file = join(w.dir, "child.pid");
|
||||
const env = slowClone(w, `(trap '' TERM; exec sleep 30) </dev/null >/dev/null 2>&1 & echo $! > '${file}'; exec sleep 30`);
|
||||
await assert.rejects(ensureCheckouts({ workspace: w.workspace, projects: { stack: { root: w.root } }, env, limits: { timeoutMs: 500, graceMs: 20000 } }), /killed by SIGTERM/);
|
||||
const pid = await pidIn(file);
|
||||
t.after(() => kill(pid));
|
||||
assert.equal(await gone(pid, 1500), true, "the sleep that ignored SIGTERM is gone well inside the grace period");
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user