morgana: row 257 build (Q198 S1, #1762): @mosaic/supervisor pressure parsers and pure D4 rules evaluator, tests and mutant plan; tests wait for Sage's slot

This commit is contained in:
2026-10-21 17:09:21 -05:00
parent cca206cdce
commit e98e37ca9f
10 changed files with 1090 additions and 0 deletions
+120
View File
@@ -0,0 +1,120 @@
# Row 257: Q198 S1 pressure parser and pure rules evaluator
Issue #1762. Brief: `docs/plans/2026-10-21_post-v1-row-briefs.md`, "Brief: Q198 S1 pressure parser and
pure rules evaluator" and "Gate: Q198 S1" (cca206cdc). Design: `docs/plans/2026-10-21_q198-system-manager.md`
r6.1, section 8 row S1 and section 4 (D4). Owner Morgana, reviewer Filbert, gate owner Sage (sage-2196).
Post-v1: MVP rows and re-pin windows go ahead of it on the slot. Base: refactor cca206cdc.
## Recorded choices (sage-2196 accepted each)
1. **Placement.** A new package, `packages/supervisor` (`@mosaic/supervisor`), shaped like
packages/runs. Nothing on refactor parsed loadavg, PSI, meminfo or cgroup limits (`git grep`; the
only hit was an awk line in scripts/case2-precheck.sh). S2 adds the sampler process to the same
package. Packages aren't registered centrally, so no other file changes.
2. **Provider scope** (429, quota, Retry-After, the half-open probe) is out. The brief puts it in S3
and S5.
3. **PSI** is not required on a host that has never shown it, so a kernel built without PSI skips the
PSI rows instead of holding forever. Once a sample has carried PSI, it is required: a later missing
or stale PSI signal holds with `health-unknown`. That is the case sage-2196 asked for.
- Tradeoff: I latch on the first sample that carries PSI, not only on the very first sample. So PSI
that appears late and then disappears also holds. This is stricter than "present at the first
sample", and it never skips a signal the host has shown.
4. **Memory under a cgroup.** Available memory = min(MemAvailable, limit - memory.current) at each
limited level. memory.current counts page cache, so this errs toward holding.
- Follow-up, not in this row: subtract reclaimable cache using memory.stat `inactive_file`, if S2's
live samples show holds that the host's MemAvailable doesn't support.
Other readings of the D4 table, made in the build:
- **Memory floors.** "Whichever demands more free memory" means the larger floor applies. Hold below
max(15% x limit, 1 GiB); release above max(25% x limit, 2 GiB).
- **Edges as written.** Hold at ≥ for load and PSI, < for memory. Release at ≤ 0.55 for load, < for
PSI, > for memory. Critical below 5% of the limit and below 1 GiB of disk.
- **Sample counts.** "For 3 samples" means 3 consecutive samples. The count resets after a gap of more
than 6 s (the stale limit) between samples. The PSI row counts as one row: any of its three
conditions on 3 consecutive samples holds, and the reasons name the resources over on the latest
sample. Counting per resource would be weaker.
- **Memory holds on one sample.** The table's memory row has no sample count.
- **Release** needs every applicable release condition on every sample for 30 s, by the samples' own
times (`>= 30000` ms after the first qualifying sample). A dead-band sample or any hold restarts the
window, and so does a gap of more than 6 s.
- **Start held.** `initialState()` is held, because nothing has been seen yet. The first healthy
sample starts the 30 s window.
- **Missing or stale.** Required signals are load, memory, oom and disk, plus PSI once seen. A signal
holds at once with `health-unknown` if it is:
- missing;
- non-numeric, negative or zero where a divisor;
- more than 6 s older than its sample (6 s exactly is fresh);
- stamped later than its sample.
Disk and OOM are required because the critical row can't be evaluated without them.
- **Host-only OOM.** `/proc/vmstat` `oom_kill` gives the OOM counter when there's no cgroup
memory.events. The parser returns null for a kernel without it, and the sampler then has no OOM
signal (health-unknown).
- **Critical** latches. It ends only when the caller passes `reconciled` with no critical condition in
the sample, then goes through held and the normal 30 s release. The evaluator never sets
`reconciled` and has no kill-switch path. Critical reasons accumulate while latched.
- **OOM counter.** The first OOM sample is the baseline. A falling counter (a recreated cgroup) resets
the baseline with a note, not an event.
- **Clock going backwards.** A sample earlier than the last one holds with `health-unknown` (unknown
`clock`, note `clock-backwards`).
- **Swap activity** is parsed (`pswpin`, `pswpout`) and has no rule. D4 lists it as a signal with no
threshold.
- **Settings.** `resolveRules` refuses:
- unknown keys and non-numbers;
- a release at or past its hold (load, each PSI resource);
- memory release floors at or below the hold floors;
- fractions outside (0, 1);
- PSI values outside 0 to 100;
- sample counts that aren't integers ≥ 1;
- a negative release window;
- a stale limit at or below the sample interval;
- a critical memory fraction at or above the hold fraction;
- a non-positive disk floor.
- **Large byte values.** Parsed with `Number`. A memory.max above 2^53 loses precision, but it only
takes part in a min against MemTotal.
## Files
- `packages/supervisor/package.json`, `README.md`
- `src/errors.mjs`: `SupervisorError` with `kind` parse, rules or sample.
- `src/pressure.mjs`: the nine parsers, `effectiveCpus` and `effectiveMemory`.
- `src/rules.mjs`: `DEFAULT_RULES`, `resolveRules`, `initialState`, `evaluate`.
- `src/index.mjs`
- `tests/pressure.test.mjs`: parsers on kernel-shaped fixture text, with malformed, truncated and
repeated-key cases. Effective CPU cases cover the host-only fallback, cpuset, fractional and nested
`cpu.max`, and `max`. Effective memory cases cover the host-only fallback, `max`, nested levels,
usage over the limit, a limit above MemTotal and a missing memory.current.
- `tests/rules.test.mjs`: the D4 table on a fake clock.
- Each hold and release threshold on both sides of its edge.
- Three-sample counting and the gap reset.
- The 30 s release at 29 999 and 30 000 ms, and the dead-band restart.
- Missing, stale, future and malformed signals.
- PSI never present, and PSI lost after it was present.
- Critical OOM, memory and disk at their edges, the reconciliation latch and the counter reset.
- A backwards clock, input state left unchanged, and settings refusals and overrides.
- `tests/purity.test.mjs`: a static check that src imports only its own files and uses no
fs, child_process, net, os, process, dynamic import, Date or timers.
No test reads /proc, /sys or a cgroup, starts a build, opens a network socket or generates load. No new
dependency.
## Mutants (each must fail its named test; the brief requires M1 and M2)
| Mutant | Edit | Named test |
|---|---|---|
| M1 hold edge | `ratio >= rules.load.hold` becomes `>` | load holds at 0.80 for 3 samples and not at 0.7999 |
| M2 early release | `now - next.releaseSince >= rules.releaseMs` becomes `>= 0` | the first state is held, and healthy samples open it after exactly 30 s |
| M3 memory floor | the hold floor `Math.max` becomes `Math.min` | memory holds below the larger floor, on one sample |
| M4 sample count | `next.loadCount >= rules.load.samples` becomes `>= 1` | load holds at 0.80 for 3 samples and not at 0.7999 |
| M5 stale | the `now - signal.at > rules.staleMs` check is removed | a signal is stale at more than 6 s old and fresh at exactly 6 s |
| M6 effective CPU | the quota `Math.min` becomes `Math.max` | effective CPUs take the smallest of host, cpuset and every quota level |
| M7 critical latch | the `!reconciled` critical branch is removed | critical stays until reconciliation, then still waits the 30 s release |
## Gate (Sage's slot, on CLEAR only)
- The three test files, each whole, then `node --test tests/` in packages/supervisor as the package
suite.
- Then M1 to M7, each on its named test, restored after each.
- umask 022, TMPDIR from `mktemp -d /mnt/storage/scratch/tmp/gXXXXXX`.
- The kit is /mnt/storage/scratch/morgana-r257 (slot257.sh, mutate.py). It refuses unless HEAD is the
candidate and the tree is clean, and stops after two unexpected failures (mos-0823).
- Static checks before the slot: `node --check` on every file, run from `/`. No test has run yet.
+80
View File
@@ -0,0 +1,80 @@
# Supervisor
Q198 S1 (row 257, #1762): parsers for the host pressure inputs in design D4
(`docs/plans/2026-10-21_q198-system-manager.md`), and a pure evaluator that turns timestamped samples
into a hold or release decision. Nothing here samples the host, runs a process or timer, or enforces
anything. The live sampler, the Console view and admission are S2 and S3.
```sh
node --test packages/supervisor/tests/
```
## Parsers
Each parser takes file text the caller already read. Text that isn't the shape the kernel writes throws
a `SupervisorError` with `kind: "parse"`; a sampler treats that as a missing signal.
| Export | Input | Returns |
|---|---|---|
| `parseLoadavg` | `/proc/loadavg` | `{ load1, load5, load15 }` |
| `parseMeminfo` | `/proc/meminfo` | `{ memTotal, memAvailable, swapTotal, swapFree }` in bytes |
| `parsePsi(text, resource)` | `/proc/pressure/{cpu,memory,io}` or a cgroup `*.pressure` | `{ some, full }`, each `{ avg10, avg60, avg300, total }`; `full` is null only for cpu |
| `parseCpuList` | `/sys/devices/system/cpu/online`, `cpuset.cpus.effective` | how many CPUs the list names |
| `parseCpuMax` | `cpu.max` | the quota in CPUs, or null for `max` |
| `parseMemoryMax` | `memory.max` | bytes, or null for `max` |
| `parseMemoryCurrent` | `memory.current` | bytes |
| `parseMemoryEvents` | `memory.events` | `{ oom, oomKill }` |
| `parseVmstat` | `/proc/vmstat` | `{ pswpin, pswpout, oomKill }`; `oomKill` is the host-only OOM counter, null when absent |
`effectiveCpus({ online, cpuset, cpuMax })` is the smallest of the host count, the cpuset and the quota
at every cgroup level given; a fractional quota stays fractional. `effectiveMemory({ meminfo, levels })`
takes cgroup levels as `{ max, current }` text. The limit is the smallest of `MemTotal` and every numeric
`memory.max`. Available memory is the smallest of `MemAvailable` and `max - current` at each limited level.
`memory.current` counts page cache, so this errs toward holding. With no cgroup input both functions use
the host's figures.
## Evaluator
`evaluate(state, sample, rules, { reconciled })` returns `{ state, decision }`. It starts from
`initialState()`, which is held: no signal has been seen yet. Time is the sample's `at` in milliseconds.
The evaluator never reads the clock.
A sample has these signals: `load { at, load1, cpus }`, `memory { at, available, limit }`,
`psi { at, cpuSome, memoryFull, ioFull }` (avg10 percentages), `oom { at, kills }` and `disk { at, free }`.
| Signal | Hold | Release |
|---|---|---|
| `load1 / cpus` | ≥ 0.80 on 3 consecutive samples | ≤ 0.55 |
| available memory | < max(15% of limit, 1 GiB), on one sample | > max(25% of limit, 2 GiB) |
| PSI avg10 | cpu some ≥ 20, memory full ≥ 2 or io full ≥ 10, on 3 consecutive samples | cpu some < 10, memory full < 0.5 and io full < 3 |
| critical | OOM kill counter rises, memory < 5% of limit, or disk free < 1 GiB | the caller passes `reconciled`, then the normal release |
- **Release** needs every applicable release condition on every sample for 30 s. A dead-band
sample (between hold and release) or a new hold restarts the window.
- **Missing or stale:** a required signal that is missing, malformed, more than 6 s older than the
sample, or later than it holds at once with `health-unknown`. `load`, `memory`, `oom` and `disk` are
always required. PSI becomes required once a sample has carried it. A host without PSI skips the PSI
rows, and one that loses it holds.
- **Consecutive counts** reset after a gap of more than 6 s between samples. The PSI count runs across
resources: any PSI hold condition on 3 consecutive samples holds.
- **OOM:** the first OOM sample is the baseline. A counter that falls (a recreated cgroup) resets the
baseline with the note `oom-counter-reset`.
- **Critical** never ends on thresholds alone. `reconciled` only counts once no critical condition
remains, and the 30 s release still follows. The evaluator never sets it and never touches a kill
switch.
- **A clock that goes backwards** holds with `health-unknown` and the note `clock-backwards`.
`decision` is `{ mode, reasons, unknown, critical, releaseAt, notes }`.
- `mode` is `open`, `held` or `critical`.
- `reasons` are `health-unknown`, `load-high`, `memory-low`, `psi-cpu`, `psi-memory`, `psi-io`,
`hysteresis`, `release-pending` or the critical codes `critical-oom`, `critical-memory` and
`critical-disk`.
- `releaseAt` is set while a release is pending.
`DEFAULT_RULES` is the D4 table, frozen. `resolveRules(overrides)` merges overrides key by key. It refuses
with `kind: "rules"` on an unknown key, a non-number or a contradictory pair, for example:
- a release threshold at or past its hold;
- a release floor at or below its hold floor;
- a critical memory fraction at or above the hold fraction;
- a stale limit at or below the sample interval.
Provider scope (429, quota, Retry-After) is not here; it belongs to S3 and S5.
+11
View File
@@ -0,0 +1,11 @@
{
"name": "@mosaic/supervisor",
"version": "0.1.0",
"private": true,
"description": "Q198 S1: parsers for host pressure inputs and a pure hold and release rules evaluator. No sampler, process or enforcement.",
"license": "UNLICENSED",
"type": "module",
"engines": { "node": ">=24" },
"exports": { ".": "./src/index.mjs" },
"scripts": { "test": "node --test tests/" }
}
+10
View File
@@ -0,0 +1,10 @@
// Every refusal in this package is a SupervisorError. `kind` is "parse" for
// input text that isn't the shape the kernel writes, and "rules" for settings
// the evaluator refuses.
export class SupervisorError extends Error {
constructor(message, kind) {
super(message);
this.name = "SupervisorError";
this.kind = kind;
}
}
+10
View File
@@ -0,0 +1,10 @@
// @mosaic/supervisor: Q198 S1. Parsers for host pressure inputs and a pure
// rules evaluator. Nothing here reads a file, the clock or the environment,
// and nothing enforces: no sampler, process, timer, bus or Console (S2, S3).
export { SupervisorError } from "./errors.mjs";
export {
parseLoadavg, parseMeminfo, parsePsi, parseCpuList, parseCpuMax, parseMemoryMax, parseMemoryCurrent,
parseMemoryEvents, parseVmstat, effectiveCpus, effectiveMemory,
} from "./pressure.mjs";
export { DEFAULT_RULES, resolveRules, initialState, evaluate } from "./rules.mjs";
+165
View File
@@ -0,0 +1,165 @@
// Parsers for the host pressure inputs Q198 D4 names, and the effective CPU
// count and memory limit they give. Every function takes file text the caller
// already read and returns numbers. Nothing here opens a file. Text that isn't
// the shape the kernel writes throws a SupervisorError of kind "parse"; the
// caller turns that into a missing signal.
import { SupervisorError } from "./errors.mjs";
const fail = (what, why) => { throw new SupervisorError(`${what}: ${why}`, "parse"); };
function text(what, value) {
if (typeof value !== "string") fail(what, "expected file text");
return value.endsWith("\n") ? value.slice(0, -1) : value;
}
function integer(what, value) {
if (!/^\d+$/.test(value)) fail(what, `not a non-negative integer: ${JSON.stringify(value)}`);
return Number(value);
}
function decimal(what, value) {
if (!/^\d+(\.\d+)?$/.test(value)) fail(what, `not a non-negative number: ${JSON.stringify(value)}`);
return Number(value);
}
// "key value" lines (memory.events, /proc/vmstat). A repeated key throws.
function keyedIntegers(what, raw) {
const out = new Map();
for (const line of text(what, raw).split("\n")) {
const m = /^([a-z0-9_]+) (\d+)$/.exec(line);
if (!m) fail(what, `bad line ${JSON.stringify(line)}`);
if (out.has(m[1])) fail(what, `repeated key ${m[1]}`);
out.set(m[1], integer(what, m[2]));
}
return out;
}
// /proc/loadavg: "0.52 0.58 0.59 1/1234 56789".
export function parseLoadavg(raw) {
const m = /^(\d+\.\d+) (\d+\.\d+) (\d+\.\d+) (\d+)\/(\d+) (\d+)$/.exec(text("loadavg", raw));
if (!m) fail("loadavg", "not the /proc/loadavg shape");
return { load1: Number(m[1]), load5: Number(m[2]), load15: Number(m[3]) };
}
// /proc/meminfo. Values are returned in bytes; lines without a unit (the
// HugePages counts) are kept as plain counts.
export function parseMeminfo(raw) {
const fields = new Map();
for (const line of text("meminfo", raw).split("\n")) {
const m = /^([A-Za-z0-9_()]+):\s+(\d+)( kB)?$/.exec(line);
if (!m) fail("meminfo", `bad line ${JSON.stringify(line)}`);
if (fields.has(m[1])) fail("meminfo", `repeated key ${m[1]}`);
fields.set(m[1], integer("meminfo", m[2]) * (m[3] ? 1024 : 1));
}
for (const key of ["MemTotal", "MemAvailable"]) if (!fields.has(key)) fail("meminfo", `no ${key}`);
return {
memTotal: fields.get("MemTotal"),
memAvailable: fields.get("MemAvailable"),
swapTotal: fields.get("SwapTotal") ?? null,
swapFree: fields.get("SwapFree") ?? null,
};
}
// /proc/pressure/{cpu,memory,io} or a cgroup's *.pressure file. The cpu file
// may lack the "full" line on older kernels; memory and io must have it.
export function parsePsi(raw, resource) {
if (!["cpu", "memory", "io"].includes(resource)) fail("psi", `unknown resource ${JSON.stringify(resource)}`);
const what = `psi ${resource}`;
const out = { some: null, full: null };
for (const line of text(what, raw).split("\n")) {
const m = /^(some|full) avg10=(\S+) avg60=(\S+) avg300=(\S+) total=(\S+)$/.exec(line);
if (!m) fail(what, `bad line ${JSON.stringify(line)}`);
if (out[m[1]]) fail(what, `repeated ${m[1]} line`);
out[m[1]] = { avg10: decimal(what, m[2]), avg60: decimal(what, m[3]), avg300: decimal(what, m[4]), total: integer(what, m[5]) };
}
if (!out.some) fail(what, "no some line");
if (!out.full && resource !== "cpu") fail(what, "no full line");
return out;
}
// A CPU list such as /sys/devices/system/cpu/online or cpuset.cpus.effective:
// "0-3,6,8-9". Returns how many distinct CPUs it names.
export function parseCpuList(raw) {
const body = text("cpu list", raw);
if (body === "") fail("cpu list", "empty");
const cpus = new Set();
for (const part of body.split(",")) {
const m = /^(\d+)(?:-(\d+))?$/.exec(part);
if (!m) fail("cpu list", `bad range ${JSON.stringify(part)}`);
const lo = integer("cpu list", m[1]);
const hi = m[2] === undefined ? lo : integer("cpu list", m[2]);
if (hi < lo) fail("cpu list", `range ${part} runs backwards`);
for (let c = lo; c <= hi; c++) cpus.add(c);
}
return cpus.size;
}
// cgroup v2 cpu.max: "max 100000" (no quota) or "50000 100000". Returns the
// CPUs the quota allows, or null for no quota.
export function parseCpuMax(raw) {
const m = /^(max|\d+) (\d+)$/.exec(text("cpu.max", raw));
if (!m) fail("cpu.max", "not the cpu.max shape");
const period = integer("cpu.max", m[2]);
if (period === 0) fail("cpu.max", "period is 0");
return m[1] === "max" ? null : integer("cpu.max", m[1]) / period;
}
// cgroup v2 memory.max: "max" (no limit) or bytes. Returns null for "max".
export function parseMemoryMax(raw) {
const body = text("memory.max", raw);
return body === "max" ? null : integer("memory.max", body);
}
// cgroup v2 memory.current: bytes.
export function parseMemoryCurrent(raw) {
return integer("memory.current", text("memory.current", raw));
}
// cgroup v2 memory.events. oom_kill is the counter the critical row watches.
export function parseMemoryEvents(raw) {
const fields = keyedIntegers("memory.events", raw);
if (!fields.has("oom_kill")) fail("memory.events", "no oom_kill");
return { oom: fields.get("oom") ?? null, oomKill: fields.get("oom_kill") };
}
// /proc/vmstat. Swap activity is reported only (D4 has no threshold for it).
// oom_kill is the host-only OOM counter; it is null on a kernel without it.
export function parseVmstat(raw) {
const fields = keyedIntegers("vmstat", raw);
for (const key of ["pswpin", "pswpout"]) if (!fields.has(key)) fail("vmstat", `no ${key}`);
return { pswpin: fields.get("pswpin"), pswpout: fields.get("pswpout"), oomKill: fields.get("oom_kill") ?? null };
}
// Effective CPUs: the smallest of the host's online CPUs, the cpuset and the
// quota at every cgroup level given. With no cgroup input it is the host count.
// A fractional quota stays fractional.
export function effectiveCpus({ online, cpuset = null, cpuMax = [] }) {
let cpus = parseCpuList(online);
if (cpuset !== null) cpus = Math.min(cpus, parseCpuList(cpuset));
for (const level of cpuMax) {
const quota = parseCpuMax(level);
if (quota !== null) cpus = Math.min(cpus, quota);
}
return cpus;
}
// Effective memory. `levels` lists the cgroup levels that apply, each
// { max, current } as file text. The limit is the smallest of MemTotal and every
// numeric memory.max. Available memory is the smallest of MemAvailable and
// (max - current) at every limited level. memory.current counts page cache,
// so this errs toward holding. With no levels it is the host's figures.
export function effectiveMemory({ meminfo, levels = [] }) {
const host = parseMeminfo(meminfo);
let limit = host.memTotal;
let available = host.memAvailable;
for (const level of levels) {
const max = parseMemoryMax(level.max);
if (max === null) continue;
if (level.current === undefined || level.current === null) fail("memory.current", "missing for a limited level");
const current = parseMemoryCurrent(level.current);
limit = Math.min(limit, max);
available = Math.min(available, Math.max(0, max - current));
}
return { limit, available };
}
+201
View File
@@ -0,0 +1,201 @@
// The Q198 D4 hold and release rules as a pure evaluator. evaluate() takes the
// previous state, one timestamped sample and the rules, and returns the next
// state and a decision. It never reads the clock, a file or the environment:
// time is the sample's `at`, in milliseconds. Nothing here enforces anything;
// the decision is advice for S2 and S3.
import { SupervisorError } from "./errors.mjs";
const GiB = 1024 ** 3;
// The D4 table. Defaults are the design's proposals, not measured limits.
export const DEFAULT_RULES = deepFreeze({
sampleMs: 2000,
staleMs: 6000,
releaseMs: 30000,
load: { hold: 0.8, release: 0.55, samples: 3 },
memory: { holdFraction: 0.15, holdBytes: 1 * GiB, releaseFraction: 0.25, releaseBytes: 2 * GiB },
psi: {
samples: 3,
hold: { cpuSome: 20, memoryFull: 2, ioFull: 10 },
release: { cpuSome: 10, memoryFull: 0.5, ioFull: 3 },
},
critical: { memoryFraction: 0.05, diskBytes: 1 * GiB },
});
function deepFreeze(o) {
for (const v of Object.values(o)) if (v && typeof v === "object") deepFreeze(v);
return Object.freeze(o);
}
const refuse = (why) => { throw new SupervisorError(`rules: ${why}`, "rules"); };
// Overrides merge onto the defaults key by key. An unknown key, a value of the
// wrong type or a contradictory pair refuses.
function merge(base, over, at) {
if (over === undefined) return structuredClone(base);
if (!over || typeof over !== "object" || Array.isArray(over)) refuse(`${at || "rules"} must be an object`);
const out = {};
for (const key of Object.keys(over)) if (!(key in base)) refuse(`unknown key ${at ? `${at}.` : ""}${key}`);
for (const [key, value] of Object.entries(base)) {
const name = at ? `${at}.${key}` : key;
if (typeof value === "object") out[key] = merge(value, over[key], name);
else {
const v = over[key] ?? value;
if (typeof v !== "number" || !Number.isFinite(v)) refuse(`${name} must be a finite number`);
out[key] = v;
}
}
return out;
}
export function resolveRules(overrides = {}) {
const r = merge(DEFAULT_RULES, overrides, "");
const check = (ok, why) => { if (!ok) refuse(why); };
const fraction = (v) => v > 0 && v < 1;
const percent = (v) => v >= 0 && v <= 100;
check(r.sampleMs > 0, "sampleMs must be above 0");
check(r.staleMs > r.sampleMs, "staleMs must be above sampleMs");
check(r.releaseMs >= 0, "releaseMs must not be negative");
for (const k of ["load", "psi"]) check(Number.isInteger(r[k].samples) && r[k].samples >= 1, `${k}.samples must be an integer of at least 1`);
check(r.load.release > 0 && r.load.release < r.load.hold, "load.release must be above 0 and below load.hold");
check(fraction(r.memory.holdFraction) && fraction(r.memory.releaseFraction), "memory fractions must be between 0 and 1");
check(r.memory.releaseFraction > r.memory.holdFraction, "memory.releaseFraction must be above memory.holdFraction");
check(r.memory.holdBytes >= 0 && r.memory.releaseBytes > r.memory.holdBytes, "memory.releaseBytes must be above memory.holdBytes");
for (const k of ["cpuSome", "memoryFull", "ioFull"]) {
check(percent(r.psi.hold[k]) && percent(r.psi.release[k]), `psi ${k} values must be percentages`);
check(r.psi.release[k] > 0 && r.psi.release[k] < r.psi.hold[k], `psi.release.${k} must be above 0 and below psi.hold.${k}`);
}
check(fraction(r.critical.memoryFraction) && r.critical.memoryFraction < r.memory.holdFraction,
"critical.memoryFraction must be between 0 and memory.holdFraction");
check(r.critical.diskBytes > 0, "critical.diskBytes must be above 0");
return deepFreeze(r);
}
// The state before any sample: held, because no signal has been seen yet.
export function initialState() {
return { mode: "held", lastAt: null, loadCount: 0, psiCount: 0, psiSeen: false, oomBaseline: null, releaseSince: null, critical: [] };
}
const SIGNAL_FIELDS = {
load: ["load1", "cpus"],
memory: ["available", "limit"],
psi: ["cpuSome", "memoryFull", "ioFull"],
oom: ["kills"],
disk: ["free"],
};
// A signal is usable when every field is a non-negative finite number, cpus
// and limit are above 0, and its own timestamp is no later than the sample and
// no more than staleMs before it.
function fresh(name, signal, now, rules) {
if (!signal || typeof signal !== "object") return false;
if (typeof signal.at !== "number" || !Number.isFinite(signal.at)) return false;
if (signal.at > now || now - signal.at > rules.staleMs) return false;
for (const f of SIGNAL_FIELDS[name]) if (typeof signal[f] !== "number" || !Number.isFinite(signal[f]) || signal[f] < 0) return false;
if (name === "load" && signal.cpus <= 0) return false;
if (name === "memory" && signal.limit <= 0) return false;
return true;
}
// Sample: { at, load: { at, load1, cpus }, memory: { at, available, limit },
// psi: { at, cpuSome, memoryFull, ioFull } (avg10 percentages),
// oom: { at, kills }, disk: { at, free } }. `reconciled` is the caller's word
// that reconciliation finished; only it lets a critical hold end.
export function evaluate(state, sample, rules = DEFAULT_RULES, { reconciled = false } = {}) {
if (!sample || typeof sample.at !== "number" || !Number.isFinite(sample.at)) {
throw new SupervisorError("sample: at must be a finite number of milliseconds", "sample");
}
const now = sample.at;
const next = { ...state, critical: [...state.critical], lastAt: now };
const unknown = [];
const notes = [];
const gap = state.lastAt === null ? Infinity : now - state.lastAt;
// A count of consecutive samples survives only a gap of at most staleMs.
const contiguous = gap >= 0 && gap <= rules.staleMs;
if (gap < 0) { unknown.push("clock"); notes.push("clock-backwards"); }
let loadHold = false, loadReleases = false;
if (fresh("load", sample.load, now, rules)) {
const ratio = sample.load.load1 / sample.load.cpus;
next.loadCount = ratio >= rules.load.hold ? (contiguous ? state.loadCount : 0) + 1 : 0;
loadHold = next.loadCount >= rules.load.samples;
loadReleases = ratio <= rules.load.release;
} else { unknown.push("load"); next.loadCount = 0; }
let memoryLow = false, memoryReleases = false, criticalMemory = false;
if (fresh("memory", sample.memory, now, rules)) {
const { available, limit } = sample.memory;
const m = rules.memory;
// "Whichever demands more free memory": the larger floor applies.
memoryLow = available < Math.max(m.holdFraction * limit, m.holdBytes);
memoryReleases = available > Math.max(m.releaseFraction * limit, m.releaseBytes);
criticalMemory = available < rules.critical.memoryFraction * limit;
} else unknown.push("memory");
// PSI is required once it has been seen. A host that never had it skips the
// PSI rows; one that loses it holds with health-unknown.
let psiHold = false, psiReleases = true;
const psiOver = [];
if (sample.psi !== undefined && sample.psi !== null) next.psiSeen = true;
if (fresh("psi", sample.psi, now, rules)) {
const p = sample.psi, h = rules.psi.hold, r = rules.psi.release;
if (p.cpuSome >= h.cpuSome) psiOver.push("psi-cpu");
if (p.memoryFull >= h.memoryFull) psiOver.push("psi-memory");
if (p.ioFull >= h.ioFull) psiOver.push("psi-io");
next.psiCount = psiOver.length ? (contiguous ? state.psiCount : 0) + 1 : 0;
psiHold = next.psiCount >= rules.psi.samples;
psiReleases = p.cpuSome < r.cpuSome && p.memoryFull < r.memoryFull && p.ioFull < r.ioFull;
} else {
next.psiCount = 0;
if (next.psiSeen) { unknown.push("psi"); psiReleases = false; }
}
let criticalOom = false;
if (fresh("oom", sample.oom, now, rules)) {
const kills = sample.oom.kills;
if (state.oomBaseline !== null && kills > state.oomBaseline) criticalOom = true;
if (state.oomBaseline !== null && kills < state.oomBaseline) notes.push("oom-counter-reset");
next.oomBaseline = kills;
} else unknown.push("oom");
let criticalDisk = false;
if (fresh("disk", sample.disk, now, rules)) criticalDisk = sample.disk.free < rules.critical.diskBytes;
else unknown.push("disk");
const criticalNow = [];
if (criticalOom) criticalNow.push("critical-oom");
if (criticalMemory) criticalNow.push("critical-memory");
if (criticalDisk) criticalNow.push("critical-disk");
const decide = (mode, reasons, releaseAt = null) => {
next.mode = mode;
return { state: next, decision: { mode, reasons, unknown, critical: [...next.critical], releaseAt, notes } };
};
if (criticalNow.length) {
for (const c of criticalNow) if (!next.critical.includes(c)) next.critical.push(c);
next.releaseSince = null;
return decide("critical", [...next.critical]);
}
// Critical ends only on the caller's reconciliation, never on thresholds alone.
if (state.mode === "critical" && !reconciled) {
next.releaseSince = null;
return decide("critical", [...next.critical]);
}
const mode = state.mode === "critical" ? "held" : state.mode;
next.critical = [];
const reasons = [];
if (unknown.length) reasons.push("health-unknown");
if (loadHold) reasons.push("load-high");
if (memoryLow) reasons.push("memory-low");
if (psiHold) reasons.push(...psiOver);
if (reasons.length) { next.releaseSince = null; return decide("held", reasons); }
if (mode === "open") { next.releaseSince = null; return decide("open", []); }
if (!(loadReleases && memoryReleases && psiReleases)) { next.releaseSince = null; return decide("held", ["hysteresis"]); }
next.releaseSince = state.releaseSince !== null && contiguous ? state.releaseSince : now;
if (now - next.releaseSince >= rules.releaseMs) { next.releaseSince = null; return decide("open", []); }
return decide("held", ["release-pending"], next.releaseSince + rules.releaseMs);
}
+133
View File
@@ -0,0 +1,133 @@
// Parser and effective-limit tests. Every input is fixture text shaped like
// the kernel's files; nothing reads /proc, /sys or a cgroup.
import { test } from "node:test";
import assert from "node:assert/strict";
import {
SupervisorError, parseLoadavg, parseMeminfo, parsePsi, parseCpuList, parseCpuMax, parseMemoryMax,
parseMemoryCurrent, parseMemoryEvents, parseVmstat, effectiveCpus, effectiveMemory,
} from "../src/index.mjs";
const GiB = 1024 ** 3;
const isParse = (re) => (e) => e instanceof SupervisorError && e.kind === "parse" && re.test(e.message);
const MEMINFO = [
"MemTotal: 32768000 kB",
"MemFree: 1024000 kB",
"MemAvailable: 8192000 kB",
"SwapTotal: 4096000 kB",
"SwapFree: 4000000 kB",
"HugePages_Total: 0",
"Hugepagesize: 2048 kB",
"",
].join("\n");
test("loadavg parses the three loads", () => {
assert.deepEqual(parseLoadavg("0.52 1.58 12.00 3/1234 56789\n"), { load1: 0.52, load5: 1.58, load15: 12 });
});
test("a malformed loadavg throws", () => {
for (const bad of ["", "0.52 0.58 0.59\n", "0.52 0.58 0.59 1/2\n", "x 0.58 0.59 1/2 3\n", "0.52 0.58 0.59 1/2 3 4\n", 42]) {
assert.throws(() => parseLoadavg(bad), isParse(/loadavg/), JSON.stringify(bad));
}
});
test("meminfo returns bytes and keeps unitless counts", () => {
assert.deepEqual(parseMeminfo(MEMINFO), {
memTotal: 32768000 * 1024, memAvailable: 8192000 * 1024, swapTotal: 4096000 * 1024, swapFree: 4000000 * 1024,
});
});
test("meminfo without MemAvailable, with a repeated key or a bad line throws", () => {
assert.throws(() => parseMeminfo("MemTotal: 1 kB\n"), isParse(/no MemAvailable/));
assert.throws(() => parseMeminfo(MEMINFO + "MemTotal: 1 kB\n"), isParse(/repeated key MemTotal/));
assert.throws(() => parseMeminfo(MEMINFO + "Broken line\n"), isParse(/bad line/));
assert.throws(() => parseMeminfo(MEMINFO.replace("8192000", "-1")), isParse(/bad line/));
});
test("PSI parses some and full, and cpu may lack full", () => {
const mem = "some avg10=1.50 avg60=0.20 avg300=0.00 total=123\nfull avg10=0.75 avg60=0.10 avg300=0.00 total=45\n";
assert.deepEqual(parsePsi(mem, "memory"), {
some: { avg10: 1.5, avg60: 0.2, avg300: 0, total: 123 },
full: { avg10: 0.75, avg60: 0.1, avg300: 0, total: 45 },
});
assert.equal(parsePsi("some avg10=20.00 avg60=0.00 avg300=0.00 total=9\n", "cpu").full, null);
});
test("PSI without a full line for memory or io, or with a bad line, throws", () => {
const someOnly = "some avg10=0.00 avg60=0.00 avg300=0.00 total=0\n";
assert.throws(() => parsePsi(someOnly, "memory"), isParse(/no full line/));
assert.throws(() => parsePsi(someOnly, "io"), isParse(/no full line/));
assert.throws(() => parsePsi("full avg10=0.00 avg60=0.00 avg300=0.00 total=0\n", "cpu"), isParse(/no some line/));
assert.throws(() => parsePsi(someOnly + someOnly, "cpu"), isParse(/repeated some/));
assert.throws(() => parsePsi("some avg10=x avg60=0.00 avg300=0.00 total=0\n", "cpu"), isParse(/not a non-negative number/));
assert.throws(() => parsePsi(someOnly, "gpu"), isParse(/unknown resource/));
});
test("CPU lists count distinct CPUs", () => {
assert.equal(parseCpuList("0-3,6,8-9\n"), 7);
assert.equal(parseCpuList("0\n"), 1);
assert.equal(parseCpuList("0-3,2-5"), 6);
for (const bad of ["", "\n", "3-1", "0,,1", "a-b", "0-"]) assert.throws(() => parseCpuList(bad), isParse(/cpu list/), bad);
});
test("cpu.max gives the quota in CPUs, or null for max", () => {
assert.equal(parseCpuMax("max 100000\n"), null);
assert.equal(parseCpuMax("50000 100000\n"), 0.5);
assert.equal(parseCpuMax("200000 100000\n"), 2);
for (const bad of ["max", "50000 0", "-1 100000", "max max"]) assert.throws(() => parseCpuMax(bad), isParse(/cpu.max/), bad);
});
test("memory.max, memory.current and memory.events", () => {
assert.equal(parseMemoryMax("max\n"), null);
assert.equal(parseMemoryMax("4294967296\n"), 4 * GiB);
assert.throws(() => parseMemoryMax("4G\n"), isParse(/memory.max/));
assert.equal(parseMemoryCurrent("123456\n"), 123456);
assert.throws(() => parseMemoryCurrent(""), isParse(/memory.current/));
assert.deepEqual(parseMemoryEvents("low 0\nhigh 4\nmax 2\noom 1\noom_kill 1\noom_group_kill 0\n"), { oom: 1, oomKill: 1 });
assert.throws(() => parseMemoryEvents("low 0\noom 1\n"), isParse(/no oom_kill/));
assert.throws(() => parseMemoryEvents("oom_kill 1\noom_kill 2\n"), isParse(/repeated key/));
});
test("vmstat gives swap counters and the host OOM counter", () => {
assert.deepEqual(parseVmstat("nr_free_pages 100\npswpin 7\npswpout 9\noom_kill 3\n"), { pswpin: 7, pswpout: 9, oomKill: 3 });
assert.deepEqual(parseVmstat("pswpin 0\npswpout 0\n"), { pswpin: 0, pswpout: 0, oomKill: null });
assert.throws(() => parseVmstat("pswpin 0\n"), isParse(/no pswpout/));
});
test("effective CPUs fall back to the host count with no cgroup input", () => {
assert.equal(effectiveCpus({ online: "0-15\n" }), 16);
});
test("effective CPUs take the smallest of host, cpuset and every quota level", () => {
assert.equal(effectiveCpus({ online: "0-15\n", cpuset: "0-3\n" }), 4);
assert.equal(effectiveCpus({ online: "0-15\n", cpuset: "0-3\n", cpuMax: ["max 100000\n", "150000 100000\n"] }), 1.5);
// The outer level is the tighter one.
assert.equal(effectiveCpus({ online: "0-15\n", cpuMax: ["800000 100000\n", "200000 100000\n"] }), 2);
// A quota above the host count doesn't raise it.
assert.equal(effectiveCpus({ online: "0-1\n", cpuMax: ["800000 100000\n"] }), 2);
assert.equal(effectiveCpus({ online: "0-15\n", cpuMax: ["max 100000\n"] }), 16);
});
test("effective memory falls back to the host with no cgroup levels, or only max", () => {
const host = { limit: 32768000 * 1024, available: 8192000 * 1024 };
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO }), host);
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: "max\n" }] }), host);
});
test("effective memory uses the tightest limited level", () => {
// 4 GiB limit with 3 GiB used: 1 GiB available, below the host's MemAvailable.
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${4 * GiB}\n`, current: `${3 * GiB}\n` }] }),
{ limit: 4 * GiB, available: 1 * GiB });
// Nested: the inner level is loose, the outer one is nearly full.
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [
{ max: "max\n" }, { max: `${16 * GiB}\n`, current: `${2 * GiB}\n` }, { max: `${6 * GiB}\n`, current: `${5.5 * GiB}\n` },
] }), { limit: 6 * GiB, available: 0.5 * GiB });
// Usage above the limit floors at 0.
assert.equal(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: "100\n", current: "200\n" }] }).available, 0);
// A limit above MemTotal doesn't raise the limit.
assert.equal(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${64 * GiB}\n`, current: "0\n" }] }).limit, 32768000 * 1024);
});
test("a limited level without memory.current throws", () => {
assert.throws(() => effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${4 * GiB}\n` }] }), isParse(/missing for a limited level/));
});
+28
View File
@@ -0,0 +1,28 @@
// S1 has no live enforcement: the source must not reach the filesystem, a
// process, the network or the clock. This reads the package's own source text.
import { test } from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
const SRC = path.join(import.meta.dirname, "..", "src");
const FORBIDDEN = [
/\bfrom\s+["'](node:)?(fs|fs\/promises|child_process|net|http|https|dgram|tls|worker_threads|os)["']/,
/\bimport\s*\(/,
/\brequire\s*\(/,
/\bprocess\./,
/\bDate\.now\b/,
/\bnew Date\b/,
/\bperformance\.now\b/,
/\bset(Timeout|Interval|Immediate)\b/,
];
test("src reads no file, process, network or clock", () => {
const files = fs.readdirSync(SRC).filter((f) => f.endsWith(".mjs"));
assert.deepEqual(files.sort(), ["errors.mjs", "index.mjs", "pressure.mjs", "rules.mjs"]);
for (const f of files) {
const body = fs.readFileSync(path.join(SRC, f), "utf8");
for (const re of FORBIDDEN) assert.doesNotMatch(body, re, `${f} matches ${re}`);
for (const m of body.matchAll(/\bfrom\s+["']([^"']+)["']/g)) assert.match(m[1], /^\.\/[a-z]+\.mjs$/, `${f} imports ${m[1]}`);
}
});
+332
View File
@@ -0,0 +1,332 @@
// Evaluator tests on a fake clock: each sample's `at` is the time. Synthetic
// numbers only; nothing samples the host or generates load.
import { test } from "node:test";
import assert from "node:assert/strict";
import { SupervisorError, DEFAULT_RULES, resolveRules, initialState, evaluate } from "../src/index.mjs";
const GiB = 1024 ** 3;
const LIMIT = 32 * GiB;
const R = DEFAULT_RULES;
// A healthy sample at `at`. `patch` changes fields per signal; null drops one.
function sample(at, patch = {}) {
const base = {
load: { load1: 0.1, cpus: 1 },
memory: { available: 16 * GiB, limit: LIMIT },
psi: { cpuSome: 0, memoryFull: 0, ioFull: 0 },
oom: { kills: 0 },
disk: { free: 100 * GiB },
};
const out = { at };
for (const [name, fields] of Object.entries(base)) {
if (patch[name] === null) continue;
out[name] = { at, ...fields, ...patch[name] };
}
return out;
}
// Fold samples through evaluate; returns every decision and the last state.
function run(state, samples, rules = R, opts) {
const decisions = [];
for (const s of samples) {
const r = evaluate(state, s, rules, typeof opts === "function" ? opts(s) : opts);
state = r.state;
decisions.push(r.decision);
}
return { state, decisions, last: decisions.at(-1) };
}
// Samples every 2 s from `from` to `to` inclusive.
function every2s(from, to, patch) {
const out = [];
for (let at = from; at <= to; at += 2000) out.push(sample(at, typeof patch === "function" ? patch(at) : patch));
return out;
}
// An open state whose last sample was at 30000.
function opened() {
const { state, last } = run(initialState(), every2s(0, 30000));
assert.equal(last.mode, "open");
return state;
}
// A state held by a load hold, last sample at 36000.
function heldByLoad() {
const { state, last } = run(opened(), every2s(32000, 36000, { load: { load1: 0.9 } }));
assert.deepEqual(last.reasons, ["load-high"]);
return state;
}
const modes = (r) => r.decisions.map((d) => d.mode);
test("the first state is held, and healthy samples open it after exactly 30 s", () => {
assert.equal(initialState().mode, "held");
const first = evaluate(initialState(), sample(0));
assert.equal(first.decision.mode, "held");
assert.deepEqual(first.decision.reasons, ["release-pending"]);
assert.equal(first.decision.releaseAt, 30000);
const r = run(initialState(), [sample(0), sample(2000), sample(6000), sample(10000), sample(14000), sample(18000),
sample(22000), sample(26000), sample(29999), sample(30000)]);
assert.deepEqual(r.decisions.at(-2).reasons, ["release-pending"]);
assert.equal(r.decisions.at(-2).mode, "held");
assert.equal(r.last.mode, "open");
});
test("load holds at 0.80 for 3 samples and not at 0.7999", () => {
const at = run(opened(), every2s(32000, 36000, { load: { load1: 0.8 } }));
assert.deepEqual(modes(at), ["open", "open", "held"]);
assert.deepEqual(at.last.reasons, ["load-high"]);
const below = run(opened(), every2s(32000, 50000, { load: { load1: 0.7999 } }));
assert.ok(below.decisions.every((d) => d.mode === "open"));
});
test("load divides by effective CPUs, fractional ones included", () => {
const r = run(opened(), every2s(32000, 36000, { load: { load1: 0.4, cpus: 0.5 } }));
assert.equal(r.last.mode, "held");
const r2 = run(opened(), every2s(32000, 36000, { load: { load1: 3.1, cpus: 4 } }));
assert.equal(r2.last.mode, "open");
});
test("a gap of more than 6 s resets the consecutive count", () => {
const high = { load: { load1: 0.9 } };
const r = run(opened(), [sample(32000, high), sample(34000, high), sample(40001, high), sample(42001, high), sample(44001, high)]);
assert.deepEqual(modes(r), ["open", "open", "open", "open", "held"]);
const ok = run(opened(), [sample(32000, high), sample(34000, high), sample(40000, high)]);
assert.equal(ok.last.mode, "held");
});
test("load releases at 0.55 and not at 0.5501", () => {
const dead = run(heldByLoad(), every2s(38000, 80000, { load: { load1: 0.5501 } }));
assert.ok(dead.decisions.every((d) => d.mode === "held" && d.reasons[0] === "hysteresis"));
const r = run(heldByLoad(), every2s(38000, 68000, { load: { load1: 0.55 } }));
assert.equal(r.decisions.at(-2).mode, "held");
assert.equal(r.last.mode, "open");
});
test("a dead-band sample restarts the 30 s release window", () => {
const r = run(heldByLoad(), [
...every2s(38000, 60000),
sample(62000, { load: { load1: 0.6 } }),
...every2s(64000, 92000),
sample(94000),
]);
const at = (t) => r.decisions[r.decisions.length - 1 - (94000 - t) / 2000];
assert.equal(at(62000).reasons[0], "hysteresis");
assert.equal(at(92000).mode, "held");
assert.equal(at(94000).mode, "open");
});
test("memory holds below the larger floor, on one sample", () => {
const floor = Math.max(R.memory.holdFraction * LIMIT, R.memory.holdBytes);
assert.ok(floor > R.memory.holdBytes); // 15% of 32 GiB is the larger floor here
const just = Math.ceil(floor);
assert.equal(evaluate(opened(), sample(32000, { memory: { available: just } })).decision.mode, "open");
const low = evaluate(opened(), sample(32000, { memory: { available: just - 1 } })).decision;
assert.equal(low.mode, "held");
assert.deepEqual(low.reasons, ["memory-low"]);
});
test("on a small limit the 1 GiB floor is the larger one", () => {
const small = 4 * GiB; // 15% is 0.6 GiB, so 1 GiB applies
const s = (available) => sample(32000, { memory: { available, limit: small } });
assert.equal(evaluate(opened(), s(1 * GiB)).decision.mode, "open");
assert.deepEqual(evaluate(opened(), s(1 * GiB - 1)).decision.reasons, ["memory-low"]);
});
test("memory releases only above the larger release floor", () => {
// 3 GiB is below the 15% floor (4.8 GiB) and above the 5% critical line (1.6 GiB).
const held = evaluate(opened(), sample(32000, { memory: { available: 3 * GiB } })).state;
assert.equal(held.mode, "held");
const rel = Math.max(R.memory.releaseFraction * LIMIT, R.memory.releaseBytes);
const stay = run(held, every2s(34000, 80000, { memory: { available: Math.floor(rel) } }));
assert.ok(stay.decisions.every((d) => d.mode === "held"));
const go = run(held, every2s(34000, 64000, { memory: { available: Math.floor(rel) + 1 } }));
assert.equal(go.last.mode, "open");
const small = { limit: 4 * GiB }; // 25% is 1 GiB, so 2 GiB applies
const stay2 = run(held, every2s(34000, 80000, { memory: { ...small, available: 2 * GiB } }));
assert.ok(stay2.decisions.every((d) => d.mode === "held"));
const go2 = run(held, every2s(34000, 64000, { memory: { ...small, available: 2 * GiB + 1 } }));
assert.equal(go2.last.mode, "open");
});
test("each PSI hold threshold holds at its value for 3 samples and not just below", () => {
for (const [field, code] of [["cpuSome", "psi-cpu"], ["memoryFull", "psi-memory"], ["ioFull", "psi-io"]]) {
const at = run(opened(), every2s(32000, 36000, { psi: { [field]: R.psi.hold[field] } }));
assert.deepEqual(modes(at), ["open", "open", "held"], field);
assert.deepEqual(at.last.reasons, [code], field);
const below = run(opened(), every2s(32000, 50000, { psi: { [field]: R.psi.hold[field] - 0.01 } }));
assert.ok(below.decisions.every((d) => d.mode === "open"), field);
}
});
test("the PSI count runs across resources", () => {
const r = run(opened(), [sample(32000, { psi: { cpuSome: 25 } }), sample(34000, { psi: { memoryFull: 3 } }), sample(36000, { psi: { ioFull: 12 } })]);
assert.deepEqual(modes(r), ["open", "open", "held"]);
assert.deepEqual(r.last.reasons, ["psi-io"]);
});
test("each PSI release threshold blocks at its value and releases just below", () => {
const held = run(opened(), every2s(32000, 36000, { psi: { cpuSome: 30 } })).state;
for (const field of ["cpuSome", "memoryFull", "ioFull"]) {
const stay = run(held, every2s(38000, 80000, { psi: { [field]: R.psi.release[field] } }));
assert.ok(stay.decisions.every((d) => d.mode === "held" && d.reasons[0] === "hysteresis"), field);
const go = run(held, every2s(38000, 68000, { psi: { [field]: R.psi.release[field] - 0.01 } }));
assert.equal(go.last.mode, "open", field);
}
});
test("a missing required signal holds at once with health-unknown", () => {
for (const name of ["load", "memory", "oom", "disk"]) {
const d = evaluate(opened(), sample(32000, { [name]: null })).decision;
assert.equal(d.mode, "held", name);
assert.deepEqual(d.reasons, ["health-unknown"], name);
assert.deepEqual(d.unknown, [name], name);
}
});
test("a signal is stale at more than 6 s old and fresh at exactly 6 s", () => {
const stale = { at: 32000, load1: 0.1, cpus: 1 };
const s = (signalAt) => ({ ...sample(32000), load: { ...stale, at: signalAt } });
assert.equal(evaluate(opened(), s(26000)).decision.mode, "open");
assert.deepEqual(evaluate(opened(), s(25999)).decision.unknown, ["load"]);
assert.deepEqual(evaluate(opened(), s(32001)).decision.unknown, ["load"]); // later than the sample
});
test("malformed signal values count as missing", () => {
for (const patch of [{ load: { load1: NaN } }, { load: { cpus: 0 } }, { load: { load1: -1 } }, { memory: { limit: 0 } },
{ memory: { available: "1" } }, { oom: { kills: Infinity } }, { disk: { free: undefined } }]) {
const d = evaluate(opened(), sample(32000, patch)).decision;
assert.deepEqual(d.reasons, ["health-unknown"], JSON.stringify(patch));
}
});
test("after health-unknown, recovery waits the full 30 s", () => {
const unknown = evaluate(opened(), sample(32000, { load: null })).state;
const r = run(unknown, every2s(34000, 64000));
assert.equal(r.decisions.at(-2).mode, "held");
assert.equal(r.last.mode, "open");
});
test("a host that never had PSI skips the PSI rows", () => {
const r = run(initialState(), every2s(0, 30000, { psi: null }));
assert.equal(r.last.mode, "open");
assert.deepEqual(r.last.unknown, []);
});
test("PSI that disappears after it was present holds with health-unknown", () => {
const d = evaluate(opened(), sample(32000, { psi: null })).decision;
assert.equal(d.mode, "held");
assert.deepEqual(d.reasons, ["health-unknown"]);
assert.deepEqual(d.unknown, ["psi"]);
// Also when it first appears late, then goes.
const late = run(initialState(), [...every2s(0, 30000, { psi: null }), sample(32000), sample(34000, { psi: null })]);
assert.deepEqual(late.last.unknown, ["psi"]);
// A stale PSI signal counts the same.
const stalePsi = { ...sample(32000), psi: { at: 25000, cpuSome: 0, memoryFull: 0, ioFull: 0 } };
assert.deepEqual(evaluate(opened(), stalePsi).decision.unknown, ["psi"]);
});
test("a rise in the OOM kill counter is critical; the first sample is the baseline", () => {
const first = evaluate(initialState(), sample(0, { oom: { kills: 5 } }));
assert.notEqual(first.decision.mode, "critical");
const r = run(opened(), [sample(32000, { oom: { kills: 0 } }), sample(34000, { oom: { kills: 1 } })]);
assert.equal(r.last.mode, "critical");
assert.deepEqual(r.last.reasons, ["critical-oom"]);
});
test("a falling OOM counter resets the baseline with a note, not an event", () => {
const s = run(initialState(), every2s(0, 30000, { oom: { kills: 6 } })).state;
assert.equal(s.mode, "open");
const r = run(s, [sample(32000, { oom: { kills: 2 } }), sample(34000, { oom: { kills: 2 } })]);
assert.deepEqual(r.decisions[0].notes, ["oom-counter-reset"]);
assert.ok(r.decisions.every((d) => d.mode === "open"));
});
test("memory under 5% and disk under 1 GiB are critical, at their edges", () => {
const five = R.critical.memoryFraction * LIMIT;
assert.equal(evaluate(opened(), sample(32000, { memory: { available: Math.ceil(five) } })).decision.mode, "held");
const mem = evaluate(opened(), sample(32000, { memory: { available: Math.ceil(five) - 1 } })).decision;
assert.equal(mem.mode, "critical");
assert.deepEqual(mem.reasons, ["critical-memory"]);
assert.equal(evaluate(opened(), sample(32000, { disk: { free: 1 * GiB } })).decision.mode, "open");
assert.deepEqual(evaluate(opened(), sample(32000, { disk: { free: 1 * GiB - 1 } })).decision.reasons, ["critical-disk"]);
});
test("critical stays until reconciliation, then still waits the 30 s release", () => {
const crit = evaluate(opened(), sample(32000, { disk: { free: 0 } })).state;
const waiting = run(crit, every2s(34000, 120000));
assert.ok(waiting.decisions.every((d) => d.mode === "critical" && d.reasons[0] === "critical-disk"));
// Reconciled while the condition is still there: still critical.
assert.equal(evaluate(crit, sample(34000, { disk: { free: 0 } }), R, { reconciled: true }).decision.mode, "critical");
// Reconciled and healthy: held, then open 30 s later.
const after = run(waiting.state, every2s(122000, 152000), R, (s) => ({ reconciled: s.at === 122000 }));
assert.deepEqual(after.decisions[0].reasons, ["release-pending"]);
assert.deepEqual(after.decisions[0].critical, []);
assert.equal(after.decisions.at(-2).mode, "held");
assert.equal(after.last.mode, "open");
});
test("critical reasons accumulate while critical", () => {
const r = run(opened(), [sample(32000, { disk: { free: 0 } }), sample(34000, { oom: { kills: 3 } })]);
assert.deepEqual(r.last.reasons, ["critical-disk", "critical-oom"]);
});
test("a clock that goes backwards holds with health-unknown", () => {
const d = evaluate(opened(), sample(20000)).decision;
assert.equal(d.mode, "held");
assert.deepEqual(d.unknown, ["clock"]);
assert.deepEqual(d.notes, ["clock-backwards"]);
});
test("a sample without a numeric time throws", () => {
for (const bad of [null, {}, { at: "1" }, { at: NaN }]) {
assert.throws(() => evaluate(initialState(), bad), (e) => e instanceof SupervisorError && e.kind === "sample");
}
});
test("evaluate doesn't change the state it was given", () => {
const s = Object.freeze({ ...opened(), critical: Object.freeze([]) });
assert.doesNotThrow(() => evaluate(s, sample(32000, { disk: { free: 0 } })));
assert.equal(s.mode, "open");
});
test("resolveRules with no overrides is the D4 table, frozen", () => {
assert.deepEqual(resolveRules(), DEFAULT_RULES);
assert.ok(Object.isFrozen(DEFAULT_RULES) && Object.isFrozen(DEFAULT_RULES.psi.hold));
assert.deepEqual(DEFAULT_RULES.load, { hold: 0.8, release: 0.55, samples: 3 });
assert.equal(DEFAULT_RULES.releaseMs, 30000);
assert.equal(DEFAULT_RULES.staleMs, 6000);
});
test("overrides apply in the evaluator", () => {
const rules = resolveRules({ releaseMs: 10000, load: { samples: 1 } });
const r = run(initialState(), every2s(0, 10000), rules);
assert.equal(r.last.mode, "open");
assert.equal(evaluate(r.state, sample(12000, { load: { load1: 0.9 } }), rules).decision.mode, "held");
});
test("contradictory or unknown settings refuse", () => {
const cases = [
[{ load: { release: 0.8 } }, /load.release must be above 0 and below load.hold/],
[{ load: { hold: 0.5 } }, /load.release must be above 0 and below load.hold/],
[{ memory: { releaseFraction: 0.15 } }, /releaseFraction must be above/],
[{ memory: { releaseBytes: 1 * GiB } }, /releaseBytes must be above/],
[{ memory: { holdFraction: 1.2 } }, /between 0 and 1/],
[{ psi: { release: { ioFull: 10 } } }, /psi.release.ioFull must be above 0 and below psi.hold.ioFull/],
[{ psi: { hold: { cpuSome: 120 } } }, /percentages/],
[{ psi: { samples: 0 } }, /psi.samples must be an integer/],
[{ load: { samples: 1.5 } }, /load.samples must be an integer/],
[{ staleMs: 2000 }, /staleMs must be above sampleMs/],
[{ releaseMs: -1 }, /releaseMs must not be negative/],
[{ critical: { memoryFraction: 0.15 } }, /critical.memoryFraction/],
[{ critical: { diskBytes: 0 } }, /critical.diskBytes/],
[{ loadd: {} }, /unknown key loadd/],
[{ psi: { hold: { gpu: 1 } } }, /unknown key psi.hold.gpu/],
[{ load: { hold: "0.8" } }, /load.hold must be a finite number/],
[{ load: 1 }, /load must be an object/],
[[], /rules must be an object/],
];
for (const [over, re] of cases) {
assert.throws(() => resolveRules(over), (e) => e instanceof SupervisorError && e.kind === "rules" && re.test(e.message), JSON.stringify(over));
}
});