morgana: row 257 build (Q198 S1, #1762): @mosaic/supervisor pressure parsers and pure D4 rules evaluator, tests and mutant plan; tests wait for Sage's slot
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# Row 257: Q198 S1 pressure parser and pure rules evaluator
|
||||
|
||||
Issue #1762. Brief: `docs/plans/2026-10-21_post-v1-row-briefs.md`, "Brief: Q198 S1 pressure parser and
|
||||
pure rules evaluator" and "Gate: Q198 S1" (cca206cdc). Design: `docs/plans/2026-10-21_q198-system-manager.md`
|
||||
r6.1, section 8 row S1 and section 4 (D4). Owner Morgana, reviewer Filbert, gate owner Sage (sage-2196).
|
||||
Post-v1: MVP rows and re-pin windows go ahead of it on the slot. Base: refactor cca206cdc.
|
||||
|
||||
## Recorded choices (sage-2196 accepted each)
|
||||
|
||||
1. **Placement.** A new package, `packages/supervisor` (`@mosaic/supervisor`), shaped like
|
||||
packages/runs. Nothing on refactor parsed loadavg, PSI, meminfo or cgroup limits (`git grep`; the
|
||||
only hit was an awk line in scripts/case2-precheck.sh). S2 adds the sampler process to the same
|
||||
package. Packages aren't registered centrally, so no other file changes.
|
||||
2. **Provider scope** (429, quota, Retry-After, the half-open probe) is out. The brief puts it in S3
|
||||
and S5.
|
||||
3. **PSI** is not required on a host that has never shown it, so a kernel built without PSI skips the
|
||||
PSI rows instead of holding forever. Once a sample has carried PSI, it is required: a later missing
|
||||
or stale PSI signal holds with `health-unknown`. That is the case sage-2196 asked for.
|
||||
- Tradeoff: I latch on the first sample that carries PSI, not only on the very first sample. So PSI
|
||||
that appears late and then disappears also holds. This is stricter than "present at the first
|
||||
sample", and it never skips a signal the host has shown.
|
||||
4. **Memory under a cgroup.** Available memory = min(MemAvailable, limit - memory.current) at each
|
||||
limited level. memory.current counts page cache, so this errs toward holding.
|
||||
- Follow-up, not in this row: subtract reclaimable cache using memory.stat `inactive_file`, if S2's
|
||||
live samples show holds that the host's MemAvailable doesn't support.
|
||||
|
||||
Other readings of the D4 table, made in the build:
|
||||
- **Memory floors.** "Whichever demands more free memory" means the larger floor applies. Hold below
|
||||
max(15% x limit, 1 GiB); release above max(25% x limit, 2 GiB).
|
||||
- **Edges as written.** Hold at ≥ for load and PSI, < for memory. Release at ≤ 0.55 for load, < for
|
||||
PSI, > for memory. Critical below 5% of the limit and below 1 GiB of disk.
|
||||
- **Sample counts.** "For 3 samples" means 3 consecutive samples. The count resets after a gap of more
|
||||
than 6 s (the stale limit) between samples. The PSI row counts as one row: any of its three
|
||||
conditions on 3 consecutive samples holds, and the reasons name the resources over on the latest
|
||||
sample. Counting per resource would be weaker.
|
||||
- **Memory holds on one sample.** The table's memory row has no sample count.
|
||||
- **Release** needs every applicable release condition on every sample for 30 s, by the samples' own
|
||||
times (`>= 30000` ms after the first qualifying sample). A dead-band sample or any hold restarts the
|
||||
window, and so does a gap of more than 6 s.
|
||||
- **Start held.** `initialState()` is held, because nothing has been seen yet. The first healthy
|
||||
sample starts the 30 s window.
|
||||
- **Missing or stale.** Required signals are load, memory, oom and disk, plus PSI once seen. A signal
|
||||
holds at once with `health-unknown` if it is:
|
||||
- missing;
|
||||
- non-numeric, negative or zero where a divisor;
|
||||
- more than 6 s older than its sample (6 s exactly is fresh);
|
||||
- stamped later than its sample.
|
||||
Disk and OOM are required because the critical row can't be evaluated without them.
|
||||
- **Host-only OOM.** `/proc/vmstat` `oom_kill` gives the OOM counter when there's no cgroup
|
||||
memory.events. The parser returns null for a kernel without it, and the sampler then has no OOM
|
||||
signal (health-unknown).
|
||||
- **Critical** latches. It ends only when the caller passes `reconciled` with no critical condition in
|
||||
the sample, then goes through held and the normal 30 s release. The evaluator never sets
|
||||
`reconciled` and has no kill-switch path. Critical reasons accumulate while latched.
|
||||
- **OOM counter.** The first OOM sample is the baseline. A falling counter (a recreated cgroup) resets
|
||||
the baseline with a note, not an event.
|
||||
- **Clock going backwards.** A sample earlier than the last one holds with `health-unknown` (unknown
|
||||
`clock`, note `clock-backwards`).
|
||||
- **Swap activity** is parsed (`pswpin`, `pswpout`) and has no rule. D4 lists it as a signal with no
|
||||
threshold.
|
||||
- **Settings.** `resolveRules` refuses:
|
||||
- unknown keys and non-numbers;
|
||||
- a release at or past its hold (load, each PSI resource);
|
||||
- memory release floors at or below the hold floors;
|
||||
- fractions outside (0, 1);
|
||||
- PSI values outside 0 to 100;
|
||||
- sample counts that aren't integers ≥ 1;
|
||||
- a negative release window;
|
||||
- a stale limit at or below the sample interval;
|
||||
- a critical memory fraction at or above the hold fraction;
|
||||
- a non-positive disk floor.
|
||||
- **Large byte values.** Parsed with `Number`. A memory.max above 2^53 loses precision, but it only
|
||||
takes part in a min against MemTotal.
|
||||
|
||||
## Files
|
||||
|
||||
- `packages/supervisor/package.json`, `README.md`
|
||||
- `src/errors.mjs`: `SupervisorError` with `kind` parse, rules or sample.
|
||||
- `src/pressure.mjs`: the nine parsers, `effectiveCpus` and `effectiveMemory`.
|
||||
- `src/rules.mjs`: `DEFAULT_RULES`, `resolveRules`, `initialState`, `evaluate`.
|
||||
- `src/index.mjs`
|
||||
- `tests/pressure.test.mjs`: parsers on kernel-shaped fixture text, with malformed, truncated and
|
||||
repeated-key cases. Effective CPU cases cover the host-only fallback, cpuset, fractional and nested
|
||||
`cpu.max`, and `max`. Effective memory cases cover the host-only fallback, `max`, nested levels,
|
||||
usage over the limit, a limit above MemTotal and a missing memory.current.
|
||||
- `tests/rules.test.mjs`: the D4 table on a fake clock.
|
||||
- Each hold and release threshold on both sides of its edge.
|
||||
- Three-sample counting and the gap reset.
|
||||
- The 30 s release at 29 999 and 30 000 ms, and the dead-band restart.
|
||||
- Missing, stale, future and malformed signals.
|
||||
- PSI never present, and PSI lost after it was present.
|
||||
- Critical OOM, memory and disk at their edges, the reconciliation latch and the counter reset.
|
||||
- A backwards clock, input state left unchanged, and settings refusals and overrides.
|
||||
- `tests/purity.test.mjs`: a static check that src imports only its own files and uses no
|
||||
fs, child_process, net, os, process, dynamic import, Date or timers.
|
||||
|
||||
No test reads /proc, /sys or a cgroup, starts a build, opens a network socket or generates load. No new
|
||||
dependency.
|
||||
|
||||
## Mutants (each must fail its named test; the brief requires M1 and M2)
|
||||
|
||||
| Mutant | Edit | Named test |
|
||||
|---|---|---|
|
||||
| M1 hold edge | `ratio >= rules.load.hold` becomes `>` | load holds at 0.80 for 3 samples and not at 0.7999 |
|
||||
| M2 early release | `now - next.releaseSince >= rules.releaseMs` becomes `>= 0` | the first state is held, and healthy samples open it after exactly 30 s |
|
||||
| M3 memory floor | the hold floor `Math.max` becomes `Math.min` | memory holds below the larger floor, on one sample |
|
||||
| M4 sample count | `next.loadCount >= rules.load.samples` becomes `>= 1` | load holds at 0.80 for 3 samples and not at 0.7999 |
|
||||
| M5 stale | the `now - signal.at > rules.staleMs` check is removed | a signal is stale at more than 6 s old and fresh at exactly 6 s |
|
||||
| M6 effective CPU | the quota `Math.min` becomes `Math.max` | effective CPUs take the smallest of host, cpuset and every quota level |
|
||||
| M7 critical latch | the `!reconciled` critical branch is removed | critical stays until reconciliation, then still waits the 30 s release |
|
||||
|
||||
## Gate (Sage's slot, on CLEAR only)
|
||||
|
||||
- The three test files, each whole, then `node --test tests/` in packages/supervisor as the package
|
||||
suite.
|
||||
- Then M1 to M7, each on its named test, restored after each.
|
||||
- umask 022, TMPDIR from `mktemp -d /mnt/storage/scratch/tmp/gXXXXXX`.
|
||||
- The kit is /mnt/storage/scratch/morgana-r257 (slot257.sh, mutate.py). It refuses unless HEAD is the
|
||||
candidate and the tree is clean, and stops after two unexpected failures (mos-0823).
|
||||
- Static checks before the slot: `node --check` on every file, run from `/`. No test has run yet.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Supervisor
|
||||
|
||||
Q198 S1 (row 257, #1762): parsers for the host pressure inputs in design D4
|
||||
(`docs/plans/2026-10-21_q198-system-manager.md`), and a pure evaluator that turns timestamped samples
|
||||
into a hold or release decision. Nothing here samples the host, runs a process or timer, or enforces
|
||||
anything. The live sampler, the Console view and admission are S2 and S3.
|
||||
|
||||
```sh
|
||||
node --test packages/supervisor/tests/
|
||||
```
|
||||
|
||||
## Parsers
|
||||
|
||||
Each parser takes file text the caller already read. Text that isn't the shape the kernel writes throws
|
||||
a `SupervisorError` with `kind: "parse"`; a sampler treats that as a missing signal.
|
||||
|
||||
| Export | Input | Returns |
|
||||
|---|---|---|
|
||||
| `parseLoadavg` | `/proc/loadavg` | `{ load1, load5, load15 }` |
|
||||
| `parseMeminfo` | `/proc/meminfo` | `{ memTotal, memAvailable, swapTotal, swapFree }` in bytes |
|
||||
| `parsePsi(text, resource)` | `/proc/pressure/{cpu,memory,io}` or a cgroup `*.pressure` | `{ some, full }`, each `{ avg10, avg60, avg300, total }`; `full` is null only for cpu |
|
||||
| `parseCpuList` | `/sys/devices/system/cpu/online`, `cpuset.cpus.effective` | how many CPUs the list names |
|
||||
| `parseCpuMax` | `cpu.max` | the quota in CPUs, or null for `max` |
|
||||
| `parseMemoryMax` | `memory.max` | bytes, or null for `max` |
|
||||
| `parseMemoryCurrent` | `memory.current` | bytes |
|
||||
| `parseMemoryEvents` | `memory.events` | `{ oom, oomKill }` |
|
||||
| `parseVmstat` | `/proc/vmstat` | `{ pswpin, pswpout, oomKill }`; `oomKill` is the host-only OOM counter, null when absent |
|
||||
|
||||
`effectiveCpus({ online, cpuset, cpuMax })` is the smallest of the host count, the cpuset and the quota
|
||||
at every cgroup level given; a fractional quota stays fractional. `effectiveMemory({ meminfo, levels })`
|
||||
takes cgroup levels as `{ max, current }` text. The limit is the smallest of `MemTotal` and every numeric
|
||||
`memory.max`. Available memory is the smallest of `MemAvailable` and `max - current` at each limited level.
|
||||
`memory.current` counts page cache, so this errs toward holding. With no cgroup input both functions use
|
||||
the host's figures.
|
||||
|
||||
## Evaluator
|
||||
|
||||
`evaluate(state, sample, rules, { reconciled })` returns `{ state, decision }`. It starts from
|
||||
`initialState()`, which is held: no signal has been seen yet. Time is the sample's `at` in milliseconds.
|
||||
The evaluator never reads the clock.
|
||||
|
||||
A sample has these signals: `load { at, load1, cpus }`, `memory { at, available, limit }`,
|
||||
`psi { at, cpuSome, memoryFull, ioFull }` (avg10 percentages), `oom { at, kills }` and `disk { at, free }`.
|
||||
|
||||
| Signal | Hold | Release |
|
||||
|---|---|---|
|
||||
| `load1 / cpus` | ≥ 0.80 on 3 consecutive samples | ≤ 0.55 |
|
||||
| available memory | < max(15% of limit, 1 GiB), on one sample | > max(25% of limit, 2 GiB) |
|
||||
| PSI avg10 | cpu some ≥ 20, memory full ≥ 2 or io full ≥ 10, on 3 consecutive samples | cpu some < 10, memory full < 0.5 and io full < 3 |
|
||||
| critical | OOM kill counter rises, memory < 5% of limit, or disk free < 1 GiB | the caller passes `reconciled`, then the normal release |
|
||||
|
||||
- **Release** needs every applicable release condition on every sample for 30 s. A dead-band
|
||||
sample (between hold and release) or a new hold restarts the window.
|
||||
- **Missing or stale:** a required signal that is missing, malformed, more than 6 s older than the
|
||||
sample, or later than it holds at once with `health-unknown`. `load`, `memory`, `oom` and `disk` are
|
||||
always required. PSI becomes required once a sample has carried it. A host without PSI skips the PSI
|
||||
rows, and one that loses it holds.
|
||||
- **Consecutive counts** reset after a gap of more than 6 s between samples. The PSI count runs across
|
||||
resources: any PSI hold condition on 3 consecutive samples holds.
|
||||
- **OOM:** the first OOM sample is the baseline. A counter that falls (a recreated cgroup) resets the
|
||||
baseline with the note `oom-counter-reset`.
|
||||
- **Critical** never ends on thresholds alone. `reconciled` only counts once no critical condition
|
||||
remains, and the 30 s release still follows. The evaluator never sets it and never touches a kill
|
||||
switch.
|
||||
- **A clock that goes backwards** holds with `health-unknown` and the note `clock-backwards`.
|
||||
|
||||
`decision` is `{ mode, reasons, unknown, critical, releaseAt, notes }`.
|
||||
- `mode` is `open`, `held` or `critical`.
|
||||
- `reasons` are `health-unknown`, `load-high`, `memory-low`, `psi-cpu`, `psi-memory`, `psi-io`,
|
||||
`hysteresis`, `release-pending` or the critical codes `critical-oom`, `critical-memory` and
|
||||
`critical-disk`.
|
||||
- `releaseAt` is set while a release is pending.
|
||||
|
||||
`DEFAULT_RULES` is the D4 table, frozen. `resolveRules(overrides)` merges overrides key by key. It refuses
|
||||
with `kind: "rules"` on an unknown key, a non-number or a contradictory pair, for example:
|
||||
- a release threshold at or past its hold;
|
||||
- a release floor at or below its hold floor;
|
||||
- a critical memory fraction at or above the hold fraction;
|
||||
- a stale limit at or below the sample interval.
|
||||
Provider scope (429, quota, Retry-After) is not here; it belongs to S3 and S5.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "@mosaic/supervisor",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Q198 S1: parsers for host pressure inputs and a pure hold and release rules evaluator. No sampler, process or enforcement.",
|
||||
"license": "UNLICENSED",
|
||||
"type": "module",
|
||||
"engines": { "node": ">=24" },
|
||||
"exports": { ".": "./src/index.mjs" },
|
||||
"scripts": { "test": "node --test tests/" }
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
// Every refusal in this package is a SupervisorError. `kind` is "parse" for
|
||||
// input text that isn't the shape the kernel writes, and "rules" for settings
|
||||
// the evaluator refuses.
|
||||
export class SupervisorError extends Error {
|
||||
constructor(message, kind) {
|
||||
super(message);
|
||||
this.name = "SupervisorError";
|
||||
this.kind = kind;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
// @mosaic/supervisor: Q198 S1. Parsers for host pressure inputs and a pure
|
||||
// rules evaluator. Nothing here reads a file, the clock or the environment,
|
||||
// and nothing enforces: no sampler, process, timer, bus or Console (S2, S3).
|
||||
|
||||
export { SupervisorError } from "./errors.mjs";
|
||||
export {
|
||||
parseLoadavg, parseMeminfo, parsePsi, parseCpuList, parseCpuMax, parseMemoryMax, parseMemoryCurrent,
|
||||
parseMemoryEvents, parseVmstat, effectiveCpus, effectiveMemory,
|
||||
} from "./pressure.mjs";
|
||||
export { DEFAULT_RULES, resolveRules, initialState, evaluate } from "./rules.mjs";
|
||||
@@ -0,0 +1,165 @@
|
||||
// Parsers for the host pressure inputs Q198 D4 names, and the effective CPU
|
||||
// count and memory limit they give. Every function takes file text the caller
|
||||
// already read and returns numbers. Nothing here opens a file. Text that isn't
|
||||
// the shape the kernel writes throws a SupervisorError of kind "parse"; the
|
||||
// caller turns that into a missing signal.
|
||||
|
||||
import { SupervisorError } from "./errors.mjs";
|
||||
|
||||
const fail = (what, why) => { throw new SupervisorError(`${what}: ${why}`, "parse"); };
|
||||
|
||||
function text(what, value) {
|
||||
if (typeof value !== "string") fail(what, "expected file text");
|
||||
return value.endsWith("\n") ? value.slice(0, -1) : value;
|
||||
}
|
||||
|
||||
function integer(what, value) {
|
||||
if (!/^\d+$/.test(value)) fail(what, `not a non-negative integer: ${JSON.stringify(value)}`);
|
||||
return Number(value);
|
||||
}
|
||||
|
||||
function decimal(what, value) {
|
||||
if (!/^\d+(\.\d+)?$/.test(value)) fail(what, `not a non-negative number: ${JSON.stringify(value)}`);
|
||||
return Number(value);
|
||||
}
|
||||
|
||||
// "key value" lines (memory.events, /proc/vmstat). A repeated key throws.
|
||||
function keyedIntegers(what, raw) {
|
||||
const out = new Map();
|
||||
for (const line of text(what, raw).split("\n")) {
|
||||
const m = /^([a-z0-9_]+) (\d+)$/.exec(line);
|
||||
if (!m) fail(what, `bad line ${JSON.stringify(line)}`);
|
||||
if (out.has(m[1])) fail(what, `repeated key ${m[1]}`);
|
||||
out.set(m[1], integer(what, m[2]));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// /proc/loadavg: "0.52 0.58 0.59 1/1234 56789".
|
||||
export function parseLoadavg(raw) {
|
||||
const m = /^(\d+\.\d+) (\d+\.\d+) (\d+\.\d+) (\d+)\/(\d+) (\d+)$/.exec(text("loadavg", raw));
|
||||
if (!m) fail("loadavg", "not the /proc/loadavg shape");
|
||||
return { load1: Number(m[1]), load5: Number(m[2]), load15: Number(m[3]) };
|
||||
}
|
||||
|
||||
// /proc/meminfo. Values are returned in bytes; lines without a unit (the
|
||||
// HugePages counts) are kept as plain counts.
|
||||
export function parseMeminfo(raw) {
|
||||
const fields = new Map();
|
||||
for (const line of text("meminfo", raw).split("\n")) {
|
||||
const m = /^([A-Za-z0-9_()]+):\s+(\d+)( kB)?$/.exec(line);
|
||||
if (!m) fail("meminfo", `bad line ${JSON.stringify(line)}`);
|
||||
if (fields.has(m[1])) fail("meminfo", `repeated key ${m[1]}`);
|
||||
fields.set(m[1], integer("meminfo", m[2]) * (m[3] ? 1024 : 1));
|
||||
}
|
||||
for (const key of ["MemTotal", "MemAvailable"]) if (!fields.has(key)) fail("meminfo", `no ${key}`);
|
||||
return {
|
||||
memTotal: fields.get("MemTotal"),
|
||||
memAvailable: fields.get("MemAvailable"),
|
||||
swapTotal: fields.get("SwapTotal") ?? null,
|
||||
swapFree: fields.get("SwapFree") ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
// /proc/pressure/{cpu,memory,io} or a cgroup's *.pressure file. The cpu file
|
||||
// may lack the "full" line on older kernels; memory and io must have it.
|
||||
export function parsePsi(raw, resource) {
|
||||
if (!["cpu", "memory", "io"].includes(resource)) fail("psi", `unknown resource ${JSON.stringify(resource)}`);
|
||||
const what = `psi ${resource}`;
|
||||
const out = { some: null, full: null };
|
||||
for (const line of text(what, raw).split("\n")) {
|
||||
const m = /^(some|full) avg10=(\S+) avg60=(\S+) avg300=(\S+) total=(\S+)$/.exec(line);
|
||||
if (!m) fail(what, `bad line ${JSON.stringify(line)}`);
|
||||
if (out[m[1]]) fail(what, `repeated ${m[1]} line`);
|
||||
out[m[1]] = { avg10: decimal(what, m[2]), avg60: decimal(what, m[3]), avg300: decimal(what, m[4]), total: integer(what, m[5]) };
|
||||
}
|
||||
if (!out.some) fail(what, "no some line");
|
||||
if (!out.full && resource !== "cpu") fail(what, "no full line");
|
||||
return out;
|
||||
}
|
||||
|
||||
// A CPU list such as /sys/devices/system/cpu/online or cpuset.cpus.effective:
|
||||
// "0-3,6,8-9". Returns how many distinct CPUs it names.
|
||||
export function parseCpuList(raw) {
|
||||
const body = text("cpu list", raw);
|
||||
if (body === "") fail("cpu list", "empty");
|
||||
const cpus = new Set();
|
||||
for (const part of body.split(",")) {
|
||||
const m = /^(\d+)(?:-(\d+))?$/.exec(part);
|
||||
if (!m) fail("cpu list", `bad range ${JSON.stringify(part)}`);
|
||||
const lo = integer("cpu list", m[1]);
|
||||
const hi = m[2] === undefined ? lo : integer("cpu list", m[2]);
|
||||
if (hi < lo) fail("cpu list", `range ${part} runs backwards`);
|
||||
for (let c = lo; c <= hi; c++) cpus.add(c);
|
||||
}
|
||||
return cpus.size;
|
||||
}
|
||||
|
||||
// cgroup v2 cpu.max: "max 100000" (no quota) or "50000 100000". Returns the
|
||||
// CPUs the quota allows, or null for no quota.
|
||||
export function parseCpuMax(raw) {
|
||||
const m = /^(max|\d+) (\d+)$/.exec(text("cpu.max", raw));
|
||||
if (!m) fail("cpu.max", "not the cpu.max shape");
|
||||
const period = integer("cpu.max", m[2]);
|
||||
if (period === 0) fail("cpu.max", "period is 0");
|
||||
return m[1] === "max" ? null : integer("cpu.max", m[1]) / period;
|
||||
}
|
||||
|
||||
// cgroup v2 memory.max: "max" (no limit) or bytes. Returns null for "max".
|
||||
export function parseMemoryMax(raw) {
|
||||
const body = text("memory.max", raw);
|
||||
return body === "max" ? null : integer("memory.max", body);
|
||||
}
|
||||
|
||||
// cgroup v2 memory.current: bytes.
|
||||
export function parseMemoryCurrent(raw) {
|
||||
return integer("memory.current", text("memory.current", raw));
|
||||
}
|
||||
|
||||
// cgroup v2 memory.events. oom_kill is the counter the critical row watches.
|
||||
export function parseMemoryEvents(raw) {
|
||||
const fields = keyedIntegers("memory.events", raw);
|
||||
if (!fields.has("oom_kill")) fail("memory.events", "no oom_kill");
|
||||
return { oom: fields.get("oom") ?? null, oomKill: fields.get("oom_kill") };
|
||||
}
|
||||
|
||||
// /proc/vmstat. Swap activity is reported only (D4 has no threshold for it).
|
||||
// oom_kill is the host-only OOM counter; it is null on a kernel without it.
|
||||
export function parseVmstat(raw) {
|
||||
const fields = keyedIntegers("vmstat", raw);
|
||||
for (const key of ["pswpin", "pswpout"]) if (!fields.has(key)) fail("vmstat", `no ${key}`);
|
||||
return { pswpin: fields.get("pswpin"), pswpout: fields.get("pswpout"), oomKill: fields.get("oom_kill") ?? null };
|
||||
}
|
||||
|
||||
// Effective CPUs: the smallest of the host's online CPUs, the cpuset and the
|
||||
// quota at every cgroup level given. With no cgroup input it is the host count.
|
||||
// A fractional quota stays fractional.
|
||||
export function effectiveCpus({ online, cpuset = null, cpuMax = [] }) {
|
||||
let cpus = parseCpuList(online);
|
||||
if (cpuset !== null) cpus = Math.min(cpus, parseCpuList(cpuset));
|
||||
for (const level of cpuMax) {
|
||||
const quota = parseCpuMax(level);
|
||||
if (quota !== null) cpus = Math.min(cpus, quota);
|
||||
}
|
||||
return cpus;
|
||||
}
|
||||
|
||||
// Effective memory. `levels` lists the cgroup levels that apply, each
|
||||
// { max, current } as file text. The limit is the smallest of MemTotal and every
|
||||
// numeric memory.max. Available memory is the smallest of MemAvailable and
|
||||
// (max - current) at every limited level. memory.current counts page cache,
|
||||
// so this errs toward holding. With no levels it is the host's figures.
|
||||
export function effectiveMemory({ meminfo, levels = [] }) {
|
||||
const host = parseMeminfo(meminfo);
|
||||
let limit = host.memTotal;
|
||||
let available = host.memAvailable;
|
||||
for (const level of levels) {
|
||||
const max = parseMemoryMax(level.max);
|
||||
if (max === null) continue;
|
||||
if (level.current === undefined || level.current === null) fail("memory.current", "missing for a limited level");
|
||||
const current = parseMemoryCurrent(level.current);
|
||||
limit = Math.min(limit, max);
|
||||
available = Math.min(available, Math.max(0, max - current));
|
||||
}
|
||||
return { limit, available };
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
// The Q198 D4 hold and release rules as a pure evaluator. evaluate() takes the
|
||||
// previous state, one timestamped sample and the rules, and returns the next
|
||||
// state and a decision. It never reads the clock, a file or the environment:
|
||||
// time is the sample's `at`, in milliseconds. Nothing here enforces anything;
|
||||
// the decision is advice for S2 and S3.
|
||||
|
||||
import { SupervisorError } from "./errors.mjs";
|
||||
|
||||
const GiB = 1024 ** 3;
|
||||
|
||||
// The D4 table. Defaults are the design's proposals, not measured limits.
|
||||
export const DEFAULT_RULES = deepFreeze({
|
||||
sampleMs: 2000,
|
||||
staleMs: 6000,
|
||||
releaseMs: 30000,
|
||||
load: { hold: 0.8, release: 0.55, samples: 3 },
|
||||
memory: { holdFraction: 0.15, holdBytes: 1 * GiB, releaseFraction: 0.25, releaseBytes: 2 * GiB },
|
||||
psi: {
|
||||
samples: 3,
|
||||
hold: { cpuSome: 20, memoryFull: 2, ioFull: 10 },
|
||||
release: { cpuSome: 10, memoryFull: 0.5, ioFull: 3 },
|
||||
},
|
||||
critical: { memoryFraction: 0.05, diskBytes: 1 * GiB },
|
||||
});
|
||||
|
||||
function deepFreeze(o) {
|
||||
for (const v of Object.values(o)) if (v && typeof v === "object") deepFreeze(v);
|
||||
return Object.freeze(o);
|
||||
}
|
||||
|
||||
const refuse = (why) => { throw new SupervisorError(`rules: ${why}`, "rules"); };
|
||||
|
||||
// Overrides merge onto the defaults key by key. An unknown key, a value of the
|
||||
// wrong type or a contradictory pair refuses.
|
||||
function merge(base, over, at) {
|
||||
if (over === undefined) return structuredClone(base);
|
||||
if (!over || typeof over !== "object" || Array.isArray(over)) refuse(`${at || "rules"} must be an object`);
|
||||
const out = {};
|
||||
for (const key of Object.keys(over)) if (!(key in base)) refuse(`unknown key ${at ? `${at}.` : ""}${key}`);
|
||||
for (const [key, value] of Object.entries(base)) {
|
||||
const name = at ? `${at}.${key}` : key;
|
||||
if (typeof value === "object") out[key] = merge(value, over[key], name);
|
||||
else {
|
||||
const v = over[key] ?? value;
|
||||
if (typeof v !== "number" || !Number.isFinite(v)) refuse(`${name} must be a finite number`);
|
||||
out[key] = v;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function resolveRules(overrides = {}) {
|
||||
const r = merge(DEFAULT_RULES, overrides, "");
|
||||
const check = (ok, why) => { if (!ok) refuse(why); };
|
||||
const fraction = (v) => v > 0 && v < 1;
|
||||
const percent = (v) => v >= 0 && v <= 100;
|
||||
check(r.sampleMs > 0, "sampleMs must be above 0");
|
||||
check(r.staleMs > r.sampleMs, "staleMs must be above sampleMs");
|
||||
check(r.releaseMs >= 0, "releaseMs must not be negative");
|
||||
for (const k of ["load", "psi"]) check(Number.isInteger(r[k].samples) && r[k].samples >= 1, `${k}.samples must be an integer of at least 1`);
|
||||
check(r.load.release > 0 && r.load.release < r.load.hold, "load.release must be above 0 and below load.hold");
|
||||
check(fraction(r.memory.holdFraction) && fraction(r.memory.releaseFraction), "memory fractions must be between 0 and 1");
|
||||
check(r.memory.releaseFraction > r.memory.holdFraction, "memory.releaseFraction must be above memory.holdFraction");
|
||||
check(r.memory.holdBytes >= 0 && r.memory.releaseBytes > r.memory.holdBytes, "memory.releaseBytes must be above memory.holdBytes");
|
||||
for (const k of ["cpuSome", "memoryFull", "ioFull"]) {
|
||||
check(percent(r.psi.hold[k]) && percent(r.psi.release[k]), `psi ${k} values must be percentages`);
|
||||
check(r.psi.release[k] > 0 && r.psi.release[k] < r.psi.hold[k], `psi.release.${k} must be above 0 and below psi.hold.${k}`);
|
||||
}
|
||||
check(fraction(r.critical.memoryFraction) && r.critical.memoryFraction < r.memory.holdFraction,
|
||||
"critical.memoryFraction must be between 0 and memory.holdFraction");
|
||||
check(r.critical.diskBytes > 0, "critical.diskBytes must be above 0");
|
||||
return deepFreeze(r);
|
||||
}
|
||||
|
||||
// The state before any sample: held, because no signal has been seen yet.
|
||||
export function initialState() {
|
||||
return { mode: "held", lastAt: null, loadCount: 0, psiCount: 0, psiSeen: false, oomBaseline: null, releaseSince: null, critical: [] };
|
||||
}
|
||||
|
||||
const SIGNAL_FIELDS = {
|
||||
load: ["load1", "cpus"],
|
||||
memory: ["available", "limit"],
|
||||
psi: ["cpuSome", "memoryFull", "ioFull"],
|
||||
oom: ["kills"],
|
||||
disk: ["free"],
|
||||
};
|
||||
|
||||
// A signal is usable when every field is a non-negative finite number, cpus
|
||||
// and limit are above 0, and its own timestamp is no later than the sample and
|
||||
// no more than staleMs before it.
|
||||
function fresh(name, signal, now, rules) {
|
||||
if (!signal || typeof signal !== "object") return false;
|
||||
if (typeof signal.at !== "number" || !Number.isFinite(signal.at)) return false;
|
||||
if (signal.at > now || now - signal.at > rules.staleMs) return false;
|
||||
for (const f of SIGNAL_FIELDS[name]) if (typeof signal[f] !== "number" || !Number.isFinite(signal[f]) || signal[f] < 0) return false;
|
||||
if (name === "load" && signal.cpus <= 0) return false;
|
||||
if (name === "memory" && signal.limit <= 0) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Sample: { at, load: { at, load1, cpus }, memory: { at, available, limit },
|
||||
// psi: { at, cpuSome, memoryFull, ioFull } (avg10 percentages),
|
||||
// oom: { at, kills }, disk: { at, free } }. `reconciled` is the caller's word
|
||||
// that reconciliation finished; only it lets a critical hold end.
|
||||
export function evaluate(state, sample, rules = DEFAULT_RULES, { reconciled = false } = {}) {
|
||||
if (!sample || typeof sample.at !== "number" || !Number.isFinite(sample.at)) {
|
||||
throw new SupervisorError("sample: at must be a finite number of milliseconds", "sample");
|
||||
}
|
||||
const now = sample.at;
|
||||
const next = { ...state, critical: [...state.critical], lastAt: now };
|
||||
const unknown = [];
|
||||
const notes = [];
|
||||
const gap = state.lastAt === null ? Infinity : now - state.lastAt;
|
||||
// A count of consecutive samples survives only a gap of at most staleMs.
|
||||
const contiguous = gap >= 0 && gap <= rules.staleMs;
|
||||
if (gap < 0) { unknown.push("clock"); notes.push("clock-backwards"); }
|
||||
|
||||
let loadHold = false, loadReleases = false;
|
||||
if (fresh("load", sample.load, now, rules)) {
|
||||
const ratio = sample.load.load1 / sample.load.cpus;
|
||||
next.loadCount = ratio >= rules.load.hold ? (contiguous ? state.loadCount : 0) + 1 : 0;
|
||||
loadHold = next.loadCount >= rules.load.samples;
|
||||
loadReleases = ratio <= rules.load.release;
|
||||
} else { unknown.push("load"); next.loadCount = 0; }
|
||||
|
||||
let memoryLow = false, memoryReleases = false, criticalMemory = false;
|
||||
if (fresh("memory", sample.memory, now, rules)) {
|
||||
const { available, limit } = sample.memory;
|
||||
const m = rules.memory;
|
||||
// "Whichever demands more free memory": the larger floor applies.
|
||||
memoryLow = available < Math.max(m.holdFraction * limit, m.holdBytes);
|
||||
memoryReleases = available > Math.max(m.releaseFraction * limit, m.releaseBytes);
|
||||
criticalMemory = available < rules.critical.memoryFraction * limit;
|
||||
} else unknown.push("memory");
|
||||
|
||||
// PSI is required once it has been seen. A host that never had it skips the
|
||||
// PSI rows; one that loses it holds with health-unknown.
|
||||
let psiHold = false, psiReleases = true;
|
||||
const psiOver = [];
|
||||
if (sample.psi !== undefined && sample.psi !== null) next.psiSeen = true;
|
||||
if (fresh("psi", sample.psi, now, rules)) {
|
||||
const p = sample.psi, h = rules.psi.hold, r = rules.psi.release;
|
||||
if (p.cpuSome >= h.cpuSome) psiOver.push("psi-cpu");
|
||||
if (p.memoryFull >= h.memoryFull) psiOver.push("psi-memory");
|
||||
if (p.ioFull >= h.ioFull) psiOver.push("psi-io");
|
||||
next.psiCount = psiOver.length ? (contiguous ? state.psiCount : 0) + 1 : 0;
|
||||
psiHold = next.psiCount >= rules.psi.samples;
|
||||
psiReleases = p.cpuSome < r.cpuSome && p.memoryFull < r.memoryFull && p.ioFull < r.ioFull;
|
||||
} else {
|
||||
next.psiCount = 0;
|
||||
if (next.psiSeen) { unknown.push("psi"); psiReleases = false; }
|
||||
}
|
||||
|
||||
let criticalOom = false;
|
||||
if (fresh("oom", sample.oom, now, rules)) {
|
||||
const kills = sample.oom.kills;
|
||||
if (state.oomBaseline !== null && kills > state.oomBaseline) criticalOom = true;
|
||||
if (state.oomBaseline !== null && kills < state.oomBaseline) notes.push("oom-counter-reset");
|
||||
next.oomBaseline = kills;
|
||||
} else unknown.push("oom");
|
||||
|
||||
let criticalDisk = false;
|
||||
if (fresh("disk", sample.disk, now, rules)) criticalDisk = sample.disk.free < rules.critical.diskBytes;
|
||||
else unknown.push("disk");
|
||||
|
||||
const criticalNow = [];
|
||||
if (criticalOom) criticalNow.push("critical-oom");
|
||||
if (criticalMemory) criticalNow.push("critical-memory");
|
||||
if (criticalDisk) criticalNow.push("critical-disk");
|
||||
|
||||
const decide = (mode, reasons, releaseAt = null) => {
|
||||
next.mode = mode;
|
||||
return { state: next, decision: { mode, reasons, unknown, critical: [...next.critical], releaseAt, notes } };
|
||||
};
|
||||
|
||||
if (criticalNow.length) {
|
||||
for (const c of criticalNow) if (!next.critical.includes(c)) next.critical.push(c);
|
||||
next.releaseSince = null;
|
||||
return decide("critical", [...next.critical]);
|
||||
}
|
||||
// Critical ends only on the caller's reconciliation, never on thresholds alone.
|
||||
if (state.mode === "critical" && !reconciled) {
|
||||
next.releaseSince = null;
|
||||
return decide("critical", [...next.critical]);
|
||||
}
|
||||
const mode = state.mode === "critical" ? "held" : state.mode;
|
||||
next.critical = [];
|
||||
|
||||
const reasons = [];
|
||||
if (unknown.length) reasons.push("health-unknown");
|
||||
if (loadHold) reasons.push("load-high");
|
||||
if (memoryLow) reasons.push("memory-low");
|
||||
if (psiHold) reasons.push(...psiOver);
|
||||
if (reasons.length) { next.releaseSince = null; return decide("held", reasons); }
|
||||
if (mode === "open") { next.releaseSince = null; return decide("open", []); }
|
||||
|
||||
if (!(loadReleases && memoryReleases && psiReleases)) { next.releaseSince = null; return decide("held", ["hysteresis"]); }
|
||||
next.releaseSince = state.releaseSince !== null && contiguous ? state.releaseSince : now;
|
||||
if (now - next.releaseSince >= rules.releaseMs) { next.releaseSince = null; return decide("open", []); }
|
||||
return decide("held", ["release-pending"], next.releaseSince + rules.releaseMs);
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
// Parser and effective-limit tests. Every input is fixture text shaped like
|
||||
// the kernel's files; nothing reads /proc, /sys or a cgroup.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
SupervisorError, parseLoadavg, parseMeminfo, parsePsi, parseCpuList, parseCpuMax, parseMemoryMax,
|
||||
parseMemoryCurrent, parseMemoryEvents, parseVmstat, effectiveCpus, effectiveMemory,
|
||||
} from "../src/index.mjs";
|
||||
|
||||
const GiB = 1024 ** 3;
|
||||
const isParse = (re) => (e) => e instanceof SupervisorError && e.kind === "parse" && re.test(e.message);
|
||||
|
||||
const MEMINFO = [
|
||||
"MemTotal: 32768000 kB",
|
||||
"MemFree: 1024000 kB",
|
||||
"MemAvailable: 8192000 kB",
|
||||
"SwapTotal: 4096000 kB",
|
||||
"SwapFree: 4000000 kB",
|
||||
"HugePages_Total: 0",
|
||||
"Hugepagesize: 2048 kB",
|
||||
"",
|
||||
].join("\n");
|
||||
|
||||
test("loadavg parses the three loads", () => {
|
||||
assert.deepEqual(parseLoadavg("0.52 1.58 12.00 3/1234 56789\n"), { load1: 0.52, load5: 1.58, load15: 12 });
|
||||
});
|
||||
|
||||
test("a malformed loadavg throws", () => {
|
||||
for (const bad of ["", "0.52 0.58 0.59\n", "0.52 0.58 0.59 1/2\n", "x 0.58 0.59 1/2 3\n", "0.52 0.58 0.59 1/2 3 4\n", 42]) {
|
||||
assert.throws(() => parseLoadavg(bad), isParse(/loadavg/), JSON.stringify(bad));
|
||||
}
|
||||
});
|
||||
|
||||
test("meminfo returns bytes and keeps unitless counts", () => {
|
||||
assert.deepEqual(parseMeminfo(MEMINFO), {
|
||||
memTotal: 32768000 * 1024, memAvailable: 8192000 * 1024, swapTotal: 4096000 * 1024, swapFree: 4000000 * 1024,
|
||||
});
|
||||
});
|
||||
|
||||
test("meminfo without MemAvailable, with a repeated key or a bad line throws", () => {
|
||||
assert.throws(() => parseMeminfo("MemTotal: 1 kB\n"), isParse(/no MemAvailable/));
|
||||
assert.throws(() => parseMeminfo(MEMINFO + "MemTotal: 1 kB\n"), isParse(/repeated key MemTotal/));
|
||||
assert.throws(() => parseMeminfo(MEMINFO + "Broken line\n"), isParse(/bad line/));
|
||||
assert.throws(() => parseMeminfo(MEMINFO.replace("8192000", "-1")), isParse(/bad line/));
|
||||
});
|
||||
|
||||
test("PSI parses some and full, and cpu may lack full", () => {
|
||||
const mem = "some avg10=1.50 avg60=0.20 avg300=0.00 total=123\nfull avg10=0.75 avg60=0.10 avg300=0.00 total=45\n";
|
||||
assert.deepEqual(parsePsi(mem, "memory"), {
|
||||
some: { avg10: 1.5, avg60: 0.2, avg300: 0, total: 123 },
|
||||
full: { avg10: 0.75, avg60: 0.1, avg300: 0, total: 45 },
|
||||
});
|
||||
assert.equal(parsePsi("some avg10=20.00 avg60=0.00 avg300=0.00 total=9\n", "cpu").full, null);
|
||||
});
|
||||
|
||||
test("PSI without a full line for memory or io, or with a bad line, throws", () => {
|
||||
const someOnly = "some avg10=0.00 avg60=0.00 avg300=0.00 total=0\n";
|
||||
assert.throws(() => parsePsi(someOnly, "memory"), isParse(/no full line/));
|
||||
assert.throws(() => parsePsi(someOnly, "io"), isParse(/no full line/));
|
||||
assert.throws(() => parsePsi("full avg10=0.00 avg60=0.00 avg300=0.00 total=0\n", "cpu"), isParse(/no some line/));
|
||||
assert.throws(() => parsePsi(someOnly + someOnly, "cpu"), isParse(/repeated some/));
|
||||
assert.throws(() => parsePsi("some avg10=x avg60=0.00 avg300=0.00 total=0\n", "cpu"), isParse(/not a non-negative number/));
|
||||
assert.throws(() => parsePsi(someOnly, "gpu"), isParse(/unknown resource/));
|
||||
});
|
||||
|
||||
test("CPU lists count distinct CPUs", () => {
|
||||
assert.equal(parseCpuList("0-3,6,8-9\n"), 7);
|
||||
assert.equal(parseCpuList("0\n"), 1);
|
||||
assert.equal(parseCpuList("0-3,2-5"), 6);
|
||||
for (const bad of ["", "\n", "3-1", "0,,1", "a-b", "0-"]) assert.throws(() => parseCpuList(bad), isParse(/cpu list/), bad);
|
||||
});
|
||||
|
||||
test("cpu.max gives the quota in CPUs, or null for max", () => {
|
||||
assert.equal(parseCpuMax("max 100000\n"), null);
|
||||
assert.equal(parseCpuMax("50000 100000\n"), 0.5);
|
||||
assert.equal(parseCpuMax("200000 100000\n"), 2);
|
||||
for (const bad of ["max", "50000 0", "-1 100000", "max max"]) assert.throws(() => parseCpuMax(bad), isParse(/cpu.max/), bad);
|
||||
});
|
||||
|
||||
test("memory.max, memory.current and memory.events", () => {
|
||||
assert.equal(parseMemoryMax("max\n"), null);
|
||||
assert.equal(parseMemoryMax("4294967296\n"), 4 * GiB);
|
||||
assert.throws(() => parseMemoryMax("4G\n"), isParse(/memory.max/));
|
||||
assert.equal(parseMemoryCurrent("123456\n"), 123456);
|
||||
assert.throws(() => parseMemoryCurrent(""), isParse(/memory.current/));
|
||||
assert.deepEqual(parseMemoryEvents("low 0\nhigh 4\nmax 2\noom 1\noom_kill 1\noom_group_kill 0\n"), { oom: 1, oomKill: 1 });
|
||||
assert.throws(() => parseMemoryEvents("low 0\noom 1\n"), isParse(/no oom_kill/));
|
||||
assert.throws(() => parseMemoryEvents("oom_kill 1\noom_kill 2\n"), isParse(/repeated key/));
|
||||
});
|
||||
|
||||
test("vmstat gives swap counters and the host OOM counter", () => {
|
||||
assert.deepEqual(parseVmstat("nr_free_pages 100\npswpin 7\npswpout 9\noom_kill 3\n"), { pswpin: 7, pswpout: 9, oomKill: 3 });
|
||||
assert.deepEqual(parseVmstat("pswpin 0\npswpout 0\n"), { pswpin: 0, pswpout: 0, oomKill: null });
|
||||
assert.throws(() => parseVmstat("pswpin 0\n"), isParse(/no pswpout/));
|
||||
});
|
||||
|
||||
test("effective CPUs fall back to the host count with no cgroup input", () => {
|
||||
assert.equal(effectiveCpus({ online: "0-15\n" }), 16);
|
||||
});
|
||||
|
||||
test("effective CPUs take the smallest of host, cpuset and every quota level", () => {
|
||||
assert.equal(effectiveCpus({ online: "0-15\n", cpuset: "0-3\n" }), 4);
|
||||
assert.equal(effectiveCpus({ online: "0-15\n", cpuset: "0-3\n", cpuMax: ["max 100000\n", "150000 100000\n"] }), 1.5);
|
||||
// The outer level is the tighter one.
|
||||
assert.equal(effectiveCpus({ online: "0-15\n", cpuMax: ["800000 100000\n", "200000 100000\n"] }), 2);
|
||||
// A quota above the host count doesn't raise it.
|
||||
assert.equal(effectiveCpus({ online: "0-1\n", cpuMax: ["800000 100000\n"] }), 2);
|
||||
assert.equal(effectiveCpus({ online: "0-15\n", cpuMax: ["max 100000\n"] }), 16);
|
||||
});
|
||||
|
||||
test("effective memory falls back to the host with no cgroup levels, or only max", () => {
|
||||
const host = { limit: 32768000 * 1024, available: 8192000 * 1024 };
|
||||
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO }), host);
|
||||
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: "max\n" }] }), host);
|
||||
});
|
||||
|
||||
test("effective memory uses the tightest limited level", () => {
|
||||
// 4 GiB limit with 3 GiB used: 1 GiB available, below the host's MemAvailable.
|
||||
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${4 * GiB}\n`, current: `${3 * GiB}\n` }] }),
|
||||
{ limit: 4 * GiB, available: 1 * GiB });
|
||||
// Nested: the inner level is loose, the outer one is nearly full.
|
||||
assert.deepEqual(effectiveMemory({ meminfo: MEMINFO, levels: [
|
||||
{ max: "max\n" }, { max: `${16 * GiB}\n`, current: `${2 * GiB}\n` }, { max: `${6 * GiB}\n`, current: `${5.5 * GiB}\n` },
|
||||
] }), { limit: 6 * GiB, available: 0.5 * GiB });
|
||||
// Usage above the limit floors at 0.
|
||||
assert.equal(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: "100\n", current: "200\n" }] }).available, 0);
|
||||
// A limit above MemTotal doesn't raise the limit.
|
||||
assert.equal(effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${64 * GiB}\n`, current: "0\n" }] }).limit, 32768000 * 1024);
|
||||
});
|
||||
|
||||
test("a limited level without memory.current throws", () => {
|
||||
assert.throws(() => effectiveMemory({ meminfo: MEMINFO, levels: [{ max: `${4 * GiB}\n` }] }), isParse(/missing for a limited level/));
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
// S1 has no live enforcement: the source must not reach the filesystem, a
|
||||
// process, the network or the clock. This reads the package's own source text.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const SRC = path.join(import.meta.dirname, "..", "src");
|
||||
const FORBIDDEN = [
|
||||
/\bfrom\s+["'](node:)?(fs|fs\/promises|child_process|net|http|https|dgram|tls|worker_threads|os)["']/,
|
||||
/\bimport\s*\(/,
|
||||
/\brequire\s*\(/,
|
||||
/\bprocess\./,
|
||||
/\bDate\.now\b/,
|
||||
/\bnew Date\b/,
|
||||
/\bperformance\.now\b/,
|
||||
/\bset(Timeout|Interval|Immediate)\b/,
|
||||
];
|
||||
|
||||
test("src reads no file, process, network or clock", () => {
|
||||
const files = fs.readdirSync(SRC).filter((f) => f.endsWith(".mjs"));
|
||||
assert.deepEqual(files.sort(), ["errors.mjs", "index.mjs", "pressure.mjs", "rules.mjs"]);
|
||||
for (const f of files) {
|
||||
const body = fs.readFileSync(path.join(SRC, f), "utf8");
|
||||
for (const re of FORBIDDEN) assert.doesNotMatch(body, re, `${f} matches ${re}`);
|
||||
for (const m of body.matchAll(/\bfrom\s+["']([^"']+)["']/g)) assert.match(m[1], /^\.\/[a-z]+\.mjs$/, `${f} imports ${m[1]}`);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,332 @@
|
||||
// Evaluator tests on a fake clock: each sample's `at` is the time. Synthetic
|
||||
// numbers only; nothing samples the host or generates load.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { SupervisorError, DEFAULT_RULES, resolveRules, initialState, evaluate } from "../src/index.mjs";
|
||||
|
||||
const GiB = 1024 ** 3;
|
||||
const LIMIT = 32 * GiB;
|
||||
const R = DEFAULT_RULES;
|
||||
|
||||
// A healthy sample at `at`. `patch` changes fields per signal; null drops one.
|
||||
function sample(at, patch = {}) {
|
||||
const base = {
|
||||
load: { load1: 0.1, cpus: 1 },
|
||||
memory: { available: 16 * GiB, limit: LIMIT },
|
||||
psi: { cpuSome: 0, memoryFull: 0, ioFull: 0 },
|
||||
oom: { kills: 0 },
|
||||
disk: { free: 100 * GiB },
|
||||
};
|
||||
const out = { at };
|
||||
for (const [name, fields] of Object.entries(base)) {
|
||||
if (patch[name] === null) continue;
|
||||
out[name] = { at, ...fields, ...patch[name] };
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Fold samples through evaluate; returns every decision and the last state.
|
||||
function run(state, samples, rules = R, opts) {
|
||||
const decisions = [];
|
||||
for (const s of samples) {
|
||||
const r = evaluate(state, s, rules, typeof opts === "function" ? opts(s) : opts);
|
||||
state = r.state;
|
||||
decisions.push(r.decision);
|
||||
}
|
||||
return { state, decisions, last: decisions.at(-1) };
|
||||
}
|
||||
|
||||
// Samples every 2 s from `from` to `to` inclusive.
|
||||
function every2s(from, to, patch) {
|
||||
const out = [];
|
||||
for (let at = from; at <= to; at += 2000) out.push(sample(at, typeof patch === "function" ? patch(at) : patch));
|
||||
return out;
|
||||
}
|
||||
|
||||
// An open state whose last sample was at 30000.
|
||||
function opened() {
|
||||
const { state, last } = run(initialState(), every2s(0, 30000));
|
||||
assert.equal(last.mode, "open");
|
||||
return state;
|
||||
}
|
||||
|
||||
// A state held by a load hold, last sample at 36000.
|
||||
function heldByLoad() {
|
||||
const { state, last } = run(opened(), every2s(32000, 36000, { load: { load1: 0.9 } }));
|
||||
assert.deepEqual(last.reasons, ["load-high"]);
|
||||
return state;
|
||||
}
|
||||
|
||||
const modes = (r) => r.decisions.map((d) => d.mode);
|
||||
|
||||
test("the first state is held, and healthy samples open it after exactly 30 s", () => {
|
||||
assert.equal(initialState().mode, "held");
|
||||
const first = evaluate(initialState(), sample(0));
|
||||
assert.equal(first.decision.mode, "held");
|
||||
assert.deepEqual(first.decision.reasons, ["release-pending"]);
|
||||
assert.equal(first.decision.releaseAt, 30000);
|
||||
const r = run(initialState(), [sample(0), sample(2000), sample(6000), sample(10000), sample(14000), sample(18000),
|
||||
sample(22000), sample(26000), sample(29999), sample(30000)]);
|
||||
assert.deepEqual(r.decisions.at(-2).reasons, ["release-pending"]);
|
||||
assert.equal(r.decisions.at(-2).mode, "held");
|
||||
assert.equal(r.last.mode, "open");
|
||||
});
|
||||
|
||||
test("load holds at 0.80 for 3 samples and not at 0.7999", () => {
|
||||
const at = run(opened(), every2s(32000, 36000, { load: { load1: 0.8 } }));
|
||||
assert.deepEqual(modes(at), ["open", "open", "held"]);
|
||||
assert.deepEqual(at.last.reasons, ["load-high"]);
|
||||
const below = run(opened(), every2s(32000, 50000, { load: { load1: 0.7999 } }));
|
||||
assert.ok(below.decisions.every((d) => d.mode === "open"));
|
||||
});
|
||||
|
||||
test("load divides by effective CPUs, fractional ones included", () => {
|
||||
const r = run(opened(), every2s(32000, 36000, { load: { load1: 0.4, cpus: 0.5 } }));
|
||||
assert.equal(r.last.mode, "held");
|
||||
const r2 = run(opened(), every2s(32000, 36000, { load: { load1: 3.1, cpus: 4 } }));
|
||||
assert.equal(r2.last.mode, "open");
|
||||
});
|
||||
|
||||
test("a gap of more than 6 s resets the consecutive count", () => {
|
||||
const high = { load: { load1: 0.9 } };
|
||||
const r = run(opened(), [sample(32000, high), sample(34000, high), sample(40001, high), sample(42001, high), sample(44001, high)]);
|
||||
assert.deepEqual(modes(r), ["open", "open", "open", "open", "held"]);
|
||||
const ok = run(opened(), [sample(32000, high), sample(34000, high), sample(40000, high)]);
|
||||
assert.equal(ok.last.mode, "held");
|
||||
});
|
||||
|
||||
test("load releases at 0.55 and not at 0.5501", () => {
|
||||
const dead = run(heldByLoad(), every2s(38000, 80000, { load: { load1: 0.5501 } }));
|
||||
assert.ok(dead.decisions.every((d) => d.mode === "held" && d.reasons[0] === "hysteresis"));
|
||||
const r = run(heldByLoad(), every2s(38000, 68000, { load: { load1: 0.55 } }));
|
||||
assert.equal(r.decisions.at(-2).mode, "held");
|
||||
assert.equal(r.last.mode, "open");
|
||||
});
|
||||
|
||||
test("a dead-band sample restarts the 30 s release window", () => {
|
||||
const r = run(heldByLoad(), [
|
||||
...every2s(38000, 60000),
|
||||
sample(62000, { load: { load1: 0.6 } }),
|
||||
...every2s(64000, 92000),
|
||||
sample(94000),
|
||||
]);
|
||||
const at = (t) => r.decisions[r.decisions.length - 1 - (94000 - t) / 2000];
|
||||
assert.equal(at(62000).reasons[0], "hysteresis");
|
||||
assert.equal(at(92000).mode, "held");
|
||||
assert.equal(at(94000).mode, "open");
|
||||
});
|
||||
|
||||
test("memory holds below the larger floor, on one sample", () => {
|
||||
const floor = Math.max(R.memory.holdFraction * LIMIT, R.memory.holdBytes);
|
||||
assert.ok(floor > R.memory.holdBytes); // 15% of 32 GiB is the larger floor here
|
||||
const just = Math.ceil(floor);
|
||||
assert.equal(evaluate(opened(), sample(32000, { memory: { available: just } })).decision.mode, "open");
|
||||
const low = evaluate(opened(), sample(32000, { memory: { available: just - 1 } })).decision;
|
||||
assert.equal(low.mode, "held");
|
||||
assert.deepEqual(low.reasons, ["memory-low"]);
|
||||
});
|
||||
|
||||
test("on a small limit the 1 GiB floor is the larger one", () => {
|
||||
const small = 4 * GiB; // 15% is 0.6 GiB, so 1 GiB applies
|
||||
const s = (available) => sample(32000, { memory: { available, limit: small } });
|
||||
assert.equal(evaluate(opened(), s(1 * GiB)).decision.mode, "open");
|
||||
assert.deepEqual(evaluate(opened(), s(1 * GiB - 1)).decision.reasons, ["memory-low"]);
|
||||
});
|
||||
|
||||
test("memory releases only above the larger release floor", () => {
|
||||
// 3 GiB is below the 15% floor (4.8 GiB) and above the 5% critical line (1.6 GiB).
|
||||
const held = evaluate(opened(), sample(32000, { memory: { available: 3 * GiB } })).state;
|
||||
assert.equal(held.mode, "held");
|
||||
const rel = Math.max(R.memory.releaseFraction * LIMIT, R.memory.releaseBytes);
|
||||
const stay = run(held, every2s(34000, 80000, { memory: { available: Math.floor(rel) } }));
|
||||
assert.ok(stay.decisions.every((d) => d.mode === "held"));
|
||||
const go = run(held, every2s(34000, 64000, { memory: { available: Math.floor(rel) + 1 } }));
|
||||
assert.equal(go.last.mode, "open");
|
||||
const small = { limit: 4 * GiB }; // 25% is 1 GiB, so 2 GiB applies
|
||||
const stay2 = run(held, every2s(34000, 80000, { memory: { ...small, available: 2 * GiB } }));
|
||||
assert.ok(stay2.decisions.every((d) => d.mode === "held"));
|
||||
const go2 = run(held, every2s(34000, 64000, { memory: { ...small, available: 2 * GiB + 1 } }));
|
||||
assert.equal(go2.last.mode, "open");
|
||||
});
|
||||
|
||||
test("each PSI hold threshold holds at its value for 3 samples and not just below", () => {
|
||||
for (const [field, code] of [["cpuSome", "psi-cpu"], ["memoryFull", "psi-memory"], ["ioFull", "psi-io"]]) {
|
||||
const at = run(opened(), every2s(32000, 36000, { psi: { [field]: R.psi.hold[field] } }));
|
||||
assert.deepEqual(modes(at), ["open", "open", "held"], field);
|
||||
assert.deepEqual(at.last.reasons, [code], field);
|
||||
const below = run(opened(), every2s(32000, 50000, { psi: { [field]: R.psi.hold[field] - 0.01 } }));
|
||||
assert.ok(below.decisions.every((d) => d.mode === "open"), field);
|
||||
}
|
||||
});
|
||||
|
||||
test("the PSI count runs across resources", () => {
|
||||
const r = run(opened(), [sample(32000, { psi: { cpuSome: 25 } }), sample(34000, { psi: { memoryFull: 3 } }), sample(36000, { psi: { ioFull: 12 } })]);
|
||||
assert.deepEqual(modes(r), ["open", "open", "held"]);
|
||||
assert.deepEqual(r.last.reasons, ["psi-io"]);
|
||||
});
|
||||
|
||||
test("each PSI release threshold blocks at its value and releases just below", () => {
|
||||
const held = run(opened(), every2s(32000, 36000, { psi: { cpuSome: 30 } })).state;
|
||||
for (const field of ["cpuSome", "memoryFull", "ioFull"]) {
|
||||
const stay = run(held, every2s(38000, 80000, { psi: { [field]: R.psi.release[field] } }));
|
||||
assert.ok(stay.decisions.every((d) => d.mode === "held" && d.reasons[0] === "hysteresis"), field);
|
||||
const go = run(held, every2s(38000, 68000, { psi: { [field]: R.psi.release[field] - 0.01 } }));
|
||||
assert.equal(go.last.mode, "open", field);
|
||||
}
|
||||
});
|
||||
|
||||
test("a missing required signal holds at once with health-unknown", () => {
|
||||
for (const name of ["load", "memory", "oom", "disk"]) {
|
||||
const d = evaluate(opened(), sample(32000, { [name]: null })).decision;
|
||||
assert.equal(d.mode, "held", name);
|
||||
assert.deepEqual(d.reasons, ["health-unknown"], name);
|
||||
assert.deepEqual(d.unknown, [name], name);
|
||||
}
|
||||
});
|
||||
|
||||
test("a signal is stale at more than 6 s old and fresh at exactly 6 s", () => {
|
||||
const stale = { at: 32000, load1: 0.1, cpus: 1 };
|
||||
const s = (signalAt) => ({ ...sample(32000), load: { ...stale, at: signalAt } });
|
||||
assert.equal(evaluate(opened(), s(26000)).decision.mode, "open");
|
||||
assert.deepEqual(evaluate(opened(), s(25999)).decision.unknown, ["load"]);
|
||||
assert.deepEqual(evaluate(opened(), s(32001)).decision.unknown, ["load"]); // later than the sample
|
||||
});
|
||||
|
||||
test("malformed signal values count as missing", () => {
|
||||
for (const patch of [{ load: { load1: NaN } }, { load: { cpus: 0 } }, { load: { load1: -1 } }, { memory: { limit: 0 } },
|
||||
{ memory: { available: "1" } }, { oom: { kills: Infinity } }, { disk: { free: undefined } }]) {
|
||||
const d = evaluate(opened(), sample(32000, patch)).decision;
|
||||
assert.deepEqual(d.reasons, ["health-unknown"], JSON.stringify(patch));
|
||||
}
|
||||
});
|
||||
|
||||
test("after health-unknown, recovery waits the full 30 s", () => {
|
||||
const unknown = evaluate(opened(), sample(32000, { load: null })).state;
|
||||
const r = run(unknown, every2s(34000, 64000));
|
||||
assert.equal(r.decisions.at(-2).mode, "held");
|
||||
assert.equal(r.last.mode, "open");
|
||||
});
|
||||
|
||||
test("a host that never had PSI skips the PSI rows", () => {
|
||||
const r = run(initialState(), every2s(0, 30000, { psi: null }));
|
||||
assert.equal(r.last.mode, "open");
|
||||
assert.deepEqual(r.last.unknown, []);
|
||||
});
|
||||
|
||||
test("PSI that disappears after it was present holds with health-unknown", () => {
|
||||
const d = evaluate(opened(), sample(32000, { psi: null })).decision;
|
||||
assert.equal(d.mode, "held");
|
||||
assert.deepEqual(d.reasons, ["health-unknown"]);
|
||||
assert.deepEqual(d.unknown, ["psi"]);
|
||||
// Also when it first appears late, then goes.
|
||||
const late = run(initialState(), [...every2s(0, 30000, { psi: null }), sample(32000), sample(34000, { psi: null })]);
|
||||
assert.deepEqual(late.last.unknown, ["psi"]);
|
||||
// A stale PSI signal counts the same.
|
||||
const stalePsi = { ...sample(32000), psi: { at: 25000, cpuSome: 0, memoryFull: 0, ioFull: 0 } };
|
||||
assert.deepEqual(evaluate(opened(), stalePsi).decision.unknown, ["psi"]);
|
||||
});
|
||||
|
||||
test("a rise in the OOM kill counter is critical; the first sample is the baseline", () => {
|
||||
const first = evaluate(initialState(), sample(0, { oom: { kills: 5 } }));
|
||||
assert.notEqual(first.decision.mode, "critical");
|
||||
const r = run(opened(), [sample(32000, { oom: { kills: 0 } }), sample(34000, { oom: { kills: 1 } })]);
|
||||
assert.equal(r.last.mode, "critical");
|
||||
assert.deepEqual(r.last.reasons, ["critical-oom"]);
|
||||
});
|
||||
|
||||
test("a falling OOM counter resets the baseline with a note, not an event", () => {
|
||||
const s = run(initialState(), every2s(0, 30000, { oom: { kills: 6 } })).state;
|
||||
assert.equal(s.mode, "open");
|
||||
const r = run(s, [sample(32000, { oom: { kills: 2 } }), sample(34000, { oom: { kills: 2 } })]);
|
||||
assert.deepEqual(r.decisions[0].notes, ["oom-counter-reset"]);
|
||||
assert.ok(r.decisions.every((d) => d.mode === "open"));
|
||||
});
|
||||
|
||||
test("memory under 5% and disk under 1 GiB are critical, at their edges", () => {
|
||||
const five = R.critical.memoryFraction * LIMIT;
|
||||
assert.equal(evaluate(opened(), sample(32000, { memory: { available: Math.ceil(five) } })).decision.mode, "held");
|
||||
const mem = evaluate(opened(), sample(32000, { memory: { available: Math.ceil(five) - 1 } })).decision;
|
||||
assert.equal(mem.mode, "critical");
|
||||
assert.deepEqual(mem.reasons, ["critical-memory"]);
|
||||
assert.equal(evaluate(opened(), sample(32000, { disk: { free: 1 * GiB } })).decision.mode, "open");
|
||||
assert.deepEqual(evaluate(opened(), sample(32000, { disk: { free: 1 * GiB - 1 } })).decision.reasons, ["critical-disk"]);
|
||||
});
|
||||
|
||||
test("critical stays until reconciliation, then still waits the 30 s release", () => {
|
||||
const crit = evaluate(opened(), sample(32000, { disk: { free: 0 } })).state;
|
||||
const waiting = run(crit, every2s(34000, 120000));
|
||||
assert.ok(waiting.decisions.every((d) => d.mode === "critical" && d.reasons[0] === "critical-disk"));
|
||||
// Reconciled while the condition is still there: still critical.
|
||||
assert.equal(evaluate(crit, sample(34000, { disk: { free: 0 } }), R, { reconciled: true }).decision.mode, "critical");
|
||||
// Reconciled and healthy: held, then open 30 s later.
|
||||
const after = run(waiting.state, every2s(122000, 152000), R, (s) => ({ reconciled: s.at === 122000 }));
|
||||
assert.deepEqual(after.decisions[0].reasons, ["release-pending"]);
|
||||
assert.deepEqual(after.decisions[0].critical, []);
|
||||
assert.equal(after.decisions.at(-2).mode, "held");
|
||||
assert.equal(after.last.mode, "open");
|
||||
});
|
||||
|
||||
test("critical reasons accumulate while critical", () => {
|
||||
const r = run(opened(), [sample(32000, { disk: { free: 0 } }), sample(34000, { oom: { kills: 3 } })]);
|
||||
assert.deepEqual(r.last.reasons, ["critical-disk", "critical-oom"]);
|
||||
});
|
||||
|
||||
test("a clock that goes backwards holds with health-unknown", () => {
|
||||
const d = evaluate(opened(), sample(20000)).decision;
|
||||
assert.equal(d.mode, "held");
|
||||
assert.deepEqual(d.unknown, ["clock"]);
|
||||
assert.deepEqual(d.notes, ["clock-backwards"]);
|
||||
});
|
||||
|
||||
test("a sample without a numeric time throws", () => {
|
||||
for (const bad of [null, {}, { at: "1" }, { at: NaN }]) {
|
||||
assert.throws(() => evaluate(initialState(), bad), (e) => e instanceof SupervisorError && e.kind === "sample");
|
||||
}
|
||||
});
|
||||
|
||||
test("evaluate doesn't change the state it was given", () => {
|
||||
const s = Object.freeze({ ...opened(), critical: Object.freeze([]) });
|
||||
assert.doesNotThrow(() => evaluate(s, sample(32000, { disk: { free: 0 } })));
|
||||
assert.equal(s.mode, "open");
|
||||
});
|
||||
|
||||
test("resolveRules with no overrides is the D4 table, frozen", () => {
|
||||
assert.deepEqual(resolveRules(), DEFAULT_RULES);
|
||||
assert.ok(Object.isFrozen(DEFAULT_RULES) && Object.isFrozen(DEFAULT_RULES.psi.hold));
|
||||
assert.deepEqual(DEFAULT_RULES.load, { hold: 0.8, release: 0.55, samples: 3 });
|
||||
assert.equal(DEFAULT_RULES.releaseMs, 30000);
|
||||
assert.equal(DEFAULT_RULES.staleMs, 6000);
|
||||
});
|
||||
|
||||
test("overrides apply in the evaluator", () => {
|
||||
const rules = resolveRules({ releaseMs: 10000, load: { samples: 1 } });
|
||||
const r = run(initialState(), every2s(0, 10000), rules);
|
||||
assert.equal(r.last.mode, "open");
|
||||
assert.equal(evaluate(r.state, sample(12000, { load: { load1: 0.9 } }), rules).decision.mode, "held");
|
||||
});
|
||||
|
||||
test("contradictory or unknown settings refuse", () => {
|
||||
const cases = [
|
||||
[{ load: { release: 0.8 } }, /load.release must be above 0 and below load.hold/],
|
||||
[{ load: { hold: 0.5 } }, /load.release must be above 0 and below load.hold/],
|
||||
[{ memory: { releaseFraction: 0.15 } }, /releaseFraction must be above/],
|
||||
[{ memory: { releaseBytes: 1 * GiB } }, /releaseBytes must be above/],
|
||||
[{ memory: { holdFraction: 1.2 } }, /between 0 and 1/],
|
||||
[{ psi: { release: { ioFull: 10 } } }, /psi.release.ioFull must be above 0 and below psi.hold.ioFull/],
|
||||
[{ psi: { hold: { cpuSome: 120 } } }, /percentages/],
|
||||
[{ psi: { samples: 0 } }, /psi.samples must be an integer/],
|
||||
[{ load: { samples: 1.5 } }, /load.samples must be an integer/],
|
||||
[{ staleMs: 2000 }, /staleMs must be above sampleMs/],
|
||||
[{ releaseMs: -1 }, /releaseMs must not be negative/],
|
||||
[{ critical: { memoryFraction: 0.15 } }, /critical.memoryFraction/],
|
||||
[{ critical: { diskBytes: 0 } }, /critical.diskBytes/],
|
||||
[{ loadd: {} }, /unknown key loadd/],
|
||||
[{ psi: { hold: { gpu: 1 } } }, /unknown key psi.hold.gpu/],
|
||||
[{ load: { hold: "0.8" } }, /load.hold must be a finite number/],
|
||||
[{ load: 1 }, /load must be an object/],
|
||||
[[], /rules must be an object/],
|
||||
];
|
||||
for (const [over, re] of cases) {
|
||||
assert.throws(() => resolveRules(over), (e) => e instanceof SupervisorError && e.kind === "rules" && re.test(e.message), JSON.stringify(over));
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user