Files

11 KiB

Row 257: Q198 S1 pressure parser and pure rules evaluator

Issue #1762. Brief: docs/plans/2026-10-21_post-v1-row-briefs.md, "Brief: Q198 S1 pressure parser and pure rules evaluator" and "Gate: Q198 S1" (cca206cdc). Design: docs/plans/2026-10-21_q198-system-manager.md r6.1, section 8 row S1 and section 4 (D4). Owner Morgana, reviewer Filbert, gate owner Sage (sage-2196). Post-v1: MVP rows and re-pin windows go ahead of it on the slot. Base: refactor cca206cdc.

Recorded choices (sage-2196 accepted each)

  1. Placement. A new package, packages/supervisor (@mosaic/supervisor), shaped like packages/runs. Nothing on refactor parsed loadavg, PSI, meminfo or cgroup limits (git grep; the only hit was an awk line in scripts/case2-precheck.sh). S2 adds the sampler process to the same package. Packages aren't registered centrally, so no other file changes.
  2. Provider scope (429, quota, Retry-After, the half-open probe) is out. The brief puts it in S3 and S5.
  3. PSI is not required on a host that has never shown it, so a kernel built without PSI skips the PSI rows instead of holding forever. Once a sample has carried PSI, it is required: a later missing or stale PSI signal holds with health-unknown. That is the case sage-2196 asked for.
    • Tradeoff: I latch on the first sample that carries PSI, not only on the very first sample. So PSI that appears late and then disappears also holds. This is stricter than "present at the first sample", and it never skips a signal the host has shown.
  4. Memory under a cgroup. Available memory = min(MemAvailable, limit - memory.current) at each limited level. memory.current counts page cache, so this errs toward holding.
    • Follow-up, not in this row: subtract reclaimable cache using memory.stat inactive_file, if S2's live samples show holds that the host's MemAvailable doesn't support.

Rulings on two readings from the build (sage-2202, re morgana-0057):

  1. PSI latch. PSI becomes required on the first sample that carries it. This is stricter than "present at the first sample" and fails closed: a PSI file that disappears after it was seen is a missing signal, so health-unknown holds.
  2. PSI count across resources. Design line 152 says "CPU some ≥20, memory full ≥2, or I/O full ≥10, for 3 samples". The window applies to the whole disjunction, so any PSI condition on 3 consecutive samples holds. If Filbert reads it per resource at review, the review finding decides, and the fix is a single counter.

Other readings of the D4 table, made in the build:

  • Memory floors. "Whichever demands more free memory" means the larger floor applies. Hold below max(15% x limit, 1 GiB); release above max(25% x limit, 2 GiB).
  • Edges as written. Hold at ≥ for load and PSI, < for memory. Release at ≤ 0.55 for load, < for PSI, > for memory. Critical below 5% of the limit and below 1 GiB of disk.
  • Sample counts. "For 3 samples" means 3 consecutive samples. The count resets after a gap of more than 6 s (the stale limit) between samples. The PSI row counts as one row (ruling 6), and the reasons name the resources over on the latest sample.
  • Memory holds on one sample. The table's memory row has no sample count.
  • Release needs every applicable release condition on every sample for 30 s, by the samples' own times (>= 30000 ms after the first qualifying sample). A dead-band sample or any hold restarts the window, and so does a gap of more than 6 s.
  • Start held. initialState() is held, because nothing has been seen yet. The first healthy sample starts the 30 s window.
  • Missing or stale. Required signals are load, memory, oom and disk, plus PSI once seen. A signal holds at once with health-unknown if it is:
    • missing;
    • non-numeric, negative or zero where a divisor;
    • more than 6 s older than its sample (6 s exactly is fresh);
    • stamped later than its sample. Disk and OOM are required because the critical row can't be evaluated without them.
  • Host-only OOM. /proc/vmstat oom_kill gives the OOM counter when there's no cgroup memory.events. The parser returns null for a kernel without it, and the sampler then has no OOM signal (health-unknown).
  • Critical latches. It ends only when the caller passes reconciled with no critical condition in the sample, then goes through held and the normal 30 s release. The evaluator never sets reconciled and has no kill-switch path. Critical reasons accumulate while latched.
  • OOM counter. The first OOM sample is the baseline. A falling counter (a recreated cgroup) resets the baseline with a note, not an event.
  • Clock going backwards. A sample earlier than the last one holds with health-unknown (unknown clock, note clock-backwards).
  • Swap activity is parsed (pswpin, pswpout) and has no rule. D4 lists it as a signal with no threshold.
  • Settings. resolveRules refuses:
    • unknown keys and non-numbers;
    • a release at or past its hold (load, each PSI resource);
    • memory release floors at or below the hold floors;
    • fractions outside (0, 1);
    • PSI values outside 0 to 100;
    • sample counts that aren't integers ≥ 1;
    • a negative release window;
    • a stale limit at or below the sample interval;
    • a critical memory fraction at or above the hold fraction;
    • a non-positive disk floor.
  • Large byte values. Parsed with Number. A memory.max above 2^53 loses precision, but it only takes part in a min against MemTotal.

Round 2 (Filbert r1 APPROVE c30188 notes, sage-2212)

  • N1: merge checked unknown keys with key in base, so inherited names (toString, constructor, __proto__) passed the check and were dropped instead of refused. It now uses Object.hasOwn(base, key), and the refusal test has those three cases.

  • N2: the critical-memory edge was tested only on a 32 GiB limit, where 5% is not an integer, so a < to <= mutant survived. The test adds a 20 GiB limit, where 5% is exactly 1 GiB, and M8 is that mutant. I checked the other memory edges for the same gap:

    • The hold floor is non-integer on 32 GiB but is tested at exactly 1 GiB on a 4 GiB limit.
    • The release floors are 8 GiB on 32 GiB and 2 GiB on 4 GiB, both integers.
    • The disk edge is 1 GiB.

    None has the gap.

Files

  • packages/supervisor/package.json, README.md
  • src/errors.mjs: SupervisorError with kind parse, rules or sample.
  • src/pressure.mjs: the nine parsers, effectiveCpus and effectiveMemory.
  • src/rules.mjs: DEFAULT_RULES, resolveRules, initialState, evaluate.
  • src/index.mjs
  • tests/pressure.test.mjs: parsers on kernel-shaped fixture text, with malformed, truncated and repeated-key cases. Effective CPU cases cover the host-only fallback, cpuset, fractional and nested cpu.max, and max. Effective memory cases cover the host-only fallback, max, nested levels, usage over the limit, a limit above MemTotal and a missing memory.current.
  • tests/rules.test.mjs: the D4 table on a fake clock.
    • Each hold and release threshold on both sides of its edge.
    • Three-sample counting and the gap reset.
    • The 30 s release at 29 999 and 30 000 ms, and the dead-band restart.
    • Missing, stale, future and malformed signals.
    • PSI never present, and PSI lost after it was present.
    • Critical OOM, memory and disk at their edges, the reconciliation latch and the counter reset.
    • A backwards clock, input state left unchanged, and settings refusals and overrides.
  • tests/purity.test.mjs: a static check that src imports only its own files and uses no fs, child_process, net, os, process, dynamic import, Date or timers.

No test reads /proc, /sys or a cgroup, starts a build, opens a network socket or generates load. No new dependency.

Mutants (each must fail its named test; the brief requires M1 and M2)

Mutant Edit Named test
M1 hold edge ratio >= rules.load.hold becomes > load holds at 0.80 for 3 samples and not at 0.7999
M2 early release now - next.releaseSince >= rules.releaseMs becomes >= 0 the first state is held, and healthy samples open it after exactly 30 s
M3 memory floor the hold floor Math.max becomes Math.min memory holds below the larger floor, on one sample
M4 sample count next.loadCount >= rules.load.samples becomes >= 1 load holds at 0.80 for 3 samples and not at 0.7999
M5 stale the now - signal.at > rules.staleMs check is removed a signal is stale at more than 6 s old and fresh at exactly 6 s
M6 effective CPU the quota Math.min becomes Math.max effective CPUs take the smallest of host, cpuset and every quota level
M7 critical latch the !reconciled critical branch is removed critical stays until reconciliation, then still waits the 30 s release
M8 critical memory edge (r2) available < rules.critical.memoryFraction * limit becomes <= memory under 5% and disk under 1 GiB are critical, at their edges

Gate (Sage's slot, on CLEAR only)

  • The three test files, each whole, then node --test tests/ in packages/supervisor as the package suite.
  • Then M1 to M8, each on its named test, restored after each.
  • umask 022, TMPDIR from mktemp -d /mnt/storage/scratch/tmp/gXXXXXX.
  • The kit is /mnt/storage/scratch/morgana-r257 (slot257.sh, mutate.py). It refuses unless HEAD is the candidate and the tree is clean, and stops after two unexpected failures (mos-0823).
  • Static checks before the slot: node --check on every file, run from /. No test has run yet.

Slot result (sage-4159 CLEAR, 2026-10-24 17:00:27Z to 17:00:30Z, candidate 1261fa2d8)

Run as busy.sh && timeout -k 10 1500 slot257.sh in its own session (setsid). The busy check read CLEAR. The worktree was checked out at 1261fa2d8 with a clean tree. Filbert's approved code is unchanged. The summary is in gate-run.log and the per-run node output in mutants-run.log.

Run tests pass fail skip cancelled todo
tests/pressure.test.mjs whole 15 15 0 0 0 0
tests/rules.test.mjs whole 29 29 0 0 0 0
tests/purity.test.mjs whole 1 1 0 0 0 0
package suite node --test tests/ 45 45 0 0 0 0

Mutants: 8 of 8 killed. Each one ran only its named test (1 test, rc=1), and each failure was an AssertionError in that test, not a crash: M1 hold edge, M2 early release, M3 memory floor, M4 sample count, M5 stale, M6 effective CPU, M7 critical latch, M8 critical memory edge.

Restore check: git checkout -- packages/supervisor ran after each mutant, and the script found the tree clean every time. At the end HEAD was 1261fa2d8 and git status --porcelain was empty. After the run no node, python or bash process had its cwd under the kit or the worktree, and no g257 TMPDIR was left behind. The run reached no scripts/test-*.sh.