167 lines
11 KiB
Markdown
167 lines
11 KiB
Markdown
# Row 257: Q198 S1 pressure parser and pure rules evaluator
|
|
|
|
Issue #1762. Brief: `docs/plans/2026-10-21_post-v1-row-briefs.md`, "Brief: Q198 S1 pressure parser and
|
|
pure rules evaluator" and "Gate: Q198 S1" (cca206cdc). Design: `docs/plans/2026-10-21_q198-system-manager.md`
|
|
r6.1, section 8 row S1 and section 4 (D4). Owner Morgana, reviewer Filbert, gate owner Sage (sage-2196).
|
|
Post-v1: MVP rows and re-pin windows go ahead of it on the slot. Base: refactor cca206cdc.
|
|
|
|
## Recorded choices (sage-2196 accepted each)
|
|
|
|
1. **Placement.** A new package, `packages/supervisor` (`@mosaic/supervisor`), shaped like
|
|
packages/runs. Nothing on refactor parsed loadavg, PSI, meminfo or cgroup limits (`git grep`; the
|
|
only hit was an awk line in scripts/case2-precheck.sh). S2 adds the sampler process to the same
|
|
package. Packages aren't registered centrally, so no other file changes.
|
|
2. **Provider scope** (429, quota, Retry-After, the half-open probe) is out. The brief puts it in S3
|
|
and S5.
|
|
3. **PSI** is not required on a host that has never shown it, so a kernel built without PSI skips the
|
|
PSI rows instead of holding forever. Once a sample has carried PSI, it is required: a later missing
|
|
or stale PSI signal holds with `health-unknown`. That is the case sage-2196 asked for.
|
|
- Tradeoff: I latch on the first sample that carries PSI, not only on the very first sample. So PSI
|
|
that appears late and then disappears also holds. This is stricter than "present at the first
|
|
sample", and it never skips a signal the host has shown.
|
|
4. **Memory under a cgroup.** Available memory = min(MemAvailable, limit - memory.current) at each
|
|
limited level. memory.current counts page cache, so this errs toward holding.
|
|
- Follow-up, not in this row: subtract reclaimable cache using memory.stat `inactive_file`, if S2's
|
|
live samples show holds that the host's MemAvailable doesn't support.
|
|
|
|
Rulings on two readings from the build (sage-2202, re morgana-0057):
|
|
|
|
5. **PSI latch.** PSI becomes required on the first sample that carries it. This is stricter than
|
|
"present at the first sample" and fails closed: a PSI file that disappears after it was seen is a
|
|
missing signal, so health-unknown holds.
|
|
6. **PSI count across resources.** Design line 152 says "CPU some ≥20, memory full ≥2, or I/O full
|
|
≥10, for 3 samples". The window applies to the whole disjunction, so any PSI condition on 3
|
|
consecutive samples holds. If Filbert reads it per resource at review, the review finding decides,
|
|
and the fix is a single counter.
|
|
|
|
Other readings of the D4 table, made in the build:
|
|
- **Memory floors.** "Whichever demands more free memory" means the larger floor applies. Hold below
|
|
max(15% x limit, 1 GiB); release above max(25% x limit, 2 GiB).
|
|
- **Edges as written.** Hold at ≥ for load and PSI, < for memory. Release at ≤ 0.55 for load, < for
|
|
PSI, > for memory. Critical below 5% of the limit and below 1 GiB of disk.
|
|
- **Sample counts.** "For 3 samples" means 3 consecutive samples. The count resets after a gap of more
|
|
than 6 s (the stale limit) between samples. The PSI row counts as one row (ruling 6), and the
|
|
reasons name the resources over on the latest sample.
|
|
- **Memory holds on one sample.** The table's memory row has no sample count.
|
|
- **Release** needs every applicable release condition on every sample for 30 s, by the samples' own
|
|
times (`>= 30000` ms after the first qualifying sample). A dead-band sample or any hold restarts the
|
|
window, and so does a gap of more than 6 s.
|
|
- **Start held.** `initialState()` is held, because nothing has been seen yet. The first healthy
|
|
sample starts the 30 s window.
|
|
- **Missing or stale.** Required signals are load, memory, oom and disk, plus PSI once seen. A signal
|
|
holds at once with `health-unknown` if it is:
|
|
- missing;
|
|
- non-numeric, negative or zero where a divisor;
|
|
- more than 6 s older than its sample (6 s exactly is fresh);
|
|
- stamped later than its sample.
|
|
Disk and OOM are required because the critical row can't be evaluated without them.
|
|
- **Host-only OOM.** `/proc/vmstat` `oom_kill` gives the OOM counter when there's no cgroup
|
|
memory.events. The parser returns null for a kernel without it, and the sampler then has no OOM
|
|
signal (health-unknown).
|
|
- **Critical** latches. It ends only when the caller passes `reconciled` with no critical condition in
|
|
the sample, then goes through held and the normal 30 s release. The evaluator never sets
|
|
`reconciled` and has no kill-switch path. Critical reasons accumulate while latched.
|
|
- **OOM counter.** The first OOM sample is the baseline. A falling counter (a recreated cgroup) resets
|
|
the baseline with a note, not an event.
|
|
- **Clock going backwards.** A sample earlier than the last one holds with `health-unknown` (unknown
|
|
`clock`, note `clock-backwards`).
|
|
- **Swap activity** is parsed (`pswpin`, `pswpout`) and has no rule. D4 lists it as a signal with no
|
|
threshold.
|
|
- **Settings.** `resolveRules` refuses:
|
|
- unknown keys and non-numbers;
|
|
- a release at or past its hold (load, each PSI resource);
|
|
- memory release floors at or below the hold floors;
|
|
- fractions outside (0, 1);
|
|
- PSI values outside 0 to 100;
|
|
- sample counts that aren't integers ≥ 1;
|
|
- a negative release window;
|
|
- a stale limit at or below the sample interval;
|
|
- a critical memory fraction at or above the hold fraction;
|
|
- a non-positive disk floor.
|
|
- **Large byte values.** Parsed with `Number`. A memory.max above 2^53 loses precision, but it only
|
|
takes part in a min against MemTotal.
|
|
|
|
## Round 2 (Filbert r1 APPROVE c30188 notes, sage-2212)
|
|
|
|
- N1: `merge` checked unknown keys with `key in base`, so inherited names (`toString`, `constructor`,
|
|
`__proto__`) passed the check and were dropped instead of refused. It now uses
|
|
`Object.hasOwn(base, key)`, and the refusal test has those three cases.
|
|
- N2: the critical-memory edge was tested only on a 32 GiB limit, where 5% is not an integer, so a
|
|
`<` to `<=` mutant survived. The test adds a 20 GiB limit, where 5% is exactly 1 GiB, and M8 is that
|
|
mutant. I checked the other memory edges for the same gap:
|
|
- The hold floor is non-integer on 32 GiB but is tested at exactly 1 GiB on a 4 GiB limit.
|
|
- The release floors are 8 GiB on 32 GiB and 2 GiB on 4 GiB, both integers.
|
|
- The disk edge is 1 GiB.
|
|
|
|
None has the gap.
|
|
|
|
## Files
|
|
|
|
- `packages/supervisor/package.json`, `README.md`
|
|
- `src/errors.mjs`: `SupervisorError` with `kind` parse, rules or sample.
|
|
- `src/pressure.mjs`: the nine parsers, `effectiveCpus` and `effectiveMemory`.
|
|
- `src/rules.mjs`: `DEFAULT_RULES`, `resolveRules`, `initialState`, `evaluate`.
|
|
- `src/index.mjs`
|
|
- `tests/pressure.test.mjs`: parsers on kernel-shaped fixture text, with malformed, truncated and
|
|
repeated-key cases. Effective CPU cases cover the host-only fallback, cpuset, fractional and nested
|
|
`cpu.max`, and `max`. Effective memory cases cover the host-only fallback, `max`, nested levels,
|
|
usage over the limit, a limit above MemTotal and a missing memory.current.
|
|
- `tests/rules.test.mjs`: the D4 table on a fake clock.
|
|
- Each hold and release threshold on both sides of its edge.
|
|
- Three-sample counting and the gap reset.
|
|
- The 30 s release at 29 999 and 30 000 ms, and the dead-band restart.
|
|
- Missing, stale, future and malformed signals.
|
|
- PSI never present, and PSI lost after it was present.
|
|
- Critical OOM, memory and disk at their edges, the reconciliation latch and the counter reset.
|
|
- A backwards clock, input state left unchanged, and settings refusals and overrides.
|
|
- `tests/purity.test.mjs`: a static check that src imports only its own files and uses no
|
|
fs, child_process, net, os, process, dynamic import, Date or timers.
|
|
|
|
No test reads /proc, /sys or a cgroup, starts a build, opens a network socket or generates load. No new
|
|
dependency.
|
|
|
|
## Mutants (each must fail its named test; the brief requires M1 and M2)
|
|
|
|
| Mutant | Edit | Named test |
|
|
|---|---|---|
|
|
| M1 hold edge | `ratio >= rules.load.hold` becomes `>` | load holds at 0.80 for 3 samples and not at 0.7999 |
|
|
| M2 early release | `now - next.releaseSince >= rules.releaseMs` becomes `>= 0` | the first state is held, and healthy samples open it after exactly 30 s |
|
|
| M3 memory floor | the hold floor `Math.max` becomes `Math.min` | memory holds below the larger floor, on one sample |
|
|
| M4 sample count | `next.loadCount >= rules.load.samples` becomes `>= 1` | load holds at 0.80 for 3 samples and not at 0.7999 |
|
|
| M5 stale | the `now - signal.at > rules.staleMs` check is removed | a signal is stale at more than 6 s old and fresh at exactly 6 s |
|
|
| M6 effective CPU | the quota `Math.min` becomes `Math.max` | effective CPUs take the smallest of host, cpuset and every quota level |
|
|
| M7 critical latch | the `!reconciled` critical branch is removed | critical stays until reconciliation, then still waits the 30 s release |
|
|
| M8 critical memory edge (r2) | `available < rules.critical.memoryFraction * limit` becomes `<=` | memory under 5% and disk under 1 GiB are critical, at their edges |
|
|
|
|
## Gate (Sage's slot, on CLEAR only)
|
|
|
|
- The three test files, each whole, then `node --test tests/` in packages/supervisor as the package
|
|
suite.
|
|
- Then M1 to M8, each on its named test, restored after each.
|
|
- umask 022, TMPDIR from `mktemp -d /mnt/storage/scratch/tmp/gXXXXXX`.
|
|
- The kit is /mnt/storage/scratch/morgana-r257 (slot257.sh, mutate.py). It refuses unless HEAD is the
|
|
candidate and the tree is clean, and stops after two unexpected failures (mos-0823).
|
|
- Static checks before the slot: `node --check` on every file, run from `/`. No test has run yet.
|
|
|
|
## Slot result (sage-4159 CLEAR, 2026-10-24 17:00:27Z to 17:00:30Z, candidate 1261fa2d8)
|
|
|
|
Run as `busy.sh && timeout -k 10 1500 slot257.sh` in its own session (setsid). The busy check read
|
|
CLEAR. The worktree was checked out at 1261fa2d8 with a clean tree. Filbert's approved code is
|
|
unchanged. The summary is in `gate-run.log` and the per-run node output in `mutants-run.log`.
|
|
|
|
| Run | tests | pass | fail | skip | cancelled | todo |
|
|
|---|---|---|---|---|---|---|
|
|
| tests/pressure.test.mjs whole | 15 | 15 | 0 | 0 | 0 | 0 |
|
|
| tests/rules.test.mjs whole | 29 | 29 | 0 | 0 | 0 | 0 |
|
|
| tests/purity.test.mjs whole | 1 | 1 | 0 | 0 | 0 | 0 |
|
|
| package suite `node --test tests/` | 45 | 45 | 0 | 0 | 0 | 0 |
|
|
|
|
Mutants: 8 of 8 killed. Each one ran only its named test (1 test, rc=1), and each failure was an
|
|
AssertionError in that test, not a crash: M1 hold edge, M2 early release, M3 memory floor, M4 sample
|
|
count, M5 stale, M6 effective CPU, M7 critical latch, M8 critical memory edge.
|
|
|
|
Restore check: `git checkout -- packages/supervisor` ran after each mutant, and the script found the
|
|
tree clean every time. At the end HEAD was 1261fa2d8 and `git status --porcelain` was empty. After the
|
|
run no node, python or bash process had its cwd under the kit or the worktree, and no g257 TMPDIR was
|
|
left behind. The run reached no scripts/test-*.sh.
|