670 lines
41 KiB
Markdown
670 lines
41 KiB
Markdown
# Row 270 build notes (Gizmoduck)
|
||
|
||
264c, T1 item 7: the compaction count, exact for Pi. Gitea #1775. Brief:
|
||
docs/plans/2026-10-21_t1-sessions.md section 4 (D6), with RULINGS Q6 (Pi exact, Claude `unknown` with
|
||
the Vikunja 466 reason). Reviewer Duckworth, gate owner Sage. Branch gizmoduck/row270 from
|
||
gizmoduck/row269 2d0d407ba (sage-2797, Q220). Row 269 lands first, and this branch rebases with it.
|
||
|
||
## Contract for 264e (pinned for Huey, row 272)
|
||
|
||
The surface reads the count from `session.list`. Each row gains one key, `compactions`:
|
||
|
||
```json
|
||
{ "id": "...", "at": "...", "name": "...", "predecessor": null, "default": true,
|
||
"compactions": { "count": 3, "known": true, "reason": null, "at": "2026-10-22T16:40:00.000Z" } }
|
||
```
|
||
|
||
- `known: true`: `count` is the exact number of completed compactions over the session's lifetime,
|
||
an integer of 0 or more. `reason` is null.
|
||
- `known: false`: `count` is a lower bound (an integer) or null when there is no bound. Show
|
||
`unknown` with the reason, never a bare 0. `reason` is one of:
|
||
- `harness-unsupported`: Claude Code, until Vikunja 466 pins its format.
|
||
- `not-collected`: no collection has run for this session yet.
|
||
- `store-unreadable`: a store file couldn't be read, or isn't a Pi session.
|
||
- `store-incomplete`: a stream file got shorter, was replaced, or has unreadable lines, so
|
||
entries may be missing.
|
||
- `at`: ISO time of the last collection, or null if none has run.
|
||
- A clear successor (264d) starts at `known: true, count: 0` once its first collection runs. The
|
||
retired session keeps its row and its count.
|
||
- On a base without 264c, the key is absent. The surface shows the count as unavailable, which is
|
||
what Huey's plan does (Sage Q1).
|
||
|
||
No new verb and no new input key. Access is `session.list`'s: the thread owner, or the tester.
|
||
|
||
## Pi source contract (pinned offline at 0.85.1)
|
||
|
||
`@earendil-works/pi-coding-agent` 0.85.1 (package.json pin), read in node_modules:
|
||
|
||
- One line per completed compaction: `{"type":"compaction","id":...,"parentId":...,"timestamp":...,
|
||
"summary":...,"firstKeptEntryId":...,"tokensBefore":...}` from `appendCompaction`
|
||
(dist/core/session-manager.js:818-833). The two callers write it after the summary exists
|
||
(dist/core/agent-session.js:1539, :1826). A start or an aborted compaction writes no entry.
|
||
- What doesn't count: a `message` entry with role `compactionSummary` (a rendered summary,
|
||
docs/session-format.md:154), any message inside a compaction's `retainedTail`, and
|
||
`branch_summary` entries.
|
||
- Identity: a compaction is its (entry `id`, entry `timestamp`) across the whole store, not per
|
||
stream. Pi's ids are unique only within a file (`generateId(this.byId)`, session-manager.js:21-29),
|
||
and `createBranchedSession` (:1093-1140) and `forkFrom` (:1264-1300) copy entries into a new
|
||
`*.jsonl` in the same directory with their id and timestamp unchanged. A (stream, id) key would
|
||
count a compaction made before a branch or fork once per copy. Reported to Sage in
|
||
gizmoduck-0262; accepted in sage-2814. The ledger line records the stream it was first seen in.
|
||
- Residual (sage-2814): two distinct compactions that share an id and a timestamp count once. That
|
||
needs an 8-hex collision between two files at the same millisecond. It is the one undercount
|
||
this key allows, and it never overcounts.
|
||
- Partial trailing line: wait. Only lines that end with a newline are read, as
|
||
packages/conversation/src/pi.mjs:35-38 does.
|
||
|
||
## Storage: A, a host ledger (sage-2806)
|
||
|
||
Sage ruled A over B. A keeps the count in `<store dir>/compactions.jsonl`, a file beside the
|
||
mounted `session/` and `claude-projects/`, in the store directory no bind mounts
|
||
(sandbox.mjs:122-127). B would have been a bus table. The tradeoff: A needs no schema step, so 271's
|
||
clear journal stays the only new schema step in the 264 rows, and the count lives with the native
|
||
store it is collected from. The cost is that the bus reads a file on each `session.list`, through a
|
||
reader runtime.mjs injects, because broker.mjs is a RUNNER_FILES file and can't import seat code.
|
||
|
||
### Ledger
|
||
|
||
`packages/seat/src/compactions.mjs`. One JSON object per line, `v: 1`:
|
||
|
||
- `{kind: "compaction", entry, completedAt, stream, at}`: once per (entry, completedAt) key.
|
||
- `{kind: "cursor", stream, bytes, tail, at}`: the stream was read through `bytes`. `tail` is the
|
||
sha256 of the up to 4096 bytes ending there. Written only when it moved.
|
||
- `{kind: "collected", at, known, reason, harness, version}`: one per collection.
|
||
|
||
Conditions (sage-2806):
|
||
|
||
1. Append-only, the launcher its only writer, a key never written twice, nothing rewrites or
|
||
truncates. The key is the one above, not (stream, id).
|
||
2. Mode 0600, opened `O_WRONLY|O_APPEND|O_CREAT|O_NOFOLLOW`. **Lock used: the session's workspace
|
||
lock** (`lockSessionWorkspace`, taken in launcher.mjs `start()` and released in the run's exit
|
||
`finally` or on a start failure). Collection runs at run start, after `check()` gives the
|
||
harness and before the spawn, and at run end, before the lock is released. Each collection is
|
||
one `writeSync` loop on one O_APPEND descriptor. A collection error is logged and recorded as a
|
||
`compactions` launch-log event and never stops or fails the run.
|
||
3. `compactionCount(dir)`:
|
||
|
||
| Ledger | Answer |
|
||
| --- | --- |
|
||
| missing | `not-collected`, count null |
|
||
| unreadable, not a regular file, a symlink | `store-unreadable`, count null |
|
||
| any bad line, a torn last line included | `store-incomplete`, count a lower bound or null |
|
||
| any collection under a harness other than pi | `harness-unsupported`, lower bound or null |
|
||
| any collection that found a stream shrunk, changed under its cursor or gone, or a bad stream line | `store-incomplete`, for good |
|
||
| the last collection hit an unreadable stream or a non-session first line | `store-unreadable` until a clean collection |
|
||
| otherwise | `known: true`, the exact count |
|
||
|
||
Never count 0 when unknown. After a torn line, the next write starts on a fresh line, so the torn
|
||
line stays one bad line and the new lines stay whole.
|
||
4. `packages/seat/tests/compactions.test.mjs` "the ledger is under no sandbox mount": for a legacy
|
||
and a non-legacy store, no `binds()` source contains the ledger path.
|
||
5. This section.
|
||
|
||
### Wiring
|
||
|
||
- launcher.mjs: `compactions(store, model.harness, ...)` at start (after `newRun`) and in the exit
|
||
`finally` before `lock?.release()`. Thread sessions only.
|
||
- broker.mjs: a `compactions` constructor option, TypeError-checked like `access`. `session.list`
|
||
selects `op` to tell a legacy session (`legacy-3l`) and adds `compactions` to each row; `op` is
|
||
not returned. No reader gives `not-collected`; a reader that throws gives `store-unreadable`.
|
||
No import added.
|
||
- runtime.mjs: passes `({business, thread, session, legacy}) => compactionCount(storeDir(...))`.
|
||
- packages/bus/README.md: the `session.list` row.
|
||
|
||
Huey's 272 broker hunk (reply citation predicate, near broker.mjs:5167 on huey/row272) doesn't
|
||
touch `session.list`.
|
||
|
||
## Tests
|
||
|
||
New files only, so no pre-existing test changes:
|
||
|
||
- packages/seat/tests/compactions.test.mjs: zero, a failed start and non-compaction entries; one and
|
||
three; a compaction copied into a branched stream counts once (sage-2814); one copy in two
|
||
streams in one collection; the same id at another time counts twice; resume with cursors and
|
||
keys written once; ledger mode 0600; a partial trailing stream line waits; a missing, a directory
|
||
and a symlinked ledger; a torn and a bad ledger line; shrink, rewrite and remove sticky; a
|
||
symlinked stream and a non-header first line; Claude harness-unsupported and sticky; `storeDir`
|
||
against `sessionStore`; the mount test.
|
||
- packages/bus/tests/session-compactions.test.mjs: the key per row with the store identity
|
||
asked for, legacy true for `legacy-3l`, no reader and a throwing reader, the TypeError.
|
||
|
||
Run limits (mos-1275): each file whole with `timeout -k 10 120`, TMPDIR from
|
||
`mktemp -d /mnt/storage/scratch/tmp/gXXX`. Then the touched files' suites: seat session.test,
|
||
sandbox.test; cli thread-launch, session-launch; bus sessions, project-runtime, tester-writes.
|
||
|
||
## Mutants
|
||
|
||
Each must turn a new test red:
|
||
|
||
- M1 key per stream at write: `keyOf(f.entry, f.completedAt)` → `keyOf(f.stream + f.entry, ...)` in
|
||
`collectCompactions`. The reader still dedupes, so the count holds; the kill is the
|
||
branched-copy test's ledger lines (`[["c1","a.jsonl"],["c2","b.jsonl"]]`) and the two-stream
|
||
test's one line.
|
||
- M2 key without the timestamp: `keyOf(entry, null)` everywhere. Kill: same-id-another-time test.
|
||
- M3 count `compactionSummary` messages: count `e.message?.role === "compactionSummary"` too. Kill:
|
||
the zero test.
|
||
- M4 ignore the cursor tail: `digest(tail) === c.tail` → `true`. Kill: rewrite subtest.
|
||
- M5 torn line skipped: `bad = torn = true` → `torn = true`. Kill: torn-line test.
|
||
- M6 no newline after a torn line: drop the `ledger.torn ? "\n"` prefix. Kill: torn-line test's bad
|
||
lines.
|
||
- M7 incomplete not sticky: drop `if (ledger.incomplete)`. Kill: the "later clean collection"
|
||
assertion.
|
||
- M8 harness not sticky: drop `if (ledger.unsupported)`. Kill: Claude then Pi test.
|
||
- M9 0 when unknown: `count: count || null` → `count`. Kill: torn-line test's null.
|
||
- M10 removed stream ignored: drop the missing-cursor loop. Kill: remove subtest.
|
||
- M11 broker legacy flag: `op === 'legacy-3l'` → `false`. Kill: bus legacy test.
|
||
- M12 broker default 0: no reader → `{count: 0, known: true, ...}`. Kill: bus no-reader test.
|
||
|
||
## Round 1 for review (sage-3024)
|
||
|
||
270's code reached the 271 branch in 09b20784f and 48c34539c and has no verdict and no mutant run.
|
||
This round submits it on its own: the manifest is the paths 270's commits change, hashed at
|
||
7d199f033 (the 271 r3 tip). queue-271/BUILD.md, which 8365e4e5b also touched, stays with 271.
|
||
broker.mjs, runtime.mjs and launcher.mjs carry 271's changes too, so their blobs match the 271 round.
|
||
|
||
Mutants: `mutants.py` beside this file, anchors at 7d199f033, each matching once (`check`, python
|
||
only). Each kill below is read from the test, not run yet.
|
||
|
||
| Mutant | Edit | Kill: first failing assertion |
|
||
| --- | --- | --- |
|
||
| M1 | collect key `keyOf(f.stream + f.entry, ...)` | seat compactions.test.mjs:64, ledger lines gain `[c1,b.jsonl]` (counts hold, readLedger dedupes) |
|
||
| M2 | `keyOf` drops the timestamp | :81, count 1 not 2 |
|
||
| M3 | count `compactionSummary` messages | :40, m2 counts 1 |
|
||
| M4 | tail digest check `true` | :170 in `rewrite`, known true |
|
||
| M5 | `bad = torn = true` to `torn = true` | :131, the torn line is popped unparsed, count reads known 0 |
|
||
| M6 | no newline after a torn line | :133, c1 joins the torn line, count null not 1 |
|
||
| M7 | drop `if (ledger.incomplete)` | :174 in `shrink` (`remove` stays incomplete by its missing cursor) |
|
||
| M8 | drop `if (ledger.unsupported)` | :201, Pi run reads known |
|
||
| M9 | `count \|\| null` to `count` | :131, count 0 not null |
|
||
| M10 | drop the missing-cursor loop | :170 in `remove` |
|
||
| M11 | broker `op === 'legacy-3l'` to `false` | bus session-compactions.test.mjs:76 |
|
||
| M12 | broker no reader gives `{count: 0, known: true}` | bus session-compactions.test.mjs:81 |
|
||
|
||
Run plan, on a CLEAR only: each mutant runs its test file whole under `timeout -k 10 120` (M1-M10
|
||
seat compactions.test.mjs, M11-M12 bus session-compactions.test.mjs), TMPDIR from
|
||
`mktemp -d /mnt/storage/scratch/tmp/gXXX`, restore with `git checkout` and a clean-tree check after
|
||
each. A kill is an AssertionError at the named line. M11 and M12 edit broker.mjs, which 271 also
|
||
changed, so after their restore bus session-clear.test.mjs and sessions.test.mjs run whole, each
|
||
under `timeout -k 10 300`. compactions.mjs has no 271 change.
|
||
|
||
Not covered: launcher.mjs's two `compactions(store, ...)` calls (run start, run end) and runtime.mjs's
|
||
reader closure have no test in either file, and no test elsewhere names compactions. A mutant
|
||
dropping either launcher call or making the closure answer a constant would survive. This is the
|
||
T3 Pi half as built; the Tests section above never claimed the wiring. Adding a cli and a runtime
|
||
test is a scope call for Sage.
|
||
|
||
Prior coverage: none. 271's M1-M14 edit no 270 line, and no past verdict names 270.
|
||
|
||
## Round 2 (duckworth-0223 B1-B3, sage-3030, sage-3031, mos-1352)
|
||
|
||
New commits on 8a5eb3c8b (round 1, on 7d199f033). Nothing here has run: no test, no mutant, no node.
|
||
Each kill below is read from the test.
|
||
|
||
### B1: the read budget, a human-only setting
|
||
|
||
`scanPi` reads a stream a CHUNK (1 MiB) at a time from its cursor, carries a line split across
|
||
reads, and stops when the collection's budget runs out, over all streams together. The whole-range
|
||
`Buffer.alloc(to - from)` that compactions.mjs:57 and :101 had is gone from the stream read;
|
||
`readRange` now serves only the tail digest reads, at most 4096 bytes. A collection the budget stops
|
||
short records `known: false`, reason `not-collected`, and cursors at the last whole line it read. The
|
||
reason isn't sticky: `compactionCount` takes `last.reason`, so a later collection that reads to the
|
||
end reads known again. When a collection has more than one problem the order is store-incomplete,
|
||
store-unreadable, then not-collected.
|
||
|
||
The setting (mos-1352, DECISIONS Q222). It's `compactionReadBytes`, a top-level key of the instance's
|
||
system config, the file `MOSAIC_CONFIG` names (`~/.config/mosaic-NAME/config.json`). Under invariant 2
|
||
a human writes that file and nothing auto-writes it. The launcher reads it once, at host start,
|
||
through `loadSystem`; nothing reads it from the session mount, the workspace or an environment
|
||
variable an agent's process sets. `readBudget` (compactions.mjs) takes a whole number of bytes from
|
||
1 MiB to 1 GiB and gives the default for anything else, missing included. The launcher logs a bad
|
||
value once at start, naming the default. A change takes effect at the next host start.
|
||
|
||
The schema change, for review. `scripts/mosaic-config.mjs` refused any top-level key outside
|
||
`configVersion, environment, dataRoot, execution, console` (`unsupported configuration key`). It now
|
||
also takes `compactionReadBytes`, and passes it on as written, a bad value included, because the
|
||
fallback is the launcher's and a bad value must not refuse the config. The output carries the key
|
||
only when the file has it, so a config without it prints what it always did (test-config.sh's
|
||
"config without console" case still pins that). The old list literal stays as it was and the key is
|
||
added by `.concat`, because packages/cli/tests/stack-release-fixture.mjs:300 edits that literal to
|
||
make an older loader.
|
||
|
||
A consequence for Duckworth to rule on. A loader from before this round refuses a config that sets
|
||
the key. scripts/stack-release.sh `config_checks` (:1079-1097) runs the target copy's loader only when
|
||
the config has a `console` key. So a human who sets `compactionReadBytes` and then rolls back to a
|
||
copy from before 270 passes that check, and the bus fails at its next config read. Nobody has to set
|
||
the key, since the default needs none. I've left stack-release.sh alone: it's outside this row's
|
||
manifest. The bounded follow-up is to make `config_checks` run the target loader whenever the config
|
||
has any key the target might refuse, not just `console`.
|
||
|
||
The default, 64 MiB, from sizes measured 2026-10-22 by stat and line length only (no content printed,
|
||
nothing under the live stack read):
|
||
|
||
| Store | Files | p50 | p90 | p99 | max |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| Pi sessions, `~/.pi/agent/sessions` | 224 | 298 KB | 1.1 MB | 4.7 MB | 11.6 MB |
|
||
| Claude Code transcripts, `~/.claude/projects` (not read by the scanner) | 549 | 607 KB | 6.6 MB | 120 MB | 603 MB |
|
||
|
||
The longest Pi line is 0.84 MB, and no Pi line is over 1 MiB. 64 MiB is more than five times the
|
||
largest Pi stream, so a session's first collection reads it whole, and later ones read only what
|
||
was appended. Memory per collection is one 1 MiB chunk plus the carry of one unfinished line, and
|
||
the budget bounds both. The Claude numbers show how far a long agent session can grow. A Pi session
|
||
that size would take several collections to read through, each one reading unknown until the last.
|
||
I haven't measured how long a 64 MiB collection takes; the sparse test reads 64 MiB of a hole, and its
|
||
wall time in the slot run will be the first number.
|
||
|
||
Known limits:
|
||
- A single line longer than the budget is never read past. The count stays not-collected until a
|
||
human raises the setting. The longest Pi line measured is 0.84 MB.
|
||
- `readLedger` still reads the ledger whole; B1 named the stream reads. Ledger lines are 120 bytes
|
||
(collected), 226 (cursor) and 199 (compaction), so a run with two collections, one changed stream
|
||
and one compaction adds about 0.9 KB. One session reaches 64 MiB of ledger after about 75,000 runs.
|
||
|
||
### B3: Claude history from before the ledger
|
||
|
||
When a Pi collection finds no ledger and the store's `claude-projects/` (`store.projects`) isn't
|
||
empty, it skips the scan and writes one collected record: `known: false`, reason
|
||
`harness-unsupported`, harness `pi`. `readLedger` now sets `unsupported` for a record whose harness
|
||
isn't `pi` or whose reason is `harness-unsupported`, so the count stays unknown for good, as after a
|
||
Claude run. A `claude-projects/` that can't be read throws, so nothing is recorded and the next
|
||
collection checks again. An empty one is no history. Once a ledger exists the check doesn't run: a
|
||
later Claude run collects for itself.
|
||
|
||
A Pi run mounts `claude-projects/` read-write (sandbox.mjs:125). The run-start collection comes before
|
||
the agent's first turn, though, so an agent can't put a file there before a fresh session's first
|
||
collection. The exception is a run whose start collection failed. Even then, the agent can only
|
||
make the count unknown, never wrong.
|
||
|
||
The Vikunja 466 reason. For a Claude run, seat compactions.test.mjs:199 asserts `harness-unsupported`,
|
||
and :201 asserts that it stays after a Pi run. For history from before the ledger,
|
||
compactions-budget.test.mjs:88 and :89 assert it (sage-3030 item 2).
|
||
|
||
### B2: the run wiring, the reader and a clear
|
||
|
||
The new tests live in files 271 doesn't touch:
|
||
- `packages/seat/tests/compactions-budget.test.mjs`, B1 and B3.
|
||
- `packages/cli/tests/compactions-launch.test.mjs`, with its stub adapter
|
||
`packages/cli/tests/fixtures/compaction-adapter.mjs`, real runners in bwrap.
|
||
|
||
compactions-launch tests:
|
||
1. A run's start and end collections, across a resume and two host restarts. session.list reads the
|
||
ledger through the runtime's reader (:166-198).
|
||
2. A collection that fails, with the ledger's path made a directory. Each failure is logged as
|
||
`run X: compactions: ...` and recorded as `store-unreadable`. The run answers, and it ends with the
|
||
same reason, exit code and signal as the next run, whose collections count (:200-224).
|
||
3. The setting. At 1 MiB, a 2 MiB hole is not-collected at both collections. With `"x"`, the host
|
||
logs the fallback once, and the default reads the hole through (:226-261).
|
||
4. A clear. The retired session keeps its count, 1, and the successor reads `{count: 0, known: true}`
|
||
after its first collection. A legacy session (op legacy-3l, seeded with the SQL from
|
||
bus session-compactions.test.mjs:72-74) reads the thread store's ledger through the runtime's reader
|
||
after a restart (:263-308).
|
||
|
||
### Mutants M13-M21
|
||
|
||
`mutants.py`, with all 21 anchors matching once at this round's tip (`check`, python only).
|
||
|
||
| Mutant | Edit | Kill: first failing assertion |
|
||
| --- | --- | --- |
|
||
| M13 | drop the run-start `compactions(...)` call (launcher.mjs:501) | cli compactions-launch.test.mjs:182, run 1 logs only `[D,2,true,null]` |
|
||
| M14 | drop the run-end call (launcher.mjs:642) | :182, run 1 logs only `[D,1,true,null]` |
|
||
| M15 | the runtime's reader returns `null` (runtime.mjs:268) | :197, compactions null; :286 and :306 fail the same way |
|
||
| M16 | no budget, `stop = size` | seat compactions-budget.test.mjs:48, the hole is read whole: known true |
|
||
| M17 | `readBudget` returns `value ?? READ_BYTES` | :37, `"8388608"` comes back as the string |
|
||
| M18 | the launcher ignores the setting, `budget = READ_BYTES` | cli :248, the 2 MiB hole reads known twice |
|
||
| M19 | no carry, `take(view.subarray(i, nl))` | seat :79, c1's halves fail to parse: store-incomplete |
|
||
| M20 | no Claude history check | :88, c1 counts known |
|
||
| M21 | `harness-unsupported` not sticky on a `pi` record | :89, the second collection reads known |
|
||
|
||
The files they touch, sha256 at this round's tip, to check each restore against (sage-3030 item 3):
|
||
|
||
```
|
||
45a3744598cfb8aa1cfcffd8a0ee2dd2bb631061efd192d8461f1901141993ee packages/seat/src/compactions.mjs
|
||
d7cdc8a17e337e00dfe2d1bf6249cc2c2e696b16c5b426b2fe616089e5559509 packages/bus/src/broker.mjs
|
||
5d36cc0fa19ab8e478ecf9a807e3afab8181cb650778e80b13efe785e6c44995 packages/cli/src/launcher.mjs
|
||
5ae423438a178b1c8eac3b8ce0e0c22f5bb7caebb9011ce7098a36d95b53f1ca packages/bus/src/runtime.mjs
|
||
```
|
||
|
||
broker.mjs and runtime.mjs are byte-identical to 7d199f033.
|
||
|
||
### Run plan, on a CLEAR only
|
||
|
||
Every run uses TMPDIR from `mktemp -d /mnt/storage/scratch/tmp/gXXX` and `timeout -k 10`. A first-run
|
||
red in a new test is placed and reported after the run (mos-1272). A red in an old test, a product
|
||
fix or a second red goes to Sage and Mos first.
|
||
|
||
1. Each mutant runs its kill file whole: M1-M10 seat compactions.test.mjs, M16, M17 and M19-M21 seat
|
||
compactions-budget.test.mjs (120 s each), M11-M12 bus session-compactions.test.mjs (120 s), and
|
||
M13-M15 and M18 cli compactions-launch.test.mjs (600 s). After each one: `git checkout` the file,
|
||
check its sha256 against the table, and confirm the tree is clean. A byte-identical restore needs
|
||
no other rerun.
|
||
2. The new test files, whole: seat compactions-budget.test.mjs and cli compactions-launch.test.mjs.
|
||
3. Every file that reads a path this round touches, whole:
|
||
- compactions.mjs: seat compactions.test.mjs, bus session-compactions.test.mjs.
|
||
- mosaic-config.mjs: scripts/test-config.sh, cli stack-instances.test.mjs, cli stack-release.test.mjs
|
||
(through stack-release-fixture.mjs), webui pinned-copy.test.mjs, and the business suite (it
|
||
writes configs through helpers.mjs:79).
|
||
- launcher.mjs: the cli files that import it, all in the cli suite.
|
||
- README.md: no test reads it.
|
||
4. The gate (sage-3031) runs on the combined 270+271 tip in one slot after Gander's 276 run5. It
|
||
covers bus, seat and cli whole, the shell suites, M1-M21 and 271's mutants. compactions.mjs is in
|
||
271's manifest, so 271's earlier gate evidence is stale. Webui pinned-copy and the business suite
|
||
join it because the loader changed.
|
||
|
||
### Known residuals (duckworth-0223, sage-3031)
|
||
|
||
- N1. A session.list read can briefly see a half-written line. The bus reads without the workspace
|
||
lock, so a read during an append sees a torn last line. That call alone reports store-incomplete,
|
||
and nothing is persisted. Accepted.
|
||
- N2 (README). A launcher killed mid-write leaves a torn ledger line, and the count stays unknown
|
||
(store-incomplete) for good, by design (sage-2806 condition 3). This is now a line in the bus
|
||
README's `session.list` row.
|
||
- N3. Pi 0.85.1 rewrites a session file only when it's empty (session-manager.js:625-632), on a
|
||
version migration (:675-677), or for a new branched file (:1173). None of these moves bytes under a
|
||
0.85.1 cursor, so the 4096-byte tail check is enough. A non-Pi writer that rewrites earlier bytes and
|
||
keeps the tail goes unseen. That residual is acceptable.
|
||
|
||
## Round 3 (duckworth-0226 B4, sage-3038)
|
||
|
||
Round 2 at 62a279cf3 got CHANGES on B4. B1-B3 were met, and Duckworth accepted the schema change.
|
||
|
||
### B4: a cap on the streams one collection opens
|
||
|
||
Before this round, one collection opened every `*.jsonl` in the session directory. It also checked
|
||
each ledger cursor with `names.includes`, which is O(cursors × names), and wrote a cursor line for
|
||
every stream, empty ones included. The agent can write its rw session directory (sandbox.mjs:124),
|
||
and nothing limits its file count. So 100,000 empty files meant about 22 MB of cursor lines at one
|
||
run end, then about 10^10 string compares on the launcher's loop at the next.
|
||
|
||
Now:
|
||
|
||
- scanPi (compactions.mjs:123-125) sorts the `*.jsonl` names and opens at most `streams` of them.
|
||
When there are more, the collection adds not-collected. The streams it didn't open get no cursor
|
||
line. A stream that already has a cursor keeps its old one.
|
||
- The missing-stream check uses a Set (compactions.mjs:121-122).
|
||
- The cap is a setting, per sage-3038 (the same reason as mos-1352 and Q222): top-level
|
||
`compactionStreams` in the instance system config. It follows compactionReadBytes. It is
|
||
human-only. `readStreams()` accepts a safe integer from 32 to 65536 and returns the default, 1024,
|
||
for anything else. The launcher reads it once and logs one `system config:` line for a bad value.
|
||
mosaic-config.mjs adds the key to the same `.concat` and passes it on verbatim. The fixture literal
|
||
at stack-release-fixture.mjs:300 still matches.
|
||
- Name order stays sorted, so a collection under a larger cap, or with fewer files, catches up.
|
||
Under the same cap and the same files, each collection opens the same streams and adds no cursor
|
||
lines for streams that haven't moved.
|
||
|
||
Default, from stat only (`find ~/.pi/agent/sessions -name '*.jsonl'`, counted per directory, no
|
||
content read). Each of these directories holds every session for one cwd, so a single session's
|
||
count is at most these:
|
||
|
||
| directories | p50 | p90 | p99 | max |
|
||
|---|---|---|---|---|
|
||
| 40 | 3 | 19 | 27 | 27 |
|
||
|
||
1024 is about 38 times the max. The cursor lines one collection writes are bounded by 1024 × 226 B,
|
||
about 231 KB. At the setting's 65536 maximum the bound is about 14.8 MB.
|
||
|
||
### N4: docs/TOOLS.md
|
||
|
||
docs/TOOLS.md has a paragraph on both keys, after the `config-console-key` paragraph. It gives the
|
||
range, the default, that only a human writes them, the fallback with its log line, not-collected,
|
||
and C1. No test reads TOOLS.md. stack-release.test.mjs:78 only names it inside a message string.
|
||
|
||
### C1 (duckworth-0226, sage-3038)
|
||
|
||
Any re-pin or rollback plan whose OLD copy is before row 270 removes both `compactionReadBytes` and
|
||
`compactionStreams` from config.json first. A pre-270 loader refuses either key. stack-release.sh
|
||
config_checks (:1079-1097) runs the target loader only when there is a `console` key, so the release
|
||
doesn't catch this. Sage is drafting the config_checks follow-up row.
|
||
|
||
### Tests
|
||
|
||
- seat compactions-budget.test.mjs:114-120: readStreams over good and bad values.
|
||
- :122-134: four streams under a cap of 3. Both collections are not-collected, and the cursors are
|
||
only a, b and c, with no new cursor lines on the second. A cap of 4 then reads count 4, known.
|
||
- :136-143: 1025 empty streams under the default cap. The result is not-collected with 1024 cursor
|
||
lines, the last one e1023.
|
||
- cli compactions-launch.test.mjs:264-298: compactionStreams 32 with 34 streams. pi.jsonl sorts
|
||
last, so its c0 isn't opened and both collections are `[D,null,false,not-collected]`. Then `"x"`
|
||
logs the fallback, and the next run's collections read `[D,1,true,null]`.
|
||
- scripts/test-config.sh: 2048 and `"x"` load and pass through, as for compactionReadBytes.
|
||
|
||
### Mutants
|
||
|
||
M10's anchor moved to the Set line. The mutant is the same, removing the missing-stream check. Every
|
||
anchor in mutants.py matches once (`check`). Round 3 shifted round 2's kill lines: seat budget
|
||
:37/:48/:79/:88/:89 are now :39/:50/:81/:90/:91, and cli launch :182/:197/:248 are now
|
||
:183/:198/:249. mutants.py K has the current lines. The launcher call sites are now
|
||
launcher.mjs:507 and :648.
|
||
|
||
| id | change | kill |
|
||
|---|---|---|
|
||
| M22 | no cap, `for (const name of visit)` | seat compactions-budget.test.mjs:127, all four are opened: count 4 known; :141 fails the same way |
|
||
| M23 | `readStreams` returns `value ?? STREAMS` | :118, `"2048"` comes back as the string |
|
||
| M24 | the launcher ignores the setting, `streams = STREAMS` | cli compactions-launch.test.mjs:285, all 34 open: `[D,1,true,null]` twice |
|
||
|
||
sha256 at this round's tip, to check each restore against. These replace round 2's table:
|
||
|
||
```
|
||
b5a078a096e447c4afe15812696d1f2f8c3f2b7665868c02e61a4e1edc8cd874 packages/seat/src/compactions.mjs
|
||
d7cdc8a17e337e00dfe2d1bf6249cc2c2e696b16c5b426b2fe616089e5559509 packages/bus/src/broker.mjs
|
||
21a7e910d81848aeb99036028d5b87354a2a515f3642779105c881b43a8b6e05 packages/cli/src/launcher.mjs
|
||
5ae423438a178b1c8eac3b8ce0e0c22f5bb7caebb9011ce7098a36d95b53f1ca packages/bus/src/runtime.mjs
|
||
```
|
||
|
||
### Run plan
|
||
|
||
This is round 2's plan with M1-M24. M22 and M23 run seat compactions-budget.test.mjs (120 s). M24
|
||
runs cli compactions-launch.test.mjs (600 s). Per sage-3038, the slot goes to Megavolt's 207, then
|
||
203, then this row's combined 270+271 gate after Duckworth approves r3.
|
||
|
||
### Residuals
|
||
|
||
- N5. readdirSync still lists the whole session directory, and the names are sorted before the
|
||
cap. Both are O(files) in memory and time on the launcher. The cap bounds opens, reads and ledger
|
||
lines, not the listing. Bounding the listing needs opendirSync with a stop. That would lose name
|
||
order, and name order is what lets a later collection catch up. Accepted for this row
|
||
(duckworth-0226).
|
||
- The ledger's cursor map still grows when the agent deletes and recreates streams, by at most the
|
||
cap per collection. Each deletion of a stream with a cursor is store-incomplete for good. This is
|
||
the known limit that the bus reads the ledger whole (round 2).
|
||
|
||
## Round 4: rebase onto row 203 (sage-3073)
|
||
|
||
Rows 270 and 271 are rebased onto origin/refactor 25c09f19f (sage-3078): row 203 (FED R2) at
|
||
a4da4e6df, plus three docs-only commits for the O2 design (BUILD-LOG.md, docs/SESSIONS.md and
|
||
docs/plans/2026-10-21_q195-writing-editor.md, none of them in either manifest). The new branch is
|
||
gizmoduck/row270-271-r4. gizmoduck/row271-r2 stays at b9da7697f. The old
|
||
merge-base was 0cbe2e9b3, and the rebase moved 28 commits.
|
||
|
||
`git range-diff 0cbe2e9b3..b9da7697f 25c09f19f..HEAD~1` shows 26 commits
|
||
unchanged. Round 2 and round 3 differ, and only in scripts/mosaic-config.mjs:
|
||
|
||
- The allowlist keeps 203's literal and adds both keys after it:
|
||
`["configVersion", "environment", "dataRoot", "execution", "console", "federation"].concat(["compactionReadBytes", "compactionStreams"])`.
|
||
203 changed packages/cli/tests/stack-release-fixture.mjs, and its LOADER_OLD replace is now at
|
||
:301 (rounds 2 and 3 say :300). It matches the literal with `"federation"`, which is still
|
||
intact before the `.concat`.
|
||
- The output has 203's `federation` line, then the two `Object.hasOwn` lines.
|
||
|
||
Rows 203, 270 and 271 also all touch scripts/test-config.sh, scripts/stack-release.sh and
|
||
packages/cli/tests/stack-release.test.mjs. Those merged without conflict, and the hunks are far
|
||
apart:
|
||
|
||
- test-config.sh: 203's federation cases start at :194. Mine are at :266-:281, after the
|
||
no-console case.
|
||
- stack-release.sh: 203 changed config_checks at :1078-:1097. Row 271's change is COUNTED at
|
||
:420, still the line 271's BUILD.md cites.
|
||
- stack-release.test.mjs: 203 changed :1468 and :1582. Row 271's LISTED is still at :861 and T46
|
||
at :913.
|
||
|
||
203 widens config_checks to run the target loader for a `federation` key as well as `console`.
|
||
It still does not run the loader for compactionReadBytes or compactionStreams, so C1 stands as
|
||
written.
|
||
|
||
`mutants.py check` passes: every anchor matches once. The restore hashes in round 3's table are
|
||
unchanged, because 203 touches none of those four files. No test pins a sha256 of mosaic-config.mjs,
|
||
test-config.sh, stack-release.sh or stack-release.test.mjs.
|
||
|
||
Nothing has run on this base yet. The combined gate is sage-3073 item 4, on Sage's CLEAR.
|
||
|
||
## Combined 270+271 gate (C2), written before the first run (sage-3073 item 4, sage-3086, mos-1351)
|
||
|
||
The gate runs on the commit that adds this section. Its code is ac44d0de9's: `git diff --name-only
|
||
ac44d0de9 HEAD` is this file only. The review keys stay manifest-270-r4 4f5e7041… and manifest-271-r4
|
||
3ab1f5c8…, Duckworth approves at revs 3379 and 3380. Worktree /mnt/storage/scratch/gizmo-270, with
|
||
node_modules linked to the canonical checkout. Base 25c09f19f. Script
|
||
/mnt/storage/scratch/gizmo-271-out/run-r4.sh, run with pipefail and tee, logs in
|
||
/mnt/storage/scratch/gizmo-271-out/r4/. TMPDIR comes from `mktemp -d /mnt/storage/scratch/tmp/gXXX`.
|
||
|
||
### Blob table
|
||
|
||
sha256 of every source file a mutant edits, at the gate commit. Each restore is `git checkout --` the
|
||
file, then this hash, then a clean tree.
|
||
|
||
```
|
||
b5a078a096e447c4afe15812696d1f2f8c3f2b7665868c02e61a4e1edc8cd874 packages/seat/src/compactions.mjs (270)
|
||
d7cdc8a17e337e00dfe2d1bf6249cc2c2e696b16c5b426b2fe616089e5559509 packages/bus/src/broker.mjs (270, 271)
|
||
21a7e910d81848aeb99036028d5b87354a2a515f3642779105c881b43a8b6e05 packages/cli/src/launcher.mjs (270, 271)
|
||
5ae423438a178b1c8eac3b8ce0e0c22f5bb7caebb9011ce7098a36d95b53f1ca packages/bus/src/runtime.mjs (270)
|
||
```
|
||
|
||
### Timeout table (mos-1275)
|
||
|
||
Every run is `timeout -k 10 LIMIT`. The longest records are 271's gate on e8af389e6 (BUILD.md queue-271
|
||
"Gate on e8af389e6") and 271's "Run limits" table.
|
||
|
||
| Run | Longest recorded | LIMIT |
|
||
| --- | --- | --- |
|
||
| stack-release.test alone | 575 s | 1800 s |
|
||
| each other touched test file alone, stack-instances.test, webui pinned-copy.test | 15 s | 300 s |
|
||
| business suite | none recorded | 900 s |
|
||
| bus suite | 24 s | 900 s |
|
||
| seat suite | 6 s | 600 s |
|
||
| cli suite | 591 s | 1800 s |
|
||
| each scripts/test-*.sh | 42 s (test-foundation; 39 s at e8af389e6) | 900 s |
|
||
| 270 mutant, seat or bus kill file | none recorded | 120 s |
|
||
| 270 mutant, cli compactions-launch.test | none recorded | 600 s |
|
||
| 271 mutant, each session-clear.test file | 57 s for both files (e8af389e6) | 300 s |
|
||
|
||
A shell-phase timeout goes to Mos unplaced. A timeout with no failed test is a hang and goes to Sage.
|
||
|
||
### GREEN and RED
|
||
|
||
- GREEN for a test run: rc 0, `fail 0`, and at least one pass for a single file. A skip is named in the
|
||
report.
|
||
- RED: any other rc, a failed test, or zero passes.
|
||
- A mutant is killed when its kill file is RED with a `not ok` on the test K names. 270's K is in
|
||
queue-270/mutants.py. 271's kills are the bus or cli session-clear.test failures in queue-271/BUILD.md.
|
||
A mutant whose kill file stays GREEN survived.
|
||
- A first-run red in a new test (seat compactions-budget.test, cli compactions-launch.test, 271's new
|
||
tests) is placed and reported after (mos-1272). A red in an old test, a product change, a looser
|
||
assertion or a second red on a row stops the run and goes to Sage before any change (sage-3086).
|
||
- Mutants run only on a GREEN base.
|
||
|
||
### Order
|
||
|
||
1. The 20 touched test files whole, then cli stack-instances.test, webui pinned-copy.test and the
|
||
business suite (the config loader's readers, round 2 run plan item 3).
|
||
2. bus, seat and cli suites whole.
|
||
3. The ten scripts/test-*.sh, test-config.sh among them, with skips named.
|
||
4. 270 M1-M24, each on its K file whole. Then 271 M1-M14, each on bus and cli session-clear.test whole.
|
||
|
||
### Run 1: a first-run red, placed (mos-1272)
|
||
|
||
Run 1 started 00:06:57Z on 0f5e5d62d. In phase 1, the 15 bus files and session-compactions passed.
|
||
Then cli compactions-launch.test.mjs, new in this row and on its first run, went RED: 4 of 5 pass. The
|
||
failure was in the clear test, "REQ-THREAD-2 T3 T4 a clear leaves the retired session's count...", at
|
||
:319 (now :324):
|
||
`[done?.state, done?.start]` was `[undefined, undefined]`, not `["retired", "started"]`, after the
|
||
30 s settle (31433 ms).
|
||
|
||
The cause is in the test, not the product. The test's host `ops` table had no entries for row 271's
|
||
five clear ops: sessionClears, sessionClearExited, sessionClearUncertain, sessionClearRetire and
|
||
sessionClearStart. The launcher's clear calls through host.op were refused as invalid-host-request,
|
||
so the successor never started. cli session-clear.test.mjs:96-100 wires the same five, and its clear
|
||
tests passed in this run (5 of 5).
|
||
|
||
The placement adds those five lines, copied from session-clear.test.mjs, after landRelease. No assertion
|
||
changed, and no product file changed. The file alone then passed 5 of 5
|
||
(/mnt/storage/scratch/gizmo-271-out/place-compactions-launch.log). The five lines move 270's
|
||
compactions-launch kill lines by +5 in queue-270/mutants.py K: :188, :203, :254 and :290. Round 2
|
||
and round 3 above cite the old lines.
|
||
|
||
Run 1 stopped at 00:07:59Z (the log's last write), after phase 1's cli session-clear.test, so that the gate runs on one
|
||
tree. No node process was left, and the tree was clean. Its log is
|
||
/mnt/storage/scratch/gizmo-271-out/r4-run1/. This is the row's first red. Run 2 is the whole gate
|
||
again on the commit that adds the placement. That commit changes the 270 manifest (compactions-launch.test.mjs,
|
||
mutants.py and this file), so Sage and Duckworth see it before landing.
|
||
|
||
### Run 2: a second red, two test defects (mos-1364, sage-3093)
|
||
|
||
Run 2 started 00:09:50Z on 5f697db99. Phase 1 passed except one file: the 15 bus files, session-compactions
|
||
4/4, compactions-launch 5/5, cli session-clear 5/5, stack-release 219/219, seat compactions 20/20,
|
||
stack-instances 24/24, webui pinned-copy 10/10 and business 231/231. seat compactions-budget.test.mjs
|
||
went RED: 8 of 10 pass. The file is new in round 2, and this was its first run. Rounds 2 and 3 were not
|
||
run. This is the row's second red, so I stopped the run (r4.out's last write 00:20:12Z) and changed
|
||
nothing until Mos ruled yes on both fixes (mos-1364, passed on in sage-3093). Phase 2 did not start. Its
|
||
log is /mnt/storage/scratch/gizmo-271-out/r4-run2/.
|
||
|
||
Both failures are defects in the test, not in compactions.mjs.
|
||
|
||
F1, :39 in "the budget is the setting's whole number of bytes...". `readBudget(8.5 * MiB)` returned
|
||
8912896, not the default 67108864. 8.5 MiB is 8912896 bytes: a whole number, in range. readBudget
|
||
(compactions.mjs:80-82, Number.isSafeInteger and the 1 MiB to 1 GiB bounds) is right to take it. The case
|
||
meant a value that is not a whole number of bytes. The fix changes only that input, to `MiB + 0.5`. The
|
||
other twelve bad values stay. M17's kill line is this :39, so M17 had no valid kill before the fix.
|
||
|
||
F2, :107 in "a claude-projects/ that can't be read...". The hook `t.after(() => chmodSync(projects,
|
||
0o700))` threw ENOENT. A test's after hooks run in the order they were registered. setup()'s rmSync of the
|
||
root (:20) was registered first, so the tree was gone when this hook ran. The test's assertions had passed.
|
||
The fix puts the 0o000 window in try/finally, which restores 0o700 before the rm even when an assertion
|
||
fails, and drops the hook and the inline chmod. It keeps the same assertions in the same order, and none
|
||
is removed. Net +2 lines, so in mutants.py K, M23 moves :118 to :120 and M22 :127 to :129. M22's note
|
||
cites the default-cap test's :141, now :143. M15's note cites the clear test's :323 and the legacy :343,
|
||
which are :328 and :348 since run 1. That change is to note text only. `mutants.py check` shows all 24
|
||
anchors matching once.
|
||
|
||
The file alone then passed 10 of 10 (/mnt/storage/scratch/gizmo-271-out/place-compactions-budget.log).
|
||
Run 3 is the whole combined gate again, on one tree, on the commit with F1 and F2. A third red on 270
|
||
stops it and goes to Sage and Mos before any change.
|
||
|
||
### Runs 3 and 4: two old-test races, not 270 or 271 (mos-1367, mos-1369)
|
||
|
||
Run 3 started 00:23:35Z on 53406f3e4. Phase 1 passed whole (business 231/231), bus 873/873 and seat
|
||
202/202. cli went RED, 904 of 905: case1-accept.test.mjs:368, the P10 inner hop probe took 1.08 s, under
|
||
its 1.25 s floor. The file is old and unchanged in 25c09f19f..53406f3e4. I stopped the run during phase 3
|
||
(r4.out's last write 00:46:24Z) and changed nothing. Mos ruled (mos-1367, passed on in sage-3097): A1 run
|
||
case1-accept alone, which passed 50/50 (00:50:01Z to 00:54:17Z, load 3.19 3.58 3.71); A2 run 4 on the
|
||
same tip. Vikunja 573 (owner gander) owns the P10 band fix. Log: /mnt/storage/scratch/gizmo-271-out/r4-run3/.
|
||
|
||
Run 4 started 00:54:25Z on 53406f3e4 (load 4.45 4.20 3.94). Phase 1 passed 23/23 files and bus 873/873.
|
||
seat went RED, 201 of 202: checkout.test.mjs:239 "a dirty clone whose fetch fails is kept, fetch-failed,
|
||
exactly as it was" failed :91 "the root is unchanged" at its first once(w) (:241). The file is old and
|
||
unchanged in 25c09f19f..53406f3e4, and it passed in run 3. Load was 8.19 at 01:06Z. I stopped the run
|
||
(r4.out's last write 01:06:31Z) and changed nothing. The cause is the fixture, not 270 or 271: its git
|
||
helper (checkout.test.mjs:35) runs with gitEnv() (checkout.mjs:80-90), which does not set
|
||
maintenance.auto=false, so its commit can leave a detached `git maintenance run --auto` writing in the
|
||
root that once() fingerprints. Mos watched objects/maintenance.lock vanish after commit returned in 1 of
|
||
30 fresh repos (mos-1369, passed on in sage-3106). Vikunja 574 (owner gander) owns the fixture fix; no test
|
||
changed in this slot. Log: /mnt/storage/scratch/gizmo-271-out/r4-run4/.
|
||
|
||
### Run 5: GREEN (mos-1369, sage-3106)
|
||
|
||
Run 5 waited until the one-minute load was under 4, then started 01:17:11Z on 53406f3e4, unchanged since
|
||
run 3 (loadavg 3.86 4.91 4.69, up 9 days 11:37; origin/refactor 062646487, no drift from 25c09f19f in
|
||
packages or scripts). It ended 01:49:41Z, wall 1950 s, exit 0.
|
||
|
||
| Phase | Result | Wall |
|
||
| --- | --- | --- |
|
||
| 1, touched test files whole and the loader's readers | 23/23 files pass; stack-release 219/219, compactions-budget 10/10, business 231/231 | 673 s |
|
||
| 2, bus suite | 873/873 | 25 s |
|
||
| 2, seat suite | 202/202 | 7 s |
|
||
| 2, cli suite | 905/905; P10 inner hop probe 2.31 s, outer 4.14 s | 682 s |
|
||
| 3, scripts/test-*.sh | all ten pass; tree clean after | |
|
||
| 4, 270 M1-M24 | 24/24 killed, each on its K file, each restore matched | |
|
||
| 4, 271 M1-M14 | 14/14 killed in the file the 271 table names (271 BUILD.md:713), each restore matched | |
|
||
|
||
No zero-pass file. Three skips, the same as before: test-queue's queue verify and render --check (this
|
||
checkout is not the queue's canonical root) and test-task's live provider cases, twice (row 244). No node
|
||
process was left with its cwd in the worktree. Sage ran one `git fetch` from the canonical checkout at
|
||
about 01:38:10Z, inside the cli suite (sage-3122); the suite had no red. Log:
|
||
/mnt/storage/scratch/gizmo-271-out/r4/ and r4.out.
|