diff --git a/agents/codex/work/queue-333/BUILD.md b/agents/codex/work/queue-333/BUILD.md index 58f1085a1..3637ecb38 100644 --- a/agents/codex/work/queue-333/BUILD.md +++ b/agents/codex/work/queue-333/BUILD.md @@ -199,3 +199,97 @@ M4b is extra; it shows the ruling 4 test finds a hidden path and not only an ext - The settings writer still stamps a new `since` on a renamed leaf (Departures above). Recommended follow-up, not built. - No migration, per the brief: no live stack holds proxy model records yet. - No push, queue, Vikunja, Gitea or PR action, per the brief. + +## R4 since carry + +Written 2026-11-03 by Claude Opus 5.5, implementation sub-agent of the Mosaic Stack lead (T3 c5c1e8ba). Issue #1831. Brief: /mnt/storage/scratch/mosaic-lead/briefs/row333-r4-fix.md. Branch t3/row333-r4-fix on the R3 tip fe3ef5128 (base b37cc5087, no rebase). Product, tests and docs are commit f65faf9df. The R3 section above is unchanged; where R4 changes something it says, this section says so. + +### Ruling 1: a moved key quota keeps its count + +- The lookup. runtime.mjs builds `keyLineage(business, keyId)` as `modelKeyLineage(credentials.status(), business, keyId)` and passes it to `launchSettings`. `modelKeyLineage` (limits.mjs) returns the lineage in that business holding the id, or `[]` when none does. It is built on `modelKeyLineages`, so the same host-wide uniqueness check runs and a duplicate id throws. It reads the bus's own credential records through their safe status: ids, never a key value. The writer hands it only a business id and a key id, so no caller-supplied lineage reaches it, and the runtime spreads `organization` before `keyLineage`, so an organization option can't replace it. +- The writer. `stamp` keeps its rules: a leaf with a stored `since` at the same path keeps it, and a caller-supplied `since` that differs refuses `invalid-request` (on a new leaf any caller `since` differs, so it always refuses). New: a `tokens.keys` leaf with no stored `since` that is a daily key quota asks the lookup for its key's lineage and takes the earliest stored `since` among the old document's `tokens.keys` daily key quotas whose ids are in it. Whether the same write removes the old leaf doesn't matter. With none it takes now. `reset` runs after `stamp` in the same write, so a reset sets now, in the move write or a later one. +- Lookup failure. `launchSettings`'s default `keyLineage` throws, so a writer built without one refuses. A throw, an answer that isn't an array, an id that isn't twelve lowercase hex, or a non-empty answer without the asked id refuses `OrganizationError('proxy-key-refused')` and writes nothing. The broker already passes that code through by name. A write that adds no new daily key quota never calls the lookup: account limits, legacy leaves, a changed or unchanged quota, a reset. +- Remove then re-add. A quota removed in one write and added back in a later one finds no old leaf, so it is a new quota and gets now. Documented in the `carried` comment and the READMEs. +- Docs. packages/plugins/usage/cliproxy-queue/README.md: the last paragraph's "stamps a new `since` on a renamed leaf ... restarts its count" is replaced by the carry rule, the remove-then-re-add rule and the lookup refusal. packages/plugins/README.md (daily-quota section) and packages/plugins/limit-check/daily-quota/README.md (model-key section) gain the carry sentence. packages/business/README.md had no `since` rule and, as the R3 review said, no model-key leaf; it now has a paragraph after `launch.limits.tokens.daily` on the leaf, the writer's `since` rules with the carry, and `keyLineage`. + +Choices and departures, and why: + +- Which leaves ask. Only a new leaf that is a daily key quota once stamped (`isDailyModelQuota` with `since` set to now). A twelve-hex-named plain `{limit}` leaf is a legacy account limit under row 333's rule, so it keeps the plain new-leaf rule and the lookup is not called. Source leaves must be daily key quotas too, so a legacy leaf's `since` never carries into a daily quota. +- A leaf that changes shape at the same path, a legacy `{limit, since}` becoming a daily quota, keeps its stored `since` under the existing same-path rule. Unchanged; found while writing the default-lookup case, which now adds a fresh id. +- Earliest against latest. The business test separates them: A from T0, B added from A (T0) and then reset to T1, both moved to C, and C takes T0 where the latest would give T1. Mutant M3 shows the test tells them apart. +- The real-runtime test uses a re-issue, not a rotation. The runtime keeps its `ModelKeys` inside and exposes no rotate, and its sweep rotates only at `rotateBy`, 30 days after issue, which no daily window spans without mocking the runtime's clock. The re-issue gives the same shape of lineage (B current, A among the earlier ids) and goes through the real `startBroker` wiring, the broker's `limits.set` and the runtime's own lookup. The rotation path through the writer is p5-key-lineage's R4 writer test, with the real `Credentials`, `ModelKeys`, `createLimits` and daily-quota module. +- The R3 "settings-writer move" test, which pinned a count of 100, now requires 160 and is renamed for R4. The R3 departure 3 and its "Not done" item are superseded by this ruling. + +### Ruling 2: the orphan set on every refresh + +`observeOrphans` (limits.mjs) runs in `refresh`, which the runtime calls after each committed verb and at boot, and in `orphanedKeys`. With no daily key quota left it stores an empty set for the business; otherwise it reads the lineages and calls `noteOrphans`, which logs each id not in the stored set and stores the current set. When the lineages can't be read it leaves the stored set as it was and logs nothing. So a quota removed and added back logs again, and an unchanged orphan logs once however many refreshes see it. The set is in memory: a restart is a fresh observation, and a quota still orphaned after it logs once more. No marker is persisted. This is in the `orphansLogged` comment and the cliproxy-queue README. + +Consequence: the orphan line now comes at the first refresh that sees the orphan, which can be boot or the refresh at the start of a launch decision, before that decision's own refusal line. Two R3 tests asserted the old order and now expect the orphan line first: the fire and re-hire test (the line comes at the keyless launch before the re-hire, not at the first launch after it) and the ruling 2 test. The number of orphan lines is unchanged in both. + +### Ruling 3: the barrier's last yield + +p5-telemetry-fixture.mjs captures `realSetImmediate` at module load with the other real timers, and `pollSettledAfterTick` yields on it inside the deadline, so a mocked `setImmediate` can't hold the barrier and the deadline covers the whole wait. Both options in the ruling, since together they cost nothing. The new p5-background test mocks `Date`, `setInterval`, `setTimeout` and `setImmediate`, advances one poll tick through the barrier, never ticks again, and requires the barrier to settle within a real bound of twice `WAIT_MS`, with the second queue read seen and the proxy state `available`. Against the R3 fixture it fails "pollSettledAfterTick did not settle" (about 10 s, the outer bound); I checked that before keeping the test. + +### Ruling 4: records + +- The 64-id bound is a limit. A lineage holds the current id, the retiring ids and the newest 64 earlier ids. Once an id falls out of the newest 64, a quota named by it is in no lineage, so it is orphaned while the agent stays hired: it limits nothing and the usage view lists it. Moving the quota to a current id fixes it, and since the old id is no longer in the lineage the writer can't carry its `since`, so the moved quota starts at now. Recorded here and in the cliproxy-queue README. +- R3 departure 4, corrected. Its reason, that proxy queue events "can only name keys the proxy still accepts, so earlier ids would never match", is wrong. A queued event the proxy accepted before the key left the record and that the bus drains after it names an id that is no longer current or retiring. proxy-usage.mjs:42-67 builds its owner map from current and retiring ids only, so that event goes to the reconcile log as `proxy-usage-unknown-key` and is not recorded. That loses telemetry attribution for the event. It is not a quota bypass: quotas count the native transcript records, which carry the run's bound key id. The owner map is unchanged in R4. + +### Tests + +Added: + +- packages/bus/tests/p5-key-lineage.test.mjs. The R3 writer test becomes the R4 writer test: A's quota is set at the start, the agent uses 60 on A, rotates, the human moves the unchanged quota to B, B's `since` equals A's, and after 30 more on A and 70 on B the launch refuses with count 160 and the report shows 160; a reset after the move sets now, the count is 0 and the launch goes ahead. New: a lineage that can't be read (an id in two agents' lineages) refuses a write adding a key quota with `proxy-key-refused`, the file unchanged, and a write adding none (a changed limit and a legacy hex leaf) saves without asking. New: an orphaned quota removed and added back logs twice, three more refreshes, a launch and `orphanedKeys` add nothing, and after a restart three refreshes add exactly one. +- packages/bus/tests/p5-runtime.test.mjs. Real `startBroker`, fake loopback proxy: a 2-token quota on A, a turn spends 2 on A, the launch refuses, the proxy stops listing A, `modelPreflight` issues B, `limits.set` moves the unchanged quota to B with no reset, the stored leaf is B with A's `since`, no key value is in the limits view, and the next `mayLaunch` refuses. +- packages/bus/tests/p5-background.test.mjs. The ruling 3 barrier test. +- packages/business/tests/token-limit-settings.test.mjs, with a stub lookup. One test: B added beside A carries A's time; earliest over latest; an id in no lineage gets now; a caller-supplied `since` refuses on a new leaf, a moved leaf and with the carried time itself; a reset in the move write sets now; removed and added back gets now. The other: no lookup for account limits, legacy hex leaves, a changed quota or a reset; a lookup that throws (with and without a code) or answers a string, a list holding a number, a list holding an uppercase id, a list without the asked id or an object refuses `proxy-key-refused` with the file unchanged; the default writer refuses a new daily key quota and saves account and legacy limits. + +Changed: the existing P5 business test passes a lookup answering `[]`, since a writer without one now refuses that leaf. In p5-key-lineage, the R3 fire and re-hire test and the ruling 2 test expect the orphan line first (Ruling 2 above). + +### Runs + +One at a time, each under a 30-minute limit (`timeout -k 10 1800 node --test`), Node v26.8.1, with TMPDIR under /mnt/storage/scratch/tmp and output to files there. Each whole run is `node --test` over the package's tests directory. All times are 2026-11-03 UTC. The first runs were on the working tree that became f65faf9df, committed with no edit between; the cli stack runs and the closing whole-package runs were on f65faf9df itself, the latter after the mutants were restored. + +| Run | Tree | Tests | Pass | Fail | Skip | Time | +|---|---|---|---|---|---|---| +| bus whole | before commit | 1397 | 1397 | 0 | 0 | 50 s, 00:55:51Z to 00:56:41Z | +| plugins whole | before commit | 132 | 132 | 0 | 0 | under 1 s, 00:56:53Z | +| business whole | before commit | 295 | 295 | 0 | 0 | 4 s, 00:56:53Z to 00:56:57Z | +| webui whole, run 1 | before commit | 499 | 498 | 1 | 0 | 88 s, 00:57:00Z to 00:58:28Z | +| webui editor-browser.test.mjs alone | before commit | 16 | 16 | 0 | 0 | 15 s, 00:58:47Z to 00:59:02Z | +| webui whole, run 2 | before commit | 499 | 499 | 0 | 0 | 119 s, 00:59:06Z to 01:01:05Z | +| cli launch-limit-checks.test.mjs | before commit | 5 | 5 | 0 | 0 | under 1 s, 01:01:23Z | +| cli stack-release.test.mjs | f65faf9df | 225 | 225 | 0 | 0 | 741 s, 01:02:50Z to 01:15:11Z | +| cli stack-release-external.test.mjs | f65faf9df | 25 | 25 | 0 | 0 | 98 s, 01:15:25Z to 01:17:03Z | +| cli stack-upgrade.test.mjs | f65faf9df | 25 | 25 | 0 | 0 | 148 s, 01:17:07Z to 01:19:35Z | +| harness tools.test.mjs | f65faf9df | 23 | 23 | 0 | 0 | under 1 s, 01:20:41Z | +| bus whole | f65faf9df | 1397 | 1397 | 0 | 0 | 54 s, 01:22:35Z to 01:23:29Z | +| plugins whole | f65faf9df | 132 | 132 | 0 | 0 | 1 s, 01:23:29Z to 01:23:30Z | +| business whole | f65faf9df | 295 | 295 | 0 | 0 | 4 s, 01:23:30Z to 01:23:34Z | +| webui whole | f65faf9df | 499 | 499 | 0 | 0 | 131 s, 01:23:38Z to 01:25:49Z | + +The webui run 1 failure was editor-browser.test.mjs:93, the CSP editor test: `Condition timed out: !document.querySelector("#doc-save").disabled` at :122, with the host's load average between 45 and 60. R4 changes nothing webui loads. The file passed alone and the whole package passed twice after, so I read it as load, not R4; both runs are listed. The model-final-completion.test.mjs:69 flake (16 !== 19) did not appear in either bus run. Against R3, bus is up 4 (p5-key-lineage's lookup-failure and orphan re-add tests, the p5-runtime move test, the p5-background barrier test; the writer test is renamed, not added) and business up 2. Plugins and webui are unchanged. + +Pins. No test pins any changed path with a hash: the grep of every test, script and fixture naming a changed file's basename for a 64-hex string found only packages/queue/tests/fixtures/queue-marked.md, which names none of the four READMEs. The tests that read a changed file by path (limit-checks.test.mjs:319 and :323, host-run.test.mjs:429) are in packages/bus, run whole. Outside the four packages, harness tools.test.mjs imports `SETTINGS_KEYS` from launch-settings.mjs and ran whole (above). Nineteen packages/cli test files import `startBroker` from bus runtime.mjs; they import, they don't pin, and beyond the four cli files the brief names they were not run (Not done). + +### Mutants + +Each was applied to f65faf9df, run against the named files, and restored with `git checkout -- `; `git status --short` was empty after each. + +| Mutant | Change | Killed by | Result | +|---|---|---|---| +| M1 | launch-settings.mjs:73 `entry.since = kept ?? now;`, so stamp ignores the lineage | p5-key-lineage.test.mjs:173, the writer test, at :185: B's `since` was the move time `2026-10-24T12:01:00.000Z` where A's `2026-10-24T12:00:00.000Z` was expected. p5-runtime.test.mjs:62 at :106: the stored B leaf had the move time, not A's. token-limit-settings.test.mjs:178 at :184: B took the write time 08:00 where A's 07:00 was expected. Also p5-key-lineage :198 at :206 and token-limit-settings :204 at :212, both because the lookup is never called (`asked` was 0 where 1 was expected, and `[]` where `[A]` was) | p5-key-lineage 2 of 11 fail, p5-runtime 1 of 2, token-limit-settings 2 of 9 | +| M2 | runtime.mjs:346 passes `launchSettings` no `keyLineage` | p5-runtime.test.mjs:62 at :88: `proxy-key-refused` from the broker at the first `limits.set` that adds a key quota, since the writer's default lookup refuses. The R3 P5 runtime test at :39 fails the same way | p5-runtime 2 of 2 fail | +| M3 | launch-settings.mjs:96 sorts descending, so the latest `since` wins | token-limit-settings.test.mjs:178 at :189: C had `since` 08:00 (T1, B's reset time) where 07:00 (T0, A's) was expected | token-limit-settings 1 of 9 fail. p5-key-lineage and p5-runtime pass (13 of 13): each of their moves has one source leaf, so earliest and latest are the same there | +| M4 | limits.mjs:130, `refresh` no longer calls `observeOrphans` | p5-key-lineage.test.mjs:348, the orphan re-add test, at :358: one `limit-key-orphaned` line where two were expected; the set was never cleared, so the re-added leaf was still remembered. Also the R3 fire and re-hire test at :299 and the R3 ruling 2 test at :341, each missing the orphan line the refresh now logs first | p5-key-lineage 3 of 11 fail | +| M4b | limits.mjs:90 returns `[]` with no daily key quota but leaves the remembered set alone | p5-key-lineage.test.mjs:348 at :358, the same one-line-for-two failure | p5-key-lineage 1 of 11 fail | +| M5 | p5-telemetry-fixture.mjs restored to its R3 content | p5-background.test.mjs:71, the barrier test: `pollSettledAfterTick did not settle` after 10 s | p5-background 1 of 5 fail | + +M4b and M5 are extra. M4b shows the empty set itself is needed, not only the refresh call. M5 shows the ruling 3 test fails on the R3 barrier, whose last yield used the mocked `setImmediate` outside the deadline. + +### Not done + +- No push, queue, Vikunja, Gitea or PR action, per the brief. +- No migration or persisted orphan marker, per the rulings. +- The proxy-usage owner map still holds only current and retiring ids (Ruling 4 above). +- The nineteen packages/cli test files that import bus runtime.mjs were not run beyond the four the brief names: they import it and pin nothing, and the runtime change adds only the lookup, used when a write adds a daily key quota.