Files
jason.woltje a13050012a docs(row 333 R4): BUILD.md R4 section (#1831)
Rulings 1 to 4 as built, choices and departures, tests, runs, mutants M1 to M5, and what was not done. The R3 section is unchanged.
2026-11-02 19:27:04 -06:00

48 KiB
Raw Permalink Blame History

row333 P5 implementation — Mos1961

Base: 09a847eda. Branch: t3/model-proxy-p5-quota-usage. Actor: Codex. Accepted brief: docs/plans/2026-10-24_model-proxy-rows.md P5; associated design D18–D20 with mos-1600 correction.

Queue next codex refused a linked worktree; no canonical mutation. Parent supplied accepted assignment/authority and coordinated runtime collector ownership. No remote issue posting or live configuration/credential reads.

Ruling: preserve existing schema by carrying safe modelKeyId in usage cursor metadata; proxy telemetry uses a separate source excluded from turnRecords and every quota check. Existing account limits stay compatible. A model key daily quota is keys.<12-hex-id> with reset/timeZone; unset remains no quota. Human-only protection applies to model-key limit changes and resets. No launcher/session/harness/business route edits.

Verification uses serial nice19 node tests with private HOME/XDG_CONFIG_HOME/TMPDIR/MOSAIC_CONFIG and unchanged test budgets. Broad/full gates reserved to parent after row334 target returns. Receipts below are actual command outputs; failures retained.

Parent steering receipts: runtime ownership confirmed createUsage/createLimits collector wiring/tick/close exclusively P5; Fethry avoids this area. Broker changes only usage option accepted by human plugins.set and mustStop run identity. No launch/harness/session/business-route bytes changed.

Exact additive old expectation update authorized by parent: kinds.test.mjs old five-entry Object.keys(KINDS) expected array retains all five in their original order and adds usage. loader.test.mjs complete loaded() expected objects retain all original members and add usage:[]. No old assertion was removed, guard relaxed, timeout increased or sleep added.

First RED receipts: 01-token-red (native key limit incorrectly allows); 02-contract-red (daily key quota, key schema and HTTP method absent); 04-integration-red (new collector absent; native attribution and plugins.set config missing); 08-edges-red (lookup fails open, nonhuman usage writer allowed, Console labels absent); 10-lookup-behavior-red (existing runtime lookup extracted unchanged: missing exception for missing/foreign/ambiguous ownership). 09 captured the preceding missing export, not behavior proof. 05 captured duplicate local variable syntax error, fixed before 06. Runtime fixture failures 12/13 were missing required process identity then incorrect cap extraction; corrected fixture follows actual runtime contract. 14 proves actual model-key issue/preflight, native accounting at key quota, stopping, later launch denial and broker human plugin setting path.

Unexpected existing named failure (reported to parent before any rerun): receipt18 was 169/170, failing REQ-LIMIT-4 a role run at its limit stops at the end of its turn: RESULT and read land first, the claim is revoked by limit-check at limit-stop.test.mjs:61. Cause: additional run in the broker's second mustStop argument violated its existing exact subject contract. Fix keeps that argument unchanged; run moves to a separate internal third context and is consumed by host lookup only. The old assertion remains byte-for-byte unchanged. Correction to initial parent message: the quoted name here is the exact failing test name (the first report paraphrased it).

Scoped receipt19: original contract corrected, 170/170 current whole-file tests GREEN. Parent then required direct native-context/no-bypass regression: receipt20 genuinely RED (stop:false on a supplied safe-looking modelKeyId when lookup returned null). Removed that fallback entirely and stripped caller modelKeyId/run from the public check subject. Only trusted host resolver can provide the ID; only the separate third internal context provides run. Receipt21: direct context tests, original unmodified limit-stop file and proxy telemetry whole files 15/15 GREEN. Retiring/current mapping rejects wrong business/instance/run and unknown or ambiguous retained IDs.

Final implementation decisions and evidence:

  • Native source records carry only the immutable run modelKeyId. New and retained generations resolve from safe Credentials.status metadata; missing, duplicate, foreign or no-longer-retained mappings fail proxy-key-refused. Third mustStop context is internal and associated by the broker from its authenticated session, never an agent argument. Missing context is fail-closed (receipt23 genuine RED stop:false, fixed in receipt24).
  • tokens.keys.<12hex> accepts the daily quota shape, uses daily-quota when reset/timeZone are present, and remains unset by default. Existing named account keys and account daily quotas retain their plugins and contracts. All model-key quota/reset changes and usage module/poll changes are fenced at both organization validation points for every writer, tester included.
  • Management queue HTTP GET is fixed to loopback with count=100, original 5-second default and 8 MiB response cap, no retries/redirects. Client api_key and upstream source hash immediately to the first 12 SHA-256 hex characters. Fields are whitelisted; malformed batches are atomic. No raw values, upstream body or arbitrary error text reaches product store/log/view/audit.
  • Exactly one bus writer owns the collector, shares one management client, drains once and assigns metadata owners, including retained keys. Unknown msk IDs get reconcile receipts; disabled businesses and foreign unmanaged keys get no telemetry. Polling defaults 15 seconds, accepts integer 1..59 and serializes concurrent ticks. @proxy/cliproxy-queue records are append-only but excluded before every native turn record/total/limit decision. Availability and account counts are explicitly proxy-reported in the Console.
  • Named upstream-value mutant changed accountId: sha256(event.source).slice(0,12) to the raw event.source: receipt16 exit1, four tests failed. Restoration matched original client SHA256 f5f86cad661034f223a76352d877fa8d7883bfa921a7a01282347a9a03272441, receipt17 positive 15/15. Final receipt24 also runs after restoration and is GREEN.
  • Final current whole-file verification: bash agents/codex/work/queue-333/verify-scoped.sh → 174/174 PASS, 24 explicitly listed whole files, serial nice19, private HOME/XDG_CONFIG_HOME/TMPDIR/MOSAIC_CONFIG, original test budgets. Raw output: receipts/24-final-current-wholefiles-green.txt SHA256 3358af061c35d42b00d5ee1e001939534c5e7563d4f5418ea85431605d0b9f6d. Runner SHA256 c8a2bf4539bffe38f3198308fd60e9c1ad338841b406af160ab0739c15394290.
  • Source freeze: source-inputs.sha256 enumerates all 202 consumed repository inputs (182 modules/scripts plus 20 schema/PIN/other files), traced on final execution and rehashed unchanged after it. Manifest SHA256 b4662dcafb2aeb27a06e5dbd3b9d0b6d184e8624e344fed351ff20e3b255951b. Raw per-process trace is receipts/source-inputs.raw.jsonl. candidate-sourcepaths.txt and candidate-sourcehashes.sha256 list every changed package file for parent combination; receipts.sha256 covers every raw run and source trace. Runner/trace use no content logging.
  • Cleanup: original private mktemp root removed and absence verified; scoped runner EXIT trap removes its separate private root. All fixture servers/stores close through test cleanup. Mutant backup and temporary root marker removed. Isolated worktree retained for parent review; no production process, real proxy, account, config, credential path or provider was contacted.

Limits and handoff: scoped GREEN is not the full landing gate. No package/full gates, Chromium/browser gates, acceptance binary, canonical queue mutation, remote issue post, push, merge or landing was run. Full/browser/combined gates remain with the parent under 1014.Q264 after admission; fresh independent Codex review is the parent's next task. No self-approval is claimed. Base remains 09a847eda and no unlanded662/657/658 feature bytes were imported.

R2 fixtures

Base 09a847eda, row 333 tip ddb3e961f, R2 fixture commit 12eba4b59. Author: Claude (sub-agent of the Mosaic Stack lead, T3 c5c1e8ba). Issue #1831. No product code, business document schema or test assertion changed.

Cause. Row 333 added packages/plugins/src/model-quota.mjs and imports it from packages/business/src/launch.mjs:3 and packages/business/src/organization.mjs:11. The cli release fixtures build smaller copies than a real stack copy: they copy packages/business/src and add only packages/plugins/src/kinds.mjs. Their business loader therefore failed with "Cannot find module .../packages/plugins/src/model-quota.mjs". Real copies are unaffected, because scripts/stack-copy.sh ships all of packages/plugins (COPY_PATHS, line 41). R1's evidence ran the 24 touched test files only (verify-scoped.sh, 174/174) and no package or cli gate, so it never built a release fixture copy and could not see this.

Fixture changes (tests only):

  • packages/cli/tests/business-imports.mjs (new, 26 lines): businessImportedFiles(repo) reads the relative imports of every .mjs under packages/business/src and returns, sorted and repo-relative, the files outside that directory they reach, following those files' own imports. An import of a missing file throws. At this tip it returns packages/library/src/limits.mjs, mapping.mjs, problems.mjs, packages/plugins/src/kinds.mjs and packages/plugins/src/model-quota.mjs.
  • packages/cli/tests/stack-release-fixture.mjs:314 (copies for stack-release.test.mjs, stack-release-external.test.mjs and stack-upgrade.test.mjs): after the business, library and roles copy, copy each returned file that is not already in the copy. kinds.mjs is skipped on purpose, because the fixture has already written its KINDS_OLD or KINDS_NEW form there.
  • packages/cli/tests/stack-release.test.mjs:1994 (the RCOPIES copies): same loop.

Searched for other fixtures that copy packages/business/src without the whole plugins directory: the grep over packages/*/tests, scripts and scripts/testing finds only those two. packages/cli/tests/case2-install.test.mjs:74 copies packages/business/src/vocabulary.mjs alone, which has no imports. stack-copy.test.mjs and stack-release-external.test.mjs:450 copy tracked files or the whole repository, so they carry model-quota.mjs already.

Production check (every place that copies, mounts or packages business code or plugins/src/kinds.mjs without the whole plugins directory):

  • scripts/stack-copy.sh:32 copies packages/business/src and :41 copies the whole packages/plugins, so model-quota.mjs reaches every real copy. scripts/case2-install.sh:185 calls it.
  • scripts/stack-release.sh:371 and :378 import the copy's own packages/business/src/business.mjs, which reaches launch.mjs and so model-quota.mjs. That copy comes from stack-copy.sh, which takes both directories from the same commit.
  • packages/seat/src/sandbox.mjs RUNNER_FILES (:50-:72) mounts packages/business/src/vocabulary.mjs (:67) and packages/plugins/src/kinds.mjs (:68) only. A transitive import walk from all 19 listed files reaches no file of packages/business or packages/plugins outside that list; launch.mjs, organization.mjs, business.mjs, loader.mjs and model-quota.mjs are not reached. vocabulary.mjs and kinds.mjs import nothing. packages/seat/src imports only vocabulary.mjs from business (compactions.mjs:46, sandbox.mjs:40, session.mjs:42). model-quota.mjs is therefore not needed in the sandbox.
  • scripts/test-darkwing-launch.mjs:34 and scripts/test-rocko-launch.mjs:25 copy packages/seat/src alone, which needs only vocabulary.mjs.
  • Containerfile:23-26 copies contracts, src, two processing-control files and adapters; src imports nothing from packages. compose.yaml mounts the data root and the auth file only.
  • scripts/mosaic:35 and scripts/case2-precheck.sh:85,171 import business code from the repository or copy they sit in, which holds all of packages/plugins.

Result: no production path misses model-quota.mjs. No product change was needed or made.

RED, on ddb3e961f before the change (whole files, from packages/cli, one at a time, TMPDIR /mnt/storage/scratch/tmp/QYOMZY):

  • stack-release.test.mjs: 224 tests, 56 pass, 168 fail, 0 skip, 186 s. 127 log lines carry the missing-module message.
  • stack-release-external.test.mjs: 25 tests, 4 pass, 21 fail, 0 skip, 32 s.
  • stack-upgrade.test.mjs: 21 tests, 7 pass, 14 fail, 0 skip, 25 s.

These match the counts the join child recorded (56/168, 4/21, 7/14).

GREEN, on 12eba4b59:

  • stack-release.test.mjs: 224/224 pass, 0 fail, 0 skip, 694 s. stack-release-external.test.mjs: 25/25, 88 s. stack-upgrade.test.mjs: 21/21, 66 s.
  • cli whole (node --test tests/*.test.mjs): 1045 tests, 1044 pass, 0 fail, 1 skip, 656 s. The skip is "FED R7 scratch two-instance acceptance", which names scripts/test-federation-acceptance.sh as the way to run it.
  • business 286/286, 3 s. bus 1380/1380, 49 s. plugins 123/123, 1 s. model-proxy 108/108, 1.4 s. webui 492/492, 63 s. No skips in these. The model-final-completion.test.mjs:69 host flake (16 !== 19) did not appear, so no rerun was needed.
  • The counts for business (286) and plugins (123) are lower than the join's (292, 128) because the join included row 332's tests; this branch stays on base 09a847eda.

Mutant. With the helper changed to drop model-quota.mjs from its result (a one-line filter in business-imports.mjs), stack-release.test.mjs gave 224 tests, 56 pass, 168 fail in 183 s, with 127 "Cannot find module .../packages/plugins/src/model-quota.mjs" lines. The helper then printed four files, without model-quota.mjs. git checkout restored the file; git status --short printed nothing afterwards.

Join onto landed 332

Written 2026-11-02T22:49Z by Claude (sub-agent of the Mosaic Stack lead, T3 c5c1e8ba), author of this join. Issue #1831. Checkout branch t3/row-333-join-replay.

Base and heads. Row 333 R2 head 0ef269da9 on base 09a847eda (R1 503b91e56, overlay 2627321f6, docs c012b2e0a and ddb3e961f, R2 fixtures 12eba4b59 and 0ef269da9). Row 332 landed on refactor at 5df0e9d6f (R2 and the lead correction). The branch started at 5df0e9d6f with no commits of its own and a clean tree. Command: git -c core.attributesFile=/mnt/storage/scratch/q284gate/union.attributes merge --no-ff --no-commit 0ef269da9. Merge commit c0c291cda, parents 5df0e9d6f and 0ef269da9, made at 22:33:47Z.

Why a replay. The earlier join (eabeb1aa8) was made on 332 R1. Row 332 then changed (R2 and a lead fix) and row 333 gained two fixture commits. Both rows change packages/business/src/launch.mjs and packages/plugins/README.md.

Conflicts. Exactly two paths, the two named in the brief. No other path conflicted.

  • packages/business/src/launch.mjs, import block (lines 3 to 8 before resolution). 332 changed the errors import to { BusinessError, refuse } for the proxy route refusals in checkModels. 333 added import { isDailyModelQuota } from "../../plugins/src/model-quota.mjs" for the daily model leaves in checkTokenLimits. They do not contradict. Resolution: both imports, the model-quota line first, then { BusinessError, refuse }. The rest of launch.mjs merged without conflict: 332's checkModels route handling and the checkAccount proxy credential, and 333's checkTokenLimits keys block.
  • packages/plugins/README.md, end of file (lines 331 to 378 before resolution). 332 appended "Per-session proxy context (P3)" and 333 appended "Proxy usage telemetry (row333 / P5)" after the same last paragraph. Resolution: keep both, P3 section first, one blank line between, P5 section last.

Other paths touched by both sides, merged without a conflict and not edited by hand: BUILD-LOG.md and docs/SESSIONS.md (union attribute, both sides' lines kept), and packages/cli/tests/stack-release.test.mjs (333's 3 lines for the business-imports helper, 332's R2 hermetic-launch edits). No product or test file was edited by hand.

Comparison with the reference resolution (/mnt/storage/scratch/tmp/iLeQSZ/join-resolution.patch). I took git diff 5df0e9d6f for the two files and compared it with the reference patch after removing the index lines. The result is identical, and the hunk headers (-1,5 +1,6; -76,8 +77,16; -365,3 +365,12) are equal too, so there is no difference to explain. The reference was a diff against the 332 R1 blobs. launch.mjs is d9ef01b40 and README.md is d4625c9f6 on both 332 R1 and landed 332, so R2 and the lead fix did not change these two files, which is why the earlier resolution carries over unchanged. A correction to the merge commit message: it says the files match the reference "apart from hunk offsets", but the hunk offsets match as well. The commit stays as made.

Tests, on c0c291cda. node_modules linked from /mnt/storage/src/mosaic-stack (a link at the root and a real directory of links under packages/webui), TMPDIR /mnt/storage/scratch/tmp/Z280Ko (31 bytes), each suite whole, one at a time, from its package directory with node --test tests/*.test.mjs, 30-minute limit, started 22:34:09Z. Nothing was skipped, failed or rerun except as noted.

Suite Tests Pass Fail Skip Time
business 293 293 0 0 2 s
plugins 129 129 0 0 1 s
bus 1382 1382 0 0 48 s
model-proxy 108 108 0 0 1 s
harness 205 205 0 0 13 s
seat 223 223 0 0 6 s
webui 498 498 0 0 132 s
cli 1102 1101 0 1 669 s
bash scripts/test-federation-acceptance.sh 1 1 0 0 5 s

The one cli skip is "FED R7 scratch two-instance acceptance", which names scripts/test-federation-acceptance.sh as the way to run it. That script passed 1/1 in the last row, with "R7 cleanup: no child groups, ports or sockets remain". The model-final-completion.test.mjs:69 host flake (16 !== 19) did not appear, so no rerun was needed. Business and plugins are one test higher than the earlier join's 292 and 128, and bus 1382 is two above R2's 1380 on the 09a847eda base; this record does not attribute those differences to a cause.

Cleanup. No test process of mine remained after the runs. The node_modules links and the TMPDIR were removed; git ls-files -co --exclude-standard | grep node_modules printed nothing, and git status --short showed only this file and docs/SESSIONS.md before this commit.

R3 rotation fix

Written 2026-11-03 by Claude Opus 5.5, implementation sub-agent of the Mosaic Stack lead (T3 c5c1e8ba). Issue #1831. Brief: /mnt/storage/scratch/mosaic-lead/briefs/row333-r3-fix.md. Review: /mnt/storage/scratch/mosaic-lead/reviews/row333-opus.md, findings 1 to 5. Branch t3/row333-r3-fix on base b37cc5087. Product, tests and docs are commit 3692fca52.

Ruling 1: the quota follows the agent's key lineage

  • Model record. Credentials.#format accepts an optional previous: a non-empty array of at most 64 entries, each exactly { keyId, retiredAt } with a 12-hex id and a canonical ISO time. Anything else refuses credential-format, on write and when the file is opened. A record without the field loads unchanged. Safe status shows previous (ids and times) next to retiring, and an empty array for a record without it. No key value is in it.
  • When an id is added. ModelKeys adds an id when the key leaves the record. On a rotation the old key first becomes retiring, as before, and its id moves to previous when the sweep removes it, at grace expiry or at once when no run is bound to it. On a re-issue (the #issue path where the proxy no longer lists the old key) the old id goes straight to previous. The field stays absent until the first id leaves. Past 64 the writer drops the oldest, so a record never carries 65; the format refuses 65 as well. A fire removes the record, as before, so a re-hired agent starts a new lineage. The revocation proofs still compare whole records with isDeepStrictEqual; a test fires an agent whose record carries previous and the proof holds.
  • Lineage. An agent's lineage is its current id, its retiring ids, then its earlier ids. modelUsageKey checks uniqueness over every lineage on the host: an id in two agents' lineages, or twice in one, refuses proxy-key-refused. The new modelKeyLineages lists the business's lineages and applies the same host-wide check. createLimits takes a keyLineages hook, which refuses by default, and passes lineages to the checks only when a daily key leaf exists. The subject's key must be in exactly one lineage read in the same call, or the launch refuses and the turn stops. report shows every limit check failed: proxy-key-refused when the lineages can't be read.
  • daily-quota. A key leaf applies to a subject whose key is in the lineage holding the leaf's id, and its tally sums the day's records for every id in that lineage. Usage records and the Console per-key counts stay per key id. Missing or malformed lineages throws bad-lineages; account quotas never need them.
  • mustStop resolves the run's bound key, and that key's lineage decides the leaf. A run may be bound only to the current or a retiring key. A run whose bound id has moved to previous refuses proxy-key-refused, which stops it.
  • Orphaned leaves. A leaf whose id is in no hired agent's lineage counts its own id only and matches no subject, so it refuses no launch. The limits code logs {event: 'limit-key-orphaned', business, path} once when a leaf becomes orphaned, not per launch, and again only if it stops being orphaned and becomes orphaned later. orphanedKeys feeds orphanedKeyQuotas in the usage view, and the Console usage view lists each one: "Key quota matches no agent, so it limits nothing. A fire and re-hire gives the agent a new key; set the quota on the new key id." A fire and re-hire therefore leaves the old leaf orphaned, and the human sees it in that view. The Token limits panel can only reset leaves, not rename them, so the text points at the new key id rather than at that panel.
  • Docs. packages/plugins/usage/cliproxy-queue/README.md (where row 333 documents tokens.keys leaves), packages/plugins/README.md (the limit-check interface gains lineages and subject.modelKeyId, plus the daily-quota section) and packages/plugins/limit-check/daily-quota/README.md each gain the rotation and orphan sentences. The business package documents no model-key leaf, so it has no doc change.
  • Rollback. A release from before this round refuses a record that carries previous with credential-format, which fails closed. No live stack holds proxy model records yet (no proxy cutover until Vikunja #774), so no migration was built.

Departures from the expected shape, and why:

  • 65 entries: both refuse and trim. The format refuses 65 (a hand-edited or foreign record), and the writer trims to the newest 64 so the bus never writes one. Both are pinned.
  • A keyless agent under a key quota refuses even when every key leaf is orphaned. An orphaned leaf matches no subject, so the refusal is not needed to enforce it, but resolving the subject's key comes first and I kept that fail-closed order rather than add a second path that skips it.
  • Moving a leaf through the human writer. The brief says moving a leaf from A to B on rotation day must not reset the count. The tally does sum A and B: a leaf named B whose since is the day's start counts 160 (90 on A, 70 on B) and refuses. But the human settings writer treats a rename as a new leaf and stamps since with the move time, so a leaf moved through the writer counts only the usage after the move (100 in the test, not 160). I did not change the writer: the since stamping is the row 125/126 human-writer contract, in the business package, and changing it widens this round. With the lineage, the move is not needed, since the leaf on A keeps applying after rotation. Both cases are pinned in p5-key-lineage.test.mjs. Recommended follow-up if the lead wants a moved leaf to keep its count: let the writer carry since over when the old and new ids are in one lineage.
  • The proxy-usage owner map stays current plus retiring. It attributes proxy queue events, which can only name keys the proxy still accepts, so earlier ids would never match. That is telemetry, not the quota, and I left it.

Ruling 2: key-quota-no-model-key

modelUsageKey throws key-quota-no-model-key when the agent has no model record, or one with no key issued. createLimits passes that code and proxy-key-refused through by name; any other lookup failure is still proxy-key-refused. Pass-through, after grepping proxy-key-refused across packages/*/src: the broker's organization-error pass-through list gains the code (bus/src/broker.mjs; no organization writer raises it today, but it now passes through the same way as the other proxy key codes). The launcher maps every bus refusal to limit-check-failed with message bus: <code>, and the turn-end stop reason is limit: bus failed <code>; neither needed a change, and both are pinned for the new code (cli launch-limit-checks.test.mjs, bus limit-stop.test.mjs). With no key quota there is no lookup and no refusal.

Ruling 3: rotation between preflight and mayLaunch

The lineage covers it. Finding 3's case is preflight admitting A, the sweep rotating A to B, and mayLaunch checking B. B and A are now one lineage, so a leaf named A matches the subject B and counts A and B: the launch refuses. The rotation test asserts exactly this state (A retiring, B current, mayLaunch refuses under the A leaf). After a re-issue, A is not current or retiring, so binding a run to A refuses. I found no gap, so mayLaunch does not take the admitted key id.

Ruling 4: the usage module gets readUsageQueue only

createProxyUsage hands the module Object.freeze({ readUsageQueue: () => management.readUsageQueue() }). cliproxy-queue reads read({ readUsageQueue }). The test wraps the real loader to capture the argument. It has exactly one own key, readUsageQueue, a function, is frozen with Object.prototype as its prototype, and a walk over every own property, accessor and prototype reachable from it finds neither the client, its prototype, its class nor any of its other methods by name or by reference. Spies on every client method show only readUsageQueue was called, and the event was still recorded.

Ruling 5: fixture deadlines

Each wait in p5-telemetry-fixture.mjs (waitLog, waitQueue, settleInitial, pollSettledAfterTick) races a deadline on the real setTimeout, captured when the module loads, before any test mocks timers. The default is WAIT_MS = 5000 ms; the fixture takes waitMs. The rejection names what it waited for, for example p5 fixture: no usage queue read number 1 within 50 ms. The new test mocks every timer API, never advances it, and asserts all four rejections by exact message with waitMs: 50. The p5-background.test.mjs header is updated.

Tests

Added: packages/bus/tests/p5-key-lineage.test.mjs (9 tests). It uses the real Credentials, ManagementClient, ModelKeys and its sweep, createLimits, modelUsageKey, modelKeyLineages and the daily-quota module, with a fake loopback management server, a made-up management key and a fresh temp root. Only Date is mocked.

  • Rotation mid-day under a hit leaf named A: refuses before, right after (A retiring, B current), inside the grace, after the grace (A in previous) and after a restart that reloads the record. The run bound to A stops at its turn end after the rotation. Another agent stays allowed.
  • Re-issue: the proxy no longer lists A at preflight, B is issued, A is in previous, and the leaf still refuses, including after a restart. A limit of 101 allows.
  • The leaf moved from A to B on rotation day still refuses with count 160, and the run bound to A stops.
  • The settings-writer move: since is the move time and the count is 100.
  • Lineage uniqueness: an id in two agents' lineages refuses both, and the report shows check failed. An id twice in one lineage refuses too.
  • Format: a record without previous loads; thirteen malformed values refuse credential-format on write and on open, including 65 entries, a key value in place of an id and an empty array; 64 earlier plus 64 retiring load within the 16384-byte cap.
  • A re-issue on a record holding 64 drops the oldest.
  • Fire and re-hire: the revocation proof holds with previous present, the old leaf is orphaned, three launches go ahead, one log line, the view lists it, and no key value is in the logs, view, receipts, status or store files.
  • Ruling 2 on modelUsageKey, mayLaunch and mustStop, and no lookup without a key quota.

Also added: three daily-quota tests (lineage match and count, orphan, bad-lineages), one usage-render test (the orphan list, escaped), the ruling 4 test in proxy-usage.test.mjs, the ruling 5 test in p5-background.test.mjs, and the new code in limit-stop.test.mjs and cli launch-limit-checks.test.mjs. Changed for the new rules: limit-checks.test.mjs (a leaf on a retiring key now applies to the agent's current key too, so the launch refuses; an unrelated lineage still allows), p5-key-context.test.mjs (the two no-owner cases now expect key-quota-no-model-key; a wrong run stays proxy-key-refused), p5-key-compatibility.test.mjs (passes lineages to the module) and the p5-telemetry-fixture.mjs waits.

Runs

All on 3692fca52, one at a time, each under a 30-minute limit, with TMPDIR under /mnt/storage/scratch/tmp and output to files there. Each whole run is node --test over the package's tests directory.

Run Tests Pass Fail Skip Time
bus whole 1393 1393 0 0 49 s, 23:55:28Z to 23:56:17Z on 2026-11-02
plugins whole 132 132 0 0 0.3 s, 23:56Z
business whole 293 293 0 0 2.6 s, 23:56Z
webui whole 499 499 0 0 66 s, 23:56:50Z to 23:57:56Z
cli stack-release.test.mjs 225 225 0 0 664 s, 23:58:03Z on 2026-11-02 to 00:09:07Z on 2026-11-03
cli stack-release-external.test.mjs 25 25 0 0 88 s, 00:09:07Z to 00:10:35Z
cli stack-upgrade.test.mjs 25 25 0 0 69 s, 00:10:35Z to 00:11:44Z
cli launch-limit-checks.test.mjs 5 5 0 0 0.2 s, 00:11:44Z

The model-final-completion.test.mjs:69 host flake (16 !== 19) did not appear, so no rerun was needed. Against the join record above, bus is up 11 (the 9 lineage tests, the ruling 4 test and the ruling 5 test), plugins up 3 (the daily-quota tests) and webui up 1 (the usage-render test); business is unchanged. The other changed tests changed existing cases or added table rows, not tests.

Mutants

Each was applied to the committed tree, run against the named file, and restored with git checkout -- <file>; git status --short was clean after each.

Mutant Change Killed by Result
M1 daily-quota matches a key leaf to the subject by the leaf's own id only p5-key-lineage.test.mjs:101, the rotation test: right after the rotation the launch was allowed where a refusal with count 100 was expected. Also :138 (re-issue), :156 (moved leaf, run not stopped), :252 (fire and re-hire) and daily-quota.test.mjs:197 lineage 4 of 9 fail, daily-quota 1 of 17 fail
M2 the tally counts only the leaf's own id p5-key-lineage.test.mjs:155, the moved-leaf test: allowed where limit 160 and count 160 were expected. Also :179 (writer move), :252 (count 100, expected 105) and daily-quota.test.mjs:197 lineage 3 of 9 fail, daily-quota 1 of 17 fail
M3 #issue records no old id on a re-issue p5-key-lineage.test.mjs:137, the re-issue test: previous was undefined where [{ keyId: 'f6548f12575e', retiredAt: '2026-10-24T12:01:00.000Z' }] was expected. Also :241 (trim at 64) lineage 2 of 9 fail
M4 the usage module gets { client: management, readUsageQueue } proxy-usage.test.mjs:89: own keys were ['client', 'readUsageQueue'] 1 of 8 fail
M4b the client hidden as a non-enumerable property on the readUsageQueue function proxy-usage.test.mjs:98: the walk reached the client 1 of 8 fail

M4b is extra; it shows the ruling 4 test finds a hidden path and not only an extra own key.

Not done

  • The settings writer still stamps a new since on a renamed leaf (Departures above). Recommended follow-up, not built.
  • No migration, per the brief: no live stack holds proxy model records yet.
  • No push, queue, Vikunja, Gitea or PR action, per the brief.

R4 since carry

Written 2026-11-03 by Claude Opus 5.5, implementation sub-agent of the Mosaic Stack lead (T3 c5c1e8ba). Issue #1831. Brief: /mnt/storage/scratch/mosaic-lead/briefs/row333-r4-fix.md. Branch t3/row333-r4-fix on the R3 tip fe3ef5128 (base b37cc5087, no rebase). Product, tests and docs are commit f65faf9df. The R3 section above is unchanged; where R4 changes something it says, this section says so.

Ruling 1: a moved key quota keeps its count

  • The lookup. runtime.mjs builds keyLineage(business, keyId) as modelKeyLineage(credentials.status(), business, keyId) and passes it to launchSettings. modelKeyLineage (limits.mjs) returns the lineage in that business holding the id, or [] when none does. It is built on modelKeyLineages, so the same host-wide uniqueness check runs and a duplicate id throws. It reads the bus's own credential records through their safe status: ids, never a key value. The writer hands it only a business id and a key id, so no caller-supplied lineage reaches it, and the runtime spreads organization before keyLineage, so an organization option can't replace it.
  • The writer. stamp keeps its rules: a leaf with a stored since at the same path keeps it, and a caller-supplied since that differs refuses invalid-request (on a new leaf any caller since differs, so it always refuses). New: a tokens.keys leaf with no stored since that is a daily key quota asks the lookup for its key's lineage and takes the earliest stored since among the old document's tokens.keys daily key quotas whose ids are in it. Whether the same write removes the old leaf doesn't matter. With none it takes now. reset runs after stamp in the same write, so a reset sets now, in the move write or a later one.
  • Lookup failure. launchSettings's default keyLineage throws, so a writer built without one refuses. A throw, an answer that isn't an array, an id that isn't twelve lowercase hex, or a non-empty answer without the asked id refuses OrganizationError('proxy-key-refused') and writes nothing. The broker already passes that code through by name. A write that adds no new daily key quota never calls the lookup: account limits, legacy leaves, a changed or unchanged quota, a reset.
  • Remove then re-add. A quota removed in one write and added back in a later one finds no old leaf, so it is a new quota and gets now. Documented in the carried comment and the READMEs.
  • Docs. packages/plugins/usage/cliproxy-queue/README.md: the last paragraph's "stamps a new since on a renamed leaf ... restarts its count" is replaced by the carry rule, the remove-then-re-add rule and the lookup refusal. packages/plugins/README.md (daily-quota section) and packages/plugins/limit-check/daily-quota/README.md (model-key section) gain the carry sentence. packages/business/README.md had no since rule and, as the R3 review said, no model-key leaf; it now has a paragraph after launch.limits.tokens.daily on the leaf, the writer's since rules with the carry, and keyLineage.

Choices and departures, and why:

  • Which leaves ask. Only a new leaf that is a daily key quota once stamped (isDailyModelQuota with since set to now). A twelve-hex-named plain {limit} leaf is a legacy account limit under row 333's rule, so it keeps the plain new-leaf rule and the lookup is not called. Source leaves must be daily key quotas too, so a legacy leaf's since never carries into a daily quota.
  • A leaf that changes shape at the same path, a legacy {limit, since} becoming a daily quota, keeps its stored since under the existing same-path rule. Unchanged; found while writing the default-lookup case, which now adds a fresh id.
  • Earliest against latest. The business test separates them: A from T0, B added from A (T0) and then reset to T1, both moved to C, and C takes T0 where the latest would give T1. Mutant M3 shows the test tells them apart.
  • The real-runtime test uses a re-issue, not a rotation. The runtime keeps its ModelKeys inside and exposes no rotate, and its sweep rotates only at rotateBy, 30 days after issue, which no daily window spans without mocking the runtime's clock. The re-issue gives the same shape of lineage (B current, A among the earlier ids) and goes through the real startBroker wiring, the broker's limits.set and the runtime's own lookup. The rotation path through the writer is p5-key-lineage's R4 writer test, with the real Credentials, ModelKeys, createLimits and daily-quota module.
  • The R3 "settings-writer move" test, which pinned a count of 100, now requires 160 and is renamed for R4. The R3 departure 3 and its "Not done" item are superseded by this ruling.

Ruling 2: the orphan set on every refresh

observeOrphans (limits.mjs) runs in refresh, which the runtime calls after each committed verb and at boot, and in orphanedKeys. With no daily key quota left it stores an empty set for the business; otherwise it reads the lineages and calls noteOrphans, which logs each id not in the stored set and stores the current set. When the lineages can't be read it leaves the stored set as it was and logs nothing. So a quota removed and added back logs again, and an unchanged orphan logs once however many refreshes see it. The set is in memory: a restart is a fresh observation, and a quota still orphaned after it logs once more. No marker is persisted. This is in the orphansLogged comment and the cliproxy-queue README.

Consequence: the orphan line now comes at the first refresh that sees the orphan, which can be boot or the refresh at the start of a launch decision, before that decision's own refusal line. Two R3 tests asserted the old order and now expect the orphan line first: the fire and re-hire test (the line comes at the keyless launch before the re-hire, not at the first launch after it) and the ruling 2 test. The number of orphan lines is unchanged in both.

Ruling 3: the barrier's last yield

p5-telemetry-fixture.mjs captures realSetImmediate at module load with the other real timers, and pollSettledAfterTick yields on it inside the deadline, so a mocked setImmediate can't hold the barrier and the deadline covers the whole wait. Both options in the ruling, since together they cost nothing. The new p5-background test mocks Date, setInterval, setTimeout and setImmediate, advances one poll tick through the barrier, never ticks again, and requires the barrier to settle within a real bound of twice WAIT_MS, with the second queue read seen and the proxy state available. Against the R3 fixture it fails "pollSettledAfterTick did not settle" (about 10 s, the outer bound); I checked that before keeping the test.

Ruling 4: records

  • The 64-id bound is a limit. A lineage holds the current id, the retiring ids and the newest 64 earlier ids. Once an id falls out of the newest 64, a quota named by it is in no lineage, so it is orphaned while the agent stays hired: it limits nothing and the usage view lists it. Moving the quota to a current id fixes it, and since the old id is no longer in the lineage the writer can't carry its since, so the moved quota starts at now. Recorded here and in the cliproxy-queue README.
  • R3 departure 4, corrected. Its reason, that proxy queue events "can only name keys the proxy still accepts, so earlier ids would never match", is wrong. A queued event the proxy accepted before the key left the record and that the bus drains after it names an id that is no longer current or retiring. proxy-usage.mjs:42-67 builds its owner map from current and retiring ids only, so that event goes to the reconcile log as proxy-usage-unknown-key and is not recorded. That loses telemetry attribution for the event. It is not a quota bypass: quotas count the native transcript records, which carry the run's bound key id. The owner map is unchanged in R4.

Tests

Added:

  • packages/bus/tests/p5-key-lineage.test.mjs. The R3 writer test becomes the R4 writer test: A's quota is set at the start, the agent uses 60 on A, rotates, the human moves the unchanged quota to B, B's since equals A's, and after 30 more on A and 70 on B the launch refuses with count 160 and the report shows 160; a reset after the move sets now, the count is 0 and the launch goes ahead. New: a lineage that can't be read (an id in two agents' lineages) refuses a write adding a key quota with proxy-key-refused, the file unchanged, and a write adding none (a changed limit and a legacy hex leaf) saves without asking. New: an orphaned quota removed and added back logs twice, three more refreshes, a launch and orphanedKeys add nothing, and after a restart three refreshes add exactly one.
  • packages/bus/tests/p5-runtime.test.mjs. Real startBroker, fake loopback proxy: a 2-token quota on A, a turn spends 2 on A, the launch refuses, the proxy stops listing A, modelPreflight issues B, limits.set moves the unchanged quota to B with no reset, the stored leaf is B with A's since, no key value is in the limits view, and the next mayLaunch refuses.
  • packages/bus/tests/p5-background.test.mjs. The ruling 3 barrier test.
  • packages/business/tests/token-limit-settings.test.mjs, with a stub lookup. One test: B added beside A carries A's time; earliest over latest; an id in no lineage gets now; a caller-supplied since refuses on a new leaf, a moved leaf and with the carried time itself; a reset in the move write sets now; removed and added back gets now. The other: no lookup for account limits, legacy hex leaves, a changed quota or a reset; a lookup that throws (with and without a code) or answers a string, a list holding a number, a list holding an uppercase id, a list without the asked id or an object refuses proxy-key-refused with the file unchanged; the default writer refuses a new daily key quota and saves account and legacy limits.

Changed: the existing P5 business test passes a lookup answering [], since a writer without one now refuses that leaf. In p5-key-lineage, the R3 fire and re-hire test and the ruling 2 test expect the orphan line first (Ruling 2 above).

Runs

One at a time, each under a 30-minute limit (timeout -k 10 1800 node --test), Node v26.8.1, with TMPDIR under /mnt/storage/scratch/tmp and output to files there. Each whole run is node --test over the package's tests directory. All times are 2026-11-03 UTC. The first runs were on the working tree that became f65faf9df, committed with no edit between; the cli stack runs and the closing whole-package runs were on f65faf9df itself, the latter after the mutants were restored.

Run Tree Tests Pass Fail Skip Time
bus whole before commit 1397 1397 0 0 50 s, 00:55:51Z to 00:56:41Z
plugins whole before commit 132 132 0 0 under 1 s, 00:56:53Z
business whole before commit 295 295 0 0 4 s, 00:56:53Z to 00:56:57Z
webui whole, run 1 before commit 499 498 1 0 88 s, 00:57:00Z to 00:58:28Z
webui editor-browser.test.mjs alone before commit 16 16 0 0 15 s, 00:58:47Z to 00:59:02Z
webui whole, run 2 before commit 499 499 0 0 119 s, 00:59:06Z to 01:01:05Z
cli launch-limit-checks.test.mjs before commit 5 5 0 0 under 1 s, 01:01:23Z
cli stack-release.test.mjs f65faf9df 225 225 0 0 741 s, 01:02:50Z to 01:15:11Z
cli stack-release-external.test.mjs f65faf9df 25 25 0 0 98 s, 01:15:25Z to 01:17:03Z
cli stack-upgrade.test.mjs f65faf9df 25 25 0 0 148 s, 01:17:07Z to 01:19:35Z
harness tools.test.mjs f65faf9df 23 23 0 0 under 1 s, 01:20:41Z
bus whole f65faf9df 1397 1397 0 0 54 s, 01:22:35Z to 01:23:29Z
plugins whole f65faf9df 132 132 0 0 1 s, 01:23:29Z to 01:23:30Z
business whole f65faf9df 295 295 0 0 4 s, 01:23:30Z to 01:23:34Z
webui whole f65faf9df 499 499 0 0 131 s, 01:23:38Z to 01:25:49Z

The webui run 1 failure was editor-browser.test.mjs:93, the CSP editor test: Condition timed out: !document.querySelector("#doc-save").disabled at :122, with the host's load average between 45 and 60. R4 changes nothing webui loads. The file passed alone and the whole package passed twice after, so I read it as load, not R4; both runs are listed. The model-final-completion.test.mjs:69 flake (16 !== 19) did not appear in either bus run. Against R3, bus is up 4 (p5-key-lineage's lookup-failure and orphan re-add tests, the p5-runtime move test, the p5-background barrier test; the writer test is renamed, not added) and business up 2. Plugins and webui are unchanged.

Pins. No test pins any changed path with a hash: the grep of every test, script and fixture naming a changed file's basename for a 64-hex string found only packages/queue/tests/fixtures/queue-marked.md, which names none of the four READMEs. The tests that read a changed file by path (limit-checks.test.mjs:319 and :323, host-run.test.mjs:429) are in packages/bus, run whole. Outside the four packages, harness tools.test.mjs imports SETTINGS_KEYS from launch-settings.mjs and ran whole (above). Nineteen packages/cli test files import startBroker from bus runtime.mjs; they import, they don't pin, and beyond the four cli files the brief names they were not run (Not done).

Mutants

Each was applied to f65faf9df, run against the named files, and restored with git checkout -- <file>; git status --short was empty after each.

Mutant Change Killed by Result
M1 launch-settings.mjs:73 entry.since = kept ?? now;, so stamp ignores the lineage p5-key-lineage.test.mjs:173, the writer test, at :185: B's since was the move time 2026-10-24T12:01:00.000Z where A's 2026-10-24T12:00:00.000Z was expected. p5-runtime.test.mjs:62 at :106: the stored B leaf had the move time, not A's. token-limit-settings.test.mjs:178 at :184: B took the write time 08:00 where A's 07:00 was expected. Also p5-key-lineage :198 at :206 and token-limit-settings :204 at :212, both because the lookup is never called (asked was 0 where 1 was expected, and [] where [A] was) p5-key-lineage 2 of 11 fail, p5-runtime 1 of 2, token-limit-settings 2 of 9
M2 runtime.mjs:346 passes launchSettings no keyLineage p5-runtime.test.mjs:62 at :88: proxy-key-refused from the broker at the first limits.set that adds a key quota, since the writer's default lookup refuses. The R3 P5 runtime test at :39 fails the same way p5-runtime 2 of 2 fail
M3 launch-settings.mjs:96 sorts descending, so the latest since wins token-limit-settings.test.mjs:178 at :189: C had since 08:00 (T1, B's reset time) where 07:00 (T0, A's) was expected token-limit-settings 1 of 9 fail. p5-key-lineage and p5-runtime pass (13 of 13): each of their moves has one source leaf, so earliest and latest are the same there
M4 limits.mjs:130, refresh no longer calls observeOrphans p5-key-lineage.test.mjs:348, the orphan re-add test, at :358: one limit-key-orphaned line where two were expected; the set was never cleared, so the re-added leaf was still remembered. Also the R3 fire and re-hire test at :299 and the R3 ruling 2 test at :341, each missing the orphan line the refresh now logs first p5-key-lineage 3 of 11 fail
M4b limits.mjs:90 returns [] with no daily key quota but leaves the remembered set alone p5-key-lineage.test.mjs:348 at :358, the same one-line-for-two failure p5-key-lineage 1 of 11 fail
M5 p5-telemetry-fixture.mjs restored to its R3 content p5-background.test.mjs:71, the barrier test: pollSettledAfterTick did not settle after 10 s p5-background 1 of 5 fail

M4b and M5 are extra. M4b shows the empty set itself is needed, not only the refresh call. M5 shows the ruling 3 test fails on the R3 barrier, whose last yield used the mocked setImmediate outside the deadline.

Not done

  • No push, queue, Vikunja, Gitea or PR action, per the brief.
  • No migration or persisted orphan marker, per the rulings.
  • The proxy-usage owner map still holds only current and retiring ids (Ruling 4 above).
  • The nineteen packages/cli test files that import bus runtime.mjs were not run beyond the four the brief names: they import it and pin nothing, and the runtime change adds only the lookup, used when a write adds a daily key quota.