mosaicstack/stack
25
0 Fork 1
Code Issues 406 Pull Requests 13 Packages Projects Releases 15 Wiki Activity

P1: library reply retry usage under ${id}:retry is refused by the broker; retry tokens go unrecorded #1853

New Issue
Open
opened 2026-11-06 21:40:36 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-11-06 21:40:36 +00:00
Owner
Copy link
Copy source

Summary

The runner's library reply retry records its usage under the turn id ${m.id}:retry. The broker refuses that id, so the tokens the retry turn uses are never recorded. The campaign budget supervisor and the limit checks count usage_records, so a cap undercounts by whatever retry turns spend.

Found while ruling on the #1852 fix row. It is not part of that row.

Code (read at 3598839e6)

  • packages/harness/src/runner.mjs:708 calls client.call("usage.begin", { turn: \${m.id}:retry` }), and :712callsusage.end` with the same id. This happens on the library path, after a reply is refused with a guard code or request-too-large, before the one retry turn.

  • packages/bus/src/broker.mjs:6012 (#requestUsage) accepts a turn id only if:

    • it is a message delivered to this run (messages JOIN deliveries ... d.op='delivered' AND d.holder_run=?), or
    • for a run with no thread, it is a landing decision the run holds (begin), or it already has a begin row (end).

    ${m.id}:retry is neither, so both calls fail with not-recipient (:6025). The id passes the format check (broker.mjs:283 allows :).

  • The runner catches both refusals and only logs runner: usage baseline unavailable / usage outcome unavailable (runner.mjs:709, :713). No usage_records row is written for the retry. The main turn's begin/end pair on m.id closed at :663, before the retry ran, so its delta (packages/bus/src/usage.mjs delta()) doesn't include the retry's transcript entries either.

  • #limitCheck runs after a successful usage.end, so no limit check runs after the retry turn.

Live check

In the ui797 store on .53 (read-only, 2026-11-06T21:4xZ), there are 0 refused usage events with a :retry turn and 0 usage_records with a :retry turn. So no retry has run on ui797 so far. This is a code-path defect, not one seen in the incident data.

Effect

Every library reply retry is a full model turn whose tokens are missing from usage_records. Campaign caps, token limits and usage views undercount by that amount, and the next limit check comes later than it should. The undercount is silent: the only sign is a runner log line.

Fix direction

The smallest change follows the #1852 approach: keep the retry inside the m.id usage bracket. Move the main turn's usage.end (m.id) to after the retry, and drop the ${m.id}:retry calls. The single end row's transcript delta then covers both turns, with no broker contract change. Tests: one begin and one end for m.id around both turns, and the end row's delta includes the retry's entries.

Before a Round B go

Per the Lead: this is on record before any Round B rerun on @campaign5, because unrecorded retry tokens make the cap undercount.

## Summary The runner's library reply retry records its usage under the turn id `${m.id}:retry`. The broker refuses that id, so the tokens the retry turn uses are never recorded. The campaign budget supervisor and the limit checks count usage_records, so a cap undercounts by whatever retry turns spend. Found while ruling on the #1852 fix row. It is not part of that row. ## Code (read at 3598839e6) - `packages/harness/src/runner.mjs:708` calls `client.call("usage.begin", { turn: \`${m.id}:retry\` })`, and `:712` calls `usage.end` with the same id. This happens on the library path, after a reply is refused with a guard code or request-too-large, before the one retry turn. - `packages/bus/src/broker.mjs:6012` (`#requestUsage`) accepts a turn id only if: - it is a message delivered to this run (`messages JOIN deliveries ... d.op='delivered' AND d.holder_run=?`), or - for a run with no thread, it is a landing decision the run holds (begin), or it already has a begin row (end). `${m.id}:retry` is neither, so both calls fail with `not-recipient` (`:6025`). The id passes the format check (`broker.mjs:283` allows `:`). - The runner catches both refusals and only logs `runner: usage baseline unavailable` / `usage outcome unavailable` (`runner.mjs:709`, `:713`). No usage_records row is written for the retry. The main turn's begin/end pair on `m.id` closed at `:663`, before the retry ran, so its delta (`packages/bus/src/usage.mjs` `delta()`) doesn't include the retry's transcript entries either. - `#limitCheck` runs after a successful `usage.end`, so no limit check runs after the retry turn. ## Live check In the ui797 store on .53 (read-only, 2026-11-06T21:4xZ), there are 0 refused usage events with a `:retry` turn and 0 usage_records with a `:retry` turn. So no retry has run on ui797 so far. This is a code-path defect, not one seen in the incident data. ## Effect Every library reply retry is a full model turn whose tokens are missing from usage_records. Campaign caps, token limits and usage views undercount by that amount, and the next limit check comes later than it should. The undercount is silent: the only sign is a runner log line. ## Fix direction The smallest change follows the #1852 approach: keep the retry inside the `m.id` usage bracket. Move the main turn's `usage.end` (`m.id`) to after the retry, and drop the `${m.id}:retry` calls. The single end row's transcript delta then covers both turns, with no broker contract change. Tests: one begin and one end for `m.id` around both turns, and the end row's delta includes the retry's entries. ## Before a Round B go Per the Lead: this is on record before any Round B rerun on @campaign5, because unrecorded retry tokens make the cap undercount.
jason.woltje added the bug label 2026-11-06 21:40:36 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate mosaic-stack-coder-bot (mosaic-stack coder bot) mosaic-stack-cto-bot (mosaic-stack cto bot) mosaic-stack-pm-bot (mosaic-stack pm bot) ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1853