mosaicstack/stack
25
0 Fork 1
Code Issues 397 Pull Requests 13 Packages Projects Releases 15 Wiki Activity

P1: duplicate continuation notices drive repeated instance-running launch refusals (ui797, 13 refusals) #1848

New Issue
Closed
opened 2026-11-06 19:48:02 +00:00 by jason.woltje · 1 comment
jason.woltje commented 2026-11-06 19:48:02 +00:00
Owner
Copy link
Copy source

Filed by the MVP3 lead #797 agent for the Lead (Mos), key roundb-2-halt. Source: ui797 Round B, campaign3, 2026-11-06. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/c3halt/busnotice2.txt (notices interleaved with launches and refusals) and haltrep.txt. Store /srv/mosaic/.mosaic-ui797/bus/bus.sqlite on .53, read only. Support tree 1186cf7b1.

What happened

Chief rebc27bb0b748 got 13 instance-running launch refusals between 19:22:45Z and 19:25:57Z: events seq 2672, 2677, 2683, 2686, 2689, 2691, 2695, 2698, 2701, 2704, 2709, 2712, 2715. They rotated through pm (3), cto (5) and reviewer (5). Every refusal fell in its own chief turn (usage seq 601-656), and those 13 turns cost 1,198,607 tokens.

Correction to the first report (19:33:41Z): I called these model-level retries with no code loop. The store shows something else. Each refusal answers a separate stale _bus notice that the chief read on that turn. The notices are duplicates generated by the bus.

The notices

Three request continuations were waiting when campaign2 halted: 624f7981 (to the reviewer), f599ccba (to the pm) and 5bd5ec27 (to the cto). The bus sent the chief a "Message X from _bus to Y waits: Y has no running session. Launch Y…" notice (#notice, broker.mjs:8163) at each of these times:

  • 19:00:04 (seq 303, cto)
  • 19:01:00 (seq 311-313) and 19:01:02 (seq 314-316): after the kill-switch session.ended events at 19:00:59, while processing was halted
  • 19:15:05 (seq 317-319): 1 s after chief rebc27bb0b748 launched
  • 19:16:51 (seq 320, 322): when pm r605148e4905b launched
  • 19:17:14 (seq 325, 326): when assistant r1ee370991943 launched
  • 19:18:10 (seq 328): when cto r38bcce184bdb launched

That is 15 notices for 3 waiting messages. The reviewer's 624f7981 alone got 6 (seq 311, 314, 317, 320, 325, 328). The chief read them one per turn from 19:22:30 to 19:26:04, after the targets were already up: pm since 19:16:51, cto since 19:18:09, reviewer since 19:22:17. On each one it called launch as the notice told it to, and got instance-running.

Cause (code at 1186cf7b1)

  • request-continuation.mjs attempt() (around lines 181-190) re-notifies through continuationRetry whenever the token waiting:<prerequisite> is new. The comment says "one chief notice per prerequisite version, not per tick".
  • #continuationContext (broker.mjs, around line 700) builds the prerequisite digest from revision = max seq of all session.launched/session.ended/launch.* events in the business. So any role's launch or end, including the kill-switch ends during a halt, makes a new version and a new notice for every waiting continuation, whatever the target's state.
  • The comment above #notice (broker.mjs:8160) says "One notice per waiting message". The continuation retry path breaks that.
  • A notice already queued is not withdrawn or marked stale when its target launches, so the chief reads it later and acts on it.
  • The tool error the chief sees is refused: instance-running: already running; wait for the existing run (harness tools.mjs:432). It doesn't say that the running instance already holds the waiting message, or that the notice can be dropped. The chief acknowledged each one, relayed it to the assistant as an INFO (which feeds the echo loop in the companion issue), and moved on to the next duplicate.

MVP fix scope (fix row with the echo-loop defect)

The smallest change that stops the loop, plus a regression test that replays this sequence (halt with 3 waiting continuations → kill-switch ends → resume → staggered launches of chief/pm/assistant/cto/reviewer). The test should show:

  1. At most one chief notice per waiting message while its target's own state is unchanged, and none generated while processing is halted.
  2. An instance-running refusal for a target that holds or will receive the waiting message tells the chief so, and tells it not to relaunch.

The builder picks the mechanism. Keying the prerequisite on the target role's own holder/launch state rather than the business-wide session revision is one candidate.

P1.

Filed by the MVP3 lead #797 agent for the Lead (Mos), key roundb-2-halt. Source: ui797 Round B, campaign3, 2026-11-06. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/c3halt/busnotice2.txt (notices interleaved with launches and refusals) and haltrep.txt. Store /srv/mosaic/.mosaic-ui797/bus/bus.sqlite on .53, read only. Support tree 1186cf7b1. ## What happened Chief rebc27bb0b748 got 13 `instance-running` launch refusals between 19:22:45Z and 19:25:57Z: events seq 2672, 2677, 2683, 2686, 2689, 2691, 2695, 2698, 2701, 2704, 2709, 2712, 2715. They rotated through pm (3), cto (5) and reviewer (5). Every refusal fell in its own chief turn (usage seq 601-656), and those 13 turns cost 1,198,607 tokens. Correction to the first report (19:33:41Z): I called these model-level retries with no code loop. The store shows something else. Each refusal answers a separate stale `_bus` notice that the chief read on that turn. The notices are duplicates generated by the bus. ### The notices Three request continuations were waiting when campaign2 halted: 624f7981 (to the reviewer), f599ccba (to the pm) and 5bd5ec27 (to the cto). The bus sent the chief a "Message X from _bus to Y waits: Y has no running session. Launch Y…" notice (`#notice`, broker.mjs:8163) at each of these times: - 19:00:04 (seq 303, cto) - 19:01:00 (seq 311-313) and 19:01:02 (seq 314-316): after the kill-switch session.ended events at 19:00:59, while processing was halted - 19:15:05 (seq 317-319): 1 s after chief rebc27bb0b748 launched - 19:16:51 (seq 320, 322): when pm r605148e4905b launched - 19:17:14 (seq 325, 326): when assistant r1ee370991943 launched - 19:18:10 (seq 328): when cto r38bcce184bdb launched That is 15 notices for 3 waiting messages. The reviewer's 624f7981 alone got 6 (seq 311, 314, 317, 320, 325, 328). The chief read them one per turn from 19:22:30 to 19:26:04, after the targets were already up: pm since 19:16:51, cto since 19:18:09, reviewer since 19:22:17. On each one it called launch as the notice told it to, and got `instance-running`. ### Cause (code at 1186cf7b1) - request-continuation.mjs `attempt()` (around lines 181-190) re-notifies through `continuationRetry` whenever the token `waiting:<prerequisite>` is new. The comment says "one chief notice per prerequisite version, not per tick". - `#continuationContext` (broker.mjs, around line 700) builds the prerequisite digest from `revision` = max seq of all `session.launched`/`session.ended`/`launch.*` events in the business. So any role's launch or end, including the kill-switch ends during a halt, makes a new version and a new notice for every waiting continuation, whatever the target's state. - The comment above `#notice` (broker.mjs:8160) says "One notice per waiting message". The continuation retry path breaks that. - A notice already queued is not withdrawn or marked stale when its target launches, so the chief reads it later and acts on it. - The tool error the chief sees is `refused: instance-running: already running; wait for the existing run` (harness tools.mjs:432). It doesn't say that the running instance already holds the waiting message, or that the notice can be dropped. The chief acknowledged each one, relayed it to the assistant as an INFO (which feeds the echo loop in the companion issue), and moved on to the next duplicate. ## MVP fix scope (fix row with the echo-loop defect) The smallest change that stops the loop, plus a regression test that replays this sequence (halt with 3 waiting continuations → kill-switch ends → resume → staggered launches of chief/pm/assistant/cto/reviewer). The test should show: 1. At most one chief notice per waiting message while its target's own state is unchanged, and none generated while processing is halted. 2. An `instance-running` refusal for a target that holds or will receive the waiting message tells the chief so, and tells it not to relaunch. The builder picks the mechanism. Keying the prerequisite on the target role's own holder/launch state rather than the business-wide session revision is one candidate. P1.
jason.woltje added the bug label 2026-11-06 19:48:02 +00:00
jason.woltje commented 2026-11-06 20:46:38 +00:00
Author
Owner
Copy link
Copy source

Fixed on refactor; pushed by the Lead at 20:45Z on 2026-11-06 (origin/refactor 1186cf7b1..3598839e6, fast-forward). Posted by the MVP3 lead #797 agent.

  • 4048e0f60 fix(bus,harness): a continuation's prerequisite version uses only the target role's own session.launched/session.ended events, launch.revoked/launch.restored, and the target's own role_claims. Unrelated launches and ends, including kill-switch ends, no longer create a new version. No notice is sent while processing is halted (the retry refuses and rolls back) or while the target's launch has not ended. The instance-running refusal now says the running instance receives its waiting messages when it claims, and that the chief must not launch it again or relay the notice. Regression test in packages/bus/tests/launch-notice.test.mjs, which fails on 1186cf7b1.
  • c04529c7b fix(bus): two Sol xhigh review findings.
    • P1: the new halted-retry refusal escaped the unconditional startup recover(), so a halted broker could not restart with a waiting continuation. Startup recovery now waits while processing is halted or invalid, as the 30-second pass does, and processing.resume schedules it again.
    • P2: any older launch with no end suppressed notices, so one lost end silenced every later run's end. Now only the target's latest lifecycle event counts.
    • Two regression tests in packages/bus/tests/request-continuation.test.mjs; both fail on 4048e0f60.

Residual: if the target's latest launch has lost its end, notices for that target stay suppressed until host recovery writes host-lost. That waits on #1851.

Gate on c04529c7b, Node 26.12.1:

  • touched 123/0
  • bus except project-files 1648/0
  • harness 318/0
  • webui 679/0 on rerun

Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/fix-1847-1848-gate/.

Fixed on refactor; pushed by the Lead at 20:45Z on 2026-11-06 (origin/refactor 1186cf7b1..3598839e6, fast-forward). Posted by the MVP3 lead #797 agent. - 4048e0f60 fix(bus,harness): a continuation's prerequisite version uses only the target role's own `session.launched`/`session.ended` events, `launch.revoked`/`launch.restored`, and the target's own role_claims. Unrelated launches and ends, including kill-switch ends, no longer create a new version. No notice is sent while processing is halted (the retry refuses and rolls back) or while the target's launch has not ended. The `instance-running` refusal now says the running instance receives its waiting messages when it claims, and that the chief must not launch it again or relay the notice. Regression test in packages/bus/tests/launch-notice.test.mjs, which fails on 1186cf7b1. - c04529c7b fix(bus): two Sol xhigh review findings. - P1: the new halted-retry refusal escaped the unconditional startup `recover()`, so a halted broker could not restart with a waiting continuation. Startup recovery now waits while processing is halted or invalid, as the 30-second pass does, and `processing.resume` schedules it again. - P2: any older launch with no end suppressed notices, so one lost end silenced every later run's end. Now only the target's latest lifecycle event counts. - Two regression tests in packages/bus/tests/request-continuation.test.mjs; both fail on 4048e0f60. Residual: if the target's latest launch has lost its end, notices for that target stay suppressed until host recovery writes `host-lost`. That waits on #1851. Gate on c04529c7b, Node 26.12.1: - touched 123/0 - bus except project-files 1648/0 - harness 318/0 - webui 679/0 on rerun Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/step14/round-b/fix-1847-1848-gate/.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate mosaic-stack-coder-bot (mosaic-stack coder bot) mosaic-stack-cto-bot (mosaic-stack cto bot) mosaic-stack-pm-bot (mosaic-stack pm bot) ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1848