mosaicstack/stack
25
0 Fork 1
Code Issues 397 Pull Requests 13 Packages Projects Releases 15 Wiki Activity

P2: launcher drops a run from sessions.json when endLaunch is refused at shutdown (stale claim) #1851

New Issue
Open
opened 2026-11-06 19:48:04 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-11-06 19:48:04 +00:00
Owner
Copy link
Copy source

Filed by the MVP3 lead #797 agent for the Lead (Mos), key roundb-2-halt. Source: uidev, 2026-11-06 18:24Z; the full write-up is in /mnt/storage/scratch/mosaic-lead/uidev/dev-owner-report.md, section "Stale chief claim on uidev". Support tree 1186cf7b1.

Symptom

After uidev-bus moved to a new host (dmfQbb), chief run r0ede41bc6d13 ended claim-refused (runner exit 21). role_claims still showed the chief claimed by r05526af39b1d, the run that was killed when the earlier host stopped.

Cause (launcher)

At host stop, the broker exited first, so endLaunch for r05526af39b1d was refused (broker-exited). The launcher logged "ended: killed" and dropped the run from bus-host/sessions.json anyway. The next host start's recover() reads only sessions.json, so it never called recoverLaunch, and the claim stayed with a dead run. mosaic stop reads the same file, so it could not clear the claim either.

Recovery done by hand (uidev only)

  1. Stopped uidev-bus.
  2. Wrote the dead run's entry back into sessions.json, with the exact launch proof from the broker's own session.launched event: pid 1597281, startTime "6766001", harness claude-code. Both pids were confirmed gone, so nothing was signalled.
  3. Restarted uidev-bus. The log shows "run r05526af39b1d (chief) from an earlier host ended: host-lost" and "launched chief: run re2b5fa809c88". role_claims: claim (r055), release (r055), claim (re2b).

The previous file is saved as uidev/sessions.json.before-recover.

Fix

The launcher must keep a run in sessions.json when endLaunch is refused at shutdown, so the next host recovers it.

Regression test: stop the broker before endLaunch at host stop, then start a host and assert the claim is released with host-lost.

P2.

Filed by the MVP3 lead #797 agent for the Lead (Mos), key roundb-2-halt. Source: uidev, 2026-11-06 18:24Z; the full write-up is in /mnt/storage/scratch/mosaic-lead/uidev/dev-owner-report.md, section "Stale chief claim on uidev". Support tree 1186cf7b1. ## Symptom After uidev-bus moved to a new host (dmfQbb), chief run r0ede41bc6d13 ended `claim-refused` (runner exit 21). role_claims still showed the chief claimed by r05526af39b1d, the run that was killed when the earlier host stopped. ## Cause (launcher) At host stop, the broker exited first, so `endLaunch` for r05526af39b1d was refused (`broker-exited`). The launcher logged "ended: killed" and dropped the run from bus-host/sessions.json anyway. The next host start's `recover()` reads only sessions.json, so it never called `recoverLaunch`, and the claim stayed with a dead run. `mosaic stop` reads the same file, so it could not clear the claim either. ## Recovery done by hand (uidev only) 1. Stopped uidev-bus. 2. Wrote the dead run's entry back into sessions.json, with the exact launch proof from the broker's own session.launched event: pid 1597281, startTime "6766001", harness claude-code. Both pids were confirmed gone, so nothing was signalled. 3. Restarted uidev-bus. The log shows "run r05526af39b1d (chief) from an earlier host ended: host-lost" and "launched chief: run re2b5fa809c88". role_claims: claim (r055), release (r055), claim (re2b). The previous file is saved as uidev/sessions.json.before-recover. ## Fix The launcher must keep a run in sessions.json when endLaunch is refused at shutdown, so the next host recovers it. Regression test: stop the broker before endLaunch at host stop, then start a host and assert the claim is released with host-lost. P2.
jason.woltje added the bug label 2026-11-06 19:48:04 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate mosaic-stack-coder-bot (mosaic-stack coder bot) mosaic-stack-cto-bot (mosaic-stack cto bot) mosaic-stack-pm-bot (mosaic-stack pm bot) ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1851