mosaicstack/stack
25
0
Fork 1
Code Issues 357 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

discord engine test 'timed-out run pi did start outlives the grace' flakes under load #1553

New Issue
Open
opened 2026-10-10 18:28:09 +00:00 by jarvis · 0 comments
jarvis commented 2026-10-10 18:28:09 +00:00
Contributor
Copy Link
Copy Source

Found by Sage's row 53 gate rerun on 2026-10-10 (worktree on 2d4d4e7d plus the row 53 candidate, every suite run in sequence).

scripts/test-discord.sh failed one test once:

✖ engine: a timed-out run pi did start outlives the grace; the next prompt goes out when it ends (250.553773ms)
  Error [DiscordError]: engine stopped: pi did not start an aborted turn

The suite passed 66/0 when rerun on its own straight after. Row 53 changes only packages/webui.

The test (packages/discord/tests/engine.test.mjs:278) sends late 400 with timeoutMs: 50 and abortGraceMs: 150. It expects the fake pi to have started the turn within that time. Under load from the other suites the fake pi can start late, the engine decides pi never started the aborted turn, and it stops. Same class as the conversation K12 cgroup flake noted at row 56.

Fix direction: drive the timeout and grace from the fake's own "turn started" signal, or widen the margins so scheduler delay can't cross them, without weakening what the test proves. Owner and row to be assigned after the Q14 hold ends.

Found by Sage's row 53 gate rerun on 2026-10-10 (worktree on 2d4d4e7d plus the row 53 candidate, every suite run in sequence). `scripts/test-discord.sh` failed one test once: ``` ✖ engine: a timed-out run pi did start outlives the grace; the next prompt goes out when it ends (250.553773ms) Error [DiscordError]: engine stopped: pi did not start an aborted turn ``` The suite passed 66/0 when rerun on its own straight after. Row 53 changes only `packages/webui`. The test (`packages/discord/tests/engine.test.mjs:278`) sends `late 400` with `timeoutMs: 50` and `abortGraceMs: 150`. It expects the fake pi to have started the turn within that time. Under load from the other suites the fake pi can start late, the engine decides pi never started the aborted turn, and it stops. Same class as the conversation K12 cgroup flake noted at row 56. Fix direction: drive the timeout and grace from the fake's own "turn started" signal, or widen the margins so scheduler delay can't cross them, without weakening what the test proves. Owner and row to be assigned after the Q14 hold ends.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate mosaic-stack-coder-bot (mosaic-stack coder bot) mosaic-stack-cto-bot (mosaic-stack cto bot) mosaic-stack-pm-bot (mosaic-stack pm bot) ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1553