mosaicstack/stack
1
0
Fork 1
Code Issues 200 Pull Requests 10 Actions Packages Projects Releases 15 Wiki Activity

flaky: test-fleet-units.sh 'contaminated loader' fails ~1 in 50 on clean main (pre-existing, blocks unrelated PRs) #1096

New Issue
Open
opened 2026-08-07 06:51:00 +00:00 by Mos · 0 comments
Mos commented 2026-08-07 06:51:00 +00:00
Contributor
Copy Link
Copy Source

test-fleet-units.sh has an intermittent race: "holder execution triggered a contaminated loader"

Pre-existing on main. Not introduced by any current PR.

Evidence

Measured by be-coder-06 while rebasing #1073, on clean main @ aa0a7b5f, without PR changes:

main, exact archive:      first 20 runs → 0 failures
main, repeated-until-fail → FAIL at attempt 53 (of 200 max)
                            "FAIL: holder execution triggered a contaminated loader"
rebased #1073 branch:      identical failure 1/20, passed on retry

The PR's only diff in test-fleet-units.sh adds MOSAIC_GIT_IDENTITY after the failure point, and start-tmux-holder.sh has no PR delta. The failure is upstream of both.

Why this matters

Roughly 1-in-50, and it can land on any PR. It is the second intermittent found tonight that blocks unrelated work — #1090 was a wall-clock tolerance in apps/gateway that held #1085 red for hours. This one is in the framework's own systemd suite, which every framework PR runs.

An intermittent that fails ~2% of runs will, across enough PRs, look like a different problem every time it appears.

What is not established

  • Root cause. "Contaminated loader" is the assertion's own message; nobody has traced what actually leaks. The suite asserts the holder unit must use env -i … --noprofile --norc and must not start tmux through a login shell — so the check exists precisely to catch environment bleed, and something intermittently bleeds.
  • Whether it is the same root cause as the installed-unit divergence in #1092. web1's installed holder unit uses /bin/bash -lc (a login shell, no env -i) where the canonical unit uses env -i … --noprofile --norc. That is consistent in shape, but the suite reads the repo's unit file, not the installed one, so it is not obviously the same thing. I have not tested the connection and am not asserting it.
  • How long it has been present.

Safety note for whoever picks this up

be-coder-06 reproduced this using isolated tmux sockets and did not restart systemd or touch the live holder/owner file. That caution is required, not optional: per #1092, systemctl --user restart mosaic-tmux-holder runs kill-server on the fleet socket and would destroy 39 live sessions. Do not reproduce this by restarting the live unit.

Effect on #1073

None. #1073 is unblocked by this finding — its rebase is clean and the failure is pre-existing. It should not be held for a main defect it did not cause.

Found by be-coder-06, using repeated-until-fail rather than accepting a single green run — which is the only instrument that surfaces a 1-in-50.

## `test-fleet-units.sh` has an intermittent race: "holder execution triggered a contaminated loader" Pre-existing on `main`. Not introduced by any current PR. ## Evidence Measured by `be-coder-06` while rebasing #1073, on **clean `main` @ `aa0a7b5f`, without PR changes**: ``` main, exact archive: first 20 runs → 0 failures main, repeated-until-fail → FAIL at attempt 53 (of 200 max) "FAIL: holder execution triggered a contaminated loader" rebased #1073 branch: identical failure 1/20, passed on retry ``` The PR's only diff in `test-fleet-units.sh` adds `MOSAIC_GIT_IDENTITY` **after** the failure point, and `start-tmux-holder.sh` has no PR delta. The failure is upstream of both. ## Why this matters **Roughly 1-in-50, and it can land on any PR.** It is the second intermittent found tonight that blocks unrelated work — #1090 was a wall-clock tolerance in `apps/gateway` that held #1085 red for hours. This one is in the framework's own systemd suite, which every framework PR runs. An intermittent that fails ~2% of runs will, across enough PRs, look like a different problem every time it appears. ## What is not established - **Root cause.** "Contaminated loader" is the assertion's own message; nobody has traced what actually leaks. The suite asserts the holder unit must use `env -i … --noprofile --norc` and must not start tmux through a login shell — so the check exists precisely to catch environment bleed, and something intermittently bleeds. - **Whether it is the same root cause as the installed-unit divergence in #1092.** web1's installed holder unit uses `/bin/bash -lc` (a login shell, no `env -i`) where the canonical unit uses `env -i … --noprofile --norc`. That is consistent in shape, but the suite reads the **repo's** unit file, not the installed one, so it is not obviously the same thing. **I have not tested the connection and am not asserting it.** - **How long it has been present.** ## Safety note for whoever picks this up `be-coder-06` reproduced this using **isolated tmux sockets** and did **not** restart systemd or touch the live holder/owner file. That caution is required, not optional: per #1092, `systemctl --user restart mosaic-tmux-holder` runs `kill-server` on the fleet socket and would destroy 39 live sessions. **Do not reproduce this by restarting the live unit.** ## Effect on #1073 None. `#1073` is unblocked by this finding — its rebase is clean and the failure is pre-existing. It should not be held for a `main` defect it did not cause. Found by `be-coder-06`, using repeated-until-fail rather than accepting a single green run — which is the only instrument that surfaces a 1-in-50.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 f10-coder jason.woltje (Jason Woltje) merge-gate mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-security-02 scooby (Scooby)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1096