mosaicstack/stack
14
0
Fork 1
Code Issues 315 Pull Requests 11 Actions Packages Projects Releases 15 Wiki Activity

test-fleet-units.sh: contaminated-loader assertion attributes fixture-server activity to the holder (flaky ~1 in 5 with tmux present) #1260

New Issue
Open
opened 2026-08-16 20:08:27 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-16 20:08:27 +00:00
Copy Link
Copy Source

test-fleet-units.sh blames the holder for loader activity it did not cause. Flaky at roughly 1 in 5 on a host with tmux, and it is an assertion defect rather than a holder defect.

The assertion

packages/mosaic/framework/systemd/user/test-fleet-units.sh:

: > "$MARKER"
if /usr/bin/env -i HOME="$HOLDER_HOME" ... "$HOLDER_START" >"$TEST_ROOT/holder.out" 2>&1; then
  fail "holder adopted contaminated named server"
fi
...
[ ! -s "$MARKER" ] || fail "holder execution triggered a contaminated loader"

The marker is truncated, the holder runs, the marker is checked. Any write in
that window is attributed to the holder.

Measurement

On sb-it-1-dt (origin/next, clean detached worktree), tmux and cc both present:

run 1 rc=1  FAIL: holder execution triggered a contaminated loader
run 2 rc=0
run 3 rc=0
run 4 rc=0
run 5 rc=1  FAIL: holder execution triggered a contaminated loader
run 6 rc=0
run 7 rc=0

Two of seven, always the same assertion. In a CI shape (/usr/bin minus tmux,
cc, systemd-analyze) the block self-skips and the suite is a clean rc=0.

The write is not the holder's

I rebuilt the fixture standalone — same marker.c, same contaminated
tmux new-session — then truncated the marker and ran no holder at all,
only the same observation window:

iter 1: marker WRITTEN with no holder involved (14 bytes)
iter 2..10: clean

1 in 10 with the holder entirely absent. The contaminated server does work of
its own inside the window — its LD_PRELOAD is in the server's global
environment, so anything it spawns loads the constructor and appends. The
assertion cannot tell that from a holder-caused load.

The other three assertions in that block are sound: the holder does fail, does
print the contract message, and does leave the server pid unchanged. Only the
attribution one is unsafe.

Why it is worth fixing rather than deleting

The property is real and worth keeping — the holder must not trigger the
contaminated loader. It needs to be attributed to the holder's own process tree
instead of to a time window. A marker keyed per-process, or comparing against a
control window with no holder, would both do it. I have not picked one; whoever
takes it should, since the fix is a design choice rather than a one-liner.

Why it matters now

#1017 task 2(b) asks whether tmux belongs in the CI image. PR #1259 enumerates
this suite, which is safe today precisely because no runner has tmux. Adding
tmux would import this flake. That is now measured rather than suspected, so
2(b) should be decided with it in view.

Found while verifying @scooby's enumeration finding for #1017.

-- fred (sb-it-1-dt)

`test-fleet-units.sh` blames the holder for loader activity it did not cause. Flaky at roughly 1 in 5 on a host with tmux, and it is an assertion defect rather than a holder defect. ## The assertion `packages/mosaic/framework/systemd/user/test-fleet-units.sh`: ```bash : > "$MARKER" if /usr/bin/env -i HOME="$HOLDER_HOME" ... "$HOLDER_START" >"$TEST_ROOT/holder.out" 2>&1; then fail "holder adopted contaminated named server" fi ... [ ! -s "$MARKER" ] || fail "holder execution triggered a contaminated loader" ``` The marker is truncated, the holder runs, the marker is checked. Any write in that window is attributed to the holder. ## Measurement On sb-it-1-dt (`origin/next`, clean detached worktree), tmux and cc both present: ``` run 1 rc=1 FAIL: holder execution triggered a contaminated loader run 2 rc=0 run 3 rc=0 run 4 rc=0 run 5 rc=1 FAIL: holder execution triggered a contaminated loader run 6 rc=0 run 7 rc=0 ``` Two of seven, always the same assertion. In a CI shape (`/usr/bin` minus tmux, cc, systemd-analyze) the block self-skips and the suite is a clean rc=0. ## The write is not the holder's I rebuilt the fixture standalone — same `marker.c`, same contaminated `tmux new-session` — then truncated the marker and ran **no holder at all**, only the same observation window: ``` iter 1: marker WRITTEN with no holder involved (14 bytes) iter 2..10: clean ``` 1 in 10 with the holder entirely absent. The contaminated server does work of its own inside the window — its `LD_PRELOAD` is in the server's global environment, so anything it spawns loads the constructor and appends. The assertion cannot tell that from a holder-caused load. The other three assertions in that block are sound: the holder does fail, does print the contract message, and does leave the server pid unchanged. Only the attribution one is unsafe. ## Why it is worth fixing rather than deleting The property is real and worth keeping — the holder must not trigger the contaminated loader. It needs to be attributed to the holder's own process tree instead of to a time window. A marker keyed per-process, or comparing against a control window with no holder, would both do it. I have not picked one; whoever takes it should, since the fix is a design choice rather than a one-liner. ## Why it matters now #1017 task 2(b) asks whether tmux belongs in the CI image. PR #1259 enumerates this suite, which is safe today precisely because no runner has tmux. Adding tmux would import this flake. That is now measured rather than suspected, so 2(b) should be decided with it in view. Found while verifying @scooby's enumeration finding for #1017. -- fred (sb-it-1-dt)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1260