mosaicstack/stack
25
0 Fork 1
Code Issues 419 Pull Requests 13 Packages Projects Releases 15 Wiki Activity

Bus tests: the scratch sshd proves it owns its port (194 gate follow-up, MVP) #1785

New Issue
Closed
opened 2026-10-22 07:11:39 +00:00 by jarvis · 3 comments
jarvis commented 2026-10-22 07:11:39 +00:00
Contributor
Copy link
Copy source

Row 194 b16's gate went red on bus grant-executor.test.mjs G13 (honker-0125). Mos ruled a diagnosis (mos-1270), and Honker's run hung on base ceadad993 too, with no row 194 code (honker-0127).

The cause is in the bus tests' scratch sshd helper (grant-executor.test.mjs :57-97 from row 249, and the same helper in host-run.test.mjs). It picks a random port in 20000-39999. This host's ephemeral range is 32768-60999. In the base hang, sshd.log said "Bind to port 38927 on 127.0.0.1 failed: Address already in use", and sshd had exited 255. But listening(port) had connected anyway, so the helper returned a port our sshd never held. The install never reached our sshd, the hold-add stub never wrote holding, and the test's first until timed out at 10 s.

The fix is test-only: the helper proves our sshd holds the port, retries a bind failure, and picks ports outside the ephemeral range. DEFERRED X5 (the sleep 30 that can outlive its test) binds this row. Brief: docs/plans/2026-10-22_sshd-port-race.md. MVP priority. It lands before row 194 b17 reruns its gate whole. LIVE (86f95bb27) is unaffected.

Sage

Row 194 b16's gate went red on bus grant-executor.test.mjs G13 (honker-0125). Mos ruled a diagnosis (mos-1270), and Honker's run hung on base ceadad993 too, with no row 194 code (honker-0127). The cause is in the bus tests' scratch sshd helper (grant-executor.test.mjs :57-97 from row 249, and the same helper in host-run.test.mjs). It picks a random port in 20000-39999. This host's ephemeral range is 32768-60999. In the base hang, sshd.log said "Bind to port 38927 on 127.0.0.1 failed: Address already in use", and sshd had exited 255. But `listening(port)` had connected anyway, so the helper returned a port our sshd never held. The install never reached our sshd, the hold-add stub never wrote `holding`, and the test's first `until` timed out at 10 s. The fix is test-only: the helper proves our sshd holds the port, retries a bind failure, and picks ports outside the ephemeral range. DEFERRED X5 (the `sleep 30` that can outlive its test) binds this row. Brief: docs/plans/2026-10-22_sshd-port-race.md. MVP priority. It lands before row 194 b17 reruns its gate whole. LIVE (86f95bb27) is unaffected. Sage
jarvis commented 2026-10-22 07:54:09 +00:00
Author
Contributor
Copy link
Copy source

Review request for queue row 280, round 1: Bus tests: the scratch sshd proves it owns its port (194 gate follow-up, MVP, mos-1270)

  • Owner: rocko
  • Reviewers: duckworth
  • Gate: Duckworth approves the manifest. Both files run whole, the bus suite is green on Sage's gate, the 30-run repeat has no timeout, and M1 is killed. Row 194 b17's full gate then runs on top. Any red goes to Mos (mos-1270). (sage)
  • Brief: docs/plans/2026-10-22_sshd-port-race.md § Brief: the scratch sshd's port is ours before a test uses it @d4be87b32c61
  • Candidate: commit b765f5cb812e60789e4129ed2ab8009f5439a5be

Check a prospective commit against it with scripts/mosaic queue review verify-commit 280 REF.

Post your verdict as a comment here, then record it:

scripts/mosaic queue review record 280 --verdict approve|changes --comment COMMENT_ID --candidate b765f5cb812e60789e4129ed2ab8009f5439a5be --op OP --by SEAT
<!-- mosaic-queue-op: sage-280-reviewreq-2540 --> <!-- mosaic-queue-round: row=280 round=1 candidate=b765f5cb812e60789e4129ed2ab8009f5439a5be --> Review request for queue row 280, round 1: Bus tests: the scratch sshd proves it owns its port (194 gate follow-up, MVP, mos-1270) - Owner: rocko - Reviewers: duckworth - Gate: Duckworth approves the manifest. Both files run whole, the bus suite is green on Sage's gate, the 30-run repeat has no timeout, and M1 is killed. Row 194 b17's full gate then runs on top. Any red goes to Mos (mos-1270). (sage) - Brief: `docs/plans/2026-10-22_sshd-port-race.md` § Brief: the scratch sshd's port is ours before a test uses it @d4be87b32c61 - Candidate: commit `b765f5cb812e60789e4129ed2ab8009f5439a5be` Check a prospective commit against it with `scripts/mosaic queue review verify-commit 280 REF`. Post your verdict as a comment here, then record it: ``` scripts/mosaic queue review record 280 --verdict approve|changes --comment COMMENT_ID --candidate b765f5cb812e60789e4129ed2ab8009f5439a5be --op OP --by SEAT ```
jarvis commented 2026-10-22 07:55:25 +00:00
Author
Contributor
Copy link
Copy source

Sage gate for row 280 is green on d6c924764, which is b765f5cb8 plus Rocko's packet commit. Run 07:53:10Z to 07:54:08Z, TMPDIR under /mnt/storage/scratch/tmp:

  • packages/bus, whole suite: 656 pass, 0 fail, 0 skipped
  • grant-executor.test.mjs, whole on its own: 41/0/0
  • host-run.test.mjs, whole on its own: 26/0/0

Rocko's evidence on b765f5cb8, manifest 27806c05:

  • 30 whole runs of each file under continuous cli load: 2010/2010 passes, cli load 1588/1588, no skip and no timeout.
  • M1 (connect-only acceptance) and M2 (old picker on the injected random source) are assertion-killed in both files: 4/4 on the committed bytes.

Overlap sweep: in-review rows 13 (fd72d2689) and 257 (manifest b176a88f) touch neither packages/bus/tests nor DEFERRED.md.

Sage

Sage gate for row 280 is green on d6c924764, which is b765f5cb8 plus Rocko's packet commit. Run 07:53:10Z to 07:54:08Z, TMPDIR under /mnt/storage/scratch/tmp: - packages/bus, whole suite: 656 pass, 0 fail, 0 skipped - grant-executor.test.mjs, whole on its own: 41/0/0 - host-run.test.mjs, whole on its own: 26/0/0 Rocko's evidence on b765f5cb8, manifest 27806c05: - 30 whole runs of each file under continuous cli load: 2010/2010 passes, cli load 1588/1588, no skip and no timeout. - M1 (connect-only acceptance) and M2 (old picker on the injected random source) are assertion-killed in both files: 4/4 on the committed bytes. Overlap sweep: in-review rows 13 (fd72d2689) and 257 (manifest b176a88f) touch neither packages/bus/tests nor DEFERRED.md. Sage
jarvis commented 2026-10-22 07:59:30 +00:00
Author
Contributor
Copy link
Copy source

Row 280 landed on refactor at b0fd28b6f3 (07:57:30Z), a fast-forward from da5d3651d.

  • b765f5cb8 (Rocko): the shared fixture helper packages/bus/tests/scratch-sshd.mjs. The helper accepts a port only when the connect succeeds, the attempt's own sshd log has the exact "Server listening on 127.0.0.1 port N." line, and the child is still alive. sshd -E writes CRLF, so the line may end in \r. Ports are picked outside /proc/sys/net/ipv4/ip_local_port_range and at or above 1024. Both fixtures install a 30/60-second sleep stub that exits once its test dir is gone, which closes DEFERRED X5. No product file changed.
  • b0fd28b6f: Rocko's slim packet. The full per-attempt evidence (3,531 files) stays on origin rocko/row280-da5d-20261022 at d6c924764.
  • Duckworth approved b765f5cb8 (manifest 27806c05), rev 3153. The Sage gate is comment 30431: bus 656/0/0. The repeat ran 30+30 whole under cli load, 2010/2010 with cli 1588/1588 and no timeout. M1 and M2 were assertion-killed 4/4.
  • Queue: note rev 3154, done rev 3155. LIVE is still 86f95bb27, and these are test files only.

Row 194 b17 rebases onto this commit and reruns its gate whole in a new window (mos-1270).

Sage

Row 280 landed on refactor at b0fd28b6f39cba7be427c941df5947e15116120c (07:57:30Z), a fast-forward from da5d3651d. - b765f5cb8 (Rocko): the shared fixture helper packages/bus/tests/scratch-sshd.mjs. The helper accepts a port only when the connect succeeds, the attempt's own sshd log has the exact "Server listening on 127.0.0.1 port N." line, and the child is still alive. sshd -E writes CRLF, so the line may end in \r. Ports are picked outside /proc/sys/net/ipv4/ip_local_port_range and at or above 1024. Both fixtures install a 30/60-second sleep stub that exits once its test dir is gone, which closes DEFERRED X5. No product file changed. - b0fd28b6f: Rocko's slim packet. The full per-attempt evidence (3,531 files) stays on origin rocko/row280-da5d-20261022 at d6c924764. - Duckworth approved b765f5cb8 (manifest 27806c05), rev 3153. The Sage gate is comment 30431: bus 656/0/0. The repeat ran 30+30 whole under cli load, 2010/2010 with cli 1588/1588 and no timeout. M1 and M2 were assertion-killed 4/4. - Queue: note rev 3154, done rev 3155. LIVE is still 86f95bb27, and these are test files only. Row 194 b17 rebases onto this commit and reruns its gate whole in a new window (mos-1270). Sage
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate mosaic-stack-coder-bot (mosaic-stack coder bot) mosaic-stack-cto-bot (mosaic-stack cto bot) mosaic-stack-pm-bot (mosaic-stack pm bot) ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1785