mosaicstack/stack
1
0
Fork 1
Code Issues 140 Pull Requests 6 Actions Packages Projects Releases 15 Wiki Activity

detector lock fd 9 is not close-on-exec: every child inherits it, and the unbounded ones (source adapter, untimed beacon sink) can hold the single-instance lock forever #992

New Issue
Open
opened 2026-07-31 08:51:51 +00:00 by Mos · 1 comment
Mos commented 2026-07-31 08:51:51 +00:00
Contributor
Copy Link
Copy Source

Summary

detector.sh holds the per-host single-instance lock on fd 9, which is not close-on-exec. Every child process it spawns inherits the file descriptor, and flock holds on the open file description — so any inherited copy keeps the lock alive after the detector itself is gone.

#966 fixes the mildest instance of this (the sleep in the poll loop) with a per-site sleep "$interval" 9>&-. That fix is correct and should land as scoped. This issue is the class it does not close.

Measured by mos-dt; the bash semantics below measured independently here.

Severity ordering — the fixed case is the least bad one

child bound outcome
sleep "$interval" interval, ≤ 30s self-heals — this is what #966 closes
WAKE_DETECTOR_SOURCE_CMD adapter none operator-supplied, network-shaped
beacon.sh:262 sink via sh -c "$WAKE_BEACON_SINK_CMD" none — no timeout operator-supplied, network-shaped

Measured: the source adapter reported INHERITED-FD9 on 2 of 2 invocations in a single --once cycle.

A hung sink or a hung adapter holds the single-instance lock indefinitely after the detector has exited. A supervised restart then fails loud — "another detector instance already holds the lock; refusing" — against a lock owned by nothing but a stuck network call. That is strictly worse than the bounded case, and it is unbounded rather than thirty seconds.

Why a per-site fix cannot be the answer

9>&- at each spawn site is a convention, not a mechanism. It binds only authors who know it, a forgotten site is silent — no test fails, the lock simply outlives its owner again — and the number of sites grows with the code.

The decisive argument is pepper's, and it is about who writes the children:

A convention protects our code from colliding with fd 9, but no convention can protect fd 9 from being inherited, because inheritance is the default and conventions only bind authors who know them. Operator-supplied commands — the source adapter, the untimed sh -c sink — are authored by people who have never read our conventions. Close-on-exec is the only defence that binds without being known.

Supporting evidence that the convention approach is already load-bearing and already fragile: _wake-common.sh allocates its saved fds dynamically at ≥10 specifically to stay clear of lock fd 8 and detector fd 9. That is a convention-shaped defence against this same hazard, and it protects only the code that knows about it.

The obvious mechanism does not exist — measured, so nobody re-derives it

The natural candidate is "allocate the lock with exec {var}> so bash marks it close-on-exec." It is false. Measured on bash 5.2.15:

parent: fd9=open  dyn=10 (open)
child : fd9  -> INHERITED
child : fd10 -> INHERITED

Bash does not set FD_CLOEXEC on {var}> redirections. Auto-allocated descriptors are inherited exactly like literal ones.

So a real mechanism has to come from somewhere else — a spawn helper that closes the fd for every child, restructuring so the lock is not held across child spawns, or an external means of setting the flag. This is recorded specifically so the next person does not assume bash provides it; the coordinator nearly ruled that from memory and the measurement killed it.

Not in scope

#966's one-liner. It is correct, verified in both directions against the real detector, preserves the single-instance invariant, and should land unchanged. Widening a deliberately scoped fix is how batches stop being reviewable.

## Summary `detector.sh` holds the per-host single-instance lock on **fd 9**, which is **not close-on-exec**. Every child process it spawns inherits the file descriptor, and `flock` holds on the *open file description* — so **any inherited copy keeps the lock alive after the detector itself is gone.** `#966` fixes the mildest instance of this (the `sleep` in the poll loop) with a per-site `sleep "$interval" 9>&-`. That fix is correct and should land as scoped. **This issue is the class it does not close.** Measured by **mos-dt**; the bash semantics below measured independently here. ## Severity ordering — the fixed case is the least bad one | child | bound | outcome | |---|---|---| | `sleep "$interval"` | interval, **≤ 30s** | self-heals — this is what `#966` closes | | `WAKE_DETECTOR_SOURCE_CMD` adapter | **none** | operator-supplied, network-shaped | | `beacon.sh:262` sink via `sh -c "$WAKE_BEACON_SINK_CMD"` | **none — no timeout** | operator-supplied, network-shaped | Measured: the source adapter reported **INHERITED-FD9 on 2 of 2 invocations** in a single `--once` cycle. **A hung sink or a hung adapter holds the single-instance lock indefinitely after the detector has exited.** A supervised restart then fails loud — *"another detector instance already holds the lock; refusing"* — against a lock owned by nothing but a stuck network call. That is strictly worse than the bounded case, and it is unbounded rather than thirty seconds. ## Why a per-site fix cannot be the answer `9>&-` at each spawn site is **a convention, not a mechanism**. It binds only authors who know it, a forgotten site is silent — no test fails, the lock simply outlives its owner again — and the number of sites grows with the code. The decisive argument is **pepper's**, and it is about who writes the children: > A convention protects *our* code from colliding with fd 9, but **no convention can protect fd 9 from being inherited, because inheritance is the default and conventions only bind authors who know them.** Operator-supplied commands — the source adapter, the untimed `sh -c` sink — are authored by people who have never read our conventions. **Close-on-exec is the only defence that binds without being known.** Supporting evidence that the convention approach is already load-bearing and already fragile: `_wake-common.sh` allocates its saved fds **dynamically at ≥10** specifically to stay clear of lock fd 8 and detector fd 9. That is a convention-shaped defence against this same hazard, and it protects only the code that knows about it. ## The obvious mechanism does not exist — measured, so nobody re-derives it The natural candidate is *"allocate the lock with `exec {var}>` so bash marks it close-on-exec."* **It is false.** Measured on **bash 5.2.15**: ``` parent: fd9=open dyn=10 (open) child : fd9 -> INHERITED child : fd10 -> INHERITED ``` **Bash does not set `FD_CLOEXEC` on `{var}>` redirections.** Auto-allocated descriptors are inherited exactly like literal ones. So a real mechanism has to come from somewhere else — a spawn helper that closes the fd for every child, restructuring so the lock is not held across child spawns, or an external means of setting the flag. **This is recorded specifically so the next person does not assume bash provides it; the coordinator nearly ruled that from memory and the measurement killed it.** ## Not in scope `#966`'s one-liner. It is correct, verified in both directions against the real detector, preserves the single-instance invariant, and should land unchanged. Widening a deliberately scoped fix is how batches stop being reviewable.
Mos commented 2026-07-31 08:52:44 +00:00
Author
Contributor
Copy Link
Copy Source

Two caveats on this issue's own framing, added by its author

1. The fd 9 framing is a scope assumption, not a measured boundary

This issue is titled and written around fd 9. That framing is inherited from where the defect was first observed, and it has not been established that fd 9 is the only lock descriptor at risk.

_wake-common.sh allocates its saved fds at ≥10 specifically to stay clear of lock fd 8 and detector fd 9 — so at least one other lock descriptor exists, and nothing here measures whether it is inherited.

The open question is prior to this issue, not settled by it: which lock fds exist, and which of them are inherited? Credit to pepper, which flagged the fd-9-only framing as a risk carried by every document written on this so far — including this one.

Do not read this issue as bounding the class to fd 9.

2. This issue contains a measurement that will contaminate an independent derivation

The section above measuring bash's {var}> behaviour is a conclusion, and a second derivation of this problem is scheduled to a different seat precisely so the mechanism gets established twice, independently.

Anyone assigned that derivation should derive first and read this issue afterwards — or read it and drop the independence claim. Both are legitimate; silently doing the second while reporting the first is not.

Recording this because the contamination is my fault: I wrote the measurement into the issue before the derivation was scheduled, which is the ordinary way a second opinion quietly becomes an echo. mos-dt deliberately abstained from measuring fd 8 to preserve exactly this independence — it gave up an easy measurement to keep the second derivation worth staffing, and it would be wasteful for this issue to spend what that abstention bought.

## Two caveats on this issue's own framing, added by its author ### 1. The `fd 9` framing is a scope assumption, not a measured boundary This issue is titled and written around **fd 9**. That framing is inherited from where the defect was first observed, and it has **not been established that fd 9 is the only lock descriptor at risk.** `_wake-common.sh` allocates its saved fds at ≥10 specifically to stay clear of **lock fd 8** *and* detector fd 9 — so at least one other lock descriptor exists, and nothing here measures whether it is inherited. **The open question is prior to this issue, not settled by it:** *which lock fds exist, and which of them are inherited?* Credit to **pepper**, which flagged the fd-9-only framing as a risk carried by **every document written on this so far — including this one.** Do not read this issue as bounding the class to fd 9. ### 2. This issue contains a measurement that will contaminate an independent derivation The section above measuring bash's `{var}>` behaviour is a **conclusion**, and a second derivation of this problem is scheduled to a different seat precisely so the mechanism gets established twice, independently. **Anyone assigned that derivation should derive first and read this issue afterwards** — or read it and drop the independence claim. Both are legitimate; silently doing the second while reporting the first is not. Recording this because the contamination is my fault: I wrote the measurement into the issue before the derivation was scheduled, which is the ordinary way a second opinion quietly becomes an echo. **mos-dt deliberately abstained from measuring fd 8 to preserve exactly this independence** — it gave up an easy measurement to keep the second derivation worth staffing, and it would be wasteful for this issue to spend what that abstention bought.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
coder-mos1 coder-mos2 f10-coder jason.woltje (Jason Woltje) mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1))
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#992