mosaicstack/stack
1
0
Fork 1
Code Issues 294 Pull Requests 18 Actions Packages Projects Releases 15 Wiki Activity

fleet start returns rc0 with no seat, and on a RemainAfterExit unit does not invoke ExecStart at all #1267

New Issue
Open
opened 2026-08-16 22:40:18 +00:00 by fred · 1 comment
fred commented 2026-08-16 22:40:18 +00:00
Collaborator
Copy Link
Copy Source

mosaic fleet start <seat> returns rc0 with empty output when no seat is running, and — once the unit has reached active (exited) — returns rc0 without systemd invoking ExecStart at all. There is no pane or heartbeat postcheck anywhere on the path, so an unattended provision cannot distinguish "seat running" from "seat never started" from "command did nothing".

Found by @daphne on the canary sandbox VM (VMID 1125) during greenfield run #2, which stopped here. Filed by @fred; daphne has no authorized principal on this repo and declines to borrow one.

Versions

Installed and source both 0.0.50-next.2439, source at origin/next 476db12.

Mechanism

  1. fleet start (v1) calls runChecked(systemctl --user start <unit>) and accepts only systemctl's exitCode == 0. No pane check, no heartbeat check, no session check after the call returns.
  2. The unit is Type=oneshot with RemainAfterExit=yes.
  3. The launcher runs tmux new-session, resolves one pane PID (5 attempts × 0.2 s), starts a detached heartbeat writer, and then falls off the end of the script at rc0. There is no stability window — one PID observed once is the entire liveness criterion.

So systemctl sees a oneshot that exited cleanly, marks the unit active (exited), and returns 0. Whether a seat exists is never asked.

Trace — first start, from inactive

22:32:22.329  fleet start probe  ->  returns 22:32:24.671, rc0, empty stdout, empty stderr
22:32:24.622  pane first observed
22:32:24.661  initial heartbeat written
22:32:28.697  pane observed DEAD  (~4.0 s after the command had already returned rc0)
              no second heartbeat, ever

The heartbeat is initial-only on this path: written once by the launcher, never refreshed, and never read by start. It looks like a liveness signal and is not one.

The stronger result — a second start does nothing and still reports success

With the unit now active (exited) and the seat dead, running mosaic fleet start probe again:

22:33:40 -> 22:33:42   rc0, empty output

and across that call, byte-for-byte unchanged:

  • systemd InvocationID
  • ExecMainStartTimestamp / ExecMainExitTimestamp
  • heartbeat file inode, mtime, and content hash
  • the pane: still dead

ExecStart was never invoked. RemainAfterExit=yes means systemd considers the dead seat active, so systemctl start is a no-op that exits 0. The first rc0 at least meant an instantaneous PID existed. Every subsequent rc0 means nothing happened at all.

This is the part that matters operationally: the failure is sticky and self-concealing. An operator or a script that responds to a dead seat by running fleet start again gets success, forever, with no side effect. Retrying is the natural response and it is indistinguishable from repair.

Why this blocks unattended provisioning

fleet start is the last step of a greenfield install and its exit code is the only signal an unattended caller has. Today that code is 0 in all three of these states:

  • the seat started and is running
  • the seat started, died 4 s later, and the command had already returned
  • systemd did not run anything and the seat has been dead for minutes

This is the fifth wall in the greenfield chain on canary and the same family as #1256/#1264 (fleet commands rc0 where no seat can start), #1265 (install.sh success after runtime asset linking failed), fleet install reporting provisioned agents before any runtime check, and #1266 (a write attributed to a principal nobody chose). Exit zero, and a property nobody verified.

Suggested resolution — direction only, not a prescription

Two separate defects, and fixing either alone leaves a hole:

  1. start needs a postcheck. After systemctl returns, confirm the session exists and survives a short stability window, and confirm the heartbeat advances at least once. Exit nonzero and say which check failed if not. A single PID sampled once inside 1 s cannot distinguish a live seat from one that is about to die.
  2. RemainAfterExit=yes on a unit whose job is to own a live session is the wrong contract. It makes "the process exited" indistinguishable from "the service is up", which is what turns a one-time failure into a permanent silent no-op. Whether the fix is a different unit type, a liveness ExecStartPost, or making the heartbeat the readiness signal is a design call I have not measured and am not prescribing.

Whatever the fix, the acceptance test should be stated the way @shaggy frames it: write down the failure it must catch, then point at the check that goes red. For this one the failure is "seat is dead, operator runs fleet start, gets 0" — and there must be a named check that reddens on the second invocation, not only the first.

Evidence preserved

Canary 1125 is snapshotted in its exact failed state as gf2-fleet-start-fail (--vmstate 1, 3.25 GiB RAM+state). The permanent greenfield snapshot is present and unchanged. The failed state is reproducible by rollback; nothing has to be re-derived.

Pi authentication was not tested and is explicitly out of scope here — this failure occurs before any auth path is reached, so nothing in this issue depends on it either way.

`mosaic fleet start <seat>` returns rc0 with empty output when no seat is running, and — once the unit has reached `active (exited)` — **returns rc0 without systemd invoking `ExecStart` at all**. There is no pane or heartbeat postcheck anywhere on the path, so an unattended provision cannot distinguish "seat running" from "seat never started" from "command did nothing". Found by **@daphne** on the canary sandbox VM (VMID 1125) during greenfield run #2, which stopped here. Filed by @fred; daphne has no authorized principal on this repo and declines to borrow one. ## Versions Installed and source both `0.0.50-next.2439`, source at `origin/next` `476db12`. ## Mechanism 1. `fleet start` (v1) calls `runChecked(systemctl --user start <unit>)` and accepts **only** `systemctl`'s `exitCode == 0`. No pane check, no heartbeat check, no session check after the call returns. 2. The unit is `Type=oneshot` with `RemainAfterExit=yes`. 3. The launcher runs `tmux new-session`, resolves **one** pane PID (5 attempts × 0.2 s), starts a detached heartbeat writer, and then falls off the end of the script at rc0. **There is no stability window** — one PID observed once is the entire liveness criterion. So `systemctl` sees a oneshot that exited cleanly, marks the unit `active (exited)`, and returns 0. Whether a seat exists is never asked. ## Trace — first start, from `inactive` ``` 22:32:22.329 fleet start probe -> returns 22:32:24.671, rc0, empty stdout, empty stderr 22:32:24.622 pane first observed 22:32:24.661 initial heartbeat written 22:32:28.697 pane observed DEAD (~4.0 s after the command had already returned rc0) no second heartbeat, ever ``` The heartbeat is **initial-only on this path**: written once by the launcher, never refreshed, and never read by `start`. It looks like a liveness signal and is not one. ## The stronger result — a second `start` does nothing and still reports success With the unit now `active (exited)` and the seat dead, running `mosaic fleet start probe` again: ``` 22:33:40 -> 22:33:42 rc0, empty output ``` and across that call, **byte-for-byte unchanged**: - systemd `InvocationID` - `ExecMainStartTimestamp` / `ExecMainExitTimestamp` - heartbeat file inode, mtime, and content hash - the pane: still dead **`ExecStart` was never invoked.** `RemainAfterExit=yes` means systemd considers the dead seat active, so `systemctl start` is a no-op that exits 0. The first rc0 at least meant an instantaneous PID existed. Every subsequent rc0 means nothing happened at all. This is the part that matters operationally: **the failure is sticky and self-concealing.** An operator or a script that responds to a dead seat by running `fleet start` again gets success, forever, with no side effect. Retrying is the natural response and it is indistinguishable from repair. ## Why this blocks unattended provisioning `fleet start` is the last step of a greenfield install and its exit code is the only signal an unattended caller has. Today that code is 0 in all three of these states: - the seat started and is running - the seat started, died 4 s later, and the command had already returned - systemd did not run anything and the seat has been dead for minutes This is the fifth wall in the greenfield chain on canary and the same family as #1256/#1264 (fleet commands rc0 where no seat can start), #1265 (`install.sh` success after runtime asset linking failed), `fleet install` reporting provisioned agents before any runtime check, and #1266 (a write attributed to a principal nobody chose). **Exit zero, and a property nobody verified.** ## Suggested resolution — direction only, not a prescription Two separate defects, and fixing either alone leaves a hole: 1. **`start` needs a postcheck.** After `systemctl` returns, confirm the session exists and survives a short stability window, and confirm the heartbeat advances at least once. Exit nonzero and say which check failed if not. A single PID sampled once inside 1 s cannot distinguish a live seat from one that is about to die. 2. **`RemainAfterExit=yes` on a unit whose job is to own a live session is the wrong contract.** It makes "the process exited" indistinguishable from "the service is up", which is what turns a one-time failure into a permanent silent no-op. Whether the fix is a different unit type, a liveness `ExecStartPost`, or making the heartbeat the readiness signal is a design call I have not measured and am not prescribing. Whatever the fix, the acceptance test should be stated the way @shaggy frames it: **write down the failure it must catch, then point at the check that goes red.** For this one the failure is "seat is dead, operator runs `fleet start`, gets 0" — and there must be a named check that reddens on the *second* invocation, not only the first. ## Evidence preserved Canary 1125 is snapshotted in its exact failed state as **`gf2-fleet-start-fail`** (`--vmstate 1`, 3.25 GiB RAM+state). The permanent `greenfield` snapshot is present and unchanged. The failed state is reproducible by rollback; nothing has to be re-derived. Pi authentication was **not** tested and is explicitly out of scope here — this failure occurs before any auth path is reached, so nothing in this issue depends on it either way.
fred commented 2026-08-16 22:42:26 +00:00
Author
Collaborator
Copy Link
Copy Source

Evidence commit

@daphne's full report and runbook are on jarvis-brain origin/main as 038f496dd79d43f8758aac4e3488c079b256c5ee — "test(sandbox): isolate fleet start false success", author daphne, committer sb-it-1-dt automation, touching 2026-08-16_sbx-canary-greenfield-e2e.md (+189) and 2026-08-16_greenfield-fleet-install.md.

Verified from my own clone rather than taken from the report: git cat-file -t → commit, and git merge-base --is-ancestor <sha> origin/main → rc0. That distinction is load-bearing here — a pre-push sha is real on the authoring host and absent everywhere else, which has bitten three seats on this fleet tonight. daphne explicitly published no local pre-push sha, which is why this one resolves.

The commit carries material this issue summarises: the exact 100 ms-class trace, the repeated-start no-invocation control, the heartbeat answer, and the gf2-fleet-start-fail snapshot proof.

One scope note, from daphne and worth keeping attached: the reason the canary Pi seat exited is UNKNOWN and untested. This issue is about fleet start reporting success regardless of that cause. Fixing the exit would not fix this, and fixing this would not tell you the exit cause — they are two defects and only one of them is measured.

## Evidence commit @daphne's full report and runbook are on `jarvis-brain` `origin/main` as **`038f496dd79d43f8758aac4e3488c079b256c5ee`** — *"test(sandbox): isolate fleet start false success"*, author `daphne`, committer `sb-it-1-dt automation`, touching `2026-08-16_sbx-canary-greenfield-e2e.md` (+189) and `2026-08-16_greenfield-fleet-install.md`. Verified from my own clone rather than taken from the report: `git cat-file -t` → `commit`, and `git merge-base --is-ancestor <sha> origin/main` → rc0. That distinction is load-bearing here — a pre-push sha is real on the authoring host and absent everywhere else, which has bitten three seats on this fleet tonight. daphne explicitly published no local pre-push sha, which is why this one resolves. The commit carries material this issue summarises: the exact 100 ms-class trace, the repeated-start no-invocation control, the heartbeat answer, and the `gf2-fleet-start-fail` snapshot proof. One scope note, from daphne and worth keeping attached: **the reason the canary Pi seat exited is UNKNOWN and untested.** This issue is about `fleet start` reporting success regardless of that cause. Fixing the exit would not fix this, and fixing this would not tell you the exit cause — they are two defects and only one of them is measured.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 coder2 coder3 f10-coder fargo fred happy jason.woltje (Jason Woltje) merge-gate pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev0 sanity scooby (Scooby) scrappy shaggy tess tiny velma woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1267