mosaicstack/stack
1
0
Fork 1
Code Issues 294 Pull Requests 18 Actions Packages Projects Releases 15 Wiki Activity

Greenfield install completes clean and no fleet seat can launch: shipped roster requires 'pi' (never installed), and the pane PATH omits the Node the installer just bootstrapped #1256

New Issue
Open
opened 2026-08-16 19:35:42 +00:00 by mos-dt-0 · 1 comment
mos-dt-0 commented 2026-08-16 19:35:42 +00:00
Collaborator
Copy Link
Copy Source

A greenfield install finishes clean and produces a fleet that cannot launch a single seat. Two independent causes, one hiding behind the other. Both measured on a Debian 13 VM rolled back to a clean snapshot, installed from next with the documented one-liner, on 0.0.49.

This does not reproduce on any host that already had Node installed, which is why it has not been seen: it only fires on exactly the hosts the Node bootstrap exists to serve.

Blocker 1 — the shipped roster requires pi; the installer never installs it

fleet/roster.yaml:22 ships a generalist seat, and the install enables [email protected] in default.target.wants. pi is not installed by the installer and is not present on the box.

$ systemctl --user start [email protected]
Job for [email protected] failed because the control process exited with error code.

$ journalctl --user -u [email protected] -n 20 --output=cat
Starting [email protected] - Mosaic tmux fleet agent generalist...
ERROR: agent launch aborted: code=missing-binary agent=generalist 'pi' is not on the pane PATH
  (/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin)
[email protected]: Main process exited, code=exited, status=69/UNAVAILABLE

$ ls ~/.npm-global/bin/
claude  mosaic  mosaic-gateway  mosaic-wizard

Credit where it is due: #1241's guard did exactly its job. It resolved the binary before any effect and produced a diagnostic that names the agent, the binary and the PATH it searched, instead of a pane that dies in under a second inside a session nobody is attached to. Every fact in this report came out of that one line. Without it this would have presented as "the fleet is quiet."

mosaic doctor does not warn about this. On a host without tmux it warns "this host has a roster and no seat can launch" — the correct sentence — but a roster naming an uninstalled runtime produces no warning at all. The check covers the transport and not the runtime the roster asks for.

Fix is a product decision, so I am not proposing one: install the runtimes the shipped roster references, ship a roster that references only what is installed, or have doctor/install fail loudly when the roster names a runtime that is absent.

Blocker 2 — the pane PATH omits the Node the installer itself just bootstrapped

Latent behind blocker 1 on this host. Proven directly rather than inferred.

On a host with no system Node, tools/install.sh bootstraps one into ~/.mosaic/node/ and writes both bin directories to ~/.profile. That is correct for an interactive login shell. The fleet never sees it:

ExecStart=/usr/bin/env -i HOME=%h MOSAIC_AGENT_NAME=%i PATH=/usr/bin:/bin \
  /bin/bash --noprofile --norc %h/.config/mosaic/tools/fleet/start-agent-session.sh %i

env -i plus --noprofile --norc is deliberate and I am not arguing with it. start-agent-session.sh then rebuilds a PANE_PATH from candidates — $MOSAIC_RUNTIME_BIN, $(npm config get prefix)/bin, ~/.npm-global/bin, ~/.local/bin. ~/.mosaic/node/current/bin is not among them, and the npm config get prefix branch is guarded by command -v npm, which on a bootstrap-Node host is itself only in ~/.mosaic/node/current/bin. So that branch is dead on precisely the hosts that need it.

Measured PANE_PATH, from the error line above: /home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin. No Node anywhere in it. And mosaic is a Node script:

$ head -1 ~/.npm-global/bin/mosaic
#!/usr/bin/env node

$ env -i HOME=/home/mosaic PATH=/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin \
    /home/mosaic/.npm-global/bin/mosaic --version
env: 'node': No such file or directory

$ env -i HOME=/home/mosaic PATH=/home/mosaic/.mosaic/node/current/bin:/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin \
    /home/mosaic/.npm-global/bin/mosaic --version
0.0.49

$ ls -l /usr/bin/node /usr/local/bin/node
ls: cannot access '/usr/bin/node': No such file or directory
ls: cannot access '/usr/local/bin/node': No such file or directory

So once blocker 1 is fixed, the pane reaches mosaic yolo <runtime> and dies on env: 'node': No such file or directory. Suggested fix is small and local: add $HOME/.mosaic/node/current/bin to the candidate list in _build_runtime_bin_prefix, ahead of the npm config get prefix probe so that probe can also succeed.

Why these two are the same bug twice

Both mechanisms protect the surface their author was picturing and miss the one that actually executes:

  • The installer's PATH work targets the interactive login shell (~/.profile). The fleet is a systemd --user unit, which never reads it — by explicit design in the unit.
  • The PATH repair that exists to solve exactly that targets npm-global and .local/bin, and misses the directory the same installer created two steps earlier.
  • doctor's roster check covers the transport (tmux) and not the runtime the roster names.

Same shape as #1249 and #1255. It is worth stating as a review question rather than a one-off fix: when a check or a repair enumerates a set, which executable set does it actually cover, and is that the set that runs?

What a greenfield host looks like right now

mosaic --version → 0.0.49 from a login shell. mosaic doctor → 11 warnings, none of them this. systemctl --user is-enabled mosaic-agent@generalist → enabled. Seats running: zero, and no tmux server at all. Every individual signal reads like a successful install.

Measured on mosaic-sbx-canary, reproduced independently on mosaic-sbx-dev (same install state, same missing pi, same absent system Node). Both are throwaway snapshot-revertible VMs; happy to run any probe you want on them.

-- fred (sb-it-1-dt)

A greenfield install finishes clean and produces a fleet that cannot launch a single seat. Two independent causes, one hiding behind the other. Both measured on a Debian 13 VM rolled back to a clean snapshot, installed from `next` with the documented one-liner, on `0.0.49`. This does not reproduce on any host that already had Node installed, which is why it has not been seen: it only fires on exactly the hosts the Node bootstrap exists to serve. ## Blocker 1 — the shipped roster requires `pi`; the installer never installs it `fleet/roster.yaml:22` ships a `generalist` seat, and the install enables `[email protected]` in `default.target.wants`. `pi` is not installed by the installer and is not present on the box. ``` $ systemctl --user start [email protected] Job for [email protected] failed because the control process exited with error code. $ journalctl --user -u [email protected] -n 20 --output=cat Starting [email protected] - Mosaic tmux fleet agent generalist... ERROR: agent launch aborted: code=missing-binary agent=generalist 'pi' is not on the pane PATH (/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin) [email protected]: Main process exited, code=exited, status=69/UNAVAILABLE $ ls ~/.npm-global/bin/ claude mosaic mosaic-gateway mosaic-wizard ``` Credit where it is due: **#1241's guard did exactly its job.** It resolved the binary before any effect and produced a diagnostic that names the agent, the binary and the PATH it searched, instead of a pane that dies in under a second inside a session nobody is attached to. Every fact in this report came out of that one line. Without it this would have presented as "the fleet is quiet." `mosaic doctor` does not warn about this. On a host without tmux it warns *"this host has a roster and no seat can launch"* — the correct sentence — but a roster naming an uninstalled **runtime** produces no warning at all. The check covers the transport and not the runtime the roster asks for. Fix is a product decision, so I am not proposing one: install the runtimes the shipped roster references, ship a roster that references only what is installed, or have `doctor`/install fail loudly when the roster names a runtime that is absent. ## Blocker 2 — the pane PATH omits the Node the installer itself just bootstrapped Latent behind blocker 1 on this host. Proven directly rather than inferred. On a host with no system Node, `tools/install.sh` bootstraps one into `~/.mosaic/node/` and writes both bin directories to `~/.profile`. That is correct for an interactive login shell. The fleet never sees it: ``` ExecStart=/usr/bin/env -i HOME=%h MOSAIC_AGENT_NAME=%i PATH=/usr/bin:/bin \ /bin/bash --noprofile --norc %h/.config/mosaic/tools/fleet/start-agent-session.sh %i ``` `env -i` plus `--noprofile --norc` is deliberate and I am not arguing with it. `start-agent-session.sh` then rebuilds a PANE_PATH from candidates — `$MOSAIC_RUNTIME_BIN`, `$(npm config get prefix)/bin`, `~/.npm-global/bin`, `~/.local/bin`. `~/.mosaic/node/current/bin` is not among them, and the `npm config get prefix` branch is guarded by `command -v npm`, which on a bootstrap-Node host is itself only in `~/.mosaic/node/current/bin`. So that branch is dead on precisely the hosts that need it. Measured PANE_PATH, from the error line above: `/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin`. No Node anywhere in it. And `mosaic` is a Node script: ``` $ head -1 ~/.npm-global/bin/mosaic #!/usr/bin/env node $ env -i HOME=/home/mosaic PATH=/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin \ /home/mosaic/.npm-global/bin/mosaic --version env: 'node': No such file or directory $ env -i HOME=/home/mosaic PATH=/home/mosaic/.mosaic/node/current/bin:/home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin \ /home/mosaic/.npm-global/bin/mosaic --version 0.0.49 $ ls -l /usr/bin/node /usr/local/bin/node ls: cannot access '/usr/bin/node': No such file or directory ls: cannot access '/usr/local/bin/node': No such file or directory ``` So once blocker 1 is fixed, the pane reaches `mosaic yolo <runtime>` and dies on `env: 'node': No such file or directory`. Suggested fix is small and local: add `$HOME/.mosaic/node/current/bin` to the candidate list in `_build_runtime_bin_prefix`, ahead of the `npm config get prefix` probe so that probe can also succeed. ## Why these two are the same bug twice Both mechanisms protect the surface their author was picturing and miss the one that actually executes: - The installer's PATH work targets the interactive login shell (`~/.profile`). The fleet is a systemd `--user` unit, which never reads it — by explicit design in the unit. - The PATH repair that exists to solve exactly that targets `npm-global` and `.local/bin`, and misses the directory the same installer created two steps earlier. - `doctor`'s roster check covers the transport (`tmux`) and not the runtime the roster names. Same shape as #1249 and #1255. It is worth stating as a review question rather than a one-off fix: *when a check or a repair enumerates a set, which executable set does it actually cover, and is that the set that runs?* ## What a greenfield host looks like right now `mosaic --version` → `0.0.49` from a login shell. `mosaic doctor` → 11 warnings, none of them this. `systemctl --user is-enabled mosaic-agent@generalist` → `enabled`. Seats running: zero, and no tmux server at all. Every individual signal reads like a successful install. Measured on `mosaic-sbx-canary`, reproduced independently on `mosaic-sbx-dev` (same install state, same missing `pi`, same absent system Node). Both are throwaway snapshot-revertible VMs; happy to run any probe you want on them. -- fred (sb-it-1-dt)
mos-dt-0 commented 2026-08-16 21:20:15 +00:00
Author
Collaborator
Copy Link
Copy Source

Confirmed end-to-end on a greenfield host: this is where a clean install stops. fleet start exits 69 missing-binary because nothing ever installs pi.

Measured by @daphne on mosaic-sbx-canary (VMID 1125), reverted to the greenfield snapshot, next
installer, CLI 0.0.50-next.2439. No credentials, no workarounds, every step recorded.

The v1 chain runs clean right up to the pane:

mosaic fleet init --profile general --write     rc=0
mosaic fleet install                            rc=0   (generated envs materialized)
mosaic fleet install-systemd                    rc=0   (user units written)
mosaic fleet add probe --runtime pi --class probe --no-start   rc=0
mosaic fleet start probe                        rc=1
  -> service exits 69 missing-binary
  -> pane PATH: /home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin
  -> mosaic fleet ps: probe failed/enabled, PANE dead

pi is not on that PATH and nothing in the install ever put it there. The shipped general profile
declares a pi worker, so the default roster the CLI itself generates cannot start on the host the
CLI itself just provisioned. That is the whole distance between "installer exits 0" and "the fleet has
a live pane" on a clean machine.

This is blocker 1 of this issue, now with a greenfield reproduction rather than an inference from a
customized host. Three things fall out that are worth separating when it gets fixed:

  1. exit 69 missing-binary is the right failure and the wrong message. It does not name the
    binary, the PATH it searched, or how to supply it. An operator on a fresh host gets a number.
  2. The default profile should not declare a runtime the installer does not provide — either the
    installer installs pi, or fleet init emits a roster whose runtimes it can verify are present,
    or fleet install fails loudly at install time instead of letting start fail per-agent later.
    Failing at install is the cheapest of the three: it is one preflight over the roster's distinct
    runtimes, and it moves the error from "an agent died" to "this host cannot run this roster yet."
  3. tmux has the same shape — the installer warns rather than installing it. Same class of gap,
    same host, found in the same run.

Full run record: docs/reports/2026-08-16_sbx-canary-greenfield-e2e.md in jarvis-brain (runs 1-5).
Related: the v1/v2 roster split and the swallowed reconciler error are filed separately as #1261 —
that one is what made run 4 look like a dead install when it was not.

-- fred (sb-it-1-dt)

## Confirmed end-to-end on a greenfield host: this is where a clean install stops. `fleet start` exits 69 `missing-binary` because nothing ever installs `pi`. Measured by @daphne on `mosaic-sbx-canary` (VMID 1125), reverted to the `greenfield` snapshot, `next` installer, CLI `0.0.50-next.2439`. No credentials, no workarounds, every step recorded. The v1 chain runs clean right up to the pane: ```text mosaic fleet init --profile general --write rc=0 mosaic fleet install rc=0 (generated envs materialized) mosaic fleet install-systemd rc=0 (user units written) mosaic fleet add probe --runtime pi --class probe --no-start rc=0 mosaic fleet start probe rc=1 -> service exits 69 missing-binary -> pane PATH: /home/mosaic/.npm-global/bin:/usr/local/bin:/usr/bin:/bin -> mosaic fleet ps: probe failed/enabled, PANE dead ``` `pi` is not on that PATH and nothing in the install ever put it there. The shipped `general` profile declares a `pi` worker, so the default roster the CLI itself generates cannot start on the host the CLI itself just provisioned. That is the whole distance between "installer exits 0" and "the fleet has a live pane" on a clean machine. This is blocker 1 of this issue, now with a greenfield reproduction rather than an inference from a customized host. Three things fall out that are worth separating when it gets fixed: 1. **`exit 69 missing-binary` is the right failure and the wrong message.** It does not name the binary, the PATH it searched, or how to supply it. An operator on a fresh host gets a number. 2. **The default profile should not declare a runtime the installer does not provide** — either the installer installs `pi`, or `fleet init` emits a roster whose runtimes it can verify are present, or `fleet install` fails loudly at install time instead of letting `start` fail per-agent later. Failing at `install` is the cheapest of the three: it is one preflight over the roster's distinct runtimes, and it moves the error from "an agent died" to "this host cannot run this roster yet." 3. **`tmux` has the same shape** — the installer warns rather than installing it. Same class of gap, same host, found in the same run. Full run record: `docs/reports/2026-08-16_sbx-canary-greenfield-e2e.md` in jarvis-brain (runs 1-5). Related: the v1/v2 roster split and the swallowed reconciler error are filed separately as #1261 — that one is what made run 4 look like a dead install when it was not. -- fred (sb-it-1-dt)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 coder2 coder3 f10-coder fargo fred happy jason.woltje (Jason Woltje) merge-gate pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev0 sanity scooby (Scooby) scrappy shaggy tess tiny velma woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1256