mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

Brain-split host: fleet start returns 0 and starts nothing — [email protected] gates on the config home, the reconciler writes the brain home #1310

New Issue
Open
opened 2026-08-18 18:57:20 +00:00 by fred · 0 comments
fred commented 2026-08-18 18:57:20 +00:00
Collaborator
Copy Link
Copy Source

On a brain-split host (~/.mosaic adopted as the brain home), mosaic fleet start returns 0, reports nothing, and starts no seat. All three units are skipped, not failed, so every status surface agrees the fleet is fine.

Measured 2026-08-18 on mosaic-sbx-dev (VMID 1124), rolled to the permanent greenfield snapshot, next lane, package 0.0.50-next.2507.

Cause

packages/mosaic/framework/systemd/user/[email protected] gates on the config home:

ConditionPathExists=%h/.config/mosaic/fleet/agents/%i.env.generated

On a roster-v2 fleet in split mode the reconciler writes the generated env to the brain home. After a clean fleet init --write + fleet install + fleet start:

present: ~/.mosaic/fleet/agents/{orchestrator,enhancer,generalist}.env.generated
absent : ~/.config/mosaic/fleet/agents/          (directory does not exist)

The condition can therefore never be met, and an unmet condition is reported as success:

orchestrator  ConditionResult=no  Result=success  ExecMainStatus=0  inactive/dead
enhancer      ConditionResult=no  Result=success  ExecMainStatus=0  inactive/dead
generalist    ConditionResult=no  Result=success  ExecMainStatus=0  inactive/dead

journal: [email protected] ... skipped, unmet condition check ConditionPathExists=/home/mosaic/.config/mosaic/fleet/agents/orchestrator.env.generated

Causal test, not correlation

One seat's env file copied into the config home. Nothing else changed. The other two seats are the untouched negative controls:

orchestrator  ConditionResult=yes  ActiveState=active   session appears on the fleet socket
enhancer      ConditionResult=no   inactive/dead        (control)
generalist    ConditionResult=no   inactive/dead        (control)

The started seat's pane shows the Mosaic identity wizard, i.e. it reaches the authentication boundary — so nothing downstream of the condition is implicated.

Controls on the measurement

  • mosaic fleet no-such-subcommand-zq7 → RC=1, run first, so the RC=0 results are not a CLI that returns 0 for everything.
  • tmux -L mosaic-fleet has-session zzz-nonce-8d2e → can't find session, so the _holder-only session list is a real reading of that socket.
  • [email protected] on the VM is sha256 3614731f70a3326e4409c5fe1e1708289d8afb6fed4ff0d4dd87b11d75f074f2, byte-identical to origin/next — this is the shipped unit, not a local edit.

Caution for anyone reproducing: a unit that does not exist also reports ConditionResult=no, so that field alone cannot tell "skipped" from "absent". I lost time to this by querying mosaic-fleet-holder.service, which is not a real unit name (mosaic-tmux-holder.service is). Confirm the unit exists first.

The existing test cannot observe this

packages/mosaic/src/commands/fleet-roster-v2-dispatch.spec.ts:214-255 names exactly this drift:

it('guards exactly the file the fleet writes, so the two cannot drift apart', ...)

Its fixture calls v2Home(), a custom mosaic-home. resolveBrainHome() deliberately never adopts ~/.mosaic when --mosaic-home is custom (hermetic sandboxes), so inside the fixture the brain home always equals the config home and the two paths cannot come apart. The assertion passes in both the broken and the fixed state. Any fix needs a fixture that reaches split mode — a real default home, or MOSAIC_BRAIN_HOME set explicitly — or it will stay green over the change.

Fix direction

The condition is a deliberate boot-window guard; the in-file comment explains that install enables the unit before the reconciler has written env, and that a skipped unit is the honest state for "enabled but not yet configured". A fix should keep that intent:

  • (preferred) have fleet install write a per-host drop-in that clears and re-points the condition at the resolved brain home — ConditionPathExists= (empty, to reset) followed by the correct path with %i; or
  • have the brain-home-aware launcher own the check and exit cleanly when env is absent, dropping the condition.

Deleting the condition outright loses the boot-window behaviour the comment documents.

Scope

Measured on one host, one VM, one package version, split mode only. Legacy (non-split) mode on the same snapshot does not hit this — its blocker is #1256. I have not measured whether web1's current install is split or legacy.

This blocks the web1 fleet migration: on a host where ~/.mosaic/fleet/agents exists, the fleet reports healthy and is empty.

-- fred (sb-it-1-dt)

On a brain-split host (`~/.mosaic` adopted as the brain home), `mosaic fleet start` returns 0, reports nothing, and starts no seat. All three units are *skipped*, not failed, so every status surface agrees the fleet is fine. Measured 2026-08-18 on `mosaic-sbx-dev` (VMID 1124), rolled to the permanent `greenfield` snapshot, `next` lane, package `0.0.50-next.2507`. ## Cause `packages/mosaic/framework/systemd/user/[email protected]` gates on the **config** home: ``` ConditionPathExists=%h/.config/mosaic/fleet/agents/%i.env.generated ``` On a roster-v2 fleet in split mode the reconciler writes the generated env to the **brain** home. After a clean `fleet init --write` + `fleet install` + `fleet start`: ``` present: ~/.mosaic/fleet/agents/{orchestrator,enhancer,generalist}.env.generated absent : ~/.config/mosaic/fleet/agents/ (directory does not exist) ``` The condition can therefore never be met, and an unmet condition is reported as success: ``` orchestrator ConditionResult=no Result=success ExecMainStatus=0 inactive/dead enhancer ConditionResult=no Result=success ExecMainStatus=0 inactive/dead generalist ConditionResult=no Result=success ExecMainStatus=0 inactive/dead ``` journal: `[email protected] ... skipped, unmet condition check ConditionPathExists=/home/mosaic/.config/mosaic/fleet/agents/orchestrator.env.generated` ## Causal test, not correlation One seat's env file copied into the config home. Nothing else changed. The other two seats are the untouched negative controls: ``` orchestrator ConditionResult=yes ActiveState=active session appears on the fleet socket enhancer ConditionResult=no inactive/dead (control) generalist ConditionResult=no inactive/dead (control) ``` The started seat's pane shows the Mosaic identity wizard, i.e. it reaches the authentication boundary — so nothing downstream of the condition is implicated. ## Controls on the measurement - `mosaic fleet no-such-subcommand-zq7` → RC=1, run first, so the RC=0 results are not a CLI that returns 0 for everything. - `tmux -L mosaic-fleet has-session zzz-nonce-8d2e` → `can't find session`, so the `_holder`-only session list is a real reading of that socket. - `[email protected]` on the VM is sha256 `3614731f70a3326e4409c5fe1e1708289d8afb6fed4ff0d4dd87b11d75f074f2`, byte-identical to `origin/next` — this is the shipped unit, not a local edit. Caution for anyone reproducing: **a unit that does not exist also reports `ConditionResult=no`**, so that field alone cannot tell "skipped" from "absent". I lost time to this by querying `mosaic-fleet-holder.service`, which is not a real unit name (`mosaic-tmux-holder.service` is). Confirm the unit exists first. ## The existing test cannot observe this `packages/mosaic/src/commands/fleet-roster-v2-dispatch.spec.ts:214-255` names exactly this drift: ```ts it('guards exactly the file the fleet writes, so the two cannot drift apart', ...) ``` Its fixture calls `v2Home()`, a custom mosaic-home. `resolveBrainHome()` deliberately never adopts `~/.mosaic` when `--mosaic-home` is custom (hermetic sandboxes), so inside the fixture the brain home always equals the config home and the two paths cannot come apart. The assertion passes in both the broken and the fixed state. Any fix needs a fixture that reaches split mode — a real default home, or `MOSAIC_BRAIN_HOME` set explicitly — or it will stay green over the change. ## Fix direction The condition is a deliberate boot-window guard; the in-file comment explains that `install` enables the unit before the reconciler has written env, and that a skipped unit is the honest state for "enabled but not yet configured". A fix should keep that intent: - **(preferred)** have `fleet install` write a per-host drop-in that clears and re-points the condition at the resolved brain home — `ConditionPathExists=` (empty, to reset) followed by the correct path with `%i`; or - have the brain-home-aware launcher own the check and exit cleanly when env is absent, dropping the condition. Deleting the condition outright loses the boot-window behaviour the comment documents. ## Scope Measured on one host, one VM, one package version, split mode only. Legacy (non-split) mode on the same snapshot does **not** hit this — its blocker is #1256. I have not measured whether web1's current install is split or legacy. This blocks the web1 fleet migration: on a host where `~/.mosaic/fleet/agents` exists, the fleet reports healthy and is empty. -- fred (sb-it-1-dt)
fred added the bug label 2026-08-18 18:57:20 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1310