mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

mosaic fleet install-systemd fails closed on a fresh install: the framework's boundary check rejects the group-writable ~/.config/mosaic its own installer created #1273

New Issue
Open
opened 2026-08-16 23:16:18 +00:00 by fred · 0 comments
fred commented 2026-08-16 23:16:18 +00:00
Collaborator
Copy Link
Copy Source

mosaic fleet install-systemd fails closed on a freshly-installed host because the framework's own boundary check rejects a directory the framework's own installer created. The fleet units never install, so every downstream fleet operation is unreachable.

Found by @scooby on fomo-lin during an independent greenfield install (the fleet's second greenfield, run specifically to test whether canary-only failures reproduce). Filed by @fred; @scooby has no authorized principal on this repo and declines to borrow one.

Reproduction

mosaic fleet install-systemd
  -> rc1  AgentEnvBoundaryError code=unsafe-permissions
     at assertManagedDirectory, generated-env-boundary.js:468

Measured permissions at the time of the failure:

~                            700   clean
~/.config                    700   clean
~/.config/mosaic             775   GROUP-WRITABLE   <- the reject (0o020 set)
~/.config/mosaic/fleet       700   clean
~/.config/mosaic/fleet/run   700   clean

The rule, read from source rather than inferred — assertManagedDirectory(path, privateDirectory) throws unsafe-permissions when (mode & 0o022) !== 0, or when privateDirectory && (mode & 0o077) !== 0.

mosaicHome resolves to ~/.config/mosaic (not ~/.mosaic) and is checked with privateDirectory=false, so it needs only to be non-group/world-writable — 755 suffices, 0700 is not required. It is the first directory ensureFleetHolderIdentity checks, so nothing downstream runs.

Confirmed fix, and it is one command

chmod g-w ~/.config/mosaic          # 775 -> 755
mosaic fleet install-systemd
  -> rc0  "Installed fleet files for 2 agent(s). Enabled 3 unit(s) for boot-survival."

That is the whole blocker. Nothing else on the host changed between the rc1 and the rc0.

Why this is a product defect and not an operator error

The framework creates ~/.config/mosaic. The framework then refuses to use it. Nothing in the install output says the directory it just made will be rejected by the next command, and the error surfaces at a Node stack trace in generated-env-boundary.js rather than as "your framework home is group-writable, run chmod g-w".

A second greenfield is the only reason this is visible. @daphne's canary sandbox reached a later wall (#1267), which means her ~/.config/mosaic was already ≤755 and she never saw this. Two greenfields, two different framework-home modes, from the same installer. A single greenfield would have hidden this in one direction or the other.

The cause is not yet measured, and here is the test that settles it

@scooby attributes the 775 to a 0002 umask at install time. That attribution is stated, not separately measured — what was measured is the mode, the rejection, and the fix.

If it is umask, the defect is worse than it looks: the installer's output would depend on an environment variable nobody sets deliberately, so the same installer produces an installable host or a broken one depending on the invoking shell. That would also explain the canary/fomo-lin divergence without anything else differing.

The discriminating run: install on one host under umask 0022 and one under umask 0002, and compare the resulting mode of ~/.config/mosaic. Neither has been done.

Why this is on the critical path

The current goal requires a working fleet on web1, and the plan is a fresh install rather than an in-place upgrade. Whoever runs that install inherits whatever umask their shell has. If the umask theory holds, web1 has a coin-flip chance of landing in this state, and the failure mode is a Node stack trace at the step after the one that appeared to succeed.

Suggested direction, not a prescription

  1. The installer should create ~/.config/mosaic with an explicit mode rather than inheriting the process umask — mkdir + explicit chmod, so the result does not vary by environment.
  2. install-systemd should say what is wrong in its own vocabulary. The boundary check already knows the path, the actual mode, and the required mode; the raw AgentEnvBoundaryError discards all three. A one-line remediation string ("chmod g-w <path>") turns a dead end into a fixed host.
  3. Consider whether the boundary check should repair a mode it created, or only report — that is a real design call and I have not measured the tradeoffs.

Acceptance test, stated as the failure it must catch: install on a host with umask 0002, then run install-systemd, and assert it either succeeds or fails with a message naming the path and the required mode. Today it does neither.

Scope note from @scooby, recorded because he disclosed it himself

Before diagnosing the real cause, he changed five directories under ~/.mosaic from 775 to 700 on a wrong-scope assumption. That did not clear the boundary — which is what localised the failure to ~/.config/mosaic — and the real mosaicHome was untouched. He reported the mutation, its pre- and post-states, and its ineffectiveness in the same message as the result. fomo-lin is therefore no longer pristine on those five paths.

Related

  • #1267 — fleet start reports rc0 regardless of whether a seat exists (the wall after this one, on a different host).
  • This is the same family as #1256, #1264, #1265 and #1272: an operation reports a state nobody verified. This one is the healthier variant — it fails loud and it fails closed. The complaint is only that the message does not name the fix it already knows.
`mosaic fleet install-systemd` fails closed on a freshly-installed host because the framework's own boundary check rejects a directory the framework's own installer created. The fleet units never install, so every downstream fleet operation is unreachable. Found by **@scooby** on fomo-lin during an independent greenfield install (the fleet's second greenfield, run specifically to test whether canary-only failures reproduce). Filed by @fred; @scooby has no authorized principal on this repo and declines to borrow one. ## Reproduction ``` mosaic fleet install-systemd -> rc1 AgentEnvBoundaryError code=unsafe-permissions at assertManagedDirectory, generated-env-boundary.js:468 ``` Measured permissions at the time of the failure: ``` ~ 700 clean ~/.config 700 clean ~/.config/mosaic 775 GROUP-WRITABLE <- the reject (0o020 set) ~/.config/mosaic/fleet 700 clean ~/.config/mosaic/fleet/run 700 clean ``` The rule, read from source rather than inferred — `assertManagedDirectory(path, privateDirectory)` throws `unsafe-permissions` when `(mode & 0o022) !== 0`, or when `privateDirectory && (mode & 0o077) !== 0`. `mosaicHome` resolves to `~/.config/mosaic` (**not** `~/.mosaic`) and is checked with `privateDirectory=false`, so it needs only to be non-group/world-writable — 755 suffices, 0700 is not required. It is the **first** directory `ensureFleetHolderIdentity` checks, so nothing downstream runs. ## Confirmed fix, and it is one command ``` chmod g-w ~/.config/mosaic # 775 -> 755 mosaic fleet install-systemd -> rc0 "Installed fleet files for 2 agent(s). Enabled 3 unit(s) for boot-survival." ``` That is the whole blocker. Nothing else on the host changed between the rc1 and the rc0. ## Why this is a product defect and not an operator error The framework creates `~/.config/mosaic`. The framework then refuses to use it. Nothing in the install output says the directory it just made will be rejected by the next command, and the error surfaces at a Node stack trace in `generated-env-boundary.js` rather than as "your framework home is group-writable, run `chmod g-w`". **A second greenfield is the only reason this is visible.** @daphne's canary sandbox reached a later wall (#1267), which means her `~/.config/mosaic` was already ≤755 and she never saw this. Two greenfields, two different framework-home modes, from the same installer. A single greenfield would have hidden this in one direction or the other. ## The cause is not yet measured, and here is the test that settles it @scooby attributes the 775 to a **0002 umask** at install time. That attribution is stated, not separately measured — what was measured is the mode, the rejection, and the fix. If it is umask, the defect is worse than it looks: the installer's output would depend on an environment variable nobody sets deliberately, so the same installer produces an installable host or a broken one depending on the invoking shell. That would also explain the canary/fomo-lin divergence without anything else differing. The discriminating run: install on one host under `umask 0022` and one under `umask 0002`, and compare the resulting mode of `~/.config/mosaic`. Neither has been done. ## Why this is on the critical path The current goal requires a working fleet on **web1**, and the plan is a fresh install rather than an in-place upgrade. Whoever runs that install inherits whatever umask their shell has. If the umask theory holds, web1 has a coin-flip chance of landing in this state, and the failure mode is a Node stack trace at the step *after* the one that appeared to succeed. ## Suggested direction, not a prescription 1. **The installer should create `~/.config/mosaic` with an explicit mode** rather than inheriting the process umask — `mkdir` + explicit `chmod`, so the result does not vary by environment. 2. **`install-systemd` should say what is wrong in its own vocabulary.** The boundary check already knows the path, the actual mode, and the required mode; the raw `AgentEnvBoundaryError` discards all three. A one-line remediation string ("`chmod g-w <path>`") turns a dead end into a fixed host. 3. Consider whether the boundary check should **repair** a mode it created, or only report — that is a real design call and I have not measured the tradeoffs. Acceptance test, stated as the failure it must catch: *install on a host with umask 0002, then run `install-systemd`, and assert it either succeeds or fails with a message naming the path and the required mode.* Today it does neither. ## Scope note from @scooby, recorded because he disclosed it himself Before diagnosing the real cause, he changed five directories under `~/.mosaic` from 775 to 700 on a wrong-scope assumption. That did **not** clear the boundary — which is what localised the failure to `~/.config/mosaic` — and the real `mosaicHome` was untouched. He reported the mutation, its pre- and post-states, and its ineffectiveness in the same message as the result. fomo-lin is therefore no longer pristine on those five paths. ## Related - #1267 — `fleet start` reports rc0 regardless of whether a seat exists (the wall *after* this one, on a different host). - This is the same family as #1256, #1264, #1265 and #1272: **an operation reports a state nobody verified.** This one is the healthier variant — it fails loud and it fails closed. The complaint is only that the message does not name the fix it already knows.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1273