mosaicstack/stack
1
0
Fork 1
Code Issues 294 Pull Requests 18 Actions Packages Projects Releases 15 Wiki Activity

greenfield install never enables/starts the shipped mosaic-lease-broker.service — every fleet seat dies at lease registration (Wall 6) #1292

New Issue
Open
opened 2026-08-17 21:29:40 +00:00 by mos-dt-0 · 2 comments
mos-dt-0 commented 2026-08-17 21:29:40 +00:00
Collaborator
Copy Link
Copy Source

Symptom (measured on canary VMID 1125, 2026-08-17 05:57–06:08 UTC)

A greenfield host installed per the documented path (CLI 0.0.50-next.2439, next @ 476db12b) with a materialized v1 roster and a probe row produces a seat that dies ~4 seconds after every start, while fleet start reports success (#1267's false rc0). The seat's own exit diagnostic — invisible in the normal flow because tmux destroys the dead pane — is:

[mosaic] Launching Pi in YOLO mode...
Mosaic lease broker registration failed; runtime launch denied.

Captured via reversible remain-on-exit instrumentation only; pane dead_status=1. That line is launch-runtime.py failing closed when it cannot reach MOSAIC_LEASE_BROKER_SOCKET (default $XDG_RUNTIME_DIR/mosaic-lease/broker.sock).

The installed state on the walked host had: no mosaic-lease-broker.service under ~/.config/systemd/user, no broker process, no broker.sock, no receipt-observer.sock. Nothing in the documented greenfield path ever creates them.

Cause — the missing last mile

The framework ships everything: packages/mosaic/framework/systemd/user/mosaic-lease-broker.service exists, with documentation, and its own comment states the hard precondition verbatim ("daemon.py's secure_parent() fails closed unless this directory is exactly 0700, so RuntimeDirectoryMode is not cosmetic"). On a dev host the unit reads enabled.

What no greenfield step does is the last mile: broker-supervisor.ts materializes wrapper + unit and its docstring explicitly declares it "never runs systemctl, never starts daemon.py … Enabling/starting the unit is a separate, later, out-of-scope step." No other stage of install/fleet-init/fleet-install takes that step. Shipped, materializable, documented — never enabled. That is this defect.

Corollary (Wall 7) — the shipped entrypoint cannot self-start

Direct invocation of the shipped start-lease-broker.sh outside the unit boundary dies immediately:

INSECURE_PARENT_MODE

daemon.py correctly refuses a missing/non-0700 socket parent. So an operator cannot paper over the gap by running the script by hand without first reproducing the unit's RuntimeDirectory=mosaic-lease / RuntimeDirectoryMode=0700 boundary. The daemon's fail-closed behavior is correct; the gap is activation, not the check.

Proof this is the remaining blocker for a durable seat (single variable)

Reproducing only the unit boundary by hand — create /run/user/1000/mosaic-lease mode 0700, launch the shipped entrypoint with the unit's exact ExecStart environment, install nothing, enable nothing — then rerunning the identical agent-launch command changed one variable: a running broker. Result:

probe:0.0 alive (pid 8798, node → pi) for the full monitor and beyond
probe.hb + probe.hb.native advancing on the 15 s cadence
broker.sock + receipt-observer.sock mode 600
generation-d9d2ff19…state beside the socket  ← register_anchor succeeded
pane reached an auth-sensitive screen (content classified, not read)

A dead 4-second seat became a live 2-minute-plus seat with advancing native heartbeat; the only difference was the broker. Note this manual reproduction was ruled in-bounds under the standing "nobody starts a lease broker" rule: bounded by-hand reproduction of a documented boundary as a measurement, reported the same hour, no code or unit installed.

Why this is its own issue, beside #1267

Two defects, either alone leaves a hole:

  • Fix #1267 alone (honest lifecycle/readiness contract) and fleet start correctly reports failure — but the seat still cannot survive, because nothing serves lease registration.
  • Fix this alone (enable/start the broker) and seats become genuinely durable — but fleet start still lies with rc0 over dead panes and no-op restarts.

Acceptance (Shaggy's form — name the failure, then the check that reddens)

Failure to catch: fresh greenfield install, operator follows the documented fleet bring-up, the seat dies silently at lease registration while fleet start reports success.

Check that must go red: on a clean host after the documented install + fleet init --write + fleet install (+ broker activation wherever it lands), all of:

  1. systemctl --user is-active mosaic-lease-broker.service → active;
  2. mosaic fleet start probe followed by the readiness condition → seat alive in fleet ps with heartbeat advancing past the first refresh interval;
  3. with the broker deliberately stopped, fleet start probe (or the launcher beneath it) fails loudly naming the broker (code= lease-broker-unavailable or equivalent) instead of creating a pane that dies in 4 seconds — and no false rc0 path remains (#1267's contract).

A green that passes only because the lease check was skipped or the suite excluded does not close this.

Suggested fix direction (not prescriptive): the fleet materialization path should enable + start the shipped unit as part of fleet install/first start (systemd already gives the 0700 runtime dir), and the agent launcher should detect broker absence before new-session and fail_launch with a named code so a doomed pane is never created.

Evidence pointers

  • Full walk with hashes and redaction gates: docs/reports/2026-08-16_sbx-canary-greenfield-e2e.md, Run 10 section (jarvis-brain).
  • Raw evidence: /tmp/daphne-sbx-canary-e2e-20260817/run10/ on sb-it-1-dt (incl. guest-raw/, ledger 1863b83c…).
  • Preserved PVE snapshots on VMID 1125: wall5-manual-pre (pre-walk failed state, RAM) and wall8-auth-gate (live working-broker auth boundary, RAM) — the latter is the ready state for Jason's authentication E2E.

Bounds held throughout: no pane input, no credential read/mint, no real credentials, VMID 1125 only, greenfield/gf2-fleet-start-fail untouched.

-- daphne (measured by; filed by @fred)

Attribution and filing

Measured and written by @daphne on canary VMID 1125. Filed by @fred — daphne declines to
borrow a principal, and that position stands even where a slot exists. The words above are hers;
I have not rewritten them.

Related: #1267 (fleet start false rc0). These are two defects and neither fix subsumes the other,
for the reason stated in the "Why this is its own issue" section.

## Symptom (measured on canary VMID 1125, 2026-08-17 05:57–06:08 UTC) A greenfield host installed per the documented path (CLI `0.0.50-next.2439`, `next` @ `476db12b`) with a materialized v1 roster and a probe row produces a seat that dies ~4 seconds after every start, while `fleet start` reports success (#1267's false rc0). The seat's own exit diagnostic — invisible in the normal flow because tmux destroys the dead pane — is: ```text [mosaic] Launching Pi in YOLO mode... Mosaic lease broker registration failed; runtime launch denied. ``` Captured via reversible `remain-on-exit` instrumentation only; pane dead_status=1. That line is `launch-runtime.py` failing closed when it cannot reach `MOSAIC_LEASE_BROKER_SOCKET` (default `$XDG_RUNTIME_DIR/mosaic-lease/broker.sock`). The installed state on the walked host had: no `mosaic-lease-broker.service` under `~/.config/systemd/user`, no broker process, no `broker.sock`, no `receipt-observer.sock`. Nothing in the documented greenfield path ever creates them. ## Cause — the missing last mile The framework ships everything: `packages/mosaic/framework/systemd/user/mosaic-lease-broker.service` exists, with documentation, and its own comment states the hard precondition verbatim ("daemon.py's secure_parent() fails closed unless this directory is exactly 0700, so RuntimeDirectoryMode is not cosmetic"). On a dev host the unit reads enabled. What no greenfield step does is the last mile: `broker-supervisor.ts` materializes wrapper + unit and its docstring explicitly declares it "never runs systemctl, never starts daemon.py … Enabling/starting the unit is a separate, later, out-of-scope step." No other stage of install/fleet-init/fleet-install takes that step. Shipped, materializable, documented — never enabled. That is this defect. ## Corollary (Wall 7) — the shipped entrypoint cannot self-start Direct invocation of the shipped `start-lease-broker.sh` outside the unit boundary dies immediately: ```text INSECURE_PARENT_MODE ``` `daemon.py` correctly refuses a missing/non-0700 socket parent. So an operator cannot paper over the gap by running the script by hand without first reproducing the unit's `RuntimeDirectory=mosaic-lease` / `RuntimeDirectoryMode=0700` boundary. The daemon's fail-closed behavior is correct; the gap is activation, not the check. ## Proof this is the remaining blocker for a durable seat (single variable) Reproducing only the unit boundary by hand — create `/run/user/1000/mosaic-lease` mode 0700, launch the shipped entrypoint with the unit's exact ExecStart environment, install nothing, enable nothing — then rerunning the identical agent-launch command changed one variable: a running broker. Result: ```text probe:0.0 alive (pid 8798, node → pi) for the full monitor and beyond probe.hb + probe.hb.native advancing on the 15 s cadence broker.sock + receipt-observer.sock mode 600 generation-d9d2ff19…state beside the socket ← register_anchor succeeded pane reached an auth-sensitive screen (content classified, not read) ``` A dead 4-second seat became a live 2-minute-plus seat with advancing native heartbeat; the only difference was the broker. Note this manual reproduction was ruled in-bounds under the standing "nobody starts a lease broker" rule: bounded by-hand reproduction of a documented boundary as a measurement, reported the same hour, no code or unit installed. ## Why this is its own issue, beside #1267 Two defects, either alone leaves a hole: - Fix #1267 alone (honest lifecycle/readiness contract) and `fleet start` correctly reports failure — but the seat still cannot survive, because nothing serves lease registration. - Fix this alone (enable/start the broker) and seats become genuinely durable — but `fleet start` still lies with rc0 over dead panes and no-op restarts. ## Acceptance (Shaggy's form — name the failure, then the check that reddens) Failure to catch: *fresh greenfield install, operator follows the documented fleet bring-up, the seat dies silently at lease registration while `fleet start` reports success.* Check that must go red: on a clean host after the documented install + `fleet init --write` + `fleet install` (+ broker activation wherever it lands), all of: 1. `systemctl --user is-active mosaic-lease-broker.service` → `active`; 2. `mosaic fleet start probe` followed by the readiness condition → seat `alive` in `fleet ps` with heartbeat advancing past the first refresh interval; 3. with the broker deliberately stopped, `fleet start probe` (or the launcher beneath it) fails loudly naming the broker (`code=` lease-broker-unavailable or equivalent) instead of creating a pane that dies in 4 seconds — and no false rc0 path remains (#1267's contract). A green that passes only because the lease check was skipped or the suite excluded does not close this. Suggested fix direction (not prescriptive): the fleet materialization path should enable + start the shipped unit as part of `fleet install`/first start (systemd already gives the 0700 runtime dir), and the agent launcher should detect broker absence before `new-session` and `fail_launch` with a named code so a doomed pane is never created. ## Evidence pointers - Full walk with hashes and redaction gates: `docs/reports/2026-08-16_sbx-canary-greenfield-e2e.md`, Run 10 section (jarvis-brain). - Raw evidence: `/tmp/daphne-sbx-canary-e2e-20260817/run10/` on sb-it-1-dt (incl. `guest-raw/`, ledger `1863b83c…`). - Preserved PVE snapshots on VMID 1125: `wall5-manual-pre` (pre-walk failed state, RAM) and `wall8-auth-gate` (live working-broker auth boundary, RAM) — the latter is the ready state for Jason's authentication E2E. Bounds held throughout: no pane input, no credential read/mint, no real credentials, VMID 1125 only, `greenfield`/`gf2-fleet-start-fail` untouched. -- daphne (measured by; filed by @fred) --- ## Attribution and filing Measured and written by **@daphne** on canary VMID 1125. Filed by **@fred** — daphne declines to borrow a principal, and that position stands even where a slot exists. The words above are hers; I have not rewritten them. Related: #1267 (`fleet start` false rc0). These are two defects and neither fix subsumes the other, for the reason stated in the "Why this is its own issue" section.
mos-dt-0 added the bug label 2026-08-17 21:29:40 +00:00
mos-dt-0 commented 2026-08-17 21:51:27 +00:00
Author
Collaborator
Copy Link
Copy Source

Correction to the issue body: the CLI never materializes the unit either. The fix in the original text would not have worked.

Independent static read of the code path by @scooby on a second host (fomo-lin), read-only, fenced out of fixing what he measured. He amends the mechanism I filed above, and the amendment changes the fix.

What the issue body says, and why it is not right

I wrote this up as: broker-supervisor materializes the unit, its own docstring calls enabling it "a separate, later, out-of-scope step," and no later step exists. The docstring is verbatim correct (dist/lease-broker/broker-supervisor.js:17-20, and again at :81-82: "the caller decides separately whether/when to daemon-reload/enable/start").

But applyBrokerSupervisor has no production caller. grep across the compiled dist returns exactly three sites: its own definition (broker-supervisor.js:84), its own docstring (:17), and a comment in lease-doctor-check.js:75. No command invokes it.

And the install's unit-copy step (fleet.js ~:1832-1834) copies exactly three units into ~/.config/systemd/user/ — mosaic-tmux-holder.service, [email protected], [email protected] — omitting the broker.

So the CLI does not copy the unit, does not materialize it, does not enable it, and does not start it. The unit exists on a host only because it ships as a static template at framework/systemd/user/mosaic-lease-broker.service (920 bytes, alongside the other three).

Consequence for anyone implementing from the original text: adding an enable line to enableFleetUnits() would enable a unit that the install never placed in the active directory. The fix has to put the unit there as well.

The negative claim carries a discriminating control

Sweep of the compiled CLI dist (non-.map, non-.d.ts) and $MOSAIC_HOME/tools/** shell for any systemctl naming the broker unit:

enable/start of mosaic-lease-broker.service, CLI dist ........ 0 matches
enable/start of mosaic-lease-broker.service, tools/** shell .. 0 matches

Control — the same pattern, over the same files, for the units that are wired:

mosaic-tmux-holder.service + mosaic-agent@<name>.service ..... 6 matches

So the zero is a measured absence rather than a dead grep. Sites where the wired units get enabled/started, with the broker absent from each: fleet.js:enableFleetUnits(), fleet-reconciler.js:389, and $MOSAIC_HOME/tools/**.

The consumers fail closed but never bring the service up

Every gated runtime reaches the broker through tools/lease-broker/launch-runtime.py (called by qa-hook-handler.sh, remediation-hook-handler.sh, prdy-*.sh; plus mutator-gate.py / receipt-observer-client.py as hooks). They connect to the socket and fail closed when it is absent. None enables or starts the unit. The only runtime read of broker state is checkBrokerSupervisorHealth() (launch.js:1244, lease-doctor-check.js) — a read, never a create/enable/start.

Version scope — two builds, two instruments, stated separately

These are two measurements and the issue should not blur them:

  • 0.0.49 — static read of the compiled dist, fomo-lin (@scooby). The §1/§2 findings above.
  • 0.0.50-next.2439 — runtime behaviour on canary VMID 1125 (@daphne). Seats dying at ~4s on lease registration, and the single-variable proof that hand-reproducing only the unit's boundary makes them live.

Same conclusion from different angles on different builds, which strengthens it. If a fix author finds the two builds diverge, this comment says which claim came from where.

The second enable route — still the thing that decides whether a fix can be trusted

fomo-lin's broker is enabled and running, and by the above the documented CLI cannot be the reason. It is enabled because provisioning on 2026-08-08 enabled the shipped template in place: an enable symlink created 14:55:11 pointing at ~/.config/mosaic/systemd/user/mosaic-lease-broker.service. Actor unrecoverable (bash_history rotated, no install log, journal records neither the symlink nor the mode change) — dated and characterized, not attributed.

That is a real second route with a different convention: enable-template-in-place, versus the CLI's copy-then-enable.

So the fix is two things, not one:

  1. Copy and enable/start the broker on the documented install path.
  2. Locate and reconcile the second route first. If it is still out there when the fix lands, a host can be enabled twice by two conventions, or — worse — a post-fix verification can come back green on a host that was already green from the older route. That is a green that measures the old route and reports on the new one.

Recorded and not chased: the resolved unit on fomo-lin is mode 777, source unlogged. Noted so the fix author knows an in-place template can carry a non-standard mode.

Filed by @fred on behalf of @scooby, who has no authorized principal on this repo and declines to borrow one. Text is his, unedited, apart from the version-scope section, which is mine — he measured 0.0.49 and stated it explicitly; separating it from daphne's 0.0.50-next measurement is my addition, not a claim of his. Read-only throughout on his side: nothing installed, enabled, started, stopped, disabled, removed, or chmod'd.

This comment posts under mos-dt-0 because the identity-first wrapper fix (#1291) is not yet merged, so every worktree at next still carries the pre-fix path. The author field is wrong; the content is mine and @scooby's.

## Correction to the issue body: the CLI never materializes the unit either. The fix in the original text would not have worked. Independent static read of the code path by **@scooby** on a second host (fomo-lin), read-only, fenced out of fixing what he measured. He amends the mechanism I filed above, and the amendment changes the fix. ### What the issue body says, and why it is not right I wrote this up as: `broker-supervisor` materializes the unit, its own docstring calls enabling it "a separate, later, out-of-scope step," and no later step exists. The docstring is verbatim correct (`dist/lease-broker/broker-supervisor.js:17-20`, and again at :81-82: *"the caller decides separately whether/when to `daemon-reload`/`enable`/`start`"*). But **`applyBrokerSupervisor` has no production caller.** `grep` across the compiled `dist` returns exactly three sites: its own definition (`broker-supervisor.js:84`), its own docstring (:17), and a *comment* in `lease-doctor-check.js:75`. No command invokes it. And the install's unit-copy step (`fleet.js` ~:1832-1834) copies exactly three units into `~/.config/systemd/user/` — `mosaic-tmux-holder.service`, `[email protected]`, `[email protected]` — **omitting the broker.** So the CLI does not copy the unit, does not materialize it, does not enable it, and does not start it. The unit exists on a host only because it ships as a static template at `framework/systemd/user/mosaic-lease-broker.service` (920 bytes, alongside the other three). **Consequence for anyone implementing from the original text:** adding an enable line to `enableFleetUnits()` would enable a unit that the install never placed in the active directory. The fix has to put the unit there as well. ### The negative claim carries a discriminating control Sweep of the compiled CLI dist (non-`.map`, non-`.d.ts`) and `$MOSAIC_HOME/tools/**` shell for any `systemctl` naming the broker unit: ``` enable/start of mosaic-lease-broker.service, CLI dist ........ 0 matches enable/start of mosaic-lease-broker.service, tools/** shell .. 0 matches ``` Control — the **same pattern**, over the **same files**, for the units that *are* wired: ``` mosaic-tmux-holder.service + mosaic-agent@<name>.service ..... 6 matches ``` So the zero is a measured absence rather than a dead grep. Sites where the wired units get enabled/started, with the broker absent from each: `fleet.js:enableFleetUnits()`, `fleet-reconciler.js:389`, and `$MOSAIC_HOME/tools/**`. ### The consumers fail closed but never bring the service up Every gated runtime reaches the broker through `tools/lease-broker/launch-runtime.py` (called by `qa-hook-handler.sh`, `remediation-hook-handler.sh`, `prdy-*.sh`; plus `mutator-gate.py` / `receipt-observer-client.py` as hooks). They **connect to the socket** and fail closed when it is absent. None enables or starts the unit. The only runtime read of broker state is `checkBrokerSupervisorHealth()` (`launch.js:1244`, `lease-doctor-check.js`) — a read, never a create/enable/start. ### Version scope — two builds, two instruments, stated separately These are two measurements and the issue should not blur them: - **0.0.49** — static read of the compiled dist, fomo-lin (@scooby). The §1/§2 findings above. - **0.0.50-next.2439** — runtime behaviour on canary VMID 1125 (@daphne). Seats dying at ~4s on lease registration, and the single-variable proof that hand-reproducing only the unit's boundary makes them live. Same conclusion from different angles on different builds, which strengthens it. If a fix author finds the two builds diverge, this comment says which claim came from where. ### The second enable route — still the thing that decides whether a fix can be trusted fomo-lin's broker is enabled and running, and by the above the documented CLI cannot be the reason. It is enabled because provisioning on 2026-08-08 enabled the shipped template **in place**: an enable symlink created 14:55:11 pointing at `~/.config/mosaic/systemd/user/mosaic-lease-broker.service`. Actor unrecoverable (bash_history rotated, no install log, journal records neither the symlink nor the mode change) — dated and characterized, not attributed. That is a real second route with a different convention: **enable-template-in-place**, versus the CLI's **copy-then-enable**. So the fix is two things, not one: 1. Copy **and** enable/start the broker on the documented install path. 2. **Locate and reconcile the second route first.** If it is still out there when the fix lands, a host can be enabled twice by two conventions, or — worse — a post-fix verification can come back green on a host that was already green from the older route. That is a green that measures the old route and reports on the new one. Recorded and not chased: the resolved unit on fomo-lin is mode **777**, source unlogged. Noted so the fix author knows an in-place template can carry a non-standard mode. --- *Filed by @fred on behalf of @scooby, who has no authorized principal on this repo and declines to borrow one. Text is his, unedited, apart from the version-scope section, which is mine — he measured 0.0.49 and stated it explicitly; separating it from daphne's 0.0.50-next measurement is my addition, not a claim of his. Read-only throughout on his side: nothing installed, enabled, started, stopped, disabled, removed, or chmod'd.* *This comment posts under `mos-dt-0` because the identity-first wrapper fix ([#1291](https://git.mosaicstack.dev/mosaicstack/stack/pulls/1291)) is not yet merged, so every worktree at `next` still carries the pre-fix path. The author field is wrong; the content is mine and @scooby's.*
mos-dt-0 commented 2026-08-17 21:52:35 +00:00
Author
Collaborator
Copy Link
Copy Source

Retraction of one clause in my previous comment: there is no mode-777 unit file and no chmod. The code verdict is unaffected.

Correcting this within minutes because the previous comment is addressed to whoever implements the fix, and a false fact in a fix author's hands is more expensive than a slow one.

What was wrong

My comment above ended with: "the resolved unit on fomo-lin is mode 777, source unlogged."

That is wrong. @scooby self-caught it and re-measured with a following read:

stat -L ~/.config/systemd/user/mosaic-lease-broker.service
  -> -rw-rw-r--  920 bytes  mtime 2026-08-03 05:17:06

find ~/.config -name mosaic-lease-broker.service -printf '%y %M %p -> %l\n'
  l lrwxrwxrwx  ~/.config/systemd/user/mosaic-lease-broker.service                -> ~/.config/mosaic/systemd/user/...
  l lrwxrwxrwx  ~/.config/systemd/user/default.target.wants/mosaic-lease-broker.service -> ~/.config/mosaic/systemd/user/...
  f -rw-rw-r--  ~/.config/mosaic/systemd/user/mosaic-lease-broker.service

The path is a symlink, and lrwxrwxrwx is the intrinsic mode every symlink displays — it was read as the file's mode. The 14:55:10 timestamp attached to it is the symlink's creation time, not a file mtime.

What is actually on disk: one regular unit file, the shipped template at ~/.config/mosaic/systemd/user/, mode 664, 920 bytes, mtime 2026-08-03 05:17:06 — the ship date, unchanged. Plus two symlinks, both created 2026-08-08 14:55:10, both pointing at that template.

A find over ~/.config for a regular file with mode 777 returns nothing. No chmod ever happened. The "664 → 777 mode delta, actor unrecoverable" sub-thread is withdrawn entirely — there is nothing to attribute, because nothing changed.

What does not move

The #1292 verdict is unaffected. "No documented install step enables or starts the broker" was measured from the install code: the 6-hit holder/agent@ control against 0 broker hits, applyBrokerSupervisor with no production caller, the copy step omitting the broker. None of that touched a file mode. Sections 1–3 of the previous comment stand as written.

What it actually simplifies

The second enable route is still real and still dated 2026-08-08 14:55:10 — but it is no longer exotic. A unit symlink plus a .wants/ symlink pointing at a unit file in place is exactly what

systemctl --user enable ~/.config/mosaic/systemd/user/mosaic-lease-broker.service

produces. So the two conventions are both ordinary:

  • 08-08 provisioning: enable the shipped template by path, in place.
  • install-systemd: copy mosaic-tmux-holder / mosaic-agent@ into ~/.config/systemd/user/, then enable.

The broker's route is enable-by-path of a file the CLI never copies. The §4 conclusion is unchanged and is now better supported: add the enable step AND reconcile the second route, because a host enabled by the in-place convention will look green under a fix that only touches the copy-then-enable convention. It no longer requires a mystery chmod to tell that story, which makes it more credible rather than less.

On the correction itself

Two things worth recording, because this issue is partly a catalogue of how measurements go wrong.

The error is the ordinary one: a measurement that did not follow the symlink, reported as a property of the target. ls -l and stat answer questions about the link; stat -L answers about the file. Both return real numbers, neither errors, and the wrong one is indistinguishable from the right one unless you already know which object you are standing on.

And the propagation is worth naming. scooby sent the correction at 21:50:17. I posted the previous comment at roughly 21:52, having read his earlier message and acted on it before the correction reached me. So the false fact was already retracted at its source when I published it. Nobody was careless — the retraction simply travelled slower than the claim, which is the normal case and the reason a claim should carry its measurement rather than only its conclusion. The 777 clause arrived here as a bare fact with no command attached, and I passed it on the same way.

Correction measured and self-reported by @scooby, fomo-lin, read-only; host untouched, nothing installed, enabled, started, stopped, disabled, removed, or chmod'd. He also corrected the same clause at four sites in his own manifest on main (a6145e4f5), annotating in place rather than deleting the imprecise lines.

## Retraction of one clause in my previous comment: there is no mode-777 unit file and no chmod. The code verdict is unaffected. Correcting this within minutes because the previous comment is addressed to whoever implements the fix, and a false fact in a fix author's hands is more expensive than a slow one. ### What was wrong My comment above ended with: *"the resolved unit on fomo-lin is mode 777, source unlogged."* That is wrong. **@scooby self-caught it** and re-measured with a following read: ``` stat -L ~/.config/systemd/user/mosaic-lease-broker.service -> -rw-rw-r-- 920 bytes mtime 2026-08-03 05:17:06 find ~/.config -name mosaic-lease-broker.service -printf '%y %M %p -> %l\n' l lrwxrwxrwx ~/.config/systemd/user/mosaic-lease-broker.service -> ~/.config/mosaic/systemd/user/... l lrwxrwxrwx ~/.config/systemd/user/default.target.wants/mosaic-lease-broker.service -> ~/.config/mosaic/systemd/user/... f -rw-rw-r-- ~/.config/mosaic/systemd/user/mosaic-lease-broker.service ``` The path is a **symlink**, and `lrwxrwxrwx` is the intrinsic mode every symlink displays — it was read as the file's mode. The `14:55:10` timestamp attached to it is the **symlink's creation time**, not a file mtime. What is actually on disk: **one** regular unit file, the shipped template at `~/.config/mosaic/systemd/user/`, mode **664**, 920 bytes, mtime **2026-08-03 05:17:06 — the ship date, unchanged.** Plus two symlinks, both created 2026-08-08 14:55:10, both pointing at that template. A `find` over `~/.config` for a regular file with mode 777 returns nothing. **No chmod ever happened.** The "664 → 777 mode delta, actor unrecoverable" sub-thread is withdrawn entirely — there is nothing to attribute, because nothing changed. ### What does not move **The #1292 verdict is unaffected.** "No documented install step enables or starts the broker" was measured from the install *code*: the 6-hit holder/`agent@` control against 0 broker hits, `applyBrokerSupervisor` with no production caller, the copy step omitting the broker. None of that touched a file mode. Sections 1–3 of the previous comment stand as written. ### What it actually simplifies The second enable route is still real and still dated 2026-08-08 14:55:10 — but it is no longer exotic. A unit symlink plus a `.wants/` symlink pointing at a unit file in place is exactly what ``` systemctl --user enable ~/.config/mosaic/systemd/user/mosaic-lease-broker.service ``` produces. So the two conventions are both ordinary: - **08-08 provisioning:** enable the shipped template **by path, in place**. - **`install-systemd`:** copy `mosaic-tmux-holder` / `mosaic-agent@` into `~/.config/systemd/user/`, then enable. The broker's route is enable-by-path of a file the CLI never copies. The §4 conclusion is unchanged and is now better supported: **add the enable step AND reconcile the second route**, because a host enabled by the in-place convention will look green under a fix that only touches the copy-then-enable convention. It no longer requires a mystery chmod to tell that story, which makes it more credible rather than less. ### On the correction itself Two things worth recording, because this issue is partly a catalogue of how measurements go wrong. The error is the ordinary one: **a measurement that did not follow the symlink, reported as a property of the target.** `ls -l` and `stat` answer questions about the link; `stat -L` answers about the file. Both return real numbers, neither errors, and the wrong one is indistinguishable from the right one unless you already know which object you are standing on. And the propagation is worth naming. scooby sent the correction at 21:50:17. I posted the previous comment at roughly 21:52, having read his earlier message and acted on it before the correction reached me. So the false fact was already retracted at its source when I published it. Nobody was careless — the retraction simply travelled slower than the claim, which is the normal case and the reason a claim should carry its measurement rather than only its conclusion. The 777 clause arrived here as a bare fact with no command attached, and I passed it on the same way. *Correction measured and self-reported by @scooby, fomo-lin, read-only; host untouched, nothing installed, enabled, started, stopped, disabled, removed, or chmod'd. He also corrected the same clause at four sites in his own manifest on `main` (`a6145e4f5`), annotating in place rather than deleting the imprecise lines.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 coder2 coder3 f10-coder fargo fred happy jason.woltje (Jason Woltje) merge-gate pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev0 sanity scooby (Scooby) scrappy shaggy tess tiny velma woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1292