fix(fleet): lease-broker activation, symlink-safe unit placement, named launch refusal — Wall 6 (#1292) #1297

2026-08-17T23:39:12Z

fargo commented

2026-08-17 23:39:12 +00:00

Wall 6 / #1292 — lease-broker activation, safe unit placement, and named launch refusal

Card implemented under fred's wave-3 rulings (design proposal → symlink measurement → amended design → GO with widened placement scope), branch fix/1292-lease-broker-activation, base next @ 8199261. Activation lands in the control plane, not the launcher.

The measured foundations (all on throwaway systemd user instances + fomo-lin read-only probes)

systemctl enable <name> does not rewrite an existing by-path wants-symlink — rc=0, silent, the host stays on the old copy indefinitely.
Node copyFile follows a by-path symlink at the destination and overwrites the seed template (verified with cp AND fs.copyFile).
fomo-lin (the only in-the-wild by-path host) survived #2 only because its one symlink was the one unit the install never copied — the residue/copy sets were disjoint by accident. Adding the broker to the copy set made them intersect on the first run, so unlink-before-copy is required for a host that exists today, not a hypothetical.

The fix

placeUnitFile — every unit the install places (all four, including the broker, previously omitted entirely) goes through one helper: unlink destination symlink → copy → remove stale wants-symlink (via readlink, not readFile) → daemon-reload. Idempotent (double-install convergence asserted). SET-INDEPENDENCE warning documented for whoever adds a fifth unit.
enableFleetUnits enables the broker first, alongside the holder.
fleet start / reconciler start the broker before any holder/agent effect, then re-check the socket (never unit ActiveState) and refuse nonzero with a named code if it did not appear — re-probed on every invocation, so RemainAfterExit=yes can never make retry look like repair (the sticky half).
Reconciler plan carries broker {unitInstalled, socketPresent} as a first-class member — "broker died an hour ago" is now reportable drift, not invisible.
start-agent-session.sh preflights the socket before any tmux effect: absent → exit 75 (EX_TEMPFAIL), named broker-absent message with socket path + remedy, no doomed pane. Exit 75 validated against the agent@ unit's measured Restart= policy (oneshot, none — no restart loop). The preflight detects and refuses; it never starts the broker.
mosaic doctor names one convention-neutral remedy: mosaic fleet install (it reconciles either enable convention).

Tests — name the failure, point at the check

Failure written down: a greenfield host's seat dies ~4s at lease registration while fleet start reports rc0; on retry, fleet start reports success again while the seat is still dead.

Placement (finding 2, the seed-integrity check fred added): by-path residue fixture → after install the seed template's bytes AND mtime are unchanged — the only check that can redden for copy-through. Plus the clean-host negative control, wants-residue clearing, and double-install idempotence.
Resulting-symlink assertions (finding 1): the residue cases assert on the resulting wants-symlink target and destination object type, never on the enable call's argument.
First start: broker absent → refused, zero holder/agent start calls emitted.
Second start (the sticky half): socket never present → second invocation refuses identically; no ActiveState trust.
Launcher: absent socket → exit 75 + no tmux session created (fake tmux asserts it); live socket passes; explicit env wins; --stop not fenced.
Reconciler: broker plan member with the enabled-but-dead shape (unitInstalled: true, socketPresent: false); broker-before-agent ordering in both command and apply paths.

Sabotage controls (all restored byte-identically, sha256-verified): placement unlink removed → exactly the seed-integrity test reddens (1/8); socket re-check disabled → exactly the two preflight specs redden; shell preflight removed → the bash suite reddens rc=1. All green after restore.

Gates (rc-honest): vitest 1563/1563 · lint rc=0 · root build 25/25 · root typecheck 45/45 · bash preflight suite green. The new bash suite is CI-fit by construction (self-controlled preconditions — fake tmux, real unix socket, hermetic env; comment on the file explains why the older test-start-agent-session.sh is not, and stays unwired per #1270).

Not in this PR

Test 6 — greenfield on sandbox 1124 (documented install → seat alive past 2min with advancing heartbeats → second fleet start still refuses if the broker is down) — runs on the sandbox after daphne's Wall-6 baseline reproduction on the same VM, coordinated with fred. This PR is the code half; 1124 is the boundary half.

Implementation note (disclosed)

Mid-work, my own sabotage-restore erased my uncommitted fleet.ts (the git checkout -- I used as "restore" was the pre-change file — copy-then-unlink applied to my own working tree). The implementation was reconstructed from the session's edit history, and all gates + all three sabotage controls were re-run on the reconstructed file — the results above are from the rebuilt version, not the erased one.

## Wall 6 / #1292 — lease-broker activation, safe unit placement, and named launch refusal Card implemented under fred's wave-3 rulings (design proposal → symlink measurement → amended design → GO with widened placement scope), branch `fix/1292-lease-broker-activation`, base `next` @ 8199261. Activation lands in the **control plane**, not the launcher. ## The measured foundations (all on throwaway systemd user instances + fomo-lin read-only probes) 1. `systemctl enable <name>` does **not** rewrite an existing by-path wants-symlink — rc=0, silent, the host stays on the old copy indefinitely. 2. Node `copyFile` **follows** a by-path symlink at the destination and overwrites the **seed template** (verified with `cp` AND `fs.copyFile`). 3. fomo-lin (the only in-the-wild by-path host) survived #2 only because its one symlink was the one unit the install never copied — the residue/copy sets were disjoint **by accident**. Adding the broker to the copy set made them intersect on the first run, so unlink-before-copy is required for a host that exists today, not a hypothetical. ## The fix - **`placeUnitFile`** — every unit the install places (all four, including the broker, previously omitted entirely) goes through one helper: unlink destination symlink → copy → remove stale wants-symlink (via `readlink`, not `readFile`) → `daemon-reload`. Idempotent (double-install convergence asserted). SET-INDEPENDENCE warning documented for whoever adds a fifth unit. - **`enableFleetUnits`** enables the broker first, alongside the holder. - **`fleet start` / reconciler** start the broker before any holder/agent effect, then **re-check the socket** (never unit ActiveState) and refuse nonzero with a named code if it did not appear — re-probed on **every** invocation, so `RemainAfterExit=yes` can never make retry look like repair (the sticky half). - **Reconciler plan** carries `broker {unitInstalled, socketPresent}` as a first-class member — "broker died an hour ago" is now reportable drift, not invisible. - **`start-agent-session.sh`** preflights the socket **before any tmux effect**: absent → exit **75** (EX_TEMPFAIL), named `broker-absent` message with socket path + remedy, no doomed pane. Exit 75 validated against the agent@ unit's measured `Restart=` policy (oneshot, none — no restart loop). The preflight detects and refuses; it never starts the broker. - **`mosaic doctor`** names one convention-neutral remedy: `mosaic fleet install (it reconciles either enable convention)`. ## Tests — name the failure, point at the check **Failure written down:** *a greenfield host's seat dies ~4s at lease registration while `fleet start` reports rc0; on retry, `fleet start` reports success again while the seat is still dead.* - **Placement (finding 2, the seed-integrity check fred added):** by-path residue fixture → after install the **seed template's bytes AND mtime are unchanged** — the only check that can redden for copy-through. Plus the clean-host negative control, wants-residue clearing, and double-install idempotence. - **Resulting-symlink assertions (finding 1):** the residue cases assert on the resulting wants-symlink target and destination object type, never on the enable call's argument. - **First start:** broker absent → refused, zero holder/agent start calls emitted. - **Second start (the sticky half):** socket never present → second invocation refuses identically; no ActiveState trust. - **Launcher:** absent socket → exit 75 + no tmux session created (fake tmux asserts it); live socket passes; explicit env wins; `--stop` not fenced. - **Reconciler:** broker plan member with the enabled-but-dead shape (`unitInstalled: true, socketPresent: false`); broker-before-agent ordering in both command and apply paths. **Sabotage controls (all restored byte-identically, sha256-verified):** placement unlink removed → exactly the seed-integrity test reddens (1/8); socket re-check disabled → exactly the two preflight specs redden; shell preflight removed → the bash suite reddens rc=1. All green after restore. **Gates (rc-honest):** vitest **1563/1563** · lint rc=0 · root build **25/25** · root typecheck **45/45** · bash preflight suite green. The new bash suite is CI-fit by construction (self-controlled preconditions — fake tmux, real unix socket, hermetic env; comment on the file explains why the older `test-start-agent-session.sh` is not, and stays unwired per #1270). ## Not in this PR **Test 6** — greenfield on sandbox 1124 (documented install → seat alive past 2min with advancing heartbeats → second `fleet start` still refuses if the broker is down) — runs on the sandbox after daphne's Wall-6 baseline reproduction on the same VM, coordinated with fred. This PR is the code half; 1124 is the boundary half. ## Implementation note (disclosed) Mid-work, my own sabotage-restore erased my uncommitted `fleet.ts` (the `git checkout --` I used as "restore" was the pre-change file — copy-then-unlink applied to my own working tree). The implementation was reconstructed from the session's edit history, and all gates + all three sabotage controls were **re-run on the reconstructed file** — the results above are from the rebuilt version, not the erased one.

fargo added 1 commit 2026-08-17 23:39:13 +00:00

fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292 )

ci/woodpecker/pr/ci Pipeline failed

Details

9d3e22b1c1

Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.

Some required checks failed

ci/woodpecker/pr/ci Pipeline failed

Required

Details

Some required checks were not successful.

This pull request doesn't have enough required approvals yet. 0 of 1 official approvals granted.

You are not authorized to merge this pull request.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/1292-lease-broker-activation:fix/1292-lease-broker-activation

git checkout fix/1292-lease-broker-activation

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#1297