#1297 - fix(fleet): lease-broker activation, symlink-safe unit placement, named launch refusal

Conversation 0 Commits 1 Files Changed 9

+936 -17

1 Commits

Author	SHA1	Message	Date
fargo	9d3e22b1c1	fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292 ) ci/woodpecker/pr/ci Pipeline failed Details Wall 6: no documented path ever enabled or started the shipped mosaic-lease-broker.service — every gated runtime died ~4s in at lease registration while fleet start reported rc0, and a broker not in the reconciler plan could not be reported as drifted. Activation lands in the control plane, not the launcher: - fleet install places ALL FOUR units through placeUnitFile — a placement helper that unlinks any by-path-enable symlink at the destination BEFORE copying (Node copyFile follows the link and overwrites the SEED template; measured on a throwaway systemd user instance 2026-08-17, with both cp and fs.copyFile), removes a stale wants-symlink pointing outside the active dir (readlink — readFile returns the target's content, not the link path), then copies and daemon-reloads. The same measurement showed systemctl enable <name> does NOT rewrite an existing by-path wants-symlink — reconciliation must be explicit. Idempotent: second install on by-path residue converges to the identical state. Until now the copy block named three units and omitted the broker, and the residue set / copy set were disjoint only by accident (fomo-lin survived copy-through because its one symlink was the one unit not copied); adding the broker made them intersect on first run. See the SET-INDEPENDENCE note on the helper before adding a fifth unit. - enableFleetUnits enables the broker first, alongside the holder. - fleet start / reconciler start the broker BEFORE any holder/agent lifecycle effect, then RE-CHECK the socket (not unit state) and exit nonzero with a named code if it did not appear. Re-probed on every invocation — a RemainAfterExit=yes dead-looking-active unit can never make retry look like repair (the sticky-retry check). - The reconciler plan carries broker {unitInstalled, socketPresent} as a first-class member; the socket is the signal (enabled-but-dead units report socketPresent=false). - start-agent-session.sh preflights the broker socket BEFORE any tmux effect (moved ahead of the ownership probe): absent -> exit 75 (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed pane. The agent@ unit is Type=oneshot with no Restart=, so the message survives instead of looping. The preflight detects and refuses; it never starts the broker. - mosaic doctor's lease check names one convention-neutral remedy: 'mosaic fleet install (it reconciles either enable convention)' — written from the measurement; teaching a manual systemctl line could leave a host with competing wants-symlinks. Tests: fleet-place-unit.spec.ts (8: clean-host negative control, by-path residue -> seed bytes AND mtime unchanged [the finding-2 check], wants-residue cleared, idempotence single + double-install convergence); fleet.spec.ts broker-first enable ordering, refused start emits no holder/agent calls, second-start re-probe; reconciler broker plan member (enabled-but-dead shape) + broker-before-agent ordering in both command and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes, hermetic env; absent -> exit 75 + no tmux session, live socket passes, explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root build 25/25, root typecheck 45/45. Sabotage controls: placement unlink removed -> exactly the seed-integrity test reddens (1/8); socket re-check disabled -> exactly the two preflight specs redden; shell preflight removed -> the bash suite reddens (6 FAIL assertions, rc=1). All restored byte-identically (sha256-verified), all green again. Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on sandbox after daphne's baseline, coordinated with fred. Note: the preflight uses exit 75 measured against the unit's Restart= policy (oneshot, none) — no restart loop.	2026-08-17 18:38:22 -05:00

Author

SHA1

Message

Date

fargo

9d3e22b1c1

fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292 )

ci/woodpecker/pr/ci Pipeline failed

Details

Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.

2026-08-17 18:38:22 -05:00

fix(fleet): lease-broker activation, symlink-safe unit placement, named launch refusal — Wall 6 (#1292) #1297