fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292)
ci/woodpecker/pr/ci Pipeline failed
ci/woodpecker/pr/ci Pipeline failed
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.
Activation lands in the control plane, not the launcher:
- fleet install places ALL FOUR units through placeUnitFile — a placement
helper that unlinks any by-path-enable symlink at the destination
BEFORE copying (Node copyFile follows the link and overwrites the SEED
template; measured on a throwaway systemd user instance 2026-08-17,
with both cp and fs.copyFile), removes a stale wants-symlink pointing
outside the active dir (readlink — readFile returns the target's
content, not the link path), then copies and daemon-reloads. The same
measurement showed systemctl enable <name> does NOT rewrite an existing
by-path wants-symlink — reconciliation must be explicit. Idempotent:
second install on by-path residue converges to the identical state.
Until now the copy block named three units and omitted the broker, and
the residue set / copy set were disjoint only by accident (fomo-lin
survived copy-through because its one symlink was the one unit not
copied); adding the broker made them intersect on first run. See the
SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
lifecycle effect, then RE-CHECK the socket (not unit state) and exit
nonzero with a named code if it did not appear. Re-probed on every
invocation — a RemainAfterExit=yes dead-looking-active unit can never
make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
first-class member; the socket is the signal (enabled-but-dead units
report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
effect (moved ahead of the ownership probe): absent -> exit 75
(EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
pane. The agent@ unit is Type=oneshot with no Restart=, so the message
survives instead of looping. The preflight detects and refuses; it
never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
'mosaic fleet install (it reconciles either enable convention)' —
written from the measurement; teaching a manual systemctl line could
leave a host with competing wants-symlinks.
Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.
Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.
Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.
Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
This commit is contained in:
@@ -233,8 +233,36 @@ assert_owned_tmux_server() {
|
||||
fail "tmux server ownership or environment validation failed"
|
||||
}
|
||||
|
||||
# Validate exact server ownership before querying, cleaning, or creating any
|
||||
# managed session. An unmanaged or contaminated named socket is never repaired.
|
||||
# Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
|
||||
# launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
|
||||
# diagnostic invisible because tmux destroys the dead pane. This check runs
|
||||
# BEFORE any tmux effect — including the ownership probe below — so a host
|
||||
# without a broker produces a named, surviving refusal instead of a doomed
|
||||
# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
|
||||
# not ready for other reasons); the agent@ unit is Type=oneshot with no
|
||||
# Restart=, so the failed unit keeps its message instead of looping. Socket
|
||||
# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
|
||||
# This preflight DETECTS and REFUSES — it never starts the broker (activation
|
||||
# belongs to the fleet control plane; a component that both detects and fixes
|
||||
# cannot be used to measure whether the fix worked).
|
||||
broker_socket_path() {
|
||||
if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
|
||||
printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
|
||||
return 0
|
||||
fi
|
||||
local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
|
||||
printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
|
||||
}
|
||||
|
||||
if [ "$MODE" = "launch" ]; then
|
||||
_broker_socket=$(broker_socket_path)
|
||||
if [ ! -S "$_broker_socket" ]; then
|
||||
echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
|
||||
echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service (or reinstall via: mosaic fleet install)" >&2
|
||||
exit 75
|
||||
fi
|
||||
fi
|
||||
|
||||
assert_owned_tmux_server
|
||||
|
||||
if [ "$MODE" = interaction ]; then
|
||||
|
||||
Reference in New Issue
Block a user