fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292)

Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.

packages/mosaic/framework/tools/fleet/start-agent-session.sh

@@ -233,8 +233,36 @@ assert_owned_tmux_server() {
     fail "tmux server ownership or environment validation failed"
 }
 
-# Validate exact server ownership before querying, cleaning, or creating any
-# managed session. An unmanaged or contaminated named socket is never repaired.
+# Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
+# launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
+# diagnostic invisible because tmux destroys the dead pane. This check runs
+# BEFORE any tmux effect — including the ownership probe below — so a host
+# without a broker produces a named, surviving refusal instead of a doomed
+# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
+# not ready for other reasons); the agent@ unit is Type=oneshot with no
+# Restart=, so the failed unit keeps its message instead of looping. Socket
+# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
+# This preflight DETECTS and REFUSES — it never starts the broker (activation
+# belongs to the fleet control plane; a component that both detects and fixes
+# cannot be used to measure whether the fix worked).
+broker_socket_path() {
+  if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
+    printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
+    return 0
+  fi
+  local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
+  printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
+}
+
+if [ "$MODE" = "launch" ]; then
+  _broker_socket=$(broker_socket_path)
+  if [ ! -S "$_broker_socket" ]; then
+    echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
+    echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service  (or reinstall via: mosaic fleet install)" >&2
+    exit 75
+  fi
+fi
+
 assert_owned_tmux_server
 
 if [ "$MODE" = interaction ]; then

packages/mosaic/framework/tools/fleet/test-agent-session-broker-preflight.sh

@@ -0,0 +1,216 @@
+#!/usr/bin/env bash
+# CI-fit regression suite for the #1292 lease-broker socket preflight in
+# start-agent-session.sh.
+#
+# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
+# context): that older suite's precondition is "the host does not have the pi
+# binary", which a CI image that ships pi violates — its guard correctly
+# refuses to report a pass there, so it is excluded from the chain. THIS suite
+# controls its own preconditions instead of inheriting them from the host: a
+# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
+# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
+# never depends on what the host has installed, so a green here means the same
+# thing on every host. Anyone adding cases: keep that property — no case may
+# depend on host state.
+#
+# The failure this suite is written down to catch (#1292): a seat launched on a
+# host with no lease broker dies ~4 seconds in at registration, with the
+# diagnostic invisible because tmux destroys the dead pane. The preflight runs
+# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
+# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
+# so a failed unit keeps its output instead of looping.
+#
+# Cases:
+#   1. absent socket -> exit 75, message names broker-absent + socket path +
+#      remedy, and NO tmux session was ever created (the doomed-pane half).
+#   2. present socket (real unix socket in tmpdir) -> proceeds PAST the
+#      preflight (the suite then stops at the next precondition, proving the
+#      preflight was not the refusal).
+#   3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
+#   4. --stop mode does NOT require the broker (teardown must not be fenced on
+#      a component whose absence is exactly what teardown may follow).
+#
+# Sabotage control, run by the developer (not in-suite): remove the preflight
+# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
+# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
+FAKE_HOME="$WORK_DIR/home"
+BIN_DIR="$WORK_DIR/bin"
+ENV_DIR="$WORK_DIR/env"
+SOCK_DIR="$WORK_DIR/sockets"
+LOG_FILE="$WORK_DIR/tmux-calls.log"
+
+rm -rf "$WORK_DIR"
+# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
+# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
+# symlinks — plus a per-agent env projection. Build the full tree the launcher
+# expects so the suite reaches the BROKER preflight rather than dying at
+# environment validation.
+mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
+chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
+chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
+cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
+MOSAIC_AGENT_NAME=preflight-test
+MOSAIC_AGENT_CLASS=worker
+MOSAIC_AGENT_RUNTIME=pi
+MOSAIC_AGENT_MODEL=
+MOSAIC_AGENT_REASONING=
+MOSAIC_AGENT_TOOL_POLICY=code
+MOSAIC_AGENT_WORKDIR=/tmp
+MOSAIC_TMUX_SOCKET=mosaic-fleet
+ENVEOF
+chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
+
+# ─── Fake tmux: records every invocation; new-session marks the marker. ────
+: > "$LOG_FILE"
+cat > "$BIN_DIR/tmux" <<SH
+#!/usr/bin/env bash
+printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
+if [[ "\$*" == *new-session* ]]; then
+  echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
+fi
+exit 0
+SH
+chmod +x "$BIN_DIR/tmux"
+
+# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
+for bin in mosaic pi claude; do
+  printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
+  chmod +x "$BIN_DIR/$bin"
+done
+
+# ─── Minimal launch environment the script expects. ────────────────────────
+# (Enough for the preflight to be reached; later stages will still fail in
+# case 2 — that is expected and asserted.)
+run_session_script() {
+  local mode="$1"; shift
+  (
+    cd "$WORK_DIR"
+    env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
+      GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
+      MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
+      AGENT_NAME=preflight-test \
+      "$@" \
+      bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
+  )
+}
+
+fail=0
+assert() {
+  local desc="$1" expected="$2" actual="$3"
+  if [[ "$expected" != "$actual" ]]; then
+    echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
+    fail=1
+  fi
+}
+assert_contains() {
+  local desc="$1" haystack="$2" needle="$3"
+  [[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
+}
+assert_not_contains() {
+  local desc="$1" haystack="$2" needle="$3"
+  if [[ "$haystack" == *"$needle"* ]]; then
+    echo "FAIL: $desc — must not contain '$needle'" >&2
+    fail=1
+  fi
+  return 0
+}
+
+# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
+: > "$LOG_FILE"
+stderr_file="$WORK_DIR/stderr-1.tmp"
+set +e
+out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
+rc=$?
+set -e
+assert "absent socket exit code" "75" "$rc"
+err=$(cat "$stderr_file")
+assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
+assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
+assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
+log1=$(cat "$LOG_FILE")
+assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
+
+# ─── 2. Present socket → passes the preflight. ─────────────────────────────
+# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
+# minimal by design); the assertion is only that the refusal is NOT
+# broker-absent and the exit is NOT 75.
+# Create a REAL unix socket: a detached python holder binds it and stays alive
+# for the duration (bash cannot create sockets; a foreground python would
+# close the socket on exit and -S on a closed-but-unlinked path fails). Written
+# as a script file + setsid nohup so no job-control/heredoc interaction with
+# set -e can silently kill the suite.
+# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
+# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
+# with the suite). The preflight takes its socket path explicitly, so this
+# stays fully controlled.
+LIVE_SOCK=$(mktemp -u /tmp/mosaic-preflight-XXXXXX.sock)
+trap 'rm -f "$LIVE_SOCK"' EXIT
+rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
+cat > "$SOCK_DIR/holder.py" <<'PY'
+import socket, sys, time
+path = sys.argv[1]
+s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+s.bind(path)
+s.listen(1)
+time.sleep(120)
+PY
+python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
+HOLDER_PID=$!
+# Wait for the socket object to exist (bind is near-instant, but do not race it).
+for _ in $(seq 1 50); do
+  [ -S "$LIVE_SOCK" ] && break
+  sleep 0.1
+done
+if [ ! -S "$LIVE_SOCK" ]; then
+  echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
+  ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
+  cat "$SOCK_DIR/holder.err" >&2 || true
+  exit 1
+fi
+: > "$LOG_FILE"
+set +e
+out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
+rc=$?
+set -e
+# The preflight PASSED if the failure (whatever later stage refused) is NOT
+# the broker refusal, and tmux was reached or a later precondition named
+# something else.
+err2=$(cat "$WORK_DIR/stderr-2.tmp")
+assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
+if [[ "$rc" == "75" ]]; then
+  echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
+  fail=1
+fi
+
+# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
+set +e
+out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
+rc=$?
+set -e
+assert "explicit env wins (exit 75)" "75" "$rc"
+assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
+
+# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
+: > "$LOG_FILE"
+set +e
+out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
+rc=$?
+set -e
+err4=$(cat "$WORK_DIR/stderr-4.tmp")
+assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
+if [[ "$rc" == "75" ]]; then
+  echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
+  fail=1
+fi
+
+kill "$HOLDER_PID" 2>/dev/null || true
+
+if [[ "$fail" -eq 0 ]]; then
+  echo "start-agent-session lease-broker preflight regression passed"
+fi
+exit "$fail"