Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1ae7a39183 | ||
|
|
2c4adeedf0 | ||
|
|
3c4da26757 | ||
|
|
8e8879ca34 | ||
|
|
019230b72f | ||
|
|
4f22a58041 |
@@ -65,6 +65,19 @@ Each of these produced a wrong conclusion before it was written down.
|
|||||||
conclusion drawn from it describes the wrong tree. Confirm `git rev-parse --show-toplevel`
|
conclusion drawn from it describes the wrong tree. Confirm `git rev-parse --show-toplevel`
|
||||||
is the tree you think it is before trusting any git output.
|
is the tree you think it is before trusting any git output.
|
||||||
|
|
||||||
|
13. **Run the repository's PINNED tool version.** `npx <tool>` resolves a local `node_modules`
|
||||||
|
install when one is present and fetches the latest release when one is not, so the same
|
||||||
|
command answers differently depending on where it ran. A reviewer measuring in a fresh clone
|
||||||
|
or a detached worktree — which is exactly where reviewers measure — has no `node_modules` and
|
||||||
|
silently gets the latest release instead of the pinned one. Measured on mosaicstack#1313: the
|
||||||
|
lockfile pins prettier 3.8.1, under which three guides pass; a version-less `npx` in a
|
||||||
|
worktree resolved 3.9.6, under which the same three fail; and 3.0.0, the floor of the declared
|
||||||
|
`^3.0.0` range, fails a different one. Three versions, three verdicts, identical bytes. Use
|
||||||
|
`node_modules/.bin/<tool>`, or name the version the lockfile pins.
|
||||||
|
14. **A formatter or linter declared as a range is a dated verdict, not a fact.** If a lockfile
|
||||||
|
pins it, the gate is reproducible today and will disagree with itself the day the pin moves.
|
||||||
|
Report a formatting failure with the version that produced it, always.
|
||||||
|
|
||||||
### Feedback Categories
|
### Feedback Categories
|
||||||
|
|
||||||
- **Blocker**: must fix before merge (security, bugs, test failures)
|
- **Blocker**: must fix before merge (security, bugs, test failures)
|
||||||
|
|||||||
@@ -257,8 +257,36 @@ assert_owned_tmux_server() {
|
|||||||
fail "tmux server ownership or environment validation failed"
|
fail "tmux server ownership or environment validation failed"
|
||||||
}
|
}
|
||||||
|
|
||||||
# Validate exact server ownership before querying, cleaning, or creating any
|
# Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
|
||||||
# managed session. An unmanaged or contaminated named socket is never repaired.
|
# launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
|
||||||
|
# diagnostic invisible because tmux destroys the dead pane. This check runs
|
||||||
|
# BEFORE any tmux effect — including the ownership probe below — so a host
|
||||||
|
# without a broker produces a named, surviving refusal instead of a doomed
|
||||||
|
# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
|
||||||
|
# not ready for other reasons); the agent@ unit is Type=oneshot with no
|
||||||
|
# Restart=, so the failed unit keeps its message instead of looping. Socket
|
||||||
|
# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
|
||||||
|
# This preflight DETECTS and REFUSES — it never starts the broker (activation
|
||||||
|
# belongs to the fleet control plane; a component that both detects and fixes
|
||||||
|
# cannot be used to measure whether the fix worked).
|
||||||
|
broker_socket_path() {
|
||||||
|
if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
|
||||||
|
printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
|
||||||
|
printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "$MODE" = "launch" ]; then
|
||||||
|
_broker_socket=$(broker_socket_path)
|
||||||
|
if [ ! -S "$_broker_socket" ]; then
|
||||||
|
echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
|
||||||
|
echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service (or reinstall via: mosaic fleet install)" >&2
|
||||||
|
exit 75
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
assert_owned_tmux_server
|
assert_owned_tmux_server
|
||||||
|
|
||||||
if [ "$MODE" = interaction ]; then
|
if [ "$MODE" = interaction ]; then
|
||||||
|
|||||||
+222
@@ -0,0 +1,222 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# CI-fit regression suite for the #1292 lease-broker socket preflight in
|
||||||
|
# start-agent-session.sh.
|
||||||
|
#
|
||||||
|
# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
|
||||||
|
# context): that older suite's precondition is "the host does not have the pi
|
||||||
|
# binary", which a CI image that ships pi violates — its guard correctly
|
||||||
|
# refuses to report a pass there, so it is excluded from the chain. THIS suite
|
||||||
|
# controls its own preconditions instead of inheriting them from the host: a
|
||||||
|
# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
|
||||||
|
# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
|
||||||
|
# never depends on what the host has installed, so a green here means the same
|
||||||
|
# thing on every host. Anyone adding cases: keep that property — no case may
|
||||||
|
# depend on host state.
|
||||||
|
#
|
||||||
|
# The failure this suite is written down to catch (#1292): a seat launched on a
|
||||||
|
# host with no lease broker dies ~4 seconds in at registration, with the
|
||||||
|
# diagnostic invisible because tmux destroys the dead pane. The preflight runs
|
||||||
|
# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
|
||||||
|
# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
|
||||||
|
# so a failed unit keeps its output instead of looping.
|
||||||
|
#
|
||||||
|
# Cases:
|
||||||
|
# 1. absent socket -> exit 75, message names broker-absent + socket path +
|
||||||
|
# remedy, and NO tmux session was ever created (the doomed-pane half).
|
||||||
|
# 2. present socket (real unix socket in tmpdir) -> proceeds PAST the
|
||||||
|
# preflight (the suite then stops at the next precondition, proving the
|
||||||
|
# preflight was not the refusal).
|
||||||
|
# 3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
|
||||||
|
# 4. --stop mode does NOT require the broker (teardown must not be fenced on
|
||||||
|
# a component whose absence is exactly what teardown may follow).
|
||||||
|
#
|
||||||
|
# Sabotage control, run by the developer (not in-suite): remove the preflight
|
||||||
|
# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
|
||||||
|
# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
|
||||||
|
FAKE_HOME="$WORK_DIR/home"
|
||||||
|
BIN_DIR="$WORK_DIR/bin"
|
||||||
|
ENV_DIR="$WORK_DIR/env"
|
||||||
|
SOCK_DIR="$WORK_DIR/sockets"
|
||||||
|
LOG_FILE="$WORK_DIR/tmux-calls.log"
|
||||||
|
|
||||||
|
rm -rf "$WORK_DIR"
|
||||||
|
# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
|
||||||
|
# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
|
||||||
|
# symlinks — plus a per-agent env projection. Build the full tree the launcher
|
||||||
|
# expects so the suite reaches the BROKER preflight rather than dying at
|
||||||
|
# environment validation.
|
||||||
|
mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
|
||||||
|
chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
|
||||||
|
chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
|
||||||
|
cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
|
||||||
|
MOSAIC_AGENT_NAME=preflight-test
|
||||||
|
MOSAIC_GIT_IDENTITY=preflight-test
|
||||||
|
MOSAIC_AGENT_CLASS=worker
|
||||||
|
MOSAIC_AGENT_RUNTIME=pi
|
||||||
|
MOSAIC_AGENT_MODEL=
|
||||||
|
MOSAIC_AGENT_REASONING=
|
||||||
|
MOSAIC_AGENT_TOOL_POLICY=code
|
||||||
|
MOSAIC_AGENT_WORKDIR=/tmp
|
||||||
|
MOSAIC_TMUX_SOCKET=mosaic-fleet
|
||||||
|
ENVEOF
|
||||||
|
chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
|
||||||
|
|
||||||
|
# ─── Fake tmux: records every invocation; new-session marks the marker. ────
|
||||||
|
: > "$LOG_FILE"
|
||||||
|
cat > "$BIN_DIR/tmux" <<SH
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
|
||||||
|
if [[ "\$*" == *new-session* ]]; then
|
||||||
|
echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
SH
|
||||||
|
chmod +x "$BIN_DIR/tmux"
|
||||||
|
|
||||||
|
# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
|
||||||
|
for bin in mosaic pi claude; do
|
||||||
|
printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
|
||||||
|
chmod +x "$BIN_DIR/$bin"
|
||||||
|
done
|
||||||
|
|
||||||
|
# ─── Minimal launch environment the script expects. ────────────────────────
|
||||||
|
# (Enough for the preflight to be reached; later stages will still fail in
|
||||||
|
# case 2 — that is expected and asserted.)
|
||||||
|
run_session_script() {
|
||||||
|
local mode="$1"; shift
|
||||||
|
(
|
||||||
|
cd "$WORK_DIR"
|
||||||
|
env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
|
||||||
|
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
|
||||||
|
MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
|
||||||
|
AGENT_NAME=preflight-test \
|
||||||
|
"$@" \
|
||||||
|
bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fail=0
|
||||||
|
assert() {
|
||||||
|
local desc="$1" expected="$2" actual="$3"
|
||||||
|
if [[ "$expected" != "$actual" ]]; then
|
||||||
|
echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
assert_contains() {
|
||||||
|
local desc="$1" haystack="$2" needle="$3"
|
||||||
|
[[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
|
||||||
|
}
|
||||||
|
assert_not_contains() {
|
||||||
|
local desc="$1" haystack="$2" needle="$3"
|
||||||
|
if [[ "$haystack" == *"$needle"* ]]; then
|
||||||
|
echo "FAIL: $desc — must not contain '$needle'" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
|
||||||
|
: > "$LOG_FILE"
|
||||||
|
stderr_file="$WORK_DIR/stderr-1.tmp"
|
||||||
|
set +e
|
||||||
|
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
assert "absent socket exit code" "75" "$rc"
|
||||||
|
err=$(cat "$stderr_file")
|
||||||
|
assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
|
||||||
|
assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
|
||||||
|
assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
|
||||||
|
log1=$(cat "$LOG_FILE")
|
||||||
|
assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
|
||||||
|
|
||||||
|
# ─── 2. Present socket → passes the preflight. ─────────────────────────────
|
||||||
|
# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
|
||||||
|
# minimal by design); the assertion is only that the refusal is NOT
|
||||||
|
# broker-absent and the exit is NOT 75.
|
||||||
|
# Create a REAL unix socket: a detached python holder binds it and stays alive
|
||||||
|
# for the duration (bash cannot create sockets; a foreground python would
|
||||||
|
# close the socket on exit and -S on a closed-but-unlinked path fails). Written
|
||||||
|
# as a script file + setsid nohup so no job-control/heredoc interaction with
|
||||||
|
# set -e can silently kill the suite.
|
||||||
|
# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
|
||||||
|
# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
|
||||||
|
# with the suite). The preflight takes its socket path explicitly, so this
|
||||||
|
# stays fully controlled.
|
||||||
|
# A real unix socket at a SHORT absolute path (AF_UNIX limit is 108 bytes,
|
||||||
|
# so the repo-deep SOCK_DIR cannot host it). The name is composed, not
|
||||||
|
# `mktemp -u`: the CI image's mktemp dialect rejects that invocation
|
||||||
|
# (pipeline 2562: "mktemp: : Invalid argument"), and no pre-existing file is
|
||||||
|
# wanted anyway — the holder binds it fresh.
|
||||||
|
LIVE_SOCK="/tmp/mosaic-preflight-$RANDOM-$$.sock"
|
||||||
|
trap 'rm -f "$LIVE_SOCK"' EXIT
|
||||||
|
rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
|
||||||
|
cat > "$SOCK_DIR/holder.py" <<'PY'
|
||||||
|
import socket, sys, time
|
||||||
|
path = sys.argv[1]
|
||||||
|
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||||
|
s.bind(path)
|
||||||
|
s.listen(1)
|
||||||
|
time.sleep(120)
|
||||||
|
PY
|
||||||
|
python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
|
||||||
|
HOLDER_PID=$!
|
||||||
|
# Wait for the socket object to exist (bind is near-instant, but do not race it).
|
||||||
|
for _ in $(seq 1 50); do
|
||||||
|
[ -S "$LIVE_SOCK" ] && break
|
||||||
|
sleep 0.1
|
||||||
|
done
|
||||||
|
if [ ! -S "$LIVE_SOCK" ]; then
|
||||||
|
echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
|
||||||
|
ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
|
||||||
|
cat "$SOCK_DIR/holder.err" >&2 || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
: > "$LOG_FILE"
|
||||||
|
set +e
|
||||||
|
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
# The preflight PASSED if the failure (whatever later stage refused) is NOT
|
||||||
|
# the broker refusal, and tmux was reached or a later precondition named
|
||||||
|
# something else.
|
||||||
|
err2=$(cat "$WORK_DIR/stderr-2.tmp")
|
||||||
|
assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
|
||||||
|
if [[ "$rc" == "75" ]]; then
|
||||||
|
echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
|
||||||
|
set +e
|
||||||
|
out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
assert "explicit env wins (exit 75)" "75" "$rc"
|
||||||
|
assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
|
||||||
|
|
||||||
|
# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
|
||||||
|
: > "$LOG_FILE"
|
||||||
|
set +e
|
||||||
|
out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
err4=$(cat "$WORK_DIR/stderr-4.tmp")
|
||||||
|
assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
|
||||||
|
if [[ "$rc" == "75" ]]; then
|
||||||
|
echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
|
||||||
|
fail=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
kill "$HOLDER_PID" 2>/dev/null || true
|
||||||
|
|
||||||
|
if [[ "$fail" -eq 0 ]]; then
|
||||||
|
echo "start-agent-session lease-broker preflight regression passed"
|
||||||
|
fi
|
||||||
|
exit "$fail"
|
||||||
@@ -25,7 +25,7 @@
|
|||||||
"lint": "eslint src",
|
"lint": "eslint src",
|
||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
|
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
|
||||||
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/quality/scripts/test-framework-drift-doctor.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-no-status.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh && bash framework/tools/_scripts/test-brain-home-check.sh"
|
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/quality/scripts/test-framework-drift-doctor.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-no-status.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh && bash framework/tools/_scripts/test-brain-home-check.sh && bash framework/tools/fleet/test-agent-session-broker-preflight.sh"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@mosaicstack/brain": "workspace:*",
|
"@mosaicstack/brain": "workspace:*",
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
import { lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { afterEach, describe, expect, it } from 'vitest';
|
||||||
|
|
||||||
|
import { placeUnitFile, resolveLeaseBrokerSocketForPreflight } from './fleet.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unit-placement regression harness for #1292.
|
||||||
|
*
|
||||||
|
* The two measured defects this suite pins:
|
||||||
|
* 1. `systemctl enable <name>` does NOT rewrite an existing by-path
|
||||||
|
* wants-symlink — so placement must remove stale residue explicitly, and
|
||||||
|
* acceptance asserts on the RESULTING SYMLINK TARGET, never on the enable
|
||||||
|
* call's argument (asserting the call cannot see where the link ended up).
|
||||||
|
* 2. Node's copyFile FOLLOWS a by-path symlink at the destination and
|
||||||
|
* overwrites the SEED template. Acceptance asserts on the SEED's bytes
|
||||||
|
* AND mtime — unchanged — which is the only check that can redden for
|
||||||
|
* finding 2. The symlink-target assertion catches finding 1; these are
|
||||||
|
* different defects with different failure modes.
|
||||||
|
*
|
||||||
|
* Fixtures are entirely inside tmpdirs (source template, active systemd dir,
|
||||||
|
* wants dir) — no real host paths are touched by this suite.
|
||||||
|
*/
|
||||||
|
|
||||||
|
describe('placeUnitFile (#1292 unit placement)', () => {
|
||||||
|
const cleanup: string[] = [];
|
||||||
|
afterEach(async () => {
|
||||||
|
while (cleanup.length > 0) {
|
||||||
|
await rm(cleanup.pop()!, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
async function fixture() {
|
||||||
|
const root = await mkdtemp(join(tmpdir(), 'place-unit-'));
|
||||||
|
cleanup.push(root);
|
||||||
|
const seedDir = join(root, 'seed');
|
||||||
|
const activeDir = join(root, 'active');
|
||||||
|
await mkdir(seedDir, { recursive: true });
|
||||||
|
await mkdir(activeDir, { recursive: true });
|
||||||
|
const seedTemplate = join(seedDir, 'unit-under-test.service');
|
||||||
|
await writeFile(
|
||||||
|
seedTemplate,
|
||||||
|
'[Unit]\nDescription=seed template\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
|
||||||
|
);
|
||||||
|
const activeSource = join(root, 'active-source.service');
|
||||||
|
await writeFile(
|
||||||
|
activeSource,
|
||||||
|
'[Unit]\nDescription=active copy v2\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
|
||||||
|
);
|
||||||
|
return { root, seedDir, activeDir, seedTemplate, activeSource };
|
||||||
|
}
|
||||||
|
|
||||||
|
it('places a regular file on a clean host (negative control: no residue anywhere)', async () => {
|
||||||
|
const f = await fixture();
|
||||||
|
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
expect(result.unlinkedDestinationSymlink).toBe(false);
|
||||||
|
expect(result.removedStaleWantsSymlink).toBe(false);
|
||||||
|
const info = await lstat(join(f.activeDir, 'unit-under-test.service'));
|
||||||
|
expect(info.isSymbolicLink()).toBe(false);
|
||||||
|
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
|
||||||
|
'active copy v2',
|
||||||
|
);
|
||||||
|
// Seed untouched by construction — but assert it, so the clean-host case
|
||||||
|
// cannot silently regress into seed-mutation.
|
||||||
|
expect(await readFile(f.seedTemplate, 'utf8')).toContain('seed template');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('by-path residue: unlinks destination symlink, places the file, seed bytes AND mtime unchanged (finding 2)', async () => {
|
||||||
|
const f = await fixture();
|
||||||
|
const seedBefore = await readFile(f.seedTemplate, 'utf8');
|
||||||
|
const mtimeBefore = (await lstat(f.seedTemplate)).mtimeMs;
|
||||||
|
// The fomo-lin convention: by-path enable left a symlink AT the unit name
|
||||||
|
// pointing at the seed template, plus a wants-symlink doing the same.
|
||||||
|
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||||
|
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||||
|
await mkdir(wantsDir, { recursive: true });
|
||||||
|
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||||
|
|
||||||
|
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
expect(result.unlinkedDestinationSymlink).toBe(true);
|
||||||
|
expect(result.removedStaleWantsSymlink).toBe(true);
|
||||||
|
|
||||||
|
// FINDING 2's check: the seed is byte-identical and its mtime did not move.
|
||||||
|
expect(await readFile(f.seedTemplate, 'utf8')).toBe(seedBefore);
|
||||||
|
expect((await lstat(f.seedTemplate)).mtimeMs).toBe(mtimeBefore);
|
||||||
|
|
||||||
|
// The destination is now a regular file carrying the ACTIVE content.
|
||||||
|
const destInfo = await lstat(join(f.activeDir, 'unit-under-test.service'));
|
||||||
|
expect(destInfo.isSymbolicLink()).toBe(false);
|
||||||
|
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
|
||||||
|
'active copy v2',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('by-path residue: no wants-symlink remains pointing at the seed (finding 1 residue cleared)', async () => {
|
||||||
|
const f = await fixture();
|
||||||
|
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||||
|
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||||
|
await mkdir(wantsDir, { recursive: true });
|
||||||
|
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||||
|
|
||||||
|
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
|
||||||
|
// After placement the stale wants link is GONE (enable-by-name recreates
|
||||||
|
// it correctly). A link still present must not point at the seed.
|
||||||
|
try {
|
||||||
|
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
|
||||||
|
if (link.isSymbolicLink()) {
|
||||||
|
const target = await readFile(join(wantsDir, 'unit-under-test.service'), 'utf8').catch(
|
||||||
|
async () => '',
|
||||||
|
);
|
||||||
|
expect(target).not.toContain('seed template');
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// absent wants link — the expected post-placement state
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('idempotence: second placement on a reconciled host is a no-op producing the identical final state', async () => {
|
||||||
|
const f = await fixture();
|
||||||
|
// Reconciled starting state: regular file at the name, wants link to the active copy.
|
||||||
|
await writeFile(
|
||||||
|
join(f.activeDir, 'unit-under-test.service'),
|
||||||
|
await readFile(f.activeSource, 'utf8'),
|
||||||
|
);
|
||||||
|
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||||
|
await mkdir(wantsDir, { recursive: true });
|
||||||
|
await symlink(
|
||||||
|
join(f.activeDir, 'unit-under-test.service'),
|
||||||
|
join(wantsDir, 'unit-under-test.service'),
|
||||||
|
);
|
||||||
|
const before = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||||
|
|
||||||
|
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
// No destructive step fired: no unlink, no wants removal.
|
||||||
|
expect(result.unlinkedDestinationSymlink).toBe(false);
|
||||||
|
expect(result.removedStaleWantsSymlink).toBe(false);
|
||||||
|
// Identical final state.
|
||||||
|
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toBe(before);
|
||||||
|
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
|
||||||
|
expect(link.isSymbolicLink()).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('double install on by-path residue converges to the identical reconciled state', async () => {
|
||||||
|
const f = await fixture();
|
||||||
|
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||||
|
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||||
|
await mkdir(wantsDir, { recursive: true });
|
||||||
|
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||||
|
|
||||||
|
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
const first = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||||
|
const secondRun = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||||
|
const second = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||||
|
expect(secondRun.unlinkedDestinationSymlink).toBe(false);
|
||||||
|
expect(second).toBe(first);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('resolveLeaseBrokerSocketForPreflight (#1292 preflight resolution)', () => {
|
||||||
|
it('explicit MOSAIC_LEASE_BROKER_SOCKET wins', () => {
|
||||||
|
expect(
|
||||||
|
resolveLeaseBrokerSocketForPreflight({ MOSAIC_LEASE_BROKER_SOCKET: '/custom/sock' }, 1000),
|
||||||
|
).toBe('/custom/sock');
|
||||||
|
});
|
||||||
|
it('XDG_RUNTIME_DIR next', () => {
|
||||||
|
expect(resolveLeaseBrokerSocketForPreflight({ XDG_RUNTIME_DIR: '/run/user/1001' }, 1000)).toBe(
|
||||||
|
'/run/user/1001/mosaic-lease/broker.sock',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
it('falls back to /run/user/<uid>', () => {
|
||||||
|
expect(resolveLeaseBrokerSocketForPreflight({}, 1002)).toBe(
|
||||||
|
'/run/user/1002/mosaic-lease/broker.sock',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -73,6 +73,10 @@ function program(
|
|||||||
runner,
|
runner,
|
||||||
reconcileDeps: {
|
reconcileDeps: {
|
||||||
homeDirectory: '/home/mosaic',
|
homeDirectory: '/home/mosaic',
|
||||||
|
// Deterministic broker presence: without a seam the reconciler probes the
|
||||||
|
// REAL host socket (#1297 F3), making every CLI start test answer the
|
||||||
|
// host's broker state instead of its own property.
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
readHolderIdentity: async () => '11111111-1111-4111-8111-111111111111',
|
readHolderIdentity: async () => '11111111-1111-4111-8111-111111111111',
|
||||||
validateRoster: async () => undefined,
|
validateRoster: async () => undefined,
|
||||||
prepareProjections: async () => [{ agentName: 'coder0' }],
|
prepareProjections: async () => [{ agentName: 'coder0' }],
|
||||||
|
|||||||
@@ -1,11 +1,24 @@
|
|||||||
import { chmod, lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises';
|
import {
|
||||||
|
chmod,
|
||||||
|
lstat,
|
||||||
|
mkdir,
|
||||||
|
mkdtemp,
|
||||||
|
readFile,
|
||||||
|
readlink,
|
||||||
|
rm,
|
||||||
|
stat,
|
||||||
|
symlink,
|
||||||
|
writeFile,
|
||||||
|
} from 'node:fs/promises';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { dirname, join, resolve } from 'node:path';
|
import { dirname, join, resolve } from 'node:path';
|
||||||
|
import { createServer } from 'node:net';
|
||||||
import { Command } from 'commander';
|
import { Command } from 'commander';
|
||||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||||
import {
|
import {
|
||||||
acquireRestartLock,
|
acquireRestartLock,
|
||||||
addAgentToRoster,
|
addAgentToRoster,
|
||||||
|
brokerSocketPresent,
|
||||||
buildAgentSendCommand,
|
buildAgentSendCommand,
|
||||||
buildAgentWatchAttachCommand,
|
buildAgentWatchAttachCommand,
|
||||||
buildAgentWatchCommand,
|
buildAgentWatchCommand,
|
||||||
@@ -42,6 +55,7 @@ import {
|
|||||||
parseSystemdShow,
|
parseSystemdShow,
|
||||||
parseTmuxListPanes,
|
parseTmuxListPanes,
|
||||||
parseTmuxListSessions,
|
parseTmuxListSessions,
|
||||||
|
placeUnitFile,
|
||||||
registerFleetCommand,
|
registerFleetCommand,
|
||||||
removeAgentFromRoster,
|
removeAgentFromRoster,
|
||||||
resolveFleetPaths,
|
resolveFleetPaths,
|
||||||
@@ -50,6 +64,7 @@ import {
|
|||||||
RESTART_LOCK_STALE_MS,
|
RESTART_LOCK_STALE_MS,
|
||||||
RUNTIME_ACCEPTABLE_COMMANDS,
|
RUNTIME_ACCEPTABLE_COMMANDS,
|
||||||
serializeRosterToYaml,
|
serializeRosterToYaml,
|
||||||
|
UnitPlacementError,
|
||||||
VERIFY_DEFAULT_TIMEOUT_MS,
|
VERIFY_DEFAULT_TIMEOUT_MS,
|
||||||
VERIFY_POLL_INTERVAL_MS,
|
VERIFY_POLL_INTERVAL_MS,
|
||||||
type AgentPsRow,
|
type AgentPsRow,
|
||||||
@@ -836,13 +851,25 @@ describe('fleet command construction', () => {
|
|||||||
};
|
};
|
||||||
const program = new Command();
|
const program = new Command();
|
||||||
program.exitOverride();
|
program.exitOverride();
|
||||||
registerFleetCommand(program, { runner, mosaicHome: home });
|
// #1292: inject a present broker socket so the preflight passes and this
|
||||||
|
// spec keeps testing its ORIGINAL property (holder-before-agent ordering).
|
||||||
|
// The preflight's own refusal behavior has dedicated specs below.
|
||||||
|
registerFleetCommand(program, {
|
||||||
|
runner,
|
||||||
|
mosaicHome: home,
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
|
});
|
||||||
|
|
||||||
try {
|
try {
|
||||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
|
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
|
||||||
|
|
||||||
expect(calls).toEqual([
|
expect(calls).toEqual([
|
||||||
|
// #1292: fleet start enables + starts the broker FIRST (enable is
|
||||||
|
// idempotent; the unit exists after install), re-checking the socket
|
||||||
|
// before any holder/agent lifecycle effect.
|
||||||
|
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
|
||||||
|
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
|
||||||
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
|
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
|
||||||
['systemctl', '--user', 'start', '[email protected]'],
|
['systemctl', '--user', 'start', '[email protected]'],
|
||||||
['systemctl', '--user', 'stop', '[email protected]'],
|
['systemctl', '--user', 'stop', '[email protected]'],
|
||||||
@@ -853,6 +880,92 @@ describe('fleet command construction', () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('fleet start refuses with a named error when the broker socket does not appear (#1292)', async () => {
|
||||||
|
const home = await tempDir();
|
||||||
|
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||||
|
await mkdir(join(home, 'fleet'), { recursive: true });
|
||||||
|
await writeFile(
|
||||||
|
rosterPath,
|
||||||
|
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
|
||||||
|
'\n',
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const runner: CommandRunner = async (command, args) => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
};
|
||||||
|
const program = new Command();
|
||||||
|
program.exitOverride();
|
||||||
|
const errors: string[] = [];
|
||||||
|
const origError = console.error;
|
||||||
|
console.error = (...args: unknown[]) => {
|
||||||
|
errors.push(args.join(' '));
|
||||||
|
};
|
||||||
|
registerFleetCommand(program, {
|
||||||
|
runner,
|
||||||
|
mosaicHome: home,
|
||||||
|
checkBrokerSocket: async () => false,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||||
|
// Refused: no holder/agent starts were issued after the broker attempt.
|
||||||
|
expect(calls).toEqual([
|
||||||
|
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
|
||||||
|
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
|
||||||
|
]);
|
||||||
|
expect(errors.join('\n')).toContain('broker-absent');
|
||||||
|
expect(errors.join('\n')).toContain('mosaic fleet install');
|
||||||
|
} finally {
|
||||||
|
console.error = origError;
|
||||||
|
await rm(home, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fleet start re-probes the broker on the SECOND invocation — no ActiveState trust (#1292 sticky half)', async () => {
|
||||||
|
const home = await tempDir();
|
||||||
|
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||||
|
await mkdir(join(home, 'fleet'), { recursive: true });
|
||||||
|
await writeFile(
|
||||||
|
rosterPath,
|
||||||
|
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
|
||||||
|
'\n',
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const runner: CommandRunner = async (command, args) => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
};
|
||||||
|
const program = new Command();
|
||||||
|
program.exitOverride();
|
||||||
|
// Broker socket NEVER appears — the second start must refuse exactly like
|
||||||
|
// the first; RemainAfterExit-style stale unit state changes nothing
|
||||||
|
// because the check is the socket, not systemctl.
|
||||||
|
registerFleetCommand(program, {
|
||||||
|
runner,
|
||||||
|
mosaicHome: home,
|
||||||
|
checkBrokerSocket: async () => false,
|
||||||
|
});
|
||||||
|
const errors: string[] = [];
|
||||||
|
const origError = console.error;
|
||||||
|
console.error = (...args: unknown[]) => {
|
||||||
|
errors.push(args.join(' '));
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||||
|
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||||
|
// Two invocations, each refusing after its own broker attempt:
|
||||||
|
expect(
|
||||||
|
calls.filter((c) => c.join(' ') === 'systemctl --user start [email protected]'),
|
||||||
|
).toHaveLength(0);
|
||||||
|
expect(errors.filter((e) => e.includes('broker-absent')).length).toBeGreaterThanOrEqual(2);
|
||||||
|
} finally {
|
||||||
|
console.error = origError;
|
||||||
|
await rm(home, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
|
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
|
||||||
const home = await tempDir();
|
const home = await tempDir();
|
||||||
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||||
@@ -2066,8 +2179,19 @@ describe('fleet install — auto-enable units for boot-survival', () => {
|
|||||||
|
|
||||||
await enableFleetUnits(runner, minimalRoster, {});
|
await enableFleetUnits(runner, minimalRoster, {});
|
||||||
|
|
||||||
|
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-lease-broker.service']);
|
||||||
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
|
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
|
||||||
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
|
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
|
||||||
|
// The broker must be enabled BEFORE the holder and agents: a start of any
|
||||||
|
// gated runtime without the broker is exactly the #1292 4-second death.
|
||||||
|
const brokerIndex = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
|
||||||
|
);
|
||||||
|
const holderIndex = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user enable mosaic-tmux-holder.service',
|
||||||
|
);
|
||||||
|
expect(brokerIndex).toBeGreaterThanOrEqual(0);
|
||||||
|
expect(brokerIndex).toBeLessThan(holderIndex);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {
|
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {
|
||||||
@@ -4362,3 +4486,70 @@ describe('fleet ps — heartbeat path resolution', () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('#1297 review: the real broker probe, exercised without any seam', () => {
|
||||||
|
it('brokerSocketPresent answers a REAL unix socket via stat().isSocket() (access(S_IFSOCK) threw ERR_OUT_OF_RANGE)', async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const sockPath = join(dir, 'broker.sock');
|
||||||
|
const server = createServer();
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
server.listen(sockPath, resolve);
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
// A live unix socket answers true through the REAL probe — no seam.
|
||||||
|
expect(await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: sockPath })).toBe(true);
|
||||||
|
// Discrimination is by file type: a regular file that EXISTS is not a
|
||||||
|
// socket. The old implementation could not reach either verdict — it
|
||||||
|
// threw ERR_OUT_OF_RANGE (node >= 24) and the catch answered false.
|
||||||
|
const notASocket = join(dir, 'not-a-sock');
|
||||||
|
await writeFile(notASocket, 'x');
|
||||||
|
expect(await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: notASocket })).toBe(false);
|
||||||
|
// Absent path: false, not a throw.
|
||||||
|
expect(
|
||||||
|
await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: join(dir, 'gone.sock') }),
|
||||||
|
).toBe(false);
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
server.close(() => resolve());
|
||||||
|
});
|
||||||
|
}
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
// EACCES-based unlink failure requires a non-root uid: root bypasses
|
||||||
|
// directory mode bits (CAP_DAC_OVERRIDE), so the abort path cannot be
|
||||||
|
// triggered this way under CI's root runner. Skipped there, exercised on
|
||||||
|
// every non-root dev host.
|
||||||
|
const itUnlessRoot =
|
||||||
|
typeof process.getuid === 'function' && process.getuid() === 0 ? it.skip : it;
|
||||||
|
itUnlessRoot(
|
||||||
|
'placeUnitFile aborts with UnitPlacementError when unlink fails — never copies through a live symlink',
|
||||||
|
async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const unitDir = join(dir, 'systemd', 'user');
|
||||||
|
await mkdir(unitDir, { recursive: true });
|
||||||
|
// By-path residue: destination is a symlink pointing somewhere else.
|
||||||
|
const residueTarget = join(dir, 'residue-target');
|
||||||
|
await writeFile(residueTarget, 'RESIDUE-BYTES');
|
||||||
|
const destination = join(unitDir, 'x.service');
|
||||||
|
await symlink(residueTarget, destination);
|
||||||
|
const source = join(dir, 'seed.service');
|
||||||
|
await writeFile(source, 'UNIT-BYTES');
|
||||||
|
// Read-only unit dir: unlink now fails EACCES (test runs as the owner,
|
||||||
|
// not root, so mode bits are enforced).
|
||||||
|
await chmod(unitDir, 0o500);
|
||||||
|
try {
|
||||||
|
await expect(placeUnitFile(source, unitDir, 'x.service')).rejects.toThrow(
|
||||||
|
UnitPlacementError,
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
await chmod(unitDir, 0o700);
|
||||||
|
}
|
||||||
|
// The copy-through never happened: residue bytes intact, destination
|
||||||
|
// still the symlink (abort, not overwrite-through).
|
||||||
|
expect(await readFile(residueTarget, 'utf8')).toBe('RESIDUE-BYTES');
|
||||||
|
expect(await readlink(destination)).toBe(residueTarget);
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,11 +1,13 @@
|
|||||||
import { constants } from 'node:fs';
|
import { constants, type Stats } from 'node:fs';
|
||||||
import {
|
import {
|
||||||
access,
|
access,
|
||||||
chmod,
|
chmod,
|
||||||
copyFile,
|
copyFile,
|
||||||
|
lstat,
|
||||||
mkdir,
|
mkdir,
|
||||||
open,
|
open,
|
||||||
readFile,
|
readFile,
|
||||||
|
readlink,
|
||||||
stat,
|
stat,
|
||||||
unlink,
|
unlink,
|
||||||
writeFile,
|
writeFile,
|
||||||
@@ -90,6 +92,8 @@ export type SleepFn = (ms: number) => Promise<void>;
|
|||||||
|
|
||||||
export interface FleetCommandDeps {
|
export interface FleetCommandDeps {
|
||||||
runner?: CommandRunner;
|
runner?: CommandRunner;
|
||||||
|
/** Test seam for the #1292 fleet-start broker preflight (socket presence). */
|
||||||
|
checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
|
||||||
/** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */
|
/** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */
|
||||||
interactiveRunner?: InteractiveRunner;
|
interactiveRunner?: InteractiveRunner;
|
||||||
/**
|
/**
|
||||||
@@ -799,6 +803,135 @@ export function buildSystemdEnableCommand(unit: string): string[] {
|
|||||||
return ['systemctl', '--user', 'enable', unit];
|
return ['systemctl', '--user', 'enable', unit];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Place a unit file into the ACTIVE systemd user directory, never through a
|
||||||
|
* symlink (#1292, measured 2026-08-17).
|
||||||
|
*
|
||||||
|
* ⚠ SET-INDEPENDENCE (fomo-lin, 2026-08-17): the set of unit names carrying
|
||||||
|
* by-path residue and the set of unit names this install copies are
|
||||||
|
* INDEPENDENT. Until 0.0.50 they were disjoint only by accident of which
|
||||||
|
* units the install happened to name — fomo-lin survived copy-through solely
|
||||||
|
* because its one by-path symlink (the broker) was the one unit the install
|
||||||
|
* did NOT copy. Adding the broker to the copy set made the intersection
|
||||||
|
* non-empty on the first run. Whoever adds a fifth unit to the placement
|
||||||
|
* list inherits this helper and its unlink step; do not place units with a
|
||||||
|
* bare copyFile.
|
||||||
|
*
|
||||||
|
* A host provisioned by the enable-by-path convention carries a symlink AT
|
||||||
|
* the unit-name path in ~/.config/systemd/user/ pointing at the shipped
|
||||||
|
* template under ~/.config/mosaic/systemd/user/. Node's copyFile FOLLOWS
|
||||||
|
* that link and overwrites the SEED template instead of placing the active
|
||||||
|
* unit (verified with fs.copyFile on a throwaway systemd user instance) —
|
||||||
|
* silent, rc=0, and it mutates the directory every later reseed reads from.
|
||||||
|
* The same measurement showed `systemctl enable <name>` does NOT rewrite an
|
||||||
|
* existing by-path wants-symlink, so reconciliation must be explicit.
|
||||||
|
*
|
||||||
|
* Placement therefore: if the destination is a symlink, unlink it first
|
||||||
|
* (unlink → copy — copy-then-unlink would mutate the seed and then destroy
|
||||||
|
* the evidence that it did); then copy. Also removes a stale
|
||||||
|
* `default.target.wants/<name>` symlink that points outside the active
|
||||||
|
* directory (readlink — NOT readFile, which follows the link and returns the
|
||||||
|
* target's CONTENT), so the subsequent enable-by-name recreates it against
|
||||||
|
* the active copy. Idempotent: on a clean or already-reconciled destination
|
||||||
|
* every step is a no-op (the copy rewrites identical bytes).
|
||||||
|
*
|
||||||
|
* Returns what was done, for assertions and install reporting.
|
||||||
|
*/
|
||||||
|
export interface PlaceUnitResult {
|
||||||
|
readonly unit: string;
|
||||||
|
readonly destination: string;
|
||||||
|
/** A symlink at the unit-name path was unlinked (by-path residue). */
|
||||||
|
readonly unlinkedDestinationSymlink: boolean;
|
||||||
|
/** A stale wants-symlink pointing outside the active dir was removed. */
|
||||||
|
readonly removedStaleWantsSymlink: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* placeUnitFile failed. Thrown BEFORE any copy: no destination bytes were
|
||||||
|
* written, so a residue target cannot have been clobbered by a copy-through
|
||||||
|
* (#1297 review F2).
|
||||||
|
*/
|
||||||
|
export class UnitPlacementError extends Error {
|
||||||
|
constructor(
|
||||||
|
readonly unit: string,
|
||||||
|
message: string,
|
||||||
|
) {
|
||||||
|
super(message);
|
||||||
|
this.name = UnitPlacementError.name;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function isErrnoException(error: unknown): error is NodeJS.ErrnoException {
|
||||||
|
return error instanceof Error && 'code' in error && typeof error.code === 'string';
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function placeUnitFile(
|
||||||
|
source: string,
|
||||||
|
systemdUserDir: string,
|
||||||
|
unit: string,
|
||||||
|
): Promise<PlaceUnitResult> {
|
||||||
|
const destination = join(systemdUserDir, unit);
|
||||||
|
let unlinkedDestinationSymlink = false;
|
||||||
|
// Destination-absent and unlink-FAILED are different outcomes and must not
|
||||||
|
// share a catch (#1297 review F2): a swallowed unlink error used to fall
|
||||||
|
// through to copyFile through the still-live symlink, silently reintroducing
|
||||||
|
// the exact copy-through this helper exists to prevent.
|
||||||
|
let destinationInfo: Stats | undefined;
|
||||||
|
try {
|
||||||
|
destinationInfo = await lstat(destination);
|
||||||
|
} catch (error) {
|
||||||
|
if (!isErrnoException(error) || error.code !== 'ENOENT') {
|
||||||
|
throw new UnitPlacementError(
|
||||||
|
unit,
|
||||||
|
`cannot inspect destination ${destination}: ${error instanceof Error ? error.message : String(error)}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// ENOENT: absent destination — nothing to unlink, copy below is safe.
|
||||||
|
}
|
||||||
|
if (destinationInfo?.isSymbolicLink()) {
|
||||||
|
try {
|
||||||
|
await unlink(destination);
|
||||||
|
} catch (error) {
|
||||||
|
// Abort BEFORE the copy: proceeding would run copyFile through the
|
||||||
|
// still-live symlink and overwrite the residue target's bytes.
|
||||||
|
throw new UnitPlacementError(
|
||||||
|
unit,
|
||||||
|
`cannot unlink destination symlink ${destination}: ${error instanceof Error ? error.message : String(error)}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
unlinkedDestinationSymlink = true;
|
||||||
|
}
|
||||||
|
await copyFile(source, destination);
|
||||||
|
|
||||||
|
let removedStaleWantsSymlink = false;
|
||||||
|
const wantsLink = join(systemdUserDir, 'default.target.wants', unit);
|
||||||
|
try {
|
||||||
|
const wantsInfo = await lstat(wantsLink);
|
||||||
|
if (wantsInfo.isSymbolicLink()) {
|
||||||
|
// readlink — NOT readFile: readFile FOLLOWS the link and returns the
|
||||||
|
// target file's CONTENT, which is not the question being asked.
|
||||||
|
let target: string | undefined;
|
||||||
|
try {
|
||||||
|
target = await readlink(wantsLink);
|
||||||
|
} catch {
|
||||||
|
target = undefined;
|
||||||
|
}
|
||||||
|
// Normalize (systemctl writes absolute targets; a relative one resolves
|
||||||
|
// against the wants dir). A wants-symlink pointing anywhere other than
|
||||||
|
// the active copy (the by-path convention points at the seed template)
|
||||||
|
// survives enable-by-name unchanged — remove it so enable recreates it.
|
||||||
|
if (target !== undefined && resolve(dirname(wantsLink), target) !== destination) {
|
||||||
|
await unlink(wantsLink);
|
||||||
|
removedStaleWantsSymlink = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// absent wants link — nothing to reconcile
|
||||||
|
}
|
||||||
|
|
||||||
|
return { unit, destination, unlinkedDestinationSymlink, removedStaleWantsSymlink };
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Returns the systemctl --user disable command for a given unit.
|
* Returns the systemctl --user disable command for a given unit.
|
||||||
* Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
|
* Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
|
||||||
@@ -833,6 +966,22 @@ export async function enableFleetUnits(
|
|||||||
let succeeded = 0;
|
let succeeded = 0;
|
||||||
let failed = 0;
|
let failed = 0;
|
||||||
|
|
||||||
|
// The lease broker ships with the fleet and every gated runtime needs it
|
||||||
|
// (#1292): seats die at lease registration without it, and no documented
|
||||||
|
// path ever enabled it. Enabled first — alongside the holder — and the
|
||||||
|
// unit must have been placed by installFleet's placeUnitFile step.
|
||||||
|
const brokerResult = await runner(
|
||||||
|
...splitCommand(buildSystemdEnableCommand('mosaic-lease-broker.service')),
|
||||||
|
);
|
||||||
|
if (brokerResult.exitCode === 0) {
|
||||||
|
succeeded++;
|
||||||
|
} else {
|
||||||
|
failed++;
|
||||||
|
process.stderr.write(
|
||||||
|
`Warning: could not enable mosaic-lease-broker.service: ${brokerResult.stderr || brokerResult.stdout || 'non-zero exit'}\n`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
const holderResult = await runner(
|
const holderResult = await runner(
|
||||||
...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')),
|
...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')),
|
||||||
);
|
);
|
||||||
@@ -1529,7 +1678,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
.description('Install local fleet tools and user systemd units')
|
.description('Install local fleet tools and user systemd units')
|
||||||
.option('--no-enable', 'Skip enabling units for boot-survival')
|
.option('--no-enable', 'Skip enabling units for boot-survival')
|
||||||
.action(async (opts: { enable?: boolean }) => {
|
.action(async (opts: { enable?: boolean }) => {
|
||||||
await installFleet(cmd, frameworkRoot);
|
await installFleet(cmd, frameworkRoot, runner);
|
||||||
// Unit enablement needs agent names only, so it reads either version.
|
// Unit enablement needs agent names only, so it reads either version.
|
||||||
const roster = await loadRosterReadModel(cmd);
|
const roster = await loadRosterReadModel(cmd);
|
||||||
await enableFleetUnits(runner, roster, opts);
|
await enableFleetUnits(runner, roster, opts);
|
||||||
@@ -1540,7 +1689,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
.description('Install local fleet tools and user systemd units')
|
.description('Install local fleet tools and user systemd units')
|
||||||
.option('--no-enable', 'Skip enabling units for boot-survival')
|
.option('--no-enable', 'Skip enabling units for boot-survival')
|
||||||
.action(async (opts: { enable?: boolean }) => {
|
.action(async (opts: { enable?: boolean }) => {
|
||||||
await installFleet(cmd, frameworkRoot);
|
await installFleet(cmd, frameworkRoot, runner);
|
||||||
// Unit enablement needs agent names only, so it reads either version.
|
// Unit enablement needs agent names only, so it reads either version.
|
||||||
const roster = await loadRosterReadModel(cmd);
|
const roster = await loadRosterReadModel(cmd);
|
||||||
await enableFleetUnits(runner, roster, opts);
|
await enableFleetUnits(runner, roster, opts);
|
||||||
@@ -1593,6 +1742,37 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
);
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if (action === 'start') {
|
||||||
|
// Broker preflight (#1292), re-probed on EVERY invocation: a
|
||||||
|
// gated runtime started without a live lease broker dies ~4s in
|
||||||
|
// while the unit reports active (RemainAfterExit) — enabling +
|
||||||
|
// starting here and then RE-CHECKING the socket refuses loudly
|
||||||
|
// instead of reporting rc0 over a doomed start. This is the
|
||||||
|
// second-start check as much as the first: it never trusts unit
|
||||||
|
// ActiveState.
|
||||||
|
await runChecked(runner, [
|
||||||
|
'systemctl',
|
||||||
|
'--user',
|
||||||
|
'enable',
|
||||||
|
'mosaic-lease-broker.service',
|
||||||
|
]);
|
||||||
|
await runChecked(runner, [
|
||||||
|
'systemctl',
|
||||||
|
'--user',
|
||||||
|
'start',
|
||||||
|
'mosaic-lease-broker.service',
|
||||||
|
]);
|
||||||
|
if (!(await brokerSocketPresent(deps))) {
|
||||||
|
console.error(
|
||||||
|
'[fleet] broker-absent: lease broker socket did not appear after enable+start (#1292).',
|
||||||
|
);
|
||||||
|
console.error(
|
||||||
|
'[fleet] remedy: mosaic fleet install (it reconciles either enable convention)',
|
||||||
|
);
|
||||||
|
process.exitCode = 1;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (action === 'restart') {
|
if (action === 'restart') {
|
||||||
// Serialize the holder+agents teardown/relaunch behind the restart lock
|
// Serialize the holder+agents teardown/relaunch behind the restart lock
|
||||||
// so a re-entrant restart waits for clean shutdown before relaunching,
|
// so a re-entrant restart waits for clean shutdown before relaunching,
|
||||||
@@ -2351,7 +2531,11 @@ export function registerFleetAgentCommands(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> {
|
async function installFleet(
|
||||||
|
cmd: Command,
|
||||||
|
frameworkRoot: string,
|
||||||
|
runner: CommandRunner,
|
||||||
|
): Promise<void> {
|
||||||
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
|
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
|
||||||
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
|
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
|
||||||
// Read model first: every file this function places is roster-independent, and
|
// Read model first: every file this function places is roster-independent, and
|
||||||
@@ -2403,18 +2587,40 @@ async function installFleet(cmd: Command, frameworkRoot: string): Promise<void>
|
|||||||
for (const toolPath of executableToolPaths) {
|
for (const toolPath of executableToolPaths) {
|
||||||
await chmod(toolPath, 0o755);
|
await chmod(toolPath, 0o755);
|
||||||
}
|
}
|
||||||
await copyFile(
|
// Unit placement (#1292): every unit goes through placeUnitFile — never a
|
||||||
join(frameworkRoot, 'systemd', 'user', 'mosaic-tmux-holder.service'),
|
// bare copyFile — so a by-path-enable symlink at the destination is
|
||||||
join(activePaths.systemdUserDir, 'mosaic-tmux-holder.service'),
|
// unlinked rather than written through (copy-through would silently
|
||||||
|
// overwrite the SEED template, measured 2026-08-17). The lease broker unit
|
||||||
|
// is placed here too: previously the install named three units and omitted
|
||||||
|
// the broker entirely, which is why no documented path ever enabled it.
|
||||||
|
const placedUnits = await Promise.all(
|
||||||
|
[
|
||||||
|
'mosaic-tmux-holder.service',
|
||||||
|
'[email protected]',
|
||||||
|
'[email protected]',
|
||||||
|
'mosaic-lease-broker.service',
|
||||||
|
].map((unit) =>
|
||||||
|
placeUnitFile(join(frameworkRoot, 'systemd', 'user', unit), activePaths.systemdUserDir, unit),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
await copyFile(
|
const reconciled = placedUnits.filter(
|
||||||
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
|
(result) => result.unlinkedDestinationSymlink || result.removedStaleWantsSymlink,
|
||||||
join(activePaths.systemdUserDir, '[email protected]'),
|
|
||||||
);
|
|
||||||
await copyFile(
|
|
||||||
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
|
|
||||||
join(activePaths.systemdUserDir, '[email protected]'),
|
|
||||||
);
|
);
|
||||||
|
if (reconciled.length > 0) {
|
||||||
|
console.log(
|
||||||
|
`Reconciled ${reconciled.length} unit placement(s) from by-path enable residue: ${reconciled.map((r) => r.unit).join(', ')}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// systemd will not see a replaced unit file without a reload; do it once
|
||||||
|
// after all placements, before any enable call below. runCommand never
|
||||||
|
// rejects (it resolves exitCode 127 on spawn error), so a plain await with
|
||||||
|
// an exitCode check matches the rest of this file's systemctl handling.
|
||||||
|
const reloadResult = await runner(...splitCommand(['systemctl', '--user', 'daemon-reload']));
|
||||||
|
if (reloadResult.exitCode !== 0) {
|
||||||
|
process.stderr.write(
|
||||||
|
`Warning: systemctl --user daemon-reload after unit placement failed (non-systemd host?): ${reloadResult.stderr || reloadResult.stdout || 'non-zero exit'}\n`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// On roster v2 the reconciler owns the generated env: `apply` writes it and
|
// On roster v2 the reconciler owns the generated env: `apply` writes it and
|
||||||
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
|
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
|
||||||
@@ -2611,6 +2817,45 @@ function splitCommand(command: string[]): [string, string[]] {
|
|||||||
return [bin, args];
|
return [bin, args];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Lease-broker socket presence for the fleet-start preflight (#1292).
|
||||||
|
* Resolution precedence matches launch.ts's defaultLeaseBrokerSocket and
|
||||||
|
* start-agent-session.sh's broker_socket_path: explicit
|
||||||
|
* MOSAIC_LEASE_BROKER_SOCKET, else $XDG_RUNTIME_DIR/mosaic-lease/broker.sock,
|
||||||
|
* else /run/user/<uid>/mosaic-lease/broker.sock. Pure filesystem check — this
|
||||||
|
* deliberately does NOT consult systemd state: a unit can be active
|
||||||
|
* (RemainAfterExit) with no live socket, and the socket is the thing the
|
||||||
|
* gated runtime connects to. Injectable via deps for tests.
|
||||||
|
*/
|
||||||
|
export function resolveLeaseBrokerSocketForPreflight(
|
||||||
|
env: NodeJS.ProcessEnv = process.env,
|
||||||
|
uid: number = typeof process.getuid === 'function' ? process.getuid() : 0,
|
||||||
|
): string {
|
||||||
|
if (env['MOSAIC_LEASE_BROKER_SOCKET']) return env['MOSAIC_LEASE_BROKER_SOCKET'];
|
||||||
|
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
|
||||||
|
return join(runtimeDir, 'mosaic-lease', 'broker.sock');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function brokerSocketPresent(
|
||||||
|
deps: FleetCommandDeps,
|
||||||
|
env: NodeJS.ProcessEnv = process.env,
|
||||||
|
): Promise<boolean> {
|
||||||
|
const check = deps.checkBrokerSocket;
|
||||||
|
const socketPath = resolveLeaseBrokerSocketForPreflight(env);
|
||||||
|
if (check) return check(socketPath);
|
||||||
|
// S_IFSOCK (0xC000) is a file-TYPE constant, not an access() mode (0-7):
|
||||||
|
// access(path, S_IFSOCK) throws ERR_OUT_OF_RANGE on node >= 24 (measured on
|
||||||
|
// v24.18.0, #1297 review F1) and cannot succeed on any node — the old catch
|
||||||
|
// swallowed the throw, so this probe could NEVER return true and every
|
||||||
|
// un-seamed call reported the broker absent. stat() + isSocket() is the real
|
||||||
|
// check and matches the bash side's [ -S ].
|
||||||
|
try {
|
||||||
|
return (await stat(socketPath)).isSocket();
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** All supported fleet profile names. */
|
/** All supported fleet profile names. */
|
||||||
export type FleetProfile =
|
export type FleetProfile =
|
||||||
| 'general'
|
| 'general'
|
||||||
|
|||||||
@@ -205,6 +205,12 @@ export async function runLeaseEnforcementDoctorCheck(
|
|||||||
message:
|
message:
|
||||||
`Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` +
|
`Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` +
|
||||||
'Every gated tool call will fail closed and BRICK this agent (see #869). ' +
|
'Every gated tool call will fail closed and BRICK this agent (see #869). ' +
|
||||||
'Remediate by activating the lease-broker supervisor (systemd unit + socket) or by removing the enforcement hooks from ~/.claude/settings.json.',
|
// #1292: one remedy, correct under BOTH enable conventions (by-path on
|
||||||
|
// the seed template, and copy-then-enable in the active dir). Written
|
||||||
|
// from the 2026-08-17 symlink measurement: `systemctl enable` by name
|
||||||
|
// does NOT rewrite an existing by-path wants-symlink, so teaching a
|
||||||
|
// manual systemctl line here could leave a host with two competing
|
||||||
|
// wants links. fleet install reconciles either shape.
|
||||||
|
'Remedy: run `mosaic fleet install` (it reconciles either enable convention), or remove the enforcement hooks from ~/.claude/settings.json.',
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -169,6 +169,16 @@ function reconcileDeps(host: FakeLifecycleHost): FleetReconcileDeps {
|
|||||||
applyProjection: async () => undefined,
|
applyProjection: async () => undefined,
|
||||||
readRoster: async () => host.roster,
|
readRoster: async () => host.roster,
|
||||||
acquireMutationLock: async () => async () => undefined,
|
acquireMutationLock: async () => async () => undefined,
|
||||||
|
// Hermetic broker observation (#1297 F3): without this, the plan probes
|
||||||
|
// the REAL host filesystem, so the "stable JSON" fixtures answered true
|
||||||
|
// on any machine with a live lease broker and false elsewhere. Pointing
|
||||||
|
// both paths at fixtures that do not exist pins socketPresent:false and
|
||||||
|
// unitInstalled:false on every host, which is what these fixtures assert.
|
||||||
|
brokerSocketEnv: {
|
||||||
|
MOSAIC_LEASE_BROKER_SOCKET: '/nonexistent/mosaic-lease/broker.sock',
|
||||||
|
XDG_CONFIG_HOME: '/nonexistent/mosaic-config',
|
||||||
|
XDG_RUNTIME_DIR: '/nonexistent/run',
|
||||||
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -459,6 +469,7 @@ describe('FCM-M3-002 reconciler lifecycle acceptance', (): void => {
|
|||||||
plan: {
|
plan: {
|
||||||
generation: 7,
|
generation: 7,
|
||||||
holder: 'owned',
|
holder: 'owned',
|
||||||
|
broker: { unitInstalled: false, socketPresent: false },
|
||||||
agents: [
|
agents: [
|
||||||
{
|
{
|
||||||
name: 'coder0',
|
name: 'coder0',
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
|
import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
|
import { createServer } from 'node:net';
|
||||||
import { afterEach, describe, expect, it } from 'vitest';
|
import { afterEach, describe, expect, it } from 'vitest';
|
||||||
import {
|
import {
|
||||||
acquirePrivateReconcileLock,
|
acquirePrivateReconcileLock,
|
||||||
@@ -92,6 +93,179 @@ async function run(command: FleetReconcileCommand, overrides: Partial<FleetRecon
|
|||||||
}
|
}
|
||||||
|
|
||||||
describe('fleet roster-owned reconciler', (): void => {
|
describe('fleet roster-owned reconciler', (): void => {
|
||||||
|
// ── #1292: broker as first-class plan member + broker-first start ordering ──
|
||||||
|
|
||||||
|
it('reports broker unit and socket state in the plan (socket is the signal, not unit state)', async (): Promise<void> => {
|
||||||
|
const result = await run('status', {
|
||||||
|
statPath: async () => true,
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
|
});
|
||||||
|
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports a dead broker as socketPresent=false even when the unit is installed (enabled-but-dead is the #1292 shape)', async (): Promise<void> => {
|
||||||
|
const result = await run('status', {
|
||||||
|
statPath: async () => true,
|
||||||
|
checkBrokerSocket: async () => false,
|
||||||
|
});
|
||||||
|
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: false });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('probes the REAL filesystem when no seam is injected — live socket and unit report healthy, absent paths report absent (#1297 F3)', async (): Promise<void> => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'mosaic-broker-probe-'));
|
||||||
|
cleanup = dir;
|
||||||
|
const configHome = join(dir, 'config');
|
||||||
|
const unitDir = join(configHome, 'systemd', 'user');
|
||||||
|
await mkdir(unitDir, { recursive: true });
|
||||||
|
await writeFile(join(unitDir, 'mosaic-lease-broker.service'), '[Unit]\n');
|
||||||
|
const sockPath = join(dir, 'broker.sock');
|
||||||
|
const server = createServer();
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
server.listen(sockPath, resolve);
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const result = await run('status', {
|
||||||
|
brokerSocketEnv: {
|
||||||
|
MOSAIC_LEASE_BROKER_SOCKET: sockPath,
|
||||||
|
XDG_CONFIG_HOME: configHome,
|
||||||
|
XDG_RUNTIME_DIR: dir,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
|
||||||
|
// Absent paths through the SAME seam-less path answer false — this is
|
||||||
|
// the half the old default got right; healthy is the half it got wrong.
|
||||||
|
const absent = await run('status', {
|
||||||
|
brokerSocketEnv: {
|
||||||
|
MOSAIC_LEASE_BROKER_SOCKET: join(dir, 'gone.sock'),
|
||||||
|
XDG_CONFIG_HOME: join(dir, 'gone-config'),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(absent.plan.broker).toEqual({ unitInstalled: false, socketPresent: false });
|
||||||
|
} finally {
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
server.close(() => resolve());
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('command start refuses with a named error when the broker socket does not appear after enable+start (#1297 F3)', async (): Promise<void> => {
|
||||||
|
const calls: string[][] = [];
|
||||||
|
await expect(
|
||||||
|
run('start', {
|
||||||
|
checkBrokerSocket: async () => false,
|
||||||
|
runner: async (command, args) => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||||
|
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
|
||||||
|
}
|
||||||
|
if (command === 'tmux' && args.includes('show-environment')) {
|
||||||
|
return {
|
||||||
|
stdout:
|
||||||
|
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
|
||||||
|
stderr: '',
|
||||||
|
exitCode: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
).rejects.toThrow(/broker-absent/);
|
||||||
|
// Refused: broker enable+start attempted, no holder/agent unit touched.
|
||||||
|
const agentStarts = calls.filter(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start [email protected]',
|
||||||
|
);
|
||||||
|
expect(agentStarts).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('command start enables and starts the broker BEFORE the holder and any agent unit', async (): Promise<void> => {
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const result = await run('start', {
|
||||||
|
// Deterministic broker presence: without the seam this test answers the
|
||||||
|
// HOST's broker state (passes on a machine with a live broker, refuses
|
||||||
|
// on CI), not the ordering property it exists for (#1297 follow-up).
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
|
runner: async (command, args) => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||||
|
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
|
||||||
|
}
|
||||||
|
if (command === 'tmux' && args.includes('show-environment')) {
|
||||||
|
return {
|
||||||
|
stdout:
|
||||||
|
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
|
||||||
|
stderr: '',
|
||||||
|
exitCode: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(result.lifecycle).toBe('complete');
|
||||||
|
const brokerEnable = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
|
||||||
|
);
|
||||||
|
const brokerStart = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
|
||||||
|
);
|
||||||
|
const holderStart = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start mosaic-tmux-holder.service',
|
||||||
|
);
|
||||||
|
const agentStart = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start [email protected]',
|
||||||
|
);
|
||||||
|
expect(brokerEnable).toBeGreaterThanOrEqual(0);
|
||||||
|
expect(brokerStart).toBeGreaterThan(brokerEnable);
|
||||||
|
// Holder start may be absent (holder 'owned' in this fixture); if present it must follow the broker.
|
||||||
|
if (holderStart >= 0) expect(holderStart).toBeGreaterThan(brokerStart);
|
||||||
|
expect(agentStart).toBeGreaterThan(brokerStart);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('apply with a running desired agent also enables and starts the broker first', async (): Promise<void> => {
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const runningRoster: FleetRosterV2 = {
|
||||||
|
...roster,
|
||||||
|
agents: roster.agents.map((agent) =>
|
||||||
|
agent.name === 'coder0'
|
||||||
|
? { ...agent, lifecycle: { enabled: true, desiredState: 'running' as const } }
|
||||||
|
: agent,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
const result = await executeFleetReconcile({
|
||||||
|
roster: runningRoster,
|
||||||
|
command: 'apply',
|
||||||
|
expectedGeneration: 7,
|
||||||
|
deps: deps({
|
||||||
|
readRoster: async () => runningRoster,
|
||||||
|
// Deterministic broker presence (see start-ordering test note).
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
|
runner: async (command, args) => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||||
|
return { stdout: '_holder\n', stderr: '', exitCode: 0 };
|
||||||
|
}
|
||||||
|
if (command === 'tmux' && args.includes('show-environment')) {
|
||||||
|
return {
|
||||||
|
stdout:
|
||||||
|
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
|
||||||
|
stderr: '',
|
||||||
|
exitCode: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
expect(result.applied).toBe(true);
|
||||||
|
const brokerStart = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
|
||||||
|
);
|
||||||
|
const agentStart = calls.findIndex(
|
||||||
|
(c) => c.join(' ') === 'systemctl --user start [email protected]',
|
||||||
|
);
|
||||||
|
expect(brokerStart).toBeGreaterThanOrEqual(0);
|
||||||
|
expect(agentStart).toBeGreaterThan(brokerStart);
|
||||||
|
});
|
||||||
|
|
||||||
it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => {
|
it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => {
|
||||||
const home = await lockHome();
|
const home = await lockHome();
|
||||||
const fleet = join(home, 'fleet');
|
const fleet = join(home, 'fleet');
|
||||||
@@ -375,6 +549,8 @@ describe('fleet roster-owned reconciler', (): void => {
|
|||||||
expectedGeneration: 7,
|
expectedGeneration: 7,
|
||||||
deps: deps({
|
deps: deps({
|
||||||
readRoster: async () => runningRoster,
|
readRoster: async () => runningRoster,
|
||||||
|
// Deterministic broker presence (see start-ordering test note).
|
||||||
|
checkBrokerSocket: async () => true,
|
||||||
runner: async (command, args) => {
|
runner: async (command, args) => {
|
||||||
calls.push([command, ...args]);
|
calls.push([command, ...args]);
|
||||||
if (command === 'tmux' && args.includes('list-sessions')) {
|
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { constants } from 'node:fs';
|
import { constants } from 'node:fs';
|
||||||
import { lstat, open, readFile, unlink, type FileHandle } from 'node:fs/promises';
|
import { lstat, open, readFile, stat, unlink, type FileHandle } from 'node:fs/promises';
|
||||||
import { randomUUID } from 'node:crypto';
|
import { randomUUID } from 'node:crypto';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
@@ -44,6 +44,10 @@ export interface FleetReconcileDeps {
|
|||||||
readonly overrideDir?: string;
|
readonly overrideDir?: string;
|
||||||
readonly homeDirectory?: string;
|
readonly homeDirectory?: string;
|
||||||
readonly readHolderIdentity?: () => Promise<string>;
|
readonly readHolderIdentity?: () => Promise<string>;
|
||||||
|
/** Test/observation seams for the lease-broker plan member (#1292). */
|
||||||
|
readonly statPath?: (path: string) => Promise<boolean> | boolean;
|
||||||
|
readonly checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
|
||||||
|
readonly brokerSocketEnv?: NodeJS.ProcessEnv;
|
||||||
readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>;
|
readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>;
|
||||||
readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>;
|
readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>;
|
||||||
readonly applyProjection?: (prepared: unknown) => Promise<unknown>;
|
readonly applyProjection?: (prepared: unknown) => Promise<unknown>;
|
||||||
@@ -75,6 +79,17 @@ export interface FleetReconcileObservedAgent {
|
|||||||
export interface FleetReconcilePlan {
|
export interface FleetReconcilePlan {
|
||||||
readonly generation: number;
|
readonly generation: number;
|
||||||
readonly holder: 'owned' | 'missing' | 'ownership-mismatch';
|
readonly holder: 'owned' | 'missing' | 'ownership-mismatch';
|
||||||
|
/**
|
||||||
|
* Lease broker observation (#1292): every gated runtime registers with the
|
||||||
|
* broker or dies ~4s in — a broker not in the plan cannot be reported as
|
||||||
|
* drifted, which made "broker died an hour ago" and "broker fine"
|
||||||
|
* produce identical output. `unitInstalled` = unit file present in the
|
||||||
|
* active dir; `socketPresent` = live broker at the resolved socket path.
|
||||||
|
*/
|
||||||
|
readonly broker: {
|
||||||
|
readonly unitInstalled: boolean;
|
||||||
|
readonly socketPresent: boolean;
|
||||||
|
};
|
||||||
readonly agents: readonly FleetReconcileObservedAgent[];
|
readonly agents: readonly FleetReconcileObservedAgent[];
|
||||||
readonly unmanagedSessions: readonly string[];
|
readonly unmanagedSessions: readonly string[];
|
||||||
}
|
}
|
||||||
@@ -246,7 +261,17 @@ export async function executeFleetReconcile(
|
|||||||
lifecycle: 'complete',
|
lifecycle: 'complete',
|
||||||
plan,
|
plan,
|
||||||
};
|
};
|
||||||
} catch {
|
} catch (error: unknown) {
|
||||||
|
// A named lifecycle precondition (broker-absent after enable+start,
|
||||||
|
// #1297 F3) must surface as itself — converting it to the generic
|
||||||
|
// recoverable result would hide the diagnosis and report a clean
|
||||||
|
// refusal where a loud one is the point.
|
||||||
|
if (
|
||||||
|
error instanceof FleetReconcileError &&
|
||||||
|
error.code === 'lifecycle-precondition-failed'
|
||||||
|
) {
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
result = {
|
result = {
|
||||||
applied: false,
|
applied: false,
|
||||||
authoritativeRoster: 'unchanged',
|
authoritativeRoster: 'unchanged',
|
||||||
@@ -315,6 +340,63 @@ function isObservational(command: FleetReconcileCommand): boolean {
|
|||||||
return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor';
|
return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Observe the lease broker for the plan (#1292). Unit presence via systemctl
|
||||||
|
* is-system-running is NOT the signal — a unit can be enabled-but-dead. The
|
||||||
|
* authoritative signal is the socket the gated runtimes connect to, matching
|
||||||
|
* broker-supervisor.ts's `checkBrokerSupervisorHealth` (healthy ===
|
||||||
|
* socketPresent). Injectable so tests drive every branch without a broker.
|
||||||
|
*/
|
||||||
|
function resolveBrokerSocketPath(env: NodeJS.ProcessEnv): string {
|
||||||
|
const uid = typeof process.getuid === 'function' ? process.getuid() : 0;
|
||||||
|
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
|
||||||
|
return env['MOSAIC_LEASE_BROKER_SOCKET'] ?? join(runtimeDir, 'mosaic-lease', 'broker.sock');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Probe the broker socket. Seams take precedence, but with no seam injected
|
||||||
|
* the REAL stat().isSocket() runs (#1297 review F3): production passes no
|
||||||
|
* seams, and defaulting to false made plan/status/doctor report a healthy
|
||||||
|
* broker as absent — a dead broker was indistinguishable from noise.
|
||||||
|
*/
|
||||||
|
async function brokerSocketPresent(
|
||||||
|
deps: FleetReconcileDeps,
|
||||||
|
env: NodeJS.ProcessEnv,
|
||||||
|
): Promise<boolean> {
|
||||||
|
const socketPath = resolveBrokerSocketPath(env);
|
||||||
|
const check = deps.checkBrokerSocket;
|
||||||
|
if (check) return check(socketPath);
|
||||||
|
try {
|
||||||
|
return (await stat(socketPath)).isSocket();
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function observeBroker(deps: FleetReconcileDeps): Promise<FleetReconcilePlan['broker']> {
|
||||||
|
const homeDirectory = deps.homeDirectory ?? homedir();
|
||||||
|
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
|
||||||
|
const configHome = env['XDG_CONFIG_HOME'] ?? join(homeDirectory, '.config');
|
||||||
|
const unitPath = join(configHome, 'systemd', 'user', 'mosaic-lease-broker.service');
|
||||||
|
const statPath = deps.statPath;
|
||||||
|
let unitInstalled = false;
|
||||||
|
let socketPresent = false;
|
||||||
|
try {
|
||||||
|
// Same principle as the socket probe: no seam → look at the real
|
||||||
|
// filesystem. A unit file placed by installFleet (or a by-path residue
|
||||||
|
// symlink resolving to it) satisfies stat().isFile().
|
||||||
|
unitInstalled = statPath ? await statPath(unitPath) : (await stat(unitPath)).isFile();
|
||||||
|
} catch {
|
||||||
|
unitInstalled = false;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
socketPresent = await brokerSocketPresent(deps, env);
|
||||||
|
} catch {
|
||||||
|
socketPresent = false;
|
||||||
|
}
|
||||||
|
return { unitInstalled, socketPresent };
|
||||||
|
}
|
||||||
|
|
||||||
async function observeFleet(
|
async function observeFleet(
|
||||||
roster: FleetRosterV2,
|
roster: FleetRosterV2,
|
||||||
deps: FleetReconcileDeps,
|
deps: FleetReconcileDeps,
|
||||||
@@ -325,10 +407,12 @@ async function observeFleet(
|
|||||||
'-F',
|
'-F',
|
||||||
'#{session_name}',
|
'#{session_name}',
|
||||||
]);
|
]);
|
||||||
|
const broker = await observeBroker(deps);
|
||||||
if (sessionsResult.exitCode !== 0) {
|
if (sessionsResult.exitCode !== 0) {
|
||||||
return {
|
return {
|
||||||
generation: roster.generation,
|
generation: roster.generation,
|
||||||
holder: 'missing',
|
holder: 'missing',
|
||||||
|
broker,
|
||||||
agents: await observeAgents(roster, deps, new Set<string>()),
|
agents: await observeAgents(roster, deps, new Set<string>()),
|
||||||
unmanagedSessions: [],
|
unmanagedSessions: [],
|
||||||
};
|
};
|
||||||
@@ -351,6 +435,7 @@ async function observeFleet(
|
|||||||
return {
|
return {
|
||||||
generation: roster.generation,
|
generation: roster.generation,
|
||||||
holder,
|
holder,
|
||||||
|
broker,
|
||||||
agents: await observeAgents(roster, deps, sessions),
|
agents: await observeAgents(roster, deps, sessions),
|
||||||
unmanagedSessions: Object.freeze(unmanagedSessions.sort()),
|
unmanagedSessions: Object.freeze(unmanagedSessions.sort()),
|
||||||
};
|
};
|
||||||
@@ -507,6 +592,17 @@ async function executeExplicitLifecycle(
|
|||||||
plan: FleetReconcilePlan,
|
plan: FleetReconcilePlan,
|
||||||
agents: readonly FleetRosterV2Agent[],
|
agents: readonly FleetRosterV2Agent[],
|
||||||
): Promise<FleetReconcileResult> {
|
): Promise<FleetReconcileResult> {
|
||||||
|
const lifecycleApplyFailed = (): FleetReconcileResult => ({
|
||||||
|
applied: false,
|
||||||
|
authoritativeRoster: 'unchanged',
|
||||||
|
projections: 'not-applied',
|
||||||
|
lifecycle: 'incomplete',
|
||||||
|
plan,
|
||||||
|
recovery: {
|
||||||
|
code: 'lifecycle-apply-failed',
|
||||||
|
action: 'rerun-after-inspecting-owned-resources',
|
||||||
|
},
|
||||||
|
});
|
||||||
if (request.command === 'start') {
|
if (request.command === 'start') {
|
||||||
for (const agent of agents) {
|
for (const agent of agents) {
|
||||||
if (!agent.lifecycle.enabled) {
|
if (!agent.lifecycle.enabled) {
|
||||||
@@ -517,6 +613,40 @@ async function executeExplicitLifecycle(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Broker FIRST (#1292): a gated runtime started without a running lease
|
||||||
|
// broker dies ~4 seconds in at registration — enable the unit (install
|
||||||
|
// places it) and start it before any holder/agent lifecycle effect.
|
||||||
|
try {
|
||||||
|
if (request.command === 'start') {
|
||||||
|
await runChecked(request.deps, 'systemctl', [
|
||||||
|
'--user',
|
||||||
|
'enable',
|
||||||
|
'mosaic-lease-broker.service',
|
||||||
|
]);
|
||||||
|
await runChecked(request.deps, 'systemctl', [
|
||||||
|
'--user',
|
||||||
|
'start',
|
||||||
|
'mosaic-lease-broker.service',
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return lifecycleApplyFailed();
|
||||||
|
}
|
||||||
|
if (request.command === 'start') {
|
||||||
|
// Socket re-check after start, as a NAMED precondition (#1297 review
|
||||||
|
// F3) — the same protection the v1 path in commands/fleet.ts has had all
|
||||||
|
// along: the unit reporting active is not the signal; the socket is.
|
||||||
|
// Deliberately outside the try/catch above: a swallowed FleetReconcileError
|
||||||
|
// here read as a generic recoverable failure, hiding the named refusal.
|
||||||
|
// Runs BEFORE any holder/agent unit is touched so nothing doomed starts.
|
||||||
|
const env = (request.deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
|
||||||
|
if (!(await brokerSocketPresent(request.deps, env))) {
|
||||||
|
throw new FleetReconcileError(
|
||||||
|
'lifecycle-precondition-failed',
|
||||||
|
'broker-absent: lease broker socket did not appear after enable+start (#1292; #1297 F3). Remedy: mosaic fleet install.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
try {
|
try {
|
||||||
if (request.command === 'start' && plan.holder === 'missing') {
|
if (request.command === 'start' && plan.holder === 'missing') {
|
||||||
await runChecked(request.deps, 'systemctl', [
|
await runChecked(request.deps, 'systemctl', [
|
||||||
@@ -533,17 +663,7 @@ async function executeExplicitLifecycle(
|
|||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
return {
|
return lifecycleApplyFailed();
|
||||||
applied: false,
|
|
||||||
authoritativeRoster: 'unchanged',
|
|
||||||
projections: 'not-applied',
|
|
||||||
lifecycle: 'incomplete',
|
|
||||||
plan,
|
|
||||||
recovery: {
|
|
||||||
code: 'lifecycle-apply-failed',
|
|
||||||
action: 'rerun-after-inspecting-owned-resources',
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
return {
|
return {
|
||||||
applied: true,
|
applied: true,
|
||||||
@@ -563,6 +683,22 @@ async function applyDesiredLifecycle(
|
|||||||
(agent: FleetRosterV2Agent): boolean =>
|
(agent: FleetRosterV2Agent): boolean =>
|
||||||
agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running',
|
agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running',
|
||||||
);
|
);
|
||||||
|
// Broker before any running agent, same ordering and reason as the
|
||||||
|
// command-driven path above (#1292).
|
||||||
|
if (needsRunningAgent) {
|
||||||
|
await runChecked(deps, 'systemctl', ['--user', 'enable', 'mosaic-lease-broker.service']);
|
||||||
|
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-lease-broker.service']);
|
||||||
|
// Same socket re-check as the explicit start path (#1297 F3): apply with
|
||||||
|
// running desired agents starts gated runtimes too, and a broker that
|
||||||
|
// starts but never binds dooms them the same way.
|
||||||
|
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
|
||||||
|
if (!(await brokerSocketPresent(deps, env))) {
|
||||||
|
throw new FleetReconcileError(
|
||||||
|
'lifecycle-precondition-failed',
|
||||||
|
'broker-absent: lease broker socket did not appear after enable+start (#1292; #1297 F3). Remedy: mosaic fleet install.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
if (needsRunningAgent && plan.holder === 'missing') {
|
if (needsRunningAgent && plan.holder === 'missing') {
|
||||||
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']);
|
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']);
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user