Commit Graph
3 Commits
Author SHA1 Message Date
fargo 7de3374fc9 ci(#1297): wire the broker-preflight suite into test:framework-shell (#1017 enumeration)
ci/woodpecker/pr/ci Pipeline failed
Pipeline 2468 and 2556 both failed the CI 'sanitization' step. The log
(2556, step 60166) shows verify-sanitized.sh PASSED both times — the
failure was the test-enumeration guard bundled in the same step: the new
tools/fleet/test-agent-session-broker-preflight.sh existed on disk but
was not enumerated on any CI surface. UNENUMERATED, 1 failure.

The suite is CI-fit by construction (self-controlled preconditions: fake
tmux/mosaic on PATH, real tmpdir unix socket, hermetic env — see its
header), so it belongs IN the chain, not in the exclusions. Wired as the
last link of test:framework-shell; guard now reports population 53,
enumerated 37, excluded 16 (was 36/16).
2026-08-20 11:50:06 -05:00
fargo a7eff7ced7 fix(#1297 review): real socket probe, honest placement abort, observable broker state
ci/woodpecker/pr/ci Pipeline failed
rev-security-03 REQUEST_CHANGES (review note on the brain), all four
findings verified by measurement before fixing:

F1 BLOCKER — brokerSocketPresent used access(path, S_IFSOCK).
S_IFSOCK (0xC000) is a file-type constant, not an access() mode (0-7):
on node 24.18.0 the call throws ERR_OUT_OF_RANGE, so the probe could
NEVER return true and the swallow-catch made every un-seamed call report
the broker absent — on this host (roster v1) 'mosaic fleet start' would
have refused broker-absent forever. Now stat().isSocket(), matching the
bash side's [ -S ]. New spec exercises the REAL probe against a REAL
unix socket (true), a regular file (false), and an absent path (false) —
no seam, so no seam can hide this again.

F2 — placeUnitFile's single catch conflated destination-absent with
unlink-FAILED; on unlink failure it copied through the still-live
symlink (copy-through overwrite measured by the reviewer). Now: ENOENT
is the only swallowed lstat outcome; unlink failure aborts with a named
UnitPlacementError BEFORE any copy. New spec proves the residue target's
bytes survive an unlink failure and the destination link is untouched.

F3 — observeBroker defaulted unit/socket to false when seams unset, and
production injects none: plan/status/doctor reported a healthy broker as
absent. Seams still take precedence; with no seam the real stat()
probes run. The v2 start path (and apply-with-running) now re-check the
socket after enable+start with a NAMED lifecycle-precondition-failed
refusal — previously only the v1 path had that protection, and the
refusal was masked into the generic recoverable result. Acceptance
fixtures gain hermetic brokerSocketEnv paths (the old fixture asserted
the lying default).

F4 (note) — unlink race stays inside the user-owned dir; no action.

Local gates on this tree: vitest 1566/1566, typecheck, lint, prettier
3.8.1, verify-sanitized all pass. CI 2468's sanitization failure did not
reproduce locally on the identical tree; watching the fresh pipeline.
2026-08-20 11:32:27 -05:00
fargo 9d3e22b1c1 fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292)
ci/woodpecker/pr/ci Pipeline failed
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
2026-08-17 18:38:22 -05:00