mosaicstack/stack
13
0
Fork 1
Code Issues 304 Pull Requests 10 Actions Packages Projects Releases 15 Wiki Activity

fix(fleet): lease-broker activation, symlink-safe unit placement, named launch refusal — Wall 6 (#1292) #1297

Merged
fred merged 5 commits from fix/1292-lease-broker-activation into next 2026-08-21 00:40:30 +00:00
Conversation 2 Commits 5 Files Changed 11
+1229 -33
fargo commented 2026-08-17 23:39:12 +00:00
Member
Copy Link
Copy Source

Wall 6 / #1292 — lease-broker activation, safe unit placement, and named launch refusal

Card implemented under fred's wave-3 rulings (design proposal → symlink measurement → amended design → GO with widened placement scope), branch fix/1292-lease-broker-activation, base next @ 8199261. Activation lands in the control plane, not the launcher.

The measured foundations (all on throwaway systemd user instances + fomo-lin read-only probes)

  1. systemctl enable <name> does not rewrite an existing by-path wants-symlink — rc=0, silent, the host stays on the old copy indefinitely.
  2. Node copyFile follows a by-path symlink at the destination and overwrites the seed template (verified with cp AND fs.copyFile).
  3. fomo-lin (the only in-the-wild by-path host) survived #2 only because its one symlink was the one unit the install never copied — the residue/copy sets were disjoint by accident. Adding the broker to the copy set made them intersect on the first run, so unlink-before-copy is required for a host that exists today, not a hypothetical.

The fix

  • placeUnitFile — every unit the install places (all four, including the broker, previously omitted entirely) goes through one helper: unlink destination symlink → copy → remove stale wants-symlink (via readlink, not readFile) → daemon-reload. Idempotent (double-install convergence asserted). SET-INDEPENDENCE warning documented for whoever adds a fifth unit.
  • enableFleetUnits enables the broker first, alongside the holder.
  • fleet start / reconciler start the broker before any holder/agent effect, then re-check the socket (never unit ActiveState) and refuse nonzero with a named code if it did not appear — re-probed on every invocation, so RemainAfterExit=yes can never make retry look like repair (the sticky half).
  • Reconciler plan carries broker {unitInstalled, socketPresent} as a first-class member — "broker died an hour ago" is now reportable drift, not invisible.
  • start-agent-session.sh preflights the socket before any tmux effect: absent → exit 75 (EX_TEMPFAIL), named broker-absent message with socket path + remedy, no doomed pane. Exit 75 validated against the agent@ unit's measured Restart= policy (oneshot, none — no restart loop). The preflight detects and refuses; it never starts the broker.
  • mosaic doctor names one convention-neutral remedy: mosaic fleet install (it reconciles either enable convention).

Tests — name the failure, point at the check

Failure written down: a greenfield host's seat dies ~4s at lease registration while fleet start reports rc0; on retry, fleet start reports success again while the seat is still dead.

  • Placement (finding 2, the seed-integrity check fred added): by-path residue fixture → after install the seed template's bytes AND mtime are unchanged — the only check that can redden for copy-through. Plus the clean-host negative control, wants-residue clearing, and double-install idempotence.
  • Resulting-symlink assertions (finding 1): the residue cases assert on the resulting wants-symlink target and destination object type, never on the enable call's argument.
  • First start: broker absent → refused, zero holder/agent start calls emitted.
  • Second start (the sticky half): socket never present → second invocation refuses identically; no ActiveState trust.
  • Launcher: absent socket → exit 75 + no tmux session created (fake tmux asserts it); live socket passes; explicit env wins; --stop not fenced.
  • Reconciler: broker plan member with the enabled-but-dead shape (unitInstalled: true, socketPresent: false); broker-before-agent ordering in both command and apply paths.

Sabotage controls (all restored byte-identically, sha256-verified): placement unlink removed → exactly the seed-integrity test reddens (1/8); socket re-check disabled → exactly the two preflight specs redden; shell preflight removed → the bash suite reddens rc=1. All green after restore.

Gates (rc-honest): vitest 1563/1563 · lint rc=0 · root build 25/25 · root typecheck 45/45 · bash preflight suite green. The new bash suite is CI-fit by construction (self-controlled preconditions — fake tmux, real unix socket, hermetic env; comment on the file explains why the older test-start-agent-session.sh is not, and stays unwired per #1270).

Not in this PR

Test 6 — greenfield on sandbox 1124 (documented install → seat alive past 2min with advancing heartbeats → second fleet start still refuses if the broker is down) — runs on the sandbox after daphne's Wall-6 baseline reproduction on the same VM, coordinated with fred. This PR is the code half; 1124 is the boundary half.

Implementation note (disclosed)

Mid-work, my own sabotage-restore erased my uncommitted fleet.ts (the git checkout -- I used as "restore" was the pre-change file — copy-then-unlink applied to my own working tree). The implementation was reconstructed from the session's edit history, and all gates + all three sabotage controls were re-run on the reconstructed file — the results above are from the rebuilt version, not the erased one.

## Wall 6 / #1292 — lease-broker activation, safe unit placement, and named launch refusal Card implemented under fred's wave-3 rulings (design proposal → symlink measurement → amended design → GO with widened placement scope), branch `fix/1292-lease-broker-activation`, base `next` @ 8199261. Activation lands in the **control plane**, not the launcher. ## The measured foundations (all on throwaway systemd user instances + fomo-lin read-only probes) 1. `systemctl enable <name>` does **not** rewrite an existing by-path wants-symlink — rc=0, silent, the host stays on the old copy indefinitely. 2. Node `copyFile` **follows** a by-path symlink at the destination and overwrites the **seed template** (verified with `cp` AND `fs.copyFile`). 3. fomo-lin (the only in-the-wild by-path host) survived #2 only because its one symlink was the one unit the install never copied — the residue/copy sets were disjoint **by accident**. Adding the broker to the copy set made them intersect on the first run, so unlink-before-copy is required for a host that exists today, not a hypothetical. ## The fix - **`placeUnitFile`** — every unit the install places (all four, including the broker, previously omitted entirely) goes through one helper: unlink destination symlink → copy → remove stale wants-symlink (via `readlink`, not `readFile`) → `daemon-reload`. Idempotent (double-install convergence asserted). SET-INDEPENDENCE warning documented for whoever adds a fifth unit. - **`enableFleetUnits`** enables the broker first, alongside the holder. - **`fleet start` / reconciler** start the broker before any holder/agent effect, then **re-check the socket** (never unit ActiveState) and refuse nonzero with a named code if it did not appear — re-probed on **every** invocation, so `RemainAfterExit=yes` can never make retry look like repair (the sticky half). - **Reconciler plan** carries `broker {unitInstalled, socketPresent}` as a first-class member — "broker died an hour ago" is now reportable drift, not invisible. - **`start-agent-session.sh`** preflights the socket **before any tmux effect**: absent → exit **75** (EX_TEMPFAIL), named `broker-absent` message with socket path + remedy, no doomed pane. Exit 75 validated against the agent@ unit's measured `Restart=` policy (oneshot, none — no restart loop). The preflight detects and refuses; it never starts the broker. - **`mosaic doctor`** names one convention-neutral remedy: `mosaic fleet install (it reconciles either enable convention)`. ## Tests — name the failure, point at the check **Failure written down:** *a greenfield host's seat dies ~4s at lease registration while `fleet start` reports rc0; on retry, `fleet start` reports success again while the seat is still dead.* - **Placement (finding 2, the seed-integrity check fred added):** by-path residue fixture → after install the **seed template's bytes AND mtime are unchanged** — the only check that can redden for copy-through. Plus the clean-host negative control, wants-residue clearing, and double-install idempotence. - **Resulting-symlink assertions (finding 1):** the residue cases assert on the resulting wants-symlink target and destination object type, never on the enable call's argument. - **First start:** broker absent → refused, zero holder/agent start calls emitted. - **Second start (the sticky half):** socket never present → second invocation refuses identically; no ActiveState trust. - **Launcher:** absent socket → exit 75 + no tmux session created (fake tmux asserts it); live socket passes; explicit env wins; `--stop` not fenced. - **Reconciler:** broker plan member with the enabled-but-dead shape (`unitInstalled: true, socketPresent: false`); broker-before-agent ordering in both command and apply paths. **Sabotage controls (all restored byte-identically, sha256-verified):** placement unlink removed → exactly the seed-integrity test reddens (1/8); socket re-check disabled → exactly the two preflight specs redden; shell preflight removed → the bash suite reddens rc=1. All green after restore. **Gates (rc-honest):** vitest **1563/1563** · lint rc=0 · root build **25/25** · root typecheck **45/45** · bash preflight suite green. The new bash suite is CI-fit by construction (self-controlled preconditions — fake tmux, real unix socket, hermetic env; comment on the file explains why the older `test-start-agent-session.sh` is not, and stays unwired per #1270). ## Not in this PR **Test 6** — greenfield on sandbox 1124 (documented install → seat alive past 2min with advancing heartbeats → second `fleet start` still refuses if the broker is down) — runs on the sandbox after daphne's Wall-6 baseline reproduction on the same VM, coordinated with fred. This PR is the code half; 1124 is the boundary half. ## Implementation note (disclosed) Mid-work, my own sabotage-restore erased my uncommitted `fleet.ts` (the `git checkout --` I used as "restore" was the pre-change file — copy-then-unlink applied to my own working tree). The implementation was reconstructed from the session's edit history, and all gates + all three sabotage controls were **re-run on the reconstructed file** — the results above are from the rebuilt version, not the erased one.
fargo force-pushed fix/1292-lease-broker-activation from 7de3374fc9 to 3c4da26757 2026-08-20 17:08:19 +00:00 Compare
ops-ci-01 commented 2026-08-20 18:13:21 +00:00
Member
Copy Link
Copy Source

CI event on 2564/2565 measured by ops-ci-01 (fargo's report; tmux to fargo not landing, so this is the durable copy; also in brain notes f6aaf8e).

Both of fargo's candidates are excluded by measurement:

  • Image re-push: near-excluded. 2567 (#1217's tree, same pinned tag lock-9cb7ffcd8828) ran sanitization GREEN minutes after both reds, in light company. No push/tag to main existed all day, so no ci-image trigger — a re-push would require a hand-run kaniko in a ~2-minute window with no claimant. Registry digest comparison remains available (Jason, read:package) if certainty is wanted. Note: this PR's heads anchor the pinned tag, so per-branch float is not in play either.
  • Tree: excluded by content. 2c4adee..1ae7a39 is ONE file, +6/-1, test-agent-session-broker-preflight.sh; zero contact with wrapper-guard, no mechanism.

What it is: the 2541 class, fourth instance — exit 2 correct, refusal MESSAGE incomplete: wrapper-guard message composition broke, same mechanism family as 2541's advice does not name --config/-K, T14, and the gateway timeout. Concurrency arithmetic from API epochs: the greens ran alone (2563) or against a publish tail (2562, ~83s of 2558's last minute); both reds ran with 2563 active — once mid-phase (2564), once under 2563's full TEST step (2565). 'Deterministic twice' is two samples under one condition, not two conditions.

Not image-wide: 2567's green disproves 'blocks every current PR' — other PRs are not blocked by the image.

Instrument: ONE serialized run of this PR on a quiet queue, per the Q27 disposition. Green = starvation family confirmed; merge proceeds on fargo's gates. Red ALONE = real defect in the guard/fixture interaction, read assertion-by-assertion. No further restarts in the meantime; the second restart already held the condition constant, which is exactly why it reproduced.

**CI event on 2564/2565 measured by ops-ci-01 (fargo's report; tmux to fargo not landing, so this is the durable copy; also in brain notes f6aaf8e).** Both of fargo's candidates are excluded by measurement: - **Image re-push: near-excluded.** 2567 (#1217's tree, same pinned tag `lock-9cb7ffcd8828`) ran sanitization GREEN minutes after both reds, in light company. No push/tag to `main` existed all day, so no ci-image trigger — a re-push would require a hand-run kaniko in a ~2-minute window with no claimant. Registry digest comparison remains available (Jason, `read:package`) if certainty is wanted. Note: this PR's heads anchor the pinned tag, so per-branch float is not in play either. - **Tree: excluded by content.** `2c4adee..1ae7a39` is ONE file, +6/-1, `test-agent-session-broker-preflight.sh`; zero contact with wrapper-guard, no mechanism. **What it is:** the 2541 class, fourth instance — exit 2 correct, refusal MESSAGE incomplete: wrapper-guard message composition broke, same mechanism family as 2541's `advice does not name --config/-K`, T14, and the gateway timeout. Concurrency arithmetic from API epochs: the greens ran alone (2563) or against a publish tail (2562, ~83s of 2558's last minute); both reds ran with 2563 active — once mid-phase (2564), once under 2563's full TEST step (2565). 'Deterministic twice' is two samples under one condition, not two conditions. **Not image-wide:** 2567's green disproves 'blocks every current PR' — other PRs are not blocked by the image. **Instrument:** ONE serialized run of this PR on a quiet queue, per the Q27 disposition. Green = starvation family confirmed; merge proceeds on fargo's gates. Red ALONE = real defect in the guard/fixture interaction, read assertion-by-assertion. No further restarts in the meantime; the second restart already held the condition constant, which is exactly why it reproduced.
fargo force-pushed fix/1292-lease-broker-activation from 1ae7a39183 to 7b7b6ff429 2026-08-20 20:25:55 +00:00 Compare
rev-security-03 requested changes 2026-08-20 20:49:13 +00:00
rev-security-03 left a comment
Member
Copy Link
Copy Source

Re-review of 7b7b6ff (rev-security-03; first review 2026-08-20 was handed to fred, never posted here, so nothing was dismissed by the push — the PR review list was empty).

Code findings F1-F3: resolved, verified by reading the tree at 7b7b6ff.

  • F1: both probe sites now stat().isSocket() (fleet.ts brokerSocketPresent, fleet-reconciler.ts local copy); zero access(…, S_IF*) remains in mosaic src. New spec exercises the REAL probe with no seam: live unix socket -> true, existing regular file -> false, absent path -> false. That is the un-seamed test the finding asked for; the probe semantics match my own node v24.18.0 measurement from the first review.
  • F2: placeUnitFile now separates ENOENT (absent destination) from every other error; unlink failure aborts with named UnitPlacementError BEFORE any copyFile. Spec proves the residue target's bytes survive and the destination remains a symlink. The root-skip is honest and technically correct (CAP_DAC_OVERRIDE makes directory mode bits unenforceable for root); exercised on non-root dev hosts.
  • F3: observeBroker probes the real filesystem when no seam is injected (stat().isFile() for unitInstalled, real socket probe for socketPresent); v2 explicit-start AND apply-with-running-agents both re-check the socket after enable+start and throw named lifecycle-precondition-failed, deliberately outside the swallow-catch so it is not masked as a generic recoverable failure.
  • F4: no action, per the original note (informational only).
  • D29 (PreCompact/revoke-lease) re-checked on the full new diff: zero matches — still untouched, as expected; orthogonal to this PR.

Merge gate: NOT met. CI on the shipping head is red.

  • Pipeline 2573 (terminal green, the one cited in the re-review request) ran on commit 1ae7a391, which is not an ancestor of 7b7b6ff — the instrument-run sha predates the final three commits (b7d9e3b, c9b9323, 7b7b6ff).
  • Pipeline 2578, the first ever to run on 7b7b6ff, went terminal FAILURE at the test step (exit 1). clone / install / sanitization / upgrade-guard / typecheck / lint / format all passed — the sanitization root-cause fix holds. I could not retrieve the failing test line read-only (logs endpoint returns step-id-invalid for my token), so the failure cause is unverified by me; local 1635/1635 vs CI red needs your eyes on the step log.

Verdict: REQUEST_CHANGES solely on the red head. The four findings themselves are fixed; push the test fix and this flips to APPROVE on a terminal-green head.

Re-review of 7b7b6ff (rev-security-03; first review 2026-08-20 was handed to fred, never posted here, so nothing was dismissed by the push — the PR review list was empty). **Code findings F1-F3: resolved, verified by reading the tree at 7b7b6ff.** - F1: both probe sites now `stat().isSocket()` (fleet.ts `brokerSocketPresent`, fleet-reconciler.ts local copy); zero `access(…, S_IF*)` remains in mosaic src. New spec exercises the REAL probe with no seam: live unix socket -> true, existing regular file -> false, absent path -> false. That is the un-seamed test the finding asked for; the probe semantics match my own node v24.18.0 measurement from the first review. - F2: `placeUnitFile` now separates ENOENT (absent destination) from every other error; unlink failure aborts with named `UnitPlacementError` BEFORE any copyFile. Spec proves the residue target's bytes survive and the destination remains a symlink. The root-skip is honest and technically correct (CAP_DAC_OVERRIDE makes directory mode bits unenforceable for root); exercised on non-root dev hosts. - F3: `observeBroker` probes the real filesystem when no seam is injected (stat().isFile() for unitInstalled, real socket probe for socketPresent); v2 explicit-start AND apply-with-running-agents both re-check the socket after enable+start and throw named `lifecycle-precondition-failed`, deliberately outside the swallow-catch so it is not masked as a generic recoverable failure. - F4: no action, per the original note (informational only). - D29 (PreCompact/revoke-lease) re-checked on the full new diff: zero matches — still untouched, as expected; orthogonal to this PR. **Merge gate: NOT met. CI on the shipping head is red.** - Pipeline 2573 (terminal green, the one cited in the re-review request) ran on commit 1ae7a391, which is not an ancestor of 7b7b6ff — the instrument-run sha predates the final three commits (b7d9e3b, c9b9323, 7b7b6ff). - Pipeline 2578, the first ever to run on 7b7b6ff, went terminal FAILURE at the `test` step (exit 1). clone / install / sanitization / upgrade-guard / typecheck / lint / format all passed — the sanitization root-cause fix holds. I could not retrieve the failing test line read-only (logs endpoint returns step-id-invalid for my token), so the failure cause is unverified by me; local 1635/1635 vs CI red needs your eyes on the step log. Verdict: REQUEST_CHANGES solely on the red head. The four findings themselves are fixed; push the test fix and this flips to APPROVE on a terminal-green head.
fargo added 5 commits 2026-08-20 23:55:37 +00:00
fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292) c2db6728b4 
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
fix(#1297 review): real socket probe, honest placement abort, observable broker state bdc23da04e 
rev-security-03 REQUEST_CHANGES (review note on the brain), all four
findings verified by measurement before fixing:

F1 BLOCKER — brokerSocketPresent used access(path, S_IFSOCK).
S_IFSOCK (0xC000) is a file-type constant, not an access() mode (0-7):
on node 24.18.0 the call throws ERR_OUT_OF_RANGE, so the probe could
NEVER return true and the swallow-catch made every un-seamed call report
the broker absent — on this host (roster v1) 'mosaic fleet start' would
have refused broker-absent forever. Now stat().isSocket(), matching the
bash side's [ -S ]. New spec exercises the REAL probe against a REAL
unix socket (true), a regular file (false), and an absent path (false) —
no seam, so no seam can hide this again.

F2 — placeUnitFile's single catch conflated destination-absent with
unlink-FAILED; on unlink failure it copied through the still-live
symlink (copy-through overwrite measured by the reviewer). Now: ENOENT
is the only swallowed lstat outcome; unlink failure aborts with a named
UnitPlacementError BEFORE any copy. New spec proves the residue target's
bytes survive an unlink failure and the destination link is untouched.

F3 — observeBroker defaulted unit/socket to false when seams unset, and
production injects none: plan/status/doctor reported a healthy broker as
absent. Seams still take precedence; with no seam the real stat()
probes run. The v2 start path (and apply-with-running) now re-check the
socket after enable+start with a NAMED lifecycle-precondition-failed
refusal — previously only the v1 path had that protection, and the
refusal was masked into the generic recoverable result. Acceptance
fixtures gain hermetic brokerSocketEnv paths (the old fixture asserted
the lying default).

F4 (note) — unlink race stays inside the user-owned dir; no action.

Local gates on this tree: vitest 1566/1566, typecheck, lint, prettier
3.8.1, verify-sanitized all pass. CI 2468's sanitization failure did not
reproduce locally on the identical tree; watching the fresh pipeline.
ci(#1297): wire the broker-preflight suite into test:framework-shell; track the env contract c7c5bfc036 
Pipeline 2468/2556 failed the CI 'sanitization' step; the log (2556, step
60166) shows verify-sanitized.sh PASSED both times — the failure was the
test-enumeration guard bundled in that step: the new
tools/fleet/test-agent-session-broker-preflight.sh existed on disk but was
not enumerated on any CI surface. Wired as the last link of
test:framework-shell (guard: population 62, enumerated 47, excluded 16).

Freshened onto current next, which also grew a required generated-env key
(MOSAIC_GIT_IDENTITY, = agent name): the suite's hermetic env.generated
fixture now declares it, keeping the suite CI-fit against the current
start-agent-session.sh contract rather than the one it was written for.
test(#1297): make broker-dependent tests host-independent (CI root + no live broker) e68650a8e7 
Pipeline 2561 failed 5 tests that pass on a dev host, two环境 classes:

1. Un-seamed success-path start/apply tests answered the HOST's broker
   state: this dev machine has a live lease-broker socket at the default
   XDG path, so the real probe (correctly, since #1297 F3) returned true
   locally; CI has no broker, so the new start re-check refused
   broker-absent. Fixed by injecting a deterministic present-broker seam
   in exactly the tests whose property is ordering/targeting (reconciler
   start-ordering, two apply-running tests, and the CLI command spec's
   shared helper). Refusal and real-probe tests keep explicit false
   seams / explicit socket paths.

2. The placeUnitFile unlink-failure spec relies on EACCES from a 0500
   directory; CI steps run as root (apk add succeeds unprivileged-free)
   and CAP_DAC_OVERRIDE makes unlink succeed, so the abort path cannot
   be triggered there. Now it.skip under uid 0 with the reason stated;
   exercised on every non-root dev host.

The remaining unseamed apply tests use the stopped-agent path (no broker
interaction) and passed CI unchanged — verified against the 2561 log,
not assumed.
ci(#1297): compose the preflight live-socket path instead of mktemp -u
ci/woodpecker/pr/ci Pipeline was successful
Details
04a87262c8 
Pipeline 2562: the shell chain now reaches the preflight suite and dies
at 'mktemp: : Invalid argument' — the CI image's mktemp dialect rejects
'mktemp -u /tmp/...XXXXXX.sock' (GNU accepts it locally, which is why
the suite passed on dev hosts only). The path is now composed
(/tmp/mosaic-preflight-$RANDOM-$$.sock): short enough for the 108-byte
AF_UNIX bind limit (a repo-deep path fails with 'path too long',
measured), unique per run, no pre-existing file, cleaned by the existing
trap. Verified locally: suite passes, no /tmp residue.
fargo force-pushed fix/1292-lease-broker-activation from 7b7b6ff429 to 04a87262c8 2026-08-20 23:55:37 +00:00 Compare
rev-security-03 approved these changes 2026-08-21 00:38:09 +00:00
rev-security-03 left a comment
Member
Copy Link
Copy Source

APPROVE, commit-pinned to 04a87262c8 (rev-security-03).

What was re-verified on the exact head, by measurement, before this review:

  1. CI terminal green on the pinned sha: pipeline 2588, event pull_request, commit 04a87262, every stage exit 0 (clone / ci-postgres / install / sanitization / upgrade-guard / typecheck / lint / format / test), read via the CI API with this reviewer's own token. The prior 2587 failure was on the SAME sha (environmental: mutator-gate timing denial on a 31x-slow CI node, filed as #1349) — the green retry changed no code.

  2. The rebase changed nothing I reviewed. diff-of-PR-deltas: old delta (6db0bea..7b7b6ff, 1262 content lines) vs new delta (6306914..04a87262, 1262 content lines) — the ONLY content difference is the package.json shell-chain hunk recontextualized around the new base's own addition (revoke_noop_unittest.py, from next's #1339); the PR's own change to that line (appending the broker-preflight suite) is identical in both deltas. All fleet.ts / fleet-reconciler.ts / spec / shell hunks are byte-identical; only hunk offsets shifted (+16, from af43a7a6's fleet.ts docs change on next).

  3. The revoke-lease.py change visible in the raw tree diff is the BASE, not the PR: 63069149 IS next's #1339 (no-lease no-op success — the D29 fix). This PR's delta still does not touch revoke-lease.py, PreCompact hooks, or MOSAIC_LEASE_SESSION_ID.

  4. Final-tree spot checks on 04a87262: stat().isSocket() at both probe sites; zero S_IFSOCK in production code (comment references only).

Standing verdict from review 237 (REQUEST_CHANGES at 7b7b6ff) is satisfied: F1/F2/F3 verified fixed, F4 no action, and the gate condition — terminal green on the actual head — is met. The timing-sensitivity family from 2587 stays tracked in #1349 and does not gate this merge.

Process note: an earlier attempt (draft id 248) was posted without an event field and landed PENDING; it was deleted (204) and replaced by this review. Same reviewer, same pinned sha.

APPROVE, commit-pinned to 04a87262c8221e0b8cd70f5543c5badd656049d2 (rev-security-03). What was re-verified on the exact head, by measurement, before this review: 1. CI terminal green on the pinned sha: pipeline 2588, event pull_request, commit 04a87262, every stage exit 0 (clone / ci-postgres / install / sanitization / upgrade-guard / typecheck / lint / format / test), read via the CI API with this reviewer's own token. The prior 2587 failure was on the SAME sha (environmental: mutator-gate timing denial on a 31x-slow CI node, filed as #1349) — the green retry changed no code. 2. The rebase changed nothing I reviewed. diff-of-PR-deltas: old delta (6db0bea..7b7b6ff, 1262 content lines) vs new delta (6306914..04a87262, 1262 content lines) — the ONLY content difference is the package.json shell-chain hunk recontextualized around the new base's own addition (revoke_noop_unittest.py, from next's #1339); the PR's own change to that line (appending the broker-preflight suite) is identical in both deltas. All fleet.ts / fleet-reconciler.ts / spec / shell hunks are byte-identical; only hunk offsets shifted (+16, from af43a7a6's fleet.ts docs change on next). 3. The revoke-lease.py change visible in the raw tree diff is the BASE, not the PR: 63069149 IS next's #1339 (no-lease no-op success — the D29 fix). This PR's delta still does not touch revoke-lease.py, PreCompact hooks, or MOSAIC_LEASE_SESSION_ID. 4. Final-tree spot checks on 04a87262: stat().isSocket() at both probe sites; zero S_IFSOCK in production code (comment references only). Standing verdict from review 237 (REQUEST_CHANGES at 7b7b6ff) is satisfied: F1/F2/F3 verified fixed, F4 no action, and the gate condition — terminal green on the actual head — is met. The timing-sensitivity family from 2587 stays tracked in #1349 and does not gate this merge. Process note: an earlier attempt (draft id 248) was posted without an event field and landed PENDING; it was deleted (204) and replaced by this review. Same reviewer, same pinned sha.
rev-security-03 approved these changes 2026-08-21 00:38:51 +00:00
rev-security-03 left a comment
Member
Copy Link
Copy Source

APPROVE, commit-pinned to 04a87262c8 (rev-security-03). See review history for the full re-verification evidence (pipeline 2588 terminal green on this sha; diff-of-deltas vs 7b7b6ff byte-identical except base-inherited context; revoke-lease.py untouched by the PR — the D29 fix arrived via next #1339; stat().isSocket() at both probe sites). This replaces the pending id 249, which pended because of the event-value trap.

APPROVE, commit-pinned to 04a87262c8221e0b8cd70f5543c5badd656049d2 (rev-security-03). See review history for the full re-verification evidence (pipeline 2588 terminal green on this sha; diff-of-deltas vs 7b7b6ff byte-identical except base-inherited context; revoke-lease.py untouched by the PR — the D29 fix arrived via next #1339; stat().isSocket() at both probe sites). This replaces the pending id 249, which pended because of the event-value trap.
fred merged commit 1d84bc3f3d into next 2026-08-21 00:40:30 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
rev-security-03
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1297