mosaicstack/stack
21
0
Fork 1
Code Issues 340 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292)

Browse Source 
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
This commit is contained in:
fargo
2026-08-20 18:53:43 -05:00
parent 6306914965
commit c2db6728b4
9 changed files with 936 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/mosaic/src/commands/fleet.spec.ts
+110 -1
View File
@@ -836,13 +836,25 @@ describe('fleet command construction', () => {
};
const program = new Command();
program.exitOverride();
registerFleetCommand(program, { runner, mosaicHome: home });
// #1292: inject a present broker socket so the preflight passes and this
// spec keeps testing its ORIGINAL property (holder-before-agent ordering).
// The preflight's own refusal behavior has dedicated specs below.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => true,
});
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
expect(calls).toEqual([
// #1292: fleet start enables + starts the broker FIRST (enable is
// idempotent; the unit exists after install), re-checking the socket
// before any holder/agent lifecycle effect.
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
['systemctl', '--user', 'start', '[email protected]'],
['systemctl', '--user', 'stop', '[email protected]'],
@@ -853,6 +865,92 @@ describe('fleet command construction', () => {
}
});
it('fleet start refuses with a named error when the broker socket does not appear (#1292)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Refused: no holder/agent starts were issued after the broker attempt.
expect(calls).toEqual([
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
]);
expect(errors.join('\n')).toContain('broker-absent');
expect(errors.join('\n')).toContain('mosaic fleet install');
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('fleet start re-probes the broker on the SECOND invocation — no ActiveState trust (#1292 sticky half)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
// Broker socket NEVER appears — the second start must refuse exactly like
// the first; RemainAfterExit-style stale unit state changes nothing
// because the check is the socket, not systemctl.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Two invocations, each refusing after its own broker attempt:
expect(
calls.filter((c) => c.join(' ') === 'systemctl --user start [email protected]'),
).toHaveLength(0);
expect(errors.filter((e) => e.includes('broker-absent')).length).toBeGreaterThanOrEqual(2);
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
@@ -2066,8 +2164,19 @@ describe('fleet install — auto-enable units for boot-survival', () => {
await enableFleetUnits(runner, minimalRoster, {});
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-lease-broker.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
// The broker must be enabled BEFORE the holder and agents: a start of any
// gated runtime without the broker is exactly the #1292 4-second death.
const brokerIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
);
const holderIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-tmux-holder.service',
);
expect(brokerIndex).toBeGreaterThanOrEqual(0);
expect(brokerIndex).toBeLessThan(holderIndex);
});
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {