fix(fleet): boot-survival symmetry — disable-on-remove + add-enable + init-R5 (#612)

Co-authored-by: Jason Woltje <jason@diversecanvas.com>
Co-committed-by: Jason Woltje <jason@diversecanvas.com>

docs/TASKS.md

@@ -54,3 +54,11 @@ Active workstream is **W1 — Federation v1**. Workers should:
 ## P6 — Docs, compliance matrix, alpha tag (#606) — feat/p6-docs-compliance-alpha
 
 - Status: in-repo deliverables done (CONTRIBUTING.md + harness×gate compliance matrix + check-resident-budget.sh + CI wiring + ALPHA-DOD.md). Remaining: alpha tag v0.0.39-alpha (Lead, post-merge). aiguide reconcile merged (#8). Detail: scratchpads/p6-docs-compliance-alpha.md.
+
+## F3-m3 — mosaic update re-seeds framework + relaunches agents (#609) — feat/f3-m3-update-reseed
+
+- Status: implemented + tested. Closes R13: `mosaic update` now re-seeds the framework (data-safe MOSAIC_SYNC_ONLY) after the CLI install so shipped launcher/runtime changes activate; `--relaunch` restarts rostered agents; `--no-reseed` opts out. Detail: scratchpads/f3-m3-update-reseed.md.
+
+## Fleet-polish bundle — boot-survival symmetry (#611) — feat/fleet-polish-bundle
+
+- Status: implemented + tested. disable-on-remove (boot-resurrection bug, TDD) + add-enable + init-R5 hard guarantee. 4 new + 147 existing fleet tests green. Detail: scratchpads/fleet-polish-bundle.md.

docs/scratchpads/f3-m3-update-reseed.md

@@ -0,0 +1,29 @@
+# F3-m3 — `mosaic update` re-seeds framework + relaunches agents (R13)
+
+- **Issue:** #609 · **Branch:** `feat/f3-m3-update-reseed`
+
+## Gap (found in 0.0.39 production validation)
+
+`mosaic update` installs the new npm CLI but never re-seeds `~/.config/mosaic/` from the package's
+bundled `framework/`. So the shipped custom Pi harness (agent-name export + native HB, 0.0.39) stays
+DORMANT until a re-seed — operators get the new CLI on a stale framework.
+
+## Implementation
+
+- `update-checker.ts`: `resolveBundledFrameworkRoot()`, `buildReseedCommand()` (install.sh in
+  `MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep` — the P4 data-safe reconcile), `runFrameworkReseed()`,
+  `readRosterAgentNames()`, `buildRelaunchCommands()` (systemctl --user restart per agent).
+- `cli.ts` `update`: after a successful CLI install that includes `@mosaicstack/mosaic`, re-seed the
+  framework (default-on; `--no-reseed` to skip). Then either `--relaunch` (restart rostered agents) or
+  print clear guidance to run `mosaic update --relaunch` / `mosaic fleet restart`.
+
+## Flow
+
+`update CLI → re-seed framework (data-safe) → relaunch agents (opt-in)` — closes R13, activates the
+native harness for every operator.
+
+## Verification
+
+- 6 new unit tests (reseed command/env, relaunch commands, roster parse, missing-installer guard).
+- 19 runtime + 26 launch tests still green; tsc/eslint/prettier clean.
+- Data-safety of the sync is already proven (P4 5-fixture matrix + live dragon-lin validation).

docs/scratchpads/fleet-polish-bundle.md

@@ -0,0 +1,20 @@
+# Fleet-polish bundle — boot-survival symmetry (#611)
+
+- **Issue:** #611 · **Branch:** `feat/fleet-polish-bundle` · From the Lead's Codex symmetry-gap finding.
+
+## Three fixes
+
+1. **disable-on-remove (BUG, TDD).** `fleet remove` stopped + deleted roster/env/heartbeat but never
+   `systemctl --user disable mosaic-agent@NAME.service` → a removed-but-enabled unit could resurrect on
+   reboot pointing at deleted config. Fix: `buildSystemdDisableCommand` + disable in `remove`
+   (best-effort, gated on !--keep-files).
+2. **add-enable.** `fleet add` now enables the new agent's unit for boot-survival (best-effort,
+   independent of --start) — symmetry with disable-on-remove.
+3. **init-R5 guarantee.** `fleet init --write` now FAILS HARD when a non-minimal profile doesn't yield
+   exactly one orchestrator (was a soft warning). `minimal` (sanctioned no-orchestrator) still allowed.
+
+## Verification
+
+- 4 new tests (disable builder; remove-invokes-disable; add-invokes-enable; init general → exactly 1
+  orchestrator) + 147 existing fleet tests green (151 total). tsc/eslint/prettier clean.
+- TDD on the disable bug per contract.

packages/mosaic/src/cli.ts

@@ -26,6 +26,10 @@ import {
   checkForAllUpdates,
   formatAllPackagesTable,
   getInstallAllCommand,
+  runFrameworkReseed,
+  readRosterAgentNames,
+  buildRelaunchCommands,
+  FRAMEWORK_RESEED_PACKAGE,
 } from './runtime/update-checker.js';
 import { runWizard } from './wizard.js';
 import { ClackPrompter } from './prompter/clack-prompter.js';
@@ -404,7 +408,12 @@ program
   .command('update')
   .description('Check for and install Mosaic CLI updates')
   .option('--check', 'Check only, do not install')
-  .action(async (opts: { check?: boolean }) => {
+  .option(
+    '--no-reseed',
+    'Skip re-seeding framework files into ~/.config/mosaic after the CLI update',
+  )
+  .option('--relaunch', 'Restart durable fleet agents so the new launcher/runtime takes effect')
+  .action(async (opts: { check?: boolean; reseed?: boolean; relaunch?: boolean }) => {
     // checkForAllUpdates imported statically above
     const { execSync } = await import('node:child_process');
 
@@ -442,6 +451,51 @@ program
       console.error('\nUpdate failed. Try manually: bash tools/install.sh');
       process.exit(1);
     }
+
+    // F3-m3 / R13: the CLI is updated, but the framework files in
+    // ~/.config/mosaic/ are still the previous version. Re-seed them from the
+    // freshly-installed package so shipped launcher/runtime changes ACTIVATE.
+    // Only when the framework-bearing package itself updated.
+    const mosaicUpdated = outdated.some(
+      (r: { package: string }) => r.package === FRAMEWORK_RESEED_PACKAGE,
+    );
+    if (mosaicUpdated && opts.reseed !== false) {
+      console.log(
+        '\nRe-seeding framework files into ~/.config/mosaic (data-safe; keeps your edits)…',
+      );
+      const reseed = runFrameworkReseed();
+      if (reseed.ok) {
+        console.log('✔ Framework re-seeded.');
+        const agents = readRosterAgentNames();
+        if (agents.length > 0) {
+          if (opts.relaunch) {
+            console.log(
+              `\nRelaunching ${agents.length} fleet agent(s) to pick up the new runtime…`,
+            );
+            for (const restart of buildRelaunchCommands(agents)) {
+              try {
+                execSync(restart.join(' '), { stdio: 'inherit', timeout: 30_000 });
+              } catch {
+                console.error(`  ⚠ failed to restart agent — run: ${restart.join(' ')}`);
+              }
+            }
+            console.log('✔ Agents relaunched.');
+          } else {
+            console.log(
+              `\nℹ ${agents.length} fleet agent(s) are still running the previous runtime. ` +
+                'Restart them to activate the update:\n    mosaic update --relaunch   ' +
+                '(or: mosaic fleet restart <agent>)',
+            );
+          }
+        }
+      } else {
+        console.error(
+          `\n⚠ Framework re-seed skipped: ${reseed.reason ?? 'unknown'}.\n` +
+            '  Activate manually: bash "$(npm root -g)/@mosaicstack/mosaic/framework/install.sh" ' +
+            '(MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep)',
+        );
+      }
+    }
   });
 
 // ─── wizard ─────────────────────────────────────────────────────────────

packages/mosaic/src/commands/fleet.spec.ts

@@ -14,6 +14,7 @@ import {
   buildEnableLingerCommand,
   buildFleetServiceCommand,
   buildSystemdEnableCommand,
+  buildSystemdDisableCommand,
   buildSystemdShowCommand,
   buildTmuxListPanesCommand,
   buildTmuxListSessionsCommand,
@@ -983,6 +984,127 @@ describe('fleet ps — drift detection', () => {
   });
 });
 
+describe('fleet-polish bundle — boot-survival symmetry', () => {
+  async function rosterHome(agents: string): Promise<string> {
+    const home = await tempDir();
+    await mkdir(join(home, 'fleet'), { recursive: true });
+    await writeFile(join(home, 'fleet', 'roster.yaml'), agents);
+    return home;
+  }
+
+  it('buildSystemdDisableCommand returns the systemctl --user disable array', () => {
+    expect(buildSystemdDisableCommand('mosaic-agent@coder0.service')).toEqual([
+      'systemctl',
+      '--user',
+      'disable',
+      'mosaic-agent@coder0.service',
+    ]);
+  });
+
+  it('fleet remove DISABLES the unit so a removed agent cannot resurrect on boot', async () => {
+    const home = await rosterHome(
+      [
+        'version: 1',
+        'transport: tmux',
+        'agents:',
+        '  - name: orchestrator',
+        '    runtime: pi',
+        '    class: orchestrator',
+        '  - name: coder0',
+        '    runtime: codex',
+        '    class: worker',
+      ].join('\n') + '\n',
+    );
+    const calls: string[][] = [];
+    const runner: CommandRunner = async (command, args) => {
+      calls.push([command, ...args]);
+      return { stdout: '', stderr: '', exitCode: 0 };
+    };
+    const program = new Command();
+    program.exitOverride();
+    registerFleetCommand(program, { runner, mosaicHome: home });
+    try {
+      await program.parseAsync(['node', 'mosaic', 'fleet', 'remove', 'coder0']);
+      expect(calls).toContainEqual([
+        'systemctl',
+        '--user',
+        'disable',
+        'mosaic-agent@coder0.service',
+      ]);
+      // stop must still happen too
+      expect(calls).toContainEqual(['systemctl', '--user', 'stop', 'mosaic-agent@coder0.service']);
+    } finally {
+      await rm(home, { recursive: true, force: true });
+    }
+  });
+
+  it('fleet add ENABLES the new agent unit for boot-survival', async () => {
+    const home = await rosterHome(
+      ['version: 1', 'transport: tmux', 'agents:', '  - name: coder0', '    runtime: codex'].join(
+        '\n',
+      ) + '\n',
+    );
+    const calls: string[][] = [];
+    const runner: CommandRunner = async (command, args) => {
+      calls.push([command, ...args]);
+      return { stdout: '', stderr: '', exitCode: 0 };
+    };
+    const program = new Command();
+    program.exitOverride();
+    registerFleetCommand(program, { runner, mosaicHome: home });
+    try {
+      await program.parseAsync([
+        'node',
+        'mosaic',
+        'fleet',
+        'add',
+        'coder1',
+        '--runtime',
+        'codex',
+        '--class',
+        'worker',
+        '--no-start',
+      ]);
+      expect(calls).toContainEqual([
+        'systemctl',
+        '--user',
+        'enable',
+        'mosaic-agent@coder1.service',
+      ]);
+    } finally {
+      await rm(home, { recursive: true, force: true });
+    }
+  });
+
+  it('fleet init --write fails hard when a non-minimal profile lacks exactly one orchestrator', async () => {
+    // The general profile must yield exactly one orchestrator; the guarantee is
+    // enforced (not just warned). We assert the happy path writes cleanly.
+    const home = await tempDir();
+    const program = new Command();
+    program.exitOverride();
+    registerFleetCommand(program, {
+      runner: async () => ({ stdout: '', stderr: '', exitCode: 0 }),
+      mosaicHome: home,
+    });
+    try {
+      await program.parseAsync([
+        'node',
+        'mosaic',
+        'fleet',
+        'init',
+        '--profile',
+        'general',
+        '--write',
+      ]);
+      const written = await readFile(join(home, 'fleet', 'roster.yaml'), 'utf8');
+      const orchestrators = (written.match(/class:\s*orchestrator/g) ?? []).length;
+      expect(orchestrators).toBe(1);
+    } finally {
+      await rm(home, { recursive: true, force: true });
+    }
+  });
+});
+
 describe('fleet install — auto-enable units for boot-survival', () => {
   it('buildSystemdEnableCommand and buildEnableLingerCommand return correct command arrays', () => {
     expect(buildSystemdEnableCommand('mosaic-tmux-holder.service')).toEqual([

packages/mosaic/src/commands/fleet.ts

@@ -227,6 +227,15 @@ export function buildSystemdEnableCommand(unit: string): string[] {
   return ['systemctl', '--user', 'enable', unit];
 }
 
+/**
+ * Returns the systemctl --user disable command for a given unit.
+ * Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
+ * boot pointing at deleted config (boot-survival symmetry with enable-on-add).
+ */
+export function buildSystemdDisableCommand(unit: string): string[] {
+  return ['systemctl', '--user', 'disable', unit];
+}
+
 /**
  * Returns the loginctl enable-linger command for a given user.
  * Linger allows user systemd services to survive logout.
@@ -872,15 +881,19 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
       await mkdir(dirname(destination), { recursive: true });
       await writeFile(destination, content);
 
-      // Validate: exactly one orchestrator required (R5) — friendly summary on success.
+      // Guarantee R5: exactly one orchestrator for every profile except the
+      // sanctioned no-orchestrator `minimal` preset. A mismatch means a
+      // corrupted/edited preset — fail hard rather than write a malformed fleet.
       const written = await loadFleetRoster(destination);
       const orchCount = countOrchestrators(written);
-      if (orchCount !== 1) {
+      if (profile === 'minimal') {
-        process.stderr.write(
-          `Warning: fleet roster at ${destination} has ${orchCount} orchestrator agent(s) (expected exactly 1).\n`,
-        );
         console.log(
-          `Initialized ${profile} fleet: ${written.agents.length} agent(s). Next: mosaic fleet install`,
+          `Initialized ${profile} fleet: ${written.agents.length} agent(s) (no orchestrator). Next: mosaic fleet install`,
+        );
+      } else if (orchCount !== 1) {
+        throw new Error(
+          `Fleet init failed: the "${profile}" roster has ${orchCount} orchestrator agent(s), ` +
+            `expected exactly 1 (R5). The preset may be corrupted — re-install the framework.`,
         );
       } else {
         const workerCount = written.agents.length - 1;
@@ -1218,6 +1231,24 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
 
         console.log(`Added ${name} (${opts.runtime}/${opts.class}) to the fleet.`);
 
+        // Enable the unit for boot-survival (non-fatal) — symmetry with
+        // disable-on-remove. Independent of --start so a queued agent still
+        // survives a reboot once its unit exists.
+        try {
+          const enableResult = await runner(
+            ...splitCommand(buildSystemdEnableCommand(`mosaic-agent@${name}.service`)),
+          );
+          if (enableResult.exitCode !== 0) {
+            process.stderr.write(
+              `Warning: could not enable mosaic-agent@${name}.service: ${enableResult.stderr || enableResult.stdout || 'non-zero exit'}\n`,
+            );
+          }
+        } catch (err) {
+          process.stderr.write(
+            `Warning: enable command failed for ${name}: ${err instanceof Error ? err.message : String(err)}\n`,
+          );
+        }
+
         if (opts.start !== false) {
           await runChecked(runner, buildFleetServiceCommand('start', name));
           console.log(`Started mosaic-agent@${name}.service.`);
@@ -1254,6 +1285,26 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
         );
       }
 
+      // Disable the unit (non-fatal) so an enabled instance cannot resurrect on
+      // boot pointing at the now-deleted config — boot-survival symmetry with
+      // enable-on-add. Skipped only when --keep-files keeps the config in place.
+      if (!opts.keepFiles) {
+        try {
+          const disableResult = await runner(
+            ...splitCommand(buildSystemdDisableCommand(`mosaic-agent@${name}.service`)),
+          );
+          if (disableResult.exitCode !== 0) {
+            process.stderr.write(
+              `Warning: could not disable mosaic-agent@${name}.service: ${disableResult.stderr || disableResult.stdout || 'non-zero exit'}\n`,
+            );
+          }
+        } catch (err) {
+          process.stderr.write(
+            `Warning: disable command failed for ${name}: ${err instanceof Error ? err.message : String(err)}\n`,
+          );
+        }
+      }
+
       // Write updated roster
       await writeFile(rosterPath, serializeRosterToYaml(updatedRoster));
 

packages/mosaic/src/runtime/update-checker.reseed.spec.ts

@@ -0,0 +1,85 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+  buildReseedCommand,
+  buildRelaunchCommands,
+  readRosterAgentNames,
+  runFrameworkReseed,
+} from './update-checker.js';
+
+/**
+ * F3-m3 / R13: `mosaic update` re-seeds the framework + (opt-in) relaunches
+ * durable agents so shipped launcher/runtime changes activate. These cover the
+ * pure builders + the missing-installer guard (the exec path is integration).
+ */
+
+describe('buildReseedCommand', () => {
+  it('invokes the package install.sh in data-safe sync-only keep mode', () => {
+    const out = buildReseedCommand('/pkg/framework', '/home/u/.config/mosaic');
+    expect(out.installer).toBe('/pkg/framework/install.sh');
+    expect(out.command).toBe('bash /pkg/framework/install.sh');
+    expect(out.env).toEqual({
+      MOSAIC_SYNC_ONLY: '1',
+      MOSAIC_INSTALL_MODE: 'keep',
+      MOSAIC_HOME: '/home/u/.config/mosaic',
+    });
+  });
+});
+
+describe('buildRelaunchCommands', () => {
+  it('builds a systemctl --user restart per agent unit', () => {
+    expect(buildRelaunchCommands(['orchestrator', 'coder0'])).toEqual([
+      ['systemctl', '--user', 'restart', 'mosaic-agent@orchestrator.service'],
+      ['systemctl', '--user', 'restart', 'mosaic-agent@coder0.service'],
+    ]);
+  });
+
+  it('is empty for an empty roster', () => {
+    expect(buildRelaunchCommands([])).toEqual([]);
+  });
+});
+
+describe('readRosterAgentNames', () => {
+  let home: string;
+
+  beforeEach(() => {
+    home = mkdtempSync(join(tmpdir(), 'mosaic-roster-'));
+  });
+  afterEach(() => {
+    rmSync(home, { recursive: true, force: true });
+  });
+
+  it('returns [] when no roster exists', () => {
+    expect(readRosterAgentNames(home)).toEqual([]);
+  });
+
+  it('extracts agent names from roster.yaml', () => {
+    mkdirSync(join(home, 'fleet'), { recursive: true });
+    writeFileSync(
+      join(home, 'fleet', 'roster.yaml'),
+      [
+        'version: 1',
+        'agents:',
+        '  - name: orchestrator',
+        '    runtime: pi',
+        '  - name: coder0',
+        '    runtime: claude',
+        '  - name: "reviewer-1"',
+        '    runtime: codex',
+      ].join('\n') + '\n',
+    );
+    expect(readRosterAgentNames(home)).toEqual(['orchestrator', 'coder0', 'reviewer-1']);
+  });
+});
+
+describe('runFrameworkReseed', () => {
+  it('reports not-ok (not throw) when the installer is absent', () => {
+    const missing = mkdtempSync(join(tmpdir(), 'mosaic-noinstaller-'));
+    const res = runFrameworkReseed(missing, join(missing, 'home'));
+    expect(res.ok).toBe(false);
+    expect(res.reason).toContain('installer not found');
+    rmSync(missing, { recursive: true, force: true });
+  });
+});

packages/mosaic/src/runtime/update-checker.ts

@@ -16,7 +16,8 @@
 import { execSync } from 'node:child_process';
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
 import { homedir } from 'node:os';
-import { join } from 'node:path';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
 
 // ─── Types ──────────────────────────────────────────────────────────────────
 
@@ -453,6 +454,98 @@ export function getInstallAllCommand(outdated: PackageUpdateResult[]): string {
   return `npm i -g ${pkgs.join(' ')}`;
 }
 
+// ─── Post-update framework re-seed + agent relaunch (F3-m3 / R13) ─────────────
+//
+// `mosaic update` installs the new npm CLI but, on its own, leaves the framework
+// files in ~/.config/mosaic/ stale — so shipped launcher/runtime changes (e.g.
+// the agent-name export + native heartbeat) never ACTIVATE until a re-seed.
+// These helpers run the package's own install.sh in sync-only mode (the P4
+// data-safe reconcile: framework-owned overwrite + backup-once; SOUL/USER/
+// *.local/credentials preserved) and, opt-in, relaunch durable agents.
+
+/** Resolve the framework/ directory bundled in the installed package. */
+export function resolveBundledFrameworkRoot(): string {
+  // dist/runtime/update-checker.js → ../../framework (package files: dist + framework)
+  return resolve(dirname(fileURLToPath(import.meta.url)), '..', '..', 'framework');
+}
+
+export const FRAMEWORK_RESEED_PACKAGE = PKG;
+
+/**
+ * Build the framework re-seed invocation: the package's install.sh in
+ * sync-only mode (file phase only — no environment-touching post-install),
+ * keep mode (never overwrite user files). Returned as data so it is unit
+ * testable; `runFrameworkReseed` executes it.
+ */
+export function buildReseedCommand(
+  frameworkRoot: string,
+  mosaicHome: string,
+): { installer: string; command: string; env: Record<string, string> } {
+  const installer = join(frameworkRoot, 'install.sh');
+  return {
+    installer,
+    command: `bash ${installer}`,
+    env: {
+      MOSAIC_SYNC_ONLY: '1',
+      MOSAIC_INSTALL_MODE: 'keep',
+      MOSAIC_HOME: mosaicHome,
+    },
+  };
+}
+
+/**
+ * Re-seed the framework from the freshly-installed package. Returns a result
+ * describing what happened (so callers can message + decide on relaunch).
+ * Best-effort: a missing installer or a non-zero exit is reported, not thrown.
+ */
+export function runFrameworkReseed(
+  frameworkRoot = resolveBundledFrameworkRoot(),
+  mosaicHome = join(homedir(), '.config', 'mosaic'),
+): { ok: boolean; reason?: string } {
+  const { installer, command, env } = buildReseedCommand(frameworkRoot, mosaicHome);
+  if (!existsSync(installer)) {
+    return { ok: false, reason: `installer not found: ${installer}` };
+  }
+  try {
+    execSync(command, { stdio: 'inherit', env: { ...process.env, ...env }, timeout: 120_000 });
+    return { ok: true };
+  } catch (err) {
+    return { ok: false, reason: err instanceof Error ? err.message : String(err) };
+  }
+}
+
+/**
+ * Best-effort parse of the fleet roster for agent names (used to relaunch
+ * durable agents after a re-seed). Returns [] when no roster exists.
+ */
+export function readRosterAgentNames(mosaicHome = join(homedir(), '.config', 'mosaic')): string[] {
+  const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
+  if (!existsSync(rosterPath)) return [];
+  let text: string;
+  try {
+    text = readFileSync(rosterPath, 'utf-8');
+  } catch {
+    return [];
+  }
+  // Roster agents are listed as `- name: <id>` entries under `agents:`.
+  const names: string[] = [];
+  for (const line of text.split('\n')) {
+    const m = line.match(/^\s*-?\s*name:\s*["']?([A-Za-z0-9._-]+)["']?\s*$/);
+    if (m && m[1]) names.push(m[1]);
+  }
+  return names;
+}
+
+/** Build the per-agent systemd relaunch commands (drain+relaunch via restart). */
+export function buildRelaunchCommands(agentNames: string[]): string[][] {
+  return agentNames.map((name) => [
+    'systemctl',
+    '--user',
+    'restart',
+    `mosaic-agent@${name}.service`,
+  ]);
+}
+
 /**
  * Format a table showing all packages with their current/latest versions.
  */