stack/docs/guides/fleet-local-canary.md

Local Fleet Canary

The local fleet canary runs a small tmux-backed Mosaic agent fleet on an isolated tmux socket. The default socket is mosaic-fleet; the commands do not use or stop the default tmux server.

Files

Product-owned defaults:

• packages/mosaic/framework/fleet/roster.schema.json
• packages/mosaic/framework/fleet/examples/minimal.yaml
• packages/mosaic/framework/fleet/examples/local-canary.yaml
• packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service
• packages/mosaic/framework/systemd/user/mosaic-agent@.service
• packages/mosaic/framework/tools/fleet/start-agent-session.sh
• packages/mosaic/framework/tools/tmux/agent-send.sh
• packages/mosaic/framework/tools/tmux/send-message.sh

These files are published through packages/mosaic/package.json, whose files allowlist includes framework along with dist.

Site-owned local roster:

~/.config/mosaic/fleet/roster.yaml

Do not put a host-specific full roster into product defaults. Start from an example and edit the local roster after mosaic fleet init --write.

Install

Minimal canary:

mosaic fleet init --profile minimal --write
# If a site-owned roster already exists, inspect it first; overwrite only explicitly:
# mosaic fleet init --profile minimal --write --force
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet verify

Small dogfood roster:

mosaic fleet init --profile local-canary --write
# Use --force only after preserving any site-owned roster changes.
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet status

Agent Operations

mosaic agent roster
mosaic agent status
mosaic agent status canary-pi
mosaic agent send canary-pi --message "status check"
mosaic agent reset canary-pi --new
mosaic agent tail canary-pi -n 80

These commands read the roster and target the configured tmux socket. The generated systemd agent services use start-agent-session.sh; message delivery uses the tmux send tools with -L mosaic-fleet.

mosaic agent send is operator-origin traffic unless a caller explicitly says otherwise. The CLI always passes a deterministic source label to agent-send.sh with -S, defaulting to <hostname>:operator, so it does not query the target tmux socket and accidentally identify as an active agent pane. Use --source-label <label> or --source <label> only when deliberately impersonating a known handoff lane. The lower-level inter-agent wrapper agent-send.sh -S <label> remains the explicit source override for scripts.

Verification

Use these checks before expanding the roster:

tmux -L mosaic-fleet ls
tmux ls
mosaic fleet verify
systemctl --user status mosaic-tmux-holder.service

Expected results:

• tmux -L mosaic-fleet ls shows _holder and roster agent sessions.
• tmux ls shows only the default tmux server sessions and is not changed by fleet start/stop operations.
• mosaic fleet verify checks exact session targets on the isolated socket.
• systemctl --user status ... may show active (exited) for oneshot units; that means the unit ran, not that an agent pane is live. Treat tmux has-session, list-panes, process tree, and logs as the liveness evidence.

Recovery — rebuild generated env projections

Each agent's ~/.config/mosaic/fleet/agents/<name>.env.generated is a deterministic projection of roster.yaml (the SSOT) that the launcher (start-agent-session.sh) sources at start. If an upgrade or a manual mistake wipes or diverges those projections, rebuild them from the roster with mosaic fleet regen — do NOT restart the affected unit first.

mosaic fleet regen            # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json     # same plan, machine-readable
mosaic fleet regen --write    # rebuild fleet/agents/<name>.env.generated on disk

regen is projection-only and never restarts an agent — it has no path to systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the same roster→env mapping as mosaic fleet reconcile, and emits paths and counts only (never the projected KEY=value body). After --write, verify each unit resolves the intended values before restarting one unit at a time. The unit sets no EnvironmentFile= — start-agent-session.sh sources .env.generated itself — so verify the generated file directly and the launcher path, not a nonexistent EnvironmentFile property:

test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>

Full recovery runbook and the three-layer #791 protection model (manifest ownership → pre-update snapshot/restore → regen): see Upgrade Safety & Recovery.

Release Preflight

Run this checklist before cutting or dogfooding a fleet release:

• Real AI dogfood: send at least one task through mosaic agent send, then confirm the agent accepted/responded using pane, process, or log evidence.
• Restart/stop/idempotency: run mosaic fleet start, restart, stop, and a repeated start against the named socket; verify the default tmux server is unchanged.
• Liveness verification: run mosaic fleet verify and confirm roster sessions with tmux -L mosaic-fleet ls or exact has-session checks.
• Package dry-run: run npm pack --dry-run --json from packages/mosaic and confirm framework/fleet, framework/systemd/user, framework/tools/fleet, and framework/tools/tmux assets are included.
• Mosaic update test: install or upgrade from the packed artifact in a temporary Mosaic home and confirm mosaic update or the release upgrade path does not remove local roster/config files.

Rollback

Stop the local canary:

mosaic fleet stop
systemctl --user disable mosaic-agent@canary-pi.service
systemctl --user disable mosaic-tmux-holder.service
systemctl --user daemon-reload

For a full local cleanup of generated canary files:

rm -f ~/.config/systemd/user/mosaic-agent@.service
rm -f ~/.config/systemd/user/mosaic-tmux-holder.service
rm -rf ~/.config/mosaic/fleet
rm -rf ~/.config/mosaic/tools/fleet

This rollback leaves the default tmux server untouched. If a canary session is still present after service stop, remove only the isolated socket server:

tmux -L mosaic-fleet kill-server