Files
stack/docs/guides/fleet-local-canary.md
jason.woltje b5c1381e45
All checks were successful
ci/woodpecker/push/ci Pipeline was successful
ci/woodpecker/push/publish Pipeline was successful
fix(fleet): harden operator sends for release (#565)
2026-06-20 20:41:11 +00:00

4.8 KiB

Local Fleet Canary

The local fleet canary runs a small tmux-backed Mosaic agent fleet on an isolated tmux socket. The default socket is mosaic-factory; the commands do not use or stop the default tmux server.

Files

Product-owned defaults:

  • packages/mosaic/framework/fleet/roster.schema.json
  • packages/mosaic/framework/fleet/examples/minimal.yaml
  • packages/mosaic/framework/fleet/examples/local-canary.yaml
  • packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service
  • packages/mosaic/framework/systemd/user/mosaic-agent@.service
  • packages/mosaic/framework/tools/fleet/start-agent-session.sh
  • packages/mosaic/framework/tools/tmux/agent-send.sh
  • packages/mosaic/framework/tools/tmux/send-message.sh

These files are published through packages/mosaic/package.json, whose files allowlist includes framework along with dist.

Site-owned local roster:

~/.config/mosaic/fleet/roster.yaml

Do not put a host-specific full roster into product defaults. Start from an example and edit the local roster after mosaic fleet init --write.

Install

Minimal canary:

mosaic fleet init --profile minimal --write
# If a site-owned roster already exists, inspect it first; overwrite only explicitly:
# mosaic fleet init --profile minimal --write --force
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet verify

Small dogfood roster:

mosaic fleet init --profile local-canary --write
# Use --force only after preserving any site-owned roster changes.
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet status

Agent Operations

mosaic agent roster
mosaic agent status
mosaic agent status canary-pi
mosaic agent send canary-pi --message "status check"
mosaic agent reset canary-pi --new
mosaic agent tail canary-pi -n 80

These commands read the roster and target the configured tmux socket. The generated systemd agent services use start-agent-session.sh; message delivery uses the tmux send tools with -L mosaic-factory.

mosaic agent send is operator-origin traffic unless a caller explicitly says otherwise. The CLI always passes a deterministic source label to agent-send.sh with -S, defaulting to <hostname>:operator, so it does not query the target tmux socket and accidentally identify as an active agent pane. Use --source-label <label> or --source <label> only when deliberately impersonating a known handoff lane. The lower-level inter-agent wrapper agent-send.sh -S <label> remains the explicit source override for scripts.

Verification

Use these checks before expanding the roster:

tmux -L mosaic-factory ls
tmux ls
mosaic fleet verify
systemctl --user status mosaic-tmux-holder.service

Expected results:

  • tmux -L mosaic-factory ls shows _holder and roster agent sessions.
  • tmux ls shows only the default tmux server sessions and is not changed by fleet start/stop operations.
  • mosaic fleet verify checks exact session targets on the isolated socket.
  • systemctl --user status ... may show active (exited) for oneshot units; that means the unit ran, not that an agent pane is live. Treat tmux has-session, list-panes, process tree, and logs as the liveness evidence.

Release Preflight

Run this checklist before cutting or dogfooding a fleet release:

  • Real AI dogfood: send at least one task through mosaic agent send, then confirm the agent accepted/responded using pane, process, or log evidence.
  • Restart/stop/idempotency: run mosaic fleet start, restart, stop, and a repeated start against the named socket; verify the default tmux server is unchanged.
  • Liveness verification: run mosaic fleet verify and confirm roster sessions with tmux -L mosaic-factory ls or exact has-session checks.
  • Package dry-run: run npm pack --dry-run --json from packages/mosaic and confirm framework/fleet, framework/systemd/user, framework/tools/fleet, and framework/tools/tmux assets are included.
  • Mosaic update test: install or upgrade from the packed artifact in a temporary Mosaic home and confirm mosaic update or the release upgrade path does not remove local roster/config files.

Rollback

Stop the local canary:

mosaic fleet stop
systemctl --user disable mosaic-agent@canary-pi.service
systemctl --user disable mosaic-tmux-holder.service
systemctl --user daemon-reload

For a full local cleanup of generated canary files:

rm -f ~/.config/systemd/user/mosaic-agent@.service
rm -f ~/.config/systemd/user/mosaic-tmux-holder.service
rm -rf ~/.config/mosaic/fleet
rm -rf ~/.config/mosaic/tools/fleet

This rollback leaves the default tmux server untouched. If a canary session is still present after service stop, remove only the isolated socket server:

tmux -L mosaic-factory kill-server