# Local Fleet Canary

The local fleet canary runs a small tmux-backed Mosaic agent fleet on an
isolated tmux socket. The default socket is `mosaic-fleet`; the commands do
not use or stop the default tmux server.

## Files

Product-owned defaults:

- `packages/mosaic/framework/fleet/roster.schema.json`
- `packages/mosaic/framework/fleet/examples/minimal.yaml`
- `packages/mosaic/framework/fleet/examples/local-canary.yaml`
- `packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service`
- `packages/mosaic/framework/systemd/user/mosaic-agent@.service`
- `packages/mosaic/framework/tools/fleet/start-agent-session.sh`
- `packages/mosaic/framework/tools/tmux/agent-send.sh`
- `packages/mosaic/framework/tools/tmux/send-message.sh`

These files are published through `packages/mosaic/package.json`, whose `files`
allowlist includes `framework` along with `dist`.

Site-owned local roster:

```text
~/.config/mosaic/fleet/roster.yaml
```

Do not put a host-specific full roster into product defaults. Start from an
example and edit the local roster after `mosaic fleet init --write`.

## Install

Minimal canary:

```bash
mosaic fleet init --profile minimal --write
# If a site-owned roster already exists, inspect it first; overwrite only explicitly:
# mosaic fleet init --profile minimal --write --force
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet verify
```

Small dogfood roster:

```bash
mosaic fleet init --profile local-canary --write
# Use --force only after preserving any site-owned roster changes.
mosaic fleet install-systemd
systemctl --user daemon-reload
mosaic fleet start
mosaic fleet status
```

## Agent Operations

```bash
mosaic agent roster
mosaic agent status
mosaic agent status canary-pi
mosaic agent send canary-pi --message "status check"
mosaic agent reset canary-pi --new
mosaic agent tail canary-pi -n 80
```

These commands read the roster and target the configured tmux socket. The
generated systemd agent services use `start-agent-session.sh`; message delivery
uses the tmux send tools with `-L mosaic-fleet`.

`mosaic agent send` is operator-origin traffic unless a caller explicitly says
otherwise. The CLI always passes a deterministic source label to
`agent-send.sh` with `-S`, defaulting to `<hostname>:operator`, so it does not
query the target tmux socket and accidentally identify as an active agent pane.
Use `--source-label <label>` or `--source <label>` only when deliberately
impersonating a known handoff lane. The lower-level inter-agent wrapper
`agent-send.sh -S <label>` remains the explicit source override for scripts.

## Verification

Use these checks before expanding the roster:

```bash
tmux -L mosaic-fleet ls
tmux ls
mosaic fleet verify
systemctl --user status mosaic-tmux-holder.service
```

Expected results:

- `tmux -L mosaic-fleet ls` shows `_holder` and roster agent sessions.
- `tmux ls` shows only the default tmux server sessions and is not changed by
  fleet start/stop operations.
- `mosaic fleet verify` checks exact session targets on the isolated socket.
- `systemctl --user status ...` may show `active (exited)` for oneshot units;
  that means the unit ran, not that an agent pane is live. Treat tmux
  `has-session`, `list-panes`, process tree, and logs as the liveness evidence.

## Recovery — rebuild generated env projections

Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
deterministic projection of `roster.yaml` (the SSOT) that the launcher
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
wipes or diverges those projections, rebuild them from the roster with
`mosaic fleet regen` — do NOT restart the affected unit first.

```bash
mosaic fleet regen            # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json     # same plan, machine-readable
mosaic fleet regen --write    # rebuild fleet/agents/<name>.env.generated on disk
```

`regen` is projection-only and **never restarts an agent** — it has no path to
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
only (never the projected `KEY=value` body). After `--write`, verify each unit
resolves the intended values before restarting one unit at a time. The unit sets
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
so verify the generated file directly and the launcher path, not a nonexistent
`EnvironmentFile` property:

```bash
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>
```

Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](../ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md).

## Release Preflight

Run this checklist before cutting or dogfooding a fleet release:

- Real AI dogfood: send at least one task through `mosaic agent send`, then
  confirm the agent accepted/responded using pane, process, or log evidence.
- Restart/stop/idempotency: run `mosaic fleet start`, `restart`, `stop`, and a
  repeated `start` against the named socket; verify the default tmux server is
  unchanged.
- Liveness verification: run `mosaic fleet verify` and confirm roster sessions
  with `tmux -L mosaic-fleet ls` or exact `has-session` checks.
- Package dry-run: run `npm pack --dry-run --json` from `packages/mosaic` and
  confirm `framework/fleet`, `framework/systemd/user`,
  `framework/tools/fleet`, and `framework/tools/tmux` assets are included.
- Mosaic update test: install or upgrade from the packed artifact in a temporary
  Mosaic home and confirm `mosaic update` or the release upgrade path does not
  remove local roster/config files.

## Rollback

Stop the local canary:

```bash
mosaic fleet stop
systemctl --user disable mosaic-agent@canary-pi.service
systemctl --user disable mosaic-tmux-holder.service
systemctl --user daemon-reload
```

For a full local cleanup of generated canary files:

```bash
rm -f ~/.config/systemd/user/mosaic-agent@.service
rm -f ~/.config/systemd/user/mosaic-tmux-holder.service
rm -rf ~/.config/mosaic/fleet
rm -rf ~/.config/mosaic/tools/fleet
```

This rollback leaves the default tmux server untouched. If a canary session is
still present after service stop, remove only the isolated socket server:

```bash
tmux -L mosaic-fleet kill-server
```
