1eb37f6fd0
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Add `mosaic fleet regen`, a projection-only recovery command that rebuilds each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an upgrade or partial write leaves the generated projections stale or missing. - Dry-run by default; `--write` applies; `--json` for machine output. Reuses the merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv + the generated-env boundary) rather than reimplementing fleet logic. - Structurally NEVER issues a lifecycle/restart call — regen recovers config only; a recordingRunner gate proves no runner invocation ever occurs. - Serializes against agent CRUD and reconcile via BOTH fleet locks (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired mutation-then-reconcile and released in reverse; both are non-blocking `wx` locks that throw on contention, so no deadlock is possible. - Hardens the shared managed-lock helper: ownership-proving tokened lock reused for both locks with per-lock fault labels; init-failure cleanup no longer strands a just-created lock (dev/ino guard, with a persisted-token fallback when the post-create stat itself fails); acquire-unwind surfaces a lock cleanup fault instead of dropping it. - Resolves personas the SAME way reconcile does by forwarding configured rolesDir/overrideDir, so a custom-persona-root deployment cannot have reconcile accept a roster that regen rejects. - Report/output is secrev-safe: paths and counts only, never projected values. - Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note. Tests are TDD red-first with co-located specs (regen spec: 26 tests covering dry-run/write dispositions, the never-restarts gate, all lock regressions, and persona-root wiring). A residual check-then-unlink TOCTOU in the lock release remains (byte-identical to the merged reconcile lock; unreachable within the `wx` writer protocol); its true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a separate follow-up. Part of #791 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
178 lines
6.4 KiB
Markdown
178 lines
6.4 KiB
Markdown
# Local Fleet Canary
|
|
|
|
The local fleet canary runs a small tmux-backed Mosaic agent fleet on an
|
|
isolated tmux socket. The default socket is `mosaic-fleet`; the commands do
|
|
not use or stop the default tmux server.
|
|
|
|
## Files
|
|
|
|
Product-owned defaults:
|
|
|
|
- `packages/mosaic/framework/fleet/roster.schema.json`
|
|
- `packages/mosaic/framework/fleet/examples/minimal.yaml`
|
|
- `packages/mosaic/framework/fleet/examples/local-canary.yaml`
|
|
- `packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service`
|
|
- `packages/mosaic/framework/systemd/user/mosaic-agent@.service`
|
|
- `packages/mosaic/framework/tools/fleet/start-agent-session.sh`
|
|
- `packages/mosaic/framework/tools/tmux/agent-send.sh`
|
|
- `packages/mosaic/framework/tools/tmux/send-message.sh`
|
|
|
|
These files are published through `packages/mosaic/package.json`, whose `files`
|
|
allowlist includes `framework` along with `dist`.
|
|
|
|
Site-owned local roster:
|
|
|
|
```text
|
|
~/.config/mosaic/fleet/roster.yaml
|
|
```
|
|
|
|
Do not put a host-specific full roster into product defaults. Start from an
|
|
example and edit the local roster after `mosaic fleet init --write`.
|
|
|
|
## Install
|
|
|
|
Minimal canary:
|
|
|
|
```bash
|
|
mosaic fleet init --profile minimal --write
|
|
# If a site-owned roster already exists, inspect it first; overwrite only explicitly:
|
|
# mosaic fleet init --profile minimal --write --force
|
|
mosaic fleet install-systemd
|
|
systemctl --user daemon-reload
|
|
mosaic fleet start
|
|
mosaic fleet verify
|
|
```
|
|
|
|
Small dogfood roster:
|
|
|
|
```bash
|
|
mosaic fleet init --profile local-canary --write
|
|
# Use --force only after preserving any site-owned roster changes.
|
|
mosaic fleet install-systemd
|
|
systemctl --user daemon-reload
|
|
mosaic fleet start
|
|
mosaic fleet status
|
|
```
|
|
|
|
## Agent Operations
|
|
|
|
```bash
|
|
mosaic agent roster
|
|
mosaic agent status
|
|
mosaic agent status canary-pi
|
|
mosaic agent send canary-pi --message "status check"
|
|
mosaic agent reset canary-pi --new
|
|
mosaic agent tail canary-pi -n 80
|
|
```
|
|
|
|
These commands read the roster and target the configured tmux socket. The
|
|
generated systemd agent services use `start-agent-session.sh`; message delivery
|
|
uses the tmux send tools with `-L mosaic-fleet`.
|
|
|
|
`mosaic agent send` is operator-origin traffic unless a caller explicitly says
|
|
otherwise. The CLI always passes a deterministic source label to
|
|
`agent-send.sh` with `-S`, defaulting to `<hostname>:operator`, so it does not
|
|
query the target tmux socket and accidentally identify as an active agent pane.
|
|
Use `--source-label <label>` or `--source <label>` only when deliberately
|
|
impersonating a known handoff lane. The lower-level inter-agent wrapper
|
|
`agent-send.sh -S <label>` remains the explicit source override for scripts.
|
|
|
|
## Verification
|
|
|
|
Use these checks before expanding the roster:
|
|
|
|
```bash
|
|
tmux -L mosaic-fleet ls
|
|
tmux ls
|
|
mosaic fleet verify
|
|
systemctl --user status mosaic-tmux-holder.service
|
|
```
|
|
|
|
Expected results:
|
|
|
|
- `tmux -L mosaic-fleet ls` shows `_holder` and roster agent sessions.
|
|
- `tmux ls` shows only the default tmux server sessions and is not changed by
|
|
fleet start/stop operations.
|
|
- `mosaic fleet verify` checks exact session targets on the isolated socket.
|
|
- `systemctl --user status ...` may show `active (exited)` for oneshot units;
|
|
that means the unit ran, not that an agent pane is live. Treat tmux
|
|
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
|
|
|
## Recovery — rebuild generated env projections
|
|
|
|
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
|
|
deterministic projection of `roster.yaml` (the SSOT) that the launcher
|
|
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
|
|
wipes or diverges those projections, rebuild them from the roster with
|
|
`mosaic fleet regen` — do NOT restart the affected unit first.
|
|
|
|
```bash
|
|
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
|
|
mosaic fleet regen --json # same plan, machine-readable
|
|
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
|
|
```
|
|
|
|
`regen` is projection-only and **never restarts an agent** — it has no path to
|
|
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
|
|
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
|
|
only (never the projected `KEY=value` body). After `--write`, verify each unit
|
|
resolves the intended values before restarting one unit at a time. The unit sets
|
|
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
|
|
so verify the generated file directly and the launcher path, not a nonexistent
|
|
`EnvironmentFile` property:
|
|
|
|
```bash
|
|
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
|
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
|
systemctl --user restart mosaic-agent@<name>
|
|
```
|
|
|
|
Full recovery runbook and the three-layer #791 protection model (manifest
|
|
ownership → pre-update snapshot/restore → regen): see
|
|
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
|
|
|
|
## Release Preflight
|
|
|
|
Run this checklist before cutting or dogfooding a fleet release:
|
|
|
|
- Real AI dogfood: send at least one task through `mosaic agent send`, then
|
|
confirm the agent accepted/responded using pane, process, or log evidence.
|
|
- Restart/stop/idempotency: run `mosaic fleet start`, `restart`, `stop`, and a
|
|
repeated `start` against the named socket; verify the default tmux server is
|
|
unchanged.
|
|
- Liveness verification: run `mosaic fleet verify` and confirm roster sessions
|
|
with `tmux -L mosaic-fleet ls` or exact `has-session` checks.
|
|
- Package dry-run: run `npm pack --dry-run --json` from `packages/mosaic` and
|
|
confirm `framework/fleet`, `framework/systemd/user`,
|
|
`framework/tools/fleet`, and `framework/tools/tmux` assets are included.
|
|
- Mosaic update test: install or upgrade from the packed artifact in a temporary
|
|
Mosaic home and confirm `mosaic update` or the release upgrade path does not
|
|
remove local roster/config files.
|
|
|
|
## Rollback
|
|
|
|
Stop the local canary:
|
|
|
|
```bash
|
|
mosaic fleet stop
|
|
systemctl --user disable mosaic-agent@canary-pi.service
|
|
systemctl --user disable mosaic-tmux-holder.service
|
|
systemctl --user daemon-reload
|
|
```
|
|
|
|
For a full local cleanup of generated canary files:
|
|
|
|
```bash
|
|
rm -f ~/.config/systemd/user/mosaic-agent@.service
|
|
rm -f ~/.config/systemd/user/mosaic-tmux-holder.service
|
|
rm -rf ~/.config/mosaic/fleet
|
|
rm -rf ~/.config/mosaic/tools/fleet
|
|
```
|
|
|
|
This rollback leaves the default tmux server untouched. If a canary session is
|
|
still present after service stop, remove only the isolated socket server:
|
|
|
|
```bash
|
|
tmux -L mosaic-fleet kill-server
|
|
```
|