test(fleet): close documentation code grammar

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Jarvis
2026-07-17 18:39:05 -05:00
parent 862dbd5204
commit bfa08e9651
36 changed files with 966 additions and 1022 deletions

View File

@@ -10,8 +10,8 @@
The durable tmux fleet runs on the isolated `mosaic-fleet` socket. That isolation
(which protects the operator's default tmux) makes the fleet **invisible** to default
tooling, and truth is split across three planes no single command joins — systemd
(`systemctl --user`), tmux (`-L mosaic-fleet`), and the process tree (`pstree`).
`agent tail` (`capture-pane`) returns **blank for full-screen TUIs**, and `agent send`
(systemctl --user), tmux (-L mosaic-fleet), and the process tree (`pstree`).
agent tail (`capture-pane`) returns **blank for full-screen TUIs**, and agent send
confirms only keystroke injection, not acceptance. Net: the operator has near-zero
observability and no safe way to watch a session.
@@ -32,22 +32,22 @@ observability and no safe way to watch a session.
## Functional requirements
| ID | Requirement |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FR-1 | `mosaic fleet ps [--json]` prints one row per roster agent joining: name · tenant · host · runtime · systemd(active/enabled) · pane(alive/dead) · pid · idle · **last-heartbeat age** · **drift** flag (roster runtime ≠ actual pane command) · **boot-enable** warning (active but `UnitFileState=disabled`). |
| FR-2 | **Heartbeat protocol v1** (see below); `dogfood-agent.py` implements the responder. `fleet ps` issues probes (or reads last-seen) and reports health per FR-1. |
| FR-3 | `mosaic agent watch <name>` opens a **read-only** view of the pane (grouped session or `tmux attach -r`) that cannot send keystrokes and does not shrink the agent's window. |
| FR-4 | `mosaic agent attach <name>` remains the **explicit** interactive-takeover path (separate verb, documented as the only one that can type). |
| FR-5 | `mosaic agent send <name> --verify` confirms the message was **accepted** (not left as an unsubmitted draft) and returns non-zero if delivery cannot be verified. |
| FR-6 | All structured output (`--json`) includes `tenant_id` and `host` fields. |
| ID | Requirement |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| FR-1 | mosaic fleet ps [--json] prints one row per roster agent joining: name · tenant · host · runtime · systemd(active/enabled) · pane(alive/dead) · pid · idle · **last-heartbeat age** · **drift** flag (roster runtime ≠ actual pane command) · **boot-enable** warning (active but `UnitFileState=disabled`). |
| FR-2 | **Heartbeat protocol v1** (see below); `dogfood-agent.py` implements the responder. fleet ps issues probes (or reads last-seen) and reports health per FR-1. |
| FR-3 | mosaic agent watch <name> opens a **read-only** view of the pane (grouped session or tmux attach -r) that cannot send keystrokes and does not shrink the agent's window. |
| FR-4 | mosaic agent attach <name> remains the **explicit** interactive-takeover path (separate verb, documented as the only one that can type). |
| FR-5 | mosaic agent send <name> --verify confirms the message was **accepted** (not left as an unsubmitted draft) and returns non-zero if delivery cannot be verified. |
| FR-6 | All structured output (`--json`) includes `tenant_id` and `host` fields. |
## Heartbeat protocol v1
- **Probe:** operator/`fleet ps` writes a sentinel line to the agent's input or a
well-known per-agent heartbeat file path `~/.config/mosaic/fleet/run/<agent>.hb`.
- **Response:** the runtime updates `<agent>.hb` with `ts=<iso8601> pid=<pid> status=<ok|busy>`
- **Probe:** operator/fleet ps writes a sentinel line to the agent's input or a
well-known per-agent heartbeat file path ~/.config/mosaic/fleet/run/<agent>.hb.
- **Response:** the runtime updates <agent>.hb with ts=<iso8601> pid=<pid> status=<ok|busy>
on a fixed interval (default 15s) and on demand when probed.
- **Health rule:** `healthy` if `now - ts <= 3 × interval`; else `stale`; missing file = `unknown`.
- **Health rule:** `healthy` if now - ts <= 3 × interval; else `stale`; missing file = `unknown`.
- **Contract:** every runtime (dogfood stub now; claude/codex/pi/opencode in Phase 3)
MUST emit the heartbeat. The protocol is file-based so it works for headless stubs and
full-screen TUIs alike (no `capture-pane` dependency).
@@ -56,15 +56,15 @@ observability and no safe way to watch a session.
## Acceptance criteria
- `mosaic fleet ps` shows all 5 live sessions on `mosaic-fleet` with correct
- mosaic fleet ps shows all 5 live sessions on `mosaic-fleet` with correct
pane/pid/idle and flags the dogfood **drift** (`canary-pi` runtime=pi but pane runs
`dogfood-agent.py`) and the **boot-enable** gap (active but disabled).
- Killing one agent's pane flips its row to dead/stale within one `interval`.
- `agent watch` shows live output and provably cannot type into the pane; detaching
- agent watch shows live output and provably cannot type into the pane; detaching
leaves the agent's window size unchanged.
- `agent send --verify` returns success on an accepting pane and non-zero on a wedged/draft pane.
- Quality gates green: `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, plus
`pnpm --filter @mosaicstack/mosaic test`.
- agent send --verify returns success on an accepting pane and non-zero on a wedged/draft pane.
- Quality gates green: pnpm typecheck, pnpm lint, pnpm format:check, plus
pnpm --filter @mosaicstack/mosaic test.
- Independent review passed; dogfood evidence captured against the live fleet.
## Test plan
@@ -72,18 +72,18 @@ observability and no safe way to watch a session.
- Unit/CLI specs in `packages/mosaic/src/commands/fleet.spec.ts` (and a new
`fleet-ps`/`watch`/`send-verify` spec) using the injected `CommandRunner` to assert
exact tmux/systemd command construction and JSON shape (tenant+host present).
- Situational: run against the live `mosaic-fleet` fleet; capture `fleet ps` output,
a kill-and-detect cycle, a read-only `watch`, and a `send --verify` pass/fail pair.
- Situational: run against the live `mosaic-fleet` fleet; capture fleet ps output,
a kill-and-detect cycle, a read-only `watch`, and a send --verify pass/fail pair.
## Known limitations
- **Verify heuristic is best-effort:** `agent send --verify` uses a `>` -prefix draft
- **Verify heuristic is best-effort:** agent send --verify uses a > -prefix draft
heuristic that is specific to pi/claude TUIs. Draft detection for codex and opencode
TUIs is best-effort only; those runtimes may not use the same input-line indicator.
- **Pane-change check is the best Phase-2 signal; verify now polls up to a bounded
timeout:** `agent send --verify` captures a BEFORE snapshot, sends the message, then
timeout:** agent send --verify captures a BEFORE snapshot, sends the message, then
polls `capture-pane` every ~400 ms up to a configurable total timeout (default ~6 s,
controlled by `--verify-timeout <ms>`). On each poll it runs classifySendResult: if
controlled by --verify-timeout <ms>). On each poll it runs classifySendResult: if
the pane shows 'accepted' or 'draft' the loop exits immediately; while the result is
'unverifiable' (no pane change yet) it keeps polling. After the timeout with no
definitive result, it fails closed: exit 1 with "no pane change after send". This
@@ -92,15 +92,15 @@ observability and no safe way to watch a session.
requires a runtime acknowledgement (Phase-3 heartbeat-ack); the bounded pane-change
poll is the best signal available against an opaque TUI for Phase-2.
- **Blank AFTER capture fails closed:** Full-screen TUIs (claude, codex, opencode, pi)
render blank for `tmux capture-pane`. When the AFTER snapshot is empty, `send --verify`
render blank for tmux capture-pane. When the AFTER snapshot is empty, send --verify
returns non-zero with an "unverifiable" message rather than silently succeeding. This
is an intentional fail-closed design (FR-5).
- **`agent watch` uses a grouped viewer session:** `tmux attach -r` directly against the
agent session lets the viewer terminal shrink the agent's window. `agent watch` instead
- **agent watch uses a grouped viewer session:** tmux attach -r directly against the
agent session lets the viewer terminal shrink the agent's window. agent watch instead
creates a throwaway grouped session (`tmux new-session -d -t '=<agent>' -s
'<agent>-watch-<pid>'`), attaches read-only to that session, and kills it on detach.
The grouped session shares the agent's windows but has independent sizing, so the
agent's window is never affected. `tmux attach` is still interactive and requires
agent's window is never affected. tmux attach is still interactive and requires
inherited stdio; the `interactiveRunner` handles TTY passthrough.
## Surfaces & parity (MVP-X1)