mosaicstack/stack
1
0
Fork 0
Code Issues 25 Pull Requests 2 Actions Packages Projects Releases 14 Wiki Activity

feat(fleet): Phase-2 observability — fleet ps + watch + send verify (#579)
All checks were successful
ci/woodpecker/push/ci Pipeline was successful
Details
ci/woodpecker/push/publish Pipeline was successful
Details

Browse Source
This commit was merged in pull request #579.
This commit is contained in:
jason.woltje
2026-06-21 04:23:51 +00:00
parent 5118be74cb
commit af2eede7a9
6 changed files with 2041 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

109
docs/fleet/PRD.md Normal file
View File

@@ -0,0 +1,109 @@
# PRD — Fleet Phase 2: Operator Observability
> **Workstream:** W-FLEET under `mvp-20260312` · **Phase:** 2
> **North star:** [docs/fleet/north-star.md](./north-star.md)
> **Source umbrella PRD:** [docs/PRD.md](../PRD.md) (Mosaic Stack v0.1.0)
> **Tracks task:** `fleet-observability-1` — restore operator observability into fleet agent sessions.
## Problem
The durable tmux fleet runs on the isolated `mosaic-factory` socket. That isolation
(which protects the operator's default tmux) makes the fleet **invisible** to default
tooling, and truth is split across three planes no single command joins — systemd
(`systemctl --user`), tmux (`-L mosaic-factory`), and the process tree (`pstree`).
`agent tail` (`capture-pane`) returns **blank for full-screen TUIs**, and `agent send`
confirms only keystroke injection, not acceptance. Net: the operator has near-zero
observability and no safe way to watch a session.
## Goals
1. One command shows the **whole fleet's** real state, joining all three planes.
2. **Liveness is truthful**: healthy = answered a heartbeat, not "pane alive".
3. The operator can **watch** any session read-only without disrupting it.
4. `send` reports **delivered-and-accepted**, not just injected.
5. Every record/address carries **`tenant_id` + `host`** (zero foreclosure for multi-tenant/multi-host).
## Non-goals (this phase)
- No webUI (Phase 5; rides federation for cross-host).
- No `fleetd` daemon or persistent history store.
- No real-runtime swap (Phase 3) — instrument the live **dogfood stub** fleet.
- No cross-host aggregation yet (addressing is host-tagged but queries stay local).
## Functional requirements
| ID | Requirement |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FR-1 | `mosaic fleet ps [--json]` prints one row per roster agent joining: name · tenant · host · runtime · systemd(active/enabled) · pane(alive/dead) · pid · idle · **last-heartbeat age** · **drift** flag (roster runtime ≠ actual pane command) · **boot-enable** warning (active but `UnitFileState=disabled`). |
| FR-2 | **Heartbeat protocol v1** (see below); `dogfood-agent.py` implements the responder. `fleet ps` issues probes (or reads last-seen) and reports health per FR-1. |
| FR-3 | `mosaic agent watch <name>` opens a **read-only** view of the pane (grouped session or `tmux attach -r`) that cannot send keystrokes and does not shrink the agent's window. |
| FR-4 | `mosaic agent attach <name>` remains the **explicit** interactive-takeover path (separate verb, documented as the only one that can type). |
| FR-5 | `mosaic agent send <name> --verify` confirms the message was **accepted** (not left as an unsubmitted draft) and returns non-zero if delivery cannot be verified. |
| FR-6 | All structured output (`--json`) includes `tenant_id` and `host` fields. |
## Heartbeat protocol v1
- **Probe:** operator/`fleet ps` writes a sentinel line to the agent's input or a
well-known per-agent heartbeat file path `~/.config/mosaic/fleet/run/<agent>.hb`.
- **Response:** the runtime updates `<agent>.hb` with `ts=<iso8601> pid=<pid> status=<ok|busy>`
on a fixed interval (default 15s) and on demand when probed.
- **Health rule:** `healthy` if `now - ts <= 3 × interval`; else `stale`; missing file = `unknown`.
- **Contract:** every runtime (dogfood stub now; claude/codex/pi/opencode in Phase 3)
MUST emit the heartbeat. The protocol is file-based so it works for headless stubs and
full-screen TUIs alike (no `capture-pane` dependency).
- `ASSUMPTION:` file-based heartbeat (vs in-pane echo) — chosen because it is TUI-safe and
uid-scoped, fitting per-tenant isolation. Open to an OTEL-span variant in Phase 3 (MVP-X6).
## Acceptance criteria
- `mosaic fleet ps` shows all 5 live sessions on `mosaic-factory` with correct
pane/pid/idle and flags the dogfood **drift** (`canary-pi` runtime=pi but pane runs
`dogfood-agent.py`) and the **boot-enable** gap (active but disabled).
- Killing one agent's pane flips its row to dead/stale within one `interval`.
- `agent watch` shows live output and provably cannot type into the pane; detaching
leaves the agent's window size unchanged.
- `agent send --verify` returns success on an accepting pane and non-zero on a wedged/draft pane.
- Quality gates green: `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, plus
`pnpm --filter @mosaicstack/mosaic test`.
- Independent review passed; dogfood evidence captured against the live fleet.
## Test plan
- Unit/CLI specs in `packages/mosaic/src/commands/fleet.spec.ts` (and a new
`fleet-ps`/`watch`/`send-verify` spec) using the injected `CommandRunner` to assert
exact tmux/systemd command construction and JSON shape (tenant+host present).
- Situational: run against the live `mosaic-factory` fleet; capture `fleet ps` output,
a kill-and-detect cycle, a read-only `watch`, and a `send --verify` pass/fail pair.
## Known limitations
- **Verify heuristic is best-effort:** `agent send --verify` uses a `>` -prefix draft
heuristic that is specific to pi/claude TUIs. Draft detection for codex and opencode
TUIs is best-effort only; those runtimes may not use the same input-line indicator.
- **Pane-change check is the best Phase-2 signal; verify now polls up to a bounded
timeout:** `agent send --verify` captures a BEFORE snapshot, sends the message, then
polls `capture-pane` every ~400 ms up to a configurable total timeout (default ~6 s,
controlled by `--verify-timeout <ms>`). On each poll it runs classifySendResult: if
the pane shows 'accepted' or 'draft' the loop exits immediately; while the result is
'unverifiable' (no pane change yet) it keeps polling. After the timeout with no
definitive result, it fails closed: exit 1 with "no pane change after send". This
eliminates false 'unverifiable' failures for slow/loaded TUIs that were previously
caused by the old fixed 300 ms single-capture. Definitive acceptance ultimately
requires a runtime acknowledgement (Phase-3 heartbeat-ack); the bounded pane-change
poll is the best signal available against an opaque TUI for Phase-2.
- **Blank AFTER capture fails closed:** Full-screen TUIs (claude, codex, opencode, pi)
render blank for `tmux capture-pane`. When the AFTER snapshot is empty, `send --verify`
returns non-zero with an "unverifiable" message rather than silently succeeding. This
is an intentional fail-closed design (FR-5).
- **`agent watch` uses a grouped viewer session:** `tmux attach -r` directly against the
agent session lets the viewer terminal shrink the agent's window. `agent watch` instead
creates a throwaway grouped session (`tmux new-session -d -t '=<agent>' -s
'<agent>-watch-<pid>'`), attaches read-only to that session, and kills it on detach.
The grouped session shares the agent's windows but has independent sizing, so the
agent's window is never affected. `tmux attach` is still interactive and requires
inherited stdio; the `interactiveRunner` handles TTY passthrough.
## Surfaces & parity (MVP-X1)
CLI lands this phase. TUI surface follows in the `packages/mosaic` wizard; webUI in
Phase 5 via federation. PRD records the parity debt explicitly so it is not lost.

27
docs/fleet/TASKS.md Normal file
View File

@@ -0,0 +1,27 @@
# Tasks — W-FLEET (Fleet) Phase 2: Observability
> Workstream task file for the Fleet. Single-writer: Fleet workstream lead (orchestrator).
> Workers read but never modify. This is **not** the MVP rollup (`docs/TASKS.md`) — a
> rollup row is proposed to the MVP orchestrator, not written here.
>
> Mission: `mvp-20260312` · PRD: [docs/fleet/PRD.md](./PRD.md) · North star: [docs/fleet/north-star.md](./north-star.md)
> Status: `not-started` | `in-progress` | `done` | `blocked` | `failed`
| id | status | description | depends_on | agent | pr | notes |
| ------------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | --------------------- | ----------- | --- | ----------------------------------------------------------------------------------------------------------------------------- |
| FLEET-OBS-000 | done | Plan: north-star + Phase-2 PRD + workstream scaffolding | — | lead | — | persisted 2026-06-20 on `feat/fleet-observability` |
| FLEET-OBS-001 | done | Heartbeat protocol v1 spec finalized in PRD + framework doc | FLEET-OBS-000 | lead | — | file-based `~/.config/mosaic/fleet/run/<agent>.hb`; spec in PRD |
| FLEET-OBS-002 | in-progress | Implement heartbeat responder in `dogfood-agent.py` | FLEET-OBS-001 | fleet-coder | — | dispatched to ad-hoc `mosaic yolo` fleet agent (dogfood) |
| FLEET-OBS-003 | done | `mosaic fleet ps` — join systemd+tmux+proc+idle+heartbeat; tenant+host tagged; drift + boot-enable flags; `--json` | FLEET-OBS-001 | worker | — | commit ab47831; LIVE-verified on mosaic-factory; caught canary-pi DRIFT + BOOT-ENABLE. Polish: idleSeconds parse returns null |
| FLEET-OBS-004 | done | `mosaic agent watch <name>` — read-only join (no resize, no keystrokes) | FLEET-OBS-000 | worker | — | `attach -r`; verb wired |
| FLEET-OBS-005 | done | `mosaic agent send --verify` — delivery/acceptance receipt | FLEET-OBS-000 | worker | — | --verify flag; draft-heuristic verify |
| FLEET-OBS-006 | done | CLI specs for ps/watch/send-verify (tenant+host shape, command construction) | FLEET-OBS-003,004,005 | worker | — | 62 tests green (31 new); re-verified by lead |
| FLEET-OBS-007 | not-started | Framework doc: fleet observability guide + verbs | FLEET-OBS-003,004,005 | lead | — | `docs/guides/` or `framework/tools/.../README` |
| FLEET-OBS-008 | not-started | Independent review + dogfood verification on live fleet | FLEET-OBS-002..007 | reviewer | — | author ≠ reviewer; capture evidence in scratchpad |
| FLEET-OBS-009 | not-started | Open PR → green CI (queue guard) → squash-merge → close `fleet-observability-1` | FLEET-OBS-008 | lead | — | trunk merge; no direct push to main |
## Proposed MVP rollup row (for the MVP orchestrator — not written by this workstream)
```
| W-FLEET | in-progress | Fleet (agent-session execution layer) | Phase 2/5 | docs/fleet/TASKS.md | observability dogfooded on live stub fleet; control plane rides federation (W1) |
```

128
docs/fleet/north-star.md Normal file
View File

@@ -0,0 +1,128 @@
# Mosaic Fleet — North Star
> **Workstream:** W-FLEET (Fleet) under mission `mvp-20260312`
> **Umbrella:** [docs/MISSION-MANIFEST.md](../MISSION-MANIFEST.md) · [docs/PRD.md](../PRD.md) (Mosaic Stack v0.1.0)
> **Status:** doctrine — authored 2026-06-20. Owner of this file: Fleet workstream lead.
> This document does **not** modify the MVP rollup; a rollup row is proposed, not written here.
## Vision
A **customizable, multi-tenant fleet of always-on AI agents** — each defined by role,
materialized as a durable, joinable runtime session, coordinated by the proven
orchestrator/worker model, and observable end-to-end across hosts. Coding today;
finance, analytics, research as roster entries tomorrow — same primitives, different
roster. The fleet is the **agent-session execution layer** of the Mosaic Stack MVP:
the thing federation makes reachable across hosts and the webUI/TUI/CLI make visible.
The USC tmux PoC (durable sessions + `agent-send` comms) proved the model. This
workstream makes it an official, observable, multi-tenant Mosaic Stack capability.
## The Fleet as means of production (bootstrapping)
The Fleet has a **dual role**, and that is the point:
- **As product** — a multi-tenant agent-fleet capability of Mosaic Stack (this workstream).
- **As means of production** — the orchestrator/worker fleet that _actually builds the
entire MVP_ (federation W1, webUI, TUI, CLI, and the Fleet itself).
We are **building the system that builds the system.** Every other MVP workstream is
delivered _by_ the fleet, so fleet observability and control are not merely product
features — they are the **operational floor of the whole delivery effort**. If we cannot
see and steer the agents, we cannot trust what they ship. This is why Phase 2
(observability) leads: it is the instrument panel for the factory, dogfooded on the live
fleet that is, recursively, building Mosaic Stack.
The discipline that makes great power safe is the same gate chain the fleet enforces:
independent review before merge, green CI, honest completion, decide-and-inform cadence,
and no irreversible action without authority. The bootstrap is only as trustworthy as
those gates.
## Alignment with MVP cross-cutting requirements
The Fleet inherits — does not re-invent — the MVP's hard requirements:
| MVP req | What it means for the Fleet |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| MVP-X1 three-surface parity | fleet observability/control reachable via **CLI + TUI + webUI** (CLI first; webUI is required for parity, not optional) |
| MVP-X2 multi-tenant isolation | one tenant = one **Linux uid** (own `systemd --user`, socket, `~/.config/mosaic`); no cross-tenant leakage |
| MVP-X3 auth (BetterAuth/SSO) | operator→fleet and cross-host views are auth-gated through the platform's existing auth |
| MVP-X4 quality gates | `pnpm typecheck`/`lint`/`format:check` green before any push |
| MVP-X5 federated topology | cross-host fleet visibility rides the **federation** boundary (W1), not a bespoke broker |
| MVP-X6 OTEL tracing | heartbeats, sends, and lifecycle events emit spans; `traceparent` crosses the federation boundary |
| MVP-X7 trunk merge | branch from `main`, squash-merge via PR, never push to `main` |
## The stack — where every concern lives
One **definition** is the source of truth; the **session** is how it runs.
| Layer | Owner | Phase-2 reality | Destination |
| -------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| **Definition + identity + auth** | gateway / `mosaic-as` (scoped tokens, #541) | `roster.yaml` (tenant-tagged) | one definition; `mosaic agent --new` materializes it |
| **Tenancy boundary** | **Linux uid per tenant** (linger, own `systemd --user`, own socket, own `~/.config/mosaic`) | one tenant: `jarvis` = tenant zero | uid-per-tenant; federation aggregates across hosts |
| **Runtime** | per-tenant tmux session on isolated socket | dogfood stub sessions (live now on `mosaic-factory`) | claude/codex/pi/opencode TUIs |
| **Liveness** | **heartbeat protocol** every runtime answers | protocol defined + dogfood stub answers it | all runtimes answer; "healthy" ≠ "pane alive" |
| **Observation** | read-only `watch` (native tmux) + `pipe-pane` stream | CLI `watch`/`ps`; explicit opt-in `attach` for control | + auth-gated webUI streams |
| **Control plane** | **federation** across hosts × tenants | records already carry `tenant_id` + `host` | federated gateways expose fleet state; webUI in Phase 5 |
## Operating model (inherited, not reinvented)
The AI-guide law stands: one accountable **orchestrator**, isolated **workers** that
stop at PR-open, the serialized **gate chain** (independent review → green CI →
diff-sanity → squash-merge → verify), **decide-and-inform** cadence, and a durable
**board** so missions survive session death. The Fleet is the infrastructure _under_
this model. See `mosaicstack-aiguide` whitepapers 01 (inter-agent comms) and 03
(orchestration model) for the rationale.
## Invariants — "maximal vision, incremental delivery, zero foreclosure"
Every artifact, starting Phase 2, MUST:
1. Carry **`tenant_id` + `host`** in schema and message addressing — even with one of each today.
2. Treat **isolation socket ≠ invisibility** — anything isolated is surfaced by one command.
3. Define **healthy = answered a heartbeat within N seconds**, never just "pane alive".
4. Make **observation read-only by default**; control is an explicit, separate, opt-in verb.
## Observation model
| Verb | Behavior |
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
| `mosaic fleet ps` | one table joining systemd + tmux + process + idle + last-heartbeat, with drift + boot-enable flags |
| `mosaic agent watch <name>` | **read-only** join (grouped session / `-r`), no resize tyranny, no keystrokes |
| `mosaic agent attach <name>` | explicit interactive takeover (the only path that can type) |
| `mosaic agent send <name> --verify` | confirms message **accepted**, not merely keystroke-injected |
> Why the current PoC blocks observation: sessions live on the isolated `mosaic-factory`
> socket (invisible to default `tmux ls`), the only sanctioned read is `capture-pane`
> (blank for full-screen TUIs), and `attach` is read-write + resizes the session. The
> verbs above restore "join and observe" safely.
## Phased roadmap
| Phase | Outcome | Status |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| 0–1 | tmux PoC, hardening, published CLI v0.0.34 (#565–#568) | ✅ done |
| **2 — Observability** | `fleet ps` (host+tenant aware join), heartbeat protocol + dogfood stub answers it, `agent watch` (read-only), `agent send --verify` receipts | ▶ now |
| 3 — Real runtimes | claude/codex/pi/opencode answer heartbeat; **hybrid lifecycle** (core always-on: orchestrator+reviewer; ephemeral workers per lane) | planned |
| 4 — Unified definition | one agent schema in gateway; `mosaic agent --new` → materialized per-tenant session; uid-tenant provisioning | planned |
| 5 — Control plane | federation-backed cross-host × cross-tenant fleet view; **webUI** (surface chosen then) for MVP-X1 parity | planned |
## Decisions of record (2026-06-20, with Jason)
- Agent model: **config defines, session runs** (gateway = definition/identity/auth; tmux = runtime).
- Tenancy: **multi-tenant from the start**; isolation = **per-tenant Linux uid**.
- Health: **heartbeat required** (dogfood stub implements the protocol now).
- Lifecycle: **hybrid** — core always-on + ephemeral workers per lane.
- Observation: **read-only default, opt-in takeover**.
- Multi-host: **designed-for from day one**; control plane **rides federation (W1)**.
- Delivery: **CLI-first now**, dogfood against the live stub fleet; webUI deferred to Phase 5.
## Assumptions (veto-able)
- `ASSUMPTION:` first-class runtimes = claude, codex, pi, opencode; a "role" (analyst,
finance, researcher) = persona + skills + tools on top of a runtime, shipped as a
starter role library in the framework.
- `ASSUMPTION:` the cross-host control plane is the **federation** layer (W1), not a
separate `fleetd` daemon.
- `ASSUMPTION:` Fleet is workstream **W-FLEET** under `mvp-20260312`; a rollup row in
`docs/TASKS.md` and a workstream declaration in `MISSION-MANIFEST.md` are proposed to
the MVP orchestrator, not written by this workstream.