feat(fleet): Phase-2 observability — fleet ps + watch + send verify #579

Merged
jason.woltje merged 9 commits from feat/fleet-observability into main 2026-06-21 04:23:52 +00:00
2 changed files with 41 additions and 41 deletions
Showing only changes of commit ddeb200fdf - Show all commits

View File

@@ -33,7 +33,7 @@ observability and no safe way to watch a session.
## Functional requirements
| ID | Requirement |
|---|---|
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FR-1 | `mosaic fleet ps [--json]` prints one row per roster agent joining: name · tenant · host · runtime · systemd(active/enabled) · pane(alive/dead) · pid · idle · **last-heartbeat age** · **drift** flag (roster runtime ≠ actual pane command) · **boot-enable** warning (active but `UnitFileState=disabled`). |
| FR-2 | **Heartbeat protocol v1** (see below); `dogfood-agent.py` implements the responder. `fleet ps` issues probes (or reads last-seen) and reports health per FR-1. |
| FR-3 | `mosaic agent watch <name>` opens a **read-only** view of the pane (grouped session or `tmux attach -r`) that cannot send keystrokes and does not shrink the agent's window. |

View File

@@ -22,11 +22,11 @@ workstream makes it an official, observable, multi-tenant Mosaic Stack capabilit
The Fleet has a **dual role**, and that is the point:
- **As product** — a multi-tenant agent-fleet capability of Mosaic Stack (this workstream).
- **As means of production** — the orchestrator/worker fleet that *actually builds the
entire MVP* (federation W1, webUI, TUI, CLI, and the Fleet itself).
- **As means of production** — the orchestrator/worker fleet that _actually builds the
entire MVP_ (federation W1, webUI, TUI, CLI, and the Fleet itself).
We are **building the system that builds the system.** Every other MVP workstream is
delivered *by* the fleet, so fleet observability and control are not merely product
delivered _by_ the fleet, so fleet observability and control are not merely product
features — they are the **operational floor of the whole delivery effort**. If we cannot
see and steer the agents, we cannot trust what they ship. This is why Phase 2
(observability) leads: it is the instrument panel for the factory, dogfooded on the live
@@ -42,7 +42,7 @@ those gates.
The Fleet inherits — does not re-invent — the MVP's hard requirements:
| MVP req | What it means for the Fleet |
|---|---|
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| MVP-X1 three-surface parity | fleet observability/control reachable via **CLI + TUI + webUI** (CLI first; webUI is required for parity, not optional) |
| MVP-X2 multi-tenant isolation | one tenant = one **Linux uid** (own `systemd --user`, socket, `~/.config/mosaic`); no cross-tenant leakage |
| MVP-X3 auth (BetterAuth/SSO) | operator→fleet and cross-host views are auth-gated through the platform's existing auth |
@@ -56,7 +56,7 @@ The Fleet inherits — does not re-invent — the MVP's hard requirements:
One **definition** is the source of truth; the **session** is how it runs.
| Layer | Owner | Phase-2 reality | Destination |
|---|---|---|---|
| -------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| **Definition + identity + auth** | gateway / `mosaic-as` (scoped tokens, #541) | `roster.yaml` (tenant-tagged) | one definition; `mosaic agent --new` materializes it |
| **Tenancy boundary** | **Linux uid per tenant** (linger, own `systemd --user`, own socket, own `~/.config/mosaic`) | one tenant: `jarvis` = tenant zero | uid-per-tenant; federation aggregates across hosts |
| **Runtime** | per-tenant tmux session on isolated socket | dogfood stub sessions (live now on `mosaic-factory`) | claude/codex/pi/opencode TUIs |
@@ -69,7 +69,7 @@ One **definition** is the source of truth; the **session** is how it runs.
The AI-guide law stands: one accountable **orchestrator**, isolated **workers** that
stop at PR-open, the serialized **gate chain** (independent review → green CI →
diff-sanity → squash-merge → verify), **decide-and-inform** cadence, and a durable
**board** so missions survive session death. The Fleet is the infrastructure *under*
**board** so missions survive session death. The Fleet is the infrastructure _under_
this model. See `mosaicstack-aiguide` whitepapers 01 (inter-agent comms) and 03
(orchestration model) for the rationale.
@@ -85,7 +85,7 @@ Every artifact, starting Phase 2, MUST:
## Observation model
| Verb | Behavior |
|---|---|
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
| `mosaic fleet ps` | one table joining systemd + tmux + process + idle + last-heartbeat, with drift + boot-enable flags |
| `mosaic agent watch <name>` | **read-only** join (grouped session / `-r`), no resize tyranny, no keystrokes |
| `mosaic agent attach <name>` | explicit interactive takeover (the only path that can type) |
@@ -99,7 +99,7 @@ Every artifact, starting Phase 2, MUST:
## Phased roadmap
| Phase | Outcome | Status |
|---|---|---|
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| 01 | tmux PoC, hardening, published CLI v0.0.34 (#565#568) | ✅ done |
| **2 — Observability** | `fleet ps` (host+tenant aware join), heartbeat protocol + dogfood stub answers it, `agent watch` (read-only), `agent send --verify` receipts | ▶ now |
| 3 — Real runtimes | claude/codex/pi/opencode answer heartbeat; **hybrid lifecycle** (core always-on: orchestrator+reviewer; ephemeral workers per lane) | planned |