mosaicstack/stack
1
0
Fork 0
Code Issues 81 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

test(fleet): close documentation code grammar

Browse Source 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Jarvis
2026-07-17 18:39:05 -05:00
parent 862dbd5204
commit bfa08e9651
36 changed files with 966 additions and 1022 deletions
Download Patch File Download Diff File Expand all files Collapse all files

91
docs/fleet/north-star.md
View File

@@ -44,9 +44,9 @@ The Fleet inherits — does not re-invent — the MVP's hard requirements:
| MVP req | What it means for the Fleet |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| MVP-X1 three-surface parity | fleet observability/control reachable via **CLI + TUI + webUI** (CLI first; webUI is required for parity, not optional) |
| MVP-X2 multi-tenant isolation | one tenant = one **Linux uid** (own `systemd --user`, socket, `~/.config/mosaic`); no cross-tenant leakage |
| MVP-X2 multi-tenant isolation | one tenant = one **Linux uid** (own systemd --user, socket, ~/.config/mosaic); no cross-tenant leakage |
| MVP-X3 auth (BetterAuth/SSO) | operator→fleet and cross-host views are auth-gated through the platform's existing auth |
| MVP-X4 quality gates | `pnpm typecheck`/`lint`/`format:check` green before any push |
| MVP-X4 quality gates | pnpm typecheck/`lint`/`format:check` green before any push |
| MVP-X5 federated topology | cross-host fleet visibility rides the **federation** boundary (W1), not a bespoke broker |
| MVP-X6 OTEL tracing | heartbeats, sends, and lifecycle events emit spans; `traceparent` crosses the federation boundary |
| MVP-X7 trunk merge | branch from `main`, squash-merge via PR, never push to `main` |
@@ -55,20 +55,20 @@ The Fleet inherits — does not re-invent — the MVP's hard requirements:
One **definition** is the source of truth; the **session** is how it runs.
| Layer | Owner | Phase-2 reality | Destination |
| -------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Definition + identity + auth** | gateway / `mosaic-as` (scoped tokens, #541) | `roster.yaml` (tenant-tagged) | one definition; `mosaic agent --new` materializes it |
| **Tenancy boundary** | **Linux uid per tenant** (linger, own `systemd --user`, own socket, own `~/.config/mosaic`) | one tenant: `jarvis` = tenant zero | uid-per-tenant; federation aggregates across hosts |
| **Runtime** | per-tenant tmux session on isolated socket | dogfood stub sessions (live now on `mosaic-factory`) | claude/codex/pi/opencode TUIs |
| **Liveness** | **heartbeat protocol** every runtime answers | protocol defined + dogfood stub answers it | all runtimes answer; "healthy" ≠ "pane alive" |
| **Observation** | read-only `watch` (native tmux) + `pipe-pane` stream | CLI `watch`/`ps`; explicit opt-in `attach` for control | + auth-gated webUI streams |
| **Control plane** | **federation** across hosts × tenants | records already carry `tenant_id` + `host` | federated gateways expose fleet state; webUI in Phase 5 |
| **Central register** | Postgres `fleet` schema (gateway instance); access via gateway API only | _none in PoC_ (files + `roster.yaml`) | agents, missions, tasks, heartbeats, spend — single network-accessible SSOT; docs = generated projections |
| **Budget / spend governance** | **per-tenant budget policy** ingested by the orchestrator + routing layer | none today (spend is unmetered) | usage-vs-limit feedback ingested; spend auto-paced to the limit window; per-provider/per-account/concurrency/API-$ budgets enforced |
| Layer | Owner | Phase-2 reality | Destination |
| -------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Definition + identity + auth** | gateway / `mosaic-as` (scoped tokens, #541) | `roster.yaml` (tenant-tagged) | one definition; mosaic agent --new materializes it |
| **Tenancy boundary** | **Linux uid per tenant** (linger, own systemd --user, own socket, own ~/.config/mosaic) | one tenant: `jarvis` = tenant zero | uid-per-tenant; federation aggregates across hosts |
| **Runtime** | per-tenant tmux session on isolated socket | dogfood stub sessions (live now on `mosaic-factory`) | claude/codex/pi/opencode TUIs |
| **Liveness** | **heartbeat protocol** every runtime answers | protocol defined + dogfood stub answers it | all runtimes answer; "healthy" ≠ "pane alive" |
| **Observation** | read-only `watch` (native tmux) + `pipe-pane` stream | CLI `watch`/`ps`; explicit opt-in `attach` for control | + auth-gated webUI streams |
| **Control plane** | **federation** across hosts × tenants | records already carry `tenant_id` + `host` | federated gateways expose fleet state; webUI in Phase 5 |
| **Central register** | Postgres `fleet` schema (gateway instance); access via gateway API only | _none in PoC_ (files + `roster.yaml`) | agents, missions, tasks, heartbeats, spend — single network-accessible SSOT; docs = generated projections |
| **Budget / spend governance** | **per-tenant budget policy** ingested by the orchestrator + routing layer | none today (spend is unmetered) | usage-vs-limit feedback ingested; spend auto-paced to the limit window; per-provider/per-account/concurrency/API-$ budgets enforced |
> **PoC socket hygiene:** the PoC fleet runs on the **default tmux socket** (no `-L`).
> The named production-isolation socket is **`mosaic-fleet`** (matches the product brand);
> an absent roster `socket_name` means the default socket everywhere (spawn, `fleet ps`,
> an absent roster `socket_name` means the default socket everywhere (spawn, fleet ps,
> onboarding cheat-sheet). The legacy dogfood canary still runs on the old `mosaic-factory`
> socket pending migration.
@@ -177,22 +177,22 @@ routing flow**, **concurrency** (the spend multiplier), and **hard API-token $-l
are enforced at the orchestrator + routing boundary, not inside individual workers (a worker never
decides its own budget — see delegation discipline).
**Budget CLI UX (#558):** `mosaic budget set --reset-at` sets the window reset; reset-datetimes
**Budget CLI UX (#558):** mosaic budget set --reset-at sets the window reset; reset-datetimes
carry **confidence tags** (`user` / `provider` / `estimated` / `unknown`); and **urgency/criticality
is a dispatch-gate modifier** — high-urgency work may override even-spread pacing **within
authorization**. (Also feeds the budgeting workstream, not only this doc.)
## Observation model
| Verb | Behavior |
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
| `mosaic fleet ps` | one table joining systemd + tmux + process + idle + last-heartbeat, with drift + boot-enable flags |
| `mosaic agent watch <name>` | **read-only** join (grouped session / `-r`), no resize tyranny, no keystrokes |
| `mosaic agent attach <name>` | explicit interactive takeover (the only path that can type) |
| `mosaic agent send <name> --verify` | confirms message **accepted**, not merely keystroke-injected |
| Verb | Behavior |
| --------------------------------- | -------------------------------------------------------------------------------------------------- |
| mosaic fleet ps | one table joining systemd + tmux + process + idle + last-heartbeat, with drift + boot-enable flags |
| mosaic agent watch <name> | **read-only** join (grouped session / `-r`), no resize tyranny, no keystrokes |
| mosaic agent attach <name> | explicit interactive takeover (the only path that can type) |
| mosaic agent send <name> --verify | confirms message **accepted**, not merely keystroke-injected |
> Why the current PoC blocks observation: sessions live on the isolated `mosaic-factory`
> socket (invisible to default `tmux ls`), the only sanctioned read is `capture-pane`
> socket (invisible to default tmux ls), the only sanctioned read is `capture-pane`
> (blank for full-screen TUIs), and `attach` is read-write + resizes the session. The
> verbs above restore "join and observe" safely.
@@ -214,7 +214,7 @@ compromised pane cannot corrupt or exfiltrate the register.
| Layer | Responsibility | Implementation |
| ---------------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Register** | Source of truth: agents, missions, tasks, heartbeats, spend | Postgres `fleet` schema — existing stack instance (`@mosaicstack/db`) |
| **Access** | Typed, auth-gated API | Gateway `fleet/*` routes |
| **Access** | Typed, auth-gated API | Gateway fleet/\* routes |
| **Dispatcher** | Brief classification, BOD review, planning/coding/review/test/deploy sequencing + gates → fleet task dispatch | **forge pipeline engine** (`runPipeline`/`resumePipeline`, brief classifier, BOD) **+ thin `forge-exec` adapter → `agent-send.sh`**; NOT a new daemon — forge is reused, only stage→agent dispatch is new |
| **Orchestrator (Mos)** | Goals, missions, judgment, user/PA interface | Context-light; sets intent → re-engages only for decisions |
@@ -236,7 +236,7 @@ role implementation.
`docs/TASKS.md` and `MISSION-MANIFEST.md` are **generated projections** of the DB,
not hand-maintained. The dispatcher (or a scheduled job) renders Markdown from
`fleet.*` tables and commits the output. DB is authoritative; docs are for human
fleet.\* tables and commits the output. DB is authoritative; docs are for human
reference.
### Spend
@@ -266,13 +266,13 @@ re-evaluate if isolation or write-volume demands it.
## Phased roadmap
| Phase | Outcome | Status |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| 0–1 | tmux PoC, hardening, published CLI v0.0.34 (#565–#568) | ✅ done |
| **2 — Observability** | `fleet ps` (host+tenant aware join), heartbeat protocol + dogfood stub answers it, `agent watch` (read-only), `agent send --verify` receipts | ▶ now |
| 3 — Real runtimes | claude/codex/pi/opencode answer heartbeat; **hybrid lifecycle** (core always-on: **orchestrator + enhancer**; ephemeral workers per lane) | planned |
| 4 — Unified definition | one agent schema in gateway; `mosaic agent --new` → materialized per-tenant session; uid-tenant provisioning; **`fleet` schema migration + `forge-exec` TaskExecutor adapter (forge → `agent-send.sh`)** | planned |
| 5 — Control plane | federation-backed cross-host × cross-tenant fleet view; **webUI** (surface chosen then) for MVP-X1 parity; **central register live (spend ledger, docs-as-projections, multi-host Kanban)** | planned |
| Phase | Outcome | Status |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| 0–1 | tmux PoC, hardening, published CLI v0.0.34 (#565–#568) | ✅ done |
| **2 — Observability** | fleet ps (host+tenant aware join), heartbeat protocol + dogfood stub answers it, agent watch (read-only), agent send --verify receipts | ▶ now |
| 3 — Real runtimes | claude/codex/pi/opencode answer heartbeat; **hybrid lifecycle** (core always-on: **orchestrator + enhancer**; ephemeral workers per lane) | planned |
| 4 — Unified definition | one agent schema in gateway; mosaic agent --new → materialized per-tenant session; uid-tenant provisioning; **`fleet` schema migration + `forge-exec` TaskExecutor adapter (forge → `agent-send.sh`)** | planned |
| 5 — Control plane | federation-backed cross-host × cross-tenant fleet view; **webUI** (surface chosen then) for MVP-X1 parity; **central register live (spend ledger, docs-as-projections, multi-host Kanban)** | planned |
## Decisions of record (2026-06-20, with Jason)
@@ -285,9 +285,9 @@ re-evaluate if isolation or write-volume demands it.
- Delivery: **CLI-first now**, dogfood against the live stub fleet; webUI deferred to Phase 5.
- Runtimes: fleet agents default to **Codex / pi-on-Codex**; **Claude is reserved for Claude
Code only** (avoid alternate-harness API pricing). Validated durable recipe:
`mosaic yolo pi --model openai-codex/gpt-5.5:high`. Durable detached launch requires the
mosaic yolo pi --model openai-codex/gpt-5.5:high. Durable detached launch requires the
runtime-bin on PATH (baked into the pane command) + boot-survival (`enable` + linger),
which `fleet init` should automate.
which fleet init should automate.
## Decisions of record (2026-06-22, with Jason)
@@ -304,19 +304,18 @@ re-evaluate if isolation or write-volume demands it.
- **Session context cap = 200k tokens (GLOBAL to all Claude sessions):** Claude Code sessions are
capped at a **max 200k-token context window**. Long-running sessions extended toward 1M tokens
have proven **worse in practice** (degraded steering, off-plan divergence); 200k is the standard.
**Enforcement split:** the _window_ lives in **`~/.claude/settings.json`** (host-global) as
`"autoCompactWindow": 200000` + `"autoCompactEnabled": true`; the _1M-disable_ lives in **launch
**Enforcement split:** the _window_ lives in **~/.claude/settings.json** (host-global) as
"autoCompactWindow": 200000 + "autoCompactEnabled": true; the _1M-disable_ lives in **launch
ENV** (`CLAUDE_CODE_DISABLE_1M_CONTEXT=1`, plus `CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000`) wherever
a `[1m]` model can be selected (`mos-claude.service` + the fleet Claude launcher), so every Claude
a [1m] model can be selected (`mos-claude.service` + the fleet Claude launcher), so every Claude
agent is capped at spawn. (settings = window; env = 1M-disable.)
- **Worker context bound (#8):** workers are kept context-bounded via the **ephemeral-per-lane
lifecycle + native compaction**, not via the 200k knob. The explicit `autoCompactWindow` 200k knob
**stays Claude-specific** — the _principle_ (bounded context) extends to workers, the _knob_ does not.
- **Orchestrator delegation discipline:** the orchestrator **delegates all delivery work** to
subagents / workflows / ultracode / coder agents and confines its own context to \*\*orchestration
- the personal-assistant lane\*\*. Keeping delivery out of the orchestrator's window keeps its
context unpolluted and measurably reduces off-plan divergence. The orchestrator coordinates and
decides; it does not implement.
subagents / workflows / ultracode / coder agents and confines its own context to the personal-assistant
lane. Keeping delivery out of the orchestrator's window keeps its context unpolluted and measurably
reduces off-plan divergence. The orchestrator coordinates and decides; it does not implement.
- **Budget governance is fleet doctrine:** token/API-dollar budgeting is a first-class fleet concern
(see "Budget & token governance"). OAuth-sub usage-vs-limit feedback is ingested per account, spend
is **auto-paced EVEN-SPREAD over remaining time** (rapid/overspend only on explicit authorization),
@@ -344,7 +343,7 @@ re-evaluate if isolation or write-volume demands it.
### Control plane & central register
- **Store:** Postgres (existing stack instance, dedicated `fleet` schema via `@mosaicstack/db`). SQLite rejected: (1) it is a local file — structurally incompatible with a multi-host fleet; (2) concurrent multi-agent writes caused repeated corruption in Hermes. "SQLite + access service" rejected as reinventing a DB server badly; "LLM agent gating DB access" rejected as slow, expensive, and a single point of failure.
- **Access:** gateway API only (`apps/gateway`, `fleet/*` routes). No raw DB credentials in any agent/dispatcher pane — directly mitigates the tmux attack-surface concern.
- **Access:** gateway API only (`apps/gateway`, fleet/\* routes). No raw DB credentials in any agent/dispatcher pane — directly mitigates the tmux attack-surface concern.
- **Dispatcher = forge (reuse, not a new build):** the dispatcher IS `@mosaicstack/forge`'s pipeline engine (`runPipeline`/`resumePipeline` + brief classifier + BOD persona loader), a fully-implemented software-factory pipeline (brief → BOD review → 3 planning stages → coding → review/remediation → testing → deploy). We do **not** design/build a new dispatcher and do **not** re-implement sequencing, gate logic, or brief classification. The only new fleet-owned piece is a thin **`forge-exec` TaskExecutor adapter** (suggested package `packages/forge-exec`) mapping a `ForgeTask` → `agent-send.sh` dispatch to a named fleet agent — forge's single missing piece. It is tracked as a Gitea issue and built **post-PoC** (not now).
- **Register backs forge:** the Postgres `fleet` register is genuinely new (neither forge nor the fleet has cross-project state). It BACKS forge's pipeline state (durable `resumePipeline`, cross-host) plus cross-project missions/tasks/Kanban.
- **'board' role = forge BOD:** the north-star role-library 'board' role IS forge's Board-of-Directors — reused, not reinvented.
@@ -357,9 +356,9 @@ re-evaluate if isolation or write-volume demands it.
- **Per-agent model switch (operator-configurable, NOT a global lock):** model selection is
**per-agent**, never a host-global pin. Claude sessions MUST NOT be locked to a single model in
`~/.claude/settings.json`; each agent chooses its model independently. The plumbing already exists —
roster `model_hint` → `MOSAIC_AGENT_MODEL` → `start-agent-session.sh` appends `--model <hint>` to that
agent's harness (claude or pi); settable today via `mosaic fleet add|edit <agent> --model <hint>`.
~/.claude/settings.json; each agent chooses its model independently. The plumbing already exists —
roster `model_hint` → `MOSAIC_AGENT_MODEL` → `start-agent-session.sh` appends --model <hint> to that
agent's harness (claude or pi); settable today via mosaic fleet add|edit <agent> --model <hint>.
**North-star target:** surface this as a **per-agent model switch in the webUI** (with CLI/TUI parity
per MVP-X1) — read the roster, expose a per-agent model dropdown, write `model_hint` back, and restart
that one agent to apply. Unset = inherit the harness default. This **composes with** the budget
@@ -385,7 +384,7 @@ re-evaluate if isolation or write-volume demands it.
self-hosted homeserver (Conduit default, Synapse alt). Matrix is named here as the strategic
future transport — peer to tmux/Discord, not superseded by them.
- **tmux fleet attack-surface hardening.** Many always-on tmux sessions are an attack surface;
`tmux send-keys` / socket access could enable malicious action against agents directly.
tmux send-keys / socket access could enable malicious action against agents directly.
Mitigations to build toward: socket ownership/perms, per-tenant socket isolation (already an
invariant), authenticated `agent-send`, and an audit of who can write to any pane. **Post-MVP
unless a P0 surfaces.** The control-plane register reinforces this (gateway-API access = no raw
@@ -418,9 +417,9 @@ re-evaluate if isolation or write-volume demands it.
---
> **Release procedure (drift re-capture, 2026-06-22):** `mosaic update` only propagates new fleet
> **Release procedure (drift re-capture, 2026-06-22):** mosaic update only propagates new fleet
> commands when the **CLI version is bumped** — without a version bump, fleet command changes never
> reach installed hosts. The release/version-bump procedure (bump → publish → `mosaic update`
> reach installed hosts. The release/version-bump procedure (bump → publish → mosaic update
> [→ `--relaunch`]) must be documented so fleet changes actually land. (Also feeds the budgeting
> workstream.)
>