# RFC-001 — MACP: A Mosaic-Native, Matrix-Native Comms Layer - **Status:** DRAFT — for Team Lead → Orchestrator staffing - **Author:** MS-LEAD (reviewer identity `ms-lead-reviewer`) - **Sponsor / veto:** Jason (human lead) - **Date:** 2026-07-24 - **Program:** Mosaic Stack comms-evolution - **Supersedes backbone:** the Hermes MCP chat bridge (strangler-retired, see §9) - **Audience:** Team Leads, the Mosaic orchestrator, infra, and any harness maintainer (Claude Code / Codex / Pi / Goose) > This is a **design document**. No code ships from this RFC. It exists to be decomposed into missions (P1→P5, §10) with per-phase acceptance criteria. Where a claim is uncertain or needs live validation, it is flagged **[VERIFY]**. --- ## 0. TL;DR We are building a **Mosaic-native comms layer on Matrix**. We self-host a **Synapse** homeserver and register a privileged **Mosaic Application Service** (the "appservice") that the orchestrator controls. The appservice bulk-provisions one Matrix identity per agent-spin, creates and manages rooms, posts agent introductions, and tracks presence/liveness. Agents talk to it through a thin `packages/comms` client SDK. **tmux stays the P0 same-host fast path**; Matrix is the durable, presence-aware, federated layer above it, and **MACP** (the Mosaic Agent Comms Protocol) is the standard that says which path to use when. Federation is **per-site Synapse homeservers federated over TLS we control** — a direct answer to "the homelab agent went dark and took comms with it." We adopt **Buzz's patterns** (auto-detect/enroll, signed identity, unified event log, humans-and-agents on one surface) without adopting Buzz/Nostr as transport. We migrate off Hermes by the **strangler** pattern: stand native alongside, move channels as proven, retire at parity. The **first standalone shippable slice is presence** (P1). --- ## 1. Goals / Non-Goals ### 1.1 Goals - **G1 — Presence & liveness first.** A Team Lead must be able to answer "is my coordinator online, away, or dead?" in seconds, not by polling for 13 hours. Presence is the P1 slice and ships before anything else. - **G2 — A native backbone we own.** Replace *Hermes-as-backbone* with a self-hosted Synapse + a Mosaic-controlled appservice. External chat bridging becomes an optional edge, not the spine. - **G3 — Turnkey harness enrollment.** `mosaic enroll` auto-detects the harness and self-registers the agent via the appservice on spin. No hand-rolled per-bot identity juggling. - **G4 — A real protocol (MACP v1).** Structured, versioned event schema over Matrix custom event types; a documented routing contract for tmux vs Matrix; a documented escalation policy. - **G5 — No central SPOF.** Per-site homeservers federated over TLS/DNS we already control, so one site going dark cannot take the fleet's comms with it. - **G6 — Gate-action integrity.** Reviews / merges / approvals carry **signed authorship** (Buzz pattern) so a gate-critical action is cryptographically attributable, retiring the fragile "distinct bot identity" juggling. ### 1.2 Non-Goals - **NG1 — Do NOT rip out working comms mid-MVP.** tmux fast-path and the existing `mos-comms` git-branch channel keep working until their replacement is proven at parity. This RFC is strangler, not big-bang. - **NG2 — tmux is NOT being replaced.** tmux inter-agent comms remains **P0**. Matrix is *above* it, not instead of it. MACP defines the boundary; it does not move it. - **NG3 — Not adopting Buzz/Nostr as transport.** We adopt Buzz's *patterns*; the wire is Matrix. - **NG4 — Not building a new chat client in P1–P4.** HIL uses an existing Matrix client (Element or equivalent) until/unless a custom client is justified (open question, §11). - **NG5 — Not federating to the public Matrix network.** Federation is Mosaic-site-to-Mosaic-site over infrastructure we control. Public `matrix.org` federation is out of scope (and should likely be firewalled off). - **NG6 — Not a Hermes feature-clone.** We reach *parity on the channels that matter* (§9 checklist), not bug-for-bug Hermes compatibility. --- ## 2. Architecture ### 2.1 Layer diagram ``` ┌───────────────────────────────────────────────┐ │ HUMAN (Jason / HIL) │ │ Element (or custom client) — §11 │ └───────────────────────┬───────────────────────┘ │ (same Matrix surface as agents) │ ┌──────────────────────────────────────────────▼──────────────────────────────────────────────┐ │ SYNAPSE HOMESERVER (self-hosted, ours) │ │ - Client-Server API (agents + humans send/receive events) │ │ - Application Service API (privileged AS hooks: transactions, user/room namespaces) │ │ - Presence EDUs, receipts, typing │ │ - Federation API (S2S) over TLS ── to peer site homeservers (§6, P4) │ └───────▲───────────────────────────────────▲──────────────────────────────────────▲───────────┘ │ AS API (hs_token / as_token) │ C-S API (per-agent access_token) │ S2S │ │ │ ┌───────┴───────────────────────┐ ┌────────┴─────────────────┐ ┌─────────┴──────────┐ │ MOSAIC APPSERVICE │ │ packages/comms (SDK) │ │ PEER SITE Synapse │ │ (apps/matrix-appservice) │ │ used by every harness │ │ (site-B, site-C…) │ │ THE "native layer" │ │ - login/whoami │ │ own appservice │ │ replacing Hermes-backbone │ │ - send MACP events │ │ own agents │ │ │ │ - subscribe/sync │ └────────────────────┘ │ - bulk-provision MXIDs │◄───┤ - presence heartbeat │ │ (@mosaic_:site) │ │ - signed-authorship │ │ - create/manage rooms │ │ envelope (gate acts) │ │ - post introductions │ └────────────┬─────────────┘ │ - track presence/liveness │ │ in-process / IPC │ - enforce room taxonomy │ ┌─────────▼──────────────────────────────────────────┐ │ - escalation watchdog (§5) │ │ AGENT HARNESS │ │ - controlled by ORCHESTRATOR │ │ Claude Code / Codex / Pi / Goose │ └───────▲───────────────────────┘ │ `mosaic enroll` runs on spin (§4.1) │ │ orchestrator drives AS └─────────┬───────────────────────────────────────────┘ ┌───────┴───────────────────────┐ │ │ MOSAIC ORCHESTRATOR │ │ P0 FAST PATH (same host, low-latency) │ (~/.config/mosaic) │ ┌─────────▼──────────┐ tmux send-keys / pane I/O │ spins agents, owns rooms, │◄──────►│ tmux (P0) │◄─►│ peer agent on same host │ │ sets escalation policy │ MACP └────────────────────┘ └─────────────────────────┘ └───────────────────────────────┘ routing rules decide tmux vs Matrix per message (§4.6) ``` Key idea: **the appservice is the backbone.** It is a long-lived privileged process registered with Synapse via an appservice registration file (`hs_token`/`as_token`, namespaces). It is the thing that used to be "Hermes-as-backbone," except we own it, it is inside the orchestrator's control plane, and it speaks native Matrix. ### 2.2 Message flow: agent spin-up → auto-enroll → room join → introduction → presence-online ``` Orchestrator Harness (mosaic enroll) Mosaic Appservice Synapse │ │ │ │ 1. spin agent ─────────────────► │ │ │ │ │ 2. auto-detect harness │ │ │ │ (Claude/Codex/Pi/Goose) │ │ │ │ 3. POST /enroll {agent meta} ─► │ │ │ │ 4. provision MXID │ │ │ │ @mosaic_:site │ │ │ │ via AS API register ─► (201, in namespace) │ │ │ 5. mint access_token │ │ │ 6. ◄── {mxid, token, rooms}──┤ (or as_token masq) │ │ │ │ 7. invite+join rooms ─► (mission/team/fleet) │ │ 8. /sync (via packages/comms)─────────────────────────► (joined state) │ │ │ 9. post introduction ─► m.room.message + │ │ │ (mosaic.introduction) custom event → rooms │ │ 10. set presence ONLINE ─────────────────────────────► presence EDU │ │ 11. start heartbeat loop │ │ │ │ (mosaic.presence ping) │ │ │ 12. appservice reports agent │ │ │ │ ◄──── live in fleet room ────┤ (watchdog now tracks liveness) │ ``` Notes on the steps that matter: - **Step 4/5** use the **Application Service API**: the appservice can register users inside its namespace (`@mosaic_*:site`) and act on their behalf. Two viable modes: (a) mint a real per-agent `access_token` via appservice login, or (b) have the appservice **masquerade** using `user_id` query param on C-S calls with the `as_token`. **Recommendation: mint per-agent tokens** for P2 so the agent process holds only its own credential (blast-radius containment, §8); reserve masquerade for bulk/bootstrap operations the appservice itself performs. **[VERIFY]** exact token-lifetime and refresh behavior against the running Synapse version. - **Step 9** — the introduction is both a human-readable `m.room.message` *and* a structured `mosaic.introduction` custom event (so other agents can machine-parse capabilities without scraping prose). - **Step 10/11** — presence goes online immediately, then a **heartbeat** keeps liveness fresh. Native Matrix presence auto-decays to `unavailable`/`offline`, but we do **not** rely solely on it (Synapse presence timeouts are coarse and federation presence is lossy **[VERIFY]**); MACP adds an explicit `mosaic.presence` heartbeat event for deterministic liveness (§4.5, §5). --- ## 3. Repo-home decision (RESOLVED — recommendation) The core tension: **product monorepo** (`mosaicstack/stack`, this checkout `/src/mosaic-stack`) vs **framework** (`~/.config/mosaic`, the agent/harness runtime that every agent shares regardless of product). The boundary rule I am ratifying: > **Product-monorepo owns the deployed *services and libraries*. Framework owns the *agent/harness contract* — anything an agent needs the moment it spins, before any product code is checked out.** Applying that rule: | Piece | Home | Rationale | |---|---|---| | **Synapse deployment** (compose/helm, config, TLS, `.well-known`, pipelines) | **Product monorepo** → `infra/matrix/` + CI in the monorepo's pipeline dir | It is deployed infrastructure with the same lifecycle/observability as Gateway/Web/DB. Lives beside other `infra/` and Woodpecker pipelines. | | **Mosaic Appservice** | **Product monorepo** → `apps/matrix-appservice` | It is a first-class deployed service (NestJS-style app, same conventions: ESM, `@Inject()`, DTOs, OTEL-before-bootstrap). It talks to Synapse, holds `hs_token`/`as_token`, and is operated like any other app. It is **controlled by** the orchestrator but **is not** the orchestrator. | | **Client SDK** | **Product monorepo** → `packages/comms` | A workspace library consumed by product apps *and* by harnesses. Published/linked like `packages/queue`, `packages/db`. Versioned with the product. | | **MACP spec** (the standard doc + JSON schemas) | **Framework** → `~/.config/mosaic/spec/macp/` (mirrored/vendored into `packages/comms` at build) | The protocol is an agent-level contract that must exist independent of any one product checkout. Framework is the source of truth; `packages/comms` vendors a pinned copy so the SDK and spec cannot drift silently. | | **`mosaic enroll` harness glue** (auto-detect, spin hook) | **Framework** → `~/.config/mosaic/tools/enroll/` | Agents/harnesses are framework-level. Enrollment must run *before* the agent has product context; it cannot depend on `/src/mosaic-stack` being present. This is exactly parallel to the existing `~/.config/mosaic/tools/*` wrappers. | **Boundary summary:** *the wire and the services are product; the contract and the spin-time glue are framework.* The one deliberate coupling is **MACP**: framework is authoritative, but `packages/comms` pins a vendored copy and CI fails if they diverge, so an agent enrolling via framework and a service validating via `packages/comms` agree on the schema by construction. **Rejected alternative:** putting the appservice in the framework. Rejected because the appservice is a stateful, deployed, secret-holding network service that needs the product's CI/observability/secret plumbing; burying it in `~/.config/mosaic` would split its operational story from every other Mosaic service. --- ## 4. MACP v1 — the standard MACP (Mosaic Agent Comms Protocol) v1 is a **profile of Matrix**: it does not invent a transport, it constrains how Mosaic agents use Matrix so that behavior is uniform across harnesses. Versioned via a `macp_version` field on every custom event; v1 is frozen at ratification (P3). ### 4.1 Enrollment contract `mosaic enroll` MUST, on agent spin, in order: 1. **Auto-detect harness.** Detection order + signal: - Claude Code — presence of the Claude Code runtime/env (e.g. `CLAUDE_CODE_*` env, `~/.claude`) **[VERIFY exact signal per harness]** - Codex — Codex runtime markers - Pi — Pi SDK runtime (`packages/agent` / `packages/mosaic` context) - Goose — Goose runtime markers - Fallback: explicit `--harness` flag; if undetectable, enroll as `generic` and warn. 2. **Provision identity** — call appservice `POST /enroll` with `{agent_slug, harness, host, mission_id?, team_id?, capabilities[]}`. Appservice returns `{mxid, access_token, homeserver, rooms[]}` (§2.2 step 4–6). 3. **Join rooms** — accept invites / join the returned room set per taxonomy (§4.6). 4. **Introduce** — post `mosaic.introduction` (+ human-readable `m.room.message`) to each joined room. 5. **Go present** — set Matrix presence `online` and start the `mosaic.presence` heartbeat loop. Enrollment is **idempotent**: re-running `mosaic enroll` for an existing agent slug rebinds to the same MXID (re-mints token if needed) rather than creating a duplicate identity. This is what retires the "distinct bot identity juggling." ### 4.2 Structured event schema (Matrix custom event types) All MACP events carry a common envelope in `content`: ```jsonc { "macp_version": "1.0", "macp_type": "", "agent": { "mxid": "@mosaic_teamlead-3:site-a", "slug": "teamlead-3", "harness": "claude-code" }, "ts": 1753300000000, "mission_id": "KBN-101", // optional "signature": { ... } // present ONLY for gate actions, §4.4 // ...type-specific fields... } ``` Event types (Matrix `type` shown; timeline events use `m.room.message` with a custom `msgtype` where a human-visible fallback is desirable, state events use a dotted custom `type`): | MACP type | Matrix carrier | Purpose | Notable fields | |---|---|---|---| | **message** | `m.room.message`, `msgtype: "mosaic.message"` (+ `body` fallback) | ordinary agent/human chat | `body`, `thread?` | | **presence** | `m.room.message` msgtype `mosaic.presence` in fleet room **or** custom EDU-backed state | heartbeat/liveness ping | `status: online\|away\|offline`, `seq`, `interval_ms` | | **workflow-step** | state event `mosaic.workflow.step` (state_key = step id) | durable record of a workflow/mission step | `step`, `status: started\|blocked\|done\|failed`, `detail` | | **review / approval** | `m.room.message` msgtype `mosaic.review` | a review verdict or merge/approval | `subject` (PR/commit ref), `verdict: approve\|reject\|request-changes`, `signature` (REQUIRED) | | **escalation** | `m.room.message` msgtype `mosaic.escalation` | raise a stuck/dark condition to HIL/fallback | `reason`, `target`, `severity`, `since_ts` | Rationale for the carrier split: **timeline events** (`m.room.message` variants) are the durable, receipted, replayable log (this *is* Buzz's "unified event log," §7). **State events** (`mosaic.workflow.step`, presence-as-state) give last-writer-wins current status that a newly-joined agent reads instantly from room state without replaying history. Every custom event is validated against a JSON Schema shipped in the MACP spec (§3). Unknown/newer `macp_version` → consumers MUST degrade gracefully (render `body`, ignore unknown fields). ### 4.3 Agent identity model - **MXID:** appservice-namespaced `@mosaic_:`. The `mosaic_` prefix is the AS **user namespace** declared in the registration file (`namespaces.users` regex `@mosaic_.*`), so Synapse routes those users to our appservice and no human can squat the namespace. - **Provisioning:** exclusively via the appservice (AS API register). Never hand-created. One MXID per agent-spin; idempotent rebind on re-enroll (§4.1). - **Signed authorship** overlays identity for gate actions only (§4.4). MXID answers "who is this account"; signature answers "did the real key-holder authorize this gate action." ### 4.4 Signed-authorship for gate-critical actions (Buzz pattern, scoped) Gate-critical = **reviews, merges, approvals** — anything that can move code to `main` or unblock a mission gate. - Each enrolled agent is issued (or generates) an **Ed25519 keypair**; the **public** key is registered with the appservice at enrollment and published as agent profile state (`mosaic.identity.pubkey`). Private key custody per §8. - A gate action event carries `content.signature = { alg: "ed25519", key_id, sig }` over a canonical serialization of the envelope (canonical-JSON of `{macp_type, agent.mxid, subject, verdict, ts, mission_id}`). - Verifiers (the appservice gate-watcher, and any agent acting on a verdict) MUST reject an unsigned or bad-signature gate event. Non-gate events are unsigned (keeps the hot path cheap). This is deliberately **narrow**: we do not sign every chat line (Buzz signs everything; we take the pattern only where forgery has teeth). Scope may widen post-P5 if warranted. ### 4.5 Presence & liveness model Three visible states plus an explicit heartbeat: - **online** — agent set presence online AND last `mosaic.presence` heartbeat within `heartbeat_interval` (default **30s [VERIFY tuning]**) × miss-tolerance (default 2). - **away** — presence `unavailable`, or heartbeats late but < dark threshold. - **offline / dark** — no heartbeat for `dark_threshold` (default **N minutes**, policy value, §5/§11) OR presence `offline`. Why not rely on native Matrix presence alone: Synapse presence is (a) coarse-grained, (b) can be disabled for load reasons, and (c) **degrades across federation** **[VERIFY]**. So MACP layers an explicit heartbeat carried as a lightweight timeline/state event in the **fleet presence room**, giving a deterministic, federation-safe liveness signal the escalation watchdog (§5) can reason about. Native presence EDUs are still emitted (they make Element show the right dot for humans) but the *authoritative* liveness source is the heartbeat. ### 4.6 Room / channel taxonomy (orchestrator-owned) The **orchestrator** (via the appservice) owns room lifecycle. Agents never create backbone rooms ad hoc. | Room | Scope | Membership | Purpose | |---|---|---|---| | **Fleet presence room** | one per site (federated view across sites) | every enrolled agent + HIL | heartbeats, the single "who's alive" board. This is the P1 deliverable. | | **Per-mission room** | one per mission (e.g. `#mission-KBN-101`) | agents on that mission + Team Lead + HIL | workflow-steps, mission chat, reviews for that mission | | **Per-team room** | one per team | team members + Team Lead | intra-team coordination | | **HIL room** | one (or one per site) | humans + escalation-privileged agents | where escalations land; Jason's pane on the fleet | Rooms are created with orchestrator-controlled power levels: appservice = admin (PL100), Team Leads elevated, worker agents default. Room aliases (`#mission-KBN-101:site-a`) are stable handles. ### 4.7 tmux ↔ Matrix routing rules (the fast-path/durable boundary) MACP mandates this decision per message. **Default bias: if it must survive the agent, be seen by an offline party, cross a host, or be audited — Matrix. If it is same-host, synchronous, and ephemeral — tmux.** | Signal | Route | Why | |---|---|---| | Same-host, live pane-to-pane prompt/nudge, sub-second need | **tmux (P0)** | lowest latency, no server round-trip; this is the working fast path we keep | | Recipient may be offline / on another host | **Matrix** | durability + store-and-forward; tmux can't reach a dark or remote pane | | Presence / heartbeat / liveness | **Matrix** | must be observable fleet-wide, including by the watchdog and HIL | | Workflow-step, review/approval, escalation | **Matrix** | must be durable, receipted, auditable, signed (gate acts) | | Cross-site anything | **Matrix (federated)** | tmux is same-host only | | Bulk log spew / high-frequency scratch between co-located agents | **tmux**, with periodic **Matrix** checkpoints | avoid flooding the durable log; keep an audit checkpoint | Rule of thumb encoded in the SDK: `comms.send()` takes a `durability` hint; `ephemeral+same-host` short-circuits to tmux, everything else goes Matrix. A message can be **dual-routed** (tmux for immediacy + a Matrix checkpoint) when both speed and durability matter. --- ## 5. Coordinator-availability + HIL escalation (the 13h-stall / homelab-dark fix) **The failure we are killing:** a Team Lead blocked ~13h, polling every 15 min, unable to distinguish "coordinator offline" from "coordinator busy"; and a homelab agent that goes dark taking comms with it. **The fix — presence-driven, policy-encoded escalation:** 1. **Deterministic liveness (§4.5).** Every agent heartbeats into the fleet presence room. The appservice **escalation watchdog** subscribes and maintains `last_seen` per agent. No polling by the Team Lead — it *subscribes* (Matrix `/sync` long-poll) and is pushed state changes. 2. **Encoded policy in MACP:** a machine-readable escalation policy attached to each agent/role, e.g.: ```jsonc { "role": "team-lead", "coordinator": "@mosaic_coordinator-1:site-a", "dark_threshold_min": 10, // OPEN QUESTION §11 — Jason/Mos to set N "on_coordinator_dark": { "action": "escalate", "fallback": "@mosaic_coordinator-2:site-b", // cross-site fallback "then": "notify-HIL", "hil_room": "#hil:site-a" } } ``` 3. **Detection & action by the Team Lead:** when the watchdog (or the Team Lead's own SDK) observes `coordinator.last_seen > dark_threshold`, it: - emits a `mosaic.escalation` event into the mission room and the HIL room (`reason: "coordinator dark", since_ts, severity`), - **re-routes** to the declared fallback coordinator (possibly on another site — this is why federation matters), - if no fallback answers within a second threshold, pages **HIL** (Jason) in the HIL room. The Team Lead **never sits blocked polling**; a dark coordinator is a *pushed event*, and the fallback/HIL path is automatic. 4. **Homelab-dark specifically:** because heartbeats are federated into a cross-site fleet room, a whole *site* going dark is visible from other sites — the watchdog on site-B sees site-A's coordinator stop heartbeating and triggers the same escalation. A dark homelab can no longer silently strand its agents, because the liveness signal and the fallback live *off that host*. **Design invariant:** liveness authority and fallback targets must never be co-located with the thing they monitor. The watchdog for site-A's coordinator should also run (or be mirrored) on site-B. --- ## 6. Federation (P4) **Model:** each site runs its **own Synapse homeserver** with its **own Mosaic appservice** and its own agents. Sites **federate** with each other over the standard Matrix server-to-server (S2S) API, restricted to Mosaic sites. **Why per-site, not one central server:** - **No SPOF.** The homelab going dark is the founding trauma of this program. A single central homeserver would recreate exactly that risk at fleet scale. Per-site means a site outage is contained: its agents drop, but every other site's comms and the cross-site fleet room survive. - **Locality.** Same-site agents get low-latency local homeserver traffic; only cross-site events pay the federation cost. - **Blast radius.** A compromised or misbehaving site can be defederated without touching the rest. **How federation is wired (real Matrix mechanics):** - **Server discovery** via `https:///.well-known/matrix/server` returning `{"m.server": "matrix.:443"}`, and/or an `_matrix._tcp` **SRV** record. We control the DNS/domains, so we control the federation graph. **[VERIFY]** current `.well-known` vs SRV precedence for the deployed Synapse version. - **TLS:** federation requires valid TLS on the federation endpoint; we terminate with certs from our own CA/ACME on domains we own. - **Allowlist:** use Synapse `federation_domain_whitelist` to restrict federation to the set of Mosaic site domains — **no public-network federation** (NG5). This is a hard security boundary. - **Cross-site rooms:** the fleet presence room and any cross-mission rooms are federated rooms whose membership spans site homeservers. Room state replicates via S2S; presence heartbeats propagate as events (not relying on lossy presence EDUs across federation, §4.5). **Cross-site identity:** an agent on site-B is `@mosaic_:site-b`. The signed-authorship pubkey travels in profile state, so a site-A verifier can validate a site-B agent's gate action without trusting site-B's homeserver blindly (signature ≠ homeserver trust). --- ## 7. Buzz-pattern adoption map We adopt Buzz's **ideas**, on Matrix rails, phased: | Buzz idea | Adopt? | How, on Matrix | Phase | |---|---|---|---| | **Harness auto-detect / enroll** | **Yes** | `mosaic enroll` detects Claude/Codex/Pi/Goose and self-registers via the appservice (§4.1) | P2 (enroll v1); scan-machine-and-offer-integrate is **v2** | | **Keypair signed identity** | **Yes, scoped** | Ed25519 signed-authorship on **gate actions only** (reviews/merges/approvals), pubkey in profile state (§4.4) | P5 (hardening); keys issued at enroll from P2 | | **Unified event log (humans + agents, one log)** | **Yes** | Matrix room timeline *is* the unified, receipted, replayable event log; MACP custom events are first-class entries (§4.2) | P2→P3 | | **Humans and agents on the same surface** | **Yes** | HIL uses the same Matrix rooms via Element/custom client; escalations land where Jason already is (§4.6, §5) | P1 (fleet room) → P2 | | **Scan machine, offer to integrate existing tools** | **Deferred (v2)** | enroll v1 auto-detects the *harness*; scanning a host for other integratable tools is explicitly **enroll v2**, post-P5 | v2 | | **Buzz/Nostr as transport** | **No** | transport is Matrix; only patterns are borrowed (NG3) | — | --- ## 8. Security - **Homeserver hardening.** Disable open registration (`enable_registration: false`); agents come only via the appservice. `federation_domain_whitelist` to Mosaic sites only (§6). Rate-limiting on. Media repo locked down or disabled if unused. Admin API bound to localhost / behind auth. Run Synapse behind our reverse proxy with TLS termination we control. **[VERIFY]** current recommended hardening flags for the deployed Synapse version. - **Appservice token custody (`hs_token` / `as_token`).** These are the crown jewels — `as_token` lets the holder act as any user in the namespace; `hs_token` authenticates the homeserver → appservice callbacks. They live **only** in the appservice's secret store (see Vault interaction below), never in an agent process, never in the SDK, never in git. Registration file (which contains them) is a secret artifact, mounted at deploy, not committed. Rotate on suspected exposure; rotation requires updating both Synapse's registration and the appservice. - **Per-agent credentials (the hard part — fleet key management).** - Agents hold **only their own** per-agent `access_token` (minted by the appservice at enroll), not the `as_token`. Compromise of one agent ≠ compromise of the namespace. - **Issuance:** at `mosaic enroll`, the appservice mints the token and returns it over the enroll channel (which must itself be authenticated — enroll requests carry a spin-time bootstrap secret / orchestrator-signed nonce **[VERIFY design in P2]**, so a random process can't enroll a rogue agent). - **Signed-authorship keys:** Ed25519 private keys should be generated agent-side and only the public key leaves the agent (best custody: private key never transits the network). Where agents are ephemeral, keys are minted per-spin and discarded on teardown; the pubkey-in-profile record remains for audit. - **Rotation:** tokens are short-lived where the Synapse version supports refresh **[VERIFY]**; otherwise the appservice supports explicit re-issue on re-enroll. A rotation runbook is a P2 deliverable. - **Federation TLS.** Valid certs on federation endpoints; whitelist-only federation; monitor for cert expiry (a cert lapse silently defederates a site — add to observability). - **Interaction with existing Vault/secrets.** The appservice's `hs_token`/`as_token` and the enroll bootstrap secret are stored in the existing secret manager (Vault or the project's chosen store — **open question §11**) and injected at deploy, consistent with how Gateway/DB secrets are handled today. No new bespoke secret store. Per-agent tokens are *transient runtime* secrets, not persisted to Vault (they're re-mintable). **[VERIFY]** current Mosaic secret-management choice — the CLAUDE.md notes secrets/KBN work in flight, so align with whatever KBN-101 lands. --- ## 9. Migration — strangler off Hermes **Principle:** stand the native layer up *alongside* Hermes, move channels over **as each is proven at parity**, retire Hermes only when the parity checklist is green. Never a flag-day cutover (NG1). **Sequence:** 1. **Stand alongside.** Native Synapse + appservice + fleet presence room live in parallel; Hermes still carries everything it carries today. Presence (P1) is *additive* — it gives us something Hermes never had, at zero risk to existing flows. 2. **Move channels as proven.** Per channel (fleet presence → mission coordination → reviews/approvals → external-chat edge), cut traffic to Matrix, keep Hermes as hot fallback until the channel meets parity for a soak period. 3. **Retire at parity.** When every checklist item is green and soaked, decommission the Hermes MCP bridge as backbone. (External chat platforms, if still needed, can be re-attached as a *bridge at the edge* of Matrix rather than the spine.) **Parity checklist (must all be green before Hermes retires):** - [ ] Every message class Hermes carries today has a MACP equivalent (message, review/approval, escalation, workflow-step). - [ ] Presence/liveness is strictly better than today (it is: today = none). - [ ] Cross-site delivery works over federation with no central SPOF. - [ ] HIL (Jason) can see and act on escalations on the Matrix surface. - [ ] Signed-authorship enforced on gate actions (no unsigned merge/approve accepted). - [ ] `mosaic enroll` auto-onboards all four harnesses (Claude/Codex/Pi/Goose) with zero manual identity setup. - [ ] Delivery receipts / durability demonstrably ≥ Hermes (no lost messages over a soak window). - [ ] Runbooks exist: appservice token rotation, site defederation, dark-site escalation, homeserver restore. - [ ] Observability: appservice + Synapse traced into OTEL/Jaeger like the rest of the stack. - [ ] Rollback path documented (re-enable Hermes channel) for the soak period. --- ## 10. Phased delivery plan (P1 → P5) with acceptance criteria > This section is the decomposition surface: each phase → one or more missions for the orchestrator. ### P1 — Presence / availability on a minimal single-site Synapse *(the first standalone shippable slice)* **Scope:** one Synapse homeserver, a **minimal** appservice (or even a scripted provisioner) whose only job is: register a handful of agent MXIDs, create the **fleet presence room**, and carry heartbeats; a minimal `packages/comms` slice that sets presence and heartbeats; native presence surfaced to a human via Element. **Acceptance criteria:** - A1. Single-site Synapse deployed (`infra/matrix/`), reachable over TLS, open registration OFF. - A2. ≥3 agents enroll (even if semi-manually) and appear in a **fleet presence room** with a live online/away/offline indicator. - A3. `mosaic.presence` heartbeat implemented; an agent killed hard flips to **offline/dark** within `dark_threshold` deterministically (not dependent on native presence timeout alone). - A4. A human (Jason) can open Element, join the fleet room, and see fleet liveness at a glance. - A5. Zero impact to existing tmux + `mos-comms` flows (they still work untouched). ### P2 — Native appservice + orchestrator auto-enroll / room-provisioning **Scope:** full **`apps/matrix-appservice`** registered with Synapse (`hs_token`/`as_token`, namespaces); `mosaic enroll` harness auto-detect; orchestrator-owned room taxonomy; per-agent token minting; introductions. **Acceptance criteria:** - B1. Appservice registered with Synapse via registration file; owns `@mosaic_*` user namespace and room-alias namespace. - B2. `mosaic enroll` auto-detects all four harnesses (Claude/Codex/Pi/Goose) and self-registers on spin, idempotently. - B3. On spin, an agent is provisioned an MXID, minted its **own** access token, joined to the correct mission/team/fleet rooms, and posts a `mosaic.introduction`. - B4. Orchestrator can create/destroy mission & team rooms with correct power levels via the appservice. - B5. Enroll bootstrap is authenticated (a rogue local process cannot enroll a rogue agent). - B6. Appservice + Synapse traced into OTEL/Jaeger. ### P3 — MACP v1 spec ratified **Scope:** freeze the standard (§4): envelope, event types + JSON Schemas, identity model, presence model, room taxonomy, tmux↔Matrix routing rules. Spec lives in framework (`~/.config/mosaic/spec/macp`), vendored+pinned into `packages/comms` with CI drift-check. **Acceptance criteria:** - C1. MACP v1 document ratified (MS-LEAD sign-off, Jason veto window closed). - C2. JSON Schemas for all five event types published; `packages/comms` validates outbound/inbound against them. - C3. CI fails if framework spec and vendored `packages/comms` copy diverge. - C4. Routing-rule conformance test: SDK provably sends ephemeral+same-host over tmux, everything else over Matrix. - C5. Unknown-`macp_version` graceful-degrade behavior tested. ### P4 — Federation **Scope:** a second site homeserver + appservice; S2S federation over our DNS/TLS; cross-site fleet room; whitelist-only federation; cross-site escalation. **Acceptance criteria:** - D1. Two sites federate via `.well-known`/SRV over TLS we control; `federation_domain_whitelist` restricts to Mosaic sites (no public federation). - D2. A cross-site fleet presence room shows agents from both sites; heartbeats propagate as events across federation. - D3. **Homelab-dark test:** killing site-A's coordinator is observed from site-B within `dark_threshold`, and the escalation/fallback fires cross-site (§5). - D4. Cross-site gate action: a site-B agent's signed review is verified by a site-A verifier. - D5. Defederation runbook proven (a site can be cut off cleanly). ### P5 — Buzz-hardening + signed-authorship + Hermes retired **Scope:** Ed25519 signed-authorship enforced on gate actions; security hardening pass; complete the strangler and retire Hermes at parity. **Acceptance criteria:** - E1. Every merge/approve/review gate action is signed; unsigned or bad-sig gate events are rejected by the appservice watcher and by consuming agents. - E2. Token/key rotation runbooks executed at least once in anger (rotate `as_token`, rotate a per-agent key). - E3. Security review complete: homeserver hardening flags, token custody, federation TLS/whitelist all verified. - E4. **Parity checklist (§9) fully green + soaked.** - E5. Hermes MCP bridge retired as backbone (optionally re-attached as an edge bridge only). --- ## 11. Open questions for Jason / Mos (need a human/coordinator ruling) 1. **DNS / domains per site.** What domain(s) do we own and want to use per site for homeserver names and `.well-known` (e.g. `site-a.mosaicstack.dev`)? Federation identity is permanent-ish once agents mint MXIDs against it — this needs a ruling **before P1 hardens** because MXIDs bake in the domain. 2. **Secret-management choice.** Is it Vault, or whatever KBN-101 lands? The appservice `hs_token`/`as_token` and enroll bootstrap secret custody depend on this (§8). CLAUDE.md signals secrets work is in flight — need the authoritative target. 3. **N-minute escalation threshold.** What is `dark_threshold_min` for a coordinator, and the second threshold before HIL is paged (§5)? Default proposed: 10 min → fallback, +5 min → HIL. Jason/Mos to confirm per role. 4. **HIL client: Element vs custom.** Do humans use off-the-shelf **Element** (fast, free, P1-ready) or do we invest in a custom HIL client? Proposed: **Element for P1–P4**, revisit custom only if HIL ergonomics demand it. 5. **Ephemeral vs persistent agent keys.** For signed-authorship, do we mint Ed25519 keys per-spin (simplest custody, no long-term private key at rest) or issue durable per-agent keys (stable identity across spins, but key-at-rest custody problem)? Proposed: **per-spin**, pubkey retained for audit. 6. **Federation topology / trust.** Full mesh between all sites, or hub-and-spoke-with-redundancy? Full mesh maximizes no-SPOF but grows O(n²); needs a call once site count is known. 7. **Fallback-coordinator assignment authority.** Who assigns each Team Lead's fallback coordinator, and is it always cross-site? (Design invariant §5 wants the fallback off the monitored host — confirm this is acceptable operationally.) 8. **Retention / compliance.** How long do we retain the Matrix event log (the unified audit trail)? Affects Synapse storage sizing and any purge policy. --- ## Appendix A — Real Matrix concepts this RFC leans on (quick reference) - **Application Service (AS) API** — a privileged service registered with the homeserver via a registration YAML declaring `id`, `url`, `as_token`, `hs_token`, and `namespaces` (users/aliases/rooms regexes). The homeserver pushes events to the AS in transactions; the AS can register/act-as users in its namespace. *(This is our appservice backbone.)* - **`hs_token` / `as_token`** — `hs_token`: homeserver→AS authentication on pushed transactions; `as_token`: AS→homeserver authentication, grants acting-as any namespaced user. Both are high-value secrets (§8). - **Masquerade (`user_id` query param)** — the AS may act as a namespaced user on C-S calls using `as_token` + `?user_id=`. We prefer per-agent tokens for blast-radius; masquerade for AS-internal bulk ops. - **Custom event types** — timeline events via `m.room.message` with a custom `msgtype` (keeps a human-visible `body` fallback) and/or fully custom `type` (dotted, e.g. `mosaic.workflow.step`); **state events** for last-writer-wins current status readable from room state without history replay. - **Presence EDUs** — native online/unavailable/offline signals; coarse and lossy over federation, so MACP adds an explicit heartbeat event as the authoritative liveness source (§4.5). - **Federation (S2S API)** — server-to-server over TLS; discovery via `/.well-known/matrix/server` and/or `_matrix._tcp` SRV; restrictable with `federation_domain_whitelist`. - **Synapse config knobs cited** — `enable_registration`, `federation_domain_whitelist`, appservice registration file, rate-limiting, admin API binding. **[VERIFY]** exact flags/paths against the deployed Synapse version at implementation time. *All Matrix mechanics above are cited from architecture knowledge and MUST be re-verified against the actual deployed Synapse version during P1 — every **[VERIFY]** in this document is a checkpoint, not an assumption.*