Files

ci/woodpecker/pr/ci Pipeline was successful

Details

style(rfc): apply repo prettier formatting to the two RFCs

Whitespace/table-padding/emphasis-marker-style only (prettier --write),
to satisfy pnpm format:check (globs **/*.md). No wording, numbers,
[VERIFY] flags, or disclaimers changed — verified against the source
files line-for-line modulo formatting.

2026-07-24 19:39:57 -05:00

47 KiB

Raw Blame History

RFC-001 — MACP: A Mosaic-Native, Matrix-Native Comms Layer

Status: DRAFT — for Team Lead → Orchestrator staffing
Author: MS-LEAD (reviewer identity ms-lead-reviewer)
Sponsor / veto: Jason (human lead)
Date: 2026-07-24
Program: Mosaic Stack comms-evolution
Supersedes backbone: the Hermes MCP chat bridge (strangler-retired, see §9)
Audience: Team Leads, the Mosaic orchestrator, infra, and any harness maintainer (Claude Code / Codex / Pi / Goose)

This is a design document. No code ships from this RFC. It exists to be decomposed into missions (P1→P5, §10) with per-phase acceptance criteria. Where a claim is uncertain or needs live validation, it is flagged [VERIFY].

0. TL;DR

We are building a Mosaic-native comms layer on Matrix. We self-host a Synapse homeserver and register a privileged Mosaic Application Service (the "appservice") that the orchestrator controls. The appservice bulk-provisions one Matrix identity per agent-spin, creates and manages rooms, posts agent introductions, and tracks presence/liveness. Agents talk to it through a thin packages/comms client SDK. tmux stays the P0 same-host fast path; Matrix is the durable, presence-aware, federated layer above it, and MACP (the Mosaic Agent Comms Protocol) is the standard that says which path to use when. Federation is per-site Synapse homeservers federated over TLS we control — a direct answer to "the homelab agent went dark and took comms with it." We adopt Buzz's patterns (auto-detect/enroll, signed identity, unified event log, humans-and-agents on one surface) without adopting Buzz/Nostr as transport. We migrate off Hermes by the strangler pattern: stand native alongside, move channels as proven, retire at parity.

The first standalone shippable slice is presence (P1).

1. Goals / Non-Goals

1.1 Goals

G1 — Presence & liveness first. A Team Lead must be able to answer "is my coordinator online, away, or dead?" in seconds, not by polling for 13 hours. Presence is the P1 slice and ships before anything else.
G2 — A native backbone we own. Replace Hermes-as-backbone with a self-hosted Synapse + a Mosaic-controlled appservice. External chat bridging becomes an optional edge, not the spine.
G3 — Turnkey harness enrollment. mosaic enroll auto-detects the harness and self-registers the agent via the appservice on spin. No hand-rolled per-bot identity juggling.
G4 — A real protocol (MACP v1). Structured, versioned event schema over Matrix custom event types; a documented routing contract for tmux vs Matrix; a documented escalation policy.
G5 — No central SPOF. Per-site homeservers federated over TLS/DNS we already control, so one site going dark cannot take the fleet's comms with it.
G6 — Gate-action integrity. Reviews / merges / approvals carry signed authorship (Buzz pattern) so a gate-critical action is cryptographically attributable, retiring the fragile "distinct bot identity" juggling.

1.2 Non-Goals

NG1 — Do NOT rip out working comms mid-MVP. tmux fast-path and the existing mos-comms git-branch channel keep working until their replacement is proven at parity. This RFC is strangler, not big-bang.
NG2 — tmux is NOT being replaced. tmux inter-agent comms remains P0. Matrix is above it, not instead of it. MACP defines the boundary; it does not move it.
NG3 — Not adopting Buzz/Nostr as transport. We adopt Buzz's patterns; the wire is Matrix.
NG4 — Not building a new chat client in P1–P4. HIL uses an existing Matrix client (Element or equivalent) until/unless a custom client is justified (open question, §11).
NG5 — Not federating to the public Matrix network. Federation is Mosaic-site-to-Mosaic-site over infrastructure we control. Public matrix.org federation is out of scope (and should likely be firewalled off).
NG6 — Not a Hermes feature-clone. We reach parity on the channels that matter (§9 checklist), not bug-for-bug Hermes compatibility.

2. Architecture

2.1 Layer diagram

                          ┌───────────────────────────────────────────────┐
                          │                 HUMAN (Jason / HIL)            │
                          │        Element (or custom client) — §11        │
                          └───────────────────────┬───────────────────────┘
                                                  │  (same Matrix surface as agents)
                                                  │
   ┌──────────────────────────────────────────────▼──────────────────────────────────────────────┐
   │                              SYNAPSE HOMESERVER  (self-hosted, ours)                          │
   │   - Client-Server API (agents + humans send/receive events)                                   │
   │   - Application Service API (privileged AS hooks: transactions, user/room namespaces)         │
   │   - Presence EDUs, receipts, typing                                                           │
   │   - Federation API (S2S) over TLS  ── to peer site homeservers (§6, P4)                       │
   └───────▲───────────────────────────────────▲──────────────────────────────────────▲───────────┘
           │ AS API (hs_token / as_token)       │ C-S API (per-agent access_token)      │ S2S
           │                                     │                                       │
   ┌───────┴───────────────────────┐    ┌────────┴─────────────────┐          ┌─────────┴──────────┐
   │   MOSAIC APPSERVICE           │    │   packages/comms  (SDK)   │          │  PEER SITE Synapse │
   │   (apps/matrix-appservice)    │    │   used by every harness   │          │  (site-B, site-C…) │
   │   THE "native layer"          │    │   - login/whoami          │          │  own appservice    │
   │   replacing Hermes-backbone   │    │   - send MACP events      │          │  own agents        │
   │                               │    │   - subscribe/sync        │          └────────────────────┘
   │  - bulk-provision MXIDs       │◄───┤   - presence heartbeat    │
   │    (@mosaic_<agent>:site)     │    │   - signed-authorship     │
   │  - create/manage rooms        │    │     envelope (gate acts)  │
   │  - post introductions         │    └────────────┬─────────────┘
   │  - track presence/liveness    │                 │ in-process / IPC
   │  - enforce room taxonomy      │        ┌─────────▼──────────────────────────────────────────┐
   │  - escalation watchdog (§5)   │        │                 AGENT HARNESS                        │
   │  - controlled by ORCHESTRATOR │        │   Claude Code / Codex / Pi / Goose                   │
   └───────▲───────────────────────┘        │   `mosaic enroll` runs on spin (§4.1)                │
           │ orchestrator drives AS          └─────────┬───────────────────────────────────────────┘
   ┌───────┴───────────────────────┐                  │
   │   MOSAIC ORCHESTRATOR          │                  │  P0 FAST PATH (same host, low-latency)
   │   (~/.config/mosaic)           │        ┌─────────▼──────────┐   tmux send-keys / pane I/O
   │   spins agents, owns rooms,    │◄──────►│   tmux (P0)        │◄─►│ peer agent on same host │
   │   sets escalation policy       │  MACP  └────────────────────┘   └─────────────────────────┘
   └───────────────────────────────┘  routing rules decide tmux vs Matrix per message (§4.6)

Key idea: the appservice is the backbone. It is a long-lived privileged process registered with Synapse via an appservice registration file (hs_token/as_token, namespaces). It is the thing that used to be "Hermes-as-backbone," except we own it, it is inside the orchestrator's control plane, and it speaks native Matrix.

2.2 Message flow: agent spin-up → auto-enroll → room join → introduction → presence-online

Orchestrator                Harness (mosaic enroll)        Mosaic Appservice           Synapse
    │                              │                              │                        │
 1. spin agent  ─────────────────► │                              │                        │
    │                              │ 2. auto-detect harness       │                        │
    │                              │    (Claude/Codex/Pi/Goose)   │                        │
    │                              │ 3. POST /enroll {agent meta} ─►                        │
    │                              │                              │ 4. provision MXID       │
    │                              │                              │    @mosaic_<agent>:site │
    │                              │                              │    via AS API register ─►  (201, in namespace)
    │                              │                              │ 5. mint access_token   │
    │                              │ 6. ◄── {mxid, token, rooms}──┤    (or as_token masq)  │
    │                              │                              │ 7. invite+join rooms   ─► (mission/team/fleet)
    │                              │ 8. /sync (via packages/comms)─────────────────────────►  (joined state)
    │                              │                              │ 9. post introduction    ─► m.room.message +
    │                              │                              │    (mosaic.introduction)    custom event → rooms
    │                              │ 10. set presence ONLINE ─────────────────────────────►   presence EDU
    │                              │ 11. start heartbeat loop     │                        │
    │                              │     (mosaic.presence ping)   │                        │
    │ 12. appservice reports agent │                              │                        │
    │ ◄──── live in fleet room ────┤ (watchdog now tracks liveness)                         │

Notes on the steps that matter:

Step 4/5 use the Application Service API: the appservice can register users inside its namespace (@mosaic_*:site) and act on their behalf. Two viable modes: (a) mint a real per-agent access_token via appservice login, or (b) have the appservice masquerade using user_id query param on C-S calls with the as_token. Recommendation: mint per-agent tokens for P2 so the agent process holds only its own credential (blast-radius containment, §8); reserve masquerade for bulk/bootstrap operations the appservice itself performs. [VERIFY] exact token-lifetime and refresh behavior against the running Synapse version.
Step 9 — the introduction is both a human-readable m.room.message and a structured mosaic.introduction custom event (so other agents can machine-parse capabilities without scraping prose).
Step 10/11 — presence goes online immediately, then a heartbeat keeps liveness fresh. Native Matrix presence auto-decays to unavailable/offline, but we do not rely solely on it (Synapse presence timeouts are coarse and federation presence is lossy [VERIFY]); MACP adds an explicit mosaic.presence heartbeat event for deterministic liveness (§4.5, §5).

3. Repo-home decision (RESOLVED — recommendation)

The core tension: product monorepo (mosaicstack/stack, this checkout /src/mosaic-stack) vs framework (~/.config/mosaic, the agent/harness runtime that every agent shares regardless of product). The boundary rule I am ratifying:

Product-monorepo owns the deployed services and libraries. Framework owns the agent/harness contract — anything an agent needs the moment it spins, before any product code is checked out.

Applying that rule:

Piece	Home	Rationale
Synapse deployment (compose/helm, config, TLS, `.well-known`, pipelines)	Product monorepo → `infra/matrix/` + CI in the monorepo's pipeline dir	It is deployed infrastructure with the same lifecycle/observability as Gateway/Web/DB. Lives beside other `infra/` and Woodpecker pipelines.
Mosaic Appservice	Product monorepo → `apps/matrix-appservice`	It is a first-class deployed service (NestJS-style app, same conventions: ESM, `@Inject()`, DTOs, OTEL-before-bootstrap). It talks to Synapse, holds `hs_token`/`as_token`, and is operated like any other app. It is controlled by the orchestrator but is not the orchestrator.
Client SDK	Product monorepo → `packages/comms`	A workspace library consumed by product apps and by harnesses. Published/linked like `packages/queue`, `packages/db`. Versioned with the product.
MACP spec (the standard doc + JSON schemas)	Framework → `~/.config/mosaic/spec/macp/` (mirrored/vendored into `packages/comms` at build)	The protocol is an agent-level contract that must exist independent of any one product checkout. Framework is the source of truth; `packages/comms` vendors a pinned copy so the SDK and spec cannot drift silently.
`mosaic enroll` harness glue (auto-detect, spin hook)	Framework → `~/.config/mosaic/tools/enroll/`	Agents/harnesses are framework-level. Enrollment must run before the agent has product context; it cannot depend on `/src/mosaic-stack` being present. This is exactly parallel to the existing `~/.config/mosaic/tools/*` wrappers.

Boundary summary: the wire and the services are product; the contract and the spin-time glue are framework. The one deliberate coupling is MACP: framework is authoritative, but packages/comms pins a vendored copy and CI fails if they diverge, so an agent enrolling via framework and a service validating via packages/comms agree on the schema by construction.

Rejected alternative: putting the appservice in the framework. Rejected because the appservice is a stateful, deployed, secret-holding network service that needs the product's CI/observability/secret plumbing; burying it in ~/.config/mosaic would split its operational story from every other Mosaic service.

4. MACP v1 — the standard

MACP (Mosaic Agent Comms Protocol) v1 is a profile of Matrix: it does not invent a transport, it constrains how Mosaic agents use Matrix so that behavior is uniform across harnesses. Versioned via a macp_version field on every custom event; v1 is frozen at ratification (P3).

4.1 Enrollment contract

mosaic enroll MUST, on agent spin, in order:

Auto-detect harness. Detection order + signal:
- Claude Code — presence of the Claude Code runtime/env (e.g. CLAUDE_CODE_* env, ~/.claude) [VERIFY exact signal per harness]
- Codex — Codex runtime markers
- Pi — Pi SDK runtime (packages/agent / packages/mosaic context)
- Goose — Goose runtime markers
- Fallback: explicit --harness flag; if undetectable, enroll as generic and warn.
Provision identity — call appservice POST /enroll with {agent_slug, harness, host, mission_id?, team_id?, capabilities[]}. Appservice returns {mxid, access_token, homeserver, rooms[]} (§2.2 step 4–6).
Join rooms — accept invites / join the returned room set per taxonomy (§4.6).
Introduce — post mosaic.introduction (+ human-readable m.room.message) to each joined room.
Go present — set Matrix presence online and start the mosaic.presence heartbeat loop.

Enrollment is idempotent: re-running mosaic enroll for an existing agent slug rebinds to the same MXID (re-mints token if needed) rather than creating a duplicate identity. This is what retires the "distinct bot identity juggling."

4.2 Structured event schema (Matrix custom event types)

All MACP events carry a common envelope in content:

{
  "macp_version": "1.0",
  "macp_type": "<see below>",
  "agent": { "mxid": "@mosaic_teamlead-3:site-a", "slug": "teamlead-3", "harness": "claude-code" },
  "ts": 1753300000000,
  "mission_id": "KBN-101",         // optional
  "signature": { ... }             // present ONLY for gate actions, §4.4
  // ...type-specific fields...
}

Event types (Matrix type shown; timeline events use m.room.message with a custom msgtype where a human-visible fallback is desirable, state events use a dotted custom type):

MACP type	Matrix carrier	Purpose	Notable fields
message	`m.room.message`, `msgtype: "mosaic.message"` (+ `body` fallback)	ordinary agent/human chat	`body`, `thread?`
presence	`m.room.message` msgtype `mosaic.presence` in fleet room or custom EDU-backed state	heartbeat/liveness ping	`status: online\|away\|offline`, `seq`, `interval_ms`
workflow-step	state event `mosaic.workflow.step` (state_key = step id)	durable record of a workflow/mission step	`step`, `status: started\|blocked\|done\|failed`, `detail`
review / approval	`m.room.message` msgtype `mosaic.review`	a review verdict or merge/approval	`subject` (PR/commit ref), `verdict: approve\|reject\|request-changes`, `signature` (REQUIRED)
escalation	`m.room.message` msgtype `mosaic.escalation`	raise a stuck/dark condition to HIL/fallback	`reason`, `target`, `severity`, `since_ts`

Rationale for the carrier split: timeline events (m.room.message variants) are the durable, receipted, replayable log (this is Buzz's "unified event log," §7). State events (mosaic.workflow.step, presence-as-state) give last-writer-wins current status that a newly-joined agent reads instantly from room state without replaying history.

Every custom event is validated against a JSON Schema shipped in the MACP spec (§3). Unknown/newer macp_version → consumers MUST degrade gracefully (render body, ignore unknown fields).

4.3 Agent identity model

MXID: appservice-namespaced @mosaic_<slug>:<site-domain>. The mosaic_ prefix is the AS user namespace declared in the registration file (namespaces.users regex @mosaic_.*), so Synapse routes those users to our appservice and no human can squat the namespace.
Provisioning: exclusively via the appservice (AS API register). Never hand-created. One MXID per agent-spin; idempotent rebind on re-enroll (§4.1).
Signed authorship overlays identity for gate actions only (§4.4). MXID answers "who is this account"; signature answers "did the real key-holder authorize this gate action."

4.4 Signed-authorship for gate-critical actions (Buzz pattern, scoped)

Gate-critical = reviews, merges, approvals — anything that can move code to main or unblock a mission gate.

Each enrolled agent is issued (or generates) an Ed25519 keypair; the public key is registered with the appservice at enrollment and published as agent profile state (mosaic.identity.pubkey). Private key custody per §8.
A gate action event carries content.signature = { alg: "ed25519", key_id, sig } over a canonical serialization of the envelope (canonical-JSON of {macp_type, agent.mxid, subject, verdict, ts, mission_id}).
Verifiers (the appservice gate-watcher, and any agent acting on a verdict) MUST reject an unsigned or bad-signature gate event. Non-gate events are unsigned (keeps the hot path cheap).

This is deliberately narrow: we do not sign every chat line (Buzz signs everything; we take the pattern only where forgery has teeth). Scope may widen post-P5 if warranted.

4.5 Presence & liveness model

Three visible states plus an explicit heartbeat:

online — agent set presence online AND last mosaic.presence heartbeat within heartbeat_interval (default 30s [VERIFY tuning]) × miss-tolerance (default 2).
away — presence unavailable, or heartbeats late but < dark threshold.
offline / dark — no heartbeat for dark_threshold (default N minutes, policy value, §5/§11) OR presence offline.

Why not rely on native Matrix presence alone: Synapse presence is (a) coarse-grained, (b) can be disabled for load reasons, and (c) degrades across federation [VERIFY]. So MACP layers an explicit heartbeat carried as a lightweight timeline/state event in the fleet presence room, giving a deterministic, federation-safe liveness signal the escalation watchdog (§5) can reason about. Native presence EDUs are still emitted (they make Element show the right dot for humans) but the authoritative liveness source is the heartbeat.

4.6 Room / channel taxonomy (orchestrator-owned)

The orchestrator (via the appservice) owns room lifecycle. Agents never create backbone rooms ad hoc.

Room	Scope	Membership	Purpose
Fleet presence room	one per site (federated view across sites)	every enrolled agent + HIL	heartbeats, the single "who's alive" board. This is the P1 deliverable.
Per-mission room	one per mission (e.g. `#mission-KBN-101`)	agents on that mission + Team Lead + HIL	workflow-steps, mission chat, reviews for that mission
Per-team room	one per team	team members + Team Lead	intra-team coordination
HIL room	one (or one per site)	humans + escalation-privileged agents	where escalations land; Jason's pane on the fleet

Rooms are created with orchestrator-controlled power levels: appservice = admin (PL100), Team Leads elevated, worker agents default. Room aliases (#mission-KBN-101:site-a) are stable handles.

4.7 tmux ↔ Matrix routing rules (the fast-path/durable boundary)

MACP mandates this decision per message. Default bias: if it must survive the agent, be seen by an offline party, cross a host, or be audited — Matrix. If it is same-host, synchronous, and ephemeral — tmux.

Signal	Route	Why
Same-host, live pane-to-pane prompt/nudge, sub-second need	tmux (P0)	lowest latency, no server round-trip; this is the working fast path we keep
Recipient may be offline / on another host	Matrix	durability + store-and-forward; tmux can't reach a dark or remote pane
Presence / heartbeat / liveness	Matrix	must be observable fleet-wide, including by the watchdog and HIL
Workflow-step, review/approval, escalation	Matrix	must be durable, receipted, auditable, signed (gate acts)
Cross-site anything	Matrix (federated)	tmux is same-host only
Bulk log spew / high-frequency scratch between co-located agents	tmux, with periodic Matrix checkpoints	avoid flooding the durable log; keep an audit checkpoint

Rule of thumb encoded in the SDK: comms.send() takes a durability hint; ephemeral+same-host short-circuits to tmux, everything else goes Matrix. A message can be dual-routed (tmux for immediacy + a Matrix checkpoint) when both speed and durability matter.

5. Coordinator-availability + HIL escalation (the 13h-stall / homelab-dark fix)

The failure we are killing: a Team Lead blocked ~13h, polling every 15 min, unable to distinguish "coordinator offline" from "coordinator busy"; and a homelab agent that goes dark taking comms with it.

The fix — presence-driven, policy-encoded escalation:

Deterministic liveness (§4.5). Every agent heartbeats into the fleet presence room. The appservice escalation watchdog subscribes and maintains last_seen per agent. No polling by the Team Lead — it subscribes (Matrix /sync long-poll) and is pushed state changes.

Encoded policy in MACP: a machine-readable escalation policy attached to each agent/role, e.g.:

{
  "role": "team-lead",
  "coordinator": "@mosaic_coordinator-1:site-a",
  "dark_threshold_min": 10, // OPEN QUESTION §11 — Jason/Mos to set N
  "on_coordinator_dark": {
    "action": "escalate",
    "fallback": "@mosaic_coordinator-2:site-b", // cross-site fallback
    "then": "notify-HIL",
    "hil_room": "#hil:site-a",
  },
}

Detection & action by the Team Lead: when the watchdog (or the Team Lead's own SDK) observes coordinator.last_seen > dark_threshold, it:
- emits a mosaic.escalation event into the mission room and the HIL room (reason: "coordinator dark", since_ts, severity),
- re-routes to the declared fallback coordinator (possibly on another site — this is why federation matters),
- if no fallback answers within a second threshold, pages HIL (Jason) in the HIL room. The Team Lead never sits blocked polling; a dark coordinator is a pushed event, and the fallback/HIL path is automatic.
Homelab-dark specifically: because heartbeats are federated into a cross-site fleet room, a whole site going dark is visible from other sites — the watchdog on site-B sees site-A's coordinator stop heartbeating and triggers the same escalation. A dark homelab can no longer silently strand its agents, because the liveness signal and the fallback live off that host.

Design invariant: liveness authority and fallback targets must never be co-located with the thing they monitor. The watchdog for site-A's coordinator should also run (or be mirrored) on site-B.

6. Federation (P4)

Model: each site runs its own Synapse homeserver with its own Mosaic appservice and its own agents. Sites federate with each other over the standard Matrix server-to-server (S2S) API, restricted to Mosaic sites.

Why per-site, not one central server:

No SPOF. The homelab going dark is the founding trauma of this program. A single central homeserver would recreate exactly that risk at fleet scale. Per-site means a site outage is contained: its agents drop, but every other site's comms and the cross-site fleet room survive.
Locality. Same-site agents get low-latency local homeserver traffic; only cross-site events pay the federation cost.
Blast radius. A compromised or misbehaving site can be defederated without touching the rest.

How federation is wired (real Matrix mechanics):

Server discovery via https://<domain>/.well-known/matrix/server returning {"m.server": "matrix.<domain>:443"}, and/or an _matrix._tcp SRV record. We control the DNS/domains, so we control the federation graph. [VERIFY] current .well-known vs SRV precedence for the deployed Synapse version.
TLS: federation requires valid TLS on the federation endpoint; we terminate with certs from our own CA/ACME on domains we own.
Allowlist: use Synapse federation_domain_whitelist to restrict federation to the set of Mosaic site domains — no public-network federation (NG5). This is a hard security boundary.
Cross-site rooms: the fleet presence room and any cross-mission rooms are federated rooms whose membership spans site homeservers. Room state replicates via S2S; presence heartbeats propagate as events (not relying on lossy presence EDUs across federation, §4.5).

Cross-site identity: an agent on site-B is @mosaic_<slug>:site-b. The signed-authorship pubkey travels in profile state, so a site-A verifier can validate a site-B agent's gate action without trusting site-B's homeserver blindly (signature ≠ homeserver trust).

7. Buzz-pattern adoption map

We adopt Buzz's ideas, on Matrix rails, phased:

Buzz idea	Adopt?	How, on Matrix	Phase
Harness auto-detect / enroll	Yes	`mosaic enroll` detects Claude/Codex/Pi/Goose and self-registers via the appservice (§4.1)	P2 (enroll v1); scan-machine-and-offer-integrate is v2
Keypair signed identity	Yes, scoped	Ed25519 signed-authorship on gate actions only (reviews/merges/approvals), pubkey in profile state (§4.4)	P5 (hardening); keys issued at enroll from P2
Unified event log (humans + agents, one log)	Yes	Matrix room timeline is the unified, receipted, replayable event log; MACP custom events are first-class entries (§4.2)	P2→P3
Humans and agents on the same surface	Yes	HIL uses the same Matrix rooms via Element/custom client; escalations land where Jason already is (§4.6, §5)	P1 (fleet room) → P2
Scan machine, offer to integrate existing tools	Deferred (v2)	enroll v1 auto-detects the harness; scanning a host for other integratable tools is explicitly enroll v2, post-P5	v2
Buzz/Nostr as transport	No	transport is Matrix; only patterns are borrowed (NG3)	—

8. Security

Homeserver hardening. Disable open registration (enable_registration: false); agents come only via the appservice. federation_domain_whitelist to Mosaic sites only (§6). Rate-limiting on. Media repo locked down or disabled if unused. Admin API bound to localhost / behind auth. Run Synapse behind our reverse proxy with TLS termination we control. [VERIFY] current recommended hardening flags for the deployed Synapse version.
Appservice token custody (hs_token / as_token). These are the crown jewels — as_token lets the holder act as any user in the namespace; hs_token authenticates the homeserver → appservice callbacks. They live only in the appservice's secret store (see Vault interaction below), never in an agent process, never in the SDK, never in git. Registration file (which contains them) is a secret artifact, mounted at deploy, not committed. Rotate on suspected exposure; rotation requires updating both Synapse's registration and the appservice.
Per-agent credentials (the hard part — fleet key management).
- Agents hold only their own per-agent access_token (minted by the appservice at enroll), not the as_token. Compromise of one agent ≠ compromise of the namespace.
- Issuance: at mosaic enroll, the appservice mints the token and returns it over the enroll channel (which must itself be authenticated — enroll requests carry a spin-time bootstrap secret / orchestrator-signed nonce [VERIFY design in P2], so a random process can't enroll a rogue agent).
- Signed-authorship keys: Ed25519 private keys should be generated agent-side and only the public key leaves the agent (best custody: private key never transits the network). Where agents are ephemeral, keys are minted per-spin and discarded on teardown; the pubkey-in-profile record remains for audit.
- Rotation: tokens are short-lived where the Synapse version supports refresh [VERIFY]; otherwise the appservice supports explicit re-issue on re-enroll. A rotation runbook is a P2 deliverable.
Federation TLS. Valid certs on federation endpoints; whitelist-only federation; monitor for cert expiry (a cert lapse silently defederates a site — add to observability).
Interaction with existing Vault/secrets. The appservice's hs_token/as_token and the enroll bootstrap secret are stored in the existing secret manager (Vault or the project's chosen store — open question §11) and injected at deploy, consistent with how Gateway/DB secrets are handled today. No new bespoke secret store. Per-agent tokens are transient runtime secrets, not persisted to Vault (they're re-mintable). [VERIFY] current Mosaic secret-management choice — the CLAUDE.md notes secrets/KBN work in flight, so align with whatever KBN-101 lands.

9. Migration — strangler off Hermes

Principle: stand the native layer up alongside Hermes, move channels over as each is proven at parity, retire Hermes only when the parity checklist is green. Never a flag-day cutover (NG1).

Sequence:

Stand alongside. Native Synapse + appservice + fleet presence room live in parallel; Hermes still carries everything it carries today. Presence (P1) is additive — it gives us something Hermes never had, at zero risk to existing flows.
Move channels as proven. Per channel (fleet presence → mission coordination → reviews/approvals → external-chat edge), cut traffic to Matrix, keep Hermes as hot fallback until the channel meets parity for a soak period.
Retire at parity. When every checklist item is green and soaked, decommission the Hermes MCP bridge as backbone. (External chat platforms, if still needed, can be re-attached as a bridge at the edge of Matrix rather than the spine.)

Parity checklist (must all be green before Hermes retires):

Every message class Hermes carries today has a MACP equivalent (message, review/approval, escalation, workflow-step).
Presence/liveness is strictly better than today (it is: today = none).
Cross-site delivery works over federation with no central SPOF.
HIL (Jason) can see and act on escalations on the Matrix surface.
Signed-authorship enforced on gate actions (no unsigned merge/approve accepted).
mosaic enroll auto-onboards all four harnesses (Claude/Codex/Pi/Goose) with zero manual identity setup.
Delivery receipts / durability demonstrably ≥ Hermes (no lost messages over a soak window).
Runbooks exist: appservice token rotation, site defederation, dark-site escalation, homeserver restore.
Observability: appservice + Synapse traced into OTEL/Jaeger like the rest of the stack.
Rollback path documented (re-enable Hermes channel) for the soak period.

10. Phased delivery plan (P1 → P5) with acceptance criteria

This section is the decomposition surface: each phase → one or more missions for the orchestrator.

P1 — Presence / availability on a minimal single-site Synapse (the first standalone shippable slice)

Scope: one Synapse homeserver, a minimal appservice (or even a scripted provisioner) whose only job is: register a handful of agent MXIDs, create the fleet presence room, and carry heartbeats; a minimal packages/comms slice that sets presence and heartbeats; native presence surfaced to a human via Element. Acceptance criteria:

A1. Single-site Synapse deployed (infra/matrix/), reachable over TLS, open registration OFF.
A2. ≥3 agents enroll (even if semi-manually) and appear in a fleet presence room with a live online/away/offline indicator.
A3. mosaic.presence heartbeat implemented; an agent killed hard flips to offline/dark within dark_threshold deterministically (not dependent on native presence timeout alone).
A4. A human (Jason) can open Element, join the fleet room, and see fleet liveness at a glance.
A5. Zero impact to existing tmux + mos-comms flows (they still work untouched).

P2 — Native appservice + orchestrator auto-enroll / room-provisioning

Scope: full apps/matrix-appservice registered with Synapse (hs_token/as_token, namespaces); mosaic enroll harness auto-detect; orchestrator-owned room taxonomy; per-agent token minting; introductions. Acceptance criteria:

B1. Appservice registered with Synapse via registration file; owns @mosaic_* user namespace and room-alias namespace.
B2. mosaic enroll auto-detects all four harnesses (Claude/Codex/Pi/Goose) and self-registers on spin, idempotently.
B3. On spin, an agent is provisioned an MXID, minted its own access token, joined to the correct mission/team/fleet rooms, and posts a mosaic.introduction.
B4. Orchestrator can create/destroy mission & team rooms with correct power levels via the appservice.
B5. Enroll bootstrap is authenticated (a rogue local process cannot enroll a rogue agent).
B6. Appservice + Synapse traced into OTEL/Jaeger.

P3 — MACP v1 spec ratified

Scope: freeze the standard (§4): envelope, event types + JSON Schemas, identity model, presence model, room taxonomy, tmux↔Matrix routing rules. Spec lives in framework (~/.config/mosaic/spec/macp), vendored+pinned into packages/comms with CI drift-check. Acceptance criteria:

C1. MACP v1 document ratified (MS-LEAD sign-off, Jason veto window closed).
C2. JSON Schemas for all five event types published; packages/comms validates outbound/inbound against them.
C3. CI fails if framework spec and vendored packages/comms copy diverge.
C4. Routing-rule conformance test: SDK provably sends ephemeral+same-host over tmux, everything else over Matrix.
C5. Unknown-macp_version graceful-degrade behavior tested.

P4 — Federation

Scope: a second site homeserver + appservice; S2S federation over our DNS/TLS; cross-site fleet room; whitelist-only federation; cross-site escalation. Acceptance criteria:

D1. Two sites federate via .well-known/SRV over TLS we control; federation_domain_whitelist restricts to Mosaic sites (no public federation).
D2. A cross-site fleet presence room shows agents from both sites; heartbeats propagate as events across federation.
D3. Homelab-dark test: killing site-A's coordinator is observed from site-B within dark_threshold, and the escalation/fallback fires cross-site (§5).
D4. Cross-site gate action: a site-B agent's signed review is verified by a site-A verifier.
D5. Defederation runbook proven (a site can be cut off cleanly).

P5 — Buzz-hardening + signed-authorship + Hermes retired

Scope: Ed25519 signed-authorship enforced on gate actions; security hardening pass; complete the strangler and retire Hermes at parity. Acceptance criteria:

E1. Every merge/approve/review gate action is signed; unsigned or bad-sig gate events are rejected by the appservice watcher and by consuming agents.
E2. Token/key rotation runbooks executed at least once in anger (rotate as_token, rotate a per-agent key).
E3. Security review complete: homeserver hardening flags, token custody, federation TLS/whitelist all verified.
E4. Parity checklist (§9) fully green + soaked.
E5. Hermes MCP bridge retired as backbone (optionally re-attached as an edge bridge only).

11. Open questions for Jason / Mos (need a human/coordinator ruling)

DNS / domains per site. What domain(s) do we own and want to use per site for homeserver names and .well-known (e.g. site-a.mosaicstack.dev)? Federation identity is permanent-ish once agents mint MXIDs against it — this needs a ruling before P1 hardens because MXIDs bake in the domain.
Secret-management choice. Is it Vault, or whatever KBN-101 lands? The appservice hs_token/as_token and enroll bootstrap secret custody depend on this (§8). CLAUDE.md signals secrets work is in flight — need the authoritative target.
N-minute escalation threshold. What is dark_threshold_min for a coordinator, and the second threshold before HIL is paged (§5)? Default proposed: 10 min → fallback, +5 min → HIL. Jason/Mos to confirm per role.
HIL client: Element vs custom. Do humans use off-the-shelf Element (fast, free, P1-ready) or do we invest in a custom HIL client? Proposed: Element for P1–P4, revisit custom only if HIL ergonomics demand it.
Ephemeral vs persistent agent keys. For signed-authorship, do we mint Ed25519 keys per-spin (simplest custody, no long-term private key at rest) or issue durable per-agent keys (stable identity across spins, but key-at-rest custody problem)? Proposed: per-spin, pubkey retained for audit.
Federation topology / trust. Full mesh between all sites, or hub-and-spoke-with-redundancy? Full mesh maximizes no-SPOF but grows O(n²); needs a call once site count is known.
Fallback-coordinator assignment authority. Who assigns each Team Lead's fallback coordinator, and is it always cross-site? (Design invariant §5 wants the fallback off the monitored host — confirm this is acceptable operationally.)
Retention / compliance. How long do we retain the Matrix event log (the unified audit trail)? Affects Synapse storage sizing and any purge policy.

Appendix A — Real Matrix concepts this RFC leans on (quick reference)

Application Service (AS) API — a privileged service registered with the homeserver via a registration YAML declaring id, url, as_token, hs_token, and namespaces (users/aliases/rooms regexes). The homeserver pushes events to the AS in transactions; the AS can register/act-as users in its namespace. (This is our appservice backbone.)
hs_token / as_token — hs_token: homeserver→AS authentication on pushed transactions; as_token: AS→homeserver authentication, grants acting-as any namespaced user. Both are high-value secrets (§8).
Masquerade (user_id query param) — the AS may act as a namespaced user on C-S calls using as_token + ?user_id=. We prefer per-agent tokens for blast-radius; masquerade for AS-internal bulk ops.
Custom event types — timeline events via m.room.message with a custom msgtype (keeps a human-visible body fallback) and/or fully custom type (dotted, e.g. mosaic.workflow.step); state events for last-writer-wins current status readable from room state without history replay.
Presence EDUs — native online/unavailable/offline signals; coarse and lossy over federation, so MACP adds an explicit heartbeat event as the authoritative liveness source (§4.5).
Federation (S2S API) — server-to-server over TLS; discovery via /.well-known/matrix/server and/or _matrix._tcp SRV; restrictable with federation_domain_whitelist.
Synapse config knobs cited — enable_registration, federation_domain_whitelist, appservice registration file, rate-limiting, admin API binding. [VERIFY] exact flags/paths against the deployed Synapse version at implementation time.

All Matrix mechanics above are cited from architecture knowledge and MUST be re-verified against the actual deployed Synapse version during P1 — every [VERIFY] in this document is a checkpoint, not an assumption.

47 KiB Raw Blame History Unescape Escape