mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests 4 Actions Packages Projects Releases 15 Wiki Activity

docs(rfc): add RFC-001 (MACP/Matrix-native comms) + RFC-002 (install/config/topology)
Some checks failed
ci/woodpecker/pr/ci Pipeline was canceled
Details

Browse Source 
Adds two design RFCs under docs/rfcs/ authored by MS-LEAD for the
Mosaic Stack comms-evolution program (sponsor/veto: Jason). RFC-001
is the MACP architecture (self-hosted Synapse + Mosaic appservice,
protocol, presence/escalation, federation, strangler migration off
Hermes). RFC-002 is the open-source install/config/topology substrate
RFC-001's phases run on. Docs only, no code. Links added to
docs/SITEMAP.md.
This commit is contained in:
mosaic-coder
2026-07-24 19:38:08 -05:00
parent 529c177830
commit 01dd2fadb3
3 changed files with 1009 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

569
docs/rfcs/RFC-002-INSTALL-CONFIG-TOPOLOGY.md Normal file
View File

@@ -0,0 +1,569 @@
# RFC-002 — Install, Configuration & Topology for the Mosaic Matrix/MACP Comms System
- **Status:** DRAFT — for Team Lead → Orchestrator staffing
- **Author:** MS-LEAD (reviewer identity `ms-lead-reviewer`)
- **Sponsor / veto:** Jason (human lead)
- **Date:** 2026-07-24
- **Program:** Mosaic Stack comms-evolution
- **Companion to:** RFC-001 — *MACP: A Mosaic-Native, Matrix-Native Comms Layer*. RFC-001 is the architecture (self-hosted Synapse + Mosaic appservice backbone + `packages/comms` SDK + MACP standard + per-site federation). **RFC-002 is the config substrate the whole thing installs and runs on.**
- **Audience:** Team Leads, the Mosaic orchestrator, infra, harness maintainers, and — critically — **strangers who install this open-source product on hardware we will never see.**
> This is a **design document**. No code ships from this RFC. It is written to be decomposed into missions with per-phase acceptance criteria, and it slots under RFC-001's P1→P5. Every uncertain or must-live-validate claim is flagged **[VERIFY]**.
> **The one framing that governs every decision below:** this is an **open-source product**. Someone we have never met will `git clone` it and run it on their own domains, their own DNS, their own certs, their own hardware. **NOTHING may hardcode our fleet's topology.** There is no `woltje.com` in the code, no assumption that DNS exists, no assumption that a second site exists. Every topology fact is **user-supplied config**. Where this doc uses `mosaic.woltje.com` / `mosaic.uscllc.com`, those are **illustrative operator values** (Jason's real installs), never defaults and never literals in the product.
---
## 0. TL;DR
The comms system installs against a **user-supplied topology**, never a baked-in one. At install the operator declares exactly one of **three topology modes**: **(A) split-domain** (identity `server_name` ≠ homeserver host, wired via Matrix delegation — this is Jason's `mosaic.woltje.com` identity + `matrix.woltje.com` host setup), **(B) single-domain** (`server_name` == homeserver host), or **(C) IP-only standalone** (no DNS, no federation, fully supported for local/airgapped). **The PRIMARY/home instance is ALWAYS configured; federation is OPTIONAL.** A single standalone instance MUST work with zero federation.
**Federation is a hard-gated capability: it REQUIRES DNS + valid certificates. IP-only federation is not possible and is not supported.** IP-only means standalone-only, forever, until the operator acquires DNS + certs.
Certificates are **one ACME integration** with a user-chosen **directory URL**: either **step-ca** (self-hosted private ACME CA, for total control and private/internal domains public CAs can't issue for) or **Let's Encrypt** (public ACME, ease-of-use). The operator also picks a challenge type (HTTP-01 / DNS-01 / TLS-ALPN-01); **DNS-01 is the answer for private/split-horizon domains.**
Secrets go through a **pluggable `SecretBackend` interface** — no forced paid dependency. Ships with a **Vault** implementation and a **Vaultwarden** implementation; the operator picks at install. The Vaultwarden model (org + orchestrator enrolled as authority + per-agent scoped access) is designed-for, with an honest **[VERIFY]** on how far Vaultwarden's machine-account coverage has matured.
Config is **DB-backed with sane defaults and install-time overrides**. Precedence: **install-time → DB override → default.** Config is split into **install-time-immutable** (e.g. `server_name`, which is baked into every MXID and cannot change without re-homing every identity) and **runtime-tunable** (e.g. dark-threshold).
RFC-002 is the substrate; **RFC-001's P1 (presence) needs only Mode A/B single-instance clean-domain and does NOT require federation, IP-only, or the secret-backend rotation story resolved.**
---
## 1. Goals / Non-Goals
### 1.1 Goals
- **G1 — Installable by a stranger.** A person with no relationship to our fleet can install, configure, and run the comms system from published artifacts and a guided installer, on their own hardware and domains, with no edits to product code.
- **G2 — Zero hardcoded topology.** Every topology fact — `server_name`, homeserver host/IP, delegation method, federation peers, cert mode, secret backend — is **user-supplied config**, validated at install, stored in the product DB. No fleet-specific literal ships in the product.
- **G3 — Standalone MUST work.** The PRIMARY instance is always fully functional with **zero federation**, including with **no DNS at all** (Mode C, IP-only). Presence, rooms, MACP, HIL-via-Element all work single-instance.
- **G4 — Federation is optional but honestly gated.** Federation is opt-in and, when opted into, **requires DNS + valid certificates as a hard precondition.** The installer must refuse to *claim* federation is working when the DNS/cert preconditions aren't met.
- **G5 — One ACME integration, two CA choices.** Build a single ACME cert-provisioning path; the operator selects step-ca or Let's Encrypt by supplying an **ACME directory URL** plus a challenge type. No second, bespoke cert path.
- **G6 — No forced paid dependency for secrets.** A pluggable `SecretBackend` with at least Vault and Vaultwarden implementations, chosen at install. Open-source ethos: the free/self-hostable path must be first-class.
- **G7 — Defaults that just work, overrides where they matter.** DB-backed config with sane defaults so most operators change little; install-time overrides for the topology-critical values; a clear immutable-vs-tunable boundary so operators can't foot-gun `server_name`.
- **G8 — A clean upgrade path.** An operator who starts standalone can later turn on federation with a documented, honest procedure (including the real cost if they started IP-only and must now acquire a stable `server_name`).
### 1.2 Non-Goals
- **NG1 — Not hosting a managed service.** This RFC is about *self-install*. We are not building multi-tenant SaaS provisioning; each operator runs their own instance(s).
- **NG2 — Not a new cert stack.** We do not write our own CA, our own ACME client protocol, or a non-ACME cert path. We integrate ACME and let the operator point it at step-ca or Let's Encrypt. (We *may* bundle/recommend step-ca as the self-hosted CA, but via its standard ACME provisioner, not a fork.)
- **NG3 — Not a new secret manager.** We define an interface and ship adapters. We do not build a secret store; we do not force one.
- **NG4 — Not public-network Matrix federation.** Consistent with RFC-001 NG5: federation is Mosaic-site-to-Mosaic-site over infrastructure the operator controls, allowlisted. No `matrix.org` federation.
- **NG5 — Not making IP-only federate.** We will not ship a hack (self-signed S2S trust bundles, `/etc/hosts` federation) that pretends IP-only can federate. IP-only is standalone. This is a deliberate, honest boundary (§2.4, §7).
- **NG6 — Not re-homing identities silently.** We will not offer a "just change your `server_name`" button that quietly orphans every MXID. Any path that changes `server_name` is a flagged, gated, documented identity re-home (§5.3, §7).
---
## 2. The topology model
### 2.1 The core split: `server_name` vs homeserver host
Matrix has exactly the split Jason described, natively:
- **`server_name`** — the Synapse config value that is the server's **identity domain**. It is the part after the colon in every MXID (`@mosaic_agent:mosaic.woltje.com`) and every room alias (`#mission-KBN-101:mosaic.woltje.com`). It is **baked into every identity the moment that identity is minted.** Changing it re-homes everything. This is `server_name` in Synapse's `homeserver.yaml`.
- **Homeserver host** — the actual network location (hostname:port or IP:port) where the Synapse process answers federation and (optionally proxied) client traffic. It **can differ** from `server_name`. Matrix reconciles the difference through **delegation**: `https://<server_name>/.well-known/matrix/server` returning `{"m.server": "matrix.woltje.com:443"}`, and/or a `_matrix._tcp.<server_name>` **SRV** record. **[VERIFY]** `.well-known` vs SRV precedence on the deployed Synapse version (RFC-001 §6 flags the same).
So Jason's "mosaic.* app-domain + matrix.* homeserver-domain" split maps precisely: **`server_name = mosaic.woltje.com` (identity, in MXIDs), homeserver runs at `matrix.woltje.com` (discovered via delegation).** That is **Mode A**.
### 2.2 The topology config schema
One canonical config object, stored in the product DB (§5), populated at install (§6). Illustrative shape (field names decomposition-ready, not frozen):
```jsonc
{
"topology": {
"mode": "split-domain | single-domain | ip-only-standalone", // A | B | C — install-time-immutable
"identity": {
"server_name": "mosaic.woltje.com", // INSTALL-TIME-IMMUTABLE. In MXIDs. Never change without re-home (§5.3).
"server_name_kind": "domain | ip" // "ip" only legal in Mode C
},
"homeserver": {
"host": "matrix.woltje.com", // where Synapse actually listens (Mode A: differs from server_name)
"port": 8448, // federation port (default 8448) or 443 if proxied — [VERIFY] per deploy
"client_bind": "https://matrix.woltje.com", // C-S API public URL (proxied)
"bind_ip": null // Mode C: e.g. "192.168.1.50" ; Modes A/B: null (DNS-resolved)
},
"delegation": {
"method": "well-known | srv | none", // Mode A: well-known or srv ; Mode B/C: none
"well_known_server": { "m.server": "matrix.woltje.com:443" }, // if method=well-known
"srv_record": "_matrix._tcp.mosaic.woltje.com. 3600 IN SRV 10 0 443 matrix.woltje.com." // if method=srv (documented, operator provisions)
},
"federation": {
"enabled": true, // OPTIONAL. Mode C forces false.
"domain_whitelist": [ // Synapse federation_domain_whitelist — allowlist ONLY
"mosaic.woltje.com",
"mosaic.uscllc.com"
],
"peers": [ // operator-declared peer sites (for room/fleet wiring)
{ "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true }
]
},
"tls": {
"acme": {
"directory_url": "https://acme.mosaic.woltje.com/acme/acme/directory", // step-ca OR https://acme-v02.api.letsencrypt.org/directory
"ca_kind": "step-ca | letsencrypt", // informational label; the directory_url is the real switch
"challenge": "dns-01 | http-01 | tls-alpn-01",
"account_email": "ops@woltje.com", // ACME account contact
"eab": { "kid": null, "hmac_key_ref": null } // External Account Binding if the CA requires it (some step-ca provisioners) — secret via SecretBackend
},
"client_tls_mode": "acme | self-signed" // Mode C may use self-signed for local C-S TLS (weaker trust, §8)
},
"secrets": {
"backend": "vault | vaultwarden", // pluggable, install-time choice (§4)
"connection": {
"address": "https://vault.woltje.com:8200", // or Vaultwarden/Bitwarden server URL
"auth_ref": "…", // how the appservice authenticates to the backend (bootstrap, §4/§8)
"namespace_or_org": "mosaic-fleet" // Vault namespace / mount, OR Vaultwarden org id
}
}
}
}
```
### 2.3 The three modes, concretely
Exactly three supported modes (Jason's ruling — no others):
**Mode A — split-domain (identity ≠ host, delegated).** *Jason's PRIMARY.* Federation-capable. This is the recommended production shape because it lets identity live on a clean app-domain while the homeserver runs on a separate operational host.
```jsonc
// Mode A — mosaic.woltje.com identity, matrix.woltje.com host, federated with a second site
{ "topology": {
"mode": "split-domain",
"identity": { "server_name": "mosaic.woltje.com", "server_name_kind": "domain" },
"homeserver": { "host": "matrix.woltje.com", "port": 443, "client_bind": "https://matrix.woltje.com", "bind_ip": null },
"delegation": { "method": "well-known", "well_known_server": { "m.server": "matrix.woltje.com:443" } },
"federation": {
"enabled": true,
"domain_whitelist": ["mosaic.woltje.com", "mosaic.uscllc.com"],
"peers": [ { "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true } ]
},
"tls": { "acme": {
"directory_url": "https://acme-v02.api.letsencrypt.org/directory", // public LE, or a step-ca directory
"ca_kind": "letsencrypt", "challenge": "dns-01", "account_email": "ops@woltje.com" },
"client_tls_mode": "acme" },
"secrets": { "backend": "vaultwarden",
"connection": { "address": "https://vw.woltje.com", "namespace_or_org": "mosaic-fleet" } }
} }
```
MXIDs on this instance: `@mosaic_coordinator-1:mosaic.woltje.com`. A human/agent's homeserver is discovered by resolving `.well-known/matrix/server` on `mosaic.woltje.com` → `matrix.woltje.com:443`.
**Mode B — single-domain (identity == host).** Simpler; the `server_name` *is* the host. No delegation needed. Federation-capable (still needs DNS + cert on that one domain).
```jsonc
// Mode B — one domain does everything
{ "topology": {
"mode": "single-domain",
"identity": { "server_name": "matrix.example.org", "server_name_kind": "domain" },
"homeserver": { "host": "matrix.example.org", "port": 8448, "client_bind": "https://matrix.example.org", "bind_ip": null },
"delegation": { "method": "none" },
"federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // optional — off here
"tls": { "acme": {
"directory_url": "https://acme-v02.api.letsencrypt.org/directory",
"ca_kind": "letsencrypt", "challenge": "http-01", "account_email": "admin@example.org" },
"client_tls_mode": "acme" },
"secrets": { "backend": "vault",
"connection": { "address": "https://vault.example.org:8200", "namespace_or_org": "mosaic" } }
} }
```
MXIDs: `@mosaic_coordinator-1:matrix.example.org`. Here `server_name == host`, so `@:matrix.example.org` is both the identity domain and where Synapse actually answers.
**Mode C — IP-only standalone (no DNS, no federation).** Fully supported for local/airgapped/homelab-without-DNS. `server_name` is an IP:port literal. **Cannot federate — ever — in this mode** (federation requires a real domain + valid CA cert; §2.4). Client TLS may be self-signed (weaker trust, §8).
```jsonc
// Mode C — airgapped / local, no DNS, standalone only
{ "topology": {
"mode": "ip-only-standalone",
"identity": { "server_name": "192.168.1.50:8448", "server_name_kind": "ip" },
"homeserver": { "host": "192.168.1.50", "port": 8448, "client_bind": "https://192.168.1.50:8448", "bind_ip": "192.168.1.50" },
"delegation": { "method": "none" },
"federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // FORCED false in Mode C
"tls": { "acme": null, "client_tls_mode": "self-signed" }, // may use a private step-ca or self-signed for C-S TLS
"secrets": { "backend": "vaultwarden",
"connection": { "address": "http://192.168.1.51:8080", "namespace_or_org": "mosaic-local" } }
} }
```
MXIDs: `@mosaic_coordinator-1:192.168.1.50:8448`. **Warning surfaced at install:** this `server_name` is an IP literal; if the operator ever wants federation they must move to a domain, which is an **identity re-home** (§5.3, §7).
> **[VERIFY]** Synapse accepts an `ip:port` `server_name` and mints usable MXIDs against it for local/standalone use. This is believed workable for non-federated operation but must be validated against the deployed Synapse version; some Synapse versions/tools assume a DNS-resolvable `server_name`. If an IP literal is rejected, Mode C falls back to a **fabricated local domain** (e.g. `mosaic.local`) resolved via `/etc/hosts` or a local resolver — still standalone-only, same re-home caveat.
### 2.4 The hard federation gate (Jason's HARD STOP)
**Federation REQUIRES DNS + valid certificates. This is a hard stop, enforced by the installer and by the config validator, not a suggestion.**
| Precondition | Why | Enforced where |
|---|---|---|
| `server_name` resolves in public/peer DNS (or delegated target does) | S2S discovery uses `.well-known`/SRV over DNS; peers must resolve you | installer reachability check (§6); config validator rejects `federation.enabled=true` with `server_name_kind=ip` |
| Valid TLS cert on the federation endpoint, chained to a CA the peer trusts | S2S is TLS; a peer validates your cert. Self-signed/untrusted ⇒ peer refuses ⇒ you are defederated | installer cert probe (§6); cert monitor (§3.4) |
| Federation `domain_whitelist` non-empty and mutually consistent with peers | allowlist-only federation (RFC-001 NG5/§6) | config validator |
**IP-only ⇒ federation is impossible.** There is no valid public/peer CA cert for a bare IP in our trust model (and we will not ship a self-signed S2S trust hack — NG5). Therefore **Mode C is standalone-only by construction**, and the config validator makes `mode=ip-only-standalone ∧ federation.enabled=true` an **illegal state that cannot be persisted.**
This is the honest, load-bearing boundary of the whole topology model:
```
DNS + valid cert?
┌─────────────┴─────────────┐
YES NO
│ │
Mode A or B Mode C (IP-only)
federation OPTIONAL STANDALONE ONLY
(opt-in, allowlisted) (federation impossible)
```
---
## 3. Certificate provisioning — one ACME integration
### 3.1 Single integration, directory-URL as the switch
We build **exactly one** cert-provisioning integration: an **ACME client integration**. Both supported CAs are ACME CAs. The operator does **not** choose between two code paths; they choose an **ACME directory URL** and a **challenge type**. That is the entire surface.
| CA choice | What it is | ACME directory URL (illustrative) | Why an operator picks it |
|---|---|---|---|
| **step-ca (Smallstep)** | self-hosted **private** ACME CA | `https://acme.<internal-domain>/acme/<provisioner>/directory` | **Total control**; issues for **private/internal/split-horizon domains** a public CA can't (e.g. `mosaic.internal`, RFC-1918 split-horizon); airgap-friendly; you own the root |
| **Let's Encrypt** | public ACME CA | `https://acme-v02.api.letsencrypt.org/directory` (staging: `.../acme-staging-v02...`) | **Ease of use**; universally trusted chain (ISRG Root X1); zero CA to operate; ideal for public domains |
Because both speak ACME, the same client (account key, order, authorization, challenge, finalize, cert-fetch, renew) drives either. The `ca_kind` label in config is informational for UX; the **`directory_url` is the real determinant**. **[VERIFY]** whether the chosen ACME library requires per-CA quirks (LE rate limits, staging switch; step-ca **External Account Binding** on some provisioners — if EAB is required the `kid`/`hmac_key` come from the SecretBackend, §4).
### 3.2 Challenge-type matrix (which challenge for which topology)
The operator picks one challenge type per the domains they're covering. This is the crux for **public vs private/split-horizon**:
| Challenge | How it proves control | Best for | Cannot / caveat |
|---|---|---|---|
| **HTTP-01** | CA GETs `http://<domain>/.well-known/acme-challenge/<token>` on port 80 | **Public, single hostname**, port 80 reachable from CA (Mode B, or Mode A's homeserver host) | Needs inbound :80 from the CA; **cannot** do wildcards; **useless for private domains** a public CA can't reach |
| **DNS-01** | CA checks a `_acme-challenge.<domain>` **TXT** record you publish | **Private / split-horizon / internal domains**, **wildcards**, and any domain where inbound HTTP from the CA is impossible | Requires **programmatic DNS API** access to publish TXT (or manual for step-ca where you own the resolver). **This is the answer for step-ca on private domains** and for Mode A when the homeserver host isn't publicly HTTP-reachable |
| **TLS-ALPN-01** | CA connects TLS on :443 with ALPN `acme-tls/1` | Public host where **:443 is free** but :80 is blocked | Needs the ACME client to own the :443 TLS handshake briefly; awkward behind some reverse proxies — **[VERIFY]** against our proxy (RFC-001 terminates TLS at a reverse proxy) |
**Guidance baked into the installer:**
- **Let's Encrypt + public domain, port 80 open →** HTTP-01 (simplest). Wildcard or no inbound :80 → DNS-01.
- **step-ca + private/internal/split-horizon domain →** **DNS-01** (the private CA can validate against a resolver you control; public HTTP reachability is irrelevant). This is the combination that lets a private homelab domain get real certs.
- **:443-only public host →** TLS-ALPN-01.
### 3.3 Delegation setup for split-domain (Mode A)
For Mode A, the cert and the delegation must agree. Concretely, at install for `server_name=mosaic.woltje.com`, host `matrix.woltje.com`:
1. **Cert(s):** obtain a valid cert for **`matrix.woltje.com`** (the federation/host endpoint — this is where the TLS handshake actually lands). If serving `.well-known` over HTTPS on `mosaic.woltje.com`, that origin **also** needs a valid cert for `mosaic.woltje.com`. So Mode A typically provisions **two** SANs/certs: identity-domain (`mosaic.woltje.com`, serves `.well-known`) and host (`matrix.woltje.com`, serves S2S+C-S). **[VERIFY]** whether a single multi-SAN cert is preferable operationally.
2. **Delegation record**, one of:
- **`.well-known`:** serve `https://mosaic.woltje.com/.well-known/matrix/server` → `{"m.server":"matrix.woltje.com:443"}` (and `.well-known/matrix/client` for C-S discovery so agents/Element find the host).
- **SRV:** `_matrix._tcp.mosaic.woltje.com. IN SRV 10 0 443 matrix.woltje.com.` The installer **documents and validates** the record but the operator provisions it in their DNS (we don't run their DNS). **[VERIFY]** `.well-known` vs SRV precedence on the deployed Synapse.
3. **Validate:** installer fetches the operator's own `.well-known`/SRV and confirms it points at the configured host, and that the host presents a valid cert (§6). Only then does it declare Mode A "federation-ready."
### 3.4 Renewal & monitoring — a lapsed federation cert silently defederates
**This is the operational trap and it must alarm.** ACME certs are short-lived (LE = 90 days; step-ca often shorter by policy). A federation cert that lapses does **not** throw a loud error — peers simply **stop trusting the S2S handshake and the site silently drops out of federation.** From inside, everything looks fine; from peers, the site went dark. That is exactly the "homelab went dark and took comms with it" trauma (RFC-001 §5), but caused by a cert, not a host.
Requirements:
- **Auto-renew** on the standard ACME schedule (renew at ~⅓ lifetime remaining; LE guidance ~30 days before expiry). The ACME integration owns this loop.
- **Expiry monitoring as a first-class alarm.** Emit cert-days-remaining into OTEL/Jaeger metrics (consistent with RFC-001 §8's "monitor for cert expiry — a cert lapse silently defederates"). Alarm thresholds (e.g. warn <14d, critical <3d) are **runtime-tunable** config (§5).
- **Federation-health probe:** periodically resolve our own delegation and validate our own cert *as a peer would* (external vantage where possible), so a broken renewal is caught as "we would fail a peer's validation" before a peer notices.
- **Escalation tie-in:** a critical cert-expiry or federation-health failure raises a `mosaic.escalation` (RFC-001 §4.2/§5) into the HIL room. A cert lapse is a fleet-visibility incident, not a silent config drift.
---
## 4. Secret backend interface
### 4.1 The `SecretBackend` contract
A single pluggable interface. The appservice and orchestrator depend on the **interface**, never on Vault or Vaultwarden directly. Chosen at install; swappable without touching callers. Illustrative contract (decomposition-ready, not frozen):
```ts
interface SecretBackend {
// --- static secret CRUD (appservice tokens, ACME EAB, DB creds) ---
get(ref: SecretRef): Promise<SecretValue>;
put(ref: SecretRef, value: SecretValue, opts?: { immutable?: boolean }): Promise<void>;
rotate(ref: SecretRef, next: SecretValue): Promise<{ previous: SecretVersion; current: SecretVersion }>;
list(prefix: SecretRef): Promise<SecretRef[]>;
delete(ref: SecretRef): Promise<void>;
// --- agent-credential lifecycle (the fleet-identity part) ---
enrollAgent(input: {
agentSlug: string;
scope: CredentialScope; // which rooms/secrets this agent may read
ttl?: Duration; // ephemeral-by-default per RFC-001 §8
}): Promise<AgentCredentialHandle>; // wraps the per-agent access_token + optional pubkey record
revokeAgent(agentSlug: string): Promise<void>; // must be authoritative & immediate
// --- health / bootstrap ---
health(): Promise<BackendHealth>;
authenticateSelf(bootstrap: BootstrapAuth): Promise<void>; // how the appservice/orchestrator logs into the backend
}
```
Design intent: **`get/put/rotate`** cover the static crown-jewel secrets (appservice `hs_token`/`as_token`, ACME account/EAB keys, DB DSN). **`enrollAgent/revokeAgent`** cover the *fleet-identity* lifecycle — this is where RFC-001's "mint per-agent token at enroll, discard on teardown" (RFC-001 §4.1, §8) actually lands.
### 4.2 How appservice / agent tokens map onto it
RFC-001 defines three tiers of Matrix secret. They map cleanly:
| RFC-001 secret | Sensitivity | `SecretBackend` treatment |
|---|---|---|
| **`as_token`** (AS→HS, acts as any namespaced user) | crown jewel | `put(immutable-ish)` + `rotate`; **only** the appservice may `get` it; stored under a fleet-admin scope; never handed to an agent |
| **`hs_token`** (HS→AS callback auth) | crown jewel | same as `as_token`; both live only in appservice scope (RFC-001 §8) |
| **enroll bootstrap secret / orchestrator-signed nonce** | high | `get` by orchestrator + appservice only; used to authenticate `POST /enroll` so a rogue process can't enroll a rogue agent (RFC-001 §8, B5) |
| **per-agent `access_token`** | transient runtime | minted via `enrollAgent`, scoped to that agent, **short-TTL / re-mintable**, discarded on teardown; **not** durably persisted (RFC-001 §8 says per-agent tokens are transient) — the backend may hold a short-lived handle or a personal-vault entry for the agent's own lifetime |
| **Ed25519 signed-authorship keypair** | high (private key) | private key generated **agent-side**, only pubkey leaves the agent (RFC-001 §4.4/§8); the SecretBackend stores the **pubkey record** for audit; per-spin keys need no at-rest custody |
The key blast-radius property (RFC-001 §8) is preserved: agents receive **only their own** credential via `enrollAgent`; the `as_token` never leaves appservice scope.
### 4.3 Vault implementation
Vault maps naturally:
- Static secrets → **KV v2** at a mount/namespace (`mosaic-fleet/`), with versioning giving `rotate` semantics for free.
- **`enrollAgent`** → issue a scoped, TTL'd token or use **AppRole** / a scoped policy per agent; Vault's native TTL + revocation is exactly the transient per-agent model. `revokeAgent` → Vault token/lease revoke (authoritative, immediate).
- **`authenticateSelf`** → the appservice authenticates to Vault via AppRole (role_id from config, secret_id injected at deploy) or a platform auth method; consistent with how Gateway/DB secrets are handled today (RFC-001 §8). **[VERIFY]** align with whatever KBN-101 lands for Mosaic secret management (CLAUDE.md flags secrets work in flight).
- **Trade-off (honesty):** Vault is the most capable backend but is **not** the free-and-simple default for a hobbyist stranger; hence it must not be *forced* (G6).
### 4.4 Vaultwarden implementation + the org/enroll/revoke agent-account model
Vaultwarden (self-hostable Bitwarden-compatible server) is the **open-source-ethos default candidate** — free, self-hostable, familiar. Jason's model, mapped onto Bitwarden/Vaultwarden's org primitives:
1. **Operator creates one or more Bitwarden orgs** at install (e.g. `mosaic-fleet`).
2. **The orchestrator is enrolled into the org and granted authority** to enroll/revoke agent sessions — it is the org's automation principal (admin/manager over an agents **collection**).
3. **Agents get scoped credential access:** each agent gets access to a **collection** (or a personal vault provisioned for its spin) holding exactly the secrets its scope allows. `enrollAgent` = grant the agent principal access to its collection + provision its per-agent Matrix token entry; `revokeAgent` = remove the agent principal / revoke its access, immediately.
4. **User + agents share scoped access:** the human operator and the agents both hold credentials in the same org, scoped by collection — humans and agents on one secret surface, mirroring RFC-001's "humans and agents on one comms surface" pattern.
Mapping to the interface:
| Interface op | Vaultwarden mechanism |
|---|---|
| `get/put/rotate` (static) | items in an org **collection**; rotate = new item version / replace + old-version audit |
| `enrollAgent` | create/attach agent principal to its **collection**; provision per-agent token item, TTL enforced by our teardown (Vaultwarden itself is not TTL-native — see [VERIFY]) |
| `revokeAgent` | revoke the agent principal's org membership / collection access |
| `authenticateSelf` | orchestrator authenticates as the org automation principal |
**HONEST MATURITY FLAG — [VERIFY]:** Bitwarden's clean **machine-account / service-account** primitive lives in **Bitwarden Secrets Manager**, and **Vaultwarden's coverage of Secrets Manager / machine accounts is partial and evolving.** What is known to work today on Vaultwarden: **orgs, collections, per-user (incl. a per-agent "user") vaults, and collection-scoped sharing.** What **may not** be fully there: the polished **machine-account API**, native short-TTL service credentials, and fine-grained programmatic access-token issuance equivalent to hosted Bitwarden Secrets Manager. **[VERIFY]** current Vaultwarden version's Secrets Manager / machine-account support before P-level commitment.
**Why this is not a blocker:** the interface is designed so **either backend is viable**. If Vaultwarden's machine-account API isn't ready, the Vaultwarden adapter implements `enrollAgent` via the **personal-vault-per-agent + org-collection** model that works *today* (create an agent principal, share the scoped collection, we enforce TTL via orchestrator teardown rather than backend-native TTL). If an operator needs backend-native short-TTL machine credentials now, they choose the **Vault** adapter. **We are not blocked on Vaultwarden maturing**, because the `SecretBackend` abstraction lets the polished-machine-account behavior land later without changing any caller.
---
## 5. Config system
### 5.1 Storage & precedence
Config is **DB-backed** (Postgres, per the stack — CLAUDE.md/RFC-001), with sane defaults compiled into the product and install-time overrides. **Precedence, highest wins:**
```
install-time value > DB override (runtime) > compiled default
```
- **Compiled default** — ships in the product; what a stranger gets with zero config for every non-topology-critical key.
- **Install-time value** — captured by the installer (§6), written to DB, and for **install-time-immutable** keys, **locked** (marked non-overridable).
- **DB override** — runtime tuning via admin surface, allowed **only** for keys classified runtime-tunable.
> Nuance: "install-time > DB override" applies to **immutable** keys — the install-time value is frozen and a DB override of it is rejected. For **tunable** keys, the DB override is the live value and the install-time value is just the initial seed. The classification (§5.3) is what makes the precedence unambiguous per key.
### 5.2 DB schema shape
Illustrative (Drizzle/Postgres, per stack conventions):
```sql
-- one row per config key
CREATE TABLE comms_config (
key text PRIMARY KEY, -- e.g. 'topology.identity.server_name'
value jsonb NOT NULL, -- current effective value
source text NOT NULL, -- 'install' | 'db-override' | 'default'
mutability text NOT NULL, -- 'install-immutable' | 'runtime-tunable'
set_by text, -- operator/agent/system that set it
set_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT immutable_not_overridable
CHECK (NOT (mutability = 'install-immutable' AND source = 'db-override'))
);
-- append-only audit of every change (esp. attempted immutable changes)
CREATE TABLE comms_config_audit (
id bigserial PRIMARY KEY,
key text NOT NULL,
old_value jsonb,
new_value jsonb,
actor text NOT NULL,
action text NOT NULL, -- 'set' | 'override' | 'rejected-immutable'
at timestamptz NOT NULL DEFAULT now()
);
```
The DB `CHECK` is a belt-and-braces backstop; the application-layer config service enforces mutability and records rejected immutable-change attempts in the audit table. Secrets are **referenced** here (a `SecretRef`), never stored inline — actual secret values live in the `SecretBackend` (§4).
### 5.3 Install-time-immutable vs runtime-tunable — the key table
The single most important classification: **what can never change after install** vs **what an operator tunes anytime.** Getting `server_name` on the wrong side of this line is a foot-gun that orphans every identity.
| Config key | Mutability | Rationale / cost of change |
|---|---|---|
| `topology.mode` (A/B/C) | **install-immutable** | changing mode changes identity/federation semantics; a mode change is effectively a reinstall/re-home |
| `topology.identity.server_name` | **install-immutable** | **baked into every MXID and room alias.** Changing it re-homes every identity — see re-home note below. **This is THE immutable value.** |
| `topology.identity.server_name_kind` | **install-immutable** | domain↔ip change is a re-home (Mode C→A/B) |
| `topology.homeserver.host` | install-immutable (**delegation-tunable**) | in Mode A you *can* move the host if you update delegation to match — the identity is unchanged; treat as immutable-with-migration, not free |
| `topology.homeserver.bind_ip` / `port` | runtime-tunable (ops) | operational network binding; no identity impact |
| `topology.delegation.method` / records | tunable-with-care | can switch well-known↔SRV as long as both still resolve to the same host; validated on change |
| `topology.federation.enabled` | **runtime-tunable (gated)** | can flip **on** only if DNS+cert preconditions pass (§2.4); flipping standalone→federated is the §7 upgrade path |
| `topology.federation.domain_whitelist` / `peers` | runtime-tunable | add/remove peers over time; each add re-validated |
| `tls.acme.directory_url` / `ca_kind` | runtime-tunable | can switch CA (e.g. LE→step-ca); triggers re-issue; monitor for trust-chain change |
| `tls.acme.challenge` | runtime-tunable | switch challenge type if DNS/HTTP reachability changes |
| `tls.client_tls_mode` | tunable-with-care | self-signed→acme is fine; acme→self-signed weakens trust (§8) |
| `secrets.backend` | install-immutable (**migration-only**) | switching Vault↔Vaultwarden requires a secret migration; not a live flip |
| `secrets.connection.*` | runtime-tunable | rotate backend address/auth without changing which backend |
| `presence.heartbeat_interval_ms` | runtime-tunable | RFC-001 §4.5 default ~30s; pure tuning |
| `presence.miss_tolerance` | runtime-tunable | RFC-001 §4.5 default 2 |
| `escalation.dark_threshold_min` | runtime-tunable | RFC-001 §5/§11 — default 10min → fallback |
| `escalation.hil_threshold_min` | runtime-tunable | RFC-001 §5 — default +5min → HIL |
| `cert.expiry_warn_days` / `expiry_critical_days` | runtime-tunable | §3.4 alarm thresholds (default 14 / 3) |
| `federation.health_probe_interval` | runtime-tunable | §3.4 |
**Identity re-home note (the cost of changing `server_name`):** because every MXID (`@mosaic_*:server_name`) and room alias (`#…:server_name`) embeds `server_name`, changing it means: every agent gets a **new identity**, all rooms must be **recreated/re-aliased**, signed-authorship pubkey records re-published, and federation peers re-pointed. There is **no in-place rename** in Matrix. Hence `server_name` is install-immutable and the installer gates it behind an explicit warning (§6.7). Changing it is a **migration/reinstall**, honestly (§7).
### 5.4 How the appservice / homeserver read config
- **Synapse (homeserver)** reads a *rendered* `homeserver.yaml`. The config service **renders** Synapse's config (server_name, listeners, `federation_domain_whitelist`, `enable_registration: false`, appservice registration path, TLS/delegation) from the DB-backed config at deploy/reconfigure time. Synapse itself is not DB-config-aware; the source of truth is the product DB, and Synapse config is a **generated artifact**. Certain Synapse values (notably `server_name`) require a **Synapse restart** and are exactly the immutable ones — reinforcing §5.3.
- **The appservice** reads config **live** from the DB config service for runtime-tunable values (thresholds, whitelist changes, cert alarm thresholds) and from the `SecretBackend` for secrets. Immutable topology values are read once at boot (they can't change under it).
- **`packages/comms` SDK** receives the values it needs (homeserver client URL, presence intervals) from the appservice at enroll (RFC-001 §4.1 returns `{mxid, access_token, homeserver, rooms[]}`), so agents never read the config DB directly.
- **Delegation artifacts** (`.well-known/matrix/server`, `.well-known/matrix/client`) are likewise **rendered** from config and served by the reverse proxy / homeserver.
---
## 6. Installer UX flow
A guided installer (`mosaic comms install` or equivalent) that captures topology, provisions certs, wires secrets, and **validates before declaring success.** It must never report success it hasn't proven. Steps:
**6.1 — Preflight & detection.** Detect existing DNS records for a candidate domain, existing certs, an existing reachable Synapse, an existing Vault/Vaultwarden. Offer detected values as suggestions (never as silent defaults). Detect whether the host has public inbound :80/:443 (informs challenge-type guidance, §3.2).
**6.2 — Primary instance (ALWAYS).** The PRIMARY/home instance is always configured — there is no "skip primary." Prompt for its identity. This is non-optional and is what makes standalone work out of the box.
**6.3 — Pick topology mode (A/B/C).** Ask the shape:
- Do you have a domain? **No →** Mode C (IP-only standalone); warn federation is impossible here (§2.4) and that `server_name` will be an IP (re-home cost if they later want federation).
- Yes, and identity domain differs from the homeserver host? **Yes →** Mode A (split-domain); capture `server_name` + host + delegation method.
- Yes, one domain does everything? **→** Mode B (single-domain).
**6.4 — Pick CA (ACME directory).** step-ca vs Let's Encrypt → capture `directory_url`, account email, and challenge type with the §3.2 guidance surfaced (e.g. "private/internal domain? → DNS-01"). Capture EAB if the CA requires it (→ SecretBackend). For step-ca, offer to point at an existing step-ca or document standing one up.
**6.5 — Pick secret backend.** Vault vs Vaultwarden → capture connection (address, org/namespace, bootstrap auth). If Vaultwarden, walk the org/collection setup (§4.4) and **surface the machine-account maturity [VERIFY]** honestly so the operator chooses eyes-open.
**6.6 — Federation (OPTIONAL).** Only offered if Mode A/B. Ask whether to enable federation now; if yes, capture peer `server_name`s and build the `domain_whitelist`. If Mode C, federation is not offered (greyed out with the explanation). Federation-off is a first-class, fully-supported outcome.
**6.7 — The "what can't be changed later" warning gate.** Before writing immutable config, present an explicit confirmation:
```
⚠ IMMUTABLE CHOICES — read before confirming
server_name = "mosaic.woltje.com"
This becomes part of every agent identity (e.g. @mosaic_coordinator-1:mosaic.woltje.com)
and every room alias. It CANNOT be changed later without re-homing every identity
(new MXIDs for all agents, recreating all rooms). There is no in-place rename in Matrix.
topology.mode = "split-domain" — changing modes later is a reinstall.
secrets.backend = "vaultwarden" — switching backends later requires a secret migration.
Type the server_name to confirm you understand it is permanent: ____________
```
The operator must **retype `server_name`** to proceed — a deliberate friction gate on the one truly permanent value.
**6.8 — Provision & validate (no success claim until proven).** The installer then:
1. Renders Synapse config + delegation artifacts; brings up Synapse with `enable_registration: false`.
2. Runs the ACME flow; obtains cert(s); verifies they're valid and installed.
3. Authenticates to the SecretBackend; stores `hs_token`/`as_token`, enroll bootstrap; runs `health()`.
4. **Reachability & cert validation** appropriate to mode:
- Mode A: fetch our own `.well-known`/SRV, confirm it points at host; TLS-probe host cert as a peer would; confirm C-S discovery.
- Mode B: TLS-probe the single domain; confirm C-S + (if federation) S2S.
- Mode C: confirm local C-S reachability over the bind IP; confirm (self-signed or private-CA) client TLS; **explicitly report "standalone — federation not available."**
- If federation enabled: validate each peer resolves + presents a peer-trusted cert; confirm `domain_whitelist` mutual consistency. If any peer fails, **federation is reported NOT-ready** — the primary still succeeds standalone.
5. **Only now** declare success, with a per-capability report: `PRIMARY: ✅ | CERT: ✅ (expires in 90d, auto-renew on) | SECRETS: ✅ (vaultwarden) | FEDERATION: ✅ 1 peer / ⚠ not-ready / ⛔ n-a (Mode C)`.
**6.9 — Post-install.** Emit the cert-expiry monitor + federation-health probe (§3.4) into OTEL; write config to DB with correct mutability flags; print the immutable-values summary again for the record.
---
## 7. Standalone → federated upgrade path
An operator who started standalone later wants federation. The path depends on **whether they started with a domain**:
**Case 1 — started Mode A or B (had a domain), federation was just off.** *Cheap, no re-home.* `server_name` is already a real domain and identities are already minted against it. To federate:
1. Ensure DNS resolves for peers (their `server_name`s and yours) — likely already true.
2. Ensure a **valid, peer-trusted cert** on the federation endpoint (if they were running client-only self-signed, they now need a real ACME cert; if already ACME, done).
3. Set `federation.enabled = true`, populate `domain_whitelist` + `peers` (all runtime-tunable, §5.3).
4. Re-run the installer's **federation validation** (§6.8 step 4) against each peer. On green, federation is live. **No identity change** — existing MXIDs simply become reachable cross-site. This is the intended, low-friction upgrade.
**Case 2 — started Mode C (IP-only), now wants federation.** *Expensive — an identity re-home, and we say so plainly.* Federation requires DNS + a valid cert (§2.4), which an IP `server_name` can never satisfy. So the operator must:
1. **Acquire a domain** and DNS, and provision a **valid ACME cert** (LE public, or step-ca if the domain is private — but note a *private* domain can only federate with peers who trust that private CA root, §8).
2. **Change `server_name` from the IP literal to the domain** — this is the **install-immutable value**, so this is a **re-home, not a config tweak**:
- Every agent identity `@mosaic_*:192.168.1.50:8448` becomes `@mosaic_*:newdomain` — **all new MXIDs.**
- Every room + alias must be **recreated** under the new `server_name`.
- Signed-authorship pubkey records re-published under the new identities.
- Any durable references to old MXIDs (escalation policies, fallback-coordinator targets, RFC-001 §5) must be re-pointed.
3. Effectively: **treat it as a fresh install in Mode A/B with a data migration of rooms/history**, not an in-place flip. Matrix has **no in-place `server_name` rename**; this cost is intrinsic to Matrix, not to our design.
**Honest guidance the installer gives Mode C operators up front (§6.3):** "If there is *any* chance you'll want to federate later, start with a domain (Mode A/B) even if you keep federation off — flipping federation on later is free, but changing an IP `server_name` to a domain later is a full identity re-home." This lets an informed operator avoid the expensive path by choosing Mode B-with-federation-off instead of Mode C.
---
## 8. Security
**8.1 — Cert trust model per CA choice.**
- **Let's Encrypt (public):** chains to a universally-trusted root (ISRG). Peers, humans' browsers, and Element trust it with no extra distribution. Best for public domains; nothing to distribute.
- **step-ca (private):** chains to a **root you operate**. Nothing trusts it by default. Therefore the **step-ca root must be distributed** to everyone who validates certs: peer homeservers (so cross-site S2S validates — a peer must add your root to its federation trust store, **[VERIFY]** Synapse's mechanism for trusting a custom federation CA), agent hosts, and any human client. This is the price of "total control" and airgap capability. For **federation between two private-CA sites**, both sites must trust each other's roots (or a shared root). Getting this wrong reproduces the silent-defederation failure (§3.4) — a peer that doesn't trust your root silently refuses your S2S.
- **Mode C self-signed client TLS:** weakest — see 8.4.
**8.2 — Federation whitelist.** `federation_domain_whitelist` is a **hard allowlist** (RFC-001 §6/NG5): only listed Mosaic site domains may federate; no public-network federation. The installer/config validator keeps the whitelist consistent with the declared peer list. Adding a peer is an explicit, audited config change.
**8.3 — Secret-backend auth.** The appservice/orchestrator authenticate to the `SecretBackend` via a **bootstrap credential injected at deploy** (Vault AppRole secret_id, or Vaultwarden org automation principal), never committed, consistent with existing Gateway/DB secret handling (RFC-001 §8). The `as_token`/`hs_token` live **only** in backend + appservice memory; agents get only their own scoped, re-mintable token (§4.2). Enroll is authenticated (RFC-001 B5) so a rogue local process can't enroll a rogue agent. Backend access is scoped: an agent's credential can read only its collection/policy, never the fleet-admin scope holding the crown jewels.
**8.4 — Honest note: IP-only standalone with self-signed client TLS is a weaker-trust local mode.** In Mode C, client TLS may be self-signed (or a local private CA). This means: no third party vouches for the endpoint; clients must be told to trust the self-signed cert (TOFU or manual root import); there is no external validation of who's on the other end. This is **acceptable and supported for local/airgapped/homelab** use where the network is already trusted, but it is **explicitly a weaker trust posture** than a real CA. The installer states this plainly at install (§6.8 Mode C). It is one more reason Mode C cannot federate: we will not extend this weaker-trust local posture across sites (NG5).
**8.5 — Homeserver hardening** (inherited from RFC-001 §8, config-rendered here): `enable_registration: false` always (agents come only via the appservice), rate-limiting on, admin API bound to localhost/behind auth, media repo locked/disabled if unused, TLS terminated at our controlled proxy. These are **rendered from config** (§5.4) so a stranger gets them by default, not by remembering to set them. **[VERIFY]** current recommended Synapse hardening flags at implementation.
---
## 9. How RFC-002 integrates with RFC-001's P1–P5
RFC-002 is the **substrate**. Each RFC-001 phase consumes a subset of it. Critically, **P1 does not need the hard parts** — presence ships on a single clean-domain instance with no federation, no IP-only, and no secret-rotation story resolved.
| RFC-001 phase | RFC-002 pieces it NEEDS | RFC-002 pieces it does NOT need yet |
|---|---|---|
| **P1 — Presence (first shippable slice)** | **Mode A or B, single-instance, clean domain** (§2.3). One ACME cert (LE or step-ca) via the single integration (§3). Config system minimal: `server_name` immutable + presence thresholds tunable (§5.3). Installer path 6.1–6.3, 6.4 (cert), 6.7 (immutable gate), 6.8 (validate single-instance). A **minimal** SecretBackend just holding the appservice token. | **No federation** (§2.4 gate irrelevant — single site). **No IP-only** needed for P1 (P1 wants a clean domain so Element/humans connect cleanly). **No agent-credential rotation** maturity. **No peer/whitelist** config. Vaultwarden machine-account [VERIFY] does **not** block P1. |
| **P2 — Appservice + auto-enroll** | Full `SecretBackend` **`enrollAgent`/`revokeAgent`** (§4.1), `hs_token`/`as_token` custody (§4.2), enroll-bootstrap secret. Config: room taxonomy, per-agent token classification. Chosen backend (Vault or Vaultwarden) real. | Federation, cross-site, IP-only. |
| **P3 — MACP v1 spec** | Config keys for MACP versioning/thresholds as runtime-tunable (§5.3); nothing new topology-wise. | Federation, secret rotation-in-anger. |
| **P4 — Federation** | **The whole federation half of RFC-002:** Mode A delegation (§3.3), the hard DNS+cert gate (§2.4), `domain_whitelist`+peers config, cert monitoring/silent-defederation alarm (§3.4), per-CA trust distribution for cross-site (§8.1). This is where standalone→federated (§7 Case 1) and Jason's `mosaic.woltje.com`↔`mosaic.uscllc.com` shape land. | IP-only (federation excludes it by construction). |
| **P5 — Hardening + signed-authorship + Hermes retired** | Secret **rotation runbooks** executed in anger (§4, RFC-001 E2), pubkey-record custody for Ed25519 (§4.2), cert-rotation runbook (§3.4), full homeserver hardening validated (§8.5), backend auth review (§8.3). | — |
**One-line integration statement:** P1 rides on the *smallest* slice of RFC-002 (single clean-domain instance + one cert + minimal config + minimal secret storage); the federation/IP-only/backend-maturity complexity is deferred to exactly the phases that need it (mostly P4/P5). RFC-002 therefore does not gate P1.
---
## 10. Open questions
Deliberately few — most topology/cert/secret decisions are resolved by Jason's rulings and baked in above.
1. **[VERIFY] IP-only `server_name` acceptance.** Does the deployed Synapse version accept an `ip:port` `server_name` and mint usable MXIDs for standalone (§2.3)? If not, Mode C uses a fabricated local domain (`mosaic.local` via local resolver) — confirm which, since it affects the re-home wording for Mode C→A/B (§7).
2. **[VERIFY] Vaultwarden machine-account maturity.** Confirm the current Vaultwarden version's Secrets Manager / machine-account coverage (§4.4). Determines whether the Vaultwarden adapter's `enrollAgent` uses native machine accounts or the personal-vault-per-agent + collection model. Does **not** block (interface absorbs either), but sets P2 expectations.
3. **[VERIFY] step-ca root distribution for cross-site federation.** Confirm Synapse's supported mechanism for trusting a **custom federation CA root** (§8.1) so two private-CA sites can federate. If Synapse won't easily trust a private federation CA, private-domain federation may in practice require public certs (LE) on the federation SANs even when internal traffic uses step-ca.
4. **Default secret backend for the published installer.** Given the open-source ethos (G6), should the installer *default-suggest* Vaultwarden (free, self-hostable) while clearly offering Vault, or present them neutrally? Recommendation: suggest Vaultwarden as the zero-cost path with the maturity caveat surfaced, Vault as the "I need native short-TTL machine creds now" path. Jason to confirm the framing.
5. **Single multi-SAN cert vs two certs in Mode A** (§3.3) — operational preference for identity-domain + host coverage. Minor; validate during P4.
6. **Reconfigure-time Synapse restart policy.** Which rendered-config changes (§5.4) require a Synapse restart vs hot-reload on the deployed version, so the config service knows when a tunable change needs a bounce. **[VERIFY]** at implementation.
---
## Appendix A — Real mechanics this RFC leans on (quick reference)
- **`server_name`** — Synapse identity domain; the `:suffix` of every MXID/alias; install-immutable; changing it = re-home (no in-place rename). Distinct from where Synapse *listens*.
- **Delegation** — `https://<server_name>/.well-known/matrix/server` → `{"m.server":"host:port"}` and/or `_matrix._tcp.<server_name>` **SRV**; how identity-domain ≠ host is reconciled (Mode A). **[VERIFY]** precedence on deployed Synapse.
- **`.well-known/matrix/client`** — C-S discovery so agents/Element find the homeserver host from the identity domain.
- **`federation_domain_whitelist`** — Synapse allowlist; only listed domains federate; our hard no-public-federation boundary.
- **ACME** — single provisioning protocol for both CAs; operator supplies **directory URL** (step-ca vs Let's Encrypt) + **challenge type**.
- **HTTP-01 / DNS-01 / TLS-ALPN-01** — challenge types; **DNS-01 is the one for private/split-horizon/wildcard**; HTTP-01 for public single host with :80; TLS-ALPN-01 for :443-only public.
- **step-ca ACME provisioner** — Smallstep's self-hosted CA exposing an ACME directory; enables private/internal-domain certs and total control; may require **EAB**; root must be distributed to validators.
- **Let's Encrypt** — public ACME CA; universally-trusted chain; 90-day certs; staging endpoint for testing.
- **Bitwarden/Vaultwarden org + collection + machine/service-account** — org holds collections; collections scope access; machine/service accounts (Bitwarden **Secrets Manager**) are the clean automation primitive but **Vaultwarden coverage is partial/evolving [VERIFY]**; personal-vault-per-agent + org-collection works today.
- **Vault KV v2 / AppRole / lease-TTL / revoke** — the capable backend; native versioning=rotate, TTL+revoke=transient per-agent creds.
- **Silent defederation** — a lapsed/renewal-failed federation cert causes peers to stop trusting S2S with no local error; must be monitored + alarmed (§3.4).
*All Matrix/ACME/secret-backend mechanics above are cited from architecture knowledge and MUST be re-verified against the actually deployed versions during implementation — every **[VERIFY]** is a checkpoint, not an assumption. Every illustrative domain (`mosaic.woltje.com`, `mosaic.uscllc.com`, `matrix.woltje.com`) is an operator-supplied example, never a product default or literal.*