stack/docs/rfcs/RFC-002-INSTALL-CONFIG-TOPOLOGY.md

RFC-002 — Install, Configuration & Topology for the Mosaic Matrix/MACP Comms System

• Status: DRAFT — for Team Lead → Orchestrator staffing
• Author: MS-LEAD (reviewer identity ms-lead-reviewer)
• Sponsor / veto: Jason (human lead)
• Date: 2026-07-24
• Program: Mosaic Stack comms-evolution
• Companion to: RFC-001 — MACP: A Mosaic-Native, Matrix-Native Comms Layer. RFC-001 is the architecture (self-hosted Synapse + Mosaic appservice backbone + packages/comms SDK + MACP standard + per-site federation). RFC-002 is the config substrate the whole thing installs and runs on.
• Audience: Team Leads, the Mosaic orchestrator, infra, harness maintainers, and — critically — strangers who install this open-source product on hardware we will never see.

This is a design document. No code ships from this RFC. It is written to be decomposed into missions with per-phase acceptance criteria, and it slots under RFC-001's P1→P5. Every uncertain or must-live-validate claim is flagged [VERIFY].

The one framing that governs every decision below: this is an open-source product. Someone we have never met will git clone it and run it on their own domains, their own DNS, their own certs, their own hardware. NOTHING may hardcode our fleet's topology. There is no woltje.com in the code, no assumption that DNS exists, no assumption that a second site exists. Every topology fact is user-supplied config. Where this doc uses mosaic.woltje.com / mosaic.uscllc.com, those are illustrative operator values (Jason's real installs), never defaults and never literals in the product.

---

0. TL;DR

The comms system installs against a user-supplied topology, never a baked-in one. At install the operator declares exactly one of three topology modes: (A) split-domain (identity server_name ≠ homeserver host, wired via Matrix delegation — this is Jason's mosaic.woltje.com identity + matrix.woltje.com host setup), (B) single-domain (server_name == homeserver host), or (C) IP-only standalone (no DNS, no federation, fully supported for local/airgapped). The PRIMARY/home instance is ALWAYS configured; federation is OPTIONAL. A single standalone instance MUST work with zero federation.

Federation is a hard-gated capability: it REQUIRES DNS + valid certificates. IP-only federation is not possible and is not supported. IP-only means standalone-only, forever, until the operator acquires DNS + certs.

Certificates are one ACME integration with a user-chosen directory URL: either step-ca (self-hosted private ACME CA, for total control and private/internal domains public CAs can't issue for) or Let's Encrypt (public ACME, ease-of-use). The operator also picks a challenge type (HTTP-01 / DNS-01 / TLS-ALPN-01); DNS-01 is the answer for private/split-horizon domains.

Secrets go through a pluggable SecretBackend interface — no forced paid dependency. Ships with a Vault implementation and a Vaultwarden implementation; the operator picks at install. The Vaultwarden model (org + orchestrator enrolled as authority + per-agent scoped access) is designed-for, with an honest [VERIFY] on how far Vaultwarden's machine-account coverage has matured.

Config is DB-backed with sane defaults and install-time overrides. Precedence: install-time → DB override → default. Config is split into install-time-immutable (e.g. server_name, which is baked into every MXID and cannot change without re-homing every identity) and runtime-tunable (e.g. dark-threshold).

RFC-002 is the substrate; RFC-001's P1 (presence) needs only Mode A/B single-instance clean-domain and does NOT require federation, IP-only, or the secret-backend rotation story resolved.

---

1. Goals / Non-Goals

1.1 Goals

• G1 — Installable by a stranger. A person with no relationship to our fleet can install, configure, and run the comms system from published artifacts and a guided installer, on their own hardware and domains, with no edits to product code.
• G2 — Zero hardcoded topology. Every topology fact — server_name, homeserver host/IP, delegation method, federation peers, cert mode, secret backend — is user-supplied config, validated at install, stored in the product DB. No fleet-specific literal ships in the product.
• G3 — Standalone MUST work. The PRIMARY instance is always fully functional with zero federation, including with no DNS at all (Mode C, IP-only). Presence, rooms, MACP, HIL-via-Element all work single-instance.
• G4 — Federation is optional but honestly gated. Federation is opt-in and, when opted into, requires DNS + valid certificates as a hard precondition. The installer must refuse to claim federation is working when the DNS/cert preconditions aren't met.
• G5 — One ACME integration, two CA choices. Build a single ACME cert-provisioning path; the operator selects step-ca or Let's Encrypt by supplying an ACME directory URL plus a challenge type. No second, bespoke cert path.
• G6 — No forced paid dependency for secrets. A pluggable SecretBackend with at least Vault and Vaultwarden implementations, chosen at install. Open-source ethos: the free/self-hostable path must be first-class.
• G7 — Defaults that just work, overrides where they matter. DB-backed config with sane defaults so most operators change little; install-time overrides for the topology-critical values; a clear immutable-vs-tunable boundary so operators can't foot-gun server_name.
• G8 — A clean upgrade path. An operator who starts standalone can later turn on federation with a documented, honest procedure (including the real cost if they started IP-only and must now acquire a stable server_name).

1.2 Non-Goals

• NG1 — Not hosting a managed service. This RFC is about self-install. We are not building multi-tenant SaaS provisioning; each operator runs their own instance(s).
• NG2 — Not a new cert stack. We do not write our own CA, our own ACME client protocol, or a non-ACME cert path. We integrate ACME and let the operator point it at step-ca or Let's Encrypt. (We may bundle/recommend step-ca as the self-hosted CA, but via its standard ACME provisioner, not a fork.)
• NG3 — Not a new secret manager. We define an interface and ship adapters. We do not build a secret store; we do not force one.
• NG4 — Not public-network Matrix federation. Consistent with RFC-001 NG5: federation is Mosaic-site-to-Mosaic-site over infrastructure the operator controls, allowlisted. No matrix.org federation.
• NG5 — Not making IP-only federate. We will not ship a hack (self-signed S2S trust bundles, /etc/hosts federation) that pretends IP-only can federate. IP-only is standalone. This is a deliberate, honest boundary (§2.4, §7).
• NG6 — Not re-homing identities silently. We will not offer a "just change your server_name" button that quietly orphans every MXID. Any path that changes server_name is a flagged, gated, documented identity re-home (§5.3, §7).

---

2. The topology model

2.1 The core split: server_name vs homeserver host

Matrix has exactly the split Jason described, natively:

• server_name — the Synapse config value that is the server's identity domain. It is the part after the colon in every MXID (@mosaic_agent:mosaic.woltje.com) and every room alias (#mission-KBN-101:mosaic.woltje.com). It is baked into every identity the moment that identity is minted. Changing it re-homes everything. This is server_name in Synapse's homeserver.yaml.
• Homeserver host — the actual network location (hostname:port or IP:port) where the Synapse process answers federation and (optionally proxied) client traffic. It can differ from server_name. Matrix reconciles the difference through delegation: https://<server_name>/.well-known/matrix/server returning {"m.server": "matrix.woltje.com:443"}, and/or a _matrix._tcp.<server_name> SRV record. [VERIFY] .well-known vs SRV precedence on the deployed Synapse version (RFC-001 §6 flags the same).

So Jason's "mosaic._ app-domain + matrix._ homeserver-domain" split maps precisely: server_name = mosaic.woltje.com (identity, in MXIDs), homeserver runs at matrix.woltje.com (discovered via delegation). That is Mode A.

2.2 The topology config schema

One canonical config object, stored in the product DB (§5), populated at install (§6). Illustrative shape (field names decomposition-ready, not frozen):

{
  "topology": {
    "mode": "split-domain | single-domain | ip-only-standalone", // A | B | C — install-time-immutable

    "identity": {
      "server_name": "mosaic.woltje.com", // INSTALL-TIME-IMMUTABLE. In MXIDs. Never change without re-home (§5.3).
      "server_name_kind": "domain | ip", // "ip" only legal in Mode C
    },

    "homeserver": {
      "host": "matrix.woltje.com", // where Synapse actually listens (Mode A: differs from server_name)
      "port": 8448, // federation port (default 8448) or 443 if proxied — [VERIFY] per deploy
      "client_bind": "https://matrix.woltje.com", // C-S API public URL (proxied)
      "bind_ip": null, // Mode C: e.g. "192.168.1.50" ; Modes A/B: null (DNS-resolved)
    },

    "delegation": {
      "method": "well-known | srv | none", // Mode A: well-known or srv ; Mode B/C: none
      "well_known_server": { "m.server": "matrix.woltje.com:443" }, // if method=well-known
      "srv_record": "_matrix._tcp.mosaic.woltje.com. 3600 IN SRV 10 0 443 matrix.woltje.com.", // if method=srv (documented, operator provisions)
    },

    "federation": {
      "enabled": true, // OPTIONAL. Mode C forces false.
      "domain_whitelist": [
        // Synapse federation_domain_whitelist — allowlist ONLY
        "mosaic.woltje.com",
        "mosaic.uscllc.com",
      ],
      "peers": [
        // operator-declared peer sites (for room/fleet wiring)
        { "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true },
      ],
    },

    "tls": {
      "acme": {
        "directory_url": "https://acme.mosaic.woltje.com/acme/acme/directory", // step-ca OR https://acme-v02.api.letsencrypt.org/directory
        "ca_kind": "step-ca | letsencrypt", // informational label; the directory_url is the real switch
        "challenge": "dns-01 | http-01 | tls-alpn-01",
        "account_email": "ops@woltje.com", // ACME account contact
        "eab": { "kid": null, "hmac_key_ref": null }, // External Account Binding if the CA requires it (some step-ca provisioners) — secret via SecretBackend
      },
      "client_tls_mode": "acme | self-signed", // Mode C may use self-signed for local C-S TLS (weaker trust, §8)
    },

    "secrets": {
      "backend": "vault | vaultwarden", // pluggable, install-time choice (§4)
      "connection": {
        "address": "https://vault.woltje.com:8200", // or Vaultwarden/Bitwarden server URL
        "auth_ref": "…", // how the appservice authenticates to the backend (bootstrap, §4/§8)
        "namespace_or_org": "mosaic-fleet", // Vault namespace / mount, OR Vaultwarden org id
      },
    },
  },
}

2.3 The three modes, concretely

Exactly three supported modes (Jason's ruling — no others):

Mode A — split-domain (identity ≠ host, delegated). Jason's PRIMARY. Federation-capable. This is the recommended production shape because it lets identity live on a clean app-domain while the homeserver runs on a separate operational host.

// Mode A — mosaic.woltje.com identity, matrix.woltje.com host, federated with a second site
{
  "topology": {
    "mode": "split-domain",
    "identity": { "server_name": "mosaic.woltje.com", "server_name_kind": "domain" },
    "homeserver": {
      "host": "matrix.woltje.com",
      "port": 443,
      "client_bind": "https://matrix.woltje.com",
      "bind_ip": null,
    },
    "delegation": {
      "method": "well-known",
      "well_known_server": { "m.server": "matrix.woltje.com:443" },
    },
    "federation": {
      "enabled": true,
      "domain_whitelist": ["mosaic.woltje.com", "mosaic.uscllc.com"],
      "peers": [{ "server_name": "mosaic.uscllc.com", "role": "secondary", "fleet_room": true }],
    },
    "tls": {
      "acme": {
        "directory_url": "https://acme-v02.api.letsencrypt.org/directory", // public LE, or a step-ca directory
        "ca_kind": "letsencrypt",
        "challenge": "dns-01",
        "account_email": "ops@woltje.com",
      },
      "client_tls_mode": "acme",
    },
    "secrets": {
      "backend": "vaultwarden",
      "connection": { "address": "https://vw.woltje.com", "namespace_or_org": "mosaic-fleet" },
    },
  },
}

MXIDs on this instance: @mosaic_coordinator-1:mosaic.woltje.com. A human/agent's homeserver is discovered by resolving .well-known/matrix/server on mosaic.woltje.com → matrix.woltje.com:443.

Mode B — single-domain (identity == host). Simpler; the server_name is the host. No delegation needed. Federation-capable (still needs DNS + cert on that one domain).

// Mode B — one domain does everything
{
  "topology": {
    "mode": "single-domain",
    "identity": { "server_name": "matrix.example.org", "server_name_kind": "domain" },
    "homeserver": {
      "host": "matrix.example.org",
      "port": 8448,
      "client_bind": "https://matrix.example.org",
      "bind_ip": null,
    },
    "delegation": { "method": "none" },
    "federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // optional — off here
    "tls": {
      "acme": {
        "directory_url": "https://acme-v02.api.letsencrypt.org/directory",
        "ca_kind": "letsencrypt",
        "challenge": "http-01",
        "account_email": "admin@example.org",
      },
      "client_tls_mode": "acme",
    },
    "secrets": {
      "backend": "vault",
      "connection": { "address": "https://vault.example.org:8200", "namespace_or_org": "mosaic" },
    },
  },
}

MXIDs: @mosaic_coordinator-1:matrix.example.org. Here server_name == host, so @:matrix.example.org is both the identity domain and where Synapse actually answers.

Mode C — IP-only standalone (no DNS, no federation). Fully supported for local/airgapped/homelab-without-DNS. server_name is an IP:port literal. Cannot federate — ever — in this mode (federation requires a real domain + valid CA cert; §2.4). Client TLS may be self-signed (weaker trust, §8).

// Mode C — airgapped / local, no DNS, standalone only
{
  "topology": {
    "mode": "ip-only-standalone",
    "identity": { "server_name": "192.168.1.50:8448", "server_name_kind": "ip" },
    "homeserver": {
      "host": "192.168.1.50",
      "port": 8448,
      "client_bind": "https://192.168.1.50:8448",
      "bind_ip": "192.168.1.50",
    },
    "delegation": { "method": "none" },
    "federation": { "enabled": false, "domain_whitelist": [], "peers": [] }, // FORCED false in Mode C
    "tls": { "acme": null, "client_tls_mode": "self-signed" }, // may use a private step-ca or self-signed for C-S TLS
    "secrets": {
      "backend": "vaultwarden",
      "connection": { "address": "http://192.168.1.51:8080", "namespace_or_org": "mosaic-local" },
    },
  },
}

MXIDs: @mosaic_coordinator-1:192.168.1.50:8448. Warning surfaced at install: this server_name is an IP literal; if the operator ever wants federation they must move to a domain, which is an identity re-home (§5.3, §7).

[VERIFY] Synapse accepts an ip:port server_name and mints usable MXIDs against it for local/standalone use. This is believed workable for non-federated operation but must be validated against the deployed Synapse version; some Synapse versions/tools assume a DNS-resolvable server_name. If an IP literal is rejected, Mode C falls back to a fabricated local domain (e.g. mosaic.local) resolved via /etc/hosts or a local resolver — still standalone-only, same re-home caveat.

2.4 The hard federation gate (Jason's HARD STOP)

Federation REQUIRES DNS + valid certificates. This is a hard stop, enforced by the installer and by the config validator, not a suggestion.

Precondition	Why	Enforced where
server_name resolves in public/peer DNS (or delegated target does)	S2S discovery uses .well-known/SRV over DNS; peers must resolve you	installer reachability check (§6); config validator rejects federation.enabled=true with server_name_kind=ip
Valid TLS cert on the federation endpoint, chained to a CA the peer trusts	S2S is TLS; a peer validates your cert. Self-signed/untrusted ⇒ peer refuses ⇒ you are defederated	installer cert probe (§6); cert monitor (§3.4)
Federation domain_whitelist non-empty and mutually consistent with peers	allowlist-only federation (RFC-001 NG5/§6)	config validator

IP-only ⇒ federation is impossible. There is no valid public/peer CA cert for a bare IP in our trust model (and we will not ship a self-signed S2S trust hack — NG5). Therefore Mode C is standalone-only by construction, and the config validator makes mode=ip-only-standalone ∧ federation.enabled=true an illegal state that cannot be persisted.

This is the honest, load-bearing boundary of the whole topology model:

        DNS + valid cert?
        ┌─────────────┴─────────────┐
       YES                          NO
        │                            │
 Mode A or B                     Mode C (IP-only)
 federation OPTIONAL          STANDALONE ONLY
 (opt-in, allowlisted)        (federation impossible)

---

3. Certificate provisioning — one ACME integration

3.1 Single integration, directory-URL as the switch

We build exactly one cert-provisioning integration: an ACME client integration. Both supported CAs are ACME CAs. The operator does not choose between two code paths; they choose an ACME directory URL and a challenge type. That is the entire surface.

CA choice	What it is	ACME directory URL (illustrative)	Why an operator picks it
step-ca (Smallstep)	self-hosted private ACME CA	https://acme.<internal-domain>/acme/<provisioner>/directory	Total control; issues for private/internal/split-horizon domains a public CA can't (e.g. mosaic.internal, RFC-1918 split-horizon); airgap-friendly; you own the root
Let's Encrypt	public ACME CA	https://acme-v02.api.letsencrypt.org/directory (staging: .../acme-staging-v02...)	Ease of use; universally trusted chain (ISRG Root X1); zero CA to operate; ideal for public domains

Because both speak ACME, the same client (account key, order, authorization, challenge, finalize, cert-fetch, renew) drives either. The ca_kind label in config is informational for UX; the directory_url is the real determinant. [VERIFY] whether the chosen ACME library requires per-CA quirks (LE rate limits, staging switch; step-ca External Account Binding on some provisioners — if EAB is required the kid/hmac_key come from the SecretBackend, §4).

3.2 Challenge-type matrix (which challenge for which topology)

The operator picks one challenge type per the domains they're covering. This is the crux for public vs private/split-horizon:

Challenge	How it proves control	Best for	Cannot / caveat
HTTP-01	CA GETs http://<domain>/.well-known/acme-challenge/<token> on port 80	Public, single hostname, port 80 reachable from CA (Mode B, or Mode A's homeserver host)	Needs inbound :80 from the CA; cannot do wildcards; useless for private domains a public CA can't reach
DNS-01	CA checks a _acme-challenge.<domain> TXT record you publish	Private / split-horizon / internal domains, wildcards, and any domain where inbound HTTP from the CA is impossible	Requires programmatic DNS API access to publish TXT (or manual for step-ca where you own the resolver). This is the answer for step-ca on private domains and for Mode A when the homeserver host isn't publicly HTTP-reachable
TLS-ALPN-01	CA connects TLS on :443 with ALPN acme-tls/1	Public host where :443 is free but :80 is blocked	Needs the ACME client to own the :443 TLS handshake briefly; awkward behind some reverse proxies — [VERIFY] against our proxy (RFC-001 terminates TLS at a reverse proxy)

Guidance baked into the installer:

• Let's Encrypt + public domain, port 80 open → HTTP-01 (simplest). Wildcard or no inbound :80 → DNS-01.
• step-ca + private/internal/split-horizon domain → DNS-01 (the private CA can validate against a resolver you control; public HTTP reachability is irrelevant). This is the combination that lets a private homelab domain get real certs.
• :443-only public host → TLS-ALPN-01.

3.3 Delegation setup for split-domain (Mode A)

For Mode A, the cert and the delegation must agree. Concretely, at install for server_name=mosaic.woltje.com, host matrix.woltje.com:

1. Cert(s): obtain a valid cert for matrix.woltje.com (the federation/host endpoint — this is where the TLS handshake actually lands). If serving .well-known over HTTPS on mosaic.woltje.com, that origin also needs a valid cert for mosaic.woltje.com. So Mode A typically provisions two SANs/certs: identity-domain (mosaic.woltje.com, serves .well-known) and host (matrix.woltje.com, serves S2S+C-S). [VERIFY] whether a single multi-SAN cert is preferable operationally.
2. Delegation record, one of:
   • .well-known: serve https://mosaic.woltje.com/.well-known/matrix/server → {"m.server":"matrix.woltje.com:443"} (and .well-known/matrix/client for C-S discovery so agents/Element find the host).
   • SRV: _matrix._tcp.mosaic.woltje.com. IN SRV 10 0 443 matrix.woltje.com. The installer documents and validates the record but the operator provisions it in their DNS (we don't run their DNS). [VERIFY] .well-known vs SRV precedence on the deployed Synapse.
3. Validate: installer fetches the operator's own .well-known/SRV and confirms it points at the configured host, and that the host presents a valid cert (§6). Only then does it declare Mode A "federation-ready."

3.4 Renewal & monitoring — a lapsed federation cert silently defederates

This is the operational trap and it must alarm. ACME certs are short-lived (LE = 90 days; step-ca often shorter by policy). A federation cert that lapses does not throw a loud error — peers simply stop trusting the S2S handshake and the site silently drops out of federation. From inside, everything looks fine; from peers, the site went dark. That is exactly the "homelab went dark and took comms with it" trauma (RFC-001 §5), but caused by a cert, not a host.

Requirements:

• Auto-renew on the standard ACME schedule (renew at ~⅓ lifetime remaining; LE guidance ~30 days before expiry). The ACME integration owns this loop.
• Expiry monitoring as a first-class alarm. Emit cert-days-remaining into OTEL/Jaeger metrics (consistent with RFC-001 §8's "monitor for cert expiry — a cert lapse silently defederates"). Alarm thresholds (e.g. warn <14d, critical <3d) are runtime-tunable config (§5).
• Federation-health probe: periodically resolve our own delegation and validate our own cert as a peer would (external vantage where possible), so a broken renewal is caught as "we would fail a peer's validation" before a peer notices.
• Escalation tie-in: a critical cert-expiry or federation-health failure raises a mosaic.escalation (RFC-001 §4.2/§5) into the HIL room. A cert lapse is a fleet-visibility incident, not a silent config drift.

---

4. Secret backend interface

4.1 The SecretBackend contract

A single pluggable interface. The appservice and orchestrator depend on the interface, never on Vault or Vaultwarden directly. Chosen at install; swappable without touching callers. Illustrative contract (decomposition-ready, not frozen):

interface SecretBackend {
  // --- static secret CRUD (appservice tokens, ACME EAB, DB creds) ---
  get(ref: SecretRef): Promise<SecretValue>;
  put(ref: SecretRef, value: SecretValue, opts?: { immutable?: boolean }): Promise<void>;
  rotate(
    ref: SecretRef,
    next: SecretValue,
  ): Promise<{ previous: SecretVersion; current: SecretVersion }>;
  list(prefix: SecretRef): Promise<SecretRef[]>;
  delete(ref: SecretRef): Promise<void>;

  // --- agent-credential lifecycle (the fleet-identity part) ---
  enrollAgent(input: {
    agentSlug: string;
    scope: CredentialScope; // which rooms/secrets this agent may read
    ttl?: Duration; // ephemeral-by-default per RFC-001 §8
  }): Promise<AgentCredentialHandle>; // wraps the per-agent access_token + optional pubkey record

  revokeAgent(agentSlug: string): Promise<void>; // must be authoritative & immediate

  // --- health / bootstrap ---
  health(): Promise<BackendHealth>;
  authenticateSelf(bootstrap: BootstrapAuth): Promise<void>; // how the appservice/orchestrator logs into the backend
}

Design intent: get/put/rotate cover the static crown-jewel secrets (appservice hs_token/as_token, ACME account/EAB keys, DB DSN). enrollAgent/revokeAgent cover the fleet-identity lifecycle — this is where RFC-001's "mint per-agent token at enroll, discard on teardown" (RFC-001 §4.1, §8) actually lands.

4.2 How appservice / agent tokens map onto it

RFC-001 defines three tiers of Matrix secret. They map cleanly:

RFC-001 secret	Sensitivity	SecretBackend treatment
as_token (AS→HS, acts as any namespaced user)	crown jewel	put(immutable-ish) + rotate; only the appservice may get it; stored under a fleet-admin scope; never handed to an agent
hs_token (HS→AS callback auth)	crown jewel	same as as_token; both live only in appservice scope (RFC-001 §8)
enroll bootstrap secret / orchestrator-signed nonce	high	get by orchestrator + appservice only; used to authenticate POST /enroll so a rogue process can't enroll a rogue agent (RFC-001 §8, B5)
per-agent access_token	transient runtime	minted via enrollAgent, scoped to that agent, short-TTL / re-mintable, discarded on teardown; not durably persisted (RFC-001 §8 says per-agent tokens are transient) — the backend may hold a short-lived handle or a personal-vault entry for the agent's own lifetime
Ed25519 signed-authorship keypair	high (private key)	private key generated agent-side, only pubkey leaves the agent (RFC-001 §4.4/§8); the SecretBackend stores the pubkey record for audit; per-spin keys need no at-rest custody

The key blast-radius property (RFC-001 §8) is preserved: agents receive only their own credential via enrollAgent; the as_token never leaves appservice scope.

4.3 Vault implementation

Vault maps naturally:

• Static secrets → KV v2 at a mount/namespace (mosaic-fleet/), with versioning giving rotate semantics for free.
• enrollAgent → issue a scoped, TTL'd token or use AppRole / a scoped policy per agent; Vault's native TTL + revocation is exactly the transient per-agent model. revokeAgent → Vault token/lease revoke (authoritative, immediate).
• authenticateSelf → the appservice authenticates to Vault via AppRole (role_id from config, secret_id injected at deploy) or a platform auth method; consistent with how Gateway/DB secrets are handled today (RFC-001 §8). [VERIFY] align with whatever KBN-101 lands for Mosaic secret management (CLAUDE.md flags secrets work in flight).
• Trade-off (honesty): Vault is the most capable backend but is not the free-and-simple default for a hobbyist stranger; hence it must not be forced (G6).

4.4 Vaultwarden implementation + the org/enroll/revoke agent-account model

Vaultwarden (self-hostable Bitwarden-compatible server) is the open-source-ethos default candidate — free, self-hostable, familiar. Jason's model, mapped onto Bitwarden/Vaultwarden's org primitives:

1. Operator creates one or more Bitwarden orgs at install (e.g. mosaic-fleet).
2. The orchestrator is enrolled into the org and granted authority to enroll/revoke agent sessions — it is the org's automation principal (admin/manager over an agents collection).
3. Agents get scoped credential access: each agent gets access to a collection (or a personal vault provisioned for its spin) holding exactly the secrets its scope allows. enrollAgent = grant the agent principal access to its collection + provision its per-agent Matrix token entry; revokeAgent = remove the agent principal / revoke its access, immediately.
4. User + agents share scoped access: the human operator and the agents both hold credentials in the same org, scoped by collection — humans and agents on one secret surface, mirroring RFC-001's "humans and agents on one comms surface" pattern.

Mapping to the interface:

Interface op	Vaultwarden mechanism
get/put/rotate (static)	items in an org collection; rotate = new item version / replace + old-version audit
enrollAgent	create/attach agent principal to its collection; provision per-agent token item, TTL enforced by our teardown (Vaultwarden itself is not TTL-native — see [VERIFY])
revokeAgent	revoke the agent principal's org membership / collection access
authenticateSelf	orchestrator authenticates as the org automation principal

HONEST MATURITY FLAG — [VERIFY]: Bitwarden's clean machine-account / service-account primitive lives in Bitwarden Secrets Manager, and Vaultwarden's coverage of Secrets Manager / machine accounts is partial and evolving. What is known to work today on Vaultwarden: orgs, collections, per-user (incl. a per-agent "user") vaults, and collection-scoped sharing. What may not be fully there: the polished machine-account API, native short-TTL service credentials, and fine-grained programmatic access-token issuance equivalent to hosted Bitwarden Secrets Manager. [VERIFY] current Vaultwarden version's Secrets Manager / machine-account support before P-level commitment.

Why this is not a blocker: the interface is designed so either backend is viable. If Vaultwarden's machine-account API isn't ready, the Vaultwarden adapter implements enrollAgent via the personal-vault-per-agent + org-collection model that works today (create an agent principal, share the scoped collection, we enforce TTL via orchestrator teardown rather than backend-native TTL). If an operator needs backend-native short-TTL machine credentials now, they choose the Vault adapter. We are not blocked on Vaultwarden maturing, because the SecretBackend abstraction lets the polished-machine-account behavior land later without changing any caller.

---

5. Config system

5.1 Storage & precedence

Config is DB-backed (Postgres, per the stack — CLAUDE.md/RFC-001), with sane defaults compiled into the product and install-time overrides. Precedence, highest wins:

install-time value   >   DB override (runtime)   >   compiled default

• Compiled default — ships in the product; what a stranger gets with zero config for every non-topology-critical key.
• Install-time value — captured by the installer (§6), written to DB, and for install-time-immutable keys, locked (marked non-overridable).
• DB override — runtime tuning via admin surface, allowed only for keys classified runtime-tunable.

Nuance: "install-time > DB override" applies to immutable keys — the install-time value is frozen and a DB override of it is rejected. For tunable keys, the DB override is the live value and the install-time value is just the initial seed. The classification (§5.3) is what makes the precedence unambiguous per key.

5.2 DB schema shape

Illustrative (Drizzle/Postgres, per stack conventions):

-- one row per config key
CREATE TABLE comms_config (
  key            text PRIMARY KEY,            -- e.g. 'topology.identity.server_name'
  value          jsonb NOT NULL,              -- current effective value
  source         text NOT NULL,               -- 'install' | 'db-override' | 'default'
  mutability     text NOT NULL,               -- 'install-immutable' | 'runtime-tunable'
  set_by         text,                        -- operator/agent/system that set it
  set_at         timestamptz NOT NULL DEFAULT now(),
  CONSTRAINT immutable_not_overridable
    CHECK (NOT (mutability = 'install-immutable' AND source = 'db-override'))
);

-- append-only audit of every change (esp. attempted immutable changes)
CREATE TABLE comms_config_audit (
  id         bigserial PRIMARY KEY,
  key        text NOT NULL,
  old_value  jsonb,
  new_value  jsonb,
  actor      text NOT NULL,
  action     text NOT NULL,     -- 'set' | 'override' | 'rejected-immutable'
  at         timestamptz NOT NULL DEFAULT now()
);

The DB CHECK is a belt-and-braces backstop; the application-layer config service enforces mutability and records rejected immutable-change attempts in the audit table. Secrets are referenced here (a SecretRef), never stored inline — actual secret values live in the SecretBackend (§4).

5.3 Install-time-immutable vs runtime-tunable — the key table

The single most important classification: what can never change after install vs what an operator tunes anytime. Getting server_name on the wrong side of this line is a foot-gun that orphans every identity.

Config key	Mutability	Rationale / cost of change
topology.mode (A/B/C)	install-immutable	changing mode changes identity/federation semantics; a mode change is effectively a reinstall/re-home
topology.identity.server_name	install-immutable	baked into every MXID and room alias. Changing it re-homes every identity — see re-home note below. This is THE immutable value.
topology.identity.server_name_kind	install-immutable	domain↔ip change is a re-home (Mode C→A/B)
topology.homeserver.host	install-immutable (delegation-tunable)	in Mode A you can move the host if you update delegation to match — the identity is unchanged; treat as immutable-with-migration, not free
topology.homeserver.bind_ip / port	runtime-tunable (ops)	operational network binding; no identity impact
topology.delegation.method / records	tunable-with-care	can switch well-known↔SRV as long as both still resolve to the same host; validated on change
topology.federation.enabled	runtime-tunable (gated)	can flip on only if DNS+cert preconditions pass (§2.4); flipping standalone→federated is the §7 upgrade path
topology.federation.domain_whitelist / peers	runtime-tunable	add/remove peers over time; each add re-validated
tls.acme.directory_url / ca_kind	runtime-tunable	can switch CA (e.g. LE→step-ca); triggers re-issue; monitor for trust-chain change
tls.acme.challenge	runtime-tunable	switch challenge type if DNS/HTTP reachability changes
tls.client_tls_mode	tunable-with-care	self-signed→acme is fine; acme→self-signed weakens trust (§8)
secrets.backend	install-immutable (migration-only)	switching Vault↔Vaultwarden requires a secret migration; not a live flip
secrets.connection.*	runtime-tunable	rotate backend address/auth without changing which backend
presence.heartbeat_interval_ms	runtime-tunable	RFC-001 §4.5 default ~30s; pure tuning
presence.miss_tolerance	runtime-tunable	RFC-001 §4.5 default 2
escalation.dark_threshold_min	runtime-tunable	RFC-001 §5/§11 — default 10min → fallback
escalation.hil_threshold_min	runtime-tunable	RFC-001 §5 — default +5min → HIL
cert.expiry_warn_days / expiry_critical_days	runtime-tunable	§3.4 alarm thresholds (default 14 / 3)
federation.health_probe_interval	runtime-tunable	§3.4

Identity re-home note (the cost of changing server_name): because every MXID (@mosaic_*:server_name) and room alias (#…:server_name) embeds server_name, changing it means: every agent gets a new identity, all rooms must be recreated/re-aliased, signed-authorship pubkey records re-published, and federation peers re-pointed. There is no in-place rename in Matrix. Hence server_name is install-immutable and the installer gates it behind an explicit warning (§6.7). Changing it is a migration/reinstall, honestly (§7).

5.4 How the appservice / homeserver read config

• Synapse (homeserver) reads a rendered homeserver.yaml. The config service renders Synapse's config (server_name, listeners, federation_domain_whitelist, enable_registration: false, appservice registration path, TLS/delegation) from the DB-backed config at deploy/reconfigure time. Synapse itself is not DB-config-aware; the source of truth is the product DB, and Synapse config is a generated artifact. Certain Synapse values (notably server_name) require a Synapse restart and are exactly the immutable ones — reinforcing §5.3.
• The appservice reads config live from the DB config service for runtime-tunable values (thresholds, whitelist changes, cert alarm thresholds) and from the SecretBackend for secrets. Immutable topology values are read once at boot (they can't change under it).
• packages/comms SDK receives the values it needs (homeserver client URL, presence intervals) from the appservice at enroll (RFC-001 §4.1 returns {mxid, access_token, homeserver, rooms[]}), so agents never read the config DB directly.
• Delegation artifacts (.well-known/matrix/server, .well-known/matrix/client) are likewise rendered from config and served by the reverse proxy / homeserver.

---

6. Installer UX flow

A guided installer (mosaic comms install or equivalent) that captures topology, provisions certs, wires secrets, and validates before declaring success. It must never report success it hasn't proven. Steps:

6.1 — Preflight & detection. Detect existing DNS records for a candidate domain, existing certs, an existing reachable Synapse, an existing Vault/Vaultwarden. Offer detected values as suggestions (never as silent defaults). Detect whether the host has public inbound :80/:443 (informs challenge-type guidance, §3.2).

6.2 — Primary instance (ALWAYS). The PRIMARY/home instance is always configured — there is no "skip primary." Prompt for its identity. This is non-optional and is what makes standalone work out of the box.

6.3 — Pick topology mode (A/B/C). Ask the shape:

• Do you have a domain? No → Mode C (IP-only standalone); warn federation is impossible here (§2.4) and that server_name will be an IP (re-home cost if they later want federation).
• Yes, and identity domain differs from the homeserver host? Yes → Mode A (split-domain); capture server_name + host + delegation method.
• Yes, one domain does everything? → Mode B (single-domain).

6.4 — Pick CA (ACME directory). step-ca vs Let's Encrypt → capture directory_url, account email, and challenge type with the §3.2 guidance surfaced (e.g. "private/internal domain? → DNS-01"). Capture EAB if the CA requires it (→ SecretBackend). For step-ca, offer to point at an existing step-ca or document standing one up.

6.5 — Pick secret backend. Vault vs Vaultwarden → capture connection (address, org/namespace, bootstrap auth). If Vaultwarden, walk the org/collection setup (§4.4) and surface the machine-account maturity [VERIFY] honestly so the operator chooses eyes-open.

6.6 — Federation (OPTIONAL). Only offered if Mode A/B. Ask whether to enable federation now; if yes, capture peer server_names and build the domain_whitelist. If Mode C, federation is not offered (greyed out with the explanation). Federation-off is a first-class, fully-supported outcome.

6.7 — The "what can't be changed later" warning gate. Before writing immutable config, present an explicit confirmation:

⚠  IMMUTABLE CHOICES — read before confirming
   server_name = "mosaic.woltje.com"
   This becomes part of every agent identity (e.g. @mosaic_coordinator-1:mosaic.woltje.com)
   and every room alias. It CANNOT be changed later without re-homing every identity
   (new MXIDs for all agents, recreating all rooms). There is no in-place rename in Matrix.
   topology.mode = "split-domain"   — changing modes later is a reinstall.
   secrets.backend = "vaultwarden"  — switching backends later requires a secret migration.
   Type the server_name to confirm you understand it is permanent:  ____________

The operator must retype server_name to proceed — a deliberate friction gate on the one truly permanent value.

6.8 — Provision & validate (no success claim until proven). The installer then:

1. Renders Synapse config + delegation artifacts; brings up Synapse with enable_registration: false.
2. Runs the ACME flow; obtains cert(s); verifies they're valid and installed.
3. Authenticates to the SecretBackend; stores hs_token/as_token, enroll bootstrap; runs health().
4. Reachability & cert validation appropriate to mode:
   • Mode A: fetch our own .well-known/SRV, confirm it points at host; TLS-probe host cert as a peer would; confirm C-S discovery.
   • Mode B: TLS-probe the single domain; confirm C-S + (if federation) S2S.
   • Mode C: confirm local C-S reachability over the bind IP; confirm (self-signed or private-CA) client TLS; explicitly report "standalone — federation not available."
   • If federation enabled: validate each peer resolves + presents a peer-trusted cert; confirm domain_whitelist mutual consistency. If any peer fails, federation is reported NOT-ready — the primary still succeeds standalone.
5. Only now declare success, with a per-capability report: PRIMARY: ✅ | CERT: ✅ (expires in 90d, auto-renew on) | SECRETS: ✅ (vaultwarden) | FEDERATION: ✅ 1 peer / ⚠ not-ready / ⛔ n-a (Mode C).

6.9 — Post-install. Emit the cert-expiry monitor + federation-health probe (§3.4) into OTEL; write config to DB with correct mutability flags; print the immutable-values summary again for the record.

---

7. Standalone → federated upgrade path

An operator who started standalone later wants federation. The path depends on whether they started with a domain:

Case 1 — started Mode A or B (had a domain), federation was just off. Cheap, no re-home. server_name is already a real domain and identities are already minted against it. To federate:

1. Ensure DNS resolves for peers (their server_names and yours) — likely already true.
2. Ensure a valid, peer-trusted cert on the federation endpoint (if they were running client-only self-signed, they now need a real ACME cert; if already ACME, done).
3. Set federation.enabled = true, populate domain_whitelist + peers (all runtime-tunable, §5.3).
4. Re-run the installer's federation validation (§6.8 step 4) against each peer. On green, federation is live. No identity change — existing MXIDs simply become reachable cross-site. This is the intended, low-friction upgrade.

Case 2 — started Mode C (IP-only), now wants federation. Expensive — an identity re-home, and we say so plainly. Federation requires DNS + a valid cert (§2.4), which an IP server_name can never satisfy. So the operator must:

1. Acquire a domain and DNS, and provision a valid ACME cert (LE public, or step-ca if the domain is private — but note a private domain can only federate with peers who trust that private CA root, §8).
2. Change server_name from the IP literal to the domain — this is the install-immutable value, so this is a re-home, not a config tweak:
   • Every agent identity @mosaic_*:192.168.1.50:8448 becomes @mosaic_*:newdomain — all new MXIDs.
   • Every room + alias must be recreated under the new server_name.
   • Signed-authorship pubkey records re-published under the new identities.
   • Any durable references to old MXIDs (escalation policies, fallback-coordinator targets, RFC-001 §5) must be re-pointed.
3. Effectively: treat it as a fresh install in Mode A/B with a data migration of rooms/history, not an in-place flip. Matrix has no in-place server_name rename; this cost is intrinsic to Matrix, not to our design.

Honest guidance the installer gives Mode C operators up front (§6.3): "If there is any chance you'll want to federate later, start with a domain (Mode A/B) even if you keep federation off — flipping federation on later is free, but changing an IP server_name to a domain later is a full identity re-home." This lets an informed operator avoid the expensive path by choosing Mode B-with-federation-off instead of Mode C.

---

8. Security

8.1 — Cert trust model per CA choice.

• Let's Encrypt (public): chains to a universally-trusted root (ISRG). Peers, humans' browsers, and Element trust it with no extra distribution. Best for public domains; nothing to distribute.
• step-ca (private): chains to a root you operate. Nothing trusts it by default. Therefore the step-ca root must be distributed to everyone who validates certs: peer homeservers (so cross-site S2S validates — a peer must add your root to its federation trust store, [VERIFY] Synapse's mechanism for trusting a custom federation CA), agent hosts, and any human client. This is the price of "total control" and airgap capability. For federation between two private-CA sites, both sites must trust each other's roots (or a shared root). Getting this wrong reproduces the silent-defederation failure (§3.4) — a peer that doesn't trust your root silently refuses your S2S.
• Mode C self-signed client TLS: weakest — see 8.4.

8.2 — Federation whitelist. federation_domain_whitelist is a hard allowlist (RFC-001 §6/NG5): only listed Mosaic site domains may federate; no public-network federation. The installer/config validator keeps the whitelist consistent with the declared peer list. Adding a peer is an explicit, audited config change.

8.3 — Secret-backend auth. The appservice/orchestrator authenticate to the SecretBackend via a bootstrap credential injected at deploy (Vault AppRole secret_id, or Vaultwarden org automation principal), never committed, consistent with existing Gateway/DB secret handling (RFC-001 §8). The as_token/hs_token live only in backend + appservice memory; agents get only their own scoped, re-mintable token (§4.2). Enroll is authenticated (RFC-001 B5) so a rogue local process can't enroll a rogue agent. Backend access is scoped: an agent's credential can read only its collection/policy, never the fleet-admin scope holding the crown jewels.

8.4 — Honest note: IP-only standalone with self-signed client TLS is a weaker-trust local mode. In Mode C, client TLS may be self-signed (or a local private CA). This means: no third party vouches for the endpoint; clients must be told to trust the self-signed cert (TOFU or manual root import); there is no external validation of who's on the other end. This is acceptable and supported for local/airgapped/homelab use where the network is already trusted, but it is explicitly a weaker trust posture than a real CA. The installer states this plainly at install (§6.8 Mode C). It is one more reason Mode C cannot federate: we will not extend this weaker-trust local posture across sites (NG5).

8.5 — Homeserver hardening (inherited from RFC-001 §8, config-rendered here): enable_registration: false always (agents come only via the appservice), rate-limiting on, admin API bound to localhost/behind auth, media repo locked/disabled if unused, TLS terminated at our controlled proxy. These are rendered from config (§5.4) so a stranger gets them by default, not by remembering to set them. [VERIFY] current recommended Synapse hardening flags at implementation.

---

9. How RFC-002 integrates with RFC-001's P1–P5

RFC-002 is the substrate. Each RFC-001 phase consumes a subset of it. Critically, P1 does not need the hard parts — presence ships on a single clean-domain instance with no federation, no IP-only, and no secret-rotation story resolved.

RFC-001 phase	RFC-002 pieces it NEEDS	RFC-002 pieces it does NOT need yet
P1 — Presence (first shippable slice)	Mode A or B, single-instance, clean domain (§2.3). One ACME cert (LE or step-ca) via the single integration (§3). Config system minimal: server_name immutable + presence thresholds tunable (§5.3). Installer path 6.1–6.3, 6.4 (cert), 6.7 (immutable gate), 6.8 (validate single-instance). A minimal SecretBackend just holding the appservice token.	No federation (§2.4 gate irrelevant — single site). No IP-only needed for P1 (P1 wants a clean domain so Element/humans connect cleanly). No agent-credential rotation maturity. No peer/whitelist config. Vaultwarden machine-account [VERIFY] does not block P1.
P2 — Appservice + auto-enroll	Full SecretBackend enrollAgent/revokeAgent (§4.1), hs_token/as_token custody (§4.2), enroll-bootstrap secret. Config: room taxonomy, per-agent token classification. Chosen backend (Vault or Vaultwarden) real.	Federation, cross-site, IP-only.
P3 — MACP v1 spec	Config keys for MACP versioning/thresholds as runtime-tunable (§5.3); nothing new topology-wise.	Federation, secret rotation-in-anger.
P4 — Federation	The whole federation half of RFC-002: Mode A delegation (§3.3), the hard DNS+cert gate (§2.4), domain_whitelist+peers config, cert monitoring/silent-defederation alarm (§3.4), per-CA trust distribution for cross-site (§8.1). This is where standalone→federated (§7 Case 1) and Jason's mosaic.woltje.com↔mosaic.uscllc.com shape land.	IP-only (federation excludes it by construction).
P5 — Hardening + signed-authorship + Hermes retired	Secret rotation runbooks executed in anger (§4, RFC-001 E2), pubkey-record custody for Ed25519 (§4.2), cert-rotation runbook (§3.4), full homeserver hardening validated (§8.5), backend auth review (§8.3).	—

One-line integration statement: P1 rides on the smallest slice of RFC-002 (single clean-domain instance + one cert + minimal config + minimal secret storage); the federation/IP-only/backend-maturity complexity is deferred to exactly the phases that need it (mostly P4/P5). RFC-002 therefore does not gate P1.

---

10. Open questions

Deliberately few — most topology/cert/secret decisions are resolved by Jason's rulings and baked in above.

1. [VERIFY] IP-only server_name acceptance. Does the deployed Synapse version accept an ip:port server_name and mint usable MXIDs for standalone (§2.3)? If not, Mode C uses a fabricated local domain (mosaic.local via local resolver) — confirm which, since it affects the re-home wording for Mode C→A/B (§7).
2. [VERIFY] Vaultwarden machine-account maturity. Confirm the current Vaultwarden version's Secrets Manager / machine-account coverage (§4.4). Determines whether the Vaultwarden adapter's enrollAgent uses native machine accounts or the personal-vault-per-agent + collection model. Does not block (interface absorbs either), but sets P2 expectations.
3. [VERIFY] step-ca root distribution for cross-site federation. Confirm Synapse's supported mechanism for trusting a custom federation CA root (§8.1) so two private-CA sites can federate. If Synapse won't easily trust a private federation CA, private-domain federation may in practice require public certs (LE) on the federation SANs even when internal traffic uses step-ca.
4. Default secret backend for the published installer. Given the open-source ethos (G6), should the installer default-suggest Vaultwarden (free, self-hostable) while clearly offering Vault, or present them neutrally? Recommendation: suggest Vaultwarden as the zero-cost path with the maturity caveat surfaced, Vault as the "I need native short-TTL machine creds now" path. Jason to confirm the framing.
5. Single multi-SAN cert vs two certs in Mode A (§3.3) — operational preference for identity-domain + host coverage. Minor; validate during P4.
6. Reconfigure-time Synapse restart policy. Which rendered-config changes (§5.4) require a Synapse restart vs hot-reload on the deployed version, so the config service knows when a tunable change needs a bounce. [VERIFY] at implementation.

---

Appendix A — Real mechanics this RFC leans on (quick reference)

• server_name — Synapse identity domain; the :suffix of every MXID/alias; install-immutable; changing it = re-home (no in-place rename). Distinct from where Synapse listens.
• Delegation — https://<server_name>/.well-known/matrix/server → {"m.server":"host:port"} and/or _matrix._tcp.<server_name> SRV; how identity-domain ≠ host is reconciled (Mode A). [VERIFY] precedence on deployed Synapse.
• .well-known/matrix/client — C-S discovery so agents/Element find the homeserver host from the identity domain.
• federation_domain_whitelist — Synapse allowlist; only listed domains federate; our hard no-public-federation boundary.
• ACME — single provisioning protocol for both CAs; operator supplies directory URL (step-ca vs Let's Encrypt) + challenge type.
• HTTP-01 / DNS-01 / TLS-ALPN-01 — challenge types; DNS-01 is the one for private/split-horizon/wildcard; HTTP-01 for public single host with :80; TLS-ALPN-01 for :443-only public.
• step-ca ACME provisioner — Smallstep's self-hosted CA exposing an ACME directory; enables private/internal-domain certs and total control; may require EAB; root must be distributed to validators.
• Let's Encrypt — public ACME CA; universally-trusted chain; 90-day certs; staging endpoint for testing.
• Bitwarden/Vaultwarden org + collection + machine/service-account — org holds collections; collections scope access; machine/service accounts (Bitwarden Secrets Manager) are the clean automation primitive but Vaultwarden coverage is partial/evolving [VERIFY]; personal-vault-per-agent + org-collection works today.
• Vault KV v2 / AppRole / lease-TTL / revoke — the capable backend; native versioning=rotate, TTL+revoke=transient per-agent creds.
• Silent defederation — a lapsed/renewal-failed federation cert causes peers to stop trusting S2S with no local error; must be monitored + alarmed (§3.4).

All Matrix/ACME/secret-backend mechanics above are cited from architecture knowledge and MUST be re-verified against the actually deployed versions during implementation — every [VERIFY] is a checkpoint, not an assumption. Every illustrative domain (mosaic.woltje.com, mosaic.uscllc.com, matrix.woltje.com) is an operator-supplied example, never a product default or literal.