chore: consolidate new foundation and archive v1 (#1495)

This commit is contained in:
2026-09-07 12:32:57 -05:00
3511 changed files with 727899 additions and 10 deletions
+3
View File
@@ -0,0 +1,3 @@
# DEV runtime state: rendered config, sqlite db, signing key, self-signed
# certs, throwaway secrets. Never committed.
.data/
+53
View File
@@ -0,0 +1,53 @@
# infra/matrix — DEV Synapse for RFC-001 P1 (presence)
> **DEV-ONLY. LOCAL SANDBOX.** This stack is for local presence validation. It
> must **never** be pointed at, or run alongside, a live/production homeserver.
> It is single-instance, federation-OFF, self-signed TLS — the smallest slice
> RFC-002 §9 says P1 needs (Mode B single-domain, no federation, no IP-only, no
> secret-rotation story).
## What this is
A single Synapse homeserver rendered from committed **templates** (no hardcoded
topology — RFC-002 G2). Every topology fact is an environment variable with a
dev default:
| Var | Default | Meaning |
| -------------------- | ------------------ | --------------------------------------------------- |
| `MATRIX_SERVER_NAME` | `matrix.localhost` | Synapse `server_name` (the `:suffix` of every MXID) |
| `MOSAIC_AS_ID` | `mosaic-as` | appservice id / registration filename |
| `MATRIX_HTTP_PORT` | `18008` | host port → Synapse 8008 (plain HTTP) |
| `MATRIX_TLS_PORT` | `18448` | host port → Synapse 8448 (self-signed TLS) |
Secrets (`as_token`, `hs_token`, Synapse macaroon/form/registration secrets)
are **throwaway values generated at bring-up** into `.data/dev-secrets.env`
(gitignored). In production these are crown-jewel secrets held by the
SecretBackend (RFC-001 §8 / RFC-002 §4) — never committed.
## Files
- `docker-compose.dev.yml` — Synapse (+ optional Element under `--profile element`).
- `synapse/homeserver.dev.yaml.tpl` — rendered Synapse config (Mode B, `enable_registration: false`, appservice wired, native TLS).
- `synapse/log.config` — Synapse logging.
- `appservice/mosaic-as.dev.yaml.tpl` — minimal AS registration (declares the `@agent-*` user namespace).
- `dev-up.sh` / `dev-down.sh` — bring up / tear down (`--purge` wipes `.data`).
- `.data/`**gitignored** runtime state (rendered config, sqlite db, signing key, self-signed certs, dev secrets).
## Usage
```bash
./dev-up.sh # render config, gen signing key + TLS cert, boot Synapse
# ... run the validation harness (tools/matrix-presence-harness/run.sh) ...
./dev-down.sh # stop, keep .data
./dev-down.sh --purge # stop and wipe .data for a pristine next boot
# optional human view (A4) — Element pointed at the dev server:
docker compose -f docker-compose.dev.yml --profile element up -d element
# -> http://127.0.0.1:18080
```
## Acceptance evidence (A1)
- TLS (self-signed) reachable: `curl -sk https://127.0.0.1:18448/_matrix/client/versions``200`.
- Open registration OFF: `POST /_matrix/client/v3/register``M_FORBIDDEN "Registration has been disabled"`.
- AS-token registration bypasses the flag by design (that is how agents are provisioned).
@@ -0,0 +1,30 @@
# ============================================================================
# mosaic-as.dev.yaml.tpl — Mosaic Appservice registration (DEV)
# ============================================================================
#
# *** DEV-ONLY. The tokens below are throwaway placeholders rendered at
# bring-up by dev-up.sh. NEVER commit real hs_token/as_token — in
# production they are crown-jewel secrets held by the SecretBackend
# (RFC-001 §8, RFC-002 §4). ***
#
# This is the MINIMAL P1 registration: it declares the @agent-* user
# namespace so the P1 provisioner can register a few virtual agent MXIDs and
# carry heartbeats. It intentionally does NOT model the full P2 taxonomy /
# token-minting appservice.
#
# Rendered by infra/matrix/dev-up.sh (envsubst -> .data/${MOSAIC_AS_ID}.yaml).
# ----------------------------------------------------------------------------
id: "${MOSAIC_AS_ID}"
url: null # P1: provisioner drives the AS API directly; Synapse pushes no txns.
as_token: "${MOSAIC_AS_TOKEN}"
hs_token: "${MOSAIC_HS_TOKEN}"
sender_localpart: "mosaic-as"
rate_limited: false
namespaces:
users:
- exclusive: true
regex: "@agent-.*:${MATRIX_SERVER_NAME}"
aliases:
- exclusive: false
regex: "#mosaic-.*:${MATRIX_SERVER_NAME}"
rooms: []
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
# dev-down.sh — tear down the DEV Synapse (matrix-p1-dev). DEV-ONLY.
# ./dev-down.sh # stop + remove containers, keep ./.data
# ./dev-down.sh --purge # also delete ./.data (fresh next bring-up)
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
docker compose -f "${HERE}/docker-compose.dev.yml" --profile element down -v --remove-orphans || true
if [[ "${1:-}" == "--purge" ]]; then
rm -rf "${HERE}/.data"
echo "[dev-down] purged ${HERE}/.data"
fi
echo "[dev-down] matrix-p1-dev stopped"
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# ============================================================================
# dev-up.sh — bring up the DEV Synapse for RFC-001 P1 (presence)
# ============================================================================
#
# *** DEV-ONLY. LOCAL SANDBOX. This script must never be pointed at live
# infra. It writes only into infra/matrix/.data (gitignored) and drives
# a dedicated compose project (matrix-p1-dev). ***
#
# It renders the Synapse config + appservice registration from the committed
# templates (envsubst), generates a DEV signing key + self-signed TLS cert,
# and starts Synapse. All topology facts come from the environment with dev
# defaults (RFC-002 G2: nothing hardcoded).
# ----------------------------------------------------------------------------
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DATA="${HERE}/.data"
COMPOSE=(docker compose -f "${HERE}/docker-compose.dev.yml")
# ---- Topology inputs (dev defaults; override via env) ----------------------
export MATRIX_SERVER_NAME="${MATRIX_SERVER_NAME:-matrix.localhost}"
export MOSAIC_AS_ID="${MOSAIC_AS_ID:-mosaic-as}"
export MATRIX_HTTP_PORT="${MATRIX_HTTP_PORT:-18008}"
export MATRIX_TLS_PORT="${MATRIX_TLS_PORT:-18448}"
# ---- DEV secrets (throwaway; regenerated if absent) ------------------------
SECRETS_ENV="${DATA}/dev-secrets.env"
mkdir -p "${DATA}"
if [[ ! -f "${SECRETS_ENV}" ]]; then
{
echo "MOSAIC_AS_TOKEN=devas_$(openssl rand -hex 16)"
echo "MOSAIC_HS_TOKEN=devhs_$(openssl rand -hex 16)"
echo "SYNAPSE_REG_SHARED_SECRET=devreg_$(openssl rand -hex 16)"
echo "SYNAPSE_MACAROON_SECRET=devmac_$(openssl rand -hex 16)"
echo "SYNAPSE_FORM_SECRET=devform_$(openssl rand -hex 16)"
} > "${SECRETS_ENV}"
echo "[dev-up] generated throwaway DEV secrets -> ${SECRETS_ENV}"
fi
# shellcheck disable=SC1090
set -a; source "${SECRETS_ENV}"; set +a
# DEV: let the synapse container user (uid 991) write the sqlite db / media.
chmod 0777 "${DATA}" || true
# ---- Render config from templates ------------------------------------------
envsubst < "${HERE}/synapse/homeserver.dev.yaml.tpl" > "${DATA}/homeserver.yaml"
envsubst < "${HERE}/appservice/mosaic-as.dev.yaml.tpl" > "${DATA}/${MOSAIC_AS_ID}.yaml"
cp "${HERE}/synapse/log.config" "${DATA}/log.config"
echo "[dev-up] rendered homeserver.yaml + ${MOSAIC_AS_ID}.yaml (server_name=${MATRIX_SERVER_NAME})"
# ---- DEV signing key -------------------------------------------------------
# Generate as the synapse container user (uid 991) so the running container
# can read it; owned-by-991 mode-600 is fine (the container IS 991).
SIGNING_KEY="${DATA}/${MATRIX_SERVER_NAME}.signing.key"
if [[ ! -f "${SIGNING_KEY}" ]]; then
docker run --rm --user 991:991 -v "${DATA}:/data" --entrypoint generate_signing_key \
matrixdotorg/synapse:latest -o "/data/${MATRIX_SERVER_NAME}.signing.key"
echo "[dev-up] generated DEV signing key"
fi
# ---- DEV self-signed TLS cert (A1) -----------------------------------------
if [[ ! -f "${DATA}/dev-cert.pem" ]]; then
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout "${DATA}/dev-key.pem" -out "${DATA}/dev-cert.pem" \
-days 90 -subj "/CN=${MATRIX_SERVER_NAME}" \
-addext "subjectAltName=DNS:${MATRIX_SERVER_NAME},DNS:localhost,IP:127.0.0.1" >/dev/null 2>&1
echo "[dev-up] generated DEV self-signed TLS cert for ${MATRIX_SERVER_NAME}"
fi
# Files hermes owns must be world-readable so the synapse (uid 991) container
# can read them (DEV-only; the signing key stays owned by 991 from above).
chmod 0644 "${DATA}/dev-cert.pem" "${DATA}/dev-key.pem" \
"${DATA}/homeserver.yaml" "${DATA}/${MOSAIC_AS_ID}.yaml" "${DATA}/log.config" || true
# ---- Element config (optional human view, A4) ------------------------------
cat > "${DATA}/element-config.json" <<JSON
{
"default_server_config": {
"m.homeserver": {
"base_url": "https://localhost:${MATRIX_TLS_PORT}",
"server_name": "${MATRIX_SERVER_NAME}"
}
},
"disable_guests": true,
"brand": "Mosaic P1 (DEV)"
}
JSON
# ---- Boot ------------------------------------------------------------------
"${COMPOSE[@]}" up -d synapse
echo -n "[dev-up] waiting for Synapse health"
for _ in $(seq 1 60); do
status="$("${COMPOSE[@]}" ps --format '{{.Health}}' synapse 2>/dev/null || true)"
if [[ "${status}" == "healthy" ]]; then echo " ... healthy"; break; fi
echo -n "."; sleep 2
done
if ! curl -fsS "http://127.0.0.1:${MATRIX_HTTP_PORT}/health" >/dev/null 2>&1; then
echo "[dev-up] ERROR: Synapse did not become healthy" >&2
"${COMPOSE[@]}" logs --tail 60 synapse >&2 || true
exit 1
fi
echo "[dev-up] Synapse UP:"
echo " plain HTTP : http://127.0.0.1:${MATRIX_HTTP_PORT}"
echo " TLS : https://127.0.0.1:${MATRIX_TLS_PORT} (self-signed, DEV)"
echo " server_name: ${MATRIX_SERVER_NAME} registration: OFF"
+60
View File
@@ -0,0 +1,60 @@
# ============================================================================
# docker-compose.dev.yml — Synapse DEV homeserver for RFC-001 P1 (presence)
# ============================================================================
#
# *** DEV-ONLY. LOCAL SANDBOX. Do NOT point this at, or run alongside, any
# live/production homeserver. ***
#
# Single-instance Synapse (RFC-002 Mode B, federation OFF) + an optional
# Element web client for the human view (A4). Brought up by ./dev-up.sh, which
# renders config into ./.data (gitignored) first.
#
# Isolated by design: dedicated project name + network + high host ports so it
# never collides with other stacks on the box (A5).
#
# Host ports: ${MATRIX_HTTP_PORT:-18008} -> Synapse 8008 (plain HTTP)
# ${MATRIX_TLS_PORT:-18448} -> Synapse 8448 (self-signed TLS)
# ${ELEMENT_PORT:-18080} -> Element web (profile: element)
# ----------------------------------------------------------------------------
name: matrix-p1-dev
services:
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix-p1-synapse
restart: 'no'
# Run against the rendered config in /data (dev-up.sh puts it there).
environment:
SYNAPSE_CONFIG_DIR: /data
SYNAPSE_CONFIG_PATH: /data/homeserver.yaml
volumes:
- ./.data:/data
ports:
- '127.0.0.1:${MATRIX_HTTP_PORT:-18008}:8008'
- '127.0.0.1:${MATRIX_TLS_PORT:-18448}:8448'
networks:
- matrix-p1-net
healthcheck:
test: ['CMD-SHELL', 'curl -fsS http://localhost:8008/health || exit 1']
interval: 3s
timeout: 5s
retries: 40
start_period: 5s
# Optional human view (A4). `docker compose --profile element up -d element`.
element:
image: vectorim/element-web:latest
container_name: matrix-p1-element
restart: 'no'
profiles: [element]
volumes:
- ./.data/element-config.json:/app/config.json:ro
ports:
- '127.0.0.1:${ELEMENT_PORT:-18080}:80'
networks:
- matrix-p1-net
networks:
matrix-p1-net:
name: matrix-p1-net
driver: bridge
@@ -0,0 +1,84 @@
# ============================================================================
# homeserver.dev.yaml.tpl — Synapse DEV homeserver config (RFC-002 Mode B)
# ============================================================================
#
# *** DEV-ONLY. NOT FOR PRODUCTION. ***
#
# This is the RFC-001 P1 (presence) single-instance homeserver. It is
# RFC-002 "Mode B — single-domain" (server_name == homeserver host), with
# federation OFF (standalone), which is all P1 needs (RFC-002 §9 P1 row).
#
# NOTHING here is a production value. Every topology fact is a variable
# rendered by dev-up.sh from the environment; there is no baked-in fleet
# domain (RFC-002 G2 "zero hardcoded topology"). The secrets below are
# throwaway DEV placeholders rendered at bring-up — never reuse them.
#
# Rendered by: infra/matrix/dev-up.sh (envsubst -> .data/homeserver.yaml)
# Variables: MATRIX_SERVER_NAME, MOSAIC_AS_ID (+ dev secrets)
# ----------------------------------------------------------------------------
server_name: "${MATRIX_SERVER_NAME}"
pid_file: /data/homeserver.pid
report_stats: false
suppress_key_server_warning: true
# --- Listeners -------------------------------------------------------------
# 8008: plain HTTP (client + federation) for in-container/local tooling.
# 8448: TLS (self-signed in DEV) — satisfies A1 "reachable over TLS".
listeners:
- port: 8008
type: http
tls: false
bind_addresses: ['0.0.0.0']
x_forwarded: true
resources:
- names: [client, federation]
compress: false
- port: 8448
type: http
tls: true
bind_addresses: ['0.0.0.0']
resources:
- names: [client, federation]
compress: false
# --- DEV TLS (self-signed; generated by dev-up.sh) -------------------------
tls_certificate_path: "/data/dev-cert.pem"
tls_private_key_path: "/data/dev-key.pem"
# --- Store: sqlite is the simplest dev store (RFC-002 §1 allows it) --------
database:
name: sqlite3
args:
database: /data/homeserver.db
log_config: "/data/log.config"
media_store_path: /data/media_store
signing_key_path: "/data/${MATRIX_SERVER_NAME}.signing.key"
# --- Hardening (RFC-001 §8 / RFC-002 §8.5), rendered so a dev gets it ------
# Registration is OFF: agents come ONLY via the appservice (A1). AS user
# registration bypasses this flag, which is exactly the design.
enable_registration: false
enable_registration_without_verification: false
registration_shared_secret: "${SYNAPSE_REG_SHARED_SECRET}"
macaroon_secret_key: "${SYNAPSE_MACAROON_SECRET}"
form_secret: "${SYNAPSE_FORM_SECRET}"
# Presence EDUs ON so Element shows the native dot for humans (RFC-001 §4.5);
# the AUTHORITATIVE liveness is still the mosaic.presence heartbeat.
presence:
enabled: true
# --- Federation: OFF for P1 (standalone). Empty whitelist = federate with
# nobody (RFC-002 §2.4 / NG5). No public-network federation. ----------------
federation_domain_whitelist: []
trusted_key_servers: []
# --- Appservice registration wired in (A1). The file is rendered next to
# this one by dev-up.sh. -----------------------------------------------------
app_service_config_files:
- "/data/${MOSAIC_AS_ID}.yaml"
# Keep default rate-limiting ON (RFC-001 §8). No overrides here.
+16
View File
@@ -0,0 +1,16 @@
# Synapse DEV log config. DEV-ONLY.
version: 1
formatters:
precise:
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
handlers:
console:
class: logging.StreamHandler
formatter: precise
loggers:
synapse.storage.SQL:
level: WARNING
root:
level: INFO
handlers: [console]
disable_existing_loggers: false
+35
View File
@@ -0,0 +1,35 @@
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
exporters:
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
debug:
verbosity: basic
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/jaeger, debug]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug]
+1
View File
@@ -0,0 +1 @@
CREATE EXTENSION IF NOT EXISTS vector;
+1
View File
@@ -0,0 +1 @@
dev-only-step-ca-password-do-not-use-in-production
+90
View File
@@ -0,0 +1,90 @@
#!/bin/sh
# infra/step-ca/init.sh
#
# Idempotent first-boot initialiser for the Mosaic Federation CA.
#
# On the first run (no /home/step/config/ca.json present) this script:
# 1. Initialises Step-CA with a JWK provisioner named "mosaic-fed".
# 2. Writes the CA configuration to the persistent volume at /home/step.
# 3. Copies the federation X.509 template into the CA config directory.
# 4. Patches the mosaic-fed provisioner entry in ca.json to reference the
# template via options.x509.templateFile (using jq — must be installed
# in the container image).
#
# On subsequent runs (config already exists) this script skips init and
# starts the CA directly.
#
# The provisioner name "mosaic-fed" is consumed by:
# apps/gateway/src/federation/ca.service.ts (added in M2-04)
#
# Password source:
# Dev: mounted from ./infra/step-ca/dev-password via bind mount.
# Prod: mounted from a Docker secret at /run/secrets/ca_password.
#
# OID template:
# infra/step-ca/templates/federation.tpl emits custom OID extensions:
# 1.3.6.1.4.1.99999.1 — mosaic_grant_id
# 1.3.6.1.4.1.99999.2 — mosaic_subject_user_id
set -e
CA_CONFIG="/home/step/config/ca.json"
PASSWORD_FILE="/run/secrets/ca_password"
TEMPLATE_SRC="/etc/step-ca-templates/federation.tpl"
TEMPLATE_DEST="/home/step/templates/federation.tpl"
if [ ! -f "${CA_CONFIG}" ]; then
echo "[step-ca init] First boot detected — initialising Mosaic Federation CA..."
step ca init \
--name "Mosaic Federation CA" \
--dns "localhost" \
--dns "step-ca" \
--address ":9000" \
--provisioner "mosaic-fed" \
--password-file "${PASSWORD_FILE}" \
--provisioner-password-file "${PASSWORD_FILE}" \
--no-db
echo "[step-ca init] CA initialised."
# Copy the X.509 template into the Step-CA config directory.
if [ -f "${TEMPLATE_SRC}" ]; then
mkdir -p /home/step/templates
cp "${TEMPLATE_SRC}" "${TEMPLATE_DEST}"
echo "[step-ca init] Federation X.509 template copied to ${TEMPLATE_DEST}."
else
echo "[step-ca init] WARNING: Template source ${TEMPLATE_SRC} not found — skipping copy."
fi
# Wire the template into the mosaic-fed provisioner via jq.
# This is idempotent: the block only runs once (first boot).
#
# jq filter: find the provisioner entry with name "mosaic-fed" and set
# .options.x509.templateFile to the absolute path of the template.
# All other provisioners and config keys are left unchanged.
if [ -f "${TEMPLATE_DEST}" ] && command -v jq > /dev/null 2>&1; then
echo "[step-ca init] Patching mosaic-fed provisioner with X.509 template..."
TEMP_CONFIG="${CA_CONFIG}.tmp"
jq --arg tpl "${TEMPLATE_DEST}" '
.authority.provisioners |= map(
if .name == "mosaic-fed" then
.options.x509.templateFile = $tpl
else
.
end
)
' "${CA_CONFIG}" > "${TEMP_CONFIG}" && mv "${TEMP_CONFIG}" "${CA_CONFIG}"
echo "[step-ca init] Provisioner patched."
elif ! command -v jq > /dev/null 2>&1; then
echo "[step-ca init] WARNING: jq not found — skipping provisioner template patch."
echo "[step-ca init] Install jq in the step-ca image to enable automatic template wiring."
fi
echo "[step-ca init] Startup complete."
else
echo "[step-ca init] Config already exists — skipping init."
fi
echo "[step-ca init] Starting Step-CA on :9000..."
exec step-ca /home/step/config/ca.json --password-file "${PASSWORD_FILE}"
+56
View File
@@ -0,0 +1,56 @@
{
"subject": {{ toJson .Subject }},
"sans": {{ toJson .SANs }},
{{- /*
Mosaic Federation X.509 Certificate Template
============================================
Provisioner: mosaic-fed (JWK)
Implemented: FED-M2-04
This template emits two custom OID extensions carrying Mosaic federation
identifiers. The OTT token (built by CaService.buildOtt) MUST include the
claims `mosaic_grant_id` and `mosaic_subject_user_id` as top-level JWT
claims. step-ca exposes them under `.Token.<claim>` in this template.
OID Registry (Mosaic Internal Arc 1.3.6.1.4.1.99999):
1.3.6.1.4.1.99999.1 mosaic_grant_id (UUID, 36 ASCII chars)
1.3.6.1.4.1.99999.2 mosaic_subject_user_id (UUID, 36 ASCII chars)
DER encoding for each extension value (ASN.1 UTF8String):
Tag = 0x0C (UTF8String)
Length = 0x24 (decimal 36 the fixed length of a UUID string)
Value = 36 ASCII bytes of the UUID
The `printf` below builds the raw TLV bytes then base64-encodes them.
step-ca expects the `value` field to be base64-encoded raw DER bytes.
Fail-loud contract:
If either claim is missing from the token the printf will produce a
zero-length UUID field, making the extension malformed. step-ca will
reject the certificate rather than issuing one without the required OIDs.
Silent OID stripping is NEVER tolerated.
Step-CA template reference:
https://smallstep.com/docs/step-ca/templates
*/ -}}
"extensions": [
{
"id": "1.3.6.1.4.1.99999.1",
"critical": false,
"value": "{{ printf "\x0c%c%s" (len .Token.mosaic_grant_id) .Token.mosaic_grant_id | b64enc }}"
},
{
"id": "1.3.6.1.4.1.99999.2",
"critical": false,
"value": "{{ printf "\x0c%c%s" (len .Token.mosaic_subject_user_id) .Token.mosaic_subject_user_id | b64enc }}"
}
],
"keyUsage": ["digitalSignature"],
"extKeyUsage": ["clientAuth"],
"basicConstraints": {
"isCA": false
}
}