stack/docs/federation/MILESTONES.md

# Mosaic Stack — Federation Implementation Milestones

**Companion to:** `PRD.md`
**Approach:** Each milestone is a verifiable slice. A milestone is "done" only when its acceptance tests pass in CI against a real (not mocked) dependency stack.

---

## Milestone Dependency Graph

```
M1 (federated tier infra)
  └── M2 (Step-CA + grant schema + CLI)
        └── M3 (mTLS handshake + list/get + scope enforcement)
              ├── M4 (search + audit + rate limit)
              │     └── M5 (cache + offline degradation + OTEL)
              ├── M6 (revocation + auto-renewal)  ◄── can start after M3
              └── M7 (multi-user hardening + e2e suite)  ◄── depends on M4+M5+M6
```

M5 and M6 can run in parallel once M4 is merged.

---

## Test Strategy (applies to all milestones)

Three layers, all required before a milestone ships:

| Layer              | Scope                                         | Runtime                                                                  |
| ------------------ | --------------------------------------------- | ------------------------------------------------------------------------ |
| **Unit**           | Per-module logic, pure functions, adapters    | Vitest, no I/O                                                           |
| **Integration**    | Single gateway against real PG/Valkey/Step-CA | Vitest + Docker Compose test profile                                     |
| **Federation E2E** | Two gateways on a Docker network, real mTLS   | Playwright/custom harness (`tools/federation-harness/`) introduced in M3 |

Every milestone adds tests to these layers. A milestone cannot be claimed complete if the federation E2E harness fails (applies from M3 onward).

**Quality gates per milestone** (same as stack-wide):

- `pnpm typecheck` green
- `pnpm lint` green
- `pnpm test` green (unit + integration)
- `pnpm test:federation` green (M3+)
- Independent code review passed
- Docs updated (`docs/federation/`)
- Merged PR on `main`, CI terminal green, linked issue closed

---

## M1 — Federated Tier Infrastructure

**Goal:** A gateway can run in `federated` tier with containerized Postgres + Valkey + pgvector, with no federation logic active yet.

**Scope:**

- Add `"tier": "federated"` to `mosaic.config.json` schema and validators
- Docker Compose `federated` profile (`docker-compose.federated.yml`) adds: Postgres+pgvector (5433), Valkey (6380), dedicated volumes
- Tier detector in gateway bootstrap: reads config, asserts required services reachable, refuses to start otherwise
- `pgvector` extension installed + verified on startup
- Migration logic: safe upgrade path from `local`/`standalone` → `federated` (data export/import script, one-way)
- `mosaic doctor` reports tier + service health
- Gateway continues to serve as a normal standalone instance (no federation yet)

**Deliverables:**

- `mosaic.config.json` schema v2 (tier enum includes `federated`)
- `apps/gateway/src/bootstrap/tier-detector.ts`
- `docker-compose.federated.yml`
- `scripts/migrate-to-federated.ts`
- Updated `mosaic doctor` output
- Updated `packages/storage/src/adapters/postgres.ts` with pgvector support

**Acceptance tests:**
| # | Test | Layer |
| - | ---------------------------------------------------------------------------------------- | ----------- |
| 1 | Gateway boots in `federated` tier with all services present | Integration |
| 2 | Gateway refuses to boot in `federated` tier when Postgres unreachable (fail-fast, clear) | Integration |
| 3 | `pgvector` extension available in target DB (`SELECT * FROM pg_extension WHERE extname='vector'`) | Integration |
| 4 | Migration script moves a populated `local` (PGlite) instance to `federated` (Postgres) with no data loss | Integration |
| 5 | `mosaic doctor` reports correct tier and all services green | Unit |
| 6 | Existing standalone behavior regression: agent session works end-to-end, no federation references | E2E (single-gateway) |

**Estimated budget:** ~20K tokens (infra + config + migration script)
**Risk notes:** Pgvector install on existing PG installs is occasionally finicky; test the migration path on a realistic DB snapshot.

---

## M2 — Step-CA + Grant Schema + Admin CLI

**Goal:** An admin can create a federation grant and its counterparty can enroll. No runtime traffic flows yet.

**Scope:**

- Embed Step-CA as a Docker Compose sidecar with a persistent CA volume
- Gateway exposes a short-lived enrollment endpoint (single-use token from the grant)
- DB schema: `federation_grants`, `federation_peers`, `federation_audit_log` (table only, not yet written to)
- Sealed storage for `client_key_pem` using the existing credential sealing key
- Admin CLI:
  - `mosaic federation grant create --user <id> --peer <host> --scope <file>`
  - `mosaic federation grant list`
  - `mosaic federation grant show <id>`
  - `mosaic federation peer add <enrollment-url>`
  - `mosaic federation peer list`
- Step-CA signs the cert with SAN OIDs for `grantId` + `subjectUserId`
- Grant status transitions: `pending` → `active` on successful enrollment

**Deliverables:**

- `packages/db` migration: three federation tables + enum types
- `apps/gateway/src/federation/ca.service.ts` (Step-CA client)
- `apps/gateway/src/federation/grants.service.ts`
- `apps/gateway/src/federation/enrollment.controller.ts`
- `packages/mosaic/src/commands/federation/` (grant + peer subcommands)
- `docker-compose.federated.yml` adds Step-CA service
- Scope JSON schema + validator

**Acceptance tests:**
| # | Test | Layer |
| - | ---------------------------------------------------------------------------------------- | ----------- |
| 1 | `grant create` writes a `pending` row with a scoped bundle | Integration |
| 2 | Enrollment endpoint signs a CSR and returns a cert with expected SAN OIDs | Integration |
| 3 | Enrollment token is single-use; second attempt returns 410 | Integration |
| 4 | Cert `subjectUserId` OID matches the grant's `subject_user_id` | Unit |
| 5 | `client_key_pem` is at-rest encrypted; raw DB read shows ciphertext, not PEM | Integration |
| 6 | `peer add <url>` on Server A yields an `active` peer record with a valid cert + key | E2E (two gateways, no traffic) |
| 7 | Scope JSON with unknown resource type rejected at `grant create` | Unit |
| 8 | `grant list` and `peer list` render active / pending / revoked accurately | Unit |

**Estimated budget:** ~30K tokens (schema + CA integration + CLI + sealing)
**Risk notes:** Step-CA's API surface is well-documented but the sealing integration with existing provider-credential encryption is a cross-module concern — walk that seam deliberately.

---

## M3 — mTLS Handshake + `list` + `get` with Scope Enforcement

**Goal:** Two federated gateways exchange real data over mTLS with scope intersecting native RBAC.

**Scope:**

- `FederationClient` (outbound): picks cert from `federation_peers`, does mTLS call
- `FederationServer` (inbound): NestJS guard validates client cert, extracts `grantId` + `subjectUserId`, loads grant
- Scope enforcement pipeline:
  1. Resource allowlist / excluded-list check
  2. Native RBAC evaluation as the `subjectUserId`
  3. Scope filter intersection (`include_teams`, `include_personal`)
  4. `max_rows_per_query` cap
- Verbs: `list`, `get`, `capabilities`
- Gateway query layer accepts `source: "local" | "federated:<host>" | "all"`; fan-out for `"all"`
- **Federation E2E harness** (`tools/federation-harness/`): docker-compose.two-gateways.yml, seed script, assertion helpers — this is its own deliverable

**Deliverables:**

- `apps/gateway/src/federation/client/federation-client.service.ts`
- `apps/gateway/src/federation/server/federation-auth.guard.ts`
- `apps/gateway/src/federation/server/scope.service.ts`
- `apps/gateway/src/federation/server/verbs/{list,get,capabilities}.controller.ts`
- `apps/gateway/src/federation/client/query-source.service.ts` (fan-out/merge)
- `tools/federation-harness/` (compose + seed + test helpers)
- `packages/types` — federation request/response DTOs in `federation.dto.ts`

**Acceptance tests:**
| # | Test | Layer |
| -- | -------------------------------------------------------------------------------------------------------- | ----- |
| 1 | A→B `list tasks` returns subjectUser's tasks intersected with scope | E2E |
| 2 | A→B `list tasks` with `include_teams: [T1]` excludes T2 tasks the user owns | E2E |
| 3 | A→B `get credential <id>` returns 403 when `credentials` is in `excluded_resources` | E2E |
| 4 | Client presenting cert for grant X cannot query subjectUser of grant Y (cross-user isolation) | E2E |
| 5 | Cert signed by untrusted CA rejected at TLS layer (no NestJS handler reached) | E2E |
| 6 | Malformed SAN OIDs → 401; cert valid but grant revoked in DB → 403 | Integration |
| 7 | `max_rows_per_query` caps response; request for more paginated | Integration |
| 8 | `source: "all"` fan-out merges local + federated results, each tagged with `_source` | Integration |
| 9 | Federation responses never persist: verify DB row count unchanged after `list` round-trip | E2E |
| 10 | Scope cannot grant more than native RBAC: user without access to team T still gets [] even if scope allows T | E2E |

**Estimated budget:** ~40K tokens (largest milestone — core federation logic + harness)
**Risk notes:** This is the critical trust boundary. Code review should focus on scope enforcement bypass and cert-SAN-spoofing paths. Every 403/401 path needs a test.

---

## M4 — `search` Verb + Audit Log + Rate Limit

**Goal:** Keyword search over allowed resources with full audit and per-grant rate limiting.

**Scope:**

- `search` verb across `resources` allowlist (intersection of scope + native RBAC)
- Keyword search (reuse existing `packages/memory/src/adapters/keyword.ts`); pgvector search stays out of v1 search verb
- Every federated request (all verbs) writes to `federation_audit_log`: `grant_id`, `verb`, `resource`, `query_hash`, `outcome`, `bytes_out`, `latency_ms`
- No request body captured; `query_hash` is SHA-256 of normalized query params
- Token-bucket rate limit per grant (default 60/min, override per grant)
- 429 response with `Retry-After` header and structured body
- 90-day hot retention for audit log; cold-tier rollover deferred to M7

**Deliverables:**

- `apps/gateway/src/federation/server/verbs/search.controller.ts`
- `apps/gateway/src/federation/server/audit.service.ts` (async write, no blocking)
- `apps/gateway/src/federation/server/rate-limit.guard.ts`
- Tests in harness

**Acceptance tests:**
| # | Test | Layer |
| - | ------------------------------------------------------------------------------------------------- | ----------- |
| 1 | `search` returns ranked hits only from allowed resources | E2E |
| 2 | `search` excluding `credentials` does not return a match even when keyword matches a credential name | E2E |
| 3 | Every successful request appears in `federation_audit_log` within 1s | Integration |
| 4 | Denied request (403) is also audited with `outcome='denied'` | Integration |
| 5 | Audit row stores query hash but NOT query body | Unit |
| 6 | 61st request in 60s window returns 429 with `Retry-After` | E2E |
| 7 | Per-grant override (e.g., 600/min) takes effect without restart | Integration |
| 8 | Audit writes are async: request latency unchanged when audit write slow (simulated) | Integration |

**Estimated budget:** ~20K tokens
**Risk notes:** Ensure audit writes can't block or error-out the request path; use a bounded queue and drop-with-counter pattern rather than in-line writes.

---

## M5 — Cache + Offline Degradation + Observability

**Goal:** Sessions feel fast and stay useful when the peer is slow or down.

**Scope:**

- In-memory response cache keyed by `(grant_id, verb, resource, query_hash)`, TTL 30s default
- Cache NOT used for `search`; only `list` and `get`
- Cache flushed on cert rotation and grant revocation
- Circuit breaker per peer: after N failures, fast-fail for cooldown window
- `_source` tagging extended with `_cached: true` when served from cache
- Agent-visible "federation offline for `<peer>`" signal emitted once per session per peer
- OTEL spans: `federation.request` with attrs `grant_id`, `peer`, `verb`, `resource`, `outcome`, `latency_ms`, `cached`
- W3C `traceparent` propagated across the mTLS boundary (both directions)
- `mosaic federation status` CLI subcommand

**Deliverables:**

- `apps/gateway/src/federation/client/response-cache.service.ts`
- `apps/gateway/src/federation/client/circuit-breaker.service.ts`
- `apps/gateway/src/federation/observability/` (span helpers)
- `packages/mosaic/src/commands/federation/status.ts`

**Acceptance tests:**
| # | Test | Layer |
| - | --------------------------------------------------------------------------------------------- | ----- |
| 1 | Two identical `list` calls within 30s: second served from cache, flagged `_cached` | Integration |
| 2 | `search` is never cached: two identical searches both hit the peer | Integration |
| 3 | After grant revocation, peer's cache is flushed immediately | Integration |
| 4 | After N consecutive failures, circuit opens; subsequent requests fail-fast without network call | E2E |
| 5 | Circuit closes after cooldown and next success | E2E |
| 6 | With peer offline, session completes using local data, one "federation offline" signal surfaced | E2E |
| 7 | OTEL traces show spans on both gateways correlated by `traceparent` | E2E |
| 8 | `mosaic federation status` prints peer state, cert expiry, last success/failure, circuit state | Unit |

**Estimated budget:** ~20K tokens
**Risk notes:** Caching correctness under revocation must be provable — write tests that intentionally race revocation against cached hits.

---

## M6 — Revocation, Auto-Renewal, CRL

**Goal:** Grant lifecycle works end-to-end: admin revoke, revoke-on-delete, automatic cert renewal, CRL distribution.

**Scope:**

- `mosaic federation grant revoke <id>` → status `revoked`, CRL updated, audit entry
- DB hook: deleting a user cascades `revoke-on-delete` on all grants where that user is subject
- Step-CA CRL endpoint exposed; serving gateway enforces CRL check on every handshake (cached CRL, refresh interval 60s)
- Client-side cert renewal job: at T-7 days, submit renewal CSR; rotate cert atomically; flush cache
- On renewal failure, peer marked `degraded` and admin-visible alert emitted
- Server A detects revocation on next request (TLS handshake fails with specific error) → peer marked `revoked`, user notified

**Deliverables:**

- `apps/gateway/src/federation/server/crl.service.ts` + endpoint
- `apps/gateway/src/federation/server/revocation.service.ts`
- DB cascade trigger or ORM hook for user deletion → grant revocation
- `apps/gateway/src/federation/client/renewal.job.ts` (scheduled)
- `packages/mosaic/src/commands/federation/grant.ts` gains `revoke` subcommand

**Acceptance tests:**
| # | Test | Layer |
| - | ----------------------------------------------------------------------------------------- | ----- |
| 1 | Admin `grant revoke` → A's next request fails with TLS-level error | E2E |
| 2 | Deleting subject user on B auto-revokes all grants where that user was the subject | Integration |
| 3 | CRL endpoint serves correct list; revoked cert present | Integration |
| 4 | Server rejects cert listed in CRL even if cert itself is still time-valid | E2E |
| 5 | Cert at T-7 days triggers renewal job; new cert issued and installed without dropped requests | E2E |
| 6 | Renewal failure marks peer `degraded` and surfaces alert | Integration |
| 7 | A marks peer `revoked` after a revocation-caused handshake failure (not on transient network errors) | E2E |

**Estimated budget:** ~20K tokens
**Risk notes:** The atomic cert swap during renewal is the sharpest edge here — any in-flight request mid-swap must either complete on old or retry on new, never fail mid-call.

---

## M7 — Multi-User RBAC Hardening + Team-Scoped Grants + Acceptance Suite

**Goal:** The full multi-tenant scenario from §4 user stories works end-to-end, with no cross-user leakage under any circumstance.

**Scope:**

- Three-user scenario on Server B (E1, E2, E3) each with their own Server A
- Team-scoped grants exercised: each employee's team-data visible on their own A, but E1's personal data never visible on E2's A
- User-facing UI surfaces on both gateways for: peer list, grant list, audit log viewer, scope editor
- Negative-path test matrix (every denial path from PRD §8)
- All PRD §15 acceptance criteria mapped to automated tests in the harness
- Security review: cert-spoofing, scope-bypass, audit-bypass paths explicitly tested
- Cold-storage rollover for audit log >90 days
- Docs: operator runbook, onboarding guide, troubleshooting guide

**Deliverables:**

- Full federation acceptance suite in `tools/federation-harness/acceptance/`
- `apps/web` surfaces for peer/grant/audit management
- `docs/federation/RUNBOOK.md`, `docs/federation/ONBOARDING.md`, `docs/federation/TROUBLESHOOTING.md`
- Audit cold-tier job (daily cron, moves rows >90d to separate table or object storage)

**Acceptance tests:**
Every PRD §15 criterion must be automated and green. Additionally:

| #   | Test                                                                                                  | Layer            |
| --- | ----------------------------------------------------------------------------------------------------- | ---------------- |
| 1   | 3-employee scenario: each A sees only its user's data from B                                          | E2E              |
| 2   | Grant with team scope returns team data; same grant denied access to another employee's personal data | E2E              |
| 3   | Concurrent sessions from E1's and E2's Server A to B interleave without any leakage                   | E2E              |
| 4   | Audit log across 3-user test shows per-grant trails with no mis-attributed rows                       | E2E              |
| 5   | Scope editor UI round-trip: edit → save → next request uses new scope                                 | E2E              |
| 6   | Attempt to use a revoked grant's cert against a different grant's endpoint: rejected                  | E2E              |
| 7   | 90-day-old audit rows moved to cold tier; queryable via explicit historical query                     | Integration      |
| 8   | Runbook steps validated: an operator following the runbook can onboard, rotate, and revoke            | Manual checklist |

**Estimated budget:** ~25K tokens
**Risk notes:** This is the security-critical milestone. Budget review time here is non-negotiable — plan for two independent code reviews (internal + security-focused) before merge.

---

## Total Budget & Timeline Sketch

| Milestone | Tokens (est.) | Can parallelize?       |
| --------- | ------------- | ---------------------- |
| M1        | 20K           | No (foundation)        |
| M2        | 30K           | No (needs M1)          |
| M3        | 40K           | No (needs M2)          |
| M4        | 20K           | No (needs M3)          |
| M5        | 20K           | Yes (with M6 after M4) |
| M6        | 20K           | Yes (with M5 after M3) |
| M7        | 25K           | No (needs all)         |
| **Total** | **~175K**     |                        |

Parallelization of M5 and M6 after M4 saves one milestone's worth of serial time.

---

## Exit Criteria (federation feature complete)

All of the following must be green on `main`:

- Every PRD §15 acceptance criterion automated and passing
- Every milestone's acceptance table green
- Security review sign-off on M7
- Runbook walk-through completed by operator (not author)
- `mosaic doctor` recognizes federated tier and reports peer health accurately
- Two-gateway production deployment (woltje.com ↔ uscllc.com) operational for ≥7 days without incident

---

## Next Step After This Doc Is Approved

1. File tracking issues on `git.mosaicstack.dev/mosaicstack/stack` — one per milestone, labeled `epic:federation`
2. Populate `docs/TASKS.md` with M1's task breakdown (per-task agent assignment, budget, dependencies)
3. Begin M1 implementation