Merge pull request 'docs(federation): close FED-M1 milestone' (#481) from feat/federation-m1-close into main

2026-04-20 02:20:43 +00:00
parent 78841f228a b9fb8aab57
commit 54c422ab06
3 changed files with 128 additions and 31 deletions
--- a/docs/federation/MISSION-MANIFEST.md
+++ b/docs/federation/MISSION-MANIFEST.md
@@ -7,11 +7,11 @@

 **ID:** federation-v1-20260419
 **Statement:** Jarvis operates across 3–4 workstations in two physical locations (home, USC). The user currently reaches back to a single jarvis-brain checkout from every session; a prior OpenBrain attempt caused cache, latency, and opacity pain. This mission builds asymmetric federation between Mosaic Stack gateways so that a session on a user's home gateway can query their work gateway in real time without data ever persisting across the boundary, with full multi-tenant isolation and standard-PKI (X.509 / Step-CA) trust management.
-**Phase:** Planning complete — M1 implementation not started
-**Current Milestone:** FED-M1
-**Progress:** 0 / 7 milestones
+**Phase:** M1 complete — federated tier infrastructure ready for testing
+**Current Milestone:** FED-M2 (next; deferred to mission planning)
+**Progress:** 1 / 7 milestones
 **Status:** active
-**Last Updated:** 2026-04-19 (PRD + MILESTONES + tracking issues filed)
+**Last Updated:** 2026-04-19 (M1 complete; tag `fed-v0.1.0-m1`)
 **Parent Mission:** None — new mission

 ## Context
@@ -52,8 +52,8 @@ Key design references:
 ## Milestones

 | #   | ID     | Name                                          | Status      | Branch             | Issue | Started    | Completed  |
-| --- | ------ | --------------------------------------------- | ----------- | ------ | ----- | ------- | --------- |
-| 1   | FED-M1 | Federated tier infrastructure                 | not-started | —      | #460  | —       | —         |
+| --- | ------ | --------------------------------------------- | ----------- | ------------------ | ----- | ---------- | ---------- |
+| 1   | FED-M1 | Federated tier infrastructure                 | done        | (12 PRs #470-#481) | #460  | 2026-04-19 | 2026-04-19 |
 | 2   | FED-M2 | Step-CA + grant schema + admin CLI            | not-started | —                  | #461  | —          | —          |
 | 3   | FED-M3 | mTLS handshake + list/get + scope enforcement | not-started | —                  | #462  | —          | —          |
 | 4   | FED-M4 | search verb + audit log + rate limit          | not-started | —                  | #463  | —          | —          |
@@ -77,9 +77,12 @@ Key design references:
 ## Session History

 | Session | Date       | Runtime | Outcome                                                               |
-| ------- | ---------- | ------- | --------------------------------------------------- |
+| ------- | ---------- | ------- | --------------------------------------------------------------------- |
 | S1      | 2026-04-19 | claude  | PRD authored, MILESTONES decomposed, 7 issues filed                   |
+| S2-S4   | 2026-04-19 | claude  | FED-M1 complete: 12 tasks (PRs #470-#481) merged; tag `fed-v0.1.0-m1` |

 ## Next Step

-Begin FED-M1 implementation: federated tier infrastructure. Breakdown in `docs/federation/TASKS.md`.
+FED-M1 complete (12 PRs #470-#481, tag `fed-v0.1.0-m1`). Federated tier infrastructure is testable end-to-end: see `docs/federation/SETUP.md` and `docs/guides/migrate-tier.md`.
+
+Begin FED-M2 (Step-CA + grant schema + admin CLI) when planning is greenlit. Issue #461 tracks scope; orchestrator decomposes M2 into per-task rows in `docs/federation/TASKS.md` at the start of M2.
--- a/docs/federation/TASKS.md
+++ b/docs/federation/TASKS.md
@@ -16,7 +16,7 @@
 Goal: Gateway runs in `federated` tier with containerized PG+pgvector+Valkey. No federation logic yet. Existing standalone behavior does not regress.

 | id        | status | description                                                                                                                                                                                                        | issue | agent  | branch                             | depends_on | estimate | notes                                                                                                                                            |
-| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----- | ------ | ---------------------------------- | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| --------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----- | ------ | ---------------------------------- | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
 | FED-M1-01 | done   | Extend `mosaic.config.json` schema: add `"federated"` to `tier` enum in validator + TS types. Keep `local` and `standalone` working. Update schema docs/README where referenced.                                   | #460  | sonnet | feat/federation-m1-tier-config     | —          | 4K       | Shipped in PR #470. Renamed `team` → `standalone`; added `team` deprecation alias; added `DEFAULT_FEDERATED_CONFIG`.                             |
 | FED-M1-02 | done   | Author `docker-compose.federated.yml` as an overlay profile: Postgres 17 + pgvector extension (port 5433), Valkey (6380), named volumes, healthchecks. Compose-up should boot cleanly on a clean machine.          | #460  | sonnet | feat/federation-m1-compose         | FED-M1-01  | 5K       | Shipped in PR #471. Overlay defines `postgres-federated`/`valkey-federated`, profile-gated, with pg-init for pgvector extension.                 |
 | FED-M1-03 | done   | Add pgvector support to `packages/storage/src/adapters/postgres.ts`: create extension on init (idempotent), expose vector column type in schema helpers. No adapter changes for non-federated tiers.               | #460  | sonnet | feat/federation-m1-pgvector        | FED-M1-02  | 8K       | Shipped in PR #472. `enableVector` flag on postgres StorageConfig; idempotent CREATE EXTENSION before migrations.                                |
@@ -28,7 +28,7 @@ Goal: Gateway runs in `federated` tier with containerized PG+pgvector+Valkey. No
 | FED-M1-09 | done   | Standalone regression: full agent-session E2E on existing `standalone` tier with a gateway built from this branch. Must pass without referencing any federation module.                                            | #460  | sonnet | feat/federation-m1-regression      | FED-M1-07  | 4K       | Clean canary. 351 gateway tests + 85 storage unit tests + full pnpm test all green; only FEDERATED_INTEGRATION-gated tests skip.                 |
 | FED-M1-10 | done   | Code review pass: security-focused on the migration script (data-at-rest during migration) + tier detector (error-message sensitivity leakage). Independent reviewer, not authors of tasks 01-09.                  | #460  | sonnet | feat/federation-m1-security-review | FED-M1-09  | 8K       | 2 review rounds caught 7 issues: credential leak in pg/valkey/pgvector errors + redact-error util; missing advisory lock; SKIP_TABLES rationale. |
 | FED-M1-11 | done   | Docs update: `docs/federation/` operator notes for tier setup; README blurb on federated tier; `docs/guides/` entry for migration. Do NOT touch runbook yet (deferred to FED-M7).                                  | #460  | haiku  | feat/federation-m1-docs            | FED-M1-10  | 4K       | Shipped: `docs/federation/SETUP.md` (119 lines), `docs/guides/migrate-tier.md` (147 lines), README Configuration blurb.                          |
-| FED-M1-12 | not-started | PR, CI green, merge to main, close #460.                                                                                                                                                                           | #460  | —      | (aggregate)                        | FED-M1-11  | 3K       | Queue-guard before push; wait for green; merge squashed; tea `issue-close` #460.                                                                 |
+| FED-M1-12 | done   | PR, CI green, merge to main, close #460.                                                                                                                                                                           | #460  | sonnet | feat/federation-m1-close           | FED-M1-11  | 3K       | M1 closed. PRs #470-#480 merged across 11 tasks. Issue #460 closed; release tag `fed-v0.1.0-m1` published.                                       |

 **M1 total estimate:** ~74K tokens (over-budget vs 20K PRD estimate — explanation below)

--- a/docs/scratchpads/mvp-20260312.md
+++ b/docs/scratchpads/mvp-20260312.md
@@ -429,3 +429,97 @@ Side change: `packages/storage/package.json` gained `"type": "module"` (codebase
 - #8: confirm `packages/config/dist` not git-tracked

 **Next:** FED-M1-09 — standalone regression e2e (haiku canary, ~4K). Verifies that the existing `standalone` tier behavior still works end-to-end on the federation-touched build, since M1 changes touched shared paths (storage, config, gateway boot).
+
+---
+
+## Session 19 — 2026-04-19 — FED-M1-09 → FED-M1-12 (M1 close)
+
+**Branches landed this run:** `feat/federation-m1-regression` (PR #478, M1-09), `feat/federation-m1-security-review` (PR #479, M1-10), `feat/federation-m1-docs` (PR #480, M1-11), `feat/federation-m1-close` (PR #481, M1-12)
+**Branch active at end:** none — M1 closed, all branches deleted, issue #460 closed, release tag `fed-v0.1.0-m1` published
+
+**M1 progress:** 12 of 12 tasks done. **Milestone complete.**
+
+### FED-M1-09 — Standalone regression canary
+
+Verification-only milestone. Re-ran the existing standalone/local test suites against current `main` (with M1-01 → M1-08 merged):
+
+- 4 target gateway test files: 148/148 pass (conversation-persistence, cross-user-isolation, resource-ownership, session-hardening)
+- Full gateway suite: 351 pass, 4 skipped (FEDERATED_INTEGRATION-gated only)
+- Storage unit tests: 85 pass, 1 skipped (integration-gated)
+- Top-level `pnpm test`: all green; only env-gated skips
+
+No regression in standalone or local tier. Federation M1 changes are non-disruptive.
+
+### FED-M1-10 — Security review (two rounds, 7 findings)
+
+Independent security review surfaced three high-impact and four medium findings; all fixed in same PR.
+
+**Round 1 (4 findings):**
+
+- MEDIUM: Credential leak via `postgres`/`ioredis` driver error messages (DSN strings) re-thrown by `migrate-tier.ts` → caller; `cli.ts:402` outer catch
+- MEDIUM: Same leak in `tier-detection.ts` `probePostgresMeasured` / `probePgvectorMeasured` → emitted as JSON by `mosaic gateway doctor --json`
+- LOW-MEDIUM: No advisory lock on `migrate-tier`; two concurrent invocations could both pass `checkTargetPreconditions` (non-atomic) and race
+- ADVISORY: `SKIP_TABLES` lacked rationale comment
+
+**Fixes:**
+
+- New internal helper `packages/storage/src/redact-error.ts` — regex `(postgres(?:ql)?|rediss?):\/\/[^@\s]*@` → `<scheme>://***@`. NOT exported from package public surface. 10 unit tests covering all schemes, multi-URL, no-creds, case-insensitive.
+- `redactErrMsg` applied at all 5 leak sites
+- `PostgresMigrationTarget.tryAcquireAdvisoryLock()` / `releaseAdvisoryLock()` using session-scoped `pg_try_advisory_lock(hashtext('mosaic-migrate-tier'))`. Acquired before preflight, released in `finally`. Dry-run skips. Non-blocking.
+- `SKIP_TABLES` comment expanded with rationale for skipped tables (TTL'd / one-time / env-bound) AND why `accounts` (OAuth) and `provider_credentials` (AI keys) are intentionally migrated (durable user-bound, not deployment-bound).
+
+**Round 2 (3 findings missed by first round):**
+
+- HIGH: Round 1 regex only covered `postgres` scheme, not `redis`/`rediss` — extended to `(postgres(?:ql)?|rediss?)`
+- HIGH: `probeValkeyMeasured` was missed in Round 1 → applied `redactErrMsg`
+- MEDIUM: `cli.ts:402` migrate-tier outer catch was missed in Round 1 → applied `redactErrMsg`
+
+**Process validation:** the two-round review pattern proved load-bearing for security work. A single review-then-fix cycle would have shipped the Valkey credential leak.
+
+### FED-M1-11 — Docs (haiku)
+
+- `docs/federation/SETUP.md` (119 lines): federated tier setup — what it is, prerequisites, docker compose start, mosaic.config.json snippet, doctor health check, troubleshooting
+- `docs/guides/migrate-tier.md` (147 lines): when to migrate, dry-run first, what migrates/skips with rationale, idempotency + advisory-lock semantics, no in-place rollback
+- `README.md` Configuration blurb linking to both
+- Runbook deferred to FED-M7 per TASKS.md scope rule
+
+### FED-M1-12 — Aggregate close (this PR)
+
+- Marked M1-12 done in TASKS.md
+- MISSION-MANIFEST.md: phase → "M1 complete", progress 1/7, M1 row done with PR range #470-#481, session log appended
+- This Session 19 entry added
+- Issue #460 closed via `~/.config/mosaic/tools/git/issue-close.sh -i 460`
+- Release tag `fed-v0.1.0-m1` created and pushed to gitea
+
+### M1 PR ledger
+
+| PR   | Task                                      | Branch                             |
+| ---- | ----------------------------------------- | ---------------------------------- |
+| #470 | M1-01 (tier config schema)                | feat/federation-m1-tier-config     |
+| #471 | M1-02 (compose overlay)                   | feat/federation-m1-compose         |
+| #472 | M1-03 (pgvector adapter)                  | feat/federation-m1-pgvector        |
+| #473 | M1-04 (tier-detector)                     | feat/federation-m1-detector        |
+| #474 | M1-05 (migrate-tier script)               | feat/federation-m1-migrate         |
+| #475 | M1-06 (gateway doctor)                    | feat/federation-m1-doctor          |
+| #476 | M1-07 (boot integration tests)            | feat/federation-m1-integration     |
+| #477 | M1-08 (migrate integration test + P0 fix) | feat/federation-m1-migrate-test    |
+| #478 | M1-09 (standalone regression)             | feat/federation-m1-regression      |
+| #479 | M1-10 (security review fixes)             | feat/federation-m1-security-review |
+| #480 | M1-11 (docs)                              | feat/federation-m1-docs            |
+| #481 | M1-12 (aggregate close)                   | feat/federation-m1-close           |
+
+### Process learnings (M1 retrospective)
+
+1. **Two-round security review is non-negotiable for security work.** First round caught postgres credential leaks; second round caught equivalent valkey leaks the worker missed when extending the regex. Single-round would have shipped HIGH severity issues.
+2. **Real-services integration tests catch what mocked unit tests cannot.** M1-08 caught a P0 in M1-05 (camelCase column names) that 32 mocked unit tests missed because both source and target were mocked. Going forward: at least one real-services test per code-mutating PR where feasible.
+3. **Test-utils for live services co-locate with consumer, not in shared library.** M1-08 reviewer caught `createPgliteDbWithVector` initially being added to `@mosaicstack/db` public exports — would have polluted prod consumers with WASM bundle. Moved to `packages/storage/src/test-utils/`.
+4. **Per-task budgets including tests/review/docs more accurate than PRD's implementation-only estimates.** M1 PRD estimated 20K; actual ~74K. Future milestones should budget the full delivery cycle.
+5. **TASKS.md status updates ride feature branches, never direct-to-main.** Caught one violation early in M1; pattern held for all 12 tasks.
+6. **Subagent tier matters.** Code review needs sonnet-level reasoning (haiku missed deep issues in M1-04); claim verification (line counts, file existence) is fine on haiku.
+
+**Followup tasks still deferred (carry forward to M2):**
+
+- #7: `tier=local` hardcoded in gateway-config resume branches (~262, ~317)
+- #8: confirm `packages/config/dist` not git-tracked
+
+**Next mission step:** FED-M2 (Step-CA + grant schema + admin CLI). Per TASKS.md scope rule, M2 will be decomposed when it enters active planning. Issue #461 tracks scope.