---
kind: record
status: active
---

# Mission Scratchpad — MVP

> Append-only log. NEVER delete entries. NEVER overwrite sections.
> This is the orchestrator's working memory across sessions.

## Original Mission Prompt

```
Active mission detected: MVP. Read the mission state files and report status.
User confirmed: start the planning gate.
```

## Planning Decisions

### 2026-03-13 — Milestone and task breakdown

- PRD defines 8 phases (Phase 0–7), mapped 1:1 to Gitea milestones
- 59 issues created on git.mosaicstack.dev/mosaic/mosaic-stack (#1–#59)
- Each phase has a verification task as the final issue
- Task IDs use P{phase}-{seq} format (P0-001 through P7-008)
- Repo created as `mosaic/mosaic-stack` (private) on Gitea
- Milestones: ms-157 (Phase 0) through ms-164 (Phase 7)
- Total: 59 tasks across 8 milestones

### Phase structure

| Phase | Version | Tasks | Focus                                                      |
| ----- | ------- | ----- | ---------------------------------------------------------- |
| 0     | v0.0.1  | 9     | Foundation — monorepo, types, db, auth, OTEL, Docker, CI   |
| 1     | v0.0.2  | 9     | Core API — gateway, brain, queue, routes, WebSocket        |
| 2     | v0.0.3  | 7     | Agent Layer — Pi SDK, multi-provider, routing, coord       |
| 3     | v0.0.4  | 8     | Web Dashboard — Next.js, chat, tasks, projects, admin      |
| 4     | v0.0.5  | 7     | Memory & Intelligence — memory, log, summarization, skills |
| 5     | v0.0.6  | 5     | Remote Control — Discord, Telegram, SSO                    |
| 6     | v0.0.7  | 6     | CLI & Tools — CLI, prdy, quality-rails, installer, TUI     |
| 7     | v0.1.0  | 8     | Polish & Beta — MCP, providers, E2E, docs, release         |

## Session Log

| Session | Date       | Milestone      | Tasks Done                                     | Outcome                                                                                                                                                                                          |
| ------- | ---------- | -------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1       | 2026-03-13 | Planning       | Planning gate                                  | Milestones created, 59 issues created, TASKS.md populated, manifest updated                                                                                                                      |
| 2       | 2026-03-13 | Vertical slice | P1-001, P1-007, P1-008, P2-001, P5-002, P6-005 | Communication spine built and merged (PR #61). Gateway + TUI + Discord. 3-agent gatekeeper review, 10/16 issues remediated, 4 deferred.                                                          |
| 3       | 2026-03-13 | Foundation     | P0-002, P0-005, P0-006                         | Foundation layer merged (PR #65). Docker Compose (PG+pgvector, Valkey, OTEL Collector, Jaeger), OTEL auto-instrumentation in gateway, @mosaicstack/types with DTOs + Socket.IO typed event maps. |

### Session 4 — Docker Compose fix

| Session | Date       | Milestone  | Tasks Done | Outcome                                                                                                          |
| ------- | ---------- | ---------- | ---------- | ---------------------------------------------------------------------------------------------------------------- |
| 4       | 2026-03-12 | Foundation | (fix)      | Fixed Jaeger tag (2→2.6.0), remapped PG/Valkey ports (5433/6380) to avoid host conflicts. PR #66 merged to main. |

**Verification evidence:**

- All 4 containers healthy (PG, Valkey, OTEL Collector, Jaeger)
- OTEL pipeline proven: `mosaic-gateway` service visible in Jaeger UI
- Gateway traces flow through Collector → Jaeger

### Session 5 — Phase 0-1 completion

| Session | Date       | Milestone        | Tasks Done                                                    | Outcome                                                                                       |
| ------- | ---------- | ---------------- | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| 5       | 2026-03-12 | Phase 0, Phase 1 | P0-003, P0-004, P0-007, P0-008, P0-009, P1-002–P1-006, P1-009 | Foundation + Core API complete. DB, auth, CI, brain, queue, CRUD routes all merged and green. |

### Session 6 — Phase 2 agent layer

| Session | Date       | Milestone | Tasks Done                                     | Outcome                                                                                                     |
| ------- | ---------- | --------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| 6       | 2026-03-12 | Phase 2   | P2-002, P2-003, P2-004, P2-005, P2-006, FIX-01 | Multi-provider routing, tool registration, coord migration, session management, dispose() fix. PRs #74–#78. |

### Session 7-8 — Phase 2 verification + completion

| Session | Date       | Milestone | Tasks Done | Outcome                                                                             |
| ------- | ---------- | --------- | ---------- | ----------------------------------------------------------------------------------- |
| 7-8     | 2026-03-12 | Phase 2   | P2-007     | 19 unit tests (routing + coord). PR #79 merged, issue #25 closed. Phase 2 complete. |

### Session 11 — Phase 5 completion

| Session | Date       | Milestone | Tasks Done | Outcome                                                                                                                                         |
| ------- | ---------- | --------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| 11      | 2026-03-14 | Phase 5   | P5-005     | Wired Telegram plugin into gateway (was stubbed). Updated .env.example with all P5 env vars. PR #99 merged, issue #45 closed. Phase 5 complete. |

**Findings during verification:**

- Telegram plugin was built but not wired into gateway (stub warning in plugin.module.ts)
- Discord plugin was fully wired
- SSO/Authentik OIDC adapter was fully wired
- All three quality gates passing

### Session 11 (continued) — Phase 6 completion

| Session | Date       | Milestone | Tasks Done                             | Outcome                                                                                                          |
| ------- | ---------- | --------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| 11      | 2026-03-14 | Phase 6   | P6-002, P6-003, P6-004, P6-001, P6-006 | Full CLI & Tools migration. PRs #100-#104 merged. Also fixed 2 gateway startup bugs (PR #102). Phase 6 complete. |

**Phase 6 details:**

- P6-002: @mosaicstack/prdy migrated from v0 (~400 LOC). PR #101.
- P6-003: @mosaicstack/quality-rails migrated from v0 (~500 LOC). PR #100.
- P6-004: @mosaicstack/mosaic wizard migrated from v0 (2272 LOC, 28 files). PR #103.
- P6-001: CLI subcommands wired — tui, prdy, quality-rails, wizard all working. PR #104.
- BUG-1: PLUGIN_REGISTRY circular import fixed via plugin.tokens.ts. PR #102.
- BUG-2: AuthStorage.create() → .inMemory() to prevent silent exit. PR #102.

### Session 11 (continued) — E2E testing + bug fixes + Phase 7 rescope

**Bug fixes merged during E2E testing (PRs #107-#117):**

- CI: from_secret syntax for Woodpecker v2 (#107)
- Gateway: dotenv loading from monorepo root (#108)
- Gateway: missing @Inject() decorators causing silent hang (#109)
- Gateway: CORS + memory userId + pgvector auto-init (#110)
- Auth: BetterAuth trustedOrigins for web dashboard (#111)
- Auth: CORS headers on raw BetterAuth HTTP handler (#112)
- Husky: removed deprecated v9 shim lines (#113)
- CLI: login command + authenticated TUI sessions (#114)
- CLI: Origin header on auth requests (#115)
- Agent: Ollama provider registration with openai-completions API (#116, #117)

**E2E testing results:**

- Web UI: login works, projects list, chats list (but chat doesn't function)
- TUI: authenticated connection works, agent responds via Ollama llama3.2
- Agent tools: brain, coord, memory tools confirmed working
- Gateway: all routes mapped, providers register correctly

**Phase 7 rescoped (Jason directed):**

- Phase 7 renamed from "Polish & Beta" to "Feature Completion (v0.0.8)"
- Added 13 new tasks (P7-009 through P7-021): web UI, agent tools, CLI, coord architecture
- P7-002 (extra SSO), P7-003 (extra LLM), P7-005 (perf), P7-008 (v0.1.0 tag) moved to Phase 8
- Phase 8 added as "Polish & Beta (v0.1.0)"
- Reason: platform isn't feature-complete enough for beta — web UI is scaffolded but non-functional for real use, agent tooling is minimal, CLI needs model switching

## Open Questions

(none at this time)

## Corrections

### 2026-03-13 — Vertical slice reorder (Jason directed)

**Original plan:** Linear Phase 0 → 1 → 2 → ... execution.

**Correction:** Vertical slice first. Scaffold monorepo, then build the Pi TUI → Gateway → Discord communication spine end-to-end before backfilling auth, brain, memory, CRUD, etc.

**Why:** Validate the architecture's core message flow before investing in horizontal layers. If the communication channels don't work, nothing else matters.

**Revised execution sequence:**

| Step | Tasks (cross-phase)                                                  | What it proves           |
| ---- | -------------------------------------------------------------------- | ------------------------ |
| 1    | P0-001: Scaffold monorepo                                            | Build system works       |
| 2    | P0-005: Docker Compose (PG + Valkey)                                 | Infrastructure runs      |
| 3    | P0-002: @mosaicstack/types (minimal — gateway, agent, chat types)    | Shared contracts         |
| 4    | P1-001: Gateway scaffold (minimal NestJS + Fastify)                  | API surface boots        |
| 5    | P1-007: WebSocket server (chat streaming)                            | Real-time channel works  |
| 6    | P1-008: Basic agent dispatch (single provider)                       | LLM responds             |
| 7    | P2-001: @mosaicstack/agent — Pi SDK integration (minimal)            | Pi sessions work         |
| 8    | P6-005: Pi TUI integration (mosaic tui → gateway)                    | TUI ↔ Gateway proven     |
| 9    | P5-001: Plugin host (channel plugin interface)                       | Plugin arch works        |
| 10   | P5-002: Discord plugin (bot + channel)                               | Discord ↔ Gateway proven |
| —    | Then backfill: auth, brain, db, queue, OTEL, CI, web dashboard, etc. |

### Session 9 — Phase 3 Web Dashboard (P3-001 through P3-007)

| Session | Date       | Milestone | Tasks Done            | Outcome                                                                                                                           |
| ------- | ---------- | --------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| 9       | 2026-03-12 | Phase 3   | P3-001 through P3-007 | Full web dashboard: Next.js 16 scaffold, auth pages, chat UI, tasks (list+kanban), projects, settings, admin. PRs #82-#89 merged. |

### Session 10 — Phase 3 verification (P3-008)

| Session | Date       | Milestone | Tasks Done | Outcome                                                                                                                    |
| ------- | ---------- | --------- | ---------- | -------------------------------------------------------------------------------------------------------------------------- |
| 10      | 2026-03-13 | Phase 3   | P3-008     | Phase 3 verification: typecheck 18/18, lint 18/18, format clean, build green (10 routes), 10 tests pass. Phase 3 complete. |

### Session 10 (continued) — Phase 4 Memory & Intelligence

| Session | Date       | Milestone | Tasks Done            | Outcome                                                                                                                                                                                                                                                                                      |
| ------- | ---------- | --------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 10      | 2026-03-13 | Phase 4   | P4-001 through P4-007 | Full memory + log system: DB schema (preferences, insights w/ pgvector, agent_logs, skills, summarization_jobs), @mosaicstack/memory + @mosaicstack/log packages, embedding service, summarization pipeline w/ cron, memory tools in agent sessions, skill management CRUD. All gates green. |

### Session 12 — Phase 7 planning + execution start

| Session | Date       | Milestone | Tasks Done | Outcome                                                                                                                                                           |
| ------- | ---------- | --------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 12      | 2026-03-15 | Phase 7   | Planning   | Merged rescope PR #119. Created 15 Gitea issues (#120-#134) for P7-009 through P7-021 + FIX-02/FIX-03. Planned 10-wave execution order with 2-worker parallelism. |

**Phase 7 execution plan (10 waves, max 2 parallel workers):**

| Wave | Task A                          | Task B                                       |
| ---- | ------------------------------- | -------------------------------------------- |
| 1    | P7-009 Web chat WS (#120)       | P7-001 MCP hardening (#52)                   |
| 2    | P7-010 Conversation mgmt (#121) | P7-015 Agent tools (#126)                    |
| 3    | P7-011 Project views (#122)     | P7-016 MCP client (#127)                     |
| 4    | P7-012 Provider UI (#123)       | P7-017 Skill invocation (#128)               |
| 5    | P7-013 Settings persist (#124)  | P7-018 CLI model switch (#129)               |
| 6    | P7-014 Admin panel (#125)       | P7-019 CLI sessions (#130)                   |
| 7    | P7-020 Coord DB (#131)          | —                                            |
| 8    | FIX-02 TUI state (#133)         | FIX-03 Agent sandbox (#134)                  |
| 9    | P7-004 E2E Playwright (#55)     | P7-006 Docs (#57) + P7-007 Deploy docs (#58) |
| 10   | P7-021 Verify Phase 7 (#132)    | —                                            |

### Session 12 — Phase 7 completion summary

**All 17 Phase 7 tasks + 2 backlog fixes completed in a single session.**

PRs merged: #136, #137, #138, #139, #140, #141, #142, #143, #144, #145, #146, #147, #148, #149, #150, #151, #152, #153
Issues closed: #52, #55, #57, #58, #120-#134

**Verification evidence:**

- Typecheck: 32/32 tasks green
- Lint: 18/18 packages green
- Format: All files clean
- 19 PRs squash-merged to main, all quality gates passed

**Phase 7 delivered:**

- Web: functional chat (WS streaming), conversation management, project detail views, provider UI, settings persistence, admin panel
- Agent: 7 new tools (file/git/shell/web), MCP server (14 tools), MCP client (external server bridge), skill invocation
- CLI: model/provider switching, session management
- Infrastructure: coord DB migration, agent sandbox hardening
- Quality: E2E Playwright suite (~35 tests), comprehensive docs (user/admin/dev/deployment)
- Fixes: TUI state updater, agent session sandboxing

### Session 13 — CLI Command Architecture (P8-005, P8-006)

| Session | Date       | Milestone | Tasks Done     | Outcome                                                                                                       |
| ------- | ---------- | --------- | -------------- | ------------------------------------------------------------------------------------------------------------- |
| 13      | 2026-03-15 | Phase 8   | P8-005, P8-006 | CLI command architecture implemented. DB schema, brain repo, gateway endpoints, CLI commands. PR #158 merged. |

**Changes delivered:**

- DB: Extended agents table (projectId, ownerId, systemPrompt, allowedTools, skills, isSystem). Added agentId to conversations.
- Brain: New agents repository with findAccessible (owner's + system agents).
- Gateway: /api/agents CRUD, consolidated /api/missions with user-scoped CRUD + /tasks sub-routes, coord slimmed to file-based only, agentConfigId wired into session creation.
- CLI: `mosaic agent` (--list, --new, --show, --update, --delete), `mosaic mission` (--list, --init, --plan, --update, task subcommand), `mosaic prdy` (gateway-aware), shared with-auth + select-dialog utilities.
- TUI: --agent and --project flags, agent name display in top bar, agentId in socket payload.
- Types: agentId added to ChatMessagePayload.
- Tests: 23/23 gateway tests pass (updated ownership test for user-scoped missions).

### Session 14 — Platform Architecture Plan Augmentation + Task Breakdown

| Session | Date       | Milestone | Tasks Done | Outcome                                                       |
| ------- | ---------- | --------- | ---------- | ------------------------------------------------------------- |
| 14      | 2026-03-15 | Phase 8   | P8-018     | Augmented plan, created 13 issues, created Phase 8 milestone. |

**Decisions made:**

- This plan is Phase 7 feature extension work, not Phase 8 beta scope. P8-001–P8-004 (SSO, LLM, perf, release gate) are deferred to far future.
- `/provider` OAuth in TUI: URL-to-clipboard + Valkey poll token pattern (same as Pi agent)
- Add `mutable` column to preferences now (P8-007 DB migration)
- Teams architecture: `teams` + `team_members` tables, `teamId`/`ownerType` on projects. Workspace path branches on owner type: `users/<uid>/` vs `teams/<tid>/`.
- Phase dependency chain decided: Wave 1 (DB+Types) → Wave 2 (TUI+toolhardening) → Wave 3 (gateway registry, gating) → Wave 4 (prefs+commands) → Wave 5 (reload+GC) → Wave 6 (workspaces) → Wave 7 (autocomplete) → Wave 8 (verify).

**Plan augmentations added:**

- Teams Architecture section (DB schema, workspace paths, RBAC)
- REST Route Specifications table
- `/provider` OAuth flow (URL+clipboard+polling)
- Preferences `mutable` migration spec
- Test Strategy (per-task test files + key test cases)
- Phase Execution Order (dependency graph + wave plan)

**Issues created:** #160–#172 (Gitea milestone ms-165)
**P8-018 closed:** Spin-off stubs created (gatekeeper-service.md, task-queue-unification.md, chroot-sandboxing.md)

**Next:** Begin execution at Wave 1 — P8-007 (DB migrations) + P8-008 (Types) in parallel.

---

### Session 15 — 2026-04-19 — MVP Rollup Manifest Authored

| Session | Date       | Milestone      | Tasks Done                                                                | Outcome                                                                                                                                                                |
| ------- | ---------- | -------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 15      | 2026-04-19 | (rollup-level) | MVP-T01 (manifest), MVP-T02 (archive iuv-v2), MVP-T03 (land FED planning) | Authored MVP rollup manifest at `docs/MISSION-MANIFEST.md`. Federation v1 planning merged to `main` (PR #468 / commit `66512550`). Install-ux-v2 archived as complete. |

**Gap context:** The MVP scratchpad was last updated at Session 14 (2026-03-15). In the intervening month, two sub-missions ran outside the MVP framework: `install-ux-hardening` (complete, `mosaic-v0.0.25`) and `install-ux-v2` (complete on 2026-04-19, `0.0.27` → `0.0.29`). Both archived under `docs/archive/missions/`. The phase-based execution from Sessions 1–14 (Phases 0–8, issues #1–#172) substantially shipped during this window via those sub-missions and standalone PRs — the MVP mission was nominally active but had no rollup manifest tracking it.

**User reframe (this session):**

> There will be more in the MVP. This will inevitably become scope creep. I need a solution that works via webUI, TUI, CLI, and just works for MVP. Federation is required because I need it to work NOW, so my disparate jarvis-brain usage can be consolidated properly.

**Decisions:**

1. **MVP is the rollup mission**, not a single-purpose mission. Federation v1 is one workstream of MVP, not MVP itself. Phase 0–8 work is preserved as historical context but is no longer the primary control plane.
2. **Three-surface parity (webUI / TUI / CLI) is a cross-cutting MVP requirement** (MVP-X1), not a workstream. Encoded explicitly so it can't be silently dropped.
3. **Scope creep is named and accommodated.** Manifest has explicit "Likely Additional Workstreams" section listing PRD-derived candidates without committing execution capacity to them.
4. **Workstream isolation** — each workstream gets its own manifest under `docs/{workstream}/MISSION-MANIFEST.md`. MVP manifest is rollup only.
5. **Archive-don't-delete** — install-ux-v2 manifest moved to `docs/archive/missions/install-ux-v2-20260405/` with status corrected to `complete` (IUV-M03 closeout note added pointing at PR #446 + releases 0.0.27 → 0.0.29).
6. **Federation planning landed first** — PR #468 merged before MVP manifest authored, so the manifest references real on-`main` artifacts.

**Open items:**

- `.mosaic/orchestrator/mission.json` MVP slot remains empty (zero milestones). Tracked as MVP-T04. Defer until next session — does not block W1 kickoff. Open question: hand-edit vs. `mosaic coord init` reinit.
- Additional workstreams (web dashboard parity, TUI/CLI completion, remote control, multi-user/SSO, LLM provider expansion, MCP, brain) anticipated per PRD but not declared. Pre-staged in manifest's "Likely Additional Workstreams" list.

**Artifacts this session:**

| Artifact                                                                               | Status                                                                                     |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| PR #468 (`docs(federation): PRD, milestones, mission manifest, and M1 task breakdown`) | merged 2026-04-19 → `main` (commit `66512550`)                                             |
| `docs/MISSION-MANIFEST.md` (MVP rollup, replaces install-ux-v2 manifest)               | authored on `docs/mvp-mission-manifest` branch                                             |
| `docs/TASKS.md` (MVP rollup, points at workstream task files)                          | authored                                                                                   |
| Install-ux-v2 manifest + tasks + scratchpad + iuv-m03-design                           | moved to `docs/archive/missions/install-ux-v2-20260405/` with status corrected to complete |

**Next:** PR `docs/mvp-mission-manifest` → merge to `main` → next session begins W1 / FED-M1 from clean state.

---

## Session 16 — 2026-04-19 — claude

**Mode:** Delivery (W1 / FED-M1 execution)
**Branch:** `feat/federation-m1-tier-config`
**Context budget:** 200K, currently ~45% used (compaction-aware)

**Goal:** FED-M1-01 — extend `mosaic.config.json` schema: add `"federated"` to tier enum.

**Critical reconciliation surfaced during pre-flight:**

The federation PRD (`docs/federation/PRD.md` line 247) defines three tiers: `local | standalone | federated`.
The existing code (`packages/config/src/mosaic-config.ts`, `packages/mosaic/src/types.ts`, `packages/mosaic/src/stages/gateway-config.ts`) uses `local | team`.

`team` is the same conceptual tier as PRD `standalone` (Postgres + Valkey, no pgvector). Rather than carrying a confusing alias forever, FED-M1-01 will rename `team` → `standalone` and add `federated` as a third value, so all downstream federation work has a coherent vocabulary.

Affected files (storage-tier semantics only — Team/workspace usages unaffected):

- `packages/config/src/mosaic-config.ts` (StorageTier type, validator enum, defaults)
- `packages/mosaic/src/types.ts` (GatewayStorageTier)
- `packages/mosaic/src/stages/gateway-config.ts` (~10 references)
- `packages/mosaic/src/stages/gateway-config.spec.ts` (test references)
- Possibly `tools/e2e-install-test.sh` (referenced grep) and headless env hint string

**Worker plan:**

1. Spawn sonnet subagent with explicit task spec + the reconciliation context above.
2. Worker delivers diff; orchestrator runs `pnpm typecheck && pnpm lint && pnpm format:check`.
3. Independent `feature-dev:code-reviewer` subagent reviews diff.
4. Second independent verification subagent (general-purpose, sonnet) verifies reviewer's claims and confirms all `'team'` storage-tier references migrated, no `Team`/workspace bleed.
5. Open PR via tea CLI; wait for CI; queue-guard; squash merge; record actuals.

**Open items:**

- `MVP-T04` (sync `.mosaic/orchestrator/mission.json`) still deferred.
- `team` tier rename touches install wizard headless env vars (`MOSAIC_STORAGE_TIER=team`); will need 0.0.x deprecation note in scratchpad if release notes are written this milestone.

---

## Session 17 — 2026-04-19 — claude

**Mode:** Delivery (W1 / FED-M1 execution; resumed after compaction)
**Branches landed this run:** `feat/federation-m1-tier-config` (PR #470), `feat/federation-m1-compose` (PR #471), `feat/federation-m1-pgvector` (PR #472)
**Branch active at end:** `feat/federation-m1-detector` (FED-M1-04, ready to push)

**Tasks closed:** FED-M1-01, FED-M1-02, FED-M1-03 (all merged to `main` via squash, CI green, issue #460 still open as milestone).

**FED-M1-04 — tier-detector:** Worker delivered `apps/gateway/src/bootstrap/tier-detector.ts` (~210 lines) + `tier-detector.spec.ts` (12 tests). Independent code review (sonnet) returned `changes-required` with 3 issues:

1. CRITICAL: `probeValkey` missing `connectTimeout: 5000` on the ioredis Redis client (defaulted to 10s, violated fail-fast spec).
2. IMPORTANT: `probePgvector` catch block did not discriminate "library not installed" (use `pgvector/pgvector:pg17`) from permission errors.
3. IMPORTANT: Federated tier silently skipped Valkey probe when `queue.type !== 'bullmq'` (computed Valkey URL conditionally).

Worker fix-up round addressed all three:

- L147: `connectTimeout: 5000` added to Redis options
- L113-117: catch block branches on `extension "vector" is not available` substring → distinct remediation per failure mode
- L206-215: federated branch fails fast with `service: 'config'` if `queue.type !== 'bullmq'`, then probes Valkey unconditionally
- 4 new tests (8 → 12 total) cover each fix specifically

Independent verifier (haiku) confirmed all 6 verification claims (line numbers, test presence, suite green: 12/12 PASS).

**Process note — review pipeline working as designed:**

Initial verifier (haiku) on the first delivery returned "OK to ship" but missed the 3 deeper issues that the sonnet code-reviewer caught. This validates the user's "always verify subagent claims independently with another subagent" rule — but specifically with the **right tier** for the task: code review needs sonnet-level reasoning, while haiku is fine for verifying surface claims (line counts, file existence) once review issues are known. Going forward: code review uses sonnet (`feature-dev:code-reviewer`), claim verification uses haiku.

**Followup tasks tracked but deferred:**

- #7: `tier=local` hardcoded in gateway-config resume branches (~262, ~317) — pre-existing bug, fix during M1-06 (doctor) or M1-09 (regression).
- #8: confirm `packages/config/dist` not git-tracked.

**Next:** PR for FED-M1-04 → CI wait → merge. Then FED-M1-05 (migration script, codex/sonnet, 10K).

---

## Session 18 — 2026-04-19 — FED-M1-07 + FED-M1-08

**Branches landed this run:** `feat/federation-m1-integration` (PR #476, FED-M1-07), `feat/federation-m1-migrate-test` (PR #477, FED-M1-08)
**Branch active at end:** none — both PRs merged to main, branches deleted

**M1 progress:** 8 of 12 tasks done. Remaining: M1-09 (regression e2e, haiku), M1-10 (security review, sonnet), M1-11 (docs, haiku), M1-12 (close + release, orchestrator).

### FED-M1-07 — Integration tests for federated tier gateway boot

Three test files under `apps/gateway/src/__tests__/integration/` gated by `FEDERATED_INTEGRATION=1`:

- `federated-boot.success.integration.test.ts` — `detectAndAssertTier` resolves; `pg_extension` row for `vector` exists
- `federated-boot.pg-unreachable.integration.test.ts` — throws `TierDetectionError` with `service: 'postgres'` when PG port is closed
- `federated-pgvector.integration.test.ts` — TEMP table with `vector(3)` column round-trips data

Independent code review (sonnet) returned VERDICT: B with two IMPORTANT items, both fixed in the same PR:

- Port 5499 collision risk → replaced with `net.createServer().listen(0)` reserved-port helper
- `afterAll` and `sql` scoped outside `describe` → moved both inside `describe.skipIf` block

Independent surface verifier (haiku) confirmed all claims. 4/4 tests pass live; 4/4 skip cleanly without env var.

### FED-M1-08 — Migration integration test (caught real P0 bug)

`packages/storage/src/migrate-tier.integration.test.ts` seeds temp PGlite with cross-table data (users, teams, team_members, conversations, messages), runs `runMigrateTier`, asserts row counts + spot-checks. Gated by `FEDERATED_INTEGRATION=1`.

**P0 bug surfaced and fixed in same PR:** `DrizzleMigrationSource.readTable()` returns Drizzle's camelCase keys (`emailVerified`, `userId`); `PostgresMigrationTarget.upsertBatch()` was using them verbatim as SQL identifiers, producing `column "emailVerified" does not exist` against real federated PG. The 32 unit tests in M1-05 missed this because both source and target were mocked. Fix: `normaliseSourceRow` now applies `toSnakeCase` (`/[A-Z]/g` → `_<lowercase>`), idempotent on already-snake_case keys.

Code review (sonnet) returned VERDICT: B with one IMPORTANT and one MINOR, both fixed:

- `createPgliteDbWithVector` and `runPgliteMigrations` were initially added to `@mosaicstack/db` public exports → moved to `packages/storage/src/test-utils/pglite-with-vector.ts` (avoids polluting prod consumers with WASM bundle)
- `afterAll` did not call `cleanTarget` → added before connection close, ensuring orphan rows cleaned even on test panic

Side change: `packages/storage/package.json` gained `"type": "module"` (codebase convention; required for `import.meta.url` in test-utils). All other workspace packages already declared this.

### Process notes for this session

- Review-then-verify pipeline now battle-tested: M1-08 reviewer caught the P0 bug + the public-API leak that the worker would have shipped. Without review, both would have gone to main.
- Integration tests are paying for themselves immediately: M1-08 caught a real P0 in M1-05 that 32 mocked unit tests missed. Going forward, **at least one real-services integration test per code-mutating PR** should become a soft norm where feasible.
- TASKS.md status updates continue to ride on the matching feature branch (avoids direct-to-main commits).

**Followup tasks tracked but still deferred (no change):**

- #7: `tier=local` hardcoded in gateway-config resume branches (~262, ~317)
- #8: confirm `packages/config/dist` not git-tracked

**Next:** FED-M1-09 — standalone regression e2e (haiku canary, ~4K). Verifies that the existing `standalone` tier behavior still works end-to-end on the federation-touched build, since M1 changes touched shared paths (storage, config, gateway boot).

---

## Session 19 — 2026-04-19 — FED-M1-09 → FED-M1-12 (M1 close)

**Branches landed this run:** `feat/federation-m1-regression` (PR #478, M1-09), `feat/federation-m1-security-review` (PR #479, M1-10), `feat/federation-m1-docs` (PR #480, M1-11), `feat/federation-m1-close` (PR #481, M1-12)
**Branch active at end:** none — M1 closed, all branches deleted, issue #460 closed, release tag `fed-v0.1.0-m1` published

**M1 progress:** 12 of 12 tasks done. **Milestone complete.**

### FED-M1-09 — Standalone regression canary

Verification-only milestone. Re-ran the existing standalone/local test suites against current `main` (with M1-01 → M1-08 merged):

- 4 target gateway test files: 148/148 pass (conversation-persistence, cross-user-isolation, resource-ownership, session-hardening)
- Full gateway suite: 351 pass, 4 skipped (FEDERATED_INTEGRATION-gated only)
- Storage unit tests: 85 pass, 1 skipped (integration-gated)
- Top-level `pnpm test`: all green; only env-gated skips

No regression in standalone or local tier. Federation M1 changes are non-disruptive.

### FED-M1-10 — Security review (two rounds, 7 findings)

Independent security review surfaced three high-impact and four medium findings; all fixed in same PR.

**Round 1 (4 findings):**

- MEDIUM: Credential leak via `postgres`/`ioredis` driver error messages (DSN strings) re-thrown by `migrate-tier.ts` → caller; `cli.ts:402` outer catch
- MEDIUM: Same leak in `tier-detection.ts` `probePostgresMeasured` / `probePgvectorMeasured` → emitted as JSON by `mosaic gateway doctor --json`
- LOW-MEDIUM: No advisory lock on `migrate-tier`; two concurrent invocations could both pass `checkTargetPreconditions` (non-atomic) and race
- ADVISORY: `SKIP_TABLES` lacked rationale comment

**Fixes:**

- New internal helper `packages/storage/src/redact-error.ts` — regex `(postgres(?:ql)?|rediss?):\/\/[^@\s]*@` → `<scheme>://***@`. NOT exported from package public surface. 10 unit tests covering all schemes, multi-URL, no-creds, case-insensitive.
- `redactErrMsg` applied at all 5 leak sites
- `PostgresMigrationTarget.tryAcquireAdvisoryLock()` / `releaseAdvisoryLock()` using session-scoped `pg_try_advisory_lock(hashtext('mosaic-migrate-tier'))`. Acquired before preflight, released in `finally`. Dry-run skips. Non-blocking.
- `SKIP_TABLES` comment expanded with rationale for skipped tables (TTL'd / one-time / env-bound) AND why `accounts` (OAuth) and `provider_credentials` (AI keys) are intentionally migrated (durable user-bound, not deployment-bound).

**Round 2 (3 findings missed by first round):**

- HIGH: Round 1 regex only covered `postgres` scheme, not `redis`/`rediss` — extended to `(postgres(?:ql)?|rediss?)`
- HIGH: `probeValkeyMeasured` was missed in Round 1 → applied `redactErrMsg`
- MEDIUM: `cli.ts:402` migrate-tier outer catch was missed in Round 1 → applied `redactErrMsg`

**Process validation:** the two-round review pattern proved load-bearing for security work. A single review-then-fix cycle would have shipped the Valkey credential leak.

### FED-M1-11 — Docs (haiku)

- `docs/federation/SETUP.md` (119 lines): federated tier setup — what it is, prerequisites, docker compose start, mosaic.config.json snippet, doctor health check, troubleshooting
- `docs/guides/migrate-tier.md` (147 lines): when to migrate, dry-run first, what migrates/skips with rationale, idempotency + advisory-lock semantics, no in-place rollback
- `README.md` Configuration blurb linking to both
- Runbook deferred to FED-M7 per TASKS.md scope rule

### FED-M1-12 — Aggregate close (this PR)

- Marked M1-12 done in TASKS.md
- MISSION-MANIFEST.md: phase → "M1 complete", progress 1/7, M1 row done with PR range #470-#481, session log appended
- This Session 19 entry added
- Issue #460 closed via `~/.config/mosaic/tools/git/issue-close.sh -i 460`
- Release tag `fed-v0.1.0-m1` created and pushed to gitea

### M1 PR ledger

| PR   | Task                                      | Branch                             |
| ---- | ----------------------------------------- | ---------------------------------- |
| #470 | M1-01 (tier config schema)                | feat/federation-m1-tier-config     |
| #471 | M1-02 (compose overlay)                   | feat/federation-m1-compose         |
| #472 | M1-03 (pgvector adapter)                  | feat/federation-m1-pgvector        |
| #473 | M1-04 (tier-detector)                     | feat/federation-m1-detector        |
| #474 | M1-05 (migrate-tier script)               | feat/federation-m1-migrate         |
| #475 | M1-06 (gateway doctor)                    | feat/federation-m1-doctor          |
| #476 | M1-07 (boot integration tests)            | feat/federation-m1-integration     |
| #477 | M1-08 (migrate integration test + P0 fix) | feat/federation-m1-migrate-test    |
| #478 | M1-09 (standalone regression)             | feat/federation-m1-regression      |
| #479 | M1-10 (security review fixes)             | feat/federation-m1-security-review |
| #480 | M1-11 (docs)                              | feat/federation-m1-docs            |
| #481 | M1-12 (aggregate close)                   | feat/federation-m1-close           |

### Process learnings (M1 retrospective)

1. **Two-round security review is non-negotiable for security work.** First round caught postgres credential leaks; second round caught equivalent valkey leaks the worker missed when extending the regex. Single-round would have shipped HIGH severity issues.
2. **Real-services integration tests catch what mocked unit tests cannot.** M1-08 caught a P0 in M1-05 (camelCase column names) that 32 mocked unit tests missed because both source and target were mocked. Going forward: at least one real-services test per code-mutating PR where feasible.
3. **Test-utils for live services co-locate with consumer, not in shared library.** M1-08 reviewer caught `createPgliteDbWithVector` initially being added to `@mosaicstack/db` public exports — would have polluted prod consumers with WASM bundle. Moved to `packages/storage/src/test-utils/`.
4. **Per-task budgets including tests/review/docs more accurate than PRD's implementation-only estimates.** M1 PRD estimated 20K; actual ~74K. Future milestones should budget the full delivery cycle.
5. **TASKS.md status updates ride feature branches, never direct-to-main.** Caught one violation early in M1; pattern held for all 12 tasks.
6. **Subagent tier matters.** Code review needs sonnet-level reasoning (haiku missed deep issues in M1-04); claim verification (line counts, file existence) is fine on haiku.

**Followup tasks still deferred (carry forward to M2):**

- #7: `tier=local` hardcoded in gateway-config resume branches (~262, ~317)
- #8: confirm `packages/config/dist` not git-tracked

**Next mission step:** FED-M2 (Step-CA + grant schema + admin CLI). Per TASKS.md scope rule, M2 will be decomposed when it enters active planning. Issue #461 tracks scope.

## Session 20 — 2026-04-21 — FED-M2 kickoff

### Decisions

- **Workstream split**: parallel CODE (M2-01..M2-13, ~72K) + DEPLOY (DEPLOY-01..DEPLOY-05, ~16K) tracks; re-converge at M2-10 E2E.
- **Test hosts**: `mos-test-1.woltje.com` (querying side / Server A), `mos-test-2.woltje.com` (serving side / Server B). Wildcard `*.woltje.com` A→174.137.97.162 already exists; Traefik wildcard cert covers both subdomains. No DNS or cert work needed pre-deploy.
- **Portainer access**: requires `PORTAINER_INSECURE=1` flag added to mosaic wrappers (self-signed cert at `https://10.1.1.43:9443`). PR pending on `feat/mosaic-portainer-tls-flag`.
- **Image policy**: deploy by digest (immutable) per Mosaic policy. `gateway:fed-v0.1.0-m1` digest = `sha256:9b72e202a9eecc27d31920b87b475b9e96e483c0323acc57856be4b1355db1ec`.

### DEPLOY-01 — image manifest verified

- Tag `fed-v0.1.0-m1` exists at `git.mosaicstack.dev/mosaicstack/stack/gateway`
- Digest: `sha256:9b72e202a9eecc27d31920b87b475b9e96e483c0323acc57856be4b1355db1ec`
- 9 layers, ~530MB total
- Use this digest in DEPLOY-02 stack template (do NOT reference `:fed-v0.1.0-m1` tag in stack — pin to digest)

### Registry auth note

- Gitea container registry uses Bearer token flow (`/v2/token?service=container_registry&scope=repository:<repo>:pull`)
- Username: `jarvis` (NOT `mosaicstack`); password: `gitea.mosaicstack.token` from credentials.json
- Direct `Authorization: Bearer <pat>` does NOT work — must exchange PAT for registry token first

### Active PRs

- #483 — docs: M2 mission planning (TASKS decomposition + manifest update) — CI running
- (pending) `feat/mosaic-portainer-tls-flag` — wrapper PORTAINER_INSECURE flag (sonnet subagent in progress)
- (pending) `feat/federation-m2-schema` — FED-M2-01 DB schema migration (sonnet subagent in progress)

### MISSION-MANIFEST layout fix

- Initial M2 commit had Test Infrastructure block inserted by lint-staged prettier between "Last Updated" and "Parent Mission" — split mission frontmatter
- Fixed in 3d001fdb: moved Parent Mission back to frontmatter, kept Test Infrastructure as standalone H2 between Mission and Context

## Session 21 — 2026-04-21/22 — DEPLOY-02 merged, gateway image bug discovered, M2-01 in remediation

### PRs merged

- **#483** — docs(federation): M2 mission planning (TASKS decomposition + manifest update)
- **#484** — feat(mosaic-portainer): PORTAINER_INSECURE flag for self-signed TLS (wrapper sync to `~/.config/mosaic/tools/portainer/` done manually due to broken `mosaic upgrade` `set -o pipefail` on dash)
- **#485** — feat(deploy): portainer stack template `deploy/portainer/federated-test.stack.yml` for federation test instances [DEPLOY-02]

### Stack deployed (mos-test-1, mos-test-2)

- Both stacks created on Portainer endpoint 3 (`local` Swarm @ 10.1.1.43, the only endpoint with traefik-public + woltje.com wildcard cert)
- Swarm ID `l7z67tfpd4bvj4979ufpkyi50`
- Image pinned to digest `sha256:9b72e202a9eecc27d31920b87b475b9e96e483c0323acc57856be4b1355db1ec`
- Traefik labels target `${HOST_FQDN}` per env

### CRITICAL FINDING — gateway image runtime-broken

- `docker run` against `gateway:fed-v0.1.0-m1` fails immediately:  
  `Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'dotenv' imported from /app/dist/main.js`
- Root cause: `docker/gateway.Dockerfile` copies `/app/node_modules` from builder — but pnpm puts deps in the content-addressed `.pnpm/` store with symlinks at `apps/gateway/node_modules/*`. The runner stage misses the symlinks → Node can't resolve workspace deps.
- M1 release was never runtime-tested as a stripped container; CI passed because tests run in dev tree where pnpm symlinks are intact.
- **Fix in flight** (subagent `a78a9ab0ddae91fbc`): switch builder to `pnpm --filter @mosaic/gateway --prod deploy /deploy`, then runner copies `/deploy/node_modules` + `/deploy/dist` + `/deploy/package.json`.

### M2-01 schema review verdict — NEEDS CHANGES

- PR #486 (`feat/federation-m2-schema`) — independent reviewer (sonnet) found 2 real issues:
  1. `federation_audit_log` time-range indexes missing `.desc()` on `created_at` (3 places)
  2. Reserved columns missing per TASKS.md M2-01 spec: `query_hash`, `outcome`, `bytes_out` (M4 will write; spec said reserve now)
- Also notes (advisory): subject_user_id correctly `text` (matches BetterAuth users.id; spec defect, not code defect); peer→grant cascade test not present (would be trivial to add)
- **Remediation in flight** (subagent `a673dd9355dc26f82` in worktree `agent-a4404ac1`): apply DESC + reserved cols, regenerate migration in place (preferred) or stack 0009 (fallback), force-push, post PR comment.

### Process notes

- Branch race incident: schema subagent + wrapper subagent both ran in main checkout → schema files appeared on wrapper branch. Recovered by TaskStop, `git checkout --` to clean, respawned schema subagent with `isolation: "worktree"`. **Rule going forward:** any subagent doing code edits gets `isolation: "worktree"` unless work is single-file and the orchestrator confirms no other branch will touch overlapping files.
- `pr-create.sh` shell-quotes backticks badly → use `tea pr create --repo mosaicstack/stack` directly (matches CLI-skill behavior). Will leave a followup to harden pr-create.sh.
- Gitea registry auth: bearer-token exchange flow (`/v2/token?service=container_registry&scope=repository:<repo>:pull`) — direct `Authorization: Bearer <pat>` returns 401.
- Portainer Swarm stack create endpoint: `POST /api/stacks/create/swarm/string?endpointId=<id>` (NOT `/api/stacks?type=1` — deprecated and rejected with 400).

### In-flight at compaction boundary

- Subagent `a78a9ab0ddae91fbc` — Dockerfile pnpm-deploy fix → PR (not yet opened at handoff)
- Subagent `a673dd9355dc26f82` — M2-01 schema remediation (DESC + reserved cols) → force-push to PR #486
- Both will trigger CI; orchestrator must independently re-review fixes (especially the security-adjacent schema work) per "always verify subagent claims" rule.

### Next after subagents return

1. Independent re-review of schema remediation (different subagent, fresh context)
2. Merge #486 if green
3. Merge Dockerfile fix PR if green → triggers Kaniko CI rebuild → capture new digest
4. Update `deploy/portainer/federated-test.stack.yml` to new digest in a small PR
5. Redeploy mos-test-1 + mos-test-2 (Portainer stack update via API)
6. Verify HTTPS reachability + `/health` endpoint at both hosts
7. DEPLOY-03/04 acceptance probes (`mosaic gateway doctor --json`, pgvector `vector(3)` round-trip)
8. DEPLOY-05: author `docs/federation/TEST-INFRA.md`
9. M2-02 (Step-CA sidecar) kicks off after image health is green

### Session 23 — 2026-04-21 — M2 close + M3 decomposition

**Closed at compaction boundary:** all 13 M2 tasks done, PRs #494–#503 merged to `main`, tag `fed-v0.2.0-m2` published, Gitea release notes posted, issue #461 closed. Main at `4ece6dc6`.

**M2 hardening landed in PR #501** (security review remediation):

- CRIT-1: post-issuance OID verification in `ca.service.ts` (rejects cert if `mosaic_grant_id` / `mosaic_subject_user_id` extensions missing or mismatched)
- CRIT-2: atomic activation guard `WHERE status='pending'` on grant + `WHERE state='pending'` on peer; throws `ConflictException` if lost race
- HIGH-2: removed try/catch fallback in `extractCertNotAfter` — parse failures propagate as 500 (no silent 90-day default)
- HIGH-4: token slice for logging (`${token.slice(0, 8)}...`) — no full token in stdout
- HIGH-5: `redeem()` wrapped in try/catch with best-effort failure audit; uses `null` (not `'unknown'`) for nullable UUID FK fallback
- MED-3: `createToken` validates `grant.peerId === dto.peerId`; `BadRequestException` on mismatch

**Remaining M2 security findings deferred to M3+:**

- HIGH-1: peerId/subjectUserId tenancy validation on `createGrant` (M3 ScopeService work surfaces this)
- HIGH-3: Step-CA cert SHA-256 fingerprint pinning (M5 cert handling)
- MED-1: token entropy already 32 bytes — wontfix
- MED-2: per-route rate limit on enrollment endpoint (M4 rate limit work)
- MED-4: CSR CN binding to peer's commonName (M3 AuthGuard work)

**M3 decomposition landed in this session:**

- 14 tasks (M3-01..M3-14), ~100K estimate
- Structure mirrors M1/M2 pattern: foundation → server stream + client stream + harness in parallel → integration → E2E → security review → docs → close
- M3-02 ships local two-gateway docker-compose (`tools/federation-harness/`) so M3-11 E2E is not blocked on the Portainer test bed (which is still blocked on `FED-M2-DEPLOY-IMG-FIX`)

**Subagent doctrine retained from M2:**

- All worker subagents use `isolation: "worktree"` to prevent branch-race incidents
- Code review is independent (different subagent, no overlap with author of work)
- `tea pr create --repo mosaicstack/stack --login mosaicstack` is the working PR-create path; `pr-create.sh` has shell-quoting bugs (followup #45 if not already filed)
- Cost tier: foundational implementation = sonnet, docs = haiku, complex multi-file architecture (security review, scope service) = sonnet with two review rounds

**Next concrete step:**

1. PR for the M3 planning artifact (this commit) — branch `docs/federation-m3-planning`
2. After merge, kickoff M3-01 (DTOs) on `feat/federation-m3-types` with sonnet subagent in worktree
3. Once M3-01 lands, fan out: M3-02 (harness) || M3-03 (AuthGuard) → M3-04 (ScopeService) || M3-08 (FederationClient)
4. Re-converge at M3-10 (Integration) → M3-11 (E2E)
