Move old docs into _old_structure dir. Create blank GETTING_STARTED.md document

2026-08-10 14:53:21 -05:00
parent 1655b1579a
commit cd4409abc3
273 changed files with 278 additions and 247 deletions
@@ -0,0 +1,290 @@
+# Design — #791: Framework upgrades must not destroy operator-owned config under `~/.config/mosaic`
+
+- **Issue:** mosaicstack/stack#791
+- **Branch:** `feat/791-upgrade-config-protection` (off `origin/main` `9745bc3f`)
+- **Author:** ms-791 worker lane
+- **Status:** Phase 1 — DESIGN, awaiting MS-LEAD confirmation before implementation
+- **Ratified scope (Mos-approved, not re-litigated):** deliver **(b) strict ownership separation [PRIMARY]** + **(a) transactional pre-update snapshot [safety net]** + **(d) regeneration-from-SSOT [recovery]**. **(c) periodic backup timer is DEFERRED** — noted as future work only.
+
+---
+
+## 1. Current updater behavior + exact wipe mechanism (evidence)
+
+### 1.1 What runs on `mosaic update`
+
+`mosaic update` re-seeds the framework by invoking the **bash installer** in sync-only, keep mode:
+
+- `packages/mosaic/src/runtime/update-checker.ts:509` `buildReseedCommand()` returns
+  `bash <frameworkRoot>/install.sh` with env `MOSAIC_SYNC_ONLY=1`, `MOSAIC_INSTALL_MODE=keep`,
+  `MOSAIC_HOME=<mosaicHome>`.
+- The same `install.sh` is the direct/`tools/install.sh` upgrade path and the framework-vN migration path.
+
+So the destructive surface is **`packages/mosaic/framework/install.sh`**.
+
+### 1.2 The wipe
+
+`sync_framework()` (`install.sh:177`) performs, in `keep` mode:
+
+```
+rsync -a --delete --exclude .git --exclude .framework-version --exclude '*.pre-constitution.bak' \
+      [--exclude "/$path" for each PRESERVE_PATHS entry] SOURCE_DIR/ TARGET_DIR/
+```
+
+- `install.sh:199` — `rsync -a --delete`. **`--delete` prunes every path in `~/.config/mosaic`
+  that is NOT present in the shipped framework source**, unless excluded.
+- `install.sh:47` — `PRESERVE_PATHS` is the **only** thing standing between `--delete` and operator
+  data. It is a _denylist of exclusions_:
+  ```
+  PRESERVE_PATHS=("CONSTITUTION.md" "AGENTS.md" "SOUL.md" "USER.md" "TOOLS.md" "STANDARDS.md"
+    "memory" "sources" "credentials" "fleet/roster.yaml" "fleet/roster.json" "fleet/agents"
+    "fleet/run" "fleet/backlog" "fleet/roles.local")
+  ```
+- The cp-fallback (no rsync) is equally destructive: `install.sh:223`
+  `find "$TARGET_DIR" -mindepth 1 -maxdepth 1 ... -exec rm -rf {} +` then re-copies source, restoring
+  only PRESERVE_PATHS globs.
+
+**Root-cause model:** _"Everything under `~/.config/mosaic` is framework-owned and pruneable UNLESS
+explicitly preserved."_ Any operator path the list forgets is destroyed on the next upgrade.
+
+### 1.3 The exact operator paths wiped
+
+Cross-referencing the issue's operator-owned list against `PRESERVE_PATHS`:
+
+| Operator path (issue #791)                                        | In PRESERVE_PATHS?                      | Fate on `mosaic update` |
+| ----------------------------------------------------------------- | --------------------------------------- | ----------------------- |
+| `agents/*.conf` (per-agent runtime)                               | **NO**                                  | **WIPED**               |
+| `policy/*.md` (operator overlays)                                 | **NO**                                  | **WIPED**               |
+| `*.local.md` (SOUL/USER/STANDARDS)                                | **NO**                                  | **WIPED**               |
+| harvester / SOP artifacts + timers                                | **NO**                                  | **WIPED**               |
+| `tools/_lib/credentials.json`                                     | **NO** (`credentials/` dir ≠ this path) | **WIPED**               |
+| `fleet/agents/*.env`                                              | yes (`fleet/agents`, added by #631)     | survives                |
+| `memory/`, `fleet/roster.*`, `fleet/backlog`, `fleet/roles.local` | yes                                     | survives                |
+
+The `fleet/agents`, `memory`, `fleet/backlog` entries were **retro-added after prior incidents**
+(#631). This whack-a-mole is the structural signature of a denylist.
+
+**Stale-comment evidence:** `update-checker.ts:492` claims the reseed preserves
+"`SOUL/USER/*.local/credentials`" — but `PRESERVE_PATHS` contains **no `*.local` entry**. The code
+documents protection it does not deliver.
+
+### 1.4 Second code path (TS) — already non-destructive, but drifted
+
+`FileConfigAdapter.syncFramework()` (`packages/mosaic/src/config/file-adapter.ts:157`) →
+`syncDirectory()` (`packages/mosaic/src/platform/file-ops.ts:66`) is a **copy-overlay**: it copies
+source over target and skips preserved paths, but **never deletes** target paths absent from source
+(`file-ops.ts:77-109`). It is used by the wizard/init flow, not `mosaic update`.
+
+Two problems remain:
+
+1. Its `preservePaths` (`file-adapter.ts:164-185`) has **already diverged** from `install.sh` — it is
+   **missing `fleet/backlog` and `fleet/roles.local`**. Two hand-maintained denylists, drifted. This
+   is direct evidence for a single shared SSOT manifest.
+2. Even non-destructive, it will happily _overwrite_ an operator file that collides with a
+   framework-shipped path unless that path is on its (incomplete) preserve list.
+
+### 1.5 Existing snapshot is inadequate for rollback
+
+`make_snapshot()`/`restore_snapshot()` (`install.sh:76-87`) copy `TARGET_DIR` to `mktemp -d` under
+`/tmp`, restore **only on `ERR/INT/TERM` trap**, and are **deleted on success** (`cleanup_snapshot`,
+`install.sh:345`). Consequences: ephemeral `/tmp`, no retention, no post-success rollback, and **no
+`mosaic restore`**. It is crash-safety only, not the transactional safety net #791 requires.
+
+---
+
+## 2. Fix (b) — Strict ownership separation [PRIMARY / root cause]
+
+### 2.1 Ownership model (invert to allow-list)
+
+Replace _"framework-owned unless preserved"_ with _"operator-owned unless framework-owned"_, resolved
+**per target path** with operator carve-outs winning inside shared framework subtrees.
+
+Two declared lists, one SSOT data file shipped in the framework
+(`framework/framework-manifest.json`), consumed by **both** bash and TS:
+
+- **`framework` globs** — paths the updater is entitled to create / overwrite / prune. Authored to
+  match exactly what the framework ships in `packages/mosaic/framework/` (e.g. `CONSTITUTION.md`,
+  `AGENTS.md`, `STANDARDS.md`, `TOOLS.md`, `guides/**`, `constitution/**`, `templates/**`, `tools/**`,
+  `skills/**`, `mcp/**`, `defaults/**`, `fleet/examples/**`, `fleet/roles/**`, `fleet/profiles/**`,
+  `fleet/roster.schema.json`).
+- **`operatorReserved` globs** — NEVER written or pruned, even nested inside a `framework` subtree;
+  these **win** over `framework` (deny-wins / most-specific-wins). At minimum:
+  `agents/**`, `policy/**`, `memory/**`, `sources/**`, `credentials/**`, `*.local.md`,
+  `tools/_lib/credentials.json`, `fleet/roster.yaml`, `fleet/roster.json`, `fleet/agents/**`,
+  `fleet/run/**`, `fleet/backlog/**`, `fleet/roles.local/**`, plus operator harvester/SOP artifacts.
+
+### 2.2 Ownership resolution for a target path `P`
+
+1. `P` matches `operatorReserved` → **operator-owned**: updater MUST NOT write, MUST NOT delete.
+2. else `P` matches `framework` → **framework-owned**: may overwrite; may prune **only if absent from
+   the current SOURCE** (a genuinely retired framework file).
+3. else (matches neither) → **UNKNOWN ⇒ operator-owned by default (fail-safe)**: never delete.
+
+Rule 3 is the actual root-cause fix: an operator path the manifest authors forget is still protected,
+because _unknown defaults to operator_. A denylist can never provide this guarantee.
+
+### 2.3 Sync mechanism change (the mechanically-critical part)
+
+`--delete` cannot express "prune only framework-owned" without re-enumerating every operator path
+(the denylist trap). So:
+
+1. **Drop `--delete` from the bulk sync.** Copy `SOURCE → TARGET` non-destructively (writes/overwrites
+   all framework files; deletes nothing). rsync without `--delete`, or the existing overlay copy.
+2. **Explicit manifest-scoped prune pass.** Iterate the **`framework` manifest** (not the whole tree);
+   for each framework path present in `TARGET` but **absent in `SOURCE`**, delete it — after
+   re-checking it does not match `operatorReserved`. Because the prune iterates only declared
+   framework globs, operator/unknown paths are **structurally unreachable** by deletion.
+
+This is implemented in both bash `sync_framework()` and TS `syncFramework()` from the shared manifest.
+A pure **prune-planner** function (TS) computes the delete-set from
+`(manifest, sourceListing, targetListing)` so the invariant is unit-testable in isolation.
+`PRESERVE_PATHS` becomes redundant (kept as a defense-in-depth alias mapping to `operatorReserved`, or
+removed) — either way the two lists stop drifting because they read one file.
+
+### 2.4 HARD GATE test — "upgrade touches no path outside the manifest"
+
+Filesystem-observation test in the existing `test-install-migration.sh` harness pattern (mktemp
+`MOSAIC_HOME`, `MOSAIC_SYNC_ONLY=1`), plus TS specs:
+
+1. Seed a throwaway `TARGET` with a realistic operator mix — one sentinel per operator class:
+   `agents/x.conf`, `policy/p.md`, `SOUL.local.md`, `memory/m.md`,
+   `tools/_lib/credentials.json` (with a secret value), `fleet/agents/a.env`, `fleet/roster.yaml`,
+   `harvester/sop.md`, **and a deliberately-unanticipated `unknown-operator-dir/x`**.
+2. Record hash+mtime of every sentinel.
+3. Run the upgrade from a `SOURCE` containing none of those operator paths.
+4. **Assert:** every sentinel exists, byte-identical, **mtime unchanged** (not even rewritten). The
+   `unknown-operator-dir` surviving proves the fail-safe default — a denylist could not pass this case.
+5. **Positive controls:** framework files WERE updated; a retired framework file WAS pruned.
+6. **Property test** (TS prune-planner): for fuzzed operator paths, `deleteSet ⊆ {matches framework ∧
+in target ∧ not in source}` and `deleteSet ∩ operatorReserved = ∅`.
+
+---
+
+## 3. Fix (a) — Transactional pre-update snapshot [safety net]
+
+- **Destination:** `${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-ts>/`.
+  **Outside `~/.config/mosaic`** (so no future sync can sweep it) and outside any repo.
+- **Perms:** dir `0700`, files `0600` — enforced with `umask 077` around the copy **and** explicit
+  `chmod`. Never world-readable.
+- **Scope:** the operator-owned surface (`operatorReserved` paths that exist) — bounded; does not copy
+  the framework tree.
+- **Timing:** taken before ANY mutation in the upgrade flow.
+- **Post-sync verify + selective restore:** after sync, diff the operator surface against the snapshot;
+  since (b) should never touch operator paths, any diff means a manifest bug — restore the affected
+  paths from the snapshot and warn loudly. This is precisely (a) catching a miss in (b).
+- **Retention:** keep N most-recent (default 5; `MOSAIC_BACKUP_RETENTION` override); prune older.
+- **`mosaic restore`:** `--list` (default, dry-run) enumerates snapshots by timestamp;
+  `--from <ts>` restores that snapshot over the operator surface, confirmation-gated. Reports
+  counts/paths only.
+- **Secret-safety:** snapshot copy and restore never emit file **contents**; only paths/counts.
+  Tests assert `0700/0600` and that no secret value appears in stdout/stderr.
+
+---
+
+## 4. Fix (d) — Regeneration-from-SSOT [recovery]
+
+The incident's live blast radius: `fleet/agents/*.env` (systemd `EnvironmentFile` sources) gone →
+`mosaic-agent@<name>` boots **unit defaults** on restart (because `EnvironmentFile=-...` is
+absent-tolerant) → **silent identity/runtime/workdir downgrade**.
+
+The SSOT for those `.env` files is the roster. The reconciler **already** separates a
+`regenerate-projections-from-roster` projection phase from lifecycle
+(`packages/mosaic/src/fleet/fleet-reconciler.ts:93,234`; env rendering in
+`generated-env-boundary.ts:149-264`).
+
+**`mosaic fleet regen`** is therefore a **thin recovery-framed wrapper over the existing projection
+phase** — it does NOT reimplement fleet logic and does NOT preempt in-flight FCM cards (M4/M5):
+
+- Regenerates derivable config (per-agent `*.env.generated`, unit files) from roster SSOT.
+- **Preview-first:** dry-run default; `--write` to apply. Idempotent.
+- **Never restarts agents** (the recovery order forbids restart-before-verify).
+- Prints the runbook's next step (verify `EnvironmentFile` resolves, THEN restart).
+
+Alternatively documentable as `install.sh --relink` per the issue; `mosaic fleet regen` is preferred
+because it reuses the merged reconciler plumbing.
+
+---
+
+## 5. Secret-safety approach (secrev surface)
+
+- Snapshots/backups: `0700`/`0600`, outside any repo, never world-readable. (§3)
+- No secret **value** ever emitted to logs/stdout/stderr by snapshot, restore, sync, or regen —
+  paths/counts only. Adversarial test: a secret value placed in `tools/_lib/credentials.json` must
+  never appear in installer or command output.
+- `tools/_lib/credentials.json` is an explicit `operatorReserved` carve-out inside the framework-owned
+  `tools/**` subtree — it is never overwritten or pruned.
+- The HARD GATE test doubles as a secret-safety test (asserts the credentials sentinel is untouched).
+
+---
+
+## 6. Test plan (TDD, tests-first, ≥85% on new code, co-located `*.spec.ts`)
+
+1. **Manifest SSOT parity** — bash and TS resolve identical framework/operator sets from the one file;
+   a test fails if either path hard-codes a divergent list.
+2. **Manifest completeness** — every path shipped in `framework/` is covered by a `framework` glob (so
+   a new shipped file cannot silently fall outside the manifest and become un-prunable/undeclared).
+3. **HARD GATE** — upgrade touches nothing outside the manifest, incl. the unanticipated-path case
+   (§2.4).
+4. **Prune-planner** unit + property tests (§2.4.6).
+5. **Snapshot** — perms `0700/0600`, correct destination, retention prune, secret value absent from
+   output.
+6. **Restore** — `--list` / `--from` round-trip restores operator surface byte-exact; confirmation
+   gate; no secret leakage.
+7. **Regen** — roster→env projection deterministic + idempotent; dry-run makes no writes; `--write`
+   restores `*.env`; **never** issues a lifecycle/restart call.
+8. **Cross-path regression** — TS `syncFramework` and bash `install.sh` agree on a shared fixture
+   (closes the current #631-style drift).
+
+Gates before every push: `pnpm typecheck && pnpm lint && pnpm format:check` + mosaic package tests
+green. Never `--no-verify`.
+
+---
+
+## 7. web1 recovery runbook (operator-agnostic; web1 specifics live in the issue as evidence only)
+
+For a currently-wiped fleet EnvironmentFile state — **do NOT service-restart while
+`fleet/agents/*.env` is absent** (a restart boots unit defaults and silently downgrades identity):
+
+1. **Regenerate:** `mosaic fleet regen --write` — rebuild `~/.config/mosaic/fleet/agents/*.env` from
+   roster SSOT.
+2. **Verify each unit resolves to the intended runtime/workdir** _before_ any restart:
+   `systemctl --user show mosaic-agent@<name> -p EnvironmentFile` and confirm the generated env exists
+   and carries the intended `MOSAIC_AGENT_*` runtime/workdir values.
+3. **Only then** `systemctl --user restart mosaic-agent@<name>`, one unit at a time.
+
+If config (not just fleet env) was lost, `mosaic restore --list` → `mosaic restore --from <ts>` before
+step 1.
+
+---
+
+## 8. Proposed PR split (reviewable; DAG-ordered)
+
+| PR  | Scope                                                                                                                                                    | Depends | Review focus               |
+| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------- |
+| PR1 | **PRIMARY** — shared `framework-manifest.json` + ownership resolver + non-deleting sync + scoped prune (bash + TS) + **HARD GATE** + prune-planner tests | —       | correctness (root fix)     |
+| PR2 | **Safety net** — pre-update snapshot (`~/.local/state`, 0700/0600, retention) + post-sync verify/restore + `mosaic restore`                              | PR1     | **secrev** (backup/secret) |
+| PR3 | **Recovery** — `mosaic fleet regen` (projection-only, preview-first, no restart) + docs (upgrade-safety + recovery runbook)                              | PR1     | correctness + docs         |
+
+Rationale: PR1 closes the failure class on its own; if PR2/PR3 slip, the class stays fixed. Each PR is
+one reviewable unit with its own tests ≥85%. Independent review (author≠reviewer) on all; **secrev** on
+PR2 (and PR1's secret-sentinel assertions).
+
+## 9. Deferred (noted per scope)
+
+**(c) periodic backup timer** — a systemd user timer snapshotting operator dirs on a cadence
+(defense-in-depth for non-upgrade losses). Explicitly **out of scope now**; future phase.
+
+## 10. Constraints honored
+
+- **Framework-PR firewall:** manifest + logic are operator-agnostic; no SOUL/USER/operator specifics
+  in framework code; web1 details are issue evidence only.
+- **Capacity-fill:** must not preempt M5-001 or #790; `fleet regen` reuses merged FCM-M3 plumbing and
+  does not overlap FCM-M4/M5 migration cards.
+- **Delivery gates:** TDD tests-first, ≥85% new-code coverage, trunk-based squash PRs, independent
+  review + secrev, completion = merged PR + descendant-main green + #791 closed.
+
+---
+
+**Requesting MS-LEAD confirmation of:** (1) the manifest allow-list + non-deleting-sync + scoped-prune
+approach as the (b) root-cause fix; (2) snapshot destination/retention + `mosaic restore` UX;
+(3) `mosaic fleet regen` as a projection-only wrapper; (4) the 3-PR split. Implementation begins only
+on your confirmation.
@@ -0,0 +1,75 @@
+# Constitution Alpha — Definition-of-Done checklist + release notes
+
+Drafted for the `v0.0.39-alpha` tag (Lead cuts after P5 #605 → P6 #607 → aiguide #8 merge).
+Maps every DoD §8 acceptance criterion to its merged evidence. Legend:
+**✅ merged on main** · **⏳ review-ready PR (pending merge)** · **🔲 Lead action**.
+
+## DoD §8 green-checklist
+
+| #   | Acceptance criterion (DESIGN §8)                                                                       | Status | Evidence / PR     |
+| --- | ------------------------------------------------------------------------------------------------------ | ------ | ----------------- |
+| 1   | MIT `LICENSE` (root + framework) + `"license":"MIT"` in package.json                                   | ✅     | P0 #570           |
+| 2   | Three credential-path sites + hook URL fast-failed (no private paths in `*.sh`/hooks)                  | ✅     | P0 #570           |
+| 3   | `verify-sanitized.sh` (two-class, `*.sh`+`*.md`, self-tested) wired **blocking** in CI                 | ✅     | P1 #572           |
+| 4   | Operator data purged from the full set (guides / tools / init-generator)                               | ✅     | P2 #572           |
+| 5   | `rails/`→`tools/` in **both** template families                                                        | ✅     | P2 #572           |
+| 6   | `jarvis-loop.json` deleted; `defaults/SOUL.md` → **neutral sanitized persona** (Q10 decision)          | ✅     | P2 #572           |
+| 7   | `CONSTITUTION.md` extracted (gates one place, capability-verb, §1.4 split, no false "already loaded")  | ✅     | P3 #575 / #577    |
+| 8   | `AGENTS.md`/`STANDARDS.md` out of `PRESERVE_PATHS` + seed-semantics → overwrite in **both** installers | ✅     | P4 #590           |
+| 9   | Snapshot + v2→v3 migration moving user edits to `.local`/`.bak`; `FRAMEWORK_VERSION=3`                 | ✅     | P4 #590 / #593    |
+| 10  | `mosaic-init --non-interactive` fail-closed persona                                                    | ✅     | P4 #590           |
+| 11  | **5-fixture migration matrix** green against **both** installers asserting **injected bytes**          | ✅     | P4 #590 / #593    |
+| 12  | `compose-contract` built + composer unit test (per-tier anchor + Tier-3 byte-equality)                 | ⏳     | P5 #605           |
+| 13  | Resident line-count ceiling enforced (framework-owned resident files)                                  | ⏳     | P6 #607           |
+| 14  | `CONTRIBUTING.md` + harness×gate compliance matrix                                                     | ⏳     | P6 #607           |
+| 15  | `aiguide` reconciled with the Constitution                                                             | ⏳     | aiguide #8        |
+| 16  | Each phase PR CI-green; alpha tag pushed + Gitea release published                                     | 🔲     | Lead (post-merge) |
+
+**Note on #6:** the DoD's literal "delete `defaults/SOUL.md`" was superseded by the resolved
+**Q10** decision — ship a _neutral, operator-agnostic_ example persona instead of deleting it. Main
+carries the sanitized 2.6 KB neutral SOUL.md ("Mosaic agent", no operator identity); the sanitization
+gate confirms it is PII-clean. Criterion met in spirit (no operator persona leaks) via the better option.
+
+**Gate to flip 12–14 → ✅:** merge P5 #605 → P6 #607 (rebase auto-drops the dup format fix
+`adc7df2`/`9f6da92`) → aiguide #8, with `ci.yml` terminal-green on the merged head.
+
+---
+
+## Release notes — `v0.0.39-alpha` (Mosaic Framework Constitution, alpha)
+
+### Mosaic Framework Constitution — Alpha
+
+This release makes the Mosaic framework a **safe-to-open-source, fork-and-customize agent
+operating layer**. It separates the non-negotiable law from operator identity, makes
+customization survive upgrades, and wires the guarantees into CI.
+
+**Highlights**
+
+- **Constitution (L0).** The hard gates now live in one place — `CONSTITUTION.md` — authored in
+  capability verbs, with a thin `AGENTS.md` dispatcher that references the law instead of restating
+  it. Governance model in `constitution/LAYER-MODEL.md`.
+- **Public & sanitized.** MIT-licensed; all operator identity, private paths, and credential sites
+  removed from shipped files. A self-tested `verify-sanitized.sh` gate (two rule classes) runs
+  **blocking** in CI so re-contamination can't merge.
+- **Upgrade-safe customization.** Framework-owned files overwrite cleanly on upgrade while
+  `SOUL.md`/`USER.md`/`*.local.md`/`credentials` are preserved. The v2→v3 migration snapshots first
+  and moves any user-edited `AGENTS.md`/`STANDARDS.md` to `.pre-constitution.bak`/`.local.md` —
+  never silently lost. Verified by a 5-fixture matrix across **both** installers.
+- **Operator overlays.** `mosaic compose-contract <harness>` merges your `*.local.md` deltas into
+  the contract per harness, so customization reaches the model as one pre-merged blob.
+- **Cross-harness.** Single L0 source referenced (never restated) by Claude / Codex / OpenCode / Pi;
+  tiered injection with a byte-equal Tier-3 fallback read.
+- **Guardrails in CI.** Resident line-count ceiling over framework-owned resident files; composer
+  unit test; sanitization gate — all blocking.
+- **Docs.** `CONTRIBUTING.md` with the layer model, dual-installer parity rule, and a harness×gate
+  **compliance matrix** (the Codex/OpenCode/Pi hook-parity gap is tracked for v2).
+
+**Known limitations (accepted, documented in `CONTRIBUTING.md` §9)**
+
+- Bare launches that bypass `mosaic` get base contracts only (no `*.local` overlays) and are not
+  drift-checked by `mosaic doctor` — mitigated by the unconditional Tier-3 self-load + a nudge.
+- Codex/OpenCode/Pi mechanical hook parity, `policy/*.md` composition, and live-launch cross-harness
+  verification are **v2**.
+
+**Phase lineage:** P0 #570 · P1+P2 #572 · P3 #575/#577 · P4 #590/#593 · P5 #605 · P6 #607 ·
+aiguide #8 (umbrella #542).
@@ -0,0 +1,63 @@
+# npm `@next` prerelease lane
+
+Status: **IMPLEMENTED**
+
+## Current behavior
+
+`tools/install.sh --next` provides the prerelease integration lane for the permanent `next` branch.
+
+The lane is fast-by-default:
+
+1. Install framework files from the `next` source archive.
+2. Resolve the Gitea npm registry `next` dist-tag for the globally installed packages:
+
+   ```bash
+   npm view @mosaicstack/gateway@next version
+   npm view @mosaicstack/mosaic@next version
+   ```
+
+3. Require both resolved versions to share the same `next.<pipeline>` suffix, then install the exact resolved versions.
+4. If either `@next` package is missing, unreachable, mismatched, or fails to install, fall back to the source-build path at `next`.
+
+`--next` never hard-fails solely because the prerelease npm dist-tag is unavailable.
+
+## Published packages
+
+The `next` publish pipeline publishes non-private `@mosaicstack/*` packages to the Mosaic Gitea npm registry:
+
+```text
+https://git.mosaicstack.dev/api/packages/mosaicstack/npm/
+```
+
+Observed `next` dist-tags after enabling the pipeline:
+
+```text
+@mosaicstack/mosaic@next  -> 0.0.49-next.1633
+@mosaicstack/gateway@next -> 0.0.7-next.1633
+```
+
+The gateway also publishes a Docker image as `gateway:sha-<short>` on `next` merges. The installer fast path uses the npm gateway package when available; the Docker image is for deployed gateway/runtime harness flows.
+
+## Explicit source lanes
+
+Source builds remain available and are still the authority for explicit ref validation:
+
+- `--dev` always builds from source.
+- `--ref <ref>` / `MOSAIC_REF=<ref>` wins over `--next` and uses the source path for that exact ref.
+
+## Pipeline shape
+
+1. Trigger on `next` merges.
+2. Compute the next prerelease version from the upcoming stable version plus the Woodpecker pipeline number (`<target-stable>-next.<CI_PIPELINE_NUMBER>`).
+3. Build and publish non-private packages in CI.
+4. Publish to the Mosaic Gitea npm registry with dist-tag `next`.
+5. Keep `latest` untouched; only main/release promotion can update `latest`.
+6. Publish gateway Docker images from `next` as `gateway:sha-<short>` only.
+
+## Guardrails
+
+- `@next` is mutable prerelease convenience, not a deployment pin.
+- Stable installs continue to use `@latest`.
+- Contributor validation remains available through `--dev --ref <branch>`.
+- Pipeline output traces every prerelease package back to the source commit on `next`.
+- The installer falls back to source rather than hard-failing on prerelease registry issues.
@@ -0,0 +1,559 @@
+# Storage & Queue Abstraction — Middleware Architecture
+
+Design
+Status: Design (retrofit required)
+date: 2026-04-02
+context: Agents coupled directly to infrastructure backends, bypassing intended middleware layer
+
+---
+
+## The Problem
+
+Current packages are **direct adapters**, not **middleware**:
+| Package | Current State | Intended Design |
+|---------|---------------|-----------------|
+| `@mosaicstack/queue` | `ioredis` hardcoded | Interface → BullMQ OR local-files |
+| `@mosaicstack/db` | Drizzle + Postgres hardcoded | Interface → Postgres OR SQLite OR JSON/MD |
+| `@mosaicstack/memory` | pgvector required | Interface → pgvector OR sqlite-vec OR keyword-search |
+
+## The gateway and TUI import these packages directly, which means they they're coupled to specific infrastructure. Users cannot run Mosaic Stack without Postgres + Valkey.
+
+## The Intended Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     Gateway / TUI / CLI                          │
+│            (agnostic of storage backend, talks to middleware)    │
+└───────────────────────────┬─────────────────────────────────────┘
+                            │
+        ┌───────────────────┼───────────────────┐
+        │                 │                       │
+        ▼─────────────────┴─────────────────┴─────────────────┘
+        |                 |                     |                 |
+        ▼─────────────────┴───────────────────┴─────────────────┘
+        |                 |                     |                 |
+   Queue               Storage             Memory
+        |                 |                     |                 |
+   ┌─────────┬─────────┬─────────┬─────────────────────────────────┐
+   | BullMQ | | Local  | | Postgres | SQLite | JSON/MD | pgvector | sqlite-vec | keyword |
+   |(Valkey)| |(files) |         |             |         |          |          |
+   └─────────┴─────────┴─────────┴─────────────────────────────────┘
+```
+
+The gateway imports the interface, not the backend. At startup it reads config and instantiates the correct adapter.
+
+## The Drift
+
+```typescript
+// What should have happened:
+gateway/queue.service.ts → @mosaicstack/queue (interface) → queue.adapter.ts
+
+// What actually happened:
+gateway/queue.service.ts → @mosaicstack/queue → ioredis (hardcoded)
+```
+
+## The Current State Analysis
+
+### `@mosaicstack/queue` (packages/queue/src/queue.ts)
+
+```typescript
+import Redis from 'ioredis'; // ← Direct import of backend
+
+export function createQueue(config?: QueueConfig): QueueHandle {
+  const url = config?.url ?? process.env['VALKEY_URL'] ?? DEFAULT_VALKEY_URL;
+  const redis = new Redis(url, { maxRetriesPerRequest: 3 });
+  // ...queue ops directly on redis...
+}
+```
+
+**Problem:** `ioredis` is imported in the package, not the adapter interface. Consumers cannot swap backends.
+
+### `@mosaicstack/db` (packages/db/src/client.ts)
+
+> **Historical design specimen — status-only, not an operator instruction.** KBN-101 supersedes
+> this pre-split `DATABASE_URL` fallback shape; it cannot authorize runtime migration, DDL, or a
+> connection-string fallback. See the KBN-101 runner/role contract for the produced interface.
+
+```typescript
+import { drizzle, type PostgresJsDatabase } from 'drizzle-orm/postgres-js';
+import postgres from 'postgres';
+
+export function createDb(url?: string): DbHandle {
+  const connectionString = url ?? process.env['DATABASE_URL'] ?? DEFAULT_DATABASE_URL;
+  const sql = postgres(connectionString, { max: 20, idle_timeout: 30, connect_timeout: 5 });
+  const db = drizzle(sql, { schema });
+  // ...
+}
+```
+
+**Problem:** Drizzle + Postgres is hardcoded. No SQLite, JSON, or file-based options.
+
+### `@mosaicstack/memory` (packages/memory/src/memory.ts)
+
+```typescript
+import type { Db } from '@mosaicstack/db'; // ← Depends on Drizzle/PG
+
+export function createMemory(db: Db): Memory {
+  return {
+    preferences: createPreferencesRepo(db),
+    insights: createInsightsRepo(db),
+  };
+}
+```
+
+**Problem:** Memory package is tightly coupled to `@mosaicstack/db` (which is Postgres-only). No alternative storage backends.
+
+## The Target Interfaces
+
+### Queue Interface
+
+```typescript
+// packages/queue/src/types.ts
+export interface QueueAdapter {
+  readonly name: string;
+
+  enqueue(queueName: string, payload: TaskPayload): Promise<void>;
+  dequeue(queueName: string): Promise<TaskPayload | null>;
+  length(queueName: string): Promise<number>;
+  publish(channel: string, message: string): Promise<void>;
+  subscribe(channel: string, handler: (message: string) => void): () => void;
+  close(): Promise<void>;
+}
+
+export interface TaskPayload {
+  id: string;
+  type: string;
+  data: Record<string, unknown>;
+  createdAt: string;
+}
+
+export interface QueueConfig {
+  type: 'bullmq' | 'local';
+  url?: string; // For bullmq: Valkey/Redis URL
+  dataDir?: string; // For local: directory for JSON persistence
+}
+```
+
+### Storage Interface
+
+```typescript
+// packages/storage/src/types.ts
+export interface StorageAdapter {
+  readonly name: string;
+
+  // Entity CRUD
+  create<T>(collection: string, data: O): Promise<T>;
+  read<T>(collection: string, id: string): Promise<T | null>;
+  update<T>(collection: string, id: string, data: Partial<O>): Promise<T | null>;
+  delete(collection: string, id: string): Promise<boolean>;
+
+  // Queries
+  find<T>(collection: string, filter: Record<string, unknown>): Promise<T[]>;
+  findOne<T>(collection: string, filter: Record<string, unknown): Promise<T | null>;
+
+  // Bulk operations
+  createMany<T>(collection: string, items: O[]): Promise<T[]>;
+  updateMany<T>(collection: string, ids: string[], data: Partial<O>): Promise<number>;
+  deleteMany(collection: string, ids: string[]): Promise<number>;
+
+  // Raw queries (for complex queries)
+  query<T>(collection: string, query: string, params?: unknown[]): Promise<T[]>;
+
+  // Transaction support
+  transaction<T>(fn: (tx: StorageTransaction) => Promise<T>): Promise<T>;
+
+  close(): Promise<void>;
+}
+
+export interface StorageTransaction {
+  commit(): Promise<void>;
+  rollback(): Promise<void>;
+}
+
+export interface StorageConfig {
+  type: 'postgres' | 'sqlite' | 'files';
+  url?: string;      // For postgres
+  path?: string;      // For sqlite/files
+}
+```
+
+### Memory Interface (Vector + Preferences)
+
+```typescript
+// packages/memory/src/types.ts
+export interface MemoryAdapter {
+  readonly name: string;
+
+  // Preferences (key-value storage)
+  getPreference(userId: string, key: string): Promise<unknown | null>;
+  setPreference(userId: string, key: string, value: unknown): Promise<void>;
+  deletePreference(userId: string, key: string): Promise<boolean>;
+  listPreferences(
+    userId: string,
+    category?: string,
+  ): Promise<Array<{ key: string; value: unknown }>>;
+
+  // Insights (with optional vector search)
+  storeInsight(insight: NewInsight): Promise<Insight>;
+  getInsight(id: string): Promise<Insight | null>;
+  searchInsights(query: string, limit?: number, filter?: InsightFilter): Promise<SearchResult[]>;
+  deleteInsight(id: string): Promise<boolean>;
+
+  // Embedding provider (optional, null = no vector search)
+  readonly embedder?: EmbeddingProvider | null;
+
+  close(): Promise<void>;
+}
+
+export interface NewInsight {
+  id: string;
+  userId: string;
+  content: string;
+  embedding?: number[]; // If embedder is available
+  source: 'agent' | 'user' | 'summarization' | 'system';
+  category: 'decision' | 'learning' | 'preference' | 'fact' | 'pattern' | 'general';
+  relevanceScore: number;
+  metadata?: Record<string, unknown>;
+  createdAt: Date;
+  decayedAt?: Date;
+}
+
+export interface InsightFilter {
+  userId?: string;
+  category?: string;
+  source?: string;
+  minRelevance?: number;
+  fromDate?: Date;
+  toDate?: Date;
+}
+
+export interface SearchResult {
+  documentId: string;
+  content: string;
+  distance: number;
+  metadata?: Record<string, unknown>;
+}
+
+export interface MemoryConfig {
+  type: 'pgvector' | 'sqlite-vec' | 'keyword';
+  storage: StorageAdapter;
+  embedder?: EmbeddingProvider;
+}
+
+export interface EmbeddingProvider {
+  embed(text: string): Promise<number[]>;
+  embedBatch(texts: string[]): Promise<number[][]>;
+  readonly dimensions: number;
+}
+```
+
+## Three Tiers
+
+### Tier 1: Local (Zero Dependencies)
+
+**Target:** Single user, single machine, no external services
+
+| Component | Backend                                       | Storage      |
+| --------- | --------------------------------------------- | ------------ |
+| Queue     | In-process + JSON files in `~/.mosaic/queue/` |
+| Storage   | SQLite (better-sqlite3) `~/.mosaic/data.db`   |
+| Memory    | Keyword search                                | SQLite table |
+| Vector    | None                                          | N/A          |
+
+**Dependencies:**
+
+- `better-sqlite3` (bundled)
+- No Postgres, No Valkey, No pgvector
+
+**Upgrade path:**
+
+1. Run `mosaic gateway configure` → select "local" tier
+2. Gateway starts with SQLite database
+3. Optional: run `mosaic gateway upgrade --tier team` to migrate to Postgres
+
+### Tier 2: Team (Postgres + Valkey)
+
+**Target:** Multiple users, shared server, CI/CD environments
+
+| Component | Backend        | Storage                        |
+| --------- | -------------- | ------------------------------ |
+| Queue     | BullMQ         | Valkey                         |
+| Storage   | Postgres       | Shared PG instance             |
+| Memory    | pgvector       | Postgres with vector extension |
+| Vector    | LLM embeddings | Configured provider            |
+
+**Dependencies:**
+
+- PostgreSQL 17+ with pgvector extension
+- Valkey (Redis-compatible)
+- LLM provider for embeddings
+
+**Migration from Local → Team:**
+
+1. `mosaic gateway backup` → creates dump of SQLite database
+2. `mosaic gateway upgrade --tier team` → restores to Postgres
+3. Queue replays from BullMQ (may need manual reconciliation for in-flight jobs)
+4. Memory embeddings regenerated if vector search was new
+
+### Tier 3: Enterprise (Clustered)
+
+**Target:** Large teams, multi-region, high availability
+
+| Component | Backend                     | Storage                       |
+| --------- | --------------------------- | ----------------------------- |
+| Queue     | BullMQ cluster              | Multiple Valkey nodes         |
+| Storage   | Postgres cluster            | Primary + replicas            |
+| Memory    | Dedicated vector DB         | Qdrant, Pinecone, or pgvector |
+| Vector    | Dedicated embedding service | Separate microservice         |
+
+## MarkdownDB Integration
+
+For file-based storage, we use [MarkdownDB](https://markdowndb.com) to parse MD files into queryable data.
+
+**What it provides:**
+
+- Parses frontmatter (YAML/JSON/TOML)
+- Extracts links, tags, metadata
+- Builds index in JSON or SQLite
+- Queryable via SQL-like interface
+
+**Usage in Mosaic:**
+
+```typescript
+// Local tier with MD files for documents
+const storage = createStorageAdapter({
+  type: 'files',
+  path: path.join(mosaicHome, 'docs'),
+  markdowndb: {
+    parseFrontmatter: true,
+    extractLinks: true,
+    indexFile: 'index.json',
+  },
+});
+```
+
+## Dream Mode — Memory Consolidation
+
+Automated equivalent to Claude Code's "Dream: Memory Consolidation" cycle
+
+**Trigger:** Every 24 hours (if 5+ sessions active)
+
+**Phases:**
+
+1. **Orient** — What happened, what's the current state
+   - Scan recent session logs
+   - Identify active tasks, missions, conversations
+   - Calculate time window (last 24h)
+
+2. **Gather** — Pull in relevant context
+   - Load conversations, decisions, agent logs
+   - Extract key interactions and outcomes
+   - Identify patterns and learnings
+
+3. **Consolidate** — Summarize and compress
+   - Generate summary of the last 24h
+   - Extract key decisions and their rationale
+   - Identify recurring patterns
+   - Compress verbose logs into concise insights
+
+4. **Prune** — Archive and cleanup
+   - Archive raw session files to dated folders
+   - Delete redundant/temporary data
+   - Update MEMORY.md with consolidated content
+   - Update insight relevance scores
+
+**Implementation:**
+
+```typescript
+// In @mosaicstack/dream (new package)
+export async function runDreamCycle(config: DreamConfig): Promise<DreamResult> {
+  const memory = await loadMemoryAdapter(config.storage);
+
+  // Orient
+  const sessions = await memory.getRecentSessions(24 * 60 * 60 * 1000);
+  if (sessions.length < 5) return { skipped: true, reason: 'insufficient_sessions' };
+
+  // Gather
+  const context = await gatherContext(memory, sessions);
+
+  // Consolidate
+  const consolidated = await consolidateWithLLM(context, config.llm);
+
+  // Prune
+  await pruneArchivedData(memory, config.retention);
+
+  // Store consolidated insights
+  await memory.storeInsights(consolidated.insights);
+
+  return {
+    sessionsProcessed: sessions.length,
+    insightsCreated: consolidated.insights.length,
+    bytesPruned: consolidated.bytesRemoved,
+  };
+}
+```
+
+---
+
+## Retrofit Plan
+
+### Phase 1: Interface Extraction (2-3 days)
+
+**Goal:** Define interfaces without changing existing behavior
+
+1. Create `packages/queue/src/types.ts` with `QueueAdapter` interface
+2. Create `packages/storage/src/types.ts` with `StorageAdapter` interface
+3. Create `packages/memory/src/types.ts` with `MemoryAdapter` interface (refactor existing)
+4. Add adapter registry pattern to each package
+5. No breaking changes — existing code continues to work
+
+### Phase 2: Refactor Existing to Adapters (3-5 days)
+
+**Goal:** Move existing implementations behind adapters
+
+#### 2.1 Queue Refactor
+
+1. Rename `packages/queue/src/queue.ts` → `packages/queue/src/adapters/bullmq.ts`
+2. Create `packages/queue/src/index.ts` to export factory function
+3. Factory function reads config, instantiates correct adapter
+4. Update gateway imports to use factory
+
+#### 2.2 Storage Refactor
+
+1. Create `packages/storage/` (new package)
+2. Move Drizzle logic to `packages/storage/src/adapters/postgres.ts`
+3. Create SQLite adapter in `packages/storage/src/adapters/sqlite.ts`
+4. Update gateway to use storage factory
+5. Deprecate direct `@mosaicstack/db` imports
+
+#### 2.3 Memory Refactor
+
+1. Extract existing logic to `packages/memory/src/adapters/pgvector.ts`
+2. Create keyword adapter in `packages/memory/src/adapters/keyword.ts`
+3. Update vector-store.ts to be adapter-agnostic
+
+### Phase 3: Local Tier Implementation (2-3 days)
+
+**Goal:** Zero-dependency baseline
+
+1. Implement `packages/queue/src/adapters/local.ts` (in-process + JSON persistence)
+2. Implement `packages/storage/src/adapters/files.ts` (JSON + MD via MarkdownDB)
+3. Implement `packages/memory/src/adapters/keyword.ts` (TF-IDF search)
+4. Add `packages/dream/` for consolidation cycle
+5. Wire up local tier in gateway startup
+
+### Phase 4: Configuration System (1-2 days)
+
+**Goal:** Runtime backend selection
+
+1. Create `packages/config/src/storage.ts` for storage configuration
+2. Add `mosaic.config.ts` schema with storage tier settings
+3. Update gateway to read config on startup
+4. Add `mosaic gateway configure` CLI command
+5. Add tier migration commands (`mosaic gateway upgrade`)
+
+### Phase 5: Testing & Documentation (2-3 days)
+
+1. Unit tests for each adapter
+2. Integration tests for factory pattern
+3. Migration tests (local → team)
+4. Update README and architecture docs
+5. Add configuration guide
+
+---
+
+## File Changes Summary
+
+### New Files
+
+```
+packages/
+├── config/
+│   └── src/
+│       ├── storage.ts          # Storage config schema
+│       └── index.ts
+├── dream/                    # NEW: Dream mode consolidation
+│   ├── src/
+│   │   ├── index.ts
+│   │   ├── orient.ts
+│   │   ├── gather.ts
+│   │   ├── consolidate.ts
+│   │   └── prune.ts
+│   └── package.json
+├── queue/
+│   └── src/
+│       ├── types.ts           # NEW: QueueAdapter interface
+│       ├── index.ts           # NEW: Factory function
+│       └── adapters/
+│           ├── bullmq.ts      # MOVED from queue.ts
+│           └── local.ts       # NEW: In-process adapter
+├── storage/                  # NEW: Storage abstraction
+│   ├── src/
+│   │   ├── types.ts           # StorageAdapter interface
+│   │   ├── index.ts           # Factory function
+│   │   └── adapters/
+│   │       ├── postgres.ts    # MOVED from @mosaicstack/db
+│   │       ├── sqlite.ts      # NEW: SQLite adapter
+│   │       └── files.ts       # NEW: JSON/MD adapter
+│   └── package.json
+└── memory/
+    └── src/
+        ├── types.ts           # UPDATED: MemoryAdapter interface
+        ├── index.ts           # UPDATED: Factory function
+        └── adapters/
+            ├── pgvector.ts    # EXTRACTED from existing code
+            ├── sqlite-vec.ts  # NEW: SQLite with vectors
+            └── keyword.ts     # NEW: TF-IDF search
+```
+
+### Modified Files
+
+```
+packages/
+├── db/                       # DEPRECATED: Logic moved to storage adapters
+├── queue/
+│   └── src/
+│       └── queue.ts          # → adapters/bullmq.ts
+├── memory/
+│   ├── src/
+│   │   ├── memory.ts         # → use factory
+│   │   ├── insights.ts       # → use factory
+│   │   └── preferences.ts    # → use factory
+│   └── package.json          # Remove pgvector from dependencies
+└── gateway/
+    └── src/
+        ├── database/
+        │   └── database.module.ts  # Update to use storage factory
+        ├── memory/
+        │   └── memory.module.ts  # Update to use memory factory
+        └── queue/
+            └── queue.module.ts  # Update to use queue factory
+```
+
+---
+
+## Breaking Changes
+
+1. **`@mosaicstack/db`** → **`@mosaicstack/storage`** (with migration guide)
+2. Direct `ioredis` imports → Use `@mosaicstack/queue` factory
+3. Direct `pgvector` queries → Use `@mosaicstack/memory` factory
+4. Gateway startup now requires storage config (defaults to local)
+
+## Non-Breaking Migration Path
+
+1. Existing deployments with Postgres/Valkey continue to work (default config)
+2. New deployments can choose local tier
+3. Migration commands available when ready to upgrade
+
+---
+
+## Success Criteria
+
+- [ ] Local tier runs with zero external dependencies
+- [ ] All three tiers (local, team, enterprise) work correctly
+- [ ] Factory pattern correctly selects backend at runtime
+- [ ] Migration from local → team preserves all data
+- [ ] Dream mode consolidates 24h of sessions
+- [ ] Documentation covers all three tiers and migration paths
+- [ ] All existing tests pass
+- [ ] New adapters have >80% coverage