chore: consolidate new foundation and archive v1 (#1495)
This commit is contained in:
@@ -0,0 +1,755 @@
|
||||
# Durable tmux Fleet Installation Plan
|
||||
|
||||
> **For Mosaic/Hermes:** This is an implementation plan for making the tmux-backed Mosaic software-factory fleet durable on this server and reusable in generic Mosaic Stack installs. Keep local USC/Mosaic defaults in profiles; keep framework behavior customizable.
|
||||
|
||||
**Goal:** Add a supported Mosaic tmux-fleet installation path: holder-owned tmux server, per-agent reusable sessions, reliable send/reset/status tools, local roster customization, and a documented cutover for this server.
|
||||
|
||||
**Architecture:** Mosaic should ship generic tmux fleet primitives in the framework, then layer local rosters through configuration. The holder service owns the tmux socket; each agent service joins the holder-owned server and runs `mosaic yolo <runtime>`. The orchestrator addresses agents through `mosaic agent ...` abstractions so tmux can later be replaced by Matrix-backed agent comms without changing mission flow.
|
||||
|
||||
**Reference:** AI Guide `playbooks/tmux-fleet.md` at commit `2a0b0b5` documents the organization-neutral holder-service pattern, exact-match `=<name>` stop targets, and coupled-server cutover/verification sequence. The Stack implementation should treat that as the lifecycle model and keep concrete Mosaic unit/tooling details here.
|
||||
|
||||
**Tech Stack:** Bash, tmux, user systemd units, Mosaic CLI/framework installer, JSON/YAML roster config, existing `packages/mosaic/framework/tools/tmux/{agent-send.sh,send-message.sh}`.
|
||||
|
||||
---
|
||||
|
||||
## Current evidence from this server
|
||||
|
||||
Checked 2026-06-19:
|
||||
|
||||
- Host: `W-jarvis`
|
||||
- User: `jarvis`
|
||||
- tmux: `/usr/bin/tmux`, version `3.4`
|
||||
- user systemd: active
|
||||
- existing tmux sessions: `ai-bma-0`, `dyor-1`, `melaniewoltje-3`, `sage-2`
|
||||
- existing Mosaic runtime: `/home/jarvis/.npm-global/bin/mosaic`, version `0.0.31`
|
||||
- installed `~/.config/mosaic/tools/tmux` was not present even though the stack repo contains `packages/mosaic/framework/tools/tmux/`
|
||||
|
||||
Implication: do not kill the current tmux server casually. This server has active ad-hoc/service sessions. The durable fleet cutover must be planned, with either a separate socket first or a scheduled fleet recycle.
|
||||
|
||||
## Design decisions
|
||||
|
||||
### 1. Generic framework, local profile
|
||||
|
||||
The Mosaic framework should ship:
|
||||
|
||||
- systemd unit templates;
|
||||
- tmux fleet CLI wrappers;
|
||||
- roster schema and examples;
|
||||
- install/enable/status/reset commands;
|
||||
- docs and verification scripts.
|
||||
|
||||
Local environments should provide:
|
||||
|
||||
- agent names;
|
||||
- runtime per slot (`claude`, `pi`, `codex`, etc.);
|
||||
- default role class;
|
||||
- launch directory;
|
||||
- optional kickstart prompt;
|
||||
- model/provider hints;
|
||||
- transport selection (`tmux` now, `matrix` later).
|
||||
|
||||
Do not bake the USC roster into generic install code. Ship it as an example profile.
|
||||
|
||||
### 2. Durable sessions, disposable task context
|
||||
|
||||
Session names are durable operational addresses. Task persona is disposable. Reusable worker slots should be reset with `/clear` or `/new` and then receive a fresh task kickstart.
|
||||
|
||||
Persistent/semi-persistent personas:
|
||||
|
||||
- lead orchestrator;
|
||||
- final/adversarial reviewer;
|
||||
- architecture/enhancement lane.
|
||||
|
||||
Disposable slots:
|
||||
|
||||
- implementers;
|
||||
- ordinary reviewers;
|
||||
- security reviewers unless actively holding a security mission.
|
||||
|
||||
### 3. Transport abstraction now
|
||||
|
||||
Add commands around tmux instead of calling tmux directly from orchestration:
|
||||
|
||||
```bash
|
||||
mosaic agent send <agent> --message "..."
|
||||
mosaic agent status [--json]
|
||||
mosaic agent reset <agent> [--clear|--new]
|
||||
mosaic agent roster [--json]
|
||||
mosaic fleet install|start|stop|restart|status|verify
|
||||
```
|
||||
|
||||
Today these call tmux/systemd. Later the same command surface can target Matrix or per-agent gateways.
|
||||
|
||||
### 4. Avoid shared-server ownership bug
|
||||
|
||||
Use the AI Guide holder pattern:
|
||||
|
||||
```text
|
||||
mosaic-tmux-holder.service owns the tmux server/socket
|
||||
mosaic-agent@<name>.service joins the existing holder-owned socket
|
||||
ExecStop kills only session =<name>
|
||||
```
|
||||
|
||||
Use exact tmux targets: `=<session>`.
|
||||
|
||||
### 5. Prefer separate named socket for Mosaic factory
|
||||
|
||||
To avoid disturbing existing tmux work, the default fleet should use a named socket such as:
|
||||
|
||||
```text
|
||||
$XDG_RUNTIME_DIR/mosaic-factory.tmux
|
||||
```
|
||||
|
||||
or tmux socket name:
|
||||
|
||||
```bash
|
||||
tmux -L mosaic-factory ...
|
||||
```
|
||||
|
||||
This avoids collision with ordinary `tmux ls` sessions. The send tools need socket support.
|
||||
|
||||
---
|
||||
|
||||
## Target USC-style roster example
|
||||
|
||||
Ship as example only, not default:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
transport: tmux
|
||||
tmux:
|
||||
socket_name: mosaic-factory
|
||||
holder_session: _holder
|
||||
working_directory: ~/src
|
||||
agents:
|
||||
- name: mos-claude
|
||||
runtime: claude
|
||||
class: orchestrator
|
||||
model_hint: Claude Opus
|
||||
persistent_persona: true
|
||||
- name: coder0
|
||||
runtime: claude
|
||||
class: implementer
|
||||
model_hint: Claude Opus
|
||||
reset_between_tasks: true
|
||||
- name: coder1
|
||||
runtime: claude
|
||||
class: implementer
|
||||
model_hint: Claude Opus
|
||||
reset_between_tasks: true
|
||||
- name: coder2
|
||||
runtime: pi
|
||||
class: implementer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: coder3
|
||||
runtime: pi
|
||||
class: implementer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: coder4
|
||||
runtime: claude
|
||||
class: implementer
|
||||
model_hint: Claude Opus
|
||||
reset_between_tasks: true
|
||||
- name: coder5
|
||||
runtime: claude
|
||||
class: implementer
|
||||
model_hint: Claude Opus
|
||||
reset_between_tasks: true
|
||||
- name: enhance
|
||||
runtime: claude
|
||||
class: enhancer
|
||||
model_hint: Claude Opus
|
||||
persistent_persona: semi
|
||||
- name: rev0
|
||||
runtime: pi
|
||||
class: reviewer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: rev1
|
||||
runtime: pi
|
||||
class: reviewer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: secrev0
|
||||
runtime: pi
|
||||
class: security_reviewer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: secrev1
|
||||
runtime: pi
|
||||
class: security_reviewer
|
||||
model_hint: Pi GPT-5.5
|
||||
reset_between_tasks: true
|
||||
- name: ultron
|
||||
runtime: pi
|
||||
class: final_reviewer
|
||||
model_hint: Pi GPT-5.5
|
||||
persistent_persona: semi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Confirm install surfaces
|
||||
|
||||
### Task 0.1: Inspect installer copy behavior
|
||||
|
||||
**Objective:** Confirm how framework files under `packages/mosaic/framework/` become installed under `~/.config/mosaic/`.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: `tools/install.sh`
|
||||
- Read: `packages/mosaic/framework/install.sh`
|
||||
- Read: `packages/mosaic/src/runtime/install-manifest.ts`
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Verify `packages/mosaic/framework/install.sh` rsyncs `tools/tmux`.
|
||||
2. Verify whether npm-packaged installs include `framework/tools/tmux`.
|
||||
3. Confirm whether installed hosts should run `mosaic update`, `bash tools/install.sh`, or `packages/mosaic/framework/install.sh` to receive new tmux tools.
|
||||
4. Record exact propagation command in docs.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
bash packages/mosaic/framework/install.sh --help || true
|
||||
npm pack --dry-run --json | jq '.[0].files[].path' | grep 'framework/tools/tmux'
|
||||
```
|
||||
|
||||
Expected: tmux tools are included in installable package or packaging fix is identified.
|
||||
|
||||
### Task 0.2: Inspect current yolo launch semantics
|
||||
|
||||
**Objective:** Confirm `mosaic yolo claude` and `mosaic yolo pi` accept optional initial prompt text and behave well under systemd/tmux.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: `packages/mosaic/src/**`
|
||||
- Read: `packages/mosaic/framework/runtime/claude/RUNTIME.md`
|
||||
- Read: `packages/mosaic/framework/runtime/pi/RUNTIME.md`
|
||||
|
||||
**Verification commands:**
|
||||
|
||||
```bash
|
||||
mosaic yolo claude --help
|
||||
mosaic yolo pi --help
|
||||
```
|
||||
|
||||
Expected: a systemd `ExecStart` can launch the runtime either with no prompt or with a kickstart prompt file/string.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Framework tmux primitives
|
||||
|
||||
### Task 1.1: Add socket support to send tools
|
||||
|
||||
**Objective:** Allow `agent-send.sh` and `send-message.sh` to target a named Mosaic tmux socket without affecting default tmux sessions.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/mosaic/framework/tools/tmux/send-message.sh`
|
||||
- Modify: `packages/mosaic/framework/tools/tmux/agent-send.sh`
|
||||
- Modify: `packages/mosaic/framework/tools/tmux/README.md`
|
||||
- Test: `packages/mosaic/framework/tools/tmux/test-send-message.sh` (new)
|
||||
|
||||
**Design:**
|
||||
|
||||
Add optional flags:
|
||||
|
||||
```bash
|
||||
-L SOCKET_NAME # tmux -L socket name
|
||||
-SOCKET PATH # optional later if needed; avoid conflict with existing -S source label in agent-send
|
||||
```
|
||||
|
||||
Because `agent-send.sh` already uses `-S` for source label, prefer `-L` for socket name and `-T` or `--socket-path` only if long-option parsing is added.
|
||||
|
||||
**Implementation notes:**
|
||||
|
||||
- Build a tmux command array:
|
||||
|
||||
```bash
|
||||
tmux_cmd=(tmux)
|
||||
if [ -n "$SOCKET_NAME" ]; then tmux_cmd+=( -L "$SOCKET_NAME" ); fi
|
||||
```
|
||||
|
||||
- Replace raw `tmux ...` calls with `"${tmux_cmd[@]}" ...`.
|
||||
- Pass `-L` through remote ssh invocation.
|
||||
- Include socket name in verbose output.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
tmux -L mosaic-test new-session -d -s target 'cat'
|
||||
packages/mosaic/framework/tools/tmux/send-message.sh -L mosaic-test -t target -m 'hello'
|
||||
tmux -L mosaic-test capture-pane -t target -p | grep hello
|
||||
tmux -L mosaic-test kill-server
|
||||
```
|
||||
|
||||
Expected: message lands in the named socket session; default `tmux ls` is untouched.
|
||||
|
||||
### Task 1.2: Add exact target validation helper
|
||||
|
||||
**Objective:** Prevent accidental prefix targeting in all tmux fleet operations.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/mosaic/framework/tools/tmux/_lib.sh`
|
||||
- Modify: `send-message.sh`
|
||||
- Modify: `agent-send.sh`
|
||||
|
||||
**Behavior:**
|
||||
|
||||
- For session-only agent names, normalize target to `=<name>` before kill/status/reset operations.
|
||||
- For explicit pane targets like `session:window.pane`, allow as advanced path but document the risk.
|
||||
|
||||
**Verification:**
|
||||
|
||||
Create sessions `agent` and `agent0`; verify killing/resetting `agent` does not affect `agent0`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — systemd unit templates
|
||||
|
||||
### Task 2.1: Add holder service template
|
||||
|
||||
**Objective:** Ship a user systemd unit template that owns the Mosaic factory tmux server.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service`
|
||||
- Create: `packages/mosaic/framework/tools/fleet/install-user-units.sh`
|
||||
|
||||
**Unit shape:**
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Mosaic tmux fleet holder
|
||||
Documentation=https://git.mosaicstack.dev/mosaicstack/aiguide
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
Environment=MOSAIC_TMUX_SOCKET=mosaic-factory
|
||||
ExecStart=/usr/bin/tmux -L ${MOSAIC_TMUX_SOCKET} new-session -d -s _holder 'while true; do sleep 3600; done'
|
||||
ExecStop=-/usr/bin/tmux -L ${MOSAIC_TMUX_SOCKET} kill-server
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
```
|
||||
|
||||
**Important:** systemd environment expansion in `ExecStart` is limited. Verify syntax; if `%E`/environment expansion is awkward, generate concrete units from config instead of relying on dynamic expansion.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
systemd-analyze --user verify ~/.config/systemd/user/mosaic-tmux-holder.service
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user start mosaic-tmux-holder.service
|
||||
tmux -L mosaic-factory ls | grep _holder
|
||||
```
|
||||
|
||||
### Task 2.2: Add agent service template
|
||||
|
||||
**Objective:** Ship a user systemd template that starts one configured agent slot.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/mosaic/framework/systemd/user/[email protected]`
|
||||
- Modify: `packages/mosaic/framework/tools/fleet/install-user-units.sh`
|
||||
|
||||
**Unit shape:**
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Mosaic agent session %i
|
||||
Requires=mosaic-tmux-holder.service
|
||||
After=mosaic-tmux-holder.service
|
||||
PartOf=mosaic-tmux-holder.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
WorkingDirectory=%h/src
|
||||
Environment=MOSAIC_TMUX_SOCKET=mosaic-factory
|
||||
ExecStart=/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" new-session -d -s "%i" "mosaic yolo $(mosaic fleet runtime %i)"'
|
||||
ExecStop=-/usr/bin/tmux -L mosaic-factory kill-session -t '=%i'
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
```
|
||||
|
||||
**Design warning:** command substitution in unit files can become brittle. Prefer a generated per-agent EnvironmentFile:
|
||||
|
||||
```text
|
||||
~/.config/mosaic/fleet/agents/coder0.env
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```bash
|
||||
MOSAIC_AGENT_NAME=coder0
|
||||
MOSAIC_AGENT_RUNTIME=claude
|
||||
MOSAIC_AGENT_WORKDIR=/home/jarvis/src
|
||||
MOSAIC_TMUX_SOCKET=mosaic-factory
|
||||
```
|
||||
|
||||
Then `ExecStart` calls a wrapper:
|
||||
|
||||
```bash
|
||||
~/.config/mosaic/tools/fleet/start-agent-session.sh
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
systemd-analyze --user verify ~/.config/systemd/user/[email protected]
|
||||
systemctl --user start [email protected]
|
||||
tmux -L mosaic-factory has-session -t '=coder0'
|
||||
systemctl --user restart [email protected]
|
||||
```
|
||||
|
||||
Expected: holder server PID remains unchanged; only `coder0` session recycles.
|
||||
|
||||
### Task 2.3: Add start-agent wrapper
|
||||
|
||||
**Objective:** Keep systemd units simple by moving config lookup and launch command construction into a script.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/mosaic/framework/tools/fleet/start-agent-session.sh`
|
||||
|
||||
**Behavior:**
|
||||
|
||||
Inputs:
|
||||
|
||||
```bash
|
||||
start-agent-session.sh <agent-name>
|
||||
```
|
||||
|
||||
Reads:
|
||||
|
||||
```text
|
||||
$MOSAIC_HOME/fleet/agents/<agent-name>.env
|
||||
```
|
||||
|
||||
Starts:
|
||||
|
||||
```bash
|
||||
tmux -L "$MOSAIC_TMUX_SOCKET" new-session -d -s "$MOSAIC_AGENT_NAME" -c "$MOSAIC_AGENT_WORKDIR" "mosaic yolo $MOSAIC_AGENT_RUNTIME"
|
||||
```
|
||||
|
||||
Guardrails:
|
||||
|
||||
- fail if runtime is empty;
|
||||
- fail if workdir does not exist;
|
||||
- no duplicate sessions unless `--replace` is passed;
|
||||
- exact session names only.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — roster config and CLI wrappers
|
||||
|
||||
### Task 3.1: Add fleet config schema and examples
|
||||
|
||||
**Objective:** Define customizable install-time roster without hardcoding USC.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `packages/mosaic/framework/fleet/roster.schema.json`
|
||||
- Create: `packages/mosaic/framework/fleet/examples/minimal.yaml`
|
||||
- Create: `packages/mosaic/framework/fleet/examples/usc-software-factory.yaml`
|
||||
- Create: `packages/mosaic/framework/fleet/README.md`
|
||||
|
||||
**Schema concepts:**
|
||||
|
||||
- `transport`: `tmux` now; `matrix` later.
|
||||
- `tmux.socket_name`
|
||||
- `tmux.holder_session`
|
||||
- `defaults.working_directory`
|
||||
- `agents[].name`
|
||||
- `agents[].runtime`
|
||||
- `agents[].class`
|
||||
- `agents[].model_hint`
|
||||
- `agents[].persistent_persona`
|
||||
- `agents[].reset_between_tasks`
|
||||
- `agents[].kickstart_template`
|
||||
|
||||
**Verification:**
|
||||
|
||||
Use `jq` for JSON examples or add a small Python/YAML validator if YAML is chosen. If no YAML parser is guaranteed, store examples as JSON or support both with Python stdlib JSON first.
|
||||
|
||||
### Task 3.2: Add `mosaic fleet` commands
|
||||
|
||||
**Objective:** Provide operator-safe commands for install/status/start/stop/restart/verify.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `packages/mosaic/src/cli.ts` or the current commander entrypoint.
|
||||
- Create scripts under: `packages/mosaic/framework/tools/fleet/`
|
||||
|
||||
**Commands:**
|
||||
|
||||
```bash
|
||||
mosaic fleet init --profile minimal|usc --write
|
||||
mosaic fleet install-systemd
|
||||
mosaic fleet start [agent]
|
||||
mosaic fleet stop [agent]
|
||||
mosaic fleet restart [agent]
|
||||
mosaic fleet status --json
|
||||
mosaic fleet verify
|
||||
```
|
||||
|
||||
**Implementation path:**
|
||||
|
||||
Start by wrapping framework shell scripts from the TypeScript CLI. Do not overbuild a TypeScript service manager in the first pass.
|
||||
|
||||
### Task 3.3: Add `mosaic agent` commands
|
||||
|
||||
**Objective:** Provide transport-stable per-agent operations.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: Mosaic CLI entrypoint.
|
||||
- Create: `packages/mosaic/framework/tools/agent/` or reuse `tools/tmux` + `tools/fleet`.
|
||||
|
||||
**Commands:**
|
||||
|
||||
```bash
|
||||
mosaic agent roster [--json]
|
||||
mosaic agent status [agent] [--json]
|
||||
mosaic agent send <agent> --message "..."
|
||||
mosaic agent reset <agent> --clear|--new
|
||||
mosaic agent tail <agent> [-n 80]
|
||||
```
|
||||
|
||||
**Reset behavior:**
|
||||
|
||||
For tmux transport, `reset --clear` sends `/clear` then Enter through `send-message.sh`.
|
||||
|
||||
For Claude/Pi differences, keep reset command configurable per runtime:
|
||||
|
||||
```yaml
|
||||
runtimes:
|
||||
claude:
|
||||
reset_command: /clear
|
||||
pi:
|
||||
reset_command: /new
|
||||
```
|
||||
|
||||
If a runtime does not support a known reset command, restart the service and send a fresh kickstart.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — this-server rollout strategy
|
||||
|
||||
### Task 4.1: Install on separate socket first
|
||||
|
||||
**Objective:** Prove the holder pattern without disturbing existing sessions.
|
||||
|
||||
**Commands after implementation lands locally:**
|
||||
|
||||
```bash
|
||||
mosaic fleet init --profile minimal --write
|
||||
mosaic fleet install-systemd
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user start mosaic-tmux-holder.service
|
||||
mosaic fleet verify
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `tmux -L mosaic-factory ls` shows `_holder`.
|
||||
- normal `tmux ls` still shows existing sessions unchanged.
|
||||
|
||||
### Task 4.2: Start one canary agent
|
||||
|
||||
**Objective:** Validate single-agent start/restart isolation.
|
||||
|
||||
Use a harmless canary first, not the full fleet.
|
||||
|
||||
Example roster addition:
|
||||
|
||||
```yaml
|
||||
- name: canary-pi
|
||||
runtime: pi
|
||||
class: canary
|
||||
working_directory: /home/jarvis/src
|
||||
```
|
||||
|
||||
Commands:
|
||||
|
||||
```bash
|
||||
systemctl --user start [email protected]
|
||||
SRV=$(tmux -L mosaic-factory display-message -p '#{pid}')
|
||||
systemctl --user restart [email protected]
|
||||
test "$SRV" = "$(tmux -L mosaic-factory display-message -p '#{pid}')"
|
||||
tmux -L mosaic-factory ls
|
||||
```
|
||||
|
||||
Expected: holder PID unchanged; `_holder` remains; `canary-pi` recreated.
|
||||
|
||||
### Task 4.3: Configure local Mosaic factory roster
|
||||
|
||||
**Objective:** Create the actual local roster for this server after canary passes.
|
||||
|
||||
Do not assume USC exact roster is desired here. Create a local profile such as:
|
||||
|
||||
```text
|
||||
~/.config/mosaic/fleet/roster.yaml
|
||||
```
|
||||
|
||||
Initial local recommendation:
|
||||
|
||||
- `mos-claude` orchestrator
|
||||
- `coder0` / `coder1` implementers
|
||||
- `rev0` reviewer
|
||||
- `secrev0` security reviewer
|
||||
- `ultron` final/adversarial reviewer
|
||||
|
||||
Scale to full USC-style pool only after resource/budget behavior is understood.
|
||||
|
||||
### Task 4.4: Cut over existing ad-hoc tmux sessions only if desired
|
||||
|
||||
**Objective:** Avoid data loss.
|
||||
|
||||
Existing sessions on this server are not on the proposed `mosaic-factory` socket. They can remain untouched. If we later want them under Mosaic fleet control:
|
||||
|
||||
1. list sessions;
|
||||
2. capture logs/handoffs;
|
||||
3. stop old processes intentionally;
|
||||
4. recreate as configured `mosaic-agent@...` services;
|
||||
5. verify comms and state.
|
||||
|
||||
Do not run `tmux kill-server` on the default socket unless Jason explicitly approves that outage.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — docs and AI Guide backfill
|
||||
|
||||
### Task 5.1: Stack docs
|
||||
|
||||
**Objective:** Document install and customization for Mosaic Stack users.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/fleet/tmux-fleet.md` or `packages/mosaic/framework/tools/fleet/README.md`
|
||||
- Modify: top-level `README.md` if appropriate.
|
||||
|
||||
Must cover:
|
||||
|
||||
- what problem holder service solves;
|
||||
- install commands;
|
||||
- customization file;
|
||||
- example rosters;
|
||||
- reset/reuse lifecycle;
|
||||
- exact-target safety;
|
||||
- separate socket default;
|
||||
- Matrix migration path.
|
||||
|
||||
### Task 5.2: AI Guide docs
|
||||
|
||||
**Objective:** Keep generic guidance in AI Guide and implementation details in Stack.
|
||||
|
||||
**Files in `mosaicstack/aiguide`:**
|
||||
|
||||
- Update: `playbooks/tmux-fleet.md` with named socket, roster/profile, and resettable-slot pattern.
|
||||
- Add or update: `reference/agent-role-matrix.md` if PR #5 lands.
|
||||
|
||||
Do not put Mosaic install commands as the only path in AI Guide. Present them as one implementation profile.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Matrix migration seam
|
||||
|
||||
### Task 6.1: Add transport enum but implement tmux only
|
||||
|
||||
**Objective:** Avoid hardcoding tmux into orchestration semantics.
|
||||
|
||||
Roster:
|
||||
|
||||
```yaml
|
||||
transport: tmux
|
||||
```
|
||||
|
||||
Future:
|
||||
|
||||
```yaml
|
||||
transport: matrix
|
||||
matrix:
|
||||
homeserver: https://matrix.example
|
||||
room_prefix: mosaic-factory
|
||||
```
|
||||
|
||||
### Task 6.2: Define transport interface docs
|
||||
|
||||
**Objective:** Make Matrix plugin work a transport swap, not a rewrite.
|
||||
|
||||
Minimum operations:
|
||||
|
||||
```text
|
||||
send(agent, message)
|
||||
reset(agent, mode)
|
||||
status(agent)
|
||||
tail(agent)
|
||||
listAgents()
|
||||
```
|
||||
|
||||
Any tmux-specific concept must stay below this line.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
The implementation is complete when:
|
||||
|
||||
- `mosaic fleet init` can write a minimal roster.
|
||||
- `mosaic fleet install-systemd` installs holder and agent units without hand editing.
|
||||
- `mosaic fleet start` starts the holder and configured agents on a named tmux socket.
|
||||
- Restarting one `[email protected]` does not change holder server PID or kill sibling sessions.
|
||||
- `mosaic agent send` can deliver a message to a named agent with a self-identifying preamble.
|
||||
- `mosaic agent reset` can clear/new a reusable slot and send a fresh kickstart.
|
||||
- `mosaic fleet verify` proves holder ownership, exact-target safety, and per-agent restart isolation.
|
||||
- Existing default tmux sessions on this server are not disturbed by default install.
|
||||
- Docs explain generic customization and include USC-style roster only as an example.
|
||||
- AI Guide remains generic; Mosaic Stack docs carry the concrete install path.
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
| --------------------------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| Killing existing tmux sessions | Use named `mosaic-factory` socket; no default `tmux kill-server`. |
|
||||
| systemd unit quoting/env expansion bugs | Move logic into shell wrappers; verify with `systemd-analyze --user verify`. |
|
||||
| Runtime reset command mismatch | Make reset command runtime-configurable; fallback to service restart + kickstart. |
|
||||
| Tool install drift | Ensure npm package includes framework tmux/fleet tools; add packaging test. |
|
||||
| Mosaic-specific assumptions leak into generic guide | Keep USC roster as example profile; AI Guide documents pattern/options. |
|
||||
| Matrix migration blocked by tmux coupling | Add `mosaic agent` abstraction now; keep tmux details below transport layer. |
|
||||
|
||||
## Suggested first PR split
|
||||
|
||||
1. **PR A — tmux tool hardening**
|
||||
- socket support;
|
||||
- exact target helpers;
|
||||
- tests/docs.
|
||||
|
||||
2. **PR B — fleet systemd primitives**
|
||||
- holder unit;
|
||||
- agent unit;
|
||||
- start-agent wrapper;
|
||||
- install-user-units script;
|
||||
- verify script.
|
||||
|
||||
3. **PR C — roster and CLI**
|
||||
- roster schema/examples;
|
||||
- `mosaic fleet ...` commands;
|
||||
- `mosaic agent ...` commands.
|
||||
|
||||
4. **PR D — local rollout and docs**
|
||||
- local roster for this server;
|
||||
- run canary;
|
||||
- document verification evidence;
|
||||
- update AI Guide with generic lessons.
|
||||
|
||||
## Immediate next action
|
||||
|
||||
Implement PR A first. It is low-risk, improves existing tools, and is required for a safe named-socket rollout on this server.
|
||||
@@ -0,0 +1,73 @@
|
||||
# 2026-08-17 — Fleet identity, comms delivery, and the ~/.mosaic tree (continuation record)
|
||||
|
||||
> **Status:** active continuation record | **Owner:** Jason (rulings) / fleet (delivery) | **Created:** 2026-08-17, sb-it-1-dt session with Jarvis (jarvis-brain)
|
||||
> **Audience:** the homelab agents continuing this effort tonight. Read this whole file before acting; it supersedes nothing but preserves structure and decisions that must not be lost.
|
||||
|
||||
---
|
||||
|
||||
## Why this exists
|
||||
|
||||
A session on sb-it-1-dt (2026-08-17) produced three architecture decisions (two awaiting Jason's ruling), one incident postmortem (#1295), interim guardrail edits in the user-owned `~/.mosaic/` contract tree, and one new tool (`ensure-watcher.sh`). The work spans jarvis-brain (P0, not retained) and this repo (the product). **This file is the stack-side anchor so continuation does not depend on jarvis-brain surviving.**
|
||||
|
||||
## 1. The `~/.mosaic` tree model — as-built, preserve this structure
|
||||
|
||||
Three-tree split (this is design intent, not accident; keep it through all framework work):
|
||||
|
||||
| Tree | Owner | Rule |
|
||||
| ------------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `~/.config/mosaic/` | framework | upgrade-managed templates; NEVER user-edited; `mosaic upgrade` may overwrite |
|
||||
| `~/.mosaic/` | user | working contracts, guides, fleet agents; upgrades reconcile with **deny-wins** (user edits never overwritten) |
|
||||
| repo satellites | repos | bootstrapped per-repo `.mosaic/` state |
|
||||
|
||||
As-built inventory of `~/.mosaic` on sb-it-1-dt (2026-08-17):
|
||||
|
||||
- **Contract core:** `CONSTITUTION.md` (L0 law), `AGENTS.md` (dispatcher + guide router + Fleet Comms Watcher requirement), `SOUL.md` (generic base for ALL fleet agents, zero persona — includes the new **Fleet Boundaries** section), `STANDARDS.md` (universal standards — includes new **session identity** + **comms watcher hygiene** sections), `SYSTEM.md` (pure communication contract, byte-identical to jarvis-brain's prompt-testing `sr_opus_5_system_prompt.md`), `USER.md`, `TOOLS.md`.
|
||||
- **`guides/`** — user-owned working copies (E2E-DELIVERY, ORCHESTRATOR(+PROTOCOL,+LEARNINGS), WAKE-DOCTRINE, VAULT-SECRETS, etc.).
|
||||
- **`fleet/agents/`** — the per-agent store (this is MOSAIC-D-002's substrate, already in use):
|
||||
- real agent dirs: `fargo/`, `orchestrator/`, `probe/`, `vision/`, `weekly-update/` — shape: `profile.json` (harness/account/overlay pointer) + `overlay.json` + `SOUL.md` (persona) + `scratch/` `work/` `notes/` subdirs (hygiene rules in root SOUL.md)
|
||||
- `*.env.generated` launch overlays: `luna` `sol` `terra` (carry `MOSAIC_AGENT_NAME`, `_CLASS`, `_RUNTIME`, `_MODEL`, `_REASONING`, `_TOOL_POLICY`) — these are the mosaic-fleet seat launch envs; `inbox.env`, `itops.env` also present
|
||||
- `probe/` is the validated layout proof: auth-bundle symlink chain, per-agent sessions, plugin-store symlink (from 2026-08-07)
|
||||
- **`auth/`, `config/`, `memory/`, `plugins/`, `skills/`, `skills-local/`** — per-tree copies/links for runtime isolation.
|
||||
- Related but outside the tree: watcher units at `~/.config/systemd/user/<agent>-comms-watcher.service`; watcher seen-state at `~/.local/state/comms-watcher-<agent>/`.
|
||||
|
||||
## 2. Decisions register (2026-08-17 session)
|
||||
|
||||
Full strict records live in jarvis-brain `docs/decisions/mosaic-stack/` (render on its dashboard); both are **Pending Jason's ruling**. Summaries so the content survives P0:
|
||||
|
||||
- **MOSAIC-D-001 — SYSTEM.md as canonical harness system prompt.** Static core (Constitution+AGENTS+USER+overlays) in one tracked file; launcher renders dynamic tail (mission/PRD/fleet/persona). Delivery: `--append-system-prompt` (repeatable) for pi/claude; symlinked core file for codex (`$CODEX_HOME/instructions.md`) and opencode (`AGENTS.md`); their dynamic tail via initial prompt (needs live verification). Static-first order is the cache win. `SYSTEM.md` in `~/.mosaic` today is the communication-contract file — D-001's SYSTEM.md is the broader composition; naming to reconcile at implementation.
|
||||
- **MOSAIC-D-002 — per-agent harness homes + mechanical profiles.** Launch with targeted config-dir env vars (e.g. `PI_CODING_AGENT_DIR=~/.mosaic/fleet/agents/<name>/pi`), NOT literal HOME. SOUL.md identity mechanically generated from roster (single writer; kills the hand-copy drift measured in `agents/vision/SOUL.md` on jarvis-brain: declared Jarvis, answered Vision). Composes: SYSTEM core → per-agent SOUL → dynamic tail. `MOSAIC_AGENT_NAME` stays load-bearing for comms.
|
||||
- **Comms delivery tooling belongs in the STACK framework, not jarvis-brain** (decided in discussion; supersedes the interim placement). jarvis-brain keeps only the transport _data_ (`comms/` tree) while it lives. Agents launch from their own repos (terra from `~/src/stack` etc.) — delivery is transport-repo-relative, so this works; but every installed watcher unit's ExecStart currently points into `~/src/jarvis-brain/scripts/` — that dependency is the P0 trap to remove. Migration = move tools + regenerate units, in one step.
|
||||
- **Watcher provisioning is instantiation duty, never running-agent duty.** Interim landed as jarvis-brain `scripts/comms/ensure-watcher.sh` (idempotent ensure + `--status` boot check + interim identity warnings: missing target session, pane `MOSAIC_AGENT_NAME` mismatch via `/proc/<child>/environ`, bare-runtime NOTE). Framework move: fold into `mosaic agent --new` + fleet launch + `mosaic doctor` drift check.
|
||||
- **Prose guardrails landed (interim fences until mechanical fixes):** `~/.mosaic/SOUL.md` Fleet Boundaries (wrong-session tripwire; comms ownership; cross-agent investigation requires tasking) · `STANDARDS.md` session identity + watcher hygiene · `AGENTS.md` Fleet Comms Watcher requirement (P0-interim script path marked transitional).
|
||||
|
||||
## 3. Incident → #1295 (already tracked here)
|
||||
|
||||
`https://git.mosaicstack.dev/mosaicstack/stack/issues/1295` — docs-seat incident: cwd-keyed session files served three lives (dev chat → goals seat → 22 watcher injections into a wedged process); name-based watcher delivery with no identity verification; wedge after pi 0.84.1→0.84.2 update passes every liveness instrument. Proposed fixes enumerated there; provisioning follow-up in comment ID 23027.
|
||||
|
||||
## 4. Open work queue (suggested sequence)
|
||||
|
||||
1. **Comms tooling migration PR** (lane `next`): move `comms-watcher.sh`, `install-watcher.sh`, `ensure-watcher.sh` into the framework tree → deploy `~/.config/mosaic/tools/comms/`; regenerate existing units' ExecStart to framework paths (one-command sweep); keep `COMMS_WATCH_REPO` per-host config (points at a brain checkout until the queue transport lands). Reference: jarvis-brain commit `701c353b1`.
|
||||
2. **Ensure-on-instantiation**: `mosaic agent --new` / fleet launch call ensure semantics; `mosaic doctor` gains the drift check (`--status --all` semantics + `fred`'s hand-written unit as the known drift case; also note daphne/docs/happy/pepper/sanity/tiny/fargo currently have no watcher — cover or consciously exempt).
|
||||
3. **MOSAIC-D-002 implementation** (after ruling): per-agent homes via targeted env vars; roster-generated SOUL.md single-writer; extend the existing `*.env.generated` pattern; launch ledger keeps `config_home` audit.
|
||||
4. **MOSAIC-D-001 implementation** (after ruling): SYSTEM.md sourcing + per-harness delivery + `compose-contract` becomes render-core+tail with drift check; bench cache-ordering before/after (jarvis-brain `domains/software-dev/mosaic-stack/prompt-testing` has the bench).
|
||||
5. **Queue transport + forced separation** (longer term): supersedes watcher path; identity-verification and wedge-detection remain valid regardless of transport.
|
||||
6. **Docs inheritance**: jarvis-brain AGENTS.md's durable comms guidance (E7 pi-glyph delivery gotchas, capture-pane rules, comms protocol) must be inherited into stack docs before P0 retirement.
|
||||
|
||||
## 5. Rules for tonight's agents
|
||||
|
||||
- Lane: **`next`** only; nothing to `main` without Jason (standing ruling).
|
||||
- Attribution caveat #1280: Gitea/git identity from this host may misattribute (issue #1295 showed as created by `@mos-dt-0`); prefer per-invocation `git -c user.name=<seat>` and verify what the remote recorded.
|
||||
- Do not delete `sb-it-1-dt:docs]` (untracked file at repo root) — it is cited fleet-wide as incident evidence.
|
||||
- Edit user contracts in `~/.mosaic/`, never the templates in `~/.config/mosaic/`.
|
||||
- Do not restart other fleet seats unilaterally (goals/scrappy/sanity restart decisions are fred's/Jason's per the docs-seat report).
|
||||
|
||||
## 6. Artifact map
|
||||
|
||||
| Artifact | Where |
|
||||
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
|
||||
| Decision records D-001/D-002 (strict, pending ruling) | jarvis-brain `docs/decisions/mosaic-stack/2026-08-17_mosaic-d-00{1,2}_*.md` |
|
||||
| Incident issue + provisioning comment | stack #1295 + comment 23027 |
|
||||
| ensure-watcher.sh (reference implementation) | jarvis-brain `scripts/comms/ensure-watcher.sh` (commit `701c353b1`) |
|
||||
| Bench for prompt A/B (pi, thinking levels, footer-token semantics) | jarvis-brain `domains/software-dev/mosaic-stack/prompt-testing/` + `docs/reports/2026-08-17-prompt-testing-glm-bench.md` |
|
||||
| Launcher inspection basis | `@mosaicstack/mosaic` 0.0.49 `dist/commands/launch.js` (composeContract / ensureRuntimeConfig / harness-home isolation) |
|
||||
| Guardrail edits | `~/.mosaic/{SOUL,AGENTS,STANDARDS}.md` on sb-it-1-dt (2026-08-17 16:53–17:04) |
|
||||
@@ -0,0 +1,66 @@
|
||||
# CLI/TUI Tools Enhancement Scratchpad
|
||||
|
||||
## Objective
|
||||
|
||||
Add 5 capability areas to the Mosaic CLI/TUI + gateway agent:
|
||||
|
||||
1. Web search tools (multi-provider: Brave, DuckDuckGo, Tavily, SearXNG)
|
||||
2. File edit tool (`fs_edit_file` with targeted text replacement)
|
||||
3. MCP management TUI commands (`/mcp status`, `/mcp reconnect`, `/mcp servers`)
|
||||
4. File reference in chat (`@file` syntax + `/attach` command)
|
||||
5. Implement `/stop` to cancel streaming
|
||||
|
||||
## Plan
|
||||
|
||||
### 1. Web Search Tools (gateway agent tools)
|
||||
|
||||
- Create `apps/gateway/src/agent/tools/search-tools.ts`
|
||||
- Providers: Brave Search API, DuckDuckGo (HTML scraping), Tavily API, SearXNG (self-hosted)
|
||||
- Each provider activated by env var (BRAVE_API_KEY, TAVILY_API_KEY, SEARXNG_URL)
|
||||
- Tools: `web_search` (unified), `web_search_news` (news-specific)
|
||||
- Export from `apps/gateway/src/agent/tools/index.ts`
|
||||
- Wire into `agent.service.ts` buildToolsForSandbox
|
||||
|
||||
### 2. File Edit Tool (gateway agent tool)
|
||||
|
||||
- Add `fs_edit_file` to `apps/gateway/src/agent/tools/file-tools.ts`
|
||||
- Parameters: path, edits[{oldText, newText}] — same semantics as pi's Edit tool
|
||||
- Validates uniqueness of each oldText, applies all edits atomically
|
||||
|
||||
### 3. MCP Management Commands (TUI + gateway)
|
||||
|
||||
- Add gateway endpoints: GET /api/mcp/status, POST /api/mcp/:name/reconnect
|
||||
- Add TUI gateway-api.ts functions for MCP
|
||||
- Add gateway slash commands: /mcp (with subcommands status, reconnect, servers)
|
||||
- Register in command manifest from gateway
|
||||
- Handle in TUI via gateway command forwarding (already works)
|
||||
|
||||
### 4. File Reference in Chat (@file syntax)
|
||||
|
||||
- TUI-side: detect @path/to/file in input, read file contents, inline into message
|
||||
- Add `/attach <path>` local command as alternative
|
||||
- gateway-api.ts helper not needed — this is purely client-side pre-processing
|
||||
- Modify InputBar or sendMessage to expand @file references before sending
|
||||
|
||||
### 5. Implement /stop
|
||||
|
||||
- Add `abort` event to ClientToServerEvents in @mosaicstack/types
|
||||
- TUI sends abort event on /stop command
|
||||
- Gateway chat handler aborts the Pi session prompt
|
||||
- Update use-socket to support abort
|
||||
- Wire /stop in app.tsx
|
||||
|
||||
## Progress
|
||||
|
||||
- [x] 1. Web search tools
|
||||
- [x] 2. File edit tool
|
||||
- [x] 3. MCP management commands
|
||||
- [x] 4. File reference in chat
|
||||
- [x] 5. Implement /stop
|
||||
|
||||
## Risks
|
||||
|
||||
- DuckDuckGo has no official API — need HTML scraping or use lite endpoint
|
||||
- SearXNG needs self-hosted instance
|
||||
- @file expansion could be large — need size limits
|
||||
- /stop requires Pi SDK abort support — need to check API
|
||||
@@ -0,0 +1,35 @@
|
||||
# Scratchpad — Thin-core prompt diet (#528)
|
||||
|
||||
**Branch:** `feat/contract-thin-core` · **Issue:** #528 · **Mode:** Delivery
|
||||
|
||||
## Objective
|
||||
|
||||
Cut the always-injected contract (`defaults/AGENTS.md` + `defaults/TOOLS.md` + `runtime/claude/RUNTIME.md`, inlined every turn by the launcher) without losing any hard gate. Restore the original "thin core + on-demand guides" intent.
|
||||
|
||||
## Change
|
||||
|
||||
- `defaults/AGENTS.md` → thin core: 12 hard gates verbatim, 37 operating rules condensed to ~15 bullets (detail already in `guides/E2E-DELIVERY.md`), Superpowers condensed, load order made on-demand (no halt-on-missing for STANDARDS), conditional guide-loading index retained.
|
||||
- `defaults/TOOLS.md` → index; full catalog moved to new `guides/TOOLS-REFERENCE.md` (read on demand).
|
||||
- `runtime/claude/RUNTIME.md` → slimmed (dedup tier table, terser pointers).
|
||||
|
||||
## Method (autoresearch-style validation)
|
||||
|
||||
1. Built a 9-probe role battery (backend/deploy/review/orchestrate/secrets/docs/simple-trap/no-stop-at-PR/agent-work) + a deterministic 18-signature gate-checklist.
|
||||
2. Headless interactive runs (Claude Max **subscription**, tmux — no API), scored by per-probe rubric.
|
||||
3. Keep-or-discard hill-climb (token cost gated by per-probe fidelity) proved the method; final design re-derived against THIS repo's content (diet-only, no drifted-deployment content imported).
|
||||
|
||||
## Validation evidence
|
||||
|
||||
- Gate-checklist: ALL gates + critical rules + mode lines + sequential-thinking + OpenBrain + Superpowers present.
|
||||
- A/B on real repo content: **thin 7/9 vs monolith 5/9** probes; strictly better on deploy/review/simple-task; composed **8,827 → 4,122 tok (−53%)**.
|
||||
- p11 (don't-stop-at-PR): 3→2/3 on one rubric line — verified a scorer/phrasing artifact (answer correctly cites gates §5/§9 + close-issue; gate verbatim-present). Variance: thin 2/2/3, v0 3/3/3.
|
||||
|
||||
## Decisions / risks
|
||||
|
||||
- **Diet-only** vs repo content (user decision). Did NOT import web1's Gate 13-15 / federated memory / OpenViking — canonical repo is behind those deployments; flagged for separate reconciliation.
|
||||
- AGENTS/TOOLS are shared across runtimes → diet benefits codex/pi/opencode too; RUNTIME change is claude-only.
|
||||
- p11 accepted as-is (user decision) — not gaming the rubric.
|
||||
|
||||
## Status
|
||||
|
||||
PR open, paused for maintainer merge ratification (fleet-governing change). `mosaic upgrade` will propagate on merge.
|
||||
@@ -0,0 +1,125 @@
|
||||
# fix(db): bootstrap migrations on local-tier gateway startup
|
||||
|
||||
## Problem
|
||||
|
||||
Fresh `mosaic gateway install` (npm-installed) leaves the gateway DB schema empty:
|
||||
|
||||
```
|
||||
relation "users" does not exist
|
||||
```
|
||||
|
||||
Sign-in 500s, `auth users create` says "Not signed in", `admin/bootstrap setup`
|
||||
also fails — every entry point queries `users` before doing anything else.
|
||||
|
||||
## Scope
|
||||
|
||||
This PR fixes the **local (PGlite) tier** end-to-end. The postgres-tier path
|
||||
has additional pre-existing bugs (see "Known issues, out of scope" below) and
|
||||
needs a separate change with real Postgres validation.
|
||||
|
||||
## Root causes addressed (5 stacked bugs on the local-tier path)
|
||||
|
||||
1. **`packages/db/package.json` `files: ["dist"]`** — the `drizzle/` SQL
|
||||
migrations folder is excluded from the published tarball. Even if a
|
||||
migrate runner existed, it would have nothing to apply.
|
||||
|
||||
2. **`packages/db/src/migrate.ts`** only supports `postgres-js`. Local-tier
|
||||
gateways use embedded PGlite, which can't be reached over a postgres wire
|
||||
protocol — so `runMigrations()` is unusable for the local tier.
|
||||
|
||||
3. **`apps/gateway/src/database/database.module.ts`** never invokes
|
||||
migrations at startup. The module creates the DB handle and storage
|
||||
adapter, but no consumer calls `.migrate()` on either. `mosaic storage
|
||||
migrate` CLI even claims "pglite runs schema setup automatically on first
|
||||
connection via `adapter.migrate()`" — but `adapter.migrate()` is only
|
||||
called by tests, never at runtime.
|
||||
|
||||
4. **`createPgliteDb` does not load the pgvector extension.** Migration 0001
|
||||
declares `CREATE EXTENSION IF NOT EXISTS vector;` for the
|
||||
`insights.embedding` column. Bare PGlite has no pgvector — the migration
|
||||
fails on extension control file lookup.
|
||||
|
||||
5. **Drizzle's PG migrator wraps every migration in one outer transaction.**
|
||||
Migration 0009 does `ALTER TYPE grant_status ADD VALUE 'pending'` and then
|
||||
`ALTER TABLE federation_grants ALTER COLUMN status SET DEFAULT 'pending'`.
|
||||
Postgres' `check_safe_enum_use` rejects the second statement because the
|
||||
new enum value isn't committed yet. Splitting the migration into two
|
||||
files doesn't help — drizzle batches all migrations into one outer tx.
|
||||
|
||||
## Fix
|
||||
|
||||
- `packages/db/package.json` — ship `drizzle/` in `files`.
|
||||
- `packages/db/src/client-pglite.ts` — load `@electric-sql/pglite/vector`.
|
||||
- `packages/db/src/migrate.ts` — add `runPgliteMigrations(handle)`. Walks the
|
||||
Drizzle journal and runs each statement-breakpoint chunk through PGlite's
|
||||
`client.exec()` (Simple Query protocol → autocommit per statement). Writes
|
||||
to the standard `drizzle.__drizzle_migrations` ledger so the result is
|
||||
interoperable with `runMigrations()` on a postgres-backed deployment.
|
||||
Per-statement try/catch surfaces which statement of which migration failed
|
||||
and the ledger row is only written on full success.
|
||||
- `packages/db/src/index.ts` — re-export.
|
||||
- `apps/gateway/src/database/database.module.ts` — implement `OnModuleInit`:
|
||||
- Local tier → `runPgliteMigrations(handle)`, then `storageAdapter.migrate()`
|
||||
(the local storage adapter has its own kv tables in a separate PGlite dir).
|
||||
- Postgres tier → `storageAdapter.migrate()` only, since
|
||||
`PostgresAdapter.migrate()` already calls `runMigrations(url)` against
|
||||
the same DATABASE_URL — we deliberately don't double-call.
|
||||
|
||||
NestJS awaits `onModuleInit` before `app.listen()`, so DB-dependent modules
|
||||
see a populated schema before any HTTP traffic is accepted.
|
||||
|
||||
- `packages/storage/src/test-utils/pglite-with-vector.ts` — **deleted**.
|
||||
The "intentionally not exported" rationale is moot now that migration 0001
|
||||
forces pgvector load anyway. `migrate-tier.integration.test.ts` switched
|
||||
to `createPgliteDb` + `runPgliteMigrations` from `@mosaicstack/db`.
|
||||
|
||||
## Tests
|
||||
|
||||
`packages/db/src/migrate.test.ts`:
|
||||
|
||||
- Verifies `runPgliteMigrations` creates the BetterAuth tables (the original
|
||||
failure mode).
|
||||
- Idempotence (transitively re-runs migration 0009).
|
||||
- Partial-failure: pre-creates a conflicting `users` table, asserts the
|
||||
thrown error includes statement context (`hash=… statement #N failed`)
|
||||
and that no ledger row was written.
|
||||
|
||||
## QA evidence
|
||||
|
||||
End-to-end on a fresh PGlite install:
|
||||
|
||||
- `[DatabaseModule] Applying PGlite schema migrations...` then
|
||||
`Initializing storage adapter (pglite)...` in startup log.
|
||||
- `GET /api/bootstrap/status` → `{"needsSetup":true}` HTTP 200 (was 500
|
||||
with `relation "users" does not exist`).
|
||||
- `POST /api/bootstrap/setup` with empty body → HTTP 400 with Zod
|
||||
validation error (was 500), confirming the request reached the
|
||||
validator past the table-existence check.
|
||||
|
||||
## Known issues, out of scope (file separately)
|
||||
|
||||
- **Postgres-tier first install is still broken.** `runMigrations()` uses
|
||||
Drizzle's `migratePostgres`, which has the same outer-transaction problem
|
||||
as PGlite's migrator. A fresh standalone-tier install would also fail at
|
||||
migration 0009. Inline TODO in `migrate.ts:31-35` flags this. Fixing it
|
||||
needs either (a) a shared per-statement loop reused for both drivers, or
|
||||
(b) splitting migration 0009.
|
||||
- **`drizzle/meta/_journal.json` has 0009 ordered before 0008** (`when`
|
||||
values `1745280000000` < `1776822435828`). `migratePostgres` skips by
|
||||
`created_at < folderMillis`, so on a postgres deployment that already
|
||||
applied 0008, 0009 would be skipped forever. Our hash-based skip in the
|
||||
PGlite path sidesteps this.
|
||||
- **No advisory lock around the migration loop.** Two gateway processes
|
||||
pointed at the same DATABASE_URL would race. PGlite is single-process by
|
||||
file lock so the local tier is fine; postgres-tier deployments should add
|
||||
`pg_advisory_lock(<deterministic-id>)` around the loop in a follow-up.
|
||||
- **`mosaic storage migrate` CLI message is misleading** — it claims
|
||||
"automatic on first connection via adapter.migrate()" but the adapter
|
||||
doesn't self-migrate. With this PR the gateway invokes it explicitly, but
|
||||
the CLI message could still be tightened.
|
||||
- **Crash mid-migration leaves a partial-state PGlite DB without a ledger
|
||||
row.** Detected loudly on next boot (the replay errors on "already
|
||||
exists"), but recovery is manual (drop the partially-applied objects or
|
||||
insert the migration hash into `drizzle.__drizzle_migrations`). A robust
|
||||
fix would add a "started_at" column to a sidecar table to detect
|
||||
half-applied state and refuse to start with actionable guidance.
|
||||
@@ -0,0 +1,45 @@
|
||||
# RI-1-002 — Publish-gate negative controls (SDLC-D-034 second half)
|
||||
|
||||
- Task: RI-1-002 (docs/release-integrity workstream, PRD item RI-N1), issue ref #1275
|
||||
- Branch: `test/ri-050-publish-gate-negative` (base `origin/next` @ d8e0aec9 = PR #1277, RI-1-001)
|
||||
- Budget: worker estimate ~45K tokens; keep scoped to the two test files + scratchpad.
|
||||
|
||||
## Objective
|
||||
|
||||
Checked-in negative-control tests that PROVE the publish gate fails when it must:
|
||||
|
||||
1. Broken mandatory check blocks every publish step (structural DAG proof from `.woodpecker/publish.yml`).
|
||||
2. Bypass shapes fail the checker: missing edge, hidden effect (non-`publish` name), detached verify, always-pass verify (`failure: ignore` / `success` override), conditional verify (`when`).
|
||||
3. Exact-commit identity: no HEAD-moving step between verify and publish effects; legitimate re-checkout requires verify to re-run after it.
|
||||
4. `verify-release.mjs` composition control: a SUBSET stage list fails the composition check.
|
||||
|
||||
## Plan
|
||||
|
||||
- NEW `scripts/publish-gate-structure.test.mjs` — self-contained structural checker (`assertPublishGateBlocksOnVerify`) + positive control on the real pipeline + one negative-control test per bypass shape (S1–S6, documented in file header) + positive control for the legitimate re-checkout shape.
|
||||
- EXTEND `scripts/verify-release.test.mjs` — refactor the stage-mirror test body into `assertStagesMirrorCi(stages, ci)`; add negative control dropping each stage one at a time (subset must throw).
|
||||
|
||||
## Conventions confirmed
|
||||
|
||||
- Root `test:checkout` = `node --test scripts/*.test.mjs` → new file auto-joins `pnpm test`.
|
||||
- Test-enumeration guard population is `*test*.sh` under `packages/mosaic/framework/tools/` only → unaffected.
|
||||
- Root eslint covers only `**/*.{ts,tsx}` → .mjs files need Prettier style only (printWidth 100, singleQuote, semi, trailingComma all).
|
||||
- Do NOT touch docs/TASKS.md, docs/release-integrity/TASKS.md, docs/scratchpads/.
|
||||
|
||||
## Progress log
|
||||
|
||||
- [x] Base verified: publish.yml `verify` step + verify-release.mjs present; HEAD contains origin/next.
|
||||
- [x] Wrote scripts/publish-gate-structure.test.mjs
|
||||
- [x] Extended scripts/verify-release.test.mjs (mirror fn + subset negative control)
|
||||
- [x] Gates: node --test scripts (31 tests pass), prettier clean on touched files, pnpm typecheck PASS, pnpm lint PASS, pnpm format:check PASS
|
||||
- [x] Committed ff585b88 + pushed, PR #1305 → next (no conflicts). Stopped before merge per task instruction.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `node --test scripts/verify-release.test.mjs scripts/publish-gate-structure.test.mjs` → 31 tests, 0 fail.
|
||||
- Mutation sanity: temporarily removing the `verify` edge from build-gateway in publish.yml → structure test goes red (verified manually during dev, then reverted).
|
||||
- Gates run from repo root on this worktree; results in Progress log.
|
||||
|
||||
## Risks / notes
|
||||
|
||||
- Effect detection (`isPublishCommand`) is deliberately over-broad (any npm/pnpm/yarn command mentioning `publish`, any kaniko/docker-push/`--destination`) — fail-closed: a false positive forces justification, a false negative is the actual hazard.
|
||||
- `git fetch` flagged as HEAD-moving even though fetch alone doesn't move HEAD — fail-closed on the classic `fetch && reset` pair.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Scratchpad — RI-4-001 One transitional PRD authority (RI-N3, #1275)
|
||||
|
||||
- Objective: single PrdService authority in `@mosaicstack/prdy`; `mosaic prdy` and
|
||||
`mission --plan` become thin adapters; mission↔PRD linkage persisted on disk;
|
||||
Markdown export is a labeled generated view (never read back); import is
|
||||
validated/conflict-aware with typed refusals.
|
||||
- Budget: ~35K tokens (card cap). Baselines: prdy build/lint rc=0, 0 tests;
|
||||
mosaic build rc=0 (after root turbo build), lint rc=0, 1548 tests pass;
|
||||
root build rc=0.
|
||||
- Plan: (1) extend store schema (version, missions linkage) (2) PrdService +
|
||||
typed errors (3) wizard/cli route through service (4) mosaic adapters
|
||||
(5) contract specs both packages (6) gates (7) sabotage control (8) report
|
||||
to /var/tmp/ri-050/ri-4-001-report.md.
|
||||
- Decisions:
|
||||
- Linkage lives ON the PRD document (`missions` array) — one authority file,
|
||||
survives restart, no sidecar sync problems.
|
||||
- `version` = content revision of sections/status (bumped by update/import
|
||||
accept). Linkage writes bump `updatedAt` only, so ids/versions stay stable
|
||||
for the card's "stable ids/versions" contract.
|
||||
- Mission version marker = `mission.updatedAt` (gateway MissionInfo has no
|
||||
numeric version field).
|
||||
- Import reads YAML documents only — never the exported Markdown (keeps the
|
||||
"no code path reads exported Markdown" invariant).
|
||||
- Import of an existing id with identical core content → `identical` no-op;
|
||||
divergent → typed `PrdImportConflictError` carrying proposed successor
|
||||
(existing.version + 1, status draft, linkages preserved). Original bytes
|
||||
untouched until explicit `acceptSuccessor`.
|
||||
- `requirementIds` default `[]` at the mission command (no requirement
|
||||
selection UI yet) — service accepts ids when a caller has them.
|
||||
- Progress log:
|
||||
- [16:35] baselines captured (prdy 0 tests; mosaic 1548 after root build; root build rc=0)
|
||||
- [16:38] store schema v2 + PrdService + wizard/cli rerouted; prdy build/lint green
|
||||
- [16:40] mosaic adapters done; prdy spec 20/20 (found+fixed: import project-path leak, empty-store typed error, YAML timestamp coercion)
|
||||
- [16:44] mosaic specs 9/9 (fixed commander from:'user' argv, vi.mock hoisting, restoreAllMocks wiping factory mocks)
|
||||
- [16:45] all gates green; 4 commits (e291bfb, 2c5d208, a23826c, 540d6f1)
|
||||
- [16:46] sabotage: linkage write removed → prdy 3 fail / mosaic 2 fail, 1548/1548 pre-existing pass; restored byte-identically; re-green 20/20 + 1557/1557
|
||||
- [16:47] report written to /var/tmp/ri-050/ri-4-001-report.md — card complete
|
||||
Reference in New Issue
Block a user