feat: MACP Phase 1 — Core Protocol Implementation (#9)
This commit was merged in pull request #9.
This commit is contained in:
324
docs/tasks/MACP-PHASE1-brief.md
Normal file
324
docs/tasks/MACP-PHASE1-brief.md
Normal file
@@ -0,0 +1,324 @@
|
||||
# MACP Phase 1 — Core Protocol Implementation
|
||||
|
||||
**Branch:** `feat/macp-phase1`
|
||||
**Repo:** `mosaic-bootstrap` (worktree at `~/src/mosaic-bootstrap-worktrees/macp-phase1`)
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Extend the existing orchestrator-matrix into **MACP (Mosaic Agent Coordination Protocol)** — a standardized protocol that lets any orchestrator delegate work to any agent runtime, track progress via quality gates, and collect structured results.
|
||||
|
||||
This is an **evolution** of the existing code, not a rewrite. Extend existing schemas, enhance the controller, and add the dispatcher layer.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Extend Task Schema (`tools/orchestrator-matrix/protocol/task.schema.json`)
|
||||
|
||||
Add these new fields to the existing schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["coding", "deploy", "research", "review", "documentation", "infrastructure"],
|
||||
"description": "Task type — determines dispatch strategy and gate requirements"
|
||||
},
|
||||
"dispatch": {
|
||||
"type": "string",
|
||||
"enum": ["yolo", "acp", "exec"],
|
||||
"description": "Execution backend: yolo=mosaic yolo (full system), acp=OpenClaw sessions_spawn (sandboxed), exec=direct shell"
|
||||
},
|
||||
"worktree": {
|
||||
"type": "string",
|
||||
"description": "Path to git worktree for this task, e.g. ~/src/repo-worktrees/task-042"
|
||||
},
|
||||
"branch": {
|
||||
"type": "string",
|
||||
"description": "Git branch name for this task"
|
||||
},
|
||||
"brief_path": {
|
||||
"type": "string",
|
||||
"description": "Path to markdown task brief relative to repo root"
|
||||
},
|
||||
"result_path": {
|
||||
"type": "string",
|
||||
"description": "Path to JSON result file relative to .mosaic/orchestrator/"
|
||||
},
|
||||
"issue": {
|
||||
"type": "string",
|
||||
"description": "Issue reference (e.g. #42)"
|
||||
},
|
||||
"pr": {
|
||||
"type": ["string", "null"],
|
||||
"description": "PR number/URL once opened"
|
||||
},
|
||||
"depends_on": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"description": "List of task IDs this task depends on"
|
||||
},
|
||||
"max_attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"default": 1
|
||||
},
|
||||
"attempts": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"default": 0
|
||||
},
|
||||
"timeout_seconds": {
|
||||
"type": "integer",
|
||||
"description": "Override default timeout for this task"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Extend the `status` enum to include: `"pending"`, `"running"`, `"gated"`, `"completed"`, `"failed"`, `"escalated"`
|
||||
|
||||
Keep all existing fields. Keep `additionalProperties: true`.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Extend Event Schema (`tools/orchestrator-matrix/protocol/event.schema.json`)
|
||||
|
||||
Add new event types to the enum:
|
||||
- `task.gated` — Worker done coding, quality gates now running
|
||||
- `task.escalated` — Requires human intervention
|
||||
- `task.retry.scheduled` — Task will be retried (already emitted by controller, just not in schema)
|
||||
|
||||
Add new source type: `"dispatcher"`
|
||||
|
||||
Keep all existing event types and fields.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Create Result Schema (`tools/orchestrator-matrix/protocol/result.schema.json`)
|
||||
|
||||
New file — standardized worker completion report:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://mosaicstack.dev/schemas/orchestrator/result.schema.json",
|
||||
"title": "MACP Task Result",
|
||||
"type": "object",
|
||||
"required": ["task_id", "status", "completed_at"],
|
||||
"properties": {
|
||||
"task_id": { "type": "string" },
|
||||
"status": { "type": "string", "enum": ["completed", "failed", "escalated"] },
|
||||
"completed_at": { "type": "string", "format": "date-time" },
|
||||
"failed_at": { "type": ["string", "null"], "format": "date-time" },
|
||||
"exit_code": { "type": ["integer", "null"] },
|
||||
"attempt": { "type": "integer" },
|
||||
"max_attempts": { "type": "integer" },
|
||||
"runtime": { "type": "string" },
|
||||
"dispatch": { "type": "string" },
|
||||
"worktree": { "type": ["string", "null"] },
|
||||
"branch": { "type": ["string", "null"] },
|
||||
"pr": { "type": ["string", "null"] },
|
||||
"summary": { "type": "string", "description": "Human-readable summary of what the worker did" },
|
||||
"files_changed": { "type": "array", "items": { "type": "string" } },
|
||||
"gate_results": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"command": { "type": "string" },
|
||||
"exit_code": { "type": "integer" },
|
||||
"type": { "type": "string", "enum": ["mechanical", "ai-review", "ci-pipeline"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"error": { "type": ["string", "null"] },
|
||||
"escalation_reason": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Build MACP Dispatcher (`tools/orchestrator-matrix/dispatcher/macp_dispatcher.py`)
|
||||
|
||||
New Python module — the core of MACP. This translates task specs into execution commands and manages the full lifecycle.
|
||||
|
||||
### Dispatcher Responsibilities:
|
||||
1. **Worktree setup** — `git worktree add` before dispatch (if task has `worktree` field)
|
||||
2. **Command generation** — Build the right command based on `dispatch` type:
|
||||
- `yolo`: `mosaic yolo codex|claude|opencode "<brief contents>"` (reads from `brief_path`)
|
||||
- `acp`: Generates a command that would be used with OpenClaw sessions_spawn (just outputs the config JSON for the caller)
|
||||
- `exec`: Direct shell command from task `command` field
|
||||
3. **Result collection** — After worker exits, write structured result JSON to `results/<task-id>.json`
|
||||
4. **Worktree cleanup** — After task completion (success or failure after max retries), remove the worktree
|
||||
|
||||
### Key Functions:
|
||||
|
||||
```python
|
||||
def setup_worktree(task: dict, repo_root: Path) -> Path:
|
||||
"""Create git worktree for task. Returns worktree path."""
|
||||
|
||||
def build_dispatch_command(task: dict, repo_root: Path) -> str:
|
||||
"""Generate execution command based on task dispatch type and runtime."""
|
||||
|
||||
def collect_result(task: dict, exit_code: int, gate_results: list, orch_dir: Path) -> dict:
|
||||
"""Build standardized result JSON after worker completion."""
|
||||
|
||||
def cleanup_worktree(task: dict) -> None:
|
||||
"""Remove git worktree after task is done."""
|
||||
|
||||
def dispatch_task(task: dict, repo_root: Path, orch_dir: Path, config: dict) -> tuple[int, str]:
|
||||
"""Full dispatch lifecycle: setup → execute → collect → cleanup. Returns (exit_code, output)."""
|
||||
```
|
||||
|
||||
### Worktree Convention:
|
||||
- Base path: `~/src/<repo-name>-worktrees/`
|
||||
- Worktree name: task ID slugified (e.g., `task-042-auth`)
|
||||
- Branch: `feat/<task-id>-<slugified-title>` (or use task `branch` field if specified)
|
||||
- Created from: `origin/main`
|
||||
|
||||
### `mosaic yolo` Command Pattern:
|
||||
The dispatcher needs to invoke `mosaic yolo` via PTY (it requires a terminal). The command pattern:
|
||||
```bash
|
||||
export PATH="$HOME/.config/mosaic/bin:$PATH"
|
||||
cd <worktree_path>
|
||||
mosaic yolo codex "<task brief contents>"
|
||||
```
|
||||
|
||||
For the dispatcher, since it's called from the controller which uses `subprocess.Popen`, the command should be wrapped appropriately. Use `script -qec` or similar for PTY simulation if needed, but prefer passing the brief via a temp file and using the worker script pattern.
|
||||
|
||||
### Important Constraints:
|
||||
- The dispatcher is a **library module** imported by the controller — NOT a standalone script
|
||||
- It should be testable (pure functions where possible, side effects isolated)
|
||||
- Error handling: if worktree creation fails, task goes to `failed` immediately
|
||||
- If `dispatch` is not set on a task, fall back to `exec` (backward compatible)
|
||||
- If `runtime` is not set, fall back to config's `worker.runtime`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Wire Controller to Use Dispatcher (`tools/orchestrator-matrix/controller/mosaic_orchestrator.py`)
|
||||
|
||||
Modify the existing controller to use the new dispatcher:
|
||||
|
||||
1. **Import the dispatcher module**
|
||||
2. **In `run_single_task()`**:
|
||||
- If task has `dispatch` field (or is MACP-aware), call `dispatcher.dispatch_task()` instead of raw `run_shell()`
|
||||
- If task does NOT have `dispatch` field, keep existing `run_shell()` behavior (backward compatibility)
|
||||
3. **Handle new statuses**: `gated` and `escalated`
|
||||
- `gated`: Set when worker completes but gates are running (transition state between running and completed/failed)
|
||||
- `escalated`: Set when task needs human intervention (gate failures after max retries, explicit worker escalation)
|
||||
4. **Emit new event types**: `task.gated` and `task.escalated`
|
||||
|
||||
### Backward Compatibility is Critical:
|
||||
- Existing tasks.json files with no `dispatch` field MUST continue to work exactly as before
|
||||
- The controller should detect MACP-aware tasks by checking for the `dispatch` field
|
||||
- All existing tests/workflows must remain functional
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Update Config Template (`templates/repo/.mosaic/orchestrator/config.json`)
|
||||
|
||||
Add MACP dispatcher configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"enabled": false,
|
||||
"transport": "matrix",
|
||||
"macp": {
|
||||
"worktree_base": "~/src/{repo}-worktrees",
|
||||
"default_dispatch": "exec",
|
||||
"default_runtime": "codex",
|
||||
"cleanup_worktrees": true,
|
||||
"brief_dir": "docs/tasks",
|
||||
"result_dir": ".mosaic/orchestrator/results"
|
||||
},
|
||||
"worker": { ... existing ... },
|
||||
"quality_gates": [ ... existing ... ]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Build `mosaic macp` CLI (`bin/mosaic-macp`)
|
||||
|
||||
Shell script that provides manual MACP interaction:
|
||||
|
||||
```bash
|
||||
mosaic macp submit --task-id TASK-001 --title "..." --type coding --runtime codex --brief docs/tasks/TASK-001.md
|
||||
mosaic macp status [--task-id TASK-001]
|
||||
mosaic macp drain # Run all pending tasks sequentially
|
||||
mosaic macp history [--task-id TASK-001] # Show events for a task
|
||||
```
|
||||
|
||||
Implementation:
|
||||
- `submit`: Adds a task to `tasks.json` with MACP fields populated
|
||||
- `status`: Pretty-prints task queue state (pending/running/gated/completed/failed/escalated counts)
|
||||
- `drain`: Calls `mosaic-orchestrator-run --until-drained`
|
||||
- `history`: Reads `events.ndjson` and filters by task ID
|
||||
|
||||
Register in the main `bin/mosaic` dispatcher so `mosaic macp ...` works.
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Update `tasks_md_sync.py`
|
||||
|
||||
Extend the sync script to handle MACP-specific columns in `docs/TASKS.md`:
|
||||
|
||||
Optional new columns: `type`, `dispatch`, `runtime`, `branch`
|
||||
|
||||
If these columns exist in the markdown table, map them to the task JSON. If they don't exist, use defaults from config.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks are complete:
|
||||
|
||||
1. **Existing behavior preserved**: Run `python3 tools/orchestrator-matrix/controller/mosaic_orchestrator.py --repo /tmp/test-repo --once` with an old-style tasks.json — must work identically
|
||||
2. **New MACP flow works**: Create a tasks.json with `dispatch: "exec"` tasks, run controller, verify worktree creation, execution, result collection, and cleanup
|
||||
3. **Schema validation**: All JSON files should validate against their schemas
|
||||
4. **CLI works**: `mosaic macp status` and `mosaic macp submit` produce correct output
|
||||
5. **No broken imports**: All Python files should pass `python3 -c "import ..."`
|
||||
|
||||
---
|
||||
|
||||
## File Map (what goes where)
|
||||
|
||||
```
|
||||
tools/orchestrator-matrix/
|
||||
├── protocol/
|
||||
│ ├── task.schema.json ← MODIFY (extend)
|
||||
│ ├── event.schema.json ← MODIFY (extend)
|
||||
│ └── result.schema.json ← NEW
|
||||
├── dispatcher/
|
||||
│ ├── __init__.py ← NEW
|
||||
│ └── macp_dispatcher.py ← NEW
|
||||
├── controller/
|
||||
│ ├── mosaic_orchestrator.py ← MODIFY (wire dispatcher)
|
||||
│ └── tasks_md_sync.py ← MODIFY (new columns)
|
||||
├── transport/
|
||||
│ └── matrix_transport.py ← UNCHANGED
|
||||
└── adapters/
|
||||
└── README.md ← UNCHANGED
|
||||
|
||||
templates/repo/.mosaic/orchestrator/
|
||||
├── config.json ← MODIFY (add macp section)
|
||||
├── tasks.json ← UNCHANGED
|
||||
└── ...
|
||||
|
||||
bin/
|
||||
├── mosaic ← MODIFY (add macp subcommand routing)
|
||||
└── mosaic-macp ← NEW
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ground Rules
|
||||
|
||||
- This is the `mosaic-bootstrap` repo — a CLI framework, NOT a web app
|
||||
- No npm/pnpm — this is bash + Python (stdlib only, no pip dependencies)
|
||||
- All Python must work with Python 3.10+
|
||||
- Keep the deterministic controller pattern — no async, no threads, no external services
|
||||
- Commit messages: `feat: <what changed>` (conventional commits)
|
||||
- Push to `feat/macp-phase1` branch when done
|
||||
64
docs/tasks/MACP-PHASE1-fixes.md
Normal file
64
docs/tasks/MACP-PHASE1-fixes.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# MACP Phase 1 — Review Fixes
|
||||
|
||||
**Branch:** `feat/macp-phase1` (amend/fix commits on top of existing)
|
||||
**Repo worktree:** `~/src/mosaic-bootstrap-worktrees/macp-phase1`
|
||||
|
||||
These are 5 findings from code review and security review. Fix all of them.
|
||||
|
||||
---
|
||||
|
||||
## Fix 1 (BLOCKER): Legacy task reclassification in tasks_md_sync.py
|
||||
|
||||
**File:** `tools/orchestrator-matrix/controller/tasks_md_sync.py`
|
||||
**Problem:** `build_task()` now always sets `task["dispatch"]` from `macp_defaults` even when the `docs/TASKS.md` table has no `dispatch` column. The controller's `is_macp_task()` check is simply `"dispatch" in task`, so ALL synced tasks get reclassified as MACP tasks and routed through the dispatcher instead of the legacy `run_shell()` path. This breaks backward compatibility.
|
||||
|
||||
**Fix:** Only populate `dispatch`, `type`, `runtime`, and `branch` from MACP defaults when the markdown table row **explicitly contains that column**. If the column is absent from the table header, do NOT inject MACP fields. Check whether the parsed headers include these column names before adding them.
|
||||
|
||||
---
|
||||
|
||||
## Fix 2 (BLOCKER): ACP dispatch is a no-op that falsely reports success
|
||||
|
||||
**File:** `tools/orchestrator-matrix/dispatcher/macp_dispatcher.py`
|
||||
**Problem:** For `dispatch == "acp"`, `build_dispatch_command()` returns a `python3 -c ...` command that only prints JSON but doesn't actually spawn an ACP session. The controller then marks the task completed even though no work ran.
|
||||
|
||||
**Fix:** For `acp` dispatch, do NOT generate a command that runs and exits 0. Instead, set the task status to `"escalated"` with `escalation_reason = "ACP dispatch requires OpenClaw integration (Phase 2)"` and return exit code 1. This makes it fail-safe — ACP tasks won't silently succeed. The controller should emit a `task.escalated` event for these.
|
||||
|
||||
---
|
||||
|
||||
## Fix 3 (SHOULD-FIX): Premature result write before quality gates
|
||||
|
||||
**File:** `tools/orchestrator-matrix/dispatcher/macp_dispatcher.py`
|
||||
**Problem:** `dispatch_task()` calls `collect_result(task, exit_code, [], orch_dir)` immediately after the worker exits, before quality gates run. If the controller crashes between this write and the final overwrite, a false "completed" result with empty gate_results sits on disk.
|
||||
|
||||
**Fix:** Remove the `collect_result()` call from `dispatch_task()`. Instead, have `dispatch_task()` return `(exit_code, output)` only. The controller in `mosaic_orchestrator.py` should call `collect_result()` AFTER quality gates have run, when the final status is known. Make sure the controller passes gate_results to collect_result.
|
||||
|
||||
---
|
||||
|
||||
## Fix 4 (SECURITY MEDIUM): Brief contents exposed in process arguments
|
||||
|
||||
**File:** `tools/orchestrator-matrix/dispatcher/macp_dispatcher.py`
|
||||
**Problem:** For `yolo` dispatch, `build_dispatch_command()` puts the full task brief text into the shell command as an argument: `mosaic yolo codex "<entire brief>"`. This is visible in `ps` output, shell logs, and crash reports.
|
||||
|
||||
**Fix:** Write the brief contents to a temporary file with restrictive permissions (0600), and pass the file path to the worker instead. The command should read: `mosaic yolo <runtime> "$(cat /path/to/brief-TASKID.tmp)"` or better, restructure so the worker script reads from a file path argument. Use the existing `orchestrator-worker.sh` pattern which already reads from a task file. Clean up the temp file after dispatch completes.
|
||||
|
||||
---
|
||||
|
||||
## Fix 5 (SECURITY MEDIUM): Unrestricted worktree cleanup path
|
||||
|
||||
**File:** `tools/orchestrator-matrix/dispatcher/macp_dispatcher.py`
|
||||
**Problem:** `cleanup_worktree()` trusts `task['worktree']` without validating it belongs to the expected worktree base directory. A tampered task could cause deletion of unrelated worktrees.
|
||||
|
||||
**Fix:** Before running `git worktree remove`, validate that the resolved worktree path starts with the configured `worktree_base` (from `config.macp.worktree_base`, with `{repo}` expanded). If the path doesn't match the expected base, log a warning and refuse to clean up. Add a helper `_is_safe_worktree_path(worktree_path, config)` for this validation.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After fixes:
|
||||
1. `python3 -c "from tools.orchestrator_matrix.dispatcher import macp_dispatcher"` should not error (fix import path if needed, or just verify `python3 tools/orchestrator-matrix/dispatcher/macp_dispatcher.py` has no syntax errors)
|
||||
2. Legacy tasks.json with no `dispatch` field must still work with the controller's `run_shell()` path
|
||||
3. ACP dispatch should NOT mark tasks completed — should escalate
|
||||
4. No brief text should appear in generated shell commands for yolo dispatch
|
||||
5. `cleanup_worktree()` should refuse paths outside the configured base
|
||||
|
||||
Commit with: `fix: address review findings — backward compat, ACP safety, result timing, security`
|
||||
Reference in New Issue
Block a user