bootstrap/docs/tasks/MACP-PHASE1-brief.md

MACP Phase 1 — Core Protocol Implementation

Branch: feat/macp-phase1 Repo: mosaic-bootstrap (worktree at ~/src/mosaic-bootstrap-worktrees/macp-phase1)

---

Objective

Extend the existing orchestrator-matrix into MACP (Mosaic Agent Coordination Protocol) — a standardized protocol that lets any orchestrator delegate work to any agent runtime, track progress via quality gates, and collect structured results.

This is an evolution of the existing code, not a rewrite. Extend existing schemas, enhance the controller, and add the dispatcher layer.

---

Task 1: Extend Task Schema (tools/orchestrator-matrix/protocol/task.schema.json)

Add these new fields to the existing schema:

{
  "type": {
    "type": "string",
    "enum": ["coding", "deploy", "research", "review", "documentation", "infrastructure"],
    "description": "Task type — determines dispatch strategy and gate requirements"
  },
  "dispatch": {
    "type": "string",
    "enum": ["yolo", "acp", "exec"],
    "description": "Execution backend: yolo=mosaic yolo (full system), acp=OpenClaw sessions_spawn (sandboxed), exec=direct shell"
  },
  "worktree": {
    "type": "string",
    "description": "Path to git worktree for this task, e.g. ~/src/repo-worktrees/task-042"
  },
  "branch": {
    "type": "string",
    "description": "Git branch name for this task"
  },
  "brief_path": {
    "type": "string",
    "description": "Path to markdown task brief relative to repo root"
  },
  "result_path": {
    "type": "string",
    "description": "Path to JSON result file relative to .mosaic/orchestrator/"
  },
  "issue": {
    "type": "string",
    "description": "Issue reference (e.g. #42)"
  },
  "pr": {
    "type": ["string", "null"],
    "description": "PR number/URL once opened"
  },
  "depends_on": {
    "type": "array",
    "items": { "type": "string" },
    "description": "List of task IDs this task depends on"
  },
  "max_attempts": {
    "type": "integer",
    "minimum": 1,
    "default": 1
  },
  "attempts": {
    "type": "integer",
    "minimum": 0,
    "default": 0
  },
  "timeout_seconds": {
    "type": "integer",
    "description": "Override default timeout for this task"
  }
}

Extend the status enum to include: "pending", "running", "gated", "completed", "failed", "escalated"

Keep all existing fields. Keep additionalProperties: true.

---

Task 2: Extend Event Schema (tools/orchestrator-matrix/protocol/event.schema.json)

Add new event types to the enum:

• task.gated — Worker done coding, quality gates now running
• task.escalated — Requires human intervention
• task.retry.scheduled — Task will be retried (already emitted by controller, just not in schema)

Add new source type: "dispatcher"

Keep all existing event types and fields.

---

Task 3: Create Result Schema (tools/orchestrator-matrix/protocol/result.schema.json)

New file — standardized worker completion report:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://mosaicstack.dev/schemas/orchestrator/result.schema.json",
  "title": "MACP Task Result",
  "type": "object",
  "required": ["task_id", "status", "completed_at"],
  "properties": {
    "task_id": { "type": "string" },
    "status": { "type": "string", "enum": ["completed", "failed", "escalated"] },
    "completed_at": { "type": "string", "format": "date-time" },
    "failed_at": { "type": ["string", "null"], "format": "date-time" },
    "exit_code": { "type": ["integer", "null"] },
    "attempt": { "type": "integer" },
    "max_attempts": { "type": "integer" },
    "runtime": { "type": "string" },
    "dispatch": { "type": "string" },
    "worktree": { "type": ["string", "null"] },
    "branch": { "type": ["string", "null"] },
    "pr": { "type": ["string", "null"] },
    "summary": { "type": "string", "description": "Human-readable summary of what the worker did" },
    "files_changed": { "type": "array", "items": { "type": "string" } },
    "gate_results": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "command": { "type": "string" },
          "exit_code": { "type": "integer" },
          "type": { "type": "string", "enum": ["mechanical", "ai-review", "ci-pipeline"] }
        }
      }
    },
    "error": { "type": ["string", "null"] },
    "escalation_reason": { "type": ["string", "null"] },
    "metadata": { "type": "object" }
  }
}

---

Task 4: Build MACP Dispatcher (tools/orchestrator-matrix/dispatcher/macp_dispatcher.py)

New Python module — the core of MACP. This translates task specs into execution commands and manages the full lifecycle.

Dispatcher Responsibilities:

1. Worktree setup — git worktree add before dispatch (if task has worktree field)
2. Command generation — Build the right command based on dispatch type:
   • yolo: mosaic yolo codex|claude|opencode "<brief contents>" (reads from brief_path)
   • acp: Generates a command that would be used with OpenClaw sessions_spawn (just outputs the config JSON for the caller)
   • exec: Direct shell command from task command field
3. Result collection — After worker exits, write structured result JSON to results/<task-id>.json
4. Worktree cleanup — After task completion (success or failure after max retries), remove the worktree

Key Functions:

def setup_worktree(task: dict, repo_root: Path) -> Path:
    """Create git worktree for task. Returns worktree path."""
    
def build_dispatch_command(task: dict, repo_root: Path) -> str:
    """Generate execution command based on task dispatch type and runtime."""
    
def collect_result(task: dict, exit_code: int, gate_results: list, orch_dir: Path) -> dict:
    """Build standardized result JSON after worker completion."""
    
def cleanup_worktree(task: dict) -> None:
    """Remove git worktree after task is done."""
    
def dispatch_task(task: dict, repo_root: Path, orch_dir: Path, config: dict) -> tuple[int, str]:
    """Full dispatch lifecycle: setup → execute → collect → cleanup. Returns (exit_code, output)."""

Worktree Convention:

• Base path: ~/src/<repo-name>-worktrees/
• Worktree name: task ID slugified (e.g., task-042-auth)
• Branch: feat/<task-id>-<slugified-title> (or use task branch field if specified)
• Created from: origin/main

mosaic yolo Command Pattern:

The dispatcher needs to invoke mosaic yolo via PTY (it requires a terminal). The command pattern:

export PATH="$HOME/.config/mosaic/bin:$PATH"
cd <worktree_path>
mosaic yolo codex "<task brief contents>"

For the dispatcher, since it's called from the controller which uses subprocess.Popen, the command should be wrapped appropriately. Use script -qec or similar for PTY simulation if needed, but prefer passing the brief via a temp file and using the worker script pattern.

Important Constraints:

• The dispatcher is a library module imported by the controller — NOT a standalone script
• It should be testable (pure functions where possible, side effects isolated)
• Error handling: if worktree creation fails, task goes to failed immediately
• If dispatch is not set on a task, fall back to exec (backward compatible)
• If runtime is not set, fall back to config's worker.runtime

---

Task 5: Wire Controller to Use Dispatcher (tools/orchestrator-matrix/controller/mosaic_orchestrator.py)

Modify the existing controller to use the new dispatcher:

1. Import the dispatcher module
2. In run_single_task():
   • If task has dispatch field (or is MACP-aware), call dispatcher.dispatch_task() instead of raw run_shell()
   • If task does NOT have dispatch field, keep existing run_shell() behavior (backward compatibility)
3. Handle new statuses: gated and escalated
   • gated: Set when worker completes but gates are running (transition state between running and completed/failed)
   • escalated: Set when task needs human intervention (gate failures after max retries, explicit worker escalation)
4. Emit new event types: task.gated and task.escalated

Backward Compatibility is Critical:

• Existing tasks.json files with no dispatch field MUST continue to work exactly as before
• The controller should detect MACP-aware tasks by checking for the dispatch field
• All existing tests/workflows must remain functional

---

Task 6: Update Config Template (templates/repo/.mosaic/orchestrator/config.json)

Add MACP dispatcher configuration:

{
  "enabled": false,
  "transport": "matrix",
  "macp": {
    "worktree_base": "~/src/{repo}-worktrees",
    "default_dispatch": "exec",
    "default_runtime": "codex",
    "cleanup_worktrees": true,
    "brief_dir": "docs/tasks",
    "result_dir": ".mosaic/orchestrator/results"
  },
  "worker": { ... existing ... },
  "quality_gates": [ ... existing ... ]
}

---

Task 7: Build mosaic macp CLI (bin/mosaic-macp)

Shell script that provides manual MACP interaction:

mosaic macp submit --task-id TASK-001 --title "..." --type coding --runtime codex --brief docs/tasks/TASK-001.md
mosaic macp status [--task-id TASK-001]
mosaic macp drain                    # Run all pending tasks sequentially
mosaic macp history [--task-id TASK-001]  # Show events for a task

Implementation:

• submit: Adds a task to tasks.json with MACP fields populated
• status: Pretty-prints task queue state (pending/running/gated/completed/failed/escalated counts)
• drain: Calls mosaic-orchestrator-run --until-drained
• history: Reads events.ndjson and filters by task ID

Register in the main bin/mosaic dispatcher so mosaic macp ... works.

---

Task 8: Update tasks_md_sync.py

Extend the sync script to handle MACP-specific columns in docs/TASKS.md:

Optional new columns: type, dispatch, runtime, branch

If these columns exist in the markdown table, map them to the task JSON. If they don't exist, use defaults from config.

---

Verification

After all tasks are complete:

1. Existing behavior preserved: Run python3 tools/orchestrator-matrix/controller/mosaic_orchestrator.py --repo /tmp/test-repo --once with an old-style tasks.json — must work identically
2. New MACP flow works: Create a tasks.json with dispatch: "exec" tasks, run controller, verify worktree creation, execution, result collection, and cleanup
3. Schema validation: All JSON files should validate against their schemas
4. CLI works: mosaic macp status and mosaic macp submit produce correct output
5. No broken imports: All Python files should pass python3 -c "import ..."

---

File Map (what goes where)

tools/orchestrator-matrix/
├── protocol/
│   ├── task.schema.json          ← MODIFY (extend)
│   ├── event.schema.json         ← MODIFY (extend)
│   └── result.schema.json        ← NEW
├── dispatcher/
│   ├── __init__.py               ← NEW
│   └── macp_dispatcher.py        ← NEW
├── controller/
│   ├── mosaic_orchestrator.py    ← MODIFY (wire dispatcher)
│   └── tasks_md_sync.py         ← MODIFY (new columns)
├── transport/
│   └── matrix_transport.py       ← UNCHANGED
└── adapters/
    └── README.md                 ← UNCHANGED

templates/repo/.mosaic/orchestrator/
├── config.json                   ← MODIFY (add macp section)
├── tasks.json                    ← UNCHANGED
└── ...

bin/
├── mosaic                        ← MODIFY (add macp subcommand routing)
└── mosaic-macp                   ← NEW

---

Ground Rules

• This is the mosaic-bootstrap repo — a CLI framework, NOT a web app
• No npm/pnpm — this is bash + Python (stdlib only, no pip dependencies)
• All Python must work with Python 3.10+
• Keep the deterministic controller pattern — no async, no threads, no external services
• Commit messages: feat: <what changed> (conventional commits)
• Push to feat/macp-phase1 branch when done