mosaicstack/bootstrap
Archived
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add MACP event bridge — watcher, webhook, Discord formatter

Browse Source
This commit is contained in:
Jarvis
2026-03-27 21:06:22 -05:00
parent 28392914a7
commit 63c30b564d
17 changed files with 736 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

103
docs/PRD.md
View File

@@ -1,4 +1,4 @@
# PRD: MACP Phase 1 Core Protocol Implementation
# PRD: MACP Phase 2A Event Bridge + Notification System
## Metadata
@@ -9,90 +9,89 @@
## Problem Statement
The current orchestrator-matrix rail can queue shell-based worker tasks, but it does not yet expose a standardized protocol for dispatch selection, worktree-aware execution, structured results, or manual MACP queue operations. MACP Phase 1 extends the existing rail so orchestrators can delegate to multiple runtimes through a consistent task model while preserving current behavior for legacy tasks.
MACP Phase 1 writes structured lifecycle events to `.mosaic/orchestrator/events.ndjson`, but no repo-local bridge consumes those events for external systems. Phase 2A adds a portable watcher, webhook delivery, and Discord-friendly formatting so MACP event streams can drive OpenClaw integrations and human-facing notifications.
## Objectives
1. Extend the existing orchestrator-matrix protocol and controller to support MACP-aware task dispatch and status tracking.
2. Add a dispatcher layer that manages worktree lifecycle, runtime command generation, and standardized results.
3. Provide a CLI entrypoint for manual MACP submission, status inspection, queue draining, and history review.
1. Add a synchronous event watcher that tails `events.ndjson` using stdlib-only file polling and persists cursor state across restarts.
2. Add a webhook adapter that can forward selected MACP events to a configured HTTP endpoint with bounded retries.
3. Add a Discord formatter that turns task lifecycle events into concise human-readable strings.
4. Extend the `mosaic macp` CLI with a `watch` command for one-shot or continuous event bridge execution.
## Scope
### In Scope
1. Extend the orchestrator task and event schemas and add a result schema.
2. Add a Python dispatcher module under `tools/orchestrator-matrix/dispatcher/`.
3. Update the controller to use the dispatcher for MACP-aware tasks while preserving legacy execution paths.
4. Update orchestrator config templates, task markdown sync logic, and CLI routing/scripts for MACP commands.
5. Add verification for backward compatibility, schema validity, imports, and basic MACP execution flow.
1. New `tools/orchestrator-matrix/events/` package with watcher, webhook adapter, and Discord formatter modules.
2. Cursor persistence at `.mosaic/orchestrator/event_cursor.json`.
3. `mosaic macp watch [--webhook] [--once]` CLI support using `.mosaic/orchestrator/config.json`.
4. Stdlib-only verification of watcher polling, webhook delivery, Discord formatting, CLI watch behavior, and cursor persistence.
5. Developer documentation and sitemap updates covering the Phase 2A event bridge.
### Out of Scope
1. Rewriting the orchestrator controller architecture.
2. Changing Matrix transport behavior beyond schema compatibility.
3. Implementing real OpenClaw `sessions_spawn` execution beyond producing the config payload/command for callers.
4. Adding non-stdlib Python dependencies or npm-based tooling.
1. Adding Discord transport or webhook server hosting inside this repository.
2. Replacing the existing Matrix transport bridge.
3. Introducing async, threads, or third-party Python packages.
4. Changing event emission behavior in the controller beyond consuming the existing event stream.
## User/Stakeholder Requirements
1. MACP must evolve the current orchestrator-matrix implementation rather than replace it.
2. Legacy task queues without `dispatch` fields must continue to run exactly as before.
3. MACP-aware tasks must support dispatch modes `yolo`, `acp`, and `exec`.
4. Results must be written in a structured JSON format suitable for audit and orchestration follow-up.
5. A manual `mosaic macp` CLI must expose submit, status, drain, and history flows.
1. External systems must be able to consume MACP events without reading the NDJSON file directly.
2. The watcher must remain portable across environments, so file polling is required instead of platform-specific file watching.
3. Restarting the watcher must not replay previously consumed events.
4. Webhook delivery failures must be logged and isolated so the watcher loop continues running.
5. Discord formatting must stay concise and useful for task lifecycle visibility.
## Functional Requirements
1. Task schema must include MACP dispatch, worktree, result, retry, branch, brief, issue/PR, and dependency fields.
2. Event schema must recognize `task.gated`, `task.escalated`, and `task.retry.scheduled`, plus a `dispatcher` source.
3. Dispatcher functions must set up worktrees, build commands, execute tasks, collect results, and clean up worktrees.
4. Controller `run_single_task()` must route MACP-aware tasks through the dispatcher and emit the correct lifecycle events/status transitions.
5. `tasks_md_sync.py` must map optional MACP table columns only when those headers are present in `docs/TASKS.md`; absent MACP headers must not inject MACP fields into legacy tasks.
6. `bin/mosaic` must route `mosaic macp ...` to a new `bin/mosaic-macp` script.
1. `EventWatcher` must watch `.mosaic/orchestrator/events.ndjson`, parse appended JSON lines, and invoke registered callbacks for matching event types.
2. `EventWatcher.poll_once()` must tolerate a missing events file, truncated/corrupt lines, and cursor positions that are stale after file truncation.
3. Cursor writes must be atomic and stored at `.mosaic/orchestrator/event_cursor.json`.
4. `send_webhook(event, config)` must POST JSON to the configured URL using `urllib.request`, optionally adding a bearer token, respecting timeout, and retrying with exponential backoff.
5. `create_webhook_callback(config)` must return a callback that swallows/logs failures instead of raising into the watcher loop.
6. `format_event(event)` must support `task.completed`, `task.failed`, `task.escalated`, `task.gated`, and `task.started`, including useful task metadata when present.
7. `format_summary(events)` must produce a short batch summary suitable for notification digests.
8. `bin/mosaic-macp` must expose `watch`, optionally enabling webhook delivery from config, and support one-shot polling with `--once`.
## Non-Functional Requirements
1. Security: no secrets embedded in generated commands, config, or results.
2. Performance: controller remains deterministic and synchronous with no async or thread-based orchestration.
3. Reliability: worktree creation/cleanup failures must be surfaced predictably and produce structured task failure/escalation states.
4. Observability: lifecycle events, logs, and result JSON must clearly show task outcome, attempts, gates, and errors.
1. Security: no secrets embedded in code or logs; auth token only sent via header when configured.
2. Performance: each webhook attempt must be bounded by `timeout_seconds`; no event-processing path may hang indefinitely.
3. Reliability: corrupt input lines and callback delivery failures must be logged to stderr and skipped without crashing the watcher.
4. Portability: Python 3.10+ stdlib only; no OS-specific file watcher APIs.
5. Observability: warnings and failures must be clear enough to diagnose cursor, parsing, and webhook problems.
## Acceptance Criteria
1. Existing legacy tasks without `dispatch` still run through the old shell path with unchanged behavior.
2. MACP-aware `exec` tasks run through the dispatcher and produce result JSON with gate outcomes.
3. New schemas validate task/event/result payload expectations for MACP fields and statuses.
4. `mosaic macp submit`, `status`, and `history` work from a bootstrapped repo state, and `drain` delegates to the existing orchestrator runner.
5. Python imports for the updated controller, dispatcher, and sync code complete without errors on Python 3.10+.
1. `EventWatcher.poll_once()` reads newly appended events, returns parsed dicts, invokes registered callbacks, and skips already-consumed events after restart.
2. Webhook delivery posts matching events to a local test endpoint, supports bearer auth configuration, and retries boundedly on failure.
3. Discord formatter returns expected concise strings for the required task lifecycle event types and a usable batch summary.
4. `mosaic macp watch --once` processes events from a bootstrapped repo state without error and honors `--webhook`.
5. Cursor persistence prevents replay on a second run and resets safely when the events file is truncated.
## Constraints and Dependencies
1. Python implementation must use stdlib only and support Python 3.10+.
2. Shell tooling must remain bash-based and fit the existing Mosaic CLI style.
3. Dispatch fallback rules must use `exec` when `dispatch` is absent and config/default runtime when `runtime` is absent.
4. Worktree convention must derive from the repository name and task metadata unless explicitly overridden by task fields.
2. Shell CLI behavior must remain bash-based and consistent with the existing Mosaic command style.
3. The watcher consumes the event schema already emitted by Phase 1 controller logic.
4. Webhook configuration lives under `.mosaic/orchestrator/config.json` at `macp.webhook`.
## Risks and Open Questions
1. Risk: yolo command execution requires a PTY, so the dispatcher needs a safe wrapper that still behaves under `subprocess`.
2. Risk: worktree cleanup could remove a path unexpectedly if task metadata is malformed.
3. Risk: old queue consumers may assume only the original task statuses and event types.
4. Open Question: whether `task.gated` should be emitted by the dispatcher or controller once worker execution ends and quality gates begin.
1. Risk: partial writes may leave an incomplete trailing JSON line that must not advance the cursor incorrectly.
2. Risk: synchronous webhook retries can slow one poll cycle if the endpoint is unavailable; timeout and retry behavior must remain bounded.
3. Risk: event payloads may omit optional metadata fields, so formatter output must degrade cleanly.
4. ASSUMPTION: the watcher should advance past corrupt lines after logging them so a single bad line does not permanently stall downstream consumption.
5. ASSUMPTION: CLI `watch` should default to no-op callback processing when no delivery option is enabled, while still updating the cursor and reporting processed count.
## Testing and Verification Expectations
1. Baseline checks: Python import validation, targeted script execution checks, JSON syntax/schema validation, and any repo-local validation applicable to changed code paths.
2. Situational testing: legacy orchestrator run with old-style tasks, MACP `exec` flow including result file generation, CLI submit/status/history behavior, and worktree lifecycle validation.
3. Evidence format: command-level results captured in the scratchpad and summarized in the final delivery report.
1. Baseline checks: Python bytecode compilation/import validation for new modules and shell syntax validation for `bin/mosaic-macp`.
2. Situational tests: temporary orchestrator state exercising watcher polling, callback filtering, webhook POST capture, CLI one-shot watch execution, and cursor persistence across repeated runs.
3. Evidence format: command-level results recorded in the scratchpad and summarized against acceptance criteria.
## Milestone / Delivery Intent
1. Target milestone/version: 0.0.x bootstrap enhancement
2. Definition of done: code merged to `main`, CI terminal green, issue `#8` closed, and verification evidence recorded against all acceptance criteria.
## Assumptions
1. ASSUMPTION: A single issue can track the full Phase 1 implementation because the user requested one bounded feature delivery rather than separate independent tickets.
2. ASSUMPTION: For `acp` dispatch in Phase 1, the controller must escalate the task immediately with a clear reason instead of pretending work ran before OpenClaw integration exists.
3. ASSUMPTION: `task.gated` should be emitted by the controller as the transition into quality-gate execution, which keeps gate-state ownership in one place alongside the existing gate loop.
1. Target milestone/version: Phase 2A observability bridge
2. Definition of done: code merged to `main`, CI terminal green, issue `#10` closed, and verification evidence recorded against all acceptance criteria.