plan(pi): benchmark and improve harness task-completion competitiveness #767

New Issue

jason.woltje · 2026-07-14T21:42:28Z

jason.woltje commented

2026-07-14 21:42:28 +00:00

Problem

The Pi harness appears to stop or idle before comparable Claude Code and Codex runs reach a verified task outcome. Treat this as a measurable harness/control-loop gap, not as evidence that an unbounded retry loop is safe.

Planning goal

Benchmark and compare Pi, Claude Code, Codex, and OpenCode under identical bounded tasks, then use opposed planning to define a competitive Pi execution policy and a dependency-ordered implementation plan.

Capability matrix

Autonomous continuation and explicit stop conditions
Tool-call retry taxonomy, bounded backoff, and idempotency
Heartbeat, progress receipts, stall detection, and wake/resume
Checkpointing, context compaction/reset, and crash recovery
Permission/tool failure handling and deterministic escalation
Test/review/verification-to-completion behavior
Background work and subagent completion handling
Cancellation, quarantine, leases, fences, and ownership
Cost/token/time budgets and operator-visible telemetry
Canaries, rollback, and failure evidence

Required planning method

Build reproducible benchmark fixtures and baseline evidence.
Produce a source-linked capability matrix; distinguish model behavior from harness behavior.
Run opposed Sol/Terra planning: completion competitiveness versus runaway cost, duplicate effects, stale authority, and state corruption.
Synthesize an explicit execution state machine owned by Coordinator policy, with bounded retries, durable checkpoints, progress receipts, stall detection, escalation, and quarantine.
Decompose implementation into independently reviewable cards with dependencies, acceptance tests, canary, and rollback gates.

Acceptance criteria for the planning issue

Benchmark protocol and fixtures are reproducible and bounded.
Comparisons use identical tasks and distinguish harness features from model quality.
Every proposed retry is classified, bounded, observable, cancellable, and safe under duplicate execution.
Heartbeats prove liveness only; they do not self-grant task, fleet, connector, or merge authority.
Resume/recovery binds to the current lease, fence, runtime incarnation, task execution generation, and policy revision.
Completion requires explicit task acceptance evidence, not activity or token consumption.
Dependency-ordered implementation cards include instrumentation, policy/state machine, adapters, UX/docs, acceptance tests, canary, and rollback.
Results are integrated into North Star and canonical project tracking before implementation.

Non-goals / boundaries

No blind or infinite retry loop.
No hidden keepalive that masks a stalled or failed task.
No implementation before the benchmark and opposed-plan gate.
No overlap with #766 cross-harness tmux target correctness.
No overlap with #758 declarative fleet configuration cards.
No change to merge, connector, federation, or roster authority.

## Problem The Pi harness appears to stop or idle before comparable Claude Code and Codex runs reach a verified task outcome. Treat this as a measurable harness/control-loop gap, not as evidence that an unbounded retry loop is safe. ## Planning goal Benchmark and compare Pi, Claude Code, Codex, and OpenCode under identical bounded tasks, then use opposed planning to define a competitive Pi execution policy and a dependency-ordered implementation plan. ## Capability matrix - Autonomous continuation and explicit stop conditions - Tool-call retry taxonomy, bounded backoff, and idempotency - Heartbeat, progress receipts, stall detection, and wake/resume - Checkpointing, context compaction/reset, and crash recovery - Permission/tool failure handling and deterministic escalation - Test/review/verification-to-completion behavior - Background work and subagent completion handling - Cancellation, quarantine, leases, fences, and ownership - Cost/token/time budgets and operator-visible telemetry - Canaries, rollback, and failure evidence ## Required planning method 1. Build reproducible benchmark fixtures and baseline evidence. 2. Produce a source-linked capability matrix; distinguish model behavior from harness behavior. 3. Run opposed Sol/Terra planning: completion competitiveness versus runaway cost, duplicate effects, stale authority, and state corruption. 4. Synthesize an explicit execution state machine owned by Coordinator policy, with bounded retries, durable checkpoints, progress receipts, stall detection, escalation, and quarantine. 5. Decompose implementation into independently reviewable cards with dependencies, acceptance tests, canary, and rollback gates. ## Acceptance criteria for the planning issue - Benchmark protocol and fixtures are reproducible and bounded. - Comparisons use identical tasks and distinguish harness features from model quality. - Every proposed retry is classified, bounded, observable, cancellable, and safe under duplicate execution. - Heartbeats prove liveness only; they do not self-grant task, fleet, connector, or merge authority. - Resume/recovery binds to the current lease, fence, runtime incarnation, task execution generation, and policy revision. - Completion requires explicit task acceptance evidence, not activity or token consumption. - Dependency-ordered implementation cards include instrumentation, policy/state machine, adapters, UX/docs, acceptance tests, canary, and rollback. - Results are integrated into North Star and canonical project tracking before implementation. ## Non-goals / boundaries - No blind or infinite retry loop. - No hidden keepalive that masks a stalled or failed task. - No implementation before the benchmark and opposed-plan gate. - No overlap with #766 cross-harness tmux target correctness. - No overlap with #758 declarative fleet configuration cards. - No change to merge, connector, federation, or roster authority.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#767