stack/docs/scratchpads/838-broker-acceptance-flake.md

Issue #838 — Broker acceptance socket flake

Objective

Eliminate high-contention uncaught JSON parse failures in lease-broker and mutator-gate acceptance helpers without laundering malformed/empty replies into passing assertions.

Constraints

• Branch: fix/838-broker-acceptance-flake in /home/hermes/agent-work/stack-838-flakefix.
• Red-first TDD with a forced empty/truncated reply path and deterministic rejection or documented retry.
• Read newline-framed replies completely; reject malformed broker replies with byte length/content context.
• Determine RIDER2 branch before finalizing: test-harness-only (a) or daemon write truncation (b).
• If product-side, prove the real Claude/Pi adapter read path fails closed; fix any allow-risk.
• Coverage >=85% per changed executable through real tests.
• Full suite and repository gates green; independent exact-head review required.
• Push and open a PR containing closes #838; do not merge.

Progress

• 2026-07-17: Reclaimed from #824. Confirmed clean worktree on fix/838-broker-acceptance-flake at main base abd2791f59b3f06f46dd08e55298ced72f6aa7c2.
• RED evidence: after dependency setup, broker-test-client.spec.ts failed to load the intentionally absent shared client module. Its contract forces empty-close retry, repeated truncated-close rejection with byte context, and newline-terminated malformed-reply rejection.
• RIDER2 verdict: branch (b). daemon.py starts one connection deadline before request reading, then may spend that budget waiting for broker_lock; after handling, remaining <= 0 returns without writing. A timed sendall failure can likewise close after a partial write. A deterministic socketpair probe held the lock past CONNECTION_DEADLINE_SECONDS and observed deadline_probe_reply_length=0.
• Adapter tripwire: real subprocess executions of mutator-gate.py for both --runtime claude and --runtime pi against actual Unix servers returning empty and truncated replies all exited 2 with GATE_UNAVAILABLE. Adapter fail-close is proven; there is no ALLOW risk.
• Residual: production broker reply loss remains an availability-denial path under extreme contention, but cannot grant mutator authority. The acceptance-only client retries early closes and otherwise rejects with response byte length, escaped bytes, and hex; malformed newline-framed replies are never retried or converted to reply objects.
• Shared client now owns newline framing and parsing for both acceptance suites. Focused suites pass 60 tests, including the real adapter tripwire.
• Coverage gate: changed helper is 100% statements/lines/functions and 90% branches; existing skill.ts remains above 85% per-file thresholds.

Scope corrections and bounded product repair

• Coordinator correction superseded the initial push/PR and retry language: this lane is BUILD-ONLY, and early-close retries are forbidden because they can mask a committed broker transaction. Nothing may be pushed or opened until explicitly cleared.
• Revised RED evidence: the no-retry empty/truncated tests failed because the first failed exchange was retried into { ok: true }; the daemon regression failed because a slow completed broker.handle() produced b'' instead of a newline-framed reply.
• Client repair: both former duplicated helpers use one shared reader. Empty, truncated, malformed, oversized, timed-out, and socket-error replies reject BrokerTransportError with a typed kind, attempt count fixed at one, response length, escaped bytes, and hex. No retry and no catch-to-reply conversion exists.
• Product repair: daemon.py now has independent bounded read, broker-lock queue, and send budgets. Lock queue exhaustion returns explicit BROKER_BUSY before broker.handle() can mutate state. Once handling starts it finishes atomically, and its reply always receives a fresh send timeout instead of being skipped because read/lock/fsync consumed a shared deadline.
• Product GREEN evidence: deterministic socketpair tests prove lock saturation returns framed BROKER_BUSY without invoking handle(), and a handle that completes after the former one-second shared deadline still returns its complete framed reply.
• RIDER2b remains fail-closed: real Claude and Pi mutator-gate.py subprocesses against empty and truncated Unix-socket replies exit 2 with GATE_UNAVAILABLE; no malformed/default ALLOW was observed.
• Residual/tripwire: an unavoidable peer disconnect or send failure can still lose acknowledgement after a valid transaction commits. The affected adapter call fails closed. A valid promote_lease may nevertheless remain VERIFIED after its acknowledgement is lost; that is authority-observability divergence requiring WI-3/Opus security review rather than expansion of #838. #838 does not add retries or attempt a protocol redesign.
• Final package evidence: recursive Mosaic dependency build passed; 73 Vitest files / 1,392 tests passed; deadline unit tests 2/2, real runtime tool tests 15/15, launch guard tests 12/12, inventory 14/14, and shell regressions passed.
• Final coverage: broker-test-client.ts 99.24% statements/lines, 86.11% branches, 100% functions under per-file >=85% thresholds. Repository typecheck 42/42, lint 23/23, and format check passed.
• Independent Codex exact-head review requested one framing fix: a valid frame followed by trailing bytes in a later socket chunk could resolve before the garbage arrived. Security review also flagged complete token-bearing reply bodies in diagnostic properties/logs.
• Review RED evidence: delayed cross-chunk garbage resolved { ok: true } instead of rejecting, and a truncated promotion_token remained in the typed error. The tests use separate timed writes to prevent kernel/event-loop coalescing.
• Review remediation: the shared client now accumulates through EOF, requires exactly one terminal newline, then parses inside a rejecting error boundary. Diagnostic bodies are capped at 256 bytes, sensitive broker fields are fully redacted, and a SHA-256 digest preserves correlation without credential disclosure.
• Post-review coverage: broker-test-client.ts 99.35% statements/lines, 86.66% branches, 100% functions. Final full suite is 73 files / 1,394 tests plus all Python/shell gates; recursive build, typecheck, lint, and format are green.
• Fresh exact-head Codex security review: risk none, no findings. Fresh code review found only that the real-adapter subprocess proof lacked a timeout; it now has a five-second bound and fails with runtime/wire context while closing the fake server. The focused Python suite remains 15/15 green.
• Mandatory Opus SECREV remains coordinator-owned and pending before any push/PR decision; this build is intentionally local-only.