stack/docs/scratchpads/1019-zero-timeout-harness.md

#1019 — Zero-timeout queue-guard harness race

• Issue: #1019 (parent status remains believed-fixed, pending jarvis validation; do not close)
• Branch: fix/1019-ci-queue-timeout-harness
• Owner: be-coder-08
• Base: origin/main at 5916aeefd6ed12bcac086c6834c7f6c4ae38e1bc
• Charter: /home/hermes/agent-work/tl-mosaic/CHARTER-1019-HARNESS-FIX.md

Objective

Make test-ci-queue-wait-tristate.sh deterministic without changing any asserted outcome. Remove the indiscriminate zero-timeout race, require every status-classification case to prove the provider was observed, and prove the harness-controlled virtual clock is active.

Scope

• In scope: packages/mosaic/framework/tools/git/test-ci-queue-wait-tristate.sh only, plus this evidence scratchpad.
• Out of scope: guard parsers, D2/D3 behavior, installer/reseed staleness, PR #1060, and issue closure.

Acceptance criteria

1. RED deterministically reproduces deadline pre-emption before the provider call.
2. Every case that intends status classification positively proves provider observation.
3. Pending observes pending before deterministic virtual-time expiration.
4. The virtual clock has a positive interception control; a broken-clock mutant makes the suite red.
5. The exact CI-base image passes the final harness repeatedly with zero failures.
6. Baseline gates, independent code/security review, exact-head CI, and coordinator-authorized squash merge pass.

Plan

1. Add deterministic RED instrumentation for the known merge/provider-unreachable pre-emption.
2. Replace global -t 0 with a nonzero timeout interpreted under an event-driven virtual clock; stub sleep without wall waiting.
3. Add provider-observation and virtual-clock positive controls without changing outcome assertions.
4. Run focused shell checks, repeat in exact CI-base image, baseline gates, and independent reviews.
5. Commit with both identity layers, queue-guard plus direct Woodpecker terminal-state verification, push, self-post PR, verify poster/head/CI, obtain coordinator merge authorization, then squash merge without closing #1019.

Budget

• No explicit token cap supplied. Keep scope to one harness file and one scratchpad; stop/report at the charter's 60% context gate.

Evidence

• RED, deterministic pre-provider expiry: evidence/1019-harness-fix/red-pre-provider-expiry.log — rc 1; merge/provider-unreachable got rc 124 instead of 75, omitted CANNOT_ASSERT, did not observe the status provider, and wrote no additional audit record (four named failures).
• GREEN host focused harness: evidence/1019-harness-fix/green-host.log — rc 0, all outcome classes passed.
• Load-bearing clock negative control: a temporary same-directory mutant replaced the virtual date body with /bin/date; evidence/1019-harness-fix/red-clock-not-intercepted.log — rc 1 with named virtual clock interception did not run failures. The mutant file was removed after the run.
• Exact CI-base repeat: git.mosaicstack.dev/mosaicstack/stack/ci-base:latest, repository mounted read-only, harness work under container /tmp; evidence/1019-harness-fix/ci-image-repeat/summary.log — 100 pass / 0 fail / 100 total.
• Synchronization design: provider-status observation creates the event marker; virtual time is 1000 before the event and 1002 afterward. Pending alone reaches the stubbed no-op sleep and a post-observation deadline check. -t 1 is uniquely load-bearing because removing it restores the 900-second default deadline at virtual time 1900, which 1002 does not cross. The numeric timeout is subject semantics under virtual time, not a wall-clock synchronization duration.

Review remediation — semantic timeout vs. liveness bound

Security review found that virtual time remained at 1000 forever before provider observation and stubbed sleep never waited. A regression looping before the status endpoint—or blocking in the first provider call—therefore could prevent run_guard from returning, so the post-return provider assertion could never fire.

General rule: A timeout usually serves two purposes: semantics and liveness. Removing wall time from semantic synchronization can silently remove the only independent hang bound. Preserve deterministic virtual time for subject semantics, but provide a separately implemented real-clock liveness watchdog and prove that watchdog fires.

Remediation:

• Every guard subject invocation is launched by absolute /usr/bin/python3 in a new session. Python's internal monotonic wait(timeout=...) provides real-clock liveness independently of PATH; expiry kills the entire isolated process group, so neither PATH-front shims nor a blocked provider descendant can retain the capture pipe.
• Watchdog expiry returns distinct harness rc 90 plus FAIL HANG watchdog, separate from subject timeout rc 124.
• A first attempt using absolute /usr/bin/timeout -s KILL passed on GNU coreutils but failed in the exact Alpine CI-base image: BusyBox killed the immediate wrapper while the guard/provider descendants survived and retained the command-substitution pipe. The process-group kill is therefore required behavior, not portability polish.
• A committed positive control hangs the branch-provider stub before the status endpoint. It must terminate through the watchdog, emit the hang-specific diagnostic, return rc 90, and prove the status provider was never reached.
• RED before remediation: a temporary ordinary-success mutant hung before provider observation; only an external control could kill the suite (rc 137), and there was no internal hang-specific diagnostic (red-watchdog-absent.log).
• The watchdog mutant/control is load-bearing: removing the internal watchdog leaves the control unable to produce its required rc 90 and diagnostic.

Post-review evidence:

• Host focused harness with process-group watchdog: rc 0 (green-watchdog-process-group-host.log).
• Exact Alpine CI-base focused harness with process-group watchdog: rc 0 (green-watchdog-ci-image.log).
• Hanging ordinary-success mutant: suite rc 1; success returned rc 90, emitted FAIL HANG watchdog, and loudly reported that provider/clock observation did not occur (red-watchdog-fires.log).
• Removed--t 1 mutant: suite rc 1; pending was terminated by the watchdog instead of producing ASSERTED_NOT_READY, proving the explicit timeout is load-bearing (red-timeout-argument-removed.log).

60% context hold

Stopped before baseline/review/commit as required by the charter. Remaining: inspect final diff, shell/static/baseline gates, independent code/security review, remediation if any, identity-bound commit/trailer verification, mandatory queue guard plus direct terminal Woodpecker mosaic enumeration, push, self-posted PR/provider poster read-back, exact-head terminal-green CI, coordinator merge authorization, squash merge, main CI verification, and leave #1019 unclosed as believed-fixed, pending jarvis validation.