stack/docs/plans/act-1-tests/2026-09-07_act-04-darkwing-test-readiness-review.md

ACT-04 test package readiness review (Darkwing)

Date: 2026-09-07. Author: Darkwing, at Jason's request. Scope: review only. No model calls, no trial execution, no session, launcher or demo-candidate changes, no staging, commit or push.

Reviewed sources: pack README, cases.json, candidate SOUL, fixtures, baseline agents/darkwing/SOUL.md, ACT-1 plan, concepts index, scripts/prepare-concept-tests.mjs, scripts/test-darkwing-launch.mjs, scripts/agent-host-dev.sh, CURRENT.md.

Verification performed

• node scripts/prepare-concept-tests.mjs --check passed: 13 Mosaic concepts, license attribution, 11 synthetic scenarios; zero model calls.
• node --test scripts/test-darkwing-launch.mjs passed 2/2 in temporary fixtures; no Docker, model or live state touched.
• Read the pinned pi CLI help (0.84.4): -p, --no-tools, --session-dir, --append-system-prompt, --no-context-files --no-skills --no-extensions are all available for headless trial runs.

Scenario review against accepted foundation decisions

All 11 cases test judgment and honesty, not runtime features, and every runtimeGate stays DEFERRED. That matches the pack's own limits. Case by case:

Case	Foundation anchor	Verdict
C01	D3 single SOUL, no fallback; fail-closed mandatory context	Sound. Fixture inventory excludes root and contract SOULs with reasons that match D3 and R16
C02	D2 directness, useful brevity	Sound; no runtime claim
C03	Write-once preservation, managed-worktree concept (proposed)	Sound; worktree-is-not-isolation expectation matches worker sandbox boundaries
C04	Pause-then-reconcile owner ruling (Q26); never guess, verify	Sound; historyGap expectation matches restart-reconciliation decisions
C05	Honest interruption semantics; design claim vs observed behavior	Sound; no-cancel-claim expectation mirrors SOUL evidence discipline
C06, C11	One controlling interface, observers without control (Q16); least privilege	Sound; attribution-does-not-grant-access matches policy invariants
C07	Startup marker is not verification; STANDARDS forbids relabeling skipped checks	Sound; the strongest discipline case in the pack
C08	Accurate failure reporting; acceptance separate from implementation	Sound; mirrors current FI-FILBERT-8 discipline
C09	Authorization never accrues from elapsed time or repetition	Sound; external-claim expectation matches role-authority rules
C10	Wake registration requires authority; no prose-note scheduling	Sound; matches the no-automatic-continuation rule

No case asks the model to claim a proposed feature exists. No case touches a real user profile, live session, credential or destructive action. The expected rubrics stay out of model input under the proposed trial mechanism.

Gaps

• G1, C01 self-reference risk. The prompt says to treat context-inventory.json as synthetic data, but the trial agent is itself launched with a SOUL from the workspace. A sharp model may answer about its own context instead of the fixture. Not a defect, but judges must require the answer to cite the fixture. Keep the synthetic-data framing verbatim when composing input.
• G2, launcher does not fit trial isolation. scripts/agent-host-dev.sh injects repo AGENTS.md, STANDARDS, the live data-root USER file, 10 skills, the goal extension and write tools, and stores sessions under $REPO/.pi/state/<agent>. Its --soul/--user overrides help, but the rest cannot be turned off. The pack requires exactly one SOUL, synthetic user context, no extensions or write tools, and a fresh session per case and variant. Running the pinned pi binary headless from the staged workspace covers all of that without adding repo code or a new launcher.
• G3, baseline and candidate are not style-only twins. The baseline SOUL carries Mosaic operating disciplines: evidence classes, preserving other agents' work, no self-granted authority. The candidate is voice only, with one honesty sentence. Eight of eleven cases largely test judgment. A candidate FAIL there measures missing content, not tone; a PASS shows the voice holds without explicit rules. Both readings are useful, but the trial record should state which question is being asked so a FAIL is not misread as a style defect.
• G4, judging protocol is unwritten. RESULTS.json rows have evidence fields but no per-expectation verdict shape. Proposal: judges record each expectation as pass or fail with a quoted line from the actual output; style calls stay with Jason, never automated.
• G5, evidence durability. The staged workspace lives in the OS temp directory and may be wiped. Actual outputs and RESULTS.json must be copied to an assigned durable location before any acceptance relies on them. Destination needs Jason's call; a trials/ subfolder here is one option and would be a repo write needing his authorization.

Proposed first trial: bounded baseline versus candidate

Scope: two cases, two variants, four model calls. Optional single repeat of an ambiguous case, capped at eight calls total.

Cases: ACT-C02 (concision; no fixture; pure style comparison) and ACT-C08 (honest completion reporting; hard rubric under the candidate's voice). Both have runtimeGate: null, so nothing in the trial can be confused with a runtime feature test. Together they answer the two questions that matter first: does the candidate's voice read better on a plain answer, and does it keep hard honesty discipline without the baseline's explicit rules.

Mechanism, reusing existing verified pieces only:

1. node scripts/prepare-concept-tests.mjs stages a private 0700 workspace. Zero model calls. No repo directory is written.
2. Per run, compose one context file: contracts/CONSTITUTION.md plus fixtures/USER.md (synthetic Casey profile) plus exactly one SOUL, the variant's. No AGENTS.md, no STANDARDS, no skills, no second SOUL.
3. Run the pinned node_modules/.bin/pi headless: -p, --no-tools (fixtures are inlined in the prompt, so there is nothing to sandbox), --no-context-files --no-skills --no-extensions --no-prompt-templates --no-themes, --append-system-prompt <composed file>, provider and model taken from the validated load_config values, --session-dir <workspace>/sessions/<variant>/<case-id> so every case and variant gets a fresh session identity.
4. Authentication is referenced in place if required, never copied into the workspace, and never printed.
5. Save each actual response under the workspace; update the four RESULTS.json cells for these cases with status, evidence and quoted expectation verdicts. All other 18 rows stay NOT_RUN.
6. Copy outputs and RESULTS.json to the durable location Jason names.

Recorded before execution, per the pack's binding requirements: harness and version (pi 0.84.4 pinned), provider/model, context file SHA-256 per run, both SOUL hashes (already in RESULTS.json variants), session directories, operator and reviewer names, and the model-call budget.

What this trial does not do: no runtime gate moves to PASS, no template is promoted, the candidate never touches active Darkwing, and nothing here is demo acceptance. A hard rubric failure fails the case regardless of tone; Jason alone judges brevity, candor and personality.

Preserved by this review

agents/darkwing/** including SOUL, launcher and CONTEXT; scripts/agent*.sh and all launch inputs; .pi/state/darkwing sessions; the Rocko and Filbert tmux sessions and frozen demo candidate; Dewey's shared git index. Nothing staged, committed or pushed. CURRENT.md's next action (await FI-FILBERT-8 r6 verdict) is unchanged by this review.

Decisions needed from Jason

1. Authorize trial execution and name operator, reviewer, and the call/cost budget (proposal: 4, hard cap 8).
2. Choose the durable evidence location.
3. Later: approve the G1 judging note before C01 ever runs, and confirm G3's interpretation rule, that a candidate judgment FAIL is a content finding first, belongs in the trial record.

Correction, appended 2026-09-07 (STANDARDS attribution)

Per Jason's direction. The C07 row above says "STANDARDS forbids relabeling skipped checks". That attribution is wrong. contracts/STANDARDS.md contains only the startup-verification answer rule: answer with the requested value, no added explanation or formatting. The "cannot relabel skipped checks as passed" language comes from the ACT-1 plan's "Current behavior to preserve and then migrate" section on startup-marker separation, and the evidence classes discipline lives in the SOUL and AGENTS.md, not in STANDARDS.

The C07 expectation itself (refusing to infer hook enforcement from a passing startup check) remains sound and anchored; only the citation was wrong. The original row text is kept as written per the corrections-are-appended rule.

G3 resolution recorded in the trial record: the first trial compares complete SOUL configurations, not a controlled style-only experiment, and no failure cause may be inferred from it. Trial evidence: .pi/evidence/act-1/2026-09-07T1625Z-c02-c08-r1/.