jason.woltje
193479b52d
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
157 lines
9.2 KiB
Markdown
157 lines
9.2 KiB
Markdown
# ACT-04 test package readiness review (Darkwing)
|
|
|
|
Date: 2026-09-07. Author: Darkwing, at Jason's request.
|
|
Scope: review only. No model calls, no trial execution, no session, launcher
|
|
or demo-candidate changes, no staging, commit or push.
|
|
|
|
Reviewed sources: [pack README](README.md), [cases.json](cases.json),
|
|
[candidate SOUL](candidate/SOUL.md), [fixtures](fixtures/), baseline
|
|
`agents/darkwing/SOUL.md`, [ACT-1 plan](../2026-09-07_agent-context-templates-and-migration.md),
|
|
[concepts index](../../concepts/README.md), `scripts/prepare-concept-tests.mjs`,
|
|
`scripts/test-darkwing-launch.mjs`, `scripts/agent-host-dev.sh`, CURRENT.md.
|
|
|
|
## Verification performed
|
|
|
|
- `node scripts/prepare-concept-tests.mjs --check` passed: 13 Mosaic concepts,
|
|
license attribution, 11 synthetic scenarios; zero model calls.
|
|
- `node --test scripts/test-darkwing-launch.mjs` passed 2/2 in temporary
|
|
fixtures; no Docker, model or live state touched.
|
|
- Read the pinned pi CLI help (0.84.4): `-p`, `--no-tools`, `--session-dir`,
|
|
`--append-system-prompt`, `--no-context-files --no-skills --no-extensions`
|
|
are all available for headless trial runs.
|
|
|
|
## Scenario review against accepted foundation decisions
|
|
|
|
All 11 cases test judgment and honesty, not runtime features, and every
|
|
`runtimeGate` stays DEFERRED. That matches the pack's own limits. Case by case:
|
|
|
|
| Case | Foundation anchor | Verdict |
|
|
|---|---|---|
|
|
| C01 | D3 single SOUL, no fallback; fail-closed mandatory context | Sound. Fixture inventory excludes root and contract SOULs with reasons that match D3 and R16 |
|
|
| C02 | D2 directness, useful brevity | Sound; no runtime claim |
|
|
| C03 | Write-once preservation, managed-worktree concept (proposed) | Sound; worktree-is-not-isolation expectation matches worker sandbox boundaries |
|
|
| C04 | Pause-then-reconcile owner ruling (Q26); never guess, verify | Sound; historyGap expectation matches restart-reconciliation decisions |
|
|
| C05 | Honest interruption semantics; design claim vs observed behavior | Sound; no-cancel-claim expectation mirrors SOUL evidence discipline |
|
|
| C06, C11 | One controlling interface, observers without control (Q16); least privilege | Sound; attribution-does-not-grant-access matches policy invariants |
|
|
| C07 | Startup marker is not verification; STANDARDS forbids relabeling skipped checks | Sound; the strongest discipline case in the pack |
|
|
| C08 | Accurate failure reporting; acceptance separate from implementation | Sound; mirrors current FI-FILBERT-8 discipline |
|
|
| C09 | Authorization never accrues from elapsed time or repetition | Sound; external-claim expectation matches role-authority rules |
|
|
| C10 | Wake registration requires authority; no prose-note scheduling | Sound; matches the no-automatic-continuation rule |
|
|
|
|
No case asks the model to claim a proposed feature exists. No case touches a
|
|
real user profile, live session, credential or destructive action. The
|
|
`expected` rubrics stay out of model input under the proposed trial mechanism.
|
|
|
|
## Gaps
|
|
|
|
- **G1, C01 self-reference risk.** The prompt says to treat
|
|
`context-inventory.json` as synthetic data, but the trial agent is itself
|
|
launched with a SOUL from the workspace. A sharp model may answer about its
|
|
own context instead of the fixture. Not a defect, but judges must require
|
|
the answer to cite the fixture. Keep the synthetic-data framing verbatim
|
|
when composing input.
|
|
- **G2, launcher does not fit trial isolation.** `scripts/agent-host-dev.sh`
|
|
injects repo AGENTS.md, STANDARDS, the live data-root USER file, 10 skills,
|
|
the goal extension and write tools, and stores sessions under
|
|
`$REPO/.pi/state/<agent>`. Its `--soul/--user` overrides help, but the rest
|
|
cannot be turned off. The pack requires exactly one SOUL, synthetic user
|
|
context, no extensions or write tools, and a fresh session per case and
|
|
variant. Running the pinned pi binary headless from the staged workspace
|
|
covers all of that without adding repo code or a new launcher.
|
|
- **G3, baseline and candidate are not style-only twins.** The baseline SOUL
|
|
carries Mosaic operating disciplines: evidence classes, preserving other
|
|
agents' work, no self-granted authority. The candidate is voice only, with
|
|
one honesty sentence. Eight of eleven cases largely test judgment. A
|
|
candidate FAIL there measures missing content, not tone; a PASS shows the
|
|
voice holds without explicit rules. Both readings are useful, but the trial
|
|
record should state which question is being asked so a FAIL is not
|
|
misread as a style defect.
|
|
- **G4, judging protocol is unwritten.** RESULTS.json rows have evidence
|
|
fields but no per-expectation verdict shape. Proposal: judges record each
|
|
expectation as pass or fail with a quoted line from the actual output;
|
|
style calls stay with Jason, never automated.
|
|
- **G5, evidence durability.** The staged workspace lives in the OS temp
|
|
directory and may be wiped. Actual outputs and RESULTS.json must be copied
|
|
to an assigned durable location before any acceptance relies on them.
|
|
Destination needs Jason's call; a `trials/` subfolder here is one option
|
|
and would be a repo write needing his authorization.
|
|
|
|
## Proposed first trial: bounded baseline versus candidate
|
|
|
|
Scope: two cases, two variants, four model calls. Optional single repeat of
|
|
an ambiguous case, capped at eight calls total.
|
|
|
|
Cases: ACT-C02 (concision; no fixture; pure style comparison) and ACT-C08
|
|
(honest completion reporting; hard rubric under the candidate's voice). Both
|
|
have `runtimeGate: null`, so nothing in the trial can be confused with a
|
|
runtime feature test. Together they answer the two questions that matter
|
|
first: does the candidate's voice read better on a plain answer, and does it
|
|
keep hard honesty discipline without the baseline's explicit rules.
|
|
|
|
Mechanism, reusing existing verified pieces only:
|
|
|
|
1. `node scripts/prepare-concept-tests.mjs` stages a private 0700 workspace.
|
|
Zero model calls. No repo directory is written.
|
|
2. Per run, compose one context file: `contracts/CONSTITUTION.md` plus
|
|
`fixtures/USER.md` (synthetic Casey profile) plus exactly one SOUL, the
|
|
variant's. No AGENTS.md, no STANDARDS, no skills, no second SOUL.
|
|
3. Run the pinned `node_modules/.bin/pi` headless: `-p`, `--no-tools`
|
|
(fixtures are inlined in the prompt, so there is nothing to sandbox),
|
|
`--no-context-files --no-skills --no-extensions --no-prompt-templates
|
|
--no-themes`, `--append-system-prompt <composed file>`, provider and model
|
|
taken from the validated `load_config` values, `--session-dir
|
|
<workspace>/sessions/<variant>/<case-id>` so every case and variant gets a
|
|
fresh session identity.
|
|
4. Authentication is referenced in place if required, never copied into the
|
|
workspace, and never printed.
|
|
5. Save each actual response under the workspace; update the four
|
|
RESULTS.json cells for these cases with status, evidence and quoted
|
|
expectation verdicts. All other 18 rows stay NOT_RUN.
|
|
6. Copy outputs and RESULTS.json to the durable location Jason names.
|
|
|
|
Recorded before execution, per the pack's binding requirements: harness and
|
|
version (pi 0.84.4 pinned), provider/model, context file SHA-256 per run,
|
|
both SOUL hashes (already in RESULTS.json variants), session directories,
|
|
operator and reviewer names, and the model-call budget.
|
|
|
|
What this trial does not do: no runtime gate moves to PASS, no template is
|
|
promoted, the candidate never touches active Darkwing, and nothing here is
|
|
demo acceptance. A hard rubric failure fails the case regardless of tone;
|
|
Jason alone judges brevity, candor and personality.
|
|
|
|
## Preserved by this review
|
|
|
|
`agents/darkwing/**` including SOUL, launcher and CONTEXT; `scripts/agent*.sh`
|
|
and all launch inputs; `.pi/state/darkwing` sessions; the Rocko and Filbert
|
|
tmux sessions and frozen demo candidate; Dewey's shared git index. Nothing
|
|
staged, committed or pushed. CURRENT.md's next action (await FI-FILBERT-8 r6
|
|
verdict) is unchanged by this review.
|
|
|
|
## Decisions needed from Jason
|
|
|
|
1. Authorize trial execution and name operator, reviewer, and the call/cost
|
|
budget (proposal: 4, hard cap 8).
|
|
2. Choose the durable evidence location.
|
|
3. Later: approve the G1 judging note before C01 ever runs, and confirm G3's
|
|
interpretation rule, that a candidate judgment FAIL is a content finding
|
|
first, belongs in the trial record.
|
|
|
|
## Correction, appended 2026-09-07 (STANDARDS attribution)
|
|
|
|
Per Jason's direction. The C07 row above says "STANDARDS forbids relabeling
|
|
skipped checks". That attribution is wrong. `contracts/STANDARDS.md` contains
|
|
only the startup-verification answer rule: answer with the requested value,
|
|
no added explanation or formatting. The "cannot relabel skipped checks as
|
|
passed" language comes from the ACT-1 plan's "Current behavior to preserve
|
|
and then migrate" section on startup-marker separation, and the evidence
|
|
classes discipline lives in the SOUL and AGENTS.md, not in STANDARDS.
|
|
|
|
The C07 expectation itself (refusing to infer hook enforcement from a passing
|
|
startup check) remains sound and anchored; only the citation was wrong. The
|
|
original row text is kept as written per the corrections-are-appended rule.
|
|
|
|
G3 resolution recorded in the trial record: the first trial compares complete
|
|
SOUL configurations, not a controlled style-only experiment, and no failure
|
|
cause may be inferred from it. Trial evidence:
|
|
`.pi/evidence/act-1/2026-09-07T1625Z-c02-c08-r1/`.
|