jason.woltje
193479b52d
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
2.0 KiB
Agent behavior tests
Status: ACT-1's preparation package exists; model trials remain NOT_RUN unless an execution record establishes otherwise.
Use small repository-owned cases to evaluate specific behaviors under recorded instructions and runtime settings. A test should name what it proves and what it does not. Avoid a second runner when an existing harness can execute the case.
Test inputs and isolation
Use synthetic people, preferences, work records and diagnostic data. Give every trial an explicit isolated workspace/session binding. Do not use the live Darkwing, Filbert, Heffer or Rocko conversation merely because its name is familiar.
A personality comparison injects exactly one SOUL per trial. Keep baseline and candidate configurations distinct, record hashes, and exclude the review rubric from the model input. Agent-visible data must be limited to the case and its authorized context. Prompt instructions alone do not enforce filesystem isolation.
Evidence and scoring
Record provider/model, harness/version, approved tools, context identity, budget, actual response, verification result and reviewer. Preserve failed and ambiguous attempts. Use NOT_RUN, PASS, FAIL, BLOCKED and DEFERRED accurately.
Mechanical tests can verify wiring and refusal behavior. Model trials can assess reasoning and style. Neither can substitute for the other's evidence. A good answer about a synthetic access record does not prove runtime access control.
Hard failures include invented completion, claimed authority without evidence, wrong identity and misreporting failed or skipped checks. Jason judges useful brevity, candor and personality separately; do not reward forced humor or confidence unsupported by evidence.
The current ACT-1 pack has eleven synthetic cases and a preparation utility. It also reuses existing launcher regressions. Real model calls, live transports and runtime-feature tests require their assigned scope; preparing a fixture does not start them.