jason.woltje
193479b52d
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
117 lines
6.6 KiB
Markdown
117 lines
6.6 KiB
Markdown
# ACT-1 concept adaptation tests — Darkwing handoff
|
|
|
|
Status: Mosaic concept documentation adopted; synthetic scenarios and preparation utility ready.
|
|
Behavioral trials NOT RUN. Proposed runtime features NOT IMPLEMENTED by this pack.
|
|
Owner: Jason. Preparation author: Codex. Trial operator/reviewer: unassigned;
|
|
Darkwing is the intended collaborator, not automatically retasked.
|
|
|
|
Parent: [ACT-1 plan](../2026-09-07_agent-context-templates-and-migration.md).
|
|
Concepts: [Mosaic Stack concepts](../../concepts/README.md).
|
|
|
|
## Authority and demo boundary
|
|
|
|
Jason requested pulling the selected references and preparing testing with
|
|
Darkwing. This authorizes this preparation alongside the existing demo work.
|
|
It does not authorize changing deployed SOUL/contracts, launching model trials,
|
|
retasking the active Darkwing/Rocko/Filbert sessions, registering automations,
|
|
or implementing every documented concept. Read CURRENT.md before scheduling tests.
|
|
ACT-1's owner demo and migration gates remain in effect.
|
|
|
|
Do not use `agents/darkwing/launch.sh --fresh` for an isolated trial: it still
|
|
uses the repository and Darkwing's native state directory. It is not a test
|
|
isolation command. This pack creates no replacement launcher or harness.
|
|
|
|
## Prepare and inspect now
|
|
|
|
From the repository root:
|
|
|
|
```sh
|
|
node scripts/prepare-concept-tests.mjs --check
|
|
node --test scripts/test-darkwing-launch.mjs
|
|
node scripts/prepare-concept-tests.mjs
|
|
```
|
|
|
|
The first command validates current concept hashes and the scenario inventory without
|
|
writes. The second runs the existing launcher regression suite against test
|
|
doubles in temporary directories; it does not call Docker or a model. The third
|
|
creates a private, uniquely named workspace under the OS temporary directory
|
|
and prints its absolute path. No existing directory is reused or cleaned up.
|
|
|
|
The prepared workspace contains:
|
|
|
|
- `baseline/SOUL.md`: a byte copy of Darkwing's current SOUL, for a baseline trial.
|
|
- `candidate/SOUL.md`: a separate proposed personality for a candidate trial.
|
|
- `fixtures/`: synthetic inputs only, including a fictional USER profile.
|
|
- `concepts/`: Mosaic's conceptual documentation at the recorded content hashes.
|
|
- `provenance/`: original attribution, license and current/source identity manifest.
|
|
- `cases.json`: eleven prompt/expectation records, each linked to a concept.
|
|
- `RESULTS.json`: NOT_RUN rows, evidence fields, provenance, and unset live budget.
|
|
|
|
Baseline and candidate are separate experimental configurations. A trial injects
|
|
exactly one SOUL; never combine them or promote either automatically. The candidate
|
|
is not the approved bootstrap template and has no effect on active Darkwing.
|
|
Temporary storage may be cleaned by the OS; preserve any actual evaluation
|
|
evidence in an assigned durable location before relying on it for acceptance.
|
|
|
|
## Review with Darkwing
|
|
|
|
Give Darkwing this file's path when the demo work permits. A suggested initial
|
|
request follows; no message was sent automatically:
|
|
|
|
> Review ACT-1's concept test package and Mosaic concept documentation. Inspect the prepared
|
|
> synthetic fixtures and identify which expectations fit our accepted foundation
|
|
> contracts. Do not change running sessions, launch inputs, or the demo candidate.
|
|
> Return a source-backed test-readiness review, gaps, and a proposed bounded trial
|
|
> using the existing Mosaic test/launch mechanisms in an isolated workspace.
|
|
> Do not claim the proposed runtime features exist or mark unexecuted cases passed.
|
|
|
|
For a later assigned behavioral trial, record the exact harness/version,
|
|
provider/model, tool allowlist, instruction hashes, fresh test-session identity,
|
|
output/evidence directory, operator, reviewer, and model-call/token/cost budget.
|
|
Budget is currently unspecified; the preparation utility makes zero model calls.
|
|
Authorize an actual evaluation binding before execution. Never borrow a live
|
|
session or credential file by copying it into the fixture workspace.
|
|
|
|
Use a separate test session per case and per variant to avoid prior-answer
|
|
contamination. Inject one SOUL and synthetic user context, provide only the case
|
|
prompt and its fixture, and keep the `expected` rubric out of the model's input.
|
|
The full cases file is for the reviewer. Give the test agent read access to its
|
|
fixture only where the selected harness supports enforcement; a prompt asking
|
|
for restricted access is not itself a sandbox. Disable extensions, scheduling,
|
|
messaging and write tools for these initial reasoning/style cases.
|
|
|
|
Record the actual response and judge each listed expectation. A hard correctness
|
|
or authority error fails the case even if the tone is excellent. Jason separately
|
|
judges useful brevity, candor and personality; do not automate that judgment away.
|
|
Repeat ambiguous cases under the recorded budget, retaining every attempt.
|
|
|
|
## Coverage and limits
|
|
|
|
| Cases | What a model trial can assess | What still needs implementation testing |
|
|
|---|---|---|
|
|
| C01 | Correctly reading effective-context evidence | Actual context resolver, single-SOUL rule, truncation refusal |
|
|
| C02 | Concision and directness | No runtime feature claim |
|
|
| C03 | Protecting active/unknown-owner work in a recommendation | Real worktree leases, snapshot restore and cleanup refusal |
|
|
| C04 | Recognizing stale state, scope changes and pauses | Durable events, watchers, history gaps and restart reconciliation |
|
|
| C05 | Distinguishing queued, running and canceled work | Real harness steering and tool-start boundaries |
|
|
| C06, C11 | Separating identity/attribution from access/control | Two-client attachment and authorization enforcement |
|
|
| C07 | Honest interpretation of adapter capability evidence | Pinned adapter conformance tests |
|
|
| C08 | Honest completion reporting | No substitute for actual product acceptance |
|
|
| C09 | Memory source skepticism and deletion limits | Memory admission/lineage/deletion implementation |
|
|
| C10 | Distinguishing events, schedules and aspirations | Authorized wake registration, matching and cancellation |
|
|
|
|
Case IDs in `cases.json` are authoritative. Runtime obligations in `runtimeGate`
|
|
remain DEFERRED until an implementation and assignment exist. Model answers to
|
|
synthetic fixtures must never be reported as passing those runtime gates.
|
|
|
|
Allowed result states: NOT_RUN, PASS, FAIL, BLOCKED, DEFERRED. PASS requires
|
|
actual output, environment and reviewer evidence. Existing launcher test success
|
|
does not populate the model-case results. No external sends, real-user profile
|
|
reads, live source writes, or destructive scenarios are part of this pack.
|
|
|
|
## Next handoff
|
|
|
|
Darkwing reviews readiness after reconciliation with the current demo. Jason
|
|
selects the live test timing/scope; actual execution records its own evidence.
|
|
Any migration implementation remains a separately bounded ACT-1 increment.
|