Files
stack/docs/design/framework-constitution/debate/redteam-contrarian.md
Jason Woltje c70b217a5c
Some checks failed
ci/woodpecker/push/ci Pipeline failed
docs(design): mosaic framework constitution — expert conference output
Conference of 7 experts (architect/moonshot/contrarian/coder/aiml/devex/steward)
debated layering, sanitization, upgrade-safety, cross-harness robustness.
Artifacts: BRIEF, 7 positions, 7 rebuttals, synthesis-v1, 3 red-team passes,
canonical DESIGN.md, OPEN-QUESTIONS.md, MISSION.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 23:47:49 -05:00

294 lines
23 KiB
Markdown

# Red-Team — Contrarian Skeptic vs. Synthesis v1
**Lens:** Contrarian Skeptic. Distrusts clever abstractions; hunts failure modes, over-engineering,
and rules that read well but degrade real agent behavior. I tried to *break* the design in
`synthesis-v1.md`, grounding every claim in the real tree. The synthesis already absorbed a lot of
contrarian input, so I went after what *survived* or was *newly introduced* by the ruling itself.
**Verdict:** The layering and sanitization decisions are sound. But the synthesis's **headline drift
fix is mechanically wrong** — it does not do what it claims, and the alpha would ship believing the
drift bug is fixed when it is not. That is a blocker. Several other claims are aspirational controls
presented as settled.
---
## R1 — BLOCKER: "Remove from PRESERVE_PATHS" does NOT make gate updates reach existing installs
This is the synthesis's central, most-repeated claim — settled-item #7, D4, §5.1, and the alpha DoD all
assert that removing `AGENTS.md`/`STANDARDS.md` from `PRESERVE_PATHS` is *"the single change [that]
makes gate updates reach every existing install (the literal drift bug)."* **It does not.** I traced
the actual install/launch code:
1. The resident, injected contract is the **root** file `~/.config/mosaic/AGENTS.md`. Proof:
`packages/mosaic/src/commands/launch.ts:326``parts.push(readFileSync(join(MOSAIC_HOME, 'AGENTS.md')))`.
It never reads `defaults/AGENTS.md`.
2. That root file is **seeded once and never re-seeded.** Proof, both install paths:
- `install.sh:235-240`: `for default_file in AGENTS.md STANDARDS.md TOOLS.md; do if [[ -f "$DEFAULTS_DIR/$default_file" ]] && [[ ! -f "$TARGET_DIR/$default_file" ]]; then cp ...` — the `! -f` guard means an existing root file is skipped.
- `file-adapter.ts:184-190`: `for (const entry of DEFAULT_SEED_FILES) { ... if (existsSync(dest)) continue; ... copyFileSync(...) }` — same seed-once semantics.
3. `defaults/` itself is rsynced into `~/.config/mosaic/defaults/` as a subdirectory, so removing the
root file from `PRESERVE_PATHS` only refreshes the *non-resident* `defaults/AGENTS.md` copy that
**nothing injects.**
**Net effect of the synthesis's fix as written:** rsync `--delete` now also deletes the user's
customized root `AGENTS.md` on every `keep` upgrade (because it's no longer preserved) — but the seed
loop will **not** put the new one back, because… actually it *will*, since the file is now absent — but
only by accident, and only on the bash path. The two sync implementations (`install.sh` and
`file-adapter.ts`) must stay byte-identical (`file-adapter.ts:148` says so explicitly) and the
synthesis **never mentions `file-adapter.ts` exists.** Any fix applied to one and not the other
silently diverges the bash-install and npm-install upgrade behavior — exactly the cross-path drift the
project already warns about in that comment.
The deeper trap: the seed mechanism is "copy if absent," which is **structurally incompatible** with
"framework-owned, overwritten every upgrade." You cannot make a file both *seeded-once-then-user-owned*
(today's model) and *clobbered-every-upgrade* (the Constitution model) by editing a path list. The
synthesis's L0 doctrine requires the seed-if-absent logic for `AGENTS.md`/`CONSTITUTION.md`/`STANDARDS.md`
to be **replaced with unconditional overwrite**, in *both* `install.sh` and `file-adapter.ts`, plus the
`DEFAULT_SEED_FILES` list at `file-adapter.ts:16` re-thought. None of that is in the plan.
**Mitigation (required before alpha):**
- Constitution model: L0 `CONSTITUTION.md` and the dispatcher `AGENTS.md` must be **unconditionally
copied/overwritten** at the root on every upgrade (not seed-if-absent), in `install.sh` AND
`file-adapter.ts`. Add a test fixture asserting that an upgrade over a *modified* root `AGENTS.md`
replaces it.
- Add `file-adapter.ts` (and `DEFAULT_SEED_FILES`) to the file-by-file plan in §2b. The synthesis is
incomplete: it plans the bash installer and the markdown, not the TS installer that ships in the npm
package.
- The migration fixture matrix in §5.5 must assert the *injected resident bytes* (what `launch.ts`
composes), not just on-disk file presence. Testing `defaults/AGENTS.md` content would pass while the
resident contract is stale.
---
## R2 — BLOCKER: the migration "snapshot/restore" is described but the restore path is a data-loss hazard
§5.4 says migration snapshots `~/.config/mosaic/``.backup-v2/` "before touching disk," and §5.5
gates the alpha on three fixtures passing "with no interactive prompt, no hang." But the real installer
(`install.sh:105-154`, `sync_framework`) does `rsync -a --delete` (or the `cp` fallback that
`find ... -exec rm -rf {} +` wipes the target first). There is **no snapshot step in the code today**,
and the synthesis describes it as if it exists. Worse:
- On the **cp fallback path** (no rsync), preservation is done by copying PRESERVE_PATHS to a tempdir,
wiping the *entire* target, then copying source + restoring preserved paths (`install.sh:128-153`).
If the process dies between the `rm -rf` (line 140) and the restore loop (line 144-151), the user's
`SOUL.md`/`USER.md`/`credentials` are **gone** — no snapshot, no transaction. The synthesis's
"snapshot to `.backup-v2/`" would fix this, but it is not written, not tested, and the DoD treats it
as already-decided rather than to-be-built.
- `--delete` + removing `AGENTS.md` from preserve means on the *first* v2→v3 upgrade, a user who edited
their root `AGENTS.md` (the install flow at `install.sh:235` explicitly invites this: "must never be
overwritten once the user has customized them") loses those edits with **no migration of intent**.
The synthesis hand-waves this with "we do not try to diff/split a user-edited flat AGENTS.md"
(§5.4) — but that *is* the population most likely to exist, since the current model encourages
editing root `AGENTS.md`. Silent loss of a customized resident contract on the very first Constitution
upgrade is the worst possible first impression for the alpha.
**Mitigation:**
- Implement the snapshot as an actual atomic step (snapshot → sync → on failure, restore) in BOTH
installers, and add a fixture that kills the process mid-sync and asserts no data loss.
- For the user-edited-root-`AGENTS.md` case: on v2→v3, if the root `AGENTS.md` differs from the shipped
v2 default, **save it to `AGENTS.md.pre-constitution.bak` and emit a doctor advisory** ("your old
AGENTS.md had local edits; the gate content now lives in CONSTITUTION.md; your edits are preserved
at <path> for review"). Don't silently delete; don't try to auto-merge.
---
## R3 — MAJOR: the cross-harness "CI smoke test asserts gates are resident" is the load-bearing control and it does not exist
D5 and §6 make the cross-harness claim *true* by leaning entirely on "a CI smoke test launches each
harness path and asserts the irreducible gates are present in the effective context." This single
sentence is doing all the work that makes "enforced consistently across Claude/Codex/Pi/OpenCode"
more than aspiration. But:
- Two of the four harnesses (Codex, OpenCode) have **no hook parity** — the synthesis itself concedes
this is "a tracked gap... not a silent inconsistency" (§6). So for those harnesses the *only*
enforcement is resident-by-value text, and the smoke test is the only thing verifying it landed.
- Launching four real agent runtimes headlessly in CI, getting their *effective context*, and asserting
text presence is a non-trivial harness — it needs each CLI installed, authed, and a way to dump the
composed system prompt. `launch.ts:518/551` build `--append-system-prompt` for Claude/Pi; there is no
evidence Codex/OpenCode expose the composed prompt for assertion. The bare-`claude` (Tier-3 pointer)
path can't be asserted at all without actually reading the model's behavior.
- The honest version is: assert what `compose-contract`/`buildPrompt` (`launch.ts:300-339`) *emits*,
per harness — a unit test on the composer, not a live-launch smoke test. That is achievable and worth
doing. The "live launch each harness" framing oversells it and will either be quietly downgraded or
block the alpha indefinitely.
**Mitigation:** Re-scope the control to a **composer unit test** (assert `buildPrompt(harness)` output
contains the irreducible-gate anchor for each tier), which is real and cheap, and demote the
"live-launch smoke test" to a post-alpha aspiration. Track Codex/OpenCode hook-parity as an explicit
known-limitation in `COMPLIANCE.md`, not as something the alpha closes.
---
## R4 — MAJOR: deleting `defaults/SOUL.md` removes the only persona an injection-failure fallback can show
The synthesis deletes `defaults/SOUL.md` (settled #3, D6, §2c) so persona ships only as a template
generated at `mosaic init`. Correct for sanitization. But consider the failure mode the synthesis
itself worries about elsewhere — **injection silently failed / bare launch / init never run**:
- `launch.ts:329` reads `SOUL.md` as **optional** (`readOptional`). If `mosaic init` was never run (or
the user `git clone`d the framework and launched a bare `claude`), there is **no `SOUL.md` at all**,
and `AGENTS.md:14` instructs "Read `~/.config/mosaic/SOUL.md`" — a file that does not exist. Today the
shipped `defaults/SOUL.md` at least seeds *a* working persona. After deletion, the out-of-box,
pre-init experience is "identity file missing," which `AGENTS.md:144` (a hard gate!) says should make
the agent **stop and report**. So the sanitization change can convert a clean first-run into a
hard-stop, unless `mosaic init` is mandatory and enforced before any launch.
- The synthesis never states whether launch is *blocked* until init completes. If it isn't, deleting
the default persona degrades first-run from "works with a generic persona" to "halts on missing core
file." If it is, that's a new gate the migration must enforce and the DoD must list.
**Mitigation:** Either (a) make `mosaic init` a hard precondition of `mosaic <harness>` with a friendly
"run init first" message (not the gate-13 hard-stop), OR (b) keep a *generic, PII-free*
`SOUL.md.default` (literally the template with safe defaults already rendered) as the seed, and let init
overwrite it — note this is exactly the "generic-defaults recreates the Jarvis bug" objection D6
rejected, so (a) is cleaner. Pick one explicitly; the current plan leaves a hole.
---
## R5 — MAJOR: the resident line-count budget (D7) is unenforceable without a defined resident set, and the set is harness-variable
D7 enforces "a resident line-count ceiling in CI" over "the always-resident set (`CONSTITUTION.md` +
`AGENTS.md` index + `SOUL.md` + `USER.md` + the resident RUNTIME slice)." Two problems:
1. **`SOUL.md` and `USER.md` are user-generated and not in the repo** (that's the whole point of D6).
CI cannot count lines of files that don't exist in the package. So the CI budget can only cover the
framework-owned files (`CONSTITUTION.md`, `AGENTS.md`, `RUNTIME.md`) — the operator can still blow
the *actual* resident budget with a 600-line `USER.md`, and CI never sees it. The budget that
matters (total tokens hitting the model) is exactly the one CI can't measure. This is "budget the
container" measuring the wrong container.
2. **The resident set differs per harness** (§6 table: Tier-1 injects L0 by value, Tier-3 injects only
a ≤5-bullet summary). So "the resident set" is not one number. A single CI ceiling either over-counts
for Tier-3 or under-counts for Tier-1.
**Mitigation:** Split the control: (a) a CI **package-side** ceiling on framework-owned resident files
(`CONSTITUTION.md` + dispatcher `AGENTS.md` + `RUNTIME.md` resident slice) — real and worth it; (b) a
**`mosaic doctor` runtime advisory** that sums the *actual* composed prompt size including `SOUL.md`/
`USER.md` and warns the operator. Don't claim CI enforces a budget it structurally cannot see.
---
## R6 — MAJOR: gate #13 (merge-authority) is being *extracted to an example*, which silently weakens a hard gate for the maintainer's own deployment
The synthesis moves the merge-authority clause (`defaults/AGENTS.md:37`, "Policy: Jason, 2026-06-11")
out of L0 into `examples/policy/merge-authority.example.md`, adopted per-deployment (D1, §2a). Sound for
sanitization. But note the BRIEF's non-negotiable: *keep the existing hard gates intact
(PR-review-before-merge, ... no forced merges)*. Gate #13 today **interacts with** the no-self-merge
rule: it says "a 'No self-merge' note means no UNREVIEWED self-merge — it does not suspend
coordinator-authorized merges." That is a *load-bearing disambiguation of an existing hard gate.* If it
becomes an opt-in example file that a deployment may or may not adopt:
- A deployment that *doesn't* adopt the policy file has **no rule** disambiguating "No self-merge" vs
coordinator-authorized merge → an orchestrator either over-blocks (waits on human, violating the
steered-autonomy gates) or, worse, an agent reads "No self-merge" literally and the coordinator flow
deadlocks. The synthesis's own "lower layers may only make stricter, never more permissive" precedence
rule (§1) means an *absent* policy file defaults to the **strictest** reading — which is "never merge
without the human," directly contradicting gates #2/#9 that the BRIEF says to preserve.
- So extraction doesn't just relocate operator data; it removes a **conflict-resolution clause between
two hard gates** from the universal law. That's a behavioral regression dressed as sanitization.
**Mitigation:** Split clause #13. The *operator-specific delegation* ("don't wait on Jason personally")
is operator policy → `examples/policy/`. The *gate-interaction rule* ("'No self-merge' = no UNREVIEWED
self-merge; coordinator-authorized merges are not self-merges") is **universal law** and must stay in
L0 `CONSTITUTION.md`, operator-agnostic. Don't ship an alpha where not-adopting an example file changes
hard-gate semantics.
---
## R7 — MINOR/MAJOR: `verify-sanitized.sh` denylist will false-positive and get disabled, OR miss the real class
D6's blocking grep matches `jarvis|jason|woltje|\bPDA\b` plus `~/src/<word>` / `/home/<word>/`. Two
predictable failures:
- **False positives that train people to bypass:** "jason" matches `jasonwebtoken`/`jsonwebtoken`
typos, `comparison`, `parse`-adjacent strings? (`\bPDA\b` is fine; bare `jason` is not anchored in the
spec). `guides/` legitimately discusses JWT, JSON, etc. A blocking CI check that fires on legitimate
content gets `# noqa`'d or the pattern narrowed until it's toothless. The synthesis says "close the
*class*, not the tokens" but then specifies **tokens** (`jarvis|jason|woltje`). The class is "this
operator's PII," which a denylist of three names cannot generalize — the next operator is named
something else, and the *agent writing future framework PRs runs with that operator's SOUL/USER in
context* (the synthesis's own §4 worry).
- The `/home/<word>/` and `~/src/<word>` patterns will hit **legitimate documentation examples** in
guides (paths are how you explain tooling). Excluding `examples/` (§4) isn't enough; guides are full
of real paths.
**Mitigation:** Keep the grep but scope it honestly: (a) **structural** rules that don't depend on
knowing the operator — unrendered `{{...}}`/`${...}` in resident files, dead `/rails/` tokens, absolute
`/home/<specific-user>/` only (not generic `/home/<word>/`); (b) a **separate allowlist-based** check
for the *known* current contaminants (`jarvis|jason|woltje|PDA`) as a one-time regression guard, clearly
labeled "current-contaminant denylist, not a general PII detector." Don't oversell a 4-name grep as
closing the PII *class*; the real class-closer is the L0 prose rule (§4) + human review, and that should
be stated as the primary control with the grep as backup, not vice-versa.
---
## R8 — MINOR: the `.local.md` overlay + `compose-contract` step is a new subsystem the DoD calls "zero new subsystems"
§5.5 claims the winning design adds "zero new subsystems (`rsync` + linear migration + overlays + a
15-line grep)." But D4/§5.2 introduce `mosaic compose-contract <harness>` that "concatenates, in
precedence order, base + `.local` deltas *before* injection." Today `launch.ts:300-339` `buildPrompt`
does a fixed concatenation with **no `.local` awareness and no precedence resolution.** Adding
per-layer overlay composition *is* a new subsystem: it needs discovery of `SOUL.local.md`/
`USER.local.md`/`STANDARDS.local.md`/`policy/*.md`, a defined precedence merge, and wiring into every
harness launch path. Calling it "zero new subsystems" understates the alpha's actual build surface and
risks it being descoped late, leaving the customization-safety promise (§5's "single sentence a user
can rely on") unimplemented while the docs claim it works.
**Mitigation:** List `compose-contract` overlay composition as an explicit DoD work item with its own
test (assert `SOUL.local.md` appends after `SOUL.md`, `policy/*.md` is tighten-only). For the alpha, if
build budget is tight, **ship only `SOUL.local.md`/`USER.local.md`** (the two files users actually
customize) and defer `STANDARDS.local.md`/`policy/` to v2 — but say so, don't imply full overlay support.
---
## R9 — MINOR: "self-load fallback" (`READ CONSTITUTION.md NOW`) reintroduces the exact false-confidence the synthesis flags in #9
Settled #9 correctly kills `defaults/AGENTS.md:11`'s false "already in your context… do not re-read."
The replacement (§1 tier-3, §6 table) is: dispatcher says *"If CONSTITUTION.md is not already in your
context, READ IT NOW."* This is better, but the conditional *"if not already in your context"* asks the
model to **introspect on its own context window** — something models are unreliable at. A model that has
a *stale* or *partial* L0 resident may conclude "it's already here" and skip the read, getting the old
gates. The honest tier-3 instruction is unconditional: *"READ `~/.config/mosaic/CONSTITUTION.md` now
before your first action"* — cheap, idempotent, no introspection. The conditional version optimizes away
a one-file read at the cost of correctness on exactly the drift-prone path it's meant to protect.
**Mitigation:** On Tier-3 (pointer) launches, make the read **unconditional**. Reserve the conditional
phrasing for Tier-1 (where injection-by-value genuinely already placed it and a re-read is wasteful).
The tier table already distinguishes these — let the *read instruction* differ by tier too.
---
## R10 — MINOR: dual-installer drift is itself an unmitigated systemic risk
`install.sh` (bash) and `file-adapter.ts` (TS) are two independent implementations of the same
upgrade/preserve/seed logic, kept in sync only by a code comment (`file-adapter.ts:148`,
`install.sh:230`). The synthesis's entire migration plan is written against `install.sh` and **never
acknowledges the TS path exists.** Every fix in §2/§5 (remove from PRESERVE_PATHS, overwrite L0,
snapshot, migration v2→v3) must be applied twice and verified equivalent, or the npm-installed users and
the curl-`install.sh` users get different upgrade behavior — a cross-harness-style inconsistency one
layer down, at install time.
**Mitigation:** Add a DoD item: a single shared test suite that runs the *same* upgrade fixtures against
both `install.sh` and `FileConfigAdapter.syncFramework`, asserting identical resulting trees. Or, better,
collapse to one implementation (have the bash installer shell out to the node CLI, or vice versa) before
piling Constitution semantics onto both.
---
## Ranked summary
| # | Risk | Severity | One-line mitigation |
|---|------|----------|---------------------|
| R1 | "Remove from PRESERVE_PATHS" does NOT update the resident root `AGENTS.md` (seed-if-absent; `launch.ts:326` reads root, not `defaults/`) — the headline drift fix is mechanically false | **BLOCKER** | Replace seed-if-absent with unconditional overwrite for L0/dispatcher in BOTH `install.sh` and `file-adapter.ts`; test injected bytes, not file presence |
| R2 | Migration snapshot/restore is described but not implemented; `cp`-fallback + `--delete` can lose `SOUL.md`/`credentials` on interrupt; user-edited root `AGENTS.md` silently lost on first upgrade | **BLOCKER** | Implement atomic snapshot→sync→restore in both installers; back up user-edited `AGENTS.md` to `.pre-constitution.bak` with a doctor advisory |
| R3 | "Live-launch CI smoke test asserts gates resident on every harness" is the load-bearing cross-harness control and is impractical (no Codex/OpenCode prompt dump, Tier-3 unassertable) | MAJOR | Re-scope to a composer unit test on `buildPrompt(harness)`; demote live-launch to v2; track hook-parity gaps in COMPLIANCE.md |
| R4 | Deleting `defaults/SOUL.md` turns a clean first-run / bare-launch into a missing-core-file hard-stop (gate #13/§144) when init wasn't run | MAJOR | Make `mosaic init` a hard precondition with a friendly message, OR seed a generic rendered `SOUL.md`; decide explicitly |
| R5 | Resident line-count budget can't see user-generated `SOUL.md`/`USER.md` and varies per harness tier — it measures the wrong container | MAJOR | CI ceiling on framework-owned resident files only; `mosaic doctor` runtime advisory for the real composed size |
| R6 | Extracting merge-authority gate #13 to an opt-in example removes a hard-gate *conflict-resolution clause*; non-adopters default (per "stricter-only" rule) to never-merge, contradicting gates #2/#9 the BRIEF preserves | MAJOR | Split #13: operator delegation → `policy/` example; the "No self-merge = no UNREVIEWED self-merge" gate-interaction rule stays universal in L0 |
| R7 | `verify-sanitized.sh` 4-name denylist false-positives (gets disabled) and can't generalize the PII *class* it claims to close | MAJOR/MINOR | Separate structural checks (always valid) from a labeled current-contaminant denylist; name human review + L0 prose rule as the primary class-closer |
| R8 | `mosaic compose-contract` overlay composition is a real new subsystem the DoD calls "zero new subsystems" | MINOR | List it as an explicit DoD item with tests; for alpha ship only `SOUL.local.md`/`USER.local.md`, defer the rest and say so |
| R9 | Conditional "if not already in context, READ CONSTITUTION.md" asks the model to introspect its context — unreliable on the drift-prone path it protects | MINOR | Make the Tier-3 pointer read **unconditional**; keep conditional only for Tier-1 |
| R10 | Two independent installers (`install.sh` + `file-adapter.ts`) kept in sync by a comment; synthesis ignores the TS path entirely | MINOR | Shared upgrade-fixture suite run against both, or collapse to one implementation before adding Constitution semantics |
**Bottom line:** Adopt the layer model and sanitization as designed. **Do not tag the alpha** until R1
and R2 are fixed in *both* installer implementations and proven by a fixture matrix that asserts the
*injected resident bytes* (not on-disk presence) — because as written, the synthesis ships an alpha that
believes it fixed the drift bug while the resident contract stays stale.