Files
stack/docs/design/framework-constitution/debate/redteam-contrarian.md
Jason Woltje c70b217a5c
Some checks failed
ci/woodpecker/push/ci Pipeline failed
docs(design): mosaic framework constitution — expert conference output
Conference of 7 experts (architect/moonshot/contrarian/coder/aiml/devex/steward)
debated layering, sanitization, upgrade-safety, cross-harness robustness.
Artifacts: BRIEF, 7 positions, 7 rebuttals, synthesis-v1, 3 red-team passes,
canonical DESIGN.md, OPEN-QUESTIONS.md, MISSION.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 23:47:49 -05:00

23 KiB

Red-Team — Contrarian Skeptic vs. Synthesis v1

Lens: Contrarian Skeptic. Distrusts clever abstractions; hunts failure modes, over-engineering, and rules that read well but degrade real agent behavior. I tried to break the design in synthesis-v1.md, grounding every claim in the real tree. The synthesis already absorbed a lot of contrarian input, so I went after what survived or was newly introduced by the ruling itself.

Verdict: The layering and sanitization decisions are sound. But the synthesis's headline drift fix is mechanically wrong — it does not do what it claims, and the alpha would ship believing the drift bug is fixed when it is not. That is a blocker. Several other claims are aspirational controls presented as settled.


R1 — BLOCKER: "Remove from PRESERVE_PATHS" does NOT make gate updates reach existing installs

This is the synthesis's central, most-repeated claim — settled-item #7, D4, §5.1, and the alpha DoD all assert that removing AGENTS.md/STANDARDS.md from PRESERVE_PATHS is "the single change [that] makes gate updates reach every existing install (the literal drift bug)." It does not. I traced the actual install/launch code:

  1. The resident, injected contract is the root file ~/.config/mosaic/AGENTS.md. Proof: packages/mosaic/src/commands/launch.ts:326parts.push(readFileSync(join(MOSAIC_HOME, 'AGENTS.md'))). It never reads defaults/AGENTS.md.
  2. That root file is seeded once and never re-seeded. Proof, both install paths:
    • install.sh:235-240: for default_file in AGENTS.md STANDARDS.md TOOLS.md; do if [[ -f "$DEFAULTS_DIR/$default_file" ]] && [[ ! -f "$TARGET_DIR/$default_file" ]]; then cp ... — the ! -f guard means an existing root file is skipped.
    • file-adapter.ts:184-190: for (const entry of DEFAULT_SEED_FILES) { ... if (existsSync(dest)) continue; ... copyFileSync(...) } — same seed-once semantics.
  3. defaults/ itself is rsynced into ~/.config/mosaic/defaults/ as a subdirectory, so removing the root file from PRESERVE_PATHS only refreshes the non-resident defaults/AGENTS.md copy that nothing injects.

Net effect of the synthesis's fix as written: rsync --delete now also deletes the user's customized root AGENTS.md on every keep upgrade (because it's no longer preserved) — but the seed loop will not put the new one back, because… actually it will, since the file is now absent — but only by accident, and only on the bash path. The two sync implementations (install.sh and file-adapter.ts) must stay byte-identical (file-adapter.ts:148 says so explicitly) and the synthesis never mentions file-adapter.ts exists. Any fix applied to one and not the other silently diverges the bash-install and npm-install upgrade behavior — exactly the cross-path drift the project already warns about in that comment.

The deeper trap: the seed mechanism is "copy if absent," which is structurally incompatible with "framework-owned, overwritten every upgrade." You cannot make a file both seeded-once-then-user-owned (today's model) and clobbered-every-upgrade (the Constitution model) by editing a path list. The synthesis's L0 doctrine requires the seed-if-absent logic for AGENTS.md/CONSTITUTION.md/STANDARDS.md to be replaced with unconditional overwrite, in both install.sh and file-adapter.ts, plus the DEFAULT_SEED_FILES list at file-adapter.ts:16 re-thought. None of that is in the plan.

Mitigation (required before alpha):

  • Constitution model: L0 CONSTITUTION.md and the dispatcher AGENTS.md must be unconditionally copied/overwritten at the root on every upgrade (not seed-if-absent), in install.sh AND file-adapter.ts. Add a test fixture asserting that an upgrade over a modified root AGENTS.md replaces it.
  • Add file-adapter.ts (and DEFAULT_SEED_FILES) to the file-by-file plan in §2b. The synthesis is incomplete: it plans the bash installer and the markdown, not the TS installer that ships in the npm package.
  • The migration fixture matrix in §5.5 must assert the injected resident bytes (what launch.ts composes), not just on-disk file presence. Testing defaults/AGENTS.md content would pass while the resident contract is stale.

R2 — BLOCKER: the migration "snapshot/restore" is described but the restore path is a data-loss hazard

§5.4 says migration snapshots ~/.config/mosaic/.backup-v2/ "before touching disk," and §5.5 gates the alpha on three fixtures passing "with no interactive prompt, no hang." But the real installer (install.sh:105-154, sync_framework) does rsync -a --delete (or the cp fallback that find ... -exec rm -rf {} + wipes the target first). There is no snapshot step in the code today, and the synthesis describes it as if it exists. Worse:

  • On the cp fallback path (no rsync), preservation is done by copying PRESERVE_PATHS to a tempdir, wiping the entire target, then copying source + restoring preserved paths (install.sh:128-153). If the process dies between the rm -rf (line 140) and the restore loop (line 144-151), the user's SOUL.md/USER.md/credentials are gone — no snapshot, no transaction. The synthesis's "snapshot to .backup-v2/" would fix this, but it is not written, not tested, and the DoD treats it as already-decided rather than to-be-built.
  • --delete + removing AGENTS.md from preserve means on the first v2→v3 upgrade, a user who edited their root AGENTS.md (the install flow at install.sh:235 explicitly invites this: "must never be overwritten once the user has customized them") loses those edits with no migration of intent. The synthesis hand-waves this with "we do not try to diff/split a user-edited flat AGENTS.md" (§5.4) — but that is the population most likely to exist, since the current model encourages editing root AGENTS.md. Silent loss of a customized resident contract on the very first Constitution upgrade is the worst possible first impression for the alpha.

Mitigation:

  • Implement the snapshot as an actual atomic step (snapshot → sync → on failure, restore) in BOTH installers, and add a fixture that kills the process mid-sync and asserts no data loss.
  • For the user-edited-root-AGENTS.md case: on v2→v3, if the root AGENTS.md differs from the shipped v2 default, save it to AGENTS.md.pre-constitution.bak and emit a doctor advisory ("your old AGENTS.md had local edits; the gate content now lives in CONSTITUTION.md; your edits are preserved at for review"). Don't silently delete; don't try to auto-merge.

R3 — MAJOR: the cross-harness "CI smoke test asserts gates are resident" is the load-bearing control and it does not exist

D5 and §6 make the cross-harness claim true by leaning entirely on "a CI smoke test launches each harness path and asserts the irreducible gates are present in the effective context." This single sentence is doing all the work that makes "enforced consistently across Claude/Codex/Pi/OpenCode" more than aspiration. But:

  • Two of the four harnesses (Codex, OpenCode) have no hook parity — the synthesis itself concedes this is "a tracked gap... not a silent inconsistency" (§6). So for those harnesses the only enforcement is resident-by-value text, and the smoke test is the only thing verifying it landed.
  • Launching four real agent runtimes headlessly in CI, getting their effective context, and asserting text presence is a non-trivial harness — it needs each CLI installed, authed, and a way to dump the composed system prompt. launch.ts:518/551 build --append-system-prompt for Claude/Pi; there is no evidence Codex/OpenCode expose the composed prompt for assertion. The bare-claude (Tier-3 pointer) path can't be asserted at all without actually reading the model's behavior.
  • The honest version is: assert what compose-contract/buildPrompt (launch.ts:300-339) emits, per harness — a unit test on the composer, not a live-launch smoke test. That is achievable and worth doing. The "live launch each harness" framing oversells it and will either be quietly downgraded or block the alpha indefinitely.

Mitigation: Re-scope the control to a composer unit test (assert buildPrompt(harness) output contains the irreducible-gate anchor for each tier), which is real and cheap, and demote the "live-launch smoke test" to a post-alpha aspiration. Track Codex/OpenCode hook-parity as an explicit known-limitation in COMPLIANCE.md, not as something the alpha closes.


R4 — MAJOR: deleting defaults/SOUL.md removes the only persona an injection-failure fallback can show

The synthesis deletes defaults/SOUL.md (settled #3, D6, §2c) so persona ships only as a template generated at mosaic init. Correct for sanitization. But consider the failure mode the synthesis itself worries about elsewhere — injection silently failed / bare launch / init never run:

  • launch.ts:329 reads SOUL.md as optional (readOptional). If mosaic init was never run (or the user git cloned the framework and launched a bare claude), there is no SOUL.md at all, and AGENTS.md:14 instructs "Read ~/.config/mosaic/SOUL.md" — a file that does not exist. Today the shipped defaults/SOUL.md at least seeds a working persona. After deletion, the out-of-box, pre-init experience is "identity file missing," which AGENTS.md:144 (a hard gate!) says should make the agent stop and report. So the sanitization change can convert a clean first-run into a hard-stop, unless mosaic init is mandatory and enforced before any launch.
  • The synthesis never states whether launch is blocked until init completes. If it isn't, deleting the default persona degrades first-run from "works with a generic persona" to "halts on missing core file." If it is, that's a new gate the migration must enforce and the DoD must list.

Mitigation: Either (a) make mosaic init a hard precondition of mosaic <harness> with a friendly "run init first" message (not the gate-13 hard-stop), OR (b) keep a generic, PII-free SOUL.md.default (literally the template with safe defaults already rendered) as the seed, and let init overwrite it — note this is exactly the "generic-defaults recreates the Jarvis bug" objection D6 rejected, so (a) is cleaner. Pick one explicitly; the current plan leaves a hole.


R5 — MAJOR: the resident line-count budget (D7) is unenforceable without a defined resident set, and the set is harness-variable

D7 enforces "a resident line-count ceiling in CI" over "the always-resident set (CONSTITUTION.md + AGENTS.md index + SOUL.md + USER.md + the resident RUNTIME slice)." Two problems:

  1. SOUL.md and USER.md are user-generated and not in the repo (that's the whole point of D6). CI cannot count lines of files that don't exist in the package. So the CI budget can only cover the framework-owned files (CONSTITUTION.md, AGENTS.md, RUNTIME.md) — the operator can still blow the actual resident budget with a 600-line USER.md, and CI never sees it. The budget that matters (total tokens hitting the model) is exactly the one CI can't measure. This is "budget the container" measuring the wrong container.
  2. The resident set differs per harness (§6 table: Tier-1 injects L0 by value, Tier-3 injects only a ≤5-bullet summary). So "the resident set" is not one number. A single CI ceiling either over-counts for Tier-3 or under-counts for Tier-1.

Mitigation: Split the control: (a) a CI package-side ceiling on framework-owned resident files (CONSTITUTION.md + dispatcher AGENTS.md + RUNTIME.md resident slice) — real and worth it; (b) a mosaic doctor runtime advisory that sums the actual composed prompt size including SOUL.md/ USER.md and warns the operator. Don't claim CI enforces a budget it structurally cannot see.


R6 — MAJOR: gate #13 (merge-authority) is being extracted to an example, which silently weakens a hard gate for the maintainer's own deployment

The synthesis moves the merge-authority clause (defaults/AGENTS.md:37, "Policy: Jason, 2026-06-11") out of L0 into examples/policy/merge-authority.example.md, adopted per-deployment (D1, §2a). Sound for sanitization. But note the BRIEF's non-negotiable: keep the existing hard gates intact (PR-review-before-merge, ... no forced merges). Gate #13 today interacts with the no-self-merge rule: it says "a 'No self-merge' note means no UNREVIEWED self-merge — it does not suspend coordinator-authorized merges." That is a load-bearing disambiguation of an existing hard gate. If it becomes an opt-in example file that a deployment may or may not adopt:

  • A deployment that doesn't adopt the policy file has no rule disambiguating "No self-merge" vs coordinator-authorized merge → an orchestrator either over-blocks (waits on human, violating the steered-autonomy gates) or, worse, an agent reads "No self-merge" literally and the coordinator flow deadlocks. The synthesis's own "lower layers may only make stricter, never more permissive" precedence rule (§1) means an absent policy file defaults to the strictest reading — which is "never merge without the human," directly contradicting gates #2/#9 that the BRIEF says to preserve.
  • So extraction doesn't just relocate operator data; it removes a conflict-resolution clause between two hard gates from the universal law. That's a behavioral regression dressed as sanitization.

Mitigation: Split clause #13. The operator-specific delegation ("don't wait on Jason personally") is operator policy → examples/policy/. The gate-interaction rule ("'No self-merge' = no UNREVIEWED self-merge; coordinator-authorized merges are not self-merges") is universal law and must stay in L0 CONSTITUTION.md, operator-agnostic. Don't ship an alpha where not-adopting an example file changes hard-gate semantics.


R7 — MINOR/MAJOR: verify-sanitized.sh denylist will false-positive and get disabled, OR miss the real class

D6's blocking grep matches jarvis|jason|woltje|\bPDA\b plus ~/src/<word> / /home/<word>/. Two predictable failures:

  • False positives that train people to bypass: "jason" matches jasonwebtoken/jsonwebtoken typos, comparison, parse-adjacent strings? (\bPDA\b is fine; bare jason is not anchored in the spec). guides/ legitimately discusses JWT, JSON, etc. A blocking CI check that fires on legitimate content gets # noqa'd or the pattern narrowed until it's toothless. The synthesis says "close the class, not the tokens" but then specifies tokens (jarvis|jason|woltje). The class is "this operator's PII," which a denylist of three names cannot generalize — the next operator is named something else, and the agent writing future framework PRs runs with that operator's SOUL/USER in context (the synthesis's own §4 worry).
  • The /home/<word>/ and ~/src/<word> patterns will hit legitimate documentation examples in guides (paths are how you explain tooling). Excluding examples/ (§4) isn't enough; guides are full of real paths.

Mitigation: Keep the grep but scope it honestly: (a) structural rules that don't depend on knowing the operator — unrendered {{...}}/${...} in resident files, dead /rails/ tokens, absolute /home/<specific-user>/ only (not generic /home/<word>/); (b) a separate allowlist-based check for the known current contaminants (jarvis|jason|woltje|PDA) as a one-time regression guard, clearly labeled "current-contaminant denylist, not a general PII detector." Don't oversell a 4-name grep as closing the PII class; the real class-closer is the L0 prose rule (§4) + human review, and that should be stated as the primary control with the grep as backup, not vice-versa.


R8 — MINOR: the .local.md overlay + compose-contract step is a new subsystem the DoD calls "zero new subsystems"

§5.5 claims the winning design adds "zero new subsystems (rsync + linear migration + overlays + a 15-line grep)." But D4/§5.2 introduce mosaic compose-contract <harness> that "concatenates, in precedence order, base + .local deltas before injection." Today launch.ts:300-339 buildPrompt does a fixed concatenation with no .local awareness and no precedence resolution. Adding per-layer overlay composition is a new subsystem: it needs discovery of SOUL.local.md/ USER.local.md/STANDARDS.local.md/policy/*.md, a defined precedence merge, and wiring into every harness launch path. Calling it "zero new subsystems" understates the alpha's actual build surface and risks it being descoped late, leaving the customization-safety promise (§5's "single sentence a user can rely on") unimplemented while the docs claim it works.

Mitigation: List compose-contract overlay composition as an explicit DoD work item with its own test (assert SOUL.local.md appends after SOUL.md, policy/*.md is tighten-only). For the alpha, if build budget is tight, ship only SOUL.local.md/USER.local.md (the two files users actually customize) and defer STANDARDS.local.md/policy/ to v2 — but say so, don't imply full overlay support.


R9 — MINOR: "self-load fallback" (READ CONSTITUTION.md NOW) reintroduces the exact false-confidence the synthesis flags in #9

Settled #9 correctly kills defaults/AGENTS.md:11's false "already in your context… do not re-read." The replacement (§1 tier-3, §6 table) is: dispatcher says "If CONSTITUTION.md is not already in your context, READ IT NOW." This is better, but the conditional "if not already in your context" asks the model to introspect on its own context window — something models are unreliable at. A model that has a stale or partial L0 resident may conclude "it's already here" and skip the read, getting the old gates. The honest tier-3 instruction is unconditional: "READ ~/.config/mosaic/CONSTITUTION.md now before your first action" — cheap, idempotent, no introspection. The conditional version optimizes away a one-file read at the cost of correctness on exactly the drift-prone path it's meant to protect.

Mitigation: On Tier-3 (pointer) launches, make the read unconditional. Reserve the conditional phrasing for Tier-1 (where injection-by-value genuinely already placed it and a re-read is wasteful). The tier table already distinguishes these — let the read instruction differ by tier too.


R10 — MINOR: dual-installer drift is itself an unmitigated systemic risk

install.sh (bash) and file-adapter.ts (TS) are two independent implementations of the same upgrade/preserve/seed logic, kept in sync only by a code comment (file-adapter.ts:148, install.sh:230). The synthesis's entire migration plan is written against install.sh and never acknowledges the TS path exists. Every fix in §2/§5 (remove from PRESERVE_PATHS, overwrite L0, snapshot, migration v2→v3) must be applied twice and verified equivalent, or the npm-installed users and the curl-install.sh users get different upgrade behavior — a cross-harness-style inconsistency one layer down, at install time.

Mitigation: Add a DoD item: a single shared test suite that runs the same upgrade fixtures against both install.sh and FileConfigAdapter.syncFramework, asserting identical resulting trees. Or, better, collapse to one implementation (have the bash installer shell out to the node CLI, or vice versa) before piling Constitution semantics onto both.


Ranked summary

# Risk Severity One-line mitigation
R1 "Remove from PRESERVE_PATHS" does NOT update the resident root AGENTS.md (seed-if-absent; launch.ts:326 reads root, not defaults/) — the headline drift fix is mechanically false BLOCKER Replace seed-if-absent with unconditional overwrite for L0/dispatcher in BOTH install.sh and file-adapter.ts; test injected bytes, not file presence
R2 Migration snapshot/restore is described but not implemented; cp-fallback + --delete can lose SOUL.md/credentials on interrupt; user-edited root AGENTS.md silently lost on first upgrade BLOCKER Implement atomic snapshot→sync→restore in both installers; back up user-edited AGENTS.md to .pre-constitution.bak with a doctor advisory
R3 "Live-launch CI smoke test asserts gates resident on every harness" is the load-bearing cross-harness control and is impractical (no Codex/OpenCode prompt dump, Tier-3 unassertable) MAJOR Re-scope to a composer unit test on buildPrompt(harness); demote live-launch to v2; track hook-parity gaps in COMPLIANCE.md
R4 Deleting defaults/SOUL.md turns a clean first-run / bare-launch into a missing-core-file hard-stop (gate #13/§144) when init wasn't run MAJOR Make mosaic init a hard precondition with a friendly message, OR seed a generic rendered SOUL.md; decide explicitly
R5 Resident line-count budget can't see user-generated SOUL.md/USER.md and varies per harness tier — it measures the wrong container MAJOR CI ceiling on framework-owned resident files only; mosaic doctor runtime advisory for the real composed size
R6 Extracting merge-authority gate #13 to an opt-in example removes a hard-gate conflict-resolution clause; non-adopters default (per "stricter-only" rule) to never-merge, contradicting gates #2/#9 the BRIEF preserves MAJOR Split #13: operator delegation → policy/ example; the "No self-merge = no UNREVIEWED self-merge" gate-interaction rule stays universal in L0
R7 verify-sanitized.sh 4-name denylist false-positives (gets disabled) and can't generalize the PII class it claims to close MAJOR/MINOR Separate structural checks (always valid) from a labeled current-contaminant denylist; name human review + L0 prose rule as the primary class-closer
R8 mosaic compose-contract overlay composition is a real new subsystem the DoD calls "zero new subsystems" MINOR List it as an explicit DoD item with tests; for alpha ship only SOUL.local.md/USER.local.md, defer the rest and say so
R9 Conditional "if not already in context, READ CONSTITUTION.md" asks the model to introspect its context — unreliable on the drift-prone path it protects MINOR Make the Tier-3 pointer read unconditional; keep conditional only for Tier-1
R10 Two independent installers (install.sh + file-adapter.ts) kept in sync by a comment; synthesis ignores the TS path entirely MINOR Shared upgrade-fixture suite run against both, or collapse to one implementation before adding Constitution semantics

Bottom line: Adopt the layer model and sanitization as designed. Do not tag the alpha until R1 and R2 are fixed in both installer implementations and proven by a fixture matrix that asserts the injected resident bytes (not on-disk presence) — because as written, the synthesis ships an alpha that believes it fixed the drift bug while the resident contract stays stale.