Files
stack/docs/reports/greenfield/fomo-lin.md
T

17 KiB
Raw Blame History

Greenfield install log — fomo-lin

Running log of a from-scratch Mosaic Stack install on Jason's test laptop fomo-lin (Debian 13, x86_64). Operator: scooby (agent). Started 2026-08-08. Channel per fred: findings → comms as they land; this file is the durable record. Branch: greenfield/fomo-lin.

Machine starting state (2026-08-08)

  • Debian 13 (kernel 6.12.101+deb13), no Node/npm, no global git config, no ~/.ssh, no ~/.config/mosaic, no ~/.mosaic, sudo requires password (agent cannot escalate).
  • Repos pre-cloned by Jason: jarvis-brain, mosaic-brain, stack, uconnect, uscllc-website (all https remotes to git.mosaicstack.dev, no credentials stored — private-repo fetch/push dead until a token was provisioned from credentials.json, usc_mos).
  • tmux session scooby running Claude Code (bare harness — not mosaic claude).

Pre-install setup that had NO framework mechanism (manual work)

  • Agent identity: MOSAIC_AGENT_NAME=scooby hand-added to ~/.bashrc + tmux env.
  • Git identity + credential store: hand-configured.
  • Comms receive path: hand-ported scooby-comms-watcher.sh from fred's watcher + hand-written systemd --user unit + loginctl enable-linger. Works (both peers verified round-trip within ~90s), but every step was artisanal — relevant input for harness-homes (W-F).

Install run (2026-08-08 ~19:09Z)

curl -fsSL https://mosaicstack.dev/install.sh | bash -s -- --yes --no-auto-launch → exit 0, framework v3 → ~/.config/mosaic/, CLI @mosaicstack/mosaic 0.0.49~/.npm-global/. Prereq path: sudo apt install nodejs npm (Debian 13's node 20.19.2 meets the ≥20 floor). Public read on the stack repo means the installer itself needs no credentials — good.

Findings (outside fred's known-gaps list of 2026-08-08)

F1 — PATH advice is print-only

Installer warns ~/.npm-global/bin is not on your PATH and suggests the rc line, but shellProfileEdits: [] in the manifest — nothing is persisted. Every fresh machine ends with mosaic not resolvable in new shells until the user hand-edits rc. Either edit the rc (with consent/flag) or make the closing summary a copy-paste block.

F2 — Installer overwrites live ~/.claude/settings.json + ~/.claude/CLAUDE.md with backup: null

.install-manifest.json runtimeAssetCopies shows dest ~/.claude/settings.json, ~/.claude/CLAUDE.md, hooks-config.json, context7-integration.md, all backup: null, written while a Claude session was LIVE on this machine. On this box the pre-existing files were near-defaults so nothing of value was lost; on any configured machine this silently destroys user settings/memory. Wants: backup-before-overwrite (populate the manifest backup field it clearly already models) + merge-not-replace for settings.json.

F3 — Fresh install fails its own doctor: 10 warnings out of the box

Immediately after a clean, successful install, mosaic doctor reports: missing USER.md; AGENTS.md missing CRITICAL HARD GATES override block; runtime file drift on ~/.claude/settings.json; 7 missing mosaic-* skills. A green install that self-reports 10 warnings erodes trust in doctor as a signal. Whatever subset is "expected until mosaic init/wizard" should be suppressed or labeled as such.

F4 — Drift check points users at the gated template (wedge hazard)

The settings the installer writes to ~/.claude/settings.json are UNGATED (no mutator-gate, no receipt-observer) — which on today's main is CORRECT, it avoids the Stop-hook wedge. But ~/.config/mosaic/runtime/claude/settings.json (the file doctor diffs against) IS the gated template. So doctor's "runtime file drift" warning invites the obvious remediation — copy the template over — which would seed the receipt-observer wedge into a live seat. The drift baseline and the seeded file should be the same artifact, or doctor should know about the gated/ungated split.

F5 — Installed skill set is disjoint from repo skills/

Skill sync installed 101 skills (six mosaic-*: deploy, gitea, orchestrator, portainer, tools, woodpecker) but NONE of the eight in stack skills/ on main (board, forge, jarvis, macp, prd, prdy, setup-cicd, standards). Doctor then flags 7 of those 8 as missing (mosaic-jarvis escapes the check). Two sources of truth for "the Mosaic skills" — the installer's bundle and the repo dir — have diverged.

Environment answers / status

  • fomo-lin → sb-it-1-dt: comms-only today. Hostname does not resolve from here and the laptop has no ssh keys. ssh reach would need Jason (key provisioning + route/VPN).
  • Gitea write to the stack repo: verified by the push of this very branch (token usc_mos).

Session 2 (2026-08-08 later) — mosaic init + first bare seat

mosaic init completed (SOUL.md / USER.md / TOOLS.md generated; TOOLS.md was backed up before overwrite — the contrast with F2 shows the codebase already knows how). Its runtime-adapter step correctly REFUSED to wire mutator-gate/receipt-observer hooks (activation half absent, #869) — loud, explained, fail-safe. Good.

First bare seat: launchedmosaic claude --model sonnet → Claude Code v2.1.226, runtime-contract injection verified from inside the seat. But it took findings F6F10 to get there; on an untouched fresh main install, install → init → launch is broken at FOUR consecutive links.

F6 — SECURITY: mosaic-init eval-injects free-text answers

tools/_scripts/mosaic-init line 142: eval "$var_name=\"$value\"". Any answer containing " crashes init mid-flow (reproduced: exit 127, USER.md never written); an answer containing $( ) would EXECUTE arbitrary commands. Fix: printf -v. Same bug in the NON_INTERACTIVE default branch. Related: init exits 1 even on success when enforcement wiring is (correctly) refused — poisons any scripted chaining.

F7 — init silently drops the installer's mcpServers block → launcher refuses to run

init's "Updating runtime adapters" rewrote ~/.claude/settings.json and removed the mcpServers.sequential-thinking block the installer had written 11 min earlier. mosaic claude hard-requires that MCP → launch refused. The prescribed fix command (mosaic-ensure-sequential-thinking --runtime claude) works. So the happy path is install → init → BROKEN → hand-run a repair script. Merge-not-replace (F2) fixes this too.

F8 — no fleet roster on a fresh install; launcher dies with a raw stack trace

mosaic claude throws an uncaught Error: Fleet communications contract unavailable: no fleet roster at ~/.config/mosaic/fleet/roster.{yaml,json} (full Node stack trace to the user). Nothing in install or init creates a roster (wizard untested here — --no-auto-launch; if the wizard seeds one, the bare-flow gap still stands). Unblocked by hand-authoring a minimal site roster from fleet/examples/minimal.yaml.

F9 — FLAGSHIP: activation-probe timeout loses to CLI cold-start on modest hardware

activation_version_gate.py gives the mosaic __lease-capability probe PROBE_TIMEOUT_SECONDS = 2.0. On fomo-lin the CLI answers CORRECTLY in ~2.552.61s every run (Node startup cost). Timeout → fail-closed → every bare mosaic claude launch aborts (exit 65) with an error blaming "mosaic not on PATH … framework/CLI version skew" — neither true. Invisible on fast dev boxes; fatal on laptops. Suggest: raise/make configurable the timeout, warm-probe cache, and split the three failure causes into distinct messages. Local workaround (documented, removable): MOSAIC_LEASE_VERSION_PROBE_COMMAND pointed at a script emitting the verified payload instantly (~/.local/bin/mosaic-lease-probe-fast).

F10 — shipped lease-broker unit is never installed → registration denied

With F9 bypassed, launch dies with "Mosaic lease broker registration failed; runtime launch denied": the broker daemon isn't running, and although the framework SHIPS systemd/user/mosaic-lease-broker.service, nothing installs/enables it. systemctl --user link + enable --now of the shipped unit → READY instantly, launch proceeds. Installer/init/wizard should own this step.

Observations (not filed as findings)

  • Launcher settings audit demands mutator-gate.py while init refuses to wire it — main's components disagree about the gated state (fold into #1113/F4).
  • Seat context: runtime contract injected ✓; SOUL.md NOT injected (seat confirmed) — matches AGENTS.md read-on-demand load order, but README says the launcher "checks for SOUL.md". Question for lead, not a finding.
  • --ref next install path verified available (flag exists, next archive HTTP 200) — not exercised; fomo-lin stays main-as-shipped per lead ruling.

Next

  • Milestone comms sent at: install complete ✓ / first seat launched ✓.
  • First gated-seat probe deliberately deferred until PR #1109 lands (known deny-only state).

Session 3 (2026-08-08 evening) — wizard + gateway; refocus to next

Directive from Jason mid-session: focus shifts to the next branch and the new structure (stock ~/.claude untouched; framework wholly under ~/.config/mosaic). Main's ~/.claude write behavior is a deprecated location — findings stand, but no further deep-testing of it.

Wizard run (main): "keep identity, update framework"; ~/.claude hooks install DECLINED per directive (wizard rewrote ~/.claude/settings.json anyway — benign, no gated hooks, MCP kept). Wizard never prompted about fleet roster or lease-broker unit → F8/F10 disambiguation partial: wizard does not visibly own those steps. Full degraded-state test dropped per refocus.

F11 — gateway "Local" tier hard-requires Redis on main (fixed on next)

Wizard gateway install, Local tier ("embedded database, no dependencies"), port 14242: daemon starts then crash-spams ioredis ECONNREFUSED; never healthy; killed manually. main..next already contains 56787fab fix(gateway): disable Redis consumers on local tier (#689). Main ships a gateway that cannot come up dependency-free; next has the cure.

F12 — wizard exits 0 on gateway failure

Terminal shows "▲ Fix the underlying error above, then re-run mosaic gateway install" and the wizard exits 0. Scripted/CI consumers read success.

Cosmetic

Skipping the optional ANTHROPIC_API_KEY prompt records the literal string "undefined".

next recon (read-only)

  • next install.sh: first-class --next prerelease lane (npm @next dist-tag CLI + framework from permanent next branch; guard against mixing @next with a different explicit --ref).
  • next does NOT carry the new structure: ~/.claude handling unchanged; no harness-homes design docs on next or main. New structure = Jason directive + fred W-F design phase.

State: bare seat launch works; gateway stopped. Holding for fred's ruling on a next-lane reinstall (proposed) and W-F design review.

Session 4 (2026-08-08 night) — --next lane reinstall (pivot confirmed by Jason)

Main uninstalled (note: uninstall removed ~/.claude/CLAUDE.md/hooks-config/context7 but LEFT its modified settings.json — asymmetric cleanup, minor). Reinstalled via next's own installer: raw/branch/next/tools/install.sh --next --yes --no-auto-launch → framework from permanent next branch + CLI 0.0.50-next.2207 / gateway 0.0.7-next.2207 from the @next registry lane. Lane works as designed.

N1 — FLAGSHIP (next-only): @next CLI requires Node 22; docs/installer floor says ≥20

On Node 20.19.2 (Debian 13's apt version, and the documented minimum) every mosaic command crashes — even --version — with ERR_REQUIRE_CYCLE_MODULE in @mosaicstack/brain/dist/projects.js. npm corroborates: [email protected] declares node >=22. Verified the same installed CLI runs clean under Node 22.23.2 (nvm). So the @next lane is dead-on-arrival on the documented minimum Node. Fix: installer gates node ≥22 for the next lane (or brain drops the require cycle). fomo-lin now runs Node 22 via nvm (user-level; system apt tops out at 20 — durable fix wants nodesource 22).

F1F12 recurrence scorecard on next

Finding On next
F1 PATH print-only RECURS (identical warning)
F2 ~/.claude writes RECURS (runtime assets copied again; per ruling, no deeper testing — W-F fixes structurally)
F3 doctor warns on fresh install RECURS (10 warnings, same classes)
F4 drift-baseline wedge RECURS (same gated template + drift warning)
F5 skill sets disjoint RECURS (same 7 missing mosaic-*)
F6 init eval injection RECURS (eval at lines 102/118/132 of next's mosaic-init)
F7 init drops mcpServers RECURS (verified: count 0 after init; ensure-script fix works)
F8 roster raw-throw RECURS in code (throw present in next launch.js; not re-triggered — roster restored from backup)
F9 probe 2.0s timeout RECURS (constant unchanged) — and compounded: probe spawns mosaic, which on ambient Node 20 crashes (N1), so the probe fails on slow AND stock-node hosts
F10 broker unit not installed RECURS (hand-relinked next's shipped unit; works)
F11 gateway Redis-on-local Expected FIXED (#689 in next); not yet live-verified — gateway install not re-run this session
F12 wizard exit-0 Untested on next (wizard.ts differs; #1120 tracks)

Chain result on next (with the same three workarounds: MCP ensure-script, restored roster, broker relink, plus probe override): install → init → launch all pass; seat up on Claude Code v2.1.226 / sonnet under Node 22.

Net: next cures nothing in F1F10 (they're all pre-W-F structural issues), carries the gateway fix, and adds one hard regression-class gap (N1 node floor). The W-F gap list stands unchanged as the fix vehicle.

Session 5 (2026-08-08 night) — ~/.mosaic prototype hand-roll (second-host cross-check)

Hand-rolled per HARNESS-HOMES prototype section, on the next-lane framework: skeleton (config/claude {}, auth/claude/jason_woltje.com with primary alias, empty plugins/skills stores), probe seat (profile.json schema 1, overlay {}, composed settings via three-layer deep-merge, credentials two-hop symlink, identity-bootstrap CLAUDE.md, seeded onboarding .claude.json, SOUL.md with positive Identity block).

Smoke test PASS (CLAUDE_CONFIG_DIR=<probe> claude --print): RC=0, auth through the two-hop chain, seat self-identified as "probe". Post-run: both symlinks survived, live credential inode unchanged, transcript in probe's own projects/, probe generated its own backups/sessions, operator ~/.claude untouched. dragon-lin's results replicate on a clean second host — the layout stands up greenfield.

Gap-bites during the roll (feed to W-F)

  • Base-template hole (F4/gap-5 adjacent, NEW): the design's composition base ~/.config/mosaic/framework/runtime/claude/settings.json does NOT exist in the shipped framework; the closest shipped artifact (runtime/claude/settings.json) is the GATED wedge template. Used the operator's vetted ungated settings as base (as dragon-lin did). W-F1 must define + ship the canonical UNGATED system base; gate hooks arrive only via promotion overlay.
  • Identity bootstrap vs permissions (NEW): in --print/restricted mode the seat was DENIED reading SOUL.md outside cwd — "read SOUL.md" bootstrap depends on tool permissions. Generator should materialize the identity INTO the generated CLAUDE.md (parameterized), keeping SOUL.md as source, not runtime dependency.
  • Gap 2 lived experience: probe exists in profile.json but not roster.yaml — the hand-rolled seat and mosaic claude are disjoint universes on the same host.
  • Gap 4 in miniature: fresh-host store is empty; nothing defines what seeds it.
  • Lease posture: hand-rolled seats launch bare claude → ungated by construction until mosaic fleet launch exists (consistent with current bare-for-real-work rule).

Addendum — gap-7 characterization (canonical ungated base)

Diffed operator vetted ungated settings vs shipped gated template: the delta is exactly three items — template-only PreToolUse mutator-gate entry, template-only Stop receipt-observer entry, operator-only mcpServers.sequential-thinking block (whose omission from the template is F7's root cause). Spec: base = template two gate hooks + mcpServers; promotion overlay = the two gate hooks, nothing more. Sent to fred (20260808T200337Z).

Box doctrine — true greenfield, repeatable full-cycle testing (Jason, 2026-08-08)

fomo-lin's defining property: the test operator (scooby) is NOT a fleet seat — comms watcher, git identity, nvm/Node, and repos live entirely outside Mosaic. Therefore Mosaic can be wiped to TRUE ZERO and reinstalled in full, repeatedly, to test protocols end-to-end per cycle (each W-F fixture drop, each next release).

Codified as ~/.local/bin/mosaic-greenfield-reset (dry-run by default, --yes to execute): removes units/gateway/npm packages/npmrc scope/~/.config/mosaic/~/.mosaic/ mosaic-written ~/.claude files (settings reset to stock)/workaround shims; preserves the operator layer (watcher, git creds, nvm, repos, ~/.claude auth + session state, baseline backup). Ends with a verify-zero checklist.

Known boundary impurities the reset explicitly handles: ~/.claude/settings.json is mosaic-written today (its QA hooks fire even in the operator's own session — observed: prevent-memory-write blocked an operator write), and the F9 probe shim sits in ~/.local/bin. Both are named in the script rather than left as ambient state.

Not executed yet — current install (next lane + prototype) is the substrate Fred's W-F1 fixtures target. First full cycle runs when the next testable artifact lands.