fix(installer): make a greenfield install actually work — node bootstrap, PATH, wizard profile #1229
Assignees
be-coder-05
be-coder-06
be-coder-07
be-coder-08
coder-mos1
coder-mos2
coder2
coder3
f10-coder
fargo
fred
happy
jason.woltje (Jason Woltje)
merge-gate
pepper
rev-974 (Rev-974 (Mosaic reviewer seat, web1))
rev-code-01
rev-code-02
rev-security-01
rev-security-02
rev0
sanity
scooby (Scooby)
scrappy
shaggy
tess
tiny
velma
woodpecker
Clear assignees
be-coder-05
be-coder-06
be-coder-07
be-coder-08
coder-mos1
coder-mos2
coder2
coder3
f10-coder
fargo
fred
happy
jason.woltje (Jason Woltje)
merge-gate
pepper
rev-974 (Rev-974 (Mosaic reviewer seat, web1))
rev-code-01
rev-code-02
rev-security-01
rev-security-02
rev0
sanity
scooby (Scooby)
scrappy
shaggy
tess
tiny
velma
woodpecker
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mosaicstack/stack#1229
Reference in New Issue
Block a user
Problem
A greenfield host could not install Mosaic. Measured on a snapshot-reverted Debian 13 image (no node, npm or git; curl present):
tools/install.sh --next --yesrequire_cmd node— "Required command not found: node", exit 1, nothing installed@mosaicstack/[email protected]+ gateway, exit 0 — andmosaicis not found, because the installer only warns that$PREFIX/binis missing from PATHSo the two ways to fail were "refuses to start" and "reports success and leaves nothing usable."
Three defects
1. Nothing installs Node.js (
tools/install.sh)The installer requires node and npm and installs neither. Fixed with
ensure_node()in preflight: fetches an official release into$HOME/.mosaic/node, verifies against that release'sSHASUMS256.txt, and refuses rather than degrades on a missing entry or a checksum mismatch.sha256sumon Linux,shasumon macOS..tar.gzover the smaller.tar.xzbecause gzip is universally present and xz is not — a minimal image is exactly the case this handles.No-op when a suitable node is already on PATH, so it never fights an operator's nvm/fnm/distro node.
MOSAIC_SKIP_NODE_BOOTSTRAP=1declines the download and fails with instructions instead.Inlined rather than factored into a sibling file because this script is fetched standalone by curl and has nothing to source.
2. PATH was a warning, not an action (
tools/install.sh)Three identical copies of a block that printed "
$PREFIX/binis not on your PATH — add this to your shell rc." An unattended install has no operator to read that. Replaced with oneensure_prefix_on_pathhelper that writes the export and is a no-op when the prefix is already on PATH or already in a profile.3. The wizard wrote PATH to
~/.bashrc(packages/mosaic/src/platform/detect.ts)getShellProfilePath()preferred~/.bashrcwhen it existed, and~/.zshrcfor zsh.setupPath()instages/finalize.tsappends the PATH export to whatever it returns. Debian's default~/.bashrcopens withso a line appended to the bottom of it never runs for
bash -lc, for systemd units, forssh host cmd, or for any agent seat — precisely the consumers that need the CLI..zshrchas the same problem; zsh only reads it for interactive shells.Now
~/.profile, which login shells read and which Debian's copy sources.bashrcfrom for interactive shells, so one line covers both. For zsh the always-sourced file is.zshenv. fish and PowerShell unchanged.This is also the root cause of the standing
[mosaic-link] ERROR: 'mosaic' CLI not found on PATHreports.Evidence
Defects 1 and 2 were first proven on a separate copy of this logic and measured end to end on canary (VMID 1125), rolled back to the
greenfieldsnapshot before each run:/home/mosaic/.profile, exit 0. Freshbash -lc: node, npm 10.9.8, npx andmosaicall resolve.mosaic --version→0.0.50-next.2413;mosaic doctor→ 9 warnings, exit 0;mosaic fleet --helpshows the full surface.Defect 3 is pinned by
packages/mosaic/__tests__/platform/detect.test.ts(6 tests), including a case asserting that no shell resolves to an interactive-only rc file. Falsified by inverting the fix: 5 failed / 1 passed; restored 6/6.ensure_prefix_on_pathwas exercised over five cases against a scratch$HOME— fresh write, idempotent re-run, already-on-PATH no-op, zsh →.zshenv, and an unwritable profile (warns, exits 0, survivesset -e).Regression check
bash -n tools/install.sh— cleanpnpm lint(packages/mosaic) — cleanorigin/next. Those failures are pre-existing and unrelated; this branch adds 6 passing tests.pnpm typecheck: 37 errors before and after, all from unbuilt workspace deps (@mosaicstack/types,@mosaicstack/storage,@mosaicstack/prdy) — unchanged by this branch.Note for reviewers
Two of these fixes also exist in
mosaic/bootstrap, which is archived and cannot accept pushes.AGENTS.mdstill documents that repo'sremote-install.shone-liner as the install path, so operators following the docs get the archived framework-only installer with nomosaic fleet. Worth a separate docs fix — not included here.getShellProfilePath() preferred ~/.bashrc when it existed, and ~/.zshrc for zsh. setupPath() in stages/finalize.ts appends the PATH export to whatever it returns. Debian's default ~/.bashrc opens with case $- in *i*) ;; *) return;; esac so a line appended to the bottom of it never runs for 'bash -lc', for systemd units, for 'ssh host cmd', or for any agent seat — precisely the consumers that need the CLI. An install could print its summary and exit 0 while leaving 'mosaic: command not found'. .zshrc has the same problem: zsh only reads it for interactive shells. Now ~/.profile, which login shells read and which Debian's copy sources .bashrc from for interactive shells, so one line covers both. For zsh the always-sourced file is .zshenv. fish and PowerShell are unchanged. __tests__/platform/detect.test.ts pins it, including a case asserting that no shell resolves to an interactive-only rc file. Falsified by inverting the fix: 5 failed / 1 passed; restored 6/6. Full package suite unchanged at 17 files / 4 tests failing, matching clean origin/next.Two defects found by the second unattended greenfield run on canary (VMID 1125, rolled back to its greenfield snapshot first). 1. ensure_node() exported the Mosaic-managed Node for the installer process and nothing wrote it down. The install finished rc=0, put $PREFIX/bin in ~/.profile, and the next login shell found `mosaic` and then died on env: 'node': No such file or directory The CLI is a Node script, so a CLI on PATH without its runtime is a successful install that produces a broken command. persist_node_on_path() now writes the runtime's bin dir to the same profile, from both the fresh-install and the already-installed-but-not-on-PATH branches. 2. The 'is it already in a shell rc file' guard was a single `grep -qslF "$dir" "${rc_files[@]}"` over four paths, most of which do not exist on a clean host. Handing grep a missing file makes the exit status implementation-defined: GNU grep 3.11 returns 0 when -q already matched an earlier file, ugrep 7.5 returns 2 for the missing one regardless. On the 2 path the caller reads 'not present yet' and appends another PATH line, so every re-install grew the profile. Measured: 3 runs produced 3 duplicate entries; with the fix, 1. path_entry_exists() now tests each file for existence and greps it on its own, so the result does not depend on the grep implementation. The profile-writing body is factored into persist_on_path(), shared by the CLI prefix and the Node runtime, since both now need identical treatment. Verified in a scratch $HOME: fresh write, idempotent across three runs, zsh routes to .zshenv, an unwritable profile warns and survives set -e, and an already-on-PATH prefix is a no-op that creates no file. Falsified by restoring the multi-file grep: duplicates return.Update — commit
00bc602f: two more defects, found by a second greenfield runThe first acceptance run used the
--nextnpm lane and passed. A second unattended run,from a fresh
greenfieldrollback of canary (VMID 1125), finished rc=0 and still left abroken install:
1. The bootstrapped Node was never persisted
ensure_node()didexport PATH=…for the installer's own process and stopped.$PREFIX/binwas written to
~/.profile; the runtime under~/.mosaic/nodewas not. The CLI is a Nodescript, so this is a successful install that produces a command which cannot start. The first
run missed it because that host already had Node reachable by other means.
persist_node_on_path()now writes the runtime's bin dir to the same profile, from both thefresh-install branch and the already-installed-but-not-on-this-shell's-PATH branch.
2. The "already in a shell rc file" guard was implementation-dependent
The guard was a single
grep -qslF "$dir" "${rc_files[@]}"over four paths, three of which donot exist on a clean host. Handing grep a missing file makes the exit status
implementation-defined:
Measured with the old guard: three installs produced three duplicate PATH lines. The new
path_entry_exists()tests each file for existence and greps it on its own, so the result doesnot depend on which grep is installed. Falsified by restoring the old guard — the duplicates
come back.
The profile-writing body is factored into a shared
persist_on_path(), since the CLI prefix andthe Node runtime now need identical treatment.
Evidence
Unit-level, in a scratch
$HOME: fresh write; idempotent across three runs (1 line each, not3); zsh routes to
.zshenv; an unwritable profile warns and survivesset -e; an already-on-PATHprefix is a no-op that creates no file.
End to end, canary rolled back to
greenfield, unattended, no TTY:Then re-running the same installer on the same host:
rc=0, stillnode: 1 cli: 1— noduplicates. And the CLI functions:
mosaic doctor→ 11 warnings, rc=0;mosaic fleet --helpprints the full command set.
bash -n tools/install.shrc=0; pre-pushprettier --checkclean.Two things this VM run does not cover
--ref <branch>sources the framework archive only. The CLI still comes from the npmregistry (
latest= 0.0.49 here), so thedetect.tschange in this PR is covered by the unittests in
packages/mosaic/__tests__/platform/detect.test.tsand not by the VM run. Only--devbuilds the CLI from a checkout.mosaic-linkstepruns before the npm CLI stage, so on a first install it always prints
ERROR: 'mosaic' CLI not found on PATH. That is an ordering defect in the framework and needsits own owner.
Review complete — scooby approves. Head is back to the four commits that were reviewed.
Review
Reviewed by scooby (greenfield install agent, fomo-lin) over the git comms channel. Scooby has no
mosaicstackGitea principal on that host and declined to borrow one, so this cannot be filed as a formal Gitea review from there. Their verdict, verbatim:What they actually did, rather than reading the diff: extracted the shipped functions by line range from the PR head (
path_entry_exists,persist_on_path,node_major_of,node_is_suitable,install_node,persist_node_on_path,ensure_node) and ran them underenv -iwithHOME/TMPDIR/MOSAIC_NODE_ROOTin a scratch dir and node/npm hidden from PATH, so the greenfield bootstrap actually fires. Real downloads from nodejs.org. Their own~/.profileand~/.bashrcwere fingerprinted before and after and came back untouched.current -> v22.23.2~/.profile,.bashrcnever createdexport PATHline after all 3file://dist base)-x bin/nodecheckMOSAIC_SKIP_NODE_BOOTSTRAP=1They also confirmed the
set -einteraction:install_nodeis invoked asif ! install_node, which suppresses-efor its dynamic extent, so the unguardedtar xzffalls through to the-x bin/nodecheck instead of aborting — and checksum verification precedes extraction, so an unverified archive is never unpacked.Non-blocking findings, all recorded, none folded into this PR:
SHASUMS256.txtgets no authenticity check. We verify the tarball matches the manifest, not that the manifest is Node's. TLS is the whole trust root, andMOSAIC_NODE_DIST_BASEwidens it to any mirror. GPG-verifyingSHASUMS256.txt.sigis the fix; it deserves its own review rather than riding in on an installer PR. Tracking as a follow-up.grep " ${tarball}\$"is a BRE, so the dots are wildcards. Only real Node filenames appear in SHASUMS. Leaving it.unamemap pulls the glibc build; on musl/Alpine the binary won't exec and you get a clean "install Node manually" exit, not a silent break. Out of scope for the Debian target.rm -rf "$target"andmv "$target.incoming" "$target"on a same-version reinstall. Repaired by any re-run.Their raw transcript is in their
pr1229/scratchpad on fomo-lin.A commit I pushed after the review, and then reverted
Full disclosure, since it briefly changed what was under review.
After scooby signed off I pushed
47e90767, which added a secondmosaic-link-runtime-assetspass after the CLI stage. The framework'sinstall.shruns that script at the end of Part 1, before Part 2 installs the CLI, so the script has no CLI to ask about lease-enforcement activation, fails safe, and writessettings.jsonwith the #828 enforcement hooks stripped. Every greenfield host therefore ends up with enforcement off because of install ordering.The fix worked. Canary rolled back to
greenfield,--next --yes, no TTY: rc=0, node v22.23.2 and CLI 0.0.50-next.2413 from a fresh login shell, 2 PATH lines in~/.profile, 0 in~/.bashrc, and both hooks wired where they had been stripped.Then
mosaic doctoron that host:So the change moves a greenfield host from "enforcement quietly off, agent works" to "enforcement wired, broker absent, agent bricks on its first gated tool call". Reverted in
fb5bb98a.The real defect is underneath:
mosaic __link-claude-settingsexits 0 (activatable) whilemosaic doctorreports the broker unhealthy, on the same host — and after a complete install there is no supervisor at all (nosystemd --userunit matching lease/broker, nothing under~/.mosaicbut the bootstrapped node, no lease/broker script in~/.config/mosaic/tools/_scripts/). Filed separately as #1234.Head is
fb5bb98a, which restores the tree to00bc602f— byte-identical to what scooby reviewed. The 5th and 6th commits are in the history as a record of the experiment and its reversal.Merge status
Author ≠ reviewer and I will not self-merge. Scooby's approval cannot be filed from fomo-lin for want of a principal. Routing the click-through to Jason, citing this comment.
Head is
07373ede— twelve lines of comment on top of the reviewed treeTwo commits since scooby's approval, neither of which changes behaviour:
fb5bb98a— the revert described in the comment above. Restores the tree to00bc602f.07373ede— F-A and F-C written up as comments ininstall_node(), as I told scooby I would.git diff --stat 00bc602f HEADis now:All twelve are comment lines. No statement in the file changed, so scooby's evidence table still describes the behaviour of this head exactly.
F-A is filed as #1235 — GPG-verifying
SHASUMS256.txt.sig, with the key-distribution question (pinned keyring vs keyserver vs verify-when-gpg-present) laid out for whoever owns it. It gets its own review instead of riding in here.Ready to merge. Author ≠ reviewer and scooby has no
mosaicstackprincipal on fomo-lin, so the click-through needs Jason.Exercised on a snapshot-reverted greenfield Debian 13.6 VM as part of the five-PR composite
e2e-compose. Node was provisioned and the PATH entry written correctly on a box with no node, npm, git or tmux to start with; the CLI was reachable on the next login shell without manual intervention. Full run and the measured stage-by-stage table are in the comment on #1244.⚠️ Pre-merge gate: both
install.shfiles are a hand-resolved keep-bothRaised by @scooby reviewing the E2E, and it is the right catch. Recording it here so it cannot be
lost between now and the merge.
The risk. #1245 and the installer PRs (#1229/#1242) both append a function after
require_cmd()intools/install.sh, and the shared trailing}closes whichever side wins.Git raises this conflict again when the PRs are merged one at a time. My greenfield E2E validated
my resolution. If whoever merges resolves it differently, the
install.shthat ships is notthe
install.shthe E2E measured, and both installer rows of that table become unproven.Same family as #1249 — the thing that runs is not the thing that shipped — except here it
happens at the merge, not in the code.
The resolution that was measured. Keep both functions, each with its own closing brace, in
this order:
tools/install.sh—ensure_prefix_on_path()at :380, closing :391; thencheck_fleet_transport()at :410, closing :428. Call sites:ensure_prefix_on_pathat:970/:983/:1001,
check_fleet_transportat :1149.packages/mosaic/framework/install.sh— #1242's mode-setting (+58:umask 022andchmod 700on
$TARGET_DIR,fleet,fleet/agents,credentials, each warn-on-failure) plus #1245'stransport-check append.
Neither side is a rewrite of the other; the conflict is purely that two additions share a brace.
The check — one command, exact values. After the five land on
next:Must print exactly:
Equivalently, and easier to eyeball:
If those match, the entire E2E table transfers to
nextunchanged. If they do not, the twoinstaller rows are unproven and I will re-run the greenfield before anyone relies on them.
e2e-composeis pushed and stays put as the reference tree until this is confirmed.