mosaicstack/stack
6
0
Fork 1
Code Issues 297 Pull Requests 19 Actions Packages Projects Releases 15 Wiki Activity

install.sh: operator-file classification is O(files x patterns) in bash with no size guard, so a large operator tree is indistinguishable from a hang #1314

New Issue
Open
opened 2026-08-18 23:19:24 +00:00 by fred · 1 comment
fred commented 2026-08-18 23:19:24 +00:00
Collaborator
Copy Link
Copy Source

Summary

install.sh classifies every file under TARGET_DIR through manifest_is_framework() to decide operator-vs-framework ownership. That helper walks ~17 manifest patterns per path in pure bash. There is no size guard, no progress output, and no warning — so on a host where the operator tree has grown, mosaic update runs for an unbounded time with a silent terminal and reads as a hang.

Measured

Host sb-it-1-dt, 2026-08-18, upgrading to the next merge d4d32a80.

Files under ~/.config/mosaic Install outcome
Before 372,576 killed at 7 min, nothing changed
After quarantine 7,271 completed rc=0 in under a minute

Of the 372,576, 366,878 were under .claude/ — a 6.0 GB stale Claude Code job checkout at .claude/jobs/<id>/tmp/, of which 351,163 files were node_modules. That is roughly 6.3 million bash string comparisons before the sync phase is even reached.

Confirmed by bash -x: the trace at the point of the kill was still inside _mo_matches, comparing a node_modules/.pnpm/... path against SOUL.md, USER.md, TOOLS.md and the ^[^/]*\.local\.md$ regex.

+ _mo_matches .claude/jobs/<id>/tmp/<repo>/node_modules/.pnpm/@[email protected]/.../index.js
+ local path=... i n=17 re pat
+ [[ ... == SOUL\.md ]]
+ [[ ... == USER\.md ]]
...

Why this is worth fixing rather than documenting

  1. The classification is correct — unknown resolves to operator, deny-wins, and that is the right fail-safe. The defect is purely that it has no cost ceiling.
  2. .claude/ under MOSAIC_HOME is not exotic. Any harness that runs with MOSAIC_HOME as its working home will accumulate job trees there, and job trees contain node_modules.
  3. The failure is silent. No count, no progress, no "scanning N files". An operator sees a dead terminal and cannot distinguish it from a deadlock, which is exactly what happened here — the first diagnosis was "hung", and that was wrong.
  4. It happens before the durable snapshot completes, so a killed run leaves no artifact explaining what it was doing.

Suggested directions

Not prescriptive, roughly in order of effort:

  • Emit a file count up front. Scanning N operator files... turns a hang into a visible, explicable wait. Cheapest fix, and it alone would have prevented the misdiagnosis.
  • Warn above a threshold. Something like: WARNING: 372,576 files under TARGET_DIR; classification will be slow. Consider moving non-Mosaic trees out of MOSAIC_HOME.
  • Prune obviously-not-framework subtrees during the scan. A -name node_modules -prune in the find cannot change ownership semantics: nothing under node_modules can ever match a framework pattern. Same for .git.
  • Short-circuit the pattern loop. Any path whose first segment is not a manifest root can skip all 17 comparisons.

Workaround

Move non-Mosaic trees out of the install target before upgrading. Rename, do not delete, and keep it on the same filesystem so it is instant and reversible:

mv ~/.config/mosaic/.claude/jobs/<id>/tmp ~/.config/mosaic-quarantine-<date>/

Not affected

Secrets and operator config were untouched throughout — ~/.config/mosaic/secrets held 32 files at inventory digest 911c7ceb both before and after, and keep-mode never writes or deletes operator paths. This is a performance and observability defect, not a correctness one.

Found while deploying #1311.

## Summary `install.sh` classifies **every file under `TARGET_DIR`** through `manifest_is_framework()` to decide operator-vs-framework ownership. That helper walks ~17 manifest patterns per path in pure bash. There is no size guard, no progress output, and no warning — so on a host where the operator tree has grown, `mosaic update` runs for an unbounded time with a silent terminal and reads as a hang. ## Measured Host `sb-it-1-dt`, 2026-08-18, upgrading to the `next` merge `d4d32a80`. | | Files under `~/.config/mosaic` | Install outcome | |---|---|---| | Before | **372,576** | killed at 7 min, **nothing changed** | | After quarantine | **7,271** | completed rc=0 in **under a minute** | Of the 372,576, **366,878 were under `.claude/`** — a 6.0 GB stale Claude Code job checkout at `.claude/jobs/<id>/tmp/`, of which 351,163 files were `node_modules`. That is roughly **6.3 million bash string comparisons** before the sync phase is even reached. Confirmed by `bash -x`: the trace at the point of the kill was still inside `_mo_matches`, comparing a `node_modules/.pnpm/...` path against `SOUL.md`, `USER.md`, `TOOLS.md` and the `^[^/]*\.local\.md$` regex. ``` + _mo_matches .claude/jobs/<id>/tmp/<repo>/node_modules/.pnpm/@smithy+[email protected]/.../index.js + local path=... i n=17 re pat + [[ ... == SOUL\.md ]] + [[ ... == USER\.md ]] ... ``` ## Why this is worth fixing rather than documenting 1. The classification is **correct** — unknown resolves to operator, deny-wins, and that is the right fail-safe. The defect is purely that it has no cost ceiling. 2. `.claude/` under `MOSAIC_HOME` is not exotic. Any harness that runs with `MOSAIC_HOME` as its working home will accumulate job trees there, and job trees contain `node_modules`. 3. The failure is **silent**. No count, no progress, no "scanning N files". An operator sees a dead terminal and cannot distinguish it from a deadlock, which is exactly what happened here — the first diagnosis was "hung", and that was wrong. 4. It happens **before** the durable snapshot completes, so a killed run leaves no artifact explaining what it was doing. ## Suggested directions Not prescriptive, roughly in order of effort: - **Emit a file count up front.** `Scanning N operator files...` turns a hang into a visible, explicable wait. Cheapest fix, and it alone would have prevented the misdiagnosis. - **Warn above a threshold.** Something like: `WARNING: 372,576 files under TARGET_DIR; classification will be slow. Consider moving non-Mosaic trees out of MOSAIC_HOME.` - **Prune obviously-not-framework subtrees during the scan.** A `-name node_modules -prune` in the `find` cannot change ownership semantics: nothing under `node_modules` can ever match a framework pattern. Same for `.git`. - **Short-circuit the pattern loop.** Any path whose first segment is not a manifest root can skip all 17 comparisons. ## Workaround Move non-Mosaic trees out of the install target before upgrading. Rename, do not delete, and keep it on the same filesystem so it is instant and reversible: ```bash mv ~/.config/mosaic/.claude/jobs/<id>/tmp ~/.config/mosaic-quarantine-<date>/ ``` ## Not affected Secrets and operator config were untouched throughout — `~/.config/mosaic/secrets` held 32 files at inventory digest `911c7ceb` both before and after, and keep-mode never writes or deletes operator paths. This is a performance and observability defect, not a correctness one. Found while deploying #1311.
fred commented 2026-08-18 23:21:02 +00:00
Author
Collaborator
Copy Link
Copy Source

Follow-up measurement that raises the priority: the framework itself now creates this condition on a clean host.

After the keep-mode upgrade completed, ~/.config/mosaic/tools/ went from 303 files to 12,118 — of which 11,824 are tools/excalidraw/ (a bundled MCP tool with its dependency tree).

That means a freshly installed, entirely operator-clean host now carries ~12k framework files under TARGET_DIR. Every subsequent mosaic update classifies all of them, at ~17 pattern comparisons each — roughly 200,000 bash string comparisons before the sync phase begins, with no operator having done anything unusual.

So this is not only a "don't put junk in MOSAIC_HOME" problem. The baseline cost grows with whatever the framework vendors, and tools/excalidraw alone moved the floor by ~40x.

The cheap mitigations get correspondingly more valuable:

  • pruning node_modules from the scan cannot change ownership semantics (nothing inside can match a framework pattern) and would remove the large majority of these 11,824
  • short-circuiting the pattern loop on a non-matching first path segment would cut the per-file constant for all of them

Measured on the same host, same upgrade, immediately after the run described above.

Follow-up measurement that raises the priority: **the framework itself now creates this condition on a clean host.** After the keep-mode upgrade completed, `~/.config/mosaic/tools/` went from 303 files to **12,118** — of which **11,824 are `tools/excalidraw/`** (a bundled MCP tool with its dependency tree). That means a freshly installed, entirely operator-clean host now carries ~12k framework files under `TARGET_DIR`. Every subsequent `mosaic update` classifies all of them, at ~17 pattern comparisons each — roughly **200,000 bash string comparisons before the sync phase begins**, with no operator having done anything unusual. So this is not only a "don't put junk in `MOSAIC_HOME`" problem. The baseline cost grows with whatever the framework vendors, and `tools/excalidraw` alone moved the floor by ~40x. The cheap mitigations get correspondingly more valuable: - pruning `node_modules` from the scan cannot change ownership semantics (nothing inside can match a framework pattern) and would remove the large majority of these 11,824 - short-circuiting the pattern loop on a non-matching first path segment would cut the per-file constant for all of them Measured on the same host, same upgrade, immediately after the run described above.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
fargo fred happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 sanity scooby (Scooby) scrappy shaggy tiny velma vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1314