mosaicstack/stack
14
0
Fork 1
Code Issues 315 Pull Requests 11 Actions Packages Projects Releases 15 Wiki Activity

install.sh can reseed ~/.config/mosaic from any checkout's branch, regressing next-only fixes (stale credential helper deployed 2026-08-21) #1368

New Issue
Open
opened 2026-08-22 03:31:46 +00:00 by code-infra-01 · 1 comment
code-infra-01 commented 2026-08-22 03:31:46 +00:00
Member
Copy Link
Copy Source

Defect

~/.config/mosaic (the deployed, upgrade-managed framework tree) was overwritten at 2026-08-21 19:28 on sb-it-1-dt with content from a main-lineage source, regressing next-only fixes. Measured victim: tools/git/git-credential-mosaic regressed from the seat-slot-aware build (landed on next in 3d2b7123, 2026-08-18) to the pre-fix build (d1f71c2f..., service-store only). Every seat's push to the brain repo then failed auth because both shared-store tokens are revoked and seat slots are the only live path, which the regressed helper cannot read. Also regressed: pr-merge.sh (lost #1280's identity-first resolution and #1334's fork-PR base-repo fix), issue-comment.sh, detect-platform.sh.

Measurements (sb-it-1-dt, 2026-08-21/22)

  1. Deployed ~/.config/mosaic/tools/git/ hashes match origin/main tip (7102ccb9) exactly across four independent files:
    • git-credential-mosaic d1f71c2f = main tip (last changed on main by #873, 07-23); = npm latest 0.0.49; ≠ next (4907b85a since 3d2b7123).
    • pr-merge.sh 023580d4 = main tip; ≠ 0.0.49 tarball (3fe21875); ≠ next (ad3e816f at 836ec3cb).
    • issue-comment.sh 6d4de792 and detect-platform.sh 98d69e36 = main tip, all four mtimes 19:28:24.
  2. The installed npm package at that moment was 0.0.50-next.2612 (package dir mtime 19:26:53; cacache fetch of the 2612 tarballs at 19:26:51), whose framework/ tree carries the FIXED helper (4907b85a).
  3. The 19:28 write cannot have been the 2612 package's own reseed:
    • keep-mode sync copies when bytes differ (cmp deployed vs package: differ), so a package reseed would have written 4907b85a;
    • repro: running the 2612 package install.sh (MOSAIC_SYNC_ONLY=1, fresh target) writes 4907b85a;
    • runFrameworkReseed resolves resolveBundledFrameworkRoot() = the package's framework dir.
  4. Corroboration that the writer was a source-tree run, not the global CLI: ~/.config/mosaic/.install-manifest.json was rewritten at 19:28:59 with cliVersion 0.0.50-next.2612, but no code in the 2612 package writes that file (only uninstall reads it); the writer had to be other code reading the global CLI version.
  5. The drift detector cannot catch this class: checkFrameworkDrift compares integer frameworkVersion (3 on both branches), so a main-over-next regression is invisible; .framework-version stayed 3.

Root cause (mechanism)

Nothing ties a ~/.config/mosaic sync to the installed package. Any checkout's packages/mosaic/framework/install.sh will "upgrade" the deployed tree with whatever branch it has checked out. The seat-slot helper fix landed on next and was never merged to main, so any main-lineage checkout run regresses it. The stable npm channel (latest = 0.0.49) is also main-based and ships the pre-fix helper, so registry-resolved installs have the same exposure.

Fix directions (for triage)

  1. install.sh keep-mode should verify its source against the installed CLI: stamp the package version into the framework tree and refuse a sync whose source version does not match (or require an explicit --from-checkout override that prints a loud warning).
  2. The framework version needs lineage/content discrimination (per-release integer or content hash), so installed < bundled can detect regressions, not just old-vs-new integers that alias across branches.
  3. Consider merging the credential helper fix (3d2b7123) to main so the stable channel stops shipping the regressed helper.

Host-side remediation is tracked separately by fred (per-repo credential.helper override); this issue is the upstream mechanism.

## Defect `~/.config/mosaic` (the deployed, upgrade-managed framework tree) was overwritten at 2026-08-21 19:28 on sb-it-1-dt with content from a **main-lineage source**, regressing next-only fixes. Measured victim: `tools/git/git-credential-mosaic` regressed from the seat-slot-aware build (landed on `next` in 3d2b7123, 2026-08-18) to the pre-fix build (`d1f71c2f...`, service-store only). Every seat's push to the brain repo then failed auth because both shared-store tokens are revoked and seat slots are the only live path, which the regressed helper cannot read. Also regressed: `pr-merge.sh` (lost #1280's identity-first resolution and #1334's fork-PR base-repo fix), `issue-comment.sh`, `detect-platform.sh`. ## Measurements (sb-it-1-dt, 2026-08-21/22) 1. Deployed `~/.config/mosaic/tools/git/` hashes match **origin/main tip (7102ccb9)** exactly across four independent files: - `git-credential-mosaic` d1f71c2f = main tip (last changed on main by #873, 07-23); = npm `latest` 0.0.49; ≠ `next` (4907b85a since 3d2b7123). - `pr-merge.sh` 023580d4 = main tip; ≠ 0.0.49 tarball (3fe21875); ≠ next (ad3e816f at 836ec3cb). - `issue-comment.sh` 6d4de792 and `detect-platform.sh` 98d69e36 = main tip, all four mtimes 19:28:24. 2. The installed npm package at that moment was `0.0.50-next.2612` (package dir mtime 19:26:53; cacache fetch of the 2612 tarballs at 19:26:51), whose `framework/` tree carries the FIXED helper (4907b85a). 3. The 19:28 write cannot have been the 2612 package's own reseed: - keep-mode sync copies when bytes differ (`cmp` deployed vs package: differ), so a package reseed would have written 4907b85a; - repro: running the 2612 package `install.sh` (MOSAIC_SYNC_ONLY=1, fresh target) writes 4907b85a; - `runFrameworkReseed` resolves `resolveBundledFrameworkRoot()` = the package's framework dir. 4. Corroboration that the writer was a source-tree run, not the global CLI: `~/.config/mosaic/.install-manifest.json` was rewritten at 19:28:59 with `cliVersion 0.0.50-next.2612`, but **no code in the 2612 package writes that file** (only uninstall reads it); the writer had to be other code reading the global CLI version. 5. The drift detector cannot catch this class: `checkFrameworkDrift` compares integer `frameworkVersion` (3 on both branches), so a main-over-next regression is invisible; `.framework-version` stayed 3. ## Root cause (mechanism) Nothing ties a `~/.config/mosaic` sync to the installed package. Any checkout's `packages/mosaic/framework/install.sh` will "upgrade" the deployed tree with whatever branch it has checked out. The seat-slot helper fix landed on `next` and was never merged to `main`, so any main-lineage checkout run regresses it. The stable npm channel (`latest` = 0.0.49) is also main-based and ships the pre-fix helper, so registry-resolved installs have the same exposure. ## Fix directions (for triage) 1. install.sh keep-mode should verify its source against the installed CLI: stamp the package version into the framework tree and refuse a sync whose source version does not match (or require an explicit `--from-checkout` override that prints a loud warning). 2. The framework version needs lineage/content discrimination (per-release integer or content hash), so `installed < bundled` can detect regressions, not just old-vs-new integers that alias across branches. 3. Consider merging the credential helper fix (3d2b7123) to `main` so the stable channel stops shipping the regressed helper. Host-side remediation is tracked separately by fred (per-repo credential.helper override); this issue is the upstream mechanism.
code-infra-01 commented 2026-08-22 03:34:27 +00:00
Author
Member
Copy Link
Copy Source

FLEET-WIDE EXPOSURE, not a local incident (fred ruling 2026-08-22): npm view @mosaicstack/mosaic dist.tarball resolves to 0.0.49 today, which is main-based and ships the pre-fix helper (md5 d1f71c2f, service-store only, no fleet/agents resolution). Any host that installs or reinstalls from the stable channel gets this outage with no guard able to see it: the integer frameworkVersion is 3 on both branches, so checkFrameworkDrift is structurally blind to a main-over-next regression, and the 0.0.49 manifest/pr-merge bytes differ from what the 19:28 writer deployed, meaning BOTH the stable channel and any main-lineage checkout run are independently sufficient to regress a host. This host is the first detection, not a special case. Triage implication: fix direction 3 (merge 3d2b7123 to main) closes the stable-channel exposure immediately and independently of the source-verification work, and is arguably the highest-leverage of the three directions.

FLEET-WIDE EXPOSURE, not a local incident (fred ruling 2026-08-22): `npm view @mosaicstack/mosaic dist.tarball` resolves to 0.0.49 today, which is main-based and ships the pre-fix helper (md5 d1f71c2f, service-store only, no fleet/agents resolution). Any host that installs or reinstalls from the stable channel gets this outage with no guard able to see it: the integer frameworkVersion is 3 on both branches, so checkFrameworkDrift is structurally blind to a main-over-next regression, and the 0.0.49 manifest/pr-merge bytes differ from what the 19:28 writer deployed, meaning BOTH the stable channel and any main-lineage checkout run are independently sufficient to regress a host. This host is the first detection, not a special case. Triage implication: fix direction 3 (merge 3d2b7123 to main) closes the stable-channel exposure immediately and independently of the source-verification work, and is arguably the highest-leverage of the three directions.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1368