mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

install.sh silently deletes symlinks under ~/.config/mosaic; operator credential stores are destroyed without a warning #1355

New Issue
Open
opened 2026-08-21 14:43:25 +00:00 by fred · 1 comment
fred commented 2026-08-21 14:43:25 +00:00
Collaborator
Copy Link
Copy Source

Defect

framework/install.sh silently deletes symlinks under $TARGET_DIR (~/.config/mosaic).
The operator-surface protection cannot see them.

verify_operator_surface() enumerates the durable pre-update snapshot with:

find "$DURABLE_SNAPSHOT_DIR" -type f -print0

-type f excludes symlinks. A symlinked path in the config tree is therefore never
snapshotted, never compared, never healed, and never warned about. The SYMLINK-LEAF-GUARD
at the rm -f further down only handles the restore-through-a-link case; it does not put a
dropped link back.

Measured, not inferred

Sandbox HOME, real install.sh, sync-only keep-mode:

ln -s "$OUTSIDE/credentials.json" "$SBX/.config/mosaic/credentials.json"
env -i HOME=$SBX ... bash framework/install.sh   # rc=0

Result: the symlink is gone, the target file is untouched, and the install log contains
no mention of it. Grep for credential|symlink|operator file in the log returns nothing.

Why it matters

tools/_lib/credentials.sh resolves the credential store in this order:

  1. $MOSAIC_CREDENTIALS_FILE
  2. $HOME/.config/mosaic/credentials.json
  3. /etc/mosaic/credentials.json

Its own header comment explicitly invites the symlink pattern:

Operators symlink /etc/mosaic/credentials.json to the host's canonical file once,
instead of exporting MOSAIC_CREDENTIALS_FILE per invocation.

So the framework documents a pattern at path 3 that operators reasonably apply at path 2,
and then the installer deletes it. On a host with no /etc/mosaic fallback, every wrapper
that calls load_credentials fails after an upgrade, with no diagnostic pointing at the
upgrade as the cause. Replayed against the sandbox, the resolver reports the file missing.

Top-level credentials.json is not in the operator carve-out list; only
tools/_lib/credentials.json is referenced there.

Proposed fix

Either is acceptable; the first is better.

  1. Preserve links. Snapshot with -type f -o -type l, compare link targets with
    readlink, and restore a dropped link with ln -sfn rather than cp.
  2. Refuse loudly. If a symlink under $TARGET_DIR would be removed, warn with its path
    and target and tell the operator how to restore it.

A keep-mode upgrade must never silently destroy operator state. A symlink is operator state.

Scope note

Self-referential symlinks the installer itself creates are out of scope. The failing case is
a link whose target lies outside $TARGET_DIR, which no reseed can regenerate.

## Defect `framework/install.sh` silently deletes symlinks under `$TARGET_DIR` (`~/.config/mosaic`). The operator-surface protection cannot see them. `verify_operator_surface()` enumerates the durable pre-update snapshot with: ```bash find "$DURABLE_SNAPSHOT_DIR" -type f -print0 ``` `-type f` **excludes symlinks**. A symlinked path in the config tree is therefore never snapshotted, never compared, never healed, and never warned about. The `SYMLINK-LEAF-GUARD` at the `rm -f` further down only handles the restore-through-a-link case; it does not put a dropped link back. ## Measured, not inferred Sandbox `HOME`, real `install.sh`, sync-only keep-mode: ```bash ln -s "$OUTSIDE/credentials.json" "$SBX/.config/mosaic/credentials.json" env -i HOME=$SBX ... bash framework/install.sh # rc=0 ``` Result: the symlink is **gone**, the target file is untouched, and the install log contains **no mention of it**. Grep for `credential|symlink|operator file` in the log returns nothing. ## Why it matters `tools/_lib/credentials.sh` resolves the credential store in this order: 1. `$MOSAIC_CREDENTIALS_FILE` 2. `$HOME/.config/mosaic/credentials.json` 3. `/etc/mosaic/credentials.json` Its own header comment explicitly invites the symlink pattern: > *Operators symlink `/etc/mosaic/credentials.json` to the host's canonical file once, > instead of exporting `MOSAIC_CREDENTIALS_FILE` per invocation.* So the framework documents a pattern at path 3 that operators reasonably apply at path 2, and then the installer deletes it. On a host with no `/etc/mosaic` fallback, every wrapper that calls `load_credentials` fails after an upgrade, with no diagnostic pointing at the upgrade as the cause. Replayed against the sandbox, the resolver reports the file missing. Top-level `credentials.json` is not in the operator carve-out list; only `tools/_lib/credentials.json` is referenced there. ## Proposed fix Either is acceptable; the first is better. 1. **Preserve links.** Snapshot with `-type f -o -type l`, compare link targets with `readlink`, and restore a dropped link with `ln -sfn` rather than `cp`. 2. **Refuse loudly.** If a symlink under `$TARGET_DIR` would be removed, warn with its path and target and tell the operator how to restore it. A keep-mode upgrade must never silently destroy operator state. A symlink is operator state. ## Scope note Self-referential symlinks the installer itself creates are out of scope. The failing case is a link whose target lies **outside** `$TARGET_DIR`, which no reseed can regenerate.
fred commented 2026-08-21 16:01:15 +00:00
Author
Collaborator
Copy Link
Copy Source

Negative result from a live upgrade on this host, added so the report is not stronger than the evidence.

The defect is still present. 0.0.50-next.2600 ships find ... -type f -print0 at install.sh lines 212 and 330. Symlinks are still excluded from both the scan and the durable snapshot.

It did not reproduce on a live host. A reseed today with MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep left both symlinked credential paths under ~/.config/mosaic intact, with the resolver still resolving. verify_operator_surface() did run before the sync-only exit.

I have not isolated why. My original reproduction was a sandbox at 0.0.50-next.2599; the live run was .2600 under sync-only. Either difference could account for it and I did not separate them.

So the original framing -- that an upgrade destroys operator symlinks -- is unproven on a real upgrade path. What is proven is narrower and still worth fixing: a symlink under the operator surface is never snapshotted, never compared, and cannot be healed. It sits outside the protection verify_operator_surface() exists to provide, whatever else happens to it.

Proposed fixes unchanged: enumerate with -type f -o -type l, compare with readlink, restore with ln -sfn; or refuse loudly on an unrepresentable path.

Negative result from a live upgrade on this host, added so the report is not stronger than the evidence. **The defect is still present.** `0.0.50-next.2600` ships `find ... -type f -print0` at install.sh lines 212 and 330. Symlinks are still excluded from both the scan and the durable snapshot. **It did not reproduce on a live host.** A reseed today with `MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep` left both symlinked credential paths under `~/.config/mosaic` intact, with the resolver still resolving. `verify_operator_surface()` did run before the sync-only exit. **I have not isolated why.** My original reproduction was a sandbox at `0.0.50-next.2599`; the live run was `.2600` under sync-only. Either difference could account for it and I did not separate them. So the original framing -- that an upgrade destroys operator symlinks -- is unproven on a real upgrade path. What is proven is narrower and still worth fixing: **a symlink under the operator surface is never snapshotted, never compared, and cannot be healed.** It sits outside the protection `verify_operator_surface()` exists to provide, whatever else happens to it. Proposed fixes unchanged: enumerate with `-type f -o -type l`, compare with `readlink`, restore with `ln -sfn`; or refuse loudly on an unrepresentable path.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1355