mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

framework-drift-check.py fails OPEN: rc=0 on a wrong --installed-root while reporting 305 not-installed #1204

New Issue
Open
opened 2026-08-13 15:46:10 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-13 15:46:10 +00:00
Copy Link
Copy Source

Summary

tools/framework-drift-check.py exits rc=0 when pointed at a wrong --installed-root, while
simultaneously reporting that essentially nothing is installed. A misconfigured input produces a
clean-looking exit, so the tool certifies "no drift" precisely when it cannot see the tree at all.

Observed

Wrong root:

not-installed=305, in-sync=0, stale=0     rc=0

Correct root (~/.config/mosaic/tools):

in-sync=193, stale=45, not-installed=67   rc=1

The first run is the dangerous one: rc=0 reads as "clean" to any script or agent consuming the
exit code, and the summary line's not-installed=305 is easy to skim past.

I only caught it because the report listed woodpecker/pipeline-list.sh as NOT_INSTALLED — a
wrapper I had been successfully invoking all session. Without that coincidence I would have
recorded "framework is in sync" and acted on it.

Why it matters

This is the same fail-open class already seen in ci-queue-wait.sh printing state=unknown at
rc=0. A verification tool that returns success when its own inputs are wrong is worse than no
tool, because it converts an unknown into a false negative and stops the investigation.

"100% not-installed" is never a real steady state for an installed framework; it is a
near-certain signal that the root is wrong.

Proposed acceptance criteria

  • Validate --installed-root before comparing: if the directory is missing, or if it contains
    none of the manifest's expected entries, exit non-zero with a message naming the path
    that was checked and what was expected there.
  • Treat in-sync=0 with not-installed>0 as a configuration error, not a drift result.
  • Never exit 0 on a run that compared nothing.
  • Add a regression test that runs the checker against an empty/incorrect root and asserts a
    non-zero exit.
  • Audit sibling verification tools for the same pattern — success on unknown inputs.
## Summary `tools/framework-drift-check.py` **exits `rc=0` when pointed at a wrong `--installed-root`**, while simultaneously reporting that essentially nothing is installed. A misconfigured input produces a clean-looking exit, so the tool certifies "no drift" precisely when it cannot see the tree at all. ## Observed Wrong root: ``` not-installed=305, in-sync=0, stale=0 rc=0 ``` Correct root (`~/.config/mosaic/tools`): ``` in-sync=193, stale=45, not-installed=67 rc=1 ``` The first run is the dangerous one: `rc=0` reads as "clean" to any script or agent consuming the exit code, and the summary line's `not-installed=305` is easy to skim past. I only caught it because the report listed `woodpecker/pipeline-list.sh` as NOT_INSTALLED — a wrapper I had been successfully invoking all session. Without that coincidence I would have recorded "framework is in sync" and acted on it. ## Why it matters This is the same fail-open class already seen in `ci-queue-wait.sh` printing `state=unknown` at `rc=0`. A verification tool that returns success when its own inputs are wrong is worse than no tool, because it converts an unknown into a false negative and stops the investigation. "100% not-installed" is never a real steady state for an installed framework; it is a near-certain signal that the root is wrong. ## Proposed acceptance criteria - [ ] Validate `--installed-root` before comparing: if the directory is missing, or if it contains none of the manifest's expected entries, **exit non-zero** with a message naming the path that was checked and what was expected there. - [ ] Treat `in-sync=0` with `not-installed>0` as a configuration error, not a drift result. - [ ] Never exit `0` on a run that compared nothing. - [ ] Add a regression test that runs the checker against an empty/incorrect root and asserts a non-zero exit. - [ ] Audit sibling verification tools for the same pattern — success on unknown inputs.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1204