stack/docs/scratchpads/1194-framework-tool-drift.md

#1194 — Installed framework-tool drift detection and refresh analysis

Decision

The reported queue-guard source defect was already fixed on main by 58b971ab; the live failure came from a stale ~/.config/mosaic/tools/git/ci-queue-wait.sh. The durable fix is therefore a detector, not a duplicate queue-guard patch.

mosaic doctor now compares the framework tools bundled with the executing Mosaic package against the deployed tools tree. Doctor is the selected visibility boundary because it is observational and operator-invoked: unlike session start, it does not add a repository/network scan to every seat launch, and it cannot silently replace identity or messaging tools while seats are active. It reports drift without changing files. --fail-on-warn converts detected drift into a non-zero doctor result.

Classification

The existing framework-manifest.txt is authoritative. The detector invokes the canonical shared tools/_lib/manifest.sh classify implementation over the complete source census and refuses missing, unreadable, malformed, incomplete, or zero-framework ownership output. Policy is therefore read rather than duplicated:

• Current policy classifies source files under tools/** as framework-owned and required in the deployed tools tree.
• Current policy explicitly classifies tools/_lib/credentials.json operator-owned and excludes it from byte comparison; future policy changes take effect without a detector edit.
• A file present only in the deployed tools tree is operator-owned/unknown by the manifest's fail-safe default. The detector reports it as INSTALLED_ONLY operator-or-unknown under --verbose but does not fail or delete it.
• Empty/partial source traversal, unreadable directories/files, symlinked census entries, root aliases, and descendant source aliases all return CANNOT_ASSERT rather than manufacturing agreement.

This means NOT_INSTALLED is not suppressed by filename guesses such as “test” or “README”: if it ships below source tools/**, the installer contract says it should be installed. Source-only implementation files outside tools/** are outside this detector population by construction.

Current host analysis (observation only; no refresh performed)

A direct source-vs-installed census showed broad drift, including identity and messaging behavior:

• Identity/provider operations: stale git/detect-platform.sh, issue-comment.sh, issue-create.sh, issue-close.sh, issue-view.sh, pr-create.sh, pr-merge.sh, pr-review.sh, pr-metadata.sh; missing pr-edit.sh and several identity/read-back regression tools.
• Messaging/session: stale tmux/agent-send.sh, tmux/send-message.sh, their regressions, and fleet/start-agent-session.sh.
• Gate enforcement: stale git/ci-queue-wait.sh; missing the queue tri-state/process-level suites and terminal-green verifier.
• Lease/QA behavior: stale lease-broker launch/mutation/receipt tools and QA hooks.

Counts vary with source head and installed local/operator files; the detector prints measured counts every run rather than baking this snapshot into policy.

Reviewed refresh command — analyse only, do not run during active seats

Use the package/release updater's manifest-driven keep-mode sync during a quiet maintenance window:

MOSAIC_SYNC_ONLY=1 \
MOSAIC_INSTALL_MODE=keep \
MOSAIC_HOME="$HOME/.config/mosaic" \
bash /path/to/reviewed/@mosaicstack/mosaic/framework/install.sh

For the globally installed package, resolve the reviewed installer rather than guessing its path:

PACKAGE_ROOT="$(dirname "$(node -p "require.resolve('@mosaicstack/mosaic/package.json')")")"
MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep MOSAIC_HOME="$HOME/.config/mosaic" \
  bash "$PACKAGE_ROOT/framework/install.sh"

Do not run this while agent seats are active: the stale set includes identity selection, provider mutation, messaging, queue/merge guards, lease enforcement, and session launch. Syncing those files in place can change behavior between a seat's preflight and mutation.

Post-refresh verification

1. Run mosaic doctor --fail-on-warn; require stale=0 not-installed=0 from the framework drift summary (other unrelated doctor warnings must also be adjudicated).
2. Re-run the constructed process-level queue probes against the installed path, not the source checkout. Use the source suite while overriding its subject path in a reviewed scratch copy, or reproduce these exact observations:
   • pending provider payload: guard must print state=pending, print the pending context, wait, and exit non-zero/timeout — never return immediately with rc 0;
   • malformed payload: guard must print state=malformed and exit non-zero;
   • unsupported but valid status vocabulary: guard must print state=unknown and exit non-zero.
3. Run provider author read-back for one deliberately low-risk wrapper operation before resuming fleet mutation work; wrapper self-report is not identity evidence.
4. Relaunch seats only after the quiet-window verification, because existing processes retain loaded environment/context.

Probe evidence

The detector regression constructs a stale installed tool plus a missing shipped tool and observes rc 1 with distinct STALE and NOT_INSTALLED lines. That case would pass or be invisible before this change because no installed-vs-shipped comparison existed. Additional review-red controls prove:

• empty and unreadable source censuses return CANNOT_ASSERT (they returned clean rc 0 at the first PR head);
• deleting the manifest returns CANNOT_ASSERT, while changing manifest ownership changes the verdict through the canonical resolver (the first head never opened the manifest);
• root and descendant symlink/source aliases cannot return clean (the first head returned clean for a source-backed installed subtree);
• a checker hung during doctor is terminated by a bounded watchdog, emits CANNOT_ASSERT, and doctor reaches its final warnings line (the first head hung and suppressed the remaining audit).

Controls retain byte-identical success, exact credential carve-out behavior, and installed-only preservation.