fix(framework): detect installed tool drift (#1194)

docs/PRD.md

@@ -1,5 +1,14 @@
 # PRD: Mosaic Stack v0.1.0
 
+## Current addendum: #1194 — Installed framework-tool drift detection
+
+- Compare the framework tools shipped with the executing Mosaic package against the deployed `$MOSAIC_HOME/tools` tree by content hash.
+- Treat every shipped `tools/**` file as framework-owned/required according to `framework-manifest.txt`, while excluding the explicit operator-owned credential carve-out and preserving installed-only operator/unknown files.
+- Distinguish and count `IN_SYNC`, `STALE`, `NOT_INSTALLED`, and installed-only classifications; fail non-zero when shipped tools are stale or absent and refuse self-comparison that would make drift unobservable.
+- Surface the observational check through `mosaic doctor`; do not refresh files, restart seats, or mutate live tooling.
+- Document identity/messaging/gate behavior changes in the current stale set, the reviewed quiet-window keep-mode refresh command, and post-refresh probes against the installed path.
+- Prove by construction that a stale and missing deployed tool are detected; that regression must fail before this checker exists.
+
 ## Metadata
 
 - **Owner:** Jason Woltje

docs/scratchpads/1194-framework-tool-drift.md

@@ -0,0 +1,63 @@
+# #1194 — Installed framework-tool drift detection and refresh analysis
+
+## Decision
+
+The reported queue-guard source defect was already fixed on `main` by `58b971ab`; the live failure came from a stale `~/.config/mosaic/tools/git/ci-queue-wait.sh`. The durable fix is therefore a detector, not a duplicate queue-guard patch.
+
+`mosaic doctor` now compares the framework tools bundled with the executing Mosaic package against the deployed tools tree. Doctor is the selected visibility boundary because it is observational and operator-invoked: unlike session start, it does not add a repository/network scan to every seat launch, and it cannot silently replace identity or messaging tools while seats are active. It reports drift without changing files. `--fail-on-warn` converts detected drift into a non-zero doctor result.
+
+## Classification
+
+The existing `framework-manifest.txt` is authoritative:
+
+- Every source file under `tools/**` is framework-owned and required in the deployed tools tree.
+- `tools/_lib/credentials.json` is explicitly operator-owned and excluded from byte comparison.
+- A file present only in the deployed tools tree is operator-owned/unknown by the manifest's fail-safe default. The detector reports it as `INSTALLED_ONLY operator-or-unknown` but does not fail or delete it.
+
+This means `NOT_INSTALLED` is not suppressed by filename guesses such as “test” or “README”: if it ships below source `tools/**`, the installer contract says it should be installed. Source-only implementation files outside `tools/**` are outside this detector population by construction.
+
+## Current host analysis (observation only; no refresh performed)
+
+A direct source-vs-installed census showed broad drift, including identity and messaging behavior:
+
+- Identity/provider operations: stale `git/detect-platform.sh`, `issue-comment.sh`, `issue-create.sh`, `issue-close.sh`, `issue-view.sh`, `pr-create.sh`, `pr-merge.sh`, `pr-review.sh`, `pr-metadata.sh`; missing `pr-edit.sh` and several identity/read-back regression tools.
+- Messaging/session: stale `tmux/agent-send.sh`, `tmux/send-message.sh`, their regressions, and `fleet/start-agent-session.sh`.
+- Gate enforcement: stale `git/ci-queue-wait.sh`; missing the queue tri-state/process-level suites and terminal-green verifier.
+- Lease/QA behavior: stale lease-broker launch/mutation/receipt tools and QA hooks.
+
+Counts vary with source head and installed local/operator files; the detector prints measured counts every run rather than baking this snapshot into policy.
+
+## Reviewed refresh command — analyse only, do not run during active seats
+
+Use the package/release updater's manifest-driven keep-mode sync during a quiet maintenance window:
+
+```bash
+MOSAIC_SYNC_ONLY=1 \
+MOSAIC_INSTALL_MODE=keep \
+MOSAIC_HOME="$HOME/.config/mosaic" \
+bash /path/to/reviewed/@mosaicstack/mosaic/framework/install.sh
+```
+
+For the globally installed package, resolve the reviewed installer rather than guessing its path:
+
+```bash
+PACKAGE_ROOT="$(dirname "$(node -p "require.resolve('@mosaicstack/mosaic/package.json')")")"
+MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep MOSAIC_HOME="$HOME/.config/mosaic" \
+  bash "$PACKAGE_ROOT/framework/install.sh"
+```
+
+Do not run this while agent seats are active: the stale set includes identity selection, provider mutation, messaging, queue/merge guards, lease enforcement, and session launch. Syncing those files in place can change behavior between a seat's preflight and mutation.
+
+## Post-refresh verification
+
+1. Run `mosaic doctor --fail-on-warn`; require `stale=0 not-installed=0` from the framework drift summary (other unrelated doctor warnings must also be adjudicated).
+2. Re-run the constructed process-level queue probes against the **installed path**, not the source checkout. Use the source suite while overriding its subject path in a reviewed scratch copy, or reproduce these exact observations:
+   - pending provider payload: guard must print `state=pending`, print the pending context, wait, and exit non-zero/timeout — never return immediately with rc 0;
+   - malformed payload: guard must print `state=malformed` and exit non-zero;
+   - unsupported but valid status vocabulary: guard must print `state=unknown` and exit non-zero.
+3. Run provider author read-back for one deliberately low-risk wrapper operation before resuming fleet mutation work; wrapper self-report is not identity evidence.
+4. Relaunch seats only after the quiet-window verification, because existing processes retain loaded environment/context.
+
+## Probe evidence
+
+The detector regression constructs a stale installed tool plus a missing shipped tool and observes rc 1 with distinct `STALE` and `NOT_INSTALLED` lines. That case would pass or be invisible before this change because no installed-vs-shipped comparison existed. Controls prove byte-identical tools pass, the credential carve-out is ignored, installed-only paths are classified without deletion/failure, and comparing a tree to itself refuses with `CANNOT_ASSERT`.

packages/mosaic/framework/tools/_scripts/mosaic-doctor

@@ -153,6 +153,23 @@ warn_if_symlink_tree_present() {
 
 echo "[mosaic-doctor] Mosaic home: $MOSAIC_HOME"
 
+# Compare the framework tools that this CLI/package ships with the deployed
+# ~/.config copy that direct wrappers and systemd units actually execute. Doctor
+# is the right boundary: observational, operator-invoked, and already designed
+# to report drift without mutating live tooling or restarting active seats.
+framework_drift_checker="$(cd -- "$(dirname -- "$0")/../quality/scripts" && pwd)/framework-drift-check.py"
+if [[ -f "$framework_drift_checker" ]]; then
+  echo "[mosaic-doctor] Checking installed framework-tool drift..."
+  if python3 "$framework_drift_checker" --installed-root "$MOSAIC_HOME/tools"; then
+    pass "Installed framework tools match shipped source"
+  else
+    drift_rc=$?
+    warn "Installed framework-tool drift detected (checker exit $drift_rc; no files changed)"
+  fi
+else
+  warn "Framework drift checker is absent from the shipped tools tree"
+fi
+
 # Canonical Mosaic checks
 expect_file "$MOSAIC_HOME/STANDARDS.md"
 expect_file "$MOSAIC_HOME/USER.md"

packages/mosaic/framework/tools/quality/scripts/framework-drift-check.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Compare deployed Mosaic framework tools with the shipped framework source.
+
+The framework ownership manifest declares tools/** framework-owned. Consequently every
+regular file shipped below source tools/ is expected below MOSAIC_HOME/tools/, except
+the explicit operator credential carve-out. Files that exist only in the deployed tree
+are operator/unknown state and are reported but never treated as framework drift.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import os
+from pathlib import Path
+import sys
+
+OPERATOR_CARVE_OUTS = {"_lib/credentials.json"}
+
+
+def digest(path: Path) -> str:
+    value = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            value.update(chunk)
+    return value.hexdigest()
+
+
+def default_source_tools() -> Path:
+    # .../tools/quality/scripts/framework-drift-check.py -> .../tools
+    return Path(__file__).resolve().parents[2]
+
+
+def normalize_source(path: Path) -> Path:
+    candidate = path.resolve()
+    if (candidate / "tools").is_dir():
+        candidate = candidate / "tools"
+    return candidate
+
+
+def files_below(root: Path) -> dict[str, Path]:
+    return {
+        path.relative_to(root).as_posix(): path
+        for path in root.rglob("*")
+        if path.is_file()
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Detect deployed Mosaic framework-tool drift")
+    parser.add_argument(
+        "--source-root",
+        type=Path,
+        default=Path(os.environ["MOSAIC_FRAMEWORK_SOURCE_ROOT"])
+        if os.environ.get("MOSAIC_FRAMEWORK_SOURCE_ROOT")
+        else default_source_tools(),
+        help="shipped framework root or tools root (default: this script's shipped tools tree)",
+    )
+    parser.add_argument(
+        "--installed-root",
+        type=Path,
+        default=Path(os.environ.get("MOSAIC_HOME", Path.home() / ".config/mosaic")) / "tools",
+        help="deployed tools root (default: $MOSAIC_HOME/tools)",
+    )
+    parser.add_argument("--verbose", action="store_true", help="list in-sync paths too")
+    args = parser.parse_args()
+
+    source = normalize_source(args.source_root)
+    installed = args.installed_root.resolve()
+    if not source.is_dir():
+        print(f"[framework-drift] CANNOT_ASSERT source tools missing: {source}", file=sys.stderr)
+        return 2
+    if not installed.is_dir():
+        print(f"[framework-drift] CANNOT_ASSERT installed tools missing: {installed}", file=sys.stderr)
+        return 2
+    if source == installed:
+        print(
+            "[framework-drift] CANNOT_ASSERT source and installed roots are identical; "
+            "run the checker from the bundled package or pass --source-root",
+            file=sys.stderr,
+        )
+        return 2
+
+    source_files = files_below(source)
+    installed_files = files_below(installed)
+    required = sorted(set(source_files) - OPERATOR_CARVE_OUTS)
+    in_sync: list[str] = []
+    stale: list[str] = []
+    not_installed: list[str] = []
+    for relative in required:
+        deployed = installed / relative
+        if not deployed.is_file():
+            not_installed.append(relative)
+        elif digest(source_files[relative]) == digest(deployed):
+            in_sync.append(relative)
+        else:
+            stale.append(relative)
+
+    installed_only = sorted(set(installed_files) - set(source_files) - OPERATOR_CARVE_OUTS)
+    if args.verbose:
+        for relative in in_sync:
+            print(f"[framework-drift] IN_SYNC {relative}")
+    for relative in stale:
+        print(f"[framework-drift] STALE {relative}")
+    for relative in not_installed:
+        print(f"[framework-drift] NOT_INSTALLED {relative}")
+    if args.verbose:
+        for relative in installed_only:
+            print(f"[framework-drift] INSTALLED_ONLY operator-or-unknown {relative}")
+
+    print(
+        "[framework-drift] summary "
+        f"in-sync={len(in_sync)} stale={len(stale)} not-installed={len(not_installed)} "
+        f"installed-only={len(installed_only)}"
+    )
+    print(
+        "[framework-drift] classification tools/**=framework-owned-required; "
+        "tools/_lib/credentials.json=operator-owned-excluded; "
+        "installed-only=operator-or-unknown-preserved"
+    )
+    if stale or not_installed:
+        print(
+            "[framework-drift] FAIL deployed framework tools do not match shipped source; "
+            "schedule a reviewed framework reseed",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

packages/mosaic/framework/tools/quality/scripts/test-framework-drift-check.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import os
+from pathlib import Path
+import subprocess
+import sys
+import tempfile
+import unittest
+
+CHECKER = Path(__file__).with_name("framework-drift-check.py")
+
+
+class FrameworkDriftCheckTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.temp = tempfile.TemporaryDirectory()
+        root = Path(self.temp.name)
+        self.source = root / "framework" / "tools"
+        self.installed = root / "home" / "tools"
+        for directory in (self.source / "git", self.source / "_lib", self.installed / "git", self.installed / "_lib"):
+            directory.mkdir(parents=True, exist_ok=True)
+        (self.source / "git" / "guard.sh").write_text("fixed\n")
+        (self.source / "git" / "new-wrapper.sh").write_text("new\n")
+        (self.source / "_lib" / "credentials.json").write_text("source-placeholder\n")
+
+    def tearDown(self) -> None:
+        self.temp.cleanup()
+
+    def run_check(self, *extra: str) -> subprocess.CompletedProcess[str]:
+        return subprocess.run(
+            [sys.executable, str(CHECKER), "--source-root", str(self.source.parent), "--installed-root", str(self.installed), *extra],
+            text=True,
+            capture_output=True,
+            check=False,
+            env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"},
+        )
+
+    def test_fails_loudly_and_classifies_stale_missing_and_installed_only(self) -> None:
+        (self.installed / "git" / "guard.sh").write_text("broken\n")
+        (self.installed / "local-helper.sh").write_text("operator\n")
+        (self.installed / "_lib" / "credentials.json").write_text("secret\n")
+
+        result = self.run_check("--verbose")
+
+        self.assertEqual(result.returncode, 1)
+        self.assertIn("STALE git/guard.sh", result.stdout)
+        self.assertIn("NOT_INSTALLED git/new-wrapper.sh", result.stdout)
+        self.assertIn("INSTALLED_ONLY operator-or-unknown local-helper.sh", result.stdout)
+        self.assertNotIn("STALE _lib/credentials.json", result.stdout)
+        self.assertNotIn("NOT_INSTALLED _lib/credentials.json", result.stdout)
+        self.assertIn("in-sync=0 stale=1 not-installed=1 installed-only=1", result.stdout)
+        self.assertIn("FAIL deployed framework tools", result.stderr)
+
+    def test_passes_only_when_every_framework_owned_source_file_matches(self) -> None:
+        (self.installed / "git" / "guard.sh").write_text("fixed\n")
+        (self.installed / "git" / "new-wrapper.sh").write_text("new\n")
+        (self.installed / "_lib" / "credentials.json").write_text("different-operator-secret\n")
+
+        result = self.run_check()
+
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertIn("in-sync=2 stale=0 not-installed=0 installed-only=0", result.stdout)
+
+    def test_refuses_self_comparison_that_would_make_drift_unobservable(self) -> None:
+        result = subprocess.run(
+            [sys.executable, str(CHECKER), "--source-root", str(self.source), "--installed-root", str(self.source)],
+            text=True,
+            capture_output=True,
+            check=False,
+        )
+        self.assertEqual(result.returncode, 2)
+        self.assertIn("source and installed roots are identical", result.stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()

packages/mosaic/package.json

@@ -25,7 +25,7 @@
     "lint": "eslint src",
     "typecheck": "tsc --noEmit",
     "test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
-    "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
+    "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
   },
   "dependencies": {
     "@mosaicstack/brain": "workspace:*",