diff --git a/docs/PRD.md b/docs/PRD.md index 806ea7d3..a551ca73 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -1,5 +1,14 @@ # PRD: Mosaic Stack v0.1.0 +## Current addendum: #1194 — Installed framework-tool drift detection + +- Compare the framework tools shipped with the executing Mosaic package against the deployed `$MOSAIC_HOME/tools` tree by content hash. +- Treat every shipped `tools/**` file as framework-owned/required according to `framework-manifest.txt`, while excluding the explicit operator-owned credential carve-out and preserving installed-only operator/unknown files. +- Distinguish and count `IN_SYNC`, `STALE`, `NOT_INSTALLED`, and installed-only classifications; fail non-zero when shipped tools are stale or absent and refuse self-comparison that would make drift unobservable. +- Surface the observational check through `mosaic doctor`; do not refresh files, restart seats, or mutate live tooling. +- Document identity/messaging/gate behavior changes in the current stale set, the reviewed quiet-window keep-mode refresh command, and post-refresh probes against the installed path. +- Prove by construction that a stale and missing deployed tool are detected; that regression must fail before this checker exists. + ## Metadata - **Owner:** Jason Woltje diff --git a/docs/scratchpads/1194-framework-tool-drift.md b/docs/scratchpads/1194-framework-tool-drift.md new file mode 100644 index 00000000..d026323f --- /dev/null +++ b/docs/scratchpads/1194-framework-tool-drift.md @@ -0,0 +1,63 @@ +# #1194 — Installed framework-tool drift detection and refresh analysis + +## Decision + +The reported queue-guard source defect was already fixed on `main` by `58b971ab`; the live failure came from a stale `~/.config/mosaic/tools/git/ci-queue-wait.sh`. The durable fix is therefore a detector, not a duplicate queue-guard patch. + +`mosaic doctor` now compares the framework tools bundled with the executing Mosaic package against the deployed tools tree. Doctor is the selected visibility boundary because it is observational and operator-invoked: unlike session start, it does not add a repository/network scan to every seat launch, and it cannot silently replace identity or messaging tools while seats are active. It reports drift without changing files. `--fail-on-warn` converts detected drift into a non-zero doctor result. + +## Classification + +The existing `framework-manifest.txt` is authoritative: + +- Every source file under `tools/**` is framework-owned and required in the deployed tools tree. +- `tools/_lib/credentials.json` is explicitly operator-owned and excluded from byte comparison. +- A file present only in the deployed tools tree is operator-owned/unknown by the manifest's fail-safe default. The detector reports it as `INSTALLED_ONLY operator-or-unknown` but does not fail or delete it. + +This means `NOT_INSTALLED` is not suppressed by filename guesses such as “test” or “README”: if it ships below source `tools/**`, the installer contract says it should be installed. Source-only implementation files outside `tools/**` are outside this detector population by construction. + +## Current host analysis (observation only; no refresh performed) + +A direct source-vs-installed census showed broad drift, including identity and messaging behavior: + +- Identity/provider operations: stale `git/detect-platform.sh`, `issue-comment.sh`, `issue-create.sh`, `issue-close.sh`, `issue-view.sh`, `pr-create.sh`, `pr-merge.sh`, `pr-review.sh`, `pr-metadata.sh`; missing `pr-edit.sh` and several identity/read-back regression tools. +- Messaging/session: stale `tmux/agent-send.sh`, `tmux/send-message.sh`, their regressions, and `fleet/start-agent-session.sh`. +- Gate enforcement: stale `git/ci-queue-wait.sh`; missing the queue tri-state/process-level suites and terminal-green verifier. +- Lease/QA behavior: stale lease-broker launch/mutation/receipt tools and QA hooks. + +Counts vary with source head and installed local/operator files; the detector prints measured counts every run rather than baking this snapshot into policy. + +## Reviewed refresh command — analyse only, do not run during active seats + +Use the package/release updater's manifest-driven keep-mode sync during a quiet maintenance window: + +```bash +MOSAIC_SYNC_ONLY=1 \ +MOSAIC_INSTALL_MODE=keep \ +MOSAIC_HOME="$HOME/.config/mosaic" \ +bash /path/to/reviewed/@mosaicstack/mosaic/framework/install.sh +``` + +For the globally installed package, resolve the reviewed installer rather than guessing its path: + +```bash +PACKAGE_ROOT="$(dirname "$(node -p "require.resolve('@mosaicstack/mosaic/package.json')")")" +MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep MOSAIC_HOME="$HOME/.config/mosaic" \ + bash "$PACKAGE_ROOT/framework/install.sh" +``` + +Do not run this while agent seats are active: the stale set includes identity selection, provider mutation, messaging, queue/merge guards, lease enforcement, and session launch. Syncing those files in place can change behavior between a seat's preflight and mutation. + +## Post-refresh verification + +1. Run `mosaic doctor --fail-on-warn`; require `stale=0 not-installed=0` from the framework drift summary (other unrelated doctor warnings must also be adjudicated). +2. Re-run the constructed process-level queue probes against the **installed path**, not the source checkout. Use the source suite while overriding its subject path in a reviewed scratch copy, or reproduce these exact observations: + - pending provider payload: guard must print `state=pending`, print the pending context, wait, and exit non-zero/timeout — never return immediately with rc 0; + - malformed payload: guard must print `state=malformed` and exit non-zero; + - unsupported but valid status vocabulary: guard must print `state=unknown` and exit non-zero. +3. Run provider author read-back for one deliberately low-risk wrapper operation before resuming fleet mutation work; wrapper self-report is not identity evidence. +4. Relaunch seats only after the quiet-window verification, because existing processes retain loaded environment/context. + +## Probe evidence + +The detector regression constructs a stale installed tool plus a missing shipped tool and observes rc 1 with distinct `STALE` and `NOT_INSTALLED` lines. That case would pass or be invisible before this change because no installed-vs-shipped comparison existed. Controls prove byte-identical tools pass, the credential carve-out is ignored, installed-only paths are classified without deletion/failure, and comparing a tree to itself refuses with `CANNOT_ASSERT`. diff --git a/packages/mosaic/framework/tools/_scripts/mosaic-doctor b/packages/mosaic/framework/tools/_scripts/mosaic-doctor index 282adf01..a31c889f 100755 --- a/packages/mosaic/framework/tools/_scripts/mosaic-doctor +++ b/packages/mosaic/framework/tools/_scripts/mosaic-doctor @@ -153,6 +153,23 @@ warn_if_symlink_tree_present() { echo "[mosaic-doctor] Mosaic home: $MOSAIC_HOME" +# Compare the framework tools that this CLI/package ships with the deployed +# ~/.config copy that direct wrappers and systemd units actually execute. Doctor +# is the right boundary: observational, operator-invoked, and already designed +# to report drift without mutating live tooling or restarting active seats. +framework_drift_checker="$(cd -- "$(dirname -- "$0")/../quality/scripts" && pwd)/framework-drift-check.py" +if [[ -f "$framework_drift_checker" ]]; then + echo "[mosaic-doctor] Checking installed framework-tool drift..." + if python3 "$framework_drift_checker" --installed-root "$MOSAIC_HOME/tools"; then + pass "Installed framework tools match shipped source" + else + drift_rc=$? + warn "Installed framework-tool drift detected (checker exit $drift_rc; no files changed)" + fi +else + warn "Framework drift checker is absent from the shipped tools tree" +fi + # Canonical Mosaic checks expect_file "$MOSAIC_HOME/STANDARDS.md" expect_file "$MOSAIC_HOME/USER.md" diff --git a/packages/mosaic/framework/tools/quality/scripts/framework-drift-check.py b/packages/mosaic/framework/tools/quality/scripts/framework-drift-check.py new file mode 100644 index 00000000..d1bb679e --- /dev/null +++ b/packages/mosaic/framework/tools/quality/scripts/framework-drift-check.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +"""Compare deployed Mosaic framework tools with the shipped framework source. + +The framework ownership manifest declares tools/** framework-owned. Consequently every +regular file shipped below source tools/ is expected below MOSAIC_HOME/tools/, except +the explicit operator credential carve-out. Files that exist only in the deployed tree +are operator/unknown state and are reported but never treated as framework drift. +""" + +from __future__ import annotations + +import argparse +import hashlib +import os +from pathlib import Path +import sys + +OPERATOR_CARVE_OUTS = {"_lib/credentials.json"} + + +def digest(path: Path) -> str: + value = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + value.update(chunk) + return value.hexdigest() + + +def default_source_tools() -> Path: + # .../tools/quality/scripts/framework-drift-check.py -> .../tools + return Path(__file__).resolve().parents[2] + + +def normalize_source(path: Path) -> Path: + candidate = path.resolve() + if (candidate / "tools").is_dir(): + candidate = candidate / "tools" + return candidate + + +def files_below(root: Path) -> dict[str, Path]: + return { + path.relative_to(root).as_posix(): path + for path in root.rglob("*") + if path.is_file() + } + + +def main() -> int: + parser = argparse.ArgumentParser(description="Detect deployed Mosaic framework-tool drift") + parser.add_argument( + "--source-root", + type=Path, + default=Path(os.environ["MOSAIC_FRAMEWORK_SOURCE_ROOT"]) + if os.environ.get("MOSAIC_FRAMEWORK_SOURCE_ROOT") + else default_source_tools(), + help="shipped framework root or tools root (default: this script's shipped tools tree)", + ) + parser.add_argument( + "--installed-root", + type=Path, + default=Path(os.environ.get("MOSAIC_HOME", Path.home() / ".config/mosaic")) / "tools", + help="deployed tools root (default: $MOSAIC_HOME/tools)", + ) + parser.add_argument("--verbose", action="store_true", help="list in-sync paths too") + args = parser.parse_args() + + source = normalize_source(args.source_root) + installed = args.installed_root.resolve() + if not source.is_dir(): + print(f"[framework-drift] CANNOT_ASSERT source tools missing: {source}", file=sys.stderr) + return 2 + if not installed.is_dir(): + print(f"[framework-drift] CANNOT_ASSERT installed tools missing: {installed}", file=sys.stderr) + return 2 + if source == installed: + print( + "[framework-drift] CANNOT_ASSERT source and installed roots are identical; " + "run the checker from the bundled package or pass --source-root", + file=sys.stderr, + ) + return 2 + + source_files = files_below(source) + installed_files = files_below(installed) + required = sorted(set(source_files) - OPERATOR_CARVE_OUTS) + in_sync: list[str] = [] + stale: list[str] = [] + not_installed: list[str] = [] + for relative in required: + deployed = installed / relative + if not deployed.is_file(): + not_installed.append(relative) + elif digest(source_files[relative]) == digest(deployed): + in_sync.append(relative) + else: + stale.append(relative) + + installed_only = sorted(set(installed_files) - set(source_files) - OPERATOR_CARVE_OUTS) + if args.verbose: + for relative in in_sync: + print(f"[framework-drift] IN_SYNC {relative}") + for relative in stale: + print(f"[framework-drift] STALE {relative}") + for relative in not_installed: + print(f"[framework-drift] NOT_INSTALLED {relative}") + if args.verbose: + for relative in installed_only: + print(f"[framework-drift] INSTALLED_ONLY operator-or-unknown {relative}") + + print( + "[framework-drift] summary " + f"in-sync={len(in_sync)} stale={len(stale)} not-installed={len(not_installed)} " + f"installed-only={len(installed_only)}" + ) + print( + "[framework-drift] classification tools/**=framework-owned-required; " + "tools/_lib/credentials.json=operator-owned-excluded; " + "installed-only=operator-or-unknown-preserved" + ) + if stale or not_installed: + print( + "[framework-drift] FAIL deployed framework tools do not match shipped source; " + "schedule a reviewed framework reseed", + file=sys.stderr, + ) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/mosaic/framework/tools/quality/scripts/test-framework-drift-check.py b/packages/mosaic/framework/tools/quality/scripts/test-framework-drift-check.py new file mode 100644 index 00000000..cae4dd6c --- /dev/null +++ b/packages/mosaic/framework/tools/quality/scripts/test-framework-drift-check.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +CHECKER = Path(__file__).with_name("framework-drift-check.py") + + +class FrameworkDriftCheckTests(unittest.TestCase): + def setUp(self) -> None: + self.temp = tempfile.TemporaryDirectory() + root = Path(self.temp.name) + self.source = root / "framework" / "tools" + self.installed = root / "home" / "tools" + for directory in (self.source / "git", self.source / "_lib", self.installed / "git", self.installed / "_lib"): + directory.mkdir(parents=True, exist_ok=True) + (self.source / "git" / "guard.sh").write_text("fixed\n") + (self.source / "git" / "new-wrapper.sh").write_text("new\n") + (self.source / "_lib" / "credentials.json").write_text("source-placeholder\n") + + def tearDown(self) -> None: + self.temp.cleanup() + + def run_check(self, *extra: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(CHECKER), "--source-root", str(self.source.parent), "--installed-root", str(self.installed), *extra], + text=True, + capture_output=True, + check=False, + env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}, + ) + + def test_fails_loudly_and_classifies_stale_missing_and_installed_only(self) -> None: + (self.installed / "git" / "guard.sh").write_text("broken\n") + (self.installed / "local-helper.sh").write_text("operator\n") + (self.installed / "_lib" / "credentials.json").write_text("secret\n") + + result = self.run_check("--verbose") + + self.assertEqual(result.returncode, 1) + self.assertIn("STALE git/guard.sh", result.stdout) + self.assertIn("NOT_INSTALLED git/new-wrapper.sh", result.stdout) + self.assertIn("INSTALLED_ONLY operator-or-unknown local-helper.sh", result.stdout) + self.assertNotIn("STALE _lib/credentials.json", result.stdout) + self.assertNotIn("NOT_INSTALLED _lib/credentials.json", result.stdout) + self.assertIn("in-sync=0 stale=1 not-installed=1 installed-only=1", result.stdout) + self.assertIn("FAIL deployed framework tools", result.stderr) + + def test_passes_only_when_every_framework_owned_source_file_matches(self) -> None: + (self.installed / "git" / "guard.sh").write_text("fixed\n") + (self.installed / "git" / "new-wrapper.sh").write_text("new\n") + (self.installed / "_lib" / "credentials.json").write_text("different-operator-secret\n") + + result = self.run_check() + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("in-sync=2 stale=0 not-installed=0 installed-only=0", result.stdout) + + def test_refuses_self_comparison_that_would_make_drift_unobservable(self) -> None: + result = subprocess.run( + [sys.executable, str(CHECKER), "--source-root", str(self.source), "--installed-root", str(self.source)], + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("source and installed roots are identical", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/mosaic/package.json b/packages/mosaic/package.json index 4a5ebd1c..618c219e 100644 --- a/packages/mosaic/package.json +++ b/packages/mosaic/package.json @@ -25,7 +25,7 @@ "lint": "eslint src", "typecheck": "tsc --noEmit", "test": "vitest run --passWithNoTests && pnpm run test:framework-shell", - "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh" + "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh" }, "dependencies": { "@mosaicstack/brain": "workspace:*",