feat(mosaic): add constrained context recovery command

2026-07-19 19:04:25 -05:00
parent 3b5513bd6e
commit 7729e6f2f7
10 changed files with 518 additions and 5 deletions
--- a/docs/SITEMAP.md
+++ b/docs/SITEMAP.md
@@ -3,7 +3,8 @@
 ## Compaction refresh lease broker

 - [Internal broker protocol](architecture/lease-broker-protocol.md) — kernel identity, ancestry and generation invariants, framed requests, responses, and persisted cycle bindings.
- [Broker operations](guides/lease-broker-operations.md) — protected paths, startup, fail-closed recovery posture, distinct-principal deployment, and residual risk.
+- [Broker operations](guides/lease-broker-operations.md) — protected paths, startup, constrained recovery, fail-closed posture, distinct-principal deployment, and residual risk.
+- [Constrained recovery skill](../packages/mosaic/framework/skills/mosaic-context-refresh/SKILL.md) — source-resident thin wrapper, receipt scope, C4 replay boundary, and T-C middle-drop disclosure.
 - [Lease-broker security notes](architecture/lease-broker-security.md) — identity, whole-class authorization, threat boundaries, and coordinator review requirements.
 - [Whole mutator-class gate](architecture/mutator-class-gate.md) — default-deny policy, revoke-first/promote-last state machine, TTL, runtime adapters, and T-B/T-C assurance boundary.
 - [Compaction revocation lifecycle](architecture/compaction-revocation.md) — Claude/Pi observer matrix, same-PID generation rollover, failure fencing, and the named bounded residual stale window.
--- a/docs/architecture/lease-broker-protocol.md
+++ b/docs/architecture/lease-broker-protocol.md
@@ -13,12 +13,14 @@ Each connection carries exactly one UTF-8 JSON object followed by one newline, c
 - `mint_token`: authenticated identity plus `binding` containing exactly `compaction_epoch`, `request_epoch`, `h_source`, `h_payload`, and `schema_version`.
 - `consume_token`: authenticated identity plus `token`.
 - `begin_verification`: authenticated identity, runtime (`claude` or `pi`), cycle `binding`, and a TTL no greater than 300 seconds. The broker revokes existing authority first, enters `PENDING_VERIFICATION`, and returns a single-use promotion token.
+- `begin_recovery`: the constrained recovery entrypoint. It rejects caller-provided receipt/challenge fields and delegates to the same `begin_verification` transition, but reports `PENDING_DELIVERY` and marks the volatile cycle as recovery-owned.
+- `complete_recovery`: authenticated identity only. It rejects caller-provided receipt/challenge fields, obtains the current recovery challenge only from broker state, and delegates to the same trusted-observer → evidence → consume → promote sequence. An observation failure revokes recovery authority; retry starts a fresh challenge.
 - `promote_lease`: authenticated identity plus the exact pending promotion token. The broker commits token consumption before making `VERIFIED` visible.
 - `revoke_lease`: authenticated observer signal; deletes pending tokens and makes the session `UNVERIFIED` immediately. WI-3 Claude/Pi hooks send this existing action; `runtime` and bounded `reason` fields are diagnostic input only and never identity authority.
 - `authorize_tool`: authenticated identity, runtime, and exact runtime-reported tool name. The broker returns an explicit allow/deny decision from the whole-class policy and current lease.

 A higher generation for the same anchor atomically replaces the stored incarnation and deletes all prior tokens and lease authority for that session. A lower generation is stale. Runtime descendants resolve the current generation from an owner-only, locked generation file created by the register-before-exec launcher; reload/new/resume/fork observers advance and `fsync` it before broker revocation. This supports generation replacement even when PID/starttime do not change. Tokens are 256-bit values from the operating-system cryptographic RNG and are single use. At most 256 pending tokens may be persisted; another mint fails with `TOKEN_CAPACITY` before mutation. Successful consumption deletes the token, while a replay still fails with `TOKEN_REPLAY`. Live v1 token records retain the existing `consumed: false` schema.

-VERIFIED leases are volatile and monotonic-time bounded: broker restart, generation change, explicit observer revocation, or expiry returns the session to `UNVERIFIED`. `begin_verification` always revokes before minting a new prerequisite. `promote_lease` is valid only from the matching pending cycle; persistence failure rolls token and lease state back, while post-rename durability uncertainty terminates the broker. The WI-1 token is the atomic promotion prerequisite substrate. A later receipt implementation must satisfy that prerequisite but cannot replace the mechanical mutator gate as safety authority.
+VERIFIED leases are volatile and monotonic-time bounded: broker restart, generation change, explicit observer revocation, or expiry returns the session to `UNVERIFIED`. `begin_verification` always revokes before minting a new prerequisite. `begin_recovery` reuses that exact transition and mints a new challenge, so a normal-path receipt/challenge cannot be replayed through recovery. `promote_lease` is valid only from the matching pending cycle; persistence failure rolls token and lease state back, while post-rename durability uncertainty terminates the broker. The WI-1 token is the atomic promotion prerequisite substrate. Receipt evidence is a T-A delivery/liveness prerequisite only; it cannot replace the mechanical mutator gate as safety authority. A tail-preserving middle drop is not receipt-detectable and remains the disclosed T-C/server-side residual.

 State replacement serializes and enforces the 4 MiB maximum before opening a temporary file, then uses a mode-`0600` temporary file, `fsync`, atomic rename, and parent-directory `fsync`. Every broker mutation snapshots the prior v1 state. A commit failure before rename restores that snapshot and leaves durable state unchanged. A failure after rename makes durability uncertain, so the store is poisoned without rolling memory back and the daemon terminates rather than serving with divergent state. Existing state is opened without following symlinks, must be a bounded regular file at mode `0600`, and is fully schema- and invariant-validated before use. Persisted tokens must be unconsumed, match their session's current generation, and remain within the 256-token cap. Session identity is uniquely keyed by `(anchor_pid,anchor_starttime)`; duplicate logical sessions for one anchor refuse startup. State integrity or mode failures refuse startup. The daemon does not log session IDs or tokens.
--- a/docs/compaction-refresh/probes/p6_constrained_recovery.py
+++ b/docs/compaction-refresh/probes/p6_constrained_recovery.py
@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""P6 constrained-recovery probe; BUILT ONLY and Mos-gated.
+
+DO NOT self-fire.  Under Mos authorization only:
+  python3 -I -S -B docs/compaction-refresh/probes/p6_constrained_recovery.py
+
+The default three isolated runs each start a fresh shipped daemon on a private
+Unix socket and drive the shipped recovery *command* out of process.  This
+probe never resets broker state, replaces the observer seam, or reimplements
+promotion: it exercises `recover-context.py` and the daemon's real socket.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import hashlib
+import importlib.util
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+REPOSITORY = HERE.parents[2]
+TOOLS = REPOSITORY / "packages/mosaic/framework/tools/lease-broker"
+DAEMON = TOOLS / "daemon.py"
+RECOVERY_COMMAND = TOOLS / "recover-context.py"
+FRAGMENTS = TOOLS / "normative_fragments.py"
+
+
+def load_shipped_fragments():
+    spec = importlib.util.spec_from_file_location("p6_shipped_fragments", FRAGMENTS)
+    if spec is None or spec.loader is None:
+        raise RuntimeError("shipped normative construction unavailable")
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def request(socket_path: Path, value: dict[str, object]) -> dict[str, object]:
+    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as connection:
+        connection.settimeout(3.0)
+        connection.connect(str(socket_path))
+        connection.sendall((json.dumps(value, separators=(",", ":")) + "\n").encode())
+        connection.shutdown(socket.SHUT_WR)
+        response = bytearray()
+        while True:
+            chunk = connection.recv(4096)
+            if not chunk:
+                break
+            response.extend(chunk)
+    if not response.endswith(b"\n") or response.count(b"\n") != 1:
+        raise AssertionError(f"unframed broker reply: {bytes(response)!r}")
+    reply = json.loads(response[:-1])
+    if not isinstance(reply, dict):
+        raise AssertionError("broker reply is not an object")
+    return reply
+
+
+def wait_ready(process: subprocess.Popen[str], socket_path: Path) -> None:
+    deadline = time.monotonic() + 5.0
+    while time.monotonic() < deadline:
+        if socket_path.exists():
+            return
+        if process.poll() is not None:
+            output = process.stdout.read() if process.stdout is not None else ""
+            raise RuntimeError(f"shipped daemon exited before READY: {output}")
+        time.sleep(0.02)
+    raise TimeoutError("shipped daemon did not create private probe socket")
+
+
+def recovery_command(arguments: list[str], environment: dict[str, str]) -> dict[str, object]:
+    completed = subprocess.run(
+        [sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), *arguments],
+        text=True,
+        capture_output=True,
+        env=environment,
+        check=False,
+    )
+    if not completed.stdout.endswith("\n"):
+        raise AssertionError(f"recovery command omitted framed result: {completed.stderr!r}")
+    reply = json.loads(completed.stdout)
+    if not isinstance(reply, dict):
+        raise AssertionError("recovery command result is not an object")
+    return reply
+
+
+def run_once(index: int) -> None:
+    # Parity guard: this is an out-of-process driver of the shipped recovery
+    # entrypoint, not a shadow receipt/promotion implementation.
+    source = RECOVERY_COMMAND.read_text(encoding="utf-8")
+    if '"action": "begin_recovery"' not in source or '"action": "complete_recovery"' not in source:
+        raise AssertionError("P6 parity guard: recovery command no longer drives shipped broker entrypoints")
+
+    fragments = load_shipped_fragments()
+    root = Path(tempfile.mkdtemp(prefix=f"mosaic-p6-recovery-{index}-"))
+    os.chmod(root, 0o700)
+    socket_path = root / "broker.sock"
+    state_path = root / "state.json"
+    observer_path = root / "test-observer.json"
+    construction_path = root / "construction.json"
+    content = b"P6 constrained recovery fixture\n"
+    construction = {
+        "manifest_version": 1,
+        "generator_version": "p6-constrained-recovery",
+        "fragments": [{
+            "source_id": "authority/p6",
+            "content_base64": base64.b64encode(content).decode("ascii"),
+            "expected_sha256": hashlib.sha256(content).hexdigest(),
+        }],
+    }
+    construction_path.write_text(json.dumps(construction), encoding="utf-8")
+    os.chmod(construction_path, 0o600)
+    process = subprocess.Popen(
+        [sys.executable, "-I", "-S", "-B", str(DAEMON), "--socket", str(socket_path),
+         "--state", str(state_path), "--test-observer-file", str(observer_path)],
+        stdin=subprocess.DEVNULL,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        text=True,
+    )
+    try:
+        wait_ready(process, socket_path)
+        registered = request(socket_path, {"action": "register_anchor", "runtime_generation": 1})
+        session_id = registered.get("session_id")
+        if registered.get("ok") is not True or not isinstance(session_id, str):
+            raise AssertionError(f"broker anchor registration failed: {registered!r}")
+        built = fragments.build_payload_from_wire(construction)
+        normal = request(socket_path, {
+            "action": "begin_verification", "session_id": session_id, "runtime_generation": 1,
+            "runtime": "pi", "construction": construction,
+            "binding": {"compaction_epoch": index, "request_epoch": index + 100,
+                        "h_source": built.h_source, "h_payload": built.h_payload, "schema_version": 1},
+        })
+        normal_challenge = normal.get("receipt_challenge")
+        normal_receipt = normal.get("receipt")
+        if not isinstance(normal_challenge, str) or not isinstance(normal_receipt, str):
+            raise AssertionError("normal path did not mint a receipt challenge")
+        environment = {
+            **os.environ,
+            "MOSAIC_LEASE_BROKER_SOCKET": str(socket_path),
+            "MOSAIC_LEASE_SESSION_ID": session_id,
+            "MOSAIC_RUNTIME_GENERATION": "1",
+            "MOSAIC_LEASE_RUNTIME": "pi",
+        }
+        recovery = recovery_command([
+            "begin", "--construction", str(construction_path), "--compaction-epoch", str(index + 10),
+            "--request-epoch", str(index + 110),
+        ], environment)
+        challenge = recovery.get("receipt_challenge")
+        receipt = recovery.get("receipt")
+        if recovery.get("state") != "PENDING_DELIVERY" or not isinstance(challenge, str) or not isinstance(receipt, str):
+            raise AssertionError(f"recovery command did not drive pending delivery: {recovery!r}")
+        if challenge == normal_challenge:
+            raise AssertionError("recovery reused a normal-path challenge")
+
+        # C4: a normal-path receipt is not an admissible recovery receipt.
+        observer_path.write_text(json.dumps({
+            "session_id": session_id, "runtime_generation": 1, "latest_assistant_message": normal_receipt,
+        }), encoding="utf-8")
+        os.chmod(observer_path, 0o600)
+        refused = recovery_command(["complete"], environment)
+        if refused.get("ok") is not False or refused.get("code") != "RECEIPT_MISMATCH":
+            raise AssertionError(f"normal-path receipt replay was not refused: {refused!r}")
+
+        # Begin a new recovery cycle, then prove shipped evidence -> consume ->
+        # promote ordering and post-consumption non-repromotion.
+        recovery = recovery_command([
+            "begin", "--construction", str(construction_path), "--compaction-epoch", str(index + 20),
+            "--request-epoch", str(index + 120),
+        ], environment)
+        receipt = recovery.get("receipt")
+        if recovery.get("state") != "PENDING_DELIVERY" or not isinstance(receipt, str):
+            raise AssertionError(f"fresh recovery retry did not pend: {recovery!r}")
+        observer_path.write_text(json.dumps({
+            "session_id": session_id, "runtime_generation": 1, "latest_assistant_message": receipt,
+        }), encoding="utf-8")
+        os.chmod(observer_path, 0o600)
+        promoted = recovery_command(["complete"], environment)
+        if promoted.get("ok") is not True or promoted.get("state") != "VERIFIED":
+            raise AssertionError(f"recovery consume-before-promote failed: {promoted!r}")
+        replay = recovery_command(["complete"], environment)
+        if replay.get("ok") is not False or replay.get("code") != "INVALID_LEASE_TRANSITION":
+            raise AssertionError(f"consumed recovery challenge re-promoted: {replay!r}")
+    finally:
+        if process.poll() is None:
+            process.terminate()
+            try:
+                process.wait(timeout=3.0)
+            except subprocess.TimeoutExpired:
+                process.kill()
+                process.wait()
+        shutil.rmtree(root, ignore_errors=True)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--runs", type=int, default=3)
+    arguments = parser.parse_args()
+    if arguments.runs != 3:
+        raise SystemExit("P6 requires exactly three isolated runs")
+    for index in range(arguments.runs):
+        run_once(index)
+    print("P6 constrained recovery probe PASS: 3 isolated shipped recovery-command runs")
+
+
+if __name__ == "__main__":
+    main()
--- a/docs/guides/lease-broker-operations.md
+++ b/docs/guides/lease-broker-operations.md
@@ -31,7 +31,11 @@ The same check runs in the Mosaic package test suite and therefore in root CI. A

 Clients must complete the request boundary before waiting for a reply. After sending the single JSON object and its terminating newline, the client **MUST half-close the socket's write side** (`shutdown(SHUT_WR)` in POSIX clients; `socket.end()` in Node) and only then await the response. Merely calling `write()` and waiting is invalid: the broker waits for EOF to enforce the exact-one-frame contract and fails closed at its one-second deadline. Do not replace `end()` with `write()` in client helpers. A delayed second frame remains malformed and is rejected.

-There is no automated recovery workflow yet. `mosaic_context_recover` is reserved as the only unverified mutator class, but its fixed payload/receipt implementation lands in a later WI. After a runtime exits, its `generation-<session>.state` file may be removed only after verifying that no process for that broker-minted session remains; stale files carry no lease authority but should be retained during incident analysis. After a broker crash, preserve the protected state file and restart only after verifying that no broker owns the socket. Restart intentionally clears all volatile VERIFIED leases. A leftover socket requires an operator to verify the owning service is stopped and remove that exact socket deliberately. Corrupt, oversized, symlinked, or non-regular state fails closed; do not overwrite it. Preserve it for incident review and establish new state only through an explicit operational decision, which invalidates prior sessions and tokens.
+`mosaic_context_recover` is the only unverified mutator class. Its durable `mosaic-context-refresh` skill is a thin wrapper over `tools/lease-broker/recover-context.py`: `begin` has the broker rebuild the validated `B_payload`/`H_payload`, revoke first, and mint a new `PENDING_DELIVERY` receipt challenge; `complete` accepts neither receipt text nor a challenge argument. It uses only the trusted `ReceiptObserver` seam for the exact latest assistant entry, commits evidence, consumes the broker-held challenge, and promotes VERIFIED last. A normal-path receipt cannot be replayed through recovery because each retry begins a distinct recovery cycle and recovery completion cannot receive caller-presented evidence.
+
+Receipt honesty is load-bearing: absent, malformed, prefix-truncated, and observable adapter-mutated terminal receipts do not promote. A tail-only case is non-promoting only where the concrete terminal payload is malformed or observably incomplete. A tail-preserving middle drop is **not receipt-detectable**; it is the disclosed T-C injection-contract residual deferred to WI-7 server-side evidence. The receipt remains a T-A delivery/liveness prerequisite, never a safety, obedience, or residency proof. The framework skill is source-resident and bridge-projected on install/upgrade; do not hand-create a live runtime symlink.
+
+After a runtime exits, its `generation-<session>.state` file may be removed only after verifying that no process for that broker-minted session remains; stale files carry no lease authority but should be retained during incident analysis. After a broker crash, preserve the protected state file and restart only after verifying that no broker owns the socket. Restart intentionally clears all volatile VERIFIED leases. A leftover socket requires an operator to verify the owning service is stopped and remove that exact socket deliberately. Corrupt, oversized, symlinked, or non-regular state fails closed; do not overwrite it. Preserve it for incident review and establish new state only through an explicit operational decision, which invalidates prior sessions and tokens.

 ## Security posture

--- a/docs/scratchpads/833-constrained-recovery-command.md
+++ b/docs/scratchpads/833-constrained-recovery-command.md
@@ -11,4 +11,5 @@
  4. Build an unfired, default-3-run P6 standalone real-socket driver; it is not added to package scripts and will not be run.
  5. Run targeted broker/mutator/receipt suites, lint, and format check; report head to `mosaic-100` and stop.
 - **Risks:** The observer can only represent an exact latest message. Tail-preserving middle-drop is intentionally not claimed receipt-detectable (T-C residual deferred to WI-7 server evidence).
+- **Evidence:** RED test committed at `3b5513bd6efad0c06b599fc759a66cff4286db04`; expected `UNKNOWN_ACTION` is logged in `/home/hermes/agent-work/reviews/833-wi6-red.log`. GREEN: `pnpm --filter @mosaicstack/mosaic run test:framework-shell` passed (deadline 2, normative 5, receipt 5, recovery 4, runtime tools 25, launch guard 12, inventory 14/14); `git diff --check`; Python `py_compile`; and a tmp-only execution of the real #824 `syncClaudeSkills` bridge projected the source skill (`WI-6 #824 tmp projection PASS`). P6 was built but not fired. The ordinary package Vitest/lint/typecheck/root-format commands cannot resolve their executables in this intentionally dependency-free fresh worktree (`node_modules` absent); no install/symlink workaround was used.
 - **Budget:** No explicit task token cap was supplied; scope is fixed to WI-6 and no unrelated behavior will be added.