mosaicstack/stack
1
0
Fork 0
Code Issues 79 Pull Requests 1 Actions Packages Projects Releases 15 Wiki Activity

fix(mosaic): wire constrained recovery runtime boundaries

Browse Source
This commit is contained in:
wjarvis mos-comms
2026-07-19 20:03:16 -05:00
parent f4beedc3e7
commit 95681510c1
14 changed files with 687 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/architecture/lease-broker-protocol.md
View File

@@ -15,6 +15,9 @@ Each connection carries exactly one UTF-8 JSON object followed by one newline, c
- `begin_verification`: authenticated identity, runtime (`claude` or `pi`), cycle `binding`, and a TTL no greater than 300 seconds. The broker revokes existing authority first, enters `PENDING_VERIFICATION`, and returns a single-use promotion token.
- `begin_recovery`: the constrained recovery entrypoint. It rejects caller-provided receipt/challenge fields and delegates to the same `begin_verification` transition, but reports `PENDING_DELIVERY` and marks the volatile cycle as recovery-owned.
- `complete_recovery`: authenticated identity only. It rejects caller-provided receipt/challenge fields, obtains the current recovery challenge only from broker state, and delegates to the same trusted-observer → evidence → consume → promote sequence. An observation failure revokes recovery authority; retry starts a fresh challenge.
The daemon owns a second protected production observer socket (mode `0600`) unless a private `--test-observer-file` fixture is selected. That transport accepts only the exact `record_runtime_observation` schema after kernel `SO_PEERCRED` plus the existing anchor/ancestry authentication; it validates the pending runtime/generation before storing one finalized assistant entry for the in-process `RuntimeReceiptObserver`. It is **not** a broker request action. Claude sends its latest assistant entry from the Stop-hook transport; Pi sends only finalized `message_end` assistant content. The public broker socket continues to reject request-supplied `latest_assistant_message` in begin, observe, and complete paths.
- `promote_lease`: authenticated identity plus the exact pending promotion token. The broker commits token consumption before making `VERIFIED` visible.
- `revoke_lease`: authenticated observer signal; deletes pending tokens and makes the session `UNVERIFIED` immediately. WI-3 Claude/Pi hooks send this existing action; `runtime` and bounded `reason` fields are diagnostic input only and never identity authority.
- `authorize_tool`: authenticated identity, runtime, and exact runtime-reported tool name. The broker returns an explicit allow/deny decision from the whole-class policy and current lease.

117
docs/compaction-refresh/probes/p6_constrained_recovery.py
View File

@@ -1,13 +1,13 @@
#!/usr/bin/env python3
"""P6 constrained-recovery probe; BUILT ONLY and Mos-gated.
DO NOT self-fire. Under Mos authorization only:
DO NOT self-fire. Under Mos authorization only:
python3 -I -S -B docs/compaction-refresh/probes/p6_constrained_recovery.py
The default three isolated runs each start a fresh shipped daemon on a private
Unix socket and drive the shipped recovery *command* out of process. This
probe never resets broker state, replaces the observer seam, or reimplements
promotion: it exercises `recover-context.py` and the daemon's real socket.
The default three isolated runs launch the shipped daemon plus its production
observer transport on private sockets. The driver invokes the shipped recovery
command and adapter gate identity; it never resets broker state, mocks promote,
or taps a live model-output stream.
"""
from __future__ import annotations
@@ -31,8 +31,12 @@ HERE = Path(__file__).resolve().parent
REPOSITORY = HERE.parents[2]
TOOLS = REPOSITORY / "packages/mosaic/framework/tools/lease-broker"
DAEMON = TOOLS / "daemon.py"
GATE = TOOLS / "mutator-gate.py"
RECOVERY_COMMAND = TOOLS / "recover-context.py"
OBSERVER_CLIENT = TOOLS / "receipt-observer-client.py"
FRAGMENTS = TOOLS / "normative_fragments.py"
CLAUDE_SETTINGS = REPOSITORY / "packages/mosaic/framework/runtime/claude/settings.json"
PI_EXTENSION = REPOSITORY / "packages/mosaic/framework/runtime/pi/mosaic-extension.ts"
def load_shipped_fragments():
@@ -76,35 +80,72 @@ def wait_ready(process: subprocess.Popen[str], socket_path: Path) -> None:
raise TimeoutError("shipped daemon did not create private probe socket")
def recovery_command(arguments: list[str], environment: dict[str, str]) -> dict[str, object]:
def run_json(command: list[str], environment: dict[str, str], input_value: object | None = None) -> dict[str, object]:
completed = subprocess.run(
[sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), *arguments],
command,
input=None if input_value is None else json.dumps(input_value),
text=True,
capture_output=True,
env=environment,
check=False,
)
if not completed.stdout.endswith("\n"):
raise AssertionError(f"recovery command omitted framed result: {completed.stderr!r}")
raise AssertionError(f"command omitted framed result: {completed.stderr!r}")
reply = json.loads(completed.stdout)
if not isinstance(reply, dict):
raise AssertionError("recovery command result is not an object")
raise AssertionError("command result is not an object")
return reply
def run_once(index: int) -> None:
# Parity guard: this is an out-of-process driver of the shipped recovery
# entrypoint, not a shadow receipt/promotion implementation.
source = RECOVERY_COMMAND.read_text(encoding="utf-8")
if '"action": "begin_recovery"' not in source or '"action": "complete_recovery"' not in source:
def gate_recovery(runtime: str, phase: str, environment: dict[str, str]) -> None:
command = [sys.executable, "-I", "-S", "-B", str(GATE), "--runtime", runtime]
if runtime == "claude":
command.extend(["--recovery-command", str(RECOVERY_COMMAND)])
recovery_invocation = (
f"python3 {RECOVERY_COMMAND} begin --construction /tmp/p6.json "
"--compaction-epoch 1 --request-epoch 1"
if phase == "begin"
else f"python3 {RECOVERY_COMMAND} complete"
)
value = {"tool_name": "Bash", "tool_input": {"command": recovery_invocation}}
else:
value = {"tool_name": "mosaic_context_recover"}
completed = subprocess.run(command, input=json.dumps(value), text=True, capture_output=True, env=environment, check=False)
if completed.returncode != 0:
raise AssertionError(f"{runtime} recovery invocation remained gated: {completed.stderr!r}")
def record_production_observation(runtime: str, message: str, root: Path, environment: dict[str, str]) -> None:
command = [sys.executable, "-I", "-S", "-B", str(OBSERVER_CLIENT), "--runtime", runtime]
if runtime == "claude":
transcript = root / "claude-transcript.jsonl"
transcript.write_text(json.dumps({"message": {"role": "assistant", "content": message}}) + "\n", encoding="utf-8")
payload = {"transcript_path": str(transcript)}
command.append("--latest-entry")
else:
payload = {"latest_assistant_message": message}
completed = subprocess.run(command, input=json.dumps(payload), text=True, capture_output=True, env=environment, check=False)
if completed.returncode != 0:
raise AssertionError(f"{runtime} production observer transport refused: {completed.stderr!r}")
def run_once(index: int, runtime: str) -> None:
# Parity guard: drive the shipped command and the repaired adapter/observer
# bytes, not a shadow receipt or promotion implementation.
recovery_source = RECOVERY_COMMAND.read_text(encoding="utf-8")
if '"action": "begin_recovery"' not in recovery_source or '"action": "complete_recovery"' not in recovery_source:
raise AssertionError("P6 parity guard: recovery command no longer drives shipped broker entrypoints")
if "--recovery-command" not in CLAUDE_SETTINGS.read_text(encoding="utf-8"):
raise AssertionError("P6 parity guard: Claude recovery mapping is missing")
if "const RECOVERY_TOOL = 'mosaic_context_recover'" not in PI_EXTENSION.read_text(encoding="utf-8"):
raise AssertionError("P6 parity guard: Pi recovery tool mapping is missing")
fragments = load_shipped_fragments()
root = Path(tempfile.mkdtemp(prefix=f"mosaic-p6-recovery-{index}-"))
os.chmod(root, 0o700)
socket_path = root / "broker.sock"
observer_socket = root / "observer.sock"
state_path = root / "state.json"
observer_path = root / "test-observer.json"
construction_path = root / "construction.json"
content = b"P6 constrained recovery fixture\n"
construction = {
@@ -120,7 +161,7 @@ def run_once(index: int) -> None:
os.chmod(construction_path, 0o600)
process = subprocess.Popen(
[sys.executable, "-I", "-S", "-B", str(DAEMON), "--socket", str(socket_path),
"--state", str(state_path), "--test-observer-file", str(observer_path)],
"--state", str(state_path), "--observer-socket", str(observer_socket)],
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
@@ -135,7 +176,7 @@ def run_once(index: int) -> None:
built = fragments.build_payload_from_wire(construction)
normal = request(socket_path, {
"action": "begin_verification", "session_id": session_id, "runtime_generation": 1,
"runtime": "pi", "construction": construction,
"runtime": runtime, "construction": construction,
"binding": {"compaction_epoch": index, "request_epoch": index + 100,
"h_source": built.h_source, "h_payload": built.h_payload, "schema_version": 1},
})
@@ -146,13 +187,15 @@ def run_once(index: int) -> None:
environment = {
**os.environ,
"MOSAIC_LEASE_BROKER_SOCKET": str(socket_path),
"MOSAIC_RECEIPT_OBSERVER_SOCKET": str(observer_socket),
"MOSAIC_LEASE_SESSION_ID": session_id,
"MOSAIC_RUNTIME_GENERATION": "1",
"MOSAIC_LEASE_RUNTIME": "pi",
"MOSAIC_LEASE_RUNTIME": runtime,
}
recovery = recovery_command([
"begin", "--construction", str(construction_path), "--compaction-epoch", str(index + 10),
"--request-epoch", str(index + 110),
gate_recovery(runtime, "begin", environment)
recovery = run_json([
sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), "begin", "--construction", str(construction_path),
"--compaction-epoch", str(index + 10), "--request-epoch", str(index + 110),
], environment)
challenge = recovery.get("receipt_challenge")
receipt = recovery.get("receipt")
@@ -161,32 +204,26 @@ def run_once(index: int) -> None:
if challenge == normal_challenge:
raise AssertionError("recovery reused a normal-path challenge")
# C4: a normal-path receipt is not an admissible recovery receipt.
observer_path.write_text(json.dumps({
"session_id": session_id, "runtime_generation": 1, "latest_assistant_message": normal_receipt,
}), encoding="utf-8")
os.chmod(observer_path, 0o600)
refused = recovery_command(["complete"], environment)
# C4: production observer content is still exact-current-cycle only.
record_production_observation(runtime, normal_receipt, root, environment)
refused = run_json([sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), "complete"], environment)
if refused.get("ok") is not False or refused.get("code") != "RECEIPT_MISMATCH":
raise AssertionError(f"normal-path receipt replay was not refused: {refused!r}")
# Begin a new recovery cycle, then prove shipped evidence -> consume ->
# promote ordering and post-consumption non-repromotion.
recovery = recovery_command([
"begin", "--construction", str(construction_path), "--compaction-epoch", str(index + 20),
"--request-epoch", str(index + 120),
gate_recovery(runtime, "begin", environment)
recovery = run_json([
sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), "begin", "--construction", str(construction_path),
"--compaction-epoch", str(index + 20), "--request-epoch", str(index + 120),
], environment)
receipt = recovery.get("receipt")
if recovery.get("state") != "PENDING_DELIVERY" or not isinstance(receipt, str):
raise AssertionError(f"fresh recovery retry did not pend: {recovery!r}")
observer_path.write_text(json.dumps({
"session_id": session_id, "runtime_generation": 1, "latest_assistant_message": receipt,
}), encoding="utf-8")
os.chmod(observer_path, 0o600)
promoted = recovery_command(["complete"], environment)
record_production_observation(runtime, receipt, root, environment)
gate_recovery(runtime, "complete", environment)
promoted = run_json([sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), "complete"], environment)
if promoted.get("ok") is not True or promoted.get("state") != "VERIFIED":
raise AssertionError(f"recovery consume-before-promote failed: {promoted!r}")
replay = recovery_command(["complete"], environment)
replay = run_json([sys.executable, "-I", "-S", "-B", str(RECOVERY_COMMAND), "complete"], environment)
if replay.get("ok") is not False or replay.get("code") != "INVALID_LEASE_TRANSITION":
raise AssertionError(f"consumed recovery challenge re-promoted: {replay!r}")
finally:
@@ -206,8 +243,8 @@ def main() -> None:
arguments = parser.parse_args()
if arguments.runs != 3:
raise SystemExit("P6 requires exactly three isolated runs")
for index in range(arguments.runs):
run_once(index)
for index, runtime in enumerate(("pi", "claude", "pi")):
run_once(index, runtime)
print("P6 constrained recovery probe PASS: 3 isolated shipped recovery-command runs")

4
docs/guides/lease-broker-operations.md
View File

@@ -31,7 +31,9 @@ The same check runs in the Mosaic package test suite and therefore in root CI. A
Clients must complete the request boundary before waiting for a reply. After sending the single JSON object and its terminating newline, the client **MUST half-close the socket's write side** (`shutdown(SHUT_WR)` in POSIX clients; `socket.end()` in Node) and only then await the response. Merely calling `write()` and waiting is invalid: the broker waits for EOF to enforce the exact-one-frame contract and fails closed at its one-second deadline. Do not replace `end()` with `write()` in client helpers. A delayed second frame remains malformed and is rejected.
`mosaic_context_recover` is the only unverified mutator class. Its durable `mosaic-context-refresh` skill is a thin wrapper over `tools/lease-broker/recover-context.py`: `begin` has the broker rebuild the validated `B_payload`/`H_payload`, revoke first, and mint a new `PENDING_DELIVERY` receipt challenge; `complete` accepts neither receipt text nor a challenge argument. It uses only the trusted `ReceiptObserver` seam for the exact latest assistant entry, commits evidence, consumes the broker-held challenge, and promotes VERIFIED last. A normal-path receipt cannot be replayed through recovery because each retry begins a distinct recovery cycle and recovery completion cannot receive caller-presented evidence.
`mosaic_context_recover` is the only unverified mutator class. Its durable `mosaic-context-refresh` skill is a thin wrapper over `tools/lease-broker/recover-context.py`: `begin` has the broker rebuild the validated `B_payload`/`H_payload`, revoke first, and mint a new `PENDING_DELIVERY` receipt challenge; `complete` accepts neither receipt text nor a challenge argument. Claude maps only the exact direct recovery executable/validated arguments to this exempt tool identity; ordinary `Bash` remains gated. Pi exposes only the `mosaic_context_recover` custom tool; ordinary `bash` and all other tools remain gated. A normal-path receipt cannot be replayed through recovery because each retry begins a distinct recovery cycle and recovery completion cannot receive caller-presented evidence.
Production daemon startup creates a separate private observer socket unless a test-only `--test-observer-file` fixture is selected. Claude's Stop hook sends its exact latest assistant entry and Pi's `message_end` handler sends only finalized assistant content to that authenticated transport; the broker public socket never accepts message text. This is byte-build and private out-of-process harness wiring only: do not activate it against a live daemon, live socket, systemd service, tmux session, or model-output stream outside the controlled integration procedure.
Receipt honesty is load-bearing: absent, malformed, prefix-truncated, and observable adapter-mutated terminal receipts do not promote. A tail-only case is non-promoting only where the concrete terminal payload is malformed or observably incomplete. A tail-preserving middle drop is **not receipt-detectable**; it is the disclosed T-C injection-contract residual deferred to WI-7 server-side evidence. The receipt remains a T-A delivery/liveness prerequisite, never a safety, obedience, or residency proof. The framework skill is source-resident and bridge-projected on install/upgrade; do not hand-create a live runtime symlink.

6
docs/scratchpads/833-constrained-recovery-command.md
View File

@@ -1,9 +1,9 @@
# #833 constrained recovery command — build scratchpad
- **Objective:** Deliver WI-6 only: the single ungated constrained recovery command, durable `mosaic-context-refresh` wrapper skill, AC-1 observable partial-delivery and C4 replay evidence, and an unfired P6 probe.
- **Objective:** Deliver WI-6 plus Mos-ruled B1/B2 repair only: authenticated narrow recovery invocation in Claude/Pi, production receipt-observer transport, AC-1/C4 preservation, and an unfired repaired P6 probe.
- **Authority:** STEP-0 SHA-256 verified 4/4 against the supplied BUILD-BRIEF, SPEC-v5, ratification, and red-team records.
- **Base:** exact `07553ead337a70a9241f826d27571650262b289c`; new branch `feat/833-constrained-recovery-command`; merge-base assertion passed before any commit.
- **Constraints:** No rebase/pull during build; no live install/symlink or live broker/socket/tmux/systemd mutation; no self-review, PR, merge, push, or P6 fire. `docs/TASKS.md` is orchestrator-owned and will not be modified.
- **Constraints:** No rebase/pull during build; no live install/symlink or live broker/socket/tmux/systemd/model-stream activation; no self-review, PR, merge, push, or P6 fire. `docs/TASKS.md` is orchestrator-owned and will not be modified.
- **Plan:**
1. Add red-first unit tests against the recovery broker entrypoint for fresh recovery challenge, normal-receipt replay refusal, observable partial-delivery refusal, and the explicit middle-drop negative capability; commit the RED test and preserve its command output.
2. Implement the recovery command as a thin driver over shared WI-5 broker transitions and the trusted observer seam; it never accepts caller receipt text.
@@ -11,5 +11,5 @@
4. Build an unfired, default-3-run P6 standalone real-socket driver; it is not added to package scripts and will not be run.
5. Run targeted broker/mutator/receipt suites, lint, and format check; report head to `mosaic-100` and stop.
- **Risks:** The observer can only represent an exact latest message. Tail-preserving middle-drop is intentionally not claimed receipt-detectable (T-C residual deferred to WI-7 server evidence).
- **Evidence:** RED test committed at `3b5513bd6efad0c06b599fc759a66cff4286db04`; expected `UNKNOWN_ACTION` is logged in `/home/hermes/agent-work/reviews/833-wi6-red.log`. GREEN: `pnpm --filter @mosaicstack/mosaic run test:framework-shell` passed (deadline 2, normative 5, receipt 5, recovery 4, runtime tools 25, launch guard 12, inventory 14/14); `git diff --check`; Python `py_compile`; and a tmp-only execution of the real #824 `syncClaudeSkills` bridge projected the source skill (`WI-6 #824 tmp projection PASS`). P6 was built but not fired. The ordinary package Vitest/lint/typecheck/root-format commands cannot resolve their executables in this intentionally dependency-free fresh worktree (`node_modules` absent); no install/symlink workaround was used.
- **Evidence:** Original RED test committed at `3b5513bd6efad0c06b599fc759a66cff4286db04`; expected `UNKNOWN_ACTION` is logged in `/home/hermes/agent-work/reviews/833-wi6-red.log`. Repair RED is committed at `f4beedc3e7ac2e142dcbdeefb0e5ee40c20d9b86` and logged in `/home/hermes/agent-work/reviews/833-wi6-repair-red.log` before B1/B2 source. The repair's private out-of-process harness proves exact Claude recovery mapping while UNVERIFIED, non-recovery Bash denial, Pi's exact custom recovery-tool registration, production Pi `message_end` and Claude latest-entry observer promotion, and S1 triple rejection. P6 was rebuilt but remains unfired. The ordinary package Vitest/lint/typecheck/root-format commands cannot resolve their executables in this intentionally dependency-free fresh worktree (`node_modules` absent); no install/symlink workaround was used.
- **Budget:** No explicit task token cap was supplied; scope is fixed to WI-6 and no unrelated behavior will be added.