stack/docs/DEVELOPER-GUIDE/testing/lease-broker-operations.md

title, type, audience, status, source_of_truth

title	type	audience	status	source_of_truth
Lease-broker operations	runbook	developer	current	false

Lease-broker operations

Status: Current for repository inspection and test verification. Live broker startup, recovery, lease mutation, service management, and cleanup remain held/non-operative.

Operational authority: This page authorizes only the non-mutating inspection and test commands in Safe static inspection. It does not authorize starting a daemon or systemd unit, connecting to a live socket, invoking recovery, promoting or revoking a lease, executing a consequential runtime tool, or changing a database.

The lease broker is a Linux-only internal process boundary. Its executable behavior is authoritative in the shipped broker daemon and tests, not in this page. The current architecture references are:

• Broker protocol — framing, kernel identity, ancestry, generations, persistence, and state transitions.
• Lease-broker security — trust boundaries, filesystem hardening, observer behavior, and residuals.
• Whole mutator-class gate — default deny, launch choke points, and broker-owned promotion order.
• Compaction revocation — Claude and Pi lifecycle observers, generation fencing, and the bounded residual.

Safe static inspection

These are the only operative commands documented here. They inspect checked-in files or run isolated tests; they do not start a user service, activate a runtime, connect to PostgreSQL, or mutate repository/product state.

Source and launch inventory

From the repository root, inspect the current implementation and its permanent runtime-launch inventory:

find packages/mosaic/framework/tools/lease-broker packages/mosaic/src/lease-broker packages/mosaic/src/mutator-gate -maxdepth 1 -type f -print | sort
python3 packages/mosaic/framework/tools/lease-broker/check-runtime-launches.py --root . --json

The launch inventory is a static completeness guard. A clean result means the checked-in production launch sites are classified by the guard; it does not prove that a broker, runtime, or service is running.

Safe unit and static contract tests

The focused standard-library tests can be run directly:

python3 packages/mosaic/src/lease-broker/daemon_deadline_unittest.py
python3 packages/mosaic/src/lease-broker/normative_fragments_unittest.py
python3 packages/mosaic/src/lease-broker/receipt_challenge_unittest.py
python3 packages/mosaic/src/lease-broker/context_recovery_unittest.py
python3 packages/mosaic/src/lease-broker/state_store_unittest.py
python3 packages/mosaic/src/lease-broker/framework_skill_portability_unittest.py
python3 packages/mosaic/src/mutator-gate/runtime_tools_unittest.py
python3 packages/mosaic/src/mutator-gate/runtime_launch_guard_unittest.py
python3 packages/mosaic/src/mutator-gate/version_coupling_unittest.py

recovery_runtime_unittest.py and recovery_b1_adversarial_unittest.py are also safe when run as tests: they use private temporary daemons, sockets, and fixtures, never the installed user service or a model stream. They are not operator recovery instructions:

python3 packages/mosaic/src/lease-broker/recovery_runtime_unittest.py
python3 packages/mosaic/src/lease-broker/recovery_b1_adversarial_unittest.py

The lease-broker Vitest acceptance suite and mutator-gate Vitest acceptance suite provide additional private-fixture coverage. Test-created child processes and Unix sockets are disposable test fixtures, not live-service authority.

Current implementation facts

Protected paths and persistence

The broker and supervisor source establish these invariants:

Object	Current contract
Parent directory	The parent containing the socket and state must already exist and have exactly mode 0700.
Broker socket	The daemon refuses an existing path or symlink, binds a new Unix socket, sets it to 0600, and removes only the inode it created during normal shutdown.
State file	A present state file must be a regular, non-symlink file protected as mode 0600, no larger than 4 MiB, and valid state-version-1 JSON. Corrupt, incompatible, oversized, or unsafe state refuses startup.
State writes	Changes are serialized through a mode-0600 temporary file, complete-write loop, fsync, atomic replace, and parent-directory fsync. Post-replace durability uncertainty poisons the store and terminates service processing.
Volatile authority	VERIFIED leases are not restored as live authority after broker restart. Persisted session identity and valid pending-token state are separate from volatile lease state.
Runtime generation	launch-runtime.py creates generation-<broker-session>.state beside the socket. It is an owner-only, locked, monotonic generation file; unsafe, non-regular, oversized, or non-private state fails closed.

The resolved socket path is, in order: explicit MOSAIC_LEASE_BROKER_SOCKET, $XDG_RUNTIME_DIR/mosaic-lease/broker.sock, or /run/user/<uid>/mosaic-lease/broker.sock. The state file is state.json next to that resolved socket. The production observer transport is a separate receipt-observer.sock by default; --test-observer-file is a private test fixture option, not a production deployment path.

The supervisor implementation can materialize a user unit, wrapper, and co-located daemon sources under caller-supplied paths, but applyBrokerSupervisor never runs systemctl, starts daemon.py, or enables the unit. The checked-in unit and start-lease-broker.sh are therefore implementation inputs, not live activation authority for this page.

Protocol and identity boundary

The protocol accepts one UTF-8 JSON object followed by one newline, capped at 64 KiB. The client must half-close its write side after the newline and before waiting for the response (shutdown(SHUT_WR) for POSIX clients or socket.end() for Node). Unterminated, multiple, delayed-second, malformed, oversized, or deadline-exceeded requests fail closed. This is an internal Unix socket protocol, not an HTTP/OpenAPI endpoint.

The broker obtains (pid, uid, gid) from kernel SO_PEERCRED, binds the session to the anchor's /proc/<pid>/stat starttime, and revalidates every walked ancestor's starttime. A caller cannot choose session_id; a sibling or unrelated process cannot authenticate with another process's session. A higher runtime generation replaces the prior incarnation and revokes its tokens and lease authority; a lower generation is stale.

Lease and tool authorization

The broker is the sole lease writer. The effective default-deny policy is:

• Claude read-only classes: Read, Grep, Glob, Ls, and Find.
• Pi read-only classes: read, grep, find, and ls.
• Both runtimes expose the fixed mosaic_context_recover identity as the constrained recovery exception.
• Every other built-in, unknown, custom, MCP, shell, edit, write, deployment, provider, or filesystem mutator is consequential and is denied while the session is not VERIFIED.

The normal broker transition is revoke-first and promote-last:

1. begin_verification authenticates the broker-minted session and current generation, revokes existing authority and pending tokens, validates the exact source construction and binding, and enters a pending state.
2. The receipt challenge and binding are broker-generated. A trusted observer must provide the exact current-cycle assistant entry; caller-supplied latest_assistant_message is rejected on the public broker socket.
3. Promotion consumes the evidence-backed one-time token before volatile VERIFIED becomes visible. A token or receipt cannot be replayed against a later cycle.
4. revoke_lease, generation replacement, broker restart, or monotonic TTL expiry removes consequential-tool authority. TTL is positive and capped at 300 seconds.

launch-runtime.py is the register-before-exec choke point for Claude and Pi. It performs the activation-capability version check, registers the anchor, creates the private generation file, exports the broker identity to descendants, and only then executes the requested runtime. Registration, capability, generation-file, broker-reply, or exec failure denies launch. Claude's raw --dangerously-skip-permissions flag is owned by this wrapper; callers request only its semantic dangerous mode.

Claude's all-tools PreToolUse hook and Pi's tool_call handler submit the runtime-reported tool name to the gate. The gate does not inspect a shell string to decide that one command is safe. Missing identity, malformed input or reply, timeout, broker unavailability, unsafe generation state, and denial all fail closed.

Lifecycle and recovery boundaries

Claude and Pi lifecycle observers use the same authenticated session and broker state machine. Compaction revocation and same-PID replacement behavior are summarized below:

Runtime event	Current source-backed behavior
Claude PreCompact	Revoke before compaction; a failed hook blocks the lifecycle transition.
Claude SessionStart matcher compact	Revoke again after compacted context starts.
Claude SessionStart matcher resume|clear	Bump the private generation, then revoke the replacement incarnation.
Pi session_before_compact	Revoke before compaction; failure cancels the transition.
Pi session_compact then first context	Run one independent post-compaction revoke; a failed post-observer locally blocks later tools until retry.
Pi session_start reason reload, new, resume, or fork	Bump the private generation before revoking and reusing the replacement session.
Fired observer with unavailable broker	Advance the private generation as a local fence and return failure; do not continue consequential work.

The source-resident context-refresh skill is a thin adapter over recover-context.py. Recovery begins with a validated normative-fragment construction, broker-side revoke-first transition, and a fresh PENDING_DELIVERY challenge. The trusted observer records only finalized Claude Stop-hook or Pi message_end content. Completion supplies no receipt or challenge argument; the broker obtains its current challenge, checks the exact observed entry, consumes evidence, and promotes last. Recovery does not accept normal-path receipt text as replayable authority.

A receipt is only a delivery/liveness prerequisite. It is not a safety, obedience, comprehension, or residency proof. Absent, malformed, prefix-truncated, and observably mutated terminal receipts do not promote. A tail-preserving middle drop is explicitly not receipt-detectable and remains a server-side/T-C residual.

Named residual

If both compaction observers are missed while a lease remains unexpired, consequential tools are ALLOWED inside the bounded residual stale window. The mutator gate makes no within-window action-count or timing claim. After the monotonic TTL expires, the next consequential tool is DENIED. This is separate from a fired observer that cannot reach the broker, which fails closed through lifecycle cancellation, a generation fence, and/or the Pi local latch. Protected-branch controls and required review/CI remain the irreducible server-side backstop for the T-C total-hook-miss boundary.

Held future procedure — non-operative

There is no current command authority for the following live procedures. The sequence below records source-backed intent for a separately approved activation/recovery work package; it must not be copied into an operator shell.

Startup and restart outline (held)

1. An activation owner would first materialize the exact framework unit, wrapper, and co-located source copies, then verify the resolved parent, socket, state, observer, and generation paths and their no-symlink/private posture.
2. The approved supervisor would start the daemon only after confirming that the exact socket path is not owned by another process. READY from the daemon and a live Unix socket would be health evidence; a unit file alone would not be healthy.
3. A broker crash would require preserving the state file, identifying the socket owner, and making an explicit restart decision. Restart intentionally clears volatile VERIFIED leases; it is not a way to restore authority.
4. A leftover socket would be handled only after the owning service is confirmed stopped and the exact path is deliberately reviewed. This page supplies no deletion, systemctl, enablement, or start command.

The source's daemon.py argument parser and the user unit show how a future activation is wired, but neither source file grants this page authority to invoke that wiring. applyBrokerSupervisor is materialization only; the supervisor source explicitly leaves enable/start as a separate held step.

Recovery outline (held)

The future adapter flow is: runtime supplies validated construction and current non-negative epochs; broker performs begin_recovery revoke-first and returns one fresh receipt; the adapter delivers that exact receipt; the authenticated observer records the finalized assistant entry; and the adapter requests completion without presenting receipt text or a challenge. Any absent, malformed, stale, duplicated, or mismatched observation leaves the session UNVERIFIED; retry starts a new recovery cycle.

Claude's adapter is restricted to the exact literal recovery argv shape checked by the gate. Pi uses only the registered mosaic_context_recover tool; Pi bash and all other tools remain gated. Do not manually invoke the revoker to restore authority, send assistant text through the public broker request, or reuse a normal-path receipt. The context-refresh skill and its runtime boundary tests are references for future adapter qualification, not an active operator route.

Lease, mutator, and incident handling outline (held)

• There is no supported operator CLI or HTTP endpoint for sending raw begin_verification, promote_lease, revoke_lease, or authorize_tool requests. Do not hand-craft JSON frames, mint tokens, replay receipts, or treat a successful read-only authorization as a promotion.
• After a runtime exits, its generation file may be removed only after an approved check establishes that no process for that broker-minted session remains. Retain stale files during incident analysis; they carry no lease authority by themselves.
• Corrupt, oversized, symlinked, or non-regular state must be preserved for review and not overwritten in place. Establishing new state is an explicit operational decision that invalidates prior sessions and tokens; no recovery command is supplied here.
• Directory 0700 and socket/state 0600 are same-principal hardening only. They do not prevent the same UID from unlinking and replacing a socket. A stronger deployment needs an external protected proxy, ACL, or service boundary that preserves the peer identity required by SO_PEERCRED and ancestry checks. No such deployment procedure is current here.

Related source and tests

• Broker daemon
• Register-and-exec launcher
• Mutator gate
• Recovery command
• Lease broker acceptance tests
• Mutator gate acceptance tests
• Pi lifecycle tests

This migration changes documentation placement and verified wording only. It does not start or change the broker, a runtime, systemd, PostgreSQL, or any other service.