diff --git a/docs/DEVELOPER-GUIDE/operations/lease-broker-operations.md b/docs/DEVELOPER-GUIDE/operations/lease-broker-operations.md new file mode 100644 index 00000000..dfc57c25 --- /dev/null +++ b/docs/DEVELOPER-GUIDE/operations/lease-broker-operations.md @@ -0,0 +1,292 @@ +--- +title: Lease-broker operations +type: runbook +audience: developer +status: current +source_of_truth: false +--- + +# Lease-broker operations + +> **Status:** Current for repository inspection and test verification. Live broker +> startup, recovery, lease mutation, service management, and cleanup remain +> **held/non-operative**. +> +> **Operational authority:** This page authorizes only the non-mutating +> inspection and test commands in [Safe static inspection](#safe-static-inspection). +> It does not authorize starting a daemon or systemd unit, connecting to a live +> socket, invoking recovery, promoting or revoking a lease, executing a +> consequential runtime tool, or changing a database. + +The lease broker is a Linux-only internal process boundary. Its executable +behavior is authoritative in the [shipped broker daemon](../../../packages/mosaic/framework/tools/lease-broker/daemon.py) +and tests, not in this page. The current architecture references are: + +- [Broker protocol](../architecture/lease-broker-protocol.md) — framing, + kernel identity, ancestry, generations, persistence, and state transitions. +- [Lease-broker security](../architecture/lease-broker-security.md) — trust + boundaries, filesystem hardening, observer behavior, and residuals. +- [Whole mutator-class gate](../architecture/mutator-class-gate.md) — default + deny, launch choke points, and broker-owned promotion order. +- [Compaction revocation](../architecture/compaction-revocation.md) — Claude + and Pi lifecycle observers, generation fencing, and the bounded residual. + +## Safe static inspection + +These are the only operative commands documented here. They inspect checked-in +files or run isolated tests; they do not start a user service, activate a +runtime, connect to PostgreSQL, or mutate repository/product state. + +### Source and launch inventory + +From the repository root, inspect the current implementation and its permanent +runtime-launch inventory: + +```bash +find packages/mosaic/framework/tools/lease-broker packages/mosaic/src/lease-broker packages/mosaic/src/mutator-gate -maxdepth 1 -type f -print | sort +python3 packages/mosaic/framework/tools/lease-broker/check-runtime-launches.py --root . --json +``` + +The launch inventory is a static completeness guard. A clean result means the +checked-in production launch sites are classified by the guard; it does not +prove that a broker, runtime, or service is running. + +### Safe unit and static contract tests + +The focused standard-library tests can be run directly: + +```bash +python3 packages/mosaic/src/lease-broker/daemon_deadline_unittest.py +python3 packages/mosaic/src/lease-broker/normative_fragments_unittest.py +python3 packages/mosaic/src/lease-broker/receipt_challenge_unittest.py +python3 packages/mosaic/src/lease-broker/context_recovery_unittest.py +python3 packages/mosaic/src/lease-broker/state_store_unittest.py +python3 packages/mosaic/src/lease-broker/framework_skill_portability_unittest.py +python3 packages/mosaic/src/mutator-gate/runtime_tools_unittest.py +python3 packages/mosaic/src/mutator-gate/runtime_launch_guard_unittest.py +python3 packages/mosaic/src/mutator-gate/version_coupling_unittest.py +``` + +`recovery_runtime_unittest.py` and `recovery_b1_adversarial_unittest.py` are +also safe when run as tests: they use private temporary daemons, sockets, and +fixtures, never the installed user service or a model stream. They are not +operator recovery instructions: + +```bash +python3 packages/mosaic/src/lease-broker/recovery_runtime_unittest.py +python3 packages/mosaic/src/lease-broker/recovery_b1_adversarial_unittest.py +``` + +The [lease-broker Vitest acceptance suite](../../../packages/mosaic/src/lease-broker/lease-broker.acceptance.spec.ts) +and [mutator-gate Vitest acceptance suite](../../../packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts) +provide additional private-fixture coverage. Test-created child processes and +Unix sockets are disposable test fixtures, not live-service authority. + +## Current implementation facts + +### Protected paths and persistence + +The broker and supervisor source establish these invariants: + +| Object | Current contract | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Parent directory | The parent containing the socket and state must already exist and have exactly mode `0700`. | +| Broker socket | The daemon refuses an existing path or symlink, binds a new Unix socket, sets it to `0600`, and removes only the inode it created during normal shutdown. | +| State file | A present state file must be a regular, non-symlink file protected as mode `0600`, no larger than 4 MiB, and valid state-version-1 JSON. Corrupt, incompatible, oversized, or unsafe state refuses startup. | +| State writes | Changes are serialized through a mode-`0600` temporary file, complete-write loop, `fsync`, atomic replace, and parent-directory `fsync`. Post-replace durability uncertainty poisons the store and terminates service processing. | +| Volatile authority | `VERIFIED` leases are not restored as live authority after broker restart. Persisted session identity and valid pending-token state are separate from volatile lease state. | +| Runtime generation | `launch-runtime.py` creates `generation-.state` beside the socket. It is an owner-only, locked, monotonic generation file; unsafe, non-regular, oversized, or non-private state fails closed. | + +The resolved socket path is, in order: explicit +`MOSAIC_LEASE_BROKER_SOCKET`, `$XDG_RUNTIME_DIR/mosaic-lease/broker.sock`, or +`/run/user//mosaic-lease/broker.sock`. The state file is `state.json` next +to that resolved socket. The production observer transport is a separate +`receipt-observer.sock` by default; `--test-observer-file` is a private test +fixture option, not a production deployment path. + +The supervisor implementation can materialize a user unit, wrapper, and +co-located daemon sources under caller-supplied paths, but +[`applyBrokerSupervisor`](../../../packages/mosaic/src/lease-broker/broker-supervisor.ts) +never runs `systemctl`, starts `daemon.py`, or enables the unit. The checked-in +unit and [`start-lease-broker.sh`](../../../packages/mosaic/framework/tools/lease-broker/start-lease-broker.sh) +are therefore implementation inputs, not live activation authority for this +page. + +### Protocol and identity boundary + +The protocol accepts one UTF-8 JSON object followed by one newline, capped at +64 KiB. The client must half-close its write side after the newline and before +waiting for the response (`shutdown(SHUT_WR)` for POSIX clients or +`socket.end()` for Node). Unterminated, multiple, delayed-second, malformed, +oversized, or deadline-exceeded requests fail closed. This is an internal Unix +socket protocol, not an HTTP/OpenAPI endpoint. + +The broker obtains `(pid, uid, gid)` from kernel `SO_PEERCRED`, binds the +session to the anchor's `/proc//stat` starttime, and revalidates every +walked ancestor's starttime. A caller cannot choose `session_id`; a sibling or +unrelated process cannot authenticate with another process's session. A higher +runtime generation replaces the prior incarnation and revokes its tokens and +lease authority; a lower generation is stale. + +### Lease and tool authorization + +The broker is the sole lease writer. The effective default-deny policy is: + +- Claude read-only classes: `Read`, `Grep`, `Glob`, `Ls`, and `Find`. +- Pi read-only classes: `read`, `grep`, `find`, and `ls`. +- Both runtimes expose the fixed `mosaic_context_recover` identity as the + constrained recovery exception. +- Every other built-in, unknown, custom, MCP, shell, edit, write, deployment, + provider, or filesystem mutator is consequential and is denied while the + session is not `VERIFIED`. + +The normal broker transition is revoke-first and promote-last: + +1. `begin_verification` authenticates the broker-minted session and current + generation, revokes existing authority and pending tokens, validates the + exact source construction and binding, and enters a pending state. +2. The receipt challenge and binding are broker-generated. A trusted observer + must provide the exact current-cycle assistant entry; caller-supplied + `latest_assistant_message` is rejected on the public broker socket. +3. Promotion consumes the evidence-backed one-time token before volatile + `VERIFIED` becomes visible. A token or receipt cannot be replayed against a + later cycle. +4. `revoke_lease`, generation replacement, broker restart, or monotonic TTL + expiry removes consequential-tool authority. TTL is positive and capped at + 300 seconds. + +`launch-runtime.py` is the register-before-`exec` choke point for Claude and +Pi. It performs the activation-capability version check, registers the anchor, +creates the private generation file, exports the broker identity to descendants, +and only then executes the requested runtime. Registration, capability, +generation-file, broker-reply, or `exec` failure denies launch. Claude's raw +`--dangerously-skip-permissions` flag is owned by this wrapper; callers request +only its semantic dangerous mode. + +Claude's all-tools `PreToolUse` hook and Pi's `tool_call` handler submit the +runtime-reported tool name to the gate. The gate does not inspect a shell string +to decide that one command is safe. Missing identity, malformed input or reply, +timeout, broker unavailability, unsafe generation state, and denial all fail +closed. + +### Lifecycle and recovery boundaries + +Claude and Pi lifecycle observers use the same authenticated session and +broker state machine. Compaction revocation and same-PID replacement behavior +are summarized below: + +| Runtime event | Current source-backed behavior | +| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Claude `PreCompact` | Revoke before compaction; a failed hook blocks the lifecycle transition. | +| Claude `SessionStart` matcher `compact` | Revoke again after compacted context starts. | +| Claude `SessionStart` matcher `resume\|clear` | Bump the private generation, then revoke the replacement incarnation. | +| Pi `session_before_compact` | Revoke before compaction; failure cancels the transition. | +| Pi `session_compact` then first `context` | Run one independent post-compaction revoke; a failed post-observer locally blocks later tools until retry. | +| Pi `session_start` reason `reload`, `new`, `resume`, or `fork` | Bump the private generation before revoking and reusing the replacement session. | +| Fired observer with unavailable broker | Advance the private generation as a local fence and return failure; do not continue consequential work. | + +The [source-resident context-refresh skill](../../../packages/mosaic/framework/skills/mosaic-context-refresh/SKILL.md) +is a thin adapter over `recover-context.py`. Recovery begins with a validated +normative-fragment construction, broker-side revoke-first transition, and a +fresh `PENDING_DELIVERY` challenge. The trusted observer records only finalized +Claude Stop-hook or Pi `message_end` content. Completion supplies no receipt or +challenge argument; the broker obtains its current challenge, checks the exact +observed entry, consumes evidence, and promotes last. Recovery does not accept +normal-path receipt text as replayable authority. + +A receipt is only a delivery/liveness prerequisite. It is not a safety, +obedience, comprehension, or residency proof. Absent, malformed, +prefix-truncated, and observably mutated terminal receipts do not promote. A +tail-preserving middle drop is explicitly not receipt-detectable and remains a +server-side/T-C residual. + +### Named residual + +If both compaction observers are missed while a lease remains unexpired, +consequential tools are **ALLOWED** inside the bounded residual stale window. +The mutator gate makes no within-window action-count or timing claim. After the +monotonic TTL expires, the next consequential tool is **DENIED**. This is +separate from a fired observer that cannot reach the broker, which fails closed +through lifecycle cancellation, a generation fence, and/or the Pi local latch. +Protected-branch controls and required review/CI remain the irreducible +server-side backstop for the T-C total-hook-miss boundary. + +## Held future procedure — non-operative + +There is no current command authority for the following live procedures. The +sequence below records source-backed intent for a separately approved +activation/recovery work package; it must not be copied into an operator shell. + +### Startup and restart outline (held) + +1. An activation owner would first materialize the exact framework unit, + wrapper, and co-located source copies, then verify the resolved parent, + socket, state, observer, and generation paths and their no-symlink/private + posture. +2. The approved supervisor would start the daemon only after confirming that + the exact socket path is not owned by another process. `READY` from the + daemon and a live Unix socket would be health evidence; a unit file alone + would not be healthy. +3. A broker crash would require preserving the state file, identifying the + socket owner, and making an explicit restart decision. Restart intentionally + clears volatile `VERIFIED` leases; it is not a way to restore authority. +4. A leftover socket would be handled only after the owning service is + confirmed stopped and the exact path is deliberately reviewed. This page + supplies no deletion, `systemctl`, enablement, or start command. + +The source's `daemon.py` argument parser and the user unit show how a future +activation is wired, but neither source file grants this page authority to +invoke that wiring. `applyBrokerSupervisor` is materialization only; the +supervisor source explicitly leaves enable/start as a separate held step. + +### Recovery outline (held) + +The future adapter flow is: runtime supplies validated construction and current +non-negative epochs; broker performs `begin_recovery` revoke-first and returns +one fresh receipt; the adapter delivers that exact receipt; the authenticated +observer records the finalized assistant entry; and the adapter requests +completion without presenting receipt text or a challenge. Any absent, +malformed, stale, duplicated, or mismatched observation leaves the session +`UNVERIFIED`; retry starts a new recovery cycle. + +Claude's adapter is restricted to the exact literal recovery argv shape checked +by the gate. Pi uses only the registered `mosaic_context_recover` tool; Pi +`bash` and all other tools remain gated. Do not manually invoke the revoker to +restore authority, send assistant text through the public broker request, or +reuse a normal-path receipt. The [context-refresh skill](../../../packages/mosaic/framework/skills/mosaic-context-refresh/SKILL.md) +and its [runtime boundary tests](../../../packages/mosaic/src/lease-broker/recovery_runtime_unittest.py) +are references for future adapter qualification, not an active operator route. + +### Lease, mutator, and incident handling outline (held) + +- There is no supported operator CLI or HTTP endpoint for sending raw + `begin_verification`, `promote_lease`, `revoke_lease`, or `authorize_tool` + requests. Do not hand-craft JSON frames, mint tokens, replay receipts, or + treat a successful read-only authorization as a promotion. +- After a runtime exits, its generation file may be removed only after an + approved check establishes that no process for that broker-minted session + remains. Retain stale files during incident analysis; they carry no lease + authority by themselves. +- Corrupt, oversized, symlinked, or non-regular state must be preserved for + review and not overwritten in place. Establishing new state is an explicit + operational decision that invalidates prior sessions and tokens; no recovery + command is supplied here. +- Directory `0700` and socket/state `0600` are same-principal hardening only. + They do not prevent the same UID from unlinking and replacing a socket. A + stronger deployment needs an external protected proxy, ACL, or service + boundary that preserves the peer identity required by `SO_PEERCRED` and + ancestry checks. No such deployment procedure is current here. + +## Related source and tests + +- [Broker daemon](../../../packages/mosaic/framework/tools/lease-broker/daemon.py) +- [Register-and-exec launcher](../../../packages/mosaic/framework/tools/lease-broker/launch-runtime.py) +- [Mutator gate](../../../packages/mosaic/framework/tools/lease-broker/mutator-gate.py) +- [Recovery command](../../../packages/mosaic/framework/tools/lease-broker/recover-context.py) +- [Lease broker acceptance tests](../../../packages/mosaic/src/lease-broker/lease-broker.acceptance.spec.ts) +- [Mutator gate acceptance tests](../../../packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts) +- [Pi lifecycle tests](../../../packages/mosaic/src/mutator-gate/pi-compaction-lifecycle.spec.ts) + +This migration changes documentation placement and verified wording only. It does +not start or change the broker, a runtime, systemd, PostgreSQL, or any other +service. diff --git a/docs/_old_structure/guides/lease-broker-operations.md b/docs/_old_structure/guides/lease-broker-operations.md deleted file mode 100644 index ee7caf57..00000000 --- a/docs/_old_structure/guides/lease-broker-operations.md +++ /dev/null @@ -1,44 +0,0 @@ -# Lease broker operations - -Place the socket and state file in a dedicated directory with mode `0700`. Start the packaged daemon with: - -```bash -python3 "$MOSAIC_HOME/tools/lease-broker/daemon.py" \ - --socket /run/user/1000/mosaic-lease/broker.sock \ - --state /run/user/1000/mosaic-lease/state.json -``` - -The broker refuses an existing parent directory whose mode is not exactly `0700`, an existing state file not at `0600`, corrupt/incompatible state, or an already-existing socket path. After bind it sets the socket to `0600`. It never silently unlinks a pre-existing socket. On normal termination it unlinks only the socket inode it created, so it does not remove a replacement path. - -Before launching Claude, Claudex, or Pi, export the socket path; `mosaic` then runs the runtime through the packaged register-and-exec wrapper: - -```bash -export MOSAIC_LEASE_BROKER_SOCKET=/run/user/1000/mosaic-lease/broker.sock -mosaic claude # or: mosaic claudex, mosaic yolo claudex, mosaic pi -``` - -The wrapper obtains a broker-minted session ID, creates a private `generation-.state` file beside the socket, and `exec`s the runtime without changing its PID/starttime anchor. The all-tools Claude `PreToolUse` hook and Pi `tool_call` handler inherit that identity and read the current generation from the file. Claudex retains its isolated proxy environment and config directory; Mosaic merges the mandatory all-tools and compaction-lifecycle hooks into that isolated `settings.json` before invoking the same wrapper. PRDY init/update, QA remediation, coord, orchestrator, and fleet launchers also converge on this boundary. Broker registration failure, unsafe isolated settings, unsafe generation state, or missing identity denies launch/tool execution fail-closed; broker timeout/unavailability and malformed replies also block tools. - -Claude `PreCompact` and `SessionStart(compact)` hooks and Pi pre-/post-compaction handlers invoke `revoke-lease.py`. Pi `session_start` reload/new/resume/fork and Claude resume/clear advance the locked generation before revocation, so a replacement session inherits no lease even when PID/starttime stay unchanged. Do not invoke the revoker manually as a way to restore authority; it only removes authority. If a lifecycle hook reports failure, stop consequential work and repair broker/generation-state availability before re-verification. - -Run the permanent launch inventory locally with: - -```bash -python3 packages/mosaic/framework/tools/lease-broker/check-runtime-launches.py --root . -``` - -The same check runs in the Mosaic package test suite and therefore in root CI. Any direct Claude/Pi binary launch must be replaced with `launch-runtime.py`, `execLeaseGatedRuntime`, or the gated `mosaic` runtime command; do not add static allowlist exceptions. - -Clients must complete the request boundary before waiting for a reply. After sending the single JSON object and its terminating newline, the client **MUST half-close the socket's write side** (`shutdown(SHUT_WR)` in POSIX clients; `socket.end()` in Node) and only then await the response. Merely calling `write()` and waiting is invalid: the broker waits for EOF to enforce the exact-one-frame contract and fails closed at its one-second deadline. Do not replace `end()` with `write()` in client helpers. A delayed second frame remains malformed and is rejected. - -`mosaic_context_recover` is the only unverified mutator class. Its durable `mosaic-context-refresh` skill is a thin wrapper over `tools/lease-broker/recover-context.py`: `begin` has the broker rebuild the validated `B_payload`/`H_payload`, revoke first, and mint a new `PENDING_DELIVERY` receipt challenge; `complete` accepts neither receipt text nor a challenge argument. Claude maps only the exact direct recovery executable/validated arguments to this exempt tool identity; ordinary `Bash` remains gated. Pi exposes only the `mosaic_context_recover` custom tool; ordinary `bash` and all other tools remain gated. A normal-path receipt cannot be replayed through recovery because each retry begins a distinct recovery cycle and recovery completion cannot receive caller-presented evidence. - -Production daemon startup creates a separate private observer socket unless a test-only `--test-observer-file` fixture is selected. Claude's Stop hook sends its exact latest assistant entry and Pi's `message_end` handler sends only finalized assistant content to that authenticated transport; the broker public socket never accepts message text. This is byte-build and private out-of-process harness wiring only: do not activate it against a live daemon, live socket, systemd service, tmux session, or model-output stream outside the controlled integration procedure. - -Receipt honesty is load-bearing: absent, malformed, prefix-truncated, and observable adapter-mutated terminal receipts do not promote. A tail-only case is non-promoting only where the concrete terminal payload is malformed or observably incomplete. A tail-preserving middle drop is **not receipt-detectable**; it is the disclosed T-C injection-contract residual deferred to WI-7 server-side evidence. The receipt remains a T-A delivery/liveness prerequisite, never a safety, obedience, or residency proof. The framework skill is source-resident and bridge-projected on install/upgrade; do not hand-create a live runtime symlink. - -After a runtime exits, its `generation-.state` file may be removed only after verifying that no process for that broker-minted session remains; stale files carry no lease authority but should be retained during incident analysis. After a broker crash, preserve the protected state file and restart only after verifying that no broker owns the socket. Restart intentionally clears all volatile VERIFIED leases. A leftover socket requires an operator to verify the owning service is stopped and remove that exact socket deliberately. Corrupt, oversized, symlinked, or non-regular state fails closed; do not overwrite it. Preserve it for incident review and establish new state only through an explicit operational decision, which invalidates prior sessions and tokens. - -## Security posture - -Directory `0700` plus socket/state `0600` is built-in same-principal hardening only: it excludes other UIDs but does **not** stop the same UID from unlinking and counterfeiting the socket. It therefore does not close T-C same-UID replacement. WI-1 does not provide a distinct-principal boundary. A stronger distinct-principal deployment requires an external protected proxy, ACL, or service boundary that clients cannot unlink or rebind and that preserves the authenticated client identity required by the broker's `SO_PEERCRED` and ancestry checks. Server-side branch protection remains the irreducible backstop.