293 lines
18 KiB
Markdown
293 lines
18 KiB
Markdown
---
|
|
title: Lease-broker operations
|
|
type: runbook
|
|
audience: developer
|
|
status: current
|
|
source_of_truth: false
|
|
---
|
|
|
|
# Lease-broker operations
|
|
|
|
> **Status:** Current for repository inspection and test verification. Live broker
|
|
> startup, recovery, lease mutation, service management, and cleanup remain
|
|
> **held/non-operative**.
|
|
>
|
|
> **Operational authority:** This page authorizes only the non-mutating
|
|
> inspection and test commands in [Safe static inspection](#safe-static-inspection).
|
|
> It does not authorize starting a daemon or systemd unit, connecting to a live
|
|
> socket, invoking recovery, promoting or revoking a lease, executing a
|
|
> consequential runtime tool, or changing a database.
|
|
|
|
The lease broker is a Linux-only internal process boundary. Its executable
|
|
behavior is authoritative in the [shipped broker daemon](../../../packages/mosaic/framework/tools/lease-broker/daemon.py)
|
|
and tests, not in this page. The current architecture references are:
|
|
|
|
- [Broker protocol](../architecture/lease-broker-protocol.md) — framing,
|
|
kernel identity, ancestry, generations, persistence, and state transitions.
|
|
- [Lease-broker security](../architecture/lease-broker-security.md) — trust
|
|
boundaries, filesystem hardening, observer behavior, and residuals.
|
|
- [Whole mutator-class gate](../architecture/mutator-class-gate.md) — default
|
|
deny, launch choke points, and broker-owned promotion order.
|
|
- [Compaction revocation](../architecture/compaction-revocation.md) — Claude
|
|
and Pi lifecycle observers, generation fencing, and the bounded residual.
|
|
|
|
## Safe static inspection
|
|
|
|
These are the only operative commands documented here. They inspect checked-in
|
|
files or run isolated tests; they do not start a user service, activate a
|
|
runtime, connect to PostgreSQL, or mutate repository/product state.
|
|
|
|
### Source and launch inventory
|
|
|
|
From the repository root, inspect the current implementation and its permanent
|
|
runtime-launch inventory:
|
|
|
|
```bash
|
|
find packages/mosaic/framework/tools/lease-broker packages/mosaic/src/lease-broker packages/mosaic/src/mutator-gate -maxdepth 1 -type f -print | sort
|
|
python3 packages/mosaic/framework/tools/lease-broker/check-runtime-launches.py --root . --json
|
|
```
|
|
|
|
The launch inventory is a static completeness guard. A clean result means the
|
|
checked-in production launch sites are classified by the guard; it does not
|
|
prove that a broker, runtime, or service is running.
|
|
|
|
### Safe unit and static contract tests
|
|
|
|
The focused standard-library tests can be run directly:
|
|
|
|
```bash
|
|
python3 packages/mosaic/src/lease-broker/daemon_deadline_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/normative_fragments_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/receipt_challenge_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/context_recovery_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/state_store_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/framework_skill_portability_unittest.py
|
|
python3 packages/mosaic/src/mutator-gate/runtime_tools_unittest.py
|
|
python3 packages/mosaic/src/mutator-gate/runtime_launch_guard_unittest.py
|
|
python3 packages/mosaic/src/mutator-gate/version_coupling_unittest.py
|
|
```
|
|
|
|
`recovery_runtime_unittest.py` and `recovery_b1_adversarial_unittest.py` are
|
|
also safe when run as tests: they use private temporary daemons, sockets, and
|
|
fixtures, never the installed user service or a model stream. They are not
|
|
operator recovery instructions:
|
|
|
|
```bash
|
|
python3 packages/mosaic/src/lease-broker/recovery_runtime_unittest.py
|
|
python3 packages/mosaic/src/lease-broker/recovery_b1_adversarial_unittest.py
|
|
```
|
|
|
|
The [lease-broker Vitest acceptance suite](../../../packages/mosaic/src/lease-broker/lease-broker.acceptance.spec.ts)
|
|
and [mutator-gate Vitest acceptance suite](../../../packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts)
|
|
provide additional private-fixture coverage. Test-created child processes and
|
|
Unix sockets are disposable test fixtures, not live-service authority.
|
|
|
|
## Current implementation facts
|
|
|
|
### Protected paths and persistence
|
|
|
|
The broker and supervisor source establish these invariants:
|
|
|
|
| Object | Current contract |
|
|
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Parent directory | The parent containing the socket and state must already exist and have exactly mode `0700`. |
|
|
| Broker socket | The daemon refuses an existing path or symlink, binds a new Unix socket, sets it to `0600`, and removes only the inode it created during normal shutdown. |
|
|
| State file | A present state file must be a regular, non-symlink file protected as mode `0600`, no larger than 4 MiB, and valid state-version-1 JSON. Corrupt, incompatible, oversized, or unsafe state refuses startup. |
|
|
| State writes | Changes are serialized through a mode-`0600` temporary file, complete-write loop, `fsync`, atomic replace, and parent-directory `fsync`. Post-replace durability uncertainty poisons the store and terminates service processing. |
|
|
| Volatile authority | `VERIFIED` leases are not restored as live authority after broker restart. Persisted session identity and valid pending-token state are separate from volatile lease state. |
|
|
| Runtime generation | `launch-runtime.py` creates `generation-<broker-session>.state` beside the socket. It is an owner-only, locked, monotonic generation file; unsafe, non-regular, oversized, or non-private state fails closed. |
|
|
|
|
The resolved socket path is, in order: explicit
|
|
`MOSAIC_LEASE_BROKER_SOCKET`, `$XDG_RUNTIME_DIR/mosaic-lease/broker.sock`, or
|
|
`/run/user/<uid>/mosaic-lease/broker.sock`. The state file is `state.json` next
|
|
to that resolved socket. The production observer transport is a separate
|
|
`receipt-observer.sock` by default; `--test-observer-file` is a private test
|
|
fixture option, not a production deployment path.
|
|
|
|
The supervisor implementation can materialize a user unit, wrapper, and
|
|
co-located daemon sources under caller-supplied paths, but
|
|
[`applyBrokerSupervisor`](../../../packages/mosaic/src/lease-broker/broker-supervisor.ts)
|
|
never runs `systemctl`, starts `daemon.py`, or enables the unit. The checked-in
|
|
unit and [`start-lease-broker.sh`](../../../packages/mosaic/framework/tools/lease-broker/start-lease-broker.sh)
|
|
are therefore implementation inputs, not live activation authority for this
|
|
page.
|
|
|
|
### Protocol and identity boundary
|
|
|
|
The protocol accepts one UTF-8 JSON object followed by one newline, capped at
|
|
64 KiB. The client must half-close its write side after the newline and before
|
|
waiting for the response (`shutdown(SHUT_WR)` for POSIX clients or
|
|
`socket.end()` for Node). Unterminated, multiple, delayed-second, malformed,
|
|
oversized, or deadline-exceeded requests fail closed. This is an internal Unix
|
|
socket protocol, not an HTTP/OpenAPI endpoint.
|
|
|
|
The broker obtains `(pid, uid, gid)` from kernel `SO_PEERCRED`, binds the
|
|
session to the anchor's `/proc/<pid>/stat` starttime, and revalidates every
|
|
walked ancestor's starttime. A caller cannot choose `session_id`; a sibling or
|
|
unrelated process cannot authenticate with another process's session. A higher
|
|
runtime generation replaces the prior incarnation and revokes its tokens and
|
|
lease authority; a lower generation is stale.
|
|
|
|
### Lease and tool authorization
|
|
|
|
The broker is the sole lease writer. The effective default-deny policy is:
|
|
|
|
- Claude read-only classes: `Read`, `Grep`, `Glob`, `Ls`, and `Find`.
|
|
- Pi read-only classes: `read`, `grep`, `find`, and `ls`.
|
|
- Both runtimes expose the fixed `mosaic_context_recover` identity as the
|
|
constrained recovery exception.
|
|
- Every other built-in, unknown, custom, MCP, shell, edit, write, deployment,
|
|
provider, or filesystem mutator is consequential and is denied while the
|
|
session is not `VERIFIED`.
|
|
|
|
The normal broker transition is revoke-first and promote-last:
|
|
|
|
1. `begin_verification` authenticates the broker-minted session and current
|
|
generation, revokes existing authority and pending tokens, validates the
|
|
exact source construction and binding, and enters a pending state.
|
|
2. The receipt challenge and binding are broker-generated. A trusted observer
|
|
must provide the exact current-cycle assistant entry; caller-supplied
|
|
`latest_assistant_message` is rejected on the public broker socket.
|
|
3. Promotion consumes the evidence-backed one-time token before volatile
|
|
`VERIFIED` becomes visible. A token or receipt cannot be replayed against a
|
|
later cycle.
|
|
4. `revoke_lease`, generation replacement, broker restart, or monotonic TTL
|
|
expiry removes consequential-tool authority. TTL is positive and capped at
|
|
300 seconds.
|
|
|
|
`launch-runtime.py` is the register-before-`exec` choke point for Claude and
|
|
Pi. It performs the activation-capability version check, registers the anchor,
|
|
creates the private generation file, exports the broker identity to descendants,
|
|
and only then executes the requested runtime. Registration, capability,
|
|
generation-file, broker-reply, or `exec` failure denies launch. Claude's raw
|
|
`--dangerously-skip-permissions` flag is owned by this wrapper; callers request
|
|
only its semantic dangerous mode.
|
|
|
|
Claude's all-tools `PreToolUse` hook and Pi's `tool_call` handler submit the
|
|
runtime-reported tool name to the gate. The gate does not inspect a shell string
|
|
to decide that one command is safe. Missing identity, malformed input or reply,
|
|
timeout, broker unavailability, unsafe generation state, and denial all fail
|
|
closed.
|
|
|
|
### Lifecycle and recovery boundaries
|
|
|
|
Claude and Pi lifecycle observers use the same authenticated session and
|
|
broker state machine. Compaction revocation and same-PID replacement behavior
|
|
are summarized below:
|
|
|
|
| Runtime event | Current source-backed behavior |
|
|
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
| Claude `PreCompact` | Revoke before compaction; a failed hook blocks the lifecycle transition. |
|
|
| Claude `SessionStart` matcher `compact` | Revoke again after compacted context starts. |
|
|
| Claude `SessionStart` matcher `resume\|clear` | Bump the private generation, then revoke the replacement incarnation. |
|
|
| Pi `session_before_compact` | Revoke before compaction; failure cancels the transition. |
|
|
| Pi `session_compact` then first `context` | Run one independent post-compaction revoke; a failed post-observer locally blocks later tools until retry. |
|
|
| Pi `session_start` reason `reload`, `new`, `resume`, or `fork` | Bump the private generation before revoking and reusing the replacement session. |
|
|
| Fired observer with unavailable broker | Advance the private generation as a local fence and return failure; do not continue consequential work. |
|
|
|
|
The [source-resident context-refresh skill](../../../packages/mosaic/framework/skills/mosaic-context-refresh/SKILL.md)
|
|
is a thin adapter over `recover-context.py`. Recovery begins with a validated
|
|
normative-fragment construction, broker-side revoke-first transition, and a
|
|
fresh `PENDING_DELIVERY` challenge. The trusted observer records only finalized
|
|
Claude Stop-hook or Pi `message_end` content. Completion supplies no receipt or
|
|
challenge argument; the broker obtains its current challenge, checks the exact
|
|
observed entry, consumes evidence, and promotes last. Recovery does not accept
|
|
normal-path receipt text as replayable authority.
|
|
|
|
A receipt is only a delivery/liveness prerequisite. It is not a safety,
|
|
obedience, comprehension, or residency proof. Absent, malformed,
|
|
prefix-truncated, and observably mutated terminal receipts do not promote. A
|
|
tail-preserving middle drop is explicitly not receipt-detectable and remains a
|
|
server-side/T-C residual.
|
|
|
|
### Named residual
|
|
|
|
If both compaction observers are missed while a lease remains unexpired,
|
|
consequential tools are **ALLOWED** inside the bounded residual stale window.
|
|
The mutator gate makes no within-window action-count or timing claim. After the
|
|
monotonic TTL expires, the next consequential tool is **DENIED**. This is
|
|
separate from a fired observer that cannot reach the broker, which fails closed
|
|
through lifecycle cancellation, a generation fence, and/or the Pi local latch.
|
|
Protected-branch controls and required review/CI remain the irreducible
|
|
server-side backstop for the T-C total-hook-miss boundary.
|
|
|
|
## Held future procedure — non-operative
|
|
|
|
There is no current command authority for the following live procedures. The
|
|
sequence below records source-backed intent for a separately approved
|
|
activation/recovery work package; it must not be copied into an operator shell.
|
|
|
|
### Startup and restart outline (held)
|
|
|
|
1. An activation owner would first materialize the exact framework unit,
|
|
wrapper, and co-located source copies, then verify the resolved parent,
|
|
socket, state, observer, and generation paths and their no-symlink/private
|
|
posture.
|
|
2. The approved supervisor would start the daemon only after confirming that
|
|
the exact socket path is not owned by another process. `READY` from the
|
|
daemon and a live Unix socket would be health evidence; a unit file alone
|
|
would not be healthy.
|
|
3. A broker crash would require preserving the state file, identifying the
|
|
socket owner, and making an explicit restart decision. Restart intentionally
|
|
clears volatile `VERIFIED` leases; it is not a way to restore authority.
|
|
4. A leftover socket would be handled only after the owning service is
|
|
confirmed stopped and the exact path is deliberately reviewed. This page
|
|
supplies no deletion, `systemctl`, enablement, or start command.
|
|
|
|
The source's `daemon.py` argument parser and the user unit show how a future
|
|
activation is wired, but neither source file grants this page authority to
|
|
invoke that wiring. `applyBrokerSupervisor` is materialization only; the
|
|
supervisor source explicitly leaves enable/start as a separate held step.
|
|
|
|
### Recovery outline (held)
|
|
|
|
The future adapter flow is: runtime supplies validated construction and current
|
|
non-negative epochs; broker performs `begin_recovery` revoke-first and returns
|
|
one fresh receipt; the adapter delivers that exact receipt; the authenticated
|
|
observer records the finalized assistant entry; and the adapter requests
|
|
completion without presenting receipt text or a challenge. Any absent,
|
|
malformed, stale, duplicated, or mismatched observation leaves the session
|
|
`UNVERIFIED`; retry starts a new recovery cycle.
|
|
|
|
Claude's adapter is restricted to the exact literal recovery argv shape checked
|
|
by the gate. Pi uses only the registered `mosaic_context_recover` tool; Pi
|
|
`bash` and all other tools remain gated. Do not manually invoke the revoker to
|
|
restore authority, send assistant text through the public broker request, or
|
|
reuse a normal-path receipt. The [context-refresh skill](../../../packages/mosaic/framework/skills/mosaic-context-refresh/SKILL.md)
|
|
and its [runtime boundary tests](../../../packages/mosaic/src/lease-broker/recovery_runtime_unittest.py)
|
|
are references for future adapter qualification, not an active operator route.
|
|
|
|
### Lease, mutator, and incident handling outline (held)
|
|
|
|
- There is no supported operator CLI or HTTP endpoint for sending raw
|
|
`begin_verification`, `promote_lease`, `revoke_lease`, or `authorize_tool`
|
|
requests. Do not hand-craft JSON frames, mint tokens, replay receipts, or
|
|
treat a successful read-only authorization as a promotion.
|
|
- After a runtime exits, its generation file may be removed only after an
|
|
approved check establishes that no process for that broker-minted session
|
|
remains. Retain stale files during incident analysis; they carry no lease
|
|
authority by themselves.
|
|
- Corrupt, oversized, symlinked, or non-regular state must be preserved for
|
|
review and not overwritten in place. Establishing new state is an explicit
|
|
operational decision that invalidates prior sessions and tokens; no recovery
|
|
command is supplied here.
|
|
- Directory `0700` and socket/state `0600` are same-principal hardening only.
|
|
They do not prevent the same UID from unlinking and replacing a socket. A
|
|
stronger deployment needs an external protected proxy, ACL, or service
|
|
boundary that preserves the peer identity required by `SO_PEERCRED` and
|
|
ancestry checks. No such deployment procedure is current here.
|
|
|
|
## Related source and tests
|
|
|
|
- [Broker daemon](../../../packages/mosaic/framework/tools/lease-broker/daemon.py)
|
|
- [Register-and-exec launcher](../../../packages/mosaic/framework/tools/lease-broker/launch-runtime.py)
|
|
- [Mutator gate](../../../packages/mosaic/framework/tools/lease-broker/mutator-gate.py)
|
|
- [Recovery command](../../../packages/mosaic/framework/tools/lease-broker/recover-context.py)
|
|
- [Lease broker acceptance tests](../../../packages/mosaic/src/lease-broker/lease-broker.acceptance.spec.ts)
|
|
- [Mutator gate acceptance tests](../../../packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts)
|
|
- [Pi lifecycle tests](../../../packages/mosaic/src/mutator-gate/pi-compaction-lifecycle.spec.ts)
|
|
|
|
This migration changes documentation placement and verified wording only. It does
|
|
not start or change the broker, a runtime, systemd, PostgreSQL, or any other
|
|
service.
|