deploy-gap: #828 lease-broker enforcement hooks shipped without a running broker daemon — bricks fleet agents (GATE_UNAVAILABLE / Stop-block) #869

New Issue

jason.woltje · 2026-07-22T17:04:49Z

jason.woltje commented

2026-07-22 17:04:49 +00:00

Severity: P0 — bricked every Claude/Pi agent on the fleet host (web1) after the 2026-07-22 04:22 framework upgrade. Currently held up only by two host-local fail-open bypasses (details below). cc @jason.woltje (owner of #828 / #860).

Summary

The #828 lease-broker enforcement hooks were deployed (wired into ~/.claude/settings.json) without the broker runtime that makes them functional. The mutator-gate.py (PreToolUse) and receipt-observer-client.py (Stop) hooks correctly fail closed when the broker identity env / socket is absent — but on web1 nothing starts the broker daemon and nothing injects the lease env, so every tool call is denied GATE_UNAVAILABLE and every turn-end is blocked. This is a deploy-ordering gap, not a code bug in the gate.

What happened

The 04:22 upgrade shipped the complete lease-broker toolkit under ~/.config/mosaic/tools/lease-broker/ — daemon.py, launch-runtime.py, mutator-gate.py, receipt-observer-client.py, recover-context.py, lease_generation.py, revoke-lease.py, … — and wired the enforcement hooks into the runtime settings. But the activation half of #828 (a supervisor that runs daemon.py, plus launch always chaining through launch-runtime.py to inject MOSAIC_LEASE_*) is not present on the host.

Evidence (host `web1`, user `hermes`)

Hook wiring is live in ~/.claude/settings.json:
- PreToolUse matcher .* → python3 …/lease-broker/mutator-gate.py --runtime claude --recovery-command …/recover-context.py
- Stop → python3 …/lease-broker/receipt-observer-client.py --runtime claude --latest-entry
No broker running / no socket: pgrep -af daemon.py → none; /run/user/<uid>/mosaic-lease/ → does not exist.
No supervisor unit: no *lease* / *broker* unit under ~/.config/systemd/user/.
Launch does not inject the lease env: the running claude process carries no MOSAIC_LEASE_SESSION_ID / MOSAIC_LEASE_BROKER_SOCKET / MOSAIC_RUNTIME_GENERATION. The fleet runner launches via mosaic yolo claude; the installed CLI is 0.0.48 (dist/cli.js bundled). In current framework source execLeaseGatedRuntime() chains claude through launch-runtime.py and sets MOSAIC_LEASE_BROKER_SOCKET, but launch-runtime.py register_anchor() requires a listening broker — which is not deployed. Whether 0.0.48 gates or execs claude directly, the outcome on this host is identical: enforcement hooks fire with no broker behind them.
Symptom in the field: mos-claude.service agent reported every Bash/Read/Edit/Write denied GATE_UNAVAILABLE; the Stop hook re-blocked deterministically (the "hook blocked N times" backstop symptom).

Why fail-open is NOT the fix

The gate's fail-closed-on-absent-identity behavior is intentional and test-locked — weakening it would (correctly) break these tests:

test_gate_entrypoint_denies_when_identity_environment_is_absent
test_environment_generation_and_request_failures_deny

So the fix is infrastructure activation, not relaxing the gate.

Root cause

Enforcement was rolled out one step ahead of the infrastructure it depends on: the hooks assume a running broker + launch-runtime.py-injected lease env that #828's activation work was supposed to guarantee on every host, but that activation never landed on the fleet host.

Proposed fix direction (owner's call)

Ship + enable a supervisor for daemon.py (systemd --user unit, or fold broker startup into the launch path) so the broker socket exists before any gated runtime starts.
Guarantee launch chaining: ensure the fleet launch path (incl. mosaic yolo claude on the deployed CLI) always goes through launch-runtime.py so MOSAIC_LEASE_* is injected — or make the hook a no-op when the host has explicitly not activated the broker (a deliberate, documented opt-out, not a silent fail-open).
Install-ordering guard: the upgrade should not wire enforcement hooks on a host where the broker supervisor is absent (fail the install/health-check loudly instead of shipping a dead gate).

Current mitigation (temporary — to be reverted once the above lands)

To restore the fleet, two host-local fail-opens are in place on web1 (originals preserved at *.pre-bypass-bak):

mutator-gate.py: returns 0 (allow) when MOSAIC_LEASE_BROKER_SOCKET is unset.
receipt-observer-client.py: replaced with a drain-stdin / exit-0 stub.

These are acknowledged as incorrect long-term (they defeat #828 enforcement) and must be reverted the moment the broker is properly activated. Do not run a mosaic framework upgrade on web1 until this is resolved — an upgrade reverts these bypasses and re-bricks the fleet.

References

#828 (WI-1 authenticated lease broker) — code merged, runtime activation gap.
#860 (mutator lease-gate flake) — same subsystem.

**Severity: P0 — bricked every Claude/Pi agent on the fleet host (`web1`) after the 2026-07-22 04:22 framework upgrade.** Currently held up only by two host-local fail-open bypasses (details below). cc @jason.woltje (owner of #828 / #860). ## Summary The #828 lease-broker **enforcement hooks** were deployed (wired into `~/.claude/settings.json`) **without the broker runtime that makes them functional**. The `mutator-gate.py` (PreToolUse) and `receipt-observer-client.py` (Stop) hooks correctly **fail closed** when the broker identity env / socket is absent — but on `web1` nothing starts the broker daemon and nothing injects the lease env, so **every tool call is denied `GATE_UNAVAILABLE` and every turn-end is blocked**. This is a deploy-ordering gap, not a code bug in the gate. ## What happened The 04:22 upgrade shipped the **complete** lease-broker toolkit under `~/.config/mosaic/tools/lease-broker/` — `daemon.py`, `launch-runtime.py`, `mutator-gate.py`, `receipt-observer-client.py`, `recover-context.py`, `lease_generation.py`, `revoke-lease.py`, … — **and** wired the enforcement hooks into the runtime settings. But the *activation* half of #828 (a supervisor that runs `daemon.py`, plus launch always chaining through `launch-runtime.py` to inject `MOSAIC_LEASE_*`) is not present on the host. ## Evidence (host `web1`, user `hermes`) - **Hook wiring is live** in `~/.claude/settings.json`: - `PreToolUse` matcher `.*` → `python3 …/lease-broker/mutator-gate.py --runtime claude --recovery-command …/recover-context.py` - `Stop` → `python3 …/lease-broker/receipt-observer-client.py --runtime claude --latest-entry` - **No broker running / no socket:** `pgrep -af daemon.py` → none; `/run/user/<uid>/mosaic-lease/` → does not exist. - **No supervisor unit:** no `*lease*` / `*broker*` unit under `~/.config/systemd/user/`. - **Launch does not inject the lease env:** the running `claude` process carries no `MOSAIC_LEASE_SESSION_ID` / `MOSAIC_LEASE_BROKER_SOCKET` / `MOSAIC_RUNTIME_GENERATION`. The fleet runner launches via `mosaic yolo claude`; the installed CLI is **0.0.48** (`dist/cli.js` bundled). In current framework source `execLeaseGatedRuntime()` chains `claude` through `launch-runtime.py` and sets `MOSAIC_LEASE_BROKER_SOCKET`, but `launch-runtime.py` `register_anchor()` requires a **listening broker** — which is not deployed. Whether 0.0.48 gates or execs claude directly, the outcome on this host is identical: enforcement hooks fire with **no broker behind them**. - **Symptom in the field:** `mos-claude.service` agent reported every `Bash`/`Read`/`Edit`/`Write` denied `GATE_UNAVAILABLE`; the Stop hook re-blocked deterministically (the "hook blocked N times" backstop symptom). ## Why fail-open is NOT the fix The gate's fail-closed-on-absent-identity behavior is **intentional and test-locked** — weakening it would (correctly) break these tests: - `test_gate_entrypoint_denies_when_identity_environment_is_absent` - `test_environment_generation_and_request_failures_deny` So the fix is **infrastructure activation**, not relaxing the gate. ## Root cause Enforcement was rolled out one step ahead of the infrastructure it depends on: the hooks assume a running broker + `launch-runtime.py`-injected lease env that #828's activation work was supposed to guarantee on every host, but that activation never landed on the fleet host. ## Proposed fix direction (owner's call) 1. **Ship + enable a supervisor** for `daemon.py` (systemd `--user` unit, or fold broker startup into the launch path) so the broker socket exists before any gated runtime starts. 2. **Guarantee launch chaining:** ensure the fleet launch path (incl. `mosaic yolo claude` on the deployed CLI) always goes through `launch-runtime.py` so `MOSAIC_LEASE_*` is injected — or make the hook a no-op when the host has explicitly not activated the broker (a deliberate, documented opt-out, **not** a silent fail-open). 3. **Install-ordering guard:** the upgrade should not wire enforcement hooks on a host where the broker supervisor is absent (fail the install/health-check loudly instead of shipping a dead gate). ## Current mitigation (temporary — to be reverted once the above lands) To restore the fleet, two host-local fail-opens are in place on `web1` (originals preserved at `*.pre-bypass-bak`): - `mutator-gate.py`: returns 0 (allow) when `MOSAIC_LEASE_BROKER_SOCKET` is unset. - `receipt-observer-client.py`: replaced with a drain-stdin / exit-0 stub. These are acknowledged as incorrect long-term (they defeat #828 enforcement) and must be reverted the moment the broker is properly activated. **Do not run a `mosaic` framework upgrade on `web1` until this is resolved** — an upgrade reverts these bypasses and re-bricks the fleet. ## References - #828 (WI-1 authenticated lease broker) — code merged, runtime activation gap. - #860 (mutator lease-gate flake) — same subsystem.

jason.woltje referenced this issue from a commit

2026-07-22 18:39:11 +00:00

feat(mosaic): leaseEnforcementActivatable() capability probe (#869 C1)

jason.woltje referenced this issue from a commit

2026-07-22 18:39:12 +00:00

chore(orchestrator): fix pre-existing Prettier drift in README.md