mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

deploy-gap: #828 lease-broker enforcement hooks shipped without a running broker daemon — bricks fleet agents (GATE_UNAVAILABLE / Stop-block) #869

New Issue
Open
opened 2026-07-22 17:04:49 +00:00 by jason.woltje · 1 comment
jason.woltje commented 2026-07-22 17:04:49 +00:00
Owner
Copy Link
Copy Source

Severity: P0 — bricked every Claude/Pi agent on the fleet host (web1) after the 2026-07-22 04:22 framework upgrade. Currently held up only by two host-local fail-open bypasses (details below). cc @jason.woltje (owner of #828 / #860).

Summary

The #828 lease-broker enforcement hooks were deployed (wired into ~/.claude/settings.json) without the broker runtime that makes them functional. The mutator-gate.py (PreToolUse) and receipt-observer-client.py (Stop) hooks correctly fail closed when the broker identity env / socket is absent — but on web1 nothing starts the broker daemon and nothing injects the lease env, so every tool call is denied GATE_UNAVAILABLE and every turn-end is blocked. This is a deploy-ordering gap, not a code bug in the gate.

What happened

The 04:22 upgrade shipped the complete lease-broker toolkit under ~/.config/mosaic/tools/lease-broker/ — daemon.py, launch-runtime.py, mutator-gate.py, receipt-observer-client.py, recover-context.py, lease_generation.py, revoke-lease.py, … — and wired the enforcement hooks into the runtime settings. But the activation half of #828 (a supervisor that runs daemon.py, plus launch always chaining through launch-runtime.py to inject MOSAIC_LEASE_*) is not present on the host.

Evidence (host web1, user hermes)

  • Hook wiring is live in ~/.claude/settings.json:
    • PreToolUse matcher .* → python3 …/lease-broker/mutator-gate.py --runtime claude --recovery-command …/recover-context.py
    • Stop → python3 …/lease-broker/receipt-observer-client.py --runtime claude --latest-entry
  • No broker running / no socket: pgrep -af daemon.py → none; /run/user/<uid>/mosaic-lease/ → does not exist.
  • No supervisor unit: no *lease* / *broker* unit under ~/.config/systemd/user/.
  • Launch does not inject the lease env: the running claude process carries no MOSAIC_LEASE_SESSION_ID / MOSAIC_LEASE_BROKER_SOCKET / MOSAIC_RUNTIME_GENERATION. The fleet runner launches via mosaic yolo claude; the installed CLI is 0.0.48 (dist/cli.js bundled). In current framework source execLeaseGatedRuntime() chains claude through launch-runtime.py and sets MOSAIC_LEASE_BROKER_SOCKET, but launch-runtime.py register_anchor() requires a listening broker — which is not deployed. Whether 0.0.48 gates or execs claude directly, the outcome on this host is identical: enforcement hooks fire with no broker behind them.
  • Symptom in the field: mos-claude.service agent reported every Bash/Read/Edit/Write denied GATE_UNAVAILABLE; the Stop hook re-blocked deterministically (the "hook blocked N times" backstop symptom).

Why fail-open is NOT the fix

The gate's fail-closed-on-absent-identity behavior is intentional and test-locked — weakening it would (correctly) break these tests:

  • test_gate_entrypoint_denies_when_identity_environment_is_absent
  • test_environment_generation_and_request_failures_deny

So the fix is infrastructure activation, not relaxing the gate.

Root cause

Enforcement was rolled out one step ahead of the infrastructure it depends on: the hooks assume a running broker + launch-runtime.py-injected lease env that #828's activation work was supposed to guarantee on every host, but that activation never landed on the fleet host.

Proposed fix direction (owner's call)

  1. Ship + enable a supervisor for daemon.py (systemd --user unit, or fold broker startup into the launch path) so the broker socket exists before any gated runtime starts.
  2. Guarantee launch chaining: ensure the fleet launch path (incl. mosaic yolo claude on the deployed CLI) always goes through launch-runtime.py so MOSAIC_LEASE_* is injected — or make the hook a no-op when the host has explicitly not activated the broker (a deliberate, documented opt-out, not a silent fail-open).
  3. Install-ordering guard: the upgrade should not wire enforcement hooks on a host where the broker supervisor is absent (fail the install/health-check loudly instead of shipping a dead gate).

Current mitigation (temporary — to be reverted once the above lands)

To restore the fleet, two host-local fail-opens are in place on web1 (originals preserved at *.pre-bypass-bak):

  • mutator-gate.py: returns 0 (allow) when MOSAIC_LEASE_BROKER_SOCKET is unset.
  • receipt-observer-client.py: replaced with a drain-stdin / exit-0 stub.

These are acknowledged as incorrect long-term (they defeat #828 enforcement) and must be reverted the moment the broker is properly activated. Do not run a mosaic framework upgrade on web1 until this is resolved — an upgrade reverts these bypasses and re-bricks the fleet.

References

  • #828 (WI-1 authenticated lease broker) — code merged, runtime activation gap.
  • #860 (mutator lease-gate flake) — same subsystem.
**Severity: P0 — bricked every Claude/Pi agent on the fleet host (`web1`) after the 2026-07-22 04:22 framework upgrade.** Currently held up only by two host-local fail-open bypasses (details below). cc @jason.woltje (owner of #828 / #860). ## Summary The #828 lease-broker **enforcement hooks** were deployed (wired into `~/.claude/settings.json`) **without the broker runtime that makes them functional**. The `mutator-gate.py` (PreToolUse) and `receipt-observer-client.py` (Stop) hooks correctly **fail closed** when the broker identity env / socket is absent — but on `web1` nothing starts the broker daemon and nothing injects the lease env, so **every tool call is denied `GATE_UNAVAILABLE` and every turn-end is blocked**. This is a deploy-ordering gap, not a code bug in the gate. ## What happened The 04:22 upgrade shipped the **complete** lease-broker toolkit under `~/.config/mosaic/tools/lease-broker/` — `daemon.py`, `launch-runtime.py`, `mutator-gate.py`, `receipt-observer-client.py`, `recover-context.py`, `lease_generation.py`, `revoke-lease.py`, … — **and** wired the enforcement hooks into the runtime settings. But the *activation* half of #828 (a supervisor that runs `daemon.py`, plus launch always chaining through `launch-runtime.py` to inject `MOSAIC_LEASE_*`) is not present on the host. ## Evidence (host `web1`, user `hermes`) - **Hook wiring is live** in `~/.claude/settings.json`: - `PreToolUse` matcher `.*` → `python3 …/lease-broker/mutator-gate.py --runtime claude --recovery-command …/recover-context.py` - `Stop` → `python3 …/lease-broker/receipt-observer-client.py --runtime claude --latest-entry` - **No broker running / no socket:** `pgrep -af daemon.py` → none; `/run/user/<uid>/mosaic-lease/` → does not exist. - **No supervisor unit:** no `*lease*` / `*broker*` unit under `~/.config/systemd/user/`. - **Launch does not inject the lease env:** the running `claude` process carries no `MOSAIC_LEASE_SESSION_ID` / `MOSAIC_LEASE_BROKER_SOCKET` / `MOSAIC_RUNTIME_GENERATION`. The fleet runner launches via `mosaic yolo claude`; the installed CLI is **0.0.48** (`dist/cli.js` bundled). In current framework source `execLeaseGatedRuntime()` chains `claude` through `launch-runtime.py` and sets `MOSAIC_LEASE_BROKER_SOCKET`, but `launch-runtime.py` `register_anchor()` requires a **listening broker** — which is not deployed. Whether 0.0.48 gates or execs claude directly, the outcome on this host is identical: enforcement hooks fire with **no broker behind them**. - **Symptom in the field:** `mos-claude.service` agent reported every `Bash`/`Read`/`Edit`/`Write` denied `GATE_UNAVAILABLE`; the Stop hook re-blocked deterministically (the "hook blocked N times" backstop symptom). ## Why fail-open is NOT the fix The gate's fail-closed-on-absent-identity behavior is **intentional and test-locked** — weakening it would (correctly) break these tests: - `test_gate_entrypoint_denies_when_identity_environment_is_absent` - `test_environment_generation_and_request_failures_deny` So the fix is **infrastructure activation**, not relaxing the gate. ## Root cause Enforcement was rolled out one step ahead of the infrastructure it depends on: the hooks assume a running broker + `launch-runtime.py`-injected lease env that #828's activation work was supposed to guarantee on every host, but that activation never landed on the fleet host. ## Proposed fix direction (owner's call) 1. **Ship + enable a supervisor** for `daemon.py` (systemd `--user` unit, or fold broker startup into the launch path) so the broker socket exists before any gated runtime starts. 2. **Guarantee launch chaining:** ensure the fleet launch path (incl. `mosaic yolo claude` on the deployed CLI) always goes through `launch-runtime.py` so `MOSAIC_LEASE_*` is injected — or make the hook a no-op when the host has explicitly not activated the broker (a deliberate, documented opt-out, **not** a silent fail-open). 3. **Install-ordering guard:** the upgrade should not wire enforcement hooks on a host where the broker supervisor is absent (fail the install/health-check loudly instead of shipping a dead gate). ## Current mitigation (temporary — to be reverted once the above lands) To restore the fleet, two host-local fail-opens are in place on `web1` (originals preserved at `*.pre-bypass-bak`): - `mutator-gate.py`: returns 0 (allow) when `MOSAIC_LEASE_BROKER_SOCKET` is unset. - `receipt-observer-client.py`: replaced with a drain-stdin / exit-0 stub. These are acknowledged as incorrect long-term (they defeat #828 enforcement) and must be reverted the moment the broker is properly activated. **Do not run a `mosaic` framework upgrade on `web1` until this is resolved** — an upgrade reverts these bypasses and re-bricks the fleet. ## References - #828 (WI-1 authenticated lease broker) — code merged, runtime activation gap. - #860 (mutator lease-gate flake) — same subsystem.
Ghost commented 2026-08-13 17:34:49 +00:00
Copy Link
Copy Source

PR #1207 merged into next as f82307c4dc0f8588d7374e600bc8edf4942515ae after independent exact-head review of 8272dfb7debdb6178b0d271168343ac34a805353, terminal-green Woodpecker pipeline 2408, and the mandatory pinned-head queue/merge gate.

Delivered here: both lease-capability probe budgets are 10 seconds; Python executable resolution honors the supplied environment PATH and fails closed for absent/empty PATH; hermetic and timeout tests plus mutation controls now bind those behaviors.

This is a bounded remediation within #869. The issue stays open: its broader activation/deployment gap—broker supervision, launch chaining, install ordering, and removal of any temporary host bypasses—requires separate end-to-end evidence and is not closed by the timeout patch.

PR #1207 merged into `next` as `f82307c4dc0f8588d7374e600bc8edf4942515ae` after independent exact-head review of `8272dfb7debdb6178b0d271168343ac34a805353`, terminal-green Woodpecker pipeline 2408, and the mandatory pinned-head queue/merge gate. Delivered here: both lease-capability probe budgets are 10 seconds; Python executable resolution honors the supplied environment PATH and fails closed for absent/empty PATH; hermetic and timeout tests plus mutation controls now bind those behaviors. This is a bounded remediation within #869. The issue stays **open**: its broader activation/deployment gap—broker supervision, launch chaining, install ordering, and removal of any temporary host bypasses—requires separate end-to-end evidence and is not closed by the timeout patch.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#869