mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

Lease broker: framework upgrade alone wedges every tool — enforcement and activation ship on different channels with no preflight #1203

New Issue
Open
opened 2026-08-13 15:45:50 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-13 15:45:50 +00:00
Copy Link
Copy Source

Summary

Upgrading the installed framework to main — on its own, with no other change — took every tool
on web1 offline. Bash, Read, and all MCP/tool calls returned:

BLOCKED: Mosaic mutator gate denied this tool (GATE_UNAVAILABLE)

The session had zero tool surface: it could not read a file, message the fleet, or self-recover.
Recovery required a human editing ~/.claude/settings.json by hand.

Root cause

The lease broker ships as two halves on two different release channels:

Half Files Channel
ENFORCEMENT tools/lease-broker/{launch-runtime,mutator-gate,revoke-lease}.py framework reseed (install.sh)
ACTIVATION execLeaseGatedRuntime() in packages/mosaic/src/commands/launch.ts npm @mosaicstack/mosaic

tools/lease-broker/activation_version_gate.py asserts enforcement may only run when activation
advertises:

EXPECTED_ACTIVATION_CAPABILITY: Final[ActivationCapability] = {
    "name": "lease-runtime-activation",
    "version": 1,
}

On this host the installed CLI is @mosaicstack/[email protected], and:

$ mosaic __lease-capability
error: unknown command

So enforcement is installed and activation is not. mutator-gate.py:157 then does
return deny("GATE_UNAVAILABLE") for every tool call, because it is wired PreToolUse
with matcher *.

Why this is a defect and not operator error

  1. install.sh upgrades one half of a two-half contract and never checks the other half.
    There is no preflight that reads the installed CLI version, and no --dry-run to discover it.
  2. The failure message names nothing actionable. GATE_UNAVAILABLE does not say which
    component is missing, which version is required, which version is present, or what command
    fixes it. Diagnosis required reading three Python files.
  3. The failure is total and self-sealing. Because the gate matches *, the very tools needed
    to diagnose or revert it are denied. A fail-closed gate that also denies Read cannot be
    recovered from inside the session.
  4. The blast radius is the whole host. The gate is wired in a global settings file, so it hits
    every session on the machine, not just the one that ran the install.
  5. Starting the broker does not help. Verified by isolation: mosaic-lease-broker.service
    active, log READY, both sockets present — all three hooks still denied. The missing daemon
    is a symptom; the skew is the cause.
  6. A second landmine survives the first recovery. revoke-lease.py is wired on PreCompact
    and SessionStart(resume|clear) and fails closed rc=2. Removing only the PreToolUse hook
    restores tools, and then the next compaction re-wedges the session.

Reproduction

  1. Host with @mosaicstack/[email protected] installed globally and lease hooks wired.
  2. git archive origin/main packages/mosaic/framework | tar -x -C /var/tmp/fw --strip-components=3
  3. /var/tmp/fw/install.sh (keep mode, rc=0, reports success).
  4. Any tool call in any Claude session on that host → GATE_UNAVAILABLE.

Proposed acceptance criteria

  • install.sh preflights the activation half: read the installed CLI's advertised capability
    before writing enforcement hooks. On mismatch, fail loudly and do not wire the hooks —
    naming both versions and the exact remediation command.
  • install.sh gains a --dry-run that reports what it would change, including any
    newly-wired hook and any cross-component version requirement.
  • GATE_UNAVAILABLE is replaced with a message stating: required capability + version,
    observed capability + version, and the one command that reconciles them.
  • The gate never denies read-only tools it does not need to mediate, so a wedged session can
    still diagnose itself.
  • A wedge caused by skew is recoverable without hand-editing a settings file — a documented
    break-glass that does not require the denied tools.

Evidence

  • Post-install drift: in-sync=305, stale=0, not-installed=0 (the refresh itself was correct and
    is not in question).
  • All five QA/guard hooks were left intact; no quality gate was weakened during recovery.
  • Five lease-broker hook entries are currently unwired on this host as the only available
    mitigation, so lease enforcement is off until the CLI can be upgraded. That upgrade replaces
    the launcher for ~40 live fleet seats and is being scheduled deliberately rather than reflexively.
## Summary Upgrading the installed framework to `main` — on its own, with no other change — took **every tool on web1 offline**. Bash, Read, and all MCP/tool calls returned: ``` BLOCKED: Mosaic mutator gate denied this tool (GATE_UNAVAILABLE) ``` The session had zero tool surface: it could not read a file, message the fleet, or self-recover. Recovery required a human editing `~/.claude/settings.json` by hand. ## Root cause The lease broker ships as two halves **on two different release channels**: | Half | Files | Channel | |---|---|---| | ENFORCEMENT | `tools/lease-broker/{launch-runtime,mutator-gate,revoke-lease}.py` | framework reseed (`install.sh`) | | ACTIVATION | `execLeaseGatedRuntime()` in `packages/mosaic/src/commands/launch.ts` | npm `@mosaicstack/mosaic` | `tools/lease-broker/activation_version_gate.py` asserts enforcement may only run when activation advertises: ```python EXPECTED_ACTIVATION_CAPABILITY: Final[ActivationCapability] = { "name": "lease-runtime-activation", "version": 1, } ``` On this host the installed CLI is `@mosaicstack/[email protected]`, and: ``` $ mosaic __lease-capability error: unknown command ``` So enforcement is installed and activation is not. `mutator-gate.py:157` then does `return deny("GATE_UNAVAILABLE")` for **every** tool call, because it is wired `PreToolUse` with matcher `*`. ## Why this is a defect and not operator error 1. **`install.sh` upgrades one half of a two-half contract and never checks the other half.** There is no preflight that reads the installed CLI version, and no `--dry-run` to discover it. 2. **The failure message names nothing actionable.** `GATE_UNAVAILABLE` does not say which component is missing, which version is required, which version is present, or what command fixes it. Diagnosis required reading three Python files. 3. **The failure is total and self-sealing.** Because the gate matches `*`, the very tools needed to diagnose or revert it are denied. A fail-closed gate that also denies `Read` cannot be recovered from inside the session. 4. **The blast radius is the whole host.** The gate is wired in a global settings file, so it hits every session on the machine, not just the one that ran the install. 5. **Starting the broker does not help.** Verified by isolation: `mosaic-lease-broker.service` `active`, log `READY`, both sockets present — all three hooks still denied. The missing daemon is a symptom; the skew is the cause. 6. **A second landmine survives the first recovery.** `revoke-lease.py` is wired on `PreCompact` and `SessionStart(resume|clear)` and fails closed `rc=2`. Removing only the `PreToolUse` hook restores tools, and then the next compaction re-wedges the session. ## Reproduction 1. Host with `@mosaicstack/[email protected]` installed globally and lease hooks wired. 2. `git archive origin/main packages/mosaic/framework | tar -x -C /var/tmp/fw --strip-components=3` 3. `/var/tmp/fw/install.sh` (keep mode, `rc=0`, reports success). 4. Any tool call in any Claude session on that host → `GATE_UNAVAILABLE`. ## Proposed acceptance criteria - [ ] `install.sh` preflights the activation half: read the installed CLI's advertised capability **before** writing enforcement hooks. On mismatch, fail loudly and **do not wire the hooks** — naming both versions and the exact remediation command. - [ ] `install.sh` gains a `--dry-run` that reports what it would change, including any newly-wired hook and any cross-component version requirement. - [ ] `GATE_UNAVAILABLE` is replaced with a message stating: required capability + version, observed capability + version, and the one command that reconciles them. - [ ] The gate never denies read-only tools it does not need to mediate, so a wedged session can still diagnose itself. - [ ] A wedge caused by skew is recoverable without hand-editing a settings file — a documented break-glass that does not require the denied tools. ## Evidence - Post-install drift: `in-sync=305, stale=0, not-installed=0` (the refresh itself was correct and is not in question). - All five QA/guard hooks were left intact; no quality gate was weakened during recovery. - Five lease-broker hook entries are currently unwired on this host as the only available mitigation, so lease enforcement is **off** until the CLI can be upgraded. That upgrade replaces the launcher for ~40 live fleet seats and is being scheduled deliberately rather than reflexively.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1203