mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

mutator-gate: mosaic_context_recover is gated by the failure it exists to recover from #1212

New Issue
Open
opened 2026-08-13 16:36:03 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-13 16:36:03 +00:00
Copy Link
Copy Source

Summary

mosaic_context_recover is documented as "the sole ungated mutator" but is gated by the same
mechanism it exists to recover from. When the lease environment is absent, the recovery path
fails with the identical GATE_UNAVAILABLE as every other mutator, leaving a seat with no way
back.

Evidence

Observed on a live pi seat (coder2, mosaic yolo pi, CLI 0.0.48):

BLOCKED: Mosaic mutator gate denied this tool (GATE_UNAVAILABLE).
Blocked: Mosaic mutator gate is unavailable, denying every tool, including required recovery:
  mosaic_context_recover phase=begin requestEpoch=1
  → GATE_UNAVAILABLE
No repository or provider action occurred.

The seat reports the contradiction itself: recovery is "required" and is denied.

Mechanism

tools/lease-broker/mutator-gate.py main():

socket_value = source_environment["MOSAIC_LEASE_BROKER_SOCKET"]
session_id   = source_environment["MOSAIC_LEASE_SESSION_ID"]
generation   = resolve_generation(source_environment)
reply = request(Path(socket_value), {...})
except (KeyError, ValueError, OSError, json.JSONDecodeError):
    return deny("GATE_UNAVAILABLE")

The recovery command is resolved by recovery_invocation_name(request_input, arguments.recovery_command)
inside the same try, and the environment reads happen before any check of whether the tool
being authorized is the recovery tool. So a missing lease environment denies recovery on the same
line it denies everything else.

Reproduced by measurement, not inference: /proc/<pid>/environ for both the mosaic yolo pi
parent and the pi child shows all three lease variables absent (checked for presence only).
KeyError is therefore guaranteed, not incidental.

The runtime side advertises the exemption that does not exist —
runtime/pi/mosaic-extension.ts:382:

'Run the constrained broker-backed context recovery flow. This is the sole ungated mutator.'

Why this is worth fixing separately from the skew that exposed it

The triggering condition here is #828-class version skew (an enforcement half from main against
a CLI with no __lease-capability), and upgrading the CLI clears the symptom. But the defect is
independent of the cause: a recovery mechanism that shares a dependency with the failure it
recovers from is not a recovery mechanism. Any future condition that empties the lease
environment — a launcher regression, a unit file losing Environment= lines, a socket path
changing — reproduces this with a different origin story, and the seat is again stranded with the
message "including required recovery" and no recovery.

Fail-closed is right for mutators. It is wrong for the one path whose entire purpose is to run
when the gate cannot answer.

Acceptance criteria

  1. Identify the recovery invocation before reading any lease environment, and authorize it on
    a path that does not require the broker to be reachable.
  2. With MOSAIC_LEASE_BROKER_SOCKET / MOSAIC_LEASE_SESSION_ID unset, mosaic_context_recover
    succeeds while every other mutating tool still denies GATE_UNAVAILABLE. Both halves asserted
    in the same test — a test that only proves recovery works does not prove the gate still holds.
  3. The recovery path stays constrained: it must not become a general bypass. If recovery can be
    named by the caller, the name must be validated against the configured recovery command, not
    accepted from the tool request.
  4. mosaic-extension.ts's "sole ungated mutator" description becomes true, or is corrected.

Not doing

No workaround applied. Hand-populating the lease environment on live seats, or routing work to
ungated runtimes to dodge this, would be bypassing the gate rather than fixing it.

## Summary `mosaic_context_recover` is documented as "the sole ungated mutator" but is gated by the same mechanism it exists to recover from. When the lease environment is absent, the recovery path fails with the identical `GATE_UNAVAILABLE` as every other mutator, leaving a seat with no way back. ## Evidence Observed on a live pi seat (`coder2`, `mosaic yolo pi`, CLI 0.0.48): ```text BLOCKED: Mosaic mutator gate denied this tool (GATE_UNAVAILABLE). Blocked: Mosaic mutator gate is unavailable, denying every tool, including required recovery: mosaic_context_recover phase=begin requestEpoch=1 → GATE_UNAVAILABLE No repository or provider action occurred. ``` The seat reports the contradiction itself: recovery is "required" and is denied. ## Mechanism `tools/lease-broker/mutator-gate.py` main(): ```python socket_value = source_environment["MOSAIC_LEASE_BROKER_SOCKET"] session_id = source_environment["MOSAIC_LEASE_SESSION_ID"] generation = resolve_generation(source_environment) reply = request(Path(socket_value), {...}) except (KeyError, ValueError, OSError, json.JSONDecodeError): return deny("GATE_UNAVAILABLE") ``` The recovery command is resolved by `recovery_invocation_name(request_input, arguments.recovery_command)` *inside* the same `try`, and the environment reads happen before any check of whether the tool being authorized is the recovery tool. So a missing lease environment denies recovery on the same line it denies everything else. Reproduced by measurement, not inference: `/proc/<pid>/environ` for both the `mosaic yolo pi` parent and the `pi` child shows **all three lease variables absent** (checked for presence only). `KeyError` is therefore guaranteed, not incidental. The runtime side advertises the exemption that does not exist — `runtime/pi/mosaic-extension.ts:382`: > 'Run the constrained broker-backed context recovery flow. This is the sole ungated mutator.' ## Why this is worth fixing separately from the skew that exposed it The triggering condition here is #828-class version skew (an enforcement half from `main` against a CLI with no `__lease-capability`), and upgrading the CLI clears the symptom. But the defect is independent of the cause: **a recovery mechanism that shares a dependency with the failure it recovers from is not a recovery mechanism.** Any future condition that empties the lease environment — a launcher regression, a unit file losing `Environment=` lines, a socket path changing — reproduces this with a different origin story, and the seat is again stranded with the message "including required recovery" and no recovery. Fail-closed is right for mutators. It is wrong for the one path whose entire purpose is to run when the gate cannot answer. ## Acceptance criteria 1. Identify the recovery invocation **before** reading any lease environment, and authorize it on a path that does not require the broker to be reachable. 2. With `MOSAIC_LEASE_BROKER_SOCKET` / `MOSAIC_LEASE_SESSION_ID` unset, `mosaic_context_recover` succeeds while every other mutating tool still denies `GATE_UNAVAILABLE`. Both halves asserted in the same test — a test that only proves recovery works does not prove the gate still holds. 3. The recovery path stays constrained: it must not become a general bypass. If recovery can be named by the caller, the name must be validated against the configured recovery command, not accepted from the tool request. 4. `mosaic-extension.ts`'s "sole ungated mutator" description becomes true, or is corrected. ## Not doing No workaround applied. Hand-populating the lease environment on live seats, or routing work to ungated runtimes to dodge this, would be bypassing the gate rather than fixing it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1212