mosaicstack/stack
1
0
Fork 1
Code Issues 207 Pull Requests 14 Actions Packages Projects Releases 15 Wiki Activity

revoke -> re-promote structurally broken on rolled-over seat: promote-begin reads frozen MOSAIC_RUNTIME_GENERATION env while daemon/mutator read the live generation file #1135

New Issue
Open
opened 2026-08-09 20:09:52 +00:00 by mos-dt-0 · 0 comments
mos-dt-0 commented 2026-08-09 20:09:52 +00:00
Collaborator
Copy Link
Copy Source

Found by scooby during the W-P1c live E2E gate (A6, 2026-08-09) on a real gated seat (fomo-lin). Root-caused with /proc evidence.

Symptom

After an in-seat rollover (/clear -> SessionStart:clear -> revoke-lease.py --bump-generation), mosaic promote <seat> permanently fails with OBSERVATION_REJECTED. The revoke half works (mutator correctly DENIES post-revoke); the re-promote half can never succeed on the same seat process.

Root cause

  • promote-begin.py:371 records the runtime observation with runtime_generation = int(env["MOSAIC_RUNTIME_GENERATION"]) — the seat's LAUNCH-time env var, frozen at its launch value (line 363's --begin promoter inherits the same stale env).
  • The rollover hook bumps the live generation FILE + daemon session (1 -> 2).
  • daemon.py:427 record_runtime_observation rejects when lease.runtime_generation (2) != request generation (1) -> OBSERVATION_UNAVAILABLE -> OBSERVATION_REJECTED.
  • Proof of drift on the live seat: /proc/<anchor_pid>/environ has MOSAIC_RUNTIME_GENERATION=1 while the generation file and daemon state are both 2.

Net: mutator-gate reads the live file (correct), promote-begin reads the frozen env (stale) — the two components disagree on "current generation", so a rolled-over seat is unpromotable for the rest of its process lifetime.

Candidate fix (scooby)

Source the current generation from MOSAIC_LEASE_GENERATION_FILE (what daemon and mutator-gate already use) instead of the frozen env var, threaded through both --begin and the observation record.

Scope note

Likely pre-existing (the old model-driven flow read the same env var), but W-P1c's mechanical promote-begin.py now owns this path. Per the A6 verdict this is tracked as a fast-follow, NOT a merge gate for #1109 (core promotion thesis — mechanical promote, model-out-of-loop, injection inertness — validated on a real seat). scooby will re-verify the revoke->re-promote cycle end-to-end when the fix lands.

Refs: PR #1109 (W-P1c), A6 E2E report in the fleet dispatch tracker (jarvis-brain docs/scratchpads/lease-remediation/dispatch/TASK-TRACKER.md, item A6-F2).

Found by scooby during the W-P1c live E2E gate (A6, 2026-08-09) on a real gated seat (fomo-lin). Root-caused with /proc evidence. ## Symptom After an in-seat rollover (`/clear` -> SessionStart:clear -> `revoke-lease.py --bump-generation`), `mosaic promote <seat>` permanently fails with `OBSERVATION_REJECTED`. The revoke half works (mutator correctly DENIES post-revoke); the re-promote half can never succeed on the same seat process. ## Root cause - `promote-begin.py:371` records the runtime observation with `runtime_generation = int(env["MOSAIC_RUNTIME_GENERATION"])` — the seat's LAUNCH-time env var, frozen at its launch value (line 363's `--begin` promoter inherits the same stale env). - The rollover hook bumps the live generation FILE + daemon session (1 -> 2). - `daemon.py:427` `record_runtime_observation` rejects when `lease.runtime_generation (2) != request generation (1)` -> `OBSERVATION_UNAVAILABLE` -> `OBSERVATION_REJECTED`. - Proof of drift on the live seat: `/proc/<anchor_pid>/environ` has `MOSAIC_RUNTIME_GENERATION=1` while the generation file and daemon state are both 2. Net: mutator-gate reads the live file (correct), promote-begin reads the frozen env (stale) — the two components disagree on "current generation", so a rolled-over seat is unpromotable for the rest of its process lifetime. ## Candidate fix (scooby) Source the current generation from `MOSAIC_LEASE_GENERATION_FILE` (what daemon and mutator-gate already use) instead of the frozen env var, threaded through both `--begin` and the observation record. ## Scope note Likely pre-existing (the old model-driven flow read the same env var), but W-P1c's mechanical `promote-begin.py` now owns this path. Per the A6 verdict this is tracked as a fast-follow, NOT a merge gate for #1109 (core promotion thesis — mechanical promote, model-out-of-loop, injection inertness — validated on a real seat). scooby will re-verify the revoke->re-promote cycle end-to-end when the fix lands. Refs: PR #1109 (W-P1c), A6 E2E report in the fleet dispatch tracker (jarvis-brain `docs/scratchpads/lease-remediation/dispatch/TASK-TRACKER.md`, item A6-F2).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
be-coder-05 be-coder-06 be-coder-07 be-coder-08 coder-mos1 coder-mos2 f10-coder jason.woltje (Jason Woltje) merge-gate mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1)) rev-security-02 scooby (Scooby) velma
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1135