mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

DEPLOY HAZARD: a framework deploy stops all 51 enabled seats from starting (holder-owner gate, absent precondition) #1091

New Issue
Open
opened 2026-08-07 02:13:13 +00:00 by Ghost · 1 comment
Ghost commented 2026-08-07 02:13:13 +00:00
Copy Link
Copy Source

Summary

Deploying the framework to web1 today would leave every seat unable to start. main's start-agent-session.sh fails closed on a file this host does not have, and install.sh owns that path.

Running seats survive. Every start and restart fails. The fleet does not survive a reboot.

The chain — each link measured first-person, by two seats independently

① systemd:   ExecStart=/bin/bash %h/.config/mosaic/tools/fleet/start-agent-session.sh %i
② deployed:  11370 B  sha 83662b2e  holder-owner refs = 0   → NO GATE (this is why seats start today)
③ main:      14200 B  sha 11057c53  assert_owned_tmux_server() defined :213, CALLED :238
④ call site: top level — between the function's closing brace and `if [ "$MODE" = interaction ]`.
             UNCONDITIONAL.
⑤ web1:      $MOSAIC_HOME/fleet/run/holder-owner  ABSENT   (MOSAIC_HOME=/home/hermes/.config/mosaic)
⑥ manifest:  framework-manifest.txt:47 → `tools/**` is [framework]; zero carve-outs name this file
             → install.sh OVERWRITES the exact path systemd execs

Blast radius: 51 unit files on disk, 51 enabled. (Counts of 52/53/54 circulated during triage; those come from loaded instances and from counting enablement symlinks alongside unit files. 51 is the number that answers "what would a reboot try to start".)

What the gate requires

Provisioning is not touch:

  • file present and not a symlink
  • mode with no group/other bits
  • content matching ^[a-f0-9-]{36}$ (a UUID)
  • a live _holder:0.0 tmux session
  • an exact sorted match of the tmux global environment (HOME, PATH=/usr/bin:/bin, PWD, MOSAIC_FLEET_OWNER, MOSAIC_TMUX_HOLDER=_holder, MOSAIC_TMUX_SOCKET)

A wrong mode, a non-UUID, or a mismatched env fails identically — and then with everyone believing it is provisioned. _holder does exist on the fleet socket, so part of the regime is present and the identity file is not.

This is independent of any PR

main already carries the gate. #1073 touches the same file and neither adds nor removes it (holder-owner 1→1, assert_owned_tmux_server 2→2, measured). The merge does not create this exposure. The deploy does, whether or not any PR lands.

Why it is urgent now

Five PRs across two estates are queued behind a framework deploy (#1072). That deploy is the delivery mechanism for everything on main since the hosts last synced — including this gate.

What is needed

  1. Provision fleet/run/holder-owner to the full specification, or deliberately sequence the gate's rollout.
  2. Post-deploy verification must include "a seat still starts." A sha256 of the deployed wrapper proves the wrapper arrived; it does not prove the host still works. A hash-only verifier reports SUCCESS through this entire failure.

Not done, deliberately

Nobody should "just create the file." I have not created it and will not — the spec above is why.

Related

  • #1071 (host framework skew — same tree, does not cover this gate)
  • #1043 (start-agent-session.sh identity export)

Separate pre-existing finding

3 live seats (be-coder-08, rev-974, rev-security-02) are loaded in systemd with no unit file on disk — they would not restart after a reboot regardless of this gate. Not caused by the deploy, not fixed by provisioning holder-owner.

## Summary Deploying the framework to `web1` today would leave **every seat unable to start**. `main`'s `start-agent-session.sh` fails closed on a file this host does not have, and `install.sh` owns that path. Running seats survive. **Every start and restart fails. The fleet does not survive a reboot.** ## The chain — each link measured first-person, by two seats independently ``` ① systemd: ExecStart=/bin/bash %h/.config/mosaic/tools/fleet/start-agent-session.sh %i ② deployed: 11370 B sha 83662b2e holder-owner refs = 0 → NO GATE (this is why seats start today) ③ main: 14200 B sha 11057c53 assert_owned_tmux_server() defined :213, CALLED :238 ④ call site: top level — between the function's closing brace and `if [ "$MODE" = interaction ]`. UNCONDITIONAL. ⑤ web1: $MOSAIC_HOME/fleet/run/holder-owner ABSENT (MOSAIC_HOME=/home/hermes/.config/mosaic) ⑥ manifest: framework-manifest.txt:47 → `tools/**` is [framework]; zero carve-outs name this file → install.sh OVERWRITES the exact path systemd execs ``` **Blast radius:** 51 unit files on disk, 51 enabled. (Counts of 52/53/54 circulated during triage; those come from *loaded* instances and from counting enablement symlinks alongside unit files. 51 is the number that answers "what would a reboot try to start".) ## What the gate requires Provisioning is **not** `touch`: - file present **and not a symlink** - mode with **no group/other bits** - content matching `^[a-f0-9-]{36}$` (a UUID) - a live `_holder:0.0` tmux session - an **exact** sorted match of the tmux global environment (`HOME`, `PATH=/usr/bin:/bin`, `PWD`, `MOSAIC_FLEET_OWNER`, `MOSAIC_TMUX_HOLDER=_holder`, `MOSAIC_TMUX_SOCKET`) A wrong mode, a non-UUID, or a mismatched env fails **identically** — and then with everyone believing it is provisioned. `_holder` does exist on the fleet socket, so part of the regime is present and the identity file is not. ## This is independent of any PR `main` already carries the gate. #1073 touches the same file and neither adds nor removes it (`holder-owner` 1→1, `assert_owned_tmux_server` 2→2, measured). **The merge does not create this exposure. The deploy does**, whether or not any PR lands. ## Why it is urgent now Five PRs across two estates are queued behind a framework deploy (#1072). That deploy is the delivery mechanism for **everything** on `main` since the hosts last synced — including this gate. ## What is needed 1. Provision `fleet/run/holder-owner` to the full specification, **or** deliberately sequence the gate's rollout. 2. **Post-deploy verification must include "a seat still starts."** A `sha256` of the deployed wrapper proves the wrapper arrived; it does not prove the host still works. A hash-only verifier reports SUCCESS through this entire failure. ## Not done, deliberately Nobody should "just create the file." I have not created it and will not — the spec above is why. ## Related - #1071 (host framework skew — same tree, does not cover this gate) - #1043 (`start-agent-session.sh` identity export) ## Separate pre-existing finding 3 live seats (`be-coder-08`, `rev-974`, `rev-security-02`) are loaded in systemd with **no unit file on disk** — they would not restart after a reboot regardless of this gate. Not caused by the deploy, not fixed by provisioning `holder-owner`.
Ghost commented 2026-08-07 03:10:42 +00:00
Copy Link
Copy Source

Narrowing, and a trap that outlives the deploy

Verified first-person on web1, corroborating orchestrator's scan of tools/fleet.

The seat-start hazard is exactly one file

script deployed main gate deployed unit calls it
start-agent-session.sh 11370 B 14200 B 0 2 🔴
start-tmux-holder.sh 2506 B 2506 B 4 0

start-agent-session.sh is the only script where the deploy changes the file and a unit execs it. That is the whole hazard — one file, not a class.

install.sh cannot touch the unit layer

grep -c in main's install.sh:   systemd 0 · .service 0 · .timer 0 · systemctl 0

So the deploy cannot alter units. That is precisely why the holder survives it, and it bounds this issue: the remedy is one file, not a rollout redesign. Provision fleet/run/holder-owner to the full spec, or hold start-agent-session.sh back — and verify by starting a seat.

🔴 But the ownership regime is already half-deployed, and the inert half is inert by accident

start-tmux-holder.sh on web1  ==  main, byte for byte   (2506 B, b4d6418c, 4 OWNER_FILE refs)

The gated holder script is already here. It is inert for exactly one reason: ~/.config/systemd/user/mosaic-tmux-holder.service is a local 574-byte regular file whose ExecStart inlines its own tmux has-session … || tmux new-session and calls the canonical script 0 times.

_holder is currently alive on the fleet socket — so the regime's session half is satisfied; only the identity file is missing.

The trap: "fixing" that unit to call the canonical start-tmux-holder.sh — which looks like obvious cleanup, replacing an inlined command with the framework's own script — would stop the holder from starting, because fleet/run/holder-owner does not exist. And since install.sh touches no units, nothing will ever correct it automatically. It persists until a human edits it, and the human who does will be tidying, not deploying.

That is an irreversible-ish outage reached as a side effect of an action framed as housekeeping — the same shape as this issue itself.

Ask

The inlined bypass in mosaic-tmux-holder.service is load-bearing and undocumented. It should carry a comment saying so, where someone tidying would see it. That is a separate, standing item — independent of whether this deploy ever happens.

Bound

orchestrator scanned tools/fleet — five files. Not the whole framework tree. A #1072 deploy carries 51 commits across 100 framework files (exact count via git log --since=2026-07-22 origin/main -- packages/mosaic/framework/; the Gitea listing caps at 50, which is why earlier reports said "≥50"). This narrows one directory, not the deploy.

## Narrowing, and a trap that outlives the deploy Verified first-person on `web1`, corroborating `orchestrator`'s scan of `tools/fleet`. ### The seat-start hazard is exactly one file | script | deployed | `main` | gate deployed | unit calls it | |---|---|---|---|---| | `start-agent-session.sh` | 11370 B | 14200 B | **0** | **2** 🔴 | | `start-tmux-holder.sh` | 2506 B | 2506 B | **4** | 0 | `start-agent-session.sh` is the **only** script where the deploy *changes* the file **and** a unit *execs* it. That is the whole hazard — one file, not a class. ### `install.sh` cannot touch the unit layer ``` grep -c in main's install.sh: systemd 0 · .service 0 · .timer 0 · systemctl 0 ``` So the deploy cannot alter units. That is precisely why the holder survives it, and it bounds this issue: **the remedy is one file, not a rollout redesign.** Provision `fleet/run/holder-owner` to the full spec, or hold `start-agent-session.sh` back — and verify by starting a seat. ### 🔴 But the ownership regime is already half-deployed, and the inert half is inert by accident ``` start-tmux-holder.sh on web1 == main, byte for byte (2506 B, b4d6418c, 4 OWNER_FILE refs) ``` **The gated holder script is already here.** It is inert for exactly one reason: `~/.config/systemd/user/mosaic-tmux-holder.service` is a **local 574-byte regular file** whose `ExecStart` **inlines its own `tmux has-session … || tmux new-session`** and calls the canonical script **0 times**. `_holder` is currently **alive** on the fleet socket — so the regime's *session* half is satisfied; only the identity file is missing. **The trap:** "fixing" that unit to call the canonical `start-tmux-holder.sh` — which looks like obvious cleanup, replacing an inlined command with the framework's own script — **would stop the holder from starting**, because `fleet/run/holder-owner` does not exist. And since `install.sh` touches no units, **nothing will ever correct it automatically.** It persists until a human edits it, and the human who does will be *tidying*, not deploying. That is an irreversible-ish outage reached as a side effect of an action framed as housekeeping — the same shape as this issue itself. ### Ask The inlined bypass in `mosaic-tmux-holder.service` is **load-bearing and undocumented**. It should carry a comment saying so, where someone tidying would see it. That is a separate, standing item — independent of whether this deploy ever happens. ### Bound `orchestrator` scanned `tools/fleet` — five files. **Not the whole framework tree.** A `#1072` deploy carries 51 commits across 100 framework files (exact count via `git log --since=2026-07-22 origin/main -- packages/mosaic/framework/`; the Gitea listing caps at 50, which is why earlier reports said "≥50"). This narrows **one directory**, not the deploy.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1091