mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

ARMED TODAY: restarting mosaic-tmux-holder runs 'tmux kill-server' and destroys all 39 fleet sessions #1092

New Issue
Open
opened 2026-08-07 03:12:13 +00:00 by Ghost · 3 comments
Ghost commented 2026-08-07 03:12:13 +00:00
Copy Link
Copy Source

systemctl --user restart mosaic-tmux-holder destroys the fleet, today

This is armed right now and requires no deploy. It is independent of #1072 and #1091.

The unit, verbatim (~/.config/systemd/user/mosaic-tmux-holder.service, 574 B, local file)

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=MOSAIC_TMUX_SOCKET=mosaic-fleet
Environment=MOSAIC_TMUX_HOLDER=_holder
ExecStart=/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" has-session -t "=${MOSAIC_TMUX_HOLDER}:0.0" 2>/dev/null || tmux -L "$MOSAIC_TMUX_SOCKET" new-session -d -s "$MOSAIC_TMUX_HOLDER" "while true; do sleep 3600; done"'
ExecStop=-/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server'

Why it is armed

ActiveState=active   RemainAfterExit=yes   CanStop=yes
sessions on mosaic-fleet: 39

systemctl --user restart is stop-then-start. The stop runs tmux kill-server on mosaic-fleet and destroys all 39 live sessions — every working seat on this host.

Scope bound: the unit sets MOSAIC_TMUX_SOCKET=mosaic-fleet, so only that socket dies. The default socket (80 sessions) is unaffected. Bad, not total.

Why the obvious cleanup makes it worse

start-tmux-holder.sh is deployed on this host byte-identical to main (2506 B, b4d6418c, 4 OWNER_FILE refs). The canonical script is already here; this unit inlines its own tmux new-session and calls it 0 times.

A seat tidying this sees an inline bash one-liner next to a canonical framework script and replaces one with the other. That is the correct-looking change. But the canonical script's ownership gate fails closed on $MOSAIC_HOME/fleet/run/holder-owner, which does not exist on this host (see #1091).

So the tidy-up is doubly destructive:

  1. the restart kills 39 seats, and
  2. the "fixed" unit then cannot recreate _holder

The unit warns nobody

0 comment lines. Zero mentions of start-tmux-holder.sh, holder-owner, or any rationale. Documentation= points at the repo root. Nothing in the file tells a reader the inline is deliberate or that ExecStop is a fleet-wide kill.

install.sh will never fix or notice this

grep -c in main's install.sh:  systemd 0 · .service 0 · .timer 0 · systemctl 0

The deploy does not touch units. This hazard is orthogonal to #1072 in both directions — the deploy neither causes it nor cures it.

Asks, cheapest first

  1. A comment in the unit stating that the inline is by design, that calling start-tmux-holder.sh requires fleet/run/holder-owner, and that ExecStop kills the whole fleet socket. Cheapest possible mitigation for the highest-consequence trap on this host.
  2. Provision fleet/run/holder-owner to the gate's full spec — this closes both this trap's second half and #1091's seat-start hazard. One file, two problems.
  3. Review ExecStop on its own merits. A holder unit that kills the entire tmux server on stop is a separate design question from the ownership gate, and it is the half that is armed today.

Not done

Nobody should restart this unit to test any of the above. I have not stopped, started, restarted, reloaded, or edited it, and I have not created holder-owner.

Found by tl-mosaic reading the whole unit; ExecStart and the byte-identity were measured by orchestrator; all figures above re-verified first-person on web1.

## `systemctl --user restart mosaic-tmux-holder` destroys the fleet, today This is **armed right now** and requires no deploy. It is independent of #1072 and #1091. ### The unit, verbatim (`~/.config/systemd/user/mosaic-tmux-holder.service`, 574 B, local file) ```ini [Service] Type=oneshot RemainAfterExit=yes Environment=MOSAIC_TMUX_SOCKET=mosaic-fleet Environment=MOSAIC_TMUX_HOLDER=_holder ExecStart=/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" has-session -t "=${MOSAIC_TMUX_HOLDER}:0.0" 2>/dev/null || tmux -L "$MOSAIC_TMUX_SOCKET" new-session -d -s "$MOSAIC_TMUX_HOLDER" "while true; do sleep 3600; done"' ExecStop=-/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server' ``` ### Why it is armed ``` ActiveState=active RemainAfterExit=yes CanStop=yes sessions on mosaic-fleet: 39 ``` `systemctl --user restart` is **stop-then-start**. The stop runs `tmux kill-server` on `mosaic-fleet` and **destroys all 39 live sessions** — every working seat on this host. **Scope bound:** the unit sets `MOSAIC_TMUX_SOCKET=mosaic-fleet`, so only that socket dies. The `default` socket (80 sessions) is unaffected. Bad, not total. ### Why the obvious cleanup makes it worse `start-tmux-holder.sh` is deployed on this host **byte-identical to `main`** (2506 B, `b4d6418c`, 4 `OWNER_FILE` refs). The canonical script is already here; this unit inlines its own `tmux new-session` and calls it **0 times**. A seat tidying this sees an inline bash one-liner next to a canonical framework script and replaces one with the other. That is the correct-looking change. But the canonical script's ownership gate fails closed on `$MOSAIC_HOME/fleet/run/holder-owner`, which **does not exist** on this host (see #1091). So the tidy-up is **doubly destructive**: 1. the restart kills 39 seats, **and** 2. the "fixed" unit then cannot recreate `_holder` ### The unit warns nobody **0 comment lines.** Zero mentions of `start-tmux-holder.sh`, `holder-owner`, or any rationale. `Documentation=` points at the repo root. Nothing in the file tells a reader the inline is deliberate or that `ExecStop` is a fleet-wide kill. ### `install.sh` will never fix or notice this ``` grep -c in main's install.sh: systemd 0 · .service 0 · .timer 0 · systemctl 0 ``` The deploy does not touch units. **This hazard is orthogonal to #1072 in both directions** — the deploy neither causes it nor cures it. ### Asks, cheapest first 1. **A comment in the unit** stating that the inline is by design, that calling `start-tmux-holder.sh` requires `fleet/run/holder-owner`, and that `ExecStop` kills the whole fleet socket. Cheapest possible mitigation for the highest-consequence trap on this host. 2. **Provision `fleet/run/holder-owner`** to the gate's full spec — this closes **both** this trap's second half and #1091's seat-start hazard. One file, two problems. 3. **Review `ExecStop` on its own merits.** A holder unit that kills the entire tmux server on stop is a separate design question from the ownership gate, and it is the half that is armed today. ### Not done Nobody should restart this unit to test any of the above. I have not stopped, started, restarted, reloaded, or edited it, and I have not created `holder-owner`. Found by `tl-mosaic` reading the whole unit; `ExecStart` and the byte-identity were measured by `orchestrator`; all figures above re-verified first-person on `web1`.
Ghost commented 2026-08-07 03:13:17 +00:00
Copy Link
Copy Source

Status: two seats can execute the cheapest mitigation; both are waiting rather than acting

orchestrator independently verified the whole unit and reached the same position I did. Recording it here so the decision does not depend on either session surviving.

Both orchestrator and I are able to write the warning comment and both have declined to do it unasked. It needs one word from the operator, to either of us.

What that edit is, precisely

3 comment lines in ~/.config/systemd/user/mosaic-tmux-holder.service:

inline by design; calling start-tmux-holder.sh requires fleet/run/holder-owner, which is not provisioned. ExecStop runs kill-server on the fleet socket — a restart kills every session.

Risk profile: comments change no behaviour · require no daemon-reload · require no restart · trivially reversible. The file is what a tidying human reads, which is the entire point.

Why neither of us just did it: it is still a write to live fleet configuration, which has been the operator's call all session. Annotating a live unit on our own judgement at 03:00 is the wrong precedent even when the edit is inert.

One clarification worth pinning

ExecStop carries a - prefix. That means systemd ignores its failure — not that the command is skipped. It runs.

And a correction each of us owes

orchestrator reports it read this file earlier with grep -nE 'ExecStart' and reported the unit's behaviour from one line of fifteen, on a 574-byte file, while writing an escalation about that very file.

Mine is the same shape: my first look used grep -c 'start-tmux-holder' and grep -E '^ExecStart'. I selected the lines that answered my question and stopped. I only saw ExecStop after tl-mosaic named it.

Three seats examined this unit; the third was the first to read all fifteen lines. For a file this small, cat cost nothing and would have surfaced line 12 to any of us.

Unchanged

The larger fix is still one file: fleet/run/holder-owner, provisioned to spec, disarms both this trap's second half and #1091's seat-start hazard. ExecStop=kill-server on a holder unit deserves review on its own merits regardless — that is a design question, not a gate question.

Nobody should test any of this. Neither seat has stopped, started, restarted, reloaded, or edited the unit, and neither has created holder-owner.

## Status: two seats can execute the cheapest mitigation; both are waiting rather than acting `orchestrator` independently verified the whole unit and reached the same position I did. Recording it here so the decision does not depend on either session surviving. **Both `orchestrator` and I are able to write the warning comment and both have declined to do it unasked.** It needs one word from the operator, to either of us. ### What that edit is, precisely 3 comment lines in `~/.config/systemd/user/mosaic-tmux-holder.service`: > inline by design; calling `start-tmux-holder.sh` requires `fleet/run/holder-owner`, which is not provisioned. `ExecStop` runs `kill-server` on the fleet socket — a restart kills every session. **Risk profile:** comments change no behaviour · require no `daemon-reload` · require no restart · trivially reversible. The file is what a tidying human reads, which is the entire point. **Why neither of us just did it:** it is still a write to live fleet configuration, which has been the operator's call all session. Annotating a live unit on our own judgement at 03:00 is the wrong precedent even when the edit is inert. ### One clarification worth pinning `ExecStop` carries a `-` prefix. That means **systemd ignores its failure** — **not** that the command is skipped. It runs. ### And a correction each of us owes `orchestrator` reports it read this file earlier with `grep -nE 'ExecStart'` and reported the unit's behaviour from one line of fifteen, on a 574-byte file, while writing an escalation about that very file. Mine is the same shape: my first look used `grep -c 'start-tmux-holder'` and `grep -E '^ExecStart'`. I selected the lines that answered my question and stopped. I only saw `ExecStop` after `tl-mosaic` named it. **Three seats examined this unit; the third was the first to read all fifteen lines.** For a file this small, `cat` cost nothing and would have surfaced line 12 to any of us. ### Unchanged The larger fix is still one file: `fleet/run/holder-owner`, provisioned to spec, disarms both this trap's second half and #1091's seat-start hazard. `ExecStop=kill-server` on a holder unit deserves review on its own merits regardless — that is a design question, not a gate question. Nobody should test any of this. Neither seat has stopped, started, restarted, reloaded, or edited the unit, and neither has created `holder-owner`.
Ghost commented 2026-08-07 03:14:47 +00:00
Copy Link
Copy Source

Correction: mosaic-tmux.service is not a second undocumented trap. It is the documented, intentional one.

tl-mosaic flagged a second ExecStop=kill-server and — correctly — bounded itself: "I have not counted mosaic-tmux.service's comments, and I am not asserting it." I counted them. The answer inverts the conclusion.

The two units are opposites

mosaic-tmux-holder.service mosaic-tmux.service
comment lines 0 4
documents the kill no yes, explicitly
socket mosaic-fleet (39) default (80)
state active / exited active / running / enabled

mosaic-tmux.service, verbatim:

Description=Mosaic shared tmux server (fleet) — sole owner of the default tmux socket
Documentation=file:/src/jarvis-brain/infra/fleet/README.md
# The ONLY intentional way to recycle the whole fleet's tmux server.
ExecStop=-/usr/bin/tmux kill-server
# This unit — and ONLY this unit — owns the tmux server's cgroup.
KillMode=control-group

That is a designed, labelled, single-owner recycle path. A reader of that file is told exactly what stopping it does, in the line immediately above the command that does it. It is the opposite of this issue's finding.

And KillMode=control-group means the kill is not even the mechanism

The tmux server runs inside that unit's cgroup. systemd tears it down on stop regardless of ExecStop. The kill-server line is belt-and-braces, not the sole destroyer — so removing it would not make stopping that unit safe, and adding a warning to it would be documenting what it already documents.

What this does and does not change

  • Unchanged: this issue stands exactly as filed. mosaic-tmux-holder.service has zero comments, routes around a canonical script that is already deployed byte-identical to main, and its obvious cleanup is destructive. That is the trap.
  • Corrected: there is one undocumented socket-killer on this host, not two. The 80-session unit is the intended fleet-recycle path and says so.
  • Still worth a separate look: whether Restart=on-failure on a unit whose stop recycles the entire fleet is the right pairing. That is a design question about mosaic-tmux.service, not a trap, and it does not belong in this issue.

Why I checked rather than amplified

"There are two socket-killers, one of them twice the size" is the more alarming sentence and it was one grep -c '^\s*#' away from being wrong. Counting the comments cost nothing. The pattern this session keeps producing is that the alarming version travels and the bounding measurement does not get run — tl-mosaic avoided that by declining to assert what it had not counted.

## Correction: `mosaic-tmux.service` is **not** a second undocumented trap. It is the documented, intentional one. `tl-mosaic` flagged a second `ExecStop=kill-server` and — correctly — bounded itself: *"I have not counted `mosaic-tmux.service`'s comments, and I am not asserting it."* I counted them. The answer inverts the conclusion. ### The two units are opposites | | `mosaic-tmux-holder.service` | `mosaic-tmux.service` | |---|---|---| | comment lines | **0** | **4** | | documents the kill | **no** | **yes, explicitly** | | socket | `mosaic-fleet` (39) | default (80) | | state | active / **exited** | active / **running** / enabled | `mosaic-tmux.service`, verbatim: ```ini Description=Mosaic shared tmux server (fleet) — sole owner of the default tmux socket Documentation=file:/src/jarvis-brain/infra/fleet/README.md # The ONLY intentional way to recycle the whole fleet's tmux server. ExecStop=-/usr/bin/tmux kill-server # This unit — and ONLY this unit — owns the tmux server's cgroup. KillMode=control-group ``` **That is a designed, labelled, single-owner recycle path.** A reader of that file is told exactly what stopping it does, in the line immediately above the command that does it. It is the *opposite* of this issue's finding. ### And `KillMode=control-group` means the kill is not even the mechanism The tmux server runs inside that unit's cgroup. **systemd tears it down on stop regardless of `ExecStop`.** The `kill-server` line is belt-and-braces, not the sole destroyer — so removing it would not make stopping that unit safe, and adding a warning to it would be documenting what it already documents. ### What this does and does not change - **Unchanged:** this issue stands exactly as filed. `mosaic-tmux-holder.service` has **zero** comments, routes around a canonical script that is already deployed byte-identical to `main`, and its obvious cleanup is destructive. That is the trap. - **Corrected:** there is **one** undocumented socket-killer on this host, not two. The 80-session unit is the intended fleet-recycle path and says so. - **Still worth a separate look:** whether `Restart=on-failure` on a unit whose stop recycles the entire fleet is the right pairing. That is a design question about `mosaic-tmux.service`, not a trap, and it does not belong in this issue. ### Why I checked rather than amplified "There are two socket-killers, one of them twice the size" is the more alarming sentence and it was one `grep -c '^\s*#'` away from being wrong. Counting the comments cost nothing. The pattern this session keeps producing is that the alarming version travels and the bounding measurement does not get run — `tl-mosaic` avoided that by declining to assert what it had not counted.
Ghost commented 2026-08-07 06:50:29 +00:00
Copy Link
Copy Source

Correction and a mechanism: the canonical unit also has ExecStop=kill-server. And web1's unit is a pre-canonical form the framework now tests against.

I filed this issue framing ExecStop=… kill-server as the hazard. That framing was wrong — the canonical unit in main has it too. Measured:

main  packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service
  ExecStart=/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet
            MOSAIC_TMUX_HOLDER=_holder /bin/bash --noprofile --norc
            %h/.config/mosaic/tools/fleet/start-tmux-holder.sh
  ExecStop=-/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet
            /bin/bash --noprofile --norc -c 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server'

web1  ~/.config/systemd/user/mosaic-tmux-holder.service
  ExecStart=/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" has-session … || tmux … new-session …'
  ExecStop=-/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server'

kill-server on stop is designed behaviour, present in both. The 39-session blast radius is real and worth an operator knowing, but it is not a defect and I should not have implied one. Withdrawn.

What the real difference is

  1. The canonical unit calls start-tmux-holder.sh. web1's inlines its own tmux new-session and calls it zero times.
  2. The canonical unit runs /usr/bin/env -i … --noprofile --norc. web1's runs /bin/bash -lc — a login shell.

And the framework already tests for exactly this

packages/mosaic/framework/systemd/user/test-fleet-units.sh asserts the canonical ExecStart string verbatim, and fails with:

"holder does not remove loader and shell-control variables"
"holder does not clear manager environment before starting tmux"
"holder must not start tmux through a login shell"

web1's installed unit violates all three. bash -lc sources the user profile — which is precisely what env -i … --noprofile --norc exists to prevent.

Why this matters for the trap

The trap I described was "someone tidying might switch the inline to the canonical script." The mechanism is stronger than that: the canonical unit already does exactly that, the framework tests for it, and web1 is simply behind. This is not a hypothetical cleanup — it is the direction the framework has already moved.

And the consequence is unchanged: the canonical ExecStart invokes start-tmux-holder.sh, whose gate fails closed on fleet/run/holder-owner — absent on web1 (#1091). Adopting the canonical unit without provisioning that file stops the holder.

Bounds

  • install.sh touches no units, so no deploy closes this gap. It persists until someone updates the unit by hand.
  • I have not run test-fleet-units.sh. It contains 4 live-action calls (systemctl / tmux new-session/kill-server); running it on this host is not safe and I did not.
  • I have not established when web1's unit diverged, or whether it ever matched the canonical form.

Surfaced while giving be-coder-06 context on a systemd-suite failure it hit rebasing #1073 — "holder execution triggered a contaminated loader." bash -lc is a loader-contaminating invocation, which is consistent, but the diagnosis is be-coder-06's and I am not claiming it.

## Correction and a mechanism: the canonical unit **also** has `ExecStop=kill-server`. And web1's unit is a pre-canonical form the framework now tests against. I filed this issue framing `ExecStop=… kill-server` as the hazard. **That framing was wrong** — the canonical unit in `main` has it too. Measured: ``` main packages/mosaic/framework/systemd/user/mosaic-tmux-holder.service ExecStart=/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet MOSAIC_TMUX_HOLDER=_holder /bin/bash --noprofile --norc %h/.config/mosaic/tools/fleet/start-tmux-holder.sh ExecStop=-/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet /bin/bash --noprofile --norc -c 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server' web1 ~/.config/systemd/user/mosaic-tmux-holder.service ExecStart=/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" has-session … || tmux … new-session …' ExecStop=-/bin/bash -lc 'tmux -L "$MOSAIC_TMUX_SOCKET" kill-server' ``` **`kill-server` on stop is designed behaviour, present in both.** The 39-session blast radius is real and worth an operator knowing, but it is not a defect and I should not have implied one. Withdrawn. ## What the real difference is 1. **The canonical unit calls `start-tmux-holder.sh`. web1's inlines its own `tmux new-session` and calls it zero times.** 2. **The canonical unit runs `/usr/bin/env -i … --noprofile --norc`. web1's runs `/bin/bash -lc` — a login shell.** ## And the framework already tests for exactly this `packages/mosaic/framework/systemd/user/test-fleet-units.sh` asserts the canonical `ExecStart` string verbatim, and fails with: ``` "holder does not remove loader and shell-control variables" "holder does not clear manager environment before starting tmux" "holder must not start tmux through a login shell" ``` **web1's installed unit violates all three.** `bash -lc` sources the user profile — which is precisely what `env -i … --noprofile --norc` exists to prevent. ## Why this matters for the trap The trap I described was *"someone tidying might switch the inline to the canonical script."* The mechanism is stronger than that: **the canonical unit already does exactly that**, the framework tests for it, and web1 is simply behind. This is not a hypothetical cleanup — it is the direction the framework has already moved. And the consequence is unchanged: the canonical `ExecStart` invokes `start-tmux-holder.sh`, whose gate fails closed on `fleet/run/holder-owner` — **absent on web1** (#1091). Adopting the canonical unit without provisioning that file stops the holder. ## Bounds - `install.sh` touches no units, so no deploy closes this gap. It persists until someone updates the unit by hand. - I have **not** run `test-fleet-units.sh`. It contains 4 live-action calls (`systemctl` / `tmux new-session`/`kill-server`); running it on this host is not safe and I did not. - I have not established when web1's unit diverged, or whether it ever matched the canonical form. Surfaced while giving `be-coder-06` context on a systemd-suite failure it hit rebasing #1073 — *"holder execution triggered a contaminated loader."* `bash -lc` is a loader-contaminating invocation, which is consistent, but the diagnosis is `be-coder-06`'s and I am not claiming it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1092