mosaicstack/stack
16
0
Fork 1
Code Issues 335 Pull Requests 15 Actions Packages Projects Releases 15 Wiki Activity

Fleet liveness: 10 timers stopped silently for 10 days — the watchers have no watcher #1088

New Issue
Open
opened 2026-08-06 22:16:22 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-06 22:16:22 +00:00
Copy Link
Copy Source

Summary

Ten systemd user timers stopped at the same second on 2026-07-27 12:50:14 and nothing noticed for ten days. No alarm, no log escalation, no degraded-mode banner. The fleet ran visibly "fine" the entire time while two of its core feedback loops were off.

This is not a request for more monitoring. It is that the watchers have no watcher, and the failure is silent by construction.

Evidence (web1, measured 2026-08-06)

gitea-pr-watch.timer          enabled · Result=success · last trigger 2026-07-27 12:50:14  (10 days)
mosaic-context-watcher.timer  enabled · Result=success · last trigger 2026-07-27 12:50:14
mosaic-heartbeat@<seat>       enabled · 6 seats, same timestamp
+ 4 others                    same second
uptime: 8 weeks  ->  no reboot. Units were STOPPED, never restarted.

All still UnitFileState=enabled. All Result=success. They did not crash or get disabled — they were stopped and nothing ever started them again.

What it cost

Two loops, two visible symptoms that were diagnosed as unrelated problems for days:

  1. gitea-pr-watch — its own final log line is tmux_notify=sent:mos-claude. It is the mechanism that tells an agent a PR is ready in its lane. Dead for 10 days ⇒ four PRs sat with nobody notified. The stall was read as a framework-tooling problem; it was silence.

  2. mosaic-context-watcher — clears idle high-context sessions. Dead for 10 days ⇒ reviewer seats drifted to 84%, 89%, 93% context with no reclamation. When a review needed dispatching, only one seat in the fleet was under threshold. That was diagnosed as "the fleet is out of reviewer capacity." It was the same outage.

One root cause, two symptoms, ten days, zero alarms.

The general shape

This is the same class as two other findings filed recently: a mechanism ships and nothing guarantees it is running.

  • a gate that requires an input nothing produces
  • a promotion client that no shipped code calls
  • a watcher that nothing watches

In each case the framework contains the capability and stops short of the step that keeps it in effect.

Suggested direction

Not prescribing the implementation; the requirement is what matters:

  • Liveness expectation as data. A timer that is enabled but has not fired within N intervals is a fault, not a quiet state. The expectation has to live somewhere machine-readable, not in an operator's memory.
  • Fail loud on absence, not just on error. Every unit above reported Result=success. Success-and-stopped is indistinguishable from success-and-running to anything currently looking.
  • One escalation path already exists — these emit to tmux/ntfy today. The gap is that nothing evaluates absence.
  • Do not add a watcher for the watcher-watcher. The check belongs in whatever already runs on a schedule and is itself observed — or in doctor, if doctor is ever run automatically (today nothing runs it).

Acceptance

  • Stop a watched timer and assert an alarm is raised within a bounded window.
  • Negative control: with all timers healthy, assert no alarm — a monitor that always fires is not a monitor.
  • The alarm names the unit and the expected interval, not just "something is wrong."

No closing keywords intended; none used.

## Summary **Ten systemd user timers stopped at the same second on 2026-07-27 12:50:14 and nothing noticed for ten days.** No alarm, no log escalation, no degraded-mode banner. The fleet ran visibly "fine" the entire time while two of its core feedback loops were off. This is not a request for more monitoring. It is that **the watchers have no watcher**, and the failure is silent by construction. ## Evidence (web1, measured 2026-08-06) ``` gitea-pr-watch.timer enabled · Result=success · last trigger 2026-07-27 12:50:14 (10 days) mosaic-context-watcher.timer enabled · Result=success · last trigger 2026-07-27 12:50:14 mosaic-heartbeat@<seat> enabled · 6 seats, same timestamp + 4 others same second uptime: 8 weeks -> no reboot. Units were STOPPED, never restarted. ``` All still `UnitFileState=enabled`. All `Result=success`. They did not crash or get disabled — they were stopped and nothing ever started them again. ## What it cost Two loops, two visible symptoms that were diagnosed as unrelated problems for days: 1. **`gitea-pr-watch`** — its own final log line is `tmux_notify=sent:mos-claude`. It is the mechanism that tells an agent a PR is ready in its lane. **Dead for 10 days ⇒ four PRs sat with nobody notified.** The stall was read as a framework-tooling problem; it was silence. 2. **`mosaic-context-watcher`** — clears idle high-context sessions. **Dead for 10 days ⇒ reviewer seats drifted to 84%, 89%, 93% context** with no reclamation. When a review needed dispatching, only one seat in the fleet was under threshold. That was diagnosed as "the fleet is out of reviewer capacity." It was the same outage. One root cause, two symptoms, ten days, zero alarms. ## The general shape This is the same class as two other findings filed recently: a mechanism ships and nothing guarantees it is *running*. - a gate that requires an input nothing produces - a promotion client that no shipped code calls - **a watcher that nothing watches** In each case the framework contains the capability and stops short of the step that keeps it in effect. ## Suggested direction Not prescribing the implementation; the requirement is what matters: - **Liveness expectation as data.** A timer that is `enabled` but has not fired within N intervals is a *fault*, not a quiet state. The expectation has to live somewhere machine-readable, not in an operator's memory. - **Fail loud on absence, not just on error.** Every unit above reported `Result=success`. Success-and-stopped is indistinguishable from success-and-running to anything currently looking. - **One escalation path already exists** — these emit to tmux/ntfy today. The gap is that nothing evaluates *absence*. - **Do not add a watcher for the watcher-watcher.** The check belongs in whatever already runs on a schedule and is itself observed — or in `doctor`, if `doctor` is ever run automatically (today nothing runs it). ## Acceptance - Stop a watched timer and assert an alarm is raised within a bounded window. - **Negative control:** with all timers healthy, assert no alarm — a monitor that always fires is not a monitor. - The alarm names the unit and the expected interval, not just "something is wrong." No closing keywords intended; none used.
Ghost added the bug label 2026-08-06 22:16:22 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1088