mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 11 Actions Packages Projects Releases 15 Wiki Activity

Feature: declarative liveness gates — timer-driven staleness checks that wake an agent session #1163

New Issue
Open
opened 2026-08-11 22:20:32 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-08-11 22:20:32 +00:00
Owner
Copy Link
Copy Source

Pattern this addresses

A recurring fleet shape: automated producers (pollers, watchers, stagers) run on systemd --user timers and stay green, while the consumer step — an agent curating/triaging/reviewing the staged output — has no trigger of its own. It runs only when a session happens to start. Result: staged output sits unreviewed for days while every dashboard and timer reports healthy.

Existing framework coverage does not catch this:

  • OnFailure=mosaic-unit-failed@ catches a run that fails.
  • mosaic-timer-liveness catches a timer that stops firing.
  • Nothing catches "producer green, consumer never ran."

Proposal

A framework primitive — e.g. mosaic gate install <spec> — where a project declares a gate instead of hand-rolling units:

# gate spec (illustrative)
name: curation-gate
producer_glob: reports/staged-*.json     # artifacts with date-stamped names
consumer_glob: reports/triage-*.md       # stamp the consumer leaves when it runs
pending_hook: scripts/count-pending.sh   # optional: only wake when real work exists
wake:
  session: <agent tmux session>          # delivered via tools/tmux/agent-send.sh
  message: "N new staged record(s) since the last triage; run the catch-up procedure when convenient."
schedule: ["*-*-* 08:27:00", "*-*-* 17:41:00"]   # off-minute, after producer passes

The framework then generates the service/timer pair, wires OnFailure=mosaic-unit-failed@%n, and auto-registers the timer in the MOSAIC_LIVENESS_WATCH list so the gate itself cannot die silently.

Design points learned from a working reference implementation

  • Self-quieting: compare newest producer stamp vs newest consumer stamp; when stale, count actual pending records before waking — a quiet producer day must not page anyone.
  • Wake failures must be loud: a missing agent-send or dead target session exits nonzero so OnFailure fires; a silent non-wake recreates exactly the lapse the gate exists to prevent.
  • Units belong to the project repo (versioned, path-substituted at install); the mechanism belongs to the framework. Operator config dirs are framework-owned and unversioned — wrong home for per-project units.
  • Neutral message register for the wake text (no urgency language), per fleet display standards.

A project-local reference implementation (check script + service/timer + installer integration + liveness registration) exists and can be generalized into the framework tool.

## Pattern this addresses A recurring fleet shape: automated **producers** (pollers, watchers, stagers) run on systemd --user timers and stay green, while the **consumer** step — an agent curating/triaging/reviewing the staged output — has no trigger of its own. It runs only when a session happens to start. Result: staged output sits unreviewed for days while every dashboard and timer reports healthy. Existing framework coverage does not catch this: - `OnFailure=mosaic-unit-failed@` catches a run that fails. - `mosaic-timer-liveness` catches a timer that stops firing. - **Nothing catches "producer green, consumer never ran."** ## Proposal A framework primitive — e.g. `mosaic gate install <spec>` — where a project declares a gate instead of hand-rolling units: ```yaml # gate spec (illustrative) name: curation-gate producer_glob: reports/staged-*.json # artifacts with date-stamped names consumer_glob: reports/triage-*.md # stamp the consumer leaves when it runs pending_hook: scripts/count-pending.sh # optional: only wake when real work exists wake: session: <agent tmux session> # delivered via tools/tmux/agent-send.sh message: "N new staged record(s) since the last triage; run the catch-up procedure when convenient." schedule: ["*-*-* 08:27:00", "*-*-* 17:41:00"] # off-minute, after producer passes ``` The framework then generates the service/timer pair, wires `OnFailure=mosaic-unit-failed@%n`, and auto-registers the timer in the `MOSAIC_LIVENESS_WATCH` list so the gate itself cannot die silently. ## Design points learned from a working reference implementation - **Self-quieting:** compare newest producer stamp vs newest consumer stamp; when stale, count actual pending records before waking — a quiet producer day must not page anyone. - **Wake failures must be loud:** a missing agent-send or dead target session exits nonzero so OnFailure fires; a silent non-wake recreates exactly the lapse the gate exists to prevent. - **Units belong to the project repo** (versioned, path-substituted at install); the *mechanism* belongs to the framework. Operator config dirs are framework-owned and unversioned — wrong home for per-project units. - **Neutral message register** for the wake text (no urgency language), per fleet display standards. A project-local reference implementation (check script + service/timer + installer integration + liveness registration) exists and can be generalized into the framework tool.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1163