feat(wake): W6 — off-host dead-man beacon + pluggable alarm-sink adapter
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

A8 of the wake canon (EPIC #892, W6): the independent liveness leg.

beacon.sh:
- MONOTONIC EMITTER (emit): strictly-increasing seq + emit_ts each cycle,
  shipped off-host via a pluggable sink adapter. The primitive the detector
  run-loop calls each poll (§1.3).
- DEAD-MAN / ABSENCE alarm (record + check): the off-host monitor's receiver
  and absence check. A missing/stale-past-SLO beacon FIRES the alarm and ROUTES
  it to a human/other-host, depending on nothing the dying host does (§1.3/§4-G1).
- PLUGGABLE alarm-sink / beacon-sink ADAPTER INTERFACE: the framework ships the
  emitter + interface; the operator owns the target endpoint, resolved BY NAME
  (load_credentials), never inlined (§1.4). FAIL-CLOSED: unconfigured OR
  unreachable target FAILS LOUD — no silent no-alarm host (§4-G2a).
- DEGRADED MODES (honest, not silent): a same-host sibling is REJECTED as
  non-independent (shares user-manager/host/sender/socket/session); an isolated
  host degrades to a FLAGGED different-supervision-root beacon; capture-pane is
  a liveness HINT only (§1.3).

detector.sh: a single minimal, opt-in W6 emit seam in the run-loop (inert unless
the operator wires WAKE_BEACON_SINK_CMD; a beacon emit failure never kills the
loop but is loud). No store/ack/digest/sign code touched.

test-wake-beacon.sh: RED-FIRST harness (10 invariant groups), wired into
test:framework-shell. manifest.txt bumped 0.4.0 -> 0.5.0 + inventory.

Part of #892

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
This commit is contained in:
mosaic-coder
2026-07-25 21:18:51 -05:00
parent 320f5bfb6f
commit 060977c732
5 changed files with 722 additions and 3 deletions

View File

@@ -0,0 +1,414 @@
#!/usr/bin/env bash
# beacon.sh — A8 of the wake canon (EPIC #892, W6): the off-host DEAD-MAN
# liveness BEACON emitter + the pluggable ALARM-SINK adapter + the beacon-ABSENCE
# alarm (the check an off-host monitor runs).
#
# CONTRACT ANCHORS (docs/scratchpads/heartbeat-planning/CONVERGED-DESIGN.md):
# §1.3 Independent liveness leg — off-host dead-man beacon. Liveness is SPLIT
# from work-triggering: the detector emits a MONOTONIC beacon each cycle
# to existing OFF-HOST monitoring, and the alarm fires on beacon
# ABSENCE — depending on NOTHING the dying component must actively do.
# A same-host sibling is CONCEDED NOT independent (shares user-manager,
# host, sender, socket, session — sol A1/A8/G9) and is REJECTED here. A
# single ISOLATED host degrades to a weaker DIFFERENT-SUPERVISION-ROOT
# beacon, FLAGGED as such — never silently pretending full independence.
# `capture-pane` is a liveness HINT ONLY; readiness is proven solely by
# the RECEIVED/CONSUMED acks (W2/W3), never by a pane scrape.
# §4 G1 Off-host dead-man response: the beacon-absence alarm is proven to FIRE
# AND ROUTE to a human/other-host within its SLO.
# §4 G2a Fail-loud semantics: an unconfigured OR unreachable target FAILS LOUD
# (no silent no-alarm host).
# §7-res6 The only real bound on a stuck drainer is an INDEPENDENT
# missing-RECEIVED/CONSUMED escalation through the off-host observer and
# its SLO — the same independent leg this tool provides.
#
# FRAMEWORK / OPERATOR BOUNDARY (§1.4, the operator-agnostic framework/operator split):
# FRAMEWORK (this file) ships:
# - the monotonic beacon EMITTER (`emit`),
# - the off-host monitor's RECEIVER + ABSENCE alarm (`record`, `check`),
# - a pluggable ALARM-SINK / BEACON-SINK ADAPTER INTERFACE.
# OPERATOR owns:
# - the TARGET ENDPOINT (off-host monitor address, alarm route). It is a
# command the operator wires; that command resolves its address/credential
# BY NAME via `load_credentials`, NEVER inlined into this framework file.
# W7 (installer, OUT OF SCOPE here) WIRES + install-validates the target. W6
# provides the emitter + adapter interface + absence-alarm + the FAIL-LOUD
# PRIMITIVE that W7's validation calls.
#
# ADAPTER INTERFACE (the pluggable seam — operator supplies the command):
# WAKE_BEACON_SINK_CMD emit ships the beacon record (JSON on stdin) via
# `sh -c "$WAKE_BEACON_SINK_CMD"`. The command is the
# transport to the OFF-HOST monitor; it resolves its
# endpoint by-name (load_credentials) internally. A
# non-zero exit = UNREACHABLE target => FAIL LOUD. Unset
# = UNCONFIGURED target => FAIL LOUD (no silent no-alarm
# host).
# WAKE_ALARM_SINK_CMD check routes the absence ALARM (JSON on stdin) via
# `sh -c "$WAKE_ALARM_SINK_CMD"` to a human/other-host
# (G1). Same fail-closed contract: unset OR non-zero
# exit => FAIL LOUD.
# WAKE_BEACON_INDEPENDENCE the operator DECLARES the independence class of the
# wired sink (W7 install-validates it is truthful):
# off-host -> full independence.
# different-supervision-root -> DEGRADED, FLAGGED.
# same-host-sibling -> REJECTED (not independent).
#
# Operator-agnostic: all state via XDG/env; no operator paths/names/secrets/hosts.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=./_wake-common.sh disable=SC1091
. "$SCRIPT_DIR/_wake-common.sh"
STATE_DIR="$(wake_state_dir)"
# Beacon state (emitter counter + the off-host monitor's received store) lives
# UNDER the wake STATE_DIR in its own subdir so it never collides with the
# store's cursor/inbox files.
BEACON_DIR="$STATE_DIR/beacon"
SEQ_FILE="$BEACON_DIR/seq"
# The off-host monitor's RECEIVED store: the last (highest-seq) beacon this
# monitor has ingested for the host. `check` reads it; `record` writes it. In a
# real deployment this lives on the MONITOR host, fed by the sink transport; the
# default keeps the primitive self-contained + testable.
RECEIVED_FILE="${WAKE_BEACON_RECEIVED:-$BEACON_DIR/received.json}"
_need_jq() {
command -v jq >/dev/null 2>&1 || {
echo "beacon.sh: jq is required" >&2
exit 3
}
}
usage() {
cat >&2 <<'EOF'
Usage: beacon.sh <command> [options]
Commands:
emit HOST side. Mint the next MONOTONIC beacon (strictly
increasing seq + emit_ts) and SHIP it off-host via the
pluggable sink adapter (WAKE_BEACON_SINK_CMD). This is
the primitive the detector's run-loop calls each poll.
FAIL-CLOSED: an UNCONFIGURED sink (unset) or an
UNREACHABLE sink (non-zero exit) FAILS LOUD — never a
silent no-alarm host. A same-host-sibling independence
declaration is REJECTED; a different-supervision-root
one is emitted DEGRADED + FLAGGED (never as healthy).
record OFF-HOST MONITOR side. Ingest a received beacon (JSON
on stdin) into the received store, keeping the highest
seq (a stale/replayed lower-or-equal seq is ignored —
the monotonic contract enforced at the receiver).
check --slo-seconds N OFF-HOST MONITOR side. The DEAD-MAN / ABSENCE check.
Reads the last received beacon; if it is MISSING or
STALE beyond the SLO, that ABSENCE FIRES the alarm and
ROUTES it via WAKE_ALARM_SINK_CMD to a human/other-host
(G1). Depends on NOTHING the dying host actively does.
Exit 0 = alive (fresh beacon); exit 1 = absence, alarm
fired+routed; exit 3 = fail-closed (missing SLO, or an
unconfigured/unreachable alarm target — loudest).
A capture-pane hint (WAKE_BEACON_PANE_HINT) is a
liveness HINT ONLY and does NOT suppress an absence.
status Print the emitter counter + last received beacon.
Options:
--slo-seconds N REQUIRED for check. The absence SLO (operator-tuned; symbolic
per-class tiers per §4 — NO default is invented). A beacon
older than N seconds is an ABSENCE.
Environment:
WAKE_BEACON_SINK_CMD Pluggable transport to the off-host monitor (emit).
WAKE_ALARM_SINK_CMD Pluggable alarm route to a human/other-host (check).
WAKE_BEACON_INDEPENDENCE off-host | different-supervision-root |
same-host-sibling (REQUIRED for emit; declared by
the operator, install-validated by W7).
WAKE_BEACON_HOST_ID Host identity carried in the beacon (default: hostname).
WAKE_BEACON_SUPERVISION_ROOT Supervision-root label for the degraded beacon.
WAKE_BEACON_RECEIVED Off-host monitor's received-beacon store path.
WAKE_STATE_HOME/WAKE_AGENT wake state namespace (see store.sh).
EOF
}
# --- monotonic emitter counter (§1.3: a monotonic beacon each cycle) --------
# _next_seq — atomically increment + return the beacon counter. Strictly
# increasing across emits (the monotonic contract the off-host monitor relies on
# to distinguish a live-advancing host from a stalled one).
_next_seq() {
mkdir -p "$BEACON_DIR"
local cur nxt
cur="$(_wake_read_int "$SEQ_FILE" 0)"
nxt=$((cur + 1))
printf '%s' "$nxt" | _atomic_write "$SEQ_FILE"
printf '%s' "$nxt"
}
# --- independence policy (§1.3: split liveness; reject a non-independent leg) -
# _resolve_independence — validate the operator's declared independence class and
# echo two space-separated tokens: "<class> <degraded 0|1>".
# same-host-sibling -> REJECTED (fail loud): shares user-manager/host/
# sender/socket/session, so it is NOT an independent
# liveness leg and must NEVER stand in for one.
# different-supervision-root -> ACCEPTED but DEGRADED=1: an isolated host with
# no off-host monitor reachable degrades to this
# weaker leg, FLAGGED — never silently "healthy".
# off-host -> ACCEPTED, DEGRADED=0: full independence.
_resolve_independence() {
local decl="${WAKE_BEACON_INDEPENDENCE:-}"
case "$decl" in
off-host)
printf 'off-host 0'
;;
different-supervision-root)
printf 'different-supervision-root 1'
;;
same-host-sibling)
echo "beacon.sh: FAIL LOUD (§1.3) — WAKE_BEACON_INDEPENDENCE=same-host-sibling is NOT an independent liveness leg (it shares user-manager/host/sender/socket/session with the component it would supervise). Refusing to emit a beacon that would falsely stand in for off-host supervision." >&2
return 2
;;
'')
echo "beacon.sh: FAIL LOUD — WAKE_BEACON_INDEPENDENCE is unset. Declare the wired sink's independence (off-host | different-supervision-root | same-host-sibling); liveness independence is never silently assumed." >&2
return 2
;;
*)
echo "beacon.sh: FAIL LOUD — WAKE_BEACON_INDEPENDENCE='$decl' is not a recognized class (off-host | different-supervision-root | same-host-sibling)." >&2
return 2
;;
esac
}
# --- host side: emit -------------------------------------------------------
cmd_emit() {
_need_jq
# Independence FIRST — a same-host-sibling / unknown declaration is rejected
# before any seq is minted, so a rejected emit never advances the counter.
local indep_class degraded parts
parts="$(_resolve_independence)" || exit 2
indep_class="${parts%% *}"
degraded="${parts##* }"
# FAIL-CLOSED, part 1 — an UNCONFIGURED off-host target is a silent no-alarm
# host. Refuse before minting a seq (nothing to ship it through).
if [ -z "${WAKE_BEACON_SINK_CMD:-}" ]; then
echo "beacon.sh: FAIL LOUD (G2a) — WAKE_BEACON_SINK_CMD is unset: no off-host beacon target is configured. A host with no beacon sink is a silent no-alarm host. Refusing to no-op." >&2
exit 3
fi
local host_id supervision_root seq emit_ts record
host_id="${WAKE_BEACON_HOST_ID:-$(hostname 2>/dev/null || echo host)}"
supervision_root="${WAKE_BEACON_SUPERVISION_ROOT:-}"
seq="$(_next_seq)"
emit_ts="$(date +%s)"
record="$(jq -cn \
--argjson seq "$seq" \
--argjson emit_ts "$emit_ts" \
--arg host_id "$host_id" \
--arg independence "$indep_class" \
--argjson degraded "$([ "$degraded" -eq 1 ] && echo true || echo false)" \
--arg supervision_root "$supervision_root" \
'{kind:"wake-beacon", beacon_seq:$seq, emit_ts:$emit_ts, host_id:$host_id,
independence:$independence, degraded:$degraded}
+ (if $supervision_root == "" then {} else {supervision_root:$supervision_root} end)')"
# DEGRADED beacons are FLAGGED loudly (§1.3) — an isolated host must never
# silently present a weaker different-supervision-root leg as full independence.
if [ "$degraded" -eq 1 ]; then
echo "beacon.sh: DEGRADED beacon (§1.3) — no off-host monitor is reachable; emitting a weaker DIFFERENT-SUPERVISION-ROOT beacon. This is FLAGGED, NOT full independence." >&2
fi
# FAIL-CLOSED, part 2 — ship via the pluggable sink adapter. A non-zero exit is
# an UNREACHABLE target => FAIL LOUD (the beacon did not reach off-host).
if ! printf '%s\n' "$record" | sh -c "$WAKE_BEACON_SINK_CMD"; then
echo "beacon.sh: FAIL LOUD (G2a) — beacon sink is UNREACHABLE (WAKE_BEACON_SINK_CMD exited non-zero); beacon seq $seq did NOT reach the off-host monitor." >&2
exit 1
fi
# Local record of the last emit (introspection / the detector seam). The
# AUTHORITATIVE liveness judgement is the off-host monitor's received store,
# never this local copy.
printf '%s\n' "$record" | _atomic_write "$BEACON_DIR/last-emit.json"
printf 'beacon_seq=%s emit_ts=%s independence=%s degraded=%s\n' \
"$seq" "$emit_ts" "$indep_class" "$([ "$degraded" -eq 1 ] && echo true || echo false)"
}
# --- off-host monitor side: record a received beacon -----------------------
cmd_record() {
_need_jq
local incoming
incoming="$(cat)"
if [ -z "$incoming" ]; then
echo "beacon.sh record: no beacon on stdin" >&2
exit 2
fi
local in_seq in_ts
in_seq="$(printf '%s' "$incoming" | jq -r '.beacon_seq // empty' 2>/dev/null)"
in_ts="$(printf '%s' "$incoming" | jq -r '.emit_ts // empty' 2>/dev/null)"
case "$in_seq" in
'' | *[!0-9]*)
echo "beacon.sh record: FAIL LOUD — received beacon has no integer beacon_seq (malformed)" >&2
exit 2
;;
esac
case "$in_ts" in
'' | *[!0-9]*)
echo "beacon.sh record: FAIL LOUD — received beacon has no integer emit_ts (malformed)" >&2
exit 2
;;
esac
mkdir -p "$(dirname "$RECEIVED_FILE")"
# MONOTONIC receiver: keep the highest-seq beacon. A lower-or-equal seq is a
# stale/replayed beacon and is IGNORED (does not roll the liveness clock back).
local last_seq=0
if [ -f "$RECEIVED_FILE" ]; then
last_seq="$(jq -r '.beacon_seq // 0' "$RECEIVED_FILE" 2>/dev/null)"
case "$last_seq" in '' | *[!0-9]*) last_seq=0 ;; esac
fi
if [ "$in_seq" -le "$last_seq" ]; then
echo "beacon.sh record: ignoring stale/replayed beacon seq $in_seq (<= recorded $last_seq)" >&2
exit 0
fi
printf '%s\n' "$incoming" | _atomic_write "$RECEIVED_FILE"
}
# --- off-host monitor side: the DEAD-MAN / ABSENCE check -------------------
# _fire_alarm REASON PAYLOAD_JSON — route an absence alarm via the pluggable
# alarm-sink adapter to a human/other-host (G1). FAIL-CLOSED: an unconfigured OR
# unreachable alarm target is the loudest failure (a silent no-alarm host is the
# exact condition G1 exists to prevent).
_fire_alarm() {
local reason="$1" payload="$2"
if [ -z "${WAKE_ALARM_SINK_CMD:-}" ]; then
echo "beacon.sh: FAIL LOUD (G1/G2a) — beacon ABSENCE detected ($reason) but WAKE_ALARM_SINK_CMD is UNSET: the alarm cannot route to a human/other-host. This is a silent no-alarm host — the exact failure G1 forbids." >&2
exit 3
fi
if ! printf '%s\n' "$payload" | sh -c "$WAKE_ALARM_SINK_CMD"; then
echo "beacon.sh: FAIL LOUD (G1/G2a) — beacon ABSENCE detected ($reason) but the alarm sink is UNREACHABLE (WAKE_ALARM_SINK_CMD exited non-zero): the alarm did NOT reach a human/other-host." >&2
exit 3
fi
echo "beacon.sh: ALARM FIRED + ROUTED (§1.3/G1) — $reason" >&2
exit 1
}
cmd_check() {
_need_jq
local slo=''
while [ $# -gt 0 ]; do
case "$1" in
--slo-seconds)
slo="${2:-}"
shift 2
;;
*)
echo "beacon.sh check: unknown option '$1'" >&2
exit 2
;;
esac
done
# No invented SLO (design law) — the per-class absence SLO MUST be supplied.
case "$slo" in
'' | *[!0-9]*)
echo "beacon.sh check: --slo-seconds is REQUIRED and must be a non-negative integer (per-class absence SLO is operator-tuned; no default is invented)." >&2
exit 3
;;
esac
# capture-pane is a liveness HINT ONLY (§1.3): its presence NEVER suppresses an
# absence. Readiness is proven solely by the received beacon clock (fed by the
# RECEIVED/CONSUMED acks' independent leg), never by a pane scrape. We note the
# hint for operators but do NOT let it gate the dead-man verdict.
if [ -n "${WAKE_BEACON_PANE_HINT:-}" ]; then
echo "beacon.sh check: note — a capture-pane hint is present; it is a liveness HINT ONLY and does not affect this absence verdict." >&2
fi
local now
now="$(date +%s)"
# ABSENCE, case 1 — NEVER received. Depends on nothing the dying host does.
if [ ! -f "$RECEIVED_FILE" ] || [ ! -s "$RECEIVED_FILE" ]; then
local payload
payload="$(jq -cn --argjson now "$now" --argjson slo "$slo" \
'{kind:"beacon-absence-alarm", reason:"no beacon ever received",
detected_ts:$now, slo_seconds:$slo}')"
_fire_alarm "no beacon ever received" "$payload"
fi
local last_ts last_seq host_id age
last_ts="$(jq -r '.emit_ts // empty' "$RECEIVED_FILE" 2>/dev/null)"
last_seq="$(jq -r '.beacon_seq // empty' "$RECEIVED_FILE" 2>/dev/null)"
host_id="$(jq -r '.host_id // "unknown"' "$RECEIVED_FILE" 2>/dev/null)"
case "$last_ts" in
'' | *[!0-9]*)
local payload
payload="$(jq -cn --argjson now "$now" --argjson slo "$slo" \
'{kind:"beacon-absence-alarm", reason:"received beacon has no valid emit_ts (corrupt)",
detected_ts:$now, slo_seconds:$slo}')"
_fire_alarm "corrupt received beacon (no emit_ts)" "$payload"
;;
esac
age=$((now - last_ts))
# ABSENCE, case 2 — STALE beyond the SLO. A stopped emitter stops advancing the
# received clock; once age > SLO the off-host monitor fires WITHOUT the dying
# host lifting a finger.
if [ "$age" -gt "$slo" ]; then
local payload
payload="$(jq -cn \
--arg host_id "$host_id" \
--argjson last_seq "${last_seq:-0}" \
--argjson last_ts "$last_ts" \
--argjson now "$now" \
--argjson age "$age" \
--argjson slo "$slo" \
'{kind:"beacon-absence-alarm", reason:"beacon stale past SLO",
host_id:$host_id, last_beacon_seq:$last_seq, last_emit_ts:$last_ts,
detected_ts:$now, age_seconds:$age, slo_seconds:$slo}')"
_fire_alarm "beacon stale (${age}s > SLO ${slo}s), host=$host_id last_seq=${last_seq:-?}" "$payload"
fi
echo "beacon.sh: ALIVE — last beacon seq ${last_seq:-?} for host '$host_id' is ${age}s old (<= SLO ${slo}s)."
}
cmd_status() {
local seq
seq="$(_wake_read_int "$SEQ_FILE" 0)"
printf 'beacon_emitter_seq=%s\n' "$seq"
if [ -f "$RECEIVED_FILE" ] && [ -s "$RECEIVED_FILE" ]; then
printf 'last_received='
cat "$RECEIVED_FILE"
else
printf 'last_received=(none)\n'
fi
}
main() {
[ $# -ge 1 ] || {
usage
exit 2
}
local cmd="$1"
shift
case "$cmd" in
emit) cmd_emit "$@" ;;
record) cmd_record "$@" ;;
check) cmd_check "$@" ;;
status) cmd_status "$@" ;;
-h | --help | help) usage ;;
*)
echo "beacon.sh: unknown command '$cmd'" >&2
usage
exit 2
;;
esac
}
main "$@"