ce53b0e646
Some checks failed
ci/woodpecker/pr/ci Pipeline failed
Separate, clearly-labeled commit for observation set (vi) from the #910 review. The HMAC-verify half is a non-trivial blast radius (emit must sign, record must verify-and-bind), so it is isolated here rather than expanding the installer diff. Both changes are additive and backward-compatible (the legacy unsigned path and emit_ts fallback are preserved when no key is configured). (vi-a) Monitor-side ingested_ts: `record` stamps the monitor's own receive-time, and `check` computes staleness from ingested_ts (falling back to emit_ts only for a legacy record). A host shipping a far-future emit_ts to defer its own staleness can no longer fool the off-host dead-man clock. (vi-b) Beacon HMAC-verify at record: when a key name is configured (WAKE_BEACON_HMAC_KEY_NAME / WAKE_HMAC_KEY_NAME), `emit` signs the beacon core via the existing sign.sh sign-digest path and `record` verifies via sign.sh verify AND binds the envelope to this beacon (content_hash == sha256(core), seq/emit_ts match). A spoofed or unsigned beacon is REJECTED fail-loud and never advances the received clock. Key is by-name only — beacon.sh inlines no key material. test-wake-beacon.sh extended with B11 (ingested_ts staleness) and B12 (HMAC-verify + spoof/unsigned rejection); test-wake-install.sh gains I7/I8 for the same invariants. Both harnesses shown red-first under targeted mutation, then green. Part of #892 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
492 lines
23 KiB
Bash
Executable File
492 lines
23 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# beacon.sh — A8 of the wake canon (EPIC #892, W6): the off-host DEAD-MAN
|
|
# liveness BEACON emitter + the pluggable ALARM-SINK adapter + the beacon-ABSENCE
|
|
# alarm (the check an off-host monitor runs).
|
|
#
|
|
# CONTRACT ANCHORS (docs/scratchpads/heartbeat-planning/CONVERGED-DESIGN.md):
|
|
# §1.3 Independent liveness leg — off-host dead-man beacon. Liveness is SPLIT
|
|
# from work-triggering: the detector emits a MONOTONIC beacon each cycle
|
|
# to existing OFF-HOST monitoring, and the alarm fires on beacon
|
|
# ABSENCE — depending on NOTHING the dying component must actively do.
|
|
# A same-host sibling is CONCEDED NOT independent (shares user-manager,
|
|
# host, sender, socket, session — sol A1/A8/G9) and is REJECTED here. A
|
|
# single ISOLATED host degrades to a weaker DIFFERENT-SUPERVISION-ROOT
|
|
# beacon, FLAGGED as such — never silently pretending full independence.
|
|
# `capture-pane` is a liveness HINT ONLY; readiness is proven solely by
|
|
# the RECEIVED/CONSUMED acks (W2/W3), never by a pane scrape.
|
|
# §4 G1 Off-host dead-man response: the beacon-absence alarm is proven to FIRE
|
|
# AND ROUTE to a human/other-host within its SLO.
|
|
# §4 G2a Fail-loud semantics: an unconfigured OR unreachable target FAILS LOUD
|
|
# (no silent no-alarm host).
|
|
# §7-res6 The only real bound on a stuck drainer is an INDEPENDENT
|
|
# missing-RECEIVED/CONSUMED escalation through the off-host observer and
|
|
# its SLO — the same independent leg this tool provides.
|
|
#
|
|
# FRAMEWORK / OPERATOR BOUNDARY (§1.4, the operator-agnostic framework/operator split):
|
|
# FRAMEWORK (this file) ships:
|
|
# - the monotonic beacon EMITTER (`emit`),
|
|
# - the off-host monitor's RECEIVER + ABSENCE alarm (`record`, `check`),
|
|
# - a pluggable ALARM-SINK / BEACON-SINK ADAPTER INTERFACE.
|
|
# OPERATOR owns:
|
|
# - the TARGET ENDPOINT (off-host monitor address, alarm route). It is a
|
|
# command the operator wires; that command resolves its address/credential
|
|
# BY NAME via `load_credentials`, NEVER inlined into this framework file.
|
|
# W7 (installer, OUT OF SCOPE here) WIRES + install-validates the target. W6
|
|
# provides the emitter + adapter interface + absence-alarm + the FAIL-LOUD
|
|
# PRIMITIVE that W7's validation calls.
|
|
#
|
|
# ADAPTER INTERFACE (the pluggable seam — operator supplies the command):
|
|
# WAKE_BEACON_SINK_CMD emit ships the beacon record (JSON on stdin) via
|
|
# `sh -c "$WAKE_BEACON_SINK_CMD"`. The command is the
|
|
# transport to the OFF-HOST monitor; it resolves its
|
|
# endpoint by-name (load_credentials) internally. A
|
|
# non-zero exit = UNREACHABLE target => FAIL LOUD. Unset
|
|
# = UNCONFIGURED target => FAIL LOUD (no silent no-alarm
|
|
# host).
|
|
# WAKE_ALARM_SINK_CMD check routes the absence ALARM (JSON on stdin) via
|
|
# `sh -c "$WAKE_ALARM_SINK_CMD"` to a human/other-host
|
|
# (G1). Same fail-closed contract: unset OR non-zero
|
|
# exit => FAIL LOUD.
|
|
# WAKE_BEACON_INDEPENDENCE the operator DECLARES the independence class of the
|
|
# wired sink (W7 install-validates it is truthful):
|
|
# off-host -> full independence.
|
|
# different-supervision-root -> DEGRADED, FLAGGED.
|
|
# same-host-sibling -> REJECTED (not independent).
|
|
#
|
|
# Operator-agnostic: all state via XDG/env; no operator paths/names/secrets/hosts.
|
|
set -uo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck source=./_wake-common.sh disable=SC1091
|
|
. "$SCRIPT_DIR/_wake-common.sh"
|
|
|
|
STATE_DIR="$(wake_state_dir)"
|
|
# Beacon state (emitter counter + the off-host monitor's received store) lives
|
|
# UNDER the wake STATE_DIR in its own subdir so it never collides with the
|
|
# store's cursor/inbox files.
|
|
BEACON_DIR="$STATE_DIR/beacon"
|
|
SEQ_FILE="$BEACON_DIR/seq"
|
|
# The off-host monitor's RECEIVED store: the last (highest-seq) beacon this
|
|
# monitor has ingested for the host. `check` reads it; `record` writes it. In a
|
|
# real deployment this lives on the MONITOR host, fed by the sink transport; the
|
|
# default keeps the primitive self-contained + testable.
|
|
RECEIVED_FILE="${WAKE_BEACON_RECEIVED:-$BEACON_DIR/received.json}"
|
|
|
|
# The non-circular HMAC signer (A5/W3). Reused VERBATIM here for beacon
|
|
# authentication (W7 monitor-integration hardening): emit signs the beacon core
|
|
# via `sign.sh sign-digest`, record verifies via the existing `sign.sh verify`
|
|
# path. Both resolve the key BY NAME (load_credentials store) — never inline.
|
|
SIGN_SH="$SCRIPT_DIR/sign.sh"
|
|
|
|
# Beacon HMAC key NAME (by-name, never the key itself). Signing is engaged ONLY
|
|
# when a key name is configured — WAKE_BEACON_HMAC_KEY_NAME, else WAKE_HMAC_KEY_NAME.
|
|
# When configured, record REQUIRES an authentic, bound envelope (a spoofed or
|
|
# unsigned beacon is REJECTED). When unset, the legacy unsigned path is preserved
|
|
# (W7 install-validate is what guarantees a key is configured in production).
|
|
_beacon_key_name() { printf '%s' "${WAKE_BEACON_HMAC_KEY_NAME:-${WAKE_HMAC_KEY_NAME:-}}"; }
|
|
|
|
# Canonical beacon CORE (stdin: a full beacon record; stdout: sorted-key JSON with
|
|
# the signature envelope and the monitor-stamped ingested_ts stripped). emit signs
|
|
# this exact projection; record recomputes it identically, so the MAC binds the
|
|
# beacon's payload and an envelope cannot be lifted onto a different beacon.
|
|
_beacon_core() { jq -cS 'del(.beacon_envelope) | del(.ingested_ts)'; }
|
|
_beacon_sha256() { openssl dgst -sha256 -r 2>/dev/null | awk '{print $1}'; }
|
|
|
|
_need_jq() {
|
|
command -v jq >/dev/null 2>&1 || {
|
|
echo "beacon.sh: jq is required" >&2
|
|
exit 3
|
|
}
|
|
}
|
|
|
|
usage() {
|
|
cat >&2 <<'EOF'
|
|
Usage: beacon.sh <command> [options]
|
|
|
|
Commands:
|
|
emit HOST side. Mint the next MONOTONIC beacon (strictly
|
|
increasing seq + emit_ts) and SHIP it off-host via the
|
|
pluggable sink adapter (WAKE_BEACON_SINK_CMD). This is
|
|
the primitive the detector's run-loop calls each poll.
|
|
FAIL-CLOSED: an UNCONFIGURED sink (unset) or an
|
|
UNREACHABLE sink (non-zero exit) FAILS LOUD — never a
|
|
silent no-alarm host. A same-host-sibling independence
|
|
declaration is REJECTED; a different-supervision-root
|
|
one is emitted DEGRADED + FLAGGED (never as healthy).
|
|
|
|
record OFF-HOST MONITOR side. Ingest a received beacon (JSON
|
|
on stdin) into the received store, keeping the highest
|
|
seq (a stale/replayed lower-or-equal seq is ignored —
|
|
the monotonic contract enforced at the receiver).
|
|
|
|
check --slo-seconds N OFF-HOST MONITOR side. The DEAD-MAN / ABSENCE check.
|
|
Reads the last received beacon; if it is MISSING or
|
|
STALE beyond the SLO, that ABSENCE FIRES the alarm and
|
|
ROUTES it via WAKE_ALARM_SINK_CMD to a human/other-host
|
|
(G1). Depends on NOTHING the dying host actively does.
|
|
Exit 0 = alive (fresh beacon); exit 1 = absence, alarm
|
|
fired+routed; exit 3 = fail-closed (missing SLO, or an
|
|
unconfigured/unreachable alarm target — loudest).
|
|
A capture-pane hint (WAKE_BEACON_PANE_HINT) is a
|
|
liveness HINT ONLY and does NOT suppress an absence.
|
|
|
|
status Print the emitter counter + last received beacon.
|
|
|
|
Options:
|
|
--slo-seconds N REQUIRED for check. The absence SLO (operator-tuned; symbolic
|
|
per-class tiers per §4 — NO default is invented). A beacon
|
|
older than N seconds is an ABSENCE.
|
|
|
|
Environment:
|
|
WAKE_BEACON_SINK_CMD Pluggable transport to the off-host monitor (emit).
|
|
WAKE_ALARM_SINK_CMD Pluggable alarm route to a human/other-host (check).
|
|
WAKE_BEACON_INDEPENDENCE off-host | different-supervision-root |
|
|
same-host-sibling (REQUIRED for emit; declared by
|
|
the operator, install-validated by W7).
|
|
WAKE_BEACON_HOST_ID Host identity carried in the beacon (default: hostname).
|
|
WAKE_BEACON_SUPERVISION_ROOT Supervision-root label for the degraded beacon.
|
|
WAKE_BEACON_RECEIVED Off-host monitor's received-beacon store path.
|
|
WAKE_STATE_HOME/WAKE_AGENT wake state namespace (see store.sh).
|
|
EOF
|
|
}
|
|
|
|
# --- monotonic emitter counter (§1.3: a monotonic beacon each cycle) --------
|
|
|
|
# _next_seq — atomically increment + return the beacon counter. Strictly
|
|
# increasing across emits (the monotonic contract the off-host monitor relies on
|
|
# to distinguish a live-advancing host from a stalled one).
|
|
_next_seq() {
|
|
mkdir -p "$BEACON_DIR"
|
|
local cur nxt
|
|
cur="$(_wake_read_int "$SEQ_FILE" 0)"
|
|
nxt=$((cur + 1))
|
|
printf '%s' "$nxt" | _atomic_write "$SEQ_FILE"
|
|
printf '%s' "$nxt"
|
|
}
|
|
|
|
# --- independence policy (§1.3: split liveness; reject a non-independent leg) -
|
|
|
|
# _resolve_independence — validate the operator's declared independence class and
|
|
# echo two space-separated tokens: "<class> <degraded 0|1>".
|
|
# same-host-sibling -> REJECTED (fail loud): shares user-manager/host/
|
|
# sender/socket/session, so it is NOT an independent
|
|
# liveness leg and must NEVER stand in for one.
|
|
# different-supervision-root -> ACCEPTED but DEGRADED=1: an isolated host with
|
|
# no off-host monitor reachable degrades to this
|
|
# weaker leg, FLAGGED — never silently "healthy".
|
|
# off-host -> ACCEPTED, DEGRADED=0: full independence.
|
|
_resolve_independence() {
|
|
local decl="${WAKE_BEACON_INDEPENDENCE:-}"
|
|
case "$decl" in
|
|
off-host)
|
|
printf 'off-host 0'
|
|
;;
|
|
different-supervision-root)
|
|
printf 'different-supervision-root 1'
|
|
;;
|
|
same-host-sibling)
|
|
echo "beacon.sh: FAIL LOUD (§1.3) — WAKE_BEACON_INDEPENDENCE=same-host-sibling is NOT an independent liveness leg (it shares user-manager/host/sender/socket/session with the component it would supervise). Refusing to emit a beacon that would falsely stand in for off-host supervision." >&2
|
|
return 2
|
|
;;
|
|
'')
|
|
echo "beacon.sh: FAIL LOUD — WAKE_BEACON_INDEPENDENCE is unset. Declare the wired sink's independence (off-host | different-supervision-root | same-host-sibling); liveness independence is never silently assumed." >&2
|
|
return 2
|
|
;;
|
|
*)
|
|
echo "beacon.sh: FAIL LOUD — WAKE_BEACON_INDEPENDENCE='$decl' is not a recognized class (off-host | different-supervision-root | same-host-sibling)." >&2
|
|
return 2
|
|
;;
|
|
esac
|
|
}
|
|
|
|
# --- host side: emit -------------------------------------------------------
|
|
|
|
cmd_emit() {
|
|
_need_jq
|
|
|
|
# Independence FIRST — a same-host-sibling / unknown declaration is rejected
|
|
# before any seq is minted, so a rejected emit never advances the counter.
|
|
local indep_class degraded parts
|
|
parts="$(_resolve_independence)" || exit 2
|
|
indep_class="${parts%% *}"
|
|
degraded="${parts##* }"
|
|
|
|
# FAIL-CLOSED, part 1 — an UNCONFIGURED off-host target is a silent no-alarm
|
|
# host. Refuse before minting a seq (nothing to ship it through).
|
|
if [ -z "${WAKE_BEACON_SINK_CMD:-}" ]; then
|
|
echo "beacon.sh: FAIL LOUD (G2a) — WAKE_BEACON_SINK_CMD is unset: no off-host beacon target is configured. A host with no beacon sink is a silent no-alarm host. Refusing to no-op." >&2
|
|
exit 3
|
|
fi
|
|
|
|
local host_id supervision_root seq emit_ts record
|
|
host_id="${WAKE_BEACON_HOST_ID:-$(hostname 2>/dev/null || echo host)}"
|
|
supervision_root="${WAKE_BEACON_SUPERVISION_ROOT:-}"
|
|
seq="$(_next_seq)"
|
|
emit_ts="$(date +%s)"
|
|
|
|
record="$(jq -cn \
|
|
--argjson seq "$seq" \
|
|
--argjson emit_ts "$emit_ts" \
|
|
--arg host_id "$host_id" \
|
|
--arg independence "$indep_class" \
|
|
--argjson degraded "$([ "$degraded" -eq 1 ] && echo true || echo false)" \
|
|
--arg supervision_root "$supervision_root" \
|
|
'{kind:"wake-beacon", beacon_seq:$seq, emit_ts:$emit_ts, host_id:$host_id,
|
|
independence:$independence, degraded:$degraded}
|
|
+ (if $supervision_root == "" then {} else {supervision_root:$supervision_root} end)')"
|
|
|
|
# HMAC-SIGN the beacon (W7 monitor-integration hardening) when a key name is
|
|
# configured. The signer is the existing non-circular sign.sh, keyed BY NAME —
|
|
# emit inlines no key. The signed envelope binds the beacon core (seq/emit_ts/
|
|
# content_hash), so the off-host monitor can reject a spoofed beacon at record.
|
|
local beacon_key; beacon_key="$(_beacon_key_name)"
|
|
if [ -n "$beacon_key" ]; then
|
|
command -v openssl >/dev/null 2>&1 || { echo "beacon.sh: openssl is required to HMAC-sign the beacon (key '$beacon_key' configured)." >&2; exit 3; }
|
|
local core envelope
|
|
core="$(printf '%s' "$record" | _beacon_core)"
|
|
if ! envelope="$(printf '%s' "$core" | "$SIGN_SH" sign-digest --key-name "$beacon_key" --observed-seq "$seq" --emit-ts "$emit_ts" 2>/dev/null)" || [ -z "$envelope" ]; then
|
|
echo "beacon.sh: FAIL LOUD — could not HMAC-sign the beacon with key '$beacon_key' (resolve it by-name in the credential store). Refusing to emit an UNSIGNED beacon when signing is configured." >&2
|
|
exit 1
|
|
fi
|
|
record="$(printf '%s' "$record" | jq -c --argjson env "$envelope" '. + {beacon_envelope:$env}')"
|
|
fi
|
|
|
|
# DEGRADED beacons are FLAGGED loudly (§1.3) — an isolated host must never
|
|
# silently present a weaker different-supervision-root leg as full independence.
|
|
if [ "$degraded" -eq 1 ]; then
|
|
echo "beacon.sh: DEGRADED beacon (§1.3) — no off-host monitor is reachable; emitting a weaker DIFFERENT-SUPERVISION-ROOT beacon. This is FLAGGED, NOT full independence." >&2
|
|
fi
|
|
|
|
# FAIL-CLOSED, part 2 — ship via the pluggable sink adapter. A non-zero exit is
|
|
# an UNREACHABLE target => FAIL LOUD (the beacon did not reach off-host).
|
|
if ! printf '%s\n' "$record" | sh -c "$WAKE_BEACON_SINK_CMD"; then
|
|
echo "beacon.sh: FAIL LOUD (G2a) — beacon sink is UNREACHABLE (WAKE_BEACON_SINK_CMD exited non-zero); beacon seq $seq did NOT reach the off-host monitor." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Local record of the last emit (introspection / the detector seam). The
|
|
# AUTHORITATIVE liveness judgement is the off-host monitor's received store,
|
|
# never this local copy.
|
|
printf '%s\n' "$record" | _atomic_write "$BEACON_DIR/last-emit.json"
|
|
printf 'beacon_seq=%s emit_ts=%s independence=%s degraded=%s\n' \
|
|
"$seq" "$emit_ts" "$indep_class" "$([ "$degraded" -eq 1 ] && echo true || echo false)"
|
|
}
|
|
|
|
# --- off-host monitor side: record a received beacon -----------------------
|
|
|
|
cmd_record() {
|
|
_need_jq
|
|
local incoming
|
|
incoming="$(cat)"
|
|
if [ -z "$incoming" ]; then
|
|
echo "beacon.sh record: no beacon on stdin" >&2
|
|
exit 2
|
|
fi
|
|
local in_seq in_ts
|
|
in_seq="$(printf '%s' "$incoming" | jq -r '.beacon_seq // empty' 2>/dev/null)"
|
|
in_ts="$(printf '%s' "$incoming" | jq -r '.emit_ts // empty' 2>/dev/null)"
|
|
case "$in_seq" in
|
|
'' | *[!0-9]*)
|
|
echo "beacon.sh record: FAIL LOUD — received beacon has no integer beacon_seq (malformed)" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
case "$in_ts" in
|
|
'' | *[!0-9]*)
|
|
echo "beacon.sh record: FAIL LOUD — received beacon has no integer emit_ts (malformed)" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
|
|
# HMAC-VERIFY the beacon (W7 monitor-integration hardening) — additive to the
|
|
# integer/monotonicity validation above. When a key name is configured, a beacon
|
|
# is accepted ONLY if it carries an envelope that is (1) AUTHENTIC under the
|
|
# by-name key (via the existing sign.sh verify path) and (2) BOUND to THIS beacon
|
|
# (its signed content_hash == sha256(core) and its seq/emit_ts match). A spoofed
|
|
# or unsigned beacon is REJECTED, so a forged liveness signal cannot roll the
|
|
# monitor's dead-man clock forward.
|
|
local beacon_key; beacon_key="$(_beacon_key_name)"
|
|
if [ -n "$beacon_key" ]; then
|
|
command -v openssl >/dev/null 2>&1 || { echo "beacon.sh record: openssl is required to HMAC-verify the beacon (key '$beacon_key' configured)." >&2; exit 3; }
|
|
local env
|
|
env="$(printf '%s' "$incoming" | jq -c '.beacon_envelope // empty' 2>/dev/null)"
|
|
if [ -z "$env" ]; then
|
|
echo "beacon.sh record: FAIL LOUD — beacon carries no signature envelope but signing is configured (key '$beacon_key'). Rejecting an UNSIGNED/spoofed beacon." >&2
|
|
exit 2
|
|
fi
|
|
if ! printf '%s' "$env" | "$SIGN_SH" verify --key-name "$beacon_key" >/dev/null 2>&1; then
|
|
echo "beacon.sh record: FAIL LOUD — beacon signature is NOT authentic under key '$beacon_key' (bad HMAC). Rejecting a spoofed beacon." >&2
|
|
exit 2
|
|
fi
|
|
local core core_sha env_chash env_seq env_ts
|
|
core="$(printf '%s' "$incoming" | _beacon_core)"
|
|
core_sha="$(printf '%s' "$core" | _beacon_sha256)"
|
|
env_chash="$(printf '%s' "$env" | jq -r '.signed.content_hash // ""')"
|
|
env_seq="$(printf '%s' "$env" | jq -r '.signed.observed_seq // ""')"
|
|
env_ts="$(printf '%s' "$env" | jq -r '.signed.emit_ts // ""')"
|
|
if [ "$core_sha" != "$env_chash" ] || [ "$env_seq" != "$in_seq" ] || [ "$env_ts" != "$in_ts" ]; then
|
|
echo "beacon.sh record: FAIL LOUD — beacon signature is valid but NOT bound to this beacon (envelope lifted onto altered fields). Rejecting a spoofed beacon." >&2
|
|
exit 2
|
|
fi
|
|
fi
|
|
|
|
mkdir -p "$(dirname "$RECEIVED_FILE")"
|
|
# MONOTONIC receiver: keep the highest-seq beacon. A lower-or-equal seq is a
|
|
# stale/replayed beacon and is IGNORED (does not roll the liveness clock back).
|
|
local last_seq=0
|
|
if [ -f "$RECEIVED_FILE" ]; then
|
|
last_seq="$(jq -r '.beacon_seq // 0' "$RECEIVED_FILE" 2>/dev/null)"
|
|
case "$last_seq" in '' | *[!0-9]*) last_seq=0 ;; esac
|
|
fi
|
|
if [ "$in_seq" -le "$last_seq" ]; then
|
|
echo "beacon.sh record: ignoring stale/replayed beacon seq $in_seq (<= recorded $last_seq)" >&2
|
|
exit 0
|
|
fi
|
|
# Stamp a MONITOR-SIDE ingested_ts at receive time (W7 monitor-integration
|
|
# hardening). Staleness (check) is computed from THIS receive-time, never the
|
|
# host-supplied emit_ts — so a host shipping a far-future emit_ts to defer its
|
|
# own staleness cannot fool the off-host dead-man clock.
|
|
local ingested_ts; ingested_ts="$(date +%s)"
|
|
printf '%s' "$incoming" | jq -c --argjson t "$ingested_ts" '. + {ingested_ts:$t}' | _atomic_write "$RECEIVED_FILE"
|
|
}
|
|
|
|
# --- off-host monitor side: the DEAD-MAN / ABSENCE check -------------------
|
|
|
|
# _fire_alarm REASON PAYLOAD_JSON — route an absence alarm via the pluggable
|
|
# alarm-sink adapter to a human/other-host (G1). FAIL-CLOSED: an unconfigured OR
|
|
# unreachable alarm target is the loudest failure (a silent no-alarm host is the
|
|
# exact condition G1 exists to prevent).
|
|
_fire_alarm() {
|
|
local reason="$1" payload="$2"
|
|
if [ -z "${WAKE_ALARM_SINK_CMD:-}" ]; then
|
|
echo "beacon.sh: FAIL LOUD (G1/G2a) — beacon ABSENCE detected ($reason) but WAKE_ALARM_SINK_CMD is UNSET: the alarm cannot route to a human/other-host. This is a silent no-alarm host — the exact failure G1 forbids." >&2
|
|
exit 3
|
|
fi
|
|
if ! printf '%s\n' "$payload" | sh -c "$WAKE_ALARM_SINK_CMD"; then
|
|
echo "beacon.sh: FAIL LOUD (G1/G2a) — beacon ABSENCE detected ($reason) but the alarm sink is UNREACHABLE (WAKE_ALARM_SINK_CMD exited non-zero): the alarm did NOT reach a human/other-host." >&2
|
|
exit 3
|
|
fi
|
|
echo "beacon.sh: ALARM FIRED + ROUTED (§1.3/G1) — $reason" >&2
|
|
exit 1
|
|
}
|
|
|
|
cmd_check() {
|
|
_need_jq
|
|
local slo=''
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--slo-seconds)
|
|
slo="${2:-}"
|
|
shift 2
|
|
;;
|
|
*)
|
|
echo "beacon.sh check: unknown option '$1'" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# No invented SLO (design law) — the per-class absence SLO MUST be supplied.
|
|
case "$slo" in
|
|
'' | *[!0-9]*)
|
|
echo "beacon.sh check: --slo-seconds is REQUIRED and must be a non-negative integer (per-class absence SLO is operator-tuned; no default is invented)." >&2
|
|
exit 3
|
|
;;
|
|
esac
|
|
|
|
# capture-pane is a liveness HINT ONLY (§1.3): its presence NEVER suppresses an
|
|
# absence. Readiness is proven solely by the received beacon clock (fed by the
|
|
# RECEIVED/CONSUMED acks' independent leg), never by a pane scrape. We note the
|
|
# hint for operators but do NOT let it gate the dead-man verdict.
|
|
if [ -n "${WAKE_BEACON_PANE_HINT:-}" ]; then
|
|
echo "beacon.sh check: note — a capture-pane hint is present; it is a liveness HINT ONLY and does not affect this absence verdict." >&2
|
|
fi
|
|
|
|
local now
|
|
now="$(date +%s)"
|
|
|
|
# ABSENCE, case 1 — NEVER received. Depends on nothing the dying host does.
|
|
if [ ! -f "$RECEIVED_FILE" ] || [ ! -s "$RECEIVED_FILE" ]; then
|
|
local payload
|
|
payload="$(jq -cn --argjson now "$now" --argjson slo "$slo" \
|
|
'{kind:"beacon-absence-alarm", reason:"no beacon ever received",
|
|
detected_ts:$now, slo_seconds:$slo}')"
|
|
_fire_alarm "no beacon ever received" "$payload"
|
|
fi
|
|
|
|
local last_ts last_seq host_id age
|
|
# Staleness clock = the MONITOR's receive-time (ingested_ts), stamped by record.
|
|
# Fall back to emit_ts only for a legacy record written before ingested_ts
|
|
# existed. A host-supplied far-future emit_ts therefore CANNOT defer staleness:
|
|
# once ingested_ts is present it governs, and emit_ts is never used for age.
|
|
last_ts="$(jq -r '.ingested_ts // .emit_ts // empty' "$RECEIVED_FILE" 2>/dev/null)"
|
|
last_seq="$(jq -r '.beacon_seq // empty' "$RECEIVED_FILE" 2>/dev/null)"
|
|
host_id="$(jq -r '.host_id // "unknown"' "$RECEIVED_FILE" 2>/dev/null)"
|
|
case "$last_ts" in
|
|
'' | *[!0-9]*)
|
|
local payload
|
|
payload="$(jq -cn --argjson now "$now" --argjson slo "$slo" \
|
|
'{kind:"beacon-absence-alarm", reason:"received beacon has no valid receive timestamp (corrupt)",
|
|
detected_ts:$now, slo_seconds:$slo}')"
|
|
_fire_alarm "corrupt received beacon (no ingested_ts/emit_ts)" "$payload"
|
|
;;
|
|
esac
|
|
|
|
age=$((now - last_ts))
|
|
|
|
# ABSENCE, case 2 — STALE beyond the SLO. A stopped emitter stops advancing the
|
|
# received clock; once age > SLO the off-host monitor fires WITHOUT the dying
|
|
# host lifting a finger.
|
|
if [ "$age" -gt "$slo" ]; then
|
|
local payload
|
|
payload="$(jq -cn \
|
|
--arg host_id "$host_id" \
|
|
--argjson last_seq "${last_seq:-0}" \
|
|
--argjson last_ts "$last_ts" \
|
|
--argjson now "$now" \
|
|
--argjson age "$age" \
|
|
--argjson slo "$slo" \
|
|
'{kind:"beacon-absence-alarm", reason:"beacon stale past SLO",
|
|
host_id:$host_id, last_beacon_seq:$last_seq, last_received_ts:$last_ts,
|
|
detected_ts:$now, age_seconds:$age, slo_seconds:$slo}')"
|
|
_fire_alarm "beacon stale (${age}s > SLO ${slo}s), host=$host_id last_seq=${last_seq:-?}" "$payload"
|
|
fi
|
|
|
|
echo "beacon.sh: ALIVE — last beacon seq ${last_seq:-?} for host '$host_id' is ${age}s old (<= SLO ${slo}s)."
|
|
}
|
|
|
|
cmd_status() {
|
|
local seq
|
|
seq="$(_wake_read_int "$SEQ_FILE" 0)"
|
|
printf 'beacon_emitter_seq=%s\n' "$seq"
|
|
if [ -f "$RECEIVED_FILE" ] && [ -s "$RECEIVED_FILE" ]; then
|
|
printf 'last_received='
|
|
cat "$RECEIVED_FILE"
|
|
else
|
|
printf 'last_received=(none)\n'
|
|
fi
|
|
}
|
|
|
|
main() {
|
|
[ $# -ge 1 ] || {
|
|
usage
|
|
exit 2
|
|
}
|
|
local cmd="$1"
|
|
shift
|
|
case "$cmd" in
|
|
emit) cmd_emit "$@" ;;
|
|
record) cmd_record "$@" ;;
|
|
check) cmd_check "$@" ;;
|
|
status) cmd_status "$@" ;;
|
|
-h | --help | help) usage ;;
|
|
*)
|
|
echo "beacon.sh: unknown command '$cmd'" >&2
|
|
usage
|
|
exit 2
|
|
;;
|
|
esac
|
|
}
|
|
|
|
main "$@"
|