feat(comms): P1 presence — minimal Synapse + fleet presence room + mosaic.presence heartbeat + liveness
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Implements RFC-001 P1 (the first shippable slice): deterministic Matrix presence/liveness on a single-instance dev Synapse. - infra/matrix/: DEV Synapse (RFC-002 Mode B, federation OFF, self-signed TLS, enable_registration:false, appservice registration wired). Rendered from parameterized templates — zero hardcoded topology. .data is gitignored. - packages/comms/: minimal MACP presence SDK — set presence, run the mosaic.presence heartbeat (seq + interval per RFC-001 §4.5), and a deterministic liveness reader (online/away/offline from heartbeat age, NOT native-presence-timeout). Liveness core written RED-FIRST. 19 vitest tests. - tools/matrix-presence-harness/: minimal provisioner (registers >=3 agent MXIDs, creates the fleet presence room, joins them — reuses the existing @mosaicstack/appservice intent lib) + an E2E validation harness proving A2/A3/A4 against a real Synapse. - eslint.config.mjs: register packages/comms/vitest.config.ts with the type-aware project service (same as other packages' vitest configs). DEV-compose validated only; production deploy is a separate coordinated step (deploy-holds respected). Part of the comms-evolution program (RFC-001 P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
This commit is contained in:
43
tools/matrix-presence-harness/README.md
Normal file
43
tools/matrix-presence-harness/README.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# tools/matrix-presence-harness — RFC-001 P1 validation (A1–A5)
|
||||
|
||||
> **DEV-ONLY. LOCAL SANDBOX.** Drives the local `infra/matrix` Synapse only.
|
||||
> Never point it at live infra.
|
||||
|
||||
The dev validation harness for the presence slice. It is the **minimal P1
|
||||
provisioner** + an end-to-end demo that proves the acceptance criteria against
|
||||
a real Synapse.
|
||||
|
||||
## Pieces
|
||||
|
||||
- `provision.ts` — the **minimal provisioner**: registers ≥3 agent MXIDs, creates
|
||||
the fleet presence room, joins the agents. Reuses the existing tested
|
||||
`@mosaicstack/appservice` intent library (register / createRoom / join). It is
|
||||
deliberately **not** the P2 appservice (no auto-enroll / taxonomy / token minting).
|
||||
- `agent-proc.ts` — one presence agent as its **own OS process** (via
|
||||
`@mosaicstack/comms`): joins the fleet room and heartbeats `mosaic.presence`.
|
||||
Standalone so the harness can `SIGKILL` it for a true hard-kill (A3).
|
||||
- `validate.ts` — provisions, spawns the agents, then asserts:
|
||||
- **A2** ≥3 agents show **online** in the fleet room.
|
||||
- **A3** a `SIGKILL`'d agent flips to **offline within `dark_threshold`**,
|
||||
deterministically (heartbeat-age based, not native-presence timeout), while
|
||||
survivors stay online.
|
||||
- **A4** prints the human-readable fleet liveness board.
|
||||
- `run.sh` — wires dev secrets + env and runs `validate.ts` over the self-signed
|
||||
TLS endpoint (A1). Exits non-zero on any failed assertion.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
../../infra/matrix/dev-up.sh # bring Synapse up first
|
||||
./run.sh # provision + validate (A2/A3/A4)
|
||||
```
|
||||
|
||||
Tunables (env): `HEARTBEAT_INTERVAL_MS` (1000), `MISS_TOLERANCE` (2),
|
||||
`DARK_THRESHOLD_MS` (6000), `AGENT_SLUGS` (`alpha,bravo,charlie`),
|
||||
`VICTIM_SLUG` (`charlie`), `MATRIX_CS_URL` (defaults to the TLS endpoint).
|
||||
|
||||
## Note on `NODE_TLS_REJECT_UNAUTHORIZED=0`
|
||||
|
||||
`run.sh` sets this **only** because the dev homeserver uses a self-signed cert.
|
||||
It is a dev convenience for exercising the SDK over TLS and must never be used
|
||||
against a real CA-issued endpoint.
|
||||
70
tools/matrix-presence-harness/agent-proc.ts
Normal file
70
tools/matrix-presence-harness/agent-proc.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* tools/matrix-presence-harness/agent-proc.ts
|
||||
*
|
||||
* *** DEV harness — LOCAL SANDBOX ONLY. ***
|
||||
*
|
||||
* Runs ONE presence agent as its own OS process: joins the fleet presence
|
||||
* room and heartbeats `mosaic.presence` via @mosaicstack/comms (RFC-001 P1).
|
||||
* Kept as a standalone process so the validation harness can `kill -9` it to
|
||||
* prove A3 (hard-kill -> offline within dark_threshold) against a real crash,
|
||||
* not a graceful shutdown.
|
||||
*
|
||||
* Auth in DEV is Application-Service masquerade: the process presents the
|
||||
* as_token and acts as its own @agent-<slug> MXID. (Per-agent minted tokens
|
||||
* are P2, RFC-001 §2.2/§8.)
|
||||
*
|
||||
* ESM / NodeNext: .js import extensions.
|
||||
*/
|
||||
import { MinimalMatrixClient, PresenceAgent, type LivenessPolicy } from '@mosaicstack/comms';
|
||||
|
||||
const env = (k: string, fallback?: string): string => {
|
||||
const v = process.env[k] ?? fallback;
|
||||
if (v === undefined) throw new Error(`missing env ${k}`);
|
||||
return v;
|
||||
};
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const homeserverUrl = env('MATRIX_CS_URL');
|
||||
const asToken = env('MOSAIC_AS_TOKEN');
|
||||
const serverName = env('MATRIX_SERVER_NAME');
|
||||
const slug = env('AGENT_SLUG');
|
||||
const roomId = env('FLEET_ROOM_ID');
|
||||
const intervalMs = Number(env('HEARTBEAT_INTERVAL_MS', '1000'));
|
||||
|
||||
const mxid = `@agent-${slug}:${serverName}`;
|
||||
const policy: LivenessPolicy = {
|
||||
heartbeatIntervalMs: intervalMs,
|
||||
missTolerance: Number(env('MISS_TOLERANCE', '2')),
|
||||
darkThresholdMs: Number(env('DARK_THRESHOLD_MS', '6000')),
|
||||
};
|
||||
|
||||
const client = new MinimalMatrixClient({
|
||||
homeserverUrl,
|
||||
accessToken: asToken,
|
||||
actAsUserId: mxid,
|
||||
});
|
||||
const agent = new PresenceAgent({
|
||||
client,
|
||||
agent: { mxid, slug, harness: 'claude-code' },
|
||||
roomId,
|
||||
intervalMs,
|
||||
policy,
|
||||
onError: (err) => console.error(`[${slug}] heartbeat error:`, (err as Error).message),
|
||||
});
|
||||
|
||||
await agent.connect();
|
||||
agent.start();
|
||||
console.log(`[${slug}] pid=${process.pid} mxid=${mxid} heartbeating every ${intervalMs}ms`);
|
||||
|
||||
// Graceful stop only on SIGTERM; the A3 test uses SIGKILL (no cleanup runs).
|
||||
process.on('SIGTERM', () => {
|
||||
void agent.stop().finally(() => process.exit(0));
|
||||
});
|
||||
// Keep the event loop alive indefinitely.
|
||||
setInterval(() => {}, 1 << 30);
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error('agent-proc fatal:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
24
tools/matrix-presence-harness/package.json
Normal file
24
tools/matrix-presence-harness/package.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"name": "@mosaicstack/matrix-presence-harness",
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"private": true,
|
||||
"description": "DEV-ONLY validation harness for RFC-001 P1 (Matrix presence). Not shipped.",
|
||||
"scripts": {
|
||||
"build": "tsc --noEmit",
|
||||
"lint": "eslint .",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "vitest run --passWithNoTests",
|
||||
"validate": "./run.sh"
|
||||
},
|
||||
"dependencies": {
|
||||
"@mosaicstack/appservice": "workspace:*",
|
||||
"@mosaicstack/comms": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.0.0",
|
||||
"tsx": "^4.19.0",
|
||||
"typescript": "^5.8.0",
|
||||
"vitest": "^2.0.0"
|
||||
}
|
||||
}
|
||||
89
tools/matrix-presence-harness/provision.ts
Normal file
89
tools/matrix-presence-harness/provision.ts
Normal file
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* tools/matrix-presence-harness/provision.ts
|
||||
*
|
||||
* *** DEV harness — LOCAL SANDBOX ONLY. ***
|
||||
*
|
||||
* The MINIMAL P1 provisioner (RFC-001 §10 P1): its ONLY job is to register a
|
||||
* few agent MXIDs, create the fleet presence room, and join the agents. It is
|
||||
* deliberately NOT the P2 appservice (no auto-enroll, no room taxonomy, no
|
||||
* token minting, no introductions).
|
||||
*
|
||||
* It reuses the existing, tested `@mosaicstack/appservice` core library
|
||||
* (AppserviceIntent: register / createRoom / join over the AS API) rather than
|
||||
* reinventing Matrix plumbing.
|
||||
*
|
||||
* ESM / NodeNext: .js import extensions.
|
||||
*/
|
||||
import { AppserviceIntent, type AppserviceConfig } from '@mosaicstack/appservice';
|
||||
|
||||
export interface ProvisionOptions {
|
||||
homeserverUrl: string;
|
||||
serverName: string;
|
||||
asToken: string;
|
||||
hsToken: string;
|
||||
agentSlugs: string[];
|
||||
fleetAlias?: string; // localpart, default "mosaic-fleet"
|
||||
}
|
||||
|
||||
export interface ProvisionResult {
|
||||
roomId: string;
|
||||
senderUserId: string;
|
||||
agents: { slug: string; mxid: string }[];
|
||||
}
|
||||
|
||||
/** Register agents, create the fleet presence room, join everyone. Idempotent. */
|
||||
export async function provision(opts: ProvisionOptions): Promise<ProvisionResult> {
|
||||
const cfg: AppserviceConfig = {
|
||||
homeserverUrl: opts.homeserverUrl,
|
||||
domain: opts.serverName,
|
||||
asToken: opts.asToken,
|
||||
hsToken: opts.hsToken,
|
||||
};
|
||||
const intent = new AppserviceIntent(cfg);
|
||||
const fleetAlias = opts.fleetAlias ?? 'mosaic-fleet';
|
||||
|
||||
// 1. Register the virtual agent MXIDs (bypasses enable_registration:false).
|
||||
const agents = [];
|
||||
for (const slug of opts.agentSlugs) {
|
||||
const mxid = await intent.ensureRegistered(slug);
|
||||
await intent.setDisplayName(slug, `agent ${slug} (DEV)`);
|
||||
agents.push({ slug, mxid });
|
||||
}
|
||||
|
||||
// 2. Create the single fleet presence room, inviting all agents (RFC-001 §4.6).
|
||||
// Idempotent: if the alias already exists, reuse that room instead.
|
||||
let roomId: string;
|
||||
try {
|
||||
const created = await intent.createRoom({
|
||||
name: 'Fleet Presence (DEV)',
|
||||
alias: fleetAlias,
|
||||
topic: 'RFC-001 P1 — mosaic.presence heartbeats. Who is alive?',
|
||||
invite: agents.map((a) => a.mxid),
|
||||
});
|
||||
roomId = created.roomId;
|
||||
} catch (err) {
|
||||
// Alias already taken (re-run): resolve the existing room id.
|
||||
const aliasFq = `#${fleetAlias}:${opts.serverName}`;
|
||||
roomId = await resolveAlias(opts, aliasFq);
|
||||
void err;
|
||||
}
|
||||
|
||||
// 3. Join every agent into the fleet room (invite + join; idempotent).
|
||||
for (const { slug } of agents) {
|
||||
await intent.ensureJoined(roomId, slug);
|
||||
}
|
||||
|
||||
return { roomId, senderUserId: intent.senderUserId, agents };
|
||||
}
|
||||
|
||||
async function resolveAlias(
|
||||
opts: Pick<ProvisionOptions, 'homeserverUrl' | 'asToken'>,
|
||||
aliasFq: string,
|
||||
): Promise<string> {
|
||||
const url = `${opts.homeserverUrl.replace(/\/$/, '')}/_matrix/client/v3/directory/room/${encodeURIComponent(aliasFq)}`;
|
||||
const res = await fetch(url, { headers: { Authorization: `Bearer ${opts.asToken}` } });
|
||||
if (!res.ok) throw new Error(`resolveAlias ${aliasFq} -> ${res.status}`);
|
||||
const data = (await res.json()) as { room_id?: string };
|
||||
if (!data.room_id) throw new Error(`resolveAlias ${aliasFq} returned no room_id`);
|
||||
return data.room_id;
|
||||
}
|
||||
47
tools/matrix-presence-harness/run.sh
Executable file
47
tools/matrix-presence-harness/run.sh
Executable file
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================================
|
||||
# run.sh — RFC-001 P1 presence validation harness (A1–A5) *** DEV-ONLY ***
|
||||
# ============================================================================
|
||||
#
|
||||
# Assumes the dev Synapse is already up (infra/matrix/dev-up.sh). Provisions 3
|
||||
# agents, heartbeats them, proves A2/A3/A4 against the LOCAL dev homeserver,
|
||||
# exercising @mosaicstack/comms over the self-signed TLS endpoint (A1).
|
||||
#
|
||||
# NODE_TLS_REJECT_UNAUTHORIZED=0 is set ONLY because the dev cert is
|
||||
# self-signed. This is a DEV convenience and must never be used in prod.
|
||||
# ----------------------------------------------------------------------------
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "${HERE}/../.." && pwd)"
|
||||
DATA="${REPO}/infra/matrix/.data"
|
||||
|
||||
if [[ ! -f "${DATA}/dev-secrets.env" ]]; then
|
||||
echo "run.sh: ${DATA}/dev-secrets.env not found — run infra/matrix/dev-up.sh first" >&2
|
||||
exit 1
|
||||
fi
|
||||
# shellcheck disable=SC1091
|
||||
set -a; source "${DATA}/dev-secrets.env"; set +a
|
||||
|
||||
export MATRIX_SERVER_NAME="${MATRIX_SERVER_NAME:-matrix.localhost}"
|
||||
export MATRIX_TLS_PORT="${MATRIX_TLS_PORT:-18448}"
|
||||
# Exercise the SDK over the self-signed TLS listener (A1). Use the plain HTTP
|
||||
# port instead by exporting MATRIX_CS_URL=http://127.0.0.1:18008 before running.
|
||||
export MATRIX_CS_URL="${MATRIX_CS_URL:-https://127.0.0.1:${MATRIX_TLS_PORT}}"
|
||||
export NODE_TLS_REJECT_UNAUTHORIZED=0
|
||||
|
||||
export HEARTBEAT_INTERVAL_MS="${HEARTBEAT_INTERVAL_MS:-1000}"
|
||||
export MISS_TOLERANCE="${MISS_TOLERANCE:-2}"
|
||||
export DARK_THRESHOLD_MS="${DARK_THRESHOLD_MS:-6000}"
|
||||
export AGENT_SLUGS="${AGENT_SLUGS:-alpha,bravo,charlie}"
|
||||
export VICTIM_SLUG="${VICTIM_SLUG:-charlie}"
|
||||
|
||||
TSX_CLI="$(ls -d "${REPO}"/node_modules/.pnpm/tsx@*/node_modules/tsx/dist/cli.mjs 2>/dev/null | head -1)"
|
||||
if [[ -z "${TSX_CLI}" ]]; then
|
||||
echo "run.sh: tsx not found under node_modules — run pnpm install first" >&2
|
||||
exit 1
|
||||
fi
|
||||
export TSX_CLI
|
||||
|
||||
echo "[run] CS=${MATRIX_CS_URL} server_name=${MATRIX_SERVER_NAME} dark_threshold=${DARK_THRESHOLD_MS}ms"
|
||||
cd "${REPO}"
|
||||
exec node "${TSX_CLI}" "${HERE}/validate.ts"
|
||||
9
tools/matrix-presence-harness/tsconfig.json
Normal file
9
tools/matrix-presence-harness/tsconfig.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"noEmit": true,
|
||||
"rootDir": "."
|
||||
},
|
||||
"include": ["*.ts"],
|
||||
"exclude": ["node_modules"]
|
||||
}
|
||||
210
tools/matrix-presence-harness/validate.ts
Normal file
210
tools/matrix-presence-harness/validate.ts
Normal file
@@ -0,0 +1,210 @@
|
||||
/**
|
||||
* tools/matrix-presence-harness/validate.ts
|
||||
*
|
||||
* *** DEV harness — LOCAL SANDBOX ONLY. Never point at live infra. ***
|
||||
*
|
||||
* End-to-end proof of RFC-001 P1 acceptance criteria against the local dev
|
||||
* Synapse (infra/matrix):
|
||||
*
|
||||
* A2 provision >=3 agents, they heartbeat, all show ONLINE in the fleet
|
||||
* presence room.
|
||||
* A3 hard-kill (SIGKILL) one agent's process; assert it flips to OFFLINE
|
||||
* within dark_threshold, DETERMINISTICALLY (heartbeat-age based, not
|
||||
* native-presence-timeout), while the survivors stay ONLINE.
|
||||
* A4 dump the human-readable fleet liveness board.
|
||||
*
|
||||
* Exits 0 on success, 1 on any failed assertion.
|
||||
* ESM / NodeNext: .js import extensions.
|
||||
*/
|
||||
import { spawn, type ChildProcess } from 'node:child_process';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
|
||||
import {
|
||||
FleetLivenessReader,
|
||||
MinimalMatrixClient,
|
||||
type AgentLiveness,
|
||||
type LivenessPolicy,
|
||||
} from '@mosaicstack/comms';
|
||||
|
||||
import { provision } from './provision.js';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
const env = (k: string, fallback?: string): string => {
|
||||
const v = process.env[k] ?? fallback;
|
||||
if (v === undefined) throw new Error(`missing env ${k}`);
|
||||
return v;
|
||||
};
|
||||
|
||||
const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
|
||||
const board = (rows: AgentLiveness[]): Record<string, string> =>
|
||||
Object.fromEntries(rows.map((r) => [r.slug, r.status]));
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const homeserverUrl = env('MATRIX_CS_URL');
|
||||
const asToken = env('MOSAIC_AS_TOKEN');
|
||||
const hsToken = env('MOSAIC_HS_TOKEN', 'unused-in-p1');
|
||||
const serverName = env('MATRIX_SERVER_NAME');
|
||||
const intervalMs = Number(env('HEARTBEAT_INTERVAL_MS', '1000'));
|
||||
const missTolerance = Number(env('MISS_TOLERANCE', '2'));
|
||||
const darkThresholdMs = Number(env('DARK_THRESHOLD_MS', '6000'));
|
||||
const slugs = env('AGENT_SLUGS', 'alpha,bravo,charlie').split(',');
|
||||
const victim = env('VICTIM_SLUG', 'charlie');
|
||||
|
||||
const policy: LivenessPolicy = {
|
||||
heartbeatIntervalMs: intervalMs,
|
||||
missTolerance,
|
||||
darkThresholdMs,
|
||||
};
|
||||
const failures: string[] = [];
|
||||
const assert = (ok: boolean, msg: string): void => {
|
||||
console.log(` ${ok ? 'PASS' : 'FAIL'} ${msg}`);
|
||||
if (!ok) failures.push(msg);
|
||||
};
|
||||
|
||||
// ── Provision (register agents, create fleet room, join) ──────────────────
|
||||
console.log(`\n[validate] provisioning ${slugs.length} agents + fleet presence room...`);
|
||||
const { roomId, senderUserId } = await provision({
|
||||
homeserverUrl,
|
||||
serverName,
|
||||
asToken,
|
||||
hsToken,
|
||||
agentSlugs: slugs,
|
||||
});
|
||||
console.log(`[validate] fleet room ${roomId} (sender ${senderUserId})`);
|
||||
|
||||
// Reader masquerades as the AS sender (a room member) to read heartbeats.
|
||||
const reader = new FleetLivenessReader({
|
||||
client: new MinimalMatrixClient({
|
||||
homeserverUrl,
|
||||
accessToken: asToken,
|
||||
actAsUserId: senderUserId,
|
||||
}),
|
||||
roomId,
|
||||
policy,
|
||||
});
|
||||
|
||||
// ── Spawn each agent as its own OS process ────────────────────────────────
|
||||
const procs = new Map<string, ChildProcess>();
|
||||
for (const slug of slugs) {
|
||||
// `--import tsx` runs agent-proc.ts IN-PROCESS (no tsx grandchild), so the
|
||||
// spawned pid IS the agent — a SIGKILL to it is a true hard-kill (A3).
|
||||
// cwd=HERE so the `tsx` specifier resolves against the harness node_modules.
|
||||
const child = spawn('node', ['--import', 'tsx', resolve(HERE, 'agent-proc.ts')], {
|
||||
cwd: HERE,
|
||||
env: {
|
||||
...process.env,
|
||||
AGENT_SLUG: slug,
|
||||
FLEET_ROOM_ID: roomId,
|
||||
MATRIX_CS_URL: homeserverUrl,
|
||||
MOSAIC_AS_TOKEN: asToken,
|
||||
MATRIX_SERVER_NAME: serverName,
|
||||
HEARTBEAT_INTERVAL_MS: String(intervalMs),
|
||||
MISS_TOLERANCE: String(missTolerance),
|
||||
DARK_THRESHOLD_MS: String(darkThresholdMs),
|
||||
},
|
||||
stdio: ['ignore', 'inherit', 'inherit'],
|
||||
});
|
||||
procs.set(slug, child);
|
||||
}
|
||||
|
||||
const hardKill = (pid: number): void => {
|
||||
try {
|
||||
process.kill(pid, 'SIGKILL'); // pid is the in-process agent — a true kill
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
};
|
||||
const cleanup = (): void => {
|
||||
for (const [, c] of procs) {
|
||||
if (c.pid) hardKill(c.pid);
|
||||
}
|
||||
};
|
||||
process.on('exit', cleanup);
|
||||
|
||||
try {
|
||||
// ── A2: let a few beats flow, assert all online ─────────────────────────
|
||||
console.log(`\n[validate] A2 — waiting for heartbeats (${intervalMs}ms interval)...`);
|
||||
await sleep(intervalMs * 4 + 1000);
|
||||
const a2 = await reader.read();
|
||||
console.log('[validate] board:', board(a2));
|
||||
for (const slug of slugs) {
|
||||
assert(a2.find((r) => r.slug === slug)?.status === 'online', `A2 ${slug} is online`);
|
||||
}
|
||||
assert(a2.length >= 3, 'A2 >=3 agents present in fleet room');
|
||||
|
||||
// ── A4: human-readable board ────────────────────────────────────────────
|
||||
console.log('\n[validate] A4 — fleet liveness board (human view):');
|
||||
console.log(await reader.formatBoard());
|
||||
|
||||
// ── A3: hard-kill the victim, measure flip-to-offline latency ───────────
|
||||
const victimProc = procs.get(victim);
|
||||
if (!victimProc?.pid) throw new Error(`no pid for victim ${victim}`);
|
||||
console.log(
|
||||
`\n[validate] A3 — SIGKILL ${victim} (pid ${victimProc.pid}); dark_threshold=${darkThresholdMs}ms`,
|
||||
);
|
||||
const killTs = Date.now();
|
||||
hardKill(victimProc.pid);
|
||||
|
||||
let victimOffline: AgentLiveness | undefined;
|
||||
let detectMs = 0;
|
||||
const deadline = killTs + darkThresholdMs + 6000;
|
||||
while (Date.now() < deadline) {
|
||||
await sleep(400);
|
||||
const rows = await reader.read();
|
||||
const v = rows.find((r) => r.slug === victim);
|
||||
const elapsed = Date.now() - killTs;
|
||||
if (v) {
|
||||
console.log(
|
||||
` t+${String(elapsed).padStart(5)}ms ${victim}=${v.status} (age=${v.ageMs}ms) ` +
|
||||
`survivors=${slugs
|
||||
.filter((s) => s !== victim)
|
||||
.map((s) => `${s}:${rows.find((r) => r.slug === s)?.status}`)
|
||||
.join(',')}`,
|
||||
);
|
||||
if (v.status === 'offline') {
|
||||
victimOffline = v;
|
||||
detectMs = elapsed;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
assert(!!victimOffline, `A3 ${victim} flipped to offline after kill`);
|
||||
if (victimOffline) {
|
||||
// Deterministic: the flip is driven by heartbeat age reaching dark_threshold.
|
||||
assert(
|
||||
victimOffline.ageMs >= darkThresholdMs,
|
||||
`A3 flip is heartbeat-deterministic (age ${victimOffline.ageMs}ms >= dark_threshold ${darkThresholdMs}ms)`,
|
||||
);
|
||||
assert(
|
||||
detectMs <= darkThresholdMs + 2000,
|
||||
`A3 detected within dark_threshold + poll slack (detected at t+${detectMs}ms)`,
|
||||
);
|
||||
}
|
||||
// Survivors unaffected.
|
||||
const finalRows = await reader.read();
|
||||
for (const slug of slugs.filter((s) => s !== victim)) {
|
||||
assert(
|
||||
finalRows.find((r) => r.slug === slug)?.status === 'online',
|
||||
`A3 survivor ${slug} still online`,
|
||||
);
|
||||
}
|
||||
|
||||
console.log('\n[validate] final board after kill:');
|
||||
console.log(await reader.formatBoard());
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
|
||||
console.log(
|
||||
`\n[validate] ${failures.length === 0 ? 'ALL PASSED ✅' : `FAILED ❌ (${failures.length})`}`,
|
||||
);
|
||||
process.exit(failures.length === 0 ? 0 : 1);
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error('[validate] fatal:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user