Compare commits
2 Commits
fix/h2-rea
...
fix/db-pgl
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7210b7391a | ||
|
|
80570f7040 |
@@ -1,66 +0,0 @@
|
|||||||
# H1 — heartbeat readiness detection
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
|
|
||||||
Add runtime-agnostic readiness classification to `mosaic fleet ps` so an agent can be reported as working/idle/stuck/stale/dead/unknown instead of treating pane liveness as progress.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
- `packages/mosaic/src/commands/fleet.ts`
|
|
||||||
- exported readiness state/types/default thresholds/helpers/classifier
|
|
||||||
- `AgentPsRow.readiness` additive JSON field
|
|
||||||
- table HB column and IDLE/STUCK flags
|
|
||||||
- `packages/mosaic/src/commands/fleet.spec.ts`
|
|
||||||
- pure classifier branch/boundary coverage
|
|
||||||
- threshold helper coverage
|
|
||||||
- legitimate render/JSON assertion updates for new HB text
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
- Branches covered: dead, unknown, stale, busy working, null-idle working, stuck boundary, idle boundary, working below idle.
|
|
||||||
- Threshold env helpers default to 300s/900s and honor positive integer env values.
|
|
||||||
- `fleet ps` rows populate `readiness` for roster and unmanaged socket sessions.
|
|
||||||
- Table HB text becomes `<age>s/<readiness>` when heartbeat age exists; remains `unknown` when absent.
|
|
||||||
- Flags include `IDLE`/`STUCK` for matching readiness.
|
|
||||||
- Local gates green: `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, fleet vitest.
|
|
||||||
- Pre-push queue guard passes; PR opened off `origin/main`; no merge by worker.
|
|
||||||
|
|
||||||
## Constraints / Assumptions
|
|
||||||
|
|
||||||
- Source branch: `origin/main` @ `e3adc6a`.
|
|
||||||
- No scope creep beyond readiness detection.
|
|
||||||
- `docs/TASKS.md` and `docs/fleet/TASKS.md` are orchestrator-owned; worker will not modify them.
|
|
||||||
- PRD alignment source: `docs/fleet/PRD.md` Phase 2 observability; this is a refinement of heartbeat observability, preserving existing unknown/stale behavior.
|
|
||||||
|
|
||||||
## Plan
|
|
||||||
|
|
||||||
1. Install dependencies with requested PNPM environment.
|
|
||||||
2. Add readiness types/helpers/classifier near heartbeat constants.
|
|
||||||
3. Add `readiness` to `AgentPsRow` and populate both row paths.
|
|
||||||
4. Update table render and flags.
|
|
||||||
5. Add unit tests and update affected ps render/JSON assertions.
|
|
||||||
6. Run build precheck + required gates.
|
|
||||||
7. Run automated independent review, remediate findings.
|
|
||||||
8. Queue guard, push, open PR.
|
|
||||||
|
|
||||||
## Progress
|
|
||||||
|
|
||||||
- 2026-06-24: Branch created from `origin/main` @ `e3adc6a`.
|
|
||||||
- 2026-06-24: Implemented readiness thresholds/classifier, JSON row field, HB column label, and IDLE/STUCK flags.
|
|
||||||
- 2026-06-24: Added classifier branch/boundary tests, threshold helper tests, JSON shape assertions, and readiness table rendering assertions.
|
|
||||||
|
|
||||||
## Verification Evidence
|
|
||||||
|
|
||||||
- `pnpm install --store-dir "$HOME/.pnpm-store"` — pass.
|
|
||||||
- `npx turbo build --filter=@mosaicstack/mosaic^...` — pass, 12/12 tasks successful.
|
|
||||||
- `pnpm typecheck` — pass, 41/41 tasks successful.
|
|
||||||
- `pnpm lint` — pass, 23/23 tasks successful.
|
|
||||||
- `pnpm format:check` — pass, all matched files use Prettier style.
|
|
||||||
- `pnpm --filter @mosaicstack/mosaic exec vitest run src/commands/fleet.spec.ts` — pass, 171 tests.
|
|
||||||
- `pnpm --filter @mosaicstack/mosaic test` — pass, 39 files / 547 tests; `fleet.spec.ts` 171 tests.
|
|
||||||
- `~/.config/mosaic/tools/codex/codex-code-review.sh --uncommitted` — approve, 0 findings (reviewed supplied diff; sandbox file-inspection limitation noted by tool).
|
|
||||||
|
|
||||||
## Risks / Blockers
|
|
||||||
|
|
||||||
- No current blocker.
|
|
||||||
- Review tool could not inspect repo files directly due sandbox wrapper limitation, but it reviewed the supplied diff and approved with no findings.
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
# H1b — tmux pane idle signal wiring
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
|
|
||||||
Feed `classifyReadiness()` a real idle signal on tmux 3.4 by deriving `idleSeconds` from the first available tmux timestamp source: pane activity, then window activity, then session activity.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
- `packages/mosaic/src/commands/fleet.ts`
|
|
||||||
- Extend `buildTmuxListPanesCommand()` format to include `#{window_activity}` and `#{session_activity}` after the existing fields.
|
|
||||||
- Update `parseTmuxListPanes()` to choose the first non-empty finite positive timestamp and clamp future idle values to 0.
|
|
||||||
- `packages/mosaic/src/commands/fleet.spec.ts`
|
|
||||||
- Cover pane/window/session activity parsing behavior, empty-field index alignment, null idle, future clamping, math correctness, and exact tmux format.
|
|
||||||
|
|
||||||
## Out of Scope
|
|
||||||
|
|
||||||
- No changes to `classifyReadiness()`, thresholds, `AgentPsRow`, or `fleet ps` rendering.
|
|
||||||
- No merge by worker; orchestrator routes review/merge.
|
|
||||||
- Workers do not modify `docs/TASKS.md`.
|
|
||||||
|
|
||||||
## PRD Alignment
|
|
||||||
|
|
||||||
Aligned with `docs/fleet/PRD.md` FR-1 and acceptance criteria for truthful `mosaic fleet ps` pane/pid/idle observability.
|
|
||||||
|
|
||||||
## Plan
|
|
||||||
|
|
||||||
1. Sync branch from latest `origin/main` and install dependencies with required pnpm env.
|
|
||||||
2. Add/confirm reproducer tests for tmux 3.4 empty `pane_activity` and new fallback behavior.
|
|
||||||
3. Implement the focused parser/format change only.
|
|
||||||
4. Run required build, baseline gates, fleet vitest, and independent review.
|
|
||||||
5. Run pre-push queue guard, push branch, and open PR to `main` with Mosaic wrapper.
|
|
||||||
|
|
||||||
## Progress
|
|
||||||
|
|
||||||
- 2026-06-24: Branch `fix/fleet-pane-idle-activity` created from `origin/main` @ `ec8dd7c` after fetching.
|
|
||||||
- 2026-06-24: Session-start generated local `.mosaic/orchestrator/*` changes on the previous release branch; stashed as `coder1 session-start state before H1b` to keep this branch clean.
|
|
||||||
- 2026-06-24: Added TDD coverage for the tmux 3.4 production case (`pane_activity` empty, `window_activity` populated), exact new list-panes format, null/future/multiple-source behavior.
|
|
||||||
- 2026-06-24: Implemented parser fallback without changing readiness classifier thresholds or render shape.
|
|
||||||
|
|
||||||
## Verification Evidence
|
|
||||||
|
|
||||||
- `pnpm install --store-dir "$HOME/.pnpm-store"` — pass.
|
|
||||||
- Reproducer before implementation: `pnpm --filter @mosaicstack/mosaic exec vitest run src/commands/fleet.spec.ts` — failed as expected (old format, no fallback, negative future idle).
|
|
||||||
- `npx turbo build --filter=@mosaicstack/mosaic^...` — pass, 12/12 tasks successful.
|
|
||||||
- `pnpm typecheck` — pass, 41/41 tasks successful.
|
|
||||||
- `pnpm lint` — pass, 23/23 tasks successful.
|
|
||||||
- `pnpm format:check` — pass, all matched files use Prettier style.
|
|
||||||
- `pnpm --filter @mosaicstack/mosaic exec vitest run src/commands/fleet.spec.ts` — pass, 176 tests.
|
|
||||||
- `~/.config/mosaic/tools/codex/codex-code-review.sh --uncommitted` — approve, 0 findings (reviewed supplied diff; sandbox file-inspection limitation noted by tool).
|
|
||||||
|
|
||||||
## Risks / Blockers
|
|
||||||
|
|
||||||
- No current blocker.
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
# H2 — readiness semantics: available, not stuck
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
|
|
||||||
Correct fleet readiness semantics so a healthy long-idle agent is reported as `available` (good/assignable) instead of `stuck` (fault). Reserve `stuck` in the type/JSON value space for future positive block evidence.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
- `packages/mosaic/src/commands/fleet.ts`
|
|
||||||
- replace `idle` readiness state with `available`
|
|
||||||
- keep `stuck` in the union but stop emitting it from idle-only heuristics
|
|
||||||
- remove stuck threshold helper/env handling
|
|
||||||
- remove IDLE/STUCK alarm flags from table rendering
|
|
||||||
- `packages/mosaic/src/commands/fleet.spec.ts`
|
|
||||||
- update classifier branch/boundary tests
|
|
||||||
- assert very long idle maps to `available`, not `stuck`
|
|
||||||
- update table/JSON assertions for available with no alarm flags
|
|
||||||
- remove stuck threshold helper tests
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
- `classifyReadiness()` remains pure/total/never-throw and maps:
|
|
||||||
- dead/stale/unknown unchanged
|
|
||||||
- busy/null/undefined/non-finite idle to `working`
|
|
||||||
- idle >= activity threshold to `available`
|
|
||||||
- idle < activity threshold to `working`
|
|
||||||
- No idle-derived path emits `stuck`.
|
|
||||||
- `MOSAIC_HEARTBEAT_IDLE_THRESHOLD` remains backward compatible as the working→available activity threshold.
|
|
||||||
- `MOSAIC_HEARTBEAT_STUCK_THRESHOLD` and helper/default are removed.
|
|
||||||
- `fleet ps` keeps the idle-seconds column header `IDLE`, renders `available` in HB label, and does not add IDLE/STUCK warning flags.
|
|
||||||
- Local gates green: build precheck, typecheck, lint, format:check, fleet vitest.
|
|
||||||
- PR opened against `main`; no merge by worker.
|
|
||||||
|
|
||||||
## Constraints / Assumptions
|
|
||||||
|
|
||||||
- Source branch: `origin/main` @ `1020cfa`.
|
|
||||||
- `docs/TASKS.md` is orchestrator-owned; worker will not modify it.
|
|
||||||
- Documentation impact is captured in this scratchpad and PR description; no user/admin guide behavior beyond CLI readiness label semantics.
|
|
||||||
|
|
||||||
## Plan
|
|
||||||
|
|
||||||
1. Install dependencies with requested PNPM environment.
|
|
||||||
2. Inspect current H1/H1b readiness implementation and tests.
|
|
||||||
3. Update classifier types/helpers/rendering.
|
|
||||||
4. Update focused tests.
|
|
||||||
5. Run build precheck + required gates.
|
|
||||||
6. Run automated code review, remediate any findings.
|
|
||||||
7. Queue guard, push, open PR.
|
|
||||||
|
|
||||||
## Progress
|
|
||||||
|
|
||||||
- 2026-06-24: Branch created from `origin/main` @ `1020cfa`.
|
|
||||||
- 2026-06-24: Replaced idle-derived `idle`/`stuck` outputs with `available`; retained `stuck` in type union for future positive block evidence.
|
|
||||||
- 2026-06-24: Removed stuck threshold env/helper plumbing and IDLE/STUCK alarm flags.
|
|
||||||
- 2026-06-24: Updated classifier and table-render tests for available semantics.
|
|
||||||
|
|
||||||
## Verification Evidence
|
|
||||||
|
|
||||||
- `pnpm install --store-dir "$HOME/.pnpm-store"` — pass.
|
|
||||||
- `npx turbo build --filter=@mosaicstack/mosaic^...` — pass, 12/12 tasks successful.
|
|
||||||
- `pnpm typecheck` — pass, 41/41 tasks successful.
|
|
||||||
- `pnpm lint` — pass, 23/23 tasks successful.
|
|
||||||
- `pnpm format:check` — pass, all matched files use Prettier style.
|
|
||||||
- `pnpm --filter @mosaicstack/mosaic exec vitest run src/commands/fleet.spec.ts` — pass, 177 tests.
|
|
||||||
- `~/.config/mosaic/tools/codex/codex-code-review.sh --uncommitted` — approve, 0 findings (reviewed supplied diff; sandbox file-inspection limitation noted by tool).
|
|
||||||
|
|
||||||
## Risks / Blockers
|
|
||||||
|
|
||||||
- No current blocker.
|
|
||||||
- Review tool could not inspect repo files directly due sandbox wrapper limitation, but it reviewed the supplied diff and approved with no findings.
|
|
||||||
@@ -122,85 +122,6 @@ fi
|
|||||||
|
|
||||||
mkdir -p "$MOSAIC_AGENT_WORKDIR"
|
mkdir -p "$MOSAIC_AGENT_WORKDIR"
|
||||||
|
|
||||||
# ── Pre-trust the workdir for the Claude runtime ─────────────────────────────
|
|
||||||
# Claude Code shows a one-time "Is this a project you trust?" folder-trust gate
|
|
||||||
# the first time it opens a directory. A fleet-launched agent has no human to
|
|
||||||
# answer it, so the pane stalls forever at the prompt while its heartbeat keeps
|
|
||||||
# reporting "healthy" (the pane process IS alive — it's just blocked).
|
|
||||||
#
|
|
||||||
# IMPORTANT: --dangerously-skip-permissions does NOT bypass this gate, and
|
|
||||||
# neither does `trustedProjectDirectories` in settings.json (verified empirically
|
|
||||||
# 2026-06-24). The ONLY thing the gate honors is the per-project record in
|
|
||||||
# ~/.claude.json: projects["<dir>"].hasTrustDialogAccepted == true (exactly what
|
|
||||||
# answering the prompt writes). So we pre-seed that record here.
|
|
||||||
#
|
|
||||||
# Idempotent, atomic, best-effort: any failure is non-fatal (the agent still
|
|
||||||
# launches — worst case it stalls on the gate, i.e. the pre-fix status quo).
|
|
||||||
# Only the claude runtime needs this; codex/pi have no such gate.
|
|
||||||
_ensure_claude_workdir_trusted() {
|
|
||||||
local workdir="$1"
|
|
||||||
# The path claude keys on is the resolved cwd it is launched in.
|
|
||||||
local rp
|
|
||||||
rp=$(cd "$workdir" 2>/dev/null && pwd -P) || rp="$workdir"
|
|
||||||
# ~/.claude.json lives next to the claude config dir; honor CLAUDE_CONFIG_DIR.
|
|
||||||
local claude_json="${MOSAIC_CLAUDE_JSON:-${CLAUDE_CONFIG_DIR:+$CLAUDE_CONFIG_DIR/.claude.json}}"
|
|
||||||
claude_json="${claude_json:-$HOME/.claude.json}"
|
|
||||||
|
|
||||||
if ! command -v python3 >/dev/null 2>&1; then
|
|
||||||
echo "WARNING: python3 not found; cannot pre-trust '$rp' for claude (agent may stall on the folder-trust gate)" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Serialize concurrent agent launches that share ~/.claude.json (flock if available).
|
|
||||||
local lock="${claude_json}.mosaic-lock"
|
|
||||||
_seed() {
|
|
||||||
MOSAIC_CJ="$claude_json" MOSAIC_TRUST_DIR="$rp" python3 - <<'PY'
|
|
||||||
import json, os, sys, tempfile
|
|
||||||
cj = os.environ["MOSAIC_CJ"]
|
|
||||||
d = os.environ["MOSAIC_TRUST_DIR"]
|
|
||||||
try:
|
|
||||||
data = json.load(open(cj)) if os.path.exists(cj) else {}
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
data = {}
|
|
||||||
except Exception:
|
|
||||||
# Never corrupt an unreadable/partial file — bail without writing.
|
|
||||||
sys.exit(2)
|
|
||||||
projects = data.setdefault("projects", {})
|
|
||||||
entry = projects.get(d)
|
|
||||||
if not isinstance(entry, dict):
|
|
||||||
entry = {}
|
|
||||||
projects[d] = entry
|
|
||||||
if entry.get("hasTrustDialogAccepted") is True:
|
|
||||||
sys.exit(0) # already trusted — nothing to do
|
|
||||||
entry["hasTrustDialogAccepted"] = True
|
|
||||||
tmp_dir = os.path.dirname(cj) or "."
|
|
||||||
fd, tmp = tempfile.mkstemp(dir=tmp_dir, prefix=".claude.json.mosaic.")
|
|
||||||
try:
|
|
||||||
with os.fdopen(fd, "w") as f:
|
|
||||||
json.dump(data, f, indent=2)
|
|
||||||
os.replace(tmp, cj) # atomic
|
|
||||||
except Exception:
|
|
||||||
try:
|
|
||||||
os.unlink(tmp)
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
sys.exit(3)
|
|
||||||
PY
|
|
||||||
}
|
|
||||||
if command -v flock >/dev/null 2>&1; then
|
|
||||||
( flock 9; _seed ) 9>"$lock" 2>/dev/null || _seed
|
|
||||||
else
|
|
||||||
_seed
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
case "$MOSAIC_AGENT_RUNTIME" in
|
|
||||||
claude)
|
|
||||||
_ensure_claude_workdir_trusted "$MOSAIC_AGENT_WORKDIR" \
|
|
||||||
|| echo "WARNING: could not pre-trust workdir for claude agent $AGENT_NAME" >&2
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
# ── Launch the tmux session (no exec — we continue to wire the heartbeat) ────
|
# ── Launch the tmux session (no exec — we continue to wire the heartbeat) ────
|
||||||
_tmux new-session -d -s "$AGENT_NAME" -c "$MOSAIC_AGENT_WORKDIR" \
|
_tmux new-session -d -s "$AGENT_NAME" -c "$MOSAIC_AGENT_WORKDIR" \
|
||||||
bash -c "$PANE_SHELL_SNIPPET"
|
bash -c "$PANE_SHELL_SNIPPET"
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@mosaicstack/mosaic",
|
"name": "@mosaicstack/mosaic",
|
||||||
"version": "0.0.44",
|
"version": "0.0.41",
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
"url": "https://git.mosaicstack.dev/mosaicstack/stack.git",
|
"url": "https://git.mosaicstack.dev/mosaicstack/stack.git",
|
||||||
|
|||||||
@@ -30,7 +30,6 @@ import {
|
|||||||
refreshActiveFleetUnits,
|
refreshActiveFleetUnits,
|
||||||
readRosterAgentNames,
|
readRosterAgentNames,
|
||||||
buildRelaunchCommands,
|
buildRelaunchCommands,
|
||||||
checkFrameworkDrift,
|
|
||||||
FRAMEWORK_RESEED_PACKAGE,
|
FRAMEWORK_RESEED_PACKAGE,
|
||||||
} from './runtime/update-checker.js';
|
} from './runtime/update-checker.js';
|
||||||
import { runWizard } from './wizard.js';
|
import { runWizard } from './wizard.js';
|
||||||
@@ -419,48 +418,6 @@ program
|
|||||||
// checkForAllUpdates imported statically above
|
// checkForAllUpdates imported statically above
|
||||||
const { execSync } = await import('node:child_process');
|
const { execSync } = await import('node:child_process');
|
||||||
|
|
||||||
// Re-seed the framework from the freshly-installed package, propagate shipped
|
|
||||||
// systemd unit fixes to the active units, and (opt-in) relaunch durable
|
|
||||||
// agents. Shared by the "packages updated" and the "framework drift" paths.
|
|
||||||
const reseedFramework = (reason: string): void => {
|
|
||||||
console.log(reason);
|
|
||||||
const reseed = runFrameworkReseed();
|
|
||||||
if (!reseed.ok) {
|
|
||||||
console.error(
|
|
||||||
`\n⚠ Framework re-seed skipped: ${reseed.reason ?? 'unknown'}.\n` +
|
|
||||||
' Activate manually: bash "$(npm root -g)/@mosaicstack/mosaic/framework/install.sh" ' +
|
|
||||||
'(MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep)',
|
|
||||||
);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
console.log('✔ Framework re-seeded.');
|
|
||||||
// Propagate shipped systemd unit fixes to the ACTIVE units (re-seed only
|
|
||||||
// touches ~/.config/mosaic/systemd/user; systemd runs ~/.config/systemd/user).
|
|
||||||
const units = refreshActiveFleetUnits();
|
|
||||||
if (units.refreshed.length > 0) {
|
|
||||||
console.log(`✔ Refreshed ${units.refreshed.length} active systemd unit(s).`);
|
|
||||||
}
|
|
||||||
const agents = readRosterAgentNames();
|
|
||||||
if (agents.length === 0) return;
|
|
||||||
if (opts.relaunch) {
|
|
||||||
console.log(`\nRelaunching ${agents.length} fleet agent(s) to pick up the new runtime…`);
|
|
||||||
for (const restart of buildRelaunchCommands(agents)) {
|
|
||||||
try {
|
|
||||||
execSync(restart.join(' '), { stdio: 'inherit', timeout: 30_000 });
|
|
||||||
} catch {
|
|
||||||
console.error(` ⚠ failed to restart agent — run: ${restart.join(' ')}`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
console.log('✔ Agents relaunched.');
|
|
||||||
} else {
|
|
||||||
console.log(
|
|
||||||
`\nℹ ${agents.length} fleet agent(s) are still running the previous runtime. ` +
|
|
||||||
'Restart them to activate the update:\n mosaic update --relaunch ' +
|
|
||||||
'(or: mosaic fleet restart <agent>)',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
console.log('Checking for updates…');
|
console.log('Checking for updates…');
|
||||||
const results = checkForAllUpdates({ skipCache: true });
|
const results = checkForAllUpdates({ skipCache: true });
|
||||||
|
|
||||||
@@ -475,18 +432,6 @@ program
|
|||||||
process.exit(1);
|
process.exit(1);
|
||||||
}
|
}
|
||||||
console.log('\n✔ All packages up to date.');
|
console.log('\n✔ All packages up to date.');
|
||||||
// #642: the CLI may have been upgraded outside `mosaic update` (e.g. a
|
|
||||||
// direct `npm i -g`), leaving the framework files stale even though no
|
|
||||||
// package is reported outdated. Detect that via the framework version and
|
|
||||||
// re-seed so shipped launcher/runtime fixes still activate.
|
|
||||||
const drift = checkFrameworkDrift();
|
|
||||||
if (drift.drifted && opts.reseed !== false) {
|
|
||||||
reseedFramework(
|
|
||||||
`\nFramework drift detected (on-disk v${drift.installed} < bundled v${drift.bundled}) — ` +
|
|
||||||
'the CLI was updated outside `mosaic update`. Re-seeding framework files into ' +
|
|
||||||
'~/.config/mosaic (data-safe; keeps your edits)…',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -511,17 +456,52 @@ program
|
|||||||
// F3-m3 / R13: the CLI is updated, but the framework files in
|
// F3-m3 / R13: the CLI is updated, but the framework files in
|
||||||
// ~/.config/mosaic/ are still the previous version. Re-seed them from the
|
// ~/.config/mosaic/ are still the previous version. Re-seed them from the
|
||||||
// freshly-installed package so shipped launcher/runtime changes ACTIVATE.
|
// freshly-installed package so shipped launcher/runtime changes ACTIVATE.
|
||||||
// Re-seed when the framework-bearing package itself updated OR the on-disk
|
// Only when the framework-bearing package itself updated.
|
||||||
// framework is older than the freshly-installed one (#642 — e.g. only
|
|
||||||
// sibling packages were outdated but the CLI was already ahead).
|
|
||||||
const mosaicUpdated = outdated.some(
|
const mosaicUpdated = outdated.some(
|
||||||
(r: { package: string }) => r.package === FRAMEWORK_RESEED_PACKAGE,
|
(r: { package: string }) => r.package === FRAMEWORK_RESEED_PACKAGE,
|
||||||
);
|
);
|
||||||
const drift = checkFrameworkDrift();
|
if (mosaicUpdated && opts.reseed !== false) {
|
||||||
if ((mosaicUpdated || drift.drifted) && opts.reseed !== false) {
|
console.log(
|
||||||
reseedFramework(
|
|
||||||
'\nRe-seeding framework files into ~/.config/mosaic (data-safe; keeps your edits)…',
|
'\nRe-seeding framework files into ~/.config/mosaic (data-safe; keeps your edits)…',
|
||||||
);
|
);
|
||||||
|
const reseed = runFrameworkReseed();
|
||||||
|
if (reseed.ok) {
|
||||||
|
console.log('✔ Framework re-seeded.');
|
||||||
|
// Propagate shipped systemd unit fixes to the ACTIVE units (re-seed only
|
||||||
|
// touches ~/.config/mosaic/systemd/user; systemd runs ~/.config/systemd/user).
|
||||||
|
const units = refreshActiveFleetUnits();
|
||||||
|
if (units.refreshed.length > 0) {
|
||||||
|
console.log(`✔ Refreshed ${units.refreshed.length} active systemd unit(s).`);
|
||||||
|
}
|
||||||
|
const agents = readRosterAgentNames();
|
||||||
|
if (agents.length > 0) {
|
||||||
|
if (opts.relaunch) {
|
||||||
|
console.log(
|
||||||
|
`\nRelaunching ${agents.length} fleet agent(s) to pick up the new runtime…`,
|
||||||
|
);
|
||||||
|
for (const restart of buildRelaunchCommands(agents)) {
|
||||||
|
try {
|
||||||
|
execSync(restart.join(' '), { stdio: 'inherit', timeout: 30_000 });
|
||||||
|
} catch {
|
||||||
|
console.error(` ⚠ failed to restart agent — run: ${restart.join(' ')}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log('✔ Agents relaunched.');
|
||||||
|
} else {
|
||||||
|
console.log(
|
||||||
|
`\nℹ ${agents.length} fleet agent(s) are still running the previous runtime. ` +
|
||||||
|
'Restart them to activate the update:\n mosaic update --relaunch ' +
|
||||||
|
'(or: mosaic fleet restart <agent>)',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.error(
|
||||||
|
`\n⚠ Framework re-seed skipped: ${reseed.reason ?? 'unknown'}.\n` +
|
||||||
|
' Activate manually: bash "$(npm root -g)/@mosaicstack/mosaic/framework/install.sh" ' +
|
||||||
|
'(MOSAIC_SYNC_ONLY=1 MOSAIC_INSTALL_MODE=keep)',
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -19,20 +19,17 @@ import {
|
|||||||
buildSystemdShowCommand,
|
buildSystemdShowCommand,
|
||||||
buildTmuxListPanesCommand,
|
buildTmuxListPanesCommand,
|
||||||
buildTmuxListSessionsCommand,
|
buildTmuxListSessionsCommand,
|
||||||
classifyReadiness,
|
|
||||||
classifySendResult,
|
classifySendResult,
|
||||||
countOrchestrators,
|
countOrchestrators,
|
||||||
countEnhancers,
|
countEnhancers,
|
||||||
detectDrift,
|
detectDrift,
|
||||||
enableFleetUnits,
|
enableFleetUnits,
|
||||||
FLEET_PROFILES,
|
FLEET_PROFILES,
|
||||||
HEARTBEAT_IDLE_THRESHOLD_SECONDS,
|
|
||||||
generateAgentEnv,
|
generateAgentEnv,
|
||||||
getDefaultOperatorSourceLabel,
|
getDefaultOperatorSourceLabel,
|
||||||
getDefaultTenantAndHost,
|
getDefaultTenantAndHost,
|
||||||
getRosterAgent,
|
getRosterAgent,
|
||||||
heartbeatPath,
|
heartbeatPath,
|
||||||
idleThresholdSeconds,
|
|
||||||
isSendAccepted,
|
isSendAccepted,
|
||||||
loadFleetRoster,
|
loadFleetRoster,
|
||||||
mergeAgentEnv,
|
mergeAgentEnv,
|
||||||
@@ -853,7 +850,7 @@ describe('fleet ps — command construction', () => {
|
|||||||
'-t',
|
'-t',
|
||||||
'=canary-pi:0.0',
|
'=canary-pi:0.0',
|
||||||
'-F',
|
'-F',
|
||||||
'#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity} #{window_activity} #{session_activity}',
|
'#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity}',
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -936,125 +933,6 @@ describe('fleet ps — heartbeat parsing', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('fleet ps — readiness thresholds', () => {
|
|
||||||
const savedIdle = process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD;
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
if (savedIdle === undefined) delete process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD;
|
|
||||||
else process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD = savedIdle;
|
|
||||||
});
|
|
||||||
|
|
||||||
it('uses the default activity threshold when env is unset', () => {
|
|
||||||
delete process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD;
|
|
||||||
|
|
||||||
expect(idleThresholdSeconds()).toBe(HEARTBEAT_IDLE_THRESHOLD_SECONDS);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('honors a positive integer activity threshold from env', () => {
|
|
||||||
process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD = '120';
|
|
||||||
|
|
||||||
expect(idleThresholdSeconds()).toBe(120);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('falls back to the default for invalid activity thresholds', () => {
|
|
||||||
process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD = '0';
|
|
||||||
|
|
||||||
expect(idleThresholdSeconds()).toBe(HEARTBEAT_IDLE_THRESHOLD_SECONDS);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('fleet ps — readiness classification', () => {
|
|
||||||
const thresholds = { idleThresholdSeconds: 300 };
|
|
||||||
|
|
||||||
it('reports dead when the pane is not alive', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: false, hbHealth: 'healthy', hbStatus: 'busy', idleSeconds: 0 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('dead');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports unknown when heartbeat health is unknown', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'unknown', hbStatus: null, idleSeconds: 0 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('unknown');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports stale when heartbeat health is stale', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'stale', hbStatus: 'busy', idleSeconds: 1_000 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('stale');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports working when heartbeat status is busy, even after the activity threshold', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'busy', idleSeconds: 2_000 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('working');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports working when pane idle seconds are null', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok', idleSeconds: null },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('working');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports working when pane idle seconds are undefined', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness({ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok' }, thresholds),
|
|
||||||
).toBe('working');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports working when pane idle seconds are non-finite', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok', idleSeconds: Number.NaN },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('working');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports available at the activity threshold boundary', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok', idleSeconds: 300 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('available');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports working below the activity threshold', () => {
|
|
||||||
expect(
|
|
||||||
classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok', idleSeconds: 299 },
|
|
||||||
thresholds,
|
|
||||||
),
|
|
||||||
).toBe('working');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports very long idle as available, not stuck', () => {
|
|
||||||
const readiness = classifyReadiness(
|
|
||||||
{ paneAlive: true, hbHealth: 'healthy', hbStatus: 'ok', idleSeconds: 100_000 },
|
|
||||||
thresholds,
|
|
||||||
);
|
|
||||||
|
|
||||||
expect(readiness).toBe('available');
|
|
||||||
expect(readiness).not.toBe('stuck');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('fleet ps — systemd show parsing', () => {
|
describe('fleet ps — systemd show parsing', () => {
|
||||||
it('parses ActiveState, SubState, UnitFileState from systemctl show output', () => {
|
it('parses ActiveState, SubState, UnitFileState from systemctl show output', () => {
|
||||||
const output = 'ActiveState=active\nSubState=running\nUnitFileState=enabled\n';
|
const output = 'ActiveState=active\nSubState=running\nUnitFileState=enabled\n';
|
||||||
@@ -1075,11 +953,9 @@ describe('fleet ps — systemd show parsing', () => {
|
|||||||
describe('fleet ps — tmux list-panes parsing', () => {
|
describe('fleet ps — tmux list-panes parsing', () => {
|
||||||
const NOW_MS = 1_700_000_000_000;
|
const NOW_MS = 1_700_000_000_000;
|
||||||
|
|
||||||
it('uses pane_activity when present', () => {
|
it('parses alive pane with pid, command, and idle time', () => {
|
||||||
const paneActivityEpoch = Math.floor((NOW_MS - 30_000) / 1000); // 30s ago
|
const activityEpoch = Math.floor((NOW_MS - 30_000) / 1000); // 30s ago
|
||||||
const windowActivityEpoch = Math.floor((NOW_MS - 60_000) / 1000); // 60s ago
|
const output = `12345 claude 0 ${activityEpoch}\n`;
|
||||||
const sessionActivityEpoch = Math.floor((NOW_MS - 90_000) / 1000); // 90s ago
|
|
||||||
const output = `12345 claude 0 ${paneActivityEpoch} ${windowActivityEpoch} ${sessionActivityEpoch}\n`;
|
|
||||||
const result = parseTmuxListPanes(output, NOW_MS);
|
const result = parseTmuxListPanes(output, NOW_MS);
|
||||||
expect(result.pid).toBe(12345);
|
expect(result.pid).toBe(12345);
|
||||||
expect(result.command).toBe('claude');
|
expect(result.command).toBe('claude');
|
||||||
@@ -1087,45 +963,8 @@ describe('fleet ps — tmux list-panes parsing', () => {
|
|||||||
expect(result.idleSeconds).toBe(30);
|
expect(result.idleSeconds).toBe(30);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('uses window_activity when pane_activity is empty', () => {
|
|
||||||
const windowActivityEpoch = Math.floor((NOW_MS - 45_000) / 1000); // 45s ago
|
|
||||||
const sessionActivityEpoch = Math.floor((NOW_MS - 90_000) / 1000); // 90s ago
|
|
||||||
const output = `12345 node 0 ${windowActivityEpoch} ${sessionActivityEpoch}\n`;
|
|
||||||
expect(output).toContain('0 '); // empty pane_activity preserves index alignment
|
|
||||||
const result = parseTmuxListPanes(output, NOW_MS);
|
|
||||||
expect(result.pid).toBe(12345);
|
|
||||||
expect(result.command).toBe('node');
|
|
||||||
expect(result.dead).toBe(false);
|
|
||||||
expect(result.idleSeconds).toBe(45);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('uses session_activity when pane_activity and window_activity are empty', () => {
|
|
||||||
const sessionActivityEpoch = Math.floor((NOW_MS - 75_000) / 1000); // 75s ago
|
|
||||||
const output = `12345 node 0 ${sessionActivityEpoch}\n`;
|
|
||||||
const result = parseTmuxListPanes(output, NOW_MS);
|
|
||||||
expect(result.idleSeconds).toBe(75);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports null idleSeconds when all activity sources are empty', () => {
|
|
||||||
const output = '12345 node 0 \n';
|
|
||||||
const result = parseTmuxListPanes(output, NOW_MS);
|
|
||||||
expect(result.idleSeconds).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('computes exact idle seconds from now minus epoch seconds', () => {
|
|
||||||
const activityEpoch = 1_699_999_877;
|
|
||||||
const result = parseTmuxListPanes(`12345 claude 0 ${activityEpoch} 0 0\n`, NOW_MS);
|
|
||||||
expect(result.idleSeconds).toBe(123);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('clamps future activity epochs to 0 idle seconds', () => {
|
|
||||||
const futureActivityEpoch = Math.floor((NOW_MS + 30_000) / 1000);
|
|
||||||
const result = parseTmuxListPanes(`12345 claude 0 ${futureActivityEpoch} 0 0\n`, NOW_MS);
|
|
||||||
expect(result.idleSeconds).toBe(0);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('reports dead pane when pane_dead=1', () => {
|
it('reports dead pane when pane_dead=1', () => {
|
||||||
const output = `0 bash 1 0 0 0\n`;
|
const output = `0 bash 1 0\n`;
|
||||||
const result = parseTmuxListPanes(output, NOW_MS);
|
const result = parseTmuxListPanes(output, NOW_MS);
|
||||||
expect(result.dead).toBe(true);
|
expect(result.dead).toBe(true);
|
||||||
});
|
});
|
||||||
@@ -1485,9 +1324,8 @@ describe('fleet ps — JSON output shape (FR-6)', () => {
|
|||||||
// boot-enable warning: active + disabled
|
// boot-enable warning: active + disabled
|
||||||
expect(row.bootEnableWarning).toBe(true);
|
expect(row.bootEnableWarning).toBe(true);
|
||||||
|
|
||||||
// heartbeat missing → unknown readiness preserves existing display semantics
|
// heartbeat missing → unknown
|
||||||
expect(row.heartbeat.health).toBe('unknown');
|
expect(row.heartbeat.health).toBe('unknown');
|
||||||
expect(row.readiness).toBe('unknown');
|
|
||||||
|
|
||||||
expect(row.name).toBe('canary-pi');
|
expect(row.name).toBe('canary-pi');
|
||||||
expect(row.runtime).toBe('pi');
|
expect(row.runtime).toBe('pi');
|
||||||
@@ -1549,88 +1387,6 @@ describe('fleet ps — command sequences issued', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('fleet ps — readiness table output', () => {
|
|
||||||
it('renders available in HB column without idle/stuck alarm flags', async () => {
|
|
||||||
const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-'));
|
|
||||||
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
|
||||||
const runDir = join(home, 'fleet', 'run');
|
|
||||||
await mkdir(runDir, { recursive: true });
|
|
||||||
await writeFile(
|
|
||||||
rosterPath,
|
|
||||||
[
|
|
||||||
'version: 1',
|
|
||||||
'transport: tmux',
|
|
||||||
'agents:',
|
|
||||||
' - name: working-agent',
|
|
||||||
' runtime: pi',
|
|
||||||
' - name: available-agent',
|
|
||||||
' runtime: pi',
|
|
||||||
].join('\n'),
|
|
||||||
);
|
|
||||||
|
|
||||||
const nowMs = 1_700_000_000_000;
|
|
||||||
const workingActivityEpoch = Math.floor((nowMs - 2_000) / 1000);
|
|
||||||
const availableActivityEpoch = Math.floor((nowMs - 40_000) / 1000);
|
|
||||||
const hbTs = new Date(nowMs - 1_000).toISOString();
|
|
||||||
await writeFile(join(runDir, 'working-agent.hb'), `ts=${hbTs}\npid=111\nstatus=ok\n`);
|
|
||||||
await writeFile(join(runDir, 'available-agent.hb'), `ts=${hbTs}\npid=222\nstatus=ok\n`);
|
|
||||||
|
|
||||||
const savedIdle = process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD;
|
|
||||||
process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD = '5';
|
|
||||||
|
|
||||||
const dateNow = vi.spyOn(Date, 'now').mockReturnValue(nowMs);
|
|
||||||
const runner: CommandRunner = async (command, args) => {
|
|
||||||
const full = [command, ...args].join(' ');
|
|
||||||
if (full.includes('list-sessions')) {
|
|
||||||
return { stdout: 'working-agent\navailable-agent\n', stderr: '', exitCode: 0 };
|
|
||||||
}
|
|
||||||
if (full.includes('=working-agent:0.0')) {
|
|
||||||
return { stdout: `111 pi 0 ${workingActivityEpoch}\n`, stderr: '', exitCode: 0 };
|
|
||||||
}
|
|
||||||
if (full.includes('=available-agent:0.0')) {
|
|
||||||
return { stdout: `222 pi 0 ${availableActivityEpoch}\n`, stderr: '', exitCode: 0 };
|
|
||||||
}
|
|
||||||
if (full.includes('systemctl') && full.includes('show')) {
|
|
||||||
return {
|
|
||||||
stdout: 'ActiveState=active\nSubState=running\nUnitFileState=enabled\n',
|
|
||||||
stderr: '',
|
|
||||||
exitCode: 0,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return { stdout: '', stderr: '', exitCode: 0 };
|
|
||||||
};
|
|
||||||
|
|
||||||
const lines: string[] = [];
|
|
||||||
const origLog = console.log;
|
|
||||||
console.log = (msg: string) => {
|
|
||||||
lines.push(msg);
|
|
||||||
};
|
|
||||||
|
|
||||||
const program = new Command();
|
|
||||||
program.exitOverride();
|
|
||||||
registerFleetCommand(program, { runner, mosaicHome: home });
|
|
||||||
|
|
||||||
try {
|
|
||||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'ps']);
|
|
||||||
} finally {
|
|
||||||
console.log = origLog;
|
|
||||||
dateNow.mockRestore();
|
|
||||||
if (savedIdle === undefined) delete process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD;
|
|
||||||
else process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD = savedIdle;
|
|
||||||
await rm(home, { recursive: true, force: true });
|
|
||||||
}
|
|
||||||
|
|
||||||
const workingLine = lines.find((line) => line.includes('working-agent'));
|
|
||||||
const availableLine = lines.find((line) => line.includes('available-agent'));
|
|
||||||
expect(workingLine).toBeDefined();
|
|
||||||
expect(workingLine).toContain('1s/working');
|
|
||||||
expect(availableLine).toBeDefined();
|
|
||||||
expect(availableLine).toContain('1s/available');
|
|
||||||
expect(availableLine).not.toMatch(/\bIDLE\b/);
|
|
||||||
expect(availableLine).not.toMatch(/\bSTUCK\b/);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('buildTmuxListSessionsCommand', () => {
|
describe('buildTmuxListSessionsCommand', () => {
|
||||||
it('builds exact list-sessions command with session_name format', () => {
|
it('builds exact list-sessions command with session_name format', () => {
|
||||||
expect(buildTmuxListSessionsCommand('mosaic-fleet')).toEqual([
|
expect(buildTmuxListSessionsCommand('mosaic-fleet')).toEqual([
|
||||||
@@ -1758,7 +1514,6 @@ describe('fleet ps — unmanaged socket sessions', () => {
|
|||||||
|
|
||||||
// driftFlag must be false for unmanaged (no roster runtime to compare)
|
// driftFlag must be false for unmanaged (no roster runtime to compare)
|
||||||
expect(unmanagedRow.driftFlag).toBe(false);
|
expect(unmanagedRow.driftFlag).toBe(false);
|
||||||
expect(unmanagedRow.readiness).toBe('unknown');
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('shows UNMANAGED flag in table output for unmanaged sessions', async () => {
|
it('shows UNMANAGED flag in table output for unmanaged sessions', async () => {
|
||||||
|
|||||||
@@ -394,7 +394,6 @@ export function buildAgentTailCommand(agentName: string, lines: number, socketNa
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
export const HEARTBEAT_INTERVAL_MS = 15_000;
|
export const HEARTBEAT_INTERVAL_MS = 15_000;
|
||||||
export const HEARTBEAT_IDLE_THRESHOLD_SECONDS = 300;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Heartbeat interval in ms, honoring MOSAIC_HEARTBEAT_INTERVAL (seconds) so the
|
* Heartbeat interval in ms, honoring MOSAIC_HEARTBEAT_INTERVAL (seconds) so the
|
||||||
@@ -405,57 +404,8 @@ export function heartbeatIntervalMs(): number {
|
|||||||
const sec = Number.parseInt(process.env.MOSAIC_HEARTBEAT_INTERVAL ?? '', 10);
|
const sec = Number.parseInt(process.env.MOSAIC_HEARTBEAT_INTERVAL ?? '', 10);
|
||||||
return Number.isFinite(sec) && sec > 0 ? sec * 1000 : HEARTBEAT_INTERVAL_MS;
|
return Number.isFinite(sec) && sec > 0 ? sec * 1000 : HEARTBEAT_INTERVAL_MS;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Activity threshold in seconds, honoring MOSAIC_HEARTBEAT_IDLE_THRESHOLD. */
|
|
||||||
export function idleThresholdSeconds(): number {
|
|
||||||
const sec = Number.parseInt(process.env.MOSAIC_HEARTBEAT_IDLE_THRESHOLD ?? '', 10);
|
|
||||||
return Number.isFinite(sec) && sec > 0 ? sec : HEARTBEAT_IDLE_THRESHOLD_SECONDS;
|
|
||||||
}
|
|
||||||
export const HEARTBEAT_HEALTHY_MULTIPLIER = 3;
|
export const HEARTBEAT_HEALTHY_MULTIPLIER = 3;
|
||||||
|
|
||||||
export type ReadinessState = 'working' | 'available' | 'stuck' | 'stale' | 'dead' | 'unknown';
|
|
||||||
|
|
||||||
export interface ReadinessSignals {
|
|
||||||
paneAlive: boolean;
|
|
||||||
hbHealth: 'healthy' | 'stale' | 'unknown';
|
|
||||||
hbStatus: 'ok' | 'busy' | null;
|
|
||||||
idleSeconds: number | null;
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface ReadinessThresholds {
|
|
||||||
idleThresholdSeconds: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Classify whether an agent is progressing based on already-parsed heartbeat/tmux signals.
|
|
||||||
* Best-effort and runtime-agnostic: it never probes, never throws, and preserves existing
|
|
||||||
* unknown/stale behavior when heartbeat data is absent or old.
|
|
||||||
*/
|
|
||||||
export function classifyReadiness(
|
|
||||||
signals: Partial<ReadinessSignals> | null | undefined,
|
|
||||||
thresholds: Partial<ReadinessThresholds> | null | undefined = {},
|
|
||||||
): ReadinessState {
|
|
||||||
try {
|
|
||||||
if (signals?.paneAlive !== true) return 'dead';
|
|
||||||
if (signals.hbHealth === 'unknown' || signals.hbHealth === undefined) return 'unknown';
|
|
||||||
if (signals.hbHealth === 'stale') return 'stale';
|
|
||||||
if (signals.hbStatus === 'busy') return 'working';
|
|
||||||
if (signals.idleSeconds === null || signals.idleSeconds === undefined) return 'working';
|
|
||||||
|
|
||||||
const idleSeconds = Number.isFinite(signals.idleSeconds) ? signals.idleSeconds : null;
|
|
||||||
if (idleSeconds === null) return 'working';
|
|
||||||
|
|
||||||
const idleThreshold = Number.isFinite(thresholds?.idleThresholdSeconds)
|
|
||||||
? Number(thresholds?.idleThresholdSeconds)
|
|
||||||
: idleThresholdSeconds();
|
|
||||||
// Follow-up: stuck pending per-agent assignment awareness: assigned task + idle past threshold => stuck.
|
|
||||||
if (idleSeconds >= idleThreshold) return 'available';
|
|
||||||
return 'working';
|
|
||||||
} catch {
|
|
||||||
return 'unknown';
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface HeartbeatInfo {
|
export interface HeartbeatInfo {
|
||||||
ts: Date | null;
|
ts: Date | null;
|
||||||
pid: number | null;
|
pid: number | null;
|
||||||
@@ -479,7 +429,6 @@ export interface AgentPsRow {
|
|||||||
paneCommand: string | null;
|
paneCommand: string | null;
|
||||||
idleSeconds: number | null;
|
idleSeconds: number | null;
|
||||||
heartbeat: HeartbeatInfo;
|
heartbeat: HeartbeatInfo;
|
||||||
readiness: ReadinessState;
|
|
||||||
/** roster runtime !== actual pane command */
|
/** roster runtime !== actual pane command */
|
||||||
driftFlag: boolean;
|
driftFlag: boolean;
|
||||||
/** active but UnitFileState=disabled */
|
/** active but UnitFileState=disabled */
|
||||||
@@ -512,7 +461,7 @@ export function buildSystemdShowCommand(agentName: string): string[] {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Returns the tmux list-panes command for an agent pane.
|
* Returns the tmux list-panes command for an agent pane.
|
||||||
* Format: `#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity} #{window_activity} #{session_activity}`
|
* Format: `#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity}`
|
||||||
*/
|
*/
|
||||||
export function buildTmuxListPanesCommand(agentName: string, socketName = ''): string[] {
|
export function buildTmuxListPanesCommand(agentName: string, socketName = ''): string[] {
|
||||||
return [
|
return [
|
||||||
@@ -522,7 +471,7 @@ export function buildTmuxListPanesCommand(agentName: string, socketName = ''): s
|
|||||||
'-t',
|
'-t',
|
||||||
`=${agentName}:0.0`,
|
`=${agentName}:0.0`,
|
||||||
'-F',
|
'-F',
|
||||||
'#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity} #{window_activity} #{session_activity}',
|
'#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity}',
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -622,8 +571,8 @@ export function parseSystemdShow(output: string): {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Parse the output of `tmux list-panes -F '#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity} #{window_activity} #{session_activity}'`
|
* Parse the output of `tmux list-panes -F '#{pane_pid} #{pane_current_command} #{pane_dead} #{pane_activity}'`
|
||||||
* Activity fields are Unix epoch timestamps (seconds), ordered most precise to coarsest.
|
* pane_activity is a Unix epoch timestamp (seconds).
|
||||||
*/
|
*/
|
||||||
export function parseTmuxListPanes(
|
export function parseTmuxListPanes(
|
||||||
output: string,
|
output: string,
|
||||||
@@ -633,18 +582,16 @@ export function parseTmuxListPanes(
|
|||||||
if (!line) {
|
if (!line) {
|
||||||
return { pid: null, command: null, dead: true, idleSeconds: null };
|
return { pid: null, command: null, dead: true, idleSeconds: null };
|
||||||
}
|
}
|
||||||
// format: <pid> <command> <dead(0|1)> <pane_activity> <window_activity> <session_activity>
|
// format: <pid> <command> <dead(0|1)> <activity_epoch>
|
||||||
const parts = line.split(' ');
|
const parts = line.split(' ');
|
||||||
const pid = parts[0] ? (Number.isFinite(Number(parts[0])) ? Number(parts[0]) : null) : null;
|
const pid = parts[0] ? (Number.isFinite(Number(parts[0])) ? Number(parts[0]) : null) : null;
|
||||||
const command = parts[1] ?? null;
|
const command = parts[1] ?? null;
|
||||||
const dead = parts[2] === '1';
|
const dead = parts[2] === '1';
|
||||||
const activityEpoch = parts
|
const activityEpoch = parts[3] ? Number(parts[3]) : NaN;
|
||||||
.slice(3, 6)
|
const idleSeconds =
|
||||||
.map((part) => (part ? Number(part) : NaN))
|
Number.isFinite(activityEpoch) && activityEpoch > 0
|
||||||
.find((epoch) => Number.isFinite(epoch) && epoch > 0);
|
? Math.floor((nowMs - activityEpoch * 1000) / 1000)
|
||||||
const idleSeconds = activityEpoch
|
: null;
|
||||||
? Math.max(0, Math.floor((nowMs - activityEpoch * 1000) / 1000))
|
|
||||||
: null;
|
|
||||||
return { pid, command, dead, idleSeconds };
|
return { pid, command, dead, idleSeconds };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1075,9 +1022,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
const nowMs = Date.now();
|
const nowMs = Date.now();
|
||||||
|
|
||||||
const rows: AgentPsRow[] = [];
|
const rows: AgentPsRow[] = [];
|
||||||
const readinessThresholds = {
|
|
||||||
idleThresholdSeconds: idleThresholdSeconds(),
|
|
||||||
};
|
|
||||||
|
|
||||||
// Build the set of roster agent names for quick lookup when filtering socket sessions.
|
// Build the set of roster agent names for quick lookup when filtering socket sessions.
|
||||||
const rosterAgentNames = new Set(roster.agents.map((a) => a.name));
|
const rosterAgentNames = new Set(roster.agents.map((a) => a.name));
|
||||||
@@ -1108,17 +1052,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
const bootEnableWarning =
|
const bootEnableWarning =
|
||||||
sysInfo.ActiveState === 'active' && sysInfo.UnitFileState === 'disabled';
|
sysInfo.ActiveState === 'active' && sysInfo.UnitFileState === 'disabled';
|
||||||
|
|
||||||
const paneAlive = !paneInfo.dead;
|
|
||||||
const readiness = classifyReadiness(
|
|
||||||
{
|
|
||||||
paneAlive,
|
|
||||||
hbHealth: hb.health,
|
|
||||||
hbStatus: hb.status,
|
|
||||||
idleSeconds: paneInfo.idleSeconds,
|
|
||||||
},
|
|
||||||
readinessThresholds,
|
|
||||||
);
|
|
||||||
|
|
||||||
rows.push({
|
rows.push({
|
||||||
name: agent.name,
|
name: agent.name,
|
||||||
tenant_id,
|
tenant_id,
|
||||||
@@ -1126,12 +1059,11 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
runtime: agent.runtime,
|
runtime: agent.runtime,
|
||||||
systemdActive: sysInfo.ActiveState,
|
systemdActive: sysInfo.ActiveState,
|
||||||
systemdEnabled: sysInfo.UnitFileState,
|
systemdEnabled: sysInfo.UnitFileState,
|
||||||
paneAlive,
|
paneAlive: !paneInfo.dead,
|
||||||
panePid: paneInfo.pid,
|
panePid: paneInfo.pid,
|
||||||
paneCommand: paneInfo.command,
|
paneCommand: paneInfo.command,
|
||||||
idleSeconds: paneInfo.idleSeconds,
|
idleSeconds: paneInfo.idleSeconds,
|
||||||
heartbeat: hb,
|
heartbeat: hb,
|
||||||
readiness,
|
|
||||||
driftFlag,
|
driftFlag,
|
||||||
bootEnableWarning,
|
bootEnableWarning,
|
||||||
managed: true,
|
managed: true,
|
||||||
@@ -1178,17 +1110,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
const bootEnableWarning =
|
const bootEnableWarning =
|
||||||
sysInfo.ActiveState === 'active' && sysInfo.UnitFileState === 'disabled';
|
sysInfo.ActiveState === 'active' && sysInfo.UnitFileState === 'disabled';
|
||||||
|
|
||||||
const paneAlive = !paneInfo.dead;
|
|
||||||
const readiness = classifyReadiness(
|
|
||||||
{
|
|
||||||
paneAlive,
|
|
||||||
hbHealth: hb.health,
|
|
||||||
hbStatus: hb.status,
|
|
||||||
idleSeconds: paneInfo.idleSeconds,
|
|
||||||
},
|
|
||||||
readinessThresholds,
|
|
||||||
);
|
|
||||||
|
|
||||||
rows.push({
|
rows.push({
|
||||||
name: sessionName,
|
name: sessionName,
|
||||||
tenant_id,
|
tenant_id,
|
||||||
@@ -1197,12 +1118,11 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
runtime: 'unknown',
|
runtime: 'unknown',
|
||||||
systemdActive: sysInfo.ActiveState,
|
systemdActive: sysInfo.ActiveState,
|
||||||
systemdEnabled: sysInfo.UnitFileState,
|
systemdEnabled: sysInfo.UnitFileState,
|
||||||
paneAlive,
|
paneAlive: !paneInfo.dead,
|
||||||
panePid: paneInfo.pid,
|
panePid: paneInfo.pid,
|
||||||
paneCommand: paneInfo.command,
|
paneCommand: paneInfo.command,
|
||||||
idleSeconds: paneInfo.idleSeconds,
|
idleSeconds: paneInfo.idleSeconds,
|
||||||
heartbeat: hb,
|
heartbeat: hb,
|
||||||
readiness,
|
|
||||||
// No roster runtime to compare — drift is not meaningful for unmanaged sessions
|
// No roster runtime to compare — drift is not meaningful for unmanaged sessions
|
||||||
driftFlag: false,
|
driftFlag: false,
|
||||||
bootEnableWarning,
|
bootEnableWarning,
|
||||||
@@ -1244,7 +1164,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
const idle = row.idleSeconds !== null ? `${row.idleSeconds}s` : '-';
|
const idle = row.idleSeconds !== null ? `${row.idleSeconds}s` : '-';
|
||||||
const hbAge =
|
const hbAge =
|
||||||
row.heartbeat.ageMs !== null
|
row.heartbeat.ageMs !== null
|
||||||
? `${Math.round(row.heartbeat.ageMs / 1000)}s/${row.readiness}`
|
? `${Math.round(row.heartbeat.ageMs / 1000)}s/${row.heartbeat.health}`
|
||||||
: `unknown`;
|
: `unknown`;
|
||||||
const model = row.heartbeat.model ?? '-';
|
const model = row.heartbeat.model ?? '-';
|
||||||
const flags: string[] = [];
|
const flags: string[] = [];
|
||||||
|
|||||||
@@ -8,9 +8,6 @@ import {
|
|||||||
readRosterAgentNames,
|
readRosterAgentNames,
|
||||||
runFrameworkReseed,
|
runFrameworkReseed,
|
||||||
refreshActiveFleetUnits,
|
refreshActiveFleetUnits,
|
||||||
readInstalledFrameworkVersion,
|
|
||||||
readBundledFrameworkVersion,
|
|
||||||
checkFrameworkDrift,
|
|
||||||
} from './update-checker.js';
|
} from './update-checker.js';
|
||||||
import { existsSync, readFileSync } from 'node:fs';
|
import { existsSync, readFileSync } from 'node:fs';
|
||||||
|
|
||||||
@@ -126,73 +123,3 @@ describe('refreshActiveFleetUnits', () => {
|
|||||||
expect(existsSync(join(configHome, 'systemd', 'user', 'mosaic-agent@.service'))).toBe(false);
|
expect(existsSync(join(configHome, 'systemd', 'user', 'mosaic-agent@.service'))).toBe(false);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
/**
|
|
||||||
* #642: re-seed when the on-disk framework is older than the bundled one even
|
|
||||||
* if no package is reported outdated (CLI upgraded outside `mosaic update`).
|
|
||||||
*/
|
|
||||||
describe('framework drift detection', () => {
|
|
||||||
let home: string; // stand-in for ~/.config/mosaic
|
|
||||||
let fw: string; // stand-in for the bundled framework root
|
|
||||||
|
|
||||||
beforeEach(() => {
|
|
||||||
const root = mkdtempSync(join(tmpdir(), 'mosaic-drift-'));
|
|
||||||
home = join(root, 'mosaic');
|
|
||||||
fw = join(root, 'framework');
|
|
||||||
mkdirSync(home, { recursive: true });
|
|
||||||
mkdirSync(fw, { recursive: true });
|
|
||||||
});
|
|
||||||
afterEach(() => {
|
|
||||||
rmSync(join(home, '..'), { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
const writeInstalled = (v: string) => writeFileSync(join(home, '.framework-version'), v);
|
|
||||||
const writeBundled = (v: string) =>
|
|
||||||
writeFileSync(join(fw, 'install.sh'), `#!/usr/bin/env bash\nFRAMEWORK_VERSION=${v}\n`);
|
|
||||||
|
|
||||||
describe('readInstalledFrameworkVersion', () => {
|
|
||||||
it('returns undefined when the version file is absent', () => {
|
|
||||||
expect(readInstalledFrameworkVersion(home)).toBeUndefined();
|
|
||||||
});
|
|
||||||
it('parses the integer (tolerating surrounding whitespace)', () => {
|
|
||||||
writeInstalled(' 3\n');
|
|
||||||
expect(readInstalledFrameworkVersion(home)).toBe(3);
|
|
||||||
});
|
|
||||||
it('returns undefined for non-numeric content', () => {
|
|
||||||
writeInstalled('not-a-number\n');
|
|
||||||
expect(readInstalledFrameworkVersion(home)).toBeUndefined();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('readBundledFrameworkVersion', () => {
|
|
||||||
it('returns undefined when install.sh is absent', () => {
|
|
||||||
expect(readBundledFrameworkVersion(fw)).toBeUndefined();
|
|
||||||
});
|
|
||||||
it('parses FRAMEWORK_VERSION=<n> from install.sh', () => {
|
|
||||||
writeBundled('4');
|
|
||||||
expect(readBundledFrameworkVersion(fw)).toBe(4);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('checkFrameworkDrift', () => {
|
|
||||||
it('reports drift when on-disk is older than bundled', () => {
|
|
||||||
writeInstalled('3');
|
|
||||||
writeBundled('4');
|
|
||||||
expect(checkFrameworkDrift(home, fw)).toEqual({ drifted: true, installed: 3, bundled: 4 });
|
|
||||||
});
|
|
||||||
it('no drift when versions match', () => {
|
|
||||||
writeInstalled('4');
|
|
||||||
writeBundled('4');
|
|
||||||
expect(checkFrameworkDrift(home, fw)).toMatchObject({ drifted: false });
|
|
||||||
});
|
|
||||||
it('no drift when on-disk is newer than bundled', () => {
|
|
||||||
writeInstalled('5');
|
|
||||||
writeBundled('4');
|
|
||||||
expect(checkFrameworkDrift(home, fw)).toMatchObject({ drifted: false });
|
|
||||||
});
|
|
||||||
it('no drift (conservative) when a version cannot be read', () => {
|
|
||||||
writeBundled('4'); // installed version file missing
|
|
||||||
expect(checkFrameworkDrift(home, fw)).toMatchObject({ drifted: false, bundled: 4 });
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|||||||
@@ -521,75 +521,6 @@ export function runFrameworkReseed(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── Framework drift detection (#642) ────────────────────────────────────────
|
|
||||||
//
|
|
||||||
// `mosaic update` only re-seeds the framework when the @mosaicstack/mosaic
|
|
||||||
// package itself is upgraded *within that command*. When the CLI is upgraded
|
|
||||||
// some OTHER way — a direct `npm i -g @mosaicstack/mosaic`, or an upgrade run
|
|
||||||
// where only sibling packages were outdated — the framework files in
|
|
||||||
// ~/.config/mosaic stay stale and shipped launcher/runtime fixes never
|
|
||||||
// activate. Comparing the on-disk framework schema version against the version
|
|
||||||
// bundled in the installed package detects exactly that situation.
|
|
||||||
|
|
||||||
/** Read the framework schema version recorded on disk (~/.config/mosaic/.framework-version). */
|
|
||||||
export function readInstalledFrameworkVersion(
|
|
||||||
mosaicHome = join(homedir(), '.config', 'mosaic'),
|
|
||||||
): number | undefined {
|
|
||||||
const vf = join(mosaicHome, '.framework-version');
|
|
||||||
if (!existsSync(vf)) return undefined;
|
|
||||||
try {
|
|
||||||
const n = parseInt(readFileSync(vf, 'utf-8').trim(), 10);
|
|
||||||
return Number.isFinite(n) ? n : undefined;
|
|
||||||
} catch {
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Read the framework schema version shipped in the installed package by parsing
|
|
||||||
* `FRAMEWORK_VERSION=<n>` out of the bundled install.sh (the authoritative
|
|
||||||
* source the installer writes to .framework-version).
|
|
||||||
*/
|
|
||||||
export function readBundledFrameworkVersion(
|
|
||||||
frameworkRoot = resolveBundledFrameworkRoot(),
|
|
||||||
): number | undefined {
|
|
||||||
const installer = join(frameworkRoot, 'install.sh');
|
|
||||||
if (!existsSync(installer)) return undefined;
|
|
||||||
try {
|
|
||||||
const m = readFileSync(installer, 'utf-8').match(/^\s*FRAMEWORK_VERSION=(\d+)/m);
|
|
||||||
const raw = m?.[1];
|
|
||||||
if (!raw) return undefined;
|
|
||||||
const n = parseInt(raw, 10);
|
|
||||||
return Number.isFinite(n) ? n : undefined;
|
|
||||||
} catch {
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface FrameworkDrift {
|
|
||||||
/** True only when both versions are known AND the on-disk one is older. */
|
|
||||||
drifted: boolean;
|
|
||||||
installed?: number;
|
|
||||||
bundled?: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Detect whether the on-disk framework is older than the framework bundled in
|
|
||||||
* the installed CLI (#642). Conservative: if either version can't be read the
|
|
||||||
* result is no-drift, so a missing/unreadable version file never triggers an
|
|
||||||
* unexpected re-seed.
|
|
||||||
*/
|
|
||||||
export function checkFrameworkDrift(
|
|
||||||
mosaicHome = join(homedir(), '.config', 'mosaic'),
|
|
||||||
frameworkRoot = resolveBundledFrameworkRoot(),
|
|
||||||
): FrameworkDrift {
|
|
||||||
const installed = readInstalledFrameworkVersion(mosaicHome);
|
|
||||||
const bundled = readBundledFrameworkVersion(frameworkRoot);
|
|
||||||
const drifted =
|
|
||||||
typeof installed === 'number' && typeof bundled === 'number' && installed < bundled;
|
|
||||||
return { drifted, installed, bundled };
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Best-effort parse of the fleet roster for agent names (used to relaunch
|
* Best-effort parse of the fleet roster for agent names (used to relaunch
|
||||||
* durable agents after a re-seed). Returns [] when no roster exists.
|
* durable agents after a re-seed). Returns [] when no roster exists.
|
||||||
|
|||||||
Reference in New Issue
Block a user