mosaicstack/stack
25
0
Fork 1
Code Issues 340 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

Fleet comms/session identity: watcher verification, per-agent session homes, wedge detection after binary updates #1295

New Issue
Open
opened 2026-08-17 21:54:15 +00:00 by Ghost · 2 comments
Ghost commented 2026-08-17 21:54:15 +00:00
Copy Link
Copy Source

Summary

Measured on sb-it-1-dt, 2026-08-17 (docs-seat incident): a non-orchestrator fleet session ended up holding and acting on other agents' context. Root cause is three mechanical gaps in comms delivery and session identity — not model behavior. Interim prose guardrails have landed in the user-owned ~/.mosaic/ contract tree (SOUL.md fleet boundaries, STANDARDS.md session identity + watcher hygiene); this issue tracks the mechanical fixes so the prose fences can retire.

The three gaps (all measured)

  1. Session files keyed by cwd, resumable by any process in that cwd. One ~/src/stack pi session file served three lives: a /goal-command dev conversation (Aug 10), the goals seat resumed onto it (Aug 13, gpt-5.6-sol turns), then 22 comms-watcher injections absorbed while the seat was wedged (Aug 17). pi --continue in a shared cwd resumes whatever ran there last.

  2. Comms watchers deliver by tmux session name with no identity verification. install-watcher.sh units target a session name; nothing verifies the pane's process is the intended agent. A renamed or reused session name silently reroutes mail. Watchers also kept injecting for hours into a seat answering nothing — no unread-backlog alarm. (Related known gotcha: delivery confirmation is runtime-glyph-dependent — E7.)

  3. No wedge detection after runtime binary updates. pi 0.84.1→0.84.2 was installed under live seats; in-memory seats fail on first provider switch (module layout mismatch: openai-completions adapter imports a symbol exported only by 0.84.2), every call dies with stopReason:"error" before spending a token, exit code still 0, and every liveness instrument (tmux flag, -icanon, foreground process, pane_alive) stays green. Third member of the dead-seat-passes-liveness family; first that passes the terminal check too.

Proposed mechanical fixes

  • Per-agent harness homes (see jarvis-brain docs/decisions/mosaic-stack/2026-08-17_mosaic-d-002_agent-home-profiles.md, Pending): sessions/transcripts/settings under ~/.mosaic/fleet/agents/<name>/, targeted config-dir env vars per harness (not literal HOME). Prototype already validated (~/.mosaic/fleet/agents/probe — auth-bundle symlink chain, per-agent sessions).
  • Watcher identity verification: before injecting, verify the target pane's process actually runs the named agent (read MOSAIC_AGENT_NAME from /proc/<pane_pid>/environ, or match the launch-ledger PID). Mismatch or unresolvable target → refuse delivery + alert, never type into a squatter.
  • Unread-backlog + wedge alarm: N consecutive injections with no assistant response = alert; repeated stopReason:"error" in the session file after a binary change = alert. Both derivable from existing artifacts (session JSONL, launch ledger).
  • Named sessions on seat launch: launcher passes --name <seat> (or equivalent) so resume is never cwd-ambiguous.

Longer term

Message queue + forced separation (planned) supersedes the watcher path entirely; the identity-verification and wedge-detection pieces remain valid regardless of transport.

## Summary Measured on sb-it-1-dt, 2026-08-17 (docs-seat incident): a non-orchestrator fleet session ended up holding and acting on other agents' context. Root cause is three mechanical gaps in comms delivery and session identity — not model behavior. Interim prose guardrails have landed in the user-owned `~/.mosaic/` contract tree (SOUL.md fleet boundaries, STANDARDS.md session identity + watcher hygiene); this issue tracks the mechanical fixes so the prose fences can retire. ## The three gaps (all measured) 1. **Session files keyed by cwd, resumable by any process in that cwd.** One `~/src/stack` pi session file served three lives: a `/goal`-command dev conversation (Aug 10), the goals seat resumed onto it (Aug 13, `gpt-5.6-sol` turns), then 22 comms-watcher injections absorbed while the seat was wedged (Aug 17). `pi --continue` in a shared cwd resumes whatever ran there last. 2. **Comms watchers deliver by tmux session name with no identity verification.** `install-watcher.sh` units target a session name; nothing verifies the pane's process is the intended agent. A renamed or reused session name silently reroutes mail. Watchers also kept injecting for hours into a seat answering nothing — no unread-backlog alarm. (Related known gotcha: delivery confirmation is runtime-glyph-dependent — E7.) 3. **No wedge detection after runtime binary updates.** pi 0.84.1→0.84.2 was installed under live seats; in-memory seats fail on first provider switch (module layout mismatch: openai-completions adapter imports a symbol exported only by 0.84.2), every call dies with `stopReason:"error"` before spending a token, exit code still 0, and every liveness instrument (tmux flag, -icanon, foreground process, pane_alive) stays green. Third member of the dead-seat-passes-liveness family; first that passes the terminal check too. ## Proposed mechanical fixes - **Per-agent harness homes** (see jarvis-brain `docs/decisions/mosaic-stack/2026-08-17_mosaic-d-002_agent-home-profiles.md`, Pending): sessions/transcripts/settings under `~/.mosaic/fleet/agents/<name>/`, targeted config-dir env vars per harness (not literal HOME). Prototype already validated (`~/.mosaic/fleet/agents/probe` — auth-bundle symlink chain, per-agent sessions). - **Watcher identity verification**: before injecting, verify the target pane's process actually runs the named agent (read `MOSAIC_AGENT_NAME` from `/proc/<pane_pid>/environ`, or match the launch-ledger PID). Mismatch or unresolvable target → refuse delivery + alert, never type into a squatter. - **Unread-backlog + wedge alarm**: N consecutive injections with no assistant response = alert; repeated `stopReason:"error"` in the session file after a binary change = alert. Both derivable from existing artifacts (session JSONL, launch ledger). - **Named sessions on seat launch**: launcher passes `--name <seat>` (or equivalent) so resume is never cwd-ambiguous. ## Longer term Message queue + forced separation (planned) supersedes the watcher path entirely; the identity-verification and wedge-detection pieces remain valid regardless of transport.
Ghost commented 2026-08-17 21:59:42 +00:00
Copy Link
Copy Source

Provisioning follow-up (from sb-it-1-dt hardening, same day): watcher setup should move into agent instantiation — mosaic agent --new (and fleet launch) should ensure the comms watcher user-service exists, is active, and targets the right session/socket, rather than leaving it to whoever remembers.

Interim implementation now lives in the brain repo: scripts/comms/ensure-watcher.sh — idempotent ensure (present+match=no-op; missing/inactive/drift=install or repair via the existing install-watcher.sh), a non-mutating --status mode for boot checks, and two interim identity warnings that directly cover the incident's failure modes:

  • target tmux session missing on the configured socket → WARN (delivering into nothing / name-follows-squatter risk)
  • target pane's process carries a different MOSAIC_AGENT_NAME (read from /proc/<child>/environ) → WARN (misrouting class); bare runtimes get a NOTE that identity is unverifiable until per-agent launches land

Documented as a fleet requirement in the user-owned ~/.mosaic/AGENTS.md ("Fleet Comms Watcher"): provisioned at instantiation, never a running agent's duty; boot check via --status. The framework-side move is: fold ensure semantics into mosaic agent --new/fleet launch + mosaic doctor (drift check), keeping the same contract.

**Provisioning follow-up (from sb-it-1-dt hardening, same day):** watcher setup should move into agent instantiation — `mosaic agent --new` (and fleet launch) should ensure the comms watcher user-service exists, is active, and targets the right session/socket, rather than leaving it to whoever remembers. Interim implementation now lives in the brain repo: `scripts/comms/ensure-watcher.sh` — idempotent ensure (present+match=no-op; missing/inactive/drift=install or repair via the existing `install-watcher.sh`), a non-mutating `--status` mode for boot checks, and two interim identity warnings that directly cover the incident's failure modes: - target tmux session missing on the configured socket → WARN (delivering into nothing / name-follows-squatter risk) - target pane's process carries a different `MOSAIC_AGENT_NAME` (read from `/proc/<child>/environ`) → WARN (misrouting class); bare runtimes get a NOTE that identity is unverifiable until per-agent launches land Documented as a fleet requirement in the user-owned `~/.mosaic/AGENTS.md` ("Fleet Comms Watcher"): provisioned at instantiation, never a running agent's duty; boot check via `--status`. The framework-side move is: fold ensure semantics into `mosaic agent --new`/fleet launch + `mosaic doctor` (drift check), keeping the same contract.
Ghost commented 2026-08-17 22:09:21 +00:00
Copy Link
Copy Source

Continuation record for tonight's follow-on work (fleet identity, comms delivery, ~/.mosaic tree preservation): scratchpads/2026-08-17-fleet-identity-comms-mosaic-tree.md, delivered via PR #1296 into next. It consolidates the decisions register (MOSAIC-D-001/-002 pending ruling in jarvis-brain, tooling-migration direction, provisioning-at-instantiation), the as-built ~/.mosaic inventory, and the sequenced work queue whose first item is the comms-tooling migration + unit regeneration.

Continuation record for tonight's follow-on work (fleet identity, comms delivery, ~/.mosaic tree preservation): scratchpads/2026-08-17-fleet-identity-comms-mosaic-tree.md, delivered via PR #1296 into next. It consolidates the decisions register (MOSAIC-D-001/-002 pending ruling in jarvis-brain, tooling-migration direction, provisioning-at-instantiation), the as-built ~/.mosaic inventory, and the sequenced work queue whose first item is the comms-tooling migration + unit regeneration.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1295