stack/docs/fleet/operations/systemd-tmux-troubleshooting.md

Systemd and tmux Troubleshooting

Start with read-only mosaic fleet status, doctor, and verify. Do not manually adopt, rename, terminate, or recreate sessions while ownership is ambiguous.

Decision table

Finding	Meaning	Safe next step
Empty roster tmux.socket_name	Literal default tmux server	Do not substitute the named mosaic-fleet socket. Use roster-derived commands only.
Non-empty socket	Exact named socket	Never target another socket or infer a per-agent socket.
holder: missing	Required exact holder absent	Inspect installation/projection readiness; do not create an unproven holder manually.
ownership-mismatch	Holder identity or global environment differs	Stop. Verify private install identity and managed paths before retry.
missing-session	Desired-running roster agent lacks exact session	Check service/runtime preconditions; review apply dry-run.
unexpected-session	Desired-stopped roster agent still has exact session	Confirm ownership; only reconciler may target the exact proven roster member.
disabled-running	Disabled roster member is observed running	Inspect and reconcile only after ownership proof.
unmanagedSessions	Unknown session exists on configured named socket	Report and investigate separately. Reconciler will not kill or adopt it.
stale/concurrent generation	Desired state changed since plan	Reload roster/generation and recompute the plan.
stale or ambiguous lock	Prior writer/cleanup cannot be proven	Inspect ownership; do not blindly remove the lock.
projection failure	Derived files incomplete	Keep roster as authority and regenerate projections.
lifecycle failure	Projections complete, runtime convergence incomplete	Inspect the exact owned resource, then rerun with current generation.

Systemd state, tmux state, heartbeat, and generated files are observations/projections, not alternate desired state. Explicit apply/reconcile honors stopped/disabled intent, but current unit enablement and launcher projections do not yet prove lifecycle-safe reboot; inspect unit enablement before reboot and treat stopped/disabled boot preservation as an FCM-M3-002 hold. Current roster-v2 status commands also do not read heartbeat files. Executable gates do not provide site cutover/rollback or package asset-revision repair.

Errors and troubleshooting output never print legacy sensitive values, credential contents, or privileged command text. Use stable codes, key names/hashes, exact roster identities, and bounded recovery actions. See status and drift and reconcile and recover.