Files
stack/docs/fleet/concepts/desired-vs-observed-state.md
Jarvis 0aee2c0981
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
docs(fleet): add operator configuration guide
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 09:27:43 -05:00

3.6 KiB

Desired, Derived, and Observed Fleet State

One writable authority

The canonical local v2 roster at <MOSAIC_HOME>/fleet/roster.yaml is desired state. Membership, stable identity, class, runtime/provider/model selection, launch policy, enablement, and persisted running/stopped intent are written only through generation-guarded roster mutations.

Derived projections are reproducible consequences of that authority:

  • <name>.env.generated;
  • exact roster-named tmux sessions on the configured socket after reconciliation;
  • systemd service targets managed by installation/reconciliation.

Current systemd unit enablement is not yet lifecycle-conformant at boot: installation can enable every agent unit, and the launcher projection does not carry enabled or desired_state. Therefore reboot preservation for stopped/disabled agents remains an FCM-M3-002 acceptance hold, not a guaranteed projection behavior.

Observed evidence available to current roster-v2 status commands includes systemd active state, tmux presence, holder ownership, and unmanaged sessions. Heartbeat files are observational in the wider fleet, but roster-v2 status, doctor, and verify do not currently read them. Observation never writes back to the roster.

Generation and ownership

generation is a positive integer concurrency fence. A mutating request must provide the current value. Successful changed CRUD increments it exactly once; stale or concurrent writers fail before mutation. Apply/reconcile rereads the canonical roster under a private exclusive lock and uses only that generation and content for effects.

Ownership is exact, never fuzzy. Reconciliation is limited to roster names, the configured socket, the exact holder session, a private installation identity, and private managed paths. An ownership mismatch, unmanaged session, unsafe path, stale generation, or ambiguous lock fails closed.

Drift decisions

Observation Interpretation Safe response
Generated file differs or is missing Derived projection drift Review apply --dry-run; regenerate from the roster.
Desired running, exact session missing missing-session Diagnose ownership/runtime, then reconcile if safe.
Desired stopped, exact session present unexpected-session Inspect; reconciliation may stop only the proven roster target.
Disabled agent running disabled-running Inspect; disabled state wins during explicit safe reconciliation.
Unknown session on the configured socket Unmanaged state Report only. Do not adopt, rename, or kill it.
Heartbeat stale in the wider fleet Liveness evidence Diagnose separately; current roster-v2 status does not read heartbeat.

status and doctor classify. verify is also observational but exits non-zero when ownership, drift, or unmanaged-state checks fail. plan/apply --dry-run validates proposed projection and lifecycle work without mutation. apply and reconcile converge only after all preconditions pass.

A partial projection failure does not roll the roster back. Treat the committed roster as authority and regenerate. A lifecycle failure after projection completion preserves both roster and projections for inspection. Sensitive legacy values are never printed; diagnostics are bounded to stable codes, key names where applicable, and hashes.