Files
stack/docs/fleet/concepts/desired-vs-observed-state.md
Jarvis 0aee2c0981
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
docs(fleet): add operator configuration guide
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 09:27:43 -05:00

43 lines
3.6 KiB
Markdown

# Desired, Derived, and Observed Fleet State
## One writable authority
The canonical local v2 roster at `<MOSAIC_HOME>/fleet/roster.yaml` is desired state. Membership, stable identity, class, runtime/provider/model selection, launch policy, enablement, and persisted `running`/`stopped` intent are written only through generation-guarded roster mutations.
Derived projections are reproducible consequences of that authority:
- `<name>.env.generated`;
- exact roster-named tmux sessions on the configured socket after reconciliation;
- systemd service targets managed by installation/reconciliation.
Current systemd unit enablement is not yet lifecycle-conformant at boot: installation can enable every
agent unit, and the launcher projection does not carry `enabled` or `desired_state`. Therefore reboot
preservation for stopped/disabled agents remains an FCM-M3-002 acceptance hold, not a guaranteed
projection behavior.
Observed evidence available to current roster-v2 status commands includes systemd active state, tmux
presence, holder ownership, and unmanaged sessions. Heartbeat files are observational in the wider fleet,
but roster-v2 `status`, `doctor`, and `verify` do not currently read them. Observation never writes back
to the roster.
## Generation and ownership
`generation` is a positive integer concurrency fence. A mutating request must provide the current value. Successful changed CRUD increments it exactly once; stale or concurrent writers fail before mutation. Apply/reconcile rereads the canonical roster under a private exclusive lock and uses only that generation and content for effects.
Ownership is exact, never fuzzy. Reconciliation is limited to roster names, the configured socket, the exact holder session, a private installation identity, and private managed paths. An ownership mismatch, unmanaged session, unsafe path, stale generation, or ambiguous lock fails closed.
## Drift decisions
| Observation | Interpretation | Safe response |
| ---------------------------------------- | ------------------------ | ---------------------------------------------------------------------- |
| Generated file differs or is missing | Derived projection drift | Review `apply --dry-run`; regenerate from the roster. |
| Desired `running`, exact session missing | `missing-session` | Diagnose ownership/runtime, then reconcile if safe. |
| Desired `stopped`, exact session present | `unexpected-session` | Inspect; reconciliation may stop only the proven roster target. |
| Disabled agent running | `disabled-running` | Inspect; disabled state wins during explicit safe reconciliation. |
| Unknown session on the configured socket | Unmanaged state | Report only. Do not adopt, rename, or kill it. |
| Heartbeat stale in the wider fleet | Liveness evidence | Diagnose separately; current roster-v2 status does not read heartbeat. |
`status` and `doctor` classify. `verify` is also observational but exits non-zero when ownership, drift, or unmanaged-state checks fail. `plan`/`apply --dry-run` validates proposed projection and lifecycle work without mutation. `apply` and `reconcile` converge only after all preconditions pass.
A partial projection failure does not roll the roster back. Treat the committed roster as authority and regenerate. A lifecycle failure after projection completion preserves both roster and projections for inspection. Sensitive legacy values are never printed; diagnostics are bounded to stable codes, key names where applicable, and hashes.