Files
stack/docs/guides/mos-connector-lease-operations.md
Jarvis dff8ce4f79
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
feat(#755): add logical agent connector fencing
Add normalized runtime-neutral identity, durable PostgreSQL CAS leases, monotonic epochs, short-lived server grants, fail-closed adapter validation, credential-safe audit, and concurrency/restart/abuse coverage.\n\nRefs #755
2026-07-14 13:17:02 -05:00

3.0 KiB

Mos Connector Lease Operations — M1

Operational status

M1 installs the durable schema and gateway policy/adapter boundary. It does not activate a connector, expose a lease administration endpoint, or cut over a channel. The default gateway connector-lease policy is deny-all until a later work package supplies an authorized server-side policy and concrete adapter.

Events to monitor

Use correlation IDs to follow connector_lease_audit_log events:

Event Meaning
acquire First holder inserted for an unused binding
renew Current holder heartbeat extended the TTL
takeover Authorized CAS replaced the holder and incremented epoch
release Current holder explicitly relinquished authority
expiry An expired current lease was observed
reject Policy, CAS, expiry, scope, or fencing validation denied an operation

Audit data is metadata-only. Raw grant objects, connector payloads, scopes, tokens, approval references, and credentials must never be added to audit output.

Incident checks

For suspected duplicate/stale connector effects:

  1. Correlate the attempted operation with its reject, takeover, or expiry event.
  2. Compare the current row's connector ID, lease UUID, epoch, expiry, and release time with the adapter's normalized execution context.
  3. Treat an old epoch, old lease UUID, expired lease, or released lease as non-authoritative. Do not retry it as the old holder.
  4. Recovery uses the authorized takeover path with the observed expected epoch. Ordinary acquire is intentionally rejected for expired/released rows.
  5. If an external effect may already have happened, preserve evidence and do not assume lease fencing provides exactly-once replay safety.

Migration and rollback safety

Migration 0016_salty_morlocks.sql is additive: it creates two new tables and indexes without modifying existing authorization/session tables. Before rollout, normal database backup and migration verification still apply. Rolling application code back leaves unused additive tables in place; dropping tables is not part of automated rollback because it would destroy lease/audit evidence.

Security constraints

  • Tenant comes from authenticated gateway context, never a connector request field.
  • Logical agent, binding, connector, and scope identifiers use normalized constrained forms.
  • Takeover requires explicit gateway policy authorization and an expected epoch.
  • Default defense-in-depth TTL caps are 5 minutes for leases and 30 seconds for grants; policy may enforce stricter limits.
  • Validation and rejection audit complete before adapter side effects.
  • Existing authz and exact-action approval controls remain additional required gates; a valid connector lease does not bypass them.