feat(#755): add logical agent connector fencing

Add normalized runtime-neutral identity, durable PostgreSQL CAS leases, monotonic epochs, short-lived server grants, fail-closed adapter validation, credential-safe audit, and concurrency/restart/abuse coverage.\n\nRefs #755
2026-07-14 13:04:52 -05:00
parent 8599fd27d8
commit d190732a55
20 changed files with 6844 additions and 0 deletions
--- a/docs/SITEMAP.md
+++ b/docs/SITEMAP.md
@@ -56,3 +56,5 @@
 - [Optional AI egress gateway ADR](architecture/ADR-MOS-EGRESS-GATEWAYS.md) — placement and gates for LiteLLM, Bifrost, and purpose-built translation proxies.
 - [Runtime-neutral Mos identity and failover mission](https://git.mosaicstack.dev/mosaicstack/stack/issues/754)
 - [Logical identity and connector lease/fencing implementation](https://git.mosaicstack.dev/mosaicstack/stack/issues/755)
+- [M1 logical identity and fencing architecture](architecture/mos-runtime-portability-m1.md)
+- [M1 connector lease operations](guides/mos-connector-lease-operations.md)
--- a/docs/architecture/mos-runtime-portability-m1.md
+++ b/docs/architecture/mos-runtime-portability-m1.md
@@ -0,0 +1,49 @@
+# Mos Runtime Portability M1 — Logical Identity and Fencing
+
+## Boundary
+
+M1 separates the logical Mosaic agent from any Claude, Pi, Codex, tmux, Matrix, or provider-native session. The normalized identity is:
+
+```text
+(tenant_id, logical_agent_id, binding_id)
+```
+
+`logical_agent_id` is a server-owned stable identifier. A connector is a replaceable holder of a lease for one binding; it is not the agent identity.
+
+## Durable lease model
+
+PostgreSQL table `logical_agent_connector_leases` has one unique row per identity/binding tuple. The current row records:
+
+- an opaque lease UUID;
+- connector ID and normalized allowed scopes;
+- a positive decimal fencing epoch stored as PostgreSQL `bigint`;
+- acquired, heartbeat, expiry, release, and update timestamps.
+
+Initial acquisition is insert-only. An existing active row causes `lease_held`. An expired or released row causes `takeover_required`; ordinary acquisition cannot recover it. Authorized takeover uses compare-and-swap against the expected epoch, rotates the lease UUID, and increments the epoch atomically. Heartbeat and release match the full identity, binding, connector, lease UUID, and epoch.
+
+The companion `connector_lease_audit_log` is append-only metadata. It stores lifecycle event, outcome/reason, identity/binding/connector, epoch, correlation ID, and timestamp. It deliberately excludes scopes, grant objects, payloads, approval references, tokens, and credentials.
+
+## Execution grants
+
+`ConnectorLeaseCoordinator` issues a short-lived internal grant only after rereading the durable current lease. Defense-in-depth caps leases at 5 minutes and grants at 30 seconds by default; constructor options may tighten these limits. A grant is bound to tenant, logical agent, binding, connector, lease UUID, scope subset, expiry, and epoch.
+
+Validation occurs immediately before adapter invocation and rereads PostgreSQL. The adapter receives only `ConnectorExecutionContext`; harness-native schemas remain behind the adapter. Validation denies:
+
+- grants not minted by the current gateway process (including cloned/forged objects);
+- expired grants or leases;
+- released leases;
+- stale epochs or replaced connector/lease UUIDs;
+- missing/cross-tenant/cross-agent/cross-binding leases;
+- scopes not authorized by both grant and current lease.
+
+A gateway restart intentionally invalidates process-local grants. The durable lease and epoch survive, and a fresh grant may be issued only after current-lease and gateway-policy validation.
+
+## Concurrency and side-effect rule
+
+The database CAS determines the sole current holder. A successful takeover makes every old-epoch validation fail. Connector adapters must consume and propagate the normalized lease epoch/context so downstream effect boundaries can also fence races that occur after gateway validation.
+
+M1 does not provide exactly-once receipts or a side-effect journal. Those remain later #754 work; callers must not infer exactly-once delivery from lease fencing.
+
+## Extension boundary
+
+`ConnectorLeaseService` is the gateway-owned policy surface. Every policy decision receives the normalized requested scopes and TTL (or explicit `null` where no TTL applies), so a concrete policy can enforce least privilege and duration limits. Its production default policy denies every lease/grant operation until a server-configured connector policy is supplied. No M1 HTTP endpoint accepts caller-controlled tenant or logical identity, and no concrete Claude/Pi/Codex adapter or channel cutover is included.
--- a/docs/guides/mos-connector-lease-operations.md
+++ b/docs/guides/mos-connector-lease-operations.md
@@ -0,0 +1,43 @@
+# Mos Connector Lease Operations — M1
+
+## Operational status
+
+M1 installs the durable schema and gateway policy/adapter boundary. It does **not** activate a connector, expose a lease administration endpoint, or cut over a channel. The default gateway connector-lease policy is deny-all until a later work package supplies an authorized server-side policy and concrete adapter.
+
+## Events to monitor
+
+Use correlation IDs to follow `connector_lease_audit_log` events:
+
+| Event      | Meaning                                                               |
+| ---------- | --------------------------------------------------------------------- |
+| `acquire`  | First holder inserted for an unused binding                           |
+| `renew`    | Current holder heartbeat extended the TTL                             |
+| `takeover` | Authorized CAS replaced the holder and incremented epoch              |
+| `release`  | Current holder explicitly relinquished authority                      |
+| `expiry`   | An expired current lease was observed                                 |
+| `reject`   | Policy, CAS, expiry, scope, or fencing validation denied an operation |
+
+Audit data is metadata-only. Raw grant objects, connector payloads, scopes, tokens, approval references, and credentials must never be added to audit output.
+
+## Incident checks
+
+For suspected duplicate/stale connector effects:
+
+1. Correlate the attempted operation with its `reject`, `takeover`, or `expiry` event.
+2. Compare the current row's connector ID, lease UUID, epoch, expiry, and release time with the adapter's normalized execution context.
+3. Treat an old epoch, old lease UUID, expired lease, or released lease as non-authoritative. Do not retry it as the old holder.
+4. Recovery uses the authorized takeover path with the observed expected epoch. Ordinary acquire is intentionally rejected for expired/released rows.
+5. If an external effect may already have happened, preserve evidence and do not assume lease fencing provides exactly-once replay safety.
+
+## Migration and rollback safety
+
+Migration `0016_salty_morlocks.sql` is additive: it creates two new tables and indexes without modifying existing authorization/session tables. Before rollout, normal database backup and migration verification still apply. Rolling application code back leaves unused additive tables in place; dropping tables is not part of automated rollback because it would destroy lease/audit evidence.
+
+## Security constraints
+
+- Tenant comes from authenticated gateway context, never a connector request field.
+- Logical agent, binding, connector, and scope identifiers use normalized constrained forms.
+- Takeover requires explicit gateway policy authorization and an expected epoch.
+- Default defense-in-depth TTL caps are 5 minutes for leases and 30 seconds for grants; policy may enforce stricter limits.
+- Validation and rejection audit complete before adapter side effects.
+- Existing authz and exact-action approval controls remain additional required gates; a valid connector lease does not bypass them.
--- a/docs/scratchpads/755-mos-logical-identity-fencing.md
+++ b/docs/scratchpads/755-mos-logical-identity-fencing.md
@@ -0,0 +1,121 @@
+# Issue #755 — Logical Mos identity and connector lease fencing
+
+- Task: `MOS-PORT-M1-001`
+- Branch: `feat/mos-logical-identity-fencing`
+- Base: `origin/main`
+- Started: 2026-07-14
+- Working budget: 38K tokens (task ledger estimate); one implementation lane, bounded to M1.
+
+## Objective
+
+Implement the first runtime-portability security boundary: normalized logical-agent identity plus a PostgreSQL-durable exclusive connector lease and server-validated fencing grants.
+
+## Scope
+
+- Normalized identity contract independent of harness/provider-native session IDs.
+- DB migration/schema/repository for one lease per tenant/logical-agent/binding.
+- CAS acquire/takeover, monotonic epoch, TTL, heartbeat, release, expiry handling.
+- Server-derived grants bound to tenant, logical agent, binding, connector, scopes, expiry, and lease epoch.
+- Reject and credential-safely audit stale, expired, forged, unauthorized, cross-tenant, and cross-binding grants before adapter side effects.
+- Runtime adapter boundary consumes normalized lease context.
+- Unit, migration, close/reopen, concurrency, abuse, and gateway integration tests.
+- Required developer/operations documentation for schema and security behavior.
+
+## Explicit exclusions
+
+No checkpoint/handoff payloads, exactly-once journal/receipts, concrete Claude/Pi/Codex harness adapter, channel cutover, or full cross-harness failover E2E.
+
+## Plan (TDD RED → GREEN → REFACTOR)
+
+1. Map existing contracts, DB/migration conventions, gateway authorization/audit boundaries, and test infrastructure.
+2. Add failing contract/repository/concurrency/restart/abuse/gateway tests and capture RED evidence.
+3. Implement the smallest normalized contracts, schema/migration/repository, grant validator, audit sink, and gateway service/adapter boundary needed to pass.
+4. Refactor for clear invariants and credential-safe observability; rerun focused suites.
+5. Run package/repo typecheck, lint, format, and appropriate tests.
+6. Run independent code + security review, remediate, and re-review.
+7. Inspect the final diff for security/scope drift; commit; queue guard; push; open PR with `Refs #755` and exact verification; stop without merge/issue closure.
+
+## Constraints and safety notes
+
+- `docs/tess/TASKS.md` is orchestrator-only and will not be edited.
+- Existing dirty `.mosaic/orchestrator/mission.json` and `.mosaic/orchestrator/session.lock` are launcher/orchestrator state and will not be staged or altered intentionally.
+- No client-supplied identity may confer authority.
+- No credential, token, or raw grant material may be persisted to audit/log output.
+- Existing authorization checks remain intact; fencing is an additional fail-closed layer.
+
+## Assumptions resolved from existing architecture
+
+- `ASSUMPTION:` M1 exposes no public lease endpoint. The gateway service is an internal policy surface with deny-all default policy because concrete connector activation/cutover is explicitly deferred.
+- `ASSUMPTION:` Fencing epochs use PostgreSQL `bigint` and cross-module decimal strings, preserving JSON portability without JavaScript number precision loss.
+- `ASSUMPTION:` Process-local grant provenance intentionally fails closed across restart; durable lease/epoch state survives and fresh grants require current policy + lease validation.
+
+## TDD evidence
+
+RED observed before implementation:
+
+- `corepack pnpm --filter @mosaicstack/types exec vitest run src/agent/connector-lease.dto.spec.ts` → failed to load missing `connector-lease.dto.js`.
+- `corepack pnpm --filter @mosaicstack/agent exec vitest run src/connector-lease.test.ts` → failed to load missing `connector-lease.js`.
+- Gateway focused tests failed before implementation because the new repository/service boundaries did not exist (workspace dependencies were then built before behavioral GREEN runs).
+
+GREEN to date:
+
+- Types contract: 6/6 passed.
+- Agent grant/fencing unit suite: 5/5 passed.
+- Gateway PGlite repository + policy/side-effect integration: 7/7 passed; 1 real-PostgreSQL test skipped when `DATABASE_URL` absent.
+- Real PostgreSQL focused run with configured `DATABASE_URL`: 1/1 passed (credential value not emitted in reports).
+
+## Documentation checklist
+
+- [x] `docs/PRD.md` contains current MOS-PORT M1 scope and acceptance criteria.
+- [x] Developer architecture: `docs/architecture/mos-runtime-portability-m1.md`.
+- [x] Admin/operations guidance: `docs/guides/mos-connector-lease-operations.md`.
+- [x] `docs/SITEMAP.md` links both pages.
+- [x] No user-guide change: M1 exposes no user-facing flow or channel cutover.
+- [x] No OpenAPI/endpoint-index change: M1 adds no HTTP endpoint.
+- [x] Migration/restart/rollback safety and credential-safe audit constraints documented.
+- [x] Canonical source remains in-repo; no external publishing action is in scope.
+- [x] Independent review confirms documentation matches implementation; implementation-specific findings were remediated.
+
+## Independent review and remediation
+
+Codex code/security review ran in multiple rounds. Findings and root-cause remediations:
+
+1. Policy could not inspect requested scope/TTL → policy subject now receives normalized requested scopes and explicit requested TTL.
+2. Unbounded authority lifetime → hard defaults cap leases at 5 minutes and grants at 30 seconds; overrides may only tighten; over-limit tests added.
+3. Cross-tenant denial could audit under submitted tenant → mismatch audit uses authenticated tenant plus sanitized `untrusted` target metadata; integration assertion added.
+4. Malformed forged grant could break the denial/audit path → runtime-safe shape validation with sanitized fallback audit; malformed-input test added.
+5. Gateway integration test depended on prior test state → denial test now seeds a unique binding itself; isolated `-t` run passed.
+6. Reviewer repeatedly identified launcher-generated `.mosaic/orchestrator/*` state; those files remain unstaged and excluded from the implementation commit.
+
+Latest independent security review: no critical/high/medium/low findings. Final commit-level code review remains to run after the intended diff is committed without launcher state.
+
+## Verification evidence
+
+- Focused contracts/fencing: types 6/6; agent 9/9.
+- Gateway focused PGlite repository/policy integration: 7/7; isolated denial test 1/1.
+- Real PostgreSQL close/reopen/CAS test: 1/1 with configured `DATABASE_URL`.
+- Root `corepack pnpm typecheck`: 42/42 Turbo tasks passed.
+- Root `corepack pnpm lint`: 23/23 Turbo tasks passed.
+- Root `corepack pnpm format:check`: all matched files passed.
+- Root `corepack pnpm test`: 42/42 Turbo tasks passed; gateway 616 passed / 12 environment-gated skipped; DB 19 passed / 7 environment-gated skipped; Mosaic 650 passed.
+
+## Known residual risks
+
+- Concrete connector policies and Claude/Pi/Codex adapters are intentionally deferred; production policy defaults deny-all.
+- Gateway pre-side-effect validation cannot make an external system exactly-once. Adapters must propagate/enforce the epoch at downstream effect boundaries; receipts/journaling are later #754 scope.
+- Migration rollback is additive-only; dropping lease/audit tables is intentionally manual to avoid destroying authority/audit evidence.
+
+## Commit-level review remediation
+
+- Commit-level Codex code review found one `should-fix`: heartbeat, release, and grant issuance authorized caller-supplied lease fields before canonical normalization.
+- TDD RED: the isolated gateway policy-boundary test showed mixed-case/padded logical agent, binding, connector, scope, and epoch values reaching policy unchanged.
+- Remediation: exported the coordinator's canonical lease normalizer and applied it at the gateway boundary before tenant/policy checks and coordinator dispatch for heartbeat, release, and grant issuance.
+- GREEN: isolated policy test 1/1; focused types 6/6, agent 9/9, gateway 8/8; root typecheck 42/42, lint 23/23, format check passed, and root tests 42/42 (gateway 617 passed / 12 environment-gated skipped).
+- Commit-level security review remained clean: no critical/high/medium/low findings.
+
+## Durable grant-expiry review remediation
+
+- Final commit review found a second `should-fix`: grant expiry was capped against submitted lease metadata after current-authority validation, rather than the durable lease row.
+- TDD RED: a crafted same-authority lease with a later submitted expiry produced a grant expiring after the durable row.
+- Remediation: grant authority fields and expiry now derive from the durable current lease; submitted scopes remain an additional narrowing constraint.
+- GREEN: focused agent fencing suite 10/10.