239 lines
13 KiB
Markdown
239 lines
13 KiB
Markdown
# Tess / Option 2 runtime-portability qualification — 2026-07-14
|
||
|
||
**Issue context:** #706–#711 and runtime-neutral Mos follow-up #754
|
||
|
||
**Qualified revision:** `d0771835542d` (`origin/main` at review time)
|
||
|
||
**Reviewer/runtime:** Independent Pi lane requested as `openai-codex/gpt-5.6-sol:high`
|
||
|
||
**Runtime resolution note:** Mosaic warned that `gpt-5.6-sol` was not present in the provider model catalog and proceeded with it as a custom model ID. This warning was part of the original qualification log and is material provenance; downstream claims must not treat catalog recognition as verified.
|
||
|
||
**Verdict:** REQUEST CHANGES
|
||
|
||
**Evidence type:** Point-in-time qualification; later commits and PR #757 must be reviewed separately
|
||
|
||
## Purpose and provenance
|
||
|
||
This report preserves the complete independent qualification that was previously available only in `/tmp/tess-option2-qualification.log`. It distinguishes passing component tests from the missing operational proof required for identity-continuous Mos failover.
|
||
|
||
No credential values, OAuth tokens, Discord tokens, device codes, or auth-file contents are included. Commands and results are retained so another environment can reproduce or challenge the findings.
|
||
|
||
---
|
||
|
||
# 1. Verdict
|
||
|
||
## **REQUEST CHANGES**
|
||
|
||
The current Option 2 implementation is a useful portability foundation, but it is **not qualified against AC-TESS-01..11** and is not equivalent to true same-Mos-identity failover.
|
||
|
||
Primary blockers:
|
||
|
||
1. **AC-TESS-01/02:** The required `mosaic tess` command does not exist; only `mosaic interaction` is registered (`packages/mosaic/src/commands/interaction.ts:60`). The cross-surface test proves CLI enrollment followed by Discord approval/stop, not bidirectional Discord/CLI chat streaming.
|
||
2. **AC-TESS-04:** Fleet/tmux and Matrix providers are implemented as libraries but are not registered in the production gateway. `AgentModule` registers only Hermes (`apps/gateway/src/agent/agent.module.ts:34`).
|
||
3. **Mos handoff is not operational or durable:** Production uses `InMemoryInteractionCoordinationPort` (`apps/gateway/src/coord/coord.module.ts:18`), with no Mos-side consumer. Restart loses handoff ownership, idempotency, activity, and results.
|
||
4. **AC-TESS-06/10:** Restart tests are good local persistence tests, but no real connector/harness failover or exercised rollback exists. Rollback is documentation-only.
|
||
5. **AC-TESS-08:** The parity suite validates a selected shared intersection using mocked transports. Matrix is not production-wired and tmux drops the runtime message idempotency key before delivery.
|
||
6. **AC-TESS-09:** M5 qualification remains `not-started`; no live Discord, Matrix homeserver, tmux/Mos consumer, Claude Code/Pi/Codex failover, or deployment rollback was tested.
|
||
7. **PR #750 mismatch:** Its description promises send-error coverage as HTTP 400, but both gateway and TUI test use HTTP 403 (`packages/mosaic/src/tui/gateway-api.interaction-errors.test.ts:25-34`).
|
||
|
||
### AC disposition
|
||
|
||
| AC | Result | Evidence |
|
||
| --- | ----------------- | ------------------------------------------------------------------------------- |
|
||
| 01 | **Fail** | No `mosaic tess`; no bidirectional same-session chat/stream E2E |
|
||
| 02 | **Fail** | Generic CLI exists, but fleet/Matrix providers are unreachable in production |
|
||
| 03 | Pass | Pi profile/model/reasoning/effective-policy tests passed |
|
||
| 04 | **Fail** | No registered fleet provider or real Mos consumer |
|
||
| 05 | Partial | Hermes normalization/fail-closed matrix passes; live capability path is limited |
|
||
| 06 | Partial | PGlite restart/idempotency passes; no actual harness failover |
|
||
| 07 | Partial | Focused denial/replay tests pass; full M5 abuse qualification absent |
|
||
| 08 | Partial | Mocked shared-intersection parity passes; Matrix not operationally wired |
|
||
| 09 | **Fail** | Baselines/CI green, but required E2E/security/rollback qualification absent |
|
||
| 10 | **Fail** | Inventory incomplete/inconsistent; rollback not exercised |
|
||
| 11 | Pass/ledger stale | Documentation and sitemap exist; plugin/catalog ledger remains unresolved |
|
||
|
||
---
|
||
|
||
# 2. Exact test commands and results
|
||
|
||
Initial focused attempts failed before collection because this detached worktree had no dependencies:
|
||
|
||
```bash
|
||
pnpm --filter @mosaicstack/agent exec vitest run ...
|
||
```
|
||
|
||
Result: startup failure, `Cannot find module 'vitest/config'`.
|
||
|
||
Setup used:
|
||
|
||
```bash
|
||
corepack pnpm --store-dir /home/jarvis/.local/share/pnpm/store/v10 \
|
||
install --frozen-lockfile --ignore-scripts
|
||
```
|
||
|
||
Result: PASS, 1,240 packages linked.
|
||
|
||
```bash
|
||
corepack pnpm turbo run build \
|
||
--filter='@mosaicstack/gateway^...' \
|
||
--filter='@mosaicstack/mosaic^...'
|
||
```
|
||
|
||
Result: **17/17 dependency builds successful**.
|
||
|
||
### Focused suites
|
||
|
||
```bash
|
||
corepack pnpm --filter @mosaicstack/agent exec vitest run \
|
||
src/runtime-provider-parity.test.ts \
|
||
src/matrix-native-runtime-provider.test.ts \
|
||
src/tmux-fleet-runtime-provider.test.ts \
|
||
src/durable-session.test.ts \
|
||
src/hermes-runtime-provider.test.ts
|
||
```
|
||
|
||
Result: **5 files, 39/39 tests passed**.
|
||
|
||
```bash
|
||
corepack pnpm --filter @mosaicstack/gateway exec vitest run \
|
||
src/agent/durable-session.repository.test.ts \
|
||
src/__tests__/integration/tess-cross-surface.integration.test.ts \
|
||
src/plugin/discord-ingress.security.spec.ts \
|
||
src/coord/interaction-coordination.service.test.ts \
|
||
src/coord/interaction-coordination.routing.e2e.test.ts \
|
||
src/agent/hermes-runtime-reachability.e2e.test.ts
|
||
```
|
||
|
||
Result: **6 files, 36/36 tests passed**. PGlite close/reopen recovery passed in 504 ms.
|
||
|
||
```bash
|
||
corepack pnpm --filter @mosaicstack/mosaic exec vitest run \
|
||
src/fleet/matrix-native-runtime-transport.test.ts \
|
||
src/fleet/tess-service-profile.test.ts \
|
||
src/commands/interaction.test.ts \
|
||
src/tui/gateway-api.interaction-errors.test.ts
|
||
```
|
||
|
||
Result: **4 files, 15/15 tests passed**.
|
||
|
||
```bash
|
||
corepack pnpm --filter @mosaicstack/coord exec vitest run \
|
||
src/__tests__/interaction-coordination.test.ts
|
||
```
|
||
|
||
Result: **1 file, 7/7 tests passed**.
|
||
|
||
```bash
|
||
corepack pnpm --filter @mosaicstack/gateway exec vitest run \
|
||
src/agent/interaction.controller.test.ts \
|
||
src/commands/command-authorization.service.spec.ts \
|
||
src/agent/__tests__/runtime-provider-registry.service.test.ts
|
||
```
|
||
|
||
Result: **3 files, 27/27 tests passed**.
|
||
|
||
Focused total: **124/124 tests passed** after dependency setup.
|
||
|
||
### Baselines
|
||
|
||
```bash
|
||
TURBO_FORCE=true corepack pnpm typecheck
|
||
```
|
||
|
||
Result: **42/42 tasks successful**.
|
||
|
||
```bash
|
||
TURBO_FORCE=true corepack pnpm lint
|
||
```
|
||
|
||
Result: **23/23 tasks successful**.
|
||
|
||
```bash
|
||
corepack pnpm format:check
|
||
```
|
||
|
||
Result: **PASS — all files matched Prettier style**.
|
||
|
||
```bash
|
||
~/.config/mosaic/tools/woodpecker/pipeline-status.sh \
|
||
-r mosaicstack/stack -n 1796
|
||
```
|
||
|
||
Result: **SUCCESS** at `d0771835542d`; all test, build, sanitization, typecheck, lint, format, and publish steps green.
|
||
|
||
No tracked files outside the pre-existing `.mosaic/orchestrator/*` launcher changes were modified.
|
||
|
||
---
|
||
|
||
# 3. Stale ledger inconsistencies
|
||
|
||
1. `docs/tess/MISSION-MANIFEST.md` still says:
|
||
- current milestone M1;
|
||
- progress 0/5;
|
||
- M2/M3/M5 not started.
|
||
2. `docs/tess/TASKS.md` says:
|
||
- M4-V failed;
|
||
- M4-W-001 and TESS-PLG-001 in progress;
|
||
- M5-V not started.
|
||
3. Provider issue state conflicts:
|
||
- #707–#709 remain open although M1–M3 rows are recorded done/pass.
|
||
- #710 and #711 are closed although M4-V failed and M5-V is not started.
|
||
4. M5 work was marked done despite depending on failed M4-V.
|
||
5. TESS-M3-002 says `mosaic tess` is done, but only `mosaic interaction` exists.
|
||
6. PR #750 removed stale service references from operational docs, but `docs/tess/TASKS.md` still contains `MosCoordinationService` in historical notes.
|
||
7. `docs/tess/MIGRATION-INVENTORY.md` remains an “initial inventory” with several capabilities marked `adapt`; `M5-MIGRATION-INVENTORY.md` marks grouped capabilities deferred/fail-closed. Neither supplies the complete owner/evidence matrix AC-TESS-10 requires.
|
||
8. TESS-M2-FUP-001 remains real: the unkeyed SHA-256 compatibility branch still exists at `durable-session.repository.ts:427-431`.
|
||
9. TESS-PLG-001 claims catalog registration was folded into W-001, but production evidence shows provider registration in the gateway—not a completed `packages/mosaic` plugin catalog.
|
||
|
||
---
|
||
|
||
# 4. Gap to true same-Mos-identity failover
|
||
|
||
Current code can relaunch the same roster name under another runtime and can rebind a durable interaction session to another provider/runtime ID. That is **replacement**, not identity-continuous failover.
|
||
|
||
Missing pieces:
|
||
|
||
- No canonical logical Mos identity independent of harness-native session IDs.
|
||
- No exclusive connector lease or monotonic fencing epoch; session rebinding is effectively last-write-wins.
|
||
- No stale-holder rejection preventing the old harness from continuing side effects.
|
||
- No normalized Claude Code/Pi/Codex checkpoint/import/export adapters.
|
||
- No durable Mos coordination transport or Mos consumer.
|
||
- No canonical handoff containing mission/task refs, git state, causal sequence, pending operations, capability requirements, and acknowledgements.
|
||
- No end-to-end receipt journal across connectors.
|
||
- Matrix has deterministic transaction IDs, but tmux delivery discards `RuntimeMessage.idempotencyKey`.
|
||
- No fault-injection test transferring Mos among Claude Code, Pi, and Codex and then rolling back.
|
||
|
||
---
|
||
|
||
# 5. Minimal follow-up issue decomposition
|
||
|
||
| Order | Issue | Minimum acceptance criteria |
|
||
| ----- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| 1 | **Logical identity and security fencing** | Server-derived `{tenant, logicalAgentId, connectorId, harness, leaseEpoch, scopes, expiry}`; signed/fenced execution grant; stale/forged/cross-tenant grants denied and audited; no connector credential in handoffs |
|
||
| 2 | **Durable connector lease** | PostgreSQL-backed exclusive lease with CAS, monotonic epoch, TTL/heartbeat, explicit takeover, and gateway rejection of stale holders; connectors for Claude Code, Pi, and Codex |
|
||
| 3 | **Canonical handoff/checkpoint** | Versioned, sealed schema containing canonical mission/task/git references, checkpoint digest, causal sequence, required capabilities, pending/ambiguous operation references, and source/destination acknowledgement; no raw secrets or mandatory harness transcript |
|
||
| 4 | **Exactly-once connector journal** | Durable operation IDs and receipts; idempotency propagated through every adapter; Matrix transaction mapping; tmux replaced or wrapped with receiver-side durable dedupe; ambiguous effects remain held for authorized reconciliation |
|
||
| 5 | **Cross-harness failover and rollback E2E** | Real Mos identity moves Claude Code → Pi → Codex and back; inject crashes before/after lease transfer, handoff persistence, send, and acknowledgement; stale connector fenced; no duplicate side effects; canonical state preserved; rollback evidence published |
|
||
| 6 | **Generic gateway research ADR** | Evaluate LiteLLM subscription OAuth and Bifrost concepts without adding either to core; include terms/security review, credential lifecycle, tenant mapping, budgets, failover semantics, and adapter-only prototype |
|
||
|
||
## Generic gateway placement
|
||
|
||
Allowed topology:
|
||
|
||
```text
|
||
Discord / CLI / web
|
||
↓
|
||
Mosaic Gateway: auth, tenant scope, policy, approvals, audit
|
||
↓
|
||
IProviderAdapter / AgentRuntimeProvider
|
||
↓
|
||
optional LiteLLM or Bifrost egress proxy
|
||
↓
|
||
upstream provider
|
||
```
|
||
|
||
- **LiteLLM ChatGPT subscription OAuth:** research-only, opt-in, behind an adapter. Subscription credentials require explicit terms, revocation, scope, token-storage, and audit review. They must never become Mosaic identity or core configuration.
|
||
- **Bifrost:** virtual keys are downstream proxy credentials, not Mosaic principals. Budget and failover concepts may inform Mosaic routing, but tenant policy, authorization, and audit remain in Mosaic.
|
||
- Neither product may introduce schemas into Mosaic core, receive direct calls from channels/agents, or bypass `IProviderAdapter`/`AgentRuntimeProvider`.
|
||
- Mosaic should also correct its existing “all providers unhealthy → use one anyway” fallback behavior before adopting more automatic failover (`routing-engine.service.ts:204-212`).
|