diff --git a/BUILD-LOG.md b/BUILD-LOG.md index 88a8df33a..f7f7eb24f 100644 --- a/BUILD-LOG.md +++ b/BUILD-LOG.md @@ -5185,3 +5185,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi 2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls. 2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls. + +2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls. diff --git a/docs/SESSIONS.md b/docs/SESSIONS.md index 4d5efd9d0..c4c1a4ba7 100644 --- a/docs/SESSIONS.md +++ b/docs/SESSIONS.md @@ -1423,3 +1423,4 @@ are never rewritten or removed; corrections are new entries. 2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge 2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls 2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls +2026-10-21T20:48:49Z | Huey (gpt-6.1-sol) | mos-1206, Q198 r5 protected mode/outage recovery, Vikunja 452 | Q205/Q206 recorded with labelled readings and probe/resume acceptance proposals; Q4 superseded, Q22 pending; doc checks only, no builds/suites/real probes/live changes/paid calls; Scrooge re-review via Mos diff --git a/docs/plans/2026-10-21_q198-system-manager.md b/docs/plans/2026-10-21_q198-system-manager.md index f086a4747..8a608fb49 100644 --- a/docs/plans/2026-10-21_q198-system-manager.md +++ b/docs/plans/2026-10-21_q198-system-manager.md @@ -4,6 +4,7 @@ Build a rules-first manager for admission, bounded retries, recovery and host he System stop and health requests bypass the work queue, without bypassing authorization. Normal and background work wait under pressure, then resume gradually and fairly. An interrupt requests cancellation. Capacity stays occupied until the owner proves exit. +Outages enter system-owned protected mode, then probe and resume automatically. Unknown external effects require reconciliation, never automatic replay. Reuse the bus, controllers, launch registry, run evidence and Console. Optional decision models rank already-permitted routes. Recommend Jev when enabled. @@ -12,8 +13,8 @@ No live trial, key installation or runtime change is authorized by this design. MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate. Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only. -Revision **r4**, mos-1203, records Jason’s DECISIONS Q204 answers to section 9 -on r3 `a36952564`. Q4 remains pending. +Revision **r5**, mos-1206, applies DECISIONS Q205/Q206 to r4 `1e8bccff1`. +Q4 is superseded. Q22 remains pending. New implementation choices are Huey’s readings. All defaults below are proposals, not measured safe operating limits or new authority. ## 1. Existing pieces and reuse @@ -83,7 +84,7 @@ handling outside that pool. This is software scheduling, not hardware/real-time content reference plus digest (content stays at its authorized source), policy revision, class, enqueue time/deadline, resource estimate, allowed routes and idempotency key. Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`, -with branches to `retry-wait`, `dead-letter`, or `uncertain`. Record every attempt, +with branches to `retry-wait`, `outage-paused`, `dead-letter`, or `uncertain`. Record every attempt, route, reason code, owner identity, heartbeat and evidence references. Materialized state is rebuildable. Re-check permission, kill state and policy at dispatch/replay. A duplicate key with the same digest returns the original request. A changed digest @@ -111,15 +112,15 @@ until a shared ledger and its authority boundary receive separate review. | Ref/class | Requests | Admission and preemption | | --- | --- | --- | | P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. | -| P1 system | Lightweight health reads, lease reconciliation, approved recovery | Reserved control capacity. Health/reconciliation can run under pressure. Recovery that launches heavy work still needs capacity and kill-clear. Never auto-clear a kill. | +| P1 system | Lightweight health reads, lease reconciliation, automatic outage recovery under Q205/Q206 | Reserved control capacity. Health/reconciliation can run under pressure. Resume needs capacity, current policy and a clear human kill switch, without a new approval per session. Never auto-clear a kill. | | P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. | | P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. | -The P1 capacity/authorization hold narrows Jason’s Q198 “take precedent” direction: -system recovery gets priority over eligible normal work, but heavy recovery cannot -bypass authorization, kill state or hard capacity. This is a proposed safety restriction, -not a claim that Jason already approved that exception. Bounded health and reconciliation -retain reserved capacity. Section 9 Q4 puts this restriction to Jason for approval. +DECISIONS Q205/Q206 supersede the old P1 per-session recovery approval question. +Huey’s reading: the configured recovery policy supplies standing authority to resume +eligible paused work. Current permissions, hard capacity and the human kill switch still +apply. Health/reconciliation retain reserved capacity. Protected mode can clear itself, +but cannot clear a human kill or grant new authority. | Work | What interrupt means | Hard-stop boundary | | --- | --- | --- | @@ -158,7 +159,8 @@ explicitly supported exclusive envelope, never an unbounded resource guess. Star then measure before raising concurrency. Provider cooldown blocks only affected provider/account/model scopes, not all host work. An open provider circuit waits at least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on -failure. Never shorten an upstream wait to fit a local retry delay. +failure. Never shorten an upstream wait to fit a local retry delay. A declared outage +uses D13–D15’s Q206 probe schedule instead of this short transient-failure circuit. These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and @@ -171,7 +173,8 @@ P0 uses a separately bounded, target-deduplicated control path. Journal failure new work and alerts, without disabling the existing emergency stop path. Normal requests expire after 30 minutes waiting, background after 24 hours, unless an -explicit authorized deadline differs. Show expiry and retain its receipt. Once healthy, +explicit authorized deadline differs. Outage-paused time is excluded under D6. +Show expiry and retain its receipt. Once healthy, weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant. After 10 minutes, background ages into normal scheduling rank, never into system priority and never through pressure holds. After control bursts, service eligible work. @@ -181,13 +184,28 @@ alert on oldest wait >10 minutes rather than promise starvation freedom under ov ## 5. Retries, dead letters and routes **D6 Retry ownership.** One logical request gets at most **3 total manager attempts**, -10 minutes of retry time and its original deadline, whichever ends first. Retry only +10 minutes of retry time and its effective deadline, whichever ends first. Retry only transient failures known to precede effects, or operations with verified idempotency. Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum. Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal retries must fit a declared per-attempt budget. If that budget is unknown, do not layer automatic manager retries over the proxy. Max one route change within the three attempts. +**Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt +allowance and remaining ten-minute retry window. Neither resets nor advances during the +pause, including the outage-causing failure once classified as an outage. It never enters +the DLQ because of that outage. Huey’s implementation reading: track provisional attempt +outcomes until classification so a threshold-crossing failure cannot first dead-letter work. Resume is continuation of +the same logical request, not D7 replay. Actual unrelated failures still count normally. + +Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline +budgets during the union of outage-paused intervals, never counting overlapping service +outages twice. Keep the original timestamp and publish the effective deadline extended +by paused time. This deliberately interprets a caller-set deadline as an execution budget, +not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation +still takes effect during a pause. Journal pause intervals and counters across restarts. +An interrupted effect remains uncertain until reconciled, regardless of the clock. + D6 covers managed work requests only. Existing notifier delivery backoff (`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff (`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own @@ -196,13 +214,16 @@ Any later consolidation needs its own design and acceptance tests. | Outcome | Action | | --- | --- | -| Connection refused before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility. | +| Mechanically classified outage (D13) | Outage-paused, counters/clocks frozen, service probes under D14. No outage DLQ. | +| Isolated connection refusal before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility and D13 does not classify an outage. | | Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. | | Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. | -| Transient retry budget/deadline exhausted, deterministic execution failure | Dead-letter with evidence and actionable reason. Admission expiry is an expired terminal item. | +| Non-outage retry budget/deadline exhausted, deterministic request failure | Dead-letter with evidence and actionable reason. Non-outage admission expiry is an expired terminal item. | | Explicit user cancellation | Terminal cancelled. Never retry automatically. | -**D7 DLQ.** A durable request state and append-only disposition history, not copied +**D7 DLQ.** Only requests that fail on their own use this path, not outage-paused work. +Validation/permission refusals remain refusals under D6, with their request evidence. +A durable request state and append-only disposition history, not copied prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted error code, idempotency/effect status and links to authorized evidence in Console “System → Requests → Dead letters”. Propose equivalent `mosaic system requests @@ -212,6 +233,8 @@ replay. Jason or an explicitly delegated role can replay after cause correction reconciliation. Replay creates a new request linked to its parent, with a fresh bounded budget and current policy. No mass replay, unattended replay chains or resurrected cancelled work. Unknown effects require explicit reconciliation evidence before replay. +D15’s automatic batch continuation of outage-paused work is not a replay chain and +requires no person to release each request. An outage never sends those requests here. **D8 Rerouting.** Candidate routes come only from the user’s configured allowlist, filtered for model capability, tool protocol, context, privacy/region, account budget @@ -221,6 +244,88 @@ or migrate a live transcript silently. Mid-turn failure is reconciled before a f attempt. Record requested and effective route and reason. New provider consent/spend is separate from enabling the decision hook. F10’s preflight fallback remains distinct. +## 5a. Protected mode and outage recovery + +**D13 Protected mode (Huey’s reading of Q205/Q206).** The supervisor owns a durable +`normal → protected → checking → canary → batch-resume → normal` state per affected +service scope, with a host/network incident able to hold all dependent scopes. This is +separate from the human kill switch. D1 remains unchanged: the manager never sets or +clears kill-switch state. Detection/status tracking are mechanical, without an LLM. + +Proposed entry thresholds, all Huey’s readings: three consecutive failed network checks +at the 2-second sample cadence before protected mode enter a network incident. +Once protected, active checks follow D14’s schedule. For a provider/account scope, +enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient +network/429/5xx failures across at least two distinct requests, or three consecutive +failed endpoint checks. One explicit account-wide quota/reset response can enter its +scope immediately. Isolated validation/permission failures do not count. Until wider +failure scope is proven, hold only that route/account. A missing required observation +holds work as unknown rather than guessing recovery. Persist incident and classification +before advancing affected request outcomes. These settings use D3’s reviewed config path. + +Fence affected new dispatch. Ask running owners to pause at safe points using F5/F6. +Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed +rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable, +record uncertain ownership and retain capacity. Do not force-stop a write or manufacture +human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects +mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected +services can continue under admission, except where D14’s default recovery order holds +them pending the JEV stage. No person releases each paused session. + +**D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first +performs mechanical network and endpoint checks. Only after the endpoint answers may a +one-turn headless probe run through a minimal harness config. A check of a dead link +makes no model call. Huey’s reading: use a fixed synthetic input, no tools, user content, +workspace effects or inherited worker environment, and separately authorized probe +credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat +request. Probe success means the endpoint plus the configured minimal harness/model +path succeeded, not merely an HTTP connection. One successful complete probe marks that +service online. Missing credentials, exhausted budget or an unknown charge keep it +unverified, without synthetic success or unauthorized spend. + +Default order is Jason’s: **JEV first**, then the other harness/provider checks start +**in parallel** after JEV succeeds. Each service has its own durable probe schedule: +wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540, +600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least +Retry-After or the usage-limit reset before probing, even if longer than 600 seconds. +The ceiling bounds this schedule, not a provider-mandated wait. Unknown reset time +keeps the scope held and visible rather than guessing a usable quota. Service probes +are separate from D10 routing calls and never retry a paused work request. + +Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check +deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup +and uncertain outcomes retain their reservations. These deadlines are independent of +the routing hook’s 500 ms candidate timeout. Scheduling retries indefinitely does not +override D11 budgets: when funds/caps are unavailable, keep mechanical checks running +on schedule and defer paid headless probes until authorized budget is available. A human +kill prevents headless probes/agent launches, while local read-only health tracking +continues. The kill can be cleared only through its existing human path. + +Huey’s reading for disabled optional JEV: if the user never enabled/configured it, +record its stage as not configured and begin the other checks without a paid JEV call. +If JEV is configured but unavailable, keep the specified JEV-first order. A JEV-only +outage can therefore delay otherwise available harnesses. Q22 proposes changing that +order and remains pending, so its alternative is not implemented by this design. + +**D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service +online proof and D4’s healthy release thresholds, reconcile paused owners/effects and +run **one test session** on an eligible online route. Use disposable synthetic work, +current policy and bounded capacity. No request with uncertain effects is used as a +test. On success resume eligible paused work in batches of at most five, with one +30-second healthy observation interval between batches and D4’s stricter dispatch +rate/reservations still enforced. A batch needs no new approval per session. If the +test session or a batch fails, return to protected mode and its probe schedule. +Request-local defects still follow D6/D7, with evidence distinguishing that failure +from a provider outage rather than falsely marking a healthy service offline. + +Huey’s proposed escalation count: after **three consecutive failed automatic exits** +(canary or batch stages) in the same incident, alert Mos/Jason and hold production +resume for diagnosis. Continue mechanical monitoring and budget-permitted probes +without an attempt limit. A fully successful exit resets the exit-failure count. This +is the repeated-exit boundary where a person is needed, not a prompt for each session +or each failed probe. Existing policy, missing credentials and unresolved external +effects remain fail-closed constraints, never reasons to improvise authority. + ## 6. Monitoring and Console **D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical @@ -235,7 +340,9 @@ an approved re-pin window after MVP. No sampler is started by this design work. **Measurements.** Pressure samples and freshness, active/reserved work, ownership uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency, retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state, -manager heartbeat and decision-model timeouts/spend/unknown charges. Store bounded +manager heartbeat and decision-model timeouts/spend/unknown charges. Huey’s reading: +also expose protected-mode scope, last/next probe, online proof, paused request counts, +canary/batch progress and consecutive exit failures. Store bounded aggregate samples for 7 days, without request bodies. Request receipts/evidence follow the instance’s retention policy, with the existing write-once run invariant preserved. @@ -245,7 +352,8 @@ Requests, Dead letters and evidence. Reuse Console read transport, auth and scop human writes (F8). No new public listener. Show stale/unknown explicitly, not green. Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and -unknown decision charge. Warn on pressure hold >60 seconds, control heartbeat stale +unknown decision charge. Huey’s reading: show protected-mode entry/recovery and alert +on D15’s repeated-exit boundary. Warn on pressure hold >60 seconds, control heartbeat stale >6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit one recovery notification. Use Console attention/trail first. External notification channels require their own configured authorization. A lightweight supervisor watchdog @@ -281,12 +389,24 @@ confidence value therefore uses rules. Discard late answers for routing, but still account for their charge. The hook cannot authorize work, choose P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the eligible set. Recheck eligibility after it returns. Never call it for safety actions. +Huey’s reading of Q205/Q206: reachable JEV may advise on resume order and route choice +among services mechanically marked online, within existing policy/capacity/budget. +It cannot mark a service online, authorize a resume, clear a human kill, replay effects +or perform safety actions. If unavailable, fixed rules choose eligible routes subject +to D14’s default probe order. Its absence does not change stop/effect proof requirements. **D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**. They live in the instance’s one system config through D3’s reviewed schema/bootstrap path. Both configured caps are enforced by the owning instance’s durable ledger. -No spending starts until Jason enables paid decisions and installs the key. Each instance +No spending starts until Jason enables paid decisions and installs the key. +Huey’s reading of Q206 probe accounting: charge all recovery headless/test-session probe +spend and decision calls to these same daily/monthly instance caps, not a separate free +probe allowance. The ledger groups the budget under the exclusive decision-key binding +and records each actual provider/key identity, including probes that use another provider. +Those probe credentials remain separate from the decision key and need their own +provider-side ceilings. Reserve/settle all probe charges under the same rules below. +No unknown charge is zero and no outage grants fresh spend authority. Each instance must use a separately provisioned key used only for that instance’s decisions, with a non-secret provider key ID bound to the instance in the approved setup record. Exclusive ownership is a setup attestation, not something the manager can prove mechanically. @@ -340,8 +460,8 @@ authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/aut | S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. | | S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. | | S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. | -| S5 | Retry/DLQ/replay and permitted reroutes, after S3/S4 | Fake provider failures with exact total-attempt/time bounds, restart during backoff, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal. | After MVP. | -| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. | +| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, relapse and three failed exits alert, human kill unchanged. No real probes. | After MVP. | +| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, JEV-first then parallel, one success online, reset wait, disabled JEV, aggregate probe spend and budget deferral, restart and hung-probe cleanup. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. | Each slice gets independent review and tests that kill a mutation of its central invariant. Sage chooses package/full-suite gates appropriate to the final diff. @@ -367,14 +487,29 @@ bounded synthetic pressure. No load experiment on the shared development host. **Recommend yes**, per role and privacy requirement. Empty list means hold/fail visibly instead of silently changing providers or model quality. **Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes” -- **Q4 System recovery priority (pending, Jason restated 2026-10-21):** approve the - P1 narrowing of Q198 so heavy recovery - takes priority over eligible normal work but still waits for authorization, kill-clear - and hard capacity? **Recommend yes.** Keep health/reconciliation on reserved control - capacity, without letting a recovery launch bypass those safeguards. - DECISIONS Q204 Q20d is conditional on Jason’s reading about returning to normal - operations, which differs from the design’s P1 recovery admission restriction. Mos - has restated the distinction to Jason. Q4 remains pending, with P1 unchanged. +- **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.** + Jason, verbatim (Q205): “It does not make sense to require the user to release each + back into operation. I would rather equip the system with the tools necessary to + detect failure, enter protected mode, monitor for cool-down periods, check for + system operational status, resume operations.” + Jason, verbatim (Q206): + + > Mosaic Stack mechanical tooling detects the outage and tracks the status without LLM need + > Continual mechanical attempt for JEV begins with a cooldown of 60, 120, 180 seconds with a ceiling of 10 minutes + > Once JEV is back online, the same cooldown starts for the other harnesses/providers in parallel + > Necessary services are marked back online when found available + > System adapts based on available services, providers, harnesses, usage limits, and usage budgets + + Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate + protected-mode recovery without per-session approval. Entry thresholds, checkpoint + handling, deadline freezing, probe deadlines/budget grouping, disabled-JEV behavior, + batch sizing and the three-failed-exit escalation count are proposed readings, not + Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged. +- **Q22 Probe independence (pending):** may other harness/provider checks start without + waiting for JEV, with fixed rules routing until JEV returns? **Recommend yes**, as + Mos’s alternative recorded under DECISIONS Q206. Huey’s reading: that avoids a JEV-only + outage holding otherwise usable services while retaining policy/usage/budget checks. + Until Jason answers, D14 follows his JEV-first, then parallel order. ## Verification and handoff