diff --git a/BUILD-LOG.md b/BUILD-LOG.md index f7f7eb24f..45517aa8d 100644 --- a/BUILD-LOG.md +++ b/BUILD-LOG.md @@ -5187,3 +5187,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi 2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls. 2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls. + +2026-10-21T21:04:37Z | Huey | Q198 design r6, mos-1208/mos-1210, DECISIONS Q207, scrooge-0003, Vikunja 452 | Before: r5 fd44f7c60 gated probes on JEV and lacked request-local resume failure separation. After: parallel independent probes; central JEV/smaller-model/rules endpoint chain, per-model timeout/calibration, 4 KiB metadata limit and isolated fallback credentials; subscription quota vs metered USD accounting and separately enabled paid-probe budget (proposed $0 defaults). S-F1–S-F7: bounded outage classification, request-local D6/two-resume DLQ rule and batch definition; escalation latch with authorized revision-bound system.recovery.release separate from kill clear; cooperative F5 pause/F6 proof only; proposed configurable 24-hour cumulative pause expiry; full Q205 kill sentence with labelled protected-mode interpretation; r3–r6 history. S5/S6 acceptance proposals updated. Verification: Q205/Q207 quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned references resolve, summary within 15 lines, manual consistency/delta review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes, credential reads or paid calls. diff --git a/docs/SESSIONS.md b/docs/SESSIONS.md index c4c1a4ba7..143a6e557 100644 --- a/docs/SESSIONS.md +++ b/docs/SESSIONS.md @@ -1424,3 +1424,4 @@ are never rewritten or removed; corrections are new entries. 2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls 2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls 2026-10-21T20:48:49Z | Huey (gpt-6.1-sol) | mos-1206, Q198 r5 protected mode/outage recovery, Vikunja 452 | Q205/Q206 recorded with labelled readings and probe/resume acceptance proposals; Q4 superseded, Q22 pending; doc checks only, no builds/suites/real probes/live changes/paid calls; Scrooge re-review via Mos +2026-10-21T21:04:37Z | Huey (gpt-6.1-sol) | mos-1208/mos-1210, Q198 r6, Q207 and scrooge-0003 S-F1–S-F7 | central fallback/parallel probes, request failure and recovery release fixes, quota/probe budgets and credential isolation documented; doc checks only, no builds/suites/probes/live changes/paid calls; Mos routes r6 to Scrooge diff --git a/docs/plans/2026-10-21_q198-system-manager.md b/docs/plans/2026-10-21_q198-system-manager.md index 8a608fb49..bf2023c43 100644 --- a/docs/plans/2026-10-21_q198-system-manager.md +++ b/docs/plans/2026-10-21_q198-system-manager.md @@ -13,8 +13,8 @@ No live trial, key installation or runtime change is authorized by this design. MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate. Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only. -Revision **r5**, mos-1206, applies DECISIONS Q205/Q206 to r4 `1e8bccff1`. -Q4 is superseded. Q22 remains pending. New implementation choices are Huey’s readings. +Revision **r6**, mos-1208/mos-1210, applies DECISIONS Q207 and scrooge-0003 to +r5 `fd44f7c60`. Q22 is resolved. New implementation choices are Huey’s readings. All defaults below are proposals, not measured safe operating limits or new authority. ## 1. Existing pieces and reuse @@ -104,7 +104,8 @@ name scan grants cancellation authority over another user or instance. Configura extends the instance’s one system config through the normal reviewed schema/bootstrap path, never a second configuration file or automatic rewrite of an existing config. Decision spending is separately scoped per exclusive key, not per host or shared -account. D11 requires one dedicated decision key per instance. Shared keys are unsupported +account. When JEV is enabled, D11 requires its dedicated key per instance. Other +endpoints use their isolated credential/billing contracts in D11/D12. Shared keys are unsupported until a shared ledger and its authority boundary receive separate review. ## 3. Classes and interrupt semantics @@ -194,9 +195,16 @@ automatic manager retries over the proxy. Max one route change within the three **Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt allowance and remaining ten-minute retry window. Neither resets nor advances during the pause, including the outage-causing failure once classified as an outage. It never enters -the DLQ because of that outage. Huey’s implementation reading: track provisional attempt -outcomes until classification so a threshold-crossing failure cannot first dead-letter work. Resume is continuation of -the same logical request, not D7 replay. Actual unrelated failures still count normally. +the DLQ because of that outage. Huey’s implementation reading: correlate a candidate +outage for at most 30 seconds, then settle each failure as either an evidenced D13 +outage or a request-local failure. A lone bad request cannot freeze itself indefinitely. +Resume continues the same logical request, not D7 replay. After a successful test session, +request-local failures in resumed work consume that request’s D6 attempts/time normally. +Persist a separate count of request-local failures across distinct resume episodes. +A request failing in two separate resumes goes to D7 with both episodes’ evidence, +even if its three-attempt allowance is not exhausted. Correlated outage failures do +not count toward that two-resume rule. These classification/counting details are Huey’s +reading. They prevent one defective request from repeatedly claiming an outage exemption. Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline budgets during the union of outage-paused intervals, never counting overlapping service @@ -205,6 +213,11 @@ by paused time. This deliberately interprets a caller-set deadline as an executi not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation still takes effect during a pause. Journal pause intervals and counters across restarts. An interrupted effect remains uncertain until reconciled, regardless of the clock. +Huey’s reading: configure a maximum outage-pause age, default **24 hours of cumulative +paused wall time per request**, through D3. On reaching it, make the request terminal +`expired`, never DLQ. Show pause age, limit and effective deadline in the Console. +Expiry prevents dispatch but never releases capacity still held by an unproven stop. +This maximum bounds the deadline extension above and does not authorize effect replay. D6 covers managed work requests only. Existing notifier delivery backoff (`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff @@ -258,19 +271,22 @@ Once protected, active checks follow D14’s schedule. For a provider/account sc enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient network/429/5xx failures across at least two distinct requests, or three consecutive failed endpoint checks. One explicit account-wide quota/reset response can enter its -scope immediately. Isolated validation/permission failures do not count. Until wider +scope immediately. Isolated validation/permission failures and model output/tool errors +do not count. Request-local failures after a successful test session use D6, never +D13 by themselves. Re-entry based on resumed requests requires the correlated failure +threshold across at least two distinct requests, not one bad request repeated. Until wider failure scope is proven, hold only that route/account. A missing required observation holds work as unknown rather than guessing recovery. Persist incident and classification before advancing affected request outcomes. These settings use D3’s reviewed config path. -Fence affected new dispatch. Ask running owners to pause at safe points using F5/F6. +Fence affected new dispatch. Pause uses only F5’s cooperative interrupt at safe points. +F6 supplies stop proof, never the pause action. Do not invoke force-stop for a pause. Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable, record uncertain ownership and retain capacity. Do not force-stop a write or manufacture human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected -services can continue under admission, except where D14’s default recovery order holds -them pending the JEV stage. No person releases each paused session. +services can continue under admission. No person releases each paused session. **D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first performs mechanical network and endpoint checks. Only after the endpoint answers may a @@ -280,11 +296,13 @@ workspace effects or inherited worker environment, and separately authorized pro credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat request. Probe success means the endpoint plus the configured minimal harness/model path succeeded, not merely an HTTP connection. One successful complete probe marks that -service online. Missing credentials, exhausted budget or an unknown charge keep it -unverified, without synthetic success or unauthorized spend. +service online. Missing credentials, exhausted applicable quota/budget or unknown +billing classification keep the headless check deferred/unverified. Subscription-only +probes need verified usage/quota accounting, not an invented per-call USD price. Paid +probe charges use D11’s separate probe budget. No synthetic success or unauthorized spend. -Default order is Jason’s: **JEV first**, then the other harness/provider checks start -**in parallel** after JEV succeeds. Each service has its own durable probe schedule: +Under Q207 and Mos’s recorded reading, **every service probes in parallel** on its own +schedule. JEV is not a gate for another service. Each has a durable probe schedule: wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540, 600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least Retry-After or the usage-limit reset before probing, even if longer than 600 seconds. @@ -295,17 +313,15 @@ are separate from D10 routing calls and never retry a paused work request. Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup and uncertain outcomes retain their reservations. These deadlines are independent of -the routing hook’s 500 ms candidate timeout. Scheduling retries indefinitely does not -override D11 budgets: when funds/caps are unavailable, keep mechanical checks running -on schedule and defer paid headless probes until authorized budget is available. A human +D10’s endpoint-specific decision timeouts. Scheduling retries indefinitely does not +override D11: when applicable quota or probe funds are unavailable, keep mechanical +checks running on schedule and defer headless probes until authorized capacity returns. A human kill prevents headless probes/agent launches, while local read-only health tracking continues. The kill can be cleared only through its existing human path. -Huey’s reading for disabled optional JEV: if the user never enabled/configured it, -record its stage as not configured and begin the other checks without a paid JEV call. -If JEV is configured but unavailable, keep the specified JEV-first order. A JEV-only -outage can therefore delay otherwise available harnesses. Q22 proposes changing that -order and remains pending, so its alternative is not implemented by this design. +Huey’s reading for optional endpoints: mark disabled/unconfigured JEV or a fallback as +not configured and make no call to it. Other services keep their independent schedules. +A failed or budget-deferred decision endpoint never blocks their probes. **D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service online proof and D4’s healthy release thresholds, reconcile paused owners/effects and @@ -313,10 +329,14 @@ run **one test session** on an eligible online route. Use disposable synthetic w current policy and bounded capacity. No request with uncertain effects is used as a test. On success resume eligible paused work in batches of at most five, with one 30-second healthy observation interval between batches and D4’s stricter dispatch -rate/reservations still enforced. A batch needs no new approval per session. If the -test session or a batch fails, return to protected mode and its probe schedule. -Request-local defects still follow D6/D7, with evidence distinguishing that failure -from a provider outage rather than falsely marking a healthy service offline. +rate/reservations still enforced. A batch needs no new approval per session. A failed +test session returns to protected mode with its evidence, without relabelling an +otherwise healthy provider as offline. After the test succeeds, “batch fails” means +D13 proves a correlated outage across distinct requests, or independent mechanical +network/endpoint evidence establishes an outage. Only that returns the scope to +protected mode. Individual resumed requests failing validation, tools or model output +consume D6 budgets and follow the two-resume/D7 rule, while healthy batch members +continue. A failed request alone never counts as a failed automatic exit. Huey’s proposed escalation count: after **three consecutive failed automatic exits** (canary or batch stages) in the same incident, alert Mos/Jason and hold production @@ -326,6 +346,18 @@ is the repeated-exit boundary where a person is needed, not a prompt for each se or each failed probe. Existing policy, missing credentials and unresolved external effects remain fail-closed constraints, never reasons to improvise authority. +Huey’s reading of release after three failed exits: persist `recovery-escalated` on the +incident. Later successful probes update service health but **do not** restart work +automatically while that latch is set. Jason, or a role already explicitly delegated +recovery authority, releases it through proposed human write `system.recovery.release` +(scope, expected incident ID and revision), also exposed as Console “System → Recovery → +Release after diagnosis”. The authenticated/scoped writer checks authority, current +incident/revision and diagnosis evidence. It clears only this escalation latch, resets +its exit counter and restarts the one-test-session stage, never bulk dispatch. It does +not clear a human kill, permissions, unresolved effects or resource holds. This design +adds no role grant. Without a delegated role, Jason is the releaser. S5 owns the new +broker/Console write and review, separate from the kill-switch clear. + ## 6. Monitoring and Console **D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical @@ -342,7 +374,7 @@ uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state, manager heartbeat and decision-model timeouts/spend/unknown charges. Huey’s reading: also expose protected-mode scope, last/next probe, online proof, paused request counts, -canary/batch progress and consecutive exit failures. Store bounded +canary/batch progress, pause age/limit and the escalation latch/release evidence. Store bounded aggregate samples for 7 days, without request bodies. Request receipts/evidence follow the instance’s retention policy, with the existing write-once run invariant preserved. @@ -364,9 +396,19 @@ reports manager death through an independent health path, not the dead manager i **D10 Interface.** `decide({version, snapshotId, policyRevision, deadline, allowedActions, state}, signal) → {actionId, confidence?, usage:{cost,currency}, providerRequestId}`. `allowedActions` contains opaque IDs for already-eligible routes -or “use rules”. Adapters normalize provider types. A deterministic rules adapter is -the default and needs no network or key. A generic structured-output adapter can support -other decision models without exposing their wire format to the scheduler. +or “use rules”. Adapters normalize provider types. **Every routing decision goes through +this one module. No agent session calls a decision endpoint or picks its own route.** +The module also chooses its own decision endpoint. Provider-neutral requests and results +keep endpoint wire formats out of sessions and the scheduler. With model assistance +disabled, the module uses built-in rules without network or key. + +When enabled, the module tries **JEV → one configured smaller fallback model → built-in +rules**. The smaller model may be Haiku, Luna or another on the user’s allowlist and +runs through a minimal harness config owned by this module. Rules last is **Mos’s +reading of Q207**, so a full outage does not require a reachable model. Huey’s reading: +one attempt per eligible model endpoint, no same-endpoint retries or unbounded chain. +Use only configured, online, calibrated and budget/quota-eligible endpoints. An +uncalibrated model never decides: the rules adapter supplies the decision in its place. Enabling paid assistance is a user setting, recommended in setup with its price/cap shown. Pin **`typesafe/jev-1.14`** as the recommended initial model, never a moving @@ -379,35 +421,60 @@ lists $0.042 per million input tokens, free output and 262,144 context tokens Use choice for route selection. Do not send images or use the large context window. Other decision models need their own adapter contract and calibrated confidence tests. -Initial timeout candidate **500 ms**, max one in-flight decision per instance, no -decision-call retry. S6 measures latency and timeout/fallback rates on delayed stand-ins, -then records the selected timeout and evidence before acceptance. Stand-ins verify the -local budget, not real provider latency. Timeout, invalid answer, stale snapshot, -missing/non-finite/out-of-range confidence, low confidence (<0.8 initially), cost -uncertainty or network failure immediately selects rules. An adapter without a calibrated -confidence value therefore uses rules. Discard late answers for -routing, but still account for their charge. The hook cannot authorize work, choose -P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the -eligible set. Recheck eligibility after it returns. Never call it for safety actions. -Huey’s reading of Q205/Q206: reachable JEV may advise on resume order and route choice -among services mechanically marked online, within existing policy/capacity/budget. -It cannot mark a service online, authorize a resume, clear a human kill, replay effects -or perform safety actions. If unavailable, fixed rules choose eligible routes subject -to D14’s default probe order. Its absence does not change stop/effect proof requirements. +**Endpoint contracts (Huey’s reading of Q207).** Each model has its own adapter, +output validation, timeout and confidence calibration. **500 ms is JEV’s candidate**. +A smaller model through a headless harness needs longer: propose **10 seconds end to +end**, including harness startup, as that fallback’s candidate. S6 uses delayed +stand-ins only to measure latency/timeout/fallback rates and records each selected +timeout and calibration. No real call or benchmark is authorized. Stand-ins measure +local behavior, not provider performance. Use at most one in-flight module decision +per instance, bounded by the caller deadline and these endpoint budgets. A fallback +never inherits JEV’s confidence threshold without its own calibration. + +Timeout/network failure advances to the next eligible endpoint within the deadline. +Invalid/stale answers, missing/non-finite/out-of-range or uncalibrated/low confidence +cannot choose a route. Use rules if no eligible calibrated model supplies a valid +answer. An exhausted caller deadline selects rules without waiting for another model. +Discard late answers for routing, retain their usage reservations, and bound owned +harness cleanup. The hook cannot authorize work, choose P0/P1 status, change thresholds, +clear a kill, replay effects or choose outside the eligible set. Recheck eligibility +at the final mechanical dispatch check. Never call models for safety actions. + +Huey’s reading of Q205–Q207: the module may advise on resume order and route choice +among services mechanically marked online, within policy/capacity/budget. It cannot +mark a service online, authorize resume, clear a human kill, replay effects or perform +safety actions. Its built-in rules remain inside the module when models are unavailable. +Stop/effect proof requirements remain unchanged. **D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**. They live in the instance’s one system config through D3’s reviewed schema/bootstrap path. Both configured caps are enforced by the owning instance’s durable ledger. No spending starts until Jason enables paid decisions and installs the key. -Huey’s reading of Q206 probe accounting: charge all recovery headless/test-session probe -spend and decision calls to these same daily/monthly instance caps, not a separate free -probe allowance. The ledger groups the budget under the exclusive decision-key binding -and records each actual provider/key identity, including probes that use another provider. -Those probe credentials remain separate from the decision key and need their own -provider-side ceilings. Reserve/settle all probe charges under the same rules below. -No unknown charge is zero and no outage grants fresh spend authority. Each instance -must use a separately provisioned key used only for that instance’s decisions, with a +**Huey’s reading of billing for Q206/Q207:** classify each adapter/credential as +subscription/quota-only or metered USD before enablement, with no automatic conversion +to a paid API. Fallback decisions using metered models share the configured D11 decision +caps with JEV. A subscription-only fallback (for example an authorized Claude OAuth or +ChatGPT-plan route) consumes its provider usage/quota limits, not a fabricated USD price. +Only a verified subscription-only contract establishes no incremental API charge. +Huey’s reading: reserve a bounded usage allowance before a subscription call, then +reconcile the provider’s measured usage. Unknown quota accounting holds that route. +Unknown billing/quota sources defer that endpoint and use the next eligible adapter. +No subscription purchase, paid overage or new model spend is authorized by fallback. + +Recovery headless probes and the synthetic resume test have a **separate paid-probe +budget**, proposed by Huey: user-set daily/monthly USD caps, **both $0 by default** +(paid probes disabled). Setup shows maximum probe frequency, estimated unit cost and +combined decision/probe budgets before explicit enablement. Subscription/quota-only +probes consume usage limits, not USD caps, and wait for Retry-After/reset when exhausted. +Their success can establish service health without a per-call dollar price. All paid +probe providers share one instance probe ledger and require provider-side key ceilings. +Reserve and settle actual paid probe charges, timeouts and unknown outcomes as below, +separately from the decision ledger, so probing cannot silently drain the decision cap. +No paid probe runs until its cap and credential are explicitly enabled. All model +fallbacks also require explicit configuration/enablement before consuming quota or money. +No unknown charge is zero and no outage grants fresh spend authority. For enabled JEV, +each instance must use a separately provisioned key used only for its decisions, with a non-secret provider key ID bound to the instance in the approved setup record. Exclusive ownership is a setup attestation, not something the manager can prove mechanically. Reject enablement without that attestation. The provider-side key limit is the real @@ -416,20 +483,27 @@ or other clients. Shared-key operation is held until a shared ledger exists. These are not account-wide caps: N enabled instances with N keys permit N times the defaults. Setup shows that combined authorized budget. Use integer currency units/decimal parsing, never binary-float threshold comparisons. -Before send, durably reserve an upper bound from a pinned price ceiling and a bounded +For metered USD requests, before send durably reserve an upper bound from a pinned +price ceiling and a bounded serialized request (≤4 KiB, verified worst-case tokens and all billable fields). Count concurrent reservations in both caps. If the adapter cannot bound cost, use rules. -Reconcile once per provider request ID using **`usage.cost`** in USD, as documented in +For metered USD adapters, reconcile once per namespaced provider request ID using +**`usage.cost`** in USD (or the fallback adapter’s verified equivalent), as documented in the [Jev guide](https://openrouter.ai/docs/guides/community/jev). Keep reservations for timeouts/crashes until settled. Missing/invalid cost, price above ceiling, or a charge above reservation disables paid decisions and alerts. Never interpret absent cost as -zero. A provider-enforced limit bound to that dedicated key is required for **every -enablement**, including trials, in addition to the local daily/monthly caps. If the +zero. A provider-enforced limit bound to the dedicated key is required for **every +metered USD enablement**, including trials, in addition to local daily/monthly caps. +Subscription-only routes instead require verified provider quota enforcement and usage +accounting, as classified above, with no implicit paid-overage permission. If the provider cannot enforce the approved key budget, keep paid decisions disabled. Local accounting alone cannot guarantee a hard ceiling against upstream misbilling. Day/month windows use UTC. No cap rollover drops an unresolved reservation. Configure caps before enabling. -**D12 Data and key.** Send only enum work type/class, numeric resource estimates, +**D12 Data and key.** The **4 KiB** serialized decision request limit and the following +metadata allowlist apply to **every endpoint**, including headless smaller-model +fallbacks. Do not attach session history or extra harness context to a fallback. +Send only enum work type/class, numeric resource estimates, coarse queue age/pressure, redacted error category, capability tags and opaque eligible route IDs. No credentials, auth headers, environment, command lines, paths, transcript, raw errors, filenames or user prompt. If a judgment requires user content, first try @@ -447,6 +521,12 @@ Do not add the distinct name to the worker allowlist or alias its value to an al provider variable. S6 must prove `sessionEnv` excludes the distinct variable using a synthetic canary. No management key access. A direct Decisions adapter may be needed unless a stand-in proves the approved proxy supports this API. +Huey’s reading of Q207 credential isolation: the fallback harness has its own configured +credential reference, injected only into the decision module’s isolated minimal harness, +never agent sessions or the shared launcher environment. Do not reuse worker credentials +or copy worker HOME/session config. A needed credential stays a slot reference in this +design. Apply the same canary/redaction checks to each fallback credential as to the +decision key. No agent endpoint credentials or route-selection tools are added. ## 8. Proposed queue rows, smallest first @@ -460,8 +540,8 @@ authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/aut | S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. | | S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. | | S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. | -| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, relapse and three failed exits alert, human kill unchanged. No real probes. | After MVP. | -| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, JEV-first then parallel, one success online, reset wait, disabled JEV, aggregate probe spend and budget deferral, restart and hung-probe cleanup. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. | +| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, request-local failures consume D6 budget, one bad request cannot re-enter protected mode, correlated distinct-request re-entry, two separate failed resumes go to D7, batch-failure definition, three-failed-exit latch, authorized revision-bound release via broker/Console without kill clear, later probes do not auto-release, maximum pause age expires without DLQ/capacity release, human kill unchanged. No real probes. | After MVP. | +| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, all service schedules parallel without a JEV gate, one success online, reset wait, disabled endpoints, subscription/quota-only success without USD, separate paid-probe caps and deferral, restart and hung-probe cleanup. Prove only the decision module selects endpoints/routes: denied session-local decision calls and route overrides, JEV → allowed smaller model → internal rules, per-model confidence/timeout calibration, 4 KiB metadata-only input at every endpoint and isolated fallback-credential canaries. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. | Each slice gets independent review and tests that kill a mutation of its central invariant. Sage chooses package/full-suite gates appropriate to the final diff. @@ -488,6 +568,11 @@ bounded synthetic pressure. No load experiment on the shared development host. visibly instead of silently changing providers or model quality. **Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes” - **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.** + Jason, verbatim (Q205): “If a catastrophe does occur, it makes sense to automatically + throw the kill switch, pause agent runs, etc.” + Huey’s reading, following Mos’s Q205 reading: protected mode does the automatic + pausing instead. The kill switch stays human-only. Jason’s Q21 reply in Q206 asked + only about cool-down, DLQ and retry. D1 remains unchanged. Jason, verbatim (Q205): “It does not make sense to require the user to release each back into operation. I would rather equip the system with the tools necessary to detect failure, enter protected mode, monitor for cool-down periods, check for @@ -500,19 +585,32 @@ bounded synthetic pressure. No load experiment on the shared development host. > Necessary services are marked back online when found available > System adapts based on available services, providers, harnesses, usage limits, and usage budgets + Q207 supersedes only the JEV-first order quoted above. D14 now probes all services + independently in parallel. Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate protected-mode recovery without per-session approval. Entry thresholds, checkpoint - handling, deadline freezing, probe deadlines/budget grouping, disabled-JEV behavior, + handling, deadline freezing/max pause age, probe deadlines/budgets and disabled-endpoint behavior, batch sizing and the three-failed-exit escalation count are proposed readings, not Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged. -- **Q22 Probe independence (pending):** may other harness/provider checks start without - waiting for JEV, with fixed rules routing until JEV returns? **Recommend yes**, as - Mos’s alternative recorded under DECISIONS Q206. Huey’s reading: that avoids a JEV-only - outage holding otherwise usable services while retaining policy/usage/budget checks. - Until Jason answers, D14 follows his JEV-first, then parallel order. +- **Q22 Resolved by DECISIONS Q207.** Jason, verbatim: + + > We would be better suited to have the requirement for decisions to always flow through the same place on the system and have that module decide the endpoint to use for decisions. If JEV is unavailable it can default to a Haiku, Luna, or some other smaller model to make the decisions from within a minimal harness. Context for the decision-maker would be kept minimal to avoid bloat and unnecessary slowdowns. I'd rather have the fallback occur at the decision module rather than at each agent session. + + Mos’s recorded reading: service probes proceed independently in parallel and the + decision module tries JEV, then a configured smaller model, then its built-in rules. + **Rules last is Mos’s reading**, not Jason’s verbatim endpoint order. Huey’s readings: + the 10-second fallback timeout candidate, per-model calibration, billing split and + isolated fallback credentials in D10–D12. No session-local endpoint or route choice. ## Verification and handoff +Revision evidence: r3 `a36952564` named the health verb/broker scope, clarified ownership +attestation and added Q4. r4 `1e8bccff1` recorded Q204 and tunable cap settings. r5 +`fd44f7c60` added Q205/Q206 protected mode, probes and resume, with D1 unchanged. r6 +records Q207, central fallback, independent probes and scrooge-0003 S-F1–S-F7 fixes. +Huey’s new thresholds, billing defaults, release/lifetime rules and timeout choices +remain labelled readings for review, not implementation or live-test evidence. + r2 addresses Scrooge findings 1–8: distinct key isolation, exclusive-key cap scope, sampler/process ownership and post-MVP re-pin, P1 interpretation, missing-confidence fallback and measured timeout selection, verified Noul spelling, retry scope exclusions,