docs: centralize decision fallback and bound outage recovery

This commit is contained in:
jason.woltje committed 2026-10-21 16:04:37 -05:00
1 parent fd44f7c603
commit 2ee68cd693
3 files changed
+167 -66

No files matched your search

+2
View File
@@ -5187,3 +5187,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi
2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls.
2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls.
2026-10-21T21:04:37Z | Huey | Q198 design r6, mos-1208/mos-1210, DECISIONS Q207, scrooge-0003, Vikunja 452 | Before: r5 fd44f7c60 gated probes on JEV and lacked request-local resume failure separation. After: parallel independent probes; central JEV/smaller-model/rules endpoint chain, per-model timeout/calibration, 4 KiB metadata limit and isolated fallback credentials; subscription quota vs metered USD accounting and separately enabled paid-probe budget (proposed $0 defaults). S-F1–S-F7: bounded outage classification, request-local D6/two-resume DLQ rule and batch definition; escalation latch with authorized revision-bound system.recovery.release separate from kill clear; cooperative F5 pause/F6 proof only; proposed configurable 24-hour cumulative pause expiry; full Q205 kill sentence with labelled protected-mode interpretation; r3–r6 history. S5/S6 acceptance proposals updated. Verification: Q205/Q207 quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned references resolve, summary within 15 lines, manual consistency/delta review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes, credential reads or paid calls.
+1
View File
@@ -1424,3 +1424,4 @@ are never rewritten or removed; corrections are new entries.
2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls
2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls
2026-10-21T20:48:49Z | Huey (gpt-6.1-sol) | mos-1206, Q198 r5 protected mode/outage recovery, Vikunja 452 | Q205/Q206 recorded with labelled readings and probe/resume acceptance proposals; Q4 superseded, Q22 pending; doc checks only, no builds/suites/real probes/live changes/paid calls; Scrooge re-review via Mos
2026-10-21T21:04:37Z | Huey (gpt-6.1-sol) | mos-1208/mos-1210, Q198 r6, Q207 and scrooge-0003 S-F1–S-F7 | central fallback/parallel probes, request failure and recovery release fixes, quota/probe budgets and credential isolation documented; doc checks only, no builds/suites/probes/live changes/paid calls; Mos routes r6 to Scrooge
+164 -66
View File
@@ -13,8 +13,8 @@ No live trial, key installation or runtime change is authorized by this design.
MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate.
Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only.
Revision **r5**, mos-1206, applies DECISIONS Q205/Q206 to r4 `1e8bccff1`.
Q4 is superseded. Q22 remains pending. New implementation choices are Huey’s readings.
Revision **r6**, mos-1208/mos-1210, applies DECISIONS Q207 and scrooge-0003 to
r5 `fd44f7c60`. Q22 is resolved. New implementation choices are Huey’s readings.
All defaults below are proposals, not measured safe operating limits or new authority.
## 1. Existing pieces and reuse
@@ -104,7 +104,8 @@ name scan grants cancellation authority over another user or instance. Configura
extends the instance’s one system config through the normal reviewed schema/bootstrap
path, never a second configuration file or automatic rewrite of an existing config.
Decision spending is separately scoped per exclusive key, not per host or shared
account. D11 requires one dedicated decision key per instance. Shared keys are unsupported
account. When JEV is enabled, D11 requires its dedicated key per instance. Other
endpoints use their isolated credential/billing contracts in D11/D12. Shared keys are unsupported
until a shared ledger and its authority boundary receive separate review.
## 3. Classes and interrupt semantics
@@ -194,9 +195,16 @@ automatic manager retries over the proxy. Max one route change within the three
**Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt
allowance and remaining ten-minute retry window. Neither resets nor advances during the
pause, including the outage-causing failure once classified as an outage. It never enters
the DLQ because of that outage. Huey’s implementation reading: track provisional attempt
outcomes until classification so a threshold-crossing failure cannot first dead-letter work. Resume is continuation of
the same logical request, not D7 replay. Actual unrelated failures still count normally.
the DLQ because of that outage. Huey’s implementation reading: correlate a candidate
outage for at most 30 seconds, then settle each failure as either an evidenced D13
outage or a request-local failure. A lone bad request cannot freeze itself indefinitely.
Resume continues the same logical request, not D7 replay. After a successful test session,
request-local failures in resumed work consume that request’s D6 attempts/time normally.
Persist a separate count of request-local failures across distinct resume episodes.
A request failing in two separate resumes goes to D7 with both episodes’ evidence,
even if its three-attempt allowance is not exhausted. Correlated outage failures do
not count toward that two-resume rule. These classification/counting details are Huey’s
reading. They prevent one defective request from repeatedly claiming an outage exemption.
Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline
budgets during the union of outage-paused intervals, never counting overlapping service
@@ -205,6 +213,11 @@ by paused time. This deliberately interprets a caller-set deadline as an executi
not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation
still takes effect during a pause. Journal pause intervals and counters across restarts.
An interrupted effect remains uncertain until reconciled, regardless of the clock.
Huey’s reading: configure a maximum outage-pause age, default **24 hours of cumulative
paused wall time per request**, through D3. On reaching it, make the request terminal
`expired`, never DLQ. Show pause age, limit and effective deadline in the Console.
Expiry prevents dispatch but never releases capacity still held by an unproven stop.
This maximum bounds the deadline extension above and does not authorize effect replay.
D6 covers managed work requests only. Existing notifier delivery backoff
(`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff
@@ -258,19 +271,22 @@ Once protected, active checks follow D14’s schedule. For a provider/account sc
enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient
network/429/5xx failures across at least two distinct requests, or three consecutive
failed endpoint checks. One explicit account-wide quota/reset response can enter its
scope immediately. Isolated validation/permission failures do not count. Until wider
scope immediately. Isolated validation/permission failures and model output/tool errors
do not count. Request-local failures after a successful test session use D6, never
D13 by themselves. Re-entry based on resumed requests requires the correlated failure
threshold across at least two distinct requests, not one bad request repeated. Until wider
failure scope is proven, hold only that route/account. A missing required observation
holds work as unknown rather than guessing recovery. Persist incident and classification
before advancing affected request outcomes. These settings use D3’s reviewed config path.
Fence affected new dispatch. Ask running owners to pause at safe points using F5/F6.
Fence affected new dispatch. Pause uses only F5’s cooperative interrupt at safe points.
F6 supplies stop proof, never the pause action. Do not invoke force-stop for a pause.
Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed
rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable,
record uncertain ownership and retain capacity. Do not force-stop a write or manufacture
human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects
mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected
services can continue under admission, except where D14’s default recovery order holds
them pending the JEV stage. No person releases each paused session.
services can continue under admission. No person releases each paused session.
**D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first
performs mechanical network and endpoint checks. Only after the endpoint answers may a
@@ -280,11 +296,13 @@ workspace effects or inherited worker environment, and separately authorized pro
credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat
request. Probe success means the endpoint plus the configured minimal harness/model
path succeeded, not merely an HTTP connection. One successful complete probe marks that
service online. Missing credentials, exhausted budget or an unknown charge keep it
unverified, without synthetic success or unauthorized spend.
service online. Missing credentials, exhausted applicable quota/budget or unknown
billing classification keep the headless check deferred/unverified. Subscription-only
probes need verified usage/quota accounting, not an invented per-call USD price. Paid
probe charges use D11’s separate probe budget. No synthetic success or unauthorized spend.
Default order is Jason’s: **JEV first**, then the other harness/provider checks start
**in parallel** after JEV succeeds. Each service has its own durable probe schedule:
Under Q207 and Mos’s recorded reading, **every service probes in parallel** on its own
schedule. JEV is not a gate for another service. Each has a durable probe schedule:
wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540,
600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least
Retry-After or the usage-limit reset before probing, even if longer than 600 seconds.
@@ -295,17 +313,15 @@ are separate from D10 routing calls and never retry a paused work request.
Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check
deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup
and uncertain outcomes retain their reservations. These deadlines are independent of
the routing hook’s 500 ms candidate timeout. Scheduling retries indefinitely does not
override D11 budgets: when funds/caps are unavailable, keep mechanical checks running
on schedule and defer paid headless probes until authorized budget is available. A human
D10’s endpoint-specific decision timeouts. Scheduling retries indefinitely does not
override D11: when applicable quota or probe funds are unavailable, keep mechanical
checks running on schedule and defer headless probes until authorized capacity returns. A human
kill prevents headless probes/agent launches, while local read-only health tracking
continues. The kill can be cleared only through its existing human path.
Huey’s reading for disabled optional JEV: if the user never enabled/configured it,
record its stage as not configured and begin the other checks without a paid JEV call.
If JEV is configured but unavailable, keep the specified JEV-first order. A JEV-only
outage can therefore delay otherwise available harnesses. Q22 proposes changing that
order and remains pending, so its alternative is not implemented by this design.
Huey’s reading for optional endpoints: mark disabled/unconfigured JEV or a fallback as
not configured and make no call to it. Other services keep their independent schedules.
A failed or budget-deferred decision endpoint never blocks their probes.
**D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service
online proof and D4’s healthy release thresholds, reconcile paused owners/effects and
@@ -313,10 +329,14 @@ run **one test session** on an eligible online route. Use disposable synthetic w
current policy and bounded capacity. No request with uncertain effects is used as a
test. On success resume eligible paused work in batches of at most five, with one
30-second healthy observation interval between batches and D4’s stricter dispatch
rate/reservations still enforced. A batch needs no new approval per session. If the
test session or a batch fails, return to protected mode and its probe schedule.
Request-local defects still follow D6/D7, with evidence distinguishing that failure
from a provider outage rather than falsely marking a healthy service offline.
rate/reservations still enforced. A batch needs no new approval per session. A failed
test session returns to protected mode with its evidence, without relabelling an
otherwise healthy provider as offline. After the test succeeds, “batch fails” means
D13 proves a correlated outage across distinct requests, or independent mechanical
network/endpoint evidence establishes an outage. Only that returns the scope to
protected mode. Individual resumed requests failing validation, tools or model output
consume D6 budgets and follow the two-resume/D7 rule, while healthy batch members
continue. A failed request alone never counts as a failed automatic exit.
Huey’s proposed escalation count: after **three consecutive failed automatic exits**
(canary or batch stages) in the same incident, alert Mos/Jason and hold production
@@ -326,6 +346,18 @@ is the repeated-exit boundary where a person is needed, not a prompt for each se
or each failed probe. Existing policy, missing credentials and unresolved external
effects remain fail-closed constraints, never reasons to improvise authority.
Huey’s reading of release after three failed exits: persist `recovery-escalated` on the
incident. Later successful probes update service health but **do not** restart work
automatically while that latch is set. Jason, or a role already explicitly delegated
recovery authority, releases it through proposed human write `system.recovery.release`
(scope, expected incident ID and revision), also exposed as Console “System → Recovery →
Release after diagnosis”. The authenticated/scoped writer checks authority, current
incident/revision and diagnosis evidence. It clears only this escalation latch, resets
its exit counter and restarts the one-test-session stage, never bulk dispatch. It does
not clear a human kill, permissions, unresolved effects or resource holds. This design
adds no role grant. Without a delegated role, Jason is the releaser. S5 owns the new
broker/Console write and review, separate from the kill-switch clear.
## 6. Monitoring and Console
**D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical
@@ -342,7 +374,7 @@ uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn
retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state,
manager heartbeat and decision-model timeouts/spend/unknown charges. Huey’s reading:
also expose protected-mode scope, last/next probe, online proof, paused request counts,
canary/batch progress and consecutive exit failures. Store bounded
canary/batch progress, pause age/limit and the escalation latch/release evidence. Store bounded
aggregate samples for 7 days, without request bodies. Request receipts/evidence follow
the instance’s retention policy, with the existing write-once run invariant preserved.
@@ -364,9 +396,19 @@ reports manager death through an independent health path, not the dead manager i
**D10 Interface.** `decide({version, snapshotId, policyRevision, deadline,
allowedActions, state}, signal) → {actionId, confidence?, usage:{cost,currency},
providerRequestId}`. `allowedActions` contains opaque IDs for already-eligible routes
or “use rules”. Adapters normalize provider types. A deterministic rules adapter is
the default and needs no network or key. A generic structured-output adapter can support
other decision models without exposing their wire format to the scheduler.
or “use rules”. Adapters normalize provider types. **Every routing decision goes through
this one module. No agent session calls a decision endpoint or picks its own route.**
The module also chooses its own decision endpoint. Provider-neutral requests and results
keep endpoint wire formats out of sessions and the scheduler. With model assistance
disabled, the module uses built-in rules without network or key.
When enabled, the module tries **JEV → one configured smaller fallback model → built-in
rules**. The smaller model may be Haiku, Luna or another on the user’s allowlist and
runs through a minimal harness config owned by this module. Rules last is **Mos’s
reading of Q207**, so a full outage does not require a reachable model. Huey’s reading:
one attempt per eligible model endpoint, no same-endpoint retries or unbounded chain.
Use only configured, online, calibrated and budget/quota-eligible endpoints. An
uncalibrated model never decides: the rules adapter supplies the decision in its place.
Enabling paid assistance is a user setting, recommended in setup with its price/cap
shown. Pin **`typesafe/jev-1.14`** as the recommended initial model, never a moving
@@ -379,35 +421,60 @@ lists $0.042 per million input tokens, free output and 262,144 context tokens
Use choice for route selection. Do not send images or use the large context window.
Other decision models need their own adapter contract and calibrated confidence tests.
Initial timeout candidate **500 ms**, max one in-flight decision per instance, no
decision-call retry. S6 measures latency and timeout/fallback rates on delayed stand-ins,
then records the selected timeout and evidence before acceptance. Stand-ins verify the
local budget, not real provider latency. Timeout, invalid answer, stale snapshot,
missing/non-finite/out-of-range confidence, low confidence (<0.8 initially), cost
uncertainty or network failure immediately selects rules. An adapter without a calibrated
confidence value therefore uses rules. Discard late answers for
routing, but still account for their charge. The hook cannot authorize work, choose
P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the
eligible set. Recheck eligibility after it returns. Never call it for safety actions.
Huey’s reading of Q205/Q206: reachable JEV may advise on resume order and route choice
among services mechanically marked online, within existing policy/capacity/budget.
It cannot mark a service online, authorize a resume, clear a human kill, replay effects
or perform safety actions. If unavailable, fixed rules choose eligible routes subject
to D14’s default probe order. Its absence does not change stop/effect proof requirements.
**Endpoint contracts (Huey’s reading of Q207).** Each model has its own adapter,
output validation, timeout and confidence calibration. **500 ms is JEV’s candidate**.
A smaller model through a headless harness needs longer: propose **10 seconds end to
end**, including harness startup, as that fallback’s candidate. S6 uses delayed
stand-ins only to measure latency/timeout/fallback rates and records each selected
timeout and calibration. No real call or benchmark is authorized. Stand-ins measure
local behavior, not provider performance. Use at most one in-flight module decision
per instance, bounded by the caller deadline and these endpoint budgets. A fallback
never inherits JEV’s confidence threshold without its own calibration.
Timeout/network failure advances to the next eligible endpoint within the deadline.
Invalid/stale answers, missing/non-finite/out-of-range or uncalibrated/low confidence
cannot choose a route. Use rules if no eligible calibrated model supplies a valid
answer. An exhausted caller deadline selects rules without waiting for another model.
Discard late answers for routing, retain their usage reservations, and bound owned
harness cleanup. The hook cannot authorize work, choose P0/P1 status, change thresholds,
clear a kill, replay effects or choose outside the eligible set. Recheck eligibility
at the final mechanical dispatch check. Never call models for safety actions.
Huey’s reading of Q205–Q207: the module may advise on resume order and route choice
among services mechanically marked online, within policy/capacity/budget. It cannot
mark a service online, authorize resume, clear a human kill, replay effects or perform
safety actions. Its built-in rules remain inside the module when models are unavailable.
Stop/effect proof requirements remain unchanged.
**D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps
are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**.
They live in the instance’s one system config through D3’s reviewed schema/bootstrap
path. Both configured caps are enforced by the owning instance’s durable ledger.
No spending starts until Jason enables paid decisions and installs the key.
Huey’s reading of Q206 probe accounting: charge all recovery headless/test-session probe
spend and decision calls to these same daily/monthly instance caps, not a separate free
probe allowance. The ledger groups the budget under the exclusive decision-key binding
and records each actual provider/key identity, including probes that use another provider.
Those probe credentials remain separate from the decision key and need their own
provider-side ceilings. Reserve/settle all probe charges under the same rules below.
No unknown charge is zero and no outage grants fresh spend authority. Each instance
must use a separately provisioned key used only for that instance’s decisions, with a
**Huey’s reading of billing for Q206/Q207:** classify each adapter/credential as
subscription/quota-only or metered USD before enablement, with no automatic conversion
to a paid API. Fallback decisions using metered models share the configured D11 decision
caps with JEV. A subscription-only fallback (for example an authorized Claude OAuth or
ChatGPT-plan route) consumes its provider usage/quota limits, not a fabricated USD price.
Only a verified subscription-only contract establishes no incremental API charge.
Huey’s reading: reserve a bounded usage allowance before a subscription call, then
reconcile the provider’s measured usage. Unknown quota accounting holds that route.
Unknown billing/quota sources defer that endpoint and use the next eligible adapter.
No subscription purchase, paid overage or new model spend is authorized by fallback.
Recovery headless probes and the synthetic resume test have a **separate paid-probe
budget**, proposed by Huey: user-set daily/monthly USD caps, **both $0 by default**
(paid probes disabled). Setup shows maximum probe frequency, estimated unit cost and
combined decision/probe budgets before explicit enablement. Subscription/quota-only
probes consume usage limits, not USD caps, and wait for Retry-After/reset when exhausted.
Their success can establish service health without a per-call dollar price. All paid
probe providers share one instance probe ledger and require provider-side key ceilings.
Reserve and settle actual paid probe charges, timeouts and unknown outcomes as below,
separately from the decision ledger, so probing cannot silently drain the decision cap.
No paid probe runs until its cap and credential are explicitly enabled. All model
fallbacks also require explicit configuration/enablement before consuming quota or money.
No unknown charge is zero and no outage grants fresh spend authority. For enabled JEV,
each instance must use a separately provisioned key used only for its decisions, with a
non-secret provider key ID bound to the instance in the approved setup record. Exclusive
ownership is a setup attestation, not something the manager can prove mechanically.
Reject enablement without that attestation. The provider-side key limit is the real
@@ -416,20 +483,27 @@ or other clients. Shared-key operation is held
until a shared ledger exists. These are not account-wide caps: N enabled instances
with N keys permit N times the defaults. Setup shows that combined authorized budget.
Use integer currency units/decimal parsing, never binary-float threshold comparisons.
Before send, durably reserve an upper bound from a pinned price ceiling and a bounded
For metered USD requests, before send durably reserve an upper bound from a pinned
price ceiling and a bounded
serialized request (≤4 KiB, verified worst-case tokens and all billable fields). Count
concurrent reservations in both caps. If the adapter cannot bound cost, use rules.
Reconcile once per provider request ID using **`usage.cost`** in USD, as documented in
For metered USD adapters, reconcile once per namespaced provider request ID using
**`usage.cost`** in USD (or the fallback adapter’s verified equivalent), as documented in
the [Jev guide](https://openrouter.ai/docs/guides/community/jev). Keep reservations for
timeouts/crashes until settled. Missing/invalid cost, price above ceiling, or a charge
above reservation disables paid decisions and alerts. Never interpret absent cost as
zero. A provider-enforced limit bound to that dedicated key is required for **every
enablement**, including trials, in addition to the local daily/monthly caps. If the
zero. A provider-enforced limit bound to the dedicated key is required for **every
metered USD enablement**, including trials, in addition to local daily/monthly caps.
Subscription-only routes instead require verified provider quota enforcement and usage
accounting, as classified above, with no implicit paid-overage permission. If the
provider cannot enforce the approved key budget, keep paid decisions disabled. Local
accounting alone cannot guarantee a hard ceiling against upstream misbilling.
Day/month windows use UTC. No cap rollover drops an unresolved reservation. Configure caps before enabling.
**D12 Data and key.** Send only enum work type/class, numeric resource estimates,
**D12 Data and key.** The **4 KiB** serialized decision request limit and the following
metadata allowlist apply to **every endpoint**, including headless smaller-model
fallbacks. Do not attach session history or extra harness context to a fallback.
Send only enum work type/class, numeric resource estimates,
coarse queue age/pressure, redacted error category, capability tags and opaque eligible
route IDs. No credentials, auth headers, environment, command lines, paths, transcript,
raw errors, filenames or user prompt. If a judgment requires user content, first try
@@ -447,6 +521,12 @@ Do not add the distinct name to the worker allowlist or alias its value to an al
provider variable. S6 must prove `sessionEnv` excludes the distinct variable using a
synthetic canary. No management key access. A direct Decisions
adapter may be needed unless a stand-in proves the approved proxy supports this API.
Huey’s reading of Q207 credential isolation: the fallback harness has its own configured
credential reference, injected only into the decision module’s isolated minimal harness,
never agent sessions or the shared launcher environment. Do not reuse worker credentials
or copy worker HOME/session config. A needed credential stays a slot reference in this
design. Apply the same canary/redaction checks to each fallback credential as to the
decision key. No agent endpoint credentials or route-selection tools are added.
## 8. Proposed queue rows, smallest first
@@ -460,8 +540,8 @@ authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/aut
| S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. |
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, relapse and three failed exits alert, human kill unchanged. No real probes. | After MVP. |
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, JEV-first then parallel, one success online, reset wait, disabled JEV, aggregate probe spend and budget deferral, restart and hung-probe cleanup. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, request-local failures consume D6 budget, one bad request cannot re-enter protected mode, correlated distinct-request re-entry, two separate failed resumes go to D7, batch-failure definition, three-failed-exit latch, authorized revision-bound release via broker/Console without kill clear, later probes do not auto-release, maximum pause age expires without DLQ/capacity release, human kill unchanged. No real probes. | After MVP. |
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, all service schedules parallel without a JEV gate, one success online, reset wait, disabled endpoints, subscription/quota-only success without USD, separate paid-probe caps and deferral, restart and hung-probe cleanup. Prove only the decision module selects endpoints/routes: denied session-local decision calls and route overrides, JEV → allowed smaller model → internal rules, per-model confidence/timeout calibration, 4 KiB metadata-only input at every endpoint and isolated fallback-credential canaries. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
Each slice gets independent review and tests that kill a mutation of its central
invariant. Sage chooses package/full-suite gates appropriate to the final diff.
@@ -488,6 +568,11 @@ bounded synthetic pressure. No load experiment on the shared development host.
visibly instead of silently changing providers or model quality.
**Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes”
- **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.**
Jason, verbatim (Q205): “If a catastrophe does occur, it makes sense to automatically
throw the kill switch, pause agent runs, etc.”
Huey’s reading, following Mos’s Q205 reading: protected mode does the automatic
pausing instead. The kill switch stays human-only. Jason’s Q21 reply in Q206 asked
only about cool-down, DLQ and retry. D1 remains unchanged.
Jason, verbatim (Q205): “It does not make sense to require the user to release each
back into operation. I would rather equip the system with the tools necessary to
detect failure, enter protected mode, monitor for cool-down periods, check for
@@ -500,19 +585,32 @@ bounded synthetic pressure. No load experiment on the shared development host.
> Necessary services are marked back online when found available
> System adapts based on available services, providers, harnesses, usage limits, and usage budgets
Q207 supersedes only the JEV-first order quoted above. D14 now probes all services
independently in parallel.
Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate
protected-mode recovery without per-session approval. Entry thresholds, checkpoint
handling, deadline freezing, probe deadlines/budget grouping, disabled-JEV behavior,
handling, deadline freezing/max pause age, probe deadlines/budgets and disabled-endpoint behavior,
batch sizing and the three-failed-exit escalation count are proposed readings, not
Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged.
- **Q22 Probe independence (pending):** may other harness/provider checks start without
waiting for JEV, with fixed rules routing until JEV returns? **Recommend yes**, as
Mos’s alternative recorded under DECISIONS Q206. Huey’s reading: that avoids a JEV-only
outage holding otherwise usable services while retaining policy/usage/budget checks.
Until Jason answers, D14 follows his JEV-first, then parallel order.
- **Q22 Resolved by DECISIONS Q207.** Jason, verbatim:
> We would be better suited to have the requirement for decisions to always flow through the same place on the system and have that module decide the endpoint to use for decisions. If JEV is unavailable it can default to a Haiku, Luna, or some other smaller model to make the decisions from within a minimal harness. Context for the decision-maker would be kept minimal to avoid bloat and unnecessary slowdowns. I'd rather have the fallback occur at the decision module rather than at each agent session.
Mos’s recorded reading: service probes proceed independently in parallel and the
decision module tries JEV, then a configured smaller model, then its built-in rules.
**Rules last is Mos’s reading**, not Jason’s verbatim endpoint order. Huey’s readings:
the 10-second fallback timeout candidate, per-model calibration, billing split and
isolated fallback credentials in D10–D12. No session-local endpoint or route choice.
## Verification and handoff
Revision evidence: r3 `a36952564` named the health verb/broker scope, clarified ownership
attestation and added Q4. r4 `1e8bccff1` recorded Q204 and tunable cap settings. r5
`fd44f7c60` added Q205/Q206 protected mode, probes and resume, with D1 unchanged. r6
records Q207, central fallback, independent probes and scrooge-0003 S-F1–S-F7 fixes.
Huey’s new thresholds, billing defaults, release/lifetime rules and timeout choices
remain labelled readings for review, not implementation or live-test evidence.
r2 addresses Scrooge findings 1–8: distinct key isolation, exclusive-key cap scope,
sampler/process ownership and post-MVP re-pin, P1 interpretation, missing-confidence
fallback and measured timeout selection, verified Noul spelling, retry scope exclusions,