docs: design protected mode and autonomous outage recovery

This commit is contained in:
jason.woltje committed 2026-10-21 15:48:49 -05:00
1 parent 1e8bccff15
commit fd44f7c603
3 files changed
+166 -28

No files matched your search

+2
View File
@@ -5185,3 +5185,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi
2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls. 2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls.
2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls. 2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls.
2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls.
+1
View File
@@ -1423,3 +1423,4 @@ are never rewritten or removed; corrections are new entries.
2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge 2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge
2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls 2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls
2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls 2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls
2026-10-21T20:48:49Z | Huey (gpt-6.1-sol) | mos-1206, Q198 r5 protected mode/outage recovery, Vikunja 452 | Q205/Q206 recorded with labelled readings and probe/resume acceptance proposals; Q4 superseded, Q22 pending; doc checks only, no builds/suites/real probes/live changes/paid calls; Scrooge re-review via Mos
+163 -28
View File
@@ -4,6 +4,7 @@ Build a rules-first manager for admission, bounded retries, recovery and host he
System stop and health requests bypass the work queue, without bypassing authorization. System stop and health requests bypass the work queue, without bypassing authorization.
Normal and background work wait under pressure, then resume gradually and fairly. Normal and background work wait under pressure, then resume gradually and fairly.
An interrupt requests cancellation. Capacity stays occupied until the owner proves exit. An interrupt requests cancellation. Capacity stays occupied until the owner proves exit.
Outages enter system-owned protected mode, then probe and resume automatically.
Unknown external effects require reconciliation, never automatic replay. Unknown external effects require reconciliation, never automatic replay.
Reuse the bus, controllers, launch registry, run evidence and Console. Reuse the bus, controllers, launch registry, run evidence and Console.
Optional decision models rank already-permitted routes. Recommend Jev when enabled. Optional decision models rank already-permitted routes. Recommend Jev when enabled.
@@ -12,8 +13,8 @@ No live trial, key installation or runtime change is authorized by this design.
MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate. MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate.
Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only. Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only.
Revision **r4**, mos-1203, records Jason’s DECISIONS Q204 answers to section 9 Revision **r5**, mos-1206, applies DECISIONS Q205/Q206 to r4 `1e8bccff1`.
on r3 `a36952564`. Q4 remains pending. Q4 is superseded. Q22 remains pending. New implementation choices are Huey’s readings.
All defaults below are proposals, not measured safe operating limits or new authority. All defaults below are proposals, not measured safe operating limits or new authority.
## 1. Existing pieces and reuse ## 1. Existing pieces and reuse
@@ -83,7 +84,7 @@ handling outside that pool. This is software scheduling, not hardware/real-time
content reference plus digest (content stays at its authorized source), policy revision, content reference plus digest (content stays at its authorized source), policy revision,
class, enqueue time/deadline, resource estimate, allowed routes and idempotency key. class, enqueue time/deadline, resource estimate, allowed routes and idempotency key.
Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`, Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`,
with branches to `retry-wait`, `dead-letter`, or `uncertain`. Record every attempt, with branches to `retry-wait`, `outage-paused`, `dead-letter`, or `uncertain`. Record every attempt,
route, reason code, owner identity, heartbeat and evidence references. Materialized route, reason code, owner identity, heartbeat and evidence references. Materialized
state is rebuildable. Re-check permission, kill state and policy at dispatch/replay. state is rebuildable. Re-check permission, kill state and policy at dispatch/replay.
A duplicate key with the same digest returns the original request. A changed digest A duplicate key with the same digest returns the original request. A changed digest
@@ -111,15 +112,15 @@ until a shared ledger and its authority boundary receive separate review.
| Ref/class | Requests | Admission and preemption | | Ref/class | Requests | Admission and preemption |
| --- | --- | --- | | --- | --- | --- |
| P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. | | P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. |
| P1 system | Lightweight health reads, lease reconciliation, approved recovery | Reserved control capacity. Health/reconciliation can run under pressure. Recovery that launches heavy work still needs capacity and kill-clear. Never auto-clear a kill. | | P1 system | Lightweight health reads, lease reconciliation, automatic outage recovery under Q205/Q206 | Reserved control capacity. Health/reconciliation can run under pressure. Resume needs capacity, current policy and a clear human kill switch, without a new approval per session. Never auto-clear a kill. |
| P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. | | P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. |
| P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. | | P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. |
The P1 capacity/authorization hold narrows Jason’s Q198 “take precedent” direction: DECISIONS Q205/Q206 supersede the old P1 per-session recovery approval question.
system recovery gets priority over eligible normal work, but heavy recovery cannot Huey’s reading: the configured recovery policy supplies standing authority to resume
bypass authorization, kill state or hard capacity. This is a proposed safety restriction, eligible paused work. Current permissions, hard capacity and the human kill switch still
not a claim that Jason already approved that exception. Bounded health and reconciliation apply. Health/reconciliation retain reserved capacity. Protected mode can clear itself,
retain reserved capacity. Section 9 Q4 puts this restriction to Jason for approval. but cannot clear a human kill or grant new authority.
| Work | What interrupt means | Hard-stop boundary | | Work | What interrupt means | Hard-stop boundary |
| --- | --- | --- | | --- | --- | --- |
@@ -158,7 +159,8 @@ explicitly supported exclusive envelope, never an unbounded resource guess. Star
then measure before raising concurrency. Provider cooldown blocks only affected then measure before raising concurrency. Provider cooldown blocks only affected
provider/account/model scopes, not all host work. An open provider circuit waits at provider/account/model scopes, not all host work. An open provider circuit waits at
least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on
failure. Never shorten an upstream wait to fit a local retry delay. failure. Never shorten an upstream wait to fit a local retry delay. A declared outage
uses D13–D15’s Q206 probe schedule instead of this short transient-failure circuit.
These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or
Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and
@@ -171,7 +173,8 @@ P0 uses a separately bounded, target-deduplicated control path. Journal failure
new work and alerts, without disabling the existing emergency stop path. new work and alerts, without disabling the existing emergency stop path.
Normal requests expire after 30 minutes waiting, background after 24 hours, unless an Normal requests expire after 30 minutes waiting, background after 24 hours, unless an
explicit authorized deadline differs. Show expiry and retain its receipt. Once healthy, explicit authorized deadline differs. Outage-paused time is excluded under D6.
Show expiry and retain its receipt. Once healthy,
weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant. weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant.
After 10 minutes, background ages into normal scheduling rank, never into system After 10 minutes, background ages into normal scheduling rank, never into system
priority and never through pressure holds. After control bursts, service eligible work. priority and never through pressure holds. After control bursts, service eligible work.
@@ -181,13 +184,28 @@ alert on oldest wait >10 minutes rather than promise starvation freedom under ov
## 5. Retries, dead letters and routes ## 5. Retries, dead letters and routes
**D6 Retry ownership.** One logical request gets at most **3 total manager attempts**, **D6 Retry ownership.** One logical request gets at most **3 total manager attempts**,
10 minutes of retry time and its original deadline, whichever ends first. Retry only 10 minutes of retry time and its effective deadline, whichever ends first. Retry only
transient failures known to precede effects, or operations with verified idempotency. transient failures known to precede effects, or operations with verified idempotency.
Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum. Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum.
Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal
retries must fit a declared per-attempt budget. If that budget is unknown, do not layer retries must fit a declared per-attempt budget. If that budget is unknown, do not layer
automatic manager retries over the proxy. Max one route change within the three attempts. automatic manager retries over the proxy. Max one route change within the three attempts.
**Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt
allowance and remaining ten-minute retry window. Neither resets nor advances during the
pause, including the outage-causing failure once classified as an outage. It never enters
the DLQ because of that outage. Huey’s implementation reading: track provisional attempt
outcomes until classification so a threshold-crossing failure cannot first dead-letter work. Resume is continuation of
the same logical request, not D7 replay. Actual unrelated failures still count normally.
Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline
budgets during the union of outage-paused intervals, never counting overlapping service
outages twice. Keep the original timestamp and publish the effective deadline extended
by paused time. This deliberately interprets a caller-set deadline as an execution budget,
not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation
still takes effect during a pause. Journal pause intervals and counters across restarts.
An interrupted effect remains uncertain until reconciled, regardless of the clock.
D6 covers managed work requests only. Existing notifier delivery backoff D6 covers managed work requests only. Existing notifier delivery backoff
(`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff (`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff
(`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own (`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own
@@ -196,13 +214,16 @@ Any later consolidation needs its own design and acceptance tests.
| Outcome | Action | | Outcome | Action |
| --- | --- | | --- | --- |
| Connection refused before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility. | | Mechanically classified outage (D13) | Outage-paused, counters/clocks frozen, service probes under D14. No outage DLQ. |
| Isolated connection refusal before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility and D13 does not classify an outage. |
| Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. | | Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. |
| Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. | | Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. |
| Transient retry budget/deadline exhausted, deterministic execution failure | Dead-letter with evidence and actionable reason. Admission expiry is an expired terminal item. | | Non-outage retry budget/deadline exhausted, deterministic request failure | Dead-letter with evidence and actionable reason. Non-outage admission expiry is an expired terminal item. |
| Explicit user cancellation | Terminal cancelled. Never retry automatically. | | Explicit user cancellation | Terminal cancelled. Never retry automatically. |
**D7 DLQ.** A durable request state and append-only disposition history, not copied **D7 DLQ.** Only requests that fail on their own use this path, not outage-paused work.
Validation/permission refusals remain refusals under D6, with their request evidence.
A durable request state and append-only disposition history, not copied
prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted
error code, idempotency/effect status and links to authorized evidence in Console error code, idempotency/effect status and links to authorized evidence in Console
“System → Requests → Dead letters”. Propose equivalent `mosaic system requests “System → Requests → Dead letters”. Propose equivalent `mosaic system requests
@@ -212,6 +233,8 @@ replay. Jason or an explicitly delegated role can replay after cause correction
reconciliation. Replay creates a new request linked to its parent, with a fresh bounded reconciliation. Replay creates a new request linked to its parent, with a fresh bounded
budget and current policy. No mass replay, unattended replay chains or resurrected budget and current policy. No mass replay, unattended replay chains or resurrected
cancelled work. Unknown effects require explicit reconciliation evidence before replay. cancelled work. Unknown effects require explicit reconciliation evidence before replay.
D15’s automatic batch continuation of outage-paused work is not a replay chain and
requires no person to release each request. An outage never sends those requests here.
**D8 Rerouting.** Candidate routes come only from the user’s configured allowlist, **D8 Rerouting.** Candidate routes come only from the user’s configured allowlist,
filtered for model capability, tool protocol, context, privacy/region, account budget filtered for model capability, tool protocol, context, privacy/region, account budget
@@ -221,6 +244,88 @@ or migrate a live transcript silently. Mid-turn failure is reconciled before a f
attempt. Record requested and effective route and reason. New provider consent/spend attempt. Record requested and effective route and reason. New provider consent/spend
is separate from enabling the decision hook. F10’s preflight fallback remains distinct. is separate from enabling the decision hook. F10’s preflight fallback remains distinct.
## 5a. Protected mode and outage recovery
**D13 Protected mode (Huey’s reading of Q205/Q206).** The supervisor owns a durable
`normal → protected → checking → canary → batch-resume → normal` state per affected
service scope, with a host/network incident able to hold all dependent scopes. This is
separate from the human kill switch. D1 remains unchanged: the manager never sets or
clears kill-switch state. Detection/status tracking are mechanical, without an LLM.
Proposed entry thresholds, all Huey’s readings: three consecutive failed network checks
at the 2-second sample cadence before protected mode enter a network incident.
Once protected, active checks follow D14’s schedule. For a provider/account scope,
enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient
network/429/5xx failures across at least two distinct requests, or three consecutive
failed endpoint checks. One explicit account-wide quota/reset response can enter its
scope immediately. Isolated validation/permission failures do not count. Until wider
failure scope is proven, hold only that route/account. A missing required observation
holds work as unknown rather than guessing recovery. Persist incident and classification
before advancing affected request outcomes. These settings use D3’s reviewed config path.
Fence affected new dispatch. Ask running owners to pause at safe points using F5/F6.
Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed
rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable,
record uncertain ownership and retain capacity. Do not force-stop a write or manufacture
human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects
mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected
services can continue under admission, except where D14’s default recovery order holds
them pending the JEV stage. No person releases each paused session.
**D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first
performs mechanical network and endpoint checks. Only after the endpoint answers may a
one-turn headless probe run through a minimal harness config. A check of a dead link
makes no model call. Huey’s reading: use a fixed synthetic input, no tools, user content,
workspace effects or inherited worker environment, and separately authorized probe
credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat
request. Probe success means the endpoint plus the configured minimal harness/model
path succeeded, not merely an HTTP connection. One successful complete probe marks that
service online. Missing credentials, exhausted budget or an unknown charge keep it
unverified, without synthetic success or unauthorized spend.
Default order is Jason’s: **JEV first**, then the other harness/provider checks start
**in parallel** after JEV succeeds. Each service has its own durable probe schedule:
wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540,
600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least
Retry-After or the usage-limit reset before probing, even if longer than 600 seconds.
The ceiling bounds this schedule, not a provider-mandated wait. Unknown reset time
keeps the scope held and visible rather than guessing a usable quota. Service probes
are separate from D10 routing calls and never retry a paused work request.
Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check
deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup
and uncertain outcomes retain their reservations. These deadlines are independent of
the routing hook’s 500 ms candidate timeout. Scheduling retries indefinitely does not
override D11 budgets: when funds/caps are unavailable, keep mechanical checks running
on schedule and defer paid headless probes until authorized budget is available. A human
kill prevents headless probes/agent launches, while local read-only health tracking
continues. The kill can be cleared only through its existing human path.
Huey’s reading for disabled optional JEV: if the user never enabled/configured it,
record its stage as not configured and begin the other checks without a paid JEV call.
If JEV is configured but unavailable, keep the specified JEV-first order. A JEV-only
outage can therefore delay otherwise available harnesses. Q22 proposes changing that
order and remains pending, so its alternative is not implemented by this design.
**D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service
online proof and D4’s healthy release thresholds, reconcile paused owners/effects and
run **one test session** on an eligible online route. Use disposable synthetic work,
current policy and bounded capacity. No request with uncertain effects is used as a
test. On success resume eligible paused work in batches of at most five, with one
30-second healthy observation interval between batches and D4’s stricter dispatch
rate/reservations still enforced. A batch needs no new approval per session. If the
test session or a batch fails, return to protected mode and its probe schedule.
Request-local defects still follow D6/D7, with evidence distinguishing that failure
from a provider outage rather than falsely marking a healthy service offline.
Huey’s proposed escalation count: after **three consecutive failed automatic exits**
(canary or batch stages) in the same incident, alert Mos/Jason and hold production
resume for diagnosis. Continue mechanical monitoring and budget-permitted probes
without an attempt limit. A fully successful exit resets the exit-failure count. This
is the repeated-exit boundary where a person is needed, not a prompt for each session
or each failed probe. Existing policy, missing credentials and unresolved external
effects remain fail-closed constraints, never reasons to improvise authority.
## 6. Monitoring and Console ## 6. Monitoring and Console
**D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical **D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical
@@ -235,7 +340,9 @@ an approved re-pin window after MVP. No sampler is started by this design work.
**Measurements.** Pressure samples and freshness, active/reserved work, ownership **Measurements.** Pressure samples and freshness, active/reserved work, ownership
uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency, uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency,
retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state, retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state,
manager heartbeat and decision-model timeouts/spend/unknown charges. Store bounded manager heartbeat and decision-model timeouts/spend/unknown charges. Huey’s reading:
also expose protected-mode scope, last/next probe, online proof, paused request counts,
canary/batch progress and consecutive exit failures. Store bounded
aggregate samples for 7 days, without request bodies. Request receipts/evidence follow aggregate samples for 7 days, without request bodies. Request receipts/evidence follow
the instance’s retention policy, with the existing write-once run invariant preserved. the instance’s retention policy, with the existing write-once run invariant preserved.
@@ -245,7 +352,8 @@ Requests, Dead letters and evidence. Reuse Console read transport, auth and scop
human writes (F8). No new public listener. Show stale/unknown explicitly, not green. human writes (F8). No new public listener. Show stale/unknown explicitly, not green.
Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and
unknown decision charge. Warn on pressure hold >60 seconds, control heartbeat stale unknown decision charge. Huey’s reading: show protected-mode entry/recovery and alert
on D15’s repeated-exit boundary. Warn on pressure hold >60 seconds, control heartbeat stale
>6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit >6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit
one recovery notification. Use Console attention/trail first. External notification one recovery notification. Use Console attention/trail first. External notification
channels require their own configured authorization. A lightweight supervisor watchdog channels require their own configured authorization. A lightweight supervisor watchdog
@@ -281,12 +389,24 @@ confidence value therefore uses rules. Discard late answers for
routing, but still account for their charge. The hook cannot authorize work, choose routing, but still account for their charge. The hook cannot authorize work, choose
P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the
eligible set. Recheck eligibility after it returns. Never call it for safety actions. eligible set. Recheck eligibility after it returns. Never call it for safety actions.
Huey’s reading of Q205/Q206: reachable JEV may advise on resume order and route choice
among services mechanically marked online, within existing policy/capacity/budget.
It cannot mark a service online, authorize a resume, clear a human kill, replay effects
or perform safety actions. If unavailable, fixed rules choose eligible routes subject
to D14’s default probe order. Its absence does not change stop/effect proof requirements.
**D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps **D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps
are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**. are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**.
They live in the instance’s one system config through D3’s reviewed schema/bootstrap They live in the instance’s one system config through D3’s reviewed schema/bootstrap
path. Both configured caps are enforced by the owning instance’s durable ledger. path. Both configured caps are enforced by the owning instance’s durable ledger.
No spending starts until Jason enables paid decisions and installs the key. Each instance No spending starts until Jason enables paid decisions and installs the key.
Huey’s reading of Q206 probe accounting: charge all recovery headless/test-session probe
spend and decision calls to these same daily/monthly instance caps, not a separate free
probe allowance. The ledger groups the budget under the exclusive decision-key binding
and records each actual provider/key identity, including probes that use another provider.
Those probe credentials remain separate from the decision key and need their own
provider-side ceilings. Reserve/settle all probe charges under the same rules below.
No unknown charge is zero and no outage grants fresh spend authority. Each instance
must use a separately provisioned key used only for that instance’s decisions, with a must use a separately provisioned key used only for that instance’s decisions, with a
non-secret provider key ID bound to the instance in the approved setup record. Exclusive non-secret provider key ID bound to the instance in the approved setup record. Exclusive
ownership is a setup attestation, not something the manager can prove mechanically. ownership is a setup attestation, not something the manager can prove mechanically.
@@ -340,8 +460,8 @@ authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/aut
| S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. | | S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. |
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. | | S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. | | S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
| S5 | Retry/DLQ/replay and permitted reroutes, after S3/S4 | Fake provider failures with exact total-attempt/time bounds, restart during backoff, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal. | After MVP. | | S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, relapse and three failed exits alert, human kill unchanged. No real probes. | After MVP. |
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. | | S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, JEV-first then parallel, one success online, reset wait, disabled JEV, aggregate probe spend and budget deferral, restart and hung-probe cleanup. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
Each slice gets independent review and tests that kill a mutation of its central Each slice gets independent review and tests that kill a mutation of its central
invariant. Sage chooses package/full-suite gates appropriate to the final diff. invariant. Sage chooses package/full-suite gates appropriate to the final diff.
@@ -367,14 +487,29 @@ bounded synthetic pressure. No load experiment on the shared development host.
**Recommend yes**, per role and privacy requirement. Empty list means hold/fail **Recommend yes**, per role and privacy requirement. Empty list means hold/fail
visibly instead of silently changing providers or model quality. visibly instead of silently changing providers or model quality.
**Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes” **Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes”
- **Q4 System recovery priority (pending, Jason restated 2026-10-21):** approve the - **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.**
P1 narrowing of Q198 so heavy recovery Jason, verbatim (Q205): “It does not make sense to require the user to release each
takes priority over eligible normal work but still waits for authorization, kill-clear back into operation. I would rather equip the system with the tools necessary to
and hard capacity? **Recommend yes.** Keep health/reconciliation on reserved control detect failure, enter protected mode, monitor for cool-down periods, check for
capacity, without letting a recovery launch bypass those safeguards. system operational status, resume operations.”
DECISIONS Q204 Q20d is conditional on Jason’s reading about returning to normal Jason, verbatim (Q206):
operations, which differs from the design’s P1 recovery admission restriction. Mos
has restated the distinction to Jason. Q4 remains pending, with P1 unchanged. > Mosaic Stack mechanical tooling detects the outage and tracks the status without LLM need
> Continual mechanical attempt for JEV begins with a cooldown of 60, 120, 180 seconds with a ceiling of 10 minutes
> Once JEV is back online, the same cooldown starts for the other harnesses/providers in parallel
> Necessary services are marked back online when found available
> System adapts based on available services, providers, harnesses, usage limits, and usage budgets
Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate
protected-mode recovery without per-session approval. Entry thresholds, checkpoint
handling, deadline freezing, probe deadlines/budget grouping, disabled-JEV behavior,
batch sizing and the three-failed-exit escalation count are proposed readings, not
Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged.
- **Q22 Probe independence (pending):** may other harness/provider checks start without
waiting for JEV, with fixed rules routing until JEV returns? **Recommend yes**, as
Mos’s alternative recorded under DECISIONS Q206. Huey’s reading: that avoids a JEV-only
outage holding otherwise usable services while retaining policy/usage/budget checks.
Until Jason answers, D14 follows his JEV-first, then parallel order.
## Verification and handoff ## Verification and handoff