docs: design protected mode and autonomous outage recovery
This commit is contained in:
1 parent
1e8bccff15
commit
fd44f7c603
3 files changed
+166
-28
No files matched your search
@@ -5185,3 +5185,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi
|
|||||||
2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls.
|
2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls.
|
||||||
|
|
||||||
2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls.
|
2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls.
|
||||||
|
|
||||||
|
2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls.
|
||||||
@@ -1423,3 +1423,4 @@ are never rewritten or removed; corrections are new entries.
|
|||||||
2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge
|
2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge
|
||||||
2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls
|
2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls
|
||||||
2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls
|
2026-10-21T20:19:53Z | Huey (gpt-6.1-sol) | mos-1203, Q198 design r4, DECISIONS Q204 | Q1–Q3 approved quotes recorded, cap settings interpretation labelled, Q4 pending and P1 unchanged; doc-only checks, no builds/live changes/paid calls
|
||||||
|
2026-10-21T20:48:49Z | Huey (gpt-6.1-sol) | mos-1206, Q198 r5 protected mode/outage recovery, Vikunja 452 | Q205/Q206 recorded with labelled readings and probe/resume acceptance proposals; Q4 superseded, Q22 pending; doc checks only, no builds/suites/real probes/live changes/paid calls; Scrooge re-review via Mos
|
||||||
@@ -4,6 +4,7 @@ Build a rules-first manager for admission, bounded retries, recovery and host he
|
|||||||
System stop and health requests bypass the work queue, without bypassing authorization.
|
System stop and health requests bypass the work queue, without bypassing authorization.
|
||||||
Normal and background work wait under pressure, then resume gradually and fairly.
|
Normal and background work wait under pressure, then resume gradually and fairly.
|
||||||
An interrupt requests cancellation. Capacity stays occupied until the owner proves exit.
|
An interrupt requests cancellation. Capacity stays occupied until the owner proves exit.
|
||||||
|
Outages enter system-owned protected mode, then probe and resume automatically.
|
||||||
Unknown external effects require reconciliation, never automatic replay.
|
Unknown external effects require reconciliation, never automatic replay.
|
||||||
Reuse the bus, controllers, launch registry, run evidence and Console.
|
Reuse the bus, controllers, launch registry, run evidence and Console.
|
||||||
Optional decision models rank already-permitted routes. Recommend Jev when enabled.
|
Optional decision models rank already-permitted routes. Recommend Jev when enabled.
|
||||||
@@ -12,8 +13,8 @@ No live trial, key installation or runtime change is authorized by this design.
|
|||||||
MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate.
|
MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate.
|
||||||
|
|
||||||
Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only.
|
Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only.
|
||||||
Revision **r4**, mos-1203, records Jason’s DECISIONS Q204 answers to section 9
|
Revision **r5**, mos-1206, applies DECISIONS Q205/Q206 to r4 `1e8bccff1`.
|
||||||
on r3 `a36952564`. Q4 remains pending.
|
Q4 is superseded. Q22 remains pending. New implementation choices are Huey’s readings.
|
||||||
All defaults below are proposals, not measured safe operating limits or new authority.
|
All defaults below are proposals, not measured safe operating limits or new authority.
|
||||||
|
|
||||||
## 1. Existing pieces and reuse
|
## 1. Existing pieces and reuse
|
||||||
@@ -83,7 +84,7 @@ handling outside that pool. This is software scheduling, not hardware/real-time
|
|||||||
content reference plus digest (content stays at its authorized source), policy revision,
|
content reference plus digest (content stays at its authorized source), policy revision,
|
||||||
class, enqueue time/deadline, resource estimate, allowed routes and idempotency key.
|
class, enqueue time/deadline, resource estimate, allowed routes and idempotency key.
|
||||||
Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`,
|
Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`,
|
||||||
with branches to `retry-wait`, `dead-letter`, or `uncertain`. Record every attempt,
|
with branches to `retry-wait`, `outage-paused`, `dead-letter`, or `uncertain`. Record every attempt,
|
||||||
route, reason code, owner identity, heartbeat and evidence references. Materialized
|
route, reason code, owner identity, heartbeat and evidence references. Materialized
|
||||||
state is rebuildable. Re-check permission, kill state and policy at dispatch/replay.
|
state is rebuildable. Re-check permission, kill state and policy at dispatch/replay.
|
||||||
A duplicate key with the same digest returns the original request. A changed digest
|
A duplicate key with the same digest returns the original request. A changed digest
|
||||||
@@ -111,15 +112,15 @@ until a shared ledger and its authority boundary receive separate review.
|
|||||||
| Ref/class | Requests | Admission and preemption |
|
| Ref/class | Requests | Admission and preemption |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. |
|
| P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. |
|
||||||
| P1 system | Lightweight health reads, lease reconciliation, approved recovery | Reserved control capacity. Health/reconciliation can run under pressure. Recovery that launches heavy work still needs capacity and kill-clear. Never auto-clear a kill. |
|
| P1 system | Lightweight health reads, lease reconciliation, automatic outage recovery under Q205/Q206 | Reserved control capacity. Health/reconciliation can run under pressure. Resume needs capacity, current policy and a clear human kill switch, without a new approval per session. Never auto-clear a kill. |
|
||||||
| P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. |
|
| P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. |
|
||||||
| P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. |
|
| P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. |
|
||||||
|
|
||||||
The P1 capacity/authorization hold narrows Jason’s Q198 “take precedent” direction:
|
DECISIONS Q205/Q206 supersede the old P1 per-session recovery approval question.
|
||||||
system recovery gets priority over eligible normal work, but heavy recovery cannot
|
Huey’s reading: the configured recovery policy supplies standing authority to resume
|
||||||
bypass authorization, kill state or hard capacity. This is a proposed safety restriction,
|
eligible paused work. Current permissions, hard capacity and the human kill switch still
|
||||||
not a claim that Jason already approved that exception. Bounded health and reconciliation
|
apply. Health/reconciliation retain reserved capacity. Protected mode can clear itself,
|
||||||
retain reserved capacity. Section 9 Q4 puts this restriction to Jason for approval.
|
but cannot clear a human kill or grant new authority.
|
||||||
|
|
||||||
| Work | What interrupt means | Hard-stop boundary |
|
| Work | What interrupt means | Hard-stop boundary |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -158,7 +159,8 @@ explicitly supported exclusive envelope, never an unbounded resource guess. Star
|
|||||||
then measure before raising concurrency. Provider cooldown blocks only affected
|
then measure before raising concurrency. Provider cooldown blocks only affected
|
||||||
provider/account/model scopes, not all host work. An open provider circuit waits at
|
provider/account/model scopes, not all host work. An open provider circuit waits at
|
||||||
least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on
|
least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on
|
||||||
failure. Never shorten an upstream wait to fit a local retry delay.
|
failure. Never shorten an upstream wait to fit a local retry delay. A declared outage
|
||||||
|
uses D13–D15’s Q206 probe schedule instead of this short transient-failure circuit.
|
||||||
|
|
||||||
These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or
|
These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or
|
||||||
Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and
|
Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and
|
||||||
@@ -171,7 +173,8 @@ P0 uses a separately bounded, target-deduplicated control path. Journal failure
|
|||||||
new work and alerts, without disabling the existing emergency stop path.
|
new work and alerts, without disabling the existing emergency stop path.
|
||||||
|
|
||||||
Normal requests expire after 30 minutes waiting, background after 24 hours, unless an
|
Normal requests expire after 30 minutes waiting, background after 24 hours, unless an
|
||||||
explicit authorized deadline differs. Show expiry and retain its receipt. Once healthy,
|
explicit authorized deadline differs. Outage-paused time is excluded under D6.
|
||||||
|
Show expiry and retain its receipt. Once healthy,
|
||||||
weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant.
|
weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant.
|
||||||
After 10 minutes, background ages into normal scheduling rank, never into system
|
After 10 minutes, background ages into normal scheduling rank, never into system
|
||||||
priority and never through pressure holds. After control bursts, service eligible work.
|
priority and never through pressure holds. After control bursts, service eligible work.
|
||||||
@@ -181,13 +184,28 @@ alert on oldest wait >10 minutes rather than promise starvation freedom under ov
|
|||||||
## 5. Retries, dead letters and routes
|
## 5. Retries, dead letters and routes
|
||||||
|
|
||||||
**D6 Retry ownership.** One logical request gets at most **3 total manager attempts**,
|
**D6 Retry ownership.** One logical request gets at most **3 total manager attempts**,
|
||||||
10 minutes of retry time and its original deadline, whichever ends first. Retry only
|
10 minutes of retry time and its effective deadline, whichever ends first. Retry only
|
||||||
transient failures known to precede effects, or operations with verified idempotency.
|
transient failures known to precede effects, or operations with verified idempotency.
|
||||||
Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum.
|
Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum.
|
||||||
Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal
|
Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal
|
||||||
retries must fit a declared per-attempt budget. If that budget is unknown, do not layer
|
retries must fit a declared per-attempt budget. If that budget is unknown, do not layer
|
||||||
automatic manager retries over the proxy. Max one route change within the three attempts.
|
automatic manager retries over the proxy. Max one route change within the three attempts.
|
||||||
|
|
||||||
|
**Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt
|
||||||
|
allowance and remaining ten-minute retry window. Neither resets nor advances during the
|
||||||
|
pause, including the outage-causing failure once classified as an outage. It never enters
|
||||||
|
the DLQ because of that outage. Huey’s implementation reading: track provisional attempt
|
||||||
|
outcomes until classification so a threshold-crossing failure cannot first dead-letter work. Resume is continuation of
|
||||||
|
the same logical request, not D7 replay. Actual unrelated failures still count normally.
|
||||||
|
|
||||||
|
Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline
|
||||||
|
budgets during the union of outage-paused intervals, never counting overlapping service
|
||||||
|
outages twice. Keep the original timestamp and publish the effective deadline extended
|
||||||
|
by paused time. This deliberately interprets a caller-set deadline as an execution budget,
|
||||||
|
not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation
|
||||||
|
still takes effect during a pause. Journal pause intervals and counters across restarts.
|
||||||
|
An interrupted effect remains uncertain until reconciled, regardless of the clock.
|
||||||
|
|
||||||
D6 covers managed work requests only. Existing notifier delivery backoff
|
D6 covers managed work requests only. Existing notifier delivery backoff
|
||||||
(`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff
|
(`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff
|
||||||
(`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own
|
(`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own
|
||||||
@@ -196,13 +214,16 @@ Any later consolidation needs its own design and acceptance tests.
|
|||||||
|
|
||||||
| Outcome | Action |
|
| Outcome | Action |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Connection refused before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility. |
|
| Mechanically classified outage (D13) | Outage-paused, counters/clocks frozen, service probes under D14. No outage DLQ. |
|
||||||
|
| Isolated connection refusal before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility and D13 does not classify an outage. |
|
||||||
| Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. |
|
| Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. |
|
||||||
| Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. |
|
| Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. |
|
||||||
| Transient retry budget/deadline exhausted, deterministic execution failure | Dead-letter with evidence and actionable reason. Admission expiry is an expired terminal item. |
|
| Non-outage retry budget/deadline exhausted, deterministic request failure | Dead-letter with evidence and actionable reason. Non-outage admission expiry is an expired terminal item. |
|
||||||
| Explicit user cancellation | Terminal cancelled. Never retry automatically. |
|
| Explicit user cancellation | Terminal cancelled. Never retry automatically. |
|
||||||
|
|
||||||
**D7 DLQ.** A durable request state and append-only disposition history, not copied
|
**D7 DLQ.** Only requests that fail on their own use this path, not outage-paused work.
|
||||||
|
Validation/permission refusals remain refusals under D6, with their request evidence.
|
||||||
|
A durable request state and append-only disposition history, not copied
|
||||||
prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted
|
prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted
|
||||||
error code, idempotency/effect status and links to authorized evidence in Console
|
error code, idempotency/effect status and links to authorized evidence in Console
|
||||||
“System → Requests → Dead letters”. Propose equivalent `mosaic system requests
|
“System → Requests → Dead letters”. Propose equivalent `mosaic system requests
|
||||||
@@ -212,6 +233,8 @@ replay. Jason or an explicitly delegated role can replay after cause correction
|
|||||||
reconciliation. Replay creates a new request linked to its parent, with a fresh bounded
|
reconciliation. Replay creates a new request linked to its parent, with a fresh bounded
|
||||||
budget and current policy. No mass replay, unattended replay chains or resurrected
|
budget and current policy. No mass replay, unattended replay chains or resurrected
|
||||||
cancelled work. Unknown effects require explicit reconciliation evidence before replay.
|
cancelled work. Unknown effects require explicit reconciliation evidence before replay.
|
||||||
|
D15’s automatic batch continuation of outage-paused work is not a replay chain and
|
||||||
|
requires no person to release each request. An outage never sends those requests here.
|
||||||
|
|
||||||
**D8 Rerouting.** Candidate routes come only from the user’s configured allowlist,
|
**D8 Rerouting.** Candidate routes come only from the user’s configured allowlist,
|
||||||
filtered for model capability, tool protocol, context, privacy/region, account budget
|
filtered for model capability, tool protocol, context, privacy/region, account budget
|
||||||
@@ -221,6 +244,88 @@ or migrate a live transcript silently. Mid-turn failure is reconciled before a f
|
|||||||
attempt. Record requested and effective route and reason. New provider consent/spend
|
attempt. Record requested and effective route and reason. New provider consent/spend
|
||||||
is separate from enabling the decision hook. F10’s preflight fallback remains distinct.
|
is separate from enabling the decision hook. F10’s preflight fallback remains distinct.
|
||||||
|
|
||||||
|
## 5a. Protected mode and outage recovery
|
||||||
|
|
||||||
|
**D13 Protected mode (Huey’s reading of Q205/Q206).** The supervisor owns a durable
|
||||||
|
`normal → protected → checking → canary → batch-resume → normal` state per affected
|
||||||
|
service scope, with a host/network incident able to hold all dependent scopes. This is
|
||||||
|
separate from the human kill switch. D1 remains unchanged: the manager never sets or
|
||||||
|
clears kill-switch state. Detection/status tracking are mechanical, without an LLM.
|
||||||
|
|
||||||
|
Proposed entry thresholds, all Huey’s readings: three consecutive failed network checks
|
||||||
|
at the 2-second sample cadence before protected mode enter a network incident.
|
||||||
|
Once protected, active checks follow D14’s schedule. For a provider/account scope,
|
||||||
|
enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient
|
||||||
|
network/429/5xx failures across at least two distinct requests, or three consecutive
|
||||||
|
failed endpoint checks. One explicit account-wide quota/reset response can enter its
|
||||||
|
scope immediately. Isolated validation/permission failures do not count. Until wider
|
||||||
|
failure scope is proven, hold only that route/account. A missing required observation
|
||||||
|
holds work as unknown rather than guessing recovery. Persist incident and classification
|
||||||
|
before advancing affected request outcomes. These settings use D3’s reviewed config path.
|
||||||
|
|
||||||
|
Fence affected new dispatch. Ask running owners to pause at safe points using F5/F6.
|
||||||
|
Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed
|
||||||
|
rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable,
|
||||||
|
record uncertain ownership and retain capacity. Do not force-stop a write or manufacture
|
||||||
|
human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects
|
||||||
|
mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected
|
||||||
|
services can continue under admission, except where D14’s default recovery order holds
|
||||||
|
them pending the JEV stage. No person releases each paused session.
|
||||||
|
|
||||||
|
**D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first
|
||||||
|
performs mechanical network and endpoint checks. Only after the endpoint answers may a
|
||||||
|
one-turn headless probe run through a minimal harness config. A check of a dead link
|
||||||
|
makes no model call. Huey’s reading: use a fixed synthetic input, no tools, user content,
|
||||||
|
workspace effects or inherited worker environment, and separately authorized probe
|
||||||
|
credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat
|
||||||
|
request. Probe success means the endpoint plus the configured minimal harness/model
|
||||||
|
path succeeded, not merely an HTTP connection. One successful complete probe marks that
|
||||||
|
service online. Missing credentials, exhausted budget or an unknown charge keep it
|
||||||
|
unverified, without synthetic success or unauthorized spend.
|
||||||
|
|
||||||
|
Default order is Jason’s: **JEV first**, then the other harness/provider checks start
|
||||||
|
**in parallel** after JEV succeeds. Each service has its own durable probe schedule:
|
||||||
|
wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540,
|
||||||
|
600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least
|
||||||
|
Retry-After or the usage-limit reset before probing, even if longer than 600 seconds.
|
||||||
|
The ceiling bounds this schedule, not a provider-mandated wait. Unknown reset time
|
||||||
|
keeps the scope held and visible rather than guessing a usable quota. Service probes
|
||||||
|
are separate from D10 routing calls and never retry a paused work request.
|
||||||
|
|
||||||
|
Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check
|
||||||
|
deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup
|
||||||
|
and uncertain outcomes retain their reservations. These deadlines are independent of
|
||||||
|
the routing hook’s 500 ms candidate timeout. Scheduling retries indefinitely does not
|
||||||
|
override D11 budgets: when funds/caps are unavailable, keep mechanical checks running
|
||||||
|
on schedule and defer paid headless probes until authorized budget is available. A human
|
||||||
|
kill prevents headless probes/agent launches, while local read-only health tracking
|
||||||
|
continues. The kill can be cleared only through its existing human path.
|
||||||
|
|
||||||
|
Huey’s reading for disabled optional JEV: if the user never enabled/configured it,
|
||||||
|
record its stage as not configured and begin the other checks without a paid JEV call.
|
||||||
|
If JEV is configured but unavailable, keep the specified JEV-first order. A JEV-only
|
||||||
|
outage can therefore delay otherwise available harnesses. Q22 proposes changing that
|
||||||
|
order and remains pending, so its alternative is not implemented by this design.
|
||||||
|
|
||||||
|
**D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service
|
||||||
|
online proof and D4’s healthy release thresholds, reconcile paused owners/effects and
|
||||||
|
run **one test session** on an eligible online route. Use disposable synthetic work,
|
||||||
|
current policy and bounded capacity. No request with uncertain effects is used as a
|
||||||
|
test. On success resume eligible paused work in batches of at most five, with one
|
||||||
|
30-second healthy observation interval between batches and D4’s stricter dispatch
|
||||||
|
rate/reservations still enforced. A batch needs no new approval per session. If the
|
||||||
|
test session or a batch fails, return to protected mode and its probe schedule.
|
||||||
|
Request-local defects still follow D6/D7, with evidence distinguishing that failure
|
||||||
|
from a provider outage rather than falsely marking a healthy service offline.
|
||||||
|
|
||||||
|
Huey’s proposed escalation count: after **three consecutive failed automatic exits**
|
||||||
|
(canary or batch stages) in the same incident, alert Mos/Jason and hold production
|
||||||
|
resume for diagnosis. Continue mechanical monitoring and budget-permitted probes
|
||||||
|
without an attempt limit. A fully successful exit resets the exit-failure count. This
|
||||||
|
is the repeated-exit boundary where a person is needed, not a prompt for each session
|
||||||
|
or each failed probe. Existing policy, missing credentials and unresolved external
|
||||||
|
effects remain fail-closed constraints, never reasons to improvise authority.
|
||||||
|
|
||||||
## 6. Monitoring and Console
|
## 6. Monitoring and Console
|
||||||
|
|
||||||
**D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical
|
**D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical
|
||||||
@@ -235,7 +340,9 @@ an approved re-pin window after MVP. No sampler is started by this design work.
|
|||||||
**Measurements.** Pressure samples and freshness, active/reserved work, ownership
|
**Measurements.** Pressure samples and freshness, active/reserved work, ownership
|
||||||
uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency,
|
uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency,
|
||||||
retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state,
|
retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state,
|
||||||
manager heartbeat and decision-model timeouts/spend/unknown charges. Store bounded
|
manager heartbeat and decision-model timeouts/spend/unknown charges. Huey’s reading:
|
||||||
|
also expose protected-mode scope, last/next probe, online proof, paused request counts,
|
||||||
|
canary/batch progress and consecutive exit failures. Store bounded
|
||||||
aggregate samples for 7 days, without request bodies. Request receipts/evidence follow
|
aggregate samples for 7 days, without request bodies. Request receipts/evidence follow
|
||||||
the instance’s retention policy, with the existing write-once run invariant preserved.
|
the instance’s retention policy, with the existing write-once run invariant preserved.
|
||||||
|
|
||||||
@@ -245,7 +352,8 @@ Requests, Dead letters and evidence. Reuse Console read transport, auth and scop
|
|||||||
human writes (F8). No new public listener. Show stale/unknown explicitly, not green.
|
human writes (F8). No new public listener. Show stale/unknown explicitly, not green.
|
||||||
|
|
||||||
Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and
|
Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and
|
||||||
unknown decision charge. Warn on pressure hold >60 seconds, control heartbeat stale
|
unknown decision charge. Huey’s reading: show protected-mode entry/recovery and alert
|
||||||
|
on D15’s repeated-exit boundary. Warn on pressure hold >60 seconds, control heartbeat stale
|
||||||
>6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit
|
>6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit
|
||||||
one recovery notification. Use Console attention/trail first. External notification
|
one recovery notification. Use Console attention/trail first. External notification
|
||||||
channels require their own configured authorization. A lightweight supervisor watchdog
|
channels require their own configured authorization. A lightweight supervisor watchdog
|
||||||
@@ -281,12 +389,24 @@ confidence value therefore uses rules. Discard late answers for
|
|||||||
routing, but still account for their charge. The hook cannot authorize work, choose
|
routing, but still account for their charge. The hook cannot authorize work, choose
|
||||||
P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the
|
P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the
|
||||||
eligible set. Recheck eligibility after it returns. Never call it for safety actions.
|
eligible set. Recheck eligibility after it returns. Never call it for safety actions.
|
||||||
|
Huey’s reading of Q205/Q206: reachable JEV may advise on resume order and route choice
|
||||||
|
among services mechanically marked online, within existing policy/capacity/budget.
|
||||||
|
It cannot mark a service online, authorize a resume, clear a human kill, replay effects
|
||||||
|
or perform safety actions. If unavailable, fixed rules choose eligible routes subject
|
||||||
|
to D14’s default probe order. Its absence does not change stop/effect proof requirements.
|
||||||
|
|
||||||
**D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps
|
**D11 Cost scope.** Huey’s reading of DECISIONS Q204 Q20b: daily and monthly USD caps
|
||||||
are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**.
|
are user settings, with defaults of **$0.10/day and $1/month per dedicated decision key**.
|
||||||
They live in the instance’s one system config through D3’s reviewed schema/bootstrap
|
They live in the instance’s one system config through D3’s reviewed schema/bootstrap
|
||||||
path. Both configured caps are enforced by the owning instance’s durable ledger.
|
path. Both configured caps are enforced by the owning instance’s durable ledger.
|
||||||
No spending starts until Jason enables paid decisions and installs the key. Each instance
|
No spending starts until Jason enables paid decisions and installs the key.
|
||||||
|
Huey’s reading of Q206 probe accounting: charge all recovery headless/test-session probe
|
||||||
|
spend and decision calls to these same daily/monthly instance caps, not a separate free
|
||||||
|
probe allowance. The ledger groups the budget under the exclusive decision-key binding
|
||||||
|
and records each actual provider/key identity, including probes that use another provider.
|
||||||
|
Those probe credentials remain separate from the decision key and need their own
|
||||||
|
provider-side ceilings. Reserve/settle all probe charges under the same rules below.
|
||||||
|
No unknown charge is zero and no outage grants fresh spend authority. Each instance
|
||||||
must use a separately provisioned key used only for that instance’s decisions, with a
|
must use a separately provisioned key used only for that instance’s decisions, with a
|
||||||
non-secret provider key ID bound to the instance in the approved setup record. Exclusive
|
non-secret provider key ID bound to the instance in the approved setup record. Exclusive
|
||||||
ownership is a setup attestation, not something the manager can prove mechanically.
|
ownership is a setup attestation, not something the manager can prove mechanically.
|
||||||
@@ -340,8 +460,8 @@ authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/aut
|
|||||||
| S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. |
|
| S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. |
|
||||||
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
|
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
|
||||||
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
|
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
|
||||||
| S5 | Retry/DLQ/replay and permitted reroutes, after S3/S4 | Fake provider failures with exact total-attempt/time bounds, restart during backoff, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal. | After MVP. |
|
| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, relapse and three failed exits alert, human kill unchanged. No real probes. | After MVP. |
|
||||||
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
|
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, JEV-first then parallel, one success online, reset wait, disabled JEV, aggregate probe spend and budget deferral, restart and hung-probe cleanup. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
|
||||||
|
|
||||||
Each slice gets independent review and tests that kill a mutation of its central
|
Each slice gets independent review and tests that kill a mutation of its central
|
||||||
invariant. Sage chooses package/full-suite gates appropriate to the final diff.
|
invariant. Sage chooses package/full-suite gates appropriate to the final diff.
|
||||||
@@ -367,14 +487,29 @@ bounded synthetic pressure. No load experiment on the shared development host.
|
|||||||
**Recommend yes**, per role and privacy requirement. Empty list means hold/fail
|
**Recommend yes**, per role and privacy requirement. Empty list means hold/fail
|
||||||
visibly instead of silently changing providers or model quality.
|
visibly instead of silently changing providers or model quality.
|
||||||
**Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes”
|
**Approved**, DECISIONS Q204. Jason, verbatim: “Q20c: yes”
|
||||||
- **Q4 System recovery priority (pending, Jason restated 2026-10-21):** approve the
|
- **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.**
|
||||||
P1 narrowing of Q198 so heavy recovery
|
Jason, verbatim (Q205): “It does not make sense to require the user to release each
|
||||||
takes priority over eligible normal work but still waits for authorization, kill-clear
|
back into operation. I would rather equip the system with the tools necessary to
|
||||||
and hard capacity? **Recommend yes.** Keep health/reconciliation on reserved control
|
detect failure, enter protected mode, monitor for cool-down periods, check for
|
||||||
capacity, without letting a recovery launch bypass those safeguards.
|
system operational status, resume operations.”
|
||||||
DECISIONS Q204 Q20d is conditional on Jason’s reading about returning to normal
|
Jason, verbatim (Q206):
|
||||||
operations, which differs from the design’s P1 recovery admission restriction. Mos
|
|
||||||
has restated the distinction to Jason. Q4 remains pending, with P1 unchanged.
|
> Mosaic Stack mechanical tooling detects the outage and tracks the status without LLM need
|
||||||
|
> Continual mechanical attempt for JEV begins with a cooldown of 60, 120, 180 seconds with a ceiling of 10 minutes
|
||||||
|
> Once JEV is back online, the same cooldown starts for the other harnesses/providers in parallel
|
||||||
|
> Necessary services are marked back online when found available
|
||||||
|
> System adapts based on available services, providers, harnesses, usage limits, and usage budgets
|
||||||
|
|
||||||
|
Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate
|
||||||
|
protected-mode recovery without per-session approval. Entry thresholds, checkpoint
|
||||||
|
handling, deadline freezing, probe deadlines/budget grouping, disabled-JEV behavior,
|
||||||
|
batch sizing and the three-failed-exit escalation count are proposed readings, not
|
||||||
|
Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged.
|
||||||
|
- **Q22 Probe independence (pending):** may other harness/provider checks start without
|
||||||
|
waiting for JEV, with fixed rules routing until JEV returns? **Recommend yes**, as
|
||||||
|
Mos’s alternative recorded under DECISIONS Q206. Huey’s reading: that avoids a JEV-only
|
||||||
|
outage holding otherwise usable services while retaining policy/usage/budget checks.
|
||||||
|
Until Jason answers, D14 follows his JEV-first, then parallel order.
|
||||||
|
|
||||||
## Verification and handoff
|
## Verification and handoff
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user