docs: design Q198 mechanical system manager

This commit is contained in:
jason.woltje committed 2026-10-21 11:34:48 -05:00
1 parent 6c75887b0c
commit add7cb5069
3 files changed
+323

No files matched your search

+2
View File
@@ -5177,3 +5177,5 @@ Before: managed project/thread verbs had no Console conversation area. After: fi
2026-10-21T01:28:05Z | Darkwing | row 232 #1734 installer I1 build, round 1 in review | Before: stack-release.sh managed only user units (systemctl --user), and stack-copy.sh had no bundle copy. After: --units external (adopt, activate, rollback --to, status, confirm, revert, gate --since, AWAITING-UNITS, stale-start, external-pm, --plan --expect-from) and stack-copy check --units external, copy --bundle FILE SHA, wrapper ID --create; user mode byte-identical to the base by test. First build 0af0c73c6 on 00d4cc1f6; rebased onto row 200 (cb5fdb064) as a0ffabc3f on darkwing/row232-r1; --sso refused under external (D9), row 200 config_checks run under external (D10). Gate on Sage's slot (sage-1618): cli 561/0, every scripts/test-*.sh green; 17/17 mutants killed. Results commit 84f1a91e6 (pushed, seat branch only). Manifest 7a3b4ea1 (9 paths), queue revs 2595-2596, request comment 29825, reviewer Filbert. Packet agents/darkwing/work/queue-232/BUILD.md. 2026-10-21T01:28:05Z | Darkwing | row 232 #1734 installer I1 build, round 1 in review | Before: stack-release.sh managed only user units (systemctl --user), and stack-copy.sh had no bundle copy. After: --units external (adopt, activate, rollback --to, status, confirm, revert, gate --since, AWAITING-UNITS, stale-start, external-pm, --plan --expect-from) and stack-copy check --units external, copy --bundle FILE SHA, wrapper ID --create; user mode byte-identical to the base by test. First build 0af0c73c6 on 00d4cc1f6; rebased onto row 200 (cb5fdb064) as a0ffabc3f on darkwing/row232-r1; --sso refused under external (D9), row 200 config_checks run under external (D10). Gate on Sage's slot (sage-1618): cli 561/0, every scripts/test-*.sh green; 17/17 mutants killed. Results commit 84f1a91e6 (pushed, seat branch only). Manifest 7a3b4ea1 (9 paths), queue revs 2595-2596, request comment 29825, reviewer Filbert. Packet agents/darkwing/work/queue-232/BUILD.md.
2026-10-21T04:50:44Z | Darkwing | row 233 #1735 installer I3 build, round 1 in review | Before: root had no upgrade driver for root-owned units and no host run sheet; stack-release.sh had no rehearse --backup last and no external plan refusals. After: scripts/stack-upgrade.sh upgrade and rollback (r6 A1-A5, A2f, R1-R4) with 20 tests, docs/guides/host-run-sheet.md, TOOLS.md lines, and on Sage's rulings stack-release.sh --backup last, BACKUP_STORE and the --plan refusals open-activation and migrate-required (3 tests). Base e65a3ab4d (row 232's landing). Gate on Sage's slot (sage-1704) on d99166ade: cli 616/0, every scripts/test-*.sh green, 1201/0 summed by awk; m1-m8 8/8 killed. Correction: the process sweep after the gate found an orphaned sleep 60 from the lock test's flock holder (N1); 08db6ac3f fixes it test only, Sage ruled the gate stands (sage-1707), rerun c0 20/0 and m1 killed with 0 left (sage-1708). Tip 1a1dc80ca (pushed, seat branch only). Manifest 464ce9eb (7 paths), queue revs 2644 and 2647, request comment 29901, reviewer Filbert. Packet agents/darkwing/work/queue-233/BUILD.md. 2026-10-21T04:50:44Z | Darkwing | row 233 #1735 installer I3 build, round 1 in review | Before: root had no upgrade driver for root-owned units and no host run sheet; stack-release.sh had no rehearse --backup last and no external plan refusals. After: scripts/stack-upgrade.sh upgrade and rollback (r6 A1-A5, A2f, R1-R4) with 20 tests, docs/guides/host-run-sheet.md, TOOLS.md lines, and on Sage's rulings stack-release.sh --backup last, BACKUP_STORE and the --plan refusals open-activation and migrate-required (3 tests). Base e65a3ab4d (row 232's landing). Gate on Sage's slot (sage-1704) on d99166ade: cli 616/0, every scripts/test-*.sh green, 1201/0 summed by awk; m1-m8 8/8 killed. Correction: the process sweep after the gate found an orphaned sleep 60 from the lock test's flock holder (N1); 08db6ac3f fixes it test only, Sage ruled the gate stands (sage-1707), rerun c0 20/0 and m1 killed with 0 left (sage-1708). Tip 1a1dc80ca (pushed, seat branch only). Manifest 464ce9eb (7 paths), queue revs 2644 and 2647, request comment 29901, reviewer Filbert. Packet agents/darkwing/work/queue-233/BUILD.md.
2026-10-21T06:37:41Z | Darkwing | row 240 #1746 installer I4 build, round 1 in review | Before: root had no host check for the bwrap AppArmor profile, its pins or a boot unit, and stack-upgrade.sh exited 1 args-failed when the args move after A5 failed, while exit 1 means the old copy runs. After: scripts/case1-hostcheck.sh (seven items, --check, --enforce, --upstream), scripts/case1-pins.sh, mosaic-hostcheck.service and 16 tests; the args move failure exits 5 args-failed, documented in TOOLS.md and run sheet section 6 and tested; run sheet section 3 puts the pinned npm on PATH and names the bin/claude.exe --claude path; sections 8 and 10 have precheck lines. Verification: slot run r1 on 47ec9fe2f failed at the singles (the new N1 test's after hook threw ENOENT after the fixture cleanup removed the root); b45852daa drops the hook; r2 on b45852daa green: singles 16/0 and 21/0, c0 plus M1-M8 8/8 killed, cli 639/0, scripts 585/0. Candidate darkwing/row240-r3 ad1b23ab2 on c771a36f4, manifest 573ece95 (9 paths), revs 2694 and 2695, comment 29943. 2026-10-21T06:37:41Z | Darkwing | row 240 #1746 installer I4 build, round 1 in review | Before: root had no host check for the bwrap AppArmor profile, its pins or a boot unit, and stack-upgrade.sh exited 1 args-failed when the args move after A5 failed, while exit 1 means the old copy runs. After: scripts/case1-hostcheck.sh (seven items, --check, --enforce, --upstream), scripts/case1-pins.sh, mosaic-hostcheck.service and 16 tests; the args move failure exits 5 args-failed, documented in TOOLS.md and run sheet section 6 and tested; run sheet section 3 puts the pinned npm on PATH and names the bin/claude.exe --claude path; sections 8 and 10 have precheck lines. Verification: slot run r1 on 47ec9fe2f failed at the singles (the new N1 test's after hook threw ENOENT after the fixture cleanup removed the root); b45852daa drops the hook; r2 on b45852daa green: singles 16/0 and 21/0, c0 plus M1-M8 8/8 killed, cli 639/0, scripts 585/0. Candidate darkwing/row240-r3 ad1b23ab2 on c771a36f4, manifest 573ece95 (9 paths), revs 2694 and 2695, comment 29943.
2026-10-21T16:34:44Z | Huey | Q198/Q199 system-manager design, mos-1183, Vikunja 452 | Before: pinned foundation 6c75887b0 has separate bus, development queue, run evidence, turn/cohort controls and launch/usage paths; the assignment requests a mechanical manager design. After: docs/plans/2026-10-21_q198-system-manager.md proposes admission/cooldown, priority and safe interruption, durable requests, bounded retry/DLQ/replay/rerouting, Console monitoring, optional decision adapters with USD reservations, six ordered slices and three recommended owner decisions. Verification: 26 pinned file:line references resolve, ten-line summary, manual source/contract review, git diff --check and installed queue guard pass. No builds/suites, runtime writes, credential reads, service changes or paid calls. Huey queue next refused the linked worktree, then returned nothing from canonical checkout; direct brief controls this assignment. Gitea issue and queue-row capture are handed to Sage under the no-credential-read/no-queue-write boundary.
+1
View File
@@ -1419,3 +1419,4 @@ are never rewritten or removed; corrections are new entries.
2026-10-21T06:47:54Z | researcher | row 228 (#1729) round 3 in review (rev 2702; start rev 2700) | rebase of r2 onto the 234 landing c771a36f4 as researcher/row228-r3 cf18c42b9 (one commit, pushed) with Webby's round 2 N1 (:711 and :718); manifest 82cf98c6 (3 files); slot sage-1796: session.test 13 + host.test 42 = 55/0, manifest OK before and after, busy228.sh copy named in the clear; r2 94/0 and M1 killed stand (seat blobs unchanged); request attempt left in requesting state (lock held by sage-238-review-2b when the outcome rev was written, exit 3; transport said failed pre-send, no token), Sage resolves and posts as jarvis; packet agents/researcher/work/queue-228/r3 2026-10-21T06:47:54Z | researcher | row 228 (#1729) round 3 in review (rev 2702; start rev 2700) | rebase of r2 onto the 234 landing c771a36f4 as researcher/row228-r3 cf18c42b9 (one commit, pushed) with Webby's round 2 N1 (:711 and :718); manifest 82cf98c6 (3 files); slot sage-1796: session.test 13 + host.test 42 = 55/0, manifest OK before and after, busy228.sh copy named in the clear; r2 94/0 and M1 killed stand (seat blobs unchanged); request attempt left in requesting state (lock held by sage-238-review-2b when the outcome rev was written, exit 3; transport said failed pre-send, no token), Sage resolves and posts as jarvis; packet agents/researcher/work/queue-228/r3
2026-10-21T07:29:23Z | researcher | row 228 (#1729) landed as 86249ac55 (pushed 07:25Z by Sage) | patch-id b6ffe31feb0d matches my r3 candidate cf18c42b9 (checked here against origin/refactor); Sage combined gate with 238: seat 165/0, cli 640/0, scripts 585/0; queue note rev 2725, done rev 2727, #1729 closed comment 29966 (sage-1824); q228r2 worktree removed, branches researcher/row228, -r2 and -r3 stay on origin and locally; packets agents/researcher/work/queue-228 r1 r2 r3 uncommitted 2026-10-21T07:29:23Z | researcher | row 228 (#1729) landed as 86249ac55 (pushed 07:25Z by Sage) | patch-id b6ffe31feb0d matches my r3 candidate cf18c42b9 (checked here against origin/refactor); Sage combined gate with 238: seat 165/0, cli 640/0, scripts 585/0; queue note rev 2725, done rev 2727, #1729 closed comment 29966 (sage-1824); q228r2 worktree removed, branches researcher/row228, -r2 and -r3 stay on origin and locally; packets agents/researcher/work/queue-228 r1 r2 r3 uncommitted
2026-10-21T15:19:14Z | Sage (Opus 5.5, 1ef1e4f8) | MVP push 10-21 afternoon: rows 249, 197 proof 3/4, 222 singles, 66 r4d, queue-commit | 249 landed 9de5a57ab (+7b9737953 DEFERRED N1, #1751 closed, Vikunja 445 c442); 197 proof 3 rc=1 on a driver defect, Herb's fix a5b636750 to Webby, proof 4 set up in sage-r197p4; 222 singles 34/34 and 5 mutants killed (sage-2028); 66 round 4 in review (82c830779, c30083/30084); queue revs 2642-2904 committed 0636721e6 2026-10-21T15:19:14Z | Sage (Opus 5.5, 1ef1e4f8) | MVP push 10-21 afternoon: rows 249, 197 proof 3/4, 222 singles, 66 r4d, queue-commit | 249 landed 9de5a57ab (+7b9737953 DEFERRED N1, #1751 closed, Vikunja 445 c442); 197 proof 3 rc=1 on a driver defect, Herb's fix a5b636750 to Webby, proof 4 set up in sage-r197p4; 222 singles 34/34 and 5 mutants killed (sage-2028); 66 round 4 in review (82c830779, c30083/30084); queue revs 2642-2904 committed 0636721e6
2026-10-21T16:34:44Z | Huey (gpt-6.1-sol) | mos-1183, Q198/Q199 system-manager design, Vikunja 452, huey/system-manager-design on 6c75887b0 | design and six proposed slices documented; pinned citations and docs checks pass; no build, suite, live action or paid call; Gitea/queue capture remains Sage-owned
@@ -0,0 +1,320 @@
# Q198: mechanical system manager
Build a rules-first manager for admission, bounded retries, recovery and host health.
System stop and health requests bypass the work queue, without bypassing authorization.
Normal and background work wait under pressure, then resume gradually and fairly.
An interrupt requests cancellation. Capacity stays occupied until the owner proves exit.
Unknown external effects require reconciliation, never automatic replay.
Reuse the bus, controllers, launch registry, run evidence and Console.
Optional decision models rank already-permitted routes. Recommend Jev when enabled.
Model calls have a timeout, a durable spend cap and a strict metadata allowlist.
No live trial, key installation or runtime change is authorized by this design.
MVP work (Vikunja 414 and 416) keeps first claim on Sage’s gate.
Brief: mos-1183, DECISIONS Q198/Q199, Vikunja **452**. Owner: Huey. Design only.
All defaults below are proposals, not measured safe operating limits or new authority.
## 1. Existing pieces and reuse
Code citations in this section mean **commit `6c75887b0`**, even when a later branch
changes the line. Inspect with `git show 6c75887b0:PATH | nl -ba`. The gate candidate
is explicitly cited at `d2de56bd2` instead. Plans establish intent, not running behavior.
| Ref | Evidence on the pinned tree | Manager boundary |
| --- | --- | --- |
| F1 Bus | WAL SQLite, event kinds and launch events: `packages/bus/schema.sql:1-30`. Broker records launch identity and checks repeat records: `packages/bus/src/broker.mjs:533-586`. | Keep the bus as authority for managed identity, claims and permissions. Add reviewed schema migrations for request records/events, not a second task bus. |
| F2 Queue | Development row states and privileged seats: `packages/queue/src/queue.mjs:13-18`. Canonical checkout/branch restrictions: `packages/queue/src/store.mjs:61-91`. | Keep Sage’s development queue intact. Runtime scheduling needs a separate request type and state machine, not automatic moves of development rows. |
| F3 Runs | Run document reader: `packages/runs/src/runs.mjs:5-6,32-71`. Retry creates a new run from the snapshots with `retriedFrom`: `scripts/mosaic-task.mjs:528-580`. | Reference original evidence and produce a fresh attempt. The retry command is a replay mechanism, not proof that replay is safe. |
| F4 Harness | Runner signals its child group with TERM then KILL: `packages/harness/src/runner.mjs:280-303`. Usage begin/end surrounds a turn, including failed turns: `packages/harness/src/runner.mjs:433-444`. | Admission belongs before turn dispatch. Preserve usage collection on failure. Do not add another harness execution loop. |
| F5 Conversation | Interrupt fences, clears and aborts through the controller, then requires settled state/proofs: `packages/conversation/src/controller.mjs:1267-1495`. Force-stop confirmation shape: `packages/conversation/src/controller.mjs:100-111`. | Request the existing controller operation with its existing capability and confirmation requirements. A scheduling policy cannot synthesize human confirmation. |
| F6 Cohort/shim | Scope identity validation, TERM, freeze, enumeration and cgroup kill: `packages/conversation/src/cohort.mjs:152-200`. Process-group fallback returns unavailable: `packages/conversation/src/cohort.mjs:156-171`. Shim implements those operations: `packages/conversation/src/shim.mjs:130-167`. | Reuse stop evidence. Never free capacity on signal delivery alone. Preserve uncertain states and quarantine. |
| F7 Seat | Launch/run paths and append log: `packages/seat/src/session.mjs:43-49,107-139`. Registry liveness checks PID start time and stop verifies it: `packages/seat/src/session.mjs:230-269`. | Register ownership before launch and use instance/run identity. Runner exit alone does not prove all external work ended. |
| F8 Board/Console | Board rescans and has reply/seen writes: `packages/control-board/src/serve.mjs:230-264`. Console already exposes usage, limits, instances and routes: `packages/webui/src/serve.mjs:60-111`. | Put manager health and request views in the Console. Reuse attention presentation concepts, without creating new Board writer endpoints. |
| F9 Limits/usage | Launch and post-turn limits: `packages/bus/src/limits.mjs:1-5`. Usage rows are append-only token counts with known/unknown status, no money column: `packages/bus/schema.sql:339-365`. | Limits remain hard constraints. Decision-model USD accounting needs a reviewed extension, not pretending existing token rows contain cost. |
| F10 Routes | Launch fallback runs only after primary preflight fails: `packages/cli/src/launcher.mjs:256-295`. | Preserve that behavior. Runtime model/provider rerouting is new, separately reviewed work. |
**F11 Kill switch and lifecycle.** Row 191’s brief requires a persistent, human-only,
per-instance stop covering launches and running work (`docs/plans/2026-10-20_kill-switch.md:12-23,37-47`).
Row 194’s brief requires per-agent vetoes, RBAC and kill-switch checks
(`docs/plans/2026-10-20_agent-lifecycle.md:27-35`). At this base, launch revoke/restore
are human-only broker operations (`packages/bus/src/broker.mjs:3189-3203`). Treat the
briefs as integration contracts, not evidence those whole features landed. The manager
must defer to their final reviewed implementations, including credential revocation.
**F12 Gate lanes are a separate owner.** On **`d2de56bd2`**, grants, private roots,
port bands, quarantined receipts and image locking are described at
`scripts/testing/README.md:3-9,21-49,59-71`. Implementation scans unmanaged test,
Docker and mutant command shapes, not arbitrary heavy work
(`scripts/testing/lane-state.mjs:48-76`, especially `:64-67`). The scan contains no
load measurement. Reuse its lease/status/recovery interface after adoption, never
replace its locks. The manager’s grant for capacity cannot grant Sage’s test slot.
**F13 Failure context.** The brief reports proof 4 failed K12 during paired gates.
The pinned K12 is the fork/freeze/enumeration test
(`packages/conversation/tests/cohort.test.mjs:263-276`). This does not establish host
saturation or prove load caused the defect. Vikunja 450 fixes K12 and 451 adds the
separate gate load refusal. Neither waits for this manager or becomes a retry workaround.
**F14 CLIProxyAPI.** Jason’s target includes built-in CLIProxyAPI with WebUI
(`/mnt/storage/src/jason-master-restart/TARGET.md:24`, owner requirement, not a Stack
implementation claim). The public [upstream configuration](https://github.com/router-for-me/CLIProxyAPI/blob/main/config.example.yaml)
(viewed 2026-10-21, mutable upstream) has routing retries, credential cooldown and
stream bootstrap retries. It does not establish the installed version’s configuration.
Keep proxy credential selection in the proxy. Stack owns host admission and request
identity. Before enabling runtime retries, verify the deployed proxy contract using
stand-ins and coordinate retry budgets. Do not read its management key or consume
management usage feeds. Do not assume its chat-compatible endpoint supports Decisions.
## 2. Ownership and durable requests
**D1 Placement.** Implement a mechanical host supervisor component, outside agent
turns, with a short asynchronous scheduler loop. The bus remains the sole writer of
its runtime request journal through a reviewed migration. No broker transaction spans
a model/network call. The supervisor reserves host capacity and calls existing owners
for launch, interrupt and stop. It cannot change roles, secrets, lifecycle vetoes or
kill-switch state. The emergency stop path must work without the scheduler, model,
provider or a healthy work backlog. Bound work-handler concurrency and keep control
handling outside that pool. This is software scheduling, not hardware/real-time priority.
**D2 Record.** Persist immutable request ID, caller/business/instance, operation,
content reference plus digest (content stays at its authorized source), policy revision,
class, enqueue time/deadline, resource estimate, allowed routes and idempotency key.
Append transitions: `held → ready → leased → running → succeeded|failed|cancelled`,
with branches to `retry-wait`, `dead-letter`, or `uncertain`. Record every attempt,
route, reason code, owner identity, heartbeat and evidence references. Materialized
state is rebuildable. Re-check permission, kill state and policy at dispatch/replay.
A duplicate key with the same digest returns the original request. A changed digest
refuses. Terminal receipts and existing runs are never rewritten.
Use one dispatcher lease per instance with generation fencing. A launch intent is
durable before spawn. After a crash or expired lease, reconcile registry/cohort and
receipts before replacement. An expired heartbeat is not evidence of death. Ambiguous
spawn/effect outcomes enter `uncertain` and continue to reserve capacity. A killed
instance remains held after restart until the authorized clear and health checks.
**D3 Scope.** Host pressure includes unmanaged work, even when the manager cannot stop
it. Owned resource envelopes are per instance. Multiple instances must not each assume
the whole host is free: use statically partitioned host budgets initially. Shared
cross-UID arbitration needs a separately approved host-service boundary. No process
name scan grants cancellation authority over another user or instance. Configuration
extends the instance’s one system config through the normal reviewed schema/bootstrap
path, never a second configuration file or automatic rewrite of an existing config.
## 3. Classes and interrupt semantics
| Ref/class | Requests | Admission and preemption |
| --- | --- | --- |
| P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. |
| P1 system | Lightweight health reads, lease reconciliation, approved recovery | Reserved control capacity. Health/reconciliation can run under pressure. Recovery that launches heavy work still needs capacity and kill-clear. Never auto-clear a kill. |
| P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. |
| P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. |
| Work | What interrupt means | Hard-stop boundary |
| --- | --- | --- |
| Agent turn | Fence new dispatch, ask its controller to abort, reconcile queued input and effects (F5). | Escalate only through an already-authorized force stop/kill path. Uncertain proofs retain the reservation (F6). No automatic reissue of the prompt. |
| Test gate | Ask its gate owner to cancel at a step boundary, record interrupted attempt, run owned cleanup. | A safety stop may terminate an owned cohort. Incomplete cleanup quarantines the lane. No green receipt, automatic lane unlock or retry-to-green. |
| Build | Cancel a registered subprocess/cohort at a declared safe point. Discard incomplete disposable output. | Publication, image tagging, package installation or release activation may have effects. No load-driven KILL there. Safety stop can still terminate, with reconciliation required. |
| Long tool call | Propagate cancellation only where the tool supports it. A transport timeout records unknown outcome. | Remote writes cannot be undone by killing the local caller. Reconcile provider operation ID/status. No blind reroute, retry or claim of rollback. |
Priority never overrides a permission refusal or a hard resource limit. No SIGSTOP
pausing by default: it can retain locks and memory while blocking the urgent work.
Reserve control-plane headroom through concurrency limits and separately bounded
process resources. Validate responsiveness under pressure before claiming an SLA.
## 4. Admission, cooldown and fairness
**D4 Proposed initial signals**, sampled every 2 seconds. Use effective CPU count and
memory limits (cpuset/cgroup as well as host), `/proc/loadavg`, `MemAvailable`, cgroup
memory current/max/events, swap activity, and CPU/memory/I/O PSI. Observe known work
reservations, disk headroom, provider concurrency and response metadata (429,
`Retry-After`, reset time, timeouts, 5xx). Do not scrape command lines or environments
into telemetry. A missing/stale required signal (>6 seconds) holds new heavy work
with `health-unknown`, while stop and bounded local diagnostics remain available.
| Signal | Hold new heavy work | Release condition |
| --- | --- | --- |
| One-minute load / effective CPUs | ≥0.80 for 3 samples | ≤0.55 |
| Available effective memory | <15% or <1 GiB, whichever demands more free memory | >25% and >2 GiB |
| PSI avg10, percentages | CPU some ≥20, memory full ≥2, or I/O full ≥10, for 3 samples | CPU some <10, memory full <0.5, I/O full <3 |
| Critical pressure | New OOM event, memory <5%, or free manager-volume space <1 GiB | Reconciliation plus healthy thresholds, no automatic kill-clear |
| Provider scope | 429 or exhausted declared quota | Retry-After/reset elapsed and one successful permitted probe |
Release only after **all applicable** low thresholds hold for 30 seconds. An admission
uses both measurements and outstanding reservations, recorded atomically before
launch. A request larger than its configured envelope refuses or waits for an
explicitly supported exclusive envelope, never an unbounded resource guess. Start with one heavy dispatch per instance and one release every 5 seconds,
then measure before raising concurrency. Provider cooldown blocks only affected
provider/account/model scopes, not all host work. An open provider circuit waits at
least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on
failure. Never shorten an upstream wait to fit a local retry delay.
These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or
Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and
pressure, with reviewed config, not model output. Reject contradictory settings.
**D5 Held work.** Return an accepted receipt and visible hold reason/next evaluation,
not a blocking connection. Default queue bound: 1,000 outstanding requests per instance,
100 per caller, envelope ≤16 KiB. Excess receives an explicit refusal with retry advice.
P0 uses a separately bounded, target-deduplicated control path. Journal failure refuses
new work and alerts, without disabling the existing emergency stop path.
Normal requests expire after 30 minutes waiting, background after 24 hours, unless an
explicit authorized deadline differs. Show expiry and retain its receipt. Once healthy,
weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant.
After 10 minutes, background ages into normal scheduling rank, never into system
priority and never through pressure holds. After control bursts, service eligible work.
Rate-limit/coalesce health reads. Continuous real emergencies can still prevent progress:
alert on oldest wait >10 minutes rather than promise starvation freedom under overload.
## 5. Retries, dead letters and routes
**D6 Retry ownership.** One logical request gets at most **3 total manager attempts**,
10 minutes of retry time and its original deadline, whichever ends first. Retry only
transient failures known to precede effects, or operations with verified idempotency.
Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum.
Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal
retries must fit a declared per-attempt budget. If that budget is unknown, do not layer
automatic manager retries over the proxy. Max one route change within the three attempts.
| Outcome | Action |
| --- | --- |
| Connection refused before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility. |
| Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. |
| Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. |
| Transient retry budget/deadline exhausted, deterministic execution failure | Dead-letter with evidence and actionable reason. Admission expiry is an expired terminal item. |
| Explicit user cancellation | Terminal cancelled. Never retry automatically. |
**D7 DLQ.** A durable request state and append-only disposition history, not copied
prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted
error code, idempotency/effect status and links to authorized evidence in Console
“System → Requests → Dead letters”. Propose equivalent `mosaic system requests
show|list` and `mosaic system replay REQUEST --expect-revision REV` verbs. These are
proposed interfaces, not available commands. Mos sees permitted items and recommends
replay. Jason or an explicitly delegated role can replay after cause correction and
reconciliation. Replay creates a new request linked to its parent, with a fresh bounded
budget and current policy. No mass replay, unattended replay chains or resurrected
cancelled work. Unknown effects require explicit reconciliation evidence before replay.
**D8 Rerouting.** Candidate routes come only from the user’s configured allowlist,
filtered for model capability, tool protocol, context, privacy/region, account budget
and role constraints. Prefer same model/provider recovery, then a configured equivalent
provider, then a pre-approved alternative model. Never downgrade required capabilities
or migrate a live transcript silently. Mid-turn failure is reconciled before a fresh
attempt. Record requested and effective route and reason. New provider consent/spend
is separate from enabling the decision hook. F10’s preflight fallback remains distinct.
## 6. Monitoring and Console
**D9 Measurements.** Pressure samples and freshness, active/reserved work, ownership
uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency,
retry counts, DLQ growth, route failures, provider cooldown/reset, stop proof state,
manager heartbeat and decision-model timeouts/spend/unknown charges. Store bounded
aggregate samples for 7 days, without request bodies. Request receipts/evidence follow
the instance’s retention policy, with the existing write-once run invariant preserved.
Console “System” shows healthy/holding/unknown, last sample time, kill state, capacity,
queue counts and oldest wait, plus provider and decision-model status. Drill down to
Requests, Dead letters and evidence. Reuse Console read transport, auth and scoped
human writes (F8). No new public listener. Show stale/unknown explicitly, not green.
Alert immediately on OOM, uncertain ownership/stop, journal failure, dead letters and
unknown decision charge. Warn on pressure hold >60 seconds, control heartbeat stale
>6 seconds, or oldest eligible wait >10 minutes. Deduplicate by reason/target and emit
one recovery notification. Use Console attention/trail first. External notification
channels require their own configured authorization. A lightweight supervisor watchdog
reports manager death through an independent health path, not the dead manager itself.
## 7. Optional decision-model hook
**D10 Interface.** `decide({version, snapshotId, policyRevision, deadline,
allowedActions, state}, signal) → {actionId, confidence?, usage:{cost,currency},
providerRequestId}`. `allowedActions` contains opaque IDs for already-eligible routes
or “use rules”. Adapters normalize provider types. A deterministic rules adapter is
the default and needs no network or key. A generic structured-output adapter can support
other decision models without exposing their wire format to the scheduler.
Enabling paid assistance is a user setting, recommended in setup with its price/cap
shown. Pin **`typesafe/jev-1.14`** as the recommended initial model, never a moving
“latest” alias. Its [OpenRouter guide](https://openrouter.ai/docs/guides/community/jev)
documents typed choice/noul/score answers with probabilities and
`POST https://openrouter.ai/api/alpha/decisions`. The [model page](https://openrouter.ai/typesafe/jev-1.14)
lists $0.042 per million input tokens, free output and 262,144 context tokens
(verified public documentation 2026-10-21, not a live call or price guarantee).
Use choice for route selection. Do not send images or use the large context window.
Other decision models need their own adapter contract and calibrated confidence tests.
Hard timeout **500 ms**, max one in-flight decision per instance, no decision-call
retry. Timeout, invalid answer, stale snapshot, low confidence (<0.8 initially), cost
uncertainty or network failure immediately selects rules. Discard late answers for
routing, but still account for their charge. The hook cannot authorize work, choose
P0/P1 status, change thresholds, clear a kill, replay effects, or choose outside the
eligible set. Recheck eligibility after it returns. Never call it for safety actions.
**D11 Cost.** Proposed opt-in defaults: **$0.10/day and $1/month**, both enforced.
Use integer currency units/decimal parsing, never binary-float threshold comparisons.
Before send, durably reserve an upper bound from a pinned price ceiling and a bounded
serialized request (≤4 KiB, verified worst-case tokens and all billable fields). Count
concurrent reservations in both caps. If the adapter cannot bound cost, use rules.
Reconcile once per provider request ID using **`usage.cost`** in USD, as documented in
the [Jev guide](https://openrouter.ai/docs/guides/community/jev). Keep reservations for
timeouts/crashes until settled. Missing/invalid cost, price above ceiling, or a charge
above reservation disables paid decisions and alerts. Never interpret absent cost as
zero. A provider-enforced spend limit is also required for an approved live trial:
local accounting alone cannot guarantee a hard ceiling against upstream misbilling.
Day/month windows use UTC. No cap rollover drops an unresolved reservation. Configure caps before enabling.
**D12 Data and key.** Send only enum work type/class, numeric resource estimates,
coarse queue age/pressure, redacted error category, capability tags and opaque eligible
route IDs. No credentials, auth headers, environment, command lines, paths, transcript,
raw errors, filenames or user prompt. If a judgment requires user content, first try
local metadata extraction. Any later minimal content field needs explicit approval
and a tested schema/redaction rule. Log sanitized decision inputs/output IDs and cost,
never wire headers. Model output is untrusted data.
Proposed slot reference only: `<instance-config-directory>/secrets/openrouter-decision.env`,
name **`OPENROUTER_API_KEY`**. This is not an existing slot claim. Jason places the value
only after live-trial approval. Runtime mounts/injects it read-only into the decision
adapter, never the model state or worker. No management key access. A direct Decisions
adapter may be needed unless a stand-in proves the approved proxy supports this API.
## 8. Proposed queue rows, smallest first
Sage assigns actual row IDs, Gitea issues, owners, reviewers and gate grants. These
are proposals under Vikunja 452, not edits to either queue file. No implementation is
authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/authority.
| Ref/order | Slice and dependency | Acceptance evidence | Before 10-27? |
| --- | --- | --- | --- |
| S1 | Pressure parser and pure rules evaluator, no live enforcement | Synthetic load/memory/PSI/missing-signal cases, exact hold/release thresholds, fake-clock hysteresis, effective CPU/memory limit cases. No builds or real pressure generation. | Candidate could land if Sage has spare review/gate capacity. Never displace MVP. |
| S2 | Read-only Console System status, after S1 | Fixture HTTP responses for healthy/held/stale/unknown and auth scoping. No writer, no secret or command-line fields. | Could land with spare capacity only. |
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
| S5 | Retry/DLQ/replay and permitted reroutes, after S3/S4 | Fake provider failures with exact total-attempt/time bounds, restart during backoff, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal. | After MVP. |
| S6 | Decision hook, USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/low-confidence answers, unapproved action, snapshot race, redaction canaries, concurrent cap reservations, missing/duplicate cost, crash and day rollover. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
Each slice gets independent review and tests that kill a mutation of its central
invariant. Sage chooses package/full-suite gates appropriate to the final diff.
Do not run them alongside an MVP gate without a grant. Before production activation,
measure emergency control responsiveness and recovery with disposable cohorts under
bounded synthetic pressure. No load experiment on the shared development host.
## 9. Questions for Jason (recommendations, not design blockers)
- **Q1 Initial authority:** should the manager ever force-stop background work just
for load? **Recommend no.** Cooperative yield for load, hard stop only through an
already-authorized safety/explicit stop. Broader authority needs a separate decision.
- **Q2 Trial spend:** accept $0.10/day, $1/month and a provider-side ceiling for the
first trial? **Recommend yes**, with rules staying default until Jason enables it
and installs the slot. Q199 provides readiness, not approval to spend now.
- **Q3 Reroute consent:** may setup authorize a named alternative model/provider list?
**Recommend yes**, per role and privacy requirement. Empty list means hold/fail
visibly instead of silently changing providers or model quality.
## Verification and handoff
This document was checked against the pinned source and public documentation only.
No build, suite, live bus write, service action, proxy inspection or paid call was
performed. The canonical `scripts/mosaic queue next huey` returned `nothing`.
The initial linked-worktree refusal was diagnosed, not bypassed. The direct mos-1183
assignment governs this branch. Gitea issue creation and implementation-row entry
remain with Sage: this brief forbids credential-file reads and queue writes.