@@ -5177,3 +5177,17 @@ Before: managed project/thread verbs had no Console conversation area. After: fi
2026-10-21T01:28:05Z | Darkwing | row 232 #1734 installer I1 build, round 1 in review | Before: stack-release.sh managed only user units (systemctl --user), and stack-copy.sh had no bundle copy. After: --units external (adopt, activate, rollback --to, status, confirm, revert, gate --since, AWAITING-UNITS, stale-start, external-pm, --plan --expect-from) and stack-copy check --units external, copy --bundle FILE SHA, wrapper ID --create; user mode byte-identical to the base by test. First build 0af0c73c6 on 00d4cc1f6; rebased onto row 200 (cb5fdb064) as a0ffabc3f on darkwing/row232-r1; --sso refused under external (D9), row 200 config_checks run under external (D10). Gate on Sage's slot (sage-1618): cli 561/0, every scripts/test-*.sh green; 17/17 mutants killed. Results commit 84f1a91e6 (pushed, seat branch only). Manifest 7a3b4ea1 (9 paths), queue revs 2595-2596, request comment 29825, reviewer Filbert. Packet agents/darkwing/work/queue-232/BUILD.md.
2026-10-21T04:50:44Z | Darkwing | row 233 #1735 installer I3 build, round 1 in review | Before: root had no upgrade driver for root-owned units and no host run sheet; stack-release.sh had no rehearse --backup last and no external plan refusals. After: scripts/stack-upgrade.sh upgrade and rollback (r6 A1-A5, A2f, R1-R4) with 20 tests, docs/guides/host-run-sheet.md, TOOLS.md lines, and on Sage's rulings stack-release.sh --backup last, BACKUP_STORE and the --plan refusals open-activation and migrate-required (3 tests). Base e65a3ab4d (row 232's landing). Gate on Sage's slot (sage-1704) on d99166ade: cli 616/0, every scripts/test-*.sh green, 1201/0 summed by awk; m1-m8 8/8 killed. Correction: the process sweep after the gate found an orphaned sleep 60 from the lock test's flock holder (N1); 08db6ac3f fixes it test only, Sage ruled the gate stands (sage-1707), rerun c0 20/0 and m1 killed with 0 left (sage-1708). Tip 1a1dc80ca (pushed, seat branch only). Manifest 464ce9eb (7 paths), queue revs 2644 and 2647, request comment 29901, reviewer Filbert. Packet agents/darkwing/work/queue-233/BUILD.md.
2026-10-21T06:37:41Z | Darkwing | row 240 #1746 installer I4 build, round 1 in review | Before: root had no host check for the bwrap AppArmor profile, its pins or a boot unit, and stack-upgrade.sh exited 1 args-failed when the args move after A5 failed, while exit 1 means the old copy runs. After: scripts/case1-hostcheck.sh (seven items, --check, --enforce, --upstream), scripts/case1-pins.sh, mosaic-hostcheck.service and 16 tests; the args move failure exits 5 args-failed, documented in TOOLS.md and run sheet section 6 and tested; run sheet section 3 puts the pinned npm on PATH and names the bin/claude.exe --claude path; sections 8 and 10 have precheck lines. Verification: slot run r1 on 47ec9fe2f failed at the singles (the new N1 test's after hook threw ENOENT after the fixture cleanup removed the root); b45852daa drops the hook; r2 on b45852daa green: singles 16/0 and 21/0, c0 plus M1-M8 8/8 killed, cli 639/0, scripts 585/0. Candidate darkwing/row240-r3 ad1b23ab2 on c771a36f4, manifest 573ece95 (9 paths), revs 2694 and 2695, comment 29943.
2026-10-21T16:34:44Z | Huey | Q198/Q199 system-manager design, mos-1183, Vikunja 452 | Before: pinned foundation 6c75887b0 has separate bus, development queue, run evidence, turn/cohort controls and launch/usage paths; the assignment requests a mechanical manager design. After: docs/plans/2026-10-21_q198-system-manager.md proposes admission/cooldown, priority and safe interruption, durable requests, bounded retry/DLQ/replay/rerouting, Console monitoring, optional decision adapters with USD reservations, six ordered slices and three recommended owner decisions. Verification: 26 pinned file:line references resolve, ten-line summary, manual source/contract review, git diff --check and installed queue guard pass. No builds/suites, runtime writes, credential reads, service changes or paid calls. Huey queue next refused the linked worktree, then returned nothing from canonical checkout; direct brief controls this assignment. Gitea issue and queue-row capture are handed to Sage under the no-credential-read/no-queue-write boundary.
2026-10-21T18:45:47Z | Huey | Q198 system-manager design r2, mos-1194, Vikunja 452 | Before: Scrooge requested changes on add7cb506 for decision-key worker exposure, cap scope and absent sampler ownership. After: distinct MOSAIC_DECISION_OPENROUTER_KEY with S6 sessionEnv canary requirement; exclusive per-instance key with per-key caps and provider limit on every enablement; S2 owns supervisor sampler and waits until after MVP/re-pin. Also records P1 interpretation, missing-confidence fallback and stand-in timeout selection, explains verified Noul spelling, excludes notifier/gateway retry loops with citations, and cites lifecycle revocation requirement. Verification: 30 pinned citation spans resolve, ten-line summary, manual diff/source review and whitespace/queue-guard checks. No builds, suites, live changes, credential reads or paid calls.
2026-10-21T19:56:11Z | Huey | Q198 design r3, mos-1201/1201a, Vikunja 452 | Before: Scrooge approved r2 7e785a08f with three notes. After: N1 names proposed system.health bus read verb and includes broker handler/authorization in S2 scope and review; N2 labels exclusive ownership as a setup attestation that cannot be mechanically proved and identifies provider key limit as real ceiling; N3 adds recommended Q4 for P1 narrowing and removes the Mos-flag instruction. Other design content retained. Verification: manual r2-to-r3 delta review, ten-line summary, whitespace and installed queue-guard checks. No builds, suites, live changes or paid calls.
2026-10-21T20:19:53Z | Huey | Q198 design r4, mos-1203, DECISIONS Q204, Vikunja 452 | Before: r3 a36952564 presented Q1–Q4 for owner approval. After: section 9 records Q20a–Q20c verbatim as approved; D11 labels Huey’s reading of configurable daily/monthly caps with $0.10/$1 defaults in the one instance config via reviewed schema; provider key ceiling and explicit enable/key requirements retained; Q4 pending, Jason restated 2026-10-21, P1 unchanged. Verification: quotes match DECISIONS ledger, unrelated sections and P1 equal r3, ten-line summary, whitespace and installed queue guard checked. No builds, suites, live changes or paid calls.
2026-10-21T20:48:49Z | Huey | Q198 design r5, mos-1206, DECISIONS Q205/Q206, Vikunja 452 | Before: r4 1e8bccff1 left Q4 pending and long outages could exhaust request retry budgets. After: protected mode separate from unchanged D1 human kill authority; mechanical detection, safe-point pause/proof, JEV-first endpoint-before-headless probes at 60-second increments to 600 seconds without attempt limit, upstream reset waits, shared D11 probe accounting, one test session then batches, proposed three-failed-exit escalation, frozen outage clocks/counters without DLQ, advisory online-service routing, Q4 superseded and Q22 pending. Thresholds/deadlines/batches/escalation explicitly labelled Huey readings. Acceptance proposals updated, no implementation. Verification: complete Q205/Q206 read, quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned citation references resolve, 11-line summary, manual consistency/diff review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes or paid calls.
2026-10-21T21:04:37Z | Huey | Q198 design r6, mos-1208/mos-1210, DECISIONS Q207, scrooge-0003, Vikunja 452 | Before: r5 fd44f7c60 gated probes on JEV and lacked request-local resume failure separation. After: parallel independent probes; central JEV/smaller-model/rules endpoint chain, per-model timeout/calibration, 4 KiB metadata limit and isolated fallback credentials; subscription quota vs metered USD accounting and separately enabled paid-probe budget (proposed $0 defaults). S-F1–S-F7: bounded outage classification, request-local D6/two-resume DLQ rule and batch definition; escalation latch with authorized revision-bound system.recovery.release separate from kill clear; cooperative F5 pause/F6 proof only; proposed configurable 24-hour cumulative pause expiry; full Q205 kill sentence with labelled protected-mode interpretation; r3–r6 history. S5/S6 acceptance proposals updated. Verification: Q205/Q207 quotes match ledger, D1/inventory/Q1–Q3 unchanged, 30 pinned references resolve, summary within 15 lines, manual consistency/delta review, whitespace and installed queue guard checked. No builds, suites, real probes, live changes, credential reads or paid calls.
2026-10-21T21:39:25Z | Huey | Q198 r6.1 touch-up, mos-1222, scrooge-0004 N1–N3, Vikunja 452 | Before: approved r6 2ee68cd69 had unprovable session-call denial wording, implicit queue/late-call behavior and a partial Q207 paragraph. After: rule/S6 tests require session decision calls/output have no routing authority; non-waiting dispatchers use internal rules immediately; timed-out JEV does not occupy fallback decision slot but retains spend reservation; full Q207 paragraph copied exactly from DECISIONS; matching Q22 reading corrected. ENV_ALLOW and all runtime code unchanged. Verification: exact full-quote equality, obsolete wording absent, D1 unchanged, summary within 15 lines, manual N1–N3 delta review, whitespace and installed queue guard checked. No builds, suites, probes, live changes or paid calls. Review/handoff via T3 as Mos authorized.
@@ -1419,3 +1419,10 @@ are never rewritten or removed; corrections are new entries.
2026-10-21T06:47:54Z | researcher | row 228 (#1729) round 3 in review (rev 2702; start rev 2700) | rebase of r2 onto the 234 landing c771a36f4 as researcher/row228-r3 cf18c42b9 (one commit, pushed) with Webby's round 2 N1 (:711 and :718); manifest 82cf98c6 (3 files); slot sage-1796: session.test 13 + host.test 42 = 55/0, manifest OK before and after, busy228.sh copy named in the clear; r2 94/0 and M1 killed stand (seat blobs unchanged); request attempt left in requesting state (lock held by sage-238-review-2b when the outcome rev was written, exit 3; transport said failed pre-send, no token), Sage resolves and posts as jarvis; packet agents/researcher/work/queue-228/r3
2026-10-21T07:29:23Z | researcher | row 228 (#1729) landed as 86249ac55 (pushed 07:25Z by Sage) | patch-id b6ffe31feb0d matches my r3 candidate cf18c42b9 (checked here against origin/refactor); Sage combined gate with 238: seat 165/0, cli 640/0, scripts 585/0; queue note rev 2725, done rev 2727, #1729 closed comment 29966 (sage-1824); q228r2 worktree removed, branches researcher/row228, -r2 and -r3 stay on origin and locally; packets agents/researcher/work/queue-228 r1 r2 r3 uncommitted
2026-10-21T15:19:14Z | Sage (Opus 5.5, 1ef1e4f8) | MVP push 10-21 afternoon: rows 249, 197 proof 3/4, 222 singles, 66 r4d, queue-commit | 249 landed 9de5a57ab (+7b9737953 DEFERRED N1, #1751 closed, Vikunja 445 c442); 197 proof 3 rc=1 on a driver defect, Herb's fix a5b636750 to Webby, proof 4 set up in sage-r197p4; 222 singles 34/34 and 5 mutants killed (sage-2028); 66 round 4 in review (82c830779, c30083/30084); queue revs 2642-2904 committed 0636721e6
2026-10-21T16:34:44Z | Huey (gpt-6.1-sol) | mos-1183, Q198/Q199 system-manager design, Vikunja 452, huey/system-manager-design on 6c75887b0 | design and six proposed slices documented; pinned citations and docs checks pass; no build, suite, live action or paid call; Gitea/queue capture remains Sage-owned
2026-10-21T18:45:47Z | Huey (gpt-6.1-sol) | mos-1194, Q198 design r2, scrooge-0001 findings 1–8, Vikunja 452 | all findings addressed in design and slice acceptance criteria; 30 citation references checked, no suites/live changes/paid calls; Mos routes r2 back to Scrooge
2026-10-21T19:56:11Z | Huey (gpt-6.1-sol) | mos-1201/1201a, Q198 design r3, scrooge-0002 N1–N3 | three doc notes applied; Q4 ready for Mos’s combined owner approval question; no builds, suites, live changes or paid calls
Revision **r6.1**, mos-1222, applies scrooge-0004 N1–N3 to approved r6 `2ee68cd69`.
Q22 is resolved. New implementation choices are Huey’s readings.
All defaults below are proposals, not measured safe operating limits or new authority.
## 1. Existing pieces and reuse
Code citations throughout this document mean **commit `6c75887b0`**, even when a later branch
changes the line. Inspect with `git show 6c75887b0:PATH | nl -ba`. The gate candidate
is explicitly cited at `d2de56bd2` instead. Plans establish intent, not running behavior.
| Ref | Evidence on the pinned tree | Manager boundary |
| --- | --- | --- |
| F1 Bus | WAL SQLite, event kinds and launch events: `packages/bus/schema.sql:1-30`. Broker records launch identity and checks repeat records: `packages/bus/src/broker.mjs:533-586`. | Keep the bus as authority for managed identity, claims and permissions. Add reviewed schema migrations for request records/events, not a second task bus. |
| F2 Queue | Development row states and privileged seats: `packages/queue/src/queue.mjs:13-18`. Canonical checkout/branch restrictions: `packages/queue/src/store.mjs:61-91`. | Keep Sage’s development queue intact. Runtime scheduling needs a separate request type and state machine, not automatic moves of development rows. |
| F3 Runs | Run document reader: `packages/runs/src/runs.mjs:5-6,32-71`. Retry creates a new run from the snapshots with `retriedFrom`: `scripts/mosaic-task.mjs:528-580`. | Reference original evidence and produce a fresh attempt. The retry command is a replay mechanism, not proof that replay is safe. |
| F4 Harness | Runner signals its child group with TERM then KILL: `packages/harness/src/runner.mjs:280-303`. Usage begin/end surrounds a turn, including failed turns: `packages/harness/src/runner.mjs:433-444`. | Admission belongs before turn dispatch. Preserve usage collection on failure. Do not add another harness execution loop. |
| F5 Conversation | Interrupt fences, clears and aborts through the controller, then requires settled state/proofs: `packages/conversation/src/controller.mjs:1267-1495`. Force-stop confirmation shape: `packages/conversation/src/controller.mjs:100-111`. | Request the existing controller operation with its existing capability and confirmation requirements. A scheduling policy cannot synthesize human confirmation. |
| F6 Cohort/shim | Scope identity validation, TERM, freeze, enumeration and cgroup kill: `packages/conversation/src/cohort.mjs:152-200`. Process-group fallback returns unavailable: `packages/conversation/src/cohort.mjs:156-171`. Shim implements those operations: `packages/conversation/src/shim.mjs:130-167`. | Reuse stop evidence. Never free capacity on signal delivery alone. Preserve uncertain states and quarantine. |
| F7 Seat | Launch/run paths and append log: `packages/seat/src/session.mjs:43-49,107-139`. Registry liveness checks PID start time and stop verifies it: `packages/seat/src/session.mjs:230-269`. | Register ownership before launch and use instance/run identity. Runner exit alone does not prove all external work ended. |
| F8 Board/Console | Board rescans and has reply/seen writes: `packages/control-board/src/serve.mjs:230-264`. Console already exposes usage, limits, instances and routes: `packages/webui/src/serve.mjs:60-111`. | Put manager health and request views in the Console. Reuse attention presentation concepts, without creating new Board writer endpoints. |
| F9 Limits/usage | Launch and post-turn limits: `packages/bus/src/limits.mjs:1-5`. Usage rows are append-only token counts with known/unknown status, no money column: `packages/bus/schema.sql:339-365`. | Limits remain hard constraints. Decision-model USD accounting needs a reviewed extension, not pretending existing token rows contain cost. |
| F10 Routes | Launch fallback runs only after primary preflight fails: `packages/cli/src/launcher.mjs:256-295`. | Preserve that behavior. Runtime model/provider rerouting is new, separately reviewed work. |
**F11 Kill switch and lifecycle.** Row 191’s brief requires a persistent, human-only,
per-instance stop covering launches and running work (`docs/plans/2026-10-20_kill-switch.md:12-23,37-47`).
Row 194’s brief requires per-agent vetoes, RBAC and kill-switch checks
(`docs/plans/2026-10-20_agent-lifecycle.md:27-35`). At this base, launch revoke/restore
are human-only broker operations (`packages/bus/src/broker.mjs:3189-3203`). Treat the
briefs as integration contracts, not evidence those whole features landed. The manager
must defer to their final reviewed implementations, including the brief’s credential
with branches to `retry-wait`, `outage-paused`, `dead-letter`, or `uncertain`. Record every attempt,
route, reason code, owner identity, heartbeat and evidence references. Materialized
state is rebuildable. Re-check permission, kill state and policy at dispatch/replay.
A duplicate key with the same digest returns the original request. A changed digest
refuses. Terminal receipts and existing runs are never rewritten.
Use one dispatcher lease per instance with generation fencing. A launch intent is
durable before spawn. After a crash or expired lease, reconcile registry/cohort and
receipts before replacement. An expired heartbeat is not evidence of death. Ambiguous
spawn/effect outcomes enter `uncertain` and continue to reserve capacity. A killed
instance remains held after restart until the authorized clear and health checks.
**D3 Scope.** Host pressure includes unmanaged work, even when the manager cannot stop
it. Owned resource envelopes are per instance. Multiple instances must not each assume
the whole host is free: use statically partitioned host budgets initially. Shared
cross-UID arbitration needs a separately approved host-service boundary. No process
name scan grants cancellation authority over another user or instance. Configuration
extends the instance’s one system config through the normal reviewed schema/bootstrap
path, never a second configuration file or automatic rewrite of an existing config.
Decision spending is separately scoped per exclusive key, not per host or shared
account. When JEV is enabled, D11 requires its dedicated key per instance. Other
endpoints use their isolated credential/billing contracts in D11/D12. Shared keys are unsupported
until a shared ledger and its authority boundary receive separate review.
## 3. Classes and interrupt semantics
| Ref/class | Requests | Admission and preemption |
| --- | --- | --- |
| P0 safety | Authorized kill switch, explicit stop/cancel, ownership fencing | Bypass backlog and load hold. Local mechanical path, no model. Deduplicate per target. Authorization is checked before accepting priority. |
| P1 system | Lightweight health reads, lease reconciliation, automatic outage recovery under Q205/Q206 | Reserved control capacity. Health/reconciliation can run under pressure. Resume needs capacity, current policy and a clear human kill switch, without a new approval per session. Never auto-clear a kill. |
| P2 normal | User/agent turns, ordinary task dispatch | FIFO within tenant, fair rotation across tenants. May request cooperative yield from background work. User text saying “urgent” does not confer system authority. |
| P3 background | Builds, indexing, maintenance, optional evaluations, granted test gates | Lowest resource admission priority. A granted gate retains Sage’s independent exclusivity and evidence rules. |
DECISIONS Q205/Q206 supersede the old P1 per-session recovery approval question.
Huey’s reading: the configured recovery policy supplies standing authority to resume
eligible paused work. Current permissions, hard capacity and the human kill switch still
apply. Health/reconciliation retain reserved capacity. Protected mode can clear itself,
but cannot clear a human kill or grant new authority.
| Work | What interrupt means | Hard-stop boundary |
| --- | --- | --- |
| Agent turn | Fence new dispatch, ask its controller to abort, reconcile queued input and effects (F5). | Escalate only through an already-authorized force stop/kill path. Uncertain proofs retain the reservation (F6). No automatic reissue of the prompt. |
| Test gate | Ask its gate owner to cancel at a step boundary, record interrupted attempt, run owned cleanup. | A safety stop may terminate an owned cohort. Incomplete cleanup quarantines the lane. No green receipt, automatic lane unlock or retry-to-green. |
| Build | Cancel a registered subprocess/cohort at a declared safe point. Discard incomplete disposable output. | Publication, image tagging, package installation or release activation may have effects. No load-driven KILL there. Safety stop can still terminate, with reconciliation required. |
| Long tool call | Propagate cancellation only where the tool supports it. A transport timeout records unknown outcome. | Remote writes cannot be undone by killing the local caller. Reconcile provider operation ID/status. No blind reroute, retry or claim of rollback. |
Priority never overrides a permission refusal or a hard resource limit. No SIGSTOP
pausing by default: it can retain locks and memory while blocking the urgent work.
Reserve control-plane headroom through concurrency limits and separately bounded
process resources. Validate responsiveness under pressure before claiming an SLA.
## 4. Admission, cooldown and fairness
**D4 Proposed initial signals**, sampled every 2 seconds. Use effective CPU count and
memory limits (cpuset/cgroup as well as host), `/proc/loadavg`, `MemAvailable`, cgroup
memory current/max/events, swap activity, and CPU/memory/I/O PSI. Observe known work
reservations, disk headroom, provider concurrency and response metadata (429,
`Retry-After`, reset time, timeouts, 5xx). Do not scrape command lines or environments
into telemetry. A missing/stale required signal (>6 seconds) holds new heavy work
with `health-unknown`, while stop and bounded local diagnostics remain available.
| Signal | Hold new heavy work | Release condition |
| Available effective memory | <15% or <1 GiB, whichever demands more free memory | >25% and >2 GiB |
| PSI avg10, percentages | CPU some ≥20, memory full ≥2, or I/O full ≥10, for 3 samples | CPU some <10, memory full <0.5, I/O full <3 |
| Critical pressure | New OOM event, memory <5%, or free manager-volume space <1 GiB | Reconciliation plus healthy thresholds, no automatic kill-clear |
| Provider scope | 429 or exhausted declared quota | Retry-After/reset elapsed and one successful permitted probe |
Release only after **all applicable** low thresholds hold for 30 seconds. An admission
uses both measurements and outstanding reservations, recorded atomically before
launch. A request larger than its configured envelope refuses or waits for an
explicitly supported exclusive envelope, never an unbounded resource guess. Start with one heavy dispatch per instance and one release every 5 seconds,
then measure before raising concurrency. Provider cooldown blocks only affected
provider/account/model scopes, not all host work. An open provider circuit waits at
least 30 seconds (or longer Retry-After), permits one half-open probe, and reopens on
failure. Never shorten an upstream wait to fit a local retry delay. A declared outage
uses D13–D15’s Q206 probe schedule instead of this short transient-failure circuit.
These thresholds do not replace the gate-specific load-under-4 rule in mos-1182 or
Vikunja 451. Adapters must satisfy both controls. Tune from observed queue wait and
pressure, with reviewed config, not model output. Reject contradictory settings.
**D5 Held work.** Return an accepted receipt and visible hold reason/next evaluation,
not a blocking connection. Default queue bound: 1,000 outstanding requests per instance,
100 per caller, envelope ≤16 KiB. Excess receives an explicit refusal with retry advice.
P0 uses a separately bounded, target-deduplicated control path. Journal failure refuses
new work and alerts, without disabling the existing emergency stop path.
Normal requests expire after 30 minutes waiting, background after 24 hours, unless an
explicit authorized deadline differs. Outage-paused time is excluded under D6.
Show expiry and retain its receipt. Once healthy,
weighted service is 4 normal : 1 background among eligible requests, FIFO per tenant.
After 10 minutes, background ages into normal scheduling rank, never into system
priority and never through pressure holds. After control bursts, service eligible work.
Rate-limit/coalesce health reads. Continuous real emergencies can still prevent progress:
alert on oldest wait >10 minutes rather than promise starvation freedom under overload.
## 5. Retries, dead letters and routes
**D6 Retry ownership.** One logical request gets at most **3 total manager attempts**,
10 minutes of retry time and its effective deadline, whichever ends first. Retry only
transient failures known to precede effects, or operations with verified idempotency.
Use full jitter `0..min(60s, 2s × 2^(attempt−1))`, with upstream Retry-After as a minimum.
Re-enter admission on every attempt. Restarts preserve counters/deadlines. Proxy/internal
retries must fit a declared per-attempt budget. If that budget is unknown, do not layer
automatic manager retries over the proxy. Max one route change within the three attempts.
**Outage exception (Q205/Q206).** Work paused by an outage keeps its remaining three-attempt
allowance and remaining ten-minute retry window. Neither resets nor advances during the
pause, including the outage-causing failure once classified as an outage. It never enters
the DLQ because of that outage. Huey’s implementation reading: correlate a candidate
outage for at most 30 seconds, then settle each failure as either an evidenced D13
outage or a request-local failure. A lone bad request cannot freeze itself indefinitely.
Resume continues the same logical request, not D7 replay. After a successful test session,
request-local failures in resumed work consume that request’s D6 attempts/time normally.
Persist a separate count of request-local failures across distinct resume episodes.
A request failing in two separate resumes goes to D7 with both episodes’ evidence,
even if its three-attempt allowance is not exhausted. Correlated outage failures do
not count toward that two-resume rule. These classification/counting details are Huey’s
reading. They prevent one defective request from repeatedly claiming an outage exemption.
Huey’s reading of deadlines: freeze queue-age expiry and caller-set execution-deadline
budgets during the union of outage-paused intervals, never counting overlapping service
outages twice. Keep the original timestamp and publish the effective deadline extended
by paused time. This deliberately interprets a caller-set deadline as an execution budget,
not an immutable wall-clock cutoff. Explain that in the acceptance receipt. Cancellation
still takes effect during a pause. Journal pause intervals and counters across restarts.
An interrupted effect remains uncertain until reconciled, regardless of the clock.
Huey’s reading: configure a maximum outage-pause age, default **24 hours of cumulative
paused wall time per request**, through D3. On reaching it, make the request terminal
`expired`, never DLQ. Show pause age, limit and effective deadline in the Console.
Expiry prevents dispatch but never releases capacity still held by an unproven stop.
This maximum bounds the deadline extension above and does not authorize effect replay.
D6 covers managed work requests only. Existing notifier delivery backoff
(`packages/cli/src/notifier.mjs:33-34,257-315`) and Discord gateway reconnect backoff
(`packages/discord/src/gateway.mjs:40,120-133`) stay out of scope and keep their own
owners. Do not wrap either loop in manager retries or count reconnects as work attempts.
Any later consolidation needs its own design and acceptance tests.
| Outcome | Action |
| --- | --- |
| Mechanically classified outage (D13) | Outage-paused, counters/clocks frozen, service probes under D14. No outage DLQ. |
| Isolated connection refusal before send, explicit 429/503 with no accepted work, safe read failure | Bounded retry/cooldown when adapter proves eligibility and D13 does not classify an outage. |
| Invalid schema, policy/capability denial, kill active, 401/403, missing configuration | Refuse or hold for authorized correction. No repeated calls or alternate route to bypass denial. |
| Partial stream, lost acknowledgment, tool write timeout, incomplete stop proof | `uncertain`. Reconcile first. Cancellation does not establish absence of effects. |
| Non-outage retry budget/deadline exhausted, deterministic request failure | Dead-letter with evidence and actionable reason. Non-outage admission expiry is an expired terminal item. |
| Explicit user cancellation | Terminal cancelled. Never retry automatically. |
**D7 DLQ.** Only requests that fail on their own use this path, not outage-paused work.
Validation/permission refusals remain refusals under D6, with their request evidence.
A durable request state and append-only disposition history, not copied
prompts or modified runs. Show request/attempt IDs, class, wait/route history, redacted
error code, idempotency/effect status and links to authorized evidence in Console
“System → Requests → Dead letters”. Propose equivalent `mosaic system requests
show|list` and `mosaic system replay REQUEST --expect-revision REV` verbs. These are
proposed interfaces, not available commands. Mos sees permitted items and recommends
replay. Jason or an explicitly delegated role can replay after cause correction and
reconciliation. Replay creates a new request linked to its parent, with a fresh bounded
budget and current policy. No mass replay, unattended replay chains or resurrected
cancelled work. Unknown effects require explicit reconciliation evidence before replay.
D15’s automatic batch continuation of outage-paused work is not a replay chain and
requires no person to release each request. An outage never sends those requests here.
**D8 Rerouting.** Candidate routes come only from the user’s configured allowlist,
filtered for model capability, tool protocol, context, privacy/region, account budget
and role constraints. Prefer same model/provider recovery, then a configured equivalent
provider, then a pre-approved alternative model. Never downgrade required capabilities
or migrate a live transcript silently. Mid-turn failure is reconciled before a fresh
attempt. Record requested and effective route and reason. New provider consent/spend
is separate from enabling the decision hook. F10’s preflight fallback remains distinct.
## 5a. Protected mode and outage recovery
**D13 Protected mode (Huey’s reading of Q205/Q206).** The supervisor owns a durable
`normal → protected → checking → canary → batch-resume → normal` state per affected
service scope, with a host/network incident able to hold all dependent scopes. This is
separate from the human kill switch. D1 remains unchanged: the manager never sets or
clears kill-switch state. Detection/status tracking are mechanical, without an LLM.
Proposed entry thresholds, all Huey’s readings: three consecutive failed network checks
at the 2-second sample cadence before protected mode enter a network incident.
Once protected, active checks follow D14’s schedule. For a provider/account scope,
enter after at least five endpoint/dispatch results in 30 seconds with ≥80% transient
network/429/5xx failures across at least two distinct requests, or three consecutive
failed endpoint checks. One explicit account-wide quota/reset response can enter its
scope immediately. Isolated validation/permission failures and model output/tool errors
do not count. Request-local failures after a successful test session use D6, never
D13 by themselves. Re-entry based on resumed requests requires the correlated failure
threshold across at least two distinct requests, not one bad request repeated. Until wider
failure scope is proven, hold only that route/account. A missing required observation
holds work as unknown rather than guessing recovery. Persist incident and classification
before advancing affected request outcomes. These settings use D3’s reviewed config path.
Fence affected new dispatch. Pause uses only F5’s cooperative interrupt at safe points.
F6 supplies stop proof, never the pause action. Do not invoke force-stop for a pause.
Pause means a proven quiescent checkpoint/turn boundary, not SIGSTOP or a presumed
rollback. Keep reservations until stop/idle proof arrives. If proof is unavailable,
record uncertain ownership and retain capacity. Do not force-stop a write or manufacture
human force-stop confirmation to obtain a checkpoint. Reconcile interrupted effects
mechanically by operation ID/receipt when possible. Ambiguity stays fenced. Unaffected
services can continue under admission. No person releases each paused session.
**D14 Probes.** Following Q206 and Mos’s recorded reading, each service check first
performs mechanical network and endpoint checks. Only after the endpoint answers may a
one-turn headless probe run through a minimal harness config. A check of a dead link
makes no model call. Huey’s reading: use a fixed synthetic input, no tools, user content,
workspace effects or inherited worker environment, and separately authorized probe
credentials. For JEV, the probe harness invokes the typed decision adapter, not a chat
request. Probe success means the endpoint plus the configured minimal harness/model
path succeeded, not merely an HTTP connection. One successful complete probe marks that
service online. Missing credentials, exhausted applicable quota/budget or unknown
billing classification keep the headless check deferred/unverified. Subscription-only
probes need verified usage/quota accounting, not an invented per-call USD price. Paid
probe charges use D11’s separate probe budget. No synthetic success or unauthorized spend.
Under Q207 and Mos’s recorded reading, **every service probes in parallel** on its own
schedule. JEV is not a gate for another service. Each has a durable probe schedule:
wait 60 seconds, then after each failure wait 120, 180, 240, 300, 360, 420, 480, 540,
600 seconds, then 600 seconds repeatedly. There is **no attempt limit**. Wait at least
Retry-After or the usage-limit reset before probing, even if longer than 600 seconds.
The ceiling bounds this schedule, not a provider-mandated wait. Unknown reset time
keeps the scope held and visible rather than guessing a usable quota. Service probes
are separate from D10 routing calls and never retry a paused work request.
Huey’s reading: permit one in-flight probe per service, with a 5-second mechanical-check
deadline and a 30-second headless-probe deadline. Hung probes fail with owned cleanup
and uncertain outcomes retain their reservations. These deadlines are independent of
D10’s endpoint-specific decision timeouts. Scheduling retries indefinitely does not
override D11: when applicable quota or probe funds are unavailable, keep mechanical
checks running on schedule and defer headless probes until authorized capacity returns. A human
kill prevents headless probes/agent launches, while local read-only health tracking
continues. The kill can be cleared only through its existing human path.
Huey’s reading for optional endpoints: mark disabled/unconfigured JEV or a fallback as
not configured and make no call to it. Other services keep their independent schedules.
A failed or budget-deferred decision endpoint never blocks their probes.
**D15 Resume (Huey’s reading of Q205/Q206 and Mos’s recorded reading).** After service
online proof and D4’s healthy release thresholds, reconcile paused owners/effects and
run **one test session** on an eligible online route. Use disposable synthetic work,
current policy and bounded capacity. No request with uncertain effects is used as a
test. On success resume eligible paused work in batches of at most five, with one
30-second healthy observation interval between batches and D4’s stricter dispatch
rate/reservations still enforced. A batch needs no new approval per session. A failed
test session returns to protected mode with its evidence, without relabelling an
otherwise healthy provider as offline. After the test succeeds, “batch fails” means
D13 proves a correlated outage across distinct requests, or independent mechanical
network/endpoint evidence establishes an outage. Only that returns the scope to
protected mode. Individual resumed requests failing validation, tools or model output
consume D6 budgets and follow the two-resume/D7 rule, while healthy batch members
continue. A failed request alone never counts as a failed automatic exit.
Huey’s proposed escalation count: after **three consecutive failed automatic exits**
(canary or batch stages) in the same incident, alert Mos/Jason and hold production
resume for diagnosis. Continue mechanical monitoring and budget-permitted probes
without an attempt limit. A fully successful exit resets the exit-failure count. This
is the repeated-exit boundary where a person is needed, not a prompt for each session
or each failed probe. Existing policy, missing credentials and unresolved external
effects remain fail-closed constraints, never reasons to improvise authority.
Huey’s reading of release after three failed exits: persist `recovery-escalated` on the
incident. Later successful probes update service health but **do not** restart work
automatically while that latch is set. Jason, or a role already explicitly delegated
recovery authority, releases it through proposed human write `system.recovery.release`
(scope, expected incident ID and revision), also exposed as Console “System → Recovery →
Release after diagnosis”. The authenticated/scoped writer checks authority, current
incident/revision and diagnosis evidence. It clears only this escalation latch, resets
its exit counter and restarts the one-test-session stage, never bulk dispatch. It does
not clear a human kill, permissions, unresolved effects or resource holds. This design
adds no role grant. Without a delegated role, Jason is the releaser. S5 owns the new
broker/Console write and review, separate from the kill-switch clear.
## 6. Monitoring and Console
**D9 Sampler ownership.** S2’s implementer owns the live sampler inside the mechanical
supervisor process proposed in D1, running in monitor-only mode before admission lands.
That process samples every 2 seconds and supplies timestamped local snapshots through
the Console’s authenticated bus read transport via the proposed `system.health` read
verb. S2 includes the broker handler and its scoped authorization review. The Console
does not read `/proc` itself. Process death/stale samples show unknown. Request/provider fields show unavailable
until their adapters land. Starting this process and updating the live Console require
an approved re-pin window after MVP. No sampler is started by this design work.
**Measurements.** Pressure samples and freshness, active/reserved work, ownership
uncertainty, queue depth/oldest wait by class, hold time/reason, admission/turn latency,
name **`MOSAIC_DECISION_OPENROUTER_KEY`**. This is not an existing slot claim. The
ordinary `OPENROUTER_API_KEY` is in `ENV_ALLOW` and `sessionEnv` copies allowed variables
(`packages/seat/src/session.mjs:83-99`), so it must never hold the decision key. Jason
places the dedicated value only after live-trial approval. Runtime injects it only into
the decision adapter, never the shared launcher environment, model state or worker.
Do not add the distinct name to the worker allowlist or alias its value to an allowed
provider variable. S6 must prove `sessionEnv` excludes the distinct variable using a
synthetic canary. No management key access. A direct Decisions
adapter may be needed unless a stand-in proves the approved proxy supports this API.
Huey’s reading of Q207 credential isolation: the fallback harness has its own configured
credential reference, injected only into the decision module’s isolated minimal harness,
never agent sessions or the shared launcher environment. Do not reuse worker credentials
or copy worker HOME/session config. A needed credential stays a slot reference in this
design. Apply the same canary/redaction checks to each fallback credential as to the
decision key. No agent endpoint credentials or route-selection tools are added.
## 8. Proposed queue rows, smallest first
Sage assigns actual row IDs, Gitea issues, owners, reviewers and gate grants. These
are proposals under Vikunja 452, not edits to either queue file. No implementation is
authorized here. Vikunja 450/451 and MVP 414/416 retain their existing order/authority.
| Ref/order | Slice and dependency | Acceptance evidence | Before 10-27? |
| --- | --- | --- | --- |
| S1 | Pressure parser and pure rules evaluator, no live enforcement | Synthetic load/memory/PSI/missing-signal cases, exact hold/release thresholds, fake-clock hysteresis, effective CPU/memory limit cases. No builds or real pressure generation. | Candidate could land if Sage has spare review/gate capacity. Never displace MVP. |
| S2 | Live sampler in the mechanical supervisor process (monitor-only), plus read-only Console System status, after S1. S2 implementer owns process lifecycle and snapshot transport (D9), including the new `system.health` bus read verb and broker handler. Scope and review include broker authorization. | Fixture sampler feeds timestamped Console responses for healthy/held/stale/unknown. Process death, unavailable adapters and auth scoping tested. No admission writer, secrets or command-line fields. | After MVP. Live sampler/Console activation needs an approved re-pin window. |
| S3 | Request journal/admission, after S1 and final kill/lifecycle contracts | Migration/rollback compatibility, duplicate-key conflict, crash before/after spawn, two dispatchers, generation fencing, queue limits/expiry, provider cooldown, fairness under sustained traffic, kill-at-launch race, instance isolation. | After MVP preferred. Schema and dispatch integration need a full owned gate. |
| S4 | Cooperative interrupts and capacity ownership, after S3 and adopted gate lanes | Agent abort proof vs uncertain, test-step cancellation/quarantine, owned build cleanup, PID reuse, remote tool unknown outcome, emergency request during full backlog. No gate grant synthesized. | After MVP. |
| S5 | Retry/DLQ/replay and permitted reroutes, plus protected-mode journal, mechanical outage detection and automatic resume policy (D13/D15), after S3/S4 | Fake-clock failure spikes/scopes, restart during pause, frozen counters/deadlines, outage never DLQ, isolated request failures still bounded, Retry-After, unknown proxy budget, partial-stream/write no replay, authorized replay lineage, incompatible-route refusal, canary then batches, request-local failures consume D6 budget, one bad request cannot re-enter protected mode, correlated distinct-request re-entry, two separate failed resumes go to D7, batch-failure definition, three-failed-exit latch, authorized revision-bound release via broker/Console without kill clear, later probes do not auto-release, maximum pause age expires without DLQ/capacity release, human kill unchanged. No real probes. | After MVP. |
| S6 | Decision hook, per-exclusive-key USD ledger and setup, after S5 | Stand-in typed responses, late/invalid/missing/low-confidence answers, unapproved action, snapshot race and redaction canaries. Measure timeout rates and choose the timeout from stand-in results. Verify `sessionEnv` drops `MOSAIC_DECISION_OPENROUTER_KEY`. Test exclusive key binding, reject shared keys/missing provider limit, independent-key budget totals, concurrent reservations, missing/duplicate cost, crash and day rollover. D14 probe adapters: endpoint-before-headless, 60-step-to-600 schedule with no attempt limit, all service schedules parallel without a JEV gate, one success online, reset wait, disabled endpoints, subscription/quota-only success without USD, separate paid-probe caps and deferral, restart and hung-probe cleanup. Prove only the decision module selects endpoints/routes: a session’s own decision call carries no routing authority and no session output enters route selection (test both directly), immediate rules for dispatchers that cannot wait, timed-out JEV not occupying the fallback slot while its spend reservation remains, JEV → allowed smaller model → internal rules, per-model confidence/timeout calibration, 4 KiB metadata-only input at every endpoint and isolated fallback-credential canaries. Network disabled by default. | Fixture implementation may proceed later. No paid trial before explicit approval. |
Each slice gets independent review and tests that kill a mutation of its central
invariant. Sage chooses package/full-suite gates appropriate to the final diff.
Do not run them alongside an MVP gate without a grant. Before production activation,
measure emergency control responsiveness and recovery with disposable cohorts under
bounded synthetic pressure. No load experiment on the shared development host.
## 9. Questions for Jason (recommendations, not design blockers)
- **Q1 Initial authority:** should the manager ever force-stop background work just
for load? **Recommend no.** Cooperative yield for load, hard stop only through an
already-authorized safety/explicit stop. Broader authority needs a separate decision.
- **Q4 System recovery priority: superseded by DECISIONS Q205 and Q206.**
Jason, verbatim (Q205): “If a catastrophe does occur, it makes sense to automatically
throw the kill switch, pause agent runs, etc.”
Huey’s reading, following Mos’s Q205 reading: protected mode does the automatic
pausing instead. The kill switch stays human-only. Jason’s Q21 reply in Q206 asked
only about cool-down, DLQ and retry. D1 remains unchanged.
Jason, verbatim (Q205): “It does not make sense to require the user to release each
back into operation. I would rather equip the system with the tools necessary to
detect failure, enter protected mode, monitor for cool-down periods, check for
system operational status, resume operations.”
Jason, verbatim (Q206):
> Mosaic Stack mechanical tooling detects the outage and tracks the status without LLM need
> Continual mechanical attempt for JEV begins with a cooldown of 60, 120, 180 seconds with a ceiling of 10 minutes
> Once JEV is back online, the same cooldown starts for the other harnesses/providers in parallel
> Necessary services are marked back online when found available
> System adapts based on available services, providers, harnesses, usage limits, and usage budgets
Q207 supersedes only the JEV-first order quoted above. D14 now probes all services
independently in parallel.
Huey’s reading, following Mos’s labelled Q205/Q206 reading: D13–D15 automate
protected-mode recovery without per-session approval. Entry thresholds, checkpoint
handling, deadline freezing/max pause age, probe deadlines/budgets and disabled-endpoint behavior,
batch sizing and the three-failed-exit escalation count are proposed readings, not
Jason’s verbatim requirements. D1’s human kill-switch boundary remains unchanged.
- **Q22 Resolved by DECISIONS Q207.** Jason, verbatim:
> Q22: That makes sense, somewhat. If the agents are allowed to directly bypass the decision-making module of Mosaic Stack, they will default to bypassing it all the time. Once service is returned, they will not go back to using the system properly until their context is cleared. We would be better suited to have the requirement for decisions to always flow through the same place on the system and have that module decide the endpoint to use for decisions. If JEV is unavailable it can default to a Haiku, Luna, or some other smaller model to make the decisions from within a minimal harness. Context for the decision-maker would be kept minimal to avoid bloat and unnecessary slowdowns. I'd rather have the fallback occur at the decision module rather than at each agent session.
Mos’s recorded reading: service probes proceed independently in parallel and the
decision module tries JEV, then a configured smaller model, then its built-in rules.
**Rules last is Mos’s reading**, not Jason’s verbatim endpoint order. Huey’s readings:
the 10-second fallback timeout candidate, per-model calibration, billing split and
isolated fallback credentials in D10–D12. Session calls/output carry no routing authority.
## Verification and handoff
Revision evidence: r3 `a36952564` named the health verb/broker scope, clarified ownership
attestation and added Q4. r4 `1e8bccff1` recorded Q204 and tunable cap settings. r5
`fd44f7c60` added Q205/Q206 protected mode, probes and resume, with D1 unchanged. r6
records Q207, central fallback, independent probes and scrooge-0003 S-F1–S-F7 fixes.
Huey’s new thresholds, billing defaults, release/lifetime rules and timeout choices
remain labelled readings for review, not implementation or live-test evidence.
sampler/process ownership and post-MVP re-pin, P1 interpretation, missing-confidence
fallback and measured timeout selection, verified Noul spelling, retry scope exclusions,
and a citation for the lifecycle brief’s revocation requirement.
This document was checked against the pinned source and public documentation only.
No build, suite, live bus write, service action, proxy inspection or paid call was
performed. The canonical `scripts/mosaic queue next huey` returned `nothing`.
The initial linked-worktree refusal was diagnosed, not bypassed. The direct mos-1183
assignment governs this branch. Gitea issue creation and implementation-row entry
remain with Sage: this brief forbids credential-file reads and queue writes.
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
jason.woltje