60 lines
8.9 KiB
Markdown
60 lines
8.9 KiB
Markdown
# #662 scope: factory finding triage, mandate budget and flaky-test policy
|
||
|
||
Owner: Sage. Authority: mos-1884; #659 S2–S4; DECISIONS Q261/Q263 (Q24 yes), with Q222 human-only settings. Starts after rows330/331: landed origin/refactor through0e1dd0a95. This scopes product work in Mosaic Stack; it does not reopen #657/#658 or alter their backlog status.
|
||
|
||
## Outcome
|
||
|
||
The next factory enforces bounded mandates, evidence-based finding triage and tracked flaky-test remediation in source. Console/CLI humans control the policy; agents can read their effective mandate and submit evidence, but cannot change guard settings, enlarge their own budget or make a failed gate pass.
|
||
|
||
## Settings and defaults
|
||
|
||
Proposed engineering namespace: `factoryPolicy` in the instance's one system config, with Console/CLI editing behind existing human authentication and writer conventions; the new policy editor is part of this task. Final wire names belong to implementation; there is no agent-writable policy copy in a task description or prompt. Effective values and the policy revision used for each decision are visible and recorded. Missing optional settings use documented defaults; explicit malformed/unknown policy refuses policy-dependent mutation and never silently widens authority.
|
||
|
||
| Control | Initial default | Effect |
|
||
|---|---|---|
|
||
| Finding blocking rule | `regression-or-red-gate` | Candidate-caused defects and red required gates block; preexisting defects receive a linked task and do not block this landing. |
|
||
| Severity vocabulary | P0 urgent / P1 high / P2 normal / P3 low | Every finding carries severity separately from causality and blocking disposition. Severity does not turn an older defect into a regression or excuse a red gate. |
|
||
| Review-round budget | 2 per mandate | Initial review and one subsequent review. This is Sage's proposed default, consistent with the existing two-round lead rule, not a new Jason-selected number. |
|
||
| Allowed seat outcome reports | done / red-with-cause / hard-rule-wait / scope-change | The coordinator receives outcomes rather than routine per-step approval requests. Evidence remains available without status-message fan-out. |
|
||
| Timing-failure action | deterministic source fix plus tracked quarantine | Preserve the failing test/assertions and raw failure; fix the timing dependency with the existing clock/readiness mechanisms. |
|
||
| Quarantine mode | tracked, temporary, no gate exemption | Record the flake under the human-owned policy and link its source-fix task. It cannot remove a required gate, count a skip as PASS or waive a red required gate. |
|
||
|
||
Only an authenticated human may change effective settings, grant an override or extend the round allowance. An agent/PM may request a scope change under its existing task authority, not grant itself a budget reset. An environment variable may still select the legitimate system config under the existing contract; it cannot inject policy values. Role names, prompts, tester sessions and task JSON are not policy authority.
|
||
|
||
## S2: finding triage
|
||
|
||
Record finding identity, severity, candidate/baseline references, reproduction or source evidence, causality (`candidate-caused`, `preexisting`, `unknown`), affected test and disposition. Derive the blocking decision from the current human policy and real gate state. A preexisting finding must retain its follow-up task/evidence; it does not trigger an open-ended design dependency. Unknown causality stays visibly unresolved until comparison establishes it; absence of evidence is not a passing test. Review receipts and the queue's existing actor/approval rules remain real, not fabricated or bypassed.
|
||
|
||
## S3: bounded mandates
|
||
|
||
Each new managed seat task has a nonempty goal, explicit acceptance criteria, scope/limits and an effective round budget tied to a mandate identity/revision. Validate this before dispatch. Count a round once when the actual review request is admitted; duplicate delivery/retry does not consume twice, and restart/relaunch does not reset the ledger. Exhaustion stops another review round and produces the bounded hard-rule-wait/scope-change outcome; it never marks work accepted or spawns replacement tasks to evade the budget. A human extension records who changed it and the old/new limits. Specify and test existing-task compatibility before enforcement; do not rewrite historical tasks or reinterpret existing prose as authenticated approval.
|
||
|
||
## S4: flaky-test policy
|
||
|
||
A timing failure reproducible under controlled load records exact test, commit/tree, raw failure, environment/load evidence and owner/source-fix task. An injected-clock/readiness regression demonstrates the source repair while retaining every original assertion. The quarantine record names the original test, reason, effective human-owned policy revision, explicit expiry/release condition and required reinstatement evidence. Recording a flake needs no per-step approval, and a quarantine record never independently authorizes exclusion from the required gate. No retries-until-green, larger deadline, assertion deletion, renamed-away failure or unreported skip substitutes for repair. Use owned deterministic fixtures instead of making acceptance depend on obtaining a slow host.
|
||
|
||
## Acceptance criteria
|
||
|
||
1. Human Console and CLI can read/update/reset the settings with defaults, validation, concurrency protection and an audit record; agent, PM and tester attempts through each actual exposed route refuse without a policy write.
|
||
2. A real managed task carries its goal/criteria/effective budget into dispatch and the review coordinator. Missing/invalid mandate data refuses before work starts; legacy task behavior is explicitly covered rather than silently migrated.
|
||
3. A controlled candidate regression blocks; the same independently demonstrated baseline defect produces a linked backlog task and permits the otherwise-green candidate. A real red required gate blocks regardless of finding label/severity. Unknown causality cannot be auto-classified as old or GREEN.
|
||
4. With the default budget, two actual rounds are accounted for and a third refuses. Duplicate requests, crash/restart and changed task descriptions cannot replenish it; only a recorded human extension permits another round.
|
||
5. Actual seat/coordinator routing accepts the four outcome types and preserves the goal/criteria across continuation. Exhaustion returns one actionable cause, not a new approval loop.
|
||
6. Reproduce a timing race using an owned deterministic test signal; demonstrate RED before the source repair and GREEN after it with all original assertions. A wall-clock delay increase cannot satisfy this control.
|
||
7. Quarantine creation/expiry/reinstatement and failure history are tested. Quarantined, skipped, unavailable, cancelled, timed-out and unrecognized outputs remain distinguishable from PASS, including gate-tool consumers. No automatic quarantine conceals a required failure.
|
||
8. Atomic policy/ledger writes, stale revision, duplicate delivery, partial write and restart tests retain history and refuse uncertainty. No secrets enter decision receipts.
|
||
9. Existing role/writer/queue/hook rules, normal task operations, system config compatibility and local test conventions remain covered. This task grants no new push/deploy/host/node authority.
|
||
10. One bounded independent review of the implementation, targeted integration checks and the applicable full gate on the accepted dedicated runner precede landing. Report actual phase outcomes and frozen commit/tree; no count-only or composite acceptance.
|
||
|
||
## Delivery boundary
|
||
|
||
Implement as one bounded feature with independently testable policy/authorization, finding disposition and mandate/quarantine consumers. Expected source owners: system config loader and human settings surface; tasks/seat dispatch and durable review accounting; review/gate disposition consumers. Exact file/schema changes and tests are settled in the implementation plan against the then-current base, not copied into agent work artifacts as a substitute for enforcement. Documentation includes defaults, human overrides, compatibility and recovery behavior.
|
||
|
||
S1/#660 infrastructure and S6/#661 node access remain Ops-owned. S5/#663 record archival is separate. No redesign of provider/model routing, deployment, source role authority or #657/#658 recovery protocols is part of #662. No new agent/review factory is launched by this scoping note.
|
||
|
||
## #660 dependency
|
||
|
||
As inspected, #660 comment1335 reports source prepared, independent review/PR merge pending and no VM/CPU write applied. Sage has supplied the actual gate commands/runtime needs in comments1329/1334. Do not claim the tool targets a dedicated runner yet. When Ops supplies the ready verified endpoint/identity, approved profile, runtime checks and merged provisioning/runbook evidence, configure the gate target, refuse local fallback and complete one full frozen gate there; confirm actual target/receipts on #660. No local substitute run or host/Proxmox change is authorized by this scope.
|
||
|
||
Stable operation: sage-4849-task662-scope-mos1884.
|