row 281 design d5: two requests per granted delete with its five conditions, a job id on every answer, event and message (sage-2575, #1786)

agents/megavolt/work/queue-281/BUILD.md

@@ -1,6 +1,6 @@
 # Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
 
-Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d4 for Dewey's approval. No code until it is
+Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d5 for Dewey's approval. No code until it is
 approved. The results sections are filled in after the runs on Sage's gate.
 
 History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
@@ -10,6 +10,9 @@ approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-
 - B3: one step G per agent, with named refusals for everything that could race it.
 - N1-N3 are folded in.
 
+d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
+answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
+
 ## Sources
 
 - Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
@@ -58,10 +61,15 @@ d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wai
   gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
   plus the Gitea calls can pass both limits.
 
-So step G never runs inside a verb. Both verbs stay synchronous and answer at once. Step G runs as tracked
+So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
+timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
 broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
 the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
 
+Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
+and every inbox message about it carry that id. The human and the chief can then match a later outcome to
+the request that started it.
+
 ### The fire
 
 1. `write('instance.fireCheck')`, as today.
@@ -70,41 +78,64 @@ the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher
    comes after `fire.stopped` and refuses (section 3d).
 3. If the set is empty and the agent has no Gitea leftover (section 5), steps B and C run synchronously,
    exactly as today. A fire with no grant answers today's result with today's events (T7).
-4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {grants}` in step A's
+4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
-   transaction, starts the tracked job, and answers at once:
+   step A's transaction, starts the job, and answers at once:
-   `{...stopped, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
+   `{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
    step B hasn't run.
-5. The tracked job runs step G, then step B (`write('instance.fire')`) and step C (`consumeAuthority`,
+5. The job runs step G, then step B (`write('instance.fire')`) and step C (`consumeAuthority`, then
-   `fire.done` with `grants`). It releases the lock when it ends, whichever way it ends.
+   `fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends.
 
 What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
-`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`. A repeat fire while the job
+`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
-runs refuses `fire-in-progress`, a new code. A repeat after it finishes takes the delisted path (webby-0094
+- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
-N1) and answers done. A repeat after a refused step G runs step G again (section 5). The human sees the outcome
+- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
-either way: `fire.done` names the grants, and a refusal sends the inbox message in section 4.
+- A repeat after a refused step G runs step G again (section 5).
+
+The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
+inbox message in section 4.
 
 To check at build time: step C's `consumeAuthority(s, ...)` runs after the verb has answered, so the job keeps
 the fields of `s` that step C needs. It must not depend on the chief's run still being live.
 
-### The delete
+### The delete (sage-2575)
 
 `lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
 grant set.
 
-- An empty set: `instances.set delete` as today.
+- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
-- A non-empty set: it takes the lock, starts the same tracked step G with verb `lifecycle.delete`, and refuses
+- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
-  `grants-revoking` at once. The launcher closes the request with that code, well inside its 10 s. That's
+  verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
-  sage-2573's Q2 path: a closed request, and the human raises the delete again.
+  closes the request with that code, well inside its 10 s.
-- When the job ends, the human gets an inbox message. Either "<agent>'s grants are revoked: raise the delete
-  again", or section 4's refusal.
-- A delete raised while the job runs refuses `grants-revoking` again, without starting a second job. The
-  delete raised after the job finds an empty set and goes as today.
 
-The tradeoff: deleting a granted instance now always takes two requests. The alternatives were to hold the
+  The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
-launcher's request open past its 10 s, which would need a host channel change and a strand path, or to make
+
-the launcher poll. Two requests changes no other package. **For Sage:** this extends Q2 from "a refused
+  > Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
-delete is raised again" to "every delete of a granted instance is raised again". Say if that's not
+  > Raise the delete again when the inbox says the revoke is done.
-acceptable.
+
+- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
+  (condition 2, the 3e lock). No second message goes out either: the first one still applies.
+- **The job ends.** The human gets one inbox message (condition 3).
+  - If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
+    delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
+  - If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
+    describes. It records `grant.revoke.refused {job, ...}`.
+- **The delete raised after a completed job** finds an empty set and goes as today.
+- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
+  the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
+  reason in a message:
+
+  > Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
+  > the delete again when the inbox says it's done, or clear <grant> by hand first.
+
+  It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
+  end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
+  it. A delete after the human cleared the grant by hand finds an empty set, or only leftover lines (Q1), and
+  goes on.
+
+The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
+cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
+launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
+widen the row into the launcher for a one-time wait.
 
 While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
 harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
@@ -265,7 +296,7 @@ the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing"
 already closed with `grants-revoking` (section 2).
 
 The verb has already answered, so the refusal is a record, not a thrown code:
-- One grant-evidence event, `grant.revoke.refused {agent, grant: 'host:<id>' | 'gitea', reason, verb:
+- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
   'instance.fire' | 'lifecycle.delete', by}`.
 - `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
 - `grant-revoke-failed` is the code in the event and in the inbox message.
@@ -319,15 +350,29 @@ way through the human's new request.
   records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
   `grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
   does. T3b tests both.
+- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
+  revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
+  done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
+  with a test:
+  1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
+     delete again when the inbox says done. A bare code isn't enough (T3).
+  2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
+  3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
+     manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
+     reason and doesn't complete over a live grant (T3b).
+  4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
+  5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
+     (T2).
 
 ## 6. The result and the event (D6)
 
-- **A fire with a non-empty set** answers at once with `revoking` and `done: false` (section 2).
+- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
-- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `grants: [{grant:
+  `fire.revoking {job, grants}` (section 2).
+- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
   'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
   So the event the human sees names them.
-- **A delete's job** records `grants.revoked.done {agent, verb: 'lifecycle.delete', grants, grantsLeft}` and
+- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
-  sends the inbox message in section 2.
+  verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
 - **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
 
 ## 7. What doesn't change (brief item 4)
@@ -352,20 +397,33 @@ same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete
 - T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
   `credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
   a record whose key file is gone.
-- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers `revoking` with all three and
+- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
-  `done: false`. After the job:
+  with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
+  the job:
   - all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
     them, plus `fire: true`;
   - the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
   - `fire.done` names all three;
   - that agent's `host.run` and Gitea use both refuse.
-- T3: the same through the delete (`lifecycleTake` then `lifecycleDelete`). The first delete refuses
+- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
-  `grants-revoking` and starts the job. The launcher's pass closes the request with that code. After the job
+  `lifecycleDelete`).
-  the human's inbox says to raise the delete again. A second request completes as today.
+  - The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
-- T3b (Q2): a delete whose job refuses (host B unreachable). The first request closes `grants-revoking`. The
+    naming the job and all three grants, and saying to raise the delete again when it's done.
-  file and the instance are unchanged, and `grant.revoke.refused` and the inbox message name the instance,
+  - A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
-  `host:B` and the reason. With B back, a new request starts a job that skips the grants already revoked, and
+    message.
-  the request after that completes.
+  - After the job, the end message names the job and each grant revoked.
+  - A third request completes as today.
+- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
+  - The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
+    `mosaic hosts revoke`.
+  - The file and the instance are unchanged.
+  - The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
+    reason and the new job, and the instance is not deleted.
+  - With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
+  - A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
+    for Gitea and the leftover line.
+- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
+  same events counted, no inbox message.
 - T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
   `host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
   - the file is unchanged and the instance is still firing;
@@ -447,6 +505,9 @@ Files (N3). Every file listed runs whole in the manifest round, touched or not:
 - M13: the verb awaits the job before it answers. Killed by T10.
 - M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
 - M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
+- M16: the delete's first refusal sends no start message. Killed by T3.
+- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
+- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
 
 ## 10. Order and rebase