row 281 design d1: a fire or a delete revokes the instance's host and Gitea grants (sage-2568, #1786)

agents/megavolt/work/queue-281/BUILD.md

@@ -0,0 +1,230 @@
+# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
+
+Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d1 for Dewey's approval. No code until it is
+approved. The results sections are filled in after the runs on Sage's gate.
+
+## Sources
+
+- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
+  mos-1271, Vikunja 524).
+- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
+  `gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
+- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
+  goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
+- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
+
+## 1. What happens today (read from the code, to be pinned by a test before the change)
+
+- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
+  agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
+  `<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
+  (gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
+  step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
+  `mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
+  roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
+  `gitea_grants` change and no Gitea call.
+- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
+  `host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
+  stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
+  The next start then finds a grant record whose key file is gone. What the start does with that isn't read
+  yet. The T1 test below pins it.
+- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
+  grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
+  still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
+  `gitea-not-declared`.
+
+## 2. Where the grant revokes sit (D1)
+
+The grant revokes run in the broker, awaited, inside `#chiefLifecycle`, after step A and before step B:
+
+1. `write('instance.fireCheck')`: the writer's checks against the file, as today. A refusal here revokes
+   nothing.
+2. Step A: the runs stop and the open requests are cancelled, as today.
+3. **New, step G:** the grant revokes (section 3). A refusal stops the fire here.
+4. Step B: `write('instance.fire')`. The writer's checks, then `beforeWrite`, which is `revoke()` for the
+   credentials, then the write. With the grant files gone after step G, `revoke()`'s store-directory check
+   passes.
+5. Step C: as today.
+
+Why not in `beforeWrite` or `revoke()`: both are synchronous and run inside the organization writer, after
+its checks. A host `remove` is an SSH call and a Gitea revoke is HTTP, so they can't run there. The business
+package also has no reach into the broker's grant tables, and shouldn't get any.
+
+Why after step A: a fire is the decision to stop the agent. If step G came first, a host that can't be reached
+would keep the fired agent running. After step A, a refused step G leaves the same state as a failed step B
+today: the instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state.
+
+The repeat fire's own path (webby-0094 N1: an instance already delisted still has its tokens revoked, with no
+write) also runs step G. Step G sits in the broker before `write('instance.fire')`, so it runs on both paths.
+
+The delete: `lifecycleDelete` (broker.mjs:3293) runs step G after its running and starting checks and before
+`instances.set delete`. It becomes async. Its one caller, the launcher's `host.op` (launcher.mjs:845), already
+awaits the reply, and `process.mjs` already returns async ops (`mayLaunch`). A refused step G answers the
+launcher with the refusal. `remove()` returns `{refusal: e.code}`, and the launcher closes the request through
+`lifecycleDone` with that code, as it does today for a writer refusal. The request is then closed, so the
+"repeat" for a delete is the human's new delete request. Grants revoked before the refusal stay revoked, and
+the new request's step G goes on from there. Q2 for Sage: is it fine that a refused delete needs a new request?
+The other choice is leaving the request taken, which today's launcher pass doesn't do for any refusal. Also to
+check at build time: the launcher's `host.op` reply timeout against an SSH remove and the Gitea calls. If it's
+shorter than they can take, the design needs a change before code.
+
+## 3. Step G: what it revokes, and how
+
+The set, for (business, agent), read in one store transaction at the start of step G:
+
+- each `host_grants` row of the agent;
+- each host named by an open host grant decision for the agent (approved, with no `grant.applied` or
+  `grant.failed`, and no `grant.revoked` of (agent, host) after it);
+- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
+
+An empty set means step G does nothing and records nothing. A fire of an instance with no grant is then
+exactly today's fire, with no new events (brief tests, fourth bullet).
+
+### 3a. A host grant: the host line first (D2)
+
+hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
+`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
+record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
+host. So the fire's host revoke reverses one step and keeps the rest the same:
+
+1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
+   `credential-unknown` branch of `#revokeHost` already does).
+2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
+   "a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
+   decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
+   (section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
+3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
+   host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
+   so the human can tell it from a manual revoke.
+4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
+5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
+
+This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
+hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
+fire and the delete call it with `lineFirst: true`.
+
+The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
+section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
+"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
+
+### 3b. The Gitea grant (D3)
+
+The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
+runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
+`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
+`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
+
+The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
+name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
+gone (section 5's reconcile). Two changes keep the fire closed:
+
+- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
+  step G refuses with nothing changed.
+- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
+  token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
+  forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
+  gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
+  kept visible here, not hidden.
+
+### 3c. Open decisions
+
+A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
+`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
+`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
+`#giteaRevokedSince`.
+
+## 4. A refusal (D4)
+
+Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
+recorded. The rest stay as they were. The fire refuses `grant-revoke-failed`, a new code. The organization
+file is unchanged, since step B never ran, and the credentials are unrevoked, since `revoke()` never ran.
+
+`BusError` carries only a code, so the names go in one grant-evidence event recorded before the throw:
+`grant.revoke.refused {agent, grant: 'host:<id>' | 'gitea', reason, verb: 'instance.fire' | 'lifecycle.delete',
+by}`. Its `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and
+`gitea-left:<ids>`. A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139):
+"Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
+is reachable." The refusal record has the verb and args, as every refusal does.
+
+## 5. A repeat fire (D5)
+
+A repeat runs the whole fire again: the checks, step A (nothing left to stop), step G, B and C.
+
+- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
+- A host grant that refused still has its row, so it's retried from 3a step 1.
+- Gitea: if the set is empty but the agent has an entry in `#giteaLeft`, or an open Gitea revoke, step G runs
+  `#sweepGitea` for the business first and then checks `#giteaLeft` again. Any entry still there for this
+  agent refuses `gitea-left`. The sweep is the human's verb, and running it here records `grant.sweep` and
+  `grant.swept` with `by` set to the firing chief. That's an internal caller of an existing method, not a new
+  verb. An entry left by an earlier manual `mosaic grants revoke` of the same agent blocks the fire the same
+  way. That's intended: no grant outlives its instance, whichever verb started its revoke.
+- A host line left by an earlier manual `hosts.revoke` (`hostLineLeft`, no row) doesn't block the fire. Its
+  record is already gone and `mosaic hosts sweep` owns it. The fire's result names it (section 6), so the human
+  sees it. **Question Q1 for Dewey and Sage:** should it block instead, as the Gitea leftovers do? Blocking would
+  make the two halves match. It would also make a fire wait on a host the human may already have retired.
+  I lean toward naming it without blocking.
+
+## 6. The result and the event (D6)
+
+The fire's result and the delete's done record gain `grants: [{grant: 'host:<id>' | 'gitea', revoke,
+granted}]` beside `credentials`. Any leftover from 5's last bullet goes in `grantsLeft`. `fire.done` (step
+C) carries the same `grants` list, so the event the human sees names them. A fire with an empty step G
+carries neither key, so its shape is exactly today's.
+
+## 7. What doesn't change (brief item 4)
+
+- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers, the same refusals (`tester-dev-only`
+  included), and the same records. hosts.revoke's own order stays local first.
+- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
+  existing grant-evidence shape.
+- `revoke()` in runtime.mjs and the organization writer don't change.
+
+## 8. Tests (each in the owning package, whole files)
+
+Bus, with the fake installer and the fake Gitea the row 235 and 236 tests use:
+
+- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
+  `credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
+  a record whose key file is gone.
+- T2: a fire with a host grant on two hosts and a Gitea grant revokes all three. Each gets `grant.revoked` and
+  `grant.revoke.closed` as the manual verbs record them, `fire: true`, the line gone on both fake hosts, every
+  token deleted and login prohibited at the fake Gitea. Afterwards that agent's `host.run` and Gitea use both
+  refuse.
+- T3: the same through the delete (`lifecycleTake` then `lifecycleDelete`).
+- T4: host B unreachable. The fire refuses `grant-revoke-failed`, and `grant.revoke.refused` names the agent,
+  `host:B` and `host-remove-failed`. The file is unchanged. Host A is revoked and recorded, host B's row and key
+  file are still there, the Gitea grant is untouched (it comes after B in the order), and the credentials are
+  unrevoked. A repeat with B reachable completes and revokes only B and Gitea.
+- T5: Gitea's admin token can't be checked. The fire refuses with the Gitea grant intact.
+- T5b: a token DELETE fails. The fire refuses `gitea-left`. A repeat with Gitea back sweeps and completes.
+- T6: an approved host grant decision that hasn't run yet, for the fired agent: the fire records
+  `grant.revoked {granted: false}`, and the executor then ends that decision `revoked`.
+- T7: an instance with no grant fires exactly as today. Same result keys, and the same events, counted.
+- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
+
+Business: none expected, since the writer doesn't change. Cli: the launcher's delete pass with a refused step
+G closes the request with `grant-revoke-failed`, if the launcher's tests stub `lifecycleDelete`.
+
+## 9. Mutants
+
+- M1: step G skips the host revokes. Killed by T2.
+- M2: step G skips the Gitea revoke. Killed by T2.
+- M3: a refused grant revoke is ignored and the fire goes on to step B. Killed by T4.
+- M4: `lifecycleDelete` skips step G. Killed by T3.
+- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
+- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
+- M7: open decisions left out of the set. Killed by T6.
+
+## 10. Order and rebase
+
+Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
+and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
+before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
+
+## Runs
+
+(after the gate)
+
+## Mutants
+
+(after the gate)