Compare commits

...
+610
View File
@@ -0,0 +1,610 @@
# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d6 for Dewey's approval. No code until it is
approved. The results sections are filled in after the runs on Sage's gate.
History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-0823):
- B1: the set takes every grant decision with no outcome, and a raise refuses a fired or firing agent.
- B2: step G runs detached from the verb, so neither caller waits on the network.
- B3: one step G per agent, with named refusals for everything that could race it.
- N1-N3 are folded in.
d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
d6 answers Dewey's CHANGES on d5 (dewey-0825):
- C1: the job's step C records authority without the chief's claim (`#authorityFor`, as a landing's record
does), processing-halted doesn't hold the job, and a step B or C failure records `fire.failed {job, step,
code}` with its own message. Tests T11 and T12, mutants M19 and M20.
- N4: boot closes a job a restart interrupted, with one message. Test T13, mutant M21.
- N5: section 7 names the refusals the manual verbs gain.
- N6: the set counts Q1 leftover lines, in fire step 3 and in the delete text.
## Sources
- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
mos-1271, Vikunja 524).
- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
`gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
## 1. Why: what happens today (read from the code, to be pinned by a test before the change)
In short: today a delete of a granted instance unlinks the key files but leaves both grant records, the host
lines and the Gitea token live, and a fire refuses `credential-revoke-failed` (sage-2573 asked for this here
and on #1786). The details:
- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
`<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
(gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
`mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
`gitea_grants` change and no Gitea call.
- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
`host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
The next start then finds a grant record whose key file is gone. What the start does with that isn't read
yet. The T1 test below pins it.
- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
`gitea-not-declared`.
- **A raise after the fire** (dewey-0823 B1). `#raiseGrant` (broker.mjs:1529) checks only that the caller is
the chief, and the grant bindings check only the roles entry. So the chief can raise a new grant for a fired
agent, and if the human approves it, it applies.
## 2. Where the grant revokes sit (D1, rewritten in d4 for B2)
d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wait that long:
- **Delete.** The launcher's `host.op` waits `REQUEST_TIMEOUT_MS = 10000` (cli/src/host.mjs:31). On a timeout,
`breakChannel` stops the whole bus host. One SSH remove can take 30 s (host-grant.mjs:231, :311), and each
Gitea call 15 s (`CALL_MS`, gitea-grant.mjs:38).
- **Fire.** server.mjs answers a verb that isn't on its async list synchronously (:154), and the async list
gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
plus the Gitea calls can pass both limits.
So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
and every inbox message about it carry that id. The human and the chief can then match a later outcome to
the request that started it.
### The fire
1. `write('instance.fireCheck')`, as today.
2. Step A, as today, with one addition. In the same transaction, after `fire.stopped`, the grant set is read
(section 3). Reading it there means a grant raise either committed before step A, and is in the set, or
comes after `fire.stopped` and refuses (section 3d).
3. If the set is empty, steps B and C run synchronously, exactly as today. A fire with no grant answers
today's result with today's events (T7). The set counts everything step G would touch (d6, N6): grant
rows, open decisions, the agent's Gitea leftovers (section 5) and its Q1 leftover host lines (section 5).
Any of them makes the set non-empty and goes to the job.
4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
step A's transaction, starts the job, and answers at once:
`{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
step B hasn't run.
5. The job runs step G, then step B (`write('instance.fire')`) and step C (`#recordAuthority`, below, then
`fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends, `fire.failed`
included.
What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
- A repeat after a refused step G runs step G again (section 5).
The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
inbox message in section 4.
**Steps B and C in the job (d6, C1).** d4 left "step C without the chief's run" as a build-time check, and
Dewey checked it: it fails.
- `#consumeAuthority` calls `#checkAuthority` (broker.mjs:2852). That runs `#processingWork()` and
`#agent(s)`, and `#agent` refuses `not-holder` once the chief's run no longer holds its claim (:749).
- After the verb answers, the job can take a minute or more. If the chief's run ends meanwhile, step C would
throw after step B had delisted. The instance would be left delisted with no `fire.done`.
So the verb does the full check, and the job only records:
- The verb runs `#checkAuthority(s, action, authority)`, as today, before step A. The chief holds its claim
then.
- The job's step C runs `#authorityFor(s, action, authority)`, the check without the holder test. It then
runs the `#spent` check and the `action.allowed` event, as `#consumeAuthority` does. That's the shape a
landing's record already uses (`recordPublish`, broker.mjs:3174, row 137): the run that asked held its
claim when it asked, and may have ended since. It's one small private method, `#recordAuthority`, shared
with nothing else.
- The job keeps a copy of the session fields `#authorityFor` and `#event` read (business, role, thread,
run). It doesn't keep the live session object.
- `processing-halted` doesn't hold the job. A halt stops agent work. Step G only removes access, and steps
B and C finish a decision the chief made while processing ran. Holding them would leave a delisted or
half-revoked instance "firing" for the length of the halt. The verb itself still refuses during a halt,
through `#checkAuthority`, as today.
**A step B or C failure in the job** gets its own record, not `grant.revoke.refused`, since every grant in the
set was revoked by then:
- It records `fire.failed {job, step: 'B' | 'C', code}`. `code` is the thrown code, for example the writer's
`credential-revoke-failed`, or `decision-consumed`.
- It sends an inbox message:
> Fire of <agent>: job <job> revoked its grants (<list>) but stopped at step <B|C> (<code>). The instance
> is still firing. Run the fire again once <code> is cleared.
- The instance stays firing. A repeat fire finds an empty set and takes today's path for that state, with the
chief's own claim: B and C after a step B failure (T12), the delisted path (webby-0094 N1) after a step C
failure.
`grant.revoke.refused` with reason `internal` stays only for an unexpected throw inside step G.
### The delete (sage-2575)
`lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
grant set.
- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
closes the request with that code, well inside its 10 s.
The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
> Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
> Raise the delete again when the inbox says the revoke is done.
- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
(condition 2, the 3e lock). No second message goes out either: the first one still applies.
- **The job ends.** The human gets one inbox message (condition 3).
- If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
- If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
describes. It records `grant.revoke.refused {job, ...}`.
- **The delete raised after a completed job** finds an empty set and goes as today.
- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
reason in a message:
> Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
> the delete again when the inbox says it's done, or clear <grant> by hand first.
It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
it. A delete after the human cleared the grant by hand doesn't find an empty set if a manual revoke left a
line (Q1). The leftover line makes the set non-empty, so that delete starts a job for the line removal
and refuses `grants-revoking` (d6, N6, and T3b's second case). If nothing is left, the set is empty and the
delete goes on.
The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
widen the row into the launcher for a one-time wait.
While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
start check.
### Why not in `beforeWrite` or `revoke()`
Both are synchronous and run inside the organization writer, after its checks. A host `remove` is SSH and a
Gitea revoke is HTTP, so they can't run there. The business package also has no reach into the broker's grant
tables, and shouldn't get any.
### Why after step A
A fire is the decision to stop the agent. If step G came first, a host that can't be reached would keep the
fired agent running. After step A, a refused step G leaves the same state as a failed step B today: the
instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state. The repeat
fire's delisted path (webby-0094 N1) also runs step G, since the set is read before step B on both paths.
## 3. Step G: what it revokes, and how
The set, for (business, agent), is read in one store transaction: step A's for a fire, the delete's checks
for a delete (section 2). It holds:
- each `host_grants` row of the agent;
- each host named by an open host grant decision for the agent;
- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
An open decision (d4, B1) is any grant decision raised for the agent with no outcome yet. That is: not denied,
not expired, no `grant.applied` or `grant.failed`, and no `grant.revoked` of the same (agent, host), or of the
agent's Gitea, recorded after its raise. A raised decision the human hasn't decided is open. d3 required
"approved", which left that one out.
An empty set means no step G and no new records. A fire of an instance with no grant is then exactly today's
fire, with no new events (brief tests, fourth bullet).
### 3a. A host grant: the host line first (D2)
hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
host. So the fire's host revoke reverses one step and keeps the rest the same:
1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
`credential-unknown` branch of `#revokeHost` already does).
2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
"a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
(section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
so the human can tell it from a manual revoke.
4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
fire and the delete call it with `lineFirst: true`.
The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
### 3b. The Gitea grant (D3)
The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
gone (section 5's reconcile). Two changes keep the fire closed:
- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
step G refuses with nothing changed.
- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
kept visible here, not hidden.
**A Gitea executor in flight** (d3 T6c, and B3 seen from the executor's side). Step G's Gitea part goes in
this order:
1. Record `grant.revoked {granted: false}` for each open Gitea decision (3c).
2. Await this agent's tracked Gitea executors and remote undos, if any are running. Each one ends `revoked` at
its next `#giteaRevokedSince` check, or it committed `applied` before step 1.
3. Read the `gitea_grants` row again and run the steps above.
`#giteaCloseRevoke` with `fire: true` lists the user's tokens and deletes every one. The list comes after
step 2, so it includes any token an in-flight executor created. `#giteaRuns` is a single Set today. It
becomes keyed by (business, agent) so step G can await one agent's executors alone.
### 3c. Open decisions
A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
`#giteaRevokedSince`.
This is the case where a decision is approved before the fire and executed after it (sage-2574). Both checks
look for a `grant.revoked` of the agent recorded after the decision was raised (broker.mjs:1156 and :1678 at
47a35bf43). A fire's `grant.revoked` is always later than the approval, so the executor ends the decision
whenever it runs. The fired agent keeps its roles entry, so `#hostDefinition` and `#giteaRole` still pass, and
this check is the only thing that stops the decision. T6 and T6b test it. A delete also removes the roles
entry, so the executor refuses there as well.
The same holds for a decision raised but not yet approved (B1). It's in the set, so it gets its
`grant.revoked {granted: false}`. If the human approves it later, its executor ends `revoked` (T6d). The
decision stays approvable in the inbox. Approving it does nothing but end it, which `grant.failed` reports as
it does for any revoked grant.
### 3d. A raise for a fired agent (d4, B1)
`#hostGrantBinding` and `#giteaGrantBinding` both run inside the raise transaction (broker.mjs:1539 and
:1548; the host raise's keyscan is awaited before that transaction). Both gain one check. The raise refuses
`instance-fired` if the agent is:
- firing (`#firing`);
- fired, by the writer's own rule (instance-settings.mjs, hire): a roles entry, not in `launch.instances`,
and not `launch.by`.
A rehire lists the name again, so a rehired agent can be granted again.
Because the check is in the transaction that records the decision, there's no gap. A raise that commits
before step A's transaction is in the set. One that commits after it refuses. A raise for an agent whose
delete's step G is running refuses `grant-revoke-running` (3e).
No executor recheck is added. A raise can't commit after `fire.stopped`, and every decision raised before it
is in the set and gets its `grant.revoked`.
### 3e. One step G per agent (d4, B3)
The broker keeps `#revoking`, a map from (business, agent) to the verb whose step G is running. The verb sets
it synchronously, in the same turn as the set read, before it answers. The job clears it in a `finally`.
While it's set:
| Caller | Answer |
|---|---|
| `instance.fire` | refuses `fire-in-progress` |
| `lifecycleDelete` | refuses `grants-revoking`, no second job |
| `hosts.revoke` / `grants.revoke` for the agent | refuse `grant-revoke-running` |
| grant raise for the agent | refuses `grant-revoke-running` |
| `grants.sweep` | skips this agent's entries and names it as `running` in its result |
Two fires started together (the chief retrying after a client timeout): the first takes the lock before it
answers, and the second refuses. The map is in memory. After a broker restart the job is gone and so is the
lock. The instance is still firing, or the delete still has a non-empty set, and the repeat starts a fresh
step G (section 5).
A manual revoke can't race step G, since it refuses while the lock is held. Step G's transactions still
check what they read: a `host_grants` row already gone at 3a step 3 records `grant.revoked {granted: false}`
and goes on, as hosts.revoke does today for a revoke with no record.
## 4. A refusal (D4)
Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
recorded, and the rest stay as they were. The job doesn't run step B: the organization file is unchanged, and
the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing". A delete's request was
already closed with `grants-revoking` (section 2).
The verb has already answered, so the refusal is a record, not a thrown code:
- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
'instance.fire' | 'lifecycle.delete', by}`.
- `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
- `grant-revoke-failed` is the code in the event and in the inbox message.
A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139). It names the instance,
the grant and the reason, with the manual way out beside the retry (N2):
> Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
> is reachable. If <host> won't come back, run `mosaic hosts revoke <agent> <host>` first: the grant record
> goes, the line is named as left, and the next fire completes and names that line (section 5).
The delete's message has the same shape, with "raise the delete again". For Gitea, the way out is
`mosaic grants sweep gitea` once Gitea is back.
A job that throws something unexpected inside step G records `grant.revoke.refused` with reason `internal`,
sends the same message, and releases the lock. There's no unhandled rejection. A failure at step B or C is
`fire.failed` (section 2).
**A restart mid-job (d6, N4).** The lock and the job are in memory, so a restart ends them with no end
record. A fire would stay "firing" until the chief happened to repeat it. A delete's request is already
closed, so no one would hear anything. Boot closes that gap:
- A job is a start record (`fire.revoking` or `grants.revoking`) whose `job` has no end record. The end
records are `fire.done`, `fire.failed`, `grants.revoked.done`, `grant.revoke.refused` and the new
`grants.job.interrupted`.
- When the broker starts, for each such job of each business, it records `grants.job.interrupted {job,
agent, verb}` and sends the human one inbox message:
> <Fire|Delete> of <agent>: job <job> was interrupted by a bus restart. Some grants may be revoked. Run the
> fire again, or raise the delete again. The repeat goes on from what's left.
- Recording the end record in the same transaction as the message means a second boot finds nothing to
send.
- Boot reads only events, with no network call, so it adds nothing to the start time beyond one query.
## 5. A repeat fire (D5)
A repeat runs the whole fire again: the checks, step A (nothing left to stop), the set read, then step G, B
and C in the job. A repeat while the job runs refuses `fire-in-progress` (3e). A repeat delete works the same
way through the human's new request.
- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
- A host grant that refused still has its row, so it's retried from 3a step 1.
- Gitea (d4, N1): if the agent has entries in `#giteaLeft` or `#giteaOpenRevokes`, step G retries only those.
It calls `#giteaCloseRevoke` for each, as `#sweepGitea` does per entry. It doesn't run the business-wide
sweep, which would retry every agent's entries and settle every open rotation under the chief's name. Any
entry still left for this agent refuses `gitea-left`. An entry left by an earlier manual
`mosaic grants revoke` of the same agent blocks the fire the same way. That's intended: no grant outlives
its instance, whichever verb started its revoke. These entries count toward a non-empty set (section 2,
fire step 3), so they go to the job and never run inside the verb.
- A host line left by an earlier manual `hosts.revoke` doesn't block the fire (Q1 below). These are the
agent's `grant.revoke.closed` records with `hostLineLeft` and no later line removal for that tag. For each
one whose host record still exists, step G sends `remove <user> <tag>` once. On success it records
`grant.revoke.closed {revoke: <the old revoke id>, hostLine: 'removed', fire: true}`, so a later fire doesn't
find it again. On failure it records nothing new and the fire goes on. The host and the tag go in
`grantsLeft` (section 6). These leftover lines also make the set non-empty, so the retry runs in the job,
inside B2's answer (dewey-0823 accepted the single retry on that condition).
## Rulings
- **Q1 (sage-2573):** a leftover line doesn't block the fire. If the host is reachable, the fire tries the
removal once, because a fired agent's key line still on a host is the exact risk this row closes. If that
try fails, the fire doesn't refuse. It names the host and the leftover line in the result and in the event
the human sees. The human accepted that residue when their manual `hosts.revoke` left it, so it shouldn't
stop a fire forever on a dead host. Dewey may push back on the retry if it widens step G more than it's
worth.
- **Q2 (sage-2573):** a delete refused at step G closes its request through the launcher's normal refusal
path, and the human raises it again. A request left taken would strand, which is the shape DEFERRED already
records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
`grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
does. T3b tests both.
- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
with a test:
1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
delete again when the inbox says done. A bare code isn't enough (T3).
2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
reason and doesn't complete over a live grant (T3b).
4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
(T2).
## 6. The result and the event (D6)
- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
`fire.revoking {job, grants}` (section 2).
- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
So the event the human sees names them.
- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
- **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
## 7. What doesn't change (brief item 4)
- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers and the same records. Their refusals stay
the same (`tester-dev-only` included) with one addition from section 3e (d6, N5): while a step G job holds
the agent's lock, `hosts.revoke` and `grants.revoke` for that agent refuse `grant-revoke-running`, and
`grants.sweep` skips the agent and names it as `running`. hosts.revoke's own order stays local first.
- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
existing grant-evidence shape.
- `revoke()` in runtime.mjs and the organization writer don't change.
- `server.mjs`, `process.mjs`, `cli/src/host.mjs` and `cli/src/launcher.mjs` don't change. Both verbs stay
synchronous. The launcher already closes a request with whatever code `lifecycleDelete` refuses.
- New codes: `fire-in-progress`, `grants-revoking`, `grant-revoke-running`, `instance-fired`, and
`grant-revoke-failed` in events and messages. New event types: `fire.revoking`, `grants.revoking`,
`grants.revoked.done`, `grant.revoke.refused`, `fire.failed` and `grants.job.interrupted`.
- The broker's start gains one events query for interrupted jobs (section 4, N4). `lifecycleDone` checks a refusal against `LIFECYCLE_CODE =
/^[a-z-]{1,64}$/` (broker.mjs:149), and `grants-revoking` matches it.
## 8. Tests (each in the owning package, whole files)
The bus tests use the fake installer and the fake Gitea from rows 235 and 236. They wait on the job with a new
trusted test API, `grantsIdle()`, which resolves once every step G job and Gitea executor has ended. It's the
same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete below is awaited that way.
- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
`credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
a record whose key file is gone.
- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
the job:
- all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
them, plus `fire: true`;
- the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
- `fire.done` names all three;
- that agent's `host.run` and Gitea use both refuse.
- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
`lifecycleDelete`).
- The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
naming the job and all three grants, and saying to raise the delete again when it's done.
- A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
message.
- After the job, the end message names the job and each grant revoked.
- A third request completes as today.
- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
- The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
`mosaic hosts revoke`.
- The file and the instance are unchanged.
- The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
reason and the new job, and the instance is not deleted.
- With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
- A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
for Gitea and the leftover line.
- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
same events counted, no inbox message.
- T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
`host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
- the file is unchanged and the instance is still firing;
- host A is revoked and recorded, and host B's row and key file are still there;
- the Gitea grant is untouched, since it comes after B in the order;
- the credentials are unrevoked.
A repeat with B reachable completes and revokes only B and Gitea.
- T4b (Q1): an earlier manual `hosts.revoke` left a line on host C. A fire with C reachable removes it and
records the close. With C unreachable, the fire completes and names C and the tag in `grantsLeft` and in
`fire.done`.
- T5: Gitea's admin token can't be checked. The job refuses with the Gitea grant intact.
- T5b: a token DELETE fails. The job refuses `gitea-left`. A repeat with Gitea back retries only this agent's
entries and completes. Another agent's left entry is still left afterwards (N1).
- T6 (sage-2574): a host grant decision approved before the fire, executed after it. The fire records
`grant.revoked {granted: false}` for (agent, host). The executor runs after `fire.done` and ends the decision
`revoked`: no `host_grants` row, no key file, no `add` sent to the host.
- T6b (sage-2574): the same for a Gitea grant decision. The executor ends it `revoked` before any Gitea call:
no `gitea_grants` row, no token file, and the fake Gitea saw no token create.
- T6c: a host or Gitea executor already in flight when step G starts.
- The host side aborts its claim (3a step 1) and ends `revoked`.
- The Gitea side: step G awaits the executor (3b), and then the fake Gitea holds no token for the user. That
includes one the executor created mid-flight.
- T6d (B1): a host decision and a Gitea decision, raised before the fire and still undecided. The fire puts
both in the set. The human approves both after `fire.done`, and both executors end `revoked`, with nothing
applied.
- T6e (B1): a host raise and a Gitea raise refuse `instance-fired`:
- while the instance is firing;
- after `fire.done`.
After a rehire of the same name, a raise goes through. While a delete's job runs, a raise refuses
`grant-revoke-running`.
- T7: an instance with no grant fires exactly as today, synchronously. Same result keys, no `revoking`, and
the same events, counted.
- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
- T9 (B3): with the job held on a remove:
- a second fire refuses `fire-in-progress`;
- a delete of the same agent refuses `grants-revoking` and starts no second job;
- `hosts.revoke` and `grants.revoke` for the agent refuse `grant-revoke-running`;
- `grants.sweep` names the agent as `running`.
After a release, one set of `grant.revoked` records exists per grant.
- T10 (B2): the fake installer holds a remove and the fake Gitea holds a DELETE, neither answering until
released.
- `instance.fire` and `lifecycleDelete` both answer before either is released. The test checks this with a
timer, not a wall-clock bound.
- Meanwhile the broker answers another verb from another session.
- After release, the jobs complete.
- A cli test drives the launcher's delete pass against a bus whose remove is held. The pass closes the
request `grants-revoking` and the host channel stays up.
- T11 (d6, C1): a fire whose job is held on a remove. While it's held, the chief's run ends and its claim is
released, and processing is halted. After the release, the job completes: `fire.done` is recorded with the
job, the consumed `action.allowed` event is there, a second use of the same authority refuses
`decision-consumed`, and the instance is out of the file. No `fire.failed` and no `grant.revoke.refused`.
- T12 (d6, C1): a fire with a grant whose step B refuses in the job. The test uses T1's setup, where
`revoke()` refuses `credential-revoke-failed`. After the job:
- `fire.failed` carries the job, step `B` and `credential-revoke-failed`, and the inbox message names the
job, the revoked grants, the step and the code;
- there's no `grant.revoke.refused` and no `fire.done`;
- the instance is still firing and still in the file.
With the cause cleared, a repeat fire finds an empty set, runs B and C synchronously and completes.
- T13 (d6, N4): a fire and a delete, each with its job held, then the broker is closed and a new one opened
on the same data root. The new broker records `grants.job.interrupted` for both jobs and sends two inbox
messages, one saying to fire again and one to raise the delete again. A third broker on the same root sends
nothing more. A job that ended before the restart gets no interrupted record.
Files (N3). Every file listed runs whole in the manifest round, touched or not:
- New: `packages/bus/tests/fire-revokes-grants.test.mjs` holds T2-T13.
- Changed:
- `grant-host.test.mjs` and `gitea-grant.test.mjs`: T1's pins, and the T8 regressions next to their
existing cases;
- `packages/cli/tests/lifecycle.test.mjs`: the T10 launcher pass;
- `fake-gitea-api.mjs` and `host-fixtures.mjs`, if T10's holds need a hook there.
- Run whole because they call `instance.fire` or `lifecycleDelete` at 47a35bf43:
- bus: `chief-of-staff`, `lifecycle-revoke`, `lifecycle-verbs`, `reload`;
- cli: `lifecycle`, `instances`;
- harness: `tools`.
- Also run: `grant-executor.test.mjs` (the executors' `#giteaRuns` change) and `gitea-grant-docker.test.mjs`
(brief gate).
- `server.mjs`, `process.mjs`, `host.mjs` and `launcher.mjs` are unchanged, so no test changes there for this
row.
## 9. Mutants
- M1: step G skips the host revokes. Killed by T2.
- M2: step G skips the Gitea revoke. Killed by T2.
- M3: a refused grant revoke is ignored and the job goes on to step B. Killed by T4.
- M4: `lifecycleDelete` skips step G. Killed by T3.
- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
- M7: open decisions left out of the set. Killed by T6 and T6b.
- M8: step G skips the leftover-line retry. Killed by T4b.
- M9: a failed leftover-line retry refuses the fire. Killed by T4b.
- M10: the set takes only approved decisions (d3's rule). Killed by T6d.
- M11: the raise bindings skip the fired check. Killed by T6e.
- M12: no `#revoking` lock. Killed by T9.
- M13: the verb awaits the job before it answers. Killed by T10.
- M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
- M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
- M16: the delete's first refusal sends no start message. Killed by T3.
- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
- M19: the job's step C uses `#consumeAuthority`, with the holder and processing checks. Killed by T11.
- M20: a step B or C failure in the job is recorded as `grant.revoke.refused internal`. Killed by T12.
- M21: boot skips the interrupted-job check. Killed by T13.
## 10. Order and rebase
Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
## Runs
(after the gate)
## Mutants
(after the gate)