Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
97535587b2 | ||
|
|
59142e9fbb | ||
|
|
41983e2659 | ||
|
|
36c2d98b81 | ||
|
|
c746ce378c | ||
|
|
dfebd16cc1 |
@@ -0,0 +1,610 @@
|
||||
# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
|
||||
|
||||
Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d6 for Dewey's approval. No code until it is
|
||||
approved. The results sections are filled in after the runs on Sage's gate.
|
||||
|
||||
History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
|
||||
approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-0823):
|
||||
- B1: the set takes every grant decision with no outcome, and a raise refuses a fired or firing agent.
|
||||
- B2: step G runs detached from the verb, so neither caller waits on the network.
|
||||
- B3: one step G per agent, with named refusals for everything that could race it.
|
||||
- N1-N3 are folded in.
|
||||
|
||||
d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
|
||||
answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
|
||||
|
||||
d6 answers Dewey's CHANGES on d5 (dewey-0825):
|
||||
- C1: the job's step C records authority without the chief's claim (`#authorityFor`, as a landing's record
|
||||
does), processing-halted doesn't hold the job, and a step B or C failure records `fire.failed {job, step,
|
||||
code}` with its own message. Tests T11 and T12, mutants M19 and M20.
|
||||
- N4: boot closes a job a restart interrupted, with one message. Test T13, mutant M21.
|
||||
- N5: section 7 names the refusals the manual verbs gain.
|
||||
- N6: the set counts Q1 leftover lines, in fire step 3 and in the delete text.
|
||||
|
||||
## Sources
|
||||
|
||||
- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
|
||||
mos-1271, Vikunja 524).
|
||||
- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
|
||||
`gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
|
||||
- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
|
||||
goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
|
||||
- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
|
||||
|
||||
## 1. Why: what happens today (read from the code, to be pinned by a test before the change)
|
||||
|
||||
In short: today a delete of a granted instance unlinks the key files but leaves both grant records, the host
|
||||
lines and the Gitea token live, and a fire refuses `credential-revoke-failed` (sage-2573 asked for this here
|
||||
and on #1786). The details:
|
||||
|
||||
- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
|
||||
agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
|
||||
`<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
|
||||
(gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
|
||||
step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
|
||||
`mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
|
||||
roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
|
||||
`gitea_grants` change and no Gitea call.
|
||||
- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
|
||||
`host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
|
||||
stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
|
||||
The next start then finds a grant record whose key file is gone. What the start does with that isn't read
|
||||
yet. The T1 test below pins it.
|
||||
- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
|
||||
grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
|
||||
still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
|
||||
`gitea-not-declared`.
|
||||
- **A raise after the fire** (dewey-0823 B1). `#raiseGrant` (broker.mjs:1529) checks only that the caller is
|
||||
the chief, and the grant bindings check only the roles entry. So the chief can raise a new grant for a fired
|
||||
agent, and if the human approves it, it applies.
|
||||
|
||||
## 2. Where the grant revokes sit (D1, rewritten in d4 for B2)
|
||||
|
||||
d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wait that long:
|
||||
|
||||
- **Delete.** The launcher's `host.op` waits `REQUEST_TIMEOUT_MS = 10000` (cli/src/host.mjs:31). On a timeout,
|
||||
`breakChannel` stops the whole bus host. One SSH remove can take 30 s (host-grant.mjs:231, :311), and each
|
||||
Gitea call 15 s (`CALL_MS`, gitea-grant.mjs:38).
|
||||
- **Fire.** server.mjs answers a verb that isn't on its async list synchronously (:154), and the async list
|
||||
gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
|
||||
plus the Gitea calls can pass both limits.
|
||||
|
||||
So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
|
||||
timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
|
||||
broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
|
||||
the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
|
||||
|
||||
Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
|
||||
and every inbox message about it carry that id. The human and the chief can then match a later outcome to
|
||||
the request that started it.
|
||||
|
||||
### The fire
|
||||
|
||||
1. `write('instance.fireCheck')`, as today.
|
||||
2. Step A, as today, with one addition. In the same transaction, after `fire.stopped`, the grant set is read
|
||||
(section 3). Reading it there means a grant raise either committed before step A, and is in the set, or
|
||||
comes after `fire.stopped` and refuses (section 3d).
|
||||
3. If the set is empty, steps B and C run synchronously, exactly as today. A fire with no grant answers
|
||||
today's result with today's events (T7). The set counts everything step G would touch (d6, N6): grant
|
||||
rows, open decisions, the agent's Gitea leftovers (section 5) and its Q1 leftover host lines (section 5).
|
||||
Any of them makes the set non-empty and goes to the job.
|
||||
4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
|
||||
step A's transaction, starts the job, and answers at once:
|
||||
`{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
|
||||
step B hasn't run.
|
||||
5. The job runs step G, then step B (`write('instance.fire')`) and step C (`#recordAuthority`, below, then
|
||||
`fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends, `fire.failed`
|
||||
included.
|
||||
|
||||
What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
|
||||
`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
|
||||
- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
|
||||
- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
|
||||
- A repeat after a refused step G runs step G again (section 5).
|
||||
|
||||
The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
|
||||
inbox message in section 4.
|
||||
|
||||
**Steps B and C in the job (d6, C1).** d4 left "step C without the chief's run" as a build-time check, and
|
||||
Dewey checked it: it fails.
|
||||
- `#consumeAuthority` calls `#checkAuthority` (broker.mjs:2852). That runs `#processingWork()` and
|
||||
`#agent(s)`, and `#agent` refuses `not-holder` once the chief's run no longer holds its claim (:749).
|
||||
- After the verb answers, the job can take a minute or more. If the chief's run ends meanwhile, step C would
|
||||
throw after step B had delisted. The instance would be left delisted with no `fire.done`.
|
||||
|
||||
So the verb does the full check, and the job only records:
|
||||
- The verb runs `#checkAuthority(s, action, authority)`, as today, before step A. The chief holds its claim
|
||||
then.
|
||||
- The job's step C runs `#authorityFor(s, action, authority)`, the check without the holder test. It then
|
||||
runs the `#spent` check and the `action.allowed` event, as `#consumeAuthority` does. That's the shape a
|
||||
landing's record already uses (`recordPublish`, broker.mjs:3174, row 137): the run that asked held its
|
||||
claim when it asked, and may have ended since. It's one small private method, `#recordAuthority`, shared
|
||||
with nothing else.
|
||||
- The job keeps a copy of the session fields `#authorityFor` and `#event` read (business, role, thread,
|
||||
run). It doesn't keep the live session object.
|
||||
- `processing-halted` doesn't hold the job. A halt stops agent work. Step G only removes access, and steps
|
||||
B and C finish a decision the chief made while processing ran. Holding them would leave a delisted or
|
||||
half-revoked instance "firing" for the length of the halt. The verb itself still refuses during a halt,
|
||||
through `#checkAuthority`, as today.
|
||||
|
||||
**A step B or C failure in the job** gets its own record, not `grant.revoke.refused`, since every grant in the
|
||||
set was revoked by then:
|
||||
- It records `fire.failed {job, step: 'B' | 'C', code}`. `code` is the thrown code, for example the writer's
|
||||
`credential-revoke-failed`, or `decision-consumed`.
|
||||
- It sends an inbox message:
|
||||
|
||||
> Fire of <agent>: job <job> revoked its grants (<list>) but stopped at step <B|C> (<code>). The instance
|
||||
> is still firing. Run the fire again once <code> is cleared.
|
||||
|
||||
- The instance stays firing. A repeat fire finds an empty set and takes today's path for that state, with the
|
||||
chief's own claim: B and C after a step B failure (T12), the delisted path (webby-0094 N1) after a step C
|
||||
failure.
|
||||
|
||||
`grant.revoke.refused` with reason `internal` stays only for an unexpected throw inside step G.
|
||||
|
||||
### The delete (sage-2575)
|
||||
|
||||
`lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
|
||||
grant set.
|
||||
|
||||
- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
|
||||
- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
|
||||
verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
|
||||
closes the request with that code, well inside its 10 s.
|
||||
|
||||
The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
|
||||
|
||||
> Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
|
||||
> Raise the delete again when the inbox says the revoke is done.
|
||||
|
||||
- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
|
||||
(condition 2, the 3e lock). No second message goes out either: the first one still applies.
|
||||
- **The job ends.** The human gets one inbox message (condition 3).
|
||||
- If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
|
||||
delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
|
||||
- If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
|
||||
describes. It records `grant.revoke.refused {job, ...}`.
|
||||
- **The delete raised after a completed job** finds an empty set and goes as today.
|
||||
- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
|
||||
the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
|
||||
reason in a message:
|
||||
|
||||
> Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
|
||||
> the delete again when the inbox says it's done, or clear <grant> by hand first.
|
||||
|
||||
It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
|
||||
end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
|
||||
it. A delete after the human cleared the grant by hand doesn't find an empty set if a manual revoke left a
|
||||
line (Q1). The leftover line makes the set non-empty, so that delete starts a job for the line removal
|
||||
and refuses `grants-revoking` (d6, N6, and T3b's second case). If nothing is left, the set is empty and the
|
||||
delete goes on.
|
||||
|
||||
The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
|
||||
cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
|
||||
launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
|
||||
widen the row into the launcher for a one-time wait.
|
||||
|
||||
While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
|
||||
harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
|
||||
start check.
|
||||
|
||||
### Why not in `beforeWrite` or `revoke()`
|
||||
|
||||
Both are synchronous and run inside the organization writer, after its checks. A host `remove` is SSH and a
|
||||
Gitea revoke is HTTP, so they can't run there. The business package also has no reach into the broker's grant
|
||||
tables, and shouldn't get any.
|
||||
|
||||
### Why after step A
|
||||
|
||||
A fire is the decision to stop the agent. If step G came first, a host that can't be reached would keep the
|
||||
fired agent running. After step A, a refused step G leaves the same state as a failed step B today: the
|
||||
instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state. The repeat
|
||||
fire's delisted path (webby-0094 N1) also runs step G, since the set is read before step B on both paths.
|
||||
|
||||
## 3. Step G: what it revokes, and how
|
||||
|
||||
The set, for (business, agent), is read in one store transaction: step A's for a fire, the delete's checks
|
||||
for a delete (section 2). It holds:
|
||||
|
||||
- each `host_grants` row of the agent;
|
||||
- each host named by an open host grant decision for the agent;
|
||||
- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
|
||||
|
||||
An open decision (d4, B1) is any grant decision raised for the agent with no outcome yet. That is: not denied,
|
||||
not expired, no `grant.applied` or `grant.failed`, and no `grant.revoked` of the same (agent, host), or of the
|
||||
agent's Gitea, recorded after its raise. A raised decision the human hasn't decided is open. d3 required
|
||||
"approved", which left that one out.
|
||||
|
||||
An empty set means no step G and no new records. A fire of an instance with no grant is then exactly today's
|
||||
fire, with no new events (brief tests, fourth bullet).
|
||||
|
||||
### 3a. A host grant: the host line first (D2)
|
||||
|
||||
hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
|
||||
`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
|
||||
record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
|
||||
host. So the fire's host revoke reverses one step and keeps the rest the same:
|
||||
|
||||
1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
|
||||
`credential-unknown` branch of `#revokeHost` already does).
|
||||
2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
|
||||
"a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
|
||||
decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
|
||||
(section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
|
||||
3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
|
||||
host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
|
||||
so the human can tell it from a manual revoke.
|
||||
4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
|
||||
5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
|
||||
|
||||
This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
|
||||
hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
|
||||
fire and the delete call it with `lineFirst: true`.
|
||||
|
||||
The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
|
||||
section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
|
||||
"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
|
||||
|
||||
### 3b. The Gitea grant (D3)
|
||||
|
||||
The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
|
||||
runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
|
||||
`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
|
||||
`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
|
||||
|
||||
The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
|
||||
name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
|
||||
gone (section 5's reconcile). Two changes keep the fire closed:
|
||||
|
||||
- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
|
||||
step G refuses with nothing changed.
|
||||
- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
|
||||
token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
|
||||
forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
|
||||
gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
|
||||
kept visible here, not hidden.
|
||||
|
||||
**A Gitea executor in flight** (d3 T6c, and B3 seen from the executor's side). Step G's Gitea part goes in
|
||||
this order:
|
||||
1. Record `grant.revoked {granted: false}` for each open Gitea decision (3c).
|
||||
2. Await this agent's tracked Gitea executors and remote undos, if any are running. Each one ends `revoked` at
|
||||
its next `#giteaRevokedSince` check, or it committed `applied` before step 1.
|
||||
3. Read the `gitea_grants` row again and run the steps above.
|
||||
|
||||
`#giteaCloseRevoke` with `fire: true` lists the user's tokens and deletes every one. The list comes after
|
||||
step 2, so it includes any token an in-flight executor created. `#giteaRuns` is a single Set today. It
|
||||
becomes keyed by (business, agent) so step G can await one agent's executors alone.
|
||||
|
||||
### 3c. Open decisions
|
||||
|
||||
A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
|
||||
`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
|
||||
`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
|
||||
`#giteaRevokedSince`.
|
||||
|
||||
This is the case where a decision is approved before the fire and executed after it (sage-2574). Both checks
|
||||
look for a `grant.revoked` of the agent recorded after the decision was raised (broker.mjs:1156 and :1678 at
|
||||
47a35bf43). A fire's `grant.revoked` is always later than the approval, so the executor ends the decision
|
||||
whenever it runs. The fired agent keeps its roles entry, so `#hostDefinition` and `#giteaRole` still pass, and
|
||||
this check is the only thing that stops the decision. T6 and T6b test it. A delete also removes the roles
|
||||
entry, so the executor refuses there as well.
|
||||
|
||||
The same holds for a decision raised but not yet approved (B1). It's in the set, so it gets its
|
||||
`grant.revoked {granted: false}`. If the human approves it later, its executor ends `revoked` (T6d). The
|
||||
decision stays approvable in the inbox. Approving it does nothing but end it, which `grant.failed` reports as
|
||||
it does for any revoked grant.
|
||||
|
||||
### 3d. A raise for a fired agent (d4, B1)
|
||||
|
||||
`#hostGrantBinding` and `#giteaGrantBinding` both run inside the raise transaction (broker.mjs:1539 and
|
||||
:1548; the host raise's keyscan is awaited before that transaction). Both gain one check. The raise refuses
|
||||
`instance-fired` if the agent is:
|
||||
- firing (`#firing`);
|
||||
- fired, by the writer's own rule (instance-settings.mjs, hire): a roles entry, not in `launch.instances`,
|
||||
and not `launch.by`.
|
||||
|
||||
A rehire lists the name again, so a rehired agent can be granted again.
|
||||
|
||||
Because the check is in the transaction that records the decision, there's no gap. A raise that commits
|
||||
before step A's transaction is in the set. One that commits after it refuses. A raise for an agent whose
|
||||
delete's step G is running refuses `grant-revoke-running` (3e).
|
||||
|
||||
No executor recheck is added. A raise can't commit after `fire.stopped`, and every decision raised before it
|
||||
is in the set and gets its `grant.revoked`.
|
||||
|
||||
### 3e. One step G per agent (d4, B3)
|
||||
|
||||
The broker keeps `#revoking`, a map from (business, agent) to the verb whose step G is running. The verb sets
|
||||
it synchronously, in the same turn as the set read, before it answers. The job clears it in a `finally`.
|
||||
While it's set:
|
||||
|
||||
| Caller | Answer |
|
||||
|---|---|
|
||||
| `instance.fire` | refuses `fire-in-progress` |
|
||||
| `lifecycleDelete` | refuses `grants-revoking`, no second job |
|
||||
| `hosts.revoke` / `grants.revoke` for the agent | refuse `grant-revoke-running` |
|
||||
| grant raise for the agent | refuses `grant-revoke-running` |
|
||||
| `grants.sweep` | skips this agent's entries and names it as `running` in its result |
|
||||
|
||||
Two fires started together (the chief retrying after a client timeout): the first takes the lock before it
|
||||
answers, and the second refuses. The map is in memory. After a broker restart the job is gone and so is the
|
||||
lock. The instance is still firing, or the delete still has a non-empty set, and the repeat starts a fresh
|
||||
step G (section 5).
|
||||
|
||||
A manual revoke can't race step G, since it refuses while the lock is held. Step G's transactions still
|
||||
check what they read: a `host_grants` row already gone at 3a step 3 records `grant.revoked {granted: false}`
|
||||
and goes on, as hosts.revoke does today for a revoke with no record.
|
||||
|
||||
## 4. A refusal (D4)
|
||||
|
||||
Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
|
||||
recorded, and the rest stay as they were. The job doesn't run step B: the organization file is unchanged, and
|
||||
the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing". A delete's request was
|
||||
already closed with `grants-revoking` (section 2).
|
||||
|
||||
The verb has already answered, so the refusal is a record, not a thrown code:
|
||||
- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
|
||||
'instance.fire' | 'lifecycle.delete', by}`.
|
||||
- `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
|
||||
- `grant-revoke-failed` is the code in the event and in the inbox message.
|
||||
|
||||
A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139). It names the instance,
|
||||
the grant and the reason, with the manual way out beside the retry (N2):
|
||||
|
||||
> Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
|
||||
> is reachable. If <host> won't come back, run `mosaic hosts revoke <agent> <host>` first: the grant record
|
||||
> goes, the line is named as left, and the next fire completes and names that line (section 5).
|
||||
|
||||
The delete's message has the same shape, with "raise the delete again". For Gitea, the way out is
|
||||
`mosaic grants sweep gitea` once Gitea is back.
|
||||
|
||||
A job that throws something unexpected inside step G records `grant.revoke.refused` with reason `internal`,
|
||||
sends the same message, and releases the lock. There's no unhandled rejection. A failure at step B or C is
|
||||
`fire.failed` (section 2).
|
||||
|
||||
**A restart mid-job (d6, N4).** The lock and the job are in memory, so a restart ends them with no end
|
||||
record. A fire would stay "firing" until the chief happened to repeat it. A delete's request is already
|
||||
closed, so no one would hear anything. Boot closes that gap:
|
||||
- A job is a start record (`fire.revoking` or `grants.revoking`) whose `job` has no end record. The end
|
||||
records are `fire.done`, `fire.failed`, `grants.revoked.done`, `grant.revoke.refused` and the new
|
||||
`grants.job.interrupted`.
|
||||
- When the broker starts, for each such job of each business, it records `grants.job.interrupted {job,
|
||||
agent, verb}` and sends the human one inbox message:
|
||||
|
||||
> <Fire|Delete> of <agent>: job <job> was interrupted by a bus restart. Some grants may be revoked. Run the
|
||||
> fire again, or raise the delete again. The repeat goes on from what's left.
|
||||
|
||||
- Recording the end record in the same transaction as the message means a second boot finds nothing to
|
||||
send.
|
||||
- Boot reads only events, with no network call, so it adds nothing to the start time beyond one query.
|
||||
|
||||
## 5. A repeat fire (D5)
|
||||
|
||||
A repeat runs the whole fire again: the checks, step A (nothing left to stop), the set read, then step G, B
|
||||
and C in the job. A repeat while the job runs refuses `fire-in-progress` (3e). A repeat delete works the same
|
||||
way through the human's new request.
|
||||
|
||||
- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
|
||||
- A host grant that refused still has its row, so it's retried from 3a step 1.
|
||||
- Gitea (d4, N1): if the agent has entries in `#giteaLeft` or `#giteaOpenRevokes`, step G retries only those.
|
||||
It calls `#giteaCloseRevoke` for each, as `#sweepGitea` does per entry. It doesn't run the business-wide
|
||||
sweep, which would retry every agent's entries and settle every open rotation under the chief's name. Any
|
||||
entry still left for this agent refuses `gitea-left`. An entry left by an earlier manual
|
||||
`mosaic grants revoke` of the same agent blocks the fire the same way. That's intended: no grant outlives
|
||||
its instance, whichever verb started its revoke. These entries count toward a non-empty set (section 2,
|
||||
fire step 3), so they go to the job and never run inside the verb.
|
||||
- A host line left by an earlier manual `hosts.revoke` doesn't block the fire (Q1 below). These are the
|
||||
agent's `grant.revoke.closed` records with `hostLineLeft` and no later line removal for that tag. For each
|
||||
one whose host record still exists, step G sends `remove <user> <tag>` once. On success it records
|
||||
`grant.revoke.closed {revoke: <the old revoke id>, hostLine: 'removed', fire: true}`, so a later fire doesn't
|
||||
find it again. On failure it records nothing new and the fire goes on. The host and the tag go in
|
||||
`grantsLeft` (section 6). These leftover lines also make the set non-empty, so the retry runs in the job,
|
||||
inside B2's answer (dewey-0823 accepted the single retry on that condition).
|
||||
|
||||
## Rulings
|
||||
|
||||
- **Q1 (sage-2573):** a leftover line doesn't block the fire. If the host is reachable, the fire tries the
|
||||
removal once, because a fired agent's key line still on a host is the exact risk this row closes. If that
|
||||
try fails, the fire doesn't refuse. It names the host and the leftover line in the result and in the event
|
||||
the human sees. The human accepted that residue when their manual `hosts.revoke` left it, so it shouldn't
|
||||
stop a fire forever on a dead host. Dewey may push back on the retry if it widens step G more than it's
|
||||
worth.
|
||||
- **Q2 (sage-2573):** a delete refused at step G closes its request through the launcher's normal refusal
|
||||
path, and the human raises it again. A request left taken would strand, which is the shape DEFERRED already
|
||||
records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
|
||||
`grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
|
||||
does. T3b tests both.
|
||||
- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
|
||||
revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
|
||||
done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
|
||||
with a test:
|
||||
1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
|
||||
delete again when the inbox says done. A bare code isn't enough (T3).
|
||||
2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
|
||||
3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
|
||||
manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
|
||||
reason and doesn't complete over a live grant (T3b).
|
||||
4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
|
||||
5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
|
||||
(T2).
|
||||
|
||||
## 6. The result and the event (D6)
|
||||
|
||||
- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
|
||||
`fire.revoking {job, grants}` (section 2).
|
||||
- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
|
||||
'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
|
||||
So the event the human sees names them.
|
||||
- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
|
||||
verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
|
||||
- **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
|
||||
|
||||
## 7. What doesn't change (brief item 4)
|
||||
|
||||
- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers and the same records. Their refusals stay
|
||||
the same (`tester-dev-only` included) with one addition from section 3e (d6, N5): while a step G job holds
|
||||
the agent's lock, `hosts.revoke` and `grants.revoke` for that agent refuse `grant-revoke-running`, and
|
||||
`grants.sweep` skips the agent and names it as `running`. hosts.revoke's own order stays local first.
|
||||
- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
|
||||
existing grant-evidence shape.
|
||||
- `revoke()` in runtime.mjs and the organization writer don't change.
|
||||
- `server.mjs`, `process.mjs`, `cli/src/host.mjs` and `cli/src/launcher.mjs` don't change. Both verbs stay
|
||||
synchronous. The launcher already closes a request with whatever code `lifecycleDelete` refuses.
|
||||
- New codes: `fire-in-progress`, `grants-revoking`, `grant-revoke-running`, `instance-fired`, and
|
||||
`grant-revoke-failed` in events and messages. New event types: `fire.revoking`, `grants.revoking`,
|
||||
`grants.revoked.done`, `grant.revoke.refused`, `fire.failed` and `grants.job.interrupted`.
|
||||
- The broker's start gains one events query for interrupted jobs (section 4, N4). `lifecycleDone` checks a refusal against `LIFECYCLE_CODE =
|
||||
/^[a-z-]{1,64}$/` (broker.mjs:149), and `grants-revoking` matches it.
|
||||
|
||||
## 8. Tests (each in the owning package, whole files)
|
||||
|
||||
The bus tests use the fake installer and the fake Gitea from rows 235 and 236. They wait on the job with a new
|
||||
trusted test API, `grantsIdle()`, which resolves once every step G job and Gitea executor has ended. It's the
|
||||
same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete below is awaited that way.
|
||||
|
||||
- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
|
||||
`credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
|
||||
a record whose key file is gone.
|
||||
- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
|
||||
with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
|
||||
the job:
|
||||
- all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
|
||||
them, plus `fire: true`;
|
||||
- the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
|
||||
- `fire.done` names all three;
|
||||
- that agent's `host.run` and Gitea use both refuse.
|
||||
- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
|
||||
`lifecycleDelete`).
|
||||
- The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
|
||||
naming the job and all three grants, and saying to raise the delete again when it's done.
|
||||
- A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
|
||||
message.
|
||||
- After the job, the end message names the job and each grant revoked.
|
||||
- A third request completes as today.
|
||||
- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
|
||||
- The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
|
||||
`mosaic hosts revoke`.
|
||||
- The file and the instance are unchanged.
|
||||
- The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
|
||||
reason and the new job, and the instance is not deleted.
|
||||
- With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
|
||||
- A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
|
||||
for Gitea and the leftover line.
|
||||
- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
|
||||
same events counted, no inbox message.
|
||||
- T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
|
||||
`host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
|
||||
- the file is unchanged and the instance is still firing;
|
||||
- host A is revoked and recorded, and host B's row and key file are still there;
|
||||
- the Gitea grant is untouched, since it comes after B in the order;
|
||||
- the credentials are unrevoked.
|
||||
A repeat with B reachable completes and revokes only B and Gitea.
|
||||
- T4b (Q1): an earlier manual `hosts.revoke` left a line on host C. A fire with C reachable removes it and
|
||||
records the close. With C unreachable, the fire completes and names C and the tag in `grantsLeft` and in
|
||||
`fire.done`.
|
||||
- T5: Gitea's admin token can't be checked. The job refuses with the Gitea grant intact.
|
||||
- T5b: a token DELETE fails. The job refuses `gitea-left`. A repeat with Gitea back retries only this agent's
|
||||
entries and completes. Another agent's left entry is still left afterwards (N1).
|
||||
- T6 (sage-2574): a host grant decision approved before the fire, executed after it. The fire records
|
||||
`grant.revoked {granted: false}` for (agent, host). The executor runs after `fire.done` and ends the decision
|
||||
`revoked`: no `host_grants` row, no key file, no `add` sent to the host.
|
||||
- T6b (sage-2574): the same for a Gitea grant decision. The executor ends it `revoked` before any Gitea call:
|
||||
no `gitea_grants` row, no token file, and the fake Gitea saw no token create.
|
||||
- T6c: a host or Gitea executor already in flight when step G starts.
|
||||
- The host side aborts its claim (3a step 1) and ends `revoked`.
|
||||
- The Gitea side: step G awaits the executor (3b), and then the fake Gitea holds no token for the user. That
|
||||
includes one the executor created mid-flight.
|
||||
- T6d (B1): a host decision and a Gitea decision, raised before the fire and still undecided. The fire puts
|
||||
both in the set. The human approves both after `fire.done`, and both executors end `revoked`, with nothing
|
||||
applied.
|
||||
- T6e (B1): a host raise and a Gitea raise refuse `instance-fired`:
|
||||
- while the instance is firing;
|
||||
- after `fire.done`.
|
||||
After a rehire of the same name, a raise goes through. While a delete's job runs, a raise refuses
|
||||
`grant-revoke-running`.
|
||||
- T7: an instance with no grant fires exactly as today, synchronously. Same result keys, no `revoking`, and
|
||||
the same events, counted.
|
||||
- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
|
||||
- T9 (B3): with the job held on a remove:
|
||||
- a second fire refuses `fire-in-progress`;
|
||||
- a delete of the same agent refuses `grants-revoking` and starts no second job;
|
||||
- `hosts.revoke` and `grants.revoke` for the agent refuse `grant-revoke-running`;
|
||||
- `grants.sweep` names the agent as `running`.
|
||||
After a release, one set of `grant.revoked` records exists per grant.
|
||||
- T10 (B2): the fake installer holds a remove and the fake Gitea holds a DELETE, neither answering until
|
||||
released.
|
||||
- `instance.fire` and `lifecycleDelete` both answer before either is released. The test checks this with a
|
||||
timer, not a wall-clock bound.
|
||||
- Meanwhile the broker answers another verb from another session.
|
||||
- After release, the jobs complete.
|
||||
- A cli test drives the launcher's delete pass against a bus whose remove is held. The pass closes the
|
||||
request `grants-revoking` and the host channel stays up.
|
||||
- T11 (d6, C1): a fire whose job is held on a remove. While it's held, the chief's run ends and its claim is
|
||||
released, and processing is halted. After the release, the job completes: `fire.done` is recorded with the
|
||||
job, the consumed `action.allowed` event is there, a second use of the same authority refuses
|
||||
`decision-consumed`, and the instance is out of the file. No `fire.failed` and no `grant.revoke.refused`.
|
||||
- T12 (d6, C1): a fire with a grant whose step B refuses in the job. The test uses T1's setup, where
|
||||
`revoke()` refuses `credential-revoke-failed`. After the job:
|
||||
- `fire.failed` carries the job, step `B` and `credential-revoke-failed`, and the inbox message names the
|
||||
job, the revoked grants, the step and the code;
|
||||
- there's no `grant.revoke.refused` and no `fire.done`;
|
||||
- the instance is still firing and still in the file.
|
||||
With the cause cleared, a repeat fire finds an empty set, runs B and C synchronously and completes.
|
||||
- T13 (d6, N4): a fire and a delete, each with its job held, then the broker is closed and a new one opened
|
||||
on the same data root. The new broker records `grants.job.interrupted` for both jobs and sends two inbox
|
||||
messages, one saying to fire again and one to raise the delete again. A third broker on the same root sends
|
||||
nothing more. A job that ended before the restart gets no interrupted record.
|
||||
|
||||
Files (N3). Every file listed runs whole in the manifest round, touched or not:
|
||||
- New: `packages/bus/tests/fire-revokes-grants.test.mjs` holds T2-T13.
|
||||
- Changed:
|
||||
- `grant-host.test.mjs` and `gitea-grant.test.mjs`: T1's pins, and the T8 regressions next to their
|
||||
existing cases;
|
||||
- `packages/cli/tests/lifecycle.test.mjs`: the T10 launcher pass;
|
||||
- `fake-gitea-api.mjs` and `host-fixtures.mjs`, if T10's holds need a hook there.
|
||||
- Run whole because they call `instance.fire` or `lifecycleDelete` at 47a35bf43:
|
||||
- bus: `chief-of-staff`, `lifecycle-revoke`, `lifecycle-verbs`, `reload`;
|
||||
- cli: `lifecycle`, `instances`;
|
||||
- harness: `tools`.
|
||||
- Also run: `grant-executor.test.mjs` (the executors' `#giteaRuns` change) and `gitea-grant-docker.test.mjs`
|
||||
(brief gate).
|
||||
- `server.mjs`, `process.mjs`, `host.mjs` and `launcher.mjs` are unchanged, so no test changes there for this
|
||||
row.
|
||||
|
||||
## 9. Mutants
|
||||
|
||||
- M1: step G skips the host revokes. Killed by T2.
|
||||
- M2: step G skips the Gitea revoke. Killed by T2.
|
||||
- M3: a refused grant revoke is ignored and the job goes on to step B. Killed by T4.
|
||||
- M4: `lifecycleDelete` skips step G. Killed by T3.
|
||||
- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
|
||||
- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
|
||||
- M7: open decisions left out of the set. Killed by T6 and T6b.
|
||||
- M8: step G skips the leftover-line retry. Killed by T4b.
|
||||
- M9: a failed leftover-line retry refuses the fire. Killed by T4b.
|
||||
- M10: the set takes only approved decisions (d3's rule). Killed by T6d.
|
||||
- M11: the raise bindings skip the fired check. Killed by T6e.
|
||||
- M12: no `#revoking` lock. Killed by T9.
|
||||
- M13: the verb awaits the job before it answers. Killed by T10.
|
||||
- M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
|
||||
- M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
|
||||
- M16: the delete's first refusal sends no start message. Killed by T3.
|
||||
- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
|
||||
- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
|
||||
- M19: the job's step C uses `#consumeAuthority`, with the holder and processing checks. Killed by T11.
|
||||
- M20: a step B or C failure in the job is recorded as `grant.revoke.refused internal`. Killed by T12.
|
||||
- M21: boot skips the interrupted-job check. Killed by T13.
|
||||
|
||||
## 10. Order and rebase
|
||||
|
||||
Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
|
||||
and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
|
||||
before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
|
||||
|
||||
## Runs
|
||||
|
||||
(after the gate)
|
||||
|
||||
## Mutants
|
||||
|
||||
(after the gate)
|
||||
Reference in new issue
Block a user