932 lines
65 KiB
Markdown
932 lines
65 KiB
Markdown
# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
|
||
|
||
Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d6 for Dewey's approval. No code until it is
|
||
approved. The results sections are filled in after the runs on Sage's gate.
|
||
|
||
History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
|
||
approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-0823):
|
||
- B1: the set takes every grant decision with no outcome, and a raise refuses a fired or firing agent.
|
||
- B2: step G runs detached from the verb, so neither caller waits on the network.
|
||
- B3: one step G per agent, with named refusals for everything that could race it.
|
||
- N1-N3 are folded in.
|
||
|
||
d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
|
||
answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
|
||
|
||
d6 answers Dewey's CHANGES on d5 (dewey-0825):
|
||
- C1: the job's step C records authority without the chief's claim (`#authorityFor`, as a landing's record
|
||
does), processing-halted doesn't hold the job, and a step B or C failure records `fire.failed {job, step,
|
||
code}` with its own message. Tests T11 and T12, mutants M19 and M20.
|
||
- N4: boot closes a job a restart interrupted, with one message. Test T13, mutant M21.
|
||
- N5: section 7 names the refusals the manual verbs gain.
|
||
- N6: the set counts Q1 leftover lines, in fire step 3 and in the delete text.
|
||
|
||
## Sources
|
||
|
||
- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
|
||
mos-1271, Vikunja 524).
|
||
- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
|
||
`gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
|
||
- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
|
||
goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
|
||
- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
|
||
|
||
## 1. Why: what happens today (read from the code, to be pinned by a test before the change)
|
||
|
||
In short: today a delete of a granted instance unlinks the key files but leaves both grant records, the host
|
||
lines and the Gitea token live, and a fire refuses `credential-revoke-failed` (sage-2573 asked for this here
|
||
and on #1786). The details:
|
||
|
||
- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
|
||
agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
|
||
`<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
|
||
(gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
|
||
step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
|
||
`mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
|
||
roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
|
||
`gitea_grants` change and no Gitea call.
|
||
- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
|
||
`host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
|
||
stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
|
||
The next start then finds a grant record whose key file is gone. What the start does with that isn't read
|
||
yet. The T1 test below pins it.
|
||
- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
|
||
grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
|
||
still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
|
||
`gitea-not-declared`.
|
||
- **A raise after the fire** (dewey-0823 B1). `#raiseGrant` (broker.mjs:1529) checks only that the caller is
|
||
the chief, and the grant bindings check only the roles entry. So the chief can raise a new grant for a fired
|
||
agent, and if the human approves it, it applies.
|
||
|
||
## 2. Where the grant revokes sit (D1, rewritten in d4 for B2)
|
||
|
||
d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wait that long:
|
||
|
||
- **Delete.** The launcher's `host.op` waits `REQUEST_TIMEOUT_MS = 10000` (cli/src/host.mjs:31). On a timeout,
|
||
`breakChannel` stops the whole bus host. One SSH remove can take 30 s (host-grant.mjs:231, :311), and each
|
||
Gitea call 15 s (`CALL_MS`, gitea-grant.mjs:38).
|
||
- **Fire.** server.mjs answers a verb that isn't on its async list synchronously (:154), and the async list
|
||
gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
|
||
plus the Gitea calls can pass both limits.
|
||
|
||
So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
|
||
timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
|
||
broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
|
||
the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
|
||
|
||
Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
|
||
and every inbox message about it carry that id. The human and the chief can then match a later outcome to
|
||
the request that started it.
|
||
|
||
### The fire
|
||
|
||
1. `write('instance.fireCheck')`, as today.
|
||
2. Step A, as today, with one addition. In the same transaction, after `fire.stopped`, the grant set is read
|
||
(section 3). Reading it there means a grant raise either committed before step A, and is in the set, or
|
||
comes after `fire.stopped` and refuses (section 3d).
|
||
3. If the set is empty, steps B and C run synchronously, exactly as today. A fire with no grant answers
|
||
today's result with today's events (T7). The set counts everything step G would touch (d6, N6): grant
|
||
rows, open decisions, the agent's Gitea leftovers (section 5) and its Q1 leftover host lines (section 5).
|
||
Any of them makes the set non-empty and goes to the job.
|
||
4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
|
||
step A's transaction, starts the job, and answers at once:
|
||
`{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
|
||
step B hasn't run.
|
||
5. The job runs step G, then step B (`write('instance.fire')`) and step C (`#recordAuthority`, below, then
|
||
`fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends, `fire.failed`
|
||
included.
|
||
|
||
What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
|
||
`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
|
||
- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
|
||
- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
|
||
- A repeat after a refused step G runs step G again (section 5).
|
||
|
||
The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
|
||
inbox message in section 4.
|
||
|
||
**Steps B and C in the job (d6, C1).** d4 left "step C without the chief's run" as a build-time check, and
|
||
Dewey checked it: it fails.
|
||
- `#consumeAuthority` calls `#checkAuthority` (broker.mjs:2852). That runs `#processingWork()` and
|
||
`#agent(s)`, and `#agent` refuses `not-holder` once the chief's run no longer holds its claim (:749).
|
||
- After the verb answers, the job can take a minute or more. If the chief's run ends meanwhile, step C would
|
||
throw after step B had delisted. The instance would be left delisted with no `fire.done`.
|
||
|
||
So the verb does the full check, and the job only records:
|
||
- The verb runs `#checkAuthority(s, action, authority)`, as today, before step A. The chief holds its claim
|
||
then.
|
||
- The job's step C runs `#authorityFor(s, action, authority)`, the check without the holder test. It then
|
||
runs the `#spent` check and the `action.allowed` event, as `#consumeAuthority` does. That's the shape a
|
||
landing's record already uses (`recordPublish`, broker.mjs:3174, row 137): the run that asked held its
|
||
claim when it asked, and may have ended since. It's one small private method, `#recordAuthority`, shared
|
||
with nothing else.
|
||
- The job keeps a copy of the session fields `#authorityFor` and `#event` read (business, role, thread,
|
||
run). It doesn't keep the live session object.
|
||
- `processing-halted` doesn't hold the job. A halt stops agent work. Step G only removes access, and steps
|
||
B and C finish a decision the chief made while processing ran. Holding them would leave a delisted or
|
||
half-revoked instance "firing" for the length of the halt. The verb itself still refuses during a halt,
|
||
through `#checkAuthority`, as today.
|
||
|
||
**A step B or C failure in the job** gets its own record, not `grant.revoke.refused`, since every grant in the
|
||
set was revoked by then:
|
||
- It records `fire.failed {job, step: 'B' | 'C', code}`. `code` is the thrown code, for example the writer's
|
||
`credential-revoke-failed`, or `decision-consumed`.
|
||
- It sends an inbox message:
|
||
|
||
> Fire of <agent>: job <job> revoked its grants (<list>) but stopped at step <B|C> (<code>). The instance
|
||
> is still firing. Run the fire again once <code> is cleared.
|
||
|
||
- The instance stays firing. A repeat fire finds an empty set and takes today's path for that state, with the
|
||
chief's own claim: B and C after a step B failure (T12), the delisted path (webby-0094 N1) after a step C
|
||
failure.
|
||
|
||
`grant.revoke.refused` with reason `internal` stays only for an unexpected throw inside step G.
|
||
|
||
### The delete (sage-2575)
|
||
|
||
`lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
|
||
grant set.
|
||
|
||
- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
|
||
- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
|
||
verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
|
||
closes the request with that code, well inside its 10 s.
|
||
|
||
The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
|
||
|
||
> Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
|
||
> Raise the delete again when the inbox says the revoke is done.
|
||
|
||
- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
|
||
(condition 2, the 3e lock). No second message goes out either: the first one still applies.
|
||
- **The job ends.** The human gets one inbox message (condition 3).
|
||
- If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
|
||
delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
|
||
- If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
|
||
describes. It records `grant.revoke.refused {job, ...}`.
|
||
- **The delete raised after a completed job** finds an empty set and goes as today.
|
||
- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
|
||
the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
|
||
reason in a message:
|
||
|
||
> Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
|
||
> the delete again when the inbox says it's done, or clear <grant> by hand first.
|
||
|
||
It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
|
||
end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
|
||
it. A delete after the human cleared the grant by hand doesn't find an empty set if a manual revoke left a
|
||
line (Q1). The leftover line makes the set non-empty, so that delete starts a job for the line removal
|
||
and refuses `grants-revoking` (d6, N6, and T3b's second case). If nothing is left, the set is empty and the
|
||
delete goes on.
|
||
|
||
The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
|
||
cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
|
||
launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
|
||
widen the row into the launcher for a one-time wait.
|
||
|
||
While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
|
||
harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
|
||
start check.
|
||
|
||
### Why not in `beforeWrite` or `revoke()`
|
||
|
||
Both are synchronous and run inside the organization writer, after its checks. A host `remove` is SSH and a
|
||
Gitea revoke is HTTP, so they can't run there. The business package also has no reach into the broker's grant
|
||
tables, and shouldn't get any.
|
||
|
||
### Why after step A
|
||
|
||
A fire is the decision to stop the agent. If step G came first, a host that can't be reached would keep the
|
||
fired agent running. After step A, a refused step G leaves the same state as a failed step B today: the
|
||
instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state. The repeat
|
||
fire's delisted path (webby-0094 N1) also runs step G, since the set is read before step B on both paths.
|
||
|
||
## 3. Step G: what it revokes, and how
|
||
|
||
The set, for (business, agent), is read in one store transaction: step A's for a fire, the delete's checks
|
||
for a delete (section 2). It holds:
|
||
|
||
- each `host_grants` row of the agent;
|
||
- each host named by an open host grant decision for the agent;
|
||
- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
|
||
|
||
An open decision (d4, B1) is any grant decision raised for the agent with no outcome yet. That is: not denied,
|
||
not expired, no `grant.applied` or `grant.failed`, and no `grant.revoked` of the same (agent, host), or of the
|
||
agent's Gitea, recorded after its raise. A raised decision the human hasn't decided is open. d3 required
|
||
"approved", which left that one out.
|
||
|
||
An empty set means no step G and no new records. A fire of an instance with no grant is then exactly today's
|
||
fire, with no new events (brief tests, fourth bullet).
|
||
|
||
### 3a. A host grant: the host line first (D2)
|
||
|
||
hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
|
||
`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
|
||
record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
|
||
host. So the fire's host revoke reverses one step and keeps the rest the same:
|
||
|
||
1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
|
||
`credential-unknown` branch of `#revokeHost` already does).
|
||
2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
|
||
"a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
|
||
decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
|
||
(section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
|
||
3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
|
||
host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
|
||
so the human can tell it from a manual revoke.
|
||
4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
|
||
5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
|
||
|
||
This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
|
||
hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
|
||
fire and the delete call it with `lineFirst: true`.
|
||
|
||
The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
|
||
section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
|
||
"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
|
||
|
||
### 3b. The Gitea grant (D3)
|
||
|
||
The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
|
||
runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
|
||
`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
|
||
`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
|
||
|
||
The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
|
||
name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
|
||
gone (section 5's reconcile). Two changes keep the fire closed:
|
||
|
||
- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
|
||
step G refuses with nothing changed.
|
||
- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
|
||
token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
|
||
forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
|
||
gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
|
||
kept visible here, not hidden.
|
||
|
||
**A Gitea executor in flight** (d3 T6c, and B3 seen from the executor's side). Step G's Gitea part goes in
|
||
this order:
|
||
1. Record `grant.revoked {granted: false}` for each open Gitea decision (3c).
|
||
2. Await this agent's tracked Gitea executors and remote undos, if any are running. Each one ends `revoked` at
|
||
its next `#giteaRevokedSince` check, or it committed `applied` before step 1.
|
||
3. Read the `gitea_grants` row again and run the steps above.
|
||
|
||
`#giteaCloseRevoke` with `fire: true` lists the user's tokens and deletes every one. The list comes after
|
||
step 2, so it includes any token an in-flight executor created. `#giteaRuns` is a single Set today. It
|
||
becomes keyed by (business, agent) so step G can await one agent's executors alone.
|
||
|
||
### 3c. Open decisions
|
||
|
||
A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
|
||
`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
|
||
`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
|
||
`#giteaRevokedSince`.
|
||
|
||
This is the case where a decision is approved before the fire and executed after it (sage-2574). Both checks
|
||
look for a `grant.revoked` of the agent recorded after the decision was raised (broker.mjs:1156 and :1678 at
|
||
47a35bf43). A fire's `grant.revoked` is always later than the approval, so the executor ends the decision
|
||
whenever it runs. The fired agent keeps its roles entry, so `#hostDefinition` and `#giteaRole` still pass, and
|
||
this check is the only thing that stops the decision. T6 and T6b test it. A delete also removes the roles
|
||
entry, so the executor refuses there as well.
|
||
|
||
The same holds for a decision raised but not yet approved (B1). It's in the set, so it gets its
|
||
`grant.revoked {granted: false}`. If the human approves it later, its executor ends `revoked` (T6d). The
|
||
decision stays approvable in the inbox. Approving it does nothing but end it, which `grant.failed` reports as
|
||
it does for any revoked grant.
|
||
|
||
### 3d. A raise for a fired agent (d4, B1)
|
||
|
||
`#hostGrantBinding` and `#giteaGrantBinding` both run inside the raise transaction (broker.mjs:1539 and
|
||
:1548; the host raise's keyscan is awaited before that transaction). Both gain one check. The raise refuses
|
||
`instance-fired` if the agent is:
|
||
- firing (`#firing`);
|
||
- fired, by the writer's own rule (instance-settings.mjs, hire): a roles entry, not in `launch.instances`,
|
||
and not `launch.by`.
|
||
|
||
A rehire lists the name again, so a rehired agent can be granted again.
|
||
|
||
Because the check is in the transaction that records the decision, there's no gap. A raise that commits
|
||
before step A's transaction is in the set. One that commits after it refuses. A raise for an agent whose
|
||
delete's step G is running refuses `grant-revoke-running` (3e).
|
||
|
||
No executor recheck is added. A raise can't commit after `fire.stopped`, and every decision raised before it
|
||
is in the set and gets its `grant.revoked`.
|
||
|
||
### 3e. One step G per agent (d4, B3)
|
||
|
||
The broker keeps `#revoking`, a map from (business, agent) to the verb whose step G is running. The verb sets
|
||
it synchronously, in the same turn as the set read, before it answers. The job clears it in a `finally`.
|
||
While it's set:
|
||
|
||
| Caller | Answer |
|
||
|---|---|
|
||
| `instance.fire` | refuses `fire-in-progress` |
|
||
| `lifecycleDelete` | refuses `grants-revoking`, no second job |
|
||
| `hosts.revoke` / `grants.revoke` for the agent | refuse `grant-revoke-running` |
|
||
| grant raise for the agent | refuses `grant-revoke-running` |
|
||
| `grants.sweep` | skips this agent's entries and names it as `running` in its result |
|
||
|
||
Two fires started together (the chief retrying after a client timeout): the first takes the lock before it
|
||
answers, and the second refuses. The map is in memory. After a broker restart the job is gone and so is the
|
||
lock. The instance is still firing, or the delete still has a non-empty set, and the repeat starts a fresh
|
||
step G (section 5).
|
||
|
||
A manual revoke can't race step G, since it refuses while the lock is held. Step G's transactions still
|
||
check what they read: a `host_grants` row already gone at 3a step 3 records `grant.revoked {granted: false}`
|
||
and goes on, as hosts.revoke does today for a revoke with no record.
|
||
|
||
## 4. A refusal (D4)
|
||
|
||
Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
|
||
recorded, and the rest stay as they were. The job doesn't run step B: the organization file is unchanged, and
|
||
the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing". A delete's request was
|
||
already closed with `grants-revoking` (section 2).
|
||
|
||
The verb has already answered, so the refusal is a record, not a thrown code:
|
||
- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
|
||
'instance.fire' | 'lifecycle.delete', by}`.
|
||
- `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
|
||
- `grant-revoke-failed` is the code in the event and in the inbox message.
|
||
|
||
A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139). It names the instance,
|
||
the grant and the reason, with the manual way out beside the retry (N2):
|
||
|
||
> Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
|
||
> is reachable. If <host> won't come back, run `mosaic hosts revoke <agent> <host>` first: the grant record
|
||
> goes, the line is named as left, and the next fire completes and names that line (section 5).
|
||
|
||
The delete's message has the same shape, with "raise the delete again". For Gitea, the way out is
|
||
`mosaic grants sweep gitea` once Gitea is back.
|
||
|
||
A job that throws something unexpected inside step G records `grant.revoke.refused` with reason `internal`,
|
||
sends the same message, and releases the lock. There's no unhandled rejection. A failure at step B or C is
|
||
`fire.failed` (section 2).
|
||
|
||
**A restart mid-job (d6, N4).** The lock and the job are in memory, so a restart ends them with no end
|
||
record. A fire would stay "firing" until the chief happened to repeat it. A delete's request is already
|
||
closed, so no one would hear anything. Boot closes that gap:
|
||
- A job is a start record (`fire.revoking` or `grants.revoking`) whose `job` has no end record. The end
|
||
records are `fire.done`, `fire.failed`, `grants.revoked.done`, `grant.revoke.refused` and the new
|
||
`grants.job.interrupted`.
|
||
- When the broker starts, for each such job of each business, it records `grants.job.interrupted {job,
|
||
agent, verb}` and sends the human one inbox message:
|
||
|
||
> <Fire|Delete> of <agent>: job <job> was interrupted by a bus restart. Some grants may be revoked. Run the
|
||
> fire again, or raise the delete again. The repeat goes on from what's left.
|
||
|
||
- Recording the end record in the same transaction as the message means a second boot finds nothing to
|
||
send.
|
||
- Boot reads only events, with no network call, so it adds nothing to the start time beyond one query.
|
||
|
||
## 5. A repeat fire (D5)
|
||
|
||
A repeat runs the whole fire again: the checks, step A (nothing left to stop), the set read, then step G, B
|
||
and C in the job. A repeat while the job runs refuses `fire-in-progress` (3e). A repeat delete works the same
|
||
way through the human's new request.
|
||
|
||
- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
|
||
- A host grant that refused still has its row, so it's retried from 3a step 1.
|
||
- Gitea (d4, N1): if the agent has entries in `#giteaLeft` or `#giteaOpenRevokes`, step G retries only those.
|
||
It calls `#giteaCloseRevoke` for each, as `#sweepGitea` does per entry. It doesn't run the business-wide
|
||
sweep, which would retry every agent's entries and settle every open rotation under the chief's name. Any
|
||
entry still left for this agent refuses `gitea-left`. An entry left by an earlier manual
|
||
`mosaic grants revoke` of the same agent blocks the fire the same way. That's intended: no grant outlives
|
||
its instance, whichever verb started its revoke. These entries count toward a non-empty set (section 2,
|
||
fire step 3), so they go to the job and never run inside the verb.
|
||
- A host line left by an earlier manual `hosts.revoke` doesn't block the fire (Q1 below). These are the
|
||
agent's `grant.revoke.closed` records with `hostLineLeft` and no later line removal for that tag. For each
|
||
one whose host record still exists, step G sends `remove <user> <tag>` once. On success it records
|
||
`grant.revoke.closed {revoke: <the old revoke id>, hostLine: 'removed', fire: true}`, so a later fire doesn't
|
||
find it again. On failure it records nothing new and the fire goes on. The host and the tag go in
|
||
`grantsLeft` (section 6). These leftover lines also make the set non-empty, so the retry runs in the job,
|
||
inside B2's answer (dewey-0823 accepted the single retry on that condition).
|
||
|
||
## Rulings
|
||
|
||
- **Q1 (sage-2573):** a leftover line doesn't block the fire. If the host is reachable, the fire tries the
|
||
removal once, because a fired agent's key line still on a host is the exact risk this row closes. If that
|
||
try fails, the fire doesn't refuse. It names the host and the leftover line in the result and in the event
|
||
the human sees. The human accepted that residue when their manual `hosts.revoke` left it, so it shouldn't
|
||
stop a fire forever on a dead host. Dewey may push back on the retry if it widens step G more than it's
|
||
worth.
|
||
- **Q2 (sage-2573):** a delete refused at step G closes its request through the launcher's normal refusal
|
||
path, and the human raises it again. A request left taken would strand, which is the shape DEFERRED already
|
||
records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
|
||
`grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
|
||
does. T3b tests both.
|
||
- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
|
||
revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
|
||
done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
|
||
with a test:
|
||
1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
|
||
delete again when the inbox says done. A bare code isn't enough (T3).
|
||
2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
|
||
3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
|
||
manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
|
||
reason and doesn't complete over a live grant (T3b).
|
||
4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
|
||
5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
|
||
(T2).
|
||
|
||
## 6. The result and the event (D6)
|
||
|
||
- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
|
||
`fire.revoking {job, grants}` (section 2).
|
||
- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
|
||
'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
|
||
So the event the human sees names them.
|
||
- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
|
||
verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
|
||
- **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
|
||
|
||
## 7. What doesn't change (brief item 4)
|
||
|
||
- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers and the same records. Their refusals stay
|
||
the same (`tester-dev-only` included) with one addition from section 3e (d6, N5): while a step G job holds
|
||
the agent's lock, `hosts.revoke` and `grants.revoke` for that agent refuse `grant-revoke-running`, and
|
||
`grants.sweep` skips the agent and names it as `running`. hosts.revoke's own order stays local first.
|
||
- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
|
||
existing grant-evidence shape.
|
||
- `revoke()` in runtime.mjs and the organization writer don't change.
|
||
- `server.mjs`, `process.mjs`, `cli/src/host.mjs` and `cli/src/launcher.mjs` don't change. Both verbs stay
|
||
synchronous. The launcher already closes a request with whatever code `lifecycleDelete` refuses.
|
||
- New codes: `fire-in-progress`, `grants-revoking`, `grant-revoke-running`, `instance-fired`, and
|
||
`grant-revoke-failed` in events and messages. New event types: `fire.revoking`, `grants.revoking`,
|
||
`grants.revoked.done`, `grant.revoke.refused`, `fire.failed` and `grants.job.interrupted`.
|
||
- The broker's start gains one events query for interrupted jobs (section 4, N4). `lifecycleDone` checks a refusal against `LIFECYCLE_CODE =
|
||
/^[a-z-]{1,64}$/` (broker.mjs:149), and `grants-revoking` matches it.
|
||
|
||
## 8. Tests (each in the owning package, whole files)
|
||
|
||
The bus tests use the fake installer and the fake Gitea from rows 235 and 236. They wait on the job with a new
|
||
trusted test API, `grantsIdle()`, which resolves once every step G job and Gitea executor has ended. It's the
|
||
same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete below is awaited that way.
|
||
|
||
- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
|
||
`credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
|
||
a record whose key file is gone.
|
||
- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
|
||
with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
|
||
the job:
|
||
- all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
|
||
them, plus `fire: true`;
|
||
- the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
|
||
- `fire.done` names all three;
|
||
- that agent's `host.run` and Gitea use both refuse.
|
||
- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
|
||
`lifecycleDelete`).
|
||
- The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
|
||
naming the job and all three grants, and saying to raise the delete again when it's done.
|
||
- A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
|
||
message.
|
||
- After the job, the end message names the job and each grant revoked.
|
||
- A third request completes as today.
|
||
- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
|
||
- The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
|
||
`mosaic hosts revoke`.
|
||
- The file and the instance are unchanged.
|
||
- The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
|
||
reason and the new job, and the instance is not deleted.
|
||
- With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
|
||
- A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
|
||
for Gitea and the leftover line.
|
||
- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
|
||
same events counted, no inbox message.
|
||
- T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
|
||
`host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
|
||
- the file is unchanged and the instance is still firing;
|
||
- host A is revoked and recorded, and host B's row and key file are still there;
|
||
- the Gitea grant is untouched, since it comes after B in the order;
|
||
- the credentials are unrevoked.
|
||
A repeat with B reachable completes and revokes only B and Gitea.
|
||
- T4b (Q1): an earlier manual `hosts.revoke` left a line on host C. A fire with C reachable removes it and
|
||
records the close. With C unreachable, the fire completes and names C and the tag in `grantsLeft` and in
|
||
`fire.done`.
|
||
- T5: Gitea's admin token can't be checked. The job refuses with the Gitea grant intact.
|
||
- T5b: a token DELETE fails. The job refuses `gitea-left`. A repeat with Gitea back retries only this agent's
|
||
entries and completes. Another agent's left entry is still left afterwards (N1).
|
||
- T6 (sage-2574): a host grant decision approved before the fire, executed after it. The fire records
|
||
`grant.revoked {granted: false}` for (agent, host). The executor runs after `fire.done` and ends the decision
|
||
`revoked`: no `host_grants` row, no key file, no `add` sent to the host.
|
||
- T6b (sage-2574): the same for a Gitea grant decision. The executor ends it `revoked` before any Gitea call:
|
||
no `gitea_grants` row, no token file, and the fake Gitea saw no token create.
|
||
- T6c: a host or Gitea executor already in flight when step G starts.
|
||
- The host side aborts its claim (3a step 1) and ends `revoked`.
|
||
- The Gitea side: step G awaits the executor (3b), and then the fake Gitea holds no token for the user. That
|
||
includes one the executor created mid-flight.
|
||
- T6d (B1): a host decision and a Gitea decision, raised before the fire and still undecided. The fire puts
|
||
both in the set. The human approves both after `fire.done`, and both executors end `revoked`, with nothing
|
||
applied.
|
||
- T6e (B1): a host raise and a Gitea raise refuse `instance-fired`:
|
||
- while the instance is firing;
|
||
- after `fire.done`.
|
||
After a rehire of the same name, a raise goes through. While a delete's job runs, a raise refuses
|
||
`grant-revoke-running`.
|
||
- T7: an instance with no grant fires exactly as today, synchronously. Same result keys, no `revoking`, and
|
||
the same events, counted.
|
||
- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
|
||
- T9 (B3): with the job held on a remove:
|
||
- a second fire refuses `fire-in-progress`;
|
||
- a delete of the same agent refuses `grants-revoking` and starts no second job;
|
||
- `hosts.revoke` and `grants.revoke` for the agent refuse `grant-revoke-running`;
|
||
- `grants.sweep` names the agent as `running`.
|
||
After a release, one set of `grant.revoked` records exists per grant.
|
||
- T10 (B2): the fake installer holds a remove and the fake Gitea holds a DELETE, neither answering until
|
||
released.
|
||
- `instance.fire` and `lifecycleDelete` both answer before either is released. The test checks this with a
|
||
timer, not a wall-clock bound.
|
||
- Meanwhile the broker answers another verb from another session.
|
||
- After release, the jobs complete.
|
||
- A cli test drives the launcher's delete pass against a bus whose remove is held. The pass closes the
|
||
request `grants-revoking` and the host channel stays up.
|
||
- T11 (d6, C1): a fire whose job is held on a remove. While it's held, the chief's run ends and its claim is
|
||
released, and processing is halted. After the release, the job completes: `fire.done` is recorded with the
|
||
job, the consumed `action.allowed` event is there, a second use of the same authority refuses
|
||
`decision-consumed`, and the instance is out of the file. No `fire.failed` and no `grant.revoke.refused`.
|
||
- T12 (d6, C1): a fire with a grant whose step B refuses in the job. The test uses T1's setup, where
|
||
`revoke()` refuses `credential-revoke-failed`. After the job:
|
||
- `fire.failed` carries the job, step `B` and `credential-revoke-failed`, and the inbox message names the
|
||
job, the revoked grants, the step and the code;
|
||
- there's no `grant.revoke.refused` and no `fire.done`;
|
||
- the instance is still firing and still in the file.
|
||
With the cause cleared, a repeat fire finds an empty set, runs B and C synchronously and completes.
|
||
- T13 (d6, N4): a fire and a delete, each with its job held, then the broker is closed and a new one opened
|
||
on the same data root. The new broker records `grants.job.interrupted` for both jobs and sends two inbox
|
||
messages, one saying to fire again and one to raise the delete again. A third broker on the same root sends
|
||
nothing more. A job that ended before the restart gets no interrupted record.
|
||
|
||
Files (N3). Every file listed runs whole in the manifest round, touched or not:
|
||
- New: `packages/bus/tests/fire-revokes-grants.test.mjs` holds T2-T13.
|
||
- Changed:
|
||
- `grant-host.test.mjs` and `gitea-grant.test.mjs`: T1's pins, and the T8 regressions next to their
|
||
existing cases;
|
||
- `packages/cli/tests/lifecycle.test.mjs`: the T10 launcher pass;
|
||
- `fake-gitea-api.mjs` and `host-fixtures.mjs`, if T10's holds need a hook there.
|
||
- Run whole because they call `instance.fire` or `lifecycleDelete` at 47a35bf43:
|
||
- bus: `chief-of-staff`, `lifecycle-revoke`, `lifecycle-verbs`, `reload`;
|
||
- cli: `lifecycle`, `instances`;
|
||
- harness: `tools`.
|
||
- Also run: `grant-executor.test.mjs` (the executors' `#giteaRuns` change) and `gitea-grant-docker.test.mjs`
|
||
(brief gate).
|
||
- `server.mjs`, `process.mjs`, `host.mjs` and `launcher.mjs` are unchanged, so no test changes there for this
|
||
row.
|
||
|
||
## 9. Mutants
|
||
|
||
- M1: step G skips the host revokes. Killed by T2.
|
||
- M2: step G skips the Gitea revoke. Killed by T2.
|
||
- M3: a refused grant revoke is ignored and the job goes on to step B. Killed by T4.
|
||
- M4: `lifecycleDelete` skips step G. Killed by T3.
|
||
- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
|
||
- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
|
||
- M7: open decisions left out of the set. Killed by T6 and T6b.
|
||
- M8: step G skips the leftover-line retry. Killed by T4b.
|
||
- M9: a failed leftover-line retry refuses the fire. Killed by T4b.
|
||
- M10: the set takes only approved decisions (d3's rule). Killed by T6d.
|
||
- M11: the raise bindings skip the fired check. Killed by T6e.
|
||
- M12: no `#revoking` lock. Killed by T9.
|
||
- M13: the verb awaits the job before it answers. Killed by T10.
|
||
- M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
|
||
- M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
|
||
- M16: the delete's first refusal sends no start message. Killed by T3.
|
||
- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
|
||
- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
|
||
- M19: the job's step C uses `#consumeAuthority`, with the holder and processing checks. Killed by T11.
|
||
- M20: a step B or C failure in the job is recorded as `grant.revoke.refused internal`. Killed by T12.
|
||
- M21: boot skips the interrupted-job check. Killed by T13.
|
||
|
||
## 10. Order and rebase
|
||
|
||
Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
|
||
and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
|
||
before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
|
||
|
||
## Build notes
|
||
|
||
Dewey approved d6 (dewey-0829, 10:11Z 10-22) with two build-time items. Both are test details, so there's no
|
||
new draft:
|
||
- N7, T12's setup. T1's setup makes `revoke()` refuse because the grant's `host.<id>` file is still in the
|
||
store. Step G's `hostKeys.revoke` removes that file before step B, so it can't be T12's cause. T12 uses a
|
||
step B refusal that survives step G, for example a stray file in the agent's store directory or a roles-entry
|
||
credential whose revoke fails. The test asserts it reaches `fire.failed` with step `B`.
|
||
- N8, `decision-consumed` in T11. A chief's fire is within-role: `#checkAuthority` returns `{ class }` with no
|
||
decision, and the lifecycle verbs take none (broker.mjs:3712). `#spent` never applies to a fire. T11 drops the
|
||
second-use `decision-consumed` assertion. Instead, a repeat fire after `fire.done` takes the delisted path and
|
||
answers done. Section 2's `decision-consumed` example for `fire.failed` can't occur, so the build uses only
|
||
codes that can.
|
||
|
||
Choices made in the code (b1, broker.mjs only so far). None changes an approved section; each fills in a
|
||
detail the design left open. Dewey, flag any you read differently.
|
||
- Record writers. `writeGrantEvent` takes only `grant.*` actions. So `fire.revoking`, `fire.done`, `fire.failed`
|
||
and `grants.job.interrupted` are not grant evidence. `fire.*` goes through `#event` under the chief's session
|
||
(the job keeps a copy, `sc`). `grants.revoking`, `grants.revoked.done` and `grants.job.interrupted` go under
|
||
the bus session `{ role: null, run: null }`, subject the agent. `grant.revoke.refused` is grant evidence, as
|
||
section 4 has it.
|
||
- A fired Gitea user. The `gitea_users` row survives a fire (#436b section 6), so on its own the row would
|
||
never let a fired agent's set empty. The user counts as pending until a later Gitea `grant.revoked` with
|
||
`fire: true` and a non-null `user` names the agent, and counts again after a new grant step (a rehire).
|
||
- Open Gitea decisions (3c) end with a `grant.revoked` that has `user: null`, `tokens: []`, `fire: false` and
|
||
`decisions: [ids]`, plus an immediate close with `serviceLeft: []`. `#giteaRevokedSince` then ends them. Since
|
||
`user` is null, the pair never counts as the user's fire.
|
||
- Leftover lines (section 5, Q1). A line is in the set as `line:<host>` when its tag is in `#hostLinesLeft`, its
|
||
host isn't one the agent holds a grant on, and no `fire.done` or `grants.revoked.done` for the agent since the
|
||
line was last named left lists it in `grantsLeft`. A line removed in step G gets `grant.revoke.closed` with
|
||
the old revoke id, `hostLine: 'removed'` and the flag. `#hostLinesLeft` now clears a tag on a later close
|
||
with `hostLine: 'removed'` and `fire` or `delete` set, so the line also leaves `hosts sweep`'s list. The
|
||
removed lines appear in `grants` as `{ grant: 'line:<host>', revoke, tag }`.
|
||
- 3a's wait for host executors. `#runHostGrant` marks itself in `#hostRuns` from its claim to its release. Step
|
||
G's host part runs `abortClaim`, waits for every mark on (agent, host), and sends the remove. If a new mark
|
||
appeared during the remove, it runs again, up to five rounds, and then refuses `host-remove-failed`. From the
|
||
last check to the step 1 commit there's no await.
|
||
- `hosts sweep gitea` (3e) skips the open revokes, settles and left entries of every agent a job holds, and
|
||
keeps those entries in `left`. Its answer has `running: [agents]` only when there's at least one. The retry
|
||
of one left entry is now `#giteaRetry`, which the sweep and step G share. The behavior is unchanged.
|
||
- Step G's Gitea retries record a sweep-shaped pair for the agent: `grant.sweep { agent, job }`, then the
|
||
closes and retries, then `grant.swept` whose `left` keeps every other agent's entries. The pair is written
|
||
first, so a serviceLeft from a close between the two can't drop out of `#giteaLeft`.
|
||
- `#giteaRuns` is now a Map from each promise to its `business/agent` key, or null for business-wide work.
|
||
Step G waits for its own agent's keys and the null ones. `#track` adds `.catch` to its `finally` chain, so a
|
||
rejected executor can't leave an unhandled rejection behind.
|
||
- A delete repeat after a refusal. `grant-revoke-failed` is returned only when the agent's last delete job end
|
||
was a `grant.revoke.refused` whose grant is still in the set. If that grant was cleared by hand and only a
|
||
leftover line remains, the repeat answers `grants-revoking`.
|
||
- The delete job's reason, in `grant.revoked`, is `lifecycle.delete <request>`.
|
||
- Step G's messages go to the human's inbox alone (no decision, so no raising role to thread to), through the
|
||
same insert and route as `#grantNotice`.
|
||
- `#recordAuthority` is `#consumeAuthority` less `#checkAuthority`'s holder and processing checks, for a
|
||
fire's step C (C1).
|
||
|
||
Choices made in the tests (b2). Each changes where a test sits or how it checks, not what it checks:
|
||
- T1 is in `lifecycle-revoke.test.mjs`, not `grant-host.test.mjs` and `gitea-grant.test.mjs`. T1 pins the
|
||
runtime's `revoke()` (step B's credential check) and the start sweep, and only `lifecycle-revoke` builds a
|
||
broker through `startBroker` with the real credential store. Its `setup` takes a new `seed` hook that writes
|
||
a host record and grant before the start.
|
||
- T8 is in `grant-executor.test.mjs` for `hosts.revoke`, where G12, G13, G26 and G27 already are, and in
|
||
`gitea-grant.test.mjs` for `grants.revoke`. G12 already pins the host verb's two records, their bodies and
|
||
their order, and G26 its role refusals, so T8 adds what no test had: `tester-dev-only` once `bus.tester` is
|
||
off, and that a verb revoke writes no `fire.*` or `grants.*` record.
|
||
- T2-T13 use a fake installer and a fake organization in `fire-revokes-grants.test.mjs`. The organization's
|
||
`instance.fire` delists and reloads the business, as the writer does, and can throw a set error, which is
|
||
how T12 gets its step B cause (N7): a `credential-revoke-failed` that survives step G.
|
||
- "A fired agent's `host.run` and Gitea use refuse" is checked through the state those uses read: no
|
||
`host_grants` row, no key file, a revoked credential, and no Gitea token for the user.
|
||
- T10's cli case runs the real runtime and launcher. The seeded grant is on a TEST-NET host
|
||
(192.0.2.10), so the job's remove fails or waits. The first request closes `grants-revoking` either way, and
|
||
the case waits on `grantsIdle()` before close.
|
||
- The mutants are `scratch/megavolt-r281/r281-mutants.sh`, one sed each on `broker.mjs`, run against
|
||
`fire-revokes-grants.test.mjs`. M13's verbs are synchronous, so its mutant makes the fire return the job's
|
||
promise, and the call waits for the job. That's what the verb awaiting its job would look like.
|
||
|
||
Rebase onto 46dedf925, after row 256 (sage-2597). It's on branch `megavolt/row281-b`, and
|
||
`megavolt/row281` stays where it is. `git range-diff 0165be266..bd0bb96ae origin/refactor..HEAD` gives `=` for
|
||
all nine commits, and no path changes on both sides. Row 256 threads the actor through `planDelete`,
|
||
`checkDelete` and `instances.check`, and adds `protectedRefusal`. That doesn't change what a delete calls in
|
||
this row:
|
||
- At the verb, `instances.set` op delete already ran `instances.check` with `s.human` before 256. A
|
||
creative instance's delete by anyone but the human now refuses there, before any request exists, so no
|
||
step G job starts for it.
|
||
- In the pass, step G runs in `lifecycleDelete` before `#organization('instances.set')`, and the actor
|
||
is still `r.by`. `planDelete` there can still refuse after step G has revoked. Before 256 a references or
|
||
last-listed refusal could do the same thing, so the order is unchanged. A delete the verb accepted only
|
||
meets a new refusal in the pass if the file changed between the two.
|
||
- `read(business, actor)` changes only the listing's `endProtected`. The broker and these tests don't read it.
|
||
The T10 cli case deletes `sec-1` (security-reviewer), which isn't END_PROTECTED.
|
||
|
||
Rebase onto 980cec88c, after row 277 (sage-2605). It's on branch `megavolt/row281-c`.
|
||
`git range-diff 46dedf925..7454be493 980cec88c..c174f37ac` gives `=` for all ten commits. Row 277's
|
||
`thread.claims` change to `broker.mjs` doesn't touch step G or any line the mutants change:
|
||
`scratch/megavolt-r281/anchors.sh` counts each M1-M21 sed anchor in the rebased `broker.mjs`, and all 28
|
||
counts are 1. The mutants script runs it after a green base and stops if a count isn't 1.
|
||
|
||
b3, after the first base run on c174f37ac (11:24:47Z) came back red. These tests had never run before: b2
|
||
was written under git and docs only. bus: 120 tests, 116 pass, 4 fail. cli lifecycle: 24 tests, 23 pass,
|
||
1 fail. Every older test in those files passed. No mutant ran. Each failure is a test fault, and b3 changes
|
||
no code:
|
||
- T4b, `fire-revokes-grants`: **test changed**. The test read `manual.revoke`, but the `hosts.revoke`
|
||
answer has never carried `revoke`: `#revokeHost` drops it, so the verb's answer keeps its pre-281 shape.
|
||
The fire's close names the manual revoke, as section 3a says. The test now takes that id from the
|
||
manual's `grant.revoked` event.
|
||
- T5b, `fire-revokes-grants`: **test changed**. The job refused `gitea-left:9,tokens`, and the test
|
||
expected `gitea-left:<digits>`. Section 4 sets the reason as `gitea-left:<ids>`, the ids left at Gitea.
|
||
For a fire, a close's `serviceLeft` is the N10 list from before 281: each token id whose DELETE failed,
|
||
then `tokens` or `login` for a fire step that failed (`#giteaCloseRevoke`). With the DELETE failing,
|
||
`#giteaFire`'s token list never clears, so `tokens` is a real entry left at Gitea. It belongs in the
|
||
reason. The test now expects `gitea-left:<id>,tokens`.
|
||
- T6b, `fire-revokes-grants`: **test changed**. After the fire, the executor made 4 Gitea calls where the
|
||
test expected none. They are step 0's admin checks C1 to C4: version, self, token scopes and the repo
|
||
check, all GET. Step 1's `#giteaRevokedSince` then ends the decision `revoked` before any write. Section
|
||
3c of d6 names that step 1 check as the only thing that stops the decision. T6b's line in section 8 says
|
||
"before any Gitea call", but the evidence it lists is no row, no token file and no token create. That
|
||
phrase was wrong, because step 0 always runs first. The approved text stays as it is, and this note is
|
||
the correction. The test now checks that every call after the fire is a GET and that no token create
|
||
happened. Moving the revoked check ahead of step 0 would change the executor for every revoke, not just
|
||
step G's, and the design doesn't ask for that.
|
||
- T8, `gitea-grant`: **test changed**. The human revoke's events between `grant.revoked` and
|
||
`grant.revoke.closed` include its 5 `grant.call`/`grant.result` pairs (`#giteaLog` logs both around every
|
||
Gitea call). T8 checks that a human revoke writes no job records, not that it makes no calls. It now
|
||
leaves the call log out before comparing.
|
||
- T10, cli `lifecycle`: **test changed** (sage-2615 asked about this one). The `credential-file` came from
|
||
`startBroker`, not from the delete and not from Gitea. The seeded host grant gives `sec-1` a `host.web1`
|
||
store reference. `Credentials` checks the store root (`<config dir>/secrets/agents`) with `privateDir` at
|
||
start, and the fixture had no store root. That check is row 199's, and it is right to fail closed. The
|
||
test now makes the store root at 0700 under `dirname(p.f.config)`, as the bus host fixtures do, never
|
||
under the real config. sage-2575's condition still holds. This business has no Gitea grant, so
|
||
`#grantSet` adds no `gitea` entry (`#giteaPending` and `#giteaUnfired` read only the store), and step G
|
||
never reaches `#stepGGitea`. A delete with no Gitea grant reads no Gitea credential file.
|
||
|
||
### b4a: the M1 hang's cause (mos-1273, sage-2636)
|
||
|
||
The first M1-M21 run on 06ee5ed6b hung on M1. All 22 tests ended in about 80 s (16 fail, 5 pass, 1
|
||
cancelled), but the file's process stayed alive until the harness killed it at 1762 s. This is a test fault,
|
||
and no code changed.
|
||
|
||
- Cause: T6c holds `web1:add`, and the fake `install` loops in `pause()` on `sleep(5)` while the key is
|
||
held. Only the fire's abort signal can end that loop, and M1 removes the host revoke that sends the
|
||
abort. In M1's log, T6c fails at its `until` on line 683, before it deletes the key on line 691. So the loop keeps a
|
||
timer alive forever. That reading comes from the log and the file. The rerun checks it: before M1-M21 it runs M1
|
||
once without `--test-force-exit`, and that run must exit by itself inside 180 s.
|
||
- Fix: setup's `t.after` clears `held` and `failing` before it closes the store, so any held loop ends
|
||
with its test. The base is unchanged because every passing test already releases its keys.
|
||
- Sage's second candidate, the Gitea `gate` in T6c and T10, isn't a live handle. The fake server's
|
||
`t.after` closes all its connections, and a parked promise with no timer doesn't keep node alive. Both
|
||
tests now also release the gate in `t.after` (`go()` only, not `off()`: a second `off()` would splice
|
||
index -1 and remove another rule).
|
||
- The runner's 180 s timeout with `--test-force-exit` (sage-2633) still caps any future hang.
|
||
|
||
### b4: T6b pins the four reads (dewey-0839, sage-2627)
|
||
|
||
- T6b: **test changed**, stricter. `slice(calls).every(GET)` also passed with zero calls. It now
|
||
deep-equals the exact list in order: `GET /api/v1/version`, `GET /api/v1/user`, `GET /api/v1/token`,
|
||
`GET /api/v1/user/repos?limit=1&page=1` (step 0's C1 to C4, `gitea-grant.mjs` adminChecks). It also fails
|
||
if step 1 stops ending the run and step 2's reads go out. The title says "before any Gitea write".
|
||
|
||
### T14 and M22: a declined decision on a removed host (sage-2653, dewey-0844, N1)
|
||
|
||
Test only, written before the slot. Not run yet.
|
||
|
||
- T14 (new, in `fire-revokes-grants.test.mjs`): coder's web1 decision is raised and left undecided. The human
|
||
then runs `hosts.remove web1`, which goes through because hosts.remove doesn't check open decisions (N1).
|
||
The fire starts a job whose set is `host:web1`. Step G refuses `host-unknown` with one
|
||
`grant.revoke.refused`, and one inbox message names the job, web1 and host-unknown. There's no fire.done.
|
||
The human resolves the decision `no`. No executor runs (`#approvedGrant` is null for a no), so the closed
|
||
check in `#openAgentDecisions` is the only thing that takes it out of the set. The repeat fire answers
|
||
with no job and records one fire.done with no job.
|
||
- M22 deletes that closed check (`broker.mjs:1684`). The declined decision stays open, so the repeat fire
|
||
starts a second job and refuses host-unknown again, and T14 fails on `repeat.job`. M10 mutates the same
|
||
line in another way (approved-only), and T6d kills it. M22 runs on the fire-revokes file like M1-M21, and
|
||
its anchor makes 29.
|
||
- N1's hint text (hosts.remove naming the open decision) is a source change. It stays out of this row and
|
||
goes to a DEFERRED line that Sage files at landing.
|
||
- If T14 is red on its first run, it goes to Sage and then Mos untouched, because 281 has had its one placed red
|
||
(mos-1272).
|
||
|
||
### Gate plan (sage-2649, dewey-0840, mos-1275)
|
||
|
||
Written before the slot. The slot runs on row281-e after the rebase onto 278's landing, and every step
|
||
stops at the first red.
|
||
|
||
1. The base: the touched files whole (fire-revokes-grants, lifecycle-revoke, grant-executor, gitea-grant
|
||
in bus; lifecycle in cli). Then the 29 anchors, the M1 exit check without `--test-force-exit`, and
|
||
M1-M22 under `timeout -k 10 180` with `--test-force-exit`.
|
||
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and
|
||
gitea-grant-docker; cli instances; harness tools. If gitea-grant-docker skips, the report names the skip
|
||
and its reason.
|
||
3. bus, business and cli whole.
|
||
4. Every `scripts/test-*.sh`, each as `timeout -k 10 LIMIT bash scripts/test-X.sh`, and no script is
|
||
edited. LIMIT is Gander's row 278 limit (`agents/gander/work/queue-278/SHELL-LIMITS.json`, sage-2646): 3x
|
||
the longest successful run from Gander's row 196 receipts, plus 120 s headroom. I checked it against two
|
||
more records. One is my row 236 gate at bd5f11ded (10-22). The other is 278's run 3
|
||
(`run3/whole/results.json`). Neither has a longer run, so every limit is at least 3x the longest of all
|
||
three. A script with no limit stops the gate before it starts. Any timeout goes to Sage, then Mos,
|
||
unplaced.
|
||
|
||
| Script | Row 236 | 196 receipts | 278 run 3 | Longest | 3x | LIMIT |
|
||
|---|---|---|---|---|---|---|
|
||
| test-auth | 1 s | 1.9 s | 0.8 s | 1.9 s | 6 s | 126 s |
|
||
| test-conductor | 6 s | 8.1 s | 6.1 s | 8.1 s | 25 s | 145 s |
|
||
| test-config | 1 s | 3.2 s | 1.9 s | 3.2 s | 10 s | 130 s |
|
||
| test-discord | 8 s | 8.8 s | 7.4 s | 8.8 s | 27 s | 147 s |
|
||
| test-extension-package | 1 s | 3.0 s | 1.4 s | 3.0 s | 10 s | 130 s |
|
||
| test-foundation | 39 s | 39.7 s | 39.9 s | 39.9 s | 120 s | 240 s |
|
||
| test-land-work | 11 s | 12.6 s | 11.3 s | 12.6 s | 38 s | 158 s |
|
||
| test-queue | 20 s | 23.4 s | 21.2 s | 23.4 s | 71 s | 191 s |
|
||
| test-release | 2 s | 12.2 s | 2.7 s | 12.2 s | 37 s | 157 s |
|
||
| test-task | 14 s | 36.8 s | 13.8 s | 36.8 s | 111 s | 231 s |
|
||
|
||
## Runs
|
||
|
||
The manifest-round slot on row281-e 7979fb41e (base 385a7194c, row 278's landing), CLEAR sage-2661. Runner
|
||
`r281-slot.sh` (scratch), from `scratch/megavolt-r222`: 12:43:17Z to 12:59:25Z, 10-22, SLOT_EXIT 0. No red,
|
||
no timeout. Every step ran in a detached worktree at the candidate, with TMPDIR under
|
||
`/mnt/storage/scratch/tmp`. The gate worktree was clean at the end, and the trap removed both worktrees.
|
||
|
||
1. The base, each file whole, plain `node --test`: bus fire-revokes-grants, lifecycle-revoke,
|
||
grant-executor and gitea-grant 121/121, 0 skipped, 22 s; cli lifecycle 24/24, 0 skipped, 13 s. T14 passed
|
||
on its first run. All 29 anchors matched once. The M1 exit check exited by itself: rc 1, 17 fails (the 16
|
||
on 3f29375a8, plus T14), 63 s. M1-M22: 22 killed, 0 survived (Mutants).
|
||
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and gitea-grant-docker
|
||
76/76, 0 skipped, 11 s (gitea-grant-docker ran, with no skip); cli instances 14/14, 0 skipped, 3 s; harness
|
||
tools 23/23, 0 skipped.
|
||
3. Whole packages: bus 819/819, 23 s; business 231/231, 3 s; cli 837/837, 591 s. Each has 0 skipped.
|
||
4. The shells, each under `timeout -k 10 LIMIT`, all rc 0:
|
||
|
||
| Script | Result | Time | LIMIT |
|
||
|---|---|---|---|
|
||
| test-auth | 15 passed, 0 failed | 1 s | 126 s |
|
||
| test-conductor | 17 passed, 0 failed | 6 s | 145 s |
|
||
| test-config | 31 passed, 0 failed | 1 s | 130 s |
|
||
| test-discord | 81 passed, 0 failed | 8 s | 147 s |
|
||
| test-extension-package | 18 passed, 0 failed | 1 s | 130 s |
|
||
| test-foundation | 44 passed, 0 failed | 36 s | 240 s |
|
||
| test-land-work | 86 passed, 0 failed | 10 s | 158 s |
|
||
| test-queue | node 156/156; 28 passed, 0 failed | 20 s | 191 s |
|
||
| test-release | 14 passed, 0 failed | 3 s | 157 s |
|
||
| test-task | 91 passed, 0 failed | 13 s | 231 s |
|
||
|
||
Skips, from a grep of every shell log for `^skip`, `# skip` and `SKIP` (dewey-0845):
|
||
|
||
- test-task, lines 87 and 93: `skip live provider cases (row 244: no provider call in the gate)`. Row 244
|
||
keeps provider calls out of the gate by design. The log has no Docker lines and no other skip.
|
||
- test-queue, line 34: `skip queue verify and render --check: this checkout (.../r281-gate-wt) is not the
|
||
queue's canonical root (/mnt/storage/src/mosaic-stack)`. The gate runs in a worktree, so the canonical-root
|
||
checks are skipped.
|
||
- No other shell log, and no node suite above, has a skip.
|
||
|
||
## Mutants
|
||
|
||
Run on b4a 3f29375a8 (base 6af85df4b), 12:16:18Z to 12:20:15Z, 10-22, slot sage-2643; runner
|
||
`r281-mutants.sh` (scratch). Base first, plain `node --test`, each file whole: bus fire-revokes-grants,
|
||
lifecycle-revoke, grant-executor and gitea-grant 120/120, 0 skipped; cli lifecycle 24/24, 0 skipped. The
|
||
worktree was clean after the base, and all 28 sed anchors matched once in `broker.mjs`.
|
||
|
||
The M1 exit check (mos-1273) applied M1 and ran fire-revokes-grants without `--test-force-exit`. It exited by
|
||
itself, rc 1, with 16 fails, in 63 s. The first run on 06ee5ed6b hung there until it was killed at 1762 s
|
||
(b4a above).
|
||
|
||
Each mutant ran fire-revokes-grants.test.mjs whole under `timeout -k 10 180 node --test --test-force-exit`
|
||
(sage-2633). Every one ran all 21 tests, with no load error and no timeout.
|
||
|
||
| Mutant | Change | Verdict | rc | Fails of 21 | Time | Section 9 names | Failing tests (from the log) |
|
||
|---|---|---|---|---|---|---|---|
|
||
| M1 | skip-host-revokes | KILLED | 1 | 16 | 63 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 |
|
||
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 12 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
|
||
| M3 | refusal-goes-on | KILLED | 1 | 3 | 3 s | T4 | T4 T5 T5b |
|
||
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
|
||
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
|
||
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||
| M7 | set-no-open-decisions | KILLED | 1 | 4 | 3 s | T6, T6b | T6 T6b T6c T6d |
|
||
| M8 | skip-line-retry | KILLED | 1 | 3 | 4 s | T4b | T3b T4b×2 |
|
||
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
|
||
| M10 | approved-only | KILLED | 1 | 1 | 3 s | T6d | T6d |
|
||
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
|
||
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
|
||
| M13 | verb-awaits-job | KILLED | 1 | 14 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 |
|
||
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
|
||
| M15 | business-wide-sweep | KILLED | 1 | 1 | 4 s | T5b | T5b |
|
||
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
|
||
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
|
||
| M18 | empty-set-job | KILLED | 1 | 4 | 4 s | T3c | T3 T3b×2 T3c |
|
||
| M19 | step-c-consume | KILLED | 1 | 1 | 4 s | T11 | T11 |
|
||
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
|
||
| M21 | no-interrupted-check | KILLED | 1 | 1 | 4 s | T13 | T13 |
|
||
|
||
Every mutant fails the test section 9 names for it; none was killed only by a different test. "T3b×2" and "T4b×2" are the two tests that share each id. The failing-test ids are taken from each log's "failing tests" list (dewey-0841).
|
||
|
||
21 killed, 0 survived. b4 makes one T6b assertion stricter after this run, and fire-revokes-grants is
|
||
rerun whole on b4 (Runs).
|
||
|
||
### On row281-e 7979fb41e (sage-2661)
|
||
|
||
12:43:17Z to 12:47:12Z, 10-22, the first part of the manifest-round slot (Runs). Same runner and flags as
|
||
above, with M22 added (T14 and M22 in the build notes). Every mutant ran all 22 tests, with no load error and
|
||
no timeout.
|
||
|
||
| Mutant | Change | Verdict | rc | Fails of 22 | Time | Section 9 names | Failing tests (from the log) |
|
||
|---|---|---|---|---|---|---|---|
|
||
| M1 | skip-host-revokes | KILLED | 1 | 17 | 62 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 T14 |
|
||
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 13 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
|
||
| M3 | refusal-goes-on | KILLED | 1 | 4 | 3 s | T4 | T4 T5 T5b T14 |
|
||
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
|
||
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
|
||
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||
| M7 | set-no-open-decisions | KILLED | 1 | 5 | 3 s | T6, T6b | T6 T6b T6c T6d T14 |
|
||
| M8 | skip-line-retry | KILLED | 1 | 3 | 3 s | T4b | T3b T4b×2 |
|
||
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
|
||
| M10 | approved-only | KILLED | 1 | 2 | 3 s | T6d | T6d T14 |
|
||
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
|
||
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
|
||
| M13 | verb-awaits-job | KILLED | 1 | 15 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 T14 |
|
||
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
|
||
| M15 | business-wide-sweep | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
|
||
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
|
||
| M18 | empty-set-job | KILLED | 1 | 4 | 3 s | T3c | T3 T3b×2 T3c |
|
||
| M19 | step-c-consume | KILLED | 1 | 1 | 3 s | T11 | T11 |
|
||
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
|
||
| M21 | no-interrupted-check | KILLED | 1 | 1 | 3 s | T13 | T13 |
|
||
| M22 | no-closed-check | KILLED | 1 | 1 | 3 s | T14 | T14 |
|
||
|
||
22 killed, 0 survived. Every mutant fails the test it's named for, and M22 fails T14 alone. Compared with
|
||
3f29375a8, the only new failures are T14's (under M1, M3, M7, M10, M13 and M22), and every other failing set is
|
||
unchanged.
|