Compare commits

...
Author SHA1 Message Date
jason.woltje e479521d3a row 281 build notes: the manifest-round slot on 7979fb41e is green; Runs has the base, anchors, M1 exit check, dewey-0840 files, bus/business/cli whole and the shells with their limits and named skips; Mutants has M1-M22, 22 killed, M22 fails only T14 (sage-2661, dewey-0845, #1786) 2026-10-22 08:00:45 -05:00
jason.woltje 7979fb41eb row 281 build notes: the gate plan for row281-e, with each scripts/test-*.sh under a limit of at least 3x its longest recorded run (sage-2649, dewey-0840, mos-1275, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 54ebc5d71c row 281 b5: T14 pins that the human's no takes a declined host decision out of the set, after hosts.remove left it with no host (N1); M22 drops the closed check and T14 kills it; test only, not yet run (sage-2653, dewey-0844, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 5e05e21794 row 281 build notes: the mutant table names section 9's killing test and each mutant's failing tests from its log; every mutant fails its named test (dewey-0841, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 6b6eb533e8 row 281 b4: T6b pins step 0's four Gitea reads in order and says 'before any Gitea write'; M1-M21 all killed on b4a 3f29375a8 with the M1 exit check, table in the build notes (dewey-0839, sage-2627, sage-2643, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje f9fcf5d7b3 row 281 b4a: the M1 hang was T6c's held web1:add loop, which only the fire's abort ends; setup's t.after clears held and failing, and T6c and T10 release their Gitea gate; test only, no code change (mos-1273, sage-2636, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje c055dddf44 row 281 b3: the first base run's five red tests were test faults, no code change; T4b reads the manual's revoke id from its event, T5b expects gitea-left:<id>,tokens (N10), T6b allows step 0's GET checks, gitea T8 leaves the call log out, cli T10 makes the 0700 store root; rebase onto 980cec88c noted (sage-2605, sage-2615, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 6d62ef04e0 row 281 build notes: rebase onto 46dedf925 after row 256, range-diff all =, 256's actor threading doesn't change what a delete calls (sage-2597, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 4982fa0adf row 281 b2: T1-T13 tests (fire-revokes-grants, lifecycle-revoke T1, T8 tester refusals, cli T10 pass) and the test choices in the build notes (#1786) 2026-10-22 07:42:10 -05:00
jason.woltje 7c2c876319 row 281 b1: step G in the broker (fire and delete jobs, the grant set, the lock, boot's interrupted check) and build notes (#1786) 2026-10-22 07:42:10 -05:00
jason.woltje 7d5f16ef95 row 281 build notes: Dewey's d6 approval with N7 (T12's step B cause survives step G) and N8 (no decision-consumed for a within-role fire) (dewey-0829, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje a7b4f6785c row 281 design d6: the job's step C without the chief's claim, fire.failed for a step B or C failure, boot closes interrupted jobs, Q1 lines in the set (dewey-0825, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 938378455d row 281 design d5: two requests per granted delete with its five conditions, a job id on every answer, event and message (sage-2575, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 5f4cd2faa6 row 281 design d4: step G detached from the verbs, open decisions and a fired-agent raise check, one step G per agent (dewey-0823, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 6faa0f1479 row 281 design d3: a grant decision approved before a fire and executed after it, tests T6/T6b/T6c (sage-2574, #1786) 2026-10-22 07:42:10 -05:00
jason.woltje 9867a6f322 row 281 design d2: Q1/Q2 rulings (sage-2573), the finding in the Why, T3b/T4b and M8/M9 (#1786) 2026-10-22 07:42:10 -05:00
jason.woltje 15ab01c0c3 row 281 design d1: a fire or a delete revokes the instance's host and Gitea grants (sage-2568, #1786) 2026-10-22 07:42:10 -05:00
7 changed files with 2538 additions and 47 deletions
+931
View File
@@ -0,0 +1,931 @@
# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d6 for Dewey's approval. No code until it is
approved. The results sections are filled in after the runs on Sage's gate.
History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-0823):
- B1: the set takes every grant decision with no outcome, and a raise refuses a fired or firing agent.
- B2: step G runs detached from the verb, so neither caller waits on the network.
- B3: one step G per agent, with named refusals for everything that could race it.
- N1-N3 are folded in.
d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
d6 answers Dewey's CHANGES on d5 (dewey-0825):
- C1: the job's step C records authority without the chief's claim (`#authorityFor`, as a landing's record
does), processing-halted doesn't hold the job, and a step B or C failure records `fire.failed {job, step,
code}` with its own message. Tests T11 and T12, mutants M19 and M20.
- N4: boot closes a job a restart interrupted, with one message. Test T13, mutant M21.
- N5: section 7 names the refusals the manual verbs gain.
- N6: the set counts Q1 leftover lines, in fire step 3 and in the delete text.
## Sources
- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
mos-1271, Vikunja 524).
- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
`gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
## 1. Why: what happens today (read from the code, to be pinned by a test before the change)
In short: today a delete of a granted instance unlinks the key files but leaves both grant records, the host
lines and the Gitea token live, and a fire refuses `credential-revoke-failed` (sage-2573 asked for this here
and on #1786). The details:
- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
`<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
(gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
`mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
`gitea_grants` change and no Gitea call.
- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
`host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
The next start then finds a grant record whose key file is gone. What the start does with that isn't read
yet. The T1 test below pins it.
- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
`gitea-not-declared`.
- **A raise after the fire** (dewey-0823 B1). `#raiseGrant` (broker.mjs:1529) checks only that the caller is
the chief, and the grant bindings check only the roles entry. So the chief can raise a new grant for a fired
agent, and if the human approves it, it applies.
## 2. Where the grant revokes sit (D1, rewritten in d4 for B2)
d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wait that long:
- **Delete.** The launcher's `host.op` waits `REQUEST_TIMEOUT_MS = 10000` (cli/src/host.mjs:31). On a timeout,
`breakChannel` stops the whole bus host. One SSH remove can take 30 s (host-grant.mjs:231, :311), and each
Gitea call 15 s (`CALL_MS`, gitea-grant.mjs:38).
- **Fire.** server.mjs answers a verb that isn't on its async list synchronously (:154), and the async list
gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
plus the Gitea calls can pass both limits.
So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
and every inbox message about it carry that id. The human and the chief can then match a later outcome to
the request that started it.
### The fire
1. `write('instance.fireCheck')`, as today.
2. Step A, as today, with one addition. In the same transaction, after `fire.stopped`, the grant set is read
(section 3). Reading it there means a grant raise either committed before step A, and is in the set, or
comes after `fire.stopped` and refuses (section 3d).
3. If the set is empty, steps B and C run synchronously, exactly as today. A fire with no grant answers
today's result with today's events (T7). The set counts everything step G would touch (d6, N6): grant
rows, open decisions, the agent's Gitea leftovers (section 5) and its Q1 leftover host lines (section 5).
Any of them makes the set non-empty and goes to the job.
4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
step A's transaction, starts the job, and answers at once:
`{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
step B hasn't run.
5. The job runs step G, then step B (`write('instance.fire')`) and step C (`#recordAuthority`, below, then
`fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends, `fire.failed`
included.
What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
- A repeat after a refused step G runs step G again (section 5).
The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
inbox message in section 4.
**Steps B and C in the job (d6, C1).** d4 left "step C without the chief's run" as a build-time check, and
Dewey checked it: it fails.
- `#consumeAuthority` calls `#checkAuthority` (broker.mjs:2852). That runs `#processingWork()` and
`#agent(s)`, and `#agent` refuses `not-holder` once the chief's run no longer holds its claim (:749).
- After the verb answers, the job can take a minute or more. If the chief's run ends meanwhile, step C would
throw after step B had delisted. The instance would be left delisted with no `fire.done`.
So the verb does the full check, and the job only records:
- The verb runs `#checkAuthority(s, action, authority)`, as today, before step A. The chief holds its claim
then.
- The job's step C runs `#authorityFor(s, action, authority)`, the check without the holder test. It then
runs the `#spent` check and the `action.allowed` event, as `#consumeAuthority` does. That's the shape a
landing's record already uses (`recordPublish`, broker.mjs:3174, row 137): the run that asked held its
claim when it asked, and may have ended since. It's one small private method, `#recordAuthority`, shared
with nothing else.
- The job keeps a copy of the session fields `#authorityFor` and `#event` read (business, role, thread,
run). It doesn't keep the live session object.
- `processing-halted` doesn't hold the job. A halt stops agent work. Step G only removes access, and steps
B and C finish a decision the chief made while processing ran. Holding them would leave a delisted or
half-revoked instance "firing" for the length of the halt. The verb itself still refuses during a halt,
through `#checkAuthority`, as today.
**A step B or C failure in the job** gets its own record, not `grant.revoke.refused`, since every grant in the
set was revoked by then:
- It records `fire.failed {job, step: 'B' | 'C', code}`. `code` is the thrown code, for example the writer's
`credential-revoke-failed`, or `decision-consumed`.
- It sends an inbox message:
> Fire of <agent>: job <job> revoked its grants (<list>) but stopped at step <B|C> (<code>). The instance
> is still firing. Run the fire again once <code> is cleared.
- The instance stays firing. A repeat fire finds an empty set and takes today's path for that state, with the
chief's own claim: B and C after a step B failure (T12), the delisted path (webby-0094 N1) after a step C
failure.
`grant.revoke.refused` with reason `internal` stays only for an unexpected throw inside step G.
### The delete (sage-2575)
`lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
grant set.
- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
closes the request with that code, well inside its 10 s.
The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
> Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
> Raise the delete again when the inbox says the revoke is done.
- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
(condition 2, the 3e lock). No second message goes out either: the first one still applies.
- **The job ends.** The human gets one inbox message (condition 3).
- If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
- If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
describes. It records `grant.revoke.refused {job, ...}`.
- **The delete raised after a completed job** finds an empty set and goes as today.
- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
reason in a message:
> Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
> the delete again when the inbox says it's done, or clear <grant> by hand first.
It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
it. A delete after the human cleared the grant by hand doesn't find an empty set if a manual revoke left a
line (Q1). The leftover line makes the set non-empty, so that delete starts a job for the line removal
and refuses `grants-revoking` (d6, N6, and T3b's second case). If nothing is left, the set is empty and the
delete goes on.
The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
widen the row into the launcher for a one-time wait.
While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
start check.
### Why not in `beforeWrite` or `revoke()`
Both are synchronous and run inside the organization writer, after its checks. A host `remove` is SSH and a
Gitea revoke is HTTP, so they can't run there. The business package also has no reach into the broker's grant
tables, and shouldn't get any.
### Why after step A
A fire is the decision to stop the agent. If step G came first, a host that can't be reached would keep the
fired agent running. After step A, a refused step G leaves the same state as a failed step B today: the
instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state. The repeat
fire's delisted path (webby-0094 N1) also runs step G, since the set is read before step B on both paths.
## 3. Step G: what it revokes, and how
The set, for (business, agent), is read in one store transaction: step A's for a fire, the delete's checks
for a delete (section 2). It holds:
- each `host_grants` row of the agent;
- each host named by an open host grant decision for the agent;
- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
An open decision (d4, B1) is any grant decision raised for the agent with no outcome yet. That is: not denied,
not expired, no `grant.applied` or `grant.failed`, and no `grant.revoked` of the same (agent, host), or of the
agent's Gitea, recorded after its raise. A raised decision the human hasn't decided is open. d3 required
"approved", which left that one out.
An empty set means no step G and no new records. A fire of an instance with no grant is then exactly today's
fire, with no new events (brief tests, fourth bullet).
### 3a. A host grant: the host line first (D2)
hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
host. So the fire's host revoke reverses one step and keeps the rest the same:
1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
`credential-unknown` branch of `#revokeHost` already does).
2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
"a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
(section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
so the human can tell it from a manual revoke.
4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
fire and the delete call it with `lineFirst: true`.
The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
### 3b. The Gitea grant (D3)
The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
gone (section 5's reconcile). Two changes keep the fire closed:
- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
step G refuses with nothing changed.
- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
kept visible here, not hidden.
**A Gitea executor in flight** (d3 T6c, and B3 seen from the executor's side). Step G's Gitea part goes in
this order:
1. Record `grant.revoked {granted: false}` for each open Gitea decision (3c).
2. Await this agent's tracked Gitea executors and remote undos, if any are running. Each one ends `revoked` at
its next `#giteaRevokedSince` check, or it committed `applied` before step 1.
3. Read the `gitea_grants` row again and run the steps above.
`#giteaCloseRevoke` with `fire: true` lists the user's tokens and deletes every one. The list comes after
step 2, so it includes any token an in-flight executor created. `#giteaRuns` is a single Set today. It
becomes keyed by (business, agent) so step G can await one agent's executors alone.
### 3c. Open decisions
A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
`#giteaRevokedSince`.
This is the case where a decision is approved before the fire and executed after it (sage-2574). Both checks
look for a `grant.revoked` of the agent recorded after the decision was raised (broker.mjs:1156 and :1678 at
47a35bf43). A fire's `grant.revoked` is always later than the approval, so the executor ends the decision
whenever it runs. The fired agent keeps its roles entry, so `#hostDefinition` and `#giteaRole` still pass, and
this check is the only thing that stops the decision. T6 and T6b test it. A delete also removes the roles
entry, so the executor refuses there as well.
The same holds for a decision raised but not yet approved (B1). It's in the set, so it gets its
`grant.revoked {granted: false}`. If the human approves it later, its executor ends `revoked` (T6d). The
decision stays approvable in the inbox. Approving it does nothing but end it, which `grant.failed` reports as
it does for any revoked grant.
### 3d. A raise for a fired agent (d4, B1)
`#hostGrantBinding` and `#giteaGrantBinding` both run inside the raise transaction (broker.mjs:1539 and
:1548; the host raise's keyscan is awaited before that transaction). Both gain one check. The raise refuses
`instance-fired` if the agent is:
- firing (`#firing`);
- fired, by the writer's own rule (instance-settings.mjs, hire): a roles entry, not in `launch.instances`,
and not `launch.by`.
A rehire lists the name again, so a rehired agent can be granted again.
Because the check is in the transaction that records the decision, there's no gap. A raise that commits
before step A's transaction is in the set. One that commits after it refuses. A raise for an agent whose
delete's step G is running refuses `grant-revoke-running` (3e).
No executor recheck is added. A raise can't commit after `fire.stopped`, and every decision raised before it
is in the set and gets its `grant.revoked`.
### 3e. One step G per agent (d4, B3)
The broker keeps `#revoking`, a map from (business, agent) to the verb whose step G is running. The verb sets
it synchronously, in the same turn as the set read, before it answers. The job clears it in a `finally`.
While it's set:
| Caller | Answer |
|---|---|
| `instance.fire` | refuses `fire-in-progress` |
| `lifecycleDelete` | refuses `grants-revoking`, no second job |
| `hosts.revoke` / `grants.revoke` for the agent | refuse `grant-revoke-running` |
| grant raise for the agent | refuses `grant-revoke-running` |
| `grants.sweep` | skips this agent's entries and names it as `running` in its result |
Two fires started together (the chief retrying after a client timeout): the first takes the lock before it
answers, and the second refuses. The map is in memory. After a broker restart the job is gone and so is the
lock. The instance is still firing, or the delete still has a non-empty set, and the repeat starts a fresh
step G (section 5).
A manual revoke can't race step G, since it refuses while the lock is held. Step G's transactions still
check what they read: a `host_grants` row already gone at 3a step 3 records `grant.revoked {granted: false}`
and goes on, as hosts.revoke does today for a revoke with no record.
## 4. A refusal (D4)
Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
recorded, and the rest stay as they were. The job doesn't run step B: the organization file is unchanged, and
the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing". A delete's request was
already closed with `grants-revoking` (section 2).
The verb has already answered, so the refusal is a record, not a thrown code:
- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
'instance.fire' | 'lifecycle.delete', by}`.
- `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
- `grant-revoke-failed` is the code in the event and in the inbox message.
A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139). It names the instance,
the grant and the reason, with the manual way out beside the retry (N2):
> Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
> is reachable. If <host> won't come back, run `mosaic hosts revoke <agent> <host>` first: the grant record
> goes, the line is named as left, and the next fire completes and names that line (section 5).
The delete's message has the same shape, with "raise the delete again". For Gitea, the way out is
`mosaic grants sweep gitea` once Gitea is back.
A job that throws something unexpected inside step G records `grant.revoke.refused` with reason `internal`,
sends the same message, and releases the lock. There's no unhandled rejection. A failure at step B or C is
`fire.failed` (section 2).
**A restart mid-job (d6, N4).** The lock and the job are in memory, so a restart ends them with no end
record. A fire would stay "firing" until the chief happened to repeat it. A delete's request is already
closed, so no one would hear anything. Boot closes that gap:
- A job is a start record (`fire.revoking` or `grants.revoking`) whose `job` has no end record. The end
records are `fire.done`, `fire.failed`, `grants.revoked.done`, `grant.revoke.refused` and the new
`grants.job.interrupted`.
- When the broker starts, for each such job of each business, it records `grants.job.interrupted {job,
agent, verb}` and sends the human one inbox message:
> <Fire|Delete> of <agent>: job <job> was interrupted by a bus restart. Some grants may be revoked. Run the
> fire again, or raise the delete again. The repeat goes on from what's left.
- Recording the end record in the same transaction as the message means a second boot finds nothing to
send.
- Boot reads only events, with no network call, so it adds nothing to the start time beyond one query.
## 5. A repeat fire (D5)
A repeat runs the whole fire again: the checks, step A (nothing left to stop), the set read, then step G, B
and C in the job. A repeat while the job runs refuses `fire-in-progress` (3e). A repeat delete works the same
way through the human's new request.
- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
- A host grant that refused still has its row, so it's retried from 3a step 1.
- Gitea (d4, N1): if the agent has entries in `#giteaLeft` or `#giteaOpenRevokes`, step G retries only those.
It calls `#giteaCloseRevoke` for each, as `#sweepGitea` does per entry. It doesn't run the business-wide
sweep, which would retry every agent's entries and settle every open rotation under the chief's name. Any
entry still left for this agent refuses `gitea-left`. An entry left by an earlier manual
`mosaic grants revoke` of the same agent blocks the fire the same way. That's intended: no grant outlives
its instance, whichever verb started its revoke. These entries count toward a non-empty set (section 2,
fire step 3), so they go to the job and never run inside the verb.
- A host line left by an earlier manual `hosts.revoke` doesn't block the fire (Q1 below). These are the
agent's `grant.revoke.closed` records with `hostLineLeft` and no later line removal for that tag. For each
one whose host record still exists, step G sends `remove <user> <tag>` once. On success it records
`grant.revoke.closed {revoke: <the old revoke id>, hostLine: 'removed', fire: true}`, so a later fire doesn't
find it again. On failure it records nothing new and the fire goes on. The host and the tag go in
`grantsLeft` (section 6). These leftover lines also make the set non-empty, so the retry runs in the job,
inside B2's answer (dewey-0823 accepted the single retry on that condition).
## Rulings
- **Q1 (sage-2573):** a leftover line doesn't block the fire. If the host is reachable, the fire tries the
removal once, because a fired agent's key line still on a host is the exact risk this row closes. If that
try fails, the fire doesn't refuse. It names the host and the leftover line in the result and in the event
the human sees. The human accepted that residue when their manual `hosts.revoke` left it, so it shouldn't
stop a fire forever on a dead host. Dewey may push back on the retry if it widens step G more than it's
worth.
- **Q2 (sage-2573):** a delete refused at step G closes its request through the launcher's normal refusal
path, and the human raises it again. A request left taken would strand, which is the shape DEFERRED already
records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
`grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
does. T3b tests both.
- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
with a test:
1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
delete again when the inbox says done. A bare code isn't enough (T3).
2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
reason and doesn't complete over a live grant (T3b).
4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
(T2).
## 6. The result and the event (D6)
- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
`fire.revoking {job, grants}` (section 2).
- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
So the event the human sees names them.
- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
- **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
## 7. What doesn't change (brief item 4)
- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers and the same records. Their refusals stay
the same (`tester-dev-only` included) with one addition from section 3e (d6, N5): while a step G job holds
the agent's lock, `hosts.revoke` and `grants.revoke` for that agent refuse `grant-revoke-running`, and
`grants.sweep` skips the agent and names it as `running`. hosts.revoke's own order stays local first.
- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
existing grant-evidence shape.
- `revoke()` in runtime.mjs and the organization writer don't change.
- `server.mjs`, `process.mjs`, `cli/src/host.mjs` and `cli/src/launcher.mjs` don't change. Both verbs stay
synchronous. The launcher already closes a request with whatever code `lifecycleDelete` refuses.
- New codes: `fire-in-progress`, `grants-revoking`, `grant-revoke-running`, `instance-fired`, and
`grant-revoke-failed` in events and messages. New event types: `fire.revoking`, `grants.revoking`,
`grants.revoked.done`, `grant.revoke.refused`, `fire.failed` and `grants.job.interrupted`.
- The broker's start gains one events query for interrupted jobs (section 4, N4). `lifecycleDone` checks a refusal against `LIFECYCLE_CODE =
/^[a-z-]{1,64}$/` (broker.mjs:149), and `grants-revoking` matches it.
## 8. Tests (each in the owning package, whole files)
The bus tests use the fake installer and the fake Gitea from rows 235 and 236. They wait on the job with a new
trusted test API, `grantsIdle()`, which resolves once every step G job and Gitea executor has ended. It's the
same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete below is awaited that way.
- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
`credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
a record whose key file is gone.
- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
the job:
- all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
them, plus `fire: true`;
- the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
- `fire.done` names all three;
- that agent's `host.run` and Gitea use both refuse.
- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
`lifecycleDelete`).
- The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
naming the job and all three grants, and saying to raise the delete again when it's done.
- A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
message.
- After the job, the end message names the job and each grant revoked.
- A third request completes as today.
- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
- The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
`mosaic hosts revoke`.
- The file and the instance are unchanged.
- The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
reason and the new job, and the instance is not deleted.
- With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
- A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
for Gitea and the leftover line.
- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
same events counted, no inbox message.
- T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
`host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
- the file is unchanged and the instance is still firing;
- host A is revoked and recorded, and host B's row and key file are still there;
- the Gitea grant is untouched, since it comes after B in the order;
- the credentials are unrevoked.
A repeat with B reachable completes and revokes only B and Gitea.
- T4b (Q1): an earlier manual `hosts.revoke` left a line on host C. A fire with C reachable removes it and
records the close. With C unreachable, the fire completes and names C and the tag in `grantsLeft` and in
`fire.done`.
- T5: Gitea's admin token can't be checked. The job refuses with the Gitea grant intact.
- T5b: a token DELETE fails. The job refuses `gitea-left`. A repeat with Gitea back retries only this agent's
entries and completes. Another agent's left entry is still left afterwards (N1).
- T6 (sage-2574): a host grant decision approved before the fire, executed after it. The fire records
`grant.revoked {granted: false}` for (agent, host). The executor runs after `fire.done` and ends the decision
`revoked`: no `host_grants` row, no key file, no `add` sent to the host.
- T6b (sage-2574): the same for a Gitea grant decision. The executor ends it `revoked` before any Gitea call:
no `gitea_grants` row, no token file, and the fake Gitea saw no token create.
- T6c: a host or Gitea executor already in flight when step G starts.
- The host side aborts its claim (3a step 1) and ends `revoked`.
- The Gitea side: step G awaits the executor (3b), and then the fake Gitea holds no token for the user. That
includes one the executor created mid-flight.
- T6d (B1): a host decision and a Gitea decision, raised before the fire and still undecided. The fire puts
both in the set. The human approves both after `fire.done`, and both executors end `revoked`, with nothing
applied.
- T6e (B1): a host raise and a Gitea raise refuse `instance-fired`:
- while the instance is firing;
- after `fire.done`.
After a rehire of the same name, a raise goes through. While a delete's job runs, a raise refuses
`grant-revoke-running`.
- T7: an instance with no grant fires exactly as today, synchronously. Same result keys, no `revoking`, and
the same events, counted.
- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
- T9 (B3): with the job held on a remove:
- a second fire refuses `fire-in-progress`;
- a delete of the same agent refuses `grants-revoking` and starts no second job;
- `hosts.revoke` and `grants.revoke` for the agent refuse `grant-revoke-running`;
- `grants.sweep` names the agent as `running`.
After a release, one set of `grant.revoked` records exists per grant.
- T10 (B2): the fake installer holds a remove and the fake Gitea holds a DELETE, neither answering until
released.
- `instance.fire` and `lifecycleDelete` both answer before either is released. The test checks this with a
timer, not a wall-clock bound.
- Meanwhile the broker answers another verb from another session.
- After release, the jobs complete.
- A cli test drives the launcher's delete pass against a bus whose remove is held. The pass closes the
request `grants-revoking` and the host channel stays up.
- T11 (d6, C1): a fire whose job is held on a remove. While it's held, the chief's run ends and its claim is
released, and processing is halted. After the release, the job completes: `fire.done` is recorded with the
job, the consumed `action.allowed` event is there, a second use of the same authority refuses
`decision-consumed`, and the instance is out of the file. No `fire.failed` and no `grant.revoke.refused`.
- T12 (d6, C1): a fire with a grant whose step B refuses in the job. The test uses T1's setup, where
`revoke()` refuses `credential-revoke-failed`. After the job:
- `fire.failed` carries the job, step `B` and `credential-revoke-failed`, and the inbox message names the
job, the revoked grants, the step and the code;
- there's no `grant.revoke.refused` and no `fire.done`;
- the instance is still firing and still in the file.
With the cause cleared, a repeat fire finds an empty set, runs B and C synchronously and completes.
- T13 (d6, N4): a fire and a delete, each with its job held, then the broker is closed and a new one opened
on the same data root. The new broker records `grants.job.interrupted` for both jobs and sends two inbox
messages, one saying to fire again and one to raise the delete again. A third broker on the same root sends
nothing more. A job that ended before the restart gets no interrupted record.
Files (N3). Every file listed runs whole in the manifest round, touched or not:
- New: `packages/bus/tests/fire-revokes-grants.test.mjs` holds T2-T13.
- Changed:
- `grant-host.test.mjs` and `gitea-grant.test.mjs`: T1's pins, and the T8 regressions next to their
existing cases;
- `packages/cli/tests/lifecycle.test.mjs`: the T10 launcher pass;
- `fake-gitea-api.mjs` and `host-fixtures.mjs`, if T10's holds need a hook there.
- Run whole because they call `instance.fire` or `lifecycleDelete` at 47a35bf43:
- bus: `chief-of-staff`, `lifecycle-revoke`, `lifecycle-verbs`, `reload`;
- cli: `lifecycle`, `instances`;
- harness: `tools`.
- Also run: `grant-executor.test.mjs` (the executors' `#giteaRuns` change) and `gitea-grant-docker.test.mjs`
(brief gate).
- `server.mjs`, `process.mjs`, `host.mjs` and `launcher.mjs` are unchanged, so no test changes there for this
row.
## 9. Mutants
- M1: step G skips the host revokes. Killed by T2.
- M2: step G skips the Gitea revoke. Killed by T2.
- M3: a refused grant revoke is ignored and the job goes on to step B. Killed by T4.
- M4: `lifecycleDelete` skips step G. Killed by T3.
- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
- M7: open decisions left out of the set. Killed by T6 and T6b.
- M8: step G skips the leftover-line retry. Killed by T4b.
- M9: a failed leftover-line retry refuses the fire. Killed by T4b.
- M10: the set takes only approved decisions (d3's rule). Killed by T6d.
- M11: the raise bindings skip the fired check. Killed by T6e.
- M12: no `#revoking` lock. Killed by T9.
- M13: the verb awaits the job before it answers. Killed by T10.
- M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
- M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
- M16: the delete's first refusal sends no start message. Killed by T3.
- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
- M19: the job's step C uses `#consumeAuthority`, with the holder and processing checks. Killed by T11.
- M20: a step B or C failure in the job is recorded as `grant.revoke.refused internal`. Killed by T12.
- M21: boot skips the interrupted-job check. Killed by T13.
## 10. Order and rebase
Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
## Build notes
Dewey approved d6 (dewey-0829, 10:11Z 10-22) with two build-time items. Both are test details, so there's no
new draft:
- N7, T12's setup. T1's setup makes `revoke()` refuse because the grant's `host.<id>` file is still in the
store. Step G's `hostKeys.revoke` removes that file before step B, so it can't be T12's cause. T12 uses a
step B refusal that survives step G, for example a stray file in the agent's store directory or a roles-entry
credential whose revoke fails. The test asserts it reaches `fire.failed` with step `B`.
- N8, `decision-consumed` in T11. A chief's fire is within-role: `#checkAuthority` returns `{ class }` with no
decision, and the lifecycle verbs take none (broker.mjs:3712). `#spent` never applies to a fire. T11 drops the
second-use `decision-consumed` assertion. Instead, a repeat fire after `fire.done` takes the delisted path and
answers done. Section 2's `decision-consumed` example for `fire.failed` can't occur, so the build uses only
codes that can.
Choices made in the code (b1, broker.mjs only so far). None changes an approved section; each fills in a
detail the design left open. Dewey, flag any you read differently.
- Record writers. `writeGrantEvent` takes only `grant.*` actions. So `fire.revoking`, `fire.done`, `fire.failed`
and `grants.job.interrupted` are not grant evidence. `fire.*` goes through `#event` under the chief's session
(the job keeps a copy, `sc`). `grants.revoking`, `grants.revoked.done` and `grants.job.interrupted` go under
the bus session `{ role: null, run: null }`, subject the agent. `grant.revoke.refused` is grant evidence, as
section 4 has it.
- A fired Gitea user. The `gitea_users` row survives a fire (#436b section 6), so on its own the row would
never let a fired agent's set empty. The user counts as pending until a later Gitea `grant.revoked` with
`fire: true` and a non-null `user` names the agent, and counts again after a new grant step (a rehire).
- Open Gitea decisions (3c) end with a `grant.revoked` that has `user: null`, `tokens: []`, `fire: false` and
`decisions: [ids]`, plus an immediate close with `serviceLeft: []`. `#giteaRevokedSince` then ends them. Since
`user` is null, the pair never counts as the user's fire.
- Leftover lines (section 5, Q1). A line is in the set as `line:<host>` when its tag is in `#hostLinesLeft`, its
host isn't one the agent holds a grant on, and no `fire.done` or `grants.revoked.done` for the agent since the
line was last named left lists it in `grantsLeft`. A line removed in step G gets `grant.revoke.closed` with
the old revoke id, `hostLine: 'removed'` and the flag. `#hostLinesLeft` now clears a tag on a later close
with `hostLine: 'removed'` and `fire` or `delete` set, so the line also leaves `hosts sweep`'s list. The
removed lines appear in `grants` as `{ grant: 'line:<host>', revoke, tag }`.
- 3a's wait for host executors. `#runHostGrant` marks itself in `#hostRuns` from its claim to its release. Step
G's host part runs `abortClaim`, waits for every mark on (agent, host), and sends the remove. If a new mark
appeared during the remove, it runs again, up to five rounds, and then refuses `host-remove-failed`. From the
last check to the step 1 commit there's no await.
- `hosts sweep gitea` (3e) skips the open revokes, settles and left entries of every agent a job holds, and
keeps those entries in `left`. Its answer has `running: [agents]` only when there's at least one. The retry
of one left entry is now `#giteaRetry`, which the sweep and step G share. The behavior is unchanged.
- Step G's Gitea retries record a sweep-shaped pair for the agent: `grant.sweep { agent, job }`, then the
closes and retries, then `grant.swept` whose `left` keeps every other agent's entries. The pair is written
first, so a serviceLeft from a close between the two can't drop out of `#giteaLeft`.
- `#giteaRuns` is now a Map from each promise to its `business/agent` key, or null for business-wide work.
Step G waits for its own agent's keys and the null ones. `#track` adds `.catch` to its `finally` chain, so a
rejected executor can't leave an unhandled rejection behind.
- A delete repeat after a refusal. `grant-revoke-failed` is returned only when the agent's last delete job end
was a `grant.revoke.refused` whose grant is still in the set. If that grant was cleared by hand and only a
leftover line remains, the repeat answers `grants-revoking`.
- The delete job's reason, in `grant.revoked`, is `lifecycle.delete <request>`.
- Step G's messages go to the human's inbox alone (no decision, so no raising role to thread to), through the
same insert and route as `#grantNotice`.
- `#recordAuthority` is `#consumeAuthority` less `#checkAuthority`'s holder and processing checks, for a
fire's step C (C1).
Choices made in the tests (b2). Each changes where a test sits or how it checks, not what it checks:
- T1 is in `lifecycle-revoke.test.mjs`, not `grant-host.test.mjs` and `gitea-grant.test.mjs`. T1 pins the
runtime's `revoke()` (step B's credential check) and the start sweep, and only `lifecycle-revoke` builds a
broker through `startBroker` with the real credential store. Its `setup` takes a new `seed` hook that writes
a host record and grant before the start.
- T8 is in `grant-executor.test.mjs` for `hosts.revoke`, where G12, G13, G26 and G27 already are, and in
`gitea-grant.test.mjs` for `grants.revoke`. G12 already pins the host verb's two records, their bodies and
their order, and G26 its role refusals, so T8 adds what no test had: `tester-dev-only` once `bus.tester` is
off, and that a verb revoke writes no `fire.*` or `grants.*` record.
- T2-T13 use a fake installer and a fake organization in `fire-revokes-grants.test.mjs`. The organization's
`instance.fire` delists and reloads the business, as the writer does, and can throw a set error, which is
how T12 gets its step B cause (N7): a `credential-revoke-failed` that survives step G.
- "A fired agent's `host.run` and Gitea use refuse" is checked through the state those uses read: no
`host_grants` row, no key file, a revoked credential, and no Gitea token for the user.
- T10's cli case runs the real runtime and launcher. The seeded grant is on a TEST-NET host
(192.0.2.10), so the job's remove fails or waits. The first request closes `grants-revoking` either way, and
the case waits on `grantsIdle()` before close.
- The mutants are `scratch/megavolt-r281/r281-mutants.sh`, one sed each on `broker.mjs`, run against
`fire-revokes-grants.test.mjs`. M13's verbs are synchronous, so its mutant makes the fire return the job's
promise, and the call waits for the job. That's what the verb awaiting its job would look like.
Rebase onto 46dedf925, after row 256 (sage-2597). It's on branch `megavolt/row281-b`, and
`megavolt/row281` stays where it is. `git range-diff 0165be266..bd0bb96ae origin/refactor..HEAD` gives `=` for
all nine commits, and no path changes on both sides. Row 256 threads the actor through `planDelete`,
`checkDelete` and `instances.check`, and adds `protectedRefusal`. That doesn't change what a delete calls in
this row:
- At the verb, `instances.set` op delete already ran `instances.check` with `s.human` before 256. A
creative instance's delete by anyone but the human now refuses there, before any request exists, so no
step G job starts for it.
- In the pass, step G runs in `lifecycleDelete` before `#organization('instances.set')`, and the actor
is still `r.by`. `planDelete` there can still refuse after step G has revoked. Before 256 a references or
last-listed refusal could do the same thing, so the order is unchanged. A delete the verb accepted only
meets a new refusal in the pass if the file changed between the two.
- `read(business, actor)` changes only the listing's `endProtected`. The broker and these tests don't read it.
The T10 cli case deletes `sec-1` (security-reviewer), which isn't END_PROTECTED.
Rebase onto 980cec88c, after row 277 (sage-2605). It's on branch `megavolt/row281-c`.
`git range-diff 46dedf925..7454be493 980cec88c..c174f37ac` gives `=` for all ten commits. Row 277's
`thread.claims` change to `broker.mjs` doesn't touch step G or any line the mutants change:
`scratch/megavolt-r281/anchors.sh` counts each M1-M21 sed anchor in the rebased `broker.mjs`, and all 28
counts are 1. The mutants script runs it after a green base and stops if a count isn't 1.
b3, after the first base run on c174f37ac (11:24:47Z) came back red. These tests had never run before: b2
was written under git and docs only. bus: 120 tests, 116 pass, 4 fail. cli lifecycle: 24 tests, 23 pass,
1 fail. Every older test in those files passed. No mutant ran. Each failure is a test fault, and b3 changes
no code:
- T4b, `fire-revokes-grants`: **test changed**. The test read `manual.revoke`, but the `hosts.revoke`
answer has never carried `revoke`: `#revokeHost` drops it, so the verb's answer keeps its pre-281 shape.
The fire's close names the manual revoke, as section 3a says. The test now takes that id from the
manual's `grant.revoked` event.
- T5b, `fire-revokes-grants`: **test changed**. The job refused `gitea-left:9,tokens`, and the test
expected `gitea-left:<digits>`. Section 4 sets the reason as `gitea-left:<ids>`, the ids left at Gitea.
For a fire, a close's `serviceLeft` is the N10 list from before 281: each token id whose DELETE failed,
then `tokens` or `login` for a fire step that failed (`#giteaCloseRevoke`). With the DELETE failing,
`#giteaFire`'s token list never clears, so `tokens` is a real entry left at Gitea. It belongs in the
reason. The test now expects `gitea-left:<id>,tokens`.
- T6b, `fire-revokes-grants`: **test changed**. After the fire, the executor made 4 Gitea calls where the
test expected none. They are step 0's admin checks C1 to C4: version, self, token scopes and the repo
check, all GET. Step 1's `#giteaRevokedSince` then ends the decision `revoked` before any write. Section
3c of d6 names that step 1 check as the only thing that stops the decision. T6b's line in section 8 says
"before any Gitea call", but the evidence it lists is no row, no token file and no token create. That
phrase was wrong, because step 0 always runs first. The approved text stays as it is, and this note is
the correction. The test now checks that every call after the fire is a GET and that no token create
happened. Moving the revoked check ahead of step 0 would change the executor for every revoke, not just
step G's, and the design doesn't ask for that.
- T8, `gitea-grant`: **test changed**. The human revoke's events between `grant.revoked` and
`grant.revoke.closed` include its 5 `grant.call`/`grant.result` pairs (`#giteaLog` logs both around every
Gitea call). T8 checks that a human revoke writes no job records, not that it makes no calls. It now
leaves the call log out before comparing.
- T10, cli `lifecycle`: **test changed** (sage-2615 asked about this one). The `credential-file` came from
`startBroker`, not from the delete and not from Gitea. The seeded host grant gives `sec-1` a `host.web1`
store reference. `Credentials` checks the store root (`<config dir>/secrets/agents`) with `privateDir` at
start, and the fixture had no store root. That check is row 199's, and it is right to fail closed. The
test now makes the store root at 0700 under `dirname(p.f.config)`, as the bus host fixtures do, never
under the real config. sage-2575's condition still holds. This business has no Gitea grant, so
`#grantSet` adds no `gitea` entry (`#giteaPending` and `#giteaUnfired` read only the store), and step G
never reaches `#stepGGitea`. A delete with no Gitea grant reads no Gitea credential file.
### b4a: the M1 hang's cause (mos-1273, sage-2636)
The first M1-M21 run on 06ee5ed6b hung on M1. All 22 tests ended in about 80 s (16 fail, 5 pass, 1
cancelled), but the file's process stayed alive until the harness killed it at 1762 s. This is a test fault,
and no code changed.
- Cause: T6c holds `web1:add`, and the fake `install` loops in `pause()` on `sleep(5)` while the key is
held. Only the fire's abort signal can end that loop, and M1 removes the host revoke that sends the
abort. In M1's log, T6c fails at its `until` on line 683, before it deletes the key on line 691. So the loop keeps a
timer alive forever. That reading comes from the log and the file. The rerun checks it: before M1-M21 it runs M1
once without `--test-force-exit`, and that run must exit by itself inside 180 s.
- Fix: setup's `t.after` clears `held` and `failing` before it closes the store, so any held loop ends
with its test. The base is unchanged because every passing test already releases its keys.
- Sage's second candidate, the Gitea `gate` in T6c and T10, isn't a live handle. The fake server's
`t.after` closes all its connections, and a parked promise with no timer doesn't keep node alive. Both
tests now also release the gate in `t.after` (`go()` only, not `off()`: a second `off()` would splice
index -1 and remove another rule).
- The runner's 180 s timeout with `--test-force-exit` (sage-2633) still caps any future hang.
### b4: T6b pins the four reads (dewey-0839, sage-2627)
- T6b: **test changed**, stricter. `slice(calls).every(GET)` also passed with zero calls. It now
deep-equals the exact list in order: `GET /api/v1/version`, `GET /api/v1/user`, `GET /api/v1/token`,
`GET /api/v1/user/repos?limit=1&page=1` (step 0's C1 to C4, `gitea-grant.mjs` adminChecks). It also fails
if step 1 stops ending the run and step 2's reads go out. The title says "before any Gitea write".
### T14 and M22: a declined decision on a removed host (sage-2653, dewey-0844, N1)
Test only, written before the slot. Not run yet.
- T14 (new, in `fire-revokes-grants.test.mjs`): coder's web1 decision is raised and left undecided. The human
then runs `hosts.remove web1`, which goes through because hosts.remove doesn't check open decisions (N1).
The fire starts a job whose set is `host:web1`. Step G refuses `host-unknown` with one
`grant.revoke.refused`, and one inbox message names the job, web1 and host-unknown. There's no fire.done.
The human resolves the decision `no`. No executor runs (`#approvedGrant` is null for a no), so the closed
check in `#openAgentDecisions` is the only thing that takes it out of the set. The repeat fire answers
with no job and records one fire.done with no job.
- M22 deletes that closed check (`broker.mjs:1684`). The declined decision stays open, so the repeat fire
starts a second job and refuses host-unknown again, and T14 fails on `repeat.job`. M10 mutates the same
line in another way (approved-only), and T6d kills it. M22 runs on the fire-revokes file like M1-M21, and
its anchor makes 29.
- N1's hint text (hosts.remove naming the open decision) is a source change. It stays out of this row and
goes to a DEFERRED line that Sage files at landing.
- If T14 is red on its first run, it goes to Sage and then Mos untouched, because 281 has had its one placed red
(mos-1272).
### Gate plan (sage-2649, dewey-0840, mos-1275)
Written before the slot. The slot runs on row281-e after the rebase onto 278's landing, and every step
stops at the first red.
1. The base: the touched files whole (fire-revokes-grants, lifecycle-revoke, grant-executor, gitea-grant
in bus; lifecycle in cli). Then the 29 anchors, the M1 exit check without `--test-force-exit`, and
M1-M22 under `timeout -k 10 180` with `--test-force-exit`.
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and
gitea-grant-docker; cli instances; harness tools. If gitea-grant-docker skips, the report names the skip
and its reason.
3. bus, business and cli whole.
4. Every `scripts/test-*.sh`, each as `timeout -k 10 LIMIT bash scripts/test-X.sh`, and no script is
edited. LIMIT is Gander's row 278 limit (`agents/gander/work/queue-278/SHELL-LIMITS.json`, sage-2646): 3x
the longest successful run from Gander's row 196 receipts, plus 120 s headroom. I checked it against two
more records. One is my row 236 gate at bd5f11ded (10-22). The other is 278's run 3
(`run3/whole/results.json`). Neither has a longer run, so every limit is at least 3x the longest of all
three. A script with no limit stops the gate before it starts. Any timeout goes to Sage, then Mos,
unplaced.
| Script | Row 236 | 196 receipts | 278 run 3 | Longest | 3x | LIMIT |
|---|---|---|---|---|---|---|
| test-auth | 1 s | 1.9 s | 0.8 s | 1.9 s | 6 s | 126 s |
| test-conductor | 6 s | 8.1 s | 6.1 s | 8.1 s | 25 s | 145 s |
| test-config | 1 s | 3.2 s | 1.9 s | 3.2 s | 10 s | 130 s |
| test-discord | 8 s | 8.8 s | 7.4 s | 8.8 s | 27 s | 147 s |
| test-extension-package | 1 s | 3.0 s | 1.4 s | 3.0 s | 10 s | 130 s |
| test-foundation | 39 s | 39.7 s | 39.9 s | 39.9 s | 120 s | 240 s |
| test-land-work | 11 s | 12.6 s | 11.3 s | 12.6 s | 38 s | 158 s |
| test-queue | 20 s | 23.4 s | 21.2 s | 23.4 s | 71 s | 191 s |
| test-release | 2 s | 12.2 s | 2.7 s | 12.2 s | 37 s | 157 s |
| test-task | 14 s | 36.8 s | 13.8 s | 36.8 s | 111 s | 231 s |
## Runs
The manifest-round slot on row281-e 7979fb41e (base 385a7194c, row 278's landing), CLEAR sage-2661. Runner
`r281-slot.sh` (scratch), from `scratch/megavolt-r222`: 12:43:17Z to 12:59:25Z, 10-22, SLOT_EXIT 0. No red,
no timeout. Every step ran in a detached worktree at the candidate, with TMPDIR under
`/mnt/storage/scratch/tmp`. The gate worktree was clean at the end, and the trap removed both worktrees.
1. The base, each file whole, plain `node --test`: bus fire-revokes-grants, lifecycle-revoke,
grant-executor and gitea-grant 121/121, 0 skipped, 22 s; cli lifecycle 24/24, 0 skipped, 13 s. T14 passed
on its first run. All 29 anchors matched once. The M1 exit check exited by itself: rc 1, 17 fails (the 16
on 3f29375a8, plus T14), 63 s. M1-M22: 22 killed, 0 survived (Mutants).
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and gitea-grant-docker
76/76, 0 skipped, 11 s (gitea-grant-docker ran, with no skip); cli instances 14/14, 0 skipped, 3 s; harness
tools 23/23, 0 skipped.
3. Whole packages: bus 819/819, 23 s; business 231/231, 3 s; cli 837/837, 591 s. Each has 0 skipped.
4. The shells, each under `timeout -k 10 LIMIT`, all rc 0:
| Script | Result | Time | LIMIT |
|---|---|---|---|
| test-auth | 15 passed, 0 failed | 1 s | 126 s |
| test-conductor | 17 passed, 0 failed | 6 s | 145 s |
| test-config | 31 passed, 0 failed | 1 s | 130 s |
| test-discord | 81 passed, 0 failed | 8 s | 147 s |
| test-extension-package | 18 passed, 0 failed | 1 s | 130 s |
| test-foundation | 44 passed, 0 failed | 36 s | 240 s |
| test-land-work | 86 passed, 0 failed | 10 s | 158 s |
| test-queue | node 156/156; 28 passed, 0 failed | 20 s | 191 s |
| test-release | 14 passed, 0 failed | 3 s | 157 s |
| test-task | 91 passed, 0 failed | 13 s | 231 s |
Skips, from a grep of every shell log for `^skip`, `# skip` and `SKIP` (dewey-0845):
- test-task, lines 87 and 93: `skip live provider cases (row 244: no provider call in the gate)`. Row 244
keeps provider calls out of the gate by design. The log has no Docker lines and no other skip.
- test-queue, line 34: `skip queue verify and render --check: this checkout (.../r281-gate-wt) is not the
queue's canonical root (/mnt/storage/src/mosaic-stack)`. The gate runs in a worktree, so the canonical-root
checks are skipped.
- No other shell log, and no node suite above, has a skip.
## Mutants
Run on b4a 3f29375a8 (base 6af85df4b), 12:16:18Z to 12:20:15Z, 10-22, slot sage-2643; runner
`r281-mutants.sh` (scratch). Base first, plain `node --test`, each file whole: bus fire-revokes-grants,
lifecycle-revoke, grant-executor and gitea-grant 120/120, 0 skipped; cli lifecycle 24/24, 0 skipped. The
worktree was clean after the base, and all 28 sed anchors matched once in `broker.mjs`.
The M1 exit check (mos-1273) applied M1 and ran fire-revokes-grants without `--test-force-exit`. It exited by
itself, rc 1, with 16 fails, in 63 s. The first run on 06ee5ed6b hung there until it was killed at 1762 s
(b4a above).
Each mutant ran fire-revokes-grants.test.mjs whole under `timeout -k 10 180 node --test --test-force-exit`
(sage-2633). Every one ran all 21 tests, with no load error and no timeout.
| Mutant | Change | Verdict | rc | Fails of 21 | Time | Section 9 names | Failing tests (from the log) |
|---|---|---|---|---|---|---|---|
| M1 | skip-host-revokes | KILLED | 1 | 16 | 63 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 |
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 12 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
| M3 | refusal-goes-on | KILLED | 1 | 3 | 3 s | T4 | T4 T5 T5b |
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
| M7 | set-no-open-decisions | KILLED | 1 | 4 | 3 s | T6, T6b | T6 T6b T6c T6d |
| M8 | skip-line-retry | KILLED | 1 | 3 | 4 s | T4b | T3b T4b×2 |
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
| M10 | approved-only | KILLED | 1 | 1 | 3 s | T6d | T6d |
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
| M13 | verb-awaits-job | KILLED | 1 | 14 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 |
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
| M15 | business-wide-sweep | KILLED | 1 | 1 | 4 s | T5b | T5b |
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
| M18 | empty-set-job | KILLED | 1 | 4 | 4 s | T3c | T3 T3b×2 T3c |
| M19 | step-c-consume | KILLED | 1 | 1 | 4 s | T11 | T11 |
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
| M21 | no-interrupted-check | KILLED | 1 | 1 | 4 s | T13 | T13 |
Every mutant fails the test section 9 names for it; none was killed only by a different test. "T3b×2" and "T4b×2" are the two tests that share each id. The failing-test ids are taken from each log's "failing tests" list (dewey-0841).
21 killed, 0 survived. b4 makes one T6b assertion stricter after this run, and fire-revokes-grants is
rerun whole on b4 (Runs).
### On row281-e 7979fb41e (sage-2661)
12:43:17Z to 12:47:12Z, 10-22, the first part of the manifest-round slot (Runs). Same runner and flags as
above, with M22 added (T14 and M22 in the build notes). Every mutant ran all 22 tests, with no load error and
no timeout.
| Mutant | Change | Verdict | rc | Fails of 22 | Time | Section 9 names | Failing tests (from the log) |
|---|---|---|---|---|---|---|---|
| M1 | skip-host-revokes | KILLED | 1 | 17 | 62 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 T14 |
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 13 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
| M3 | refusal-goes-on | KILLED | 1 | 4 | 3 s | T4 | T4 T5 T5b T14 |
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
| M7 | set-no-open-decisions | KILLED | 1 | 5 | 3 s | T6, T6b | T6 T6b T6c T6d T14 |
| M8 | skip-line-retry | KILLED | 1 | 3 | 3 s | T4b | T3b T4b×2 |
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
| M10 | approved-only | KILLED | 1 | 2 | 3 s | T6d | T6d T14 |
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
| M13 | verb-awaits-job | KILLED | 1 | 15 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 T14 |
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
| M15 | business-wide-sweep | KILLED | 1 | 1 | 3 s | T5b | T5b |
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
| M18 | empty-set-job | KILLED | 1 | 4 | 3 s | T3c | T3 T3b×2 T3c |
| M19 | step-c-consume | KILLED | 1 | 1 | 3 s | T11 | T11 |
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
| M21 | no-interrupted-check | KILLED | 1 | 1 | 3 s | T13 | T13 |
| M22 | no-closed-check | KILLED | 1 | 1 | 3 s | T14 | T14 |
22 killed, 0 survived. Every mutant fails the test it's named for, and M22 fails T14 alone. Compared with
3f29375a8, the only new failures are T14's (under M1, M3, M7, M10, M13 and M22), and every other failing set is
unchanged.
+548 -45
View File
@@ -393,8 +393,15 @@ export class Broker {
#giteaGrant;
#giteaKeys;
#giteaUndo = [];
#giteaRuns = new Set();
// Row 281 (#1786, d6 section 3b): each tracked Gitea promise and the business/agent it acts for, or
// null for business-wide work, so a step G job can await one agent's executors.
#giteaRuns = new Map();
#giteaRevokeUndo = [];
// Row 281 (d6 section 3e): business/agent to the verb whose step G job runs, the jobs themselves,
// and the host executors in flight by business/agent/host (section 3a).
#revoking = new Map();
#grantJobs = new Set();
#hostRuns = new Map();
// Row 125 (#1619, REQ-LIMIT-4): a run a limit stops, by business:run, until its next receive or state.
#stops = new Map();
#clock = 0;
@@ -1271,6 +1278,8 @@ export class Broker {
} catch {}
return code;
}
// Row 281 (d6 section 3a): marked from the claim to the release, so a step G job can await it.
const ran = this.#hostRunMark(business, g.agent, g.host);
const tag = this.#hostGrant.grantTag(business, g.agent, g.host);
const change = previous !== null;
const step = (name, extra = {}) =>
@@ -1372,8 +1381,23 @@ export class Broker {
return code;
} finally {
this.#hostKeys.release(business, g.agent, g.host, signal);
ran();
}
}
// Row 281 (d6 section 3a): marks one host executor for (agent, host) and returns its end.
#hostRunMark(business, agent, host) {
const key = business + '/' + agent + '/' + host;
let end;
const p = new Promise((resolve) => (end = resolve));
if (!this.#hostRuns.has(key)) this.#hostRuns.set(key, new Set());
this.#hostRuns.get(key).add(p);
return () => {
const set = this.#hostRuns.get(key);
set?.delete(p);
if (set?.size === 0) this.#hostRuns.delete(key);
end();
};
}
// r6 X3 and duckworth-0067 N1: credential.mint decisions for `method`, resolved yes by the human via
// cli and never spent.
#openGrantDecisions(business, method) {
@@ -1458,9 +1482,40 @@ export class Broker {
for (const m of this.#openRevokes(business)) this.#revokeUndo.push({ business, ...m });
// Row 236 (#436b r5 section 5): the Gitea grants' local part.
if (this.#giteaGrant && this.#giteaKeys) ended.push(...this.#giteaReconcileLocal(business));
this.#interruptedJobs(business);
}
return ended;
}
// Row 281 (d6 section 4, N4): a step G job is in memory, so a restart ends it with no end record. Each
// start record with no end gets grants.job.interrupted and one message to the human, in the same
// transaction, so a second boot sends nothing. A job this broker is running is left alone.
#interruptedJobs(business) {
for (const e of this.#store.all(
"SELECT subject, body FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.action') IN ('fire.revoking','grants.revoking') ORDER BY seq",
business,
)) {
const b = JSON.parse(e.body);
const fire = b.action === 'fire.revoking';
const agent = fire ? e.subject : b.agent;
if (this.#revoking.has(business + '/' + agent)) continue;
if (
this.#store.get(
"SELECT 1 FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.job')=? AND json_extract(body,'$.action') IN ('fire.done','fire.failed','grants.revoked.done','grant.revoke.refused','grants.job.interrupted') LIMIT 1",
business,
b.job,
)
)
continue;
const verb = fire ? 'instance.fire' : b.verb;
this.#store.transaction(() => {
this.#event({ business, role: null, run: null }, 'action.allowed', { action: 'grants.job.interrupted', job: b.job, agent, verb }, agent);
this.#humanNotice(
business,
`${fire ? 'Fire' : 'Delete'} of ${agent}: job ${b.job} was interrupted by a bus restart. Some grants may be revoked. ${fire ? 'Run the fire again' : 'Raise the delete again'}. The repeat goes on from what's left.`,
);
});
}
}
// Trusted host API, after the socket opens (r6 X3): the remote undo of each grant reconcileGrants left
// claimed. `remove <user> <tag>` through the installer, then grant.failed interrupted, with
// hostLineLeft if the remove failed, and the claim released. It only takes access away, so it needs no
@@ -1537,31 +1592,410 @@ export class Broker {
keys(a, ['agent', 'host', 'reason'], ['agent', 'host', 'reason']);
if (typeof a.agent !== 'string' || !ID_PATTERN.test(a.agent) || typeof a.host !== 'string' || !HOST_ID.test(a.host)) fail('invalid-request');
string(a.reason, 500);
// Row 281 (d6 section 3e): no manual revoke while a step G job holds the agent.
if (this.#revoking.has(s.business + '/' + a.agent)) fail('grant-revoke-running');
const { revoke, ...out } = await this.#revokeHostGrant(s.business, a.agent, a.host, a.reason, by);
return out;
}
// Row 281 (#1786, d6 section 3a): hosts.revoke's steps, shared with a fire's and a delete's step G.
// `lineFirst` sends the remove before anything local changes: the executor in flight aborts and ends,
// then `remove <user> <tag>`, and a refusal there (host-unknown, host-remove-failed) leaves the
// record, the key file and the memory copy as they were. An executor that started during the remove
// is aborted and awaited, and the remove runs again. The close then records the remove already done.
// `flag` ({ fire: true } or { delete: true }) goes in grant.revoked and the close.
async #revokeHostGrant(business, agent, host, reason, by, { lineFirst = false, flag = {} } = {}) {
if (!this.#hostGrant) fail('host-not-configured');
const tag = this.#hostGrant.grantTag(s.business, a.agent, a.host);
const tag = this.#hostGrant.grantTag(business, agent, host);
const revoke = randomUUID();
if (lineFirst) {
const key = business + '/' + agent + '/' + host;
for (let round = 0; ; round++) {
if (round === 5) fail('host-remove-failed');
this.#hostKeys?.abortClaim(business, agent, host);
while (this.#hostRuns.get(key)?.size) await Promise.all([...this.#hostRuns.get(key)]);
const record = this.#hostRecord(business, host);
if (!record) fail('host-unknown');
try {
await this.#hostGrant.remove(business, host, record, tag);
} catch {
fail('host-remove-failed');
}
// No await from this check to the step 1 commit, so no executor starts in between.
if (!this.#hostRuns.get(key)?.size) break;
}
}
const granted = this.#store.transaction(() => {
if (!this.#hostRecord(s.business, a.host)) fail('host-unknown');
const out = this.#store.run('DELETE FROM host_grants WHERE business=? AND agent=? AND host=?', s.business, a.agent, a.host);
if (!this.#hostRecord(business, host)) fail('host-unknown');
const out = this.#store.run('DELETE FROM host_grants WHERE business=? AND agent=? AND host=?', business, agent, host);
this.#grantEvent(
s.business,
{ action: 'grant.revoked', revoke, agent: a.agent, host: a.host, tag, hostLine: 'pending', granted: out.changes === 1, reason: a.reason, by },
a.host,
business,
{ action: 'grant.revoked', revoke, agent, host, tag, hostLine: 'pending', granted: out.changes === 1, reason, by, ...flag },
host,
);
return out.changes === 1;
});
let keyFile = 'removed';
if (this.#hostKeys) {
try {
this.#hostKeys.revoke(s.business, a.agent, a.host);
this.#hostKeys.revoke(business, agent, host);
} catch (e) {
// No reference: only a claim can be held, an executor's before its register.
if (e instanceof BusError && e.code === 'credential-unknown') this.#hostKeys.abortClaim(s.business, a.agent, a.host);
if (e instanceof BusError && e.code === 'credential-unknown') this.#hostKeys.abortClaim(business, agent, host);
else keyFile = 'left';
}
}
const closed = await this.#closeRevoke(s.business, { revoke, agent: a.agent, host: a.host, tag, by });
return { agent: a.agent, host: a.host, tag, granted, keyFile, ...(closed.hostLineLeft ? { hostLineLeft: closed.hostLineLeft } : { hostLine: 'removed' }) };
let closed;
if (lineFirst) {
closed = { action: 'grant.revoke.closed', revoke, agent, host, tag, hostLine: 'removed', ...flag, by };
this.#store.transaction(() => this.#grantEvent(business, closed, host));
} else closed = await this.#closeRevoke(business, { revoke, agent, host, tag, by });
return { agent, host, tag, revoke, granted, keyFile, ...(closed.hostLineLeft ? { hostLineLeft: closed.hostLineLeft } : { hostLine: 'removed' }) };
}
// Row 281 (#1786, d6 section 3): the step G set of (business, agent), each with its label and what it
// revokes. The caller reads it in one transaction: step A's for a fire, the delete's checks for a
// delete. Hosts first, by id, then Gitea, then the leftover lines (section 5).
#grantSet(business, agent) {
const set = [];
const open = this.#openAgentDecisions(business, agent);
const hosts = new Set(this.#store.all('SELECT host FROM host_grants WHERE business=? AND agent=?', business, agent).map((r) => r.host));
for (const d of open) if (d.grant.method === 'host') hosts.add(d.grant.host);
for (const host of [...hosts].sort()) set.push({ label: 'host:' + host, kind: 'host', host });
if (this.#giteaPending(business, agent, open)) set.push({ label: 'gitea', kind: 'gitea' });
if (this.#hostGrant)
for (const { host } of this.#store.all('SELECT host FROM hosts WHERE business=? ORDER BY host', business)) {
if (hosts.has(host)) continue;
const tag = this.#hostGrant.grantTag(business, agent, host);
if (this.#hostLinesLeft(business, host).includes(tag) && !this.#lineTried(business, agent, host, tag))
set.push({ label: 'line:' + host, kind: 'line', host, tag });
}
return set;
}
// Section 3 (d4, B1): the agent's host and Gitea grant decisions with no outcome yet. Not denied or
// expired, no grant.applied or grant.failed, and no grant.revoked of the grant since its raise. An
// undecided one counts.
#openAgentDecisions(business, agent) {
return this.#store
.all("SELECT id FROM decisions WHERE business=? AND action='credential.mint' AND route_to='human' ORDER BY at,id", business)
.map((d) => ({ id: d.id, raise: this.#raised(business, d.id) }))
.filter(({ id, raise }) => {
const g = raise?.grant;
if (g?.agent !== agent || (g.method !== 'host' && g.method !== 'gitea')) return false;
const r = this.#closed(id);
if (r && !(r.op === 'resolved' && r.choice === raise.approvalChoice)) return false;
if (
this.#store.get(
`SELECT 1 FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.grantDecision')=? AND json_extract(body,'$.action') IN ('grant.applied','grant.failed') LIMIT 1`,
business,
id,
)
)
return false;
return g.method === 'host' ? !this.#revokedSince(business, id, agent, g.host) : !this.#giteaRevokedSince(business, id, agent);
})
.map(({ id, raise }) => ({ id, grant: raise.grant }));
}
// Sections 3 and 5: whether step G has a Gitea part for the agent. A gitea_grants row, an open Gitea
// decision, a left entry, an open revoke, or a stack-made user no fire has closed.
#giteaPending(business, agent, open) {
if (this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, agent)) return true;
if (open.some((d) => d.grant.method === 'gitea')) return true;
if (this.#giteaLeft(business).some((e) => e.agent === agent)) return true;
if (this.#giteaAgentRevokes(business, agent).length) return true;
return this.#giteaUnfired(business, agent);
}
// A gitea_users row survives a fire (#436b section 6), so the row alone would never let a fired
// agent's set empty. The user counts until a grant.revoked with fire: true and a user names the
// agent after its last Gitea grant step. A new grant after a rehire makes it count again.
#giteaUnfired(business, agent) {
if (!this.#store.get('SELECT 1 FROM gitea_users WHERE business=? AND agent=?', business, agent)) return false;
const last = (where) =>
this.#store.get(
`SELECT MAX(seq) AS seq FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.method')='gitea' AND json_extract(body,'$.agent')=? AND ${where}`,
business,
agent,
)?.seq ?? 0;
const fired = last("json_extract(body,'$.action')='grant.revoked' AND json_extract(body,'$.fire')=1 AND json_extract(body,'$.user') IS NOT NULL");
return fired === 0 || fired < last("json_extract(body,'$.action')='grant.step'");
}
// The agent's open Gitea revokes, less those reconcileGitea is closing.
#giteaAgentRevokes(business, agent) {
return this.#giteaOpenRevokes(business).filter((m) => m.agent === agent && !this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke));
}
// Section 5 (Q1): a job end since the line was last named as left tried it once and listed it in
// grantsLeft. That line doesn't make the set non-empty again.
#lineTried(business, agent, host, tag) {
const since =
this.#store.get(
`SELECT MAX(seq) AS seq FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND (json_extract(body,'$.hostLineLeft')=? OR (json_extract(body,'$.action')='grant.host.swept' AND EXISTS (SELECT 1 FROM json_each(body,'$.left') WHERE value=?)))`,
business,
host,
tag,
tag,
)?.seq ?? 0;
return this.#store
.all(
"SELECT body FROM events WHERE business=? AND kind='action.allowed' AND subject=? AND json_extract(body,'$.action') IN ('fire.done','grants.revoked.done') AND seq > ?",
business,
agent,
since,
)
.some((e) => (JSON.parse(e.body).grantsLeft ?? []).some((l) => l.host === host && l.tag === tag));
}
// Section 3: step G, in the set's order. Each host goes line first (3a), then Gitea (3b, 3c and
// section 5's retries), then each leftover line once (Q1). At the first grant it can't revoke it
// throws { grantRefusal: { grant, reason } }, and what it revoked before that stays revoked and
// recorded. An unexpected throw is reason internal (section 4). Returns the grants revoked and the
// lines still left.
async #stepG(business, agent, set, { job, reason, by, flag }) {
const grants = [];
const grantsLeft = [];
let at = set[0]?.label ?? null;
const refuse = (grant, why) => {
throw { grantRefusal: { grant, reason: why } };
};
try {
for (const g of set.filter((x) => x.kind === 'host')) {
at = g.label;
let out;
try {
out = await this.#revokeHostGrant(business, agent, g.host, reason, by, { lineFirst: true, flag });
} catch (e) {
if (e instanceof BusError && ['host-unknown', 'host-remove-failed', 'host-not-configured'].includes(e.code))
refuse(g.label, e.code === 'host-unknown' ? 'host-unknown' : 'host-remove-failed');
throw e;
}
grants.push({ grant: g.label, revoke: out.revoke, granted: out.granted });
}
if (set.some((x) => x.kind === 'gitea')) {
at = 'gitea';
grants.push(...(await this.#stepGGitea(business, agent, { job, reason, by, flag }, refuse)));
}
for (const g of set.filter((x) => x.kind === 'line')) {
at = g.label;
const record = this.#hostRecord(business, g.host);
let removed = false;
try {
if (record) {
await this.#hostGrant.remove(business, g.host, record, g.tag);
removed = true;
}
} catch {}
if (!removed) {
grantsLeft.push({ host: g.host, tag: g.tag });
continue;
}
const old =
this.#store.get(
`SELECT json_extract(body,'$.revoke') AS revoke FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND json_extract(body,'$.hostLineLeft')=? ORDER BY seq DESC LIMIT 1`,
business,
g.host,
g.tag,
)?.revoke ?? null;
this.#store.transaction(() =>
this.#grantEvent(business, { action: 'grant.revoke.closed', revoke: old, agent, host: g.host, tag: g.tag, hostLine: 'removed', ...flag, by }, g.host),
);
grants.push({ grant: g.label, revoke: old, tag: g.tag });
}
} catch (e) {
if (e?.grantRefusal) throw e;
throw { grantRefusal: { grant: at, reason: 'internal' } };
}
return { grants, grantsLeft };
}
// Section 3b, 3c and 5 (d4, N1): step G's Gitea part. Open decisions end first. Then this agent's
// executors and remote undos are awaited, the admin token is checked, this agent's own left entries
// and open revokes are retried, and the grant goes with fire: true. Anything still left at Gitea
// refuses gitea-left.
async #stepGGitea(business, agent, { job, reason, by, flag }, refuse) {
const key = business + '/' + agent;
const done = [];
const open = this.#openAgentDecisions(business, agent).filter((d) => d.grant.method === 'gitea');
if (open.length) {
// No user and no token: the record only ends the decisions at #giteaRevokedSince (3c).
const revoke = randomUUID();
this.#store.transaction(() => {
this.#grantEvent(
business,
{ action: 'grant.revoked', method: 'gitea', revoke, agent, user: null, userId: null, tokens: [], fire: false, granted: false, decisions: open.map((d) => d.id), reason, by, ...flag },
agent,
);
this.#grantEvent(business, { action: 'grant.revoke.closed', method: 'gitea', revoke, agent, user: null, serviceLeft: [], by }, agent);
});
done.push({ grant: 'gitea', revoke, granted: false });
}
const mine = () => [...this.#giteaRuns].filter(([, k]) => k === key || k === null).map(([p]) => p);
while (mine().length) await Promise.allSettled(mine());
const live = this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, agent) || this.#giteaUnfired(business, agent);
const opens = this.#giteaAgentRevokes(business, agent);
const retry = this.#giteaLeft(business).some((e) => e.agent === agent);
if (!live && !opens.length && !retry) return done;
if (!this.#giteaGrant || !this.#giteaKeys) refuse('gitea', 'gitea-admin-unavailable');
// Preflight: nothing changes unless the admin token checks.
const admin = await this.#giteaAdminChecked(business, this.#giteaLog(business, job));
if (!admin) refuse('gitea', 'gitea-admin-unavailable');
const still = [];
if (opens.length || retry) {
// The sweep's records for this agent alone. Every other agent's entry stays in left.
const sweep = randomUUID();
const all = this.#store.transaction(() => {
this.#grantEvent(business, { action: 'grant.sweep', method: 'gitea', sweep, agent, job, by });
return this.#giteaLeft(business);
});
const kept = [];
const skipped = [];
const closed = [];
for (const m of opens) {
const c = await this.#giteaCloseRevoke(business, m);
closed.push(m.revoke);
for (const id of c.serviceLeft) kept.push({ agent, user: m.user, id });
}
for (const e of all.filter((x) => x.agent === agent)) {
const r = await this.#giteaRetry(business, admin, e);
if (r.skipped) skipped.push({ agent, user: e.user, id: e.id, reason: r.skipped });
else if (!r.ok) kept.push({ agent, user: e.user, id: e.id });
}
const others = all.filter((x) => x.agent !== agent).map((x) => ({ agent: x.agent, user: x.user, id: x.id }));
this.#store.transaction(() =>
this.#grantEvent(business, { action: 'grant.swept', method: 'gitea', sweep, left: [...others, ...kept], skipped, settled: [], closed, agent, job, by }),
);
still.push(...kept);
}
if (live) {
try {
this.#giteaKeys.revoke(business, agent);
} catch (e) {
if (e instanceof BusError && e.code === 'credential-unknown') this.#giteaKeys.abortClaim(business, agent);
else throw e;
}
const m = this.#giteaRevokeLocal(business, agent, reason, by, true, flag.delete ? { delete: true } : {});
const c = await this.#giteaCloseRevoke(business, m);
for (const id of c.serviceLeft) still.push({ agent, user: m.user, id });
done.push({ grant: 'gitea', revoke: m.revoke, granted: m.granted });
}
if (still.length) refuse('gitea', 'gitea-left:' + [...new Set(still.map((e) => e.id))].join(','));
return done;
}
// Row 281 (d6 section 2, 3e): a step G job. The lock is taken now, in the verb's turn, and the job
// starts after the verb answers. The lock goes in a finally, whichever way the job ends.
#startGrantJob(business, agent, verb, run) {
const key = business + '/' + agent;
this.#revoking.set(key, verb);
const job = (async () => {
try {
await new Promise((resolve) => setImmediate(resolve));
await run();
} finally {
this.#revoking.delete(key);
}
})();
this.#grantJobs.add(job);
job.catch(() => {}).finally(() => this.#grantJobs.delete(job));
}
// Section 2, the fire's job: step G, then step B and step C. A step G refusal is section 4's record. A
// step B or C failure is fire.failed (d6, C1). Step C records the authority the verb checked, with no
// holder check, since the chief's run may have ended since.
async #fireJob(sc, args, authority, write, job, set) {
const name = args.name;
let out;
try {
out = await this.#stepG(sc.business, name, set, { job, reason: args.reason, by: sc.role, flag: { fire: true } });
} catch (e) {
this.#store.transaction(() => this.#grantRefused(sc.business, name, 'instance.fire', job, e.grantRefusal, sc.role));
return;
}
let step = 'B';
try {
const result = write('instance.fire');
step = 'C';
this.#store.transaction(() => {
this.#recordAuthority(sc, 'role.fire', authority, { reason: args.reason, transaction: result.transaction ?? null });
this.#event(sc, 'action.allowed', { action: 'fire.done', transaction: result.transaction ?? null, job, grants: out.grants, grantsLeft: out.grantsLeft }, name);
});
} catch (e) {
const code = e instanceof BusError ? e.code : 'outcome-unknown';
const list = out.grants.map((g) => g.grant).join(', ') || 'none';
this.#store.transaction(() => {
this.#event(sc, 'action.allowed', { action: 'fire.failed', job, step, code }, name);
this.#humanNotice(
sc.business,
`Fire of ${name}: job ${job} revoked its grants (${list}) but stopped at step ${step} (${code}). The instance is still firing. Run the fire again once ${code} is cleared.`,
);
});
}
}
// Section 2, the delete's job: step G, then grants.revoked.done and the end message.
async #deleteJob(business, agent, job, set, by, request) {
const verb = 'lifecycle.delete';
let out;
try {
out = await this.#stepG(business, agent, set, { job, reason: `lifecycle.delete ${request}`, by, flag: { delete: true } });
} catch (e) {
this.#store.transaction(() => this.#grantRefused(business, agent, verb, job, e.grantRefusal, by));
return;
}
const left = out.grantsLeft.length ? ` Lines still left: ${out.grantsLeft.map((l) => `${l.tag} on ${l.host}`).join(', ')}.` : '';
this.#store.transaction(() => {
this.#event({ business, role: null, run: null }, 'action.allowed', { action: 'grants.revoked.done', job, agent, verb, grants: out.grants, grantsLeft: out.grantsLeft }, agent);
this.#humanNotice(
business,
`Delete of ${agent}: job ${job} is done. Revoked: ${out.grants.map((g) => g.grant).join(', ') || 'nothing'}.${left} Raise the delete again.`,
);
});
}
// Section 4: a refused step G, as grant evidence and one message to the human. The caller owns the
// transaction.
#grantRefused(business, agent, verb, job, { grant, reason }, by) {
this.#grantEvent(business, { action: 'grant.revoke.refused', job, agent, grant, reason, code: 'grant-revoke-failed', verb, by }, agent);
const fire = verb === 'instance.fire';
const what = `${fire ? 'Fire' : 'Delete'} of ${agent} stopped (job ${job}, grant-revoke-failed)`;
const again = fire ? 'Run the fire again' : 'Raise the delete again';
const host = grant?.startsWith('host:') ? grant.slice('host:'.length) : null;
this.#humanNotice(
business,
reason === 'internal'
? `${what}: step G failed at ${grant} (internal). ${again}. It goes on from what's left.`
: host
? `${what}: its grant on ${host} couldn't be revoked (${reason}). ${again} once ${host} is reachable. If ${host} won't come back, run \`mosaic hosts revoke ${agent} ${host} --reason R\` first: the grant record goes, the line is named as left, and the next ${fire ? 'fire' : 'delete'} completes and names that line.`
: `${what}: its Gitea grant couldn't be revoked (${reason}). ${again} once Gitea is back. To clear it by hand, run \`mosaic grants sweep gitea\` once Gitea is back, or \`mosaic grants revoke ${agent} gitea --reason R\`.`,
);
}
// Section 2: the agent's latest delete job end, if it refused on a grant still in the set.
#deleteRefused(business, agent, labels) {
const e = this.#store.get(
"SELECT body FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.action') IN ('grants.revoked.done','grant.revoke.refused','grants.job.interrupted') AND json_extract(body,'$.verb')='lifecycle.delete' AND json_extract(body,'$.agent')=? ORDER BY seq DESC LIMIT 1",
business,
agent,
);
const b = e && JSON.parse(e.body);
return b?.action === 'grant.revoke.refused' && labels.includes(b.grant) ? b : null;
}
// Row 281: a step G message to the human's inbox, from the bus, as #grantNotice sends one. The caller
// owns the transaction.
#humanNotice(business, body) {
const mid = randomUUID();
this.#store.run(
'INSERT INTO messages(id,at,business,from_role,from_run,to_role,class,in_reply_to,decision,corrects,body) VALUES(?,?,?,?,?,?,?,?,?,?,?)',
mid,
this.#now(),
business,
NOTICE_ROLE,
NOTICE_RUN,
'human',
'RESULT',
null,
null,
null,
body,
);
this.#route(business, mid, 'human', null);
}
// Section 3d, 3e: a grant raise refuses for an agent a fire has stopped or delisted, by the writer's
// own rule, and while a step G job holds the agent. A rehire lists the name again.
#grantableAgent(business, agent) {
if (this.#revoking.has(business + '/' + agent)) fail('grant-revoke-running');
const b = this.#business(business);
if (this.#firing(business, agent) || (Array.isArray(b.launch?.instances) && !b.launch.instances.includes(agent) && b.launch.by !== agent))
fail('instance-fired');
}
// r6 X3 sweep (B4): the tags this business recorded on the host, from grant.step and grant.revoked,
// less those of live grants.
@@ -1623,6 +2057,7 @@ export class Broker {
// key is new, and the chief can't shape it.
#hostGrantBinding(business, g) {
const definition = this.#hostDefinition(business, g.agent);
this.#grantableAgent(business, g.agent);
const h = this.#hostRecord(business, g.host);
if (!h) fail('host-unknown');
if (h.state !== 'ready') fail('host-not-ready');
@@ -1688,6 +2123,7 @@ export class Broker {
// can't shape the block: every line comes from the business, the slot and the store.
#giteaGrantBinding(business, g) {
const { b, role, definition } = this.#giteaRole(business, g.agent);
this.#grantableAgent(business, g.agent);
const targets = this.#giteaTargets(b, role, g.target.projects);
const meta = this.#giteaAdminMeta(business, role.identity.gitea.url);
const kind = this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, g.agent)
@@ -1825,11 +2261,12 @@ export class Broker {
}
return Promise.resolve(code);
}
return this.#track(this.#runGitea(business, decision, start.g, start.signal));
return this.#track(this.#runGitea(business, decision, start.g, start.signal), business + '/' + start.g.agent);
}
#track(promise) {
this.#giteaRuns.add(promise);
promise.finally(() => this.#giteaRuns.delete(promise));
// `key` is business/agent for one agent's work, null for business-wide work.
#track(promise, key = null) {
this.#giteaRuns.set(promise, key);
promise.finally(() => this.#giteaRuns.delete(promise)).catch(() => {});
return promise;
}
// Trusted API, as runHostGrant is, and no verb reaches it: one decision's executor, to its end.
@@ -1838,7 +2275,12 @@ export class Broker {
}
// Trusted API for the tests: resolves once every Gitea executor and remote undo in flight has ended.
async giteaIdle() {
while (this.#giteaRuns.size) await Promise.allSettled([...this.#giteaRuns]);
while (this.#giteaRuns.size) await Promise.allSettled([...this.#giteaRuns.keys()]);
}
// Row 281 (d6 section 8): resolves once every step G job, Gitea executor and host executor has ended.
async grantsIdle() {
while (this.#grantJobs.size || this.#giteaRuns.size || this.#hostRuns.size)
await Promise.allSettled([...this.#grantJobs, ...this.#giteaRuns.keys(), ...[...this.#hostRuns.values()].flatMap((s) => [...s])]);
}
// Section 4 steps 0 (the remote half) to 8. Never throws: the outcome is in the store and the
// notices, and the return value is 'applied', 'settle-pending' or the code grant.failed records.
@@ -2288,7 +2730,7 @@ export class Broker {
// Section 6 step 2, in one transaction: the record goes and grant.revoked records what step 3 deletes.
// That is the record's token, and both tokens of a rotation whose commit never settled (settle-pending,
// or grant-token-unknown, whose own unknown id stays for Jason).
#giteaRevokeLocal(business, agent, reason, by, fire) {
#giteaRevokeLocal(business, agent, reason, by, fire, extra = {}) {
const revoke = randomUUID();
return this.#store.transaction(() => {
const u = this.#store.get('SELECT user, id FROM gitea_users WHERE business=? AND agent=?', business, agent);
@@ -2310,7 +2752,7 @@ export class Broker {
this.#store.run('DELETE FROM gitea_grants WHERE business=? AND agent=?', business, agent);
const body = {
action: 'grant.revoked', method: 'gitea', revoke, agent, user: u?.user ?? null, userId: u?.id ?? null,
tokens: [...tokens], fire, granted: Boolean(live), reason, by,
tokens: [...tokens], fire, granted: Boolean(live), reason, by, ...extra,
};
this.#grantEvent(business, body, agent);
return body;
@@ -2409,15 +2851,18 @@ export class Broker {
const log = this.#giteaLog(business, sweep);
const left = this.#giteaLeft(business);
const closed = [];
// Row 281 (d6 section 3e): an agent a step G job holds is skipped, and named as running.
const held = (agent) => this.#revoking.has(business + '/' + agent);
const running = [...this.#revoking.keys()].filter((k) => k.startsWith(business + '/')).map((k) => k.slice(business.length + 1));
for (const m of this.#giteaOpenRevokes(business)) {
if (this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke)) continue;
if (this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke) || held(m.agent)) continue;
const c = await this.#giteaCloseRevoke(business, m);
closed.push(m.revoke);
for (const id of c.serviceLeft) if (!left.some((e) => e.agent === m.agent && e.id === id)) left.push({ agent: m.agent, user: m.user, id, fresh: true });
}
const settled = [];
for (const o of this.#giteaGrant.openGrants(this.#store, business)) {
if (o.step?.step !== 'commit' || o.step.previousTokenId === null || o.revokedAfter) continue;
if (o.step?.step !== 'commit' || o.step.previousTokenId === null || o.revokedAfter || held(o.agent)) continue;
let signal;
try {
signal = this.#giteaKeys.claim(business, o.agent);
@@ -2438,31 +2883,33 @@ export class Broker {
const still = [];
const skipped = [];
for (const e of left) {
if (e.fresh) {
if (e.fresh || held(e.agent)) {
still.push({ agent: e.agent, user: e.user, id: e.id });
continue;
}
let ok = false;
if (admin && e.user) {
if (e.id === 'tokens' || e.id === 'login') {
// A fire retry checks the user's id first, as the revoke close does: the gitea_users row
// survives a fire, and a user remade under the same name is not the one that was fired.
const u = this.#store.get('SELECT id FROM gitea_users WHERE business=? AND agent=? AND user=?', business, e.agent, e.user);
const r = await admin.api.userGet(e.user);
const reason = r.status === 404 ? 'gitea-user-gone' : r.status === 200 && (!u || r.json?.id !== u.id) ? 'gitea-user-foreign' : null;
if (reason) {
skipped.push({ agent: e.agent, user: e.user, id: e.id, reason });
continue;
}
if (r.status === 200) ok = !(await this.#giteaFire(admin.api, e.user)).includes(e.id);
} else ok = await this.#giteaDelete(admin.api, e.user, e.id).catch(() => false);
}
if (!ok) still.push({ agent: e.agent, user: e.user, id: e.id });
const r = await this.#giteaRetry(business, admin, e);
if (r.skipped) skipped.push({ agent: e.agent, user: e.user, id: e.id, reason: r.skipped });
else if (!r.ok) still.push({ agent: e.agent, user: e.user, id: e.id });
}
this.#store.transaction(() =>
this.#grantEvent(business, { action: 'grant.swept', method: 'gitea', sweep, left: still, skipped, settled, closed, by }),
);
return { left: still, skipped, settled, closed };
return { left: still, skipped, settled, closed, ...(running.length ? { running } : {}) };
}
// One left entry's retry: a fire step checks the user's id first, a token goes by id. Returns
// { ok } or { skipped: reason }. The sweep and a step G job share it (row 281).
async #giteaRetry(business, admin, e) {
if (!admin || !e.user) return { ok: false };
if (e.id === 'tokens' || e.id === 'login') {
// A fire retry checks the user's id first, as the revoke close does: the gitea_users row
// survives a fire, and a user remade under the same name is not the one that was fired.
const u = this.#store.get('SELECT id FROM gitea_users WHERE business=? AND agent=? AND user=?', business, e.agent, e.user);
const r = await admin.api.userGet(e.user);
const reason = r.status === 404 ? 'gitea-user-gone' : r.status === 200 && (!u || r.json?.id !== u.id) ? 'gitea-user-foreign' : null;
if (reason) return { skipped: reason };
return { ok: r.status === 200 && !(await this.#giteaFire(admin.api, e.user)).includes(e.id) };
}
return { ok: await this.#giteaDelete(admin.api, e.user, e.id).catch(() => false) };
}
// `mosaic grants status gitea`.
#giteaStatus(business) {
@@ -2570,7 +3017,7 @@ export class Broker {
keys(a, ['decision'], ['decision']);
identifier(a.decision);
const { g, signal } = this.#giteaBegin(s.business, a.decision);
this.#track(this.#runGitea(s.business, a.decision, g, signal));
this.#track(this.#runGitea(s.business, a.decision, g, signal), s.business + '/' + g.agent);
return { decision: a.decision, agent: g.agent, started: true };
}
if (a.service !== 'gitea') fail('invalid-request');
@@ -2587,6 +3034,8 @@ export class Broker {
if (typeof a.agent !== 'string' || !ID_PATTERN.test(a.agent)) fail('invalid-request');
string(a.reason, 500);
if (!Object.hasOwn(this.#business(s.business).roles, a.agent)) fail('credential-unknown');
// Row 281 (d6 section 3e): no manual revoke while a step G job holds the agent.
if (this.#revoking.has(s.business + '/' + a.agent)) fail('grant-revoke-running');
// Step 1: memory, the abort of a use or an executor in flight, the file.
try {
this.#giteaKeys.revoke(s.business, a.agent);
@@ -2595,7 +3044,7 @@ export class Broker {
else throw e;
}
const m = this.#giteaRevokeLocal(s.business, a.agent, a.reason, by, false);
const closed = await this.#track(this.#giteaCloseRevoke(s.business, m));
const closed = await this.#track(this.#giteaCloseRevoke(s.business, m), s.business + '/' + a.agent);
return { agent: a.agent, revoke: m.revoke, granted: m.granted, serviceLeft: closed.serviceLeft };
}
return await this.#giteaAdmin(s.business, verb, a, by);
@@ -2714,13 +3163,17 @@ export class Broker {
host,
);
const left = swept ? JSON.parse(swept.body).left : [];
// Row 281 (section 5): a fire's or a delete's close with the line removed clears its tag.
for (const e of this.#store.all(
`SELECT json_extract(body,'$.hostLineLeft') AS tag FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND json_extract(body,'$.hostLineLeft') IS NOT NULL AND seq > ? ORDER BY seq`,
`SELECT json_extract(body,'$.hostLineLeft') AS tag, json_extract(body,'$.tag') AS removed FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND (json_extract(body,'$.hostLineLeft') IS NOT NULL OR (json_extract(body,'$.action')='grant.revoke.closed' AND json_extract(body,'$.hostLine')='removed' AND (json_extract(body,'$.fire')=1 OR json_extract(body,'$.delete')=1))) AND seq > ? ORDER BY seq`,
business,
host,
swept?.seq ?? 0,
))
if (!left.includes(e.tag)) left.push(e.tag);
)) {
if (e.tag !== null) {
if (!left.includes(e.tag)) left.push(e.tag);
} else if (left.includes(e.removed)) left.splice(left.indexOf(e.removed), 1);
}
return left;
}
#hardWait(s, action, target) {
@@ -2987,6 +3440,15 @@ export class Broker {
);
return result;
}
// Row 281 (d6, C1): #consumeAuthority less the holder check, for a fire's step C in its job. The chief
// held its claim when the verb ran #checkAuthority, and its run may have ended since, as a landing's
// record allows (row 137). Caller owns the transaction.
#recordAuthority(s, action, context = {}, detail = {}) {
const result = this.#authorityFor(s, action, context);
if (result.decision && this.#spent(s.business, result.decision)) fail('decision-consumed');
this.#event(s, 'action.allowed', { ...detail, action, ...result, target: context.target ?? null }, context.target ?? null);
return result;
}
// Trusted S3 handler calls inside its own operation; not a generic socket action executor.
// Row 93 (#1587): task.close on a task with a recorded push also needs
// context.verdict, checked by #closeVerdict; a refusal there rolls the action.allowed back.
@@ -3366,6 +3828,32 @@ export class Broker {
if (this.#openRequests(record.business, r.instance).some((o) => o.request !== r.request && o.action !== 'lifecycle.end' &&
(o.state === 'taken' || !this.#expired(o)))) fail('instance-starting');
if (!this.#organization) fail('not-configured');
// Row 281 (#1786, d6 section 2, sage-2575): a delete of an instance with a grant takes two requests.
// This one starts step G as a job and refuses at once, and the human raises the delete again when
// the inbox says the job is done. With no grant, the delete goes on as before.
if (this.#revoking.has(record.business + '/' + r.instance)) fail('grants-revoking');
const set = this.#store.transaction(() => this.#grantSet(record.business, r.instance));
if (set.length) {
const job = randomUUID();
const labels = set.map((g) => g.label);
const refused = this.#deleteRefused(record.business, r.instance, labels);
this.#store.transaction(() => {
this.#event(
{ business: record.business, role: null, run: null },
'action.allowed',
{ action: 'grants.revoking', job, agent: r.instance, verb: 'lifecycle.delete', request: r.request, grants: labels },
r.instance,
);
this.#humanNotice(
record.business,
refused
? `Delete of ${r.instance} refused: job ${refused.job} couldn't revoke ${refused.grant} (${refused.reason}). Retrying now as job ${job} for: ${labels.join(', ')}. Raise the delete again when the inbox says it's done, or clear ${refused.grant} by hand first.`
: `Delete of ${r.instance} is waiting on its grants. Revoking started (job ${job}) for: ${labels.join(', ')}. Raise the delete again when the inbox says the revoke is done.`,
);
});
this.#startGrantJob(record.business, r.instance, 'lifecycle.delete', () => this.#deleteJob(record.business, r.instance, job, set, r.by, r.request));
fail(refused ? 'grant-revoke-failed' : 'grants-revoking');
}
let out;
try {
out = this.#organization(record.business, 'instances.set', { revision: r.revision, instance: r.instance, op: 'delete' }, r.by);
@@ -3757,7 +4245,7 @@ export class Broker {
this.#store.get('SELECT 1 FROM gitea_users WHERE business=? AND agent=?', s.business, a.agent)
) {
const m = this.#giteaRevokeLocal(s.business, a.agent, a.reason, s.tester ? 'tester' : s.human, verb === 'credentials.delete');
setImmediate(() => this.#track(this.#giteaCloseRevoke(s.business, m)).catch(() => {}));
setImmediate(() => this.#track(this.#giteaCloseRevoke(s.business, m), s.business + '/' + a.agent).catch(() => {}));
return { ...result, giteaRevoke: m.revoke };
}
return result;
@@ -3805,6 +4293,8 @@ export class Broker {
// The last listed name refuses before step A stops anything, as the writer would at step B.
const listed = b.launch?.instances ?? [];
if (listed.length === 1 && listed[0] === args.name) fail('invalid-request');
// Row 281 (d6 section 3e): one step G per agent. A repeat while its job runs refuses.
if (this.#revoking.has(s.business + '/' + args.name)) fail('fire-in-progress');
// webby-0094 B1: the same checks against the file. A keep this copy hasn't loaded (a reload the
// credentials refused) would otherwise pass here, and step B's refusal would leave I firing.
write('instance.fireCheck');
@@ -3827,8 +4317,21 @@ export class Broker {
if (request)
this.#event(s, 'action.allowed', { action: 'lifecycle.end', request, via: 'chief', by: s.role, runs, reason: args.reason, kill: true }, args.name);
this.#event(s, 'action.allowed', { action: 'fire.stopped', request }, args.name);
return { request, runs, cancelled, taken };
// Row 281 (#1786, d6 section 2): the grant set, read after fire.stopped, so a grant raise either
// committed before this and is in the set, or refuses instance-fired (3d).
const set = this.#grantSet(s.business, args.name);
if (!set.length) return { request, runs, cancelled, taken };
const job = randomUUID();
this.#event(s, 'action.allowed', { action: 'fire.revoking', job, grants: set.map((g) => g.label) }, args.name);
return { request, runs, cancelled, taken, job, set };
});
// A non-empty set: step G, B and C run as a job, and the verb answers now with no transaction.
if (stopped.job) {
const { job, set, ...rest } = stopped;
const sc = { business: s.business, role: s.role, run: s.run, thread: s.thread, human: null };
this.#startGrantJob(s.business, args.name, verb, () => this.#fireJob(sc, args, authority, write, job, set));
return { ...rest, job, revoking: set.map((g) => g.label), done: false };
}
// Step B: the delist. An instance already delisted counts as done.
const result = write('instance.fire');
// Step C: log the fire and record it done. A failure at B or C leaves I firing until a repeat.
@@ -0,0 +1,954 @@
// Row 281 (#1786, design d6 section 8): a fire or a delete revokes the instance's host and Gitea grants
// (step G), as a job after the verb answers. T2-T13. The broker runs bare, as chief-of-staff.test.mjs
// runs it, with a fake organization writer, a fake installer in place of the host's remote steps, and
// the fake Gitea from fake-gitea-api.mjs on 127.0.0.1. Every token is a fake the fake made; no test
// reaches git.mosaicstack.dev or any provider, and none creates a real account or token (Q188). Host
// keys are real local ssh-keygen pairs in the scratch store, and no ssh connection is made. Decisions a
// test needs outside the human's yes go in by SQL as fixtures.
import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { Broker, GRANT_EVIDENCE, registrySha256 } from '../src/broker.mjs';
import { Credentials } from '../src/credentials.mjs';
import { adminSha256, definitionSha256 as giteaDefinitionSha256, giteaGrant, giteaKeys, identitySha256 } from '../src/gitea-grant.mjs';
import { definitionSha256 as hostDefinitionSha256, grantKeys, grantTag, hostGrant } from '../src/host-grant.mjs';
import { Store } from '../src/store.mjs';
import { fakeGiteaApi } from './fake-gitea-api.mjs';
import { grantDecision, hostRecord } from './host-fixtures.mjs';
const AT = '2026-10-22T00:00:00.000Z';
const REV = 'b'.repeat(64);
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const dayFrom = (n) => new Date(Date.now() + n * 86400000).toISOString().slice(0, 10);
const sha256 = (s) => createHash('sha256').update(s).digest('hex');
const options = [{ key: 'yes', text: 'Yes' }, { key: 'no', text: 'No' }];
const none = { authority: { withinRole: ['message.send'], crossRole: [] } };
const chiefAuthority = JSON.parse(readFileSync(fileURLToPath(new URL('../../../roles/chief-of-staff.json', import.meta.url)), 'utf8')).authority;
const ENTRY = { address: '127.0.0.1', port: 2222, user: 'deploy', hostKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl'] };
const HOSTS = ['web1', 'web2', 'web3'];
const GENERATION = '0b6e7f6e-4c1d-4a8e-9a51-6d2f3c4b5a69';
async function until(fn, ms = 10000) {
const end = Date.now() + ms;
while (!fn()) {
if (Date.now() > end) throw new Error('timed out');
await sleep(10);
}
}
const rejects = (fn, code) => assert.rejects(async () => fn(), (e) => (assert.equal(e.code, code), true));
const code = (fn) => {
try {
fn();
} catch (e) {
return e.code;
}
return null;
};
function makeBusiness(url) {
const gitea = (name, user) => ({ name, email: `${user}@example.test`, gitea: { url, user } });
return {
id: 'demo',
human: 'jason',
arbiters: { technical: 'pm', delivery: 'pm' },
roles: {
chief: { definition: 'chief-of-staff', authority: chiefAuthority },
pm: { definition: 'pm', ...none },
coder: { definition: 'coder', ...none, identity: gitea('Coder', 'demo-coder'), publish: ['app'] },
helper: { definition: 'coder', ...none, identity: gitea('Helper', 'demo-helper'), publish: ['app'] },
ops: { definition: 'ops', ...none },
},
definitions: {
'chief-of-staff': { roleVersion: 2, credentials: [] },
pm: { roleVersion: 2, credentials: [] },
coder: { roleVersion: 2, credentials: [{ service: 'gitea' }, { service: 'host' }] },
ops: { roleVersion: 2, credentials: [{ service: 'host' }] },
},
projects: { app: { remote: `${url}/acme/app` } },
launch: { by: 'chief', instances: ['pm', 'coder', 'helper', 'ops'] },
vars: { 'bus.tester': true },
};
}
async function setup(t) {
const dir = mkdtempSync(join(tmpdir(), 'g281-'));
t.after(() => rmSync(dir, { recursive: true, force: true }));
const gitea = await fakeGiteaApi(t, join(dir, 'tls'));
gitea.addUser('acme');
gitea.addRepo('acme', 'app');
const configDir = join(dir, 'cfg');
mkdirSync(configDir, { mode: 0o700 });
const storeRoot = join(configDir, 'secrets', 'agents');
const runRoot = join(configDir, 'secrets', 'run');
mkdirSync(join(dir, 'data'), { mode: 0o700 });
const store = new Store(join(dir, 'data'));
const credentials = new Credentials({ references: {}, storeRoot, dataRoot: join(dir, 'data') });
t.after(() => {
// A test that fails while a host step is held must not leave pause() looping after it ends.
held.clear();
failing.clear();
credentials.close();
store.close();
});
const doc = makeBusiness(gitea.url);
const grantApi = giteaGrant({ configDir, ca: gitea.ca, timeoutMs: 5000 });
const meta = { url: gitea.url, account: 'mosaic-grant', rotateBy: dayFrom(60), setAt: AT };
grantApi.writeAdmin('demo', gitea.adminToken, meta);
const gKeys = giteaKeys({ credentials, storeRoot });
const hKeys = grantKeys({ credentials, storeRoot });
// The fake installer: each host's lines as a set of tags. `held` keeps a host's add or remove from
// answering until the test deletes the key; `failing` makes every request to a host fail.
const lines = Object.fromEntries(HOSTS.map((h) => [h, new Set()]));
const sent = [];
const held = new Set();
const failing = new Set();
const pause = async (key, signal) => {
while (held.has(key)) {
if (signal?.aborted) throw new Error('aborted');
await sleep(5);
}
if (signal?.aborted) throw new Error('aborted');
};
const installer = {
...hostGrant({ configDir, runRoot }),
async install(business, host, record, tag, publicLine, signal) {
sent.push(`add ${host} ${tag}`);
await pause(host + ':add', signal);
if (failing.has(host)) throw new Error('host-failed');
lines[host].add(tag);
},
async verify(host, record, key, signal) {
if (signal?.aborted) throw new Error('aborted');
if (failing.has(host)) throw new Error('host-failed');
},
async remove(business, host, record, tag) {
sent.push(`remove ${host} ${tag}`);
await pause(host + ':remove');
if (failing.has(host)) throw new Error('host-failed');
lines[host].delete(tag);
},
};
// The fake organization writer. A fire delists the name and a delete drops the role, each with a
// reload, as the runtime reloads after a commit. `fireError` makes the fire's write refuse.
const org = { log: [], fireError: null };
const sw = { state: { version: 1, halted: false, generation: GENERATION } };
let broker;
const organization = (business, op, args) => {
org.log.push([op, args?.name ?? args?.instance ?? null]);
if (op === 'instance.fire') {
if (org.fireError) throw org.fireError;
doc.launch.instances = doc.launch.instances.filter((n) => n !== args.name);
broker.reloadBusiness('demo', doc);
return { revision: REV, transaction: 't-' + org.log.length };
}
if (op === 'instances.set') {
delete doc.roles[args.instance];
doc.launch.instances = doc.launch.instances.filter((n) => n !== args.instance);
broker.reloadBusiness('demo', doc);
return { revision: REV, transaction: 't-' + org.log.length };
}
return { revision: REV };
};
const make = () =>
new Broker({
store,
businesses: { demo: structuredClone(doc) },
secretCheck: (value) => credentials.assertClean(value),
organization,
processing: { read: () => sw.state },
hostGrant: installer,
hostKeys: hKeys,
giteaGrant: grantApi,
giteaKeys: gKeys,
});
broker = make();
for (const host of HOSTS) hostRecord(store, { host, entry: ENTRY });
const caps = {};
for (const role of ['chief', 'pm']) {
caps[role] = broker.bindLaunch({ business: 'demo', role, run: role + '-run', harness: 'pi', address: role });
broker.request(caps[role], { verb: 'role.claim' });
}
caps.human = broker.bindHuman({ business: 'demo', human: 'jason', via: 'cli', outsideAgent: true });
caps.tester = broker.bindTester({ business: 'demo' });
const call = (who, verb, args = {}) => broker.request(caps[who], { verb, args });
const grantEvents = () =>
store.all(`SELECT body FROM events WHERE business='demo' AND ${GRANT_EVIDENCE} ORDER BY seq`).map((e) => JSON.parse(e.body));
const ended = (decision) => grantEvents().findLast((e) => e.grantDecision === decision && ['grant.applied', 'grant.failed'].includes(e.action));
const allowed = (action) =>
store
.all("SELECT actor_role, subject, body FROM events WHERE business='demo' AND kind='action.allowed' ORDER BY seq")
.map((e) => ({ ...e, body: JSON.parse(e.body) }))
.filter((e) => e.body.action === action);
// The bus's step G messages to the human: no decision, so no grant notice.
const inbox = () =>
store.all("SELECT body FROM messages WHERE business='demo' AND from_role='_bus' AND to_role='human' AND decision IS NULL ORDER BY rowid").map((r) => r.body);
const raiseHost = (agent, host, commands = [['uptime']]) =>
call('chief', 'decision.raise', {
action: 'credential.mint', question: `Grant ${agent} ${host}?`, options, recommendation: 'yes', blocking: false,
grant: { method: 'host', agent, host, commands, rotateBy: dayFrom(30) },
});
const raiseGitea = (agent) =>
call('chief', 'decision.raise', {
action: 'credential.mint', question: `Grant ${agent} Gitea?`, options, recommendation: 'yes', blocking: false,
grant: { method: 'gitea', agent, target: { projects: ['app'] }, rotateBy: dayFrom(30) },
});
const approve = async (d) => {
call('human', 'decision.resolve', { id: d.id, choice: 'yes' });
await until(() => ended(d.id));
await broker.grantsIdle();
return ended(d.id);
};
const approveHost = async (agent, host) => {
const end = await approve(await raiseHost(agent, host));
assert.equal(end.action, 'grant.applied', `${agent} on ${host}`);
return end;
};
const approveGitea = async (agent) => {
const end = await approve(await raiseGitea(agent));
assert.equal(end.action, 'grant.applied', `${agent} gitea`);
return end;
};
// A host decision approved and never run, with R4's body built here (T6).
const fixtureHost = (agent, host, commands = [['uptime']]) =>
grantDecision(store, {
grant: {
method: 'host', agent, host, commands, commandsSha256: sha256(JSON.stringify(commands)), rotateBy: dayFrom(30),
registrySha256: registrySha256({ ...ENTRY, sudo: true }), definitionSha256: hostDefinitionSha256(doc.definitions[doc.roles[agent].definition]),
},
});
// A Gitea decision approved and never run (T6b).
const fixtureGitea = (agent) =>
grantDecision(store, {
grant: {
method: 'gitea', agent, target: { projects: ['app'] }, rotateBy: dayFrom(30),
definitionSha256: giteaDefinitionSha256(doc.definitions[doc.roles[agent].definition]),
identitySha256: identitySha256(doc.roles[agent], doc.projects),
adminSha256: adminSha256(grantApi.adminMeta('demo') ?? meta),
},
});
const fire = (name) => call('chief', 'instance.fire', { name, reason: 'the work is done' });
// A delete as the launcher's pass runs it: the human's request, the take, then lifecycleDelete. A
// refused delete closes the request with its code, as the launcher does.
const remove = (instance) => {
const { request } = call('human', 'instances.set', { revision: REV, instance, op: 'delete' });
assert.equal(broker.lifecycleTake({ business: 'demo', request }).go, true);
try {
return { request, ...broker.lifecycleDelete({ business: 'demo', request }) };
} catch (e) {
if (typeof e.code !== 'string') throw e;
broker.lifecycleDone({ business: 'demo', request, refusal: e.code });
return { request, code: e.code };
}
};
const tag = (agent, host) => grantTag('demo', agent, host);
const keyFile = (agent, host) => join(storeRoot, 'demo', agent, 'host.' + host);
const tokenFile = (agent) => join(storeRoot, 'demo', agent, 'gitea');
const hostRow = (agent, host) => store.get("SELECT * FROM host_grants WHERE business='demo' AND agent=? AND host=?", agent, host);
const giteaRow = (agent) => store.get("SELECT * FROM gitea_grants WHERE business='demo' AND agent=?", agent);
const state = (agent, service) => credentials.status().find((e) => e.instance === 'demo/' + agent && e.service === service)?.state;
const userTokens = (login) => gitea.tokens(login) ?? [];
const revoked = (agent, where = () => true) => grantEvents().filter((e) => e.action === 'grant.revoked' && e.agent === agent && where(e));
const closed = (revoke) => grantEvents().filter((e) => e.action === 'grant.revoke.closed' && e.revoke === revoke);
const noLeak = () => {
const text = [
...store.all("SELECT body FROM events WHERE business='demo'").map((r) => r.body),
...store.all("SELECT body FROM messages WHERE business='demo'").map((r) => r.body),
JSON.stringify(gitea.requests),
].join('\n');
assert.ok(gitea.issued.every((tok) => !text.includes(tok)), 'a token value reached the store or the request log');
};
return {
dir, gitea, store, credentials, doc, org, sw, lines, sent, held, failing, caps, call, make, grantEvents, ended, allowed, inbox,
raiseHost, raiseGitea, approve, approveHost, approveGitea, fixtureHost, fixtureGitea, fire, remove, tag, keyFile, tokenFile,
hostRow, giteaRow, state, userTokens, revoked, closed, noLeak,
get broker() {
return broker;
},
};
}
// Coder's three grants: web1, web2 and Gitea, each through the human's yes and its executor.
async function threeGrants(f) {
await f.approveHost('coder', 'web1');
await f.approveHost('coder', 'web2');
await f.approveGitea('coder');
for (const h of ['web1', 'web2']) {
assert.ok(f.lines[h].has(f.tag('coder', h)));
assert.ok(existsSync(f.keyFile('coder', h)));
}
assert.equal(f.userTokens('demo-coder').length, 1);
assert.ok(existsSync(f.tokenFile('coder')));
}
// Every grant of coder is gone: records, key and token files, host lines, Gitea tokens, login.
function allRevoked(f, hosts = ['web1', 'web2']) {
for (const h of hosts) {
assert.equal(f.hostRow('coder', h), undefined, h);
assert.equal(existsSync(f.keyFile('coder', h)), false, h);
assert.equal(f.lines[h].has(f.tag('coder', h)), false, h);
assert.notEqual(f.state('coder', 'host.' + h), 'valid', h);
}
assert.equal(f.giteaRow('coder'), undefined);
assert.equal(existsSync(f.tokenFile('coder')), false);
assert.notEqual(f.state('coder', 'gitea'), 'valid');
assert.deepEqual(f.userTokens('demo-coder'), []);
assert.equal(f.gitea.users.get('demo-coder').prohibit_login, true);
}
test('row 281 T2: a fire with host grants on two hosts and a Gitea grant answers with a job, then revokes all three', async (t) => {
const f = await setup(t);
await threeGrants(f);
const out = f.fire('coder');
assert.equal(out.done, false);
assert.match(out.job, UUID);
assert.deepEqual(out.revoking, ['host:web1', 'host:web2', 'gitea']);
assert.ok(!('transaction' in out));
const [revoking, ...more] = f.allowed('fire.revoking');
assert.equal(more.length, 0);
assert.equal(revoking.body.job, out.job);
assert.deepEqual(revoking.body.grants, out.revoking);
assert.equal(revoking.actor_role, 'chief');
assert.equal(revoking.subject, 'coder');
await f.broker.grantsIdle();
// Each host as hosts.revoke records it, with fire: true, line first so the close says removed.
for (const h of ['web1', 'web2']) {
const [r, ...rest] = f.revoked('coder', (e) => e.host === h);
assert.equal(rest.length, 0, h);
assert.equal(r.fire, true);
assert.equal(r.granted, true);
assert.equal(r.hostLine, 'pending');
assert.equal(r.tag, f.tag('coder', h));
const [c] = f.closed(r.revoke);
assert.equal(c.hostLine, 'removed');
assert.equal(c.fire, true);
}
const [g, ...rest] = f.revoked('coder', (e) => e.method === 'gitea');
assert.equal(rest.length, 0);
assert.equal(g.user, 'demo-coder');
assert.equal(g.granted, true);
assert.deepEqual(f.closed(g.revoke).map((c) => c.serviceLeft), [[]]);
allRevoked(f);
const [done, ...again] = f.allowed('fire.done');
assert.equal(again.length, 0);
assert.equal(done.body.job, out.job);
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'host:web2', 'gitea']);
assert.deepEqual(done.body.grantsLeft, []);
assert.match(done.body.transaction, /^t-/);
assert.equal(done.actor_role, 'chief');
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
assert.ok(!f.doc.launch.instances.includes('coder'));
// No step G message for a job that completed.
assert.deepEqual(f.inbox(), []);
f.noLeak();
});
test('row 281 T3 (sage-2575): a delete with grants takes two requests: the first starts the job, the next completes', async (t) => {
const f = await setup(t);
await threeGrants(f);
f.held.add('web1:remove');
const first = f.remove('coder');
assert.equal(first.code, 'grants-revoking');
const [start, ...more] = f.allowed('grants.revoking');
assert.equal(more.length, 0);
assert.match(start.body.job, UUID);
assert.deepEqual(start.body.grants, ['host:web1', 'host:web2', 'gitea']);
assert.equal(start.body.verb, 'lifecycle.delete');
assert.equal(start.body.request, first.request);
assert.equal(start.actor_role, null);
assert.equal(start.subject, 'coder');
const job = start.body.job;
const [message] = f.inbox();
assert.equal(f.inbox().length, 1);
for (const s of [job, 'host:web1', 'host:web2', 'gitea', 'Raise the delete again']) assert.ok(message.includes(s), s);
// Held: a second delete refuses grants-revoking, with no second job and no second message.
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
assert.equal(f.remove('coder').code, 'grants-revoking');
assert.equal(f.allowed('grants.revoking').length, 1);
assert.equal(f.inbox().length, 1);
f.held.delete('web1:remove');
await f.broker.grantsIdle();
for (const h of ['web1', 'web2']) assert.equal(f.revoked('coder', (e) => e.host === h && e.delete === true).length, 1, h);
allRevoked(f);
const [done] = f.allowed('grants.revoked.done');
assert.equal(done.body.job, job);
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'host:web2', 'gitea']);
const end = f.inbox()[1];
for (const s of [job, 'host:web1', 'host:web2', 'gitea', 'Raise the delete again']) assert.ok(end.includes(s), s);
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
// The third request completes as today.
const third = f.remove('coder');
assert.equal(third.code, undefined);
assert.match(third.transaction, /^t-/);
assert.equal(third.retry, false);
assert.deepEqual(f.org.log.filter(([op]) => op === 'instances.set'), [['instances.set', 'coder']]);
assert.equal(f.allowed('grants.revoking').length, 1);
f.noLeak();
});
test('row 281 T3b (Q2): a delete whose job refuses on host B; the retry refuses grant-revoke-failed and revokes only B and Gitea', async (t) => {
const f = await setup(t);
await threeGrants(f);
f.failing.add('web2');
assert.equal(f.remove('coder').code, 'grants-revoking');
await f.broker.grantsIdle();
const job = f.allowed('grants.revoking')[0].body.job;
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
assert.equal(more.length, 0);
assert.deepEqual(
{ job: refused.job, agent: refused.agent, grant: refused.grant, reason: refused.reason, code: refused.code, verb: refused.verb },
{ job, agent: 'coder', grant: 'host:web2', reason: 'host-remove-failed', code: 'grant-revoke-failed', verb: 'lifecycle.delete' },
);
const end = f.inbox()[1];
for (const s of [job, 'coder', 'web2', 'host-remove-failed', 'mosaic hosts revoke coder web2']) assert.ok(end.includes(s), s);
// Nothing deleted: the writer saw no delete and the role is still there. Host A is revoked, B isn't.
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
assert.ok(f.doc.roles.coder);
assert.equal(f.hostRow('coder', 'web1'), undefined);
assert.ok(f.hostRow('coder', 'web2'));
assert.ok(existsSync(f.keyFile('coder', 'web2')));
assert.ok(f.giteaRow('coder'));
assert.equal(f.allowed('grants.revoked.done').length, 0);
// B back: the next delete refuses grant-revoke-failed, names the earlier reason and the new job.
f.failing.delete('web2');
assert.equal(f.remove('coder').code, 'grant-revoke-failed');
const retry = f.allowed('grants.revoking')[1].body;
assert.deepEqual(retry.grants, ['host:web2', 'gitea']);
const message = f.inbox()[2];
for (const s of [job, retry.job, 'host:web2', 'host-remove-failed']) assert.ok(message.includes(s), s);
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
await f.broker.grantsIdle();
const [done] = f.allowed('grants.revoked.done');
assert.equal(done.body.job, retry.job);
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web2', 'gitea']);
// Host A was sent one remove only, by the first job.
assert.equal(f.sent.filter((x) => x === `remove web1 ${f.tag('coder', 'web1')}`).length, 1);
allRevoked(f);
assert.match(f.remove('coder').transaction, /^t-/);
});
test('row 281 T3b: B cleared by hand with hosts.revoke; the next delete starts a job for Gitea and the leftover line only', async (t) => {
const f = await setup(t);
await threeGrants(f);
f.failing.add('web2');
assert.equal(f.remove('coder').code, 'grants-revoking');
await f.broker.grantsIdle();
const manual = await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web2', reason: 'web2 is gone' });
assert.equal(manual.hostLineLeft, f.tag('coder', 'web2'));
assert.equal(f.hostRow('coder', 'web2'), undefined);
// The refused grant isn't in the set any more, so this is a new start, not a retry.
assert.equal(f.remove('coder').code, 'grants-revoking');
const second = f.allowed('grants.revoking')[1].body;
assert.deepEqual(second.grants, ['gitea', 'line:web2']);
await f.broker.grantsIdle();
const done = f.allowed('grants.revoked.done')[0].body;
assert.equal(done.job, second.job);
assert.deepEqual(done.grants.map((x) => x.grant), ['gitea']);
assert.deepEqual(done.grantsLeft, [{ host: 'web2', tag: f.tag('coder', 'web2') }]);
assert.ok(f.inbox().at(-1).includes(`${f.tag('coder', 'web2')} on web2`));
// The line was tried once and listed: it doesn't hold the delete again.
assert.match(f.remove('coder').transaction, /^t-/);
assert.equal(f.allowed('grants.revoking').length, 2);
});
test('row 281 T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request', async (t) => {
const f = await setup(t);
const before = f.store.get("SELECT count(*) AS n FROM events WHERE business='demo'").n;
const out = f.remove('helper');
assert.equal(out.code, undefined);
assert.match(out.transaction, /^t-/);
assert.equal(out.retry, false);
assert.deepEqual(f.org.log.filter(([op]) => op === 'instances.set'), [['instances.set', 'helper']]);
assert.equal(f.allowed('grants.revoking').length, 0);
assert.deepEqual(f.inbox(), []);
// The request, the take and the delete's own records: none from step G.
const after = f.store
.all("SELECT body FROM events WHERE business='demo' ORDER BY seq")
.slice(before)
.map((e) => JSON.parse(e.body).action ?? JSON.parse(e.body).operation ?? null);
assert.ok(after.every((a) => !String(a).startsWith('grant')), after.join());
await f.broker.grantsIdle();
assert.equal(f.allowed('grants.revoked.done').length, 0);
});
test('row 281 T4: host B unreachable in a fire: the job refuses, A stays revoked, B and Gitea untouched; a repeat completes', async (t) => {
const f = await setup(t);
await threeGrants(f);
f.failing.add('web2');
const out = f.fire('coder');
await f.broker.grantsIdle();
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
assert.equal(more.length, 0);
assert.deepEqual(
{ job: refused.job, agent: refused.agent, grant: refused.grant, reason: refused.reason, verb: refused.verb },
{ job: out.job, agent: 'coder', grant: 'host:web2', reason: 'host-remove-failed', verb: 'instance.fire' },
);
const [message] = f.inbox();
for (const s of [out.job, 'web2', 'host-remove-failed', 'mosaic hosts revoke coder web2', 'Run the fire again']) assert.ok(message.includes(s), s);
// The file is unchanged and the instance still firing.
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), []);
assert.ok(f.doc.launch.instances.includes('coder'));
assert.equal(f.allowed('fire.done').length, 0);
assert.equal(f.allowed('fire.failed').length, 0);
await rejects(() => f.raiseHost('coder', 'web3'), 'instance-fired');
// A revoked; B's record, key and line there (line first: nothing local changed on B).
assert.equal(f.revoked('coder', (e) => e.host === 'web1').length, 1);
assert.equal(f.hostRow('coder', 'web1'), undefined);
assert.equal(f.revoked('coder', (e) => e.host === 'web2').length, 0);
assert.ok(f.hostRow('coder', 'web2'));
assert.ok(existsSync(f.keyFile('coder', 'web2')));
assert.equal(f.state('coder', 'host.web2'), 'valid');
// Gitea comes after B: untouched.
assert.equal(f.revoked('coder', (e) => e.method === 'gitea').length, 0);
assert.ok(f.giteaRow('coder'));
assert.equal(f.userTokens('demo-coder').length, 1);
assert.equal(f.state('coder', 'gitea'), 'valid');
// B back: the repeat revokes only B and Gitea.
f.failing.delete('web2');
const repeat = f.fire('coder');
assert.deepEqual(repeat.revoking, ['host:web2', 'gitea']);
await f.broker.grantsIdle();
const [done] = f.allowed('fire.done');
assert.equal(done.body.job, repeat.job);
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web2', 'gitea']);
allRevoked(f);
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
});
test('row 281 T4b (Q1): a line a manual hosts.revoke left on C is removed by the fire, with its close', async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
await f.approveHost('coder', 'web3');
f.failing.add('web3');
const manual = await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web3', reason: 'web3 is down' });
const tag3 = f.tag('coder', 'web3');
assert.equal(manual.hostLineLeft, tag3);
f.failing.delete('web3');
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['host:web1', 'line:web3']);
await f.broker.grantsIdle();
assert.equal(f.lines.web3.has(tag3), false);
const close = f.grantEvents().filter((e) => e.action === 'grant.revoke.closed' && e.host === 'web3' && e.fire === true);
assert.equal(close.length, 1);
assert.equal(close[0].hostLine, 'removed');
assert.equal(close[0].tag, tag3);
// hosts.revoke's answer has no revoke id; the manual's grant.revoked has it.
const [manualRevoked] = f.revoked('coder', (e) => e.host === 'web3' && e.fire !== true);
assert.equal(close[0].revoke, manualRevoked.revoke);
const [done] = f.allowed('fire.done');
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'line:web3']);
assert.deepEqual(done.body.grantsLeft, []);
});
test('row 281 T4b (Q1): with C unreachable, the fire completes and names the line in grantsLeft', async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
await f.approveHost('coder', 'web3');
f.failing.add('web3');
await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web3', reason: 'web3 is down' });
const tag3 = f.tag('coder', 'web3');
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['host:web1', 'line:web3']);
await f.broker.grantsIdle();
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
const [done] = f.allowed('fire.done');
assert.equal(done.body.job, out.job);
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1']);
assert.deepEqual(done.body.grantsLeft, [{ host: 'web3', tag: tag3 }]);
assert.ok(f.lines.web3.has(tag3));
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
});
test("row 281 T5: Gitea's admin token can't be checked: the job refuses with the Gitea grant intact", async (t) => {
const f = await setup(t);
await f.approveGitea('coder');
const off = f.gitea.rule((e) => e.method === 'GET' && e.path === '/api/v1/version', { status: 500, json: {} });
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['gitea']);
await f.broker.grantsIdle();
off();
const [refused] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
assert.deepEqual([refused.grant, refused.reason, refused.job], ['gitea', 'gitea-admin-unavailable', out.job]);
assert.ok(f.inbox()[0].includes('mosaic grants sweep gitea'));
assert.ok(f.giteaRow('coder'));
assert.ok(existsSync(f.tokenFile('coder')));
assert.equal(f.state('coder', 'gitea'), 'valid');
assert.equal(f.userTokens('demo-coder').length, 1);
assert.equal(f.gitea.users.get('demo-coder').prohibit_login, false);
assert.equal(f.revoked('coder').length, 0);
assert.equal(f.allowed('fire.done').length, 0);
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), []);
});
test("row 281 T5b (N1): a failed token DELETE refuses gitea-left; the repeat retries only this agent's entries", async (t) => {
const f = await setup(t);
await f.approveGitea('helper');
await f.approveGitea('coder');
const tokenDelete = (login) => (e) => e.method === 'DELETE' && e.path.startsWith(`/api/v1/users/${login}/tokens/`);
// Another agent's left entry, from a manual revoke whose DELETE failed.
const offHelper = f.gitea.rule(tokenDelete('demo-helper'), { status: 500, json: {} });
const manual = await f.call('human', 'grants.revoke', { service: 'gitea', agent: 'helper', reason: 'test' });
assert.equal(manual.serviceLeft.length, 1);
const helperDeletes = () => f.gitea.requests.filter(tokenDelete('demo-helper')).length;
const before = helperDeletes();
const offCoder = f.gitea.rule(tokenDelete('demo-coder'), { status: 500, json: {} });
const out = f.fire('coder');
await f.broker.grantsIdle();
const [refused] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
assert.equal(refused.grant, 'gitea');
// The close's serviceLeft (N10): the token id, and "tokens" for the fire's list that didn't clear.
assert.match(refused.reason, /^gitea-left:\d+,tokens$/);
assert.equal(refused.job, out.job);
assert.equal(f.userTokens('demo-coder').length, 1);
assert.equal(f.allowed('fire.done').length, 0);
// Gitea back for coder only. The helper's DELETE still fails, so a business-wide sweep would show.
offCoder();
const repeat = f.fire('coder');
assert.deepEqual(repeat.revoking, ['gitea']);
await f.broker.grantsIdle();
const [done] = f.allowed('fire.done');
assert.equal(done.body.job, repeat.job);
assert.deepEqual(f.userTokens('demo-coder'), []);
const swept = f.grantEvents().filter((e) => e.action === 'grant.swept' && e.job === repeat.job);
assert.equal(swept.length, 1);
assert.equal(swept[0].agent, 'coder');
assert.deepEqual(swept[0].left.map((e) => e.agent), ['helper']);
// Neither job touched the helper's token.
assert.equal(helperDeletes(), before);
assert.equal(f.userTokens('demo-helper').length, 1);
offHelper();
});
test('row 281 T6 (sage-2574): a host decision approved before the fire and run after it ends revoked, with nothing applied', async (t) => {
const f = await setup(t);
const id = f.fixtureHost('coder', 'web1');
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['host:web1']);
await f.broker.grantsIdle();
const [r] = f.revoked('coder', (e) => e.host === 'web1');
assert.equal(r.granted, false);
assert.equal(r.fire, true);
assert.equal(f.allowed('fire.done').length, 1);
assert.equal(await f.broker.runHostGrant('demo', id), 'revoked');
assert.equal(f.ended(id).code, 'revoked');
assert.equal(f.hostRow('coder', 'web1'), undefined);
assert.equal(existsSync(f.keyFile('coder', 'web1')), false);
assert.ok(!f.sent.some((x) => x.startsWith('add ')));
});
test('row 281 T6b (sage-2574): a Gitea decision approved before the fire and run after it ends revoked before any Gitea write', async (t) => {
const f = await setup(t);
const id = f.fixtureGitea('coder');
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['gitea']);
await f.broker.grantsIdle();
const [r] = f.revoked('coder', (e) => e.method === 'gitea');
assert.deepEqual([r.user, r.granted, r.fire, r.decisions], [null, false, true, [id]]);
assert.equal(f.allowed('fire.done').length, 1);
const calls = f.gitea.requests.length;
await f.broker.runGiteaGrant('demo', id);
await f.broker.grantsIdle();
assert.equal(f.ended(id).code, 'revoked');
assert.equal(f.giteaRow('coder'), undefined);
assert.equal(existsSync(f.tokenFile('coder')), false);
// Only step 0's C1 to C4 reach Gitea, all reads; step 1's recheck ends it before any write.
assert.deepEqual(f.gitea.requests.slice(calls).map((e) => `${e.method} ${e.path}`), [
'GET /api/v1/version',
'GET /api/v1/user',
'GET /api/v1/token',
'GET /api/v1/user/repos?limit=1&page=1',
]);
assert.ok(!f.gitea.requests.some((e) => e.method === 'POST' && e.path.endsWith('/tokens')));
});
test('row 281 T6c: a host executor in flight aborts and ends revoked; a Gitea executor in flight is awaited and leaves no token', async (t) => {
const f = await setup(t);
// Host: held at its add.
f.held.add('web1:add');
const host = await f.raiseHost('coder', 'web1');
f.call('human', 'decision.resolve', { id: host.id, choice: 'yes' });
await until(() => f.sent.some((x) => x.startsWith('add web1 ')));
// Gitea: held at the token create.
let go;
const gate = new Promise((r) => (go = r));
t.after(() => go());
let reached = false;
const off = f.gitea.rule(
(e) => e.method === 'POST' && e.path === '/api/v1/users/demo-coder/tokens',
async () => {
reached = true;
await gate;
return { pass: true };
},
);
const gd = await f.raiseGitea('coder');
f.call('human', 'decision.resolve', { id: gd.id, choice: 'yes' });
await until(() => reached);
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['host:web1', 'gitea']);
await until(() => f.ended(host.id));
assert.equal(f.ended(host.id).code, 'revoked');
// The job waits on the Gitea executor.
await sleep(50);
assert.equal(f.allowed('fire.done').length, 0);
go();
await f.broker.grantsIdle();
off();
f.held.delete('web1:add');
assert.equal(f.hostRow('coder', 'web1'), undefined);
assert.equal(existsSync(f.keyFile('coder', 'web1')), false);
assert.equal(f.lines.web1.has(f.tag('coder', 'web1')), false);
assert.ok(f.ended(gd.id));
assert.deepEqual(f.userTokens('demo-coder'), []);
assert.equal(f.giteaRow('coder'), undefined);
assert.equal(f.allowed('fire.done').length, 1);
f.noLeak();
});
test('row 281 T6d (B1): undecided host and Gitea decisions are in the set, and approved after the fire both end revoked', async (t) => {
const f = await setup(t);
const host = await f.raiseHost('coder', 'web1');
const gd = await f.raiseGitea('coder');
const out = f.fire('coder');
assert.deepEqual(out.revoking, ['host:web1', 'gitea']);
await f.broker.grantsIdle();
assert.equal(f.allowed('fire.done').length, 1);
for (const d of [host, gd]) {
f.call('human', 'decision.resolve', { id: d.id, choice: 'yes' });
await until(() => f.ended(d.id));
}
await f.broker.grantsIdle();
assert.equal(f.ended(host.id).code, 'revoked');
assert.equal(f.ended(gd.id).code, 'revoked');
assert.ok(!f.sent.some((x) => x.startsWith('add ')));
assert.ok(!f.gitea.requests.some((e) => e.method === 'POST' && e.path.endsWith('/tokens')));
assert.equal(f.hostRow('coder', 'web1'), undefined);
assert.equal(f.giteaRow('coder'), undefined);
});
test('row 281 T6e (B1): a grant raise refuses instance-fired while firing and after the fire, goes through after a rehire, and refuses grant-revoke-running during a delete job', async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
// Firing with no job: step B refused, so fire.stopped stands with no fire.done.
f.org.fireError = Object.assign(new Error('refused'), { code: 'credential-revoke-failed' });
f.fire('coder');
await f.broker.grantsIdle();
assert.equal(f.allowed('fire.failed').length, 1);
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
await rejects(() => f.raiseGitea('coder'), 'instance-fired');
// After fire.done: delisted.
f.org.fireError = null;
assert.match(f.fire('coder').transaction, /^t-/);
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
await rejects(() => f.raiseGitea('coder'), 'instance-fired');
// A rehire lists the name again.
f.doc.launch.instances.push('coder');
f.broker.reloadBusiness('demo', f.doc);
assert.match((await f.raiseHost('coder', 'web2')).id, UUID);
// While a delete's job holds ops.
await f.approveHost('ops', 'web3');
f.held.add('web3:remove');
assert.equal(f.remove('ops').code, 'grants-revoking');
await rejects(() => f.raiseHost('ops', 'web1'), 'grant-revoke-running');
f.held.delete('web3:remove');
await f.broker.grantsIdle();
});
test('row 281 T7: an instance with no grant fires as today, synchronously, with the same events', async (t) => {
const f = await setup(t);
const out = f.fire('helper');
assert.deepEqual(Object.keys(out).sort(), ['cancelled', 'request', 'revision', 'runs', 'taken', 'transaction']);
assert.match(out.transaction, /^t-/);
// Recorded before the verb answered: no job.
assert.equal(f.allowed('fire.stopped').length, 1);
assert.equal(f.allowed('fire.done').length, 1);
assert.equal(f.allowed('fire.done')[0].body.job, undefined);
assert.equal(f.allowed('fire.revoking').length, 0);
assert.equal(f.allowed('role.fire').length, 1);
assert.deepEqual(f.org.log.map(([op]) => op), ['instance.fireCheck', 'instance.fire']);
await f.broker.grantsIdle();
assert.deepEqual(f.inbox(), []);
assert.equal(f.grantEvents().filter((e) => e.action?.startsWith('grant.revoke')).length, 0);
});
test('row 281 T9 (B3): one step G per agent; the manual verbs refuse while it runs, and sweep names it running', async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
await f.approveGitea('coder');
f.held.add('web1:remove');
f.fire('coder');
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
assert.equal(code(() => f.fire('coder')), 'fire-in-progress');
assert.equal(f.remove('coder').code, 'grants-revoking');
assert.equal(f.allowed('grants.revoking').length, 0);
assert.equal(f.allowed('fire.revoking').length, 1);
await rejects(() => f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web1', reason: 'r' }), 'grant-revoke-running');
await rejects(() => f.call('tester', 'hosts.revoke', { agent: 'coder', host: 'web1', reason: 'r' }), 'grant-revoke-running');
await rejects(() => f.call('human', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'grant-revoke-running');
const sweep = await f.call('human', 'grants.sweep', { service: 'gitea' });
assert.deepEqual(sweep.running, ['coder']);
f.held.delete('web1:remove');
await f.broker.grantsIdle();
assert.equal(f.revoked('coder', (e) => e.host === 'web1').length, 1);
assert.equal(f.revoked('coder', (e) => e.method === 'gitea').length, 1);
assert.equal(f.allowed('fire.done').length, 1);
allRevoked(f, ['web1']);
});
test('row 281 T10 (B2): the fire and the delete answer while their jobs are held, and the broker answers other verbs', async (t) => {
const f = await setup(t);
await f.approveGitea('coder');
await f.approveHost('ops', 'web2');
let go;
const gate = new Promise((r) => (go = r));
t.after(() => go());
let reached = false;
const off = f.gitea.rule(
(e) => e.method === 'DELETE' && e.path.startsWith('/api/v1/users/demo-coder/tokens/'),
async () => {
reached = true;
await gate;
return { pass: true };
},
);
f.held.add('web2:remove');
// The verbs answer before the event loop turns, so before any part of a job has run.
let turned = false;
setImmediate(() => (turned = true));
const fired = f.fire('coder');
const deleted = f.remove('ops');
assert.equal(turned, false);
assert.equal(fired.done, false);
assert.equal(deleted.code, 'grants-revoking');
await until(() => reached && f.sent.includes(`remove web2 ${f.tag('ops', 'web2')}`));
// Held: another session's verb answers.
assert.ok(f.call('pm', 'message.send', { to: 'chief', body: 'still here' }));
assert.equal(f.allowed('fire.done').length, 0);
assert.equal(f.allowed('grants.revoked.done').length, 0);
go();
f.held.delete('web2:remove');
await f.broker.grantsIdle();
off();
assert.equal(f.allowed('fire.done').length, 1);
assert.equal(f.allowed('grants.revoked.done').length, 1);
assert.deepEqual(f.userTokens('demo-coder'), []);
assert.equal(f.hostRow('ops', 'web2'), undefined);
});
test("row 281 T11 (C1): a fire's job completes after the chief's run ends and processing halts", async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
f.held.add('web1:remove');
const out = f.fire('coder');
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
f.broker.endLaunch({ business: 'demo', run: 'chief-run', reason: 'test' });
f.sw.state = { version: 1, halted: true, generation: GENERATION };
f.held.delete('web1:remove');
await f.broker.grantsIdle();
const [done] = f.allowed('fire.done');
assert.equal(done.body.job, out.job);
assert.match(done.body.transaction, /^t-/);
const [used, ...more] = f.allowed('role.fire');
assert.equal(more.length, 0);
assert.equal(used.actor_role, 'chief');
assert.equal(used.body.transaction, done.body.transaction);
assert.ok(!f.doc.launch.instances.includes('coder'));
assert.equal(f.allowed('fire.failed').length, 0);
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
});
test("row 281 T12 (C1, N7): a step B refusal in the job is fire.failed, and the repeat runs B and C synchronously", async (t) => {
const f = await setup(t);
await f.approveHost('coder', 'web1');
// N7: the writer's own credential-revoke-failed, which step G's key revokes can't clear.
f.org.fireError = Object.assign(new Error('refused'), { code: 'credential-revoke-failed' });
const out = f.fire('coder');
await f.broker.grantsIdle();
const [failed, ...more] = f.allowed('fire.failed');
assert.equal(more.length, 0);
assert.deepEqual([failed.body.job, failed.body.step, failed.body.code], [out.job, 'B', 'credential-revoke-failed']);
assert.equal(failed.actor_role, 'chief');
const [message] = f.inbox();
for (const s of [out.job, 'host:web1', 'step B', 'credential-revoke-failed']) assert.ok(message.includes(s), s);
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
assert.equal(f.allowed('fire.done').length, 0);
assert.ok(f.doc.launch.instances.includes('coder'));
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
// The grant was revoked before step B.
assert.equal(f.hostRow('coder', 'web1'), undefined);
f.org.fireError = null;
const repeat = f.fire('coder');
assert.equal(repeat.job, undefined);
assert.match(repeat.transaction, /^t-/);
assert.equal(f.allowed('fire.done').length, 1);
assert.ok(!f.doc.launch.instances.includes('coder'));
});
test('row 281 T13 (N4): a restart records each held job interrupted once, with one message each; an ended job gets none', async (t) => {
const f = await setup(t);
await f.approveHost('helper', 'web3');
await f.approveHost('coder', 'web1');
await f.approveHost('ops', 'web2');
// A delete job that ended before the restart.
assert.equal(f.remove('helper').code, 'grants-revoking');
await f.broker.grantsIdle();
assert.equal(f.allowed('grants.revoked.done').length, 1);
f.held.add('web1:remove');
f.held.add('web2:remove');
const fired = f.fire('coder');
assert.equal(f.remove('ops').code, 'grants-revoking');
const deleteJob = f.allowed('grants.revoking').at(-1).body.job;
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`) && f.sent.includes(`remove web2 ${f.tag('ops', 'web2')}`));
const restartNotes = () => f.inbox().filter((m) => m.includes('interrupted by a bus restart'));
const second = f.make();
second.reconcileGrants();
const interrupted = f.allowed('grants.job.interrupted');
assert.deepEqual(
interrupted.map((e) => [e.body.job, e.body.agent, e.body.verb]).sort(),
[[fired.job, 'coder', 'instance.fire'], [deleteJob, 'ops', 'lifecycle.delete']].sort(),
);
assert.ok(interrupted.every((e) => e.actor_role === null));
const notes = restartNotes();
assert.equal(notes.length, 2);
assert.ok(notes.some((m) => m.includes(fired.job) && m.includes('Run the fire again')));
assert.ok(notes.some((m) => m.includes(deleteJob) && m.includes('Raise the delete again')));
// A third broker on the same root sends nothing more.
f.make().reconcileGrants();
assert.equal(f.allowed('grants.job.interrupted').length, 2);
assert.equal(restartNotes().length, 2);
f.held.clear();
await f.broker.grantsIdle();
});
test("row 281 T14 (N1, sage-2653): a host removed under an undecided decision stops the fire on host-unknown; the human's no closes it, and the repeat fire has no job", async (t) => {
const f = await setup(t);
const d = await f.raiseHost('coder', 'web1');
// N1: hosts.remove doesn't check open decisions, so the decision outlives its host.
assert.deepEqual(await f.call('human', 'hosts.remove', { host: 'web1' }), { host: 'web1', removed: true });
const out = f.fire('coder');
assert.match(out.job, UUID);
assert.deepEqual(out.revoking, ['host:web1']);
await f.broker.grantsIdle();
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
assert.equal(more.length, 0);
assert.deepEqual([refused.job, refused.grant, refused.reason, refused.verb], [out.job, 'host:web1', 'host-unknown', 'instance.fire']);
const [message, ...others] = f.inbox();
assert.equal(others.length, 0);
for (const s of [out.job, 'web1', 'host-unknown']) assert.ok(message.includes(s), s);
assert.equal(f.allowed('fire.done').length, 0);
assert.ok(f.doc.launch.instances.includes('coder'));
// The human's no: no executor runs, so the closed check alone takes it out of the set (M22).
f.call('human', 'decision.resolve', { id: d.id, choice: 'no' });
await f.broker.grantsIdle();
assert.equal(f.ended(d.id), undefined);
const repeat = f.fire('coder');
assert.equal(repeat.job, undefined);
assert.match(repeat.transaction, /^t-/);
assert.equal(f.allowed('fire.done').length, 1);
assert.equal(f.allowed('fire.done')[0].body.job, undefined);
assert.ok(!f.doc.launch.instances.includes('coder'));
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 1);
});
+21
View File
@@ -1216,6 +1216,27 @@ test('X-G14: a revoke Gitea cannot reach stays open, and a restart closes it', a
await rejects(f.call('pm', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'revoke-refused');
});
// Row 281 (#1786, d6 section 8, T8): grants.revoke shares its Gitea steps with a fire's step G now. This
// pins the tester's dev-only refusal, the verb's records and their order, and that it starts no job.
test('row 281 T8: a tester grants.revoke refuses tester-dev-only once bus.tester is off; the human revoke keeps its records and starts no job', async (t) => {
const f = await setup(t);
await f.approve();
f.broker.reloadBusiness('demo', { ...f.business, vars: { 'bus.tester': 'yes' } });
const before = f.grantEvents().length;
await rejects(f.call('tester', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'tester-dev-only');
assert.equal(f.grantEvents().length, before);
assert.ok(f.record());
await f.call('human', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' });
const after = f.grantEvents().slice(before)
.filter((e) => e.action !== 'grant.call' && e.action !== 'grant.result')
.map((e) => [e.action, e.fire ?? false]);
assert.deepEqual(after, [['grant.revoked', false], ['grant.revoke.closed', false]]);
assert.deepEqual(f.userTokens(), []);
const jobs = f.store.all("SELECT body FROM events WHERE business='demo' AND kind='action.allowed'").map((e) => JSON.parse(e.body).action)
.filter((a) => /^(fire|grants)\./.test(a));
assert.deepEqual(jobs, []);
});
test('X-G15: look-alike commit steps outside the grant writer are ignored at start, by revoke and by status', async (t) => {
const f = await setup(t);
const first = await f.approve();
@@ -780,6 +780,24 @@ test('G26: the human, the assistant and the tester revoke; the chief and a pm re
assert.deepEqual((await f.call('tester', 'hosts.sweep', { host: 'web1' })).removed, []);
});
// Row 281 (#1786, d6 section 8, T8): hosts.revoke shares its steps with a fire's step G now. G12 pins its
// records and their order; this pins the tester's dev-only refusal and that a verb revoke starts no job.
test('row 281 T8: a tester hosts.revoke refuses tester-dev-only once bus.tester is off, and a human revoke writes no job records', async (t) => {
const f = await setup(t);
await f.approve();
f.caps.tester = f.broker.bindTester({ business: 'demo' });
f.broker.reloadBusiness('demo', { ...f.business, vars: { 'bus.tester': 'yes' } });
const args = { agent: 'ops', host: 'web1', reason: 'r' };
await refusesWith(f.call('tester', 'hosts.revoke', args), 'tester-dev-only');
assert.equal(revokeBodies(f).length, 0);
assert.ok(f.record());
assert.equal((await f.call('human', 'hosts.revoke', args)).granted, true);
assert.deepEqual(revokeBodies(f).map((e) => [e.action, e.fire, e.delete]), [['grant.revoked', undefined, undefined], ['grant.revoke.closed', undefined, undefined]]);
const jobs = f.store.all("SELECT body FROM events WHERE business='demo' AND kind='action.allowed'").map((e) => JSON.parse(e.body).action)
.filter((a) => /^(fire|grants)\./.test(a));
assert.deepEqual(jobs, []);
});
test('G27: a restart after step 1 sweeps the key file and runs step 3 once the bus is up; status shows the revoke until then', async (t) => {
const f = await setup(t);
await f.approve();
+38 -1
View File
@@ -9,6 +9,8 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { busBusiness } from '../src/business.mjs';
import { startBroker } from '../src/runtime.mjs';
import { Store } from '../src/store.mjs';
import { hostGrantRecord, hostRecord } from './host-fixtures.mjs';
import { loadBusiness } from '../../business/src/business.mjs';
import { resolveInstance } from '../../business/src/resolve.mjs';
import { businessDoc, FAR_DATE, rolesCopy, systemFor, writeJson } from '../../business/tests/helpers.mjs';
@@ -19,7 +21,7 @@ const store = (agent, service) => ({ store: true, [service === 'gitea' ? 'rotate
// coder: gitea in the store, vikunja a file reference. rev-2 and rev-3: both in the store, with a
// tracker bot. `drop` names instances whose credentials the boot copy leaves out, as a restart that
// dropped the entry would: the running bus then has no entry for a file on disk (sage-1592).
async function setup(t, { drop = [] } = {}) {
async function setup(t, { drop = [], seed = null } = {}) {
const root = mkdtempSync(join(tmpdir(), 'bus-lifecycle-revoke-'));
const rolesDir = rolesCopy(root);
const doc = { ...businessDoc(root, 'demo'), plugins: { harness: ['pi'] } };
@@ -31,6 +33,12 @@ async function setup(t, { drop = [] } = {}) {
const file = writeJson(join(root, 'businesses/demo.json'), doc);
mkdirSync(join(root, 'data'), { mode: 0o700 });
mkdirSync(join(root, 'secrets', 'agents'), { recursive: true, mode: 0o700 });
// Row 281 T1: records written before the start, as an earlier bus left them.
if (seed) {
const seeded = new Store(join(root, 'data'));
seed(seeded);
seeded.close();
}
const b = loadBusiness('demo', { dir: root, rolesDir });
const resolved = {};
for (const i of Object.keys(b.roles)) resolved[i] = resolveInstance({ system: systemFor(root), business: b, instance: i });
@@ -181,3 +189,32 @@ test('T21 delete: a write between the raise and the take makes the delete refuse
assert.deepEqual(readdirSync(f.agentDir('rev-2')), ['vikunja']);
assert.deepEqual(f.status('rev-2'), { gitea: 'not-issued', vikunja: 'valid' });
});
// Row 281 (#1786, d6 section 1) T1: pins what section 1 read from the code, and what still holds after the
// change. Step B's revoke refuses while a grant's file is in the agent's store, which is why step G runs
// first. A fire with no grant record goes straight to step B, so a stray host.<id> file planted after the
// start (the start sweep would unlink it) still refuses there.
test('row 281 T1 fire: a host.<id> or granted gitea file in the store refuses step B credential-revoke-failed, file unchanged', async (t) => {
const f = await setup(t);
f.plant('coder', 'host.web1');
const before = f.bytes();
assert.equal(f.code(() => f.fire('coder')), 'credential-revoke-failed');
assert.equal(f.bytes(), before);
assert.deepEqual(readdirSync(f.agentDir('coder')), ['host.web1']);
rmSync(join(f.agentDir('coder'), 'host.web1'));
assert.equal(f.fire('coder').delisted, true);
});
// T1, the start: a host_grants record whose key file is gone (what a delete left before this row). The
// start loads the reference with no token, so the bus comes up and the entry reads not-issued.
test('row 281 T1 start: a host grant record whose key file is gone starts and reads not-issued', async (t) => {
const entry = { address: '192.0.2.10', port: 22, user: 'deploy', hostKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKeyForTests'] };
const f = await setup(t, {
seed: (store) => {
hostRecord(store, { host: 'web1', entry });
hostGrantRecord(store, { agent: 'coder', host: 'web1', commands: [['uptime']] });
},
});
assert.equal(existsSync(join(f.agentDir('coder'), 'host.web1')), false);
assert.equal(f.status('coder')['host.web1'], 'not-issued');
});
+28 -1
View File
@@ -13,11 +13,13 @@ import { spawn, spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { DatabaseSync } from "node:sqlite";
import { existsSync, lstatSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { launchDoc } from "../../business/tests/helpers.mjs";
import { BusError } from "../../bus/src/broker.mjs";
import { startBroker } from "../../bus/src/runtime.mjs";
import { Store } from "../../bus/src/store.mjs";
import { hostGrantRecord, hostRecord } from "../../bus/tests/host-fixtures.mjs";
import { gitEnv } from "../../seat/src/checkout.mjs";
import { sandboxProblem } from "../../seat/src/sandbox.mjs";
import { processIdentity, registerRuntime, sweepRuntimes, unregisterRuntime, updateRuntime } from "../../processing-control/src/runtimes.mjs";
@@ -428,6 +430,31 @@ test("row 194 V5: a delete with no workspace commits and closes with the transac
assert.equal(existsSync(join(workspaceDir(p.f.dataRoot, "acme", ".retired"))), false);
});
// Row 281 (#1786, d6 section 2, T10 B2): a delete of an instance with a grant answers at once. The pass
// closes the first request grants-revoking while step G runs as a job, and the host channel stays up:
// the next op answers. The grant is a seeded record on a TEST-NET host, so the job's remove can only
// fail or wait; either way the request has already closed.
test("row 281 T10: the delete pass closes a granted instance's first request grants-revoking and the host channel stays up", { skip, timeout: 90000 }, async (t) => {
const p = prepare(t);
// A host grant gives the agent a host.web1 store reference, so Credentials checks the store root at start.
mkdirSync(join(dirname(p.f.config), "secrets", "agents"), { recursive: true, mode: 0o700 });
const seeded = new Store(p.f.dataRoot);
const entry = { address: "192.0.2.10", port: 22, user: "deploy", hostKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKeyForTests"] };
hostRecord(seeded, { business: "acme", host: "web1", entry });
hostGrantRecord(seeded, { business: "acme", agent: "sec-1", host: "web1", commands: [["uptime"]] });
seeded.close();
const s = await start(t, p);
const del = s.call("instances.set", { revision: s.call("instances").revision, instance: "sec-1", op: "delete" }).request;
const r = await s.carry(del);
assert.equal(r.done.refusal, "grants-revoking");
assert.equal(r.done.transaction ?? null, null);
// The channel answers the next op and the next pass.
assert.ok(s.call("instances").revision);
await s.launcher.tick();
assert.ok(Object.hasOwn(JSON.parse(readFileSync(join(p.root, "config", "businesses", "acme.json"), "utf8")).roles, "sec-1"));
await s.runtime.broker.grantsIdle();
});
// Every regular file under root, by path, as a sha256. Sockets and other special files are left out.
function snapshot(root) {
const out = {};