Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e479521d3a | ||
|
|
7979fb41eb | ||
|
|
54ebc5d71c | ||
|
|
5e05e21794 | ||
|
|
6b6eb533e8 | ||
|
|
f9fcf5d7b3 | ||
|
|
c055dddf44 | ||
|
|
6d62ef04e0 | ||
|
|
4982fa0adf | ||
|
|
7c2c876319 | ||
|
|
7d5f16ef95 | ||
|
|
a7b4f6785c | ||
|
|
938378455d | ||
|
|
5f4cd2faa6 | ||
|
|
6faa0f1479 | ||
|
|
9867a6f322 | ||
|
|
15ab01c0c3 |
@@ -0,0 +1,931 @@
|
||||
# Row 281 build notes: a fire or a delete revokes the instance's grants (#1786)
|
||||
|
||||
Builder Megavolt, reviewer Dewey, gate owner Sage. Design draft d6 for Dewey's approval. No code until it is
|
||||
approved. The results sections are filled in after the runs on Sage's gate.
|
||||
|
||||
History: d1 dfebd16cc. d2 c746ce378 added Sage's Q1/Q2 rulings (sage-2573). d3 36c2d98b8 added the
|
||||
approved-before-fire tests (sage-2574). d4 answers Dewey's CHANGES on d3 (dewey-0823):
|
||||
- B1: the set takes every grant decision with no outcome, and a raise refuses a fired or firing agent.
|
||||
- B2: step G runs detached from the verb, so neither caller waits on the network.
|
||||
- B3: one step G per agent, with named refusals for everything that could race it.
|
||||
- N1-N3 are folded in.
|
||||
|
||||
d5 adds sage-2575: two requests per granted delete, with its five conditions. It also adds a `job` id to every
|
||||
answer, event and message, and tests T3, T3b and T3c with mutants M16-M18.
|
||||
|
||||
d6 answers Dewey's CHANGES on d5 (dewey-0825):
|
||||
- C1: the job's step C records authority without the chief's claim (`#authorityFor`, as a landing's record
|
||||
does), processing-halted doesn't hold the job, and a step B or C failure records `fire.failed {job, step,
|
||||
code}` with its own message. Tests T11 and T12, mutants M19 and M20.
|
||||
- N4: boot closes a job a restart interrupted, with one message. Test T13, mutant M21.
|
||||
- N5: section 7 names the refusals the manual verbs gain.
|
||||
- N6: the set counts Q1 leftover lines, in fire step 3 and in the delete text.
|
||||
|
||||
## Sources
|
||||
|
||||
- Brief: `docs/plans/2026-10-22_fire-revokes-grants.md`, "Brief: no grant outlives its instance" (sage-2568,
|
||||
mos-1271, Vikunja 524).
|
||||
- Code read at 47a35bf43: `packages/bus/src/broker.mjs`, `runtime.mjs`, `credentials.mjs`, `host-grant.mjs`,
|
||||
`gitea-grant.mjs`, `packages/business/src/instance-settings.mjs`, `tools/host/mosaic-keys-write`.
|
||||
- Row 256's delete (honker/row256-r2 at 1e761962a, not landed) changes only the delete's checks. Its write still
|
||||
goes through `lifecycleDelete` and the same `revoke()`, so this design covers it unchanged.
|
||||
- Row 235 r6 section 6 (host revoke) and #436b r5 section 6 (Gitea revoke and the fire steps).
|
||||
|
||||
## 1. Why: what happens today (read from the code, to be pinned by a test before the change)
|
||||
|
||||
In short: today a delete of a granted instance unlinks the key files but leaves both grant records, the host
|
||||
lines and the Gitea token live, and a fire refuses `credential-revoke-failed` (sage-2573 asked for this here
|
||||
and on #1786). The details:
|
||||
|
||||
- **Fire.** `revoke()` (runtime.mjs:151) revokes only the roles entry's `credentials` services, then lists the
|
||||
agent's store directory and refuses `credential-revoke-failed` if any file is left. A host grant's key is
|
||||
`<store>/<business>/<agent>/host.<id>` (host-grant.mjs:102), and a Gitea grant's token is `.../gitea`
|
||||
(gitea-grant.mjs:379). Neither is a roles-entry service, so a fire of an instance with a grant refuses at
|
||||
step B with "host.<id> still in the store". It doesn't leak, but it can't finish until a human runs
|
||||
`mosaic hosts revoke` or `mosaic grants revoke` by hand. If the business file also names `gitea` as a
|
||||
roles-entry credential, `credentials.revoke(instance, 'gitea')` unlinks the granted token file with no
|
||||
`gitea_grants` change and no Gitea call.
|
||||
- **Delete.** `credentials.remove(instance)` (credentials.mjs:400) revokes every registered reference,
|
||||
`host.<id>` and `gitea` included, and unlinks the directory. The `host_grants` and `gitea_grants` records
|
||||
stay, the host line stays on each host, and the Gitea token, user and login stay live. This is the leak.
|
||||
The next start then finds a grant record whose key file is gone. What the start does with that isn't read
|
||||
yet. The T1 test below pins it.
|
||||
- **An open grant decision.** A fire delists the instance but keeps its roles entry. An approved host or Gitea
|
||||
grant decision for that agent that hasn't run yet still passes `#hostDefinition` and `#giteaRole`, so it can
|
||||
still apply after the fire. A delete removes the roles entry, so its executor refuses `unknown-role` or
|
||||
`gitea-not-declared`.
|
||||
- **A raise after the fire** (dewey-0823 B1). `#raiseGrant` (broker.mjs:1529) checks only that the caller is
|
||||
the chief, and the grant bindings check only the roles entry. So the chief can raise a new grant for a fired
|
||||
agent, and if the human approves it, it applies.
|
||||
|
||||
## 2. Where the grant revokes sit (D1, rewritten in d4 for B2)
|
||||
|
||||
d3 awaited step G inside the verb. Dewey's B2 showed that neither caller can wait that long:
|
||||
|
||||
- **Delete.** The launcher's `host.op` waits `REQUEST_TIMEOUT_MS = 10000` (cli/src/host.mjs:31). On a timeout,
|
||||
`breakChannel` stops the whole bus host. One SSH remove can take 30 s (host-grant.mjs:231, :311), and each
|
||||
Gitea call 15 s (`CALL_MS`, gitea-grant.mjs:38).
|
||||
- **Fire.** server.mjs answers a verb that isn't on its async list synchronously (:154), and the async list
|
||||
gives 120 s (:150). The chief's runner client times out at 30 s (harness/src/runner.mjs:449). Two hosts
|
||||
plus the Gitea calls can pass both limits.
|
||||
|
||||
So step G never runs inside a verb. The fire and the delete both use the same answer-at-once job, for the same
|
||||
timeout reason (sage-2575 condition 5). Both verbs stay synchronous and answer at once. Step G runs as tracked
|
||||
broker work after the answer, as row 236's grant executors do (`#track`, `setImmediate`). No caller waits on
|
||||
the network, so `server.mjs`, `process.mjs`, `cli/src/host.mjs` and the launcher don't change.
|
||||
|
||||
Each job gets an id, `job` (a UUID), when it starts. The verb's answer, the job's start event, its end event
|
||||
and every inbox message about it carry that id. The human and the chief can then match a later outcome to
|
||||
the request that started it.
|
||||
|
||||
### The fire
|
||||
|
||||
1. `write('instance.fireCheck')`, as today.
|
||||
2. Step A, as today, with one addition. In the same transaction, after `fire.stopped`, the grant set is read
|
||||
(section 3). Reading it there means a grant raise either committed before step A, and is in the set, or
|
||||
comes after `fire.stopped` and refuses (section 3d).
|
||||
3. If the set is empty, steps B and C run synchronously, exactly as today. A fire with no grant answers
|
||||
today's result with today's events (T7). The set counts everything step G would touch (d6, N6): grant
|
||||
rows, open decisions, the agent's Gitea leftovers (section 5) and its Q1 leftover host lines (section 5).
|
||||
Any of them makes the set non-empty and goes to the job.
|
||||
4. Otherwise the verb takes the agent's step G lock (section 3e), records `fire.revoking {job, grants}` in
|
||||
step A's transaction, starts the job, and answers at once:
|
||||
`{...stopped, job, revoking: ['host:<id>' | 'gitea', ...], done: false}`. There's no `transaction`, since
|
||||
step B hasn't run.
|
||||
5. The job runs step G, then step B (`write('instance.fire')`) and step C (`#recordAuthority`, below, then
|
||||
`fire.done {job, grants}`). It releases the lock when it ends, whichever way it ends, `fire.failed`
|
||||
included.
|
||||
|
||||
What the chief sees meanwhile: the instance is "firing". That's an existing state (`#firing`, broker.mjs:645):
|
||||
`fire.stopped` with no later `fire.done`. Starts already refuse `instance-firing`.
|
||||
- A repeat fire while the job runs refuses `fire-in-progress`, a new code.
|
||||
- A repeat after the job finishes takes the delisted path (webby-0094 N1) and answers done.
|
||||
- A repeat after a refused step G runs step G again (section 5).
|
||||
|
||||
The human sees the outcome either way: `fire.done` names the job and the grants, and a refusal sends the
|
||||
inbox message in section 4.
|
||||
|
||||
**Steps B and C in the job (d6, C1).** d4 left "step C without the chief's run" as a build-time check, and
|
||||
Dewey checked it: it fails.
|
||||
- `#consumeAuthority` calls `#checkAuthority` (broker.mjs:2852). That runs `#processingWork()` and
|
||||
`#agent(s)`, and `#agent` refuses `not-holder` once the chief's run no longer holds its claim (:749).
|
||||
- After the verb answers, the job can take a minute or more. If the chief's run ends meanwhile, step C would
|
||||
throw after step B had delisted. The instance would be left delisted with no `fire.done`.
|
||||
|
||||
So the verb does the full check, and the job only records:
|
||||
- The verb runs `#checkAuthority(s, action, authority)`, as today, before step A. The chief holds its claim
|
||||
then.
|
||||
- The job's step C runs `#authorityFor(s, action, authority)`, the check without the holder test. It then
|
||||
runs the `#spent` check and the `action.allowed` event, as `#consumeAuthority` does. That's the shape a
|
||||
landing's record already uses (`recordPublish`, broker.mjs:3174, row 137): the run that asked held its
|
||||
claim when it asked, and may have ended since. It's one small private method, `#recordAuthority`, shared
|
||||
with nothing else.
|
||||
- The job keeps a copy of the session fields `#authorityFor` and `#event` read (business, role, thread,
|
||||
run). It doesn't keep the live session object.
|
||||
- `processing-halted` doesn't hold the job. A halt stops agent work. Step G only removes access, and steps
|
||||
B and C finish a decision the chief made while processing ran. Holding them would leave a delisted or
|
||||
half-revoked instance "firing" for the length of the halt. The verb itself still refuses during a halt,
|
||||
through `#checkAuthority`, as today.
|
||||
|
||||
**A step B or C failure in the job** gets its own record, not `grant.revoke.refused`, since every grant in the
|
||||
set was revoked by then:
|
||||
- It records `fire.failed {job, step: 'B' | 'C', code}`. `code` is the thrown code, for example the writer's
|
||||
`credential-revoke-failed`, or `decision-consumed`.
|
||||
- It sends an inbox message:
|
||||
|
||||
> Fire of <agent>: job <job> revoked its grants (<list>) but stopped at step <B|C> (<code>). The instance
|
||||
> is still firing. Run the fire again once <code> is cleared.
|
||||
|
||||
- The instance stays firing. A repeat fire finds an empty set and takes today's path for that state, with the
|
||||
chief's own claim: B and C after a step B failure (T12), the delisted path (webby-0094 N1) after a step C
|
||||
failure.
|
||||
|
||||
`grant.revoke.refused` with reason `internal` stays only for an unexpected throw inside step G.
|
||||
|
||||
### The delete (sage-2575)
|
||||
|
||||
`lifecycleDelete` (broker.mjs:3293) stays synchronous. After its running and starting checks, it reads the
|
||||
grant set.
|
||||
|
||||
- **No grant.** An empty set: `instances.set delete`, one request, exactly as today (condition 4, T3c).
|
||||
- **The first request.** For a non-empty set, it takes the lock and records `grants.revoking {job, agent,
|
||||
verb: 'lifecycle.delete', grants}`. It starts the job and refuses `grants-revoking` at once. The launcher
|
||||
closes the request with that code, well inside its 10 s.
|
||||
|
||||
The launcher can carry only the code, so the broker also sends the human an inbox message (condition 1):
|
||||
|
||||
> Delete of <agent> is waiting on its grants. Revoking started (job <job>) for: host:<id>, ..., gitea.
|
||||
> Raise the delete again when the inbox says the revoke is done.
|
||||
|
||||
- **A delete raised while the job runs.** It refuses `grants-revoking` again and starts no second job
|
||||
(condition 2, the 3e lock). No second message goes out either: the first one still applies.
|
||||
- **The job ends.** The human gets one inbox message (condition 3).
|
||||
- If it completed, the message names each grant revoked, with any `grantsLeft` (section 5), then "Raise the
|
||||
delete again." It records `grants.revoked.done {job, agent, verb, grants, grantsLeft}`.
|
||||
- If it refused, the message names the grant, the reason and how to clear it by hand, as section 4
|
||||
describes. It records `grant.revoke.refused {job, ...}`.
|
||||
- **The delete raised after a completed job** finds an empty set and goes as today.
|
||||
- **The delete raised after a refused job** doesn't complete over a live grant. Its set is non-empty, since
|
||||
the refused grant's record is still there. It refuses `grant-revoke-failed`, naming the earlier job's
|
||||
reason in a message:
|
||||
|
||||
> Delete of <agent> refused: job <job> couldn't revoke <grant> (<reason>). Retrying now as job <new>. Raise
|
||||
> the delete again when the inbox says it's done, or clear <grant> by hand first.
|
||||
|
||||
It starts a new job for what's left. The broker knows the last job refused because the agent's latest job
|
||||
end event is a `grant.revoke.refused` with verb `lifecycle.delete`, and no `grants.revoked.done` came after
|
||||
it. A delete after the human cleared the grant by hand doesn't find an empty set if a manual revoke left a
|
||||
line (Q1). The leftover line makes the set non-empty, so that delete starts a job for the line removal
|
||||
and refuses `grants-revoking` (d6, N6, and T3b's second case). If nothing is left, the set is empty and the
|
||||
delete goes on.
|
||||
|
||||
The tradeoff (ruled by sage-2575): deleting a granted instance now always takes two requests. That keeps the
|
||||
cli and the host channel out of row 281, and a delete of an instance with a grant is rare. Holding the
|
||||
launcher's request open past its 10 s would need a host channel change and a strand path, and polling would
|
||||
widen the row into the launcher for a one-time wait.
|
||||
|
||||
While a delete's step G runs, the instance is still listed and not firing, so it could be started. That's
|
||||
harmless: its grants are being revoked, and the repeat delete refuses `instance-running` as today. No new
|
||||
start check.
|
||||
|
||||
### Why not in `beforeWrite` or `revoke()`
|
||||
|
||||
Both are synchronous and run inside the organization writer, after its checks. A host `remove` is SSH and a
|
||||
Gitea revoke is HTTP, so they can't run there. The business package also has no reach into the broker's grant
|
||||
tables, and shouldn't get any.
|
||||
|
||||
### Why after step A
|
||||
|
||||
A fire is the decision to stop the agent. If step G came first, a host that can't be reached would keep the
|
||||
fired agent running. After step A, a refused step G leaves the same state as a failed step B today: the
|
||||
instance is "firing" (stopped, still listed) until a repeat. Row 194 already handles that state. The repeat
|
||||
fire's delisted path (webby-0094 N1) also runs step G, since the set is read before step B on both paths.
|
||||
|
||||
## 3. Step G: what it revokes, and how
|
||||
|
||||
The set, for (business, agent), is read in one store transaction: step A's for a fire, the delete's checks
|
||||
for a delete (section 2). It holds:
|
||||
|
||||
- each `host_grants` row of the agent;
|
||||
- each host named by an open host grant decision for the agent;
|
||||
- the Gitea grant if the agent has a `gitea_grants` row, a `gitea_users` row, or an open Gitea grant decision.
|
||||
|
||||
An open decision (d4, B1) is any grant decision raised for the agent with no outcome yet. That is: not denied,
|
||||
not expired, no `grant.applied` or `grant.failed`, and no `grant.revoked` of the same (agent, host), or of the
|
||||
agent's Gitea, recorded after its raise. A raised decision the human hasn't decided is open. d3 required
|
||||
"approved", which left that one out.
|
||||
|
||||
An empty set means no step G and no new records. A fire of an instance with no grant is then exactly today's
|
||||
fire, with no new events (brief tests, fourth bullet).
|
||||
|
||||
### 3a. A host grant: the host line first (D2)
|
||||
|
||||
hosts.revoke goes local first: the record drops, the key leaves memory and its file, and only then is
|
||||
`remove <user> <tag>` sent. If the host can't be reached, the revoke closes with `hostLineLeft` and the
|
||||
record is already gone. The brief says a fire may not drop a grant record while its key could still be on the
|
||||
host. So the fire's host revoke reverses one step and keeps the rest the same:
|
||||
|
||||
1. Abort an executor in flight for (agent, host), with no record change (`hostKeys.abortClaim`, as the
|
||||
`credential-unknown` branch of `#revokeHost` already does).
|
||||
2. `remove <user> <tag>` through the installer, pinned to the host record's keys. `mosaic-keys-write` says
|
||||
"a missing tag changes nothing", so this is safe to repeat and safe for a host named only by an open
|
||||
decision. If the host record is gone, or the remove doesn't exit 0, step G refuses for this host
|
||||
(section 4). Nothing local has changed for this grant: the record, the key file and the memory copy stay.
|
||||
3. hosts.revoke's step 1 transaction, unchanged: the `host_grants` row goes and `grant.revoked {revoke, agent,
|
||||
host, tag, hostLine: 'pending', granted, reason, by}` is recorded, plus `fire: true` (or `delete: true`)
|
||||
so the human can tell it from a manual revoke.
|
||||
4. hosts.revoke's step 2, unchanged: `hostKeys.revoke`, so the key leaves memory and its file goes.
|
||||
5. `grant.revoke.closed {hostLine: 'removed'}` with no second remove. Step 2 already did it.
|
||||
|
||||
This is one shared method, `#revokeHostGrant(business, agent, host, reason, by, { lineFirst })`.
|
||||
hosts.revoke calls it with `lineFirst: false` and its behavior and records don't change (brief item 4). The
|
||||
fire and the delete call it with `lineFirst: true`.
|
||||
|
||||
The tradeoff: hosts.revoke goes local first so that "the part that matters can't wait on the network" (r6
|
||||
section 6). For a fire, step A has already stopped the agent's runs, and a refused fire keeps the instance
|
||||
"firing". So waiting on the host costs nothing in exposure, and the record never lies about the host.
|
||||
|
||||
### 3b. The Gitea grant (D3)
|
||||
|
||||
The Gitea revoke stays grants.revoke's steps with `fire: true`, which is what `credentials delete` already
|
||||
runs (#436b section 6). Step 1 takes the token out of memory and unlinks the file. Step 2 is
|
||||
`#giteaRevokeLocal(..., fire: true)`. Step 3 is `#giteaCloseRevoke`: every token deleted and
|
||||
`prohibit_login: true`. The fire awaits step 3 instead of leaving it to `setImmediate`.
|
||||
|
||||
The Gitea side can't go remote first the way a host does. The record has to drop and `grant.revoked` has to
|
||||
name the tokens before the DELETE calls, or a restart in between would find a live record whose tokens are
|
||||
gone (section 5's reconcile). Two changes keep the fire closed:
|
||||
|
||||
- **Preflight.** Before step 1, `#giteaAdminChecked` runs. If the admin token can't be read or checked,
|
||||
step G refuses with nothing changed.
|
||||
- **Left means not done.** If `grant.revoke.closed` lists anything in `serviceLeft`, step G refuses. The
|
||||
token ids stay in the evidence, which is exactly what `mosaic grants sweep gitea` retries, so nothing
|
||||
forgets them. The record is gone, but no key of this stack's is still usable: the file and memory copy are
|
||||
gone, and only the Gitea side is left. That is the case the brief's host rule guards against, and it is
|
||||
kept visible here, not hidden.
|
||||
|
||||
**A Gitea executor in flight** (d3 T6c, and B3 seen from the executor's side). Step G's Gitea part goes in
|
||||
this order:
|
||||
1. Record `grant.revoked {granted: false}` for each open Gitea decision (3c).
|
||||
2. Await this agent's tracked Gitea executors and remote undos, if any are running. Each one ends `revoked` at
|
||||
its next `#giteaRevokedSince` check, or it committed `applied` before step 1.
|
||||
3. Read the `gitea_grants` row again and run the steps above.
|
||||
|
||||
`#giteaCloseRevoke` with `fire: true` lists the user's tokens and deletes every one. The list comes after
|
||||
step 2, so it includes any token an in-flight executor created. `#giteaRuns` is a single Set today. It
|
||||
becomes keyed by (business, agent) so step G can await one agent's executors alone.
|
||||
|
||||
### 3c. Open decisions
|
||||
|
||||
A host named only by an open decision has no record. Step 3a still runs `remove` (a no-op) and records
|
||||
`grant.revoked {granted: false}`, as hosts.revoke does for a revoke with no record (N4). The executor's
|
||||
`#revokedSince` then ends that decision `revoked`. The Gitea side is the same through
|
||||
`#giteaRevokedSince`.
|
||||
|
||||
This is the case where a decision is approved before the fire and executed after it (sage-2574). Both checks
|
||||
look for a `grant.revoked` of the agent recorded after the decision was raised (broker.mjs:1156 and :1678 at
|
||||
47a35bf43). A fire's `grant.revoked` is always later than the approval, so the executor ends the decision
|
||||
whenever it runs. The fired agent keeps its roles entry, so `#hostDefinition` and `#giteaRole` still pass, and
|
||||
this check is the only thing that stops the decision. T6 and T6b test it. A delete also removes the roles
|
||||
entry, so the executor refuses there as well.
|
||||
|
||||
The same holds for a decision raised but not yet approved (B1). It's in the set, so it gets its
|
||||
`grant.revoked {granted: false}`. If the human approves it later, its executor ends `revoked` (T6d). The
|
||||
decision stays approvable in the inbox. Approving it does nothing but end it, which `grant.failed` reports as
|
||||
it does for any revoked grant.
|
||||
|
||||
### 3d. A raise for a fired agent (d4, B1)
|
||||
|
||||
`#hostGrantBinding` and `#giteaGrantBinding` both run inside the raise transaction (broker.mjs:1539 and
|
||||
:1548; the host raise's keyscan is awaited before that transaction). Both gain one check. The raise refuses
|
||||
`instance-fired` if the agent is:
|
||||
- firing (`#firing`);
|
||||
- fired, by the writer's own rule (instance-settings.mjs, hire): a roles entry, not in `launch.instances`,
|
||||
and not `launch.by`.
|
||||
|
||||
A rehire lists the name again, so a rehired agent can be granted again.
|
||||
|
||||
Because the check is in the transaction that records the decision, there's no gap. A raise that commits
|
||||
before step A's transaction is in the set. One that commits after it refuses. A raise for an agent whose
|
||||
delete's step G is running refuses `grant-revoke-running` (3e).
|
||||
|
||||
No executor recheck is added. A raise can't commit after `fire.stopped`, and every decision raised before it
|
||||
is in the set and gets its `grant.revoked`.
|
||||
|
||||
### 3e. One step G per agent (d4, B3)
|
||||
|
||||
The broker keeps `#revoking`, a map from (business, agent) to the verb whose step G is running. The verb sets
|
||||
it synchronously, in the same turn as the set read, before it answers. The job clears it in a `finally`.
|
||||
While it's set:
|
||||
|
||||
| Caller | Answer |
|
||||
|---|---|
|
||||
| `instance.fire` | refuses `fire-in-progress` |
|
||||
| `lifecycleDelete` | refuses `grants-revoking`, no second job |
|
||||
| `hosts.revoke` / `grants.revoke` for the agent | refuse `grant-revoke-running` |
|
||||
| grant raise for the agent | refuses `grant-revoke-running` |
|
||||
| `grants.sweep` | skips this agent's entries and names it as `running` in its result |
|
||||
|
||||
Two fires started together (the chief retrying after a client timeout): the first takes the lock before it
|
||||
answers, and the second refuses. The map is in memory. After a broker restart the job is gone and so is the
|
||||
lock. The instance is still firing, or the delete still has a non-empty set, and the repeat starts a fresh
|
||||
step G (section 5).
|
||||
|
||||
A manual revoke can't race step G, since it refuses while the lock is held. Step G's transactions still
|
||||
check what they read: a `host_grants` row already gone at 3a step 3 records `grant.revoked {granted: false}`
|
||||
and goes on, as hosts.revoke does today for a revoke with no record.
|
||||
|
||||
## 4. A refusal (D4)
|
||||
|
||||
Step G stops at the first grant that can't be revoked. Grants already revoked in this step G stay revoked and
|
||||
recorded, and the rest stay as they were. The job doesn't run step B: the organization file is unchanged, and
|
||||
the credentials are unrevoked, since `revoke()` never ran. A fire stays "firing". A delete's request was
|
||||
already closed with `grants-revoking` (section 2).
|
||||
|
||||
The verb has already answered, so the refusal is a record, not a thrown code:
|
||||
- One grant-evidence event, `grant.revoke.refused {job, agent, grant: 'host:<id>' | 'gitea', reason, verb:
|
||||
'instance.fire' | 'lifecycle.delete', by}`.
|
||||
- `reason` is one of `host-unknown`, `host-remove-failed`, `gitea-admin-unavailable` and `gitea-left:<ids>`.
|
||||
- `grant-revoke-failed` is the code in the event and in the inbox message.
|
||||
|
||||
A message lands in the human's inbox, as `grant.failed` does today (broker.mjs:1139). It names the instance,
|
||||
the grant and the reason, with the manual way out beside the retry (N2):
|
||||
|
||||
> Fire of <agent> stopped: its grant on <host> couldn't be revoked (<reason>). Run the fire again once <host>
|
||||
> is reachable. If <host> won't come back, run `mosaic hosts revoke <agent> <host>` first: the grant record
|
||||
> goes, the line is named as left, and the next fire completes and names that line (section 5).
|
||||
|
||||
The delete's message has the same shape, with "raise the delete again". For Gitea, the way out is
|
||||
`mosaic grants sweep gitea` once Gitea is back.
|
||||
|
||||
A job that throws something unexpected inside step G records `grant.revoke.refused` with reason `internal`,
|
||||
sends the same message, and releases the lock. There's no unhandled rejection. A failure at step B or C is
|
||||
`fire.failed` (section 2).
|
||||
|
||||
**A restart mid-job (d6, N4).** The lock and the job are in memory, so a restart ends them with no end
|
||||
record. A fire would stay "firing" until the chief happened to repeat it. A delete's request is already
|
||||
closed, so no one would hear anything. Boot closes that gap:
|
||||
- A job is a start record (`fire.revoking` or `grants.revoking`) whose `job` has no end record. The end
|
||||
records are `fire.done`, `fire.failed`, `grants.revoked.done`, `grant.revoke.refused` and the new
|
||||
`grants.job.interrupted`.
|
||||
- When the broker starts, for each such job of each business, it records `grants.job.interrupted {job,
|
||||
agent, verb}` and sends the human one inbox message:
|
||||
|
||||
> <Fire|Delete> of <agent>: job <job> was interrupted by a bus restart. Some grants may be revoked. Run the
|
||||
> fire again, or raise the delete again. The repeat goes on from what's left.
|
||||
|
||||
- Recording the end record in the same transaction as the message means a second boot finds nothing to
|
||||
send.
|
||||
- Boot reads only events, with no network call, so it adds nothing to the start time beyond one query.
|
||||
|
||||
## 5. A repeat fire (D5)
|
||||
|
||||
A repeat runs the whole fire again: the checks, step A (nothing left to stop), the set read, then step G, B
|
||||
and C in the job. A repeat while the job runs refuses `fire-in-progress` (3e). A repeat delete works the same
|
||||
way through the human's new request.
|
||||
|
||||
- A host grant already revoked has no row and no open decision, so it isn't in the set. It's done.
|
||||
- A host grant that refused still has its row, so it's retried from 3a step 1.
|
||||
- Gitea (d4, N1): if the agent has entries in `#giteaLeft` or `#giteaOpenRevokes`, step G retries only those.
|
||||
It calls `#giteaCloseRevoke` for each, as `#sweepGitea` does per entry. It doesn't run the business-wide
|
||||
sweep, which would retry every agent's entries and settle every open rotation under the chief's name. Any
|
||||
entry still left for this agent refuses `gitea-left`. An entry left by an earlier manual
|
||||
`mosaic grants revoke` of the same agent blocks the fire the same way. That's intended: no grant outlives
|
||||
its instance, whichever verb started its revoke. These entries count toward a non-empty set (section 2,
|
||||
fire step 3), so they go to the job and never run inside the verb.
|
||||
- A host line left by an earlier manual `hosts.revoke` doesn't block the fire (Q1 below). These are the
|
||||
agent's `grant.revoke.closed` records with `hostLineLeft` and no later line removal for that tag. For each
|
||||
one whose host record still exists, step G sends `remove <user> <tag>` once. On success it records
|
||||
`grant.revoke.closed {revoke: <the old revoke id>, hostLine: 'removed', fire: true}`, so a later fire doesn't
|
||||
find it again. On failure it records nothing new and the fire goes on. The host and the tag go in
|
||||
`grantsLeft` (section 6). These leftover lines also make the set non-empty, so the retry runs in the job,
|
||||
inside B2's answer (dewey-0823 accepted the single retry on that condition).
|
||||
|
||||
## Rulings
|
||||
|
||||
- **Q1 (sage-2573):** a leftover line doesn't block the fire. If the host is reachable, the fire tries the
|
||||
removal once, because a fired agent's key line still on a host is the exact risk this row closes. If that
|
||||
try fails, the fire doesn't refuse. It names the host and the leftover line in the result and in the event
|
||||
the human sees. The human accepted that residue when their manual `hosts.revoke` left it, so it shouldn't
|
||||
stop a fire forever on a dead host. Dewey may push back on the retry if it widens step G more than it's
|
||||
worth.
|
||||
- **Q2 (sage-2573):** a delete refused at step G closes its request through the launcher's normal refusal
|
||||
path, and the human raises it again. A request left taken would strand, which is the shape DEFERRED already
|
||||
records for the >64-stops done. Conditions: the refusal names the instance, the grant and the reason (the
|
||||
`grant.revoke.refused` event, section 4). The repeat delete skips grants already revoked, as the repeat fire
|
||||
does. T3b tests both.
|
||||
- **Q2 extended (sage-2575):** every delete of a granted instance takes two requests. The first starts the
|
||||
revoke job and refuses `grants-revoking`, and the human raises the delete again when the inbox says it's
|
||||
done. This keeps the cli and the host channel out of the row, and such a delete is rare. Conditions, each
|
||||
with a test:
|
||||
1. The first refusal tells the human what happened and what's next: the named grants, and to raise the
|
||||
delete again when the inbox says done. A bare code isn't enough (T3).
|
||||
2. A delete raised while the job runs refuses `grants-revoking` again and starts no second job (T3, T9).
|
||||
3. The job's end message names each grant revoked. If one failed, it names the grant, the reason and the
|
||||
manual way out (`mosaic hosts revoke` or `mosaic grants revoke`). The next delete then refuses with that
|
||||
reason and doesn't complete over a live grant (T3b).
|
||||
4. A delete of an instance with no grant completes in one request, exactly as today (T3c).
|
||||
5. Section 2 says the fire uses the same answer-at-once job, and the fire's result and event name the job
|
||||
(T2).
|
||||
|
||||
## 6. The result and the event (D6)
|
||||
|
||||
- **A fire with a non-empty set** answers at once with `job`, `revoking` and `done: false`, and records
|
||||
`fire.revoking {job, grants}` (section 2).
|
||||
- **The job's outcome** is recorded, not returned. `fire.done` (step C) carries `job` and `grants: [{grant:
|
||||
'host:<id>' | 'gitea', revoke, granted}]`, plus `grantsLeft` for any leftover from section 5's last bullet.
|
||||
So the event the human sees names them.
|
||||
- **A delete's job** records `grants.revoking {job, ...}` at the start and `grants.revoked.done {job, agent,
|
||||
verb: 'lifecycle.delete', grants, grantsLeft}` at the end. It sends the inbox messages in section 2.
|
||||
- **A fire with an empty set** carries none of these keys, so its shape is exactly today's.
|
||||
|
||||
## 7. What doesn't change (brief item 4)
|
||||
|
||||
- `hosts.revoke`, `grants.revoke` and `grants.sweep`: same callers and the same records. Their refusals stay
|
||||
the same (`tester-dev-only` included) with one addition from section 3e (d6, N5): while a step G job holds
|
||||
the agent's lock, `hosts.revoke` and `grants.revoke` for that agent refuse `grant-revoke-running`, and
|
||||
`grants.sweep` skips the agent and names it as `running`. hosts.revoke's own order stays local first.
|
||||
- No new verb, no Console route, no CLI change, no schema change. Every new record is an `events` row in the
|
||||
existing grant-evidence shape.
|
||||
- `revoke()` in runtime.mjs and the organization writer don't change.
|
||||
- `server.mjs`, `process.mjs`, `cli/src/host.mjs` and `cli/src/launcher.mjs` don't change. Both verbs stay
|
||||
synchronous. The launcher already closes a request with whatever code `lifecycleDelete` refuses.
|
||||
- New codes: `fire-in-progress`, `grants-revoking`, `grant-revoke-running`, `instance-fired`, and
|
||||
`grant-revoke-failed` in events and messages. New event types: `fire.revoking`, `grants.revoking`,
|
||||
`grants.revoked.done`, `grant.revoke.refused`, `fire.failed` and `grants.job.interrupted`.
|
||||
- The broker's start gains one events query for interrupted jobs (section 4, N4). `lifecycleDone` checks a refusal against `LIFECYCLE_CODE =
|
||||
/^[a-z-]{1,64}$/` (broker.mjs:149), and `grants-revoking` matches it.
|
||||
|
||||
## 8. Tests (each in the owning package, whole files)
|
||||
|
||||
The bus tests use the fake installer and the fake Gitea from rows 235 and 236. They wait on the job with a new
|
||||
trusted test API, `grantsIdle()`, which resolves once every step G job and Gitea executor has ended. It's the
|
||||
same shape as `giteaIdle()`. Unless a test says otherwise, every fire or delete below is awaited that way.
|
||||
|
||||
- T1 (pins section 1 before the change): a fire of an instance with a host grant refuses
|
||||
`credential-revoke-failed` today. A delete leaves the `host_grants` row. Then: what the next start does with
|
||||
a record whose key file is gone.
|
||||
- T2: a fire with a host grant on two hosts and a Gitea grant. The verb answers with a `job` id, `revoking`
|
||||
with all three, and `done: false`, and `fire.revoking` carries the same job (sage-2575, condition 5). After
|
||||
the job:
|
||||
- all three are revoked, each with `grant.revoked` and `grant.revoke.closed` as the manual verbs record
|
||||
them, plus `fire: true`;
|
||||
- the line is gone on both fake hosts, every token is deleted, and login is prohibited at the fake Gitea;
|
||||
- `fire.done` names all three;
|
||||
- that agent's `host.run` and Gitea use both refuse.
|
||||
- T3 (sage-2575, conditions 1-3): the same grants through the delete (`lifecycleTake` then
|
||||
`lifecycleDelete`).
|
||||
- The first delete refuses `grants-revoking` and starts the job. The human's inbox gets the start message
|
||||
naming the job and all three grants, and saying to raise the delete again when it's done.
|
||||
- A delete raised while the job is held refuses `grants-revoking`, starts no second job and sends no second
|
||||
message.
|
||||
- After the job, the end message names the job and each grant revoked.
|
||||
- A third request completes as today.
|
||||
- T3b (Q2, sage-2575 condition 3): a delete whose job refuses (host B unreachable).
|
||||
- The end message and `grant.revoke.refused` name the instance, `host:B`, the reason and
|
||||
`mosaic hosts revoke`.
|
||||
- The file and the instance are unchanged.
|
||||
- The next delete refuses `grant-revoke-failed`, not `grants-revoking`. Its message names the earlier job's
|
||||
reason and the new job, and the instance is not deleted.
|
||||
- With B back, that retry job revokes only B and Gitea, skipping host A, and the delete after it completes.
|
||||
- A second case clears B by hand with `hosts.revoke` before the next delete. That delete starts a job only
|
||||
for Gitea and the leftover line.
|
||||
- T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request. Same result,
|
||||
same events counted, no inbox message.
|
||||
- T4: host B unreachable in a fire. The job records `grant.revoke.refused` naming the agent, `host:B` and
|
||||
`host-remove-failed`, and the inbox message names `mosaic hosts revoke` (N2). Afterwards:
|
||||
- the file is unchanged and the instance is still firing;
|
||||
- host A is revoked and recorded, and host B's row and key file are still there;
|
||||
- the Gitea grant is untouched, since it comes after B in the order;
|
||||
- the credentials are unrevoked.
|
||||
A repeat with B reachable completes and revokes only B and Gitea.
|
||||
- T4b (Q1): an earlier manual `hosts.revoke` left a line on host C. A fire with C reachable removes it and
|
||||
records the close. With C unreachable, the fire completes and names C and the tag in `grantsLeft` and in
|
||||
`fire.done`.
|
||||
- T5: Gitea's admin token can't be checked. The job refuses with the Gitea grant intact.
|
||||
- T5b: a token DELETE fails. The job refuses `gitea-left`. A repeat with Gitea back retries only this agent's
|
||||
entries and completes. Another agent's left entry is still left afterwards (N1).
|
||||
- T6 (sage-2574): a host grant decision approved before the fire, executed after it. The fire records
|
||||
`grant.revoked {granted: false}` for (agent, host). The executor runs after `fire.done` and ends the decision
|
||||
`revoked`: no `host_grants` row, no key file, no `add` sent to the host.
|
||||
- T6b (sage-2574): the same for a Gitea grant decision. The executor ends it `revoked` before any Gitea call:
|
||||
no `gitea_grants` row, no token file, and the fake Gitea saw no token create.
|
||||
- T6c: a host or Gitea executor already in flight when step G starts.
|
||||
- The host side aborts its claim (3a step 1) and ends `revoked`.
|
||||
- The Gitea side: step G awaits the executor (3b), and then the fake Gitea holds no token for the user. That
|
||||
includes one the executor created mid-flight.
|
||||
- T6d (B1): a host decision and a Gitea decision, raised before the fire and still undecided. The fire puts
|
||||
both in the set. The human approves both after `fire.done`, and both executors end `revoked`, with nothing
|
||||
applied.
|
||||
- T6e (B1): a host raise and a Gitea raise refuse `instance-fired`:
|
||||
- while the instance is firing;
|
||||
- after `fire.done`.
|
||||
After a rehire of the same name, a raise goes through. While a delete's job runs, a raise refuses
|
||||
`grant-revoke-running`.
|
||||
- T7: an instance with no grant fires exactly as today, synchronously. Same result keys, no `revoking`, and
|
||||
the same events, counted.
|
||||
- T8: hosts.revoke and grants.revoke keep their records, their order and their tester refusals.
|
||||
- T9 (B3): with the job held on a remove:
|
||||
- a second fire refuses `fire-in-progress`;
|
||||
- a delete of the same agent refuses `grants-revoking` and starts no second job;
|
||||
- `hosts.revoke` and `grants.revoke` for the agent refuse `grant-revoke-running`;
|
||||
- `grants.sweep` names the agent as `running`.
|
||||
After a release, one set of `grant.revoked` records exists per grant.
|
||||
- T10 (B2): the fake installer holds a remove and the fake Gitea holds a DELETE, neither answering until
|
||||
released.
|
||||
- `instance.fire` and `lifecycleDelete` both answer before either is released. The test checks this with a
|
||||
timer, not a wall-clock bound.
|
||||
- Meanwhile the broker answers another verb from another session.
|
||||
- After release, the jobs complete.
|
||||
- A cli test drives the launcher's delete pass against a bus whose remove is held. The pass closes the
|
||||
request `grants-revoking` and the host channel stays up.
|
||||
- T11 (d6, C1): a fire whose job is held on a remove. While it's held, the chief's run ends and its claim is
|
||||
released, and processing is halted. After the release, the job completes: `fire.done` is recorded with the
|
||||
job, the consumed `action.allowed` event is there, a second use of the same authority refuses
|
||||
`decision-consumed`, and the instance is out of the file. No `fire.failed` and no `grant.revoke.refused`.
|
||||
- T12 (d6, C1): a fire with a grant whose step B refuses in the job. The test uses T1's setup, where
|
||||
`revoke()` refuses `credential-revoke-failed`. After the job:
|
||||
- `fire.failed` carries the job, step `B` and `credential-revoke-failed`, and the inbox message names the
|
||||
job, the revoked grants, the step and the code;
|
||||
- there's no `grant.revoke.refused` and no `fire.done`;
|
||||
- the instance is still firing and still in the file.
|
||||
With the cause cleared, a repeat fire finds an empty set, runs B and C synchronously and completes.
|
||||
- T13 (d6, N4): a fire and a delete, each with its job held, then the broker is closed and a new one opened
|
||||
on the same data root. The new broker records `grants.job.interrupted` for both jobs and sends two inbox
|
||||
messages, one saying to fire again and one to raise the delete again. A third broker on the same root sends
|
||||
nothing more. A job that ended before the restart gets no interrupted record.
|
||||
|
||||
Files (N3). Every file listed runs whole in the manifest round, touched or not:
|
||||
- New: `packages/bus/tests/fire-revokes-grants.test.mjs` holds T2-T13.
|
||||
- Changed:
|
||||
- `grant-host.test.mjs` and `gitea-grant.test.mjs`: T1's pins, and the T8 regressions next to their
|
||||
existing cases;
|
||||
- `packages/cli/tests/lifecycle.test.mjs`: the T10 launcher pass;
|
||||
- `fake-gitea-api.mjs` and `host-fixtures.mjs`, if T10's holds need a hook there.
|
||||
- Run whole because they call `instance.fire` or `lifecycleDelete` at 47a35bf43:
|
||||
- bus: `chief-of-staff`, `lifecycle-revoke`, `lifecycle-verbs`, `reload`;
|
||||
- cli: `lifecycle`, `instances`;
|
||||
- harness: `tools`.
|
||||
- Also run: `grant-executor.test.mjs` (the executors' `#giteaRuns` change) and `gitea-grant-docker.test.mjs`
|
||||
(brief gate).
|
||||
- `server.mjs`, `process.mjs`, `host.mjs` and `launcher.mjs` are unchanged, so no test changes there for this
|
||||
row.
|
||||
|
||||
## 9. Mutants
|
||||
|
||||
- M1: step G skips the host revokes. Killed by T2.
|
||||
- M2: step G skips the Gitea revoke. Killed by T2.
|
||||
- M3: a refused grant revoke is ignored and the job goes on to step B. Killed by T4.
|
||||
- M4: `lifecycleDelete` skips step G. Killed by T3.
|
||||
- M5: the fire's host revoke goes local first (`lineFirst: false`). Killed by T4, because B's row is gone.
|
||||
- M6: a Gitea revoke with something in `serviceLeft` counts as done. Killed by T5b.
|
||||
- M7: open decisions left out of the set. Killed by T6 and T6b.
|
||||
- M8: step G skips the leftover-line retry. Killed by T4b.
|
||||
- M9: a failed leftover-line retry refuses the fire. Killed by T4b.
|
||||
- M10: the set takes only approved decisions (d3's rule). Killed by T6d.
|
||||
- M11: the raise bindings skip the fired check. Killed by T6e.
|
||||
- M12: no `#revoking` lock. Killed by T9.
|
||||
- M13: the verb awaits the job before it answers. Killed by T10.
|
||||
- M14: step G's Gitea part doesn't await the in-flight executor. Killed by T6c.
|
||||
- M15: step G's Gitea retry runs the business-wide `#sweepGitea`. Killed by T5b.
|
||||
- M16: the delete's first refusal sends no start message. Killed by T3.
|
||||
- M17: a delete after a refused job refuses `grants-revoking` instead of `grant-revoke-failed`. Killed by T3b.
|
||||
- M18: a delete of an instance with no grant goes through the job. Killed by T3c.
|
||||
- M19: the job's step C uses `#consumeAuthority`, with the holder and processing checks. Killed by T11.
|
||||
- M20: a step B or C failure in the job is recorded as `grant.revoke.refused internal`. Killed by T12.
|
||||
- M21: boot skips the interrupted-job check. Killed by T13.
|
||||
|
||||
## 10. Order and rebase
|
||||
|
||||
Built from 47a35bf43 for this read. The code goes on whatever refactor is when the build starts. Rows 274, 256
|
||||
and 194's follow-ups touch the broker and instance-settings, so expect a rebase and a whole rerun. The row lands
|
||||
before the .53 Case 1 install hires an instance with a grant. It doesn't hold the dragon-lin re-pin.
|
||||
|
||||
## Build notes
|
||||
|
||||
Dewey approved d6 (dewey-0829, 10:11Z 10-22) with two build-time items. Both are test details, so there's no
|
||||
new draft:
|
||||
- N7, T12's setup. T1's setup makes `revoke()` refuse because the grant's `host.<id>` file is still in the
|
||||
store. Step G's `hostKeys.revoke` removes that file before step B, so it can't be T12's cause. T12 uses a
|
||||
step B refusal that survives step G, for example a stray file in the agent's store directory or a roles-entry
|
||||
credential whose revoke fails. The test asserts it reaches `fire.failed` with step `B`.
|
||||
- N8, `decision-consumed` in T11. A chief's fire is within-role: `#checkAuthority` returns `{ class }` with no
|
||||
decision, and the lifecycle verbs take none (broker.mjs:3712). `#spent` never applies to a fire. T11 drops the
|
||||
second-use `decision-consumed` assertion. Instead, a repeat fire after `fire.done` takes the delisted path and
|
||||
answers done. Section 2's `decision-consumed` example for `fire.failed` can't occur, so the build uses only
|
||||
codes that can.
|
||||
|
||||
Choices made in the code (b1, broker.mjs only so far). None changes an approved section; each fills in a
|
||||
detail the design left open. Dewey, flag any you read differently.
|
||||
- Record writers. `writeGrantEvent` takes only `grant.*` actions. So `fire.revoking`, `fire.done`, `fire.failed`
|
||||
and `grants.job.interrupted` are not grant evidence. `fire.*` goes through `#event` under the chief's session
|
||||
(the job keeps a copy, `sc`). `grants.revoking`, `grants.revoked.done` and `grants.job.interrupted` go under
|
||||
the bus session `{ role: null, run: null }`, subject the agent. `grant.revoke.refused` is grant evidence, as
|
||||
section 4 has it.
|
||||
- A fired Gitea user. The `gitea_users` row survives a fire (#436b section 6), so on its own the row would
|
||||
never let a fired agent's set empty. The user counts as pending until a later Gitea `grant.revoked` with
|
||||
`fire: true` and a non-null `user` names the agent, and counts again after a new grant step (a rehire).
|
||||
- Open Gitea decisions (3c) end with a `grant.revoked` that has `user: null`, `tokens: []`, `fire: false` and
|
||||
`decisions: [ids]`, plus an immediate close with `serviceLeft: []`. `#giteaRevokedSince` then ends them. Since
|
||||
`user` is null, the pair never counts as the user's fire.
|
||||
- Leftover lines (section 5, Q1). A line is in the set as `line:<host>` when its tag is in `#hostLinesLeft`, its
|
||||
host isn't one the agent holds a grant on, and no `fire.done` or `grants.revoked.done` for the agent since the
|
||||
line was last named left lists it in `grantsLeft`. A line removed in step G gets `grant.revoke.closed` with
|
||||
the old revoke id, `hostLine: 'removed'` and the flag. `#hostLinesLeft` now clears a tag on a later close
|
||||
with `hostLine: 'removed'` and `fire` or `delete` set, so the line also leaves `hosts sweep`'s list. The
|
||||
removed lines appear in `grants` as `{ grant: 'line:<host>', revoke, tag }`.
|
||||
- 3a's wait for host executors. `#runHostGrant` marks itself in `#hostRuns` from its claim to its release. Step
|
||||
G's host part runs `abortClaim`, waits for every mark on (agent, host), and sends the remove. If a new mark
|
||||
appeared during the remove, it runs again, up to five rounds, and then refuses `host-remove-failed`. From the
|
||||
last check to the step 1 commit there's no await.
|
||||
- `hosts sweep gitea` (3e) skips the open revokes, settles and left entries of every agent a job holds, and
|
||||
keeps those entries in `left`. Its answer has `running: [agents]` only when there's at least one. The retry
|
||||
of one left entry is now `#giteaRetry`, which the sweep and step G share. The behavior is unchanged.
|
||||
- Step G's Gitea retries record a sweep-shaped pair for the agent: `grant.sweep { agent, job }`, then the
|
||||
closes and retries, then `grant.swept` whose `left` keeps every other agent's entries. The pair is written
|
||||
first, so a serviceLeft from a close between the two can't drop out of `#giteaLeft`.
|
||||
- `#giteaRuns` is now a Map from each promise to its `business/agent` key, or null for business-wide work.
|
||||
Step G waits for its own agent's keys and the null ones. `#track` adds `.catch` to its `finally` chain, so a
|
||||
rejected executor can't leave an unhandled rejection behind.
|
||||
- A delete repeat after a refusal. `grant-revoke-failed` is returned only when the agent's last delete job end
|
||||
was a `grant.revoke.refused` whose grant is still in the set. If that grant was cleared by hand and only a
|
||||
leftover line remains, the repeat answers `grants-revoking`.
|
||||
- The delete job's reason, in `grant.revoked`, is `lifecycle.delete <request>`.
|
||||
- Step G's messages go to the human's inbox alone (no decision, so no raising role to thread to), through the
|
||||
same insert and route as `#grantNotice`.
|
||||
- `#recordAuthority` is `#consumeAuthority` less `#checkAuthority`'s holder and processing checks, for a
|
||||
fire's step C (C1).
|
||||
|
||||
Choices made in the tests (b2). Each changes where a test sits or how it checks, not what it checks:
|
||||
- T1 is in `lifecycle-revoke.test.mjs`, not `grant-host.test.mjs` and `gitea-grant.test.mjs`. T1 pins the
|
||||
runtime's `revoke()` (step B's credential check) and the start sweep, and only `lifecycle-revoke` builds a
|
||||
broker through `startBroker` with the real credential store. Its `setup` takes a new `seed` hook that writes
|
||||
a host record and grant before the start.
|
||||
- T8 is in `grant-executor.test.mjs` for `hosts.revoke`, where G12, G13, G26 and G27 already are, and in
|
||||
`gitea-grant.test.mjs` for `grants.revoke`. G12 already pins the host verb's two records, their bodies and
|
||||
their order, and G26 its role refusals, so T8 adds what no test had: `tester-dev-only` once `bus.tester` is
|
||||
off, and that a verb revoke writes no `fire.*` or `grants.*` record.
|
||||
- T2-T13 use a fake installer and a fake organization in `fire-revokes-grants.test.mjs`. The organization's
|
||||
`instance.fire` delists and reloads the business, as the writer does, and can throw a set error, which is
|
||||
how T12 gets its step B cause (N7): a `credential-revoke-failed` that survives step G.
|
||||
- "A fired agent's `host.run` and Gitea use refuse" is checked through the state those uses read: no
|
||||
`host_grants` row, no key file, a revoked credential, and no Gitea token for the user.
|
||||
- T10's cli case runs the real runtime and launcher. The seeded grant is on a TEST-NET host
|
||||
(192.0.2.10), so the job's remove fails or waits. The first request closes `grants-revoking` either way, and
|
||||
the case waits on `grantsIdle()` before close.
|
||||
- The mutants are `scratch/megavolt-r281/r281-mutants.sh`, one sed each on `broker.mjs`, run against
|
||||
`fire-revokes-grants.test.mjs`. M13's verbs are synchronous, so its mutant makes the fire return the job's
|
||||
promise, and the call waits for the job. That's what the verb awaiting its job would look like.
|
||||
|
||||
Rebase onto 46dedf925, after row 256 (sage-2597). It's on branch `megavolt/row281-b`, and
|
||||
`megavolt/row281` stays where it is. `git range-diff 0165be266..bd0bb96ae origin/refactor..HEAD` gives `=` for
|
||||
all nine commits, and no path changes on both sides. Row 256 threads the actor through `planDelete`,
|
||||
`checkDelete` and `instances.check`, and adds `protectedRefusal`. That doesn't change what a delete calls in
|
||||
this row:
|
||||
- At the verb, `instances.set` op delete already ran `instances.check` with `s.human` before 256. A
|
||||
creative instance's delete by anyone but the human now refuses there, before any request exists, so no
|
||||
step G job starts for it.
|
||||
- In the pass, step G runs in `lifecycleDelete` before `#organization('instances.set')`, and the actor
|
||||
is still `r.by`. `planDelete` there can still refuse after step G has revoked. Before 256 a references or
|
||||
last-listed refusal could do the same thing, so the order is unchanged. A delete the verb accepted only
|
||||
meets a new refusal in the pass if the file changed between the two.
|
||||
- `read(business, actor)` changes only the listing's `endProtected`. The broker and these tests don't read it.
|
||||
The T10 cli case deletes `sec-1` (security-reviewer), which isn't END_PROTECTED.
|
||||
|
||||
Rebase onto 980cec88c, after row 277 (sage-2605). It's on branch `megavolt/row281-c`.
|
||||
`git range-diff 46dedf925..7454be493 980cec88c..c174f37ac` gives `=` for all ten commits. Row 277's
|
||||
`thread.claims` change to `broker.mjs` doesn't touch step G or any line the mutants change:
|
||||
`scratch/megavolt-r281/anchors.sh` counts each M1-M21 sed anchor in the rebased `broker.mjs`, and all 28
|
||||
counts are 1. The mutants script runs it after a green base and stops if a count isn't 1.
|
||||
|
||||
b3, after the first base run on c174f37ac (11:24:47Z) came back red. These tests had never run before: b2
|
||||
was written under git and docs only. bus: 120 tests, 116 pass, 4 fail. cli lifecycle: 24 tests, 23 pass,
|
||||
1 fail. Every older test in those files passed. No mutant ran. Each failure is a test fault, and b3 changes
|
||||
no code:
|
||||
- T4b, `fire-revokes-grants`: **test changed**. The test read `manual.revoke`, but the `hosts.revoke`
|
||||
answer has never carried `revoke`: `#revokeHost` drops it, so the verb's answer keeps its pre-281 shape.
|
||||
The fire's close names the manual revoke, as section 3a says. The test now takes that id from the
|
||||
manual's `grant.revoked` event.
|
||||
- T5b, `fire-revokes-grants`: **test changed**. The job refused `gitea-left:9,tokens`, and the test
|
||||
expected `gitea-left:<digits>`. Section 4 sets the reason as `gitea-left:<ids>`, the ids left at Gitea.
|
||||
For a fire, a close's `serviceLeft` is the N10 list from before 281: each token id whose DELETE failed,
|
||||
then `tokens` or `login` for a fire step that failed (`#giteaCloseRevoke`). With the DELETE failing,
|
||||
`#giteaFire`'s token list never clears, so `tokens` is a real entry left at Gitea. It belongs in the
|
||||
reason. The test now expects `gitea-left:<id>,tokens`.
|
||||
- T6b, `fire-revokes-grants`: **test changed**. After the fire, the executor made 4 Gitea calls where the
|
||||
test expected none. They are step 0's admin checks C1 to C4: version, self, token scopes and the repo
|
||||
check, all GET. Step 1's `#giteaRevokedSince` then ends the decision `revoked` before any write. Section
|
||||
3c of d6 names that step 1 check as the only thing that stops the decision. T6b's line in section 8 says
|
||||
"before any Gitea call", but the evidence it lists is no row, no token file and no token create. That
|
||||
phrase was wrong, because step 0 always runs first. The approved text stays as it is, and this note is
|
||||
the correction. The test now checks that every call after the fire is a GET and that no token create
|
||||
happened. Moving the revoked check ahead of step 0 would change the executor for every revoke, not just
|
||||
step G's, and the design doesn't ask for that.
|
||||
- T8, `gitea-grant`: **test changed**. The human revoke's events between `grant.revoked` and
|
||||
`grant.revoke.closed` include its 5 `grant.call`/`grant.result` pairs (`#giteaLog` logs both around every
|
||||
Gitea call). T8 checks that a human revoke writes no job records, not that it makes no calls. It now
|
||||
leaves the call log out before comparing.
|
||||
- T10, cli `lifecycle`: **test changed** (sage-2615 asked about this one). The `credential-file` came from
|
||||
`startBroker`, not from the delete and not from Gitea. The seeded host grant gives `sec-1` a `host.web1`
|
||||
store reference. `Credentials` checks the store root (`<config dir>/secrets/agents`) with `privateDir` at
|
||||
start, and the fixture had no store root. That check is row 199's, and it is right to fail closed. The
|
||||
test now makes the store root at 0700 under `dirname(p.f.config)`, as the bus host fixtures do, never
|
||||
under the real config. sage-2575's condition still holds. This business has no Gitea grant, so
|
||||
`#grantSet` adds no `gitea` entry (`#giteaPending` and `#giteaUnfired` read only the store), and step G
|
||||
never reaches `#stepGGitea`. A delete with no Gitea grant reads no Gitea credential file.
|
||||
|
||||
### b4a: the M1 hang's cause (mos-1273, sage-2636)
|
||||
|
||||
The first M1-M21 run on 06ee5ed6b hung on M1. All 22 tests ended in about 80 s (16 fail, 5 pass, 1
|
||||
cancelled), but the file's process stayed alive until the harness killed it at 1762 s. This is a test fault,
|
||||
and no code changed.
|
||||
|
||||
- Cause: T6c holds `web1:add`, and the fake `install` loops in `pause()` on `sleep(5)` while the key is
|
||||
held. Only the fire's abort signal can end that loop, and M1 removes the host revoke that sends the
|
||||
abort. In M1's log, T6c fails at its `until` on line 683, before it deletes the key on line 691. So the loop keeps a
|
||||
timer alive forever. That reading comes from the log and the file. The rerun checks it: before M1-M21 it runs M1
|
||||
once without `--test-force-exit`, and that run must exit by itself inside 180 s.
|
||||
- Fix: setup's `t.after` clears `held` and `failing` before it closes the store, so any held loop ends
|
||||
with its test. The base is unchanged because every passing test already releases its keys.
|
||||
- Sage's second candidate, the Gitea `gate` in T6c and T10, isn't a live handle. The fake server's
|
||||
`t.after` closes all its connections, and a parked promise with no timer doesn't keep node alive. Both
|
||||
tests now also release the gate in `t.after` (`go()` only, not `off()`: a second `off()` would splice
|
||||
index -1 and remove another rule).
|
||||
- The runner's 180 s timeout with `--test-force-exit` (sage-2633) still caps any future hang.
|
||||
|
||||
### b4: T6b pins the four reads (dewey-0839, sage-2627)
|
||||
|
||||
- T6b: **test changed**, stricter. `slice(calls).every(GET)` also passed with zero calls. It now
|
||||
deep-equals the exact list in order: `GET /api/v1/version`, `GET /api/v1/user`, `GET /api/v1/token`,
|
||||
`GET /api/v1/user/repos?limit=1&page=1` (step 0's C1 to C4, `gitea-grant.mjs` adminChecks). It also fails
|
||||
if step 1 stops ending the run and step 2's reads go out. The title says "before any Gitea write".
|
||||
|
||||
### T14 and M22: a declined decision on a removed host (sage-2653, dewey-0844, N1)
|
||||
|
||||
Test only, written before the slot. Not run yet.
|
||||
|
||||
- T14 (new, in `fire-revokes-grants.test.mjs`): coder's web1 decision is raised and left undecided. The human
|
||||
then runs `hosts.remove web1`, which goes through because hosts.remove doesn't check open decisions (N1).
|
||||
The fire starts a job whose set is `host:web1`. Step G refuses `host-unknown` with one
|
||||
`grant.revoke.refused`, and one inbox message names the job, web1 and host-unknown. There's no fire.done.
|
||||
The human resolves the decision `no`. No executor runs (`#approvedGrant` is null for a no), so the closed
|
||||
check in `#openAgentDecisions` is the only thing that takes it out of the set. The repeat fire answers
|
||||
with no job and records one fire.done with no job.
|
||||
- M22 deletes that closed check (`broker.mjs:1684`). The declined decision stays open, so the repeat fire
|
||||
starts a second job and refuses host-unknown again, and T14 fails on `repeat.job`. M10 mutates the same
|
||||
line in another way (approved-only), and T6d kills it. M22 runs on the fire-revokes file like M1-M21, and
|
||||
its anchor makes 29.
|
||||
- N1's hint text (hosts.remove naming the open decision) is a source change. It stays out of this row and
|
||||
goes to a DEFERRED line that Sage files at landing.
|
||||
- If T14 is red on its first run, it goes to Sage and then Mos untouched, because 281 has had its one placed red
|
||||
(mos-1272).
|
||||
|
||||
### Gate plan (sage-2649, dewey-0840, mos-1275)
|
||||
|
||||
Written before the slot. The slot runs on row281-e after the rebase onto 278's landing, and every step
|
||||
stops at the first red.
|
||||
|
||||
1. The base: the touched files whole (fire-revokes-grants, lifecycle-revoke, grant-executor, gitea-grant
|
||||
in bus; lifecycle in cli). Then the 29 anchors, the M1 exit check without `--test-force-exit`, and
|
||||
M1-M22 under `timeout -k 10 180` with `--test-force-exit`.
|
||||
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and
|
||||
gitea-grant-docker; cli instances; harness tools. If gitea-grant-docker skips, the report names the skip
|
||||
and its reason.
|
||||
3. bus, business and cli whole.
|
||||
4. Every `scripts/test-*.sh`, each as `timeout -k 10 LIMIT bash scripts/test-X.sh`, and no script is
|
||||
edited. LIMIT is Gander's row 278 limit (`agents/gander/work/queue-278/SHELL-LIMITS.json`, sage-2646): 3x
|
||||
the longest successful run from Gander's row 196 receipts, plus 120 s headroom. I checked it against two
|
||||
more records. One is my row 236 gate at bd5f11ded (10-22). The other is 278's run 3
|
||||
(`run3/whole/results.json`). Neither has a longer run, so every limit is at least 3x the longest of all
|
||||
three. A script with no limit stops the gate before it starts. Any timeout goes to Sage, then Mos,
|
||||
unplaced.
|
||||
|
||||
| Script | Row 236 | 196 receipts | 278 run 3 | Longest | 3x | LIMIT |
|
||||
|---|---|---|---|---|---|---|
|
||||
| test-auth | 1 s | 1.9 s | 0.8 s | 1.9 s | 6 s | 126 s |
|
||||
| test-conductor | 6 s | 8.1 s | 6.1 s | 8.1 s | 25 s | 145 s |
|
||||
| test-config | 1 s | 3.2 s | 1.9 s | 3.2 s | 10 s | 130 s |
|
||||
| test-discord | 8 s | 8.8 s | 7.4 s | 8.8 s | 27 s | 147 s |
|
||||
| test-extension-package | 1 s | 3.0 s | 1.4 s | 3.0 s | 10 s | 130 s |
|
||||
| test-foundation | 39 s | 39.7 s | 39.9 s | 39.9 s | 120 s | 240 s |
|
||||
| test-land-work | 11 s | 12.6 s | 11.3 s | 12.6 s | 38 s | 158 s |
|
||||
| test-queue | 20 s | 23.4 s | 21.2 s | 23.4 s | 71 s | 191 s |
|
||||
| test-release | 2 s | 12.2 s | 2.7 s | 12.2 s | 37 s | 157 s |
|
||||
| test-task | 14 s | 36.8 s | 13.8 s | 36.8 s | 111 s | 231 s |
|
||||
|
||||
## Runs
|
||||
|
||||
The manifest-round slot on row281-e 7979fb41e (base 385a7194c, row 278's landing), CLEAR sage-2661. Runner
|
||||
`r281-slot.sh` (scratch), from `scratch/megavolt-r222`: 12:43:17Z to 12:59:25Z, 10-22, SLOT_EXIT 0. No red,
|
||||
no timeout. Every step ran in a detached worktree at the candidate, with TMPDIR under
|
||||
`/mnt/storage/scratch/tmp`. The gate worktree was clean at the end, and the trap removed both worktrees.
|
||||
|
||||
1. The base, each file whole, plain `node --test`: bus fire-revokes-grants, lifecycle-revoke,
|
||||
grant-executor and gitea-grant 121/121, 0 skipped, 22 s; cli lifecycle 24/24, 0 skipped, 13 s. T14 passed
|
||||
on its first run. All 29 anchors matched once. The M1 exit check exited by itself: rc 1, 17 fails (the 16
|
||||
on 3f29375a8, plus T14), 63 s. M1-M22: 22 killed, 0 survived (Mutants).
|
||||
2. dewey-0840's files whole: bus chief-of-staff, lifecycle-verbs, reload, grant-host and gitea-grant-docker
|
||||
76/76, 0 skipped, 11 s (gitea-grant-docker ran, with no skip); cli instances 14/14, 0 skipped, 3 s; harness
|
||||
tools 23/23, 0 skipped.
|
||||
3. Whole packages: bus 819/819, 23 s; business 231/231, 3 s; cli 837/837, 591 s. Each has 0 skipped.
|
||||
4. The shells, each under `timeout -k 10 LIMIT`, all rc 0:
|
||||
|
||||
| Script | Result | Time | LIMIT |
|
||||
|---|---|---|---|
|
||||
| test-auth | 15 passed, 0 failed | 1 s | 126 s |
|
||||
| test-conductor | 17 passed, 0 failed | 6 s | 145 s |
|
||||
| test-config | 31 passed, 0 failed | 1 s | 130 s |
|
||||
| test-discord | 81 passed, 0 failed | 8 s | 147 s |
|
||||
| test-extension-package | 18 passed, 0 failed | 1 s | 130 s |
|
||||
| test-foundation | 44 passed, 0 failed | 36 s | 240 s |
|
||||
| test-land-work | 86 passed, 0 failed | 10 s | 158 s |
|
||||
| test-queue | node 156/156; 28 passed, 0 failed | 20 s | 191 s |
|
||||
| test-release | 14 passed, 0 failed | 3 s | 157 s |
|
||||
| test-task | 91 passed, 0 failed | 13 s | 231 s |
|
||||
|
||||
Skips, from a grep of every shell log for `^skip`, `# skip` and `SKIP` (dewey-0845):
|
||||
|
||||
- test-task, lines 87 and 93: `skip live provider cases (row 244: no provider call in the gate)`. Row 244
|
||||
keeps provider calls out of the gate by design. The log has no Docker lines and no other skip.
|
||||
- test-queue, line 34: `skip queue verify and render --check: this checkout (.../r281-gate-wt) is not the
|
||||
queue's canonical root (/mnt/storage/src/mosaic-stack)`. The gate runs in a worktree, so the canonical-root
|
||||
checks are skipped.
|
||||
- No other shell log, and no node suite above, has a skip.
|
||||
|
||||
## Mutants
|
||||
|
||||
Run on b4a 3f29375a8 (base 6af85df4b), 12:16:18Z to 12:20:15Z, 10-22, slot sage-2643; runner
|
||||
`r281-mutants.sh` (scratch). Base first, plain `node --test`, each file whole: bus fire-revokes-grants,
|
||||
lifecycle-revoke, grant-executor and gitea-grant 120/120, 0 skipped; cli lifecycle 24/24, 0 skipped. The
|
||||
worktree was clean after the base, and all 28 sed anchors matched once in `broker.mjs`.
|
||||
|
||||
The M1 exit check (mos-1273) applied M1 and ran fire-revokes-grants without `--test-force-exit`. It exited by
|
||||
itself, rc 1, with 16 fails, in 63 s. The first run on 06ee5ed6b hung there until it was killed at 1762 s
|
||||
(b4a above).
|
||||
|
||||
Each mutant ran fire-revokes-grants.test.mjs whole under `timeout -k 10 180 node --test --test-force-exit`
|
||||
(sage-2633). Every one ran all 21 tests, with no load error and no timeout.
|
||||
|
||||
| Mutant | Change | Verdict | rc | Fails of 21 | Time | Section 9 names | Failing tests (from the log) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| M1 | skip-host-revokes | KILLED | 1 | 16 | 63 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 |
|
||||
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 12 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
|
||||
| M3 | refusal-goes-on | KILLED | 1 | 3 | 3 s | T4 | T4 T5 T5b |
|
||||
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
|
||||
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
|
||||
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||||
| M7 | set-no-open-decisions | KILLED | 1 | 4 | 3 s | T6, T6b | T6 T6b T6c T6d |
|
||||
| M8 | skip-line-retry | KILLED | 1 | 3 | 4 s | T4b | T3b T4b×2 |
|
||||
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
|
||||
| M10 | approved-only | KILLED | 1 | 1 | 3 s | T6d | T6d |
|
||||
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
|
||||
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
|
||||
| M13 | verb-awaits-job | KILLED | 1 | 14 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 |
|
||||
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
|
||||
| M15 | business-wide-sweep | KILLED | 1 | 1 | 4 s | T5b | T5b |
|
||||
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
|
||||
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
|
||||
| M18 | empty-set-job | KILLED | 1 | 4 | 4 s | T3c | T3 T3b×2 T3c |
|
||||
| M19 | step-c-consume | KILLED | 1 | 1 | 4 s | T11 | T11 |
|
||||
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
|
||||
| M21 | no-interrupted-check | KILLED | 1 | 1 | 4 s | T13 | T13 |
|
||||
|
||||
Every mutant fails the test section 9 names for it; none was killed only by a different test. "T3b×2" and "T4b×2" are the two tests that share each id. The failing-test ids are taken from each log's "failing tests" list (dewey-0841).
|
||||
|
||||
21 killed, 0 survived. b4 makes one T6b assertion stricter after this run, and fire-revokes-grants is
|
||||
rerun whole on b4 (Runs).
|
||||
|
||||
### On row281-e 7979fb41e (sage-2661)
|
||||
|
||||
12:43:17Z to 12:47:12Z, 10-22, the first part of the manifest-round slot (Runs). Same runner and flags as
|
||||
above, with M22 added (T14 and M22 in the build notes). Every mutant ran all 22 tests, with no load error and
|
||||
no timeout.
|
||||
|
||||
| Mutant | Change | Verdict | rc | Fails of 22 | Time | Section 9 names | Failing tests (from the log) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| M1 | skip-host-revokes | KILLED | 1 | 17 | 62 s | T2 | T2 T3 T3b×2 T4 T4b×2 T6 T6c T6d T6e T9 T10 T11 T12 T13 T14 |
|
||||
| M2 | skip-gitea-revoke | KILLED | 1 | 12 | 13 s | T2 | T2 T3 T3b×2 T4 T5 T5b T6b T6c T6d T9 T10 |
|
||||
| M3 | refusal-goes-on | KILLED | 1 | 4 | 3 s | T4 | T4 T5 T5b T14 |
|
||||
| M4 | delete-skips-step-g | KILLED | 1 | 6 | 3 s | T3 | T3 T3b×2 T6e T10 T13 |
|
||||
| M5 | host-local-first | KILLED | 1 | 4 | 3 s | T4 | T2 T3b×2 T4 |
|
||||
| M6 | service-left-done | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||||
| M7 | set-no-open-decisions | KILLED | 1 | 5 | 3 s | T6, T6b | T6 T6b T6c T6d T14 |
|
||||
| M8 | skip-line-retry | KILLED | 1 | 3 | 3 s | T4b | T3b T4b×2 |
|
||||
| M9 | line-retry-refuses | KILLED | 1 | 2 | 3 s | T4b | T3b T4b |
|
||||
| M10 | approved-only | KILLED | 1 | 2 | 3 s | T6d | T6d T14 |
|
||||
| M11 | raise-no-fired-check | KILLED | 1 | 3 | 3 s | T6e | T4 T6e T12 |
|
||||
| M12 | no-lock | KILLED | 1 | 3 | 3 s | T9 | T3 T6e T9 |
|
||||
| M13 | verb-awaits-job | KILLED | 1 | 15 | 3 s | T10 | T2 T4 T4b×2 T5 T5b T6 T6b T6c T6d T10 T11 T12 T13 T14 |
|
||||
| M14 | no-executor-wait | KILLED | 1 | 1 | 3 s | T6c | T6c |
|
||||
| M15 | business-wide-sweep | KILLED | 1 | 1 | 3 s | T5b | T5b |
|
||||
| M16 | no-start-message | KILLED | 1 | 2 | 3 s | T3 | T3 T3b |
|
||||
| M17 | repeat-grants-revoking | KILLED | 1 | 1 | 3 s | T3b | T3b |
|
||||
| M18 | empty-set-job | KILLED | 1 | 4 | 3 s | T3c | T3 T3b×2 T3c |
|
||||
| M19 | step-c-consume | KILLED | 1 | 1 | 3 s | T11 | T11 |
|
||||
| M20 | bc-as-internal | KILLED | 1 | 2 | 3 s | T12 | T6e T12 |
|
||||
| M21 | no-interrupted-check | KILLED | 1 | 1 | 3 s | T13 | T13 |
|
||||
| M22 | no-closed-check | KILLED | 1 | 1 | 3 s | T14 | T14 |
|
||||
|
||||
22 killed, 0 survived. Every mutant fails the test it's named for, and M22 fails T14 alone. Compared with
|
||||
3f29375a8, the only new failures are T14's (under M1, M3, M7, M10, M13 and M22), and every other failing set is
|
||||
unchanged.
|
||||
+548
-45
@@ -393,8 +393,15 @@ export class Broker {
|
||||
#giteaGrant;
|
||||
#giteaKeys;
|
||||
#giteaUndo = [];
|
||||
#giteaRuns = new Set();
|
||||
// Row 281 (#1786, d6 section 3b): each tracked Gitea promise and the business/agent it acts for, or
|
||||
// null for business-wide work, so a step G job can await one agent's executors.
|
||||
#giteaRuns = new Map();
|
||||
#giteaRevokeUndo = [];
|
||||
// Row 281 (d6 section 3e): business/agent to the verb whose step G job runs, the jobs themselves,
|
||||
// and the host executors in flight by business/agent/host (section 3a).
|
||||
#revoking = new Map();
|
||||
#grantJobs = new Set();
|
||||
#hostRuns = new Map();
|
||||
// Row 125 (#1619, REQ-LIMIT-4): a run a limit stops, by business:run, until its next receive or state.
|
||||
#stops = new Map();
|
||||
#clock = 0;
|
||||
@@ -1271,6 +1278,8 @@ export class Broker {
|
||||
} catch {}
|
||||
return code;
|
||||
}
|
||||
// Row 281 (d6 section 3a): marked from the claim to the release, so a step G job can await it.
|
||||
const ran = this.#hostRunMark(business, g.agent, g.host);
|
||||
const tag = this.#hostGrant.grantTag(business, g.agent, g.host);
|
||||
const change = previous !== null;
|
||||
const step = (name, extra = {}) =>
|
||||
@@ -1372,8 +1381,23 @@ export class Broker {
|
||||
return code;
|
||||
} finally {
|
||||
this.#hostKeys.release(business, g.agent, g.host, signal);
|
||||
ran();
|
||||
}
|
||||
}
|
||||
// Row 281 (d6 section 3a): marks one host executor for (agent, host) and returns its end.
|
||||
#hostRunMark(business, agent, host) {
|
||||
const key = business + '/' + agent + '/' + host;
|
||||
let end;
|
||||
const p = new Promise((resolve) => (end = resolve));
|
||||
if (!this.#hostRuns.has(key)) this.#hostRuns.set(key, new Set());
|
||||
this.#hostRuns.get(key).add(p);
|
||||
return () => {
|
||||
const set = this.#hostRuns.get(key);
|
||||
set?.delete(p);
|
||||
if (set?.size === 0) this.#hostRuns.delete(key);
|
||||
end();
|
||||
};
|
||||
}
|
||||
// r6 X3 and duckworth-0067 N1: credential.mint decisions for `method`, resolved yes by the human via
|
||||
// cli and never spent.
|
||||
#openGrantDecisions(business, method) {
|
||||
@@ -1458,9 +1482,40 @@ export class Broker {
|
||||
for (const m of this.#openRevokes(business)) this.#revokeUndo.push({ business, ...m });
|
||||
// Row 236 (#436b r5 section 5): the Gitea grants' local part.
|
||||
if (this.#giteaGrant && this.#giteaKeys) ended.push(...this.#giteaReconcileLocal(business));
|
||||
this.#interruptedJobs(business);
|
||||
}
|
||||
return ended;
|
||||
}
|
||||
// Row 281 (d6 section 4, N4): a step G job is in memory, so a restart ends it with no end record. Each
|
||||
// start record with no end gets grants.job.interrupted and one message to the human, in the same
|
||||
// transaction, so a second boot sends nothing. A job this broker is running is left alone.
|
||||
#interruptedJobs(business) {
|
||||
for (const e of this.#store.all(
|
||||
"SELECT subject, body FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.action') IN ('fire.revoking','grants.revoking') ORDER BY seq",
|
||||
business,
|
||||
)) {
|
||||
const b = JSON.parse(e.body);
|
||||
const fire = b.action === 'fire.revoking';
|
||||
const agent = fire ? e.subject : b.agent;
|
||||
if (this.#revoking.has(business + '/' + agent)) continue;
|
||||
if (
|
||||
this.#store.get(
|
||||
"SELECT 1 FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.job')=? AND json_extract(body,'$.action') IN ('fire.done','fire.failed','grants.revoked.done','grant.revoke.refused','grants.job.interrupted') LIMIT 1",
|
||||
business,
|
||||
b.job,
|
||||
)
|
||||
)
|
||||
continue;
|
||||
const verb = fire ? 'instance.fire' : b.verb;
|
||||
this.#store.transaction(() => {
|
||||
this.#event({ business, role: null, run: null }, 'action.allowed', { action: 'grants.job.interrupted', job: b.job, agent, verb }, agent);
|
||||
this.#humanNotice(
|
||||
business,
|
||||
`${fire ? 'Fire' : 'Delete'} of ${agent}: job ${b.job} was interrupted by a bus restart. Some grants may be revoked. ${fire ? 'Run the fire again' : 'Raise the delete again'}. The repeat goes on from what's left.`,
|
||||
);
|
||||
});
|
||||
}
|
||||
}
|
||||
// Trusted host API, after the socket opens (r6 X3): the remote undo of each grant reconcileGrants left
|
||||
// claimed. `remove <user> <tag>` through the installer, then grant.failed interrupted, with
|
||||
// hostLineLeft if the remove failed, and the claim released. It only takes access away, so it needs no
|
||||
@@ -1537,31 +1592,410 @@ export class Broker {
|
||||
keys(a, ['agent', 'host', 'reason'], ['agent', 'host', 'reason']);
|
||||
if (typeof a.agent !== 'string' || !ID_PATTERN.test(a.agent) || typeof a.host !== 'string' || !HOST_ID.test(a.host)) fail('invalid-request');
|
||||
string(a.reason, 500);
|
||||
// Row 281 (d6 section 3e): no manual revoke while a step G job holds the agent.
|
||||
if (this.#revoking.has(s.business + '/' + a.agent)) fail('grant-revoke-running');
|
||||
const { revoke, ...out } = await this.#revokeHostGrant(s.business, a.agent, a.host, a.reason, by);
|
||||
return out;
|
||||
}
|
||||
// Row 281 (#1786, d6 section 3a): hosts.revoke's steps, shared with a fire's and a delete's step G.
|
||||
// `lineFirst` sends the remove before anything local changes: the executor in flight aborts and ends,
|
||||
// then `remove <user> <tag>`, and a refusal there (host-unknown, host-remove-failed) leaves the
|
||||
// record, the key file and the memory copy as they were. An executor that started during the remove
|
||||
// is aborted and awaited, and the remove runs again. The close then records the remove already done.
|
||||
// `flag` ({ fire: true } or { delete: true }) goes in grant.revoked and the close.
|
||||
async #revokeHostGrant(business, agent, host, reason, by, { lineFirst = false, flag = {} } = {}) {
|
||||
if (!this.#hostGrant) fail('host-not-configured');
|
||||
const tag = this.#hostGrant.grantTag(s.business, a.agent, a.host);
|
||||
const tag = this.#hostGrant.grantTag(business, agent, host);
|
||||
const revoke = randomUUID();
|
||||
if (lineFirst) {
|
||||
const key = business + '/' + agent + '/' + host;
|
||||
for (let round = 0; ; round++) {
|
||||
if (round === 5) fail('host-remove-failed');
|
||||
this.#hostKeys?.abortClaim(business, agent, host);
|
||||
while (this.#hostRuns.get(key)?.size) await Promise.all([...this.#hostRuns.get(key)]);
|
||||
const record = this.#hostRecord(business, host);
|
||||
if (!record) fail('host-unknown');
|
||||
try {
|
||||
await this.#hostGrant.remove(business, host, record, tag);
|
||||
} catch {
|
||||
fail('host-remove-failed');
|
||||
}
|
||||
// No await from this check to the step 1 commit, so no executor starts in between.
|
||||
if (!this.#hostRuns.get(key)?.size) break;
|
||||
}
|
||||
}
|
||||
const granted = this.#store.transaction(() => {
|
||||
if (!this.#hostRecord(s.business, a.host)) fail('host-unknown');
|
||||
const out = this.#store.run('DELETE FROM host_grants WHERE business=? AND agent=? AND host=?', s.business, a.agent, a.host);
|
||||
if (!this.#hostRecord(business, host)) fail('host-unknown');
|
||||
const out = this.#store.run('DELETE FROM host_grants WHERE business=? AND agent=? AND host=?', business, agent, host);
|
||||
this.#grantEvent(
|
||||
s.business,
|
||||
{ action: 'grant.revoked', revoke, agent: a.agent, host: a.host, tag, hostLine: 'pending', granted: out.changes === 1, reason: a.reason, by },
|
||||
a.host,
|
||||
business,
|
||||
{ action: 'grant.revoked', revoke, agent, host, tag, hostLine: 'pending', granted: out.changes === 1, reason, by, ...flag },
|
||||
host,
|
||||
);
|
||||
return out.changes === 1;
|
||||
});
|
||||
let keyFile = 'removed';
|
||||
if (this.#hostKeys) {
|
||||
try {
|
||||
this.#hostKeys.revoke(s.business, a.agent, a.host);
|
||||
this.#hostKeys.revoke(business, agent, host);
|
||||
} catch (e) {
|
||||
// No reference: only a claim can be held, an executor's before its register.
|
||||
if (e instanceof BusError && e.code === 'credential-unknown') this.#hostKeys.abortClaim(s.business, a.agent, a.host);
|
||||
if (e instanceof BusError && e.code === 'credential-unknown') this.#hostKeys.abortClaim(business, agent, host);
|
||||
else keyFile = 'left';
|
||||
}
|
||||
}
|
||||
const closed = await this.#closeRevoke(s.business, { revoke, agent: a.agent, host: a.host, tag, by });
|
||||
return { agent: a.agent, host: a.host, tag, granted, keyFile, ...(closed.hostLineLeft ? { hostLineLeft: closed.hostLineLeft } : { hostLine: 'removed' }) };
|
||||
let closed;
|
||||
if (lineFirst) {
|
||||
closed = { action: 'grant.revoke.closed', revoke, agent, host, tag, hostLine: 'removed', ...flag, by };
|
||||
this.#store.transaction(() => this.#grantEvent(business, closed, host));
|
||||
} else closed = await this.#closeRevoke(business, { revoke, agent, host, tag, by });
|
||||
return { agent, host, tag, revoke, granted, keyFile, ...(closed.hostLineLeft ? { hostLineLeft: closed.hostLineLeft } : { hostLine: 'removed' }) };
|
||||
}
|
||||
// Row 281 (#1786, d6 section 3): the step G set of (business, agent), each with its label and what it
|
||||
// revokes. The caller reads it in one transaction: step A's for a fire, the delete's checks for a
|
||||
// delete. Hosts first, by id, then Gitea, then the leftover lines (section 5).
|
||||
#grantSet(business, agent) {
|
||||
const set = [];
|
||||
const open = this.#openAgentDecisions(business, agent);
|
||||
const hosts = new Set(this.#store.all('SELECT host FROM host_grants WHERE business=? AND agent=?', business, agent).map((r) => r.host));
|
||||
for (const d of open) if (d.grant.method === 'host') hosts.add(d.grant.host);
|
||||
for (const host of [...hosts].sort()) set.push({ label: 'host:' + host, kind: 'host', host });
|
||||
if (this.#giteaPending(business, agent, open)) set.push({ label: 'gitea', kind: 'gitea' });
|
||||
if (this.#hostGrant)
|
||||
for (const { host } of this.#store.all('SELECT host FROM hosts WHERE business=? ORDER BY host', business)) {
|
||||
if (hosts.has(host)) continue;
|
||||
const tag = this.#hostGrant.grantTag(business, agent, host);
|
||||
if (this.#hostLinesLeft(business, host).includes(tag) && !this.#lineTried(business, agent, host, tag))
|
||||
set.push({ label: 'line:' + host, kind: 'line', host, tag });
|
||||
}
|
||||
return set;
|
||||
}
|
||||
// Section 3 (d4, B1): the agent's host and Gitea grant decisions with no outcome yet. Not denied or
|
||||
// expired, no grant.applied or grant.failed, and no grant.revoked of the grant since its raise. An
|
||||
// undecided one counts.
|
||||
#openAgentDecisions(business, agent) {
|
||||
return this.#store
|
||||
.all("SELECT id FROM decisions WHERE business=? AND action='credential.mint' AND route_to='human' ORDER BY at,id", business)
|
||||
.map((d) => ({ id: d.id, raise: this.#raised(business, d.id) }))
|
||||
.filter(({ id, raise }) => {
|
||||
const g = raise?.grant;
|
||||
if (g?.agent !== agent || (g.method !== 'host' && g.method !== 'gitea')) return false;
|
||||
const r = this.#closed(id);
|
||||
if (r && !(r.op === 'resolved' && r.choice === raise.approvalChoice)) return false;
|
||||
if (
|
||||
this.#store.get(
|
||||
`SELECT 1 FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.grantDecision')=? AND json_extract(body,'$.action') IN ('grant.applied','grant.failed') LIMIT 1`,
|
||||
business,
|
||||
id,
|
||||
)
|
||||
)
|
||||
return false;
|
||||
return g.method === 'host' ? !this.#revokedSince(business, id, agent, g.host) : !this.#giteaRevokedSince(business, id, agent);
|
||||
})
|
||||
.map(({ id, raise }) => ({ id, grant: raise.grant }));
|
||||
}
|
||||
// Sections 3 and 5: whether step G has a Gitea part for the agent. A gitea_grants row, an open Gitea
|
||||
// decision, a left entry, an open revoke, or a stack-made user no fire has closed.
|
||||
#giteaPending(business, agent, open) {
|
||||
if (this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, agent)) return true;
|
||||
if (open.some((d) => d.grant.method === 'gitea')) return true;
|
||||
if (this.#giteaLeft(business).some((e) => e.agent === agent)) return true;
|
||||
if (this.#giteaAgentRevokes(business, agent).length) return true;
|
||||
return this.#giteaUnfired(business, agent);
|
||||
}
|
||||
// A gitea_users row survives a fire (#436b section 6), so the row alone would never let a fired
|
||||
// agent's set empty. The user counts until a grant.revoked with fire: true and a user names the
|
||||
// agent after its last Gitea grant step. A new grant after a rehire makes it count again.
|
||||
#giteaUnfired(business, agent) {
|
||||
if (!this.#store.get('SELECT 1 FROM gitea_users WHERE business=? AND agent=?', business, agent)) return false;
|
||||
const last = (where) =>
|
||||
this.#store.get(
|
||||
`SELECT MAX(seq) AS seq FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.method')='gitea' AND json_extract(body,'$.agent')=? AND ${where}`,
|
||||
business,
|
||||
agent,
|
||||
)?.seq ?? 0;
|
||||
const fired = last("json_extract(body,'$.action')='grant.revoked' AND json_extract(body,'$.fire')=1 AND json_extract(body,'$.user') IS NOT NULL");
|
||||
return fired === 0 || fired < last("json_extract(body,'$.action')='grant.step'");
|
||||
}
|
||||
// The agent's open Gitea revokes, less those reconcileGitea is closing.
|
||||
#giteaAgentRevokes(business, agent) {
|
||||
return this.#giteaOpenRevokes(business).filter((m) => m.agent === agent && !this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke));
|
||||
}
|
||||
// Section 5 (Q1): a job end since the line was last named as left tried it once and listed it in
|
||||
// grantsLeft. That line doesn't make the set non-empty again.
|
||||
#lineTried(business, agent, host, tag) {
|
||||
const since =
|
||||
this.#store.get(
|
||||
`SELECT MAX(seq) AS seq FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND (json_extract(body,'$.hostLineLeft')=? OR (json_extract(body,'$.action')='grant.host.swept' AND EXISTS (SELECT 1 FROM json_each(body,'$.left') WHERE value=?)))`,
|
||||
business,
|
||||
host,
|
||||
tag,
|
||||
tag,
|
||||
)?.seq ?? 0;
|
||||
return this.#store
|
||||
.all(
|
||||
"SELECT body FROM events WHERE business=? AND kind='action.allowed' AND subject=? AND json_extract(body,'$.action') IN ('fire.done','grants.revoked.done') AND seq > ?",
|
||||
business,
|
||||
agent,
|
||||
since,
|
||||
)
|
||||
.some((e) => (JSON.parse(e.body).grantsLeft ?? []).some((l) => l.host === host && l.tag === tag));
|
||||
}
|
||||
// Section 3: step G, in the set's order. Each host goes line first (3a), then Gitea (3b, 3c and
|
||||
// section 5's retries), then each leftover line once (Q1). At the first grant it can't revoke it
|
||||
// throws { grantRefusal: { grant, reason } }, and what it revoked before that stays revoked and
|
||||
// recorded. An unexpected throw is reason internal (section 4). Returns the grants revoked and the
|
||||
// lines still left.
|
||||
async #stepG(business, agent, set, { job, reason, by, flag }) {
|
||||
const grants = [];
|
||||
const grantsLeft = [];
|
||||
let at = set[0]?.label ?? null;
|
||||
const refuse = (grant, why) => {
|
||||
throw { grantRefusal: { grant, reason: why } };
|
||||
};
|
||||
try {
|
||||
for (const g of set.filter((x) => x.kind === 'host')) {
|
||||
at = g.label;
|
||||
let out;
|
||||
try {
|
||||
out = await this.#revokeHostGrant(business, agent, g.host, reason, by, { lineFirst: true, flag });
|
||||
} catch (e) {
|
||||
if (e instanceof BusError && ['host-unknown', 'host-remove-failed', 'host-not-configured'].includes(e.code))
|
||||
refuse(g.label, e.code === 'host-unknown' ? 'host-unknown' : 'host-remove-failed');
|
||||
throw e;
|
||||
}
|
||||
grants.push({ grant: g.label, revoke: out.revoke, granted: out.granted });
|
||||
}
|
||||
if (set.some((x) => x.kind === 'gitea')) {
|
||||
at = 'gitea';
|
||||
grants.push(...(await this.#stepGGitea(business, agent, { job, reason, by, flag }, refuse)));
|
||||
}
|
||||
for (const g of set.filter((x) => x.kind === 'line')) {
|
||||
at = g.label;
|
||||
const record = this.#hostRecord(business, g.host);
|
||||
let removed = false;
|
||||
try {
|
||||
if (record) {
|
||||
await this.#hostGrant.remove(business, g.host, record, g.tag);
|
||||
removed = true;
|
||||
}
|
||||
} catch {}
|
||||
if (!removed) {
|
||||
grantsLeft.push({ host: g.host, tag: g.tag });
|
||||
continue;
|
||||
}
|
||||
const old =
|
||||
this.#store.get(
|
||||
`SELECT json_extract(body,'$.revoke') AS revoke FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND json_extract(body,'$.hostLineLeft')=? ORDER BY seq DESC LIMIT 1`,
|
||||
business,
|
||||
g.host,
|
||||
g.tag,
|
||||
)?.revoke ?? null;
|
||||
this.#store.transaction(() =>
|
||||
this.#grantEvent(business, { action: 'grant.revoke.closed', revoke: old, agent, host: g.host, tag: g.tag, hostLine: 'removed', ...flag, by }, g.host),
|
||||
);
|
||||
grants.push({ grant: g.label, revoke: old, tag: g.tag });
|
||||
}
|
||||
} catch (e) {
|
||||
if (e?.grantRefusal) throw e;
|
||||
throw { grantRefusal: { grant: at, reason: 'internal' } };
|
||||
}
|
||||
return { grants, grantsLeft };
|
||||
}
|
||||
// Section 3b, 3c and 5 (d4, N1): step G's Gitea part. Open decisions end first. Then this agent's
|
||||
// executors and remote undos are awaited, the admin token is checked, this agent's own left entries
|
||||
// and open revokes are retried, and the grant goes with fire: true. Anything still left at Gitea
|
||||
// refuses gitea-left.
|
||||
async #stepGGitea(business, agent, { job, reason, by, flag }, refuse) {
|
||||
const key = business + '/' + agent;
|
||||
const done = [];
|
||||
const open = this.#openAgentDecisions(business, agent).filter((d) => d.grant.method === 'gitea');
|
||||
if (open.length) {
|
||||
// No user and no token: the record only ends the decisions at #giteaRevokedSince (3c).
|
||||
const revoke = randomUUID();
|
||||
this.#store.transaction(() => {
|
||||
this.#grantEvent(
|
||||
business,
|
||||
{ action: 'grant.revoked', method: 'gitea', revoke, agent, user: null, userId: null, tokens: [], fire: false, granted: false, decisions: open.map((d) => d.id), reason, by, ...flag },
|
||||
agent,
|
||||
);
|
||||
this.#grantEvent(business, { action: 'grant.revoke.closed', method: 'gitea', revoke, agent, user: null, serviceLeft: [], by }, agent);
|
||||
});
|
||||
done.push({ grant: 'gitea', revoke, granted: false });
|
||||
}
|
||||
const mine = () => [...this.#giteaRuns].filter(([, k]) => k === key || k === null).map(([p]) => p);
|
||||
while (mine().length) await Promise.allSettled(mine());
|
||||
const live = this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, agent) || this.#giteaUnfired(business, agent);
|
||||
const opens = this.#giteaAgentRevokes(business, agent);
|
||||
const retry = this.#giteaLeft(business).some((e) => e.agent === agent);
|
||||
if (!live && !opens.length && !retry) return done;
|
||||
if (!this.#giteaGrant || !this.#giteaKeys) refuse('gitea', 'gitea-admin-unavailable');
|
||||
// Preflight: nothing changes unless the admin token checks.
|
||||
const admin = await this.#giteaAdminChecked(business, this.#giteaLog(business, job));
|
||||
if (!admin) refuse('gitea', 'gitea-admin-unavailable');
|
||||
const still = [];
|
||||
if (opens.length || retry) {
|
||||
// The sweep's records for this agent alone. Every other agent's entry stays in left.
|
||||
const sweep = randomUUID();
|
||||
const all = this.#store.transaction(() => {
|
||||
this.#grantEvent(business, { action: 'grant.sweep', method: 'gitea', sweep, agent, job, by });
|
||||
return this.#giteaLeft(business);
|
||||
});
|
||||
const kept = [];
|
||||
const skipped = [];
|
||||
const closed = [];
|
||||
for (const m of opens) {
|
||||
const c = await this.#giteaCloseRevoke(business, m);
|
||||
closed.push(m.revoke);
|
||||
for (const id of c.serviceLeft) kept.push({ agent, user: m.user, id });
|
||||
}
|
||||
for (const e of all.filter((x) => x.agent === agent)) {
|
||||
const r = await this.#giteaRetry(business, admin, e);
|
||||
if (r.skipped) skipped.push({ agent, user: e.user, id: e.id, reason: r.skipped });
|
||||
else if (!r.ok) kept.push({ agent, user: e.user, id: e.id });
|
||||
}
|
||||
const others = all.filter((x) => x.agent !== agent).map((x) => ({ agent: x.agent, user: x.user, id: x.id }));
|
||||
this.#store.transaction(() =>
|
||||
this.#grantEvent(business, { action: 'grant.swept', method: 'gitea', sweep, left: [...others, ...kept], skipped, settled: [], closed, agent, job, by }),
|
||||
);
|
||||
still.push(...kept);
|
||||
}
|
||||
if (live) {
|
||||
try {
|
||||
this.#giteaKeys.revoke(business, agent);
|
||||
} catch (e) {
|
||||
if (e instanceof BusError && e.code === 'credential-unknown') this.#giteaKeys.abortClaim(business, agent);
|
||||
else throw e;
|
||||
}
|
||||
const m = this.#giteaRevokeLocal(business, agent, reason, by, true, flag.delete ? { delete: true } : {});
|
||||
const c = await this.#giteaCloseRevoke(business, m);
|
||||
for (const id of c.serviceLeft) still.push({ agent, user: m.user, id });
|
||||
done.push({ grant: 'gitea', revoke: m.revoke, granted: m.granted });
|
||||
}
|
||||
if (still.length) refuse('gitea', 'gitea-left:' + [...new Set(still.map((e) => e.id))].join(','));
|
||||
return done;
|
||||
}
|
||||
// Row 281 (d6 section 2, 3e): a step G job. The lock is taken now, in the verb's turn, and the job
|
||||
// starts after the verb answers. The lock goes in a finally, whichever way the job ends.
|
||||
#startGrantJob(business, agent, verb, run) {
|
||||
const key = business + '/' + agent;
|
||||
this.#revoking.set(key, verb);
|
||||
const job = (async () => {
|
||||
try {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
await run();
|
||||
} finally {
|
||||
this.#revoking.delete(key);
|
||||
}
|
||||
})();
|
||||
this.#grantJobs.add(job);
|
||||
job.catch(() => {}).finally(() => this.#grantJobs.delete(job));
|
||||
}
|
||||
// Section 2, the fire's job: step G, then step B and step C. A step G refusal is section 4's record. A
|
||||
// step B or C failure is fire.failed (d6, C1). Step C records the authority the verb checked, with no
|
||||
// holder check, since the chief's run may have ended since.
|
||||
async #fireJob(sc, args, authority, write, job, set) {
|
||||
const name = args.name;
|
||||
let out;
|
||||
try {
|
||||
out = await this.#stepG(sc.business, name, set, { job, reason: args.reason, by: sc.role, flag: { fire: true } });
|
||||
} catch (e) {
|
||||
this.#store.transaction(() => this.#grantRefused(sc.business, name, 'instance.fire', job, e.grantRefusal, sc.role));
|
||||
return;
|
||||
}
|
||||
let step = 'B';
|
||||
try {
|
||||
const result = write('instance.fire');
|
||||
step = 'C';
|
||||
this.#store.transaction(() => {
|
||||
this.#recordAuthority(sc, 'role.fire', authority, { reason: args.reason, transaction: result.transaction ?? null });
|
||||
this.#event(sc, 'action.allowed', { action: 'fire.done', transaction: result.transaction ?? null, job, grants: out.grants, grantsLeft: out.grantsLeft }, name);
|
||||
});
|
||||
} catch (e) {
|
||||
const code = e instanceof BusError ? e.code : 'outcome-unknown';
|
||||
const list = out.grants.map((g) => g.grant).join(', ') || 'none';
|
||||
this.#store.transaction(() => {
|
||||
this.#event(sc, 'action.allowed', { action: 'fire.failed', job, step, code }, name);
|
||||
this.#humanNotice(
|
||||
sc.business,
|
||||
`Fire of ${name}: job ${job} revoked its grants (${list}) but stopped at step ${step} (${code}). The instance is still firing. Run the fire again once ${code} is cleared.`,
|
||||
);
|
||||
});
|
||||
}
|
||||
}
|
||||
// Section 2, the delete's job: step G, then grants.revoked.done and the end message.
|
||||
async #deleteJob(business, agent, job, set, by, request) {
|
||||
const verb = 'lifecycle.delete';
|
||||
let out;
|
||||
try {
|
||||
out = await this.#stepG(business, agent, set, { job, reason: `lifecycle.delete ${request}`, by, flag: { delete: true } });
|
||||
} catch (e) {
|
||||
this.#store.transaction(() => this.#grantRefused(business, agent, verb, job, e.grantRefusal, by));
|
||||
return;
|
||||
}
|
||||
const left = out.grantsLeft.length ? ` Lines still left: ${out.grantsLeft.map((l) => `${l.tag} on ${l.host}`).join(', ')}.` : '';
|
||||
this.#store.transaction(() => {
|
||||
this.#event({ business, role: null, run: null }, 'action.allowed', { action: 'grants.revoked.done', job, agent, verb, grants: out.grants, grantsLeft: out.grantsLeft }, agent);
|
||||
this.#humanNotice(
|
||||
business,
|
||||
`Delete of ${agent}: job ${job} is done. Revoked: ${out.grants.map((g) => g.grant).join(', ') || 'nothing'}.${left} Raise the delete again.`,
|
||||
);
|
||||
});
|
||||
}
|
||||
// Section 4: a refused step G, as grant evidence and one message to the human. The caller owns the
|
||||
// transaction.
|
||||
#grantRefused(business, agent, verb, job, { grant, reason }, by) {
|
||||
this.#grantEvent(business, { action: 'grant.revoke.refused', job, agent, grant, reason, code: 'grant-revoke-failed', verb, by }, agent);
|
||||
const fire = verb === 'instance.fire';
|
||||
const what = `${fire ? 'Fire' : 'Delete'} of ${agent} stopped (job ${job}, grant-revoke-failed)`;
|
||||
const again = fire ? 'Run the fire again' : 'Raise the delete again';
|
||||
const host = grant?.startsWith('host:') ? grant.slice('host:'.length) : null;
|
||||
this.#humanNotice(
|
||||
business,
|
||||
reason === 'internal'
|
||||
? `${what}: step G failed at ${grant} (internal). ${again}. It goes on from what's left.`
|
||||
: host
|
||||
? `${what}: its grant on ${host} couldn't be revoked (${reason}). ${again} once ${host} is reachable. If ${host} won't come back, run \`mosaic hosts revoke ${agent} ${host} --reason R\` first: the grant record goes, the line is named as left, and the next ${fire ? 'fire' : 'delete'} completes and names that line.`
|
||||
: `${what}: its Gitea grant couldn't be revoked (${reason}). ${again} once Gitea is back. To clear it by hand, run \`mosaic grants sweep gitea\` once Gitea is back, or \`mosaic grants revoke ${agent} gitea --reason R\`.`,
|
||||
);
|
||||
}
|
||||
// Section 2: the agent's latest delete job end, if it refused on a grant still in the set.
|
||||
#deleteRefused(business, agent, labels) {
|
||||
const e = this.#store.get(
|
||||
"SELECT body FROM events WHERE business=? AND kind='action.allowed' AND json_extract(body,'$.action') IN ('grants.revoked.done','grant.revoke.refused','grants.job.interrupted') AND json_extract(body,'$.verb')='lifecycle.delete' AND json_extract(body,'$.agent')=? ORDER BY seq DESC LIMIT 1",
|
||||
business,
|
||||
agent,
|
||||
);
|
||||
const b = e && JSON.parse(e.body);
|
||||
return b?.action === 'grant.revoke.refused' && labels.includes(b.grant) ? b : null;
|
||||
}
|
||||
// Row 281: a step G message to the human's inbox, from the bus, as #grantNotice sends one. The caller
|
||||
// owns the transaction.
|
||||
#humanNotice(business, body) {
|
||||
const mid = randomUUID();
|
||||
this.#store.run(
|
||||
'INSERT INTO messages(id,at,business,from_role,from_run,to_role,class,in_reply_to,decision,corrects,body) VALUES(?,?,?,?,?,?,?,?,?,?,?)',
|
||||
mid,
|
||||
this.#now(),
|
||||
business,
|
||||
NOTICE_ROLE,
|
||||
NOTICE_RUN,
|
||||
'human',
|
||||
'RESULT',
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
body,
|
||||
);
|
||||
this.#route(business, mid, 'human', null);
|
||||
}
|
||||
// Section 3d, 3e: a grant raise refuses for an agent a fire has stopped or delisted, by the writer's
|
||||
// own rule, and while a step G job holds the agent. A rehire lists the name again.
|
||||
#grantableAgent(business, agent) {
|
||||
if (this.#revoking.has(business + '/' + agent)) fail('grant-revoke-running');
|
||||
const b = this.#business(business);
|
||||
if (this.#firing(business, agent) || (Array.isArray(b.launch?.instances) && !b.launch.instances.includes(agent) && b.launch.by !== agent))
|
||||
fail('instance-fired');
|
||||
}
|
||||
// r6 X3 sweep (B4): the tags this business recorded on the host, from grant.step and grant.revoked,
|
||||
// less those of live grants.
|
||||
@@ -1623,6 +2057,7 @@ export class Broker {
|
||||
// key is new, and the chief can't shape it.
|
||||
#hostGrantBinding(business, g) {
|
||||
const definition = this.#hostDefinition(business, g.agent);
|
||||
this.#grantableAgent(business, g.agent);
|
||||
const h = this.#hostRecord(business, g.host);
|
||||
if (!h) fail('host-unknown');
|
||||
if (h.state !== 'ready') fail('host-not-ready');
|
||||
@@ -1688,6 +2123,7 @@ export class Broker {
|
||||
// can't shape the block: every line comes from the business, the slot and the store.
|
||||
#giteaGrantBinding(business, g) {
|
||||
const { b, role, definition } = this.#giteaRole(business, g.agent);
|
||||
this.#grantableAgent(business, g.agent);
|
||||
const targets = this.#giteaTargets(b, role, g.target.projects);
|
||||
const meta = this.#giteaAdminMeta(business, role.identity.gitea.url);
|
||||
const kind = this.#store.get('SELECT 1 FROM gitea_grants WHERE business=? AND agent=?', business, g.agent)
|
||||
@@ -1825,11 +2261,12 @@ export class Broker {
|
||||
}
|
||||
return Promise.resolve(code);
|
||||
}
|
||||
return this.#track(this.#runGitea(business, decision, start.g, start.signal));
|
||||
return this.#track(this.#runGitea(business, decision, start.g, start.signal), business + '/' + start.g.agent);
|
||||
}
|
||||
#track(promise) {
|
||||
this.#giteaRuns.add(promise);
|
||||
promise.finally(() => this.#giteaRuns.delete(promise));
|
||||
// `key` is business/agent for one agent's work, null for business-wide work.
|
||||
#track(promise, key = null) {
|
||||
this.#giteaRuns.set(promise, key);
|
||||
promise.finally(() => this.#giteaRuns.delete(promise)).catch(() => {});
|
||||
return promise;
|
||||
}
|
||||
// Trusted API, as runHostGrant is, and no verb reaches it: one decision's executor, to its end.
|
||||
@@ -1838,7 +2275,12 @@ export class Broker {
|
||||
}
|
||||
// Trusted API for the tests: resolves once every Gitea executor and remote undo in flight has ended.
|
||||
async giteaIdle() {
|
||||
while (this.#giteaRuns.size) await Promise.allSettled([...this.#giteaRuns]);
|
||||
while (this.#giteaRuns.size) await Promise.allSettled([...this.#giteaRuns.keys()]);
|
||||
}
|
||||
// Row 281 (d6 section 8): resolves once every step G job, Gitea executor and host executor has ended.
|
||||
async grantsIdle() {
|
||||
while (this.#grantJobs.size || this.#giteaRuns.size || this.#hostRuns.size)
|
||||
await Promise.allSettled([...this.#grantJobs, ...this.#giteaRuns.keys(), ...[...this.#hostRuns.values()].flatMap((s) => [...s])]);
|
||||
}
|
||||
// Section 4 steps 0 (the remote half) to 8. Never throws: the outcome is in the store and the
|
||||
// notices, and the return value is 'applied', 'settle-pending' or the code grant.failed records.
|
||||
@@ -2288,7 +2730,7 @@ export class Broker {
|
||||
// Section 6 step 2, in one transaction: the record goes and grant.revoked records what step 3 deletes.
|
||||
// That is the record's token, and both tokens of a rotation whose commit never settled (settle-pending,
|
||||
// or grant-token-unknown, whose own unknown id stays for Jason).
|
||||
#giteaRevokeLocal(business, agent, reason, by, fire) {
|
||||
#giteaRevokeLocal(business, agent, reason, by, fire, extra = {}) {
|
||||
const revoke = randomUUID();
|
||||
return this.#store.transaction(() => {
|
||||
const u = this.#store.get('SELECT user, id FROM gitea_users WHERE business=? AND agent=?', business, agent);
|
||||
@@ -2310,7 +2752,7 @@ export class Broker {
|
||||
this.#store.run('DELETE FROM gitea_grants WHERE business=? AND agent=?', business, agent);
|
||||
const body = {
|
||||
action: 'grant.revoked', method: 'gitea', revoke, agent, user: u?.user ?? null, userId: u?.id ?? null,
|
||||
tokens: [...tokens], fire, granted: Boolean(live), reason, by,
|
||||
tokens: [...tokens], fire, granted: Boolean(live), reason, by, ...extra,
|
||||
};
|
||||
this.#grantEvent(business, body, agent);
|
||||
return body;
|
||||
@@ -2409,15 +2851,18 @@ export class Broker {
|
||||
const log = this.#giteaLog(business, sweep);
|
||||
const left = this.#giteaLeft(business);
|
||||
const closed = [];
|
||||
// Row 281 (d6 section 3e): an agent a step G job holds is skipped, and named as running.
|
||||
const held = (agent) => this.#revoking.has(business + '/' + agent);
|
||||
const running = [...this.#revoking.keys()].filter((k) => k.startsWith(business + '/')).map((k) => k.slice(business.length + 1));
|
||||
for (const m of this.#giteaOpenRevokes(business)) {
|
||||
if (this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke)) continue;
|
||||
if (this.#giteaRevokeUndo.some((u) => u.m.revoke === m.revoke) || held(m.agent)) continue;
|
||||
const c = await this.#giteaCloseRevoke(business, m);
|
||||
closed.push(m.revoke);
|
||||
for (const id of c.serviceLeft) if (!left.some((e) => e.agent === m.agent && e.id === id)) left.push({ agent: m.agent, user: m.user, id, fresh: true });
|
||||
}
|
||||
const settled = [];
|
||||
for (const o of this.#giteaGrant.openGrants(this.#store, business)) {
|
||||
if (o.step?.step !== 'commit' || o.step.previousTokenId === null || o.revokedAfter) continue;
|
||||
if (o.step?.step !== 'commit' || o.step.previousTokenId === null || o.revokedAfter || held(o.agent)) continue;
|
||||
let signal;
|
||||
try {
|
||||
signal = this.#giteaKeys.claim(business, o.agent);
|
||||
@@ -2438,31 +2883,33 @@ export class Broker {
|
||||
const still = [];
|
||||
const skipped = [];
|
||||
for (const e of left) {
|
||||
if (e.fresh) {
|
||||
if (e.fresh || held(e.agent)) {
|
||||
still.push({ agent: e.agent, user: e.user, id: e.id });
|
||||
continue;
|
||||
}
|
||||
let ok = false;
|
||||
if (admin && e.user) {
|
||||
if (e.id === 'tokens' || e.id === 'login') {
|
||||
// A fire retry checks the user's id first, as the revoke close does: the gitea_users row
|
||||
// survives a fire, and a user remade under the same name is not the one that was fired.
|
||||
const u = this.#store.get('SELECT id FROM gitea_users WHERE business=? AND agent=? AND user=?', business, e.agent, e.user);
|
||||
const r = await admin.api.userGet(e.user);
|
||||
const reason = r.status === 404 ? 'gitea-user-gone' : r.status === 200 && (!u || r.json?.id !== u.id) ? 'gitea-user-foreign' : null;
|
||||
if (reason) {
|
||||
skipped.push({ agent: e.agent, user: e.user, id: e.id, reason });
|
||||
continue;
|
||||
}
|
||||
if (r.status === 200) ok = !(await this.#giteaFire(admin.api, e.user)).includes(e.id);
|
||||
} else ok = await this.#giteaDelete(admin.api, e.user, e.id).catch(() => false);
|
||||
}
|
||||
if (!ok) still.push({ agent: e.agent, user: e.user, id: e.id });
|
||||
const r = await this.#giteaRetry(business, admin, e);
|
||||
if (r.skipped) skipped.push({ agent: e.agent, user: e.user, id: e.id, reason: r.skipped });
|
||||
else if (!r.ok) still.push({ agent: e.agent, user: e.user, id: e.id });
|
||||
}
|
||||
this.#store.transaction(() =>
|
||||
this.#grantEvent(business, { action: 'grant.swept', method: 'gitea', sweep, left: still, skipped, settled, closed, by }),
|
||||
);
|
||||
return { left: still, skipped, settled, closed };
|
||||
return { left: still, skipped, settled, closed, ...(running.length ? { running } : {}) };
|
||||
}
|
||||
// One left entry's retry: a fire step checks the user's id first, a token goes by id. Returns
|
||||
// { ok } or { skipped: reason }. The sweep and a step G job share it (row 281).
|
||||
async #giteaRetry(business, admin, e) {
|
||||
if (!admin || !e.user) return { ok: false };
|
||||
if (e.id === 'tokens' || e.id === 'login') {
|
||||
// A fire retry checks the user's id first, as the revoke close does: the gitea_users row
|
||||
// survives a fire, and a user remade under the same name is not the one that was fired.
|
||||
const u = this.#store.get('SELECT id FROM gitea_users WHERE business=? AND agent=? AND user=?', business, e.agent, e.user);
|
||||
const r = await admin.api.userGet(e.user);
|
||||
const reason = r.status === 404 ? 'gitea-user-gone' : r.status === 200 && (!u || r.json?.id !== u.id) ? 'gitea-user-foreign' : null;
|
||||
if (reason) return { skipped: reason };
|
||||
return { ok: r.status === 200 && !(await this.#giteaFire(admin.api, e.user)).includes(e.id) };
|
||||
}
|
||||
return { ok: await this.#giteaDelete(admin.api, e.user, e.id).catch(() => false) };
|
||||
}
|
||||
// `mosaic grants status gitea`.
|
||||
#giteaStatus(business) {
|
||||
@@ -2570,7 +3017,7 @@ export class Broker {
|
||||
keys(a, ['decision'], ['decision']);
|
||||
identifier(a.decision);
|
||||
const { g, signal } = this.#giteaBegin(s.business, a.decision);
|
||||
this.#track(this.#runGitea(s.business, a.decision, g, signal));
|
||||
this.#track(this.#runGitea(s.business, a.decision, g, signal), s.business + '/' + g.agent);
|
||||
return { decision: a.decision, agent: g.agent, started: true };
|
||||
}
|
||||
if (a.service !== 'gitea') fail('invalid-request');
|
||||
@@ -2587,6 +3034,8 @@ export class Broker {
|
||||
if (typeof a.agent !== 'string' || !ID_PATTERN.test(a.agent)) fail('invalid-request');
|
||||
string(a.reason, 500);
|
||||
if (!Object.hasOwn(this.#business(s.business).roles, a.agent)) fail('credential-unknown');
|
||||
// Row 281 (d6 section 3e): no manual revoke while a step G job holds the agent.
|
||||
if (this.#revoking.has(s.business + '/' + a.agent)) fail('grant-revoke-running');
|
||||
// Step 1: memory, the abort of a use or an executor in flight, the file.
|
||||
try {
|
||||
this.#giteaKeys.revoke(s.business, a.agent);
|
||||
@@ -2595,7 +3044,7 @@ export class Broker {
|
||||
else throw e;
|
||||
}
|
||||
const m = this.#giteaRevokeLocal(s.business, a.agent, a.reason, by, false);
|
||||
const closed = await this.#track(this.#giteaCloseRevoke(s.business, m));
|
||||
const closed = await this.#track(this.#giteaCloseRevoke(s.business, m), s.business + '/' + a.agent);
|
||||
return { agent: a.agent, revoke: m.revoke, granted: m.granted, serviceLeft: closed.serviceLeft };
|
||||
}
|
||||
return await this.#giteaAdmin(s.business, verb, a, by);
|
||||
@@ -2714,13 +3163,17 @@ export class Broker {
|
||||
host,
|
||||
);
|
||||
const left = swept ? JSON.parse(swept.body).left : [];
|
||||
// Row 281 (section 5): a fire's or a delete's close with the line removed clears its tag.
|
||||
for (const e of this.#store.all(
|
||||
`SELECT json_extract(body,'$.hostLineLeft') AS tag FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND json_extract(body,'$.hostLineLeft') IS NOT NULL AND seq > ? ORDER BY seq`,
|
||||
`SELECT json_extract(body,'$.hostLineLeft') AS tag, json_extract(body,'$.tag') AS removed FROM events WHERE business=? AND ${GRANT_EVIDENCE} AND json_extract(body,'$.host')=? AND (json_extract(body,'$.hostLineLeft') IS NOT NULL OR (json_extract(body,'$.action')='grant.revoke.closed' AND json_extract(body,'$.hostLine')='removed' AND (json_extract(body,'$.fire')=1 OR json_extract(body,'$.delete')=1))) AND seq > ? ORDER BY seq`,
|
||||
business,
|
||||
host,
|
||||
swept?.seq ?? 0,
|
||||
))
|
||||
if (!left.includes(e.tag)) left.push(e.tag);
|
||||
)) {
|
||||
if (e.tag !== null) {
|
||||
if (!left.includes(e.tag)) left.push(e.tag);
|
||||
} else if (left.includes(e.removed)) left.splice(left.indexOf(e.removed), 1);
|
||||
}
|
||||
return left;
|
||||
}
|
||||
#hardWait(s, action, target) {
|
||||
@@ -2987,6 +3440,15 @@ export class Broker {
|
||||
);
|
||||
return result;
|
||||
}
|
||||
// Row 281 (d6, C1): #consumeAuthority less the holder check, for a fire's step C in its job. The chief
|
||||
// held its claim when the verb ran #checkAuthority, and its run may have ended since, as a landing's
|
||||
// record allows (row 137). Caller owns the transaction.
|
||||
#recordAuthority(s, action, context = {}, detail = {}) {
|
||||
const result = this.#authorityFor(s, action, context);
|
||||
if (result.decision && this.#spent(s.business, result.decision)) fail('decision-consumed');
|
||||
this.#event(s, 'action.allowed', { ...detail, action, ...result, target: context.target ?? null }, context.target ?? null);
|
||||
return result;
|
||||
}
|
||||
// Trusted S3 handler calls inside its own operation; not a generic socket action executor.
|
||||
// Row 93 (#1587): task.close on a task with a recorded push also needs
|
||||
// context.verdict, checked by #closeVerdict; a refusal there rolls the action.allowed back.
|
||||
@@ -3366,6 +3828,32 @@ export class Broker {
|
||||
if (this.#openRequests(record.business, r.instance).some((o) => o.request !== r.request && o.action !== 'lifecycle.end' &&
|
||||
(o.state === 'taken' || !this.#expired(o)))) fail('instance-starting');
|
||||
if (!this.#organization) fail('not-configured');
|
||||
// Row 281 (#1786, d6 section 2, sage-2575): a delete of an instance with a grant takes two requests.
|
||||
// This one starts step G as a job and refuses at once, and the human raises the delete again when
|
||||
// the inbox says the job is done. With no grant, the delete goes on as before.
|
||||
if (this.#revoking.has(record.business + '/' + r.instance)) fail('grants-revoking');
|
||||
const set = this.#store.transaction(() => this.#grantSet(record.business, r.instance));
|
||||
if (set.length) {
|
||||
const job = randomUUID();
|
||||
const labels = set.map((g) => g.label);
|
||||
const refused = this.#deleteRefused(record.business, r.instance, labels);
|
||||
this.#store.transaction(() => {
|
||||
this.#event(
|
||||
{ business: record.business, role: null, run: null },
|
||||
'action.allowed',
|
||||
{ action: 'grants.revoking', job, agent: r.instance, verb: 'lifecycle.delete', request: r.request, grants: labels },
|
||||
r.instance,
|
||||
);
|
||||
this.#humanNotice(
|
||||
record.business,
|
||||
refused
|
||||
? `Delete of ${r.instance} refused: job ${refused.job} couldn't revoke ${refused.grant} (${refused.reason}). Retrying now as job ${job} for: ${labels.join(', ')}. Raise the delete again when the inbox says it's done, or clear ${refused.grant} by hand first.`
|
||||
: `Delete of ${r.instance} is waiting on its grants. Revoking started (job ${job}) for: ${labels.join(', ')}. Raise the delete again when the inbox says the revoke is done.`,
|
||||
);
|
||||
});
|
||||
this.#startGrantJob(record.business, r.instance, 'lifecycle.delete', () => this.#deleteJob(record.business, r.instance, job, set, r.by, r.request));
|
||||
fail(refused ? 'grant-revoke-failed' : 'grants-revoking');
|
||||
}
|
||||
let out;
|
||||
try {
|
||||
out = this.#organization(record.business, 'instances.set', { revision: r.revision, instance: r.instance, op: 'delete' }, r.by);
|
||||
@@ -3757,7 +4245,7 @@ export class Broker {
|
||||
this.#store.get('SELECT 1 FROM gitea_users WHERE business=? AND agent=?', s.business, a.agent)
|
||||
) {
|
||||
const m = this.#giteaRevokeLocal(s.business, a.agent, a.reason, s.tester ? 'tester' : s.human, verb === 'credentials.delete');
|
||||
setImmediate(() => this.#track(this.#giteaCloseRevoke(s.business, m)).catch(() => {}));
|
||||
setImmediate(() => this.#track(this.#giteaCloseRevoke(s.business, m), s.business + '/' + a.agent).catch(() => {}));
|
||||
return { ...result, giteaRevoke: m.revoke };
|
||||
}
|
||||
return result;
|
||||
@@ -3805,6 +4293,8 @@ export class Broker {
|
||||
// The last listed name refuses before step A stops anything, as the writer would at step B.
|
||||
const listed = b.launch?.instances ?? [];
|
||||
if (listed.length === 1 && listed[0] === args.name) fail('invalid-request');
|
||||
// Row 281 (d6 section 3e): one step G per agent. A repeat while its job runs refuses.
|
||||
if (this.#revoking.has(s.business + '/' + args.name)) fail('fire-in-progress');
|
||||
// webby-0094 B1: the same checks against the file. A keep this copy hasn't loaded (a reload the
|
||||
// credentials refused) would otherwise pass here, and step B's refusal would leave I firing.
|
||||
write('instance.fireCheck');
|
||||
@@ -3827,8 +4317,21 @@ export class Broker {
|
||||
if (request)
|
||||
this.#event(s, 'action.allowed', { action: 'lifecycle.end', request, via: 'chief', by: s.role, runs, reason: args.reason, kill: true }, args.name);
|
||||
this.#event(s, 'action.allowed', { action: 'fire.stopped', request }, args.name);
|
||||
return { request, runs, cancelled, taken };
|
||||
// Row 281 (#1786, d6 section 2): the grant set, read after fire.stopped, so a grant raise either
|
||||
// committed before this and is in the set, or refuses instance-fired (3d).
|
||||
const set = this.#grantSet(s.business, args.name);
|
||||
if (!set.length) return { request, runs, cancelled, taken };
|
||||
const job = randomUUID();
|
||||
this.#event(s, 'action.allowed', { action: 'fire.revoking', job, grants: set.map((g) => g.label) }, args.name);
|
||||
return { request, runs, cancelled, taken, job, set };
|
||||
});
|
||||
// A non-empty set: step G, B and C run as a job, and the verb answers now with no transaction.
|
||||
if (stopped.job) {
|
||||
const { job, set, ...rest } = stopped;
|
||||
const sc = { business: s.business, role: s.role, run: s.run, thread: s.thread, human: null };
|
||||
this.#startGrantJob(s.business, args.name, verb, () => this.#fireJob(sc, args, authority, write, job, set));
|
||||
return { ...rest, job, revoking: set.map((g) => g.label), done: false };
|
||||
}
|
||||
// Step B: the delist. An instance already delisted counts as done.
|
||||
const result = write('instance.fire');
|
||||
// Step C: log the fire and record it done. A failure at B or C leaves I firing until a repeat.
|
||||
|
||||
@@ -0,0 +1,954 @@
|
||||
// Row 281 (#1786, design d6 section 8): a fire or a delete revokes the instance's host and Gitea grants
|
||||
// (step G), as a job after the verb answers. T2-T13. The broker runs bare, as chief-of-staff.test.mjs
|
||||
// runs it, with a fake organization writer, a fake installer in place of the host's remote steps, and
|
||||
// the fake Gitea from fake-gitea-api.mjs on 127.0.0.1. Every token is a fake the fake made; no test
|
||||
// reaches git.mosaicstack.dev or any provider, and none creates a real account or token (Q188). Host
|
||||
// keys are real local ssh-keygen pairs in the scratch store, and no ssh connection is made. Decisions a
|
||||
// test needs outside the human's yes go in by SQL as fixtures.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { Broker, GRANT_EVIDENCE, registrySha256 } from '../src/broker.mjs';
|
||||
import { Credentials } from '../src/credentials.mjs';
|
||||
import { adminSha256, definitionSha256 as giteaDefinitionSha256, giteaGrant, giteaKeys, identitySha256 } from '../src/gitea-grant.mjs';
|
||||
import { definitionSha256 as hostDefinitionSha256, grantKeys, grantTag, hostGrant } from '../src/host-grant.mjs';
|
||||
import { Store } from '../src/store.mjs';
|
||||
import { fakeGiteaApi } from './fake-gitea-api.mjs';
|
||||
import { grantDecision, hostRecord } from './host-fixtures.mjs';
|
||||
|
||||
const AT = '2026-10-22T00:00:00.000Z';
|
||||
const REV = 'b'.repeat(64);
|
||||
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
const dayFrom = (n) => new Date(Date.now() + n * 86400000).toISOString().slice(0, 10);
|
||||
const sha256 = (s) => createHash('sha256').update(s).digest('hex');
|
||||
const options = [{ key: 'yes', text: 'Yes' }, { key: 'no', text: 'No' }];
|
||||
const none = { authority: { withinRole: ['message.send'], crossRole: [] } };
|
||||
const chiefAuthority = JSON.parse(readFileSync(fileURLToPath(new URL('../../../roles/chief-of-staff.json', import.meta.url)), 'utf8')).authority;
|
||||
const ENTRY = { address: '127.0.0.1', port: 2222, user: 'deploy', hostKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl'] };
|
||||
const HOSTS = ['web1', 'web2', 'web3'];
|
||||
const GENERATION = '0b6e7f6e-4c1d-4a8e-9a51-6d2f3c4b5a69';
|
||||
async function until(fn, ms = 10000) {
|
||||
const end = Date.now() + ms;
|
||||
while (!fn()) {
|
||||
if (Date.now() > end) throw new Error('timed out');
|
||||
await sleep(10);
|
||||
}
|
||||
}
|
||||
const rejects = (fn, code) => assert.rejects(async () => fn(), (e) => (assert.equal(e.code, code), true));
|
||||
const code = (fn) => {
|
||||
try {
|
||||
fn();
|
||||
} catch (e) {
|
||||
return e.code;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
function makeBusiness(url) {
|
||||
const gitea = (name, user) => ({ name, email: `${user}@example.test`, gitea: { url, user } });
|
||||
return {
|
||||
id: 'demo',
|
||||
human: 'jason',
|
||||
arbiters: { technical: 'pm', delivery: 'pm' },
|
||||
roles: {
|
||||
chief: { definition: 'chief-of-staff', authority: chiefAuthority },
|
||||
pm: { definition: 'pm', ...none },
|
||||
coder: { definition: 'coder', ...none, identity: gitea('Coder', 'demo-coder'), publish: ['app'] },
|
||||
helper: { definition: 'coder', ...none, identity: gitea('Helper', 'demo-helper'), publish: ['app'] },
|
||||
ops: { definition: 'ops', ...none },
|
||||
},
|
||||
definitions: {
|
||||
'chief-of-staff': { roleVersion: 2, credentials: [] },
|
||||
pm: { roleVersion: 2, credentials: [] },
|
||||
coder: { roleVersion: 2, credentials: [{ service: 'gitea' }, { service: 'host' }] },
|
||||
ops: { roleVersion: 2, credentials: [{ service: 'host' }] },
|
||||
},
|
||||
projects: { app: { remote: `${url}/acme/app` } },
|
||||
launch: { by: 'chief', instances: ['pm', 'coder', 'helper', 'ops'] },
|
||||
vars: { 'bus.tester': true },
|
||||
};
|
||||
}
|
||||
|
||||
async function setup(t) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'g281-'));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
const gitea = await fakeGiteaApi(t, join(dir, 'tls'));
|
||||
gitea.addUser('acme');
|
||||
gitea.addRepo('acme', 'app');
|
||||
const configDir = join(dir, 'cfg');
|
||||
mkdirSync(configDir, { mode: 0o700 });
|
||||
const storeRoot = join(configDir, 'secrets', 'agents');
|
||||
const runRoot = join(configDir, 'secrets', 'run');
|
||||
mkdirSync(join(dir, 'data'), { mode: 0o700 });
|
||||
const store = new Store(join(dir, 'data'));
|
||||
const credentials = new Credentials({ references: {}, storeRoot, dataRoot: join(dir, 'data') });
|
||||
t.after(() => {
|
||||
// A test that fails while a host step is held must not leave pause() looping after it ends.
|
||||
held.clear();
|
||||
failing.clear();
|
||||
credentials.close();
|
||||
store.close();
|
||||
});
|
||||
const doc = makeBusiness(gitea.url);
|
||||
const grantApi = giteaGrant({ configDir, ca: gitea.ca, timeoutMs: 5000 });
|
||||
const meta = { url: gitea.url, account: 'mosaic-grant', rotateBy: dayFrom(60), setAt: AT };
|
||||
grantApi.writeAdmin('demo', gitea.adminToken, meta);
|
||||
const gKeys = giteaKeys({ credentials, storeRoot });
|
||||
const hKeys = grantKeys({ credentials, storeRoot });
|
||||
// The fake installer: each host's lines as a set of tags. `held` keeps a host's add or remove from
|
||||
// answering until the test deletes the key; `failing` makes every request to a host fail.
|
||||
const lines = Object.fromEntries(HOSTS.map((h) => [h, new Set()]));
|
||||
const sent = [];
|
||||
const held = new Set();
|
||||
const failing = new Set();
|
||||
const pause = async (key, signal) => {
|
||||
while (held.has(key)) {
|
||||
if (signal?.aborted) throw new Error('aborted');
|
||||
await sleep(5);
|
||||
}
|
||||
if (signal?.aborted) throw new Error('aborted');
|
||||
};
|
||||
const installer = {
|
||||
...hostGrant({ configDir, runRoot }),
|
||||
async install(business, host, record, tag, publicLine, signal) {
|
||||
sent.push(`add ${host} ${tag}`);
|
||||
await pause(host + ':add', signal);
|
||||
if (failing.has(host)) throw new Error('host-failed');
|
||||
lines[host].add(tag);
|
||||
},
|
||||
async verify(host, record, key, signal) {
|
||||
if (signal?.aborted) throw new Error('aborted');
|
||||
if (failing.has(host)) throw new Error('host-failed');
|
||||
},
|
||||
async remove(business, host, record, tag) {
|
||||
sent.push(`remove ${host} ${tag}`);
|
||||
await pause(host + ':remove');
|
||||
if (failing.has(host)) throw new Error('host-failed');
|
||||
lines[host].delete(tag);
|
||||
},
|
||||
};
|
||||
// The fake organization writer. A fire delists the name and a delete drops the role, each with a
|
||||
// reload, as the runtime reloads after a commit. `fireError` makes the fire's write refuse.
|
||||
const org = { log: [], fireError: null };
|
||||
const sw = { state: { version: 1, halted: false, generation: GENERATION } };
|
||||
let broker;
|
||||
const organization = (business, op, args) => {
|
||||
org.log.push([op, args?.name ?? args?.instance ?? null]);
|
||||
if (op === 'instance.fire') {
|
||||
if (org.fireError) throw org.fireError;
|
||||
doc.launch.instances = doc.launch.instances.filter((n) => n !== args.name);
|
||||
broker.reloadBusiness('demo', doc);
|
||||
return { revision: REV, transaction: 't-' + org.log.length };
|
||||
}
|
||||
if (op === 'instances.set') {
|
||||
delete doc.roles[args.instance];
|
||||
doc.launch.instances = doc.launch.instances.filter((n) => n !== args.instance);
|
||||
broker.reloadBusiness('demo', doc);
|
||||
return { revision: REV, transaction: 't-' + org.log.length };
|
||||
}
|
||||
return { revision: REV };
|
||||
};
|
||||
const make = () =>
|
||||
new Broker({
|
||||
store,
|
||||
businesses: { demo: structuredClone(doc) },
|
||||
secretCheck: (value) => credentials.assertClean(value),
|
||||
organization,
|
||||
processing: { read: () => sw.state },
|
||||
hostGrant: installer,
|
||||
hostKeys: hKeys,
|
||||
giteaGrant: grantApi,
|
||||
giteaKeys: gKeys,
|
||||
});
|
||||
broker = make();
|
||||
for (const host of HOSTS) hostRecord(store, { host, entry: ENTRY });
|
||||
const caps = {};
|
||||
for (const role of ['chief', 'pm']) {
|
||||
caps[role] = broker.bindLaunch({ business: 'demo', role, run: role + '-run', harness: 'pi', address: role });
|
||||
broker.request(caps[role], { verb: 'role.claim' });
|
||||
}
|
||||
caps.human = broker.bindHuman({ business: 'demo', human: 'jason', via: 'cli', outsideAgent: true });
|
||||
caps.tester = broker.bindTester({ business: 'demo' });
|
||||
const call = (who, verb, args = {}) => broker.request(caps[who], { verb, args });
|
||||
const grantEvents = () =>
|
||||
store.all(`SELECT body FROM events WHERE business='demo' AND ${GRANT_EVIDENCE} ORDER BY seq`).map((e) => JSON.parse(e.body));
|
||||
const ended = (decision) => grantEvents().findLast((e) => e.grantDecision === decision && ['grant.applied', 'grant.failed'].includes(e.action));
|
||||
const allowed = (action) =>
|
||||
store
|
||||
.all("SELECT actor_role, subject, body FROM events WHERE business='demo' AND kind='action.allowed' ORDER BY seq")
|
||||
.map((e) => ({ ...e, body: JSON.parse(e.body) }))
|
||||
.filter((e) => e.body.action === action);
|
||||
// The bus's step G messages to the human: no decision, so no grant notice.
|
||||
const inbox = () =>
|
||||
store.all("SELECT body FROM messages WHERE business='demo' AND from_role='_bus' AND to_role='human' AND decision IS NULL ORDER BY rowid").map((r) => r.body);
|
||||
const raiseHost = (agent, host, commands = [['uptime']]) =>
|
||||
call('chief', 'decision.raise', {
|
||||
action: 'credential.mint', question: `Grant ${agent} ${host}?`, options, recommendation: 'yes', blocking: false,
|
||||
grant: { method: 'host', agent, host, commands, rotateBy: dayFrom(30) },
|
||||
});
|
||||
const raiseGitea = (agent) =>
|
||||
call('chief', 'decision.raise', {
|
||||
action: 'credential.mint', question: `Grant ${agent} Gitea?`, options, recommendation: 'yes', blocking: false,
|
||||
grant: { method: 'gitea', agent, target: { projects: ['app'] }, rotateBy: dayFrom(30) },
|
||||
});
|
||||
const approve = async (d) => {
|
||||
call('human', 'decision.resolve', { id: d.id, choice: 'yes' });
|
||||
await until(() => ended(d.id));
|
||||
await broker.grantsIdle();
|
||||
return ended(d.id);
|
||||
};
|
||||
const approveHost = async (agent, host) => {
|
||||
const end = await approve(await raiseHost(agent, host));
|
||||
assert.equal(end.action, 'grant.applied', `${agent} on ${host}`);
|
||||
return end;
|
||||
};
|
||||
const approveGitea = async (agent) => {
|
||||
const end = await approve(await raiseGitea(agent));
|
||||
assert.equal(end.action, 'grant.applied', `${agent} gitea`);
|
||||
return end;
|
||||
};
|
||||
// A host decision approved and never run, with R4's body built here (T6).
|
||||
const fixtureHost = (agent, host, commands = [['uptime']]) =>
|
||||
grantDecision(store, {
|
||||
grant: {
|
||||
method: 'host', agent, host, commands, commandsSha256: sha256(JSON.stringify(commands)), rotateBy: dayFrom(30),
|
||||
registrySha256: registrySha256({ ...ENTRY, sudo: true }), definitionSha256: hostDefinitionSha256(doc.definitions[doc.roles[agent].definition]),
|
||||
},
|
||||
});
|
||||
// A Gitea decision approved and never run (T6b).
|
||||
const fixtureGitea = (agent) =>
|
||||
grantDecision(store, {
|
||||
grant: {
|
||||
method: 'gitea', agent, target: { projects: ['app'] }, rotateBy: dayFrom(30),
|
||||
definitionSha256: giteaDefinitionSha256(doc.definitions[doc.roles[agent].definition]),
|
||||
identitySha256: identitySha256(doc.roles[agent], doc.projects),
|
||||
adminSha256: adminSha256(grantApi.adminMeta('demo') ?? meta),
|
||||
},
|
||||
});
|
||||
const fire = (name) => call('chief', 'instance.fire', { name, reason: 'the work is done' });
|
||||
// A delete as the launcher's pass runs it: the human's request, the take, then lifecycleDelete. A
|
||||
// refused delete closes the request with its code, as the launcher does.
|
||||
const remove = (instance) => {
|
||||
const { request } = call('human', 'instances.set', { revision: REV, instance, op: 'delete' });
|
||||
assert.equal(broker.lifecycleTake({ business: 'demo', request }).go, true);
|
||||
try {
|
||||
return { request, ...broker.lifecycleDelete({ business: 'demo', request }) };
|
||||
} catch (e) {
|
||||
if (typeof e.code !== 'string') throw e;
|
||||
broker.lifecycleDone({ business: 'demo', request, refusal: e.code });
|
||||
return { request, code: e.code };
|
||||
}
|
||||
};
|
||||
const tag = (agent, host) => grantTag('demo', agent, host);
|
||||
const keyFile = (agent, host) => join(storeRoot, 'demo', agent, 'host.' + host);
|
||||
const tokenFile = (agent) => join(storeRoot, 'demo', agent, 'gitea');
|
||||
const hostRow = (agent, host) => store.get("SELECT * FROM host_grants WHERE business='demo' AND agent=? AND host=?", agent, host);
|
||||
const giteaRow = (agent) => store.get("SELECT * FROM gitea_grants WHERE business='demo' AND agent=?", agent);
|
||||
const state = (agent, service) => credentials.status().find((e) => e.instance === 'demo/' + agent && e.service === service)?.state;
|
||||
const userTokens = (login) => gitea.tokens(login) ?? [];
|
||||
const revoked = (agent, where = () => true) => grantEvents().filter((e) => e.action === 'grant.revoked' && e.agent === agent && where(e));
|
||||
const closed = (revoke) => grantEvents().filter((e) => e.action === 'grant.revoke.closed' && e.revoke === revoke);
|
||||
const noLeak = () => {
|
||||
const text = [
|
||||
...store.all("SELECT body FROM events WHERE business='demo'").map((r) => r.body),
|
||||
...store.all("SELECT body FROM messages WHERE business='demo'").map((r) => r.body),
|
||||
JSON.stringify(gitea.requests),
|
||||
].join('\n');
|
||||
assert.ok(gitea.issued.every((tok) => !text.includes(tok)), 'a token value reached the store or the request log');
|
||||
};
|
||||
return {
|
||||
dir, gitea, store, credentials, doc, org, sw, lines, sent, held, failing, caps, call, make, grantEvents, ended, allowed, inbox,
|
||||
raiseHost, raiseGitea, approve, approveHost, approveGitea, fixtureHost, fixtureGitea, fire, remove, tag, keyFile, tokenFile,
|
||||
hostRow, giteaRow, state, userTokens, revoked, closed, noLeak,
|
||||
get broker() {
|
||||
return broker;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Coder's three grants: web1, web2 and Gitea, each through the human's yes and its executor.
|
||||
async function threeGrants(f) {
|
||||
await f.approveHost('coder', 'web1');
|
||||
await f.approveHost('coder', 'web2');
|
||||
await f.approveGitea('coder');
|
||||
for (const h of ['web1', 'web2']) {
|
||||
assert.ok(f.lines[h].has(f.tag('coder', h)));
|
||||
assert.ok(existsSync(f.keyFile('coder', h)));
|
||||
}
|
||||
assert.equal(f.userTokens('demo-coder').length, 1);
|
||||
assert.ok(existsSync(f.tokenFile('coder')));
|
||||
}
|
||||
|
||||
// Every grant of coder is gone: records, key and token files, host lines, Gitea tokens, login.
|
||||
function allRevoked(f, hosts = ['web1', 'web2']) {
|
||||
for (const h of hosts) {
|
||||
assert.equal(f.hostRow('coder', h), undefined, h);
|
||||
assert.equal(existsSync(f.keyFile('coder', h)), false, h);
|
||||
assert.equal(f.lines[h].has(f.tag('coder', h)), false, h);
|
||||
assert.notEqual(f.state('coder', 'host.' + h), 'valid', h);
|
||||
}
|
||||
assert.equal(f.giteaRow('coder'), undefined);
|
||||
assert.equal(existsSync(f.tokenFile('coder')), false);
|
||||
assert.notEqual(f.state('coder', 'gitea'), 'valid');
|
||||
assert.deepEqual(f.userTokens('demo-coder'), []);
|
||||
assert.equal(f.gitea.users.get('demo-coder').prohibit_login, true);
|
||||
}
|
||||
|
||||
test('row 281 T2: a fire with host grants on two hosts and a Gitea grant answers with a job, then revokes all three', async (t) => {
|
||||
const f = await setup(t);
|
||||
await threeGrants(f);
|
||||
const out = f.fire('coder');
|
||||
assert.equal(out.done, false);
|
||||
assert.match(out.job, UUID);
|
||||
assert.deepEqual(out.revoking, ['host:web1', 'host:web2', 'gitea']);
|
||||
assert.ok(!('transaction' in out));
|
||||
const [revoking, ...more] = f.allowed('fire.revoking');
|
||||
assert.equal(more.length, 0);
|
||||
assert.equal(revoking.body.job, out.job);
|
||||
assert.deepEqual(revoking.body.grants, out.revoking);
|
||||
assert.equal(revoking.actor_role, 'chief');
|
||||
assert.equal(revoking.subject, 'coder');
|
||||
await f.broker.grantsIdle();
|
||||
// Each host as hosts.revoke records it, with fire: true, line first so the close says removed.
|
||||
for (const h of ['web1', 'web2']) {
|
||||
const [r, ...rest] = f.revoked('coder', (e) => e.host === h);
|
||||
assert.equal(rest.length, 0, h);
|
||||
assert.equal(r.fire, true);
|
||||
assert.equal(r.granted, true);
|
||||
assert.equal(r.hostLine, 'pending');
|
||||
assert.equal(r.tag, f.tag('coder', h));
|
||||
const [c] = f.closed(r.revoke);
|
||||
assert.equal(c.hostLine, 'removed');
|
||||
assert.equal(c.fire, true);
|
||||
}
|
||||
const [g, ...rest] = f.revoked('coder', (e) => e.method === 'gitea');
|
||||
assert.equal(rest.length, 0);
|
||||
assert.equal(g.user, 'demo-coder');
|
||||
assert.equal(g.granted, true);
|
||||
assert.deepEqual(f.closed(g.revoke).map((c) => c.serviceLeft), [[]]);
|
||||
allRevoked(f);
|
||||
const [done, ...again] = f.allowed('fire.done');
|
||||
assert.equal(again.length, 0);
|
||||
assert.equal(done.body.job, out.job);
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'host:web2', 'gitea']);
|
||||
assert.deepEqual(done.body.grantsLeft, []);
|
||||
assert.match(done.body.transaction, /^t-/);
|
||||
assert.equal(done.actor_role, 'chief');
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
|
||||
assert.ok(!f.doc.launch.instances.includes('coder'));
|
||||
// No step G message for a job that completed.
|
||||
assert.deepEqual(f.inbox(), []);
|
||||
f.noLeak();
|
||||
});
|
||||
|
||||
test('row 281 T3 (sage-2575): a delete with grants takes two requests: the first starts the job, the next completes', async (t) => {
|
||||
const f = await setup(t);
|
||||
await threeGrants(f);
|
||||
f.held.add('web1:remove');
|
||||
const first = f.remove('coder');
|
||||
assert.equal(first.code, 'grants-revoking');
|
||||
const [start, ...more] = f.allowed('grants.revoking');
|
||||
assert.equal(more.length, 0);
|
||||
assert.match(start.body.job, UUID);
|
||||
assert.deepEqual(start.body.grants, ['host:web1', 'host:web2', 'gitea']);
|
||||
assert.equal(start.body.verb, 'lifecycle.delete');
|
||||
assert.equal(start.body.request, first.request);
|
||||
assert.equal(start.actor_role, null);
|
||||
assert.equal(start.subject, 'coder');
|
||||
const job = start.body.job;
|
||||
const [message] = f.inbox();
|
||||
assert.equal(f.inbox().length, 1);
|
||||
for (const s of [job, 'host:web1', 'host:web2', 'gitea', 'Raise the delete again']) assert.ok(message.includes(s), s);
|
||||
// Held: a second delete refuses grants-revoking, with no second job and no second message.
|
||||
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
|
||||
assert.equal(f.remove('coder').code, 'grants-revoking');
|
||||
assert.equal(f.allowed('grants.revoking').length, 1);
|
||||
assert.equal(f.inbox().length, 1);
|
||||
f.held.delete('web1:remove');
|
||||
await f.broker.grantsIdle();
|
||||
for (const h of ['web1', 'web2']) assert.equal(f.revoked('coder', (e) => e.host === h && e.delete === true).length, 1, h);
|
||||
allRevoked(f);
|
||||
const [done] = f.allowed('grants.revoked.done');
|
||||
assert.equal(done.body.job, job);
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'host:web2', 'gitea']);
|
||||
const end = f.inbox()[1];
|
||||
for (const s of [job, 'host:web1', 'host:web2', 'gitea', 'Raise the delete again']) assert.ok(end.includes(s), s);
|
||||
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
|
||||
// The third request completes as today.
|
||||
const third = f.remove('coder');
|
||||
assert.equal(third.code, undefined);
|
||||
assert.match(third.transaction, /^t-/);
|
||||
assert.equal(third.retry, false);
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instances.set'), [['instances.set', 'coder']]);
|
||||
assert.equal(f.allowed('grants.revoking').length, 1);
|
||||
f.noLeak();
|
||||
});
|
||||
|
||||
test('row 281 T3b (Q2): a delete whose job refuses on host B; the retry refuses grant-revoke-failed and revokes only B and Gitea', async (t) => {
|
||||
const f = await setup(t);
|
||||
await threeGrants(f);
|
||||
f.failing.add('web2');
|
||||
assert.equal(f.remove('coder').code, 'grants-revoking');
|
||||
await f.broker.grantsIdle();
|
||||
const job = f.allowed('grants.revoking')[0].body.job;
|
||||
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
|
||||
assert.equal(more.length, 0);
|
||||
assert.deepEqual(
|
||||
{ job: refused.job, agent: refused.agent, grant: refused.grant, reason: refused.reason, code: refused.code, verb: refused.verb },
|
||||
{ job, agent: 'coder', grant: 'host:web2', reason: 'host-remove-failed', code: 'grant-revoke-failed', verb: 'lifecycle.delete' },
|
||||
);
|
||||
const end = f.inbox()[1];
|
||||
for (const s of [job, 'coder', 'web2', 'host-remove-failed', 'mosaic hosts revoke coder web2']) assert.ok(end.includes(s), s);
|
||||
// Nothing deleted: the writer saw no delete and the role is still there. Host A is revoked, B isn't.
|
||||
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
|
||||
assert.ok(f.doc.roles.coder);
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
assert.ok(f.hostRow('coder', 'web2'));
|
||||
assert.ok(existsSync(f.keyFile('coder', 'web2')));
|
||||
assert.ok(f.giteaRow('coder'));
|
||||
assert.equal(f.allowed('grants.revoked.done').length, 0);
|
||||
// B back: the next delete refuses grant-revoke-failed, names the earlier reason and the new job.
|
||||
f.failing.delete('web2');
|
||||
assert.equal(f.remove('coder').code, 'grant-revoke-failed');
|
||||
const retry = f.allowed('grants.revoking')[1].body;
|
||||
assert.deepEqual(retry.grants, ['host:web2', 'gitea']);
|
||||
const message = f.inbox()[2];
|
||||
for (const s of [job, retry.job, 'host:web2', 'host-remove-failed']) assert.ok(message.includes(s), s);
|
||||
assert.equal(f.org.log.filter(([op]) => op === 'instances.set').length, 0);
|
||||
await f.broker.grantsIdle();
|
||||
const [done] = f.allowed('grants.revoked.done');
|
||||
assert.equal(done.body.job, retry.job);
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web2', 'gitea']);
|
||||
// Host A was sent one remove only, by the first job.
|
||||
assert.equal(f.sent.filter((x) => x === `remove web1 ${f.tag('coder', 'web1')}`).length, 1);
|
||||
allRevoked(f);
|
||||
assert.match(f.remove('coder').transaction, /^t-/);
|
||||
});
|
||||
|
||||
test('row 281 T3b: B cleared by hand with hosts.revoke; the next delete starts a job for Gitea and the leftover line only', async (t) => {
|
||||
const f = await setup(t);
|
||||
await threeGrants(f);
|
||||
f.failing.add('web2');
|
||||
assert.equal(f.remove('coder').code, 'grants-revoking');
|
||||
await f.broker.grantsIdle();
|
||||
const manual = await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web2', reason: 'web2 is gone' });
|
||||
assert.equal(manual.hostLineLeft, f.tag('coder', 'web2'));
|
||||
assert.equal(f.hostRow('coder', 'web2'), undefined);
|
||||
// The refused grant isn't in the set any more, so this is a new start, not a retry.
|
||||
assert.equal(f.remove('coder').code, 'grants-revoking');
|
||||
const second = f.allowed('grants.revoking')[1].body;
|
||||
assert.deepEqual(second.grants, ['gitea', 'line:web2']);
|
||||
await f.broker.grantsIdle();
|
||||
const done = f.allowed('grants.revoked.done')[0].body;
|
||||
assert.equal(done.job, second.job);
|
||||
assert.deepEqual(done.grants.map((x) => x.grant), ['gitea']);
|
||||
assert.deepEqual(done.grantsLeft, [{ host: 'web2', tag: f.tag('coder', 'web2') }]);
|
||||
assert.ok(f.inbox().at(-1).includes(`${f.tag('coder', 'web2')} on web2`));
|
||||
// The line was tried once and listed: it doesn't hold the delete again.
|
||||
assert.match(f.remove('coder').transaction, /^t-/);
|
||||
assert.equal(f.allowed('grants.revoking').length, 2);
|
||||
});
|
||||
|
||||
test('row 281 T3c (sage-2575 condition 4): a delete of an instance with no grant completes in one request', async (t) => {
|
||||
const f = await setup(t);
|
||||
const before = f.store.get("SELECT count(*) AS n FROM events WHERE business='demo'").n;
|
||||
const out = f.remove('helper');
|
||||
assert.equal(out.code, undefined);
|
||||
assert.match(out.transaction, /^t-/);
|
||||
assert.equal(out.retry, false);
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instances.set'), [['instances.set', 'helper']]);
|
||||
assert.equal(f.allowed('grants.revoking').length, 0);
|
||||
assert.deepEqual(f.inbox(), []);
|
||||
// The request, the take and the delete's own records: none from step G.
|
||||
const after = f.store
|
||||
.all("SELECT body FROM events WHERE business='demo' ORDER BY seq")
|
||||
.slice(before)
|
||||
.map((e) => JSON.parse(e.body).action ?? JSON.parse(e.body).operation ?? null);
|
||||
assert.ok(after.every((a) => !String(a).startsWith('grant')), after.join());
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.allowed('grants.revoked.done').length, 0);
|
||||
});
|
||||
|
||||
test('row 281 T4: host B unreachable in a fire: the job refuses, A stays revoked, B and Gitea untouched; a repeat completes', async (t) => {
|
||||
const f = await setup(t);
|
||||
await threeGrants(f);
|
||||
f.failing.add('web2');
|
||||
const out = f.fire('coder');
|
||||
await f.broker.grantsIdle();
|
||||
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
|
||||
assert.equal(more.length, 0);
|
||||
assert.deepEqual(
|
||||
{ job: refused.job, agent: refused.agent, grant: refused.grant, reason: refused.reason, verb: refused.verb },
|
||||
{ job: out.job, agent: 'coder', grant: 'host:web2', reason: 'host-remove-failed', verb: 'instance.fire' },
|
||||
);
|
||||
const [message] = f.inbox();
|
||||
for (const s of [out.job, 'web2', 'host-remove-failed', 'mosaic hosts revoke coder web2', 'Run the fire again']) assert.ok(message.includes(s), s);
|
||||
// The file is unchanged and the instance still firing.
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), []);
|
||||
assert.ok(f.doc.launch.instances.includes('coder'));
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
assert.equal(f.allowed('fire.failed').length, 0);
|
||||
await rejects(() => f.raiseHost('coder', 'web3'), 'instance-fired');
|
||||
// A revoked; B's record, key and line there (line first: nothing local changed on B).
|
||||
assert.equal(f.revoked('coder', (e) => e.host === 'web1').length, 1);
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
assert.equal(f.revoked('coder', (e) => e.host === 'web2').length, 0);
|
||||
assert.ok(f.hostRow('coder', 'web2'));
|
||||
assert.ok(existsSync(f.keyFile('coder', 'web2')));
|
||||
assert.equal(f.state('coder', 'host.web2'), 'valid');
|
||||
// Gitea comes after B: untouched.
|
||||
assert.equal(f.revoked('coder', (e) => e.method === 'gitea').length, 0);
|
||||
assert.ok(f.giteaRow('coder'));
|
||||
assert.equal(f.userTokens('demo-coder').length, 1);
|
||||
assert.equal(f.state('coder', 'gitea'), 'valid');
|
||||
// B back: the repeat revokes only B and Gitea.
|
||||
f.failing.delete('web2');
|
||||
const repeat = f.fire('coder');
|
||||
assert.deepEqual(repeat.revoking, ['host:web2', 'gitea']);
|
||||
await f.broker.grantsIdle();
|
||||
const [done] = f.allowed('fire.done');
|
||||
assert.equal(done.body.job, repeat.job);
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web2', 'gitea']);
|
||||
allRevoked(f);
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
|
||||
});
|
||||
|
||||
test('row 281 T4b (Q1): a line a manual hosts.revoke left on C is removed by the fire, with its close', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
await f.approveHost('coder', 'web3');
|
||||
f.failing.add('web3');
|
||||
const manual = await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web3', reason: 'web3 is down' });
|
||||
const tag3 = f.tag('coder', 'web3');
|
||||
assert.equal(manual.hostLineLeft, tag3);
|
||||
f.failing.delete('web3');
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['host:web1', 'line:web3']);
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.lines.web3.has(tag3), false);
|
||||
const close = f.grantEvents().filter((e) => e.action === 'grant.revoke.closed' && e.host === 'web3' && e.fire === true);
|
||||
assert.equal(close.length, 1);
|
||||
assert.equal(close[0].hostLine, 'removed');
|
||||
assert.equal(close[0].tag, tag3);
|
||||
// hosts.revoke's answer has no revoke id; the manual's grant.revoked has it.
|
||||
const [manualRevoked] = f.revoked('coder', (e) => e.host === 'web3' && e.fire !== true);
|
||||
assert.equal(close[0].revoke, manualRevoked.revoke);
|
||||
const [done] = f.allowed('fire.done');
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1', 'line:web3']);
|
||||
assert.deepEqual(done.body.grantsLeft, []);
|
||||
});
|
||||
|
||||
test('row 281 T4b (Q1): with C unreachable, the fire completes and names the line in grantsLeft', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
await f.approveHost('coder', 'web3');
|
||||
f.failing.add('web3');
|
||||
await f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web3', reason: 'web3 is down' });
|
||||
const tag3 = f.tag('coder', 'web3');
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['host:web1', 'line:web3']);
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
|
||||
const [done] = f.allowed('fire.done');
|
||||
assert.equal(done.body.job, out.job);
|
||||
assert.deepEqual(done.body.grants.map((x) => x.grant), ['host:web1']);
|
||||
assert.deepEqual(done.body.grantsLeft, [{ host: 'web3', tag: tag3 }]);
|
||||
assert.ok(f.lines.web3.has(tag3));
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), [['instance.fire', 'coder']]);
|
||||
});
|
||||
|
||||
test("row 281 T5: Gitea's admin token can't be checked: the job refuses with the Gitea grant intact", async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveGitea('coder');
|
||||
const off = f.gitea.rule((e) => e.method === 'GET' && e.path === '/api/v1/version', { status: 500, json: {} });
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['gitea']);
|
||||
await f.broker.grantsIdle();
|
||||
off();
|
||||
const [refused] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
|
||||
assert.deepEqual([refused.grant, refused.reason, refused.job], ['gitea', 'gitea-admin-unavailable', out.job]);
|
||||
assert.ok(f.inbox()[0].includes('mosaic grants sweep gitea'));
|
||||
assert.ok(f.giteaRow('coder'));
|
||||
assert.ok(existsSync(f.tokenFile('coder')));
|
||||
assert.equal(f.state('coder', 'gitea'), 'valid');
|
||||
assert.equal(f.userTokens('demo-coder').length, 1);
|
||||
assert.equal(f.gitea.users.get('demo-coder').prohibit_login, false);
|
||||
assert.equal(f.revoked('coder').length, 0);
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
assert.deepEqual(f.org.log.filter(([op]) => op === 'instance.fire'), []);
|
||||
});
|
||||
|
||||
test("row 281 T5b (N1): a failed token DELETE refuses gitea-left; the repeat retries only this agent's entries", async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveGitea('helper');
|
||||
await f.approveGitea('coder');
|
||||
const tokenDelete = (login) => (e) => e.method === 'DELETE' && e.path.startsWith(`/api/v1/users/${login}/tokens/`);
|
||||
// Another agent's left entry, from a manual revoke whose DELETE failed.
|
||||
const offHelper = f.gitea.rule(tokenDelete('demo-helper'), { status: 500, json: {} });
|
||||
const manual = await f.call('human', 'grants.revoke', { service: 'gitea', agent: 'helper', reason: 'test' });
|
||||
assert.equal(manual.serviceLeft.length, 1);
|
||||
const helperDeletes = () => f.gitea.requests.filter(tokenDelete('demo-helper')).length;
|
||||
const before = helperDeletes();
|
||||
const offCoder = f.gitea.rule(tokenDelete('demo-coder'), { status: 500, json: {} });
|
||||
const out = f.fire('coder');
|
||||
await f.broker.grantsIdle();
|
||||
const [refused] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
|
||||
assert.equal(refused.grant, 'gitea');
|
||||
// The close's serviceLeft (N10): the token id, and "tokens" for the fire's list that didn't clear.
|
||||
assert.match(refused.reason, /^gitea-left:\d+,tokens$/);
|
||||
assert.equal(refused.job, out.job);
|
||||
assert.equal(f.userTokens('demo-coder').length, 1);
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
// Gitea back for coder only. The helper's DELETE still fails, so a business-wide sweep would show.
|
||||
offCoder();
|
||||
const repeat = f.fire('coder');
|
||||
assert.deepEqual(repeat.revoking, ['gitea']);
|
||||
await f.broker.grantsIdle();
|
||||
const [done] = f.allowed('fire.done');
|
||||
assert.equal(done.body.job, repeat.job);
|
||||
assert.deepEqual(f.userTokens('demo-coder'), []);
|
||||
const swept = f.grantEvents().filter((e) => e.action === 'grant.swept' && e.job === repeat.job);
|
||||
assert.equal(swept.length, 1);
|
||||
assert.equal(swept[0].agent, 'coder');
|
||||
assert.deepEqual(swept[0].left.map((e) => e.agent), ['helper']);
|
||||
// Neither job touched the helper's token.
|
||||
assert.equal(helperDeletes(), before);
|
||||
assert.equal(f.userTokens('demo-helper').length, 1);
|
||||
offHelper();
|
||||
});
|
||||
|
||||
test('row 281 T6 (sage-2574): a host decision approved before the fire and run after it ends revoked, with nothing applied', async (t) => {
|
||||
const f = await setup(t);
|
||||
const id = f.fixtureHost('coder', 'web1');
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['host:web1']);
|
||||
await f.broker.grantsIdle();
|
||||
const [r] = f.revoked('coder', (e) => e.host === 'web1');
|
||||
assert.equal(r.granted, false);
|
||||
assert.equal(r.fire, true);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
assert.equal(await f.broker.runHostGrant('demo', id), 'revoked');
|
||||
assert.equal(f.ended(id).code, 'revoked');
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
assert.equal(existsSync(f.keyFile('coder', 'web1')), false);
|
||||
assert.ok(!f.sent.some((x) => x.startsWith('add ')));
|
||||
});
|
||||
|
||||
test('row 281 T6b (sage-2574): a Gitea decision approved before the fire and run after it ends revoked before any Gitea write', async (t) => {
|
||||
const f = await setup(t);
|
||||
const id = f.fixtureGitea('coder');
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['gitea']);
|
||||
await f.broker.grantsIdle();
|
||||
const [r] = f.revoked('coder', (e) => e.method === 'gitea');
|
||||
assert.deepEqual([r.user, r.granted, r.fire, r.decisions], [null, false, true, [id]]);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
const calls = f.gitea.requests.length;
|
||||
await f.broker.runGiteaGrant('demo', id);
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.ended(id).code, 'revoked');
|
||||
assert.equal(f.giteaRow('coder'), undefined);
|
||||
assert.equal(existsSync(f.tokenFile('coder')), false);
|
||||
// Only step 0's C1 to C4 reach Gitea, all reads; step 1's recheck ends it before any write.
|
||||
assert.deepEqual(f.gitea.requests.slice(calls).map((e) => `${e.method} ${e.path}`), [
|
||||
'GET /api/v1/version',
|
||||
'GET /api/v1/user',
|
||||
'GET /api/v1/token',
|
||||
'GET /api/v1/user/repos?limit=1&page=1',
|
||||
]);
|
||||
assert.ok(!f.gitea.requests.some((e) => e.method === 'POST' && e.path.endsWith('/tokens')));
|
||||
});
|
||||
|
||||
test('row 281 T6c: a host executor in flight aborts and ends revoked; a Gitea executor in flight is awaited and leaves no token', async (t) => {
|
||||
const f = await setup(t);
|
||||
// Host: held at its add.
|
||||
f.held.add('web1:add');
|
||||
const host = await f.raiseHost('coder', 'web1');
|
||||
f.call('human', 'decision.resolve', { id: host.id, choice: 'yes' });
|
||||
await until(() => f.sent.some((x) => x.startsWith('add web1 ')));
|
||||
// Gitea: held at the token create.
|
||||
let go;
|
||||
const gate = new Promise((r) => (go = r));
|
||||
t.after(() => go());
|
||||
let reached = false;
|
||||
const off = f.gitea.rule(
|
||||
(e) => e.method === 'POST' && e.path === '/api/v1/users/demo-coder/tokens',
|
||||
async () => {
|
||||
reached = true;
|
||||
await gate;
|
||||
return { pass: true };
|
||||
},
|
||||
);
|
||||
const gd = await f.raiseGitea('coder');
|
||||
f.call('human', 'decision.resolve', { id: gd.id, choice: 'yes' });
|
||||
await until(() => reached);
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['host:web1', 'gitea']);
|
||||
await until(() => f.ended(host.id));
|
||||
assert.equal(f.ended(host.id).code, 'revoked');
|
||||
// The job waits on the Gitea executor.
|
||||
await sleep(50);
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
go();
|
||||
await f.broker.grantsIdle();
|
||||
off();
|
||||
f.held.delete('web1:add');
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
assert.equal(existsSync(f.keyFile('coder', 'web1')), false);
|
||||
assert.equal(f.lines.web1.has(f.tag('coder', 'web1')), false);
|
||||
assert.ok(f.ended(gd.id));
|
||||
assert.deepEqual(f.userTokens('demo-coder'), []);
|
||||
assert.equal(f.giteaRow('coder'), undefined);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
f.noLeak();
|
||||
});
|
||||
|
||||
test('row 281 T6d (B1): undecided host and Gitea decisions are in the set, and approved after the fire both end revoked', async (t) => {
|
||||
const f = await setup(t);
|
||||
const host = await f.raiseHost('coder', 'web1');
|
||||
const gd = await f.raiseGitea('coder');
|
||||
const out = f.fire('coder');
|
||||
assert.deepEqual(out.revoking, ['host:web1', 'gitea']);
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
for (const d of [host, gd]) {
|
||||
f.call('human', 'decision.resolve', { id: d.id, choice: 'yes' });
|
||||
await until(() => f.ended(d.id));
|
||||
}
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.ended(host.id).code, 'revoked');
|
||||
assert.equal(f.ended(gd.id).code, 'revoked');
|
||||
assert.ok(!f.sent.some((x) => x.startsWith('add ')));
|
||||
assert.ok(!f.gitea.requests.some((e) => e.method === 'POST' && e.path.endsWith('/tokens')));
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
assert.equal(f.giteaRow('coder'), undefined);
|
||||
});
|
||||
|
||||
test('row 281 T6e (B1): a grant raise refuses instance-fired while firing and after the fire, goes through after a rehire, and refuses grant-revoke-running during a delete job', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
// Firing with no job: step B refused, so fire.stopped stands with no fire.done.
|
||||
f.org.fireError = Object.assign(new Error('refused'), { code: 'credential-revoke-failed' });
|
||||
f.fire('coder');
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.allowed('fire.failed').length, 1);
|
||||
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
|
||||
await rejects(() => f.raiseGitea('coder'), 'instance-fired');
|
||||
// After fire.done: delisted.
|
||||
f.org.fireError = null;
|
||||
assert.match(f.fire('coder').transaction, /^t-/);
|
||||
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
|
||||
await rejects(() => f.raiseGitea('coder'), 'instance-fired');
|
||||
// A rehire lists the name again.
|
||||
f.doc.launch.instances.push('coder');
|
||||
f.broker.reloadBusiness('demo', f.doc);
|
||||
assert.match((await f.raiseHost('coder', 'web2')).id, UUID);
|
||||
// While a delete's job holds ops.
|
||||
await f.approveHost('ops', 'web3');
|
||||
f.held.add('web3:remove');
|
||||
assert.equal(f.remove('ops').code, 'grants-revoking');
|
||||
await rejects(() => f.raiseHost('ops', 'web1'), 'grant-revoke-running');
|
||||
f.held.delete('web3:remove');
|
||||
await f.broker.grantsIdle();
|
||||
});
|
||||
|
||||
test('row 281 T7: an instance with no grant fires as today, synchronously, with the same events', async (t) => {
|
||||
const f = await setup(t);
|
||||
const out = f.fire('helper');
|
||||
assert.deepEqual(Object.keys(out).sort(), ['cancelled', 'request', 'revision', 'runs', 'taken', 'transaction']);
|
||||
assert.match(out.transaction, /^t-/);
|
||||
// Recorded before the verb answered: no job.
|
||||
assert.equal(f.allowed('fire.stopped').length, 1);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
assert.equal(f.allowed('fire.done')[0].body.job, undefined);
|
||||
assert.equal(f.allowed('fire.revoking').length, 0);
|
||||
assert.equal(f.allowed('role.fire').length, 1);
|
||||
assert.deepEqual(f.org.log.map(([op]) => op), ['instance.fireCheck', 'instance.fire']);
|
||||
await f.broker.grantsIdle();
|
||||
assert.deepEqual(f.inbox(), []);
|
||||
assert.equal(f.grantEvents().filter((e) => e.action?.startsWith('grant.revoke')).length, 0);
|
||||
});
|
||||
|
||||
test('row 281 T9 (B3): one step G per agent; the manual verbs refuse while it runs, and sweep names it running', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
await f.approveGitea('coder');
|
||||
f.held.add('web1:remove');
|
||||
f.fire('coder');
|
||||
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
|
||||
assert.equal(code(() => f.fire('coder')), 'fire-in-progress');
|
||||
assert.equal(f.remove('coder').code, 'grants-revoking');
|
||||
assert.equal(f.allowed('grants.revoking').length, 0);
|
||||
assert.equal(f.allowed('fire.revoking').length, 1);
|
||||
await rejects(() => f.call('human', 'hosts.revoke', { agent: 'coder', host: 'web1', reason: 'r' }), 'grant-revoke-running');
|
||||
await rejects(() => f.call('tester', 'hosts.revoke', { agent: 'coder', host: 'web1', reason: 'r' }), 'grant-revoke-running');
|
||||
await rejects(() => f.call('human', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'grant-revoke-running');
|
||||
const sweep = await f.call('human', 'grants.sweep', { service: 'gitea' });
|
||||
assert.deepEqual(sweep.running, ['coder']);
|
||||
f.held.delete('web1:remove');
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.revoked('coder', (e) => e.host === 'web1').length, 1);
|
||||
assert.equal(f.revoked('coder', (e) => e.method === 'gitea').length, 1);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
allRevoked(f, ['web1']);
|
||||
});
|
||||
|
||||
test('row 281 T10 (B2): the fire and the delete answer while their jobs are held, and the broker answers other verbs', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveGitea('coder');
|
||||
await f.approveHost('ops', 'web2');
|
||||
let go;
|
||||
const gate = new Promise((r) => (go = r));
|
||||
t.after(() => go());
|
||||
let reached = false;
|
||||
const off = f.gitea.rule(
|
||||
(e) => e.method === 'DELETE' && e.path.startsWith('/api/v1/users/demo-coder/tokens/'),
|
||||
async () => {
|
||||
reached = true;
|
||||
await gate;
|
||||
return { pass: true };
|
||||
},
|
||||
);
|
||||
f.held.add('web2:remove');
|
||||
// The verbs answer before the event loop turns, so before any part of a job has run.
|
||||
let turned = false;
|
||||
setImmediate(() => (turned = true));
|
||||
const fired = f.fire('coder');
|
||||
const deleted = f.remove('ops');
|
||||
assert.equal(turned, false);
|
||||
assert.equal(fired.done, false);
|
||||
assert.equal(deleted.code, 'grants-revoking');
|
||||
await until(() => reached && f.sent.includes(`remove web2 ${f.tag('ops', 'web2')}`));
|
||||
// Held: another session's verb answers.
|
||||
assert.ok(f.call('pm', 'message.send', { to: 'chief', body: 'still here' }));
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
assert.equal(f.allowed('grants.revoked.done').length, 0);
|
||||
go();
|
||||
f.held.delete('web2:remove');
|
||||
await f.broker.grantsIdle();
|
||||
off();
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
assert.equal(f.allowed('grants.revoked.done').length, 1);
|
||||
assert.deepEqual(f.userTokens('demo-coder'), []);
|
||||
assert.equal(f.hostRow('ops', 'web2'), undefined);
|
||||
});
|
||||
|
||||
test("row 281 T11 (C1): a fire's job completes after the chief's run ends and processing halts", async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
f.held.add('web1:remove');
|
||||
const out = f.fire('coder');
|
||||
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`));
|
||||
f.broker.endLaunch({ business: 'demo', run: 'chief-run', reason: 'test' });
|
||||
f.sw.state = { version: 1, halted: true, generation: GENERATION };
|
||||
f.held.delete('web1:remove');
|
||||
await f.broker.grantsIdle();
|
||||
const [done] = f.allowed('fire.done');
|
||||
assert.equal(done.body.job, out.job);
|
||||
assert.match(done.body.transaction, /^t-/);
|
||||
const [used, ...more] = f.allowed('role.fire');
|
||||
assert.equal(more.length, 0);
|
||||
assert.equal(used.actor_role, 'chief');
|
||||
assert.equal(used.body.transaction, done.body.transaction);
|
||||
assert.ok(!f.doc.launch.instances.includes('coder'));
|
||||
assert.equal(f.allowed('fire.failed').length, 0);
|
||||
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
|
||||
});
|
||||
|
||||
test("row 281 T12 (C1, N7): a step B refusal in the job is fire.failed, and the repeat runs B and C synchronously", async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('coder', 'web1');
|
||||
// N7: the writer's own credential-revoke-failed, which step G's key revokes can't clear.
|
||||
f.org.fireError = Object.assign(new Error('refused'), { code: 'credential-revoke-failed' });
|
||||
const out = f.fire('coder');
|
||||
await f.broker.grantsIdle();
|
||||
const [failed, ...more] = f.allowed('fire.failed');
|
||||
assert.equal(more.length, 0);
|
||||
assert.deepEqual([failed.body.job, failed.body.step, failed.body.code], [out.job, 'B', 'credential-revoke-failed']);
|
||||
assert.equal(failed.actor_role, 'chief');
|
||||
const [message] = f.inbox();
|
||||
for (const s of [out.job, 'host:web1', 'step B', 'credential-revoke-failed']) assert.ok(message.includes(s), s);
|
||||
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 0);
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
assert.ok(f.doc.launch.instances.includes('coder'));
|
||||
await rejects(() => f.raiseHost('coder', 'web2'), 'instance-fired');
|
||||
// The grant was revoked before step B.
|
||||
assert.equal(f.hostRow('coder', 'web1'), undefined);
|
||||
f.org.fireError = null;
|
||||
const repeat = f.fire('coder');
|
||||
assert.equal(repeat.job, undefined);
|
||||
assert.match(repeat.transaction, /^t-/);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
assert.ok(!f.doc.launch.instances.includes('coder'));
|
||||
});
|
||||
|
||||
test('row 281 T13 (N4): a restart records each held job interrupted once, with one message each; an ended job gets none', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approveHost('helper', 'web3');
|
||||
await f.approveHost('coder', 'web1');
|
||||
await f.approveHost('ops', 'web2');
|
||||
// A delete job that ended before the restart.
|
||||
assert.equal(f.remove('helper').code, 'grants-revoking');
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.allowed('grants.revoked.done').length, 1);
|
||||
f.held.add('web1:remove');
|
||||
f.held.add('web2:remove');
|
||||
const fired = f.fire('coder');
|
||||
assert.equal(f.remove('ops').code, 'grants-revoking');
|
||||
const deleteJob = f.allowed('grants.revoking').at(-1).body.job;
|
||||
await until(() => f.sent.includes(`remove web1 ${f.tag('coder', 'web1')}`) && f.sent.includes(`remove web2 ${f.tag('ops', 'web2')}`));
|
||||
const restartNotes = () => f.inbox().filter((m) => m.includes('interrupted by a bus restart'));
|
||||
const second = f.make();
|
||||
second.reconcileGrants();
|
||||
const interrupted = f.allowed('grants.job.interrupted');
|
||||
assert.deepEqual(
|
||||
interrupted.map((e) => [e.body.job, e.body.agent, e.body.verb]).sort(),
|
||||
[[fired.job, 'coder', 'instance.fire'], [deleteJob, 'ops', 'lifecycle.delete']].sort(),
|
||||
);
|
||||
assert.ok(interrupted.every((e) => e.actor_role === null));
|
||||
const notes = restartNotes();
|
||||
assert.equal(notes.length, 2);
|
||||
assert.ok(notes.some((m) => m.includes(fired.job) && m.includes('Run the fire again')));
|
||||
assert.ok(notes.some((m) => m.includes(deleteJob) && m.includes('Raise the delete again')));
|
||||
// A third broker on the same root sends nothing more.
|
||||
f.make().reconcileGrants();
|
||||
assert.equal(f.allowed('grants.job.interrupted').length, 2);
|
||||
assert.equal(restartNotes().length, 2);
|
||||
f.held.clear();
|
||||
await f.broker.grantsIdle();
|
||||
});
|
||||
|
||||
test("row 281 T14 (N1, sage-2653): a host removed under an undecided decision stops the fire on host-unknown; the human's no closes it, and the repeat fire has no job", async (t) => {
|
||||
const f = await setup(t);
|
||||
const d = await f.raiseHost('coder', 'web1');
|
||||
// N1: hosts.remove doesn't check open decisions, so the decision outlives its host.
|
||||
assert.deepEqual(await f.call('human', 'hosts.remove', { host: 'web1' }), { host: 'web1', removed: true });
|
||||
const out = f.fire('coder');
|
||||
assert.match(out.job, UUID);
|
||||
assert.deepEqual(out.revoking, ['host:web1']);
|
||||
await f.broker.grantsIdle();
|
||||
const [refused, ...more] = f.grantEvents().filter((e) => e.action === 'grant.revoke.refused');
|
||||
assert.equal(more.length, 0);
|
||||
assert.deepEqual([refused.job, refused.grant, refused.reason, refused.verb], [out.job, 'host:web1', 'host-unknown', 'instance.fire']);
|
||||
const [message, ...others] = f.inbox();
|
||||
assert.equal(others.length, 0);
|
||||
for (const s of [out.job, 'web1', 'host-unknown']) assert.ok(message.includes(s), s);
|
||||
assert.equal(f.allowed('fire.done').length, 0);
|
||||
assert.ok(f.doc.launch.instances.includes('coder'));
|
||||
// The human's no: no executor runs, so the closed check alone takes it out of the set (M22).
|
||||
f.call('human', 'decision.resolve', { id: d.id, choice: 'no' });
|
||||
await f.broker.grantsIdle();
|
||||
assert.equal(f.ended(d.id), undefined);
|
||||
const repeat = f.fire('coder');
|
||||
assert.equal(repeat.job, undefined);
|
||||
assert.match(repeat.transaction, /^t-/);
|
||||
assert.equal(f.allowed('fire.done').length, 1);
|
||||
assert.equal(f.allowed('fire.done')[0].body.job, undefined);
|
||||
assert.ok(!f.doc.launch.instances.includes('coder'));
|
||||
assert.equal(f.grantEvents().filter((e) => e.action === 'grant.revoke.refused').length, 1);
|
||||
});
|
||||
@@ -1216,6 +1216,27 @@ test('X-G14: a revoke Gitea cannot reach stays open, and a restart closes it', a
|
||||
await rejects(f.call('pm', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'revoke-refused');
|
||||
});
|
||||
|
||||
// Row 281 (#1786, d6 section 8, T8): grants.revoke shares its Gitea steps with a fire's step G now. This
|
||||
// pins the tester's dev-only refusal, the verb's records and their order, and that it starts no job.
|
||||
test('row 281 T8: a tester grants.revoke refuses tester-dev-only once bus.tester is off; the human revoke keeps its records and starts no job', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approve();
|
||||
f.broker.reloadBusiness('demo', { ...f.business, vars: { 'bus.tester': 'yes' } });
|
||||
const before = f.grantEvents().length;
|
||||
await rejects(f.call('tester', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' }), 'tester-dev-only');
|
||||
assert.equal(f.grantEvents().length, before);
|
||||
assert.ok(f.record());
|
||||
await f.call('human', 'grants.revoke', { service: 'gitea', agent: 'coder', reason: 'r' });
|
||||
const after = f.grantEvents().slice(before)
|
||||
.filter((e) => e.action !== 'grant.call' && e.action !== 'grant.result')
|
||||
.map((e) => [e.action, e.fire ?? false]);
|
||||
assert.deepEqual(after, [['grant.revoked', false], ['grant.revoke.closed', false]]);
|
||||
assert.deepEqual(f.userTokens(), []);
|
||||
const jobs = f.store.all("SELECT body FROM events WHERE business='demo' AND kind='action.allowed'").map((e) => JSON.parse(e.body).action)
|
||||
.filter((a) => /^(fire|grants)\./.test(a));
|
||||
assert.deepEqual(jobs, []);
|
||||
});
|
||||
|
||||
test('X-G15: look-alike commit steps outside the grant writer are ignored at start, by revoke and by status', async (t) => {
|
||||
const f = await setup(t);
|
||||
const first = await f.approve();
|
||||
|
||||
@@ -780,6 +780,24 @@ test('G26: the human, the assistant and the tester revoke; the chief and a pm re
|
||||
assert.deepEqual((await f.call('tester', 'hosts.sweep', { host: 'web1' })).removed, []);
|
||||
});
|
||||
|
||||
// Row 281 (#1786, d6 section 8, T8): hosts.revoke shares its steps with a fire's step G now. G12 pins its
|
||||
// records and their order; this pins the tester's dev-only refusal and that a verb revoke starts no job.
|
||||
test('row 281 T8: a tester hosts.revoke refuses tester-dev-only once bus.tester is off, and a human revoke writes no job records', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approve();
|
||||
f.caps.tester = f.broker.bindTester({ business: 'demo' });
|
||||
f.broker.reloadBusiness('demo', { ...f.business, vars: { 'bus.tester': 'yes' } });
|
||||
const args = { agent: 'ops', host: 'web1', reason: 'r' };
|
||||
await refusesWith(f.call('tester', 'hosts.revoke', args), 'tester-dev-only');
|
||||
assert.equal(revokeBodies(f).length, 0);
|
||||
assert.ok(f.record());
|
||||
assert.equal((await f.call('human', 'hosts.revoke', args)).granted, true);
|
||||
assert.deepEqual(revokeBodies(f).map((e) => [e.action, e.fire, e.delete]), [['grant.revoked', undefined, undefined], ['grant.revoke.closed', undefined, undefined]]);
|
||||
const jobs = f.store.all("SELECT body FROM events WHERE business='demo' AND kind='action.allowed'").map((e) => JSON.parse(e.body).action)
|
||||
.filter((a) => /^(fire|grants)\./.test(a));
|
||||
assert.deepEqual(jobs, []);
|
||||
});
|
||||
|
||||
test('G27: a restart after step 1 sweeps the key file and runs step 3 once the bus is up; status shows the revoke until then', async (t) => {
|
||||
const f = await setup(t);
|
||||
await f.approve();
|
||||
|
||||
@@ -9,6 +9,8 @@ import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { busBusiness } from '../src/business.mjs';
|
||||
import { startBroker } from '../src/runtime.mjs';
|
||||
import { Store } from '../src/store.mjs';
|
||||
import { hostGrantRecord, hostRecord } from './host-fixtures.mjs';
|
||||
import { loadBusiness } from '../../business/src/business.mjs';
|
||||
import { resolveInstance } from '../../business/src/resolve.mjs';
|
||||
import { businessDoc, FAR_DATE, rolesCopy, systemFor, writeJson } from '../../business/tests/helpers.mjs';
|
||||
@@ -19,7 +21,7 @@ const store = (agent, service) => ({ store: true, [service === 'gitea' ? 'rotate
|
||||
// coder: gitea in the store, vikunja a file reference. rev-2 and rev-3: both in the store, with a
|
||||
// tracker bot. `drop` names instances whose credentials the boot copy leaves out, as a restart that
|
||||
// dropped the entry would: the running bus then has no entry for a file on disk (sage-1592).
|
||||
async function setup(t, { drop = [] } = {}) {
|
||||
async function setup(t, { drop = [], seed = null } = {}) {
|
||||
const root = mkdtempSync(join(tmpdir(), 'bus-lifecycle-revoke-'));
|
||||
const rolesDir = rolesCopy(root);
|
||||
const doc = { ...businessDoc(root, 'demo'), plugins: { harness: ['pi'] } };
|
||||
@@ -31,6 +33,12 @@ async function setup(t, { drop = [] } = {}) {
|
||||
const file = writeJson(join(root, 'businesses/demo.json'), doc);
|
||||
mkdirSync(join(root, 'data'), { mode: 0o700 });
|
||||
mkdirSync(join(root, 'secrets', 'agents'), { recursive: true, mode: 0o700 });
|
||||
// Row 281 T1: records written before the start, as an earlier bus left them.
|
||||
if (seed) {
|
||||
const seeded = new Store(join(root, 'data'));
|
||||
seed(seeded);
|
||||
seeded.close();
|
||||
}
|
||||
const b = loadBusiness('demo', { dir: root, rolesDir });
|
||||
const resolved = {};
|
||||
for (const i of Object.keys(b.roles)) resolved[i] = resolveInstance({ system: systemFor(root), business: b, instance: i });
|
||||
@@ -181,3 +189,32 @@ test('T21 delete: a write between the raise and the take makes the delete refuse
|
||||
assert.deepEqual(readdirSync(f.agentDir('rev-2')), ['vikunja']);
|
||||
assert.deepEqual(f.status('rev-2'), { gitea: 'not-issued', vikunja: 'valid' });
|
||||
});
|
||||
|
||||
// Row 281 (#1786, d6 section 1) T1: pins what section 1 read from the code, and what still holds after the
|
||||
// change. Step B's revoke refuses while a grant's file is in the agent's store, which is why step G runs
|
||||
// first. A fire with no grant record goes straight to step B, so a stray host.<id> file planted after the
|
||||
// start (the start sweep would unlink it) still refuses there.
|
||||
test('row 281 T1 fire: a host.<id> or granted gitea file in the store refuses step B credential-revoke-failed, file unchanged', async (t) => {
|
||||
const f = await setup(t);
|
||||
f.plant('coder', 'host.web1');
|
||||
const before = f.bytes();
|
||||
assert.equal(f.code(() => f.fire('coder')), 'credential-revoke-failed');
|
||||
assert.equal(f.bytes(), before);
|
||||
assert.deepEqual(readdirSync(f.agentDir('coder')), ['host.web1']);
|
||||
rmSync(join(f.agentDir('coder'), 'host.web1'));
|
||||
assert.equal(f.fire('coder').delisted, true);
|
||||
});
|
||||
|
||||
// T1, the start: a host_grants record whose key file is gone (what a delete left before this row). The
|
||||
// start loads the reference with no token, so the bus comes up and the entry reads not-issued.
|
||||
test('row 281 T1 start: a host grant record whose key file is gone starts and reads not-issued', async (t) => {
|
||||
const entry = { address: '192.0.2.10', port: 22, user: 'deploy', hostKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKeyForTests'] };
|
||||
const f = await setup(t, {
|
||||
seed: (store) => {
|
||||
hostRecord(store, { host: 'web1', entry });
|
||||
hostGrantRecord(store, { agent: 'coder', host: 'web1', commands: [['uptime']] });
|
||||
},
|
||||
});
|
||||
assert.equal(existsSync(join(f.agentDir('coder'), 'host.web1')), false);
|
||||
assert.equal(f.status('coder')['host.web1'], 'not-issued');
|
||||
});
|
||||
|
||||
@@ -13,11 +13,13 @@ import { spawn, spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { existsSync, lstatSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { launchDoc } from "../../business/tests/helpers.mjs";
|
||||
import { BusError } from "../../bus/src/broker.mjs";
|
||||
import { startBroker } from "../../bus/src/runtime.mjs";
|
||||
import { Store } from "../../bus/src/store.mjs";
|
||||
import { hostGrantRecord, hostRecord } from "../../bus/tests/host-fixtures.mjs";
|
||||
import { gitEnv } from "../../seat/src/checkout.mjs";
|
||||
import { sandboxProblem } from "../../seat/src/sandbox.mjs";
|
||||
import { processIdentity, registerRuntime, sweepRuntimes, unregisterRuntime, updateRuntime } from "../../processing-control/src/runtimes.mjs";
|
||||
@@ -428,6 +430,31 @@ test("row 194 V5: a delete with no workspace commits and closes with the transac
|
||||
assert.equal(existsSync(join(workspaceDir(p.f.dataRoot, "acme", ".retired"))), false);
|
||||
});
|
||||
|
||||
// Row 281 (#1786, d6 section 2, T10 B2): a delete of an instance with a grant answers at once. The pass
|
||||
// closes the first request grants-revoking while step G runs as a job, and the host channel stays up:
|
||||
// the next op answers. The grant is a seeded record on a TEST-NET host, so the job's remove can only
|
||||
// fail or wait; either way the request has already closed.
|
||||
test("row 281 T10: the delete pass closes a granted instance's first request grants-revoking and the host channel stays up", { skip, timeout: 90000 }, async (t) => {
|
||||
const p = prepare(t);
|
||||
// A host grant gives the agent a host.web1 store reference, so Credentials checks the store root at start.
|
||||
mkdirSync(join(dirname(p.f.config), "secrets", "agents"), { recursive: true, mode: 0o700 });
|
||||
const seeded = new Store(p.f.dataRoot);
|
||||
const entry = { address: "192.0.2.10", port: 22, user: "deploy", hostKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKeyForTests"] };
|
||||
hostRecord(seeded, { business: "acme", host: "web1", entry });
|
||||
hostGrantRecord(seeded, { business: "acme", agent: "sec-1", host: "web1", commands: [["uptime"]] });
|
||||
seeded.close();
|
||||
const s = await start(t, p);
|
||||
const del = s.call("instances.set", { revision: s.call("instances").revision, instance: "sec-1", op: "delete" }).request;
|
||||
const r = await s.carry(del);
|
||||
assert.equal(r.done.refusal, "grants-revoking");
|
||||
assert.equal(r.done.transaction ?? null, null);
|
||||
// The channel answers the next op and the next pass.
|
||||
assert.ok(s.call("instances").revision);
|
||||
await s.launcher.tick();
|
||||
assert.ok(Object.hasOwn(JSON.parse(readFileSync(join(p.root, "config", "businesses", "acme.json"), "utf8")).roles, "sec-1"));
|
||||
await s.runtime.broker.grantsIdle();
|
||||
});
|
||||
|
||||
// Every regular file under root, by path, as a sha256. Sockets and other special files are left out.
|
||||
function snapshot(root) {
|
||||
const out = {};
|
||||
|
||||
Reference in New Issue
Block a user