stack/docs/plans/2026-10-20_host-credentials.md

Brief: agent host access with sudo through the credential store (#424)

Sage, 2026-10-20, for Jason's rulings Q183 and Q184 relayed by Mos (mos-1056 item 3, mos-1067). Gitea #1707, Vikunja P19 #424, due 2026-10-27 (Q184: "Q5: by 10.27"). Owner Gizmoduck (design and build), reviewer Duckworth, gate owner Sage. Chained on row 199's candidate (Q177): row 199 design r6 (4d3007ec9 on megavolt/row199, approved in duckworth-0027) is not reopened. This row lands after 199 and rebases onto its landing.

What Jason asked for

Q183, through Mos: "I envision a credential store within Mosaic Stack that works as some sort of variable for the agent to call. The system will enforce access to the credentials to keep the agents sandboxed with proper security... If we can utilize the credential store in Mosaic Stack and allow agents to perform their jobs with NOPASSWD sudo on various machines, I am a happy man." Q183 also says not every agent needs that access.

Mos's requirements (mos-1067 item 1):

• An agent asks for a host credential by name.
• The stack checks a grant for that agent and that host, and does the ssh and sudo for it.
• The agent never sees the key or any secret value.
• Only agents with a grant can do it.

The PRD text is REQ-CRED-3, drafted by Honker for PRD 1.7 (sage-1422). If it misses 1.7 it goes into 1.8, and this row lands only after the version that holds it is approved.

What row 199 gives and what it doesn't (megavolt-0019)

It builds on 199 as 199 stands: the store layout under <configDir>/secrets/agents/<business>/<agent>/, the lifecycle verbs and their changes.jsonl receipts, useFor(session, service, fn) with the D5 allowlist test, assertClean on every result, and the grant shape of publish (a list on the role entry, checked by the broker, plus a hard-wait action that isn't routed).

This row's own changes:

• A host kind in the closed kind set (vocabulary.mjs SERVICES, credentials.mjs, the broker's service check, the CLI date flag, the remove regex), with a naming rule for one credential per host or per agent.
• A per-kind value check. An OpenSSH private key fails today's TOKEN pattern.
• Handing the key to ssh without the agent seeing it: an ssh-agent on a private socket, or a 0600 temp file removed on every path, the way publish's one-shot helper works.
• Pinned host keys per host, and no trust on first use.
• The command boundary: what an agent may run under sudo, and how output returns without echoing the secret. This is the main design work.
• The executor: a ticket and a host op running in the bus child, the same split row 199 uses for publish (Megavolt's deviation in 199's BUILD.md).

Steps

1. DESIGN.md first, in agents/gizmoduck/work/queue-ROW/. It answers each item above, names the grant shape (roles.<agent>.hosts and a host.sudo action, or a reasoned alternative), says who grants (Jason, through a reviewed role change, unless REQ-CRED-3 says otherwise), and lists tests and mutants. Duckworth approves it before code. Sage reads it too, because it gives agents sudo on hosts.
2. Build on 199's candidate. Single test files run freely. Suites and mutant batches need Sage's slot.
3. The live part is not this row. Putting a key in the store and granting a host is a separate window with Jason or Mos, after landing.

Hosts

The first targets are localadmin with NOPASSWD sudo on 10.1.1.53 and 10.1.1.54 (#423 step 1, applied by ops-01 at 20:05:47Z), and the servers agents already reach as localadmin (Q172). ops-01 owns the host side. Tests use scratch sshd fixtures on 127.0.0.1 or fakes, never a real host.

Boundaries

• No change to ~/.mosaic-dev, ~/.config/mosaic-dev, slot files, the live units or ~/.mosaic. No key is read, copied or printed. No ssh to any real host from a test or a seat.
• No widening of row 199's kinds or regex inside row 199. Both rows touch them, so this row rebases after 199 lands.

Gate

Duckworth approves DESIGN.md, then the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate, and the mutants are killed (one per refusal, plus one that leaks the key into the agent's output and one that skips the host key pin).

Machine grant row a: enrollment, store records and migration 3i-3j

Sage, 2026-10-20, for Vikunja P19 #436 (#436a), mos-1141 and mos-1143. Owner Gizmoduck, reviewer Duckworth, gate owner Sage. The design is agents/gizmoduck/work/machine-grant/DESIGN.md r6 at f9039682e on gizmoduck/machine-grant-design, which Duckworth approved for code once row 210 lands (duckworth-0063, mos-1141). It is not reopened here. Section 12 Q-S4 defines the split. Estimate 1.5 days. Gitea #1740.

Row a ships enrollment as the approved request (section 3), the four tools/host files, the store records and their migration 3i-3j.sql (section 5), host.run reading the records, and hosts status. It also moves row 210's tests onto the store. Every grant name is under grant.* behind one guard, as action.allowed bodies (mos-1144, gizmoduck-0086, duckworth-0065 O1). No agent creates a grant (Q188).

Order and timing:

• It builds on row 210's landing, after row 199 (mos-1135, mos-1137).
• This row owns SCHEMA_VERSION 3j. Any other row with a schema step goes 3j to 3k or later and rebases onto this row's landing: the #436b build takes 3j to 3k, then row 206 (193b2) 3k to 3l.
• It lands by 10-25 and before the .53 and .54 installs (mos-1134, mos-1145). Sage flags a slip past 10-25 to Mos. Live on dragon-lin comes in the one 10-24 window with rows 190, 199 and 210, as an agent-run --migrate under Q173, backup first (mos-1145 line 9). The run sheet strips row 210's hosts keys and role grants from the business file before that re-pin (design r5 N9).
• It needs the PRD 1.8 line before landing (Q-S4).

Gate: Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. The design's section 10 tests and mutants for row a pass, the migration test included.

Machine grant row b: raise binding, executor, reconcile and revoke

Same owner, reviewer, gate owner and design as row a. Estimate 1.75 days. Chained on row a's landing. Gitea #1741.

Row b ships the raise binding, the executor (section 5), reconcile, and revoke with the hosts.revoke verb and its marker (section 6), plus row 210's per-call decision as manual mode (section 7). After step 5 commits, the executor ignores an abort, and both undo paths update only an existing record (r6 C1, X3, X5). At start, after the grant reconcile, the bus unlinks every host.<id> key file that has no host_grants record (r6 N1, G27). It adds no schema step. If it needs one, Sage re-briefs it first and assigns the next free step (#436b holds 3j to 3k, row 206 3k to 3l).

Creative roles get no host grant (mos-1163, from Webby's N2 on row 213 r4). Since row 234, a role definition's credentials no longer gate host access, so the host grant writer itself refuses a grant whose agent is a CREATIVE_DEFINITIONS role (row 213), with a named reason and a test. The Case 2 hub runs creative agents on .54, and without this a creative role has a path to host.run. Duckworth's review checks it, and answers from the code whether a decision raised by another role can be spent by a creative one.

host.run refuses too, at spend time, when the agent's current definition is a CREATIVE_DEFINITIONS role (sage-1908, accepted in mos-1164). That covers a grant written before the role became creative. The refusal leaves the grant record in place, so hosts.revoke still finds it. Each path, the writer refusal at raise and at the step 1 recheck and the spend refusal, gets its own test and its own mutant. CREATIVE_DEFINITIONS comes from row 213, so row 213 lands before row b. If that order can't hold, Sage tells Mos before row b lands.

Gate: as row a, with section 10's row b tests and mutants, G27, G28 and the upsert mutant included, and the creative refusal tests and mutants (writer and spend).

Machine grant row b follow-up: the held stubs end with their test

The row 197 r8 D5 proof stopped on a leftover process on 2026-10-21. In packages/bus/tests/grant-executor.test.mjs, the add stub waits for release when hold-add is set. G23 ("a revoke during install") and G13 write release, finish, and t.after removes the test dir. The stub polls every 50 ms, so it usually misses release and then loops forever on a dir that is gone. Four runs of the file on Sage's slot left 7 stubs, every run 37/0.

Build: each hold loop in that file, hold-add and hold-remove, also stops when $D no longer exists. The test file is the only change. Nothing under src changes.

Gate: Webby approves the commit. On Sage's gate the whole file passes three times in a row with no stub left, checked by pgrep on the run's TMPDIR, and the bus suite is green. A mutant that drops the dir check from hold-add leaves a stub behind.

Machine grant row b follow-up: the undo can't prove the remote add has ended

Mos asked this in mos-1166. Gizmoduck answered in gizmoduck-0137 and gizmoduck-0139, and Duckworth checked the answer in duckworth-0091. Line numbers are at 6d5545d44.

An abort during install SIGKILLs the local ssh process group (host-run.mjs:136). A timeout does the same (host-run.mjs:131-134). sshd gives a non-pty command no SIGHUP when the client dies, so the remote add can keep running. The undo runs remove <user> <tag> once (broker.mjs:1017). If the add writes its line after that, the remove has already exited 0 and hostLineLeft stays unset (:1019). Then hosts.status shows the host clean (:1396), reconcile skips a grant that already ended (:1042), and nothing prompts a sweep. hosts.remove's host-line-left check (:1362) passes, and once the record is gone a sweep refuses host-unknown (:1350), so no verb reaches the line. The line authorizes a key nobody holds, under the installer's restrict prefix. No test covers this case. G24 seeds its failed grant with hostLineLeft already set.

Build:

• In #runHostGrant, mark install done right after await install(...) returns. In the catch, when install started and never returned, record hostLineLeft: tag whatever the remove's exit was. The rule keys on install not returning. It covers an abort, a timeout and a dropped connection, and doesn't depend on signal.aborted. The message names mosaic hosts sweep <host>.
• An add that exits nonzero on the host also records the hint. That hint is false but harmless, since a sweep that doesn't list the tag drops it (G24, "a left tag no longer on the host").
• Fix the comment at grant-executor.test.mjs:432, which says a killed session's add exits.
• Row a's enroll add: check it for the same race, then fix it or record in BUILD.md why it can't happen.
• Owner Gizmoduck. Reviewer Duckworth.

Tests: one abort case and one timeout case. Each uses a held stub add that writes after the remove and exits 0. Each asserts hostLineLeft on grant.failed and in hosts.status, that hosts.remove refuses host-line-left before the sweep, that the sweep removes the line, and that hosts.remove then succeeds. A mutant that drops the new catch branch fails a test.

Gate: Duckworth approves the manifest. On Sage's gate the bus suite and the whole touched test files are green, and the mutant is killed.