mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

EPIC: 'mosaic cred' — one governed CLI/broker for agent credential lifecycle + access (identity validation via token→certificate; fail-closed, audited) #1045

New Issue
Open
opened 2026-08-04 18:30:13 +00:00 by Ghost · 6 comments
Ghost commented 2026-08-04 18:30:13 +00:00
Copy Link
Copy Source

Problem — credential access is scattered, and the scatter is the failure

There is no single governed tool for agent credential lifecycle and access. What exists is split across two disconnected systems and several ad-hoc scripts:

  • _lib/credentials.sh + credentials.json — a read-only loader for shared service credentials (load_credentials <service> → env vars for gitea-mosaicstack, gitea-usc, portainer, authentik, woodpecker, cloudflare…). One token per service — the shared account.
  • Per-slot agent tokens — individual files ~/.config/mosaic/secrets/gitea-tokens/gitea-<host>-<identity>.token, resolved by git-credential-mosaic (via MOSAIC_GIT_IDENTITY) for git, and by get_gitea_token (detect-platform.sh) for the API path, and wired into each seat by hand.

Nothing owns the per-agent lifecycle (mint → register → wire → validate → rotate → revoke) or governs per-agent access (one fail-closed, audited code path). The consequences, all observed in one night of fleet dogfooding:

  • #1043 — seat bring-up mints a token but never wires MOSAIC_GIT_IDENTITY; seats reach the runtime unable to do git.
  • #1044 (security) — the two resolvers disagree on a lost identity: git-credential-mosaic fails closed, get_gitea_token falls back to the shared credential silently and authors as the owner.
  • #1013 — tokens sit as long-lived bearer secrets in files/argv, readable by same-UID processes.
  • Operational papercuts — this Gitea's WAF 403s default Python User-Agents, so api-commit.py silently fails while the shell wrappers work; every caller re-implements host detection, token lookup, UA, and fail-closed logic slightly differently.

Vision — mosaic cred: one governed entry point for credential lifecycle + access

A single CLI (growing into a broker) that owns credentials the way mosaic-delegate owns delegation: a predictable, mandatory path that every agent and script uses instead of reading token files directly.

Proposed CLI surface

mosaic cred provision <identity> [--host ...] [--scopes ...]   # mint/reuse account + token, register, wire env
mosaic cred wire <identity> [--seat-env <file>]                # write MOSAIC_GIT_IDENTITY (idempotent)
mosaic cred get <identity> --host <h> [--repo <r>]             # emit a credential, FAIL CLOSED, audited (no shared fallback under a fleet context)
mosaic cred validate <identity> --host <h> [--repo <r>]        # resolve + scope + per-repo WRITE-DIFFERENTIAL, assert by result
mosaic cred whoami                                             # what identity does THIS process resolve to, on which host
mosaic cred rotate <identity>   |   mosaic cred revoke <identity>
mosaic cred list [--stale]      |   mosaic cred audit          # inventory + who-accessed-what

Non-negotiable invariants (baked in once, not re-implemented per caller):

  • Fail closed under a fleet context — an unset/unresolvable identity NEVER falls back to a shared/privileged account (fixes #1044). Interactive/human use may still use shared creds explicitly.
  • Correct transport — sets the working User-Agent, host detection, TLS opts once (kills the api-commit 403 class).
  • Read-back / assert-by-result — validate proves the credential can actually do the operation on the target repo, not just that a file exists (a token that works on one repo proves nothing about another).
  • Audit every access — who requested which credential, for which host/repo, when.

Agent validation — token now, certificate/broker next (the part you flagged)

  • Phase 1 — CLI over the current file store. mosaic cred wraps the existing per-slot tokens + credentials.json behind the surface above. Still bearer tokens, but ONE governed, fail-closed, audited, UA-correct code path. Subsumes #1043 (provision/wire) and #1044 (fail-closed get).
  • Phase 2 — agent identity validation. Each agent holds a keypair/certificate (or an Authentik-issued service identity). The agent PROVES identity by signing a challenge — possession of a token file stops being identity. mosaic cred validates the requester before issuing anything, so an unwired/wrong agent gets a refusal, not a shared token.
  • Phase 3 — broker with short-lived, scoped credentials. mosaic cred becomes a broker/daemon (Vault approle/PKI-style) that issues short-lived, narrowly-scoped, per-operation credentials. Nothing long-lived sits in a file → kills #1013, and there is no shared credential to fall back TO.

Don't reinvent storage — front the backend

The stack already has an IdP (Authentik) and a secrets story (Vault, per VAULT-SECRETS.md). mosaic cred should be the agent-facing governance + identity layer OVER whatever backend (file store now, Vault/Authentik later) — it owns the contract and the fail-closed/audit behavior, not the secret storage itself.

Subsumes / relates to

  • #1043 (mechanize provision + wire) → mosaic cred provision / wire.
  • #1044 (fail-closed on the API path) → the get invariant.
  • #1013 (long-lived bearer tokens exposed) → Phase 3 short-lived creds.

Recommend keeping #1043/#1044 as the concrete near-term fixes and adopting THIS as the umbrella they roll up into, so the point fixes are built as the first slices of mosaic cred rather than throwaway patches.

Open design decisions (operator)

  1. Phase-2 identity: certificate/mTLS vs Authentik-issued service identity vs signed-token. Cert/mTLS is strongest and backend-agnostic; Authentik ties agent identity to the existing IdP.
  2. Broker (daemon) vs CLI-only. A daemon enables short-lived creds + central audit but adds a moving part; CLI-only is simpler but keeps some long-lived material.
  3. Backend: Vault vs Authentik vs keep the file store for phase 1, and the migration path.
  4. Scope granularity — per-host, per-repo, or per-operation credentials.
## Problem — credential access is scattered, and the scatter is the failure There is no single governed tool for **agent** credential lifecycle and access. What exists is split across two disconnected systems and several ad-hoc scripts: - **`_lib/credentials.sh` + `credentials.json`** — a read-only *loader* for **shared service** credentials (`load_credentials <service>` → env vars for `gitea-mosaicstack`, `gitea-usc`, portainer, authentik, woodpecker, cloudflare…). One token per service — the shared account. - **Per-slot agent tokens** — individual files `~/.config/mosaic/secrets/gitea-tokens/gitea-<host>-<identity>.token`, resolved by `git-credential-mosaic` (via `MOSAIC_GIT_IDENTITY`) for git, and by `get_gitea_token` (detect-platform.sh) for the API path, and wired into each seat by hand. Nothing owns the per-agent lifecycle (mint → register → wire → validate → rotate → revoke) or governs per-agent *access* (one fail-closed, audited code path). The consequences, all observed in one night of fleet dogfooding: - **#1043** — seat bring-up mints a token but never wires `MOSAIC_GIT_IDENTITY`; seats reach the runtime unable to do git. - **#1044 (security)** — the two resolvers disagree on a lost identity: `git-credential-mosaic` fails **closed**, `get_gitea_token` falls back to the **shared** credential silently and authors as the owner. - **#1013** — tokens sit as long-lived bearer secrets in files/argv, readable by same-UID processes. - **Operational papercuts** — this Gitea's WAF 403s default Python User-Agents, so `api-commit.py` silently fails while the shell wrappers work; every caller re-implements host detection, token lookup, UA, and fail-closed logic slightly differently. ## Vision — `mosaic cred`: one governed entry point for credential lifecycle + access A single CLI (growing into a broker) that owns credentials the way `mosaic-delegate` owns delegation: a predictable, mandatory path that every agent and script uses instead of reading token files directly. ### Proposed CLI surface ``` mosaic cred provision <identity> [--host ...] [--scopes ...] # mint/reuse account + token, register, wire env mosaic cred wire <identity> [--seat-env <file>] # write MOSAIC_GIT_IDENTITY (idempotent) mosaic cred get <identity> --host <h> [--repo <r>] # emit a credential, FAIL CLOSED, audited (no shared fallback under a fleet context) mosaic cred validate <identity> --host <h> [--repo <r>] # resolve + scope + per-repo WRITE-DIFFERENTIAL, assert by result mosaic cred whoami # what identity does THIS process resolve to, on which host mosaic cred rotate <identity> | mosaic cred revoke <identity> mosaic cred list [--stale] | mosaic cred audit # inventory + who-accessed-what ``` Non-negotiable invariants (baked in once, not re-implemented per caller): - **Fail closed under a fleet context** — an unset/unresolvable identity NEVER falls back to a shared/privileged account (fixes #1044). Interactive/human use may still use shared creds explicitly. - **Correct transport** — sets the working User-Agent, host detection, TLS opts once (kills the api-commit 403 class). - **Read-back / assert-by-result** — `validate` proves the credential can actually do the operation on the *target repo*, not just that a file exists (a token that works on one repo proves nothing about another). - **Audit every access** — who requested which credential, for which host/repo, when. ### Agent validation — token now, certificate/broker next (the part you flagged) - **Phase 1 — CLI over the current file store.** `mosaic cred` wraps the existing per-slot tokens + `credentials.json` behind the surface above. Still bearer tokens, but ONE governed, fail-closed, audited, UA-correct code path. Subsumes #1043 (provision/wire) and #1044 (fail-closed get). - **Phase 2 — agent identity validation.** Each agent holds a keypair/certificate (or an Authentik-issued service identity). The agent PROVES identity by signing a challenge — possession of a token file stops being identity. `mosaic cred` validates the requester before issuing anything, so an unwired/wrong agent gets a refusal, not a shared token. - **Phase 3 — broker with short-lived, scoped credentials.** `mosaic cred` becomes a broker/daemon (Vault approle/PKI-style) that issues short-lived, narrowly-scoped, per-operation credentials. Nothing long-lived sits in a file → kills #1013, and there is no shared credential to fall back TO. ### Don't reinvent storage — front the backend The stack already has an IdP (Authentik) and a secrets story (Vault, per VAULT-SECRETS.md). `mosaic cred` should be the **agent-facing governance + identity layer** OVER whatever backend (file store now, Vault/Authentik later) — it owns the contract and the fail-closed/audit behavior, not the secret storage itself. ## Subsumes / relates to - **#1043** (mechanize provision + wire) → `mosaic cred provision` / `wire`. - **#1044** (fail-closed on the API path) → the `get` invariant. - **#1013** (long-lived bearer tokens exposed) → Phase 3 short-lived creds. Recommend keeping #1043/#1044 as the concrete near-term fixes and adopting THIS as the umbrella they roll up into, so the point fixes are built as the first slices of `mosaic cred` rather than throwaway patches. ## Open design decisions (operator) 1. **Phase-2 identity: certificate/mTLS vs Authentik-issued service identity vs signed-token.** Cert/mTLS is strongest and backend-agnostic; Authentik ties agent identity to the existing IdP. 2. **Broker (daemon) vs CLI-only.** A daemon enables short-lived creds + central audit but adds a moving part; CLI-only is simpler but keeps some long-lived material. 3. **Backend: Vault vs Authentik vs keep the file store** for phase 1, and the migration path. 4. **Scope granularity** — per-host, per-repo, or per-operation credentials.
Ghost commented 2026-08-04 21:39:57 +00:00
Copy Link
Copy Source

Resolved requirement (operator, 2026-08-04) — pluggable backends + emergency-update UX + red-team

Scope confirmed: mosaic cred supplies ALL external-service creds (Gitea/GitHub/Forgejo/Matrix/Discord/…) with authN + authZ screening on every request. Authentik is at most the human-identity layer; it does not hold service secrets.

Backends are ADAPTERS behind one abstraction (both supported, chosen per-cred):

  • HashiCorp Vault — purpose-built for the machine/agent pattern (AppRole = bootstrap, dynamic short-lived leases). Already configured (.vault.addr). Poor human UX (hard to unlock/see) — NOT the human-facing surface.
  • VaultWarden (vw.uscllc.com, per-collection RBAC via Authentik SSO) — friendly human UX, good for human/shared creds and emergency user updates. CONSTRAINT (from docs/scratchpads/vaultwarden-upgrade-2025.md): items are encrypted under the user master password by design, so machine read requires either Bitwarden Secrets Manager (machine accounts/access tokens) — VERIFY VW supports it — or it cannot back agent secrets directly.

Emergency-update UX is a first-class requirement (operator): a human must be able to rotate a credential fast, in an emergency, without Vault's UX. Resolution via the abstraction — decouple the human write-path from the agent read-path: humans update in the friendly surface (VaultWarden UI); mosaic cred brokers agent reads and, where the agent-tier backend differs (Vault), an adapter syncs/propagates the change. So "update once in the easy place, agents pick it up" — safety AND emergency usability, not one at the cost of the other.

Bootstrap (secret-zero) mechanism — MUST be red-teamed before adoption (operator): proposed anchor is systemd LoadCredential= (web1 has systemd 252; tmpfs, unit-private $CREDENTIALS_DIRECTORY, not in env, not inherited by siblings — also mitigates #1013) delivering a short-lived signed JWT (sub=agent, scopes, exp ~minutes) that mosaic cred verifies (authN) before applying RBAC policy (authZ). This entire bootstrap chain is a REQUIRED adversarial-review item — do not adopt LoadCredential as "secure" without a red-team that tries to read another agent's credential dir, race the tmpfs, or forge/replay the JWT.

Open verify items: (1) does VaultWarden expose Secrets Manager / any master-password-free machine read; (2) red-team LoadCredential; (3) JWT issuer/rotation/revocation design. Prior jarvis-session VW-API research requested via scout.

## Resolved requirement (operator, 2026-08-04) — pluggable backends + emergency-update UX + red-team **Scope confirmed:** `mosaic cred` supplies ALL external-service creds (Gitea/GitHub/Forgejo/Matrix/Discord/…) with authN + authZ screening on every request. Authentik is at most the human-identity layer; it does not hold service secrets. **Backends are ADAPTERS behind one abstraction (both supported, chosen per-cred):** - **HashiCorp Vault** — purpose-built for the machine/agent pattern (AppRole = bootstrap, dynamic short-lived leases). Already configured (`.vault.addr`). Poor human UX (hard to unlock/see) — NOT the human-facing surface. - **VaultWarden** (vw.uscllc.com, per-collection RBAC via Authentik SSO) — friendly human UX, good for human/shared creds and **emergency user updates**. CONSTRAINT (from docs/scratchpads/vaultwarden-upgrade-2025.md): items are encrypted under the **user master password by design**, so machine read requires either Bitwarden **Secrets Manager** (machine accounts/access tokens) — VERIFY VW supports it — or it cannot back agent secrets directly. **Emergency-update UX is a first-class requirement (operator):** a human must be able to rotate a credential fast, in an emergency, without Vault's UX. Resolution via the abstraction — **decouple the human write-path from the agent read-path:** humans update in the friendly surface (VaultWarden UI); `mosaic cred` brokers agent reads and, where the agent-tier backend differs (Vault), an adapter syncs/propagates the change. So "update once in the easy place, agents pick it up" — safety AND emergency usability, not one at the cost of the other. **Bootstrap (secret-zero) mechanism — MUST be red-teamed before adoption (operator):** proposed anchor is systemd `LoadCredential=` (web1 has systemd 252; tmpfs, unit-private `$CREDENTIALS_DIRECTORY`, not in env, not inherited by siblings — also mitigates #1013) delivering a short-lived signed **JWT** (sub=agent, scopes, exp ~minutes) that `mosaic cred` verifies (authN) before applying RBAC policy (authZ). This entire bootstrap chain is a REQUIRED adversarial-review item — do not adopt LoadCredential as "secure" without a red-team that tries to read another agent's credential dir, race the tmpfs, or forge/replay the JWT. **Open verify items:** (1) does VaultWarden expose Secrets Manager / any master-password-free machine read; (2) red-team LoadCredential; (3) JWT issuer/rotation/revocation design. Prior jarvis-session VW-API research requested via scout.
Ghost commented 2026-08-04 21:43:14 +00:00
Copy Link
Copy Source

Correction (operator scoping, 2026-08-04): homelab-first, not USC

My earlier comment grounded on USC-network artifacts — that was a mis-grounding (mos-claude sits on the USC network; the deployment target is the homelab).

  • Initial Mosaic Stack deployment is completed FULLY on the homelab (W-jarvis). USC is not the target.
  • The VaultWarden test instance is vw.woltje.com (homelab), NOT vw.uscllc.com (that's the USC instance, a different deployment). Re-ground all VW research/config on vw.woltje.com.
  • The .vault.addr referenced earlier is the USC Vault and may not be the homelab's — do not assume it applies.
  • Instance-independent (still holds): the "master-password-required-for-decryption" constraint is Bitwarden's architecture, not a per-instance setting, so the deciding question (does VW expose a master-password-free machine read / Secrets Manager) applies to vw.woltje.com unchanged.
  • Federation / cross-deployment chatter / cross-network access controls = an explicit LATER phase, after homelab MS works; needs more discussion. The broker is designed homelab-scoped first, not cross-network.
## Correction (operator scoping, 2026-08-04): homelab-first, not USC My earlier comment grounded on USC-network artifacts — that was a mis-grounding (mos-claude sits on the USC network; the deployment target is the **homelab**). - **Initial Mosaic Stack deployment is completed FULLY on the homelab** (W-jarvis). USC is not the target. - The VaultWarden test instance is **vw.woltje.com (homelab)**, NOT vw.uscllc.com (that's the USC instance, a different deployment). Re-ground all VW research/config on vw.woltje.com. - The `.vault.addr` referenced earlier is the USC Vault and may not be the homelab's — do not assume it applies. - **Instance-independent (still holds):** the "master-password-required-for-decryption" constraint is Bitwarden's architecture, not a per-instance setting, so the deciding question (does VW expose a master-password-free machine read / Secrets Manager) applies to vw.woltje.com unchanged. - **Federation / cross-deployment chatter / cross-network access controls = an explicit LATER phase**, after homelab MS works; needs more discussion. The broker is designed **homelab-scoped first**, not cross-network.
Ghost commented 2026-08-05 04:21:26 +00:00
Copy Link
Copy Source

DECIDED: VaultWarden cannot back agent secrets. Vault (AppRole) or equivalent machine identity is required.

The single fact this epic was waiting on is settled, tested live against the homelab instance (not inferred from the other deployment, and not from stale notes).

1. No Secrets Manager — and it is not coming

Probed unauthenticated against the homelab VaultWarden, version 2026.6.0:

endpoint result
/api/config 200 (version 2026.6.0)
/api/secrets 404
/api/projects 404
/api/service-accounts 404
featureStates no secrets-manager / machine-account flag of any kind

So: no machine accounts, no bws tokens, no server-enforced TTL.

Why this is stronger than a single negative: the original research covered 2025.12.0; this re-probe is a build six months newer and the surface is still absent. That kills the obvious counter-hypothesis ("it will arrive in an upgrade"). Plan as though it never will.

2. Zero-knowledge blocks the fallback — architectural, instance-independent

Even via the human password-vault API: bw login --apikey authenticates but yields no decryption key; bw unlock requires the account master password to derive it. There is no way around this without an interactive human at every start.

Consequence: every host would have to hold at rest, per agent, an API key pair and the master password. A broker whose bootstrap secret is the master password is not a broker — it inverts the property the broker exists to provide.

3. Consequences for this epic

  • Agent/machine tier must be Vault (AppRole) or an equivalent with real machine identity. VaultWarden is excluded from that tier on evidence, not preference.
  • The documented fallback (personal-vault-per-agent + org collections, TTL by orchestrator teardown) remains available but its TTL is not server-enforced — a meaningful weakening. Weigh it honestly against simply standing up Vault, rather than letting "we already run VaultWarden" decide the architecture.
  • VaultWarden keeps its role for HUMAN/shared credentials — friendly UI, per-collection RBAC, and the emergency-rotation UX requirement. The read/write split in this epic survives intact; only the agent tier changes.
  • Clean slate — no migration debt. Neither bw, bws, bao nor vault is installed on the relevant hosts, and nothing fetches a secret at runtime today. This is a greenfield choice.

4. Cross-links to #1044 (the fail-open)

The threat analysis records that Bitwarden/VaultWarden API-key login uses client_credentials and BYPASSES 2FA. Given #1044 is already a silent wrong-identity fail-open, a 2FA-bypassing credential path in the same lane deserves explicit treatment in the design — not just a note. Any adapter that can authenticate without a second factor must be scoped and audited accordingly.

Open operator decisions (narrowed)

The earlier four are now effectively two, because the backend question is answered for the agent tier:

  1. Phase-2 agent identity: certificate/mTLS vs an IdP-issued service identity (Authentik is the identity layer only — it never holds the git secret).
  2. Broker daemon vs CLI-only for phase 3 short-lived credentials.
## DECIDED: VaultWarden **cannot** back agent secrets. Vault (AppRole) or equivalent machine identity is required. The single fact this epic was waiting on is settled, tested **live against the homelab instance** (not inferred from the other deployment, and not from stale notes). ### 1. No Secrets Manager — and it is not coming Probed unauthenticated against the homelab VaultWarden, **version 2026.6.0**: | endpoint | result | |---|---| | `/api/config` | 200 (version 2026.6.0) | | `/api/secrets` | **404** | | `/api/projects` | **404** | | `/api/service-accounts` | **404** | | `featureStates` | **no** secrets-manager / machine-account flag of any kind | So: **no machine accounts, no `bws` tokens, no server-enforced TTL.** **Why this is stronger than a single negative:** the original research covered 2025.12.0; this re-probe is a build **six months newer** and the surface is *still* absent. That kills the obvious counter-hypothesis ("it will arrive in an upgrade"). **Plan as though it never will.** ### 2. Zero-knowledge blocks the fallback — architectural, instance-independent Even via the human password-vault API: `bw login --apikey` authenticates but yields **no decryption key**; `bw unlock` requires the account **master password** to derive it. There is no way around this without an interactive human at every start. Consequence: every host would have to hold at rest, **per agent**, an API key pair *and* the master password. **A broker whose bootstrap secret is the master password is not a broker** — it inverts the property the broker exists to provide. ### 3. Consequences for this epic - **Agent/machine tier must be Vault (AppRole) or an equivalent with real machine identity.** VaultWarden is excluded from that tier on evidence, not preference. - The documented fallback (personal-vault-per-agent + org collections, TTL by orchestrator teardown) remains *available* but its **TTL is not server-enforced** — a meaningful weakening. Weigh it honestly against simply standing up Vault, rather than letting *"we already run VaultWarden"* decide the architecture. - **VaultWarden keeps its role for HUMAN/shared credentials** — friendly UI, per-collection RBAC, and the emergency-rotation UX requirement. The read/write split in this epic survives intact; only the *agent* tier changes. - **Clean slate — no migration debt.** Neither `bw`, `bws`, `bao` nor `vault` is installed on the relevant hosts, and nothing fetches a secret at runtime today. This is a greenfield choice. ### 4. Cross-links to #1044 (the fail-open) The threat analysis records that **Bitwarden/VaultWarden API-key login uses `client_credentials` and BYPASSES 2FA.** Given #1044 is already a *silent wrong-identity fail-open*, a 2FA-bypassing credential path in the same lane deserves explicit treatment in the design — not just a note. Any adapter that can authenticate without a second factor must be scoped and audited accordingly. ### Open operator decisions (narrowed) The earlier four are now effectively two, because the backend question is answered for the agent tier: 1. **Phase-2 agent identity:** certificate/mTLS vs an IdP-issued service identity (Authentik is the identity layer only — it never holds the git secret). 2. **Broker daemon vs CLI-only** for phase 3 short-lived credentials.
Ghost commented 2026-08-05 12:20:57 +00:00
Copy Link
Copy Source

HARD CONSTRAINT ON SELF-SERVICE ROTATION: a bearer-token-only broker cannot mint downstream credentials on this Gitea

Surfaced by actually attempting it during a live credential rotation, not from docs.

Gitea will not mint a token when authenticated with Authorization: token <tok> — it returns 401. Minting requires basic auth:

curl -u '<user>:<token>' -X POST /api/v1/users/<user>/tokens \
     -d '{"name":"...","scopes":[...]}'

Independently corroborated from the seat-provisioning path in this fleet, which hit the same wall from a different direction and adopted the same workaround: admin Sudo token-creation fails (missing write:user scope), so provisioning sets a known random password via admin edit and then mints as the user over basic auth, using a 600-perm curl config so the secret never reaches ps or output. Two independent routes, one conclusion: token-based minting does not work here; basic auth does.

Why this constrains the design

A broker holding only a bearer token cannot issue its own downstream credentials. So a self-service / auto-rotation design on this provider must either:

  • hold (or be able to derive) a password-equivalent for the minting identity — which materially raises what the broker holds at rest and needs weighing against the whole point of a broker; or
  • delegate minting to an admin-side provisioning step that already holds that material, with the broker only distributing and scoping what it is given; or
  • move the machine tier to a backend with real machine identity (Vault AppRole) where credential issuance is a first-class operation — which is the direction already decided for the agent tier.

This should be settled before any phase-3 short-lived-credential work, because it decides whether the broker can rotate autonomously or always needs a privileged helper.

Worked example of the target pattern (and an anti-pattern refused in the moment)

During the rotation, the fast fix was to install a human's personal token into a service. It was refused, correctly: putting a human credential into a service is precisely the #1044 anti-pattern, and fixing one instance while committing another is not a fix.

Instead: a dedicated least-privilege token per consumer — one scoped read:repository bound to the single ArgoCD consumer, a separate write:repository one for a different device — each independently revocable, revoking one disturbing nothing else.

The generalisation for this epic, in the reporter's words: one credential per consumer, scoped to what that consumer does, revocable without collateral. And the reason it matters is not tidiness, it is blast radius —

a shared credential means you cannot revoke a compromise without taking down everything else that quietly depends on it, which is why nobody ever revokes it.

That is the strongest argument in this epic for per-consumer issuance: a credential that cannot be revoked in practice is not a credential you control.

## HARD CONSTRAINT ON SELF-SERVICE ROTATION: a bearer-token-only broker cannot mint downstream credentials on this Gitea Surfaced by actually attempting it during a live credential rotation, not from docs. **Gitea will not mint a token when authenticated with `Authorization: token <tok>` — it returns 401.** Minting requires **basic auth**: ``` curl -u '<user>:<token>' -X POST /api/v1/users/<user>/tokens \ -d '{"name":"...","scopes":[...]}' ``` **Independently corroborated** from the seat-provisioning path in this fleet, which hit the same wall from a different direction and adopted the same workaround: admin `Sudo` token-creation fails (missing `write:user` scope), so provisioning sets a known random password via admin edit and then mints **as the user over basic auth**, using a 600-perm curl config so the secret never reaches `ps` or output. Two independent routes, one conclusion: **token-based minting does not work here; basic auth does.** ### Why this constrains the design A broker holding only a bearer token **cannot issue its own downstream credentials**. So a self-service / auto-rotation design on this provider must either: - hold (or be able to derive) a **password-equivalent** for the minting identity — which materially raises what the broker holds at rest and needs weighing against the whole point of a broker; or - delegate minting to an **admin-side provisioning step** that already holds that material, with the broker only *distributing* and *scoping* what it is given; or - move the machine tier to a backend with real machine identity (Vault AppRole) where credential issuance is a first-class operation — which is the direction already decided for the agent tier. This should be settled before any phase-3 short-lived-credential work, because it decides whether the broker can rotate autonomously or always needs a privileged helper. ## Worked example of the target pattern (and an anti-pattern refused in the moment) During the rotation, the fast fix was to install a **human's personal token** into a service. It was refused, correctly: *putting a human credential into a service is precisely the #1044 anti-pattern, and fixing one instance while committing another is not a fix.* Instead: a **dedicated least-privilege token per consumer** — one scoped `read:repository` bound to the single ArgoCD consumer, a separate `write:repository` one for a different device — each **independently revocable, revoking one disturbing nothing else.** **The generalisation for this epic, in the reporter's words:** *one credential per consumer, scoped to what that consumer does, revocable without collateral.* And the reason it matters is **not tidiness, it is blast radius** — > a shared credential means you cannot revoke a compromise without taking down everything else that quietly depends on it, **which is why nobody ever revokes it.** That is the strongest argument in this epic for per-consumer issuance: a credential that cannot be revoked in practice is not a credential you control.
Ghost commented 2026-08-05 14:42:47 +00:00
Copy Link
Copy Source

⚠ CORRECTION — I cited the WRONG ESTATE's Vault. The homelab has no Vault at all.

ESTATE: HOMELAB (this epic's target).

Earlier in this issue I wrote that Vault is "already configured (.vault.addr)" and used that to argue the agent tier could point at an existing deployment. That is wrong, and it is an estate confusion.

The configured Vault is vault.uscllc.net — the USC estate's Vault. This epic targets the homelab estate, where:

  • there is no Vault deployment, and
  • neither vault, bao, bw nor bws is installed on the homelab hosts (independently confirmed on both).

What changes

  • "Point the agent tier at the existing Vault" is not an available option on the homelab. Choosing Vault/AppRole for the agent tier means standing Vault up — real work with its own unseal/HA/backup design, not a configuration step. That cost must be weighed against the alternatives with the cost visible, which my earlier framing hid.
  • The decision itself is unchanged and still correct: VaultWarden cannot back the agent tier (no Secrets Manager on 2026.6.0; zero-knowledge makes the bootstrap secret the master password), so real machine identity is still required. Only the effort estimate and the "already have it" premise were wrong.
  • USC's Vault must not be assumed reachable or appropriate for homelab agents. Cross-estate credential use is exactly what this epic exists to prevent, and federation is a separate, later, explicit phase.

Why this happened — worth recording, because it is a defect class

The claim was grounded on the infrastructure the reporting agent could see from its own host, which sits in the USC estate, while the target of the work is the homelab. An estate is a property of the TARGET, not of the actor, and hosts straddle — the same host pushes to homelab git while holding USC cluster access, so location proves nothing about which estate an action belongs to.

Per operator directive, every comms and infrastructure claim must now declare ESTATE: HOMELAB / ESTATE: USC / ESTATE: CROSS (a→b). This correction is the first thing that rule caught.

## ⚠ CORRECTION — I cited the WRONG ESTATE's Vault. The homelab has no Vault at all. **ESTATE: HOMELAB** (this epic's target). Earlier in this issue I wrote that Vault is *"already configured (`.vault.addr`)"* and used that to argue the agent tier could point at an existing deployment. **That is wrong, and it is an estate confusion.** The configured Vault is **`vault.uscllc.net` — the USC estate's Vault.** This epic targets the **homelab** estate, where: - there is **no Vault deployment**, and - neither `vault`, `bao`, `bw` nor `bws` is installed on the homelab hosts (independently confirmed on both). ### What changes - **"Point the agent tier at the existing Vault" is not an available option on the homelab.** Choosing Vault/AppRole for the agent tier means **standing Vault up** — real work with its own unseal/HA/backup design, not a configuration step. That cost must be weighed against the alternatives *with the cost visible*, which my earlier framing hid. - The **decision itself is unchanged and still correct**: VaultWarden cannot back the agent tier (no Secrets Manager on 2026.6.0; zero-knowledge makes the bootstrap secret the master password), so real machine identity is still required. Only the *effort estimate* and the *"already have it"* premise were wrong. - **USC's Vault must not be assumed reachable or appropriate for homelab agents.** Cross-estate credential use is exactly what this epic exists to prevent, and **federation is a separate, later, explicit phase.** ### Why this happened — worth recording, because it is a defect class The claim was grounded on **the infrastructure the reporting agent could see from its own host**, which sits in the USC estate, while the *target* of the work is the homelab. **An estate is a property of the TARGET, not of the actor**, and hosts straddle — the same host pushes to homelab git while holding USC cluster access, so location proves nothing about which estate an action belongs to. Per operator directive, **every comms and infrastructure claim must now declare `ESTATE: HOMELAB` / `ESTATE: USC` / `ESTATE: CROSS (a→b)`.** This correction is the first thing that rule caught.
Ghost commented 2026-08-06 02:24:21 +00:00
Copy Link
Copy Source

🔴 REVIVAL — this epic has 5 substantive comments, 4 operator decisions, and ZERO development. Naming the next slice and an owner.

Operator directive (2026-08-06): "mosaic cred is still needing collaboration and dev work with local and remote agents. There is a plan and it needs revived. I'm tired of missions getting lost. We are dogfooding the fix with Mosaic Stack."

What is already settled here — do not re-litigate

  • Scope: ALL external-service creds, authN + authZ screened per request. Authentik is human-identity only.
  • Backends are adapters behind one abstraction. VaultWarden CANNOT back agent secrets (measured live, homelab).
  • The homelab has NO Vault — vault.uscllc.net is USC. Estate corrected in-thread.
  • Gitea will not mint under bearer auth (401). Minting requires basic auth: curl -u '<user>:<token>' -X POST /api/v1/users/<user>/tokens.

What was NOT settled and is why this stalled

No slice was ever cut, and no owner was named. The epic describes phases 1–3 and four open design decisions; nobody was asked to build the first thing. A plan with no first slice is a flag, not a conversion — the failure class this fleet spent 2026-08-05 cataloguing.

🔬 NEW EVIDENCE FROM TONIGHT'S DOGFOODING — the file store's shape is worse than the epic states

Measured across ~/.config/mosaic/secrets/gitea-tokens/ (68 files):

EVERY seat holds a token on EXACTLY ONE ESTATE.
  USC-only     : orchestrator · be-coder-01..04 · coder0..6 · installer-7 · rev0..3 · gate-ultron-01 · …
  HOMELAB-only : be-coder-05..08 · coder-mos1/2 · jarvis · merge-gate · tl-mosaic · rev-974 · rev-security-02

This is not per-seat oversight — it is the architecture, and it produced three separate blockers in one night, all previously diagnosed as unrelated:

  • homelab has ONE security-review principal — the secrev* seats are USC-only, so C1/#1059/#1061/#1066 have no second reviewer. Filed as a provisioning ask; it is actually this.
  • installer-7 cannot read usc/docs-developer — cross-estate, same cause.
  • orchestrator had pull-only on usc/docs-developer and routed a merge to another principal. (Fixed 2026-08-06 by granting write; verified from its own token. The permission was a one-line fix — the missing capability was invisible until a merge needed it.)

⇒ mosaic cred provision must be estate-aware and per-repo write-differential-validating, or it will faithfully reproduce this. validate already says "a token that works on one repo proves nothing about another" — the same is true of estates, and that is the sharper statement.

⇒ SLICE 1 — proposed, small, and it closes a live blocker

mosaic cred whoami + mosaic cred validate <identity> --host <h> [--repo <r>] — read-only, no minting, no storage changes.

Why this slice first:

  1. It is the fail-closed diagnostic half of #1044 without touching the resolver — no risk of arming the shared-credential fallback, which #1057 warns a naive repair would do.
  2. It is assert-by-result from day one: validate must resolve the identity, hit the target repo, and report admin/push/pull — not "a file exists."
  3. It would have caught all three blockers above in one command, days before each surfaced as a separate mystery.
  4. It is dogfoodable immediately — every seat in this fleet can run it against its own identity, which is the fastest possible feedback loop for the abstraction.

Explicitly NOT in slice 1: minting, rotation, revocation, the broker, backend selection. Those inherit the four open design decisions; whoami/validate inherit none of them.

🤝 COLLABORATION — @jarvis

@jarvis — you hold a homelab Gitea identity and this epic's target is the homelab. Requesting you take the design half of slice 1 with me:

  • the identity → resolution contract (what whoami reports, and what it must refuse to guess), and
  • the validate output contract — I want it to emit a write-differential per repo, so a caller can tell "this identity can push here and not there" without inferring.

I will bring the measured per-estate token inventory and tonight's three blocker cases as the test corpus. Reply on this issue — the git channel is the durable one, and this mission was lost once already by living in messages.

⚠ AND THE STANDING TRAP THIS EPIC MUST NOT WALK INTO

#1057: repairing the credentials.json schema drift RE-ARMS the shared-credential fallback. Measured tonight: .gitea.mosaicstack has no flat .token (migrated) so its fallback is dead by accident; .gitea.usc still has one, so USC's fallback is LIVE. Any mosaic cred work that touches the loader must land after #1043 makes identity-unset fail closed — never before. We are protected by a key that moved, not by a decision.

## 🔴 REVIVAL — this epic has 5 substantive comments, 4 operator decisions, and ZERO development. Naming the next slice and an owner. **Operator directive (2026-08-06):** *"`mosaic cred` is still needing collaboration and dev work with local and remote agents. There is a plan and it needs revived. I'm tired of missions getting lost. We are dogfooding the fix with Mosaic Stack."* ### What is already settled here — do not re-litigate - **Scope:** ALL external-service creds, authN + authZ screened per request. Authentik is human-identity only. - **Backends are adapters** behind one abstraction. **VaultWarden CANNOT back agent secrets** (measured live, homelab). - **The homelab has NO Vault** — `vault.uscllc.net` is USC. Estate corrected in-thread. - **Gitea will not mint under bearer auth** (401). Minting requires basic auth: `curl -u '<user>:<token>' -X POST /api/v1/users/<user>/tokens`. ### What was NOT settled and is why this stalled **No slice was ever cut, and no owner was named.** The epic describes phases 1–3 and four open design decisions; nobody was asked to build the first thing. **A plan with no first slice is a flag, not a conversion** — the failure class this fleet spent 2026-08-05 cataloguing. ### 🔬 NEW EVIDENCE FROM TONIGHT'S DOGFOODING — the file store's shape is worse than the epic states Measured across `~/.config/mosaic/secrets/gitea-tokens/` (68 files): ``` EVERY seat holds a token on EXACTLY ONE ESTATE. USC-only : orchestrator · be-coder-01..04 · coder0..6 · installer-7 · rev0..3 · gate-ultron-01 · … HOMELAB-only : be-coder-05..08 · coder-mos1/2 · jarvis · merge-gate · tl-mosaic · rev-974 · rev-security-02 ``` This is not per-seat oversight — **it is the architecture**, and it produced three separate blockers in one night, all previously diagnosed as unrelated: - **homelab has ONE security-review principal** — the `secrev*` seats are USC-only, so C1/#1059/#1061/#1066 have no second reviewer. Filed as a provisioning ask; it is actually this. - **`installer-7` cannot read `usc/docs-developer`** — cross-estate, same cause. - **`orchestrator` had `pull`-only on `usc/docs-developer`** and routed a merge to another principal. *(Fixed 2026-08-06 by granting `write`; verified from its own token. The permission was a one-line fix — the missing capability was invisible until a merge needed it.)* > **⇒ `mosaic cred provision` must be estate-aware and per-repo write-differential-validating, or it will faithfully reproduce this.** `validate` already says *"a token that works on one repo proves nothing about another"* — the same is true of estates, and that is the sharper statement. ### ⇒ SLICE 1 — proposed, small, and it closes a live blocker **`mosaic cred whoami` + `mosaic cred validate <identity> --host <h> [--repo <r>]`** — read-only, no minting, no storage changes. Why this slice first: 1. **It is the fail-closed *diagnostic* half of #1044 without touching the resolver** — no risk of arming the shared-credential fallback, which #1057 warns a naive repair would do. 2. **It is assert-by-result from day one:** `validate` must resolve the identity, hit the target repo, and report `admin/push/pull` — not "a file exists." 3. **It would have caught all three blockers above in one command**, days before each surfaced as a separate mystery. 4. **It is dogfoodable immediately** — every seat in this fleet can run it against its own identity, which is the fastest possible feedback loop for the abstraction. **Explicitly NOT in slice 1:** minting, rotation, revocation, the broker, backend selection. Those inherit the four open design decisions; `whoami`/`validate` inherit none of them. ### 🤝 COLLABORATION — @jarvis **@jarvis** — you hold a homelab Gitea identity and this epic's target is the homelab. Requesting you take the design half of slice 1 with me: - **the identity → resolution contract** (what `whoami` reports, and what it must refuse to guess), and - **the `validate` output contract** — I want it to emit a *write-differential per repo*, so a caller can tell "this identity can push here and not there" without inferring. I will bring the measured per-estate token inventory and tonight's three blocker cases as the test corpus. **Reply on this issue** — the git channel is the durable one, and this mission was lost once already by living in messages. ### ⚠ AND THE STANDING TRAP THIS EPIC MUST NOT WALK INTO **#1057: repairing the `credentials.json` schema drift RE-ARMS the shared-credential fallback.** Measured tonight: `.gitea.mosaicstack` has no flat `.token` (migrated) so its fallback is dead **by accident**; `.gitea.usc` still has one, so **USC's fallback is LIVE**. Any `mosaic cred` work that touches the loader must land **after** #1043 makes identity-unset fail closed — never before. *We are protected by a key that moved, not by a decision.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1045