jason.woltje
75f6d5533a
Security review: - Release verification cache keys on ctime as well as mtime, in nanoseconds. A same-size rewrite with mtime set back now re-hashes and reports proxy-install-invalid. Build review: - The setup page hides the account panels on any failed accounts read, not only proxy-down, and brings them back once the proxy answers. - The organizations table shows a dash, not a made-up 0, when a bus does not answer or sends no model list (orgCatalogue in proxy-admin.mjs). - A fence clear the bus did not answer keeps the confirmed restart for a retry. - A fence with no readable time names `mosaic proxy fence clear --proxy-restarted` on the page, in the refusal table and in the README. - CLI `init --generate-key` goes through the setup controller and writes the same init receipt as the page. - Status carries the last ten receipts, so the page shows them after a reload. An initialized proxy keeps the init form folded. - README: proxy-admin-busy is per process; the cross-process locks that serialize the writes are named.
179 lines
11 KiB
Markdown
179 lines
11 KiB
Markdown
# Model proxy (P1 #1827, P2 #1829)
|
||
|
||
The bus initializes and manages one pinned CLIProxyAPI per instance. PIN records
|
||
release URLs and independently reviewable SHA256s of the archive, executable and
|
||
panel. `mosaic proxy fetch` verifies both downloads before installing under
|
||
`<dataRoot>/opt/cliproxyapi/<version>/`; there is no runtime panel update.
|
||
|
||
From a human terminal, with the business bus running:
|
||
|
||
mosaic proxy fetch
|
||
mosaic proxy init [--port N] [--business ID]
|
||
|
||
Init prompts twice without echo and passes the key to the bus through its
|
||
path-bound human proof. An agent, tester, redirected stdin or mismatched
|
||
confirmation refuses. The management key must contain 1–4096 printable ASCII
|
||
characters with no spaces (bytes 0x21–0x7e). The bus exclusively creates proxy.json beside config.json,
|
||
the private management-key file and initial proxy directory. It never rewrites
|
||
config.json. Init does not install/start a service. The validated system config
|
||
contains `proxy.port` and `proxy.binary` only when proxy.json exists.
|
||
|
||
The management client is imported by the bus; it only GETs/PUTs the v8 list on
|
||
127.0.0.1, never follows redirects, and emits receipts with SHA256 key IDs only.
|
||
Client-key values are never returned by an agent-facing bus verb. The bus manages
|
||
per-agent lifecycle below; initialization creates only the sentinel.
|
||
|
||
The unit starts through a hash-checking runner with the pinned panel directory
|
||
and `-local-model`, so catalogs require no network. Render/install is explicit;
|
||
no test or acceptance script installs or starts a host unit.
|
||
|
||
Acceptance is agents/magica/work/queue-329/acceptance.sh, outside the generic gate.
|
||
It uses a fresh scratch root and private network namespace. The historical
|
||
v1.26.7 panel sandbox findings remain in BUILD.md. Q250 / mos-1606 drops the
|
||
Console panel forward; P4 builds a native Console page. P1 retains the pinned
|
||
panel asset and records Claude/Codex OAuth redirect URLs for that page.
|
||
|
||
A stale `.proxy-init.lock` has no automatic expiry. Only the human may clear it,
|
||
after stopping this instance's bus user unit so it cannot restart and confirming
|
||
that no initialization operation is active. Check the selected config/data root:
|
||
if `proxy.json`, `secrets/proxy-management.key` or `<dataRoot>/proxy/` exists,
|
||
retain that state and resolve the partial/completed initialization separately;
|
||
do not delete it to make init run again. If none exists and the bus is stopped,
|
||
remove only the stale lock, substituting the actual selected config directory:
|
||
|
||
rm -- "<configDir>/.proxy-init.lock"
|
||
|
||
Then start the bus normally and retry init from the human terminal. If the bus
|
||
or initialization state is uncertain, leave the lock in place. A refused fetch
|
||
may leave empty `opt/cliproxyapi/` parent directories; they are not an installed
|
||
version and need not be removed before retrying.
|
||
|
||
PIN upgrades must carry the native Console page's offline upgrade receipt.
|
||
Run `bash agents/poe/work/queue-330/acceptance.sh <verified-fetched-executable>`
|
||
under the test-slot rules, and include its output for the new PIN version in
|
||
that upgrade row's BUILD.md. Sage's landing gate checks the receipt. The script
|
||
uses a new temp root and private network namespace, with made-up credentials,
|
||
and calls the page's bus verbs. If the page cannot pass against the new PIN,
|
||
the upgrade stops; Sage takes Q46's fallback to Jason through Mos.
|
||
|
||
The native `/proxy` page uses human-only bus verbs; only the bus holds the
|
||
management client. On PIN 8.0.42, OAuth credentials are listed under
|
||
`/v8/management/credentials` and priority is patched as `{name, priority}`.
|
||
API-key groups are absent from that list: z.ai keys use the Claude-compatible
|
||
provider configuration at `/v8/management/config/api-keys/claude`, whose keys
|
||
carry proxy-generated `auth_index` identifiers. Their priority updates use
|
||
provider configuration and a read-back check. No upstream API-key value is
|
||
returned by the bus.
|
||
|
||
P2 adds an implicit `model` credential for every agent in proxy mode. The bus
|
||
confirms the v8 key-list write before storing a key at
|
||
`<configDir>/secrets/agents/<business>/<agent>/model` (0600, directories 0700).
|
||
The private file holds the key and its lifecycle metadata. Status and receipts
|
||
contain key IDs, issue time, rotation date and state; they never contain values.
|
||
No role or business `credentials.model` reference is needed or accepted.
|
||
|
||
Rotation defaults to 30 days. New launches use the replacement key. The old key
|
||
stays until its runs end or its grace expires; a failed bounded session stop
|
||
keeps both the old key and its retirement metadata for the next check. Revoke
|
||
requires a confirming management GET and polling `/v1/models` to 401 before local
|
||
cleanup. Polling has a five-second total deadline and 25ms intervals. A durable
|
||
`revoking` state blocks launches before preflight, and boot reconciliation
|
||
finishes it without restoring that key. Reconciliation preserves foreign keys
|
||
and refuses launches when this instance's sentinel is absent.
|
||
|
||
For a new instance only, human `proxy init` also accepts
|
||
`--rotate-grace-minutes N` (1–1440, default 120) and
|
||
`--first-output-seconds N` (1–3600, default 120). These are optional fields in
|
||
proxy.json; existing two-field sidecars remain valid. There is no per-agent
|
||
rotation-period setter in P2. Fetch also refuses agent-marked and tester calls
|
||
before reading config or touching the network/disk.
|
||
|
||
The trusted launcher requests `modelPreflight` with business/role and receives
|
||
exactly `{ baseUrl, key }`. That answer never passes through an agent socket or
|
||
status API. A 401 or other non-success HTTP status refuses `proxy-key-refused`;
|
||
a failed connection refuses `proxy-down`. No model response activity by the configured deadline
|
||
requests a bounded stop and records `proxy-stall` with key IDs. The runner writes
|
||
only a fixed token to `<run>/proxy-first-output`, never stdout contents. Managed
|
||
adapters also publish this token on trusted streamed model response events
|
||
(including thinking and tool-use), before their answer-only stdout. Startup
|
||
status, tool results and workspace file writes do not count. A stop
|
||
failure records/refuses `proxy-stop-failed` and retains the key.
|
||
|
||
P5 adds the bus-only `readUsageQueue()` management operation. Its fixed
|
||
`GET /v8/management/observability/usage/queue?count=100` retains the client's
|
||
existing response-size/deadline/loopback controls. It validates the entire array
|
||
(maximum 100 events), hashes client `api_key` and upstream `source` to 12 hex IDs
|
||
at the read boundary, and returns only safe IDs, counts and failure metadata.
|
||
Bodies, credential values and upstream error messages never enter receipts.
|
||
|
||
### Shared proxy for organization buses
|
||
|
||
Organization buses on one instance share the proxy using separate `mso-`
|
||
key namespaces. Their proxy management writes cooperate through an
|
||
instance-wide lock. A write with an uncertain outcome leaves a durable fence:
|
||
proxy-dependent launches and management mutations refuse until recovery, and
|
||
catalogues report unavailable. Check the fence with `mosaic proxy fence status
|
||
--json`. After actually restarting the proxy process so no old request can
|
||
still arrive, a human can run `mosaic proxy fence clear --proxy-restarted`.
|
||
The command reads the management list under the lock and records a receipt;
|
||
each organization's next start reconciles its own namespace. Recovery also
|
||
invalidates cached catalogues. The flag is the operator's statement that the
|
||
restart occurred; the command cannot verify it.
|
||
|
||
The lock is cooperative among this release's buses on the same instance. Older
|
||
buses, the proxy's own panel, other holders of the management key, and the
|
||
human `proxy fetch` and `proxy init` tools do not participate. Such writers can
|
||
still race a whole-list write or alter proxy state. A server-side fence is a
|
||
follow-up; do not run those writers concurrently with organization buses.
|
||
|
||
### Setup from the Console (#1873)
|
||
|
||
Settings > Model proxy has a Proxy setup panel for the instance owner: the
|
||
`--dev-local-owner` owner on a dev Console, or the root owner under `--sso`.
|
||
Agents, testers and grant holders get `human-required` on every setup route,
|
||
read or write. The panel shows the release (pinned version, hashes verified),
|
||
initialization, the unit (installed, active, pid, port), the fence, the init
|
||
lock, each organization's proxy (shared) and catalogue, and own proxy. Each
|
||
state offers its next step as one button. CLI parity:
|
||
|
||
mosaic proxy status [--json]
|
||
mosaic proxy fetch
|
||
mosaic proxy init --generate-key [--port N] [--rotate-grace-minutes N] [--first-output-seconds N] --business ID
|
||
mosaic proxy unit render|status|install|start|stop|restart
|
||
mosaic proxy fence clear --restart --business ID
|
||
mosaic proxy init-lock status|clear
|
||
|
||
- Fetch runs the same PIN and hash checks as `mosaic proxy fetch`; the page
|
||
shows each step as it runs.
|
||
- Init asks the bus to generate the management key (`proxy.init` with
|
||
`generate: true`, 48 CSPRNG bytes). The key goes to the key file only; no
|
||
response, event or receipt carries it. A pasted key with its confirmation is
|
||
accepted only when the page is on HTTPS or on loopback from a loopback
|
||
peer; a plain-HTTP LAN page gets the generated key only
|
||
(`proxy-key-transport`). Init keeps its exclusive-create rules and never
|
||
rewrites `config.json`. Restart the other buses afterwards.
|
||
- The unit verbs install the same user unit as `scripts/proxy-service.sh`.
|
||
Health is the unit's main process tree listening on the configured port.
|
||
- `fence clear --restart` (and the page's fence step) restarts the unit,
|
||
confirms a new process that started after the fence and listens on the port,
|
||
then clears the fence with that evidence under the lock. The receipt holds the
|
||
before and after pids. Without a confirmed restart it refuses
|
||
(`proxy-restart-unconfirmed`). A fence file with no readable time also refuses,
|
||
because a restart cannot be shown to come after it; restart the service
|
||
yourself and use `fence clear --proxy-restarted`, which is unchanged.
|
||
- `proxy-admin-busy` (one setup action at a time) is per process: the Console
|
||
and a `mosaic proxy` command each hold their own. The writes that matter are
|
||
still serialized across processes: init by the bus's init lock and
|
||
exclusive create, key writes by the instance-wide proxy lock, and unit
|
||
actions by systemd.
|
||
- The page and `status --json` carry the last ten receipts from
|
||
`proxy-admin.jsonl`, so they survive a reload. CLI `init --generate-key`
|
||
writes the same init receipt as the page.
|
||
- The init lock clears only when every bus on the instance is stopped and no
|
||
`proxy.json`, key file or proxy state directory exists. Otherwise the panel
|
||
and `init-lock status` list why not.
|
||
- Own proxy per organization is not built: the configuration has one proxy
|
||
per instance (ruling 8, deferred).
|
||
|
||
Receipts: `<dataRoot>/state/proxy-admin.jsonl` (labels, pids and port only).
|