stack/packages/model-proxy/README.md

Model proxy (P1 #1827, P2 #1829)

The bus initializes and manages one pinned CLIProxyAPI per instance. PIN records release URLs and independently reviewable SHA256s of the archive, executable and panel. mosaic proxy fetch verifies both downloads before installing under <dataRoot>/opt/cliproxyapi/<version>/; there is no runtime panel update.

From a human terminal, with the business bus running:

mosaic proxy fetch
mosaic proxy init [--port N] [--business ID]

Init prompts twice without echo and passes the key to the bus through its path-bound human proof. An agent, tester, redirected stdin or mismatched confirmation refuses. The management key must contain 1–4096 printable ASCII characters with no spaces (bytes 0x21–0x7e). The bus exclusively creates proxy.json beside config.json, the private management-key file and initial proxy directory. It never rewrites config.json. Init does not install/start a service. The validated system config contains proxy.port and proxy.binary only when proxy.json exists.

The management client is imported by the bus; it only GETs/PUTs the v8 list on 127.0.0.1, never follows redirects, and emits receipts with SHA256 key IDs only. Client-key values are never returned by an agent-facing bus verb. The bus manages per-agent lifecycle below; initialization creates only the sentinel.

The unit starts through a hash-checking runner with the pinned panel directory and -local-model, so catalogs require no network. Render/install is explicit; no test or acceptance script installs or starts a host unit.

Acceptance is agents/magica/work/queue-329/acceptance.sh, outside the generic gate. It uses a fresh scratch root and private network namespace. The historical v1.26.7 panel sandbox findings remain in BUILD.md. Q250 / mos-1606 drops the Console panel forward; P4 builds a native Console page. P1 retains the pinned panel asset and records Claude/Codex OAuth redirect URLs for that page.

A stale .proxy-init.lock has no automatic expiry. Only the human may clear it, after stopping this instance's bus user unit so it cannot restart and confirming that no initialization operation is active. Check the selected config/data root: if proxy.json, secrets/proxy-management.key or <dataRoot>/proxy/ exists, retain that state and resolve the partial/completed initialization separately; do not delete it to make init run again. If none exists and the bus is stopped, remove only the stale lock, substituting the actual selected config directory:

rm -- "<configDir>/.proxy-init.lock"

Then start the bus normally and retry init from the human terminal. If the bus or initialization state is uncertain, leave the lock in place. A refused fetch may leave empty opt/cliproxyapi/ parent directories; they are not an installed version and need not be removed before retrying.

PIN upgrades must carry the native Console page's offline upgrade receipt. Run bash agents/poe/work/queue-330/acceptance.sh <verified-fetched-executable> under the test-slot rules, and include its output for the new PIN version in that upgrade row's BUILD.md. Sage's landing gate checks the receipt. The script uses a new temp root and private network namespace, with made-up credentials, and calls the page's bus verbs. If the page cannot pass against the new PIN, the upgrade stops; Sage takes Q46's fallback to Jason through Mos.

The native /proxy page uses human-only bus verbs; only the bus holds the management client. On PIN 8.0.42, OAuth credentials are listed under /v8/management/credentials and priority is patched as {name, priority}. API-key groups are absent from that list: z.ai keys use the Claude-compatible provider configuration at /v8/management/config/api-keys/claude, whose keys carry proxy-generated auth_index identifiers. Their priority updates use provider configuration and a read-back check. No upstream API-key value is returned by the bus.

P2 adds an implicit model credential for every agent in proxy mode. The bus confirms the v8 key-list write before storing a key at <configDir>/secrets/agents/<business>/<agent>/model (0600, directories 0700). The private file holds the key and its lifecycle metadata. Status and receipts contain key IDs, issue time, rotation date and state; they never contain values. No role or business credentials.model reference is needed or accepted.

Rotation defaults to 30 days. New launches use the replacement key. The old key stays until its runs end or its grace expires; a failed bounded session stop keeps both the old key and its retirement metadata for the next check. Revoke requires a confirming management GET and polling /v1/models to 401 before local cleanup. Polling has a five-second total deadline and 25ms intervals. A durable revoking state blocks launches before preflight, and boot reconciliation finishes it without restoring that key. Reconciliation preserves foreign keys and refuses launches when this instance's sentinel is absent.

For a new instance only, human proxy init also accepts --rotate-grace-minutes N (1–1440, default 120) and --first-output-seconds N (1–3600, default 120). These are optional fields in proxy.json; existing two-field sidecars remain valid. There is no per-agent rotation-period setter in P2. Fetch also refuses agent-marked and tester calls before reading config or touching the network/disk.

The trusted launcher requests modelPreflight with business/role and receives exactly { baseUrl, key }. That answer never passes through an agent socket or status API. A 401 or other non-success HTTP status refuses proxy-key-refused; a failed connection refuses proxy-down. No model response activity by the configured deadline requests a bounded stop and records proxy-stall with key IDs. The runner writes only a fixed token to <run>/proxy-first-output, never stdout contents. Managed adapters also publish this token on trusted streamed model response events (including thinking and tool-use), before their answer-only stdout. Startup status, tool results and workspace file writes do not count. A stop failure records/refuses proxy-stop-failed and retains the key.

P5 adds the bus-only readUsageQueue() management operation. Its fixed GET /v8/management/observability/usage/queue?count=100 retains the client's existing response-size/deadline/loopback controls. It validates the entire array (maximum 100 events), hashes client api_key and upstream source to 12 hex IDs at the read boundary, and returns only safe IDs, counts and failure metadata. Bodies, credential values and upstream error messages never enter receipts.

Shared proxy for organization buses

Organization buses on one instance share the proxy using separate mso- key namespaces. Their proxy management writes cooperate through an instance-wide lock. A write with an uncertain outcome leaves a durable fence: proxy-dependent launches and management mutations refuse until recovery, and catalogues report unavailable. Check the fence with mosaic proxy fence status --json. After actually restarting the proxy process so no old request can still arrive, a human can run mosaic proxy fence clear --proxy-restarted. The command reads the management list under the lock and records a receipt; each organization's next start reconciles its own namespace. Recovery also invalidates cached catalogues. The flag is the operator's statement that the restart occurred; the command cannot verify it.

The lock is cooperative among this release's buses on the same instance. Older buses, the proxy's own panel, other holders of the management key, and the human proxy fetch and proxy init tools do not participate. Such writers can still race a whole-list write or alter proxy state. A server-side fence is a follow-up; do not run those writers concurrently with organization buses.

Setup from the Console (#1873)

Settings > Model proxy has a Proxy setup panel for the instance owner: the --dev-local-owner owner on a dev Console, or the root owner under --sso. Agents, testers and grant holders get human-required on every setup route, read or write. The panel shows the release (pinned version, hashes verified), initialization, the unit (installed, active, pid, port), the fence, the init lock, each organization's proxy (shared) and catalogue, and own proxy. Each state offers its next step as one button. CLI parity:

mosaic proxy status [--json]
mosaic proxy fetch
mosaic proxy init --generate-key [--port N] [--rotate-grace-minutes N] [--first-output-seconds N] --business ID
mosaic proxy unit render|status|install|start|stop|restart
mosaic proxy fence clear --restart --business ID
mosaic proxy init-lock status|clear

• Fetch runs the same PIN and hash checks as mosaic proxy fetch; the page shows each step as it runs.
• Init asks the bus to generate the management key (proxy.init with generate: true, 48 CSPRNG bytes). The key goes to the key file only; no response, event or receipt carries it. A pasted key with its confirmation is accepted only when the page is on HTTPS or on loopback from a loopback peer; a plain-HTTP LAN page gets the generated key only (proxy-key-transport). Init keeps its exclusive-create rules and never rewrites config.json. Restart the other buses afterwards.
• The unit verbs install the same user unit as scripts/proxy-service.sh. Health is the unit's main process tree listening on the configured port.
• fence clear --restart (and the page's fence step) restarts the unit, confirms a new process that started after the fence and listens on the port, then clears the fence with that evidence under the lock. The receipt holds the before and after pids. Without a confirmed restart it refuses (proxy-restart-unconfirmed). A fence file with no readable time also refuses, because a restart cannot be shown to come after it; restart the service yourself and use fence clear --proxy-restarted, which is unchanged.
• proxy-admin-busy (one setup action at a time) is per process: the Console and a mosaic proxy command each hold their own. The writes that matter are still serialized across processes: init by the bus's init lock and exclusive create, key writes by the instance-wide proxy lock, and unit actions by systemd.
• The page and status --json carry the last ten receipts from proxy-admin.jsonl, so they survive a reload. CLI init --generate-key writes the same init receipt as the page.
• The init lock clears only when every bus on the instance is stopped and no proxy.json, key file or proxy state directory exists. Otherwise the panel and init-lock status list why not.
• Own proxy per organization is not built: the configuration has one proxy per instance (ruling 8, deferred).

Receipts: <dataRoot>/state/proxy-admin.jsonl (labels, pids and port only).