Compare commits

...
Author SHA1 Message Date
jason.woltje f65ca7bb73 Keep setup ready when one organization answers and verify Docker cancellation 2026-11-22 00:32:33 -06:00
jason.woltje 96e157382b Correct setup evidence and share repeated dashboard reads 2026-11-22 00:32:32 -06:00
jason.woltje 650a2d2890 Show setup checks from current system state on the dashboard 2026-11-22 00:32:32 -06:00
jason.woltje 66ef78d69e Give the stack-release rehearsal copies the model-proxy files config.mjs now loads
config.mjs imports the ride refusal lines from packages/model-proxy/src/settings.mjs, and its
common.mjs reads packages/model-proxy/PIN at load. The real stack copy already carries both
(COPY_PATHS). The test's hand-built rehearsal copies did not, so 22 cases failed scratch-config.
2026-11-21 22:45:57 -06:00
jason.woltje a0ec8e028c Fix the proxy-ride review findings
Review findings 1-4 (all P3). A ride file removed while validate runs now
refuses proxy-ride-invalid, not a TypeError stack. The config.mjs comment and
docs/TOOLS.md now say an owner problem refuses every verb on the rider while
proxy-ride.json exists, that the owner should run with its own config path
because the first side to lock fixes the lock identity, and that run-task.sh
still shows a ride refusal as a parse error. A new test fails if container-cli
stops passing ride: true: the stand-in launch step answers with the proxy it
was given.
2026-11-21 22:45:57 -06:00
jason.woltje 8c5764fd03 Let one instance's worker runs ride another instance's model proxy
Jason approved five items on 2026-11-21 (option C, Q4 first slice, worker
runs only). <configDir>/proxy-ride.json {"version":1,"instance":"NAME"}
names an instance of the same OS user. mosaic proxy ride --instance NAME
writes it and mosaic proxy ride --off removes it; both are human only.

The config loader resolves the owner's proxy.json through the owner's own
config and data root claim and adds proxy.owner. A worker run reads the
owner's management key and model on/off list, and writes its worker key
under the owner's lock, fence and state dir, with the rider's own key tag.
loadSystem drops a ridden proxy unless a worker run asks, so buses, the
Console and the proxy unit see no proxy for a rider.

New refusals: proxy-ride-invalid (bad file, a path, its own instance, a
shared data root), proxy-ride-owner-missing (no valid owner proxy, or the
owner rides too), proxy-ride-conflict (proxy.json and a ride file).
2026-11-21 22:45:57 -06:00
jason.woltje 6f0427024a Say every cause of an unknown worker model standing in the view's comment 2026-11-21 21:42:35 -06:00
jason.woltje 538fe12829 Read the config fresh for the Worker model card, and run its view alongside the status read
Review round for worker-model-setup (S6):
- The card reads execution from mosaic-config.mjs validate on each status read, so a config
  edit shows without a Console restart.
- status() starts the worker view first and awaits it last; status({ worker: false }) skips it,
  and the Triage snapshot passes that.
- The card says why the model list is not known: no organization, the model service off
  everywhere, or not read.
- The "config always wins" line no longer speaks of a choice made here.
- The standing 'claude' is now 'claude-model', since display.js may not key by a provider id.
2026-11-21 21:35:35 -06:00
jason.woltje 5f4ccd0326 Show the worker model on Model setup, and let the launch read a saved choice when the config names none
A Worker model card on Settings > Model setup (owner only, read only) says which model worker
runs use, that the config names it and wins, and lists only switched-on, worker-harness, non-Claude
models from the organizations' catalogues. The view rides the existing owner-gated status read.

A small store holds a worker model choice, guarded like the Triage route store. The launch reads
it only when the config names no model, holds it to the same Claude, switched-off and served checks,
and passes the checked model to compose. The config schema still requires a model, and no save
route is built; both are asks in the build report.
2026-11-21 20:02:13 -06:00
jason.woltje 1d39235a8b Keep an empty first field when the gate test reads its compose record 2026-11-21 19:02:01 -06:00
jason.woltje 13763db199 Test that a container-cli probe writes worker output with the key taken out
Review finding 5 (P3): no test failed if container-cli dropped its redaction. The test hook now
matches the resolved URL, since container-cli imports './worker-proxy.mjs'.
2026-11-21 18:17:27 -06:00
jason.woltje 717a8b9624 Run workers through the local model proxy with a Mosaic-issued key
A host worker launch (run-task.sh, mosaic-task.mjs, verify.sh, the release
health check) now checks the model, sweeps stale launches, issues or re-adds
the worker key, starts a relay to the proxy socket and mounts it at
/run/mosaic-proxy. The key enters the container only as ANTHROPIC_API_KEY and
is redacted from stderr.txt, the response and the run cause. The run record
keeps the key id only (modelKeyId).

The pi auth mount, the ZAI_API_KEY pass-through, the caller's
ANTHROPIC_API_KEY, the worker credential check, the auth.sh worker line and
the agent.sh account choice are gone. The harness name comes from
execution.adapter, and a Claude model never runs on pi. RELEASE is 0.0.14.
2026-11-21 17:54:33 -06:00
jason.woltje e5d1a37c9a Test that a task record with a container name keeps its worker key 2026-11-21 16:34:00 -06:00
jason.woltje 95a6be3b63 Hold a worker key for a container whose launcher died, and re-add a refused fast-path key under the lock 2026-11-21 16:34:00 -06:00
jason.woltje 882c12a136 Give the Triage key module a worker purpose, a lock-free launch path and a retire guard 2026-11-21 16:34:00 -06:00
jason.woltje 0aac96dd15 Stop naming a lost proxy for an idle close, and test that an error ends both sides
Re-check of bab5f47d9 found two minor things; this fixes them and one note.

- The forwarder printed the fixed line when the proxy closed an idle
  connection that never carried a request. An upstream end now counts
  only when the forwarder had sent the client's bytes; a socket error
  still always counts.
- Nothing tested that an error on one side ends the other. A new test
  streams from a proxy that keeps writing after a FIN, drops and resets
  120 connections mid-stream, and checks that the relay's and the
  forwarder's fd counts return to their baseline. With the error path
  cut it fails at 222 and 102 fds.
- A linger timer is now cleared when its socket closes by itself.
- spawnRelay's start timeout sends SIGKILL, so a relay wedged before
  "ready" cannot outlive it.
2026-11-21 16:22:32 -06:00
jason.woltje f904d7afda Keep a slow reader's tail, run the relay with no environment, and name a lost proxy
Review of 62468f963 found five things; this fixes them.

- Both copiers destroyed the other side on close, which cut a reply
  short for a client that half-closed and read slowly. A clean close now
  ends the other side and lets it flush; it is destroyed when its peer
  closes or 10 s after its last byte. An error still ends both at once.
- spawnRelay passed the caller's whole environment. The relay now runs
  with an empty one.
- The forwarder said nothing when the proxy died after the start check.
  A connection that ends before any reply byte, while the client still
  waits, now prints the fixed line once.
- stop() could wait forever on a wedged relay. It now sends SIGKILL
  after 2 s, and a stdin error cannot escape to the caller.
- New tests: half-closed slow reader through the relay and through the
  forwarder, empty environment, stop() on a stopped relay, the ppid poll
  alone, SIGINT and SIGHUP, and the proxy gone after the start check.
2026-11-21 16:22:32 -06:00
jason.woltje c27612cf41 Add the worker proxy relay and in-container forwarder, tested, not wired
Slice S2 of worker-proxy-simple. The host relay
(packages/processing-control/src/proxy-relay.mjs) listens on a 0600 Unix
socket in a caller-made 0700 directory and copies bytes to 127.0.0.1:<port>.
It refuses any other target, a socket path of 108 bytes or more, an unsafe
directory and an existing path. It exits on stdin EOF, on TERM, INT or HUP,
and when its parent changes.

The forwarder (adapters/pi/proxy-forward.mjs) listens on 127.0.0.1:<port>
and copies bytes to the socket. One start check through the socket must get
a reply byte; otherwise it prints one fixed line and exits 1.

Nothing calls either file yet except the tests. adapter.sh, compose.yaml,
the Containerfile and containers.mjs are unchanged.
2026-11-21 16:22:32 -06:00
jason.woltje 7264134899 Correct the final-stop notes from review, and name the cap and an unknown signal plainly
The header no longer claims stderr never carries an environment value: the
error line is pi's message as sent, and run-task redacts it. A comment says
why a worker fails a length stop with text while a managed session does
not. An unknown signal exits 1 rather than 128, and the over-cap line names
the 64 MiB cap. TOOLS.md notes that the wrapper is process 1 in the
container and passes SIGINT to pi.
2026-11-21 15:58:10 -06:00
jason.woltje 66d66eb736 Judge a pi worker run by its final stop reason, and print the cause of a failed run
In text print mode pi exits 1 on an error or aborted stop, but a length
stop exits 0 with nothing printed, so run-task recorded an empty success.
The worker entrypoint now sets MOSAIC_PI_FINAL_STOP=1 for headless pi.
adapter.sh then runs pi through final-stop.mjs, which uses --mode json,
keeps the last agent_end and fails the run on an error, aborted or length
stop with one plain line on stderr. A finished stop prints the answer as
text mode does. run-task also prints the cause line that show prints. The
reason stays exit-nonzero, with no new field. The mock adapter is unchanged.
2026-11-21 15:35:49 -06:00
jason.woltje baf5714443 Keep the Triage key state through a failed read, and stop a replaced decision draw
Re-check fix round. Triage settings hold the key state apart from the
read data, so a read that fails after a key change cannot let an older
read put back the old key state. The decision view checks that its draw
is still current after reading earlier versions, so a draw a newer one
replaced does not change the kept notice results.
2026-11-21 00:04:03 -06:00
jason.woltje 5f7adb7670 Keep a reload's route across a key change, and drop a result when its decision returns
Review fix round. A key change in Triage settings no longer drops a
Reload route read that was in flight. The read still draws its route
and revision, and keeps the newer key state. A decision notice drops
its kept result when the decision comes back into the inbox after a
draw saw it gone.
2026-11-20 23:52:05 -06:00
jason.woltje fc60809b80 Keep decision notice results, and drop older triage and workflow reads
The decision notice on an inbox decision kept its result only in the
page node, so the next poll redraw lost it, and a confirmed or reversed
decision left the inbox read with its notice. The result is now kept in
page memory per decision; a redraw keeps the result line and keeps both
buttons off while an answer is in flight.

Settings > Triage applied every route read. A read from an earlier visit
or a Reload route read that answered after a save or key change put back
the old route, revision or key state, and the next save was refused as
stale. Reads are now numbered as they start; one older than what the
page has drawn is dropped, and a save or key change takes a new number.

Workflows applied plain loads in the order they answered. Each
organization's loads are now numbered too, and an older load that
answers last, or fails last, is dropped. The SP17 write epochs are
unchanged.
2026-11-20 23:41:38 -06:00
jason.woltje 9b3cbe4296 Test that a hire receipt survives an overlapping packs reload
Two forced-order tests for the hire click. They fail without the packs state fix in 5a7e6765f.
2026-11-20 23:25:29 -06:00
jason.woltje 1e8a20074d Keep a newer autonomy read that lands during a save
Re-check of 7cba209dc (P3-1): a save can commit, another writer can
commit after it, and a poll can draw that newer read before the save
answers. The save then wrote its own older values into the state, and
the new redraw painted them over the newer form. It failed closed, but
the page showed older values.

A save that answers now writes its values into the state only when the
state still holds the revision it sent or the one it got back. Any other
revision was written after the save, so it stays. The redraw draws from
that state, so it never paints older values.

The older-read test waits for the draw as well as the read, since the
dashboard draws only after all its reads (re-check P3-2). A new test
forces the newer-writer order.
2026-11-20 22:48:55 -06:00
jason.woltje 2e78c7e593 Draw an autonomy save over a clean form drawn from an older read
Review of a4a035d17 (P3-1): a reload whose read was served before a save
committed could redraw a clean form while the save was in flight. The
save then wrote its revision to the form it started on, which the page
no longer drew. The drawn form showed the old levels next to the receipt,
and a second save before the next poll was refused as a revision
conflict.

A save that answers now draws the settings section again from the saved
state when the drawn form is a different, clean form. A form the owner is
editing stays, so it saves against its own revision and fails closed.
The live section lookup skips decision notices for the hold check too.

The older-read test now waits on the held read by the revision it
carries, so a poll in the window cannot move its count (P3-3). A new test
forces the P3-1 order and checks the drawn form before any reload.
2026-11-20 22:48:55 -06:00
jason.woltje fbfcd942be Keep an autonomy save or review receipt that overlaps a reload
The autonomy panel built a new state object on every load and copied the
status from the state it found when the load began. A save kept the old
object across its own request. So a save that answered while a poll or
an organization change was reading autonomy wrote its receipt and its new
revision to an object the page no longer drew, or the load copied the
empty status forward before the receipt existed. The next draw then
showed no receipt and the old revision, so the next save was refused as
a revision conflict. A trail review wrote its line only to the node it
started on, which a redraw replaces.

Each organization now keeps one state object. A load writes its read into
it when it ends. A load that began before the last applied read, or
before a save answered, is dropped. A save or review result is kept in
that state and written to the status line the page draws now.

Browser tests hold the fixture's calls to force each order of a save
against a reload, a review against a reload, and an older read that
answers after a newer one.
2026-11-20 22:48:55 -06:00
jason.woltje 5a7e6765f8 Keep a pack preview or receipt that overlaps a packs reload
The pack picker built a new state object on every load and copied the
preview and receipt from the state it found when the load began. A click
kept the old object across its own request. So a preview or a hire receipt
that answered while a poll or an apply's page reload was reading the packs
was written to an object the page no longer drew, or was copied away
before it existed. Under host load this dropped the project preview in the
SP6 pack picker test and the hire status in the pack hire test.

Each organization now keeps one state object. A load writes its read
fields into it when it ends, and a load that began before the last one
applied is dropped, so an older list never replaces a newer one.

One browser test holds the fixture's reads to force each order of the
overlap, a second forces an older packs read to answer last, and a third
holds a reload's packs read while an apply's receipt lands.
2026-11-20 20:33:47 -06:00
jason.woltje 69b3931c35 Resolve recheck follow-up cases 2026-11-20 19:43:53 -06:00
jason.woltje b7116be9c3 Address review of proxy administration follow-ups 2026-11-20 19:43:53 -06:00
jason.woltje 0f3135e36a Show unanswered organizations with offered models 2026-11-20 19:43:53 -06:00
jason.woltje b0dd4d3e2f Recognize organizations without a launch in model survey 2026-11-20 19:43:53 -06:00
jason.woltje e8c6c7e3a4 Test the 257 model limit boundary 2026-11-20 19:43:53 -06:00
jason.woltje fe671844ea Guard shared credential names with invalid account IDs 2026-11-20 19:43:53 -06:00
jason.woltje b3eaa8fd81 Preserve account removal success when refresh fails 2026-11-20 19:43:53 -06:00
jason.woltje 07baa87381 Clarify account removal ambiguity 2026-11-20 19:43:53 -06:00
jason.woltje 5e67fd9289 test(webui): wait for restart completion 2026-11-20 18:43:48 -06:00
jason.woltje ffc7db130f test(webui): enforce switch ordering in restart check 2026-11-20 18:43:48 -06:00
jason.woltje d6de22aaf8 test(webui): isolate browser waits and offline observation 2026-11-20 18:43:48 -06:00
jason.woltje f4134fff9a Fix account repair navigation and coverage 2026-11-20 17:03:22 -06:00
jason.woltje 122659a488 Make model setup account status truthful 2026-11-20 17:03:22 -06:00
jason.woltje 40afaaf6ad fix(harness): prove MCP seals with extension-backed loaders 2026-11-20 16:32:21 -06:00
jason.woltje 8042aa13ed test(cli): exercise plugin backstop with TMPDIR unset 2026-11-20 16:23:18 -06:00
jason.woltje 82aa0c0bda test(plugins): use a multi-segment temp path for redaction 2026-11-20 16:13:28 -06:00
jason.woltje 0430c7dc32 fix(release,task,auth): health-reason review round: the worker credential check reads the auth file path as compose does (a leading ~ is $HOME, a relative path is from the checkout, a named volume is undecided), treats COMPOSE_ENV_FILES and docker-compose.override.yaml|yml as undecided like .env, and leaves the credential not checked when this user can't read the auth file (EACCES, EPERM) instead of refusing; a key variable that is only spaces is no credential, as in pi; a compose.yaml that can't be read or parsed gives fixed words, never its path or text; release.sh's reason with no run id comes from the new scripts/health-reason.mjs (the last refusal line, prefix dropped, redacted, withheld when it holds an environment value) and says "gave no run id"; the fault-injection and recursion-guard comments are back; tests cover the tilde path, COMPOSE_ENV_FILES, an unreadable auth file, a blank key, a compose warning line in a failed run's stderr, and the redacted and withheld reason; docs/TOOLS.md names each undecided case (#1873, lane health-reason) 2026-11-20 15:31:59 -06:00
jason.woltje 1433686d50 feat(release,task,auth,runs,plugins): health-reason: a failed release health check prints its run id and one plain reason before release: refusing activate: health check failed: the cause line mosaic-task show now prints for a run that did not succeed (a runner reason in words, else one redacted line from the run's stderr.txt read through the new readRunText), or, when the runner refused before a run record, its own refusal line, redacted. A pi run checks before its run directory exists that the worker container compose starts holds a credential for config execution.provider: names and types from the PI_AUTH_FILE compose mounts and the key variables compose passes with a value, judged by pi-ai's own checkAuth on placeholders (packages/plugins/harness/pi/worker-credential.mjs), so no provider list is kept; with none it refuses exit 3 naming the provider, what the container has and what to do; when it can't decide (COMPOSE_FILE, an override or .env, an env_file, an unknown provider) it says so and the run goes ahead. auth.sh status adds one line for the default run's provider and credential. Tests in test-task, test-release, test-auth and runs.test; docs/TOOLS.md (#1873, lane health-reason) 2026-11-20 15:14:07 -06:00
120 changed files with 8109 additions and 509 deletions

No files matched your search

+4 -15
View File
@@ -1,24 +1,13 @@
# Non-secret runtime settings for the mosaic-poc-agent container.
# Copy to .env if you want to override the defaults in compose.yaml.
#
# NEVER put credentials in this file. Authentication is supplied at
# runtime only, via one of the two documented paths:
# 1. read-only mounted pi auth file (default: ~/.pi/agent/auth.json,
# override the host path with PI_AUTH_FILE)
# 2. provider API key environment variable (ZAI_API_KEY or
# ANTHROPIC_API_KEY), passed through by compose.yaml when set
# NEVER put credentials in this file. A worker holds no credential of
# its own: it reaches its model through the local model proxy with the
# worker key Mosaic issues (Settings > Model setup). No pi auth file is
# mounted and no provider key variable is passed through.
# Model provider (built-in pi provider name)
PI_PROVIDER=zai
# Model ID within the provider
PI_MODEL=glm-5.3-flash
# Optional: alternative host path of the pi credential file mounted
# read-only at /home/node/.pi/agent/auth.json in the container
#PI_AUTH_FILE=/home/jwoltje/.pi/agent/auth.json
# Optional: documented env-var auth alternative (secret! set in your
# shell or a gitignored .env, never commit)
#ZAI_API_KEY=
#ANTHROPIC_API_KEY=
+3 -2
View File
@@ -35,8 +35,9 @@ RUN chmod 0555 /opt/mosaic/contracts /opt/mosaic/contracts/* \
&& chmod 0555 /opt/mosaic/adapters /opt/mosaic/adapters/*/adapter.sh
# Private per-container prompt/output paths, the fixed narrow-mount targets,
# workspace, and pi agent directory (auth.json is bind-mounted read-only at
# runtime; no host data root is mounted).
# workspace, and pi agent directory (worker-cutover: no credential file is mounted;
# a pi worker's models.json, which names the model proxy and holds no key, is
# bind-mounted read-only at runtime; no host data root is mounted).
RUN mkdir -p /var/lib/mosaic/private /var/lib/mosaic/input/user /var/lib/mosaic/input/skills /var/lib/mosaic/control /var/lib/mosaic/session /workspace /home/node/.pi/agent \
&& touch /var/lib/mosaic/input/task.json /var/lib/mosaic/input/mission.json /var/lib/mosaic/input/agent-soul.md /var/lib/mosaic/input/session-fork.jsonl /var/lib/mosaic/control/processing.json \
&& chown -R node:node /var/lib/mosaic /workspace /home/node /opt/app \
+4 -2
View File
@@ -11,8 +11,10 @@ pi from the lock, not pinned, and fit-checked before the build) runs one Pi agen
loaded in fixed order into the generated system prompt
(`/var/lib/mosaic/private/system-prompt.md`, private to that container). One real model request is sent
noninteractively; the response must equal `MOSAIC_HELLO_OK` exactly or the
verification exits nonzero. Authentication is supplied at runtime only
(read-only mounted pi auth file, or a provider API key environment variable).
verification exits nonzero. Authentication is supplied at runtime only: since
worker-cutover a worker reaches its model through the local model proxy, with
the worker key Mosaic issues passed only as `ANTHROPIC_API_KEY` and the relay
socket directory mounted read-only. No pi auth file is mounted.
## L1 — Deferred: persist and resume a named Pi session
+15 -12
View File
@@ -237,21 +237,24 @@ EXPECTED_MARKER=MOSAIC_NOT_OK scripts/verify.sh # must exit nonzero
## Authentication
Pi's documented container authentication (see the package's
`docs/containerization.md`) is used, in this order:
A worker container holds no credential of its own (worker-cutover). It
reaches its model only through the instance's local model proxy, with the
worker key Mosaic issues, re-adds and rotates itself at launch. Each launch
passes the key to the container only as `ANTHROPIC_API_KEY`, and mounts the
relay socket directory read-only at `/run/mosaic-proxy` and a `models.json`
that names the proxy (and holds no key) read-only at
`/home/node/.pi/agent/models.json`. No pi auth file is mounted, and a
caller's `ZAI_API_KEY` or `ANTHROPIC_API_KEY` never reaches a worker.
1. **Read-only mounted credential file** (default): the host pi auth file
`~/.pi/agent/auth.json` is bind-mounted read-only to
`/home/node/.pi/agent/auth.json`. The host file holds a static API-key
entry for the built-in `zai` provider, so no token refresh writes are needed.
2. **Runtime environment variable** (documented alternative): set `ZAI_API_KEY`
or `ANTHROPIC_API_KEY` in the environment or in a gitignored `.env`; compose
passes them through. Pi's documented precedence applies.
Set up once: start the proxy and add an account on Settings > Model setup.
The worker's provider and model stay `execution.provider` and
`execution.model` in the config; a Claude model is refused for a pi worker.
The launch checks and their one-line refusals are in `docs/TOOLS.md`.
Credentials are never committed, never copied into the image, and never printed.
Mosaic-managed named accounts (`agent.sh --auth`) live under the data root
(`auth/<account>.json`, 0600) — the stack never writes into `~/.pi`.
`.env.example` contains non-secret settings only.
The key's text is redacted from run evidence; a run record keeps only
`modelKeyId`. The stack never writes into `~/.pi`. `.env.example` contains
non-secret settings only.
## Boundaries honored
+1 -1
View File
@@ -1 +1 @@
0.0.13
0.0.14
+7 -1
View File
@@ -31,6 +31,9 @@ Optional, adapter-specific (documented per adapter):
| Variable | Meaning |
|---|---|
| `MOSAIC_MOCK_RESPONSE` | mock only: the verbatim response to emit |
| `MOSAIC_PI_FINAL_STOP` | pi only, `1` from `src/run-agent.sh` for a headless worker: judge the run by pi's final stop reason (`final-stop.mjs`) |
| `MOSAIC_PROXY_PORT` | pi only, set by the host launcher (worker-cutover): start `proxy-forward.mjs` on `127.0.0.1:<port>` against the relay socket mounted at `/run/mosaic-proxy/proxy.sock`. The adapter refuses (exit 1) unless the forwarder's captured stdout is exactly `ready`. Unset: no proxy and no credential, so pi fails on its own. |
| `ANTHROPIC_API_KEY` | pi only: the worker key Mosaic issued for this launch, read by the mounted `models.json` as `$ANTHROPIC_API_KEY`. It is the container's only credential; the adapter never prints it. Empty for the mock adapter. |
**Outputs:**
@@ -53,7 +56,10 @@ Optional, adapter-specific (documented per adapter):
## Included adapters
- `pi` — the `@earendil-works/pi-coding-agent` CLI (not pinned; #1873) in noninteractive
print mode (`-p`), ambient discovery disabled, stdin detached.
print mode (`-p`), ambient discovery disabled, stdin detached. A worker run
(`MOSAIC_PI_FINAL_STOP=1`) goes through `final-stop.mjs`: pi runs with
`--mode json`, the answer is printed only on a finished stop, and an error,
aborted or output-limit stop exits 1 with one line on stderr.
- `mock` — deterministic echo of `MOSAIC_MOCK_RESPONSE`. Test-only: never use
it where a real model response is required.
- `claude` — the host's `claude` CLI, for managed sessions only (below):
+20
View File
@@ -25,6 +25,18 @@ esac
: "${PI_PROVIDER:?pi adapter: PI_PROVIDER is required}"
: "${PI_MODEL:?pi adapter: PI_MODEL is required}"
# Model proxy (worker-cutover, S4): the host launcher (packages/processing-control/src/
# worker-proxy.mjs) mounts its relay's socket dir at /run/mosaic-proxy and sets
# MOSAIC_PROXY_PORT. The forwarder listens on 127.0.0.1:$MOSAIC_PROXY_PORT, where the mounted
# models.json points pi, after one request through the socket got a reply. It prints exactly
# "ready" on stdout and closes it, so the capture below returns while it keeps running. Anything
# else refuses: the forwarder already printed its one line on stderr. With no port set there is
# no proxy and no credential in this container, so pi fails on its own.
if [ -n "${MOSAIC_PROXY_PORT:-}" ]; then
ready=$(node "$(dirname "$0")/proxy-forward.mjs" /run/mosaic-proxy/proxy.sock "$MOSAIC_PROXY_PORT" &)
[ "$ready" = "ready" ] || exit 1
fi
INTERACTIVE="${MOSAIC_INTERACTIVE:-}"
if [ "$INTERACTIVE" != "1" ]; then
[ -n "${MOSAIC_REQUEST:-}" ] || { echo "pi adapter: MOSAIC_REQUEST is required" >&2; exit 2; }
@@ -148,5 +160,13 @@ if [ -n "${MOSAIC_MODEL_EVENT_MONITOR:-}" ] && [ "$INTERACTIVE" != "1" ]; then
STREAM="$(dirname "$0")/../../packages/plugins/harness/pi/model-stream.mjs"
exec node "$STREAM" pi "$MOSAIC_FIRST_OUTPUT_DIR" "${MOSAIC_PI_BIN:-pi}" "$@" --mode json
fi
# worker-stop-reason: a run-task worker (src/run-agent.sh sets MOSAIC_PI_FINAL_STOP=1) is judged
# by pi's final stop reason as well as its exit status. Text mode exits 0 on a "length" stop with
# nothing printed; final-stop.mjs runs pi with --mode json, prints the answer as text mode does on
# a finished stop, and fails the run on an error, aborted or length stop with one line on stderr.
# Managed sessions never set it and keep text mode.
if [ "${MOSAIC_PI_FINAL_STOP:-}" = "1" ] && [ "$INTERACTIVE" != "1" ]; then
exec node "$(dirname "$0")/final-stop.mjs" "${MOSAIC_PI_BIN:-pi}" "$@"
fi
# #1873 S2: the binary the harness module found (MOSAIC_PI_BIN, its path inside the sandbox).
exec "${MOSAIC_PI_BIN:-pi}" "$@"
+152
View File
@@ -0,0 +1,152 @@
#!/usr/bin/env node
// worker-stop-reason: judge a headless pi worker run by pi's final stop reason, not only its exit
// status. The worker entrypoint (src/run-agent.sh) asks for this with MOSAIC_PI_FINAL_STOP=1, and
// adapter.sh then runs `node final-stop.mjs <pi> <args...>` in place of `<pi> <args...>`.
//
// Why: in text print mode (-p) pi 0.85.1 exits 1 on an "error" or "aborted" stop, but a "length"
// stop (the model reached its output limit) exits 0 with nothing on stdout or stderr, so run-task
// recorded an empty success. Only the JSON event stream says how the run ended: its last
// agent_end event carries the run's messages, and the last assistant message's stopReason is the
// final stop. Proved against pi 0.85.1 with a fake provider on loopback (packages/plugins/tests/
// pi-final-stop.test.mjs).
//
// This runs pi with `--mode json` and keeps only the last agent_end line (bounded). Then:
// pi exited nonzero the same code, pi's own stderr already said why
// pi was stopped by a signal 128 + the signal number, one line on stderr
// final stop "stop" the text parts, each followed by "\n" (as text mode prints), exit 0
// final stop "toolUse" nothing, exit 0 (a tool ended the run; text mode prints nothing)
// final stop "error"/"aborted" exit 1, one line: pi stopped on an error: <pi's error message>
// final stop "length" exit 1, one line: the model reached its output limit
// anything else exit 1, one line (fail closed: no final answer to judge)
// stdout carries the answer only on success. An error line carries pi's error message as the
// provider sent it, as text mode printed it; run-task redacts known secret values before it shows
// the cause (runCause in scripts/mosaic-task.mjs).
import { spawn } from "node:child_process";
import { realpathSync } from "node:fs";
import { constants } from "node:os";
import { fileURLToPath } from "node:url";
export const AGENT_END_PREFIX = '{"type":"agent_end"';
// The cap on the one line kept. agent_end carries every message of the run, tool results too.
export const MAX_AGENT_END_BYTES = 64 * 1024 * 1024;
const MAX_LINE_CHARS = 2000;
const oneLine = (text) => {
const line = String(text).replace(/\s+/g, " ").trim();
return line.length > MAX_LINE_CHARS ? `${line.slice(0, MAX_LINE_CHARS)}...` : line;
};
// The judgement for the last agent_end line pi wrote (null when it wrote none):
// { ok: true, answer } or { ok: false, message }.
export function judgeFinalStop(agentEndLine) {
if (agentEndLine === null) return { ok: false, message: "pi ended without a final answer: it reported no end of run" };
let event;
try { event = JSON.parse(agentEndLine); } catch { return { ok: false, message: "pi ended without a final answer: its end of run could not be read" }; }
const messages = Array.isArray(event?.messages) ? event.messages : [];
let last = null;
for (let i = messages.length - 1; i >= 0; i--) {
if (messages[i]?.role === "assistant") { last = messages[i]; break; }
}
if (!last) return { ok: false, message: "pi ended without a final answer: the model gave no reply" };
const reason = last.stopReason;
if (reason === "stop" || reason === "toolUse") {
if (messages[messages.length - 1] !== last) return { ok: true, answer: "" };
const parts = Array.isArray(last.content) ? last.content : [];
return { ok: true, answer: parts.filter((p) => p?.type === "text" && typeof p.text === "string").map((p) => `${p.text}\n`).join("") };
}
if (reason === "error" || reason === "aborted") {
const said = typeof last.errorMessage === "string" && last.errorMessage.trim() ? oneLine(last.errorMessage) : `request ${reason}`;
return { ok: false, message: `pi stopped on an error: ${said}` };
}
// A worker run fails on "length" even with partial text: the answer was cut off. A managed
// session first turn (packages/plugins/harness/pi/model-stream.mjs) counts length with text as
// output; that turn is a conversation, a worker run is a finished job.
if (reason === "length") return { ok: false, message: "pi stopped with no finished answer: the model reached its output limit" };
return { ok: false, message: `pi ended without a final answer: it stopped for an unknown reason (${oneLine(String(reason)).slice(0, 40)})` };
}
// Keeps the last line that starts with AGENT_END_PREFIX out of a byte stream, holding at most
// one such line (up to max bytes) in memory. Other lines pass through unkept.
export function agentEndKeeper(max = MAX_AGENT_END_BYTES) {
let chunks = [];
let size = 0;
let state = "start"; // start: at a line start; keep: in an agent_end line; skip: in another line
let head = Buffer.alloc(0);
let last = null;
let tooBig = false;
const finishLine = () => {
if (state === "keep") {
if (size > max) tooBig = true;
else { last = Buffer.concat(chunks).toString("utf8"); tooBig = false; }
}
chunks = []; size = 0; state = "start"; head = Buffer.alloc(0);
};
const feedPart = (part) => {
if (state === "start") {
head = Buffer.concat([head, part]);
const prefix = Buffer.from(AGENT_END_PREFIX);
const n = Math.min(head.length, prefix.length);
if (!head.subarray(0, n).equals(prefix.subarray(0, n))) { state = "skip"; head = Buffer.alloc(0); return; }
if (head.length < prefix.length) return;
state = "keep"; part = head; head = Buffer.alloc(0);
}
if (state === "keep") {
size += part.length;
if (size <= max) chunks.push(part); else chunks = [];
}
};
return {
write(buffer) {
let at = 0;
while (at < buffer.length) {
const nl = buffer.indexOf(10, at);
if (nl === -1) { feedPart(buffer.subarray(at)); return; }
feedPart(buffer.subarray(at, nl));
finishLine();
at = nl + 1;
}
},
end() {
if (state === "keep") finishLine();
return { line: tooBig ? null : last, tooBig };
},
};
}
async function main(argv) {
const [pi, ...args] = argv;
if (!pi) { process.stderr.write("pi final stop: no pi binary was given\n"); return 2; }
const keeper = agentEndKeeper();
const child = spawn(pi, [...args, "--mode", "json"], { stdio: ["inherit", "pipe", "inherit"] });
const forward = (signal) => { try { child.kill(signal); } catch {} };
for (const signal of ["SIGTERM", "SIGINT", "SIGHUP"]) process.on(signal, () => forward(signal));
child.stdout.on("data", (buffer) => keeper.write(buffer));
const outcome = await new Promise((resolve) => {
child.on("error", (error) => resolve({ error }));
child.on("close", (code, signal) => resolve({ code, signal }));
});
if (outcome.error) {
process.stderr.write(`pi final stop: pi could not be started (${outcome.error.code ?? "spawn error"})\n`);
return 127;
}
if (outcome.signal) {
process.stderr.write(`pi final stop: pi was stopped by ${outcome.signal}\n`);
const number = constants.signals[outcome.signal];
return Number.isInteger(number) && number > 0 ? 128 + number : 1;
}
if (outcome.code !== 0) return outcome.code ?? 1;
const { line, tooBig } = keeper.end();
const verdict = tooBig
? { ok: false, message: `pi ended without a final answer: its end of run was over ${MAX_AGENT_END_BYTES / (1024 * 1024)} MiB` }
: judgeFinalStop(line);
if (!verdict.ok) {
process.stderr.write(`${verdict.message}\n`);
return 1;
}
await new Promise((resolve) => process.stdout.write(verdict.answer, resolve));
return 0;
}
let isMain = false;
try { isMain = realpathSync(process.argv[1] ?? "") === realpathSync(fileURLToPath(import.meta.url)); } catch {}
if (isMain) process.exitCode = await main(process.argv.slice(2));
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env node
// In-container forwarder for the model proxy (worker-proxy-simple, slice S2).
//
// node proxy-forward.mjs <socket path> <port>
//
// Listens on 127.0.0.1:<port> inside the container and copies bytes, both
// ways, to the host relay's Unix socket (packages/processing-control/src/
// proxy-relay.mjs), which copies them to the host's loopback model proxy.
// It parses nothing and holds no key.
//
// Start check. Before it listens it makes one connection through the socket
// and sends one small HTTP request (HEAD /). Any reply byte within five
// seconds passes: the socket, the relay and the proxy all answered. Anything
// else prints exactly one line on stderr and exits 1:
//
// worker: the container could not reach the model proxy socket.
//
// The same line is printed once, and the forwarder keeps running, when a later
// connection fails before any reply byte: the socket refuses, or the relay
// hangs up on a request with no reply, which is what a gone proxy looks like.
// An idle connection that the proxy closes before any request is not counted.
//
// Output. One line "ready" on stdout once the port listens, then stdout is
// closed. The caller captures it (for example ready=$(node ... &)), so it
// never mixes with the adapter's response on stdout. A usage or listen
// failure is one other line on stderr and exits 2.
// Not called by adapter.sh yet; the cutover slice (S4) starts it.
import { closeSync, openSync } from 'node:fs';
import net from 'node:net';
import { isAbsolute } from 'node:path';
const UNREACHABLE = 'worker: the container could not reach the model proxy socket.';
const CHECK_MS = 5000;
const CHECK_REQUEST = 'HEAD / HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n';
let told = false;
function unreachable() {
if (!told) { told = true; process.stderr.write(`${UNREACHABLE}\n`); }
}
function check(socketPath) {
return new Promise((resolveCheck) => {
let passed = false, finished = false;
const probe = net.connect({ path: socketPath });
const finish = (ok) => {
if (finished) return;
finished = true;
clearTimeout(timer);
probe.destroy();
resolveCheck(ok);
};
const timer = setTimeout(() => finish(false), CHECK_MS);
probe.once('connect', () => probe.write(CHECK_REQUEST));
probe.on('data', () => { passed = true; finish(true); });
probe.on('error', () => finish(false));
probe.on('close', () => finish(passed));
});
}
// Copy bytes both ways. An error on either side ends both at once. A clean
// close of one side only ends the other, so bytes still queued for a slow
// reader are flushed first; that side is destroyed when its peer closes too,
// or LINGER_MS after its last byte was written. Kept in step with the same
// helper in packages/processing-control/src/proxy-relay.mjs.
const LINGER_MS = 10000;
function join(a, b) {
const kill = () => { a.destroy(); b.destroy(); };
const finish = (other) => () => {
if (other.destroyed) return;
other.resume();
other.end();
const linger = () => {
const timer = setTimeout(() => other.destroy(), LINGER_MS).unref();
other.once('close', () => clearTimeout(timer));
};
if (other.writableFinished) linger(); else other.once('finish', linger);
};
a.on('error', kill); b.on('error', kill);
a.on('close', finish(b)); b.on('close', finish(a));
a.pipe(b); b.pipe(a);
}
async function main(argv) {
const port = /^[1-9][0-9]{0,4}$/.test(argv[1] ?? '') ? Number(argv[1]) : 0;
if (argv.length !== 2 || !isAbsolute(argv[0]) || port < 1 || port > 65535) {
process.stderr.write('worker: proxy forwarder usage: proxy-forward.mjs <socket path> <port>.\n');
process.exit(2);
}
const socketPath = argv[0];
if (!(await check(socketPath))) {
unreachable();
process.exit(1);
}
const server = net.createServer({ allowHalfOpen: true }, (client) => {
const upstream = net.connect({ path: socketPath, allowHalfOpen: true });
let clientDone = false;
client.once('end', () => { clientDone = true; });
client.once('close', () => { clientDone = true; });
// The socket failed, or the relay hung up on a request before any reply
// byte while the client still waited: what a gone proxy looks like. An
// idle connection the proxy closes before any request is not counted.
const lost = () => { if (!clientDone && upstream.bytesRead === 0) unreachable(); };
upstream.on('error', lost);
upstream.once('end', () => { if (upstream.bytesWritten > 0) lost(); });
join(client, upstream);
});
server.on('error', () => {
process.stderr.write(`worker: the model proxy forwarder could not listen on 127.0.0.1:${port}.\n`);
process.exit(2);
});
server.listen({ host: '127.0.0.1', port, exclusive: true }, () => {
// Node never closes fd 1 itself, so close it and put /dev/null in its place:
// a caller reading stdout to its end then gets EOF right after "ready".
process.stdout.write('ready\n', () => {
try { closeSync(1); openSync('/dev/null', 'w'); } catch {}
});
});
}
main(process.argv.slice(2)).catch(() => {
unreachable();
process.exit(1);
});
+7 -8
View File
@@ -26,15 +26,14 @@ services:
MOSAIC_AGENT_ROLE: ${MOSAIC_AGENT_ROLE:-}
# mock adapter only: verbatim response for deterministic seam tests
MOSAIC_MOCK_RESPONSE: ${MOSAIC_MOCK_RESPONSE:-}
# Documented container auth alternative: provider API key via
# runtime environment variable. Empty by default; when empty Pi
# falls back to the read-only mounted auth.json credential file.
ZAI_API_KEY: ${ZAI_API_KEY:-}
# worker-cutover: a pi worker reaches its model only through the
# instance's model proxy. The launcher (packages/processing-control/
# src/worker-proxy.mjs) sets both from the worker key it issued and
# the relay it started; it clears them for every other launch, so a
# caller's own provider key never enters the container. There is no
# credential file mount.
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
volumes:
# Runtime credential only: pi auth file mounted READ-ONLY.
# Never copied into the image.
- ${PI_AUTH_FILE:-/home/jwoltje/.pi/agent/auth.json}:/home/node/.pi/agent/auth.json:ro
MOSAIC_PROXY_PORT: ${MOSAIC_PROXY_PORT:-}
# Headless runs: the request is passed as command args by the launchers
# (run-task.sh) or defaults inside run-agent.sh (hello/verify). Never a
# fixed command here - interactive runs (scripts/agent.sh) need no args.
+150 -14
View File
@@ -39,6 +39,86 @@ Task fields: `prompt` (required), `mission` (path), `expectExact`,
`directives[]`, optional governing `capabilities.tools`. Policy: a task may
narrow a mission's tools, never widen; empty intersection = tool-free run.
Worker launch checks (worker-cutover). A run's provider and model come only
from the instance config, `execution.provider` and `execution.model`; task and
mission files have no provider field. A worker container holds no credential
of its own. Every host worker launch (`mosaic-task run` and `retry`, so
`run-task.sh` and the release health check, and `scripts/container.sh` for
`verify.sh`, `hello.sh` and `agent.sh`) goes through
`packages/processing-control/src/worker-proxy.mjs`. For a pi worker it checks,
before a run directory exists and before docker:
1. the host user is uid 1000 (the image's user);
2. the model proxy is set up (`proxy.json` beside the config);
3. the model is not a Claude model (Claude OAuth models run only in the
Claude harness);
4. the management key loads and the proxy's model list has an account that
serves `execution.provider/execution.model` for pi.
It then issues, re-adds or rotates the worker key (`msw-<tag>-worker-…`, in
`<configDir>/secrets/worker/`) under the proxy lock when needed, and reads it,
reading again when another launch rotated it in between. Each refusal is one
line on stderr, exit 3, and no run record:
| Failure | Line |
|---|---|
| Proxy not set up | `proxy-not-configured: The model proxy is not set up. Set it up on Settings > Model setup.` |
| Proxy not running | `proxy-down: The model proxy is not running. Start it on Settings > Model setup.` |
| No account for the model | `model-provider-unavailable: No account serves zai/glm-5.3-flash. Models you can use: (list). Add an account on Settings > Model setup, or set execution.model in the config.` |
| Model switched off | `model-provider-unavailable: zai/glm-5.3-flash is switched off. Switch it on in Settings > Model setup.` |
| Claude model on pi | `model-harness-incompatible: Claude models run only in the Claude harness. Pick another model in the config.` |
| Key refused | `proxy-key-refused: The proxy did not accept the worker key. Restart the proxy on Settings > Model setup, then run verify again.` |
| Another key write running | `proxy-lock-busy: Another proxy key change is running. Run verify again in a minute.` |
| Proxy write uncertain | `proxy-uncertain: A proxy key change did not finish. Restart the proxy, then run \`mosaic proxy fence clear --proxy-restarted\`.` |
| Host user is not uid 1000 | `Worker containers run as user 1000. Run Mosaic as that user.` (no code) |
`mosaic-task` prefixes a coded line with `mosaic-task: `, `container.sh` with
`processing: `. Rarer `worker-key-*` codes come from the key module with its
own fixed sentence.
The launch then registers the runtime record, makes
`<dataRoot>/worker-proxy/<id>/` (0700, `<id>` is 12 hex), writes pi's
`models.json` there (it names the proxy and reads the key from
`$ANTHROPIC_API_KEY`; it holds no key), and starts the relay at
`<id>/proxy.sock`. Docker gets the key only in `ANTHROPIC_API_KEY`, the
directory read-only at `/run/mosaic-proxy` and `models.json` read-only at
`/home/node/.pi/agent/models.json`. No pi auth file is mounted. The caller's
`ZAI_API_KEY` is removed and its `ANTHROPIC_API_KEY` replaced for every
adapter. When docker returns, the relay stops, the directory goes, and the
runtime record is marked ended (on a nonzero exit, after docker lists no
container for it). A run record holds `modelKeyId`, the key's id (12 hex of
its sha256), never the key. The key's exact text is redacted from
`stderr.txt`, the response and the cause line.
The mock adapter takes no key, no proxy and no relay, and is never checked.
`mosaic-task show <runId>` on a run that did not succeed adds one line after
`exit:`, `cause: <reason>`. A reason the runner decided is said in words
(`the run passed its time limit and was stopped`, `the response did not match
the expected text`, `processing was halted during the run`). Otherwise it is
`worker said: <line>`, one line from the run's `stderr.txt` with compose's
progress and warning lines passed over, redacted (paths, links, emails,
key-shaped strings) and withheld when it holds a value from the environment.
With no such line it says the worker container could not be started or the
worker exited with its code and wrote nothing that can be shown. `mosaic-task
run` prints the same line as `cause: <reason>` after `response:` when the run
did not succeed.
A pi worker is judged by pi's final stop reason as well as its exit status
(worker-stop-reason). In print mode pi exits 0 when the model reaches its
output limit, with nothing printed, so that run used to count as succeeded.
The container's entrypoint sets `MOSAIC_PI_FINAL_STOP=1` for a headless pi
run, and the pi adapter then runs pi through `adapters/pi/final-stop.mjs`,
which reads pi's JSON event stream. A finished stop prints the answer as
before. An error or aborted stop fails with `pi stopped on an error: <pi's
error>`, and an output-limit stop fails with `pi stopped with no finished
answer: the model reached its output limit`. The reason stays
`exit-nonzero`, and the line is the `worker said:` cause. Managed sessions
don't set the variable and keep text mode. In the container the wrapper is
process 1 and passes SIGTERM, SIGINT and SIGHUP to pi, so a SIGINT that
reaches the container now stops pi (exit 130). Before, pi was process 1 and
ignored SIGINT.
Legacy worker containers use a typed per-launch mount contract. Task, mission,
user and session-fork snapshots are read-only. A fork source is opened without
following its final symlink, limited to 16 MiB, checked for owner/type and
@@ -89,9 +169,9 @@ contract's tools are a ceiling the seat definition or `--tools` may narrow,
never escalate past. Missing/invalid contract refuses the launch; empty
intersection = loud tool-free seat. An explicit `MOSAIC_AGENTS_DIR` override
that cannot resolve the named seat also refuses (#46) — unset the override
for the M13 plain governed TUI. `--auth <account>` injects
`auth.<account>.json` (beside the active credential file) as the launch's
`PI_AUTH_FILE`; a missing/invalid account refuses (M19).
for the M13 plain governed TUI. `--auth` is retired (worker-cutover): it
refuses with exit 4 and `agent: --auth is retired. Workers use the model
proxy; set up accounts on Settings > Model setup.`
For native repository development, opt in with a **leading** `--host-dev`:
@@ -116,21 +196,25 @@ provider; resolution order `--api-key` → `auth.json` → env → models.json).
No credential material is ever printed — provider names, credential types,
and env var NAMES only.
Ownership rule (#48): `~/.pi` is read-only to the stack, permanently. The
only interaction is the existing read-only container mount of the default
credential (`PI_AUTH_FILE`, default `~/.pi/agent/auth.json`). Mosaic-managed
accounts live under the data root: `<dataRoot>/auth/<account>.json`, perms
0600 (mirroring `scripts/gitea-api.sh` hygiene — loose perms are flagged in
listings and refused by `--auth`).
Ownership rule (#48): `~/.pi` is read-only to the stack, permanently. Since
worker-cutover no worker container mounts it: a worker reaches its model
through the model proxy with the worker key Mosaic issues, and accounts are
set up on Settings > Model setup. `auth.sh` still reports the host pi file
(`PI_AUTH_FILE`, default `~/.pi/agent/auth.json`) for host-side pi use, and
lists old files under `<dataRoot>/auth/<account>.json` for cleanup; nothing
launches with them.
| Command | Purpose | Notes |
|---|---|---|
| `scripts/auth.sh status` | Report both credential sources | Default harness credential (read-only) + mosaic-managed accounts; never prints material |
| `scripts/auth.sh accounts` | List mosaic-managed accounts | Under the data root; marks the active one; flags non-0600 |
| `scripts/auth.sh status` | Report the host pi file, the default run and old accounts | Never prints material |
| `scripts/auth.sh accounts` | List old accounts under the data root | Marks the one `PI_AUTH_FILE` names; flags non-0600 |
`agent.sh --auth <account>` injects `<dataRoot>/auth/<account>.json` as the
launch's `PI_AUTH_FILE`; missing/symlinked/non-0600 accounts refuse.
Headless task runs keep the default credential.
`auth.sh status` prints one line for the default run:
```
default run: provider zai (config execution.provider); the worker uses the model proxy with a key Mosaic issues (Settings > Model setup)
default run: provider zai (config execution.provider); the adapter needs no credential
```
## Release
@@ -142,6 +226,28 @@ Headless task runs keep the default credential.
| `scripts/release.sh status` | Release, tag, active pointer, log | Safe on empty state |
| `scripts/release.sh ensure` | Self-determination: align active pointer to `RELEASE` | Fast path restores a missing/mismatched pointer without a gate; slow path packages + health-gates first. Invoked automatically at launch |
A failed health check says which run failed and why before it refuses. With
a run record, the reason is the `cause:` line `mosaic-task show` prints:
```
release: health check run: r-20261120T120000Z-a1b2c3
release: reason: the response did not match the expected text
release: refusing activate: health check failed
```
When the runner gave no run id (it refused before the run started: a launch
check from the table above, a config problem), the reason is the runner's last
refusal line, picked and redacted by `scripts/health-reason.mjs`:
```
release: health check gave no run id
release: reason: proxy-not-configured: The model proxy is not set up. Set it up on Settings > Model setup.
release: refusing activate: health check failed
```
Neither prints a key, an environment value or a host path; the run id is the
only identifier.
## Conductor (worker patches)
```bash
@@ -1379,6 +1485,36 @@ evidence (`proxy-restart-unconfirmed` otherwise). `proxy init-lock
status|clear` clears a stale init lock only when every bus is stopped and no
proxy state exists. Details: `packages/model-proxy/README.md`.
`proxy ride --instance NAME` makes this instance's worker runs use instance
NAME's model proxy (proxy-ride first slice, Jason 2026-11-21). Both instances
belong to the OS user running the command. It writes
`<configDir>/proxy-ride.json` as `{"version":1,"instance":"NAME"}`, mode 0600,
then loads the system config the way a worker run does and removes the file
again on any refusal. `proxy ride --off` removes the file without loading the
config. Both refuse `human-required` in an agent shell or with `--tester`. The
riding instance's `mosaic-config.mjs validate` prints the owner's `proxy` with
`owner: {instance, configDir, dataRoot}`; worker runs (`verify.sh`, `hello.sh`,
`agent.sh`, `run-task.sh`) then use the owner's management key, lock, fence,
state directory and model on/off list. The worker key keeps the riding
instance's own tag and file. Buses, the Console and every other verb load the
config without that proxy, as on an instance with no proxy, and
`MOSAIC_INSTANCE_PROXY_CONFIGURED` stays off, so no proxy unit is installed for
the rider. Every validate resolves the owner, so while `proxy-ride.json`
exists an owner problem refuses every verb on the rider, buses and the Console
included; `proxy ride --off` is the way out. Run the owner with its own config
path (`--instance NAME`, or `MOSAIC_CONFIG` set to exactly
`~/.config/mosaic-NAME/config.json`): the rider locks the owner's proxy state
with that path, and the first side to lock it fixes the lock identity, so the
other side refuses `proxy-lock-invalid`. Refusals (validate exit 2):
`proxy-ride-invalid` (a bad file, a bad name, the instance's own name, or an owner with the same data root),
`proxy-ride-owner-missing` (the named
instance has no valid config, claim or `proxy.json`, or rides another instance
itself), `proxy-ride-conflict` (`proxy.json` and `proxy-ride.json` both exist;
`proxy init` also refuses it on a riding instance). Each prints one line with
the owner's next step, except under `run-task.sh`, where a ride refusal still
shows as a JSON parse error, as every config refusal there does. `--off` leaves the rider's worker key in the owner's
proxy; it is not removed yet.
## Agent limits (`scripts/mosaic limits`)
View: `scripts/mosaic limits --business ID --json`. Set: `scripts/mosaic limits
+3 -3
View File
@@ -108,7 +108,7 @@ export class ProxyPage {
rows.push({ ...view(this.#providers, key.auth_index, provider, status, key.priority ?? group.priority), source: 'apikey', groupIndex, keyIndex, models: configured.map(m => m.alias || m.name) });
}
}
return { rows, groups };
return { rows, groups, credentialNames: response.files.map(row => row?.name).filter(identifier) };
}
#public(row) {
const { accountId, provider, displayName, status, priority } = row;
@@ -271,13 +271,13 @@ export class ProxyPage {
fields(args, ['accountId', 'confirm']);
if (typeof args.accountId !== 'string' || !/^[a-f0-9]{12}$/.test(args.accountId) || args.confirm !== true) refuse('invalid-request');
return this.#write(async () => {
const { rows, groups } = await this.#accounts();
const { rows, groups, credentialNames } = await this.#accounts();
const original = this.#lock ? structuredClone(groups) : null;
const matches = rows.filter(row => row.accountId === args.accountId);
if (matches.length !== 1) refuse('invalid-request');
const row = matches[0];
// The DELETE names a credential file; another listed account on the same name would go too.
if (row.source === 'oauth' && rows.some(r => r !== row && r.source === 'oauth' && r.name === row.name)) refuse('invalid-request');
if (row.source === 'oauth' && credentialNames.filter(name => name === row.name).length !== 1) refuse('invalid-request');
if (this.#lock && !isDeepStrictEqual(await this.#client.getUpstreamGroups(), original)) refuse('proxy-down');
this.#beforeMutation('proxy.account.remove');
try {
+5
View File
@@ -88,6 +88,11 @@ group("models-offered-conflict models-in-use", "R1/R4: the owner's offered model
group("proxy-key-mismatch", "R1: a pasted management key is confirmed before init", "Type the same key in both fields, or leave them empty to have the bus generate the key.");
group("proxy-lock-busy proxy-lock-invalid", "R4: one proxy key write at a time", "Wait for the running proxy write to finish and retry. An invalid lock file needs the owner to inspect the proxy state directory.");
group("proxy-not-configured proxy-unit-usage proxy-unit-exists proxy-unit-dropin proxy-unit-copy proxy-unit-path proxy-unit-missing proxy-unit-unhealthy proxy-unit-failed", "R1/R4: proxy user unit is explicit creation", "Initialize the proxy first, then install the unit from the Proxy page or with mosaic proxy unit install. An existing unit or drop-in is never replaced; remove it by hand. An unhealthy unit: read its journal by unit name and restart it.");
// proxy-ride first slice (Jason 2026-11-21): an instance's worker runs use another instance's proxy.
// Each next step is the same line packages/model-proxy/src/settings.mjs RIDE_LINES prints.
group("proxy-ride-invalid", "R1/R4: a ride file names one other instance of the same OS user", "proxy-ride.json is not a valid ride file, or it names this instance. Run mosaic proxy ride --off, then mosaic proxy ride --instance NAME with the instance whose proxy this one should use.");
group("proxy-ride-owner-missing", "R1/R4: the ridden instance owns a valid model proxy", "The instance named in proxy-ride.json has no valid model proxy, or it rides another instance itself. Set up the proxy on that instance, or run mosaic proxy ride --off.");
group("proxy-ride-conflict", "R1/R4: an instance has its own proxy or rides one, never both", "This instance has its own proxy.json and a proxy-ride.json. Run mosaic proxy ride --off to keep its own proxy.");
group("proxy-client-invalid proxy-client-refused proxy-client-response", "R1: proxy management answers", "Check the proxy unit is running and its port answers on the Proxy page, then retry. Upstream bodies are never shown.");
// proxy-scope slice 4 (Jason 2026-11-19): the Triage key lives with a bus that rides the owner's
// proxy, and a Triage session on a key past its retireAt ends before the key leaves the proxy.
+61
View File
@@ -0,0 +1,61 @@
// Existing owner thread/session/message reads only. Runner notices are not answers.
import { LENGTH_FAILURE_NOTICE_PREFIX, TURN_FAILURE_NOTICE_PREFIX } from '../../harness/src/failure-notice.mjs';
function realBody(body) {
if (typeof body !== 'string') return false;
const text = body.split('\n\n[unslop:')[0].trim();
return !!text && text !== '(the session gave no answer)' &&
!text.startsWith(TURN_FAILURE_NOTICE_PREFIX) && !text.startsWith(LENGTH_FAILURE_NOTICE_PREFIX);
}
const newest = rows => [...rows].sort((a, b) => String(b.at ?? '').localeCompare(String(a.at ?? '')) || String(b.id).localeCompare(String(a.id)));
export const setupState = Object.freeze([
{
id: 'assistant.replied', title: 'Say hello to your assistant',
purpose: 'Send one message and get an answer. That proves every step above works together.',
fix: { page: '#/assistant', label: 'Start a conversation' },
order: { after: ['agents.routed'] }, required: true,
async check(ctx) {
let fix = null, incomplete = false, pages = 0;
for (const org of ctx.members) {
if (ctx.signal?.aborted) { incomplete = true; break; }
const assistant = Object.entries(org.doc?.roles ?? {}).find(([, role]) => role.definition === 'assistant')?.[0] ?? 'assistant';
fix ??= { page: `#/orgs/${encodeURIComponent(org.id)}`, label: 'Start a conversation' };
try {
const threads = await ctx.read(org, 'threads');
if (!Array.isArray(threads)) throw Error('threads unavailable');
const own = newest(threads.filter(t => t.instance === assistant));
incomplete ||= own.length > 50;
for (const thread of own.slice(0, 50)) {
try {
const sessions = await ctx.read(org, 'sessions', { thread: thread.id });
if (!Array.isArray(sessions)) throw Error('sessions unavailable');
incomplete ||= sessions.length > 50;
for (const session of newest(sessions).slice(0, 50)) {
let after;
const human = new Set();
for (let page = 0; page < 20; page++) {
// A request-wide page budget also bounds many empty threads and sessions.
if (pages++ >= 100 || ctx.signal?.aborted) { incomplete = true; break; }
const messages = await ctx.read(org, 'messages', { thread: thread.id, session: session.id, ...(after ? { after } : {}), limit: 200 });
if (!Array.isArray(messages)) throw Error('messages unavailable');
for (const m of messages) {
if (m.from_role === 'human' && typeof m.id === 'string') human.add(m.id);
if (m.from_role === assistant && m.to_role === 'human' && typeof m.from_run === 'string' && m.from_run && m.from_run !== 'bus' && human.has(m.in_reply_to) && realBody(m.body))
return { state: 'done', reason: 'Your assistant has replied to a message in your conversation.' + (incomplete ? " Some conversation history couldn't be checked." : '') };
}
if (messages.length < 200) break;
if (page === 19 || !messages.at(-1)?.id || messages.at(-1).id === after) { incomplete = true; break; }
after = messages.at(-1).id;
}
if (pages >= 100 || ctx.signal?.aborted) break;
}
} catch { incomplete = true; }
if (pages >= 100 || ctx.signal?.aborted) break;
}
} catch { incomplete = true; }
}
return { state: 'next', reason: incomplete ? "Some conversation history couldn't be checked. Open your organization to check the conversation or send a message." : 'Open your organization and start a conversation with your assistant.', ...(fix ? { fix } : {}) };
},
},
]);
+51 -27
View File
@@ -7,6 +7,12 @@
// <configDir>/secrets/triage/model (directory 0700, file 0600, the running user's)
// { version, key, keyId, issuedAt, rotateBy, retiring: [{ key, keyId, issuedAt, retireAt }] }
//
// worker-key-purpose (S3 of reports/worker-proxy-simple.md, Jason's ruling 2026-11-21): the same
// file shape and checks serve the worker key. A purpose, `triage` (the default, unchanged) or
// `worker`, picks the key prefix, the directory, the wording and the code prefix:
// triage mst-<tag>-triage-<43> <configDir>/secrets/triage/model triage-key-*
// worker msw-<tag>-worker-<43> <configDir>/secrets/worker/model worker-key-*
//
// Messages name files, never values.
import { closeSync, constants, fchmodSync, fstatSync, lstatSync, mkdirSync, openSync, readFileSync } from "node:fs";
import { createHash } from "node:crypto";
@@ -29,16 +35,30 @@ export class TriageKeyError extends Error {
this.exitCode = exitCode;
}
}
export const unreadable = (why) => new TriageKeyError("triage-key-unreadable", `the Triage key store is unreadable: ${why}`);
// What a purpose changes. `code` prefixes every refusal code this module and triage-key.mjs raise
// for that purpose; `head` and `name` make the key prefix; `dir` is the directory under secrets.
export const PURPOSES = Object.freeze({
triage: Object.freeze({ name: "triage", head: "mst", dir: "triage", code: "triage-key", store: "the Triage key store", dirWords: "the triage secrets directory", label: "Triage" }),
worker: Object.freeze({ name: "worker", head: "msw", dir: "worker", code: "worker-key", store: "the worker key store", dirWords: "the worker secrets directory", label: "worker" }),
});
export function purposeOf(purpose = "triage") {
if (!Object.hasOwn(PURPOSES, purpose)) throw new TypeError("the key purpose must be triage or worker");
return PURPOSES[purpose];
}
// A refusal for `purpose`: `suffix` is the part after the prefix, so ("worker", "raced") is
// worker-key-raced. The Triage codes are exactly the codes this module raised before purposes.
export const keyError = (purpose, suffix, message, exitCode) => new TriageKeyError(`${purposeOf(purpose).code}-${suffix}`, message, exitCode);
export const unreadable = (why, purpose = "triage") => keyError(purpose, "unreadable", `${purposeOf(purpose).store} is unreadable: ${why}`);
export const keyIdOf = (key) => createHash("sha256").update(key).digest("hex").slice(0, 12);
export const instanceTag = (configDir) => createHash("sha256").update(configDir).digest("hex").slice(0, 12);
export const keyPrefix = (configDir) => `mst-${instanceTag(configDir)}-triage-`;
const ownKey = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir)) && SECRET.test(k.slice(keyPrefix(configDir).length));
export const keyPrefix = (configDir, purpose = "triage") => { const p = purposeOf(purpose); return `${p.head}-${instanceTag(configDir)}-${p.name}-`; };
const ownKey = (configDir, purpose) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir, purpose)) && SECRET.test(k.slice(keyPrefix(configDir, purpose).length));
export function keyPaths(configDir) {
export function keyPaths(configDir, purpose = "triage") {
const secrets = join(configDir, "secrets");
const dir = join(secrets, "triage");
const dir = join(secrets, purposeOf(purpose).dir);
return { secrets, dir, file: join(dir, "model") };
}
@@ -47,8 +67,8 @@ const exactKeys = (v, keys) => plain(v) && Object.keys(v).length === keys.length
const canonicalTime = (s) => typeof s === "string" && ISO.test(s) && !Number.isNaN(Date.parse(s)) && new Date(s).toISOString() === s;
const canonicalDay = (s) => typeof s === "string" && DAY.test(s) && !Number.isNaN(Date.parse(`${s}T00:00:00Z`)) && new Date(`${s}T00:00:00Z`).toISOString().slice(0, 10) === s;
export function validRecord(doc, configDir) {
const own = ownKey(configDir);
export function validRecord(doc, configDir, purpose = "triage") {
const own = ownKey(configDir, purpose);
return exactKeys(doc, FIELDS)
&& doc.version === KEY_VERSION
&& own(doc.key) && doc.keyId === keyIdOf(doc.key)
@@ -62,23 +82,24 @@ export function validRecord(doc, configDir) {
const uidOf = (opts) => opts.uid ?? process.getuid();
// A real directory, the running user's, mode 0700. Missing: false, or made when `create` is on.
export function checkDir(dir, uid, what, { create }) {
export function checkDir(dir, uid, what, { create, purpose = "triage" }) {
const bad = (why) => unreadable(why, purpose);
let st;
try { st = lstatSync(dir); }
catch (e) {
if (e.code !== "ENOENT") throw unreadable(`${what} cannot be read`);
if (e.code !== "ENOENT") throw bad(`${what} cannot be read`);
if (!create) return false;
try { mkdirSync(dir, { mode: 0o700 }); } catch { throw unreadable(`${what} cannot be made`); }
try { mkdirSync(dir, { mode: 0o700 }); } catch { throw bad(`${what} cannot be made`); }
try {
const fd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW);
try { fchmodSync(fd, 0o700); } finally { closeSync(fd); }
} catch { throw unreadable(`${what} cannot be made`); }
} catch { throw bad(`${what} cannot be made`); }
st = lstatSync(dir);
}
if (st.isSymbolicLink()) throw unreadable(`${what} is a link`);
if (!st.isDirectory()) throw unreadable(`${what} is not a directory`);
if (st.uid !== uid) throw unreadable(`${what} is not owned by this user`);
if ((st.mode & 0o777) !== 0o700) throw unreadable(`${what} mode is not 0700`);
if (st.isSymbolicLink()) throw bad(`${what} is a link`);
if (!st.isDirectory()) throw bad(`${what} is not a directory`);
if (st.uid !== uid) throw bad(`${what} is not owned by this user`);
if ((st.mode & 0o777) !== 0o700) throw bad(`${what} mode is not 0700`);
return true;
}
@@ -91,27 +112,30 @@ export function checkFile(fd, uid, what, fail = unreadable) {
return st;
}
// The stored record, or null when no key is issued. Throws triage-key-unreadable on anything unsafe.
// opts.uid (both directories and the file) and opts.fileUid (the file only) are test seams for the
// owner checks.
// The stored record, or null when no key is issued. Throws <purpose>-key-unreadable on anything
// unsafe. opts.purpose picks the key (default triage). opts.uid (both directories and the file) and
// opts.fileUid (the file only) are test seams for the owner checks.
export function readKey(configDir, opts = {}) {
const { secrets, dir, file } = keyPaths(configDir);
const purpose = opts.purpose ?? "triage";
const p = purposeOf(purpose);
const bad = (why) => unreadable(why, purpose);
const { secrets, dir, file } = keyPaths(configDir, purpose);
const uid = uidOf(opts);
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) return null;
if (!checkDir(dir, uid, "the triage secrets directory", { create: false })) return null;
if (!checkDir(secrets, uid, "the secrets directory", { create: false, purpose })) return null;
if (!checkDir(dir, uid, p.dirWords, { create: false, purpose })) return null;
let fd;
try { fd = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); }
catch (e) {
if (e.code === "ENOENT") return null;
if (e.code === "ELOOP") throw unreadable("the key file is a link");
throw unreadable("the key file cannot be opened");
if (e.code === "ELOOP") throw bad("the key file is a link");
throw bad("the key file cannot be opened");
}
try {
const st = checkFile(fd, opts.fileUid ?? uid, "the key file");
if (st.size > MAX_BYTES) throw unreadable("the key file is too large");
const st = checkFile(fd, opts.fileUid ?? uid, "the key file", bad);
if (st.size > MAX_BYTES) throw bad("the key file is too large");
let doc;
try { doc = JSON.parse(readFileSync(fd, "utf8")); } catch { throw unreadable("the key file is not JSON"); }
if (!validRecord(doc, configDir)) throw unreadable("the key file does not match the key shape");
try { doc = JSON.parse(readFileSync(fd, "utf8")); } catch { throw bad("the key file is not JSON"); }
if (!validRecord(doc, configDir, purpose)) throw bad("the key file does not match the key shape");
return doc;
} finally { closeSync(fd); }
}
+320 -107
View File
@@ -29,17 +29,54 @@
// Nothing here prints, logs or records a key. Receipts and output carry key ids only (the first 12
// hex of sha256(key), the id the bus uses). Messages name files and options, never values.
// The Console reads the key at each Triage spawn through triage-key-file.mjs, never this module.
import { closeSync, constants, fchmodSync, fsyncSync, lstatSync, openSync, readdirSync, renameSync, unlinkSync, writeSync } from "node:fs";
//
// worker-key-purpose (S3 of reports/worker-proxy-simple.md, Jason's ruling 2026-11-21): every export
// takes opts.purpose, `triage` (the default; nothing above changes) or `worker`. The worker key is
// `msw-<tag>-worker-<43 base64url>` in <configDir>/secrets/worker/model, and its refusal codes are
// the Triage codes with `worker-key-` for `triage-key-`. Each purpose removes names in its own
// namespace only; change() refuses any plan that would remove another name. Two worker-only rules:
// - ensureReady, the launch path: rotates when rotateBy has come and retires keys whose grace
// has passed, and
// - a retiring worker key whose grace has passed still stays while retireHeld says a live
// runtime started before the rotation (opts.runtimes, the processing-control list).
// ensureReady (both purposes) skips the lock when the plan it would run under the lock changes
// nothing, then probes the key. Nothing calls it for Triage yet.
import { closeSync, constants, fchmodSync, fsyncSync, lstatSync, openSync, readFileSync, readdirSync, renameSync, unlinkSync, writeSync } from "node:fs";
import { randomBytes } from "node:crypto";
import { join } from "node:path";
import { performance } from "node:perf_hooks";
import { isDeepStrictEqual } from "node:util";
import { createProxyLock } from "./proxy-lock.mjs";
import { loadManagementClient } from "../../model-proxy/src/client.mjs";
import { processIdentity, runtimeGroupState } from "../../processing-control/src/runtimes.mjs";
// #1873 triage-wiring: the key file's reader lives in triage-key-file.mjs (node built-ins only), so
// the Console reads the key without this module's proxy client. Re-exported here unchanged.
import { KEY_VERSION, MAX_RETIRING, SECRET, TriageKeyError, checkDir, checkFile, instanceTag, keyIdOf, keyPaths, keyPrefix, readKey, unreadable, validRecord } from "./triage-key-file.mjs";
export { KEY_VERSION, TriageKeyError, instanceTag, keyIdOf, keyPaths, keyPrefix, readKey, validRecord };
import { KEY_VERSION, MAX_RETIRING, PURPOSES, SECRET, TriageKeyError, checkDir, checkFile, instanceTag, keyError, keyIdOf, keyPaths, keyPrefix, purposeOf, readKey, unreadable, validRecord } from "./triage-key-file.mjs";
export { KEY_VERSION, PURPOSES, TriageKeyError, instanceTag, keyIdOf, keyPaths, keyPrefix, readKey, validRecord };
// The words that name a purpose's own verbs. Triage's are the words this module used before
// purposes; the worker key has no verbs, so its words name the launch.
const WORDS = {
triage: {
unrecorded: "the change was made but its receipt line was not written; run mosaic triage key status",
issued: "a Triage key is already issued; use mosaic triage key rotate",
notIssued: "no Triage key is issued; use mosaic triage key issue",
tooMany: "too many keys are retiring; run mosaic triage key retire once their grace has passed",
unwritten: (file, why, op) => `the proxy list changed but ${file} was not written (${why}); fix the file's directory, then run mosaic triage key ${op} again, which clears what this run left`,
unsynced: (file) => `${file} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status`,
raced: "the proxy's key list changed while this verb ran; run it again",
},
worker: {
unrecorded: "the change was made but its receipt line was not written; see <dataRoot>/proxy/mosaic-keys.receipts.jsonl",
issued: "a worker key is already issued",
notIssued: "no worker key is issued",
tooMany: "too many worker keys are retiring; they leave once their grace has passed and no older run is live",
unwritten: (file, why) => `the proxy list changed but ${file} was not written (${why}); fix the file's directory, then run the launch again, which clears what this run left`,
unsynced: (file) => `${file} was written but its directory could not be synced, so it may not survive a power loss; run the launch again`,
raced: "the proxy's key list changed while the worker key was being changed; run the launch again",
},
};
// The org key lifetime and the rotate grace default. The bus holds both as literals
// (packages/bus/src/model-keys.mjs #issue: 30 days to rotateBy; packages/bus/src/runtime.mjs:
@@ -50,27 +87,27 @@ export { KEY_VERSION, TriageKeyError, instanceTag, keyIdOf, keyPaths, keyPrefix,
export const KEY_LIFETIME_DAYS = 30;
export const DEFAULT_ROTATE_GRACE_MINUTES = 120;
const sentinelPrefix = (configDir) => `msk-${instanceTag(configDir)}-sentinel-`;
// Any name under the prefix is Triage's to remove, well formed or not, as the bus treats its own
// `msk-<tag>-` names. Only a well-formed key may be stored in the file.
const inNamespace = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir));
// Any name under the prefix is the purpose's to remove, well formed or not, as the bus treats its
// own `msk-<tag>-` names. Only a well-formed key may be stored in the file.
const inNamespace = (configDir, purpose = "triage") => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir, purpose));
// Both directories, checked (and the triage one made) before any proxy write, so a store that cannot
// be written refuses while the proxy is still untouched.
function prepareDirs(configDir, uid) {
const { secrets, dir } = keyPaths(configDir);
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) throw unreadable("the secrets directory is missing");
checkDir(dir, uid, "the triage secrets directory", { create: true });
// Both directories, checked (and the purpose's one made) before any proxy write, so a store that
// cannot be written refuses while the proxy is still untouched.
function prepareDirs(configDir, uid, purpose = "triage") {
const { secrets, dir } = keyPaths(configDir, purpose);
if (!checkDir(secrets, uid, "the secrets directory", { create: false, purpose })) throw unreadable("the secrets directory is missing", purpose);
checkDir(dir, uid, purposeOf(purpose).dirWords, { create: true, purpose });
}
// A temp file writeKey left when it was killed before its rename. It may hold a key.
const TEMP = /^\.model\.\d+\.[0-9a-f]{12}\.tmp$/;
const tempsIn = (dir) => { try { return readdirSync(dir).filter((n) => TEMP.test(n)); } catch { return []; } };
// Under the lock no other Triage writer runs, so every temp file left in the directory is dead.
function removeTemps(configDir) {
const { dir } = keyPaths(configDir);
// Under the lock no other writer of this purpose runs, so every temp file left in the directory is dead.
function removeTemps(configDir, purpose = "triage") {
const { dir } = keyPaths(configDir, purpose);
for (const name of tempsIn(dir)) {
try { if (!lstatSync(join(dir, name)).isDirectory()) unlinkSync(join(dir, name)); }
catch { throw unreadable("a leftover temp file in the triage secrets directory cannot be removed"); }
catch { throw unreadable(`a leftover temp file in ${purposeOf(purpose).dirWords} cannot be removed`, purpose); }
}
}
@@ -78,19 +115,19 @@ function removeTemps(configDir) {
class NotSynced extends Error {}
// Atomic: a 0600 temp file in the same directory, fsync, rename, fsync of the directory.
function writeKey(configDir, record, uid, hook = () => {}) {
const { dir, file } = keyPaths(configDir);
prepareDirs(configDir, uid);
function writeKey(configDir, record, uid, hook = () => {}, purpose = "triage") {
const { dir, file } = keyPaths(configDir, purpose);
prepareDirs(configDir, uid, purpose);
const temp = join(dir, `.model.${process.pid}.${randomBytes(6).toString("hex")}.tmp`);
let fd;
try { fd = openSync(temp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600); }
catch { throw unreadable("the key file cannot be written"); }
catch { throw unreadable("the key file cannot be written", purpose); }
try {
try { fchmodSync(fd, 0o600); writeSync(fd, `${JSON.stringify(record, null, 2)}\n`); fsyncSync(fd); }
finally { closeSync(fd); }
hook("rename");
renameSync(temp, file);
} catch { try { unlinkSync(temp); } catch {} throw unreadable("the key file cannot be written"); }
} catch { try { unlinkSync(temp); } catch {} throw unreadable("the key file cannot be written", purpose); }
try {
hook("sync");
const dfd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY);
@@ -101,15 +138,15 @@ function writeKey(configDir, record, uid, hook = () => {}) {
// One line per write in the proxy lock's receipts file, ids only. `receiptsReady` runs under the
// lock before any change, so an unsafe receipts file refuses while nothing has changed yet.
const RECEIPTS_FLAGS = constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW | constants.O_NONBLOCK;
const receiptsUnsafe = (why) => new TriageKeyError("triage-key-receipts-unsafe", `${why}; nothing was changed`);
function receiptsReady(stateDir, uid) {
const receiptsUnsafe = (purpose) => (why) => keyError(purpose, "receipts-unsafe", `${why}; nothing was changed`);
function receiptsReady(stateDir, uid, purpose = "triage") {
let fd;
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); }
catch { throw receiptsUnsafe("the proxy receipts file cannot be opened"); }
try { checkFile(fd, uid, "the proxy receipts file", receiptsUnsafe); } finally { closeSync(fd); }
catch { throw receiptsUnsafe(purpose)("the proxy receipts file cannot be opened"); }
try { checkFile(fd, uid, "the proxy receipts file", receiptsUnsafe(purpose)); } finally { closeSync(fd); }
}
function receipt(stateDir, uid, entry) {
const unrecorded = new TriageKeyError("triage-key-unrecorded", "the change was made but its receipt line was not written; run mosaic triage key status");
function receipt(stateDir, uid, entry, purpose = "triage") {
const unrecorded = keyError(purpose, "unrecorded", WORDS[purpose].unrecorded);
let fd;
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); } catch { throw unrecorded; }
try {
@@ -125,7 +162,7 @@ export const graceMinutes = (proxy) => proxy?.rotateGraceMinutes ?? DEFAULT_ROTA
export const rotateByFor = (nowMs) => new Date(nowMs + KEY_LIFETIME_DAYS * 86400_000).toISOString().slice(0, 10);
export const retireAtFor = (nowMs, proxy) => new Date(nowMs + graceMinutes(proxy) * 60_000).toISOString();
const proxyFailure = (code) => {
const proxyFailure = (code, purpose = "triage") => {
const words = {
"proxy-lock-busy": "another proxy writer holds the instance proxy lock; try again when it ends, and if no writer is running, see mosaic proxy fence status",
"proxy-lock-uncertain": "the instance proxy lock is in a state this verb will not guess about; check <dataRoot>/proxy by hand",
@@ -138,36 +175,55 @@ const proxyFailure = (code) => {
"proxy-client-redirect": "the proxy answered with a redirect",
"proxy-client-response": "the proxy gave an answer this verb cannot use",
};
return new TriageKeyError(code, words[code] ?? "the proxy write failed");
return new TriageKeyError(code === "triage-key-failed" ? `${purposeOf(purpose).code}-failed` : code, words[code] ?? "the proxy write failed");
};
// ensureReady's probe refusals, the codes a bus launch uses for the same answers.
const probeFailure = (code) => new TriageKeyError(code, code === "proxy-down" ? "the proxy did not answer the key probe" : "the proxy did not accept the key");
// opts.uid, opts.now and opts.hook(stage) are test seams; a stage hook may throw or kill the process.
// opts.client and opts.lock: the bus passes its own management client and instance proxy lock, so a
// bus change runs under the lock object its model key sweep uses and reads no key file of its own.
function context({ configDir, system, uid, fileUid, now, hook, client, lock }) {
if (!system?.proxy || !Number.isInteger(system.proxy.port)) throw new TriageKeyError("triage-key-no-proxy", "the instance has no model proxy set up (proxy.json)");
// opts.purpose: "triage" (default) or "worker". opts.runtimes (worker only): () => the
// processing-control runtime list, listRuntimes(dataRoot); without it no worker key whose grace has
// passed is removed. opts.retireGuard: test seams for retireHeld ({ marginMs, live, startedAt }).
// proxy-ride first slice (Jason 2026-11-21): a riding instance's system.proxy carries `owner`
// ({ instance, configDir, dataRoot }) from scripts/mosaic-config.mjs. Without it, the instance's own.
const proxyConfigDir = (configDir, system) => system?.proxy?.owner?.configDir ?? configDir;
const proxyStateDir = (system) => join(system?.proxy?.owner?.dataRoot ?? system.dataRoot, "proxy");
function context({ configDir, system, uid, fileUid, now, hook, client, lock, purpose = "triage", runtimes, retireGuard, planSeam }) {
const p = purposeOf(purpose);
if (!system?.proxy || !Number.isInteger(system.proxy.port)) throw keyError(purpose, "no-proxy", "the instance has no model proxy set up (proxy.json)");
return {
configDir, dataRoot: system.dataRoot, proxy: system.proxy, stateDir: join(system.dataRoot, "proxy"),
configDir, dataRoot: system.dataRoot, proxy: system.proxy, stateDir: proxyStateDir(system),
// proxy-ride: the instance whose proxy this is. Its lock identity, sentinel and management key
// are the owner's; the key file, key names and namespace stay this instance's (configDir).
proxyConfigDir: proxyConfigDir(configDir, system),
uid: uid ?? process.getuid(), fileUid, now: now ?? Date.now, hook: hook ?? (() => {}), client, lock,
purpose, p, words: WORDS[purpose],
// Tests only: rewrites a plan's result, to prove change()'s namespace check refuses it.
planSeam: typeof planSeam === "function" ? planSeam : (result) => result,
// A Triage key leaves when its grace ends. A worker key also waits for older runs to end.
held: purpose === "worker" ? (record) => retireHeld({ runtimes, rotatedAtMs: Date.parse(record.issuedAt), ...(retireGuard ?? {}) }) : () => false,
};
}
function lockFor(c) {
if (c.lock) return c.lock;
try { return createProxyLock({ stateDir: c.stateDir, identity: `${c.configDir}:${c.proxy.port}` }); }
catch (e) { throw proxyFailure(e.code ?? "proxy-lock-invalid"); }
try { return createProxyLock({ stateDir: c.stateDir, identity: `${c.proxyConfigDir}:${c.proxy.port}` }); }
catch (e) { throw proxyFailure(e.code ?? "proxy-lock-invalid", c.purpose); }
}
async function clientFor(c) {
if (c.client) return c.client;
try { return await loadManagementClient({ configDir: c.configDir, proxy: c.proxy }); }
catch (e) { throw proxyFailure(e.code ?? "proxy-management-key-invalid"); }
try { return await loadManagementClient({ configDir: c.proxyConfigDir, proxy: c.proxy }); }
catch (e) { throw proxyFailure(e.code ?? "proxy-management-key-invalid", c.purpose); }
}
function checkSentinel(c, keys) {
const prefix = sentinelPrefix(c.configDir);
const prefix = sentinelPrefix(c.proxyConfigDir);
if (!keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length)))) {
throw new TriageKeyError("triage-key-proxy-open", "the proxy's key list has no sentinel for this instance, so it may be another proxy");
throw keyError(c.purpose, "proxy-open", "the proxy's key list has no sentinel for this instance, so it may be another proxy");
}
}
@@ -182,57 +238,63 @@ async function change(c, opName, plan) {
return await lock.run(async () => {
const before = await client.getKeys();
checkSentinel(c, before);
const current = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid });
const result = plan(before, current);
const current = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid, purpose: c.purpose });
const result = c.planSeam(plan(before, current), before);
const op = result.op ?? opName;
const remove = result.remove === true && current !== null;
if (result.record) prepareDirs(c.configDir, c.uid);
removeTemps(c.configDir);
receiptsReady(c.stateDir, c.uid);
if (result.record) prepareDirs(c.configDir, c.uid, c.purpose);
removeTemps(c.configDir, c.purpose);
receiptsReady(c.stateDir, c.uid, c.purpose);
const after = result.after;
const removed = before.filter((k) => !after.includes(k));
const added = after.filter((k) => !before.includes(k));
const { removed, added } = diff(before, after);
// A key in another namespace never leaves the list.
if (removed.some((k) => !inNamespace(c.configDir)(k))) throw new TriageKeyError("triage-key-failed", "the change would remove a key outside the Triage namespace");
if (removed.some((k) => !inNamespace(c.configDir, c.purpose)(k))) throw keyError(c.purpose, "failed", `the change would remove a key outside the ${c.p.label} namespace`);
if (added.length || removed.length) {
// As the bus does: a list that moved since the first read is not overwritten.
if (!isDeepStrictEqual(await client.getKeys(), before)) throw new TriageKeyError("triage-key-raced", "the proxy's key list changed while this verb ran; run it again");
lock.fence({ org: null, op: `triage.${op}` });
if (!isDeepStrictEqual(await client.getKeys(), before)) throw keyError(c.purpose, "raced", c.words.raced);
lock.fence({ org: null, op: `${c.p.name}.${op}` });
await client.putKeys(after);
c.hook("put");
const confirmed = await client.getKeys();
if (!confirmed.some((k) => k.startsWith(sentinelPrefix(c.configDir))) || added.some((k) => !confirmed.includes(k)) || removed.some((k) => confirmed.includes(k))) {
throw new TriageKeyError("triage-key-unconfirmed", "the proxy did not confirm the change; the instance proxy fence stays set, see mosaic proxy fence status");
if (!confirmed.some((k) => k.startsWith(sentinelPrefix(c.proxyConfigDir))) || added.some((k) => !confirmed.includes(k)) || removed.some((k) => confirmed.includes(k))) {
throw keyError(c.purpose, "unconfirmed", "the proxy did not confirm the change; the instance proxy fence stays set, see mosaic proxy fence status");
}
lock.confirm();
}
c.hook("write");
let unsynced = false;
if (result.record || remove) {
try { if (remove) unlinkKey(c.configDir, c.hook); else writeKey(c.configDir, result.record, c.uid, c.hook); }
try { if (remove) unlinkKey(c.configDir, c.hook, c.purpose); else writeKey(c.configDir, result.record, c.uid, c.hook, c.purpose); }
catch (e) {
if (e instanceof NotSynced) unsynced = true;
else throw new TriageKeyError("triage-key-unwritten", `the proxy list changed but ${keyPaths(c.configDir).file} was not written (${e.message}); fix the file's directory, then run mosaic triage key ${op} again, which clears what this run left`);
else throw keyError(c.purpose, "unwritten", c.words.unwritten(keyPaths(c.configDir, c.purpose).file, e.message, op));
}
}
c.hook("receipt");
// A run that changed neither the proxy nor the file leaves no receipt.
if (added.length || removed.length || result.record || remove) receipt(c.stateDir, c.uid, { op: `triage.key.${op}`, time: new Date(c.now()).toISOString(), addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) });
if (added.length || removed.length || result.record || remove) receipt(c.stateDir, c.uid, { op: `${c.p.name}.key.${op}`, time: new Date(c.now()).toISOString(), addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) }, c.purpose);
// The file is in place and the receipt written; only its durability is in doubt.
if (unsynced) throw new TriageKeyError("triage-key-unsynced", `${keyPaths(c.configDir).file} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status`);
if (unsynced) throw keyError(c.purpose, "unsynced", c.words.unsynced(keyPaths(c.configDir, c.purpose).file));
return { ...result.done, addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) };
});
} catch (e) {
if (e instanceof TriageKeyError) throw e;
throw proxyFailure(typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "triage-key-failed");
throw proxyFailure(typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "triage-key-failed", c.purpose);
}
}
const diff = (before, after) => ({ removed: before.filter((k) => !after.includes(k)), added: after.filter((k) => !before.includes(k)) });
// True when change() would write the proxy list, the file or a receipt for this plan.
const writes = (before, current, result) => {
const { removed, added } = diff(before, result.after);
return added.length > 0 || removed.length > 0 || Boolean(result.record) || (result.remove === true && current !== null);
};
// Revoke's file step: unlink, then sync the directory, with writeKey's stages and failures.
function unlinkKey(configDir, hook = () => {}) {
const { dir, file } = keyPaths(configDir);
function unlinkKey(configDir, hook = () => {}, purpose = "triage") {
const { dir, file } = keyPaths(configDir, purpose);
try { hook("rename"); unlinkSync(file); }
catch (e) { if (e?.code !== "ENOENT") throw unreadable("the key file cannot be removed"); }
catch (e) { if (e?.code !== "ENOENT") throw unreadable("the key file cannot be removed", purpose); }
try {
hook("sync");
const dfd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY);
@@ -240,40 +302,43 @@ function unlinkKey(configDir, hook = () => {}) {
} catch { throw new NotSynced(); }
}
const newKey = (configDir) => keyPrefix(configDir) + randomBytes(32).toString("base64url");
const newKey = (configDir, purpose = "triage") => keyPrefix(configDir, purpose) + randomBytes(32).toString("base64url");
// Keys in the Triage namespace that the file does not hold: left by a crash between the PUT and the
// file write, or by a file that was removed by hand.
const orphansOf = (configDir, keys, record) => {
// Keys in the purpose's namespace that the file does not hold: left by a crash between the PUT and
// the file write, or by a file that was removed by hand.
const orphansOf = (configDir, keys, record, purpose = "triage") => {
const held = record ? [record.key, ...record.retiring.map((r) => r.key)] : [];
return keys.filter((k) => inNamespace(configDir)(k) && !held.includes(k));
return keys.filter((k) => inNamespace(configDir, purpose)(k) && !held.includes(k));
};
export async function issueKey(opts) {
const c = context(opts);
return change(c, "issue", (before, current) => {
if (current) throw new TriageKeyError("triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate");
const orphans = orphansOf(c.configDir, before, null);
const key = newKey(c.configDir), nowMs = c.now();
if (current) throw keyError(c.purpose, "issued", c.words.issued);
const orphans = orphansOf(c.configDir, before, null, c.purpose);
const key = newKey(c.configDir, c.purpose), nowMs = c.now();
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring: [] };
return { after: [...before.filter((k) => !orphans.includes(k)), key], record, done: { keyId: record.keyId, rotateBy: record.rotateBy } };
});
}
// Retiring keys whose grace (plus marginMs, 0 for the terminal verbs) has passed, or that the proxy
// no longer holds.
const dueOf = (record, keys, nowMs, marginMs = 0) => record.retiring.filter((r) => Date.parse(r.retireAt) + marginMs <= nowMs || !keys.includes(r.key));
// no longer holds. For a worker key, a key whose grace has passed stays while c.held(record) says
// an older run may still use it; a key the proxy no longer holds goes from the file either way.
const graceOver = (record, nowMs, marginMs) => record.retiring.some((r) => Date.parse(r.retireAt) + marginMs <= nowMs);
const dueOf = (record, keys, nowMs, marginMs = 0, held = false) => record.retiring.filter((r) => (!held && Date.parse(r.retireAt) + marginMs <= nowMs) || !keys.includes(r.key));
const heldFor = (c, record, nowMs, marginMs) => graceOver(record, nowMs, marginMs) && c.held(record);
const rotateDueAt = (record) => Date.parse(`${record.rotateBy}T00:00:00Z`);
// The rotate plan, shared by the terminal verb and the sweep.
function rotatePlan(c, before, current, nowMs, marginMs = 0) {
const orphans = orphansOf(c.configDir, before, current);
const due = dueOf(current, before, nowMs, marginMs);
const orphans = orphansOf(c.configDir, before, current, c.purpose);
const due = dueOf(current, before, nowMs, marginMs, heldFor(c, current, nowMs, marginMs));
const keep = before.includes(current.key);
const retiring = [...current.retiring.filter((r) => !due.includes(r)),
...(keep ? [{ key: current.key, keyId: current.keyId, issuedAt: current.issuedAt, retireAt: retireAtFor(nowMs, c.proxy) }] : [])];
if (retiring.length > MAX_RETIRING) throw new TriageKeyError("triage-key-failed", "too many keys are retiring; run mosaic triage key retire once their grace has passed");
const key = newKey(c.configDir);
if (retiring.length > MAX_RETIRING) throw keyError(c.purpose, "failed", c.words.tooMany);
const key = newKey(c.configDir, c.purpose);
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring };
const drop = [...orphans, ...due.map((r) => r.key)];
return {
@@ -285,13 +350,13 @@ function rotatePlan(c, before, current, nowMs, marginMs = 0) {
export async function rotateKey(opts) {
const c = context(opts);
return change(c, "rotate", (before, current) => {
if (!current) throw new TriageKeyError("triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue");
if (!current) throw keyError(c.purpose, "not-issued", c.words.notIssued);
return rotatePlan(c, before, current, c.now());
});
}
function retirePlan(c, before, current, nowMs, marginMs = 0) {
const orphans = orphansOf(c.configDir, before, current);
const due = current ? dueOf(current, before, nowMs, marginMs) : [];
const orphans = orphansOf(c.configDir, before, current, c.purpose);
const due = current ? dueOf(current, before, nowMs, marginMs, heldFor(c, current, nowMs, marginMs)) : [];
const record = current && due.length ? { ...current, retiring: current.retiring.filter((r) => !due.includes(r)) } : null;
const drop = [...orphans, ...due.map((r) => r.key)];
return { after: before.filter((k) => !drop.includes(k)), record, done: { droppedKeyIds: due.map((r) => r.keyId) } };
@@ -307,41 +372,188 @@ export async function retireKeys(opts) {
// issued no file: a new key (keys in the namespace the file does not hold are removed)
// held the proxy holds the file's key: only orphans and retiring keys the proxy lost go
// replaced the proxy does not hold the file's key: a new key, the old one marked lost
function ensurePlan(c, before, current, nowMs) {
const orphans = orphansOf(c.configDir, before, current, c.purpose);
const fresh = (retiring) => {
const key = newKey(c.configDir, c.purpose);
return { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring };
};
if (!current) {
const record = fresh([]);
return { after: [...before.filter((k) => !orphans.includes(k)), record.key], record, done: { action: "issued", keyId: record.keyId, rotateBy: record.rotateBy } };
}
const lost = current.retiring.filter((r) => !before.includes(r.key));
const retiring = current.retiring.filter((r) => !lost.includes(r));
if (before.includes(current.key)) {
return {
after: before.filter((k) => !orphans.includes(k)), record: lost.length ? { ...current, retiring } : null,
done: { action: "held", keyId: current.keyId, rotateBy: current.rotateBy },
};
}
const record = fresh(retiring);
return {
after: [...before.filter((k) => !orphans.includes(k)), record.key], record,
done: { action: "replaced", keyId: record.keyId, rotateBy: record.rotateBy, old: { keyId: current.keyId, lost: true } },
};
}
export async function ensureKey(opts) {
const c = context(opts);
return change(c, "ensure", (before, current) => {
const nowMs = c.now();
const orphans = orphansOf(c.configDir, before, current);
const fresh = (retiring) => {
const key = newKey(c.configDir);
return { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring };
};
if (!current) {
const record = fresh([]);
return { after: [...before.filter((k) => !orphans.includes(k)), record.key], record, done: { action: "issued", keyId: record.keyId, rotateBy: record.rotateBy } };
}
const lost = current.retiring.filter((r) => !before.includes(r.key));
const retiring = current.retiring.filter((r) => !lost.includes(r));
if (before.includes(current.key)) {
return {
after: before.filter((k) => !orphans.includes(k)), record: lost.length ? { ...current, retiring } : null,
done: { action: "held", keyId: current.keyId, rotateBy: current.rotateBy },
};
}
const record = fresh(retiring);
return {
after: [...before.filter((k) => !orphans.includes(k)), record.key], record,
done: { action: "replaced", keyId: record.keyId, rotateBy: record.rotateBy, old: { keyId: current.keyId, lost: true } },
};
return change(c, "ensure", (before, current) => ensurePlan(c, before, current, c.now()));
}
// worker-key-purpose: the launch's plan. Triage: ensureKey's plan. Worker: the same, and on a held
// key it also rotates once rotateBy has come and retires keys whose grace has passed (unless an
// older run is live, see retireHeld). The worker key has no sweep; launches are its timer.
function readyPlan(c, before, current, nowMs) {
const base = ensurePlan(c, before, current, nowMs);
if (c.purpose !== "worker" || base.done.action !== "held") return base;
if (rotateDueAt(current) <= nowMs) {
// When older runs hold so many retiring keys that one more would pass MAX_RETIRING, the launch
// keeps the current key (still in the list) and rotates at a later launch.
let plan = null;
try { plan = rotatePlan(c, before, current, nowMs); }
catch (e) { if (!(e instanceof TriageKeyError && e.message === c.words.tooMany)) throw e; }
if (plan) return { ...plan, op: "rotate", done: { action: "rotated", ...plan.done } };
}
// Retire only for a key whose grace has passed and that the list still holds; orphans and lost
// keys are the ensure plan's.
const plan = retirePlan(c, before, current, nowMs);
if (!plan.record || !current.retiring.some((r) => before.includes(r.key) && !plan.record.retiring.includes(r))) return base;
return { ...plan, op: "retire", done: { action: "retired", keyId: current.keyId, rotateBy: current.rotateBy, ...plan.done } };
}
// model-keys.mjs's ISSUE_PROBE_TIMEOUT_MS, ISSUE_REPROBE_MIN_MS and REVOKE_POLL_MS, copied so the key
// verbs do not load the broker; worker-key.test.mjs pins them equal.
export const PROBE_TIMING = Object.freeze({ timeoutMs: 2000, reprobeMinMs: 250, pollMs: 25 });
const pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
// As ModelKeys.preflight probes a just-issued key: CLIProxyAPI applies a key-list PUT by an async
// reload after it answers, so only a 401 is probed again, until a short deadline.
async function probeReady(client, key) {
const deadline = performance.now() + PROBE_TIMING.timeoutMs;
let status, first = true;
for (;;) {
const options = first ? undefined : { timeout: Math.max(PROBE_TIMING.reprobeMinMs, Math.ceil(deadline - performance.now())) };
first = false;
try { status = await client.probeKey(key, options); }
catch { throw probeFailure("proxy-down"); }
if (status !== 401 || performance.now() >= deadline) break;
await pause(PROBE_TIMING.pollMs);
}
if (status < 200 || status >= 300) throw probeFailure("proxy-key-refused");
}
// worker-key-purpose: ensure a usable key for a launch, without the lock when nothing needs writing.
// Fast path: read the file and the list (no lock, no write, not even the lock's identity file), run
// the plan change() would run, and when it writes nothing, no temp file is left and no fence is set,
// probe the file's key. Otherwise change() takes the lock, reads both again, decides again and
// writes; then the key is probed. Two launches that both see "no key" serialize on the lock, and
// the second finds the first's key held. A fast-path probe the proxy refuses (a list writer that
// dropped the key after the read) goes to the lock once, which re-adds the key, and refuses only
// when the probe after that fails too. The answer is ids only; the caller reads the key with
// readKey and must call again when readKey's keyId is not the answer's keyId: a rotate by another
// launch in between leaves a newer key that this call did not probe. opts.hook also sees "fast"
// (no lock) and "locked" (the slow path) for tests.
export async function ensureReady(opts) {
const c = context(opts);
const client = await clientFor(c);
let before;
try { before = await client.getKeys(); }
catch (e) { throw proxyFailure(typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "triage-key-failed", c.purpose); }
checkSentinel(c, before);
const current = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid, purpose: c.purpose });
const quiet = current && !writes(before, current, readyPlan(c, before, current, c.now()))
&& tempsIn(keyPaths(c.configDir, c.purpose).dir).length === 0 && !fenceSet(c.stateDir);
if (quiet) {
c.hook("fast");
try {
await probeReady(client, current.key);
return { action: "held", keyId: current.keyId, rotateBy: current.rotateBy, addedKeyIds: [], removedKeyIds: [] };
} catch (e) { if (e.code !== "proxy-key-refused") throw e; }
}
c.hook("locked");
const done = await change({ ...c, client }, "ensure", (latest, record) => readyPlan(c, latest, record, c.now()));
const record = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid, purpose: c.purpose });
if (!record) throw keyError(c.purpose, "failed", "the key file is missing after the change");
await probeReady(client, record.key);
return done;
}
// The fence by name only, as keyStatus checks it: a set fence sends the launch to the lock, which
// refuses proxy-uncertain.
function fenceSet(stateDir) {
try { lstatSync(join(stateDir, "mosaic-keys.uncertain")); return true; }
catch (e) { return e.code !== "ENOENT"; }
}
// worker-key-purpose: the retire guard (reports/worker-proxy-simple.md section 2, "Retired"). True
// when an old worker key must stay: a runtime that is not ended, and not proved dead, started before
// the rotation plus marginMs, or its start is unknown. Every doubt keeps the key.
// runtimes () => listRuntimes(dataRoot) from packages/processing-control/src/runtimes.mjs.
// Missing, throwing or not an array: the key stays.
// rotatedAtMs when the current key was issued (record.issuedAt), the latest rotation. Older
// retiring keys were replaced earlier, so this is the conservative time for all.
// marginMs a launch reads the key file before it registers its runtime; a run registered up
// to marginMs after the rotation may still hold the old key.
// live, startedAt seams; the defaults read /proc.
export const RETIRE_LAUNCH_MARGIN_MS = 10 * 60_000;
export function retireHeld({ runtimes, rotatedAtMs, marginMs = RETIRE_LAUNCH_MARGIN_MS, live = runtimeLive, startedAt = runtimeStartMs } = {}) {
if (typeof runtimes !== "function" || !Number.isFinite(rotatedAtMs)) return true;
let list;
try { list = runtimes(); } catch { return true; }
if (!Array.isArray(list)) return true;
return list.some((r) => {
try {
if (r?.phase === "ended") return false;
if (live(r) === false) return false;
const started = startedAt(r);
return !Number.isFinite(started) || started <= rotatedAtMs + marginMs;
} catch { return true; }
});
}
// false only when the runtime is proved gone: its supervisor (the host process that registered it,
// the launcher that read the key) is dead or another process, and its own pid, when it has one, is
// dead too. A failed task leaves its record behind, so "not ended" alone does not mean live.
// Liveness is processing-control's own check (a pid is dead only on ESRCH or another start time),
// so an unreadable /proc keeps the key. A container runtime has no pid, and a dead docker client is
// not proof its container exited (worker-proxy.mjs finish), so it stays unknown until its
// record is ended.
const supervisorLive = (id) => (id && typeof id === "object" ? runtimeGroupState({ pid: id.pid, startTime: id.startTime, boot: id.boot }) : null);
const sameProcess = (id) => {
if (!id || !Number.isSafeInteger(id.pid) || typeof id.startTime !== "string") return null;
const now = processIdentity(id.pid);
if (!now) return false;
return now.boot === id.boot && now.startTime === id.startTime;
};
export function runtimeLive(r) {
const supervisor = supervisorLive(r?.supervisor);
if (supervisor !== false) return supervisor === true ? true : null;
if (r.kind === "container" || r.containerName !== undefined) return null;
if (r.pid === undefined) return false;
const own = runtimeGroupState(r);
return own === false ? false : own === true ? true : null;
}
// The wall-clock start of the runtime's supervisor, or null when it is not the same process on this
// boot. Start ticks since boot plus btime, at 100 ticks a second, as model-proxy's processStartMs.
export function runtimeStartMs(r) {
const id = r?.supervisor;
if (sameProcess(id) !== true) return null;
try {
const btime = Number(/^btime ([0-9]+)$/m.exec(readFileSync("/proc/stat", "utf8"))?.[1]);
const ticks = Number(id.startTime);
if (!Number.isFinite(btime) || !Number.isFinite(ticks)) return null;
return btime * 1000 + ticks * 10;
} catch { return null; }
}
// proxy-scope slice 4: op revoke, when the owner turns Triage off. Every name in this instance's
// Triage namespace leaves the proxy list, then the key file goes. No file and no names: nothing.
export async function revokeKeys(opts) {
const c = context(opts);
return change(c, "revoke", (before, current) => {
const mine = before.filter(inNamespace(c.configDir));
const mine = before.filter(inNamespace(c.configDir, c.purpose));
return { after: before.filter((k) => !mine.includes(k)), record: null, remove: true, done: { action: mine.length || current ? "revoked" : "none" } };
});
}
@@ -377,20 +589,21 @@ export async function sweepKeys(opts, { marginMs = 0 } = {}) {
// Read only: the file, and when the proxy is set up, what its list holds. No lock, no write.
// The fence is checked by name only, so status writes nothing, not even the lock's identity file.
export async function keyStatus(opts) {
const { configDir, system, uid, fileUid } = opts;
const record = readKey(configDir, { uid, fileUid });
const proxy = { state: "not-set-up", fence: false, leftoverTemps: tempsIn(keyPaths(configDir).dir).length };
const { configDir, system, uid, fileUid, purpose = "triage" } = opts;
purposeOf(purpose);
const record = readKey(configDir, { uid, fileUid, purpose });
const proxy = { state: "not-set-up", fence: false, leftoverTemps: tempsIn(keyPaths(configDir, purpose).dir).length };
if (system?.proxy && Number.isInteger(system.proxy.port)) {
try { lstatSync(join(system.dataRoot, "proxy", "mosaic-keys.uncertain")); proxy.fence = true; }
try { lstatSync(join(proxyStateDir(system), "mosaic-keys.uncertain")); proxy.fence = true; }
catch (e) { if (e.code !== "ENOENT") proxy.fence = true; }
try {
const client = await loadManagementClient({ configDir, proxy: system.proxy });
const client = await loadManagementClient({ configDir: proxyConfigDir(configDir, system), proxy: system.proxy });
const keys = await client.getKeys();
const prefix = sentinelPrefix(configDir);
const prefix = sentinelPrefix(proxyConfigDir(configDir, system));
proxy.state = keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length))) ? "read" : "no-sentinel";
proxy.holdsKey = record ? keys.includes(record.key) : null;
proxy.retiringHeld = record ? record.retiring.map((r) => keys.includes(r.key)) : [];
proxy.orphanKeyIds = orphansOf(configDir, keys, record).map(keyIdOf);
proxy.orphanKeyIds = orphansOf(configDir, keys, record, purpose).map(keyIdOf);
} catch (e) { proxy.state = "unreadable"; proxy.code = typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "proxy-client-response"; }
}
return { record, proxy };
@@ -223,6 +223,14 @@ test('remove: two listed accounts on one credential name, or one account id list
await assert.rejects(d.call('proxy.account.remove', { accountId: id('oauth-one.json'), confirm: true }), { code: 'invalid-request' });
for (const x of [s, d]) { assert.equal(x.state.calls.some(c => !c.startsWith('GET ')), false); assert.equal(x.state.files.length, 2); }
});
test('remove: a credential with an unusable id still prevents deleting its shared name', async t => {
for (const badId of ['', null, 'x'.repeat(513)]) {
const s = await fakeManagement(t, { files: [oauth(), oauth({ id: badId, name: 'oauth-one.json' })] });
await assert.rejects(s.call('proxy.account.remove', { accountId: id('oauth-one.json'), confirm: true }), { code: 'invalid-request' });
assert.equal(s.state.calls.some(c => !c.startsWith('GET ')), false);
assert.equal(s.state.files.length, 2);
}
});
test('remove: under the lock, key groups that change between the read and the write refuse with proxy-down before the fence', async t => {
const fs = await import('node:fs'); const { join } = await import('node:path'); const { tmpdir } = await import('node:os');
const { createProxyLock } = await import('../src/proxy-lock.mjs');
+127
View File
@@ -0,0 +1,127 @@
// proxy-ride first slice (Jason 2026-11-21): a riding instance's worker key in the owner
// instance's proxy. Two scratch instances and a stand-in proxy on 127.0.0.1 port 0. The rider's
// system.proxy carries `owner`, as scripts/mosaic-config.mjs gives it. Nothing here reads a real
// config, proxy or secret.
import { test } from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
import http from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createHash, randomBytes } from "node:crypto";
import { ensureReady, keyPrefix, keyStatus, readKey, rotateKey, sweepKeys } from "../src/triage-key.mjs";
import { ModelKeys } from "../src/model-keys.mjs";
import { loadManagementClient } from "../../model-proxy/src/client.mjs";
const MANAGE = "fxride-manage";
const KEY_PATH = "/v8/management/config/access/api-keys";
const sha = (s) => createHash("sha256").update(s).digest("hex");
const tag = (dir) => sha(dir).slice(0, 12);
const b64 = () => randomBytes(32).toString("base64url");
const dir700 = (p) => { fs.mkdirSync(p, { recursive: true, mode: 0o700 }); fs.chmodSync(p, 0o700); };
async function fixture(t) {
const root = fs.realpathSync(fs.mkdtempSync(join(tmpdir(), "pr-")));
const owner = { configDir: join(root, "oc"), dataRoot: join(root, "od") };
const rider = { configDir: join(root, "rc"), dataRoot: join(root, "rd") };
for (const d of [owner.configDir, join(owner.configDir, "secrets"), owner.dataRoot, join(owner.dataRoot, "proxy"), rider.configDir, join(rider.configDir, "secrets"), rider.dataRoot]) dir700(d);
fs.writeFileSync(join(owner.configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
const O = tag(owner.configDir);
const ownerKeys = { sentinel: `msk-${O}-sentinel-${b64()}`, bus: `msk-${O}-bus-${b64()}`, org: `mso-${O}-demo-key-${b64()}` };
const lockFile = join(owner.dataRoot, "proxy", "mosaic-keys.lock");
const state = { keys: Object.values(ownerKeys), puts: [] };
const server = http.createServer(async (req, res) => {
if (req.url === "/v1/models") {
res.writeHead(state.keys.includes(req.headers.authorization?.slice(7)) ? 200 : 401); res.end("{}"); return;
}
if (req.url !== KEY_PATH || req.headers.authorization !== `Bearer ${MANAGE}`) { res.writeHead(401); res.end("{}"); return; }
if (req.method === "GET") { res.end(JSON.stringify(state.keys)); return; }
const chunks = []; for await (const c of req) chunks.push(c);
state.keys = JSON.parse(Buffer.concat(chunks));
state.puts.push({ locked: fs.existsSync(lockFile) });
res.end('{"status":"ok"}');
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const port = server.address().port;
t.after(async () => { server.closeAllConnections(); await new Promise((r) => server.close(r)); fs.rmSync(root, { recursive: true, force: true }); });
owner.system = { dataRoot: owner.dataRoot, proxy: { port, binary: "/fixture-unused" } };
rider.system = { dataRoot: rider.dataRoot, proxy: { port, binary: "/fixture-unused", owner: { instance: "owner", configDir: owner.configDir, dataRoot: owner.dataRoot } } };
const opts = (who, extra = {}) => ({ configDir: who.configDir, system: who.system, purpose: "worker", ...extra });
return { root, owner, rider, ownerKeys, state, port, lockFile, opts, receipts: join(owner.dataRoot, "proxy", "mosaic-keys.receipts.jsonl") };
}
test("a rider's worker key lands under the rider's tag, under the owner's lock, and survives the owner's own key work", async (t) => {
const f = await fixture(t);
// The owner writes first, so its lock identity file exists before the rider's write.
await ensureReady(f.opts(f.owner));
const ownerWorker = readKey(f.owner.configDir, { purpose: "worker" }).key;
const putsBefore = f.state.puts.length;
const done = await ensureReady(f.opts(f.rider));
assert.equal(done.action, "issued");
const riderKey = readKey(f.rider.configDir, { purpose: "worker" }).key;
assert.match(riderKey, new RegExp(`^msw-${tag(f.rider.configDir)}-worker-[A-Za-z0-9_-]{43}$`));
assert.ok(riderKey.startsWith(keyPrefix(f.rider.configDir, "worker")));
assert.ok(f.state.keys.includes(riderKey));
assert.ok(f.state.puts.length > putsBefore && f.state.puts.every((p) => p.locked), "every PUT ran under the owner's lock");
assert.equal(fs.existsSync(join(f.rider.dataRoot, "proxy")), false, "the rider has no proxy state of its own");
assert.equal(fs.existsSync(join(f.rider.configDir, "secrets", "proxy-management.key")), false);
const ops = fs.readFileSync(f.receipts, "utf8").trim().split("\n").map((l) => JSON.parse(l).op);
assert.deepEqual(ops, ["worker.key.ensure", "worker.key.ensure"], "the rider's receipt is in the owner's state directory");
for (const k of [...Object.values(f.ownerKeys), ownerWorker]) assert.ok(f.state.keys.includes(k), "the rider removes none of the owner's keys");
// The rider rotates and sweeps its own key; the owner's keys stay.
await rotateKey(f.opts(f.rider));
await sweepKeys(f.opts(f.rider));
for (const k of [...Object.values(f.ownerKeys), ownerWorker]) assert.ok(f.state.keys.includes(k));
const riderNow = readKey(f.rider.configDir, { purpose: "worker" }).key;
// The owner's own worker key work and its bus model key reconcile leave the rider's key.
await rotateKey(f.opts(f.owner));
await sweepKeys(f.opts(f.owner));
await ensureReady(f.opts(f.owner));
const client = await loadManagementClient({ configDir: f.owner.configDir, proxy: f.owner.system.proxy });
const manager = new ModelKeys({ credentials: { status: () => [], modelRecord: () => null }, client, configDir: f.owner.configDir, baseUrl: `http://127.0.0.1:${f.port}/v1` });
const receipt = await manager.reconcile();
assert.ok(receipt.removedKeyIds.length >= 1, "the reconcile did remove the owner's own stale bus key");
assert.ok(f.state.keys.includes(riderNow), "the rider's key survives the owner's reconcile and cleanup");
assert.equal((await ensureReady(f.opts(f.rider))).action, "held");
});
test("the rider's lock identity is the owner's: a writer naming the rider's own config directory refuses", async (t) => {
const f = await fixture(t);
await ensureReady(f.opts(f.owner));
// The same state directory under the rider's own identity is another proxy to the lock.
const wrong = { ...f.rider.system, proxy: { ...f.rider.system.proxy, owner: { ...f.rider.system.proxy.owner, configDir: f.rider.configDir } } };
// Give it every other piece, so only the lock identity differs.
fs.copyFileSync(join(f.owner.configDir, "secrets", "proxy-management.key"), join(f.rider.configDir, "secrets", "proxy-management.key"));
f.state.keys.push(`msk-${tag(f.rider.configDir)}-sentinel-${b64()}`);
const puts = f.state.puts.length;
await assert.rejects(ensureReady({ configDir: f.rider.configDir, system: wrong, purpose: "worker" }), (e) => e.code === "proxy-lock-invalid");
assert.equal(f.state.puts.length, puts, "no write without the owner's identity");
});
test("the owner's fence stops the rider's key write, and the rider's status reports it", async (t) => {
const f = await fixture(t);
await ensureReady(f.opts(f.rider));
const held = readKey(f.rider.configDir, { purpose: "worker" });
fs.writeFileSync(join(f.owner.dataRoot, "proxy", "mosaic-keys.uncertain"), "{}\n", { mode: 0o600 });
const puts = f.state.puts.length;
// A held key goes to the lock while the fence is set, and the lock refuses.
await assert.rejects(ensureReady(f.opts(f.rider)), (e) => e.code === "proxy-uncertain");
await assert.rejects(rotateKey(f.opts(f.rider)), (e) => e.code === "proxy-uncertain");
assert.equal(f.state.puts.length, puts, "no PUT while the owner's fence is set");
assert.deepEqual(readKey(f.rider.configDir, { purpose: "worker" }), held);
const status = await keyStatus(f.opts(f.rider));
assert.equal(status.proxy.fence, true);
assert.equal(status.proxy.state, "read", "status reads the list with the owner's management key and sentinel");
assert.equal(status.proxy.holdsKey, true);
assert.equal(JSON.stringify(status).includes(MANAGE), false);
});
test("a rider without its own secrets directory refuses before any proxy write", async (t) => {
const f = await fixture(t);
fs.rmdirSync(join(f.rider.configDir, "secrets"));
await assert.rejects(ensureReady(f.opts(f.rider)), (e) => e.code === "worker-key-unreadable" && /secrets directory is missing/.test(e.message));
assert.equal(f.state.puts.length, 0);
});
+51
View File
@@ -0,0 +1,51 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { setupState } from '../src/setup-reply.mjs';
const org = { id: 'team', doc: { roles: { assistant: { definition: 'assistant' } } } };
const human = { id: 'q', from_role: 'human', body: 'Hello' };
const answer = { id: 'a', from_role: 'assistant', from_run: 'real-run', to_role: 'human', in_reply_to: 'q', body: 'Hello back' };
test('hello needs an actual reply to the owner, including retired session history', async () => {
const run = messages => setupState[0].check({ members: [org], read: async (_, name) => name === 'threads' ? [{ id: 'thread', instance: 'assistant' }] : name === 'sessions' ? [{ id: 'session' }] : messages });
assert.equal((await run([human, answer])).state, 'done');
assert.equal((await run([human])).state, 'next');
assert.equal((await run([human, { ...answer, from_run: 'bus', body: 'The turn failed' }])).state, 'next');
assert.equal((await run([{ ...answer, in_reply_to: null }])).state, 'next');
assert.equal((await setupState[0].check({ members: [] })).state, 'next');
assert.match((await setupState[0].check({ members: [org], read: async () => { throw Error(); } })).reason, /couldn't be checked/);
});
import { failureBody, LENGTH_FAILURE_NOTICE_PREFIX } from '../../harness/src/failure-notice.mjs';
for (const [kind, body] of [
['failed turn', failureBody({ reason: 'adapter exited' })],
['length failure', LENGTH_FAILURE_NOTICE_PREFIX + 'answer was too long'],
['empty answer', '(the session gave no answer)'],
]) test(`hello rejects the runner ${kind} notice, including its unslop note`, async () => {
for (const note of ['', '\n\n[unslop: the check is broken, so this session\'s replies go unchecked: test]']) {
const result = await setupState[0].check({ members: [org], read: async (_, name) => name === 'threads' ? [{ id: 'thread', instance: 'assistant' }] : name === 'sessions' ? [{ id: 'session' }] : [human, { ...answer, class: 'RESULT', body: body + note }] });
assert.equal(result.state, 'next');
}
});
test('one unavailable organization does not hide a real reply in another', async () => {
const result = await setupState[0].check({ members: [{ id: 'stopped' }, org], read: async (member, name) => {
if (member.id === 'stopped') throw Error('private');
return name === 'threads' ? [{ id: 'thread', instance: 'assistant' }] : name === 'sessions' ? [{ id: 'session' }] : [human, answer];
} });
assert.equal(result.state, 'done'); assert.match(result.reason, /couldn't be checked/);
});
test('large histories scan newest first and stop at the first real reply', async () => {
const calls = [];
const result = await setupState[0].check({ members: [org], read: async (_, name, args) => {
calls.push([name, args]);
if (name === 'threads') return Array.from({ length: 51 }, (_, n) => ({ id: `t${n}`, at: String(n).padStart(3, '0'), instance: 'assistant' }));
if (name === 'sessions') return Array.from({ length: 51 }, (_, n) => ({ id: `s${n}`, at: String(n).padStart(3, '0') }));
return [human, answer, ...Array.from({ length: 198 }, () => human)];
} });
assert.equal(result.state, 'done');
assert.deepEqual(calls.map(c => c[0]), ['threads', 'sessions', 'messages']);
assert.equal(calls[1][1].thread, 't50'); assert.equal(calls[2][1].session, 's50');
});
test('bounded history without a reply stays honest and plain', async () => {
let pages = 0;
const result = await setupState[0].check({ members: [org], read: async (_, name) => name === 'threads' ? Array.from({ length: 51 }, (_, i) => ({ id: String(i), instance: 'assistant' })) : name === 'sessions' ? [{ id: 'session' }] : (pages++, Array.from({ length: 200 }, (_, i) => ({ ...human, id: `${pages}-${i}` }))) });
assert.equal(result.state, 'next'); assert.match(result.reason, /couldn't be checked/); assert.equal(pages, 100);
});
+579
View File
@@ -0,0 +1,579 @@
// worker-key-purpose (S3 of reports/worker-proxy-simple.md): triage-key.mjs with purpose "worker".
// The worker key `msw-<tag>-worker-<43>` in <configDir>/secrets/worker/model, its mirrored
// worker-key-* refusals, the lock-free launch path ensureReady, the post-write probe, the namespace
// fence in both directions and the retire guard. A stand-in proxy on 127.0.0.1 port 0 and scratch
// config directories only; nothing here reads a real config, proxy or secret.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import fs from "node:fs";
import http from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createHash, randomBytes } from "node:crypto";
import {
PROBE_TIMING, PURPOSES, ensureKey, ensureReady, issueKey, keyPaths, keyPrefix, keyStatus, readKey, retireHeld, retireKeys,
revokeKeys, rotateKey, runtimeLive, runtimeStartMs, sweepKeys, RETIRE_LAUNCH_MARGIN_MS,
} from "../src/triage-key.mjs";
import { processIdentity } from "../../processing-control/src/runtimes.mjs";
import { ISSUE_PROBE_TIMEOUT_MS, ISSUE_REPROBE_MIN_MS, REVOKE_POLL_MS } from "../src/model-keys.mjs";
const NOW = Date.parse("2026-11-21T12:00:00.000Z");
const MIN = 60_000, HOUR = 60 * MIN, DAY = 24 * HOUR;
const MANAGE = "fxwk-manage";
const KEY_PATH = "/v8/management/config/access/api-keys";
const sha = (s) => createHash("sha256").update(s).digest("hex");
const tag = (dir) => sha(dir).slice(0, 12);
const id = (key) => sha(key).slice(0, 12);
const b64 = () => randomBytes(32).toString("base64url");
const iso = (ms) => new Date(ms).toISOString();
const MODULE = new URL("../src/triage-key.mjs", import.meta.url).href;
// Every msw-/mst- key the stand-in ever saw, and every text a call produced, for the leak scans.
const SEEN = new Set();
const OUTPUTS = [];
function leaks(text, keys = SEEN) {
for (const key of keys) {
const secret = key.slice(-43);
if (text.includes(key) || text.includes(secret)) return true;
for (let i = 0; i + 8 <= secret.length; i++) if (text.includes(secret.slice(i, i + 8))) return true;
}
return false;
}
// A stand-in proxy. A PUT records whether the instance proxy lock file existed while it ran.
async function fixture(t) {
const root = fs.realpathSync(fs.mkdtempSync(join(tmpdir(), "wk-")));
const configDir = join(root, "config");
fs.mkdirSync(join(configDir, "secrets"), { recursive: true, mode: 0o700 });
fs.chmodSync(configDir, 0o700); fs.chmodSync(join(configDir, "secrets"), 0o700);
fs.writeFileSync(join(configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
fs.mkdirSync(join(root, "proxy"), { mode: 0o700 });
const T = tag(configDir);
const others = {
sentinel: `msk-${T}-sentinel-${b64()}`, bus: `msk-${T}-bus-${b64()}`, org: `mso-${T}-demo-key-${b64()}`,
foreign: "foreign-client-key", otherWorker: `msw-${"f".repeat(12)}-worker-${b64()}`,
};
const lockFile = join(root, "proxy", "mosaic-keys.lock");
const state = { keys: Object.values(others), gets: 0, puts: [], probes: [], refuse401: 0, ignorePut: false, onGet: null };
const server = http.createServer(async (req, res) => {
if (req.url === "/v1/models") {
const key = req.headers.authorization?.slice(7);
state.probes.push(key);
if (state.dropProbe) { req.socket.destroy(); return; }
if (state.refuse401 > 0) { state.refuse401--; res.writeHead(401); res.end("{}"); return; }
res.writeHead(state.keys.includes(key) ? 200 : 401); res.end("{}"); return;
}
if (req.url !== KEY_PATH || req.headers.authorization !== `Bearer ${MANAGE}`) { res.writeHead(401); res.end("{}"); return; }
if (req.method === "GET") { state.gets++; const body = JSON.stringify(state.keys); state.onGet?.(state.gets); res.end(body); return; }
const chunks = []; for await (const c of req) chunks.push(c);
const next = JSON.parse(Buffer.concat(chunks));
for (const k of next) if (/^ms[tw]-.*-[A-Za-z0-9_-]{43}$/.test(k)) SEEN.add(k);
state.puts.push({ keys: next, locked: fs.existsSync(lockFile) });
if (!state.ignorePut) state.keys = next;
res.end('{"status":"ok"}');
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const port = server.address().port;
t.after(async () => {
if (fs.existsSync(join(root, "proxy", "mosaic-keys.receipts.jsonl"))) OUTPUTS.push(fs.readFileSync(join(root, "proxy", "mosaic-keys.receipts.jsonl"), "utf8"));
server.closeAllConnections(); await new Promise((resolve) => server.close(resolve));
fs.rmSync(root, { recursive: true, force: true });
});
const f = {
root, configDir, state, others, port, lockFile, now: NOW,
system: { dataRoot: root, proxy: { port, binary: "/fixture-unused" } },
worker: keyPaths(configDir, "worker"), triage: keyPaths(configDir, "triage"),
receipts: join(root, "proxy", "mosaic-keys.receipts.jsonl"),
opts(extra = {}) {
const stages = [];
const o = { configDir, system: f.system, now: () => f.now, purpose: "worker", hook: (s) => stages.push(s), ...extra };
o.stages = stages;
return o;
},
mine: (purpose = "worker") => state.keys.filter((k) => k.startsWith(keyPrefix(configDir, purpose))),
read: (purpose = "worker") => readKey(configDir, { purpose }),
// Every file under the root with its size and mtime, for "wrote nothing".
snapshot() {
const out = {};
const walk = (dir) => { for (const name of fs.readdirSync(dir)) { const p = join(dir, name), st = fs.lstatSync(p); out[p] = st.isDirectory() ? "dir" : `${st.size}:${st.mtimeMs}`; if (st.isDirectory()) walk(p); } };
walk(root);
return out;
},
// Writes a worker record by hand: the current key, plus retiring entries.
seed({ issuedAt = NOW - DAY, rotateBy, retiring = [], inList = true } = {}) {
const key = keyPrefix(configDir, "worker") + b64();
SEEN.add(key);
const record = { version: 1, key, keyId: id(key), issuedAt: iso(issuedAt), rotateBy: rotateBy ?? iso(issuedAt + 30 * DAY).slice(0, 10), retiring };
fs.mkdirSync(f.worker.dir, { recursive: true, mode: 0o700 });
fs.writeFileSync(f.worker.file, JSON.stringify(record), { mode: 0o600 });
if (inList) state.keys.push(key);
for (const r of retiring) if (r.inList !== false && !state.keys.includes(r.key)) state.keys.push(r.key);
return record;
},
retiring(retireAt, { inList = true } = {}) {
const key = keyPrefix(configDir, "worker") + b64();
SEEN.add(key);
return { key, keyId: id(key), issuedAt: iso(NOW - 40 * DAY), retireAt: iso(retireAt), ...(inList ? {} : { inList }) };
},
};
return f;
}
// readKey refuses unknown fields, so the seed drops the test-only inList marker.
const clean = (r) => { const { inList, ...rest } = r; return rest; };
async function refused(promise, code) {
try { await promise; } catch (e) { OUTPUTS.push(e.message); assert.equal(e.code, code, e.message); return e; }
assert.fail(`expected ${code}`);
}
const deadRuntime = () => ({ phase: "running", supervisor: { pid: 999_999_999, startTime: "1", boot: "00000000-0000-4000-8000-000000000000" } });
test("the worker purpose names its own prefix, file and codes, and the Triage purpose is unchanged", async (t) => {
const f = await fixture(t);
assert.equal(keyPrefix(f.configDir), `mst-${tag(f.configDir)}-triage-`);
assert.equal(keyPrefix(f.configDir, "worker"), `msw-${tag(f.configDir)}-worker-`);
assert.equal(f.triage.file, join(f.configDir, "secrets", "triage", "model"));
assert.equal(f.worker.file, join(f.configDir, "secrets", "worker", "model"));
assert.deepEqual(Object.keys(PURPOSES), ["triage", "worker"]);
assert.throws(() => keyPrefix(f.configDir, "org"), TypeError);
await assert.rejects(ensureReady(f.opts({ purpose: "bus" })), TypeError);
const done = await ensureReady(f.opts());
assert.equal(done.action, "issued");
const record = f.read();
assert.match(record.key, new RegExp(`^msw-${tag(f.configDir)}-worker-[A-Za-z0-9_-]{43}$`));
assert.equal((fs.statSync(f.worker.file).mode & 0o777), 0o600);
assert.equal((fs.statSync(f.worker.dir).mode & 0o777), 0o700);
assert.equal(fs.existsSync(f.triage.dir), false, "a worker call leaves the Triage directory alone");
assert.equal(readKey(f.configDir), null, "the Triage reader does not see the worker key");
assert.deepEqual(f.mine(), [record.key]);
for (const k of Object.values(f.others)) assert.ok(f.state.keys.includes(k));
const lines = fs.readFileSync(f.receipts, "utf8").trim().split("\n").map((l) => JSON.parse(l));
assert.deepEqual(lines.map((l) => l.op), ["worker.key.ensure"]);
assert.deepEqual(lines[0].addedKeyIds, [record.keyId]);
assert.ok(f.state.puts.every((p) => p.locked), "every PUT ran under the lock");
assert.equal(f.state.probes.at(-1), record.key, "the key was probed after the write");
});
test("each worker-key-* refusal mirrors its triage-key-* code", async (t) => {
const f = await fixture(t);
const noProxy = { configDir: f.configDir, system: { dataRoot: f.root } };
await refused(ensureReady({ ...noProxy, purpose: "worker" }), "worker-key-no-proxy");
await refused(ensureKey({ ...noProxy }), "triage-key-no-proxy");
await refused(rotateKey(f.opts()), "worker-key-not-issued");
await issueKey(f.opts());
const issued = await refused(issueKey(f.opts()), "worker-key-issued");
assert.equal(issued.message, "a worker key is already issued");
fs.chmodSync(f.worker.file, 0o644);
await refused(ensureReady(f.opts()), "worker-key-unreadable");
fs.chmodSync(f.worker.file, 0o600);
const sentinel = f.others.sentinel;
f.state.keys = f.state.keys.filter((k) => k !== sentinel);
await refused(ensureReady(f.opts()), "worker-key-proxy-open");
f.state.keys.push(sentinel);
f.state.onGet = (n) => { if (n % 2 === 0) f.state.keys.push(`unrelated-${n}`); };
const raced = await refused(rotateKey(f.opts()), "worker-key-raced");
assert.match(raced.message, /run the launch again/);
assert.doesNotMatch(raced.message, /verb/);
f.state.onGet = null;
f.state.ignorePut = true;
await refused(rotateKey(f.opts()), "worker-key-unconfirmed");
f.state.ignorePut = false;
fs.rmSync(join(f.root, "proxy", "mosaic-keys.uncertain"), { force: true });
fs.chmodSync(f.receipts, 0o644);
await refused(rotateKey(f.opts()), "worker-key-receipts-unsafe");
fs.chmodSync(f.receipts, 0o600);
const unwritten = await refused(rotateKey(f.opts({ hook: (s) => { if (s === "rename") throw new Error("stop"); } })), "worker-key-unwritten");
assert.match(unwritten.message, /run the launch again/);
assert.doesNotMatch(unwritten.message, /triage/i);
const unsynced = await refused(rotateKey(f.opts({ hook: (s) => { if (s === "sync") throw new Error("stop"); } })), "worker-key-unsynced");
assert.match(unsynced.message, /secrets\/worker\/model/);
});
test("ensureReady on a held worker key takes no lock, writes nothing and probes the key", async (t) => {
const f = await fixture(t);
await ensureReady(f.opts());
const record = f.read();
// The lock's identity file is a write too; the fast path must not make it.
fs.rmSync(join(f.root, "proxy", "mosaic-proxy.identity"));
const before = f.snapshot(), puts = f.state.puts.length, probes = f.state.probes.length;
const o = f.opts();
const done = await ensureReady(o);
assert.deepEqual(o.stages, ["fast"]);
assert.deepEqual(done, { action: "held", keyId: record.keyId, rotateBy: record.rotateBy, addedKeyIds: [], removedKeyIds: [] });
assert.deepEqual(f.snapshot(), before, "no file changed or appeared");
assert.equal(f.state.puts.length, puts);
assert.deepEqual(f.state.probes.slice(probes), [record.key]);
// A Triage key takes the same fast path.
await ensureReady(f.opts({ purpose: "triage" }));
const triageRecord = f.read("triage");
fs.rmSync(join(f.root, "proxy", "mosaic-proxy.identity"));
const o2 = f.opts({ purpose: "triage" });
const snap = f.snapshot();
assert.equal((await ensureReady(o2)).action, "held");
assert.deepEqual(o2.stages, ["fast"]);
assert.deepEqual(f.snapshot(), snap);
assert.equal(f.state.probes.at(-1), triageRecord.key);
});
test("ensureReady takes the lock for every state that needs a write, and only writes under it", async (t) => {
const cases = [
["no file", (f) => {}, "issued"],
["key gone from the list", (f) => { const r = f.seed({ inList: false }); return r; }, "replaced"],
["an orphan in the worker namespace", (f) => { f.seed(); f.state.keys.push(keyPrefix(f.configDir, "worker") + b64()); }, "held"],
["a malformed name in the worker namespace", (f) => { f.seed(); f.state.keys.push(keyPrefix(f.configDir, "worker") + "short"); }, "held"],
["a retiring key the proxy lost", (f) => { const r = f.retiring(NOW + HOUR, { inList: false }); f.seed({ retiring: [clean(r)] }); f.state.keys = f.state.keys.filter((k) => k !== r.key); }, "held"],
["a leftover temp file", (f) => { f.seed(); fs.writeFileSync(join(f.worker.dir, `.model.1.${"a".repeat(12)}.tmp`), "x", { mode: 0o600 }); }, "held"],
["rotateBy has come", (f) => { f.seed({ issuedAt: NOW - 31 * DAY }); }, "rotated"],
["a retiring key past its grace, no live runs", (f) => { const r = f.retiring(NOW - MIN); f.seed({ retiring: [clean(r)] }); }, "retired"],
];
for (const [name, setup, action] of cases) {
await t.test(name, async (t) => {
const f = await fixture(t);
setup(f);
const o = f.opts({ runtimes: () => [] });
const done = await ensureReady(o);
assert.equal(o.stages[0], "locked", name);
assert.equal(done.action, action);
assert.ok(f.state.puts.every((p) => p.locked), "no PUT ran without the lock");
assert.deepEqual(fs.readdirSync(f.worker.dir), ["model"], "no temp file is left");
const record = f.read();
assert.deepEqual([...f.mine()].sort(), [record.key, ...record.retiring.map((r) => r.key)].sort(), "the list holds exactly the file's keys");
// The next launch finds nothing to write.
const o2 = f.opts({ runtimes: () => [] });
await ensureReady(o2);
assert.deepEqual(o2.stages, ["fast"], `${name}: settled`);
});
}
await t.test("a set fence sends the launch to the lock, which refuses", async (t) => {
const f = await fixture(t);
f.seed();
fs.writeFileSync(join(f.root, "proxy", "mosaic-keys.uncertain"), "{}", { mode: 0o600 });
const o = f.opts();
await refused(ensureReady(o), "proxy-uncertain");
assert.deepEqual(o.stages, ["locked"]);
assert.equal(f.state.puts.length, 0);
});
});
test("ensureReady probes a just-written key again while the proxy answers 401, then refuses", async (t) => {
assert.deepEqual({ ...PROBE_TIMING }, { timeoutMs: ISSUE_PROBE_TIMEOUT_MS, reprobeMinMs: ISSUE_REPROBE_MIN_MS, pollMs: REVOKE_POLL_MS }, "the bus's probe timing");
const f = await fixture(t);
f.state.refuse401 = 3;
const done = await ensureReady(f.opts());
assert.equal(done.action, "issued");
assert.equal(f.state.probes.length, 4, "three 401s, then the 200");
f.state.refuse401 = 1e9;
const started = Date.now();
await refused(ensureReady(f.opts()), "proxy-key-refused");
assert.ok(Date.now() - started >= 1500, "the 401 was probed until the deadline");
f.state.refuse401 = 0;
// A proxy that drops the model route gives proxy-down, on the fast path and after a write.
f.state.dropProbe = true;
await refused(ensureReady(f.opts()), "proxy-down");
fs.rmSync(f.worker.file);
await refused(ensureReady(f.opts()), "proxy-down");
f.state.dropProbe = false;
assert.equal((await ensureReady(f.opts())).action, "held", "the key written before the probe failed is kept");
});
test("a fast-path probe the proxy refuses goes to the lock once, which re-adds the key", async (t) => {
const f = await fixture(t);
const record = f.seed();
// A list writer outside the lock drops the key after this launch read the list.
f.state.onGet = (n) => { if (n === 1) f.state.keys = f.state.keys.filter((k) => k !== record.key); };
const o = f.opts();
const done = await ensureReady(o);
f.state.onGet = null;
assert.deepEqual(o.stages.filter((s) => s === "fast" || s === "locked"), ["fast", "locked"]);
assert.equal(done.action, "replaced");
assert.ok(f.state.puts.length > 0 && f.state.puts.every((p) => p.locked), "the re-add ran under the lock");
assert.equal(f.read().keyId, done.keyId);
assert.ok(f.state.keys.includes(f.read().key));
// A key the list holds but the proxy still refuses: the locked pass changes nothing, then refuses.
f.state.refuse401 = 1e9;
const puts = f.state.puts.length;
const o2 = f.opts();
await refused(ensureReady(o2), "proxy-key-refused");
f.state.refuse401 = 0;
assert.deepEqual(o2.stages.filter((s) => s === "fast" || s === "locked"), ["fast", "locked"]);
assert.equal(f.state.puts.length, puts, "nothing was written");
});
test("change() refuses a plan that would remove a key outside the purpose's namespace", async (t) => {
for (const purpose of ["worker", "triage"]) {
await t.test(purpose, async (t) => {
const f = await fixture(t);
await issueKey(f.opts({ purpose }));
const puts = f.state.puts.length;
const other = purpose === "worker" ? f.others.otherWorker : f.others.sentinel;
const planSeam = (result, before) => ({ ...result, after: (result.after ?? before).filter((k) => k !== other) });
const e = await refused(rotateKey(f.opts({ purpose, planSeam })), `${purpose}-key-failed`);
assert.match(e.message, /outside the (worker|Triage) namespace/);
assert.equal(f.state.puts.length, puts, "no PUT ran");
assert.ok(f.state.keys.includes(other));
});
}
});
test("two parallel worker ensureReady calls both succeed and leave exactly one key", async (t) => {
const f = await fixture(t);
const results = await Promise.all([ensureReady(f.opts()), ensureReady(f.opts())]);
assert.deepEqual(results.map((r) => r.action).sort(), ["held", "issued"]);
const record = f.read();
assert.deepEqual(f.mine(), [record.key]);
assert.equal(new Set(results.map((r) => r.keyId)).size, 1);
assert.ok(f.state.puts.length === 1 && f.state.puts[0].locked);
// The same from two processes.
fs.rmSync(f.worker.file);
f.state.keys = f.state.keys.filter((k) => !k.startsWith(keyPrefix(f.configDir, "worker")));
const script = `
import { ensureReady } from ${JSON.stringify(MODULE)};
const p = JSON.parse(process.env.WK_OPTS);
const done = await ensureReady({ ...p, purpose: "worker" });
process.stdout.write(JSON.stringify(done));`;
const child = () => new Promise((resolve) => {
const c = spawn(process.execPath, ["--input-type=module", "-e", script], { env: { PATH: process.env.PATH, HOME: process.env.HOME, TMPDIR: process.env.TMPDIR, WK_OPTS: JSON.stringify({ configDir: f.configDir, system: f.system }) }, stdio: ["ignore", "pipe", "pipe"] });
let out = "", err = "";
c.stdout.on("data", (d) => { out += d; }); c.stderr.on("data", (d) => { err += d; });
c.on("close", (code) => resolve({ code, out, err }));
});
const kids = await Promise.all([child(), child(), child()]);
for (const k of kids) { OUTPUTS.push(k.out, k.err); assert.equal(k.code, 0, k.err); }
const actions = kids.map((k) => JSON.parse(k.out).action).sort();
assert.deepEqual(actions, ["held", "held", "issued"]);
assert.deepEqual(f.mine(), [f.read().key]);
assert.ok(f.state.puts.every((p) => p.locked));
});
test("a Triage call never removes a worker key, and a worker call never removes a Triage, org, bus or foreign key", async (t) => {
await t.test("Triage calls leave msw- names alone", async (t) => {
const f = await fixture(t);
const workerKey = keyPrefix(f.configDir, "worker") + b64();
const malformed = keyPrefix(f.configDir, "worker") + "x";
f.state.keys.push(workerKey, malformed);
const keep = [workerKey, malformed, ...Object.values(f.others)];
const triage = (extra = {}) => ({ configDir: f.configDir, system: f.system, now: () => f.now, ...extra });
const steps = [
() => issueKey(triage()), () => ensureKey(triage()), () => ensureReady(triage()), () => rotateKey(triage()),
() => { f.now = NOW + DAY; return retireKeys(triage()); }, () => { f.now = NOW + 40 * DAY; return sweepKeys(triage()); },
() => revokeKeys(triage()),
];
for (const step of steps) {
await step();
for (const k of keep) assert.ok(f.state.keys.includes(k), "a non-Triage key stayed");
}
assert.deepEqual(f.mine("triage"), []);
assert.equal(fs.existsSync(f.worker.file), false);
});
await t.test("worker calls leave mst-, mso-, msk- and foreign names alone", async (t) => {
const f = await fixture(t);
const triageKey = keyPrefix(f.configDir, "triage") + b64();
const triageMalformed = keyPrefix(f.configDir, "triage") + "x";
f.state.keys.push(triageKey, triageMalformed);
const keep = [triageKey, triageMalformed, ...Object.values(f.others)];
const steps = [
() => issueKey(f.opts()), () => ensureKey(f.opts()), () => ensureReady(f.opts()), () => rotateKey(f.opts()),
() => { f.now = NOW + DAY; return retireKeys(f.opts({ runtimes: () => [] })); },
() => { f.now = NOW + 40 * DAY; return ensureReady(f.opts({ runtimes: () => [] })); },
() => { f.now = NOW + 80 * DAY; return sweepKeys(f.opts({ runtimes: () => [] })); },
() => revokeKeys(f.opts()),
];
for (const step of steps) {
await step();
for (const k of keep) assert.ok(f.state.keys.includes(k), "a non-worker key stayed");
}
assert.deepEqual(f.mine(), []);
assert.equal(fs.existsSync(f.worker.file), false, "revoke removed the worker file");
assert.equal(fs.existsSync(f.triage.dir), false, "no Triage file was made");
// The other instance's worker key (another tag) is another namespace too.
assert.ok(f.state.keys.includes(f.others.otherWorker));
});
});
test("retireHeld keeps an old worker key on every doubt and lets it go only when no older run is live", () => {
const rotatedAtMs = NOW;
const run = (started, extra = {}) => ({ phase: "running", started, ...extra });
const seams = { live: (r) => r.live ?? true, startedAt: (r) => r.started };
assert.equal(retireHeld({ rotatedAtMs, ...seams }), true, "no runtimes source");
assert.equal(retireHeld({ runtimes: () => { throw new Error("x"); }, rotatedAtMs, ...seams }), true, "the source throws");
assert.equal(retireHeld({ runtimes: () => ({}), rotatedAtMs, ...seams }), true, "not a list");
assert.equal(retireHeld({ runtimes: () => [], rotatedAtMs: NaN, ...seams }), true, "no rotation time");
assert.equal(retireHeld({ runtimes: () => [], rotatedAtMs, ...seams }), false, "no runs");
assert.equal(retireHeld({ runtimes: () => [run(NOW - HOUR)], rotatedAtMs, ...seams }), true, "a run from before the rotation");
assert.equal(retireHeld({ runtimes: () => [run(NOW + RETIRE_LAUNCH_MARGIN_MS)], rotatedAtMs, ...seams }), true, "a run inside the launch margin");
assert.equal(retireHeld({ runtimes: () => [run(NOW + RETIRE_LAUNCH_MARGIN_MS + 1)], rotatedAtMs, ...seams }), false, "a run after the margin");
assert.equal(RETIRE_LAUNCH_MARGIN_MS, 10 * MIN);
assert.equal(retireHeld({ runtimes: () => [run(NOW + 5 * MIN)], rotatedAtMs, ...seams }), true, "a run five minutes after the rotation may hold the old key");
assert.equal(retireHeld({ runtimes: () => [run(NOW + 5 * MIN)], rotatedAtMs, marginMs: MIN, ...seams }), false, "marginMs is a seam");
assert.equal(retireHeld({ runtimes: () => [run(null)], rotatedAtMs, ...seams }), true, "a run whose start is unknown");
assert.equal(retireHeld({ runtimes: () => [run(NOW - HOUR, { phase: "ended" })], rotatedAtMs, ...seams }), false, "an ended run");
assert.equal(retireHeld({ runtimes: () => [run(NOW - HOUR, { live: false })], rotatedAtMs, ...seams }), false, "a run proved dead");
assert.equal(retireHeld({ runtimes: () => [run(NOW - HOUR, { live: null })], rotatedAtMs, ...seams }), true, "liveness unknown");
assert.equal(retireHeld({ runtimes: () => [run(NOW - HOUR)], rotatedAtMs, live: () => { throw new Error("x"); }, startedAt: () => 0 }), true, "a seam throws");
assert.equal(retireHeld({ runtimes: () => [run(NOW + 2 * RETIRE_LAUNCH_MARGIN_MS), run(NOW - HOUR)], rotatedAtMs, ...seams }), true, "one older run is enough");
});
test("runtimeLive and runtimeStartMs read the supervisor from /proc", async () => {
const me = processIdentity(process.pid);
const record = { phase: "running", supervisor: me };
assert.equal(runtimeLive(record), true);
const expected = Date.now() - process.uptime() * 1000;
assert.ok(Math.abs(runtimeStartMs(record) - expected) < 3000, "start ticks at 100 a second plus btime match this process's start");
assert.equal(runtimeLive({ ...record, supervisor: { ...me, startTime: String(Number(me.startTime) + 1) } }), false, "another process on the same pid");
assert.equal(runtimeStartMs({ ...record, supervisor: { ...me, startTime: "1" } }), null);
assert.equal(runtimeLive({ phase: "running" }), null, "no supervisor: unknown");
const child = spawn(process.execPath, ["-e", "setTimeout(() => {}, 60000)"], { stdio: "ignore" });
await new Promise((resolve) => setTimeout(resolve, 100));
const kid = processIdentity(child.pid);
assert.equal(runtimeLive({ phase: "running", supervisor: kid }), true);
child.kill("SIGKILL");
await new Promise((resolve) => child.on("exit", resolve));
assert.equal(runtimeLive({ phase: "running", supervisor: kid }), false, "a dead supervisor with no pid of its own");
assert.equal(runtimeLive({ phase: "running", supervisor: kid, pid: process.pid, startTime: me.startTime, boot: me.boot }), true, "the runtime's own pid lives on");
assert.equal(runtimeLive(deadRuntime()), false);
// A container runtime has no pid, and its dead docker client is no proof the container exited.
const container = { phase: "running", kind: "container", containerName: `mosaic-${"a".repeat(12)}-${"0".repeat(8)}-0000-4000-8000-${"0".repeat(12)}`, supervisor: kid };
assert.equal(runtimeLive(container), null, "a container whose launcher died is unknown, not gone");
assert.equal(runtimeLive({ ...container, containerName: undefined }), null, "kind container alone is enough");
assert.equal(runtimeLive({ ...container, kind: "task" }), null, "a task record with a container name, the shape a real worker run registers");
assert.equal(retireHeld({ runtimes: () => [container], rotatedAtMs: NOW }), true, "so it holds the old key");
assert.equal(retireHeld({ runtimes: () => [{ ...container, phase: "ended" }], rotatedAtMs: NOW }), false, "until its record is ended");
assert.equal(runtimeLive({ ...container, supervisor: me }), true, "a live launcher is live");
// A pid this process may not signal or read is unknown, never dead (processing-control's own check).
assert.equal(runtimeLive({ phase: "running", supervisor: { ...me, boot: "not-a-boot-id" } }), null, "an identity the check cannot use");
});
test("a worker key past its grace stays while an older run is live, and goes when none is", async (t) => {
const f = await fixture(t);
const old = f.retiring(NOW - MIN);
const lost = f.retiring(NOW - MIN, { inList: false });
const record = f.seed({ issuedAt: NOW - HOUR, retiring: [clean(old), clean(lost)] });
f.state.keys = f.state.keys.filter((k) => k !== lost.key);
const me = processIdentity(process.pid);
const live = () => [{ phase: "running", supervisor: me }];
// This process is live (read from /proc); the seam dates its start before the rotation.
const retireGuard = { startedAt: () => NOW - 2 * HOUR };
// A lost key leaves the file even while held; the old key stays in the list and the file.
const o = f.opts({ runtimes: live, retireGuard });
await ensureReady(o);
assert.equal(o.stages[0], "locked");
assert.ok(f.state.keys.includes(old.key));
assert.deepEqual(f.read().retiring.map((r) => r.keyId), [old.keyId]);
// Held: the launch takes the fast path and leaves the old key.
const o2 = f.opts({ runtimes: live, retireGuard });
await ensureReady(o2);
assert.deepEqual(o2.stages, ["fast"]);
assert.ok(f.state.keys.includes(old.key));
// No runtimes source at all: held too, for every verb.
await retireKeys(f.opts());
await sweepKeys(f.opts());
assert.ok(f.state.keys.includes(old.key));
// A dead run from a failed launch does not hold it.
const o3 = f.opts({ runtimes: () => [deadRuntime(), { phase: "ended", supervisor: me }], retireGuard });
assert.equal((await ensureReady(o3)).action, "retired");
assert.equal(f.state.keys.includes(old.key), false);
assert.deepEqual(f.read().retiring, []);
assert.equal(f.read().key, record.key);
// Triage is not held: its retire drops a key past its grace with no runtimes source.
await issueKey({ configDir: f.configDir, system: f.system, now: () => NOW - 2 * DAY });
await rotateKey({ configDir: f.configDir, system: f.system, now: () => NOW - 2 * DAY });
const tOld = f.read("triage").retiring[0].key;
await retireKeys({ configDir: f.configDir, system: f.system, now: () => NOW });
assert.equal(f.state.keys.includes(tOld), false);
});
test("a due rotation waits while older runs hold MAX_RETIRING keys", async (t) => {
const f = await fixture(t);
const retiring = Array.from({ length: 16 }, () => clean(f.retiring(NOW - MIN)));
const record = f.seed({ issuedAt: NOW - 31 * DAY, retiring });
const me = processIdentity(process.pid);
const retireGuard = { startedAt: () => NOW - 32 * DAY };
const o = f.opts({ runtimes: () => [{ phase: "running", supervisor: me }], retireGuard });
const done = await ensureReady(o);
assert.equal(done.action, "held");
assert.deepEqual(o.stages, ["fast"]);
assert.equal(f.read().key, record.key);
// The terminal rotate still refuses, in worker words.
const e = await refused(rotateKey(f.opts({ runtimes: () => [{ phase: "running", supervisor: me }], retireGuard })), "worker-key-failed");
assert.match(e.message, /too many worker keys/);
});
test("keyStatus reads the worker file and its orphans", async (t) => {
const f = await fixture(t);
await ensureReady(f.opts());
const orphan = keyPrefix(f.configDir, "worker") + b64();
f.state.keys.push(orphan);
const s = await keyStatus({ configDir: f.configDir, system: f.system, purpose: "worker" });
assert.equal(s.proxy.state, "read");
assert.equal(s.proxy.holdsKey, true);
assert.deepEqual(s.proxy.orphanKeyIds, [id(orphan)]);
const triage = await keyStatus({ configDir: f.configDir, system: f.system });
assert.equal(triage.record, null);
assert.deepEqual(triage.proxy.orphanKeyIds, []);
// keyStatus returns the record itself (the Triage CLI prints only ids); scan the proxy half.
OUTPUTS.push(JSON.stringify(s.proxy), JSON.stringify(triage.proxy));
});
test("no output, error or receipt carries key material", () => {
assert.ok(SEEN.size > 10);
assert.ok(OUTPUTS.length > 10);
for (const text of OUTPUTS) assert.equal(leaks(text), false, "an output, error or receipt leaked a key");
});
// The Triage refusals keep their codes and words exactly as they were before purposes.
test("Triage refusals, receipts and fence keep their exact codes and words", async (t) => {
const f = await fixture(t);
const triage = (extra = {}) => ({ configDir: f.configDir, system: f.system, now: () => f.now, ...extra });
const exact = async (promise, code, message) => {
const e = await refused(promise, code);
if (message instanceof RegExp) assert.match(e.message, message); else assert.equal(e.message, message);
};
await exact(ensureKey({ configDir: f.configDir, system: { dataRoot: f.root } }), "triage-key-no-proxy", "the instance has no model proxy set up (proxy.json)");
await exact(rotateKey(triage()), "triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue");
await issueKey(triage());
await exact(issueKey(triage()), "triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate");
fs.chmodSync(f.triage.file, 0o644);
await exact(ensureKey(triage()), "triage-key-unreadable", /^the Triage key store is unreadable: /);
fs.chmodSync(f.triage.file, 0o600);
const sentinel = f.others.sentinel;
f.state.keys = f.state.keys.filter((k) => k !== sentinel);
await exact(ensureKey(triage()), "triage-key-proxy-open", "the proxy's key list has no sentinel for this instance, so it may be another proxy");
f.state.keys.push(sentinel);
f.state.onGet = (n) => { if (n % 2 === 0) f.state.keys.push(`unrelated-${n}`); };
await exact(rotateKey(triage()), "triage-key-raced", "the proxy's key list changed while this verb ran; run it again");
f.state.onGet = null;
f.state.ignorePut = true;
await exact(rotateKey(triage()), "triage-key-unconfirmed", "the proxy did not confirm the change; the instance proxy fence stays set, see mosaic proxy fence status");
f.state.ignorePut = false;
assert.match(fs.readFileSync(join(f.root, "proxy", "mosaic-keys.uncertain"), "utf8"), /"triage\.rotate"/);
fs.rmSync(join(f.root, "proxy", "mosaic-keys.uncertain"));
fs.chmodSync(f.receipts, 0o644);
await exact(rotateKey(triage()), "triage-key-receipts-unsafe", /; nothing was changed$/);
fs.chmodSync(f.receipts, 0o600);
await exact(rotateKey(triage({ hook: (s) => { if (s === "rename") throw new Error("stop"); } })), "triage-key-unwritten",
`the proxy list changed but ${f.triage.file} was not written (the Triage key store is unreadable: the key file cannot be written); fix the file's directory, then run mosaic triage key rotate again, which clears what this run left`);
await exact(rotateKey(triage({ hook: (s) => { if (s === "sync") throw new Error("stop"); } })), "triage-key-unsynced",
`${f.triage.file} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status`);
for (let i = 0; i < 20 && f.read("triage").retiring.length < 16; i++) await rotateKey(triage());
await exact(rotateKey(triage()), "triage-key-failed", "too many keys are retiring; run mosaic triage key retire once their grace has passed");
const ops = fs.readFileSync(f.receipts, "utf8").trim().split("\n").map((l) => JSON.parse(l).op);
assert.ok(ops.length > 3 && ops.every((op) => /^triage\.key\.(issue|rotate)$/.test(op)), ops.join(","));
});
+1 -1
View File
@@ -66,7 +66,7 @@ export const USAGE = `usage:
${PLUGINS_USAGE}
${CREDENTIALS_USAGE}
mosaic proxy fetch | status [--json] | fence status [--json] | fence clear --proxy-restarted | fence clear --restart [--business ID] | init [--generate-key] [--port N] [--business ID] [--rotate-grace-minutes N] [--first-output-seconds N]
mosaic proxy unit render|install|start|stop|restart|status [--json] | init-lock status|clear
mosaic proxy unit render|install|start|stop|restart|status [--json] | init-lock status|clear | ride --instance NAME | ride --off
${HOSTS_USAGE}
${GRANTS_USAGE}
${ORG_INSTANCES_USAGE}
+17 -3
View File
@@ -17,6 +17,7 @@ import { systemConfigProblem } from "../../organization/src/system-config-proble
import { businessFilePath } from "../../organization/src/organization-config.mjs";
import { busOrganization } from "../../bus/src/organization-context.mjs";
import { BusError } from "../../bus/src/broker.mjs";
import { RIDE_LINES } from "../../model-proxy/src/settings.mjs";
import { CliError } from "./errors.mjs";
import { accessContextFor } from "./access.mjs";
@@ -57,6 +58,9 @@ function instanceProblem(problem, label) {
m = new RegExp(`^instance-config-mismatch: --instance (${INSTANCE}) names `).exec(body);
if (m) return `instance-config-mismatch: --instance ${m[1]} names its own config, but MOSAIC_CONFIG names another`;
if (body.startsWith("instance-config-mismatch: ")) return "instance-config-mismatch: --instance names its own config, but MOSAIC_CONFIG names another";
// proxy-ride refusals carry their fixed owner line from settings.mjs, never the child's text.
m = /^(proxy-ride-(?:invalid|owner-missing|conflict)): /.exec(body);
if (m) return `${m[1]}: ${RIDE_LINES[m[1]]}`;
if (body.startsWith("instance-name-invalid: ")) return `instance-name-invalid: an instance name must match /^${INSTANCE}$/`;
if (body.startsWith("configuration not found: ")) {
m = new RegExp(`\\(run (scripts/bootstrap\\.sh(?: --instance ${INSTANCE})?) to create it\\)$`).exec(body);
@@ -65,7 +69,14 @@ function instanceProblem(problem, label) {
return null;
}
export function loadSystem({ env = process.env, instance = null } = {}) {
// proxy-ride first slice (Jason 2026-11-21): on an instance whose proxy-ride.json names another
// instance, validate prints the owner's proxy with an `owner` object. Only worker runs use it
// (`ride: true`, packages/processing-control/src/container-cli.mjs). Every other caller, the
// bus boot and the Console among them, gets the system config without that proxy, so a riding
// instance's buses run exactly as an instance with no proxy runs. That holds while the owner is
// valid: validate resolves the owner on every call, so while proxy-ride.json exists an owner
// problem refuses every verb on the rider, buses and the Console included.
export function loadSystem({ env = process.env, instance = null, ride = false } = {}) {
const proc = spawnSync(process.execPath, [join(REPO, "scripts", "mosaic-config.mjs"), "validate", ...(instance === null ? [] : ["--instance", instance])], {
encoding: "utf8",
maxBuffer: 1024 * 1024,
@@ -76,11 +87,14 @@ export function loadSystem({ env = process.env, instance = null } = {}) {
const known = instanceProblem(String(proc.stderr ?? "").trim(), label);
throw new CliError(known ? `system config problem (mosaic-config exit ${proc.status}): ${known}` : systemConfigProblem(proc, env, label), 3);
}
let system;
try {
return JSON.parse(proc.stdout);
system = JSON.parse(proc.stdout);
} catch {
throw new CliError("system config: mosaic-config printed something that isn't JSON", 3);
}
if (!ride && system?.proxy?.owner) delete system.proxy;
return system;
}
export function rolesDir(env = process.env) {
@@ -207,7 +221,7 @@ export function bootConfig({ system, businessId, orgRoot = system.dataRoot, env
const config = {
dataRoot: orgRoot,
...(orgRoot === system.dataRoot ? {} : { instanceRoot: system.dataRoot, org: { id: b.id }, instanceDataRoot: system.dataRoot, instanceConfigDir: configDir(env) }),
...(system.proxy ? { proxy: system.proxy } : {}),
...(system.proxy && !system.proxy.owner ? { proxy: system.proxy } : {}),
businesses: { [b.id]: busOrganization(b, resolved) },
organization: {dir:configDir(env),rolesDir:rolesDir(env),system:vars},
launches: [],
+41 -1
View File
@@ -1,12 +1,15 @@
import { lstatSync } from 'node:fs';
import { join } from 'node:path';
import { configDir } from '../../organization/src/organization-config.mjs';
import { createProxyLock } from '../../bus/src/proxy-lock.mjs';
import { loadManagementClient } from '../../model-proxy/src/client.mjs';
import { CliError } from './errors.mjs';
import { loadSystem } from './config.mjs';
import { orgSelector } from './org-selector.mjs';
import { refuseInsideAgent } from './transport.mjs';
import { fetchPinned } from '../../model-proxy/src/fetch.mjs';
import { validManagementKey, proxyTiming } from '../../model-proxy/src/common.mjs';
import { RIDE_LINES, RIDE_NAME, readRide, createRide, removeRide } from '../../model-proxy/src/settings.mjs';
import { createProxyAdmin } from '../../model-proxy/src/admin-control.mjs';
import { listOrgs } from '../../organization/src/org-registry.mjs';
import { busesStopped, instanceFromConfig } from './proxy-admin-host.mjs';
@@ -91,13 +94,50 @@ async function setupCommand(verb, rest, io, { system, deps }) {
throw failure('proxy-usage', 4);
}
// proxy-ride first slice (Jason 2026-11-21): `mosaic proxy ride --instance NAME` writes this
// instance's proxy-ride.json, then loads the system config exactly as a worker run loads it; any
// refusal removes the file again, so a file the loader refuses is never left behind. `--off`
// removes the file without loading the config, so it works when the owner is gone. Neither form
// changes the owner instance, its proxy or any key.
const RIDE_CODES = Object.keys(RIDE_LINES);
const rideFailure = (code) => Object.assign(new CliError(`${code}: ${RIDE_LINES[code]}`, 2), { code });
const rideCode = (error) => RIDE_CODES.find(code => error?.code === code || new RegExp(`(?:^|: )${code}: `).test(error?.message ?? ''));
async function rideCommand(rest, io, { validate }) {
const dir = configDir(io.env);
if (rest.length === 1 && rest[0] === '--off') {
let removed;
try { removed = removeRide(dir); } catch { throw rideFailure('proxy-ride-invalid'); }
return print(io, { riding: null, removed });
}
if (rest.length !== 2 || rest[0] !== '--instance') throw failure('proxy-usage', 4);
const name = rest[1];
if (!RIDE_NAME.test(name)) throw rideFailure('proxy-ride-invalid');
let current;
try { current = readRide(dir); } catch (error) { throw rideFailure(rideCode(error) ?? 'proxy-ride-invalid'); }
if (current && current.instance !== name) {
throw Object.assign(new CliError(`proxy-usage: this instance already rides instance ${current.instance}; run mosaic proxy ride --off first`, 4), { code: 'proxy-usage' });
}
let undo = () => {};
if (!current) {
try { lstatSync(join(dir, 'proxy.json')); throw rideFailure('proxy-ride-conflict'); }
catch (error) { if (error instanceof CliError) throw error; if (error.code !== 'ENOENT') throw rideFailure('proxy-ride-conflict'); }
try { undo = createRide(dir, name); } catch { throw rideFailure('proxy-ride-invalid'); }
}
let system;
try { system = validate(); }
catch (error) { undo(); const code = rideCode(error); if (code) throw rideFailure(code); throw error; }
if (system?.proxy?.owner?.instance !== name || !Number.isInteger(system.proxy.port)) { undo(); throw rideFailure('proxy-ride-owner-missing'); }
return print(io, { riding: name, port: system.proxy.port });
}
export async function proxyCommand(args, io, { system, callFor, prompt = promptProxyKey, fetch = fetchPinned, ...deps }) {
const [verb, ...rest] = args;
if (args.includes('--tester')) throw failure('human-required', 3);
// #1873: every proxy verb, read or write, runs only from a human shell.
if (['init', 'fetch', 'fence', 'status', 'unit', 'init-lock'].includes(verb)) {
if (['init', 'fetch', 'fence', 'status', 'unit', 'init-lock', 'ride'].includes(verb)) {
try { refuseInsideAgent(io.env); } catch { throw failure('human-required', 3); }
}
if (verb === 'ride') return rideCommand(rest, io, { validate: deps.rideValidate ?? (() => loadSystem({ env: io.env, ride: true })) });
if (['status', 'unit', 'init-lock'].includes(verb)) return setupCommand(verb, rest, io, { system, deps: { ...deps, fetch } });
if (verb === 'fence' && rest[0] === 'clear' && rest.includes('--restart')) {
// #1873: restart the proxy unit, then send the restart evidence to the bus, which re-proves it
+14 -12
View File
@@ -66,35 +66,37 @@ exit 0
const env = { PATH: `${bin}:${process.env.PATH}`, HOME: process.env.HOME, TMPDIR: tmp, LANG: 'C.UTF-8', FAKE_DOCKER_RECORD: record, ...extra };
const r = spawnSync('bash', [join(checkout, 'scripts', script)], { env, encoding: 'utf8', timeout: 600000 });
t.diagnostic(JSON.stringify({ script, extra: Object.keys(extra), wallMs: Math.round(performance.now() - begin), status: r.status }));
const runs = existsSync(record) ? readFileSync(record, 'utf8').trim().split('\n').filter(Boolean).map(l => { const [file, kind, adapter] = l.split('\t'); return { file, kind, adapter }; }) : [];
const runs = existsSync(record) ? readFileSync(record, 'utf8').split('\n').filter(Boolean).map(l => { const [file, kind, adapter] = l.split('\t'); return { file, kind, adapter }; }) : [];
return { out: r.stdout + r.stderr, runs, tmp };
};
}
const fixtureRuns = (runs, tmp) => runs.every(r => r.file !== HOST_AUTH && r.file.startsWith(tmp + '/') && r.kind === 'regular file 600');
// worker-cutover: the suites export no PI_AUTH_FILE and compose mounts no credential file, so no
// compose run carries one, and none ever names the host's.
const noAuthRuns = runs => runs.every(r => r.file === '' && r.kind === 'unset');
test('test-release.sh activates through the mock adapter with a 0600 fixture auth file', t => {
const { out, runs, tmp } = suiteLane(t)('test-release.sh');
test('test-release.sh activates through the mock adapter with no auth file', t => {
const { out, runs } = suiteLane(t)('test-release.sh');
assert.ok(runs.length >= 3, `fault, healthy and repeat activations reach Compose\n${out}`);
assert.ok(fixtureRuns(runs, tmp), JSON.stringify(runs));
assert.ok(noAuthRuns(runs), JSON.stringify(runs));
assert.deepEqual([...new Set(runs.map(r => r.adapter))], ['mock'], 'ruling A: no pi adapter in the gate');
assert.match(out, /OK +healthy activation succeeds/); assert.match(out, /OK +fault-injected activation refuses/);
});
test('test-task.sh mounts only its fixture, uses only the mock adapter and prints the live skip twice', t => {
const { out, runs, tmp } = suiteLane(t)('test-task.sh');
test('test-task.sh passes no auth file, uses only the mock adapter and prints the live skip twice', t => {
const { out, runs } = suiteLane(t)('test-task.sh');
assert.ok(runs.length >= 1, out);
assert.ok(fixtureRuns(runs, tmp), JSON.stringify(runs));
assert.ok(noAuthRuns(runs), JSON.stringify(runs));
assert.deepEqual([...new Set(runs.map(r => r.adapter))], ['mock']);
assert.equal(out.split('\n').filter(l => l === LIVE_SKIP).length, 2, out);
assert.match(out, /OK +hello task succeeds with exact marker/);
});
test('M-T: MOSAIC_LIVE_PROVIDER=1 runs the real-model cases, which fail under a fixture auth file', t => {
test('M-T: MOSAIC_LIVE_PROVIDER=1 runs the real-model cases, which fail before Compose with no model proxy', t => {
const lane = suiteLane(t), auth = join(scratch(t), 'operator-auth.json'); writeFileSync(auth, '{}'); chmodSync(auth, 0o600);
const { out, runs } = lane('test-task.sh', { MOSAIC_LIVE_PROVIDER: '1', PI_AUTH_FILE: auth });
assert.ok(!out.includes(LIVE_SKIP), out);
// The fake answers a pi run with nothing: the fork base misses REMEMBERED, recall misses the name.
// worker-cutover: an operator's PI_AUTH_FILE no longer runs a worker. The suite's pi config has no
// model proxy next to it, so each pi launch refuses before Compose: the fork base and the recall fail.
assert.match(out, /FAIL fork base: teach succeeds/); assert.match(out, /FAIL recalled user name/);
assert.ok(runs.some(r => r.adapter === 'pi'), JSON.stringify(runs));
assert.ok(runs.every(r => r.file === auth), 'live mode keeps the operator\'s PI_AUTH_FILE');
assert.ok(runs.length >= 1 && runs.every(r => r.adapter === 'mock'), JSON.stringify(runs));
});
@@ -468,10 +468,10 @@ test("W3 host watch: the default stat key reads the plugin root's plugins, and a
const dataRoot = scratch(t);
// The plugin root sits beside TMPDIR, not in it: TMPDIR is an env value the backstop already
// knows, so a root under it would be caught without the plugin root as a needle of its own.
// When TMPDIR's parent can't take it, the test skips: a root under TMPDIR would pass with the
// plugin root left out of the backstop.
// With TMPDIR unset, its value cannot mask the check, so a separate scratch directory works.
// Otherwise skip if the parent cannot take a sibling: a child would give a false pass.
let pluginRoot;
try { pluginRoot = mkdtempSync(join(dirname(tmpdir()), "mosaic-hw-plugins-")); }
try { pluginRoot = process.env.TMPDIR ? mkdtempSync(join(dirname(tmpdir()), "mosaic-hw-plugins-")) : scratch(t); }
catch { t.skip("the plugin root must sit beside TMPDIR, and TMPDIR's parent is not writable"); return; }
t.after(() => rmSync(pluginRoot, { recursive: true, force: true }));
const clock = fakeClock();
@@ -487,6 +487,7 @@ test("W3 host watch: the default stat key reads the plugin root's plugins, and a
// A plugin lock that can't be read changes the plugin root's key (the loader refuses it): a full pass.
writeFileSync(pluginPaths(pluginRoot).lock, "{", { mode: 0o600 });
line = `${pluginRoot}/plugins/pi`;
assert.equal(leakChecker({ env: {}, scratch: null, dataRoot })({ line }), false, "without the plugin root the backstop does not catch this path");
clock.fire(); await handle.settled();
assert.equal(stub.calls.length, 2, "the key read the plugin root, so its change ran a full pass");
assert.match(logs.join("\n"), /a state line held a host path or an env value/, "and the record that names the plugin root was not written");
+148
View File
@@ -0,0 +1,148 @@
// proxy-ride first slice (Jason 2026-11-21): `mosaic proxy ride --instance NAME | --off`, the CLI
// loader's view of a riding instance, the bus boot config, and the worker launcher's refusal line.
// The rider is the cli fixture's config; the owner is named instance "owner" under a made-up HOME.
// The real scripts/mosaic-config.mjs runs; nothing reads a real config, proxy or secret.
import test from 'node:test';
import assert from 'node:assert/strict';
import { existsSync, lstatSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { spawnSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { main } from '../src/cli.mjs';
import { bootConfig, loadSystem } from '../src/config.mjs';
import { proxyCommand } from '../src/model-proxy.mjs';
import { RIDE_LINES, installPaths } from '../../model-proxy/src/settings.mjs';
import { fixture, io, tmp } from './helpers.mjs';
const MANAGE = 'ride-cli-management-key';
const CONTAINER_CLI = fileURLToPath(new URL('../../processing-control/src/container-cli.mjs', import.meta.url));
// The stand-in launch step of worker-proxy-task.test.mjs; FAKE_WP=proxy answers with the proxy it got.
const HOOK = fileURLToPath(new URL('../../processing-control/tests/fixtures/worker-proxy-hook.mjs', import.meta.url));
const code = (c) => (e) => e.code === c;
function setup(t) {
const root = tmp(t, 'rcl-');
const f = fixture(root, 'acme');
f.env.HOME = root;
const ownerDir = join(root, '.config', 'mosaic-owner'), ownerData = join(root, 'od');
mkdirSync(join(ownerDir, 'secrets'), { recursive: true, mode: 0o700 }); mkdirSync(ownerData, { mode: 0o700 });
writeFileSync(join(ownerDir, 'config.json'), JSON.stringify({ configVersion: 1, environment: 'development', dataRoot: ownerData, execution: { backend: 'docker', provider: 'fixture', model: 'fixture' } }));
writeFileSync(join(ownerData, 'instance.json'), JSON.stringify({ instanceVersion: 1, name: 'owner' }));
const value = { port: 18452, binary: installPaths(ownerData).binary };
writeFileSync(join(ownerDir, 'proxy.json'), JSON.stringify(value));
writeFileSync(join(ownerDir, 'secrets', 'proxy-management.key'), MANAGE, { mode: 0o600 });
const riderDir = join(root, 'config');
return { root, f, env: f.env, riderDir, rideFile: join(riderDir, 'proxy-ride.json'), owner: { dir: ownerDir, dataRoot: ownerData, value } };
}
const ride = (s, args, env = s.env) => { const x = io(env); return main(['proxy', 'ride', ...args], x).then(() => x); };
test('ride --instance writes the file, prints the owner port, and a second identical ride holds it', async (t) => {
const s = setup(t);
let x = await ride(s, ['--instance', 'owner']);
assert.deepEqual(JSON.parse(x.out.text), { riding: 'owner', port: 18452 });
assert.equal(readFileSync(s.rideFile, 'utf8'), '{"version":1,"instance":"owner"}\n');
assert.equal(lstatSync(s.rideFile).mode & 0o777, 0o600);
assert.equal(x.out.text.includes(MANAGE) || x.err.text.includes(MANAGE), false);
x = await ride(s, ['--instance', 'owner']);
assert.deepEqual(JSON.parse(x.out.text), { riding: 'owner', port: 18452 });
// Nothing of the owner's changed.
assert.deepEqual(JSON.parse(readFileSync(join(s.owner.dir, 'proxy.json'), 'utf8')), s.owner.value);
assert.equal(existsSync(join(s.riderDir, 'proxy.json')), false);
});
test('ride and ride --off run only from a human shell', async (t) => {
const s = setup(t);
for (const args of [['--instance', 'owner'], ['--off']]) {
await assert.rejects(ride(s, args, { ...s.env, CLAUDECODE: '1' }), code('human-required'));
await assert.rejects(ride(s, [...args, '--tester']), code('human-required'));
}
assert.equal(existsSync(s.rideFile), false);
});
test('a ride the loader refuses is removed again, with the ride code and its owner line', async (t) => {
const s = setup(t);
const cases = [
[['--instance', 'nobody'], 'proxy-ride-owner-missing', () => {}],
[['--instance', 'owner'], 'proxy-ride-owner-missing', () => rmSync(join(s.owner.dir, 'proxy.json'))],
];
for (const [args, c, prep] of cases) {
prep();
await assert.rejects(ride(s, args), (e) => e.code === c && e.message === `${c}: ${RIDE_LINES[c]}` && e.exitCode === 2);
assert.equal(existsSync(s.rideFile), false, `${c}: the refused file is gone`);
}
await assert.rejects(ride(s, ['--instance', '../x']), code('proxy-ride-invalid'));
assert.equal(existsSync(s.rideFile), false);
});
test('ride refuses beside its own proxy.json, and over a ride of another instance', async (t) => {
const s = setup(t);
writeFileSync(join(s.riderDir, 'proxy.json'), '{}');
await assert.rejects(ride(s, ['--instance', 'owner']), code('proxy-ride-conflict'));
assert.equal(existsSync(s.rideFile), false);
rmSync(join(s.riderDir, 'proxy.json'));
writeFileSync(s.rideFile, '{"version":1,"instance":"other"}\n', { mode: 0o600 });
await assert.rejects(ride(s, ['--instance', 'owner']), (e) => e.code === 'proxy-usage' && e.exitCode === 4 && /already rides instance other/.test(e.message));
assert.equal(readFileSync(s.rideFile, 'utf8'), '{"version":1,"instance":"other"}\n', 'the existing ride is left as it was');
});
test('ride --off removes the file, also when the owner is gone, and says when there was none', async (t) => {
const s = setup(t);
await ride(s, ['--instance', 'owner']);
rmSync(s.owner.dir, { recursive: true });
let x = await ride(s, ['--off']);
assert.deepEqual(JSON.parse(x.out.text), { riding: null, removed: true });
assert.equal(existsSync(s.rideFile), false);
x = await ride(s, ['--off']);
assert.deepEqual(JSON.parse(x.out.text), { riding: null, removed: false });
for (const args of [[], ['--instance'], ['--off', '--instance', 'owner'], ['--on']]) await assert.rejects(ride(s, args), code('proxy-usage'));
});
test('a validate seam that answers another owner or no port removes the file', async (t) => {
const s = setup(t);
for (const answer of [{ proxy: { port: 1, owner: { instance: 'elsewhere' } } }, { proxy: { owner: { instance: 'owner' } } }, {}]) {
const x = io(s.env);
await assert.rejects(proxyCommand(['ride', '--instance', 'owner'], x, { system: () => ({}), rideValidate: () => answer }), code('proxy-ride-owner-missing'));
assert.equal(existsSync(s.rideFile), false);
}
});
test('loadSystem drops the ridden proxy unless a worker run asks; the bus boot config is the one with no proxy', async (t) => {
const s = setup(t);
const plain = bootConfig({ system: loadSystem({ env: s.env }), businessId: 'acme', env: s.env });
await ride(s, ['--instance', 'owner']);
const off = loadSystem({ env: s.env });
assert.equal(Object.hasOwn(off, 'proxy'), false, 'every caller but a worker run sees no proxy');
const on = loadSystem({ env: s.env, ride: true });
assert.deepEqual(on.proxy, { ...s.owner.value, owner: { instance: 'owner', configDir: s.owner.dir, dataRoot: s.owner.dataRoot } });
for (const system of [off, on]) {
const boot = bootConfig({ system, businessId: 'acme', env: s.env });
assert.equal(Object.hasOwn(boot, 'proxy'), false);
assert.deepEqual(boot, plain, "the rider's bus boots exactly as it did before the ride");
}
// An instance with its own proxy keeps it in the boot config, as before.
const own = bootConfig({ system: { ...off, proxy: { port: 18460, binary: '/x' } }, businessId: 'acme', env: s.env });
assert.deepEqual(own.proxy, { port: 18460, binary: '/x' });
});
test('a ride refusal reaches loadSystem and the worker launcher as its code and owner line', async (t) => {
const s = setup(t);
writeFileSync(s.rideFile, '{"version":1,"instance":"owner","path":"/x"}', { mode: 0o600 });
const line = `proxy-ride-invalid: ${RIDE_LINES['proxy-ride-invalid']}`;
assert.throws(() => loadSystem({ env: s.env }), (e) => e.exitCode === 3 && e.message === `system config problem (mosaic-config exit 2): ${line}`);
const r = spawnSync(process.execPath, [CONTAINER_CLI, '--check'], { env: s.env, encoding: 'utf8', timeout: 20000 });
assert.equal(r.status, 3, r.stderr);
assert.equal(r.stderr, `processing: ${line}\n`);
assert.equal(r.stderr.includes(s.root), false);
});
test("a worker launch through container-cli is given the owner's proxy", async (t) => {
const s = setup(t);
const launch = () => spawnSync(process.execPath, ['--import', HOOK, CONTAINER_CLI], { env: { ...s.env, FAKE_WP: 'proxy' }, encoding: 'utf8', timeout: 20000 });
let r = launch();
assert.equal(r.status, 0, r.stderr);
assert.equal(r.stdout, 'no-owner:no-proxy', 'no ride file: no proxy, as before');
await ride(s, ['--instance', 'owner']);
r = launch();
assert.equal(r.status, 0, r.stderr);
assert.equal(r.stdout, 'owner:18452');
});
@@ -240,8 +240,8 @@ export async function lifecycleFixture(t) {
}
async function consoleBrowser(t, f) {
const b = await browser(); t.after(() => b.close());
b.on('Fetch.requestPaused', ({ requestId, request: incoming }) => {
let b = await browser(); const consolePage = b; t.after(() => consolePage.close());
consolePage.on('Fetch.requestPaused', ({ requestId, request: incoming }) => {
const url = new URL(incoming.url);
if (incoming.method === 'POST') f.recordOwner(url.pathname);
const req = httpRequest({ host: '127.0.0.1', port: f.port, path: url.pathname + url.search, method: incoming.method,
@@ -249,9 +249,9 @@ async function consoleBrowser(t, f) {
const parts = []; res.on('data', part => parts.push(part)); res.on('end', () => {
const responseHeaders = Object.entries(res.headers).filter(([name]) => !['connection','keep-alive','transfer-encoding'].includes(name))
.flatMap(([name, value]) => (Array.isArray(value) ? value : [value]).map(value => ({ name, value: String(value) })));
b.call('Fetch.fulfillRequest', { requestId, responseCode: res.statusCode, responseHeaders, body: Buffer.concat(parts).toString('base64') }).catch(() => {});
consolePage.call('Fetch.fulfillRequest', { requestId, responseCode: res.statusCode, responseHeaders, body: Buffer.concat(parts).toString('base64') }).catch(() => {});
});
}); req.on('error', () => b.call('Fetch.failRequest', { requestId, errorReason: 'Failed' }).catch(() => {})); req.end(incoming.postData);
}); req.on('error', () => consolePage.call('Fetch.failRequest', { requestId, errorReason: 'Failed' }).catch(() => {})); req.end(incoming.postData);
});
await b.call('Fetch.enable', { patterns: [{ urlPattern: ORIGIN + '/*', requestStage: 'Request' }] });
const s = await f.current(); await b.navigate(`${ORIGIN}/#/inbox/${s.selection.decision}`);
@@ -267,7 +267,10 @@ async function consoleBrowser(t, f) {
await wait('document.querySelector("#decide-status")?.textContent.startsWith("Recorded")');
},
async offline(html) {
await b.call('Fetch.disable');
// The Console can still poll during navigation. Close it before observing the downloaded file.
await b.close();
b = await browser(); const offlinePage = b; t.after(() => offlinePage.close());
external.length = 0;
await b.call('Page.addScriptToEvaluateOnNewDocument', { source: `window.__now=1000;Date.now=()=>window.__now;window.__ticks=[];window.setInterval=fn=>{window.__ticks.push(fn);return window.__ticks.length;};window.__sounds=0;window.AudioContext=class{currentTime=0;destination={};resume(){}createOscillator(){return{connect(){},frequency:{value:0},start(){window.__sounds++},stop(){}}}createGain(){return{connect(){},gain:{value:0}}}};` });
await b.call('Network.enable'); await b.call('Network.emulateNetworkConditions', { offline: true, latency: 0, downloadThroughput: 0, uploadThroughput: 0 });
b.on('Network.requestWillBeSent', ({ request }) => { if (/^https?:/.test(request.url)) external.push(request.url); });
+3 -1
View File
@@ -2210,10 +2210,12 @@ export class Store {
// C1's scripts/mosaic runs B's host, as a copy-check failure needs.
// Row 294: config.mjs imports access/transport. Task 5b also imports the built-in catalogue and
// its task, bus, seat and processing-control dependencies; the rehearsal needs those files too.
// proxy-ride-1: config.mjs imports the ride refusal lines from model-proxy/src/settings.mjs, whose
// common.mjs reads model-proxy/PIN when it loads.
const RCOPIES=join(base,'rcopies');mkdirSync(RCOPIES);
for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],[C1,'3g',['3f']]]){
const d=join(RCOPIES,id);
for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/cli/src/access.mjs','packages/cli/src/org-selector.mjs','packages/cli/src/transport.mjs','packages/bus/src/business.mjs','packages/bus/src/organization-context.mjs','packages/bus/src/request-store.mjs','packages/bus/src/requirement-catalog.mjs','packages/tasks/src','packages/seat/src/landing.mjs','packages/seat/src/checkout.mjs','packages/processing-control/src','packages/plugins/src/kinds.mjs','packages/organization/src','packages/business/src','packages/library/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/cli/src/access.mjs','packages/cli/src/org-selector.mjs','packages/cli/src/transport.mjs','packages/bus/src/business.mjs','packages/bus/src/organization-context.mjs','packages/bus/src/request-store.mjs','packages/bus/src/requirement-catalog.mjs','packages/tasks/src','packages/seat/src/landing.mjs','packages/seat/src/checkout.mjs','packages/processing-control/src','packages/model-proxy/src','packages/model-proxy/PIN','packages/plugins/src/kinds.mjs','packages/organization/src','packages/business/src','packages/library/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
// Row 333 R2 (#1831): what the business loader imports from outside its directory, model-quota.mjs included.
for(const p of businessImportedFiles(repo))if(!existsSync(join(d,p)))cpSync(join(repo,p),join(d,p));
if(id===C1)writeFileSync(join(d,'scripts','mosaic'),`#!/usr/bin/env bash\nexec node "${join(RCOPIES,B,'packages','cli','src','cli.mjs')}" "$@"\n`,{mode:0o755});
+9 -2
View File
@@ -72,8 +72,15 @@ trust and `--no-approve` would then hide a seal flag the binary ignores.
Then one sealed rpc run per loader flag leaves out only that flag: its
kind must show, and no other kind (an MCP server may show beside an
extension when `--help` lists no `--no-mcp`, since pi then ties MCP to
`--no-extensions`). So each loader flag is shown to keep its own kind
out, and no other seal flag, nor their order, can stand in for it. Five
`--no-extensions`). If `--no-extensions` also disables the MCP loader,
the check leaves out both it and `--no-mcp` in one additional rpc run.
That run must show MCP and no kind beyond MCP and extensions. Together
with the run leaving out only `--no-extensions` (extensions load, MCP
does not), this proves `--no-mcp` with its loader enabled. The planted
MCP servers have distinct names so a trusted project entry cannot hide
the user entry. No version comparison selects these checks.
So each loader flag is shown to keep its own kind out, and no other
seal flag, nor their order, can stand in for it. Five
sealed `-p` runs have a bus session's shape: the arguments the adapter
passes for a session, in its order, from `sessionShape` in
`src/pi-fit.mjs`, with two `-e` and `--session-dir`. The adapter's four
+2 -1
View File
@@ -149,7 +149,8 @@ export function turnCause(stdout, secrets = []) {
}
// The chat body for a failed turn: the plain cause first, the technical reason after.
export const TURN_FAILURE_NOTICE_PREFIX = 'The turn failed: ';
export const DETAILS = "\nDetails: ";
export function failureBody(turn) {
return turn.cause ? `The turn failed: ${turn.cause}${DETAILS}${turn.reason}` : `The turn failed: ${turn.reason}`;
return turn.cause ? `${TURN_FAILURE_NOTICE_PREFIX}${turn.cause}${DETAILS}${turn.reason}` : `${TURN_FAILURE_NOTICE_PREFIX}${turn.reason}`;
}
+22 -5
View File
@@ -28,7 +28,9 @@
// kind it keeps out must show, and no other kind (pi without --no-mcp ties MCP to
// --no-extensions, so an MCP server may show beside an extension). So each loader flag
// is shown to keep its own kind out, and no other seal flag, nor the order they come
// in, can stand in for it.
// in, can stand in for it. If --no-extensions also disables the MCP loader, a run
// without both it and --no-mcp must show MCP and extensions only. Paired with the
// --no-extensions one-out run, this proves --no-mcp with its loader enabled.
// session five -p runs in a bus session's shape, with the arguments adapters/pi/adapter.sh
// passes for a session, in its order (sessionShape below): two -e, --session-dir in
// place of --no-session, a fixed --provider and --model, -p and a message. The
@@ -570,13 +572,26 @@ async function judge(found, { levels, who, digest, run }) {
// Each loader flag on its own: with every other seal flag on, its kind must show.
for (const [flag, kind, pending] of oneOut) {
const r = await pending;
const but = `with every seal flag but ${flag}`;
let r = await pending;
let but = `with every seal flag but ${flag}`;
if (r.stopped) return unsure(`the start ${but} ${r.stopped}`);
if (!r.answer.state || !r.answer.commands) return unsure(`it didn't answer get_state and get_commands ${but}`);
if (r.prompt === null) return unsure(`its -e extension didn't see the system prompt ${but}`);
// pi-152: MCP can itself be a built-in extension. Removing --no-mcp alone then leaves its
// loader disabled. The earlier --no-extensions one-out run proved that extensions load but
// MCP stays off with --no-mcp. Remove that flag too: MCP must now run, and no unrelated kind
// may load. This pair proves the flag, without demanding that the two seals be independent.
let pairedMcp = false;
if (flag === "--no-mcp" && r.seen.length === 0) {
pairedMcp = true;
but = "with every seal flag but --no-extensions and --no-mcp";
r = await rpcRun(without("--no-extensions", "--no-mcp"));
if (r.stopped) return unsure(`the start ${but} ${r.stopped}`);
if (!r.answer.state || !r.answer.commands) return unsure(`it didn't answer get_state and get_commands ${but}`);
if (r.prompt === null) return unsure(`its -e extension didn't see the system prompt ${but}`);
}
if (!r.seen.some((s) => s.kind === kind)) return unsure(`it didn't load a planted ${kind} ${but}`);
const others = r.seen.filter((s) => s.kind !== kind && !(flag === "--no-extensions" && !noMcp && s.kind === "MCP server"));
const others = r.seen.filter((s) => s.kind !== kind && !(pairedMcp && s.kind === "extension") && !(flag === "--no-extensions" && !noMcp && s.kind === "MCP server"));
if (others.length) return refuse(`loaded a planted ${others.map((s) => s.source).join(", ")} ${but}`);
}
@@ -896,7 +911,9 @@ function plant(scratch) {
const marks = join(scratch, "marks");
const put = (path, text) => { mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, text); };
const ext = (source) => `import { writeFileSync } from "node:fs";\nexport default function (): void { writeFileSync(${JSON.stringify(join(marks, markerName(source)))}, ""); }\n`;
const mcp = (source) => JSON.stringify({ mcpServers: { fit: { command: "/bin/sh", args: ["-c", `: > ${join(marks, markerName(source))}`] } } });
// A trusted project's server replaces a user server with the same name (pi 1.5.2). The
// control must see a user source independently of project trust, so use a name per source.
const mcp = (source) => JSON.stringify({ mcpServers: { [markerName(source)]: { command: "/bin/sh", args: ["-c", `: > ${join(marks, markerName(source))}`] } } });
const skill = (name) => `---\nname: ${name}\ndescription: a planted skill\n---\nA planted skill.\n`;
const prompt = "A planted prompt template.\n";
const agent = join(scratch, "home", ".pi", "agent");
+53 -5
View File
@@ -124,6 +124,52 @@ test("pi-unpin S3: a stand-in that touches the planted MCP marker under the seal
for (const [kind, line] of Object.entries(lines)) assert.equal((await fit(standIn(t, { sealLeak: [kind] }).env)).message, `pi 2.0.0 (MOSAIC_PI_BIN) ${line}`, kind);
});
// pi-152: a trusted project MCP entry replaces a user entry of the same name. The positive
// control needs a user source, so the fixture must keep the two servers distinct.
test("pi-152: project MCP name precedence does not hide the user positive control", SKIP, async t => {
const s = standIn(t, { extraFlags: ["--no-mcp"], mcpProjectOverride: true });
const r = await fit(s.env);
assert.equal(r.ok, true, r.message);
assert.equal(r.noMcp, true);
});
test("pi-152: MCP name precedence still refuses a user server loaded under the seal flags", SKIP, async t => {
const s = standIn(t, { extraFlags: ["--no-mcp"], mcpProjectOverride: true, sealLeak: ["mcp"] });
const r = await fit(s.env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) loaded a planted user mcp.json with the seal flags on");
noPath(r.message, s.paths);
});
test("pi-152: --no-mcp is proved even when --no-extensions also disables the MCP loader", SKIP, async t => {
const s = standIn(t, { extraFlags: ["--no-mcp"], mcpProjectOverride: true, mcpAlso: "--no-extensions" });
const r = await fit(s.env);
assert.equal(r.ok, true, r.message);
assert.equal(r.noMcp, true);
});
test("pi-152: --no-extensions cannot mask an ignored --no-mcp", SKIP, async t => {
const s = standIn(t, { extraFlags: ["--no-mcp"], mcpProjectOverride: true, standIn: ["--no-mcp", "--no-extensions"] });
const r = await fit(s.env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.equal(r.message, "pi 2.0.0 (MOSAIC_PI_BIN) loaded a planted user mcp.json with every seal flag but --no-extensions");
noPath(r.message, s.paths);
});
test("pi-152: the paired MCP control still refuses unrelated ambient sources", SKIP, async t => {
const s = standIn(t, {
extraFlags: ["--no-mcp"], mcpProjectOverride: true, mcpAlso: "--no-extensions",
leakWith: ["--no-context-files"], leakWithoutAll: ["--no-extensions", "--no-mcp"],
});
const r = await fit(s.env);
assert.equal(r.ok, false);
assert.equal(r.undecided, false);
assert.match(r.message, /loaded a planted .*user skill.*user AGENTS.md.* with every seal flag but --no-extensions and --no-mcp$/);
noPath(r.message, s.paths);
});
// pi-skills: the worker's shape. pi 0.85.1 adds a --skill directory to discovery, so the adapter
// keeps --no-skills in front of each --skill; the fit check's last -p run passes the adapter's -p
// argv with MOSAIC_SKILLS (workerShape) and two --skill directories.
@@ -423,11 +469,11 @@ test("pi-unpin S3 re-check 7: a pi that loads every kind when TERM, TZ or LC_ALL
}
});
// Re-check 5 item 2: the --no-mcp run is the only one that shows --no-mcp alone keeps MCP out.
// This pi keeps MCP out with --no-mcp or with --no-context-files, so every other run hides that
// --no-mcp does nothing on its own.
// Re-check 5 item 2: another seal must not hide whether --no-mcp works. This pi keeps MCP out
// with --no-mcp or --no-context-files. Even the paired control that enables extensions cannot
// show MCP, so the check still cannot tell.
test("pi-unpin S3 re-check 5: a pi whose --no-mcp works only beside another seal flag is can't tell at the --no-mcp run", SKIP, async t => {
assert.equal((await fit(standIn(t, { extraFlags: ["--no-mcp"], mcpAlso: "--no-context-files" }).env)).message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't load a planted MCP server with every seal flag but --no-mcp");
assert.equal((await fit(standIn(t, { extraFlags: ["--no-mcp"], mcpAlso: "--no-context-files" }).env)).message, "pi 2.0.0 (MOSAIC_PI_BIN) can't be checked: it didn't load a planted MCP server with every seal flag but --no-extensions and --no-mcp");
assert.equal((await fit(standIn(t, { extraFlags: ["--no-mcp"] }).env)).ok, true);
});
@@ -1257,7 +1303,9 @@ const SHELL_SET = ["IFS"];
test("pi-unpin S3 re-check 6: every argv the adapter builds in a bus session's environment is one sessionShape builds, over each tool count, -c case, thinking level and provider, and each pair of its branches is one a fit run passes", SKIP, async t => {
const adapter = join(REPO, "adapters", "pi", "adapter.sh");
// Set only by worker containers (packages/processing-control) and the TUI (scripts/agent.sh).
const NOT_BUS = ["MOSAIC_SKILLS", "MOSAIC_SESSION_FORK", "MOSAIC_INTERACTIVE"];
// MOSAIC_PROXY_PORT: only a host worker launch sets it (worker-proxy.mjs), for the model proxy
// forwarder; a bus session reaches its model another way and leaves it unset.
const NOT_BUS = ["MOSAIC_SKILLS", "MOSAIC_SESSION_FORK", "MOSAIC_INTERACTIVE", "MOSAIC_PROXY_PORT"];
for (const name of NOT_BUS) assert.ok(!ENV_ALLOW.includes(name), name);
const lines = readFileSync(adapter, "utf8").split("\n");
const firstSet = new Map();
+12 -6
View File
@@ -37,8 +37,11 @@
// none. `needs`: a flag the loader flags work only with; without it they do
// nothing. `leakWithout`: flags; when any of them is absent, every kind loads.
// `leakWith`: flags; when all of them are given, every kind loads (with
// `leakWithout` too, both must hold). `mcpAlso`: a flag that keeps MCP servers out
// as well as their own flag does. `leakNoAgentDir`: every kind loads when
// `leakWithout` too, both must hold). `leakWithoutAll`: every listed flag must be
// absent, combined with the other leak conditions. `mcpAlso`: a flag that keeps MCP servers out
// as well as their own flag does. `mcpProjectOverride`: like pi 1.5.2, merge the
// user and trusted project's .pi/mcp.json by name, with project entries winning;
// ignore .mcp.json. `leakNoAgentDir`: every kind loads when
// PI_CODING_AGENT_DIR is unset, as in a bus session. `leakNoVar`: names; every
// kind loads when any of them is unset (TERM, TZ, LC_ALL in a unit-started bus).
// `leakNoMatch`: with -c, every
@@ -233,7 +236,7 @@ if (modelRun) {
if (fs.existsSync(marker)) ignoreAll = true;
else { fs.writeFileSync(marker, ""); leak = new Set(ALL); }
}
if ((K.leakWith || K.leakWithout) && (K.leakWith || []).every(has) && (!K.leakWithout || K.leakWithout.some((f) => !has(f)))) leak = new Set(ALL);
if ((K.leakWith || K.leakWithout || K.leakWithoutAll) && (K.leakWith || []).every(has) && (!K.leakWithout || K.leakWithout.some((f) => !has(f))) && (!K.leakWithoutAll || K.leakWithoutAll.every((f) => !has(f)))) leak = new Set(ALL);
if (K.leakNoAgentDir && !process.env.PI_CODING_AGENT_DIR) leak = new Set(ALL);
if (K.leakNoVar && K.leakNoVar.some((n) => process.env[n] === undefined)) leak = new Set(ALL);
if (K.leakNoMatch && has("-c") && !storedFor(val("--session-dir"))) leak = new Set(ALL);
@@ -280,9 +283,12 @@ if (modelRun) {
}
const mcpFlag = (K.extraFlags || []).includes("--no-mcp") ? "--no-mcp" : "--no-extensions";
if (!ignoreAll && (!(sealed(mcpFlag) || (K.mcpAlso && has(K.mcpAlso))) || leak.has("mcp"))) {
for (const f of [path.join(agent, "mcp.json"), path.join(cwd, ".pi", "mcp.json"), path.join(cwd, ".mcp.json")]) {
for (const s of Object.values(json(f)?.mcpServers || {})) cp.spawnSync(s.command, s.args || [], { stdio: "ignore" });
}
const configs = K.mcpProjectOverride
? [path.join(agent, "mcp.json"), ...(trusted ? [path.join(cwd, ".pi", "mcp.json")] : [])]
: [path.join(agent, "mcp.json"), path.join(cwd, ".pi", "mcp.json"), path.join(cwd, ".mcp.json")];
const entries = configs.map((f) => json(f)?.mcpServers || {});
const servers = K.mcpProjectOverride ? Object.values(Object.assign({}, ...entries)) : entries.flatMap(Object.values);
for (const s of servers) cp.spawnSync(s.command, s.args || [], { stdio: "ignore" });
}
if (on("skills", "--no-skills") || (K.skillLeak && has("--skill") && !ignoreAll)) {
const home = process.env.HOME || "/";
+1
View File
@@ -162,6 +162,7 @@ state offers its next step as one button. CLI parity:
mosaic proxy unit render|status|install|start|stop|restart
mosaic proxy fence clear --restart --business ID
mosaic proxy init-lock status|clear
mosaic proxy ride --instance NAME | ride --off (CLI only; no Console control)
- Fetch runs the same PIN and hash checks as `mosaic proxy fetch`; the page
shows each step as it runs.
+10 -2
View File
@@ -82,7 +82,10 @@ const isoTime = value => (typeof value === 'string' && /^[0-9]{4}-[0-9]{2}-[0-9]
// reads it ({document, revision}), read only. `harnesses()` lists the installed harness programs.
// `rootAgents()` reads the root agent file (loadRootAgents); with none given, or one that throws,
// no organization that has the program on is written (unreadable).
export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir = null, unit = null, fetch = fetchPinned, buses, orgs = async () => [], riders = async () => [], harnesses = () => installedHarnesses(dataRoot), rootAgents = null, ownerModels = async () => ({ proxy: false, riding: false, models: [] }), move = null, now = () => new Date(), env = process.env, pin = PIN, providers = PROVIDERS }) {
// worker-model-setup: `workerModel()`, when given, answers the Worker model card's view
// (webui/src/worker-model-view.mjs); status() carries it as `worker`, or null when it throws.
// status({ worker: false }) leaves it out and does not call it (the Triage snapshot).
export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir = null, unit = null, fetch = fetchPinned, buses, orgs = async () => [], riders = async () => [], harnesses = () => installedHarnesses(dataRoot), rootAgents = null, ownerModels = async () => ({ proxy: false, riding: false, models: [] }), move = null, now = () => new Date(), env = process.env, pin = PIN, providers = PROVIDERS, workerModel = null }) {
const port = () => { try { return initState(configDir, dataRoot).settings?.port ?? null; } catch { return null; } };
const control = unit ?? createUnitControl({ instance, unitDir, port, env });
let job = { state: 'idle', step: null, receipt: null, error: null };
@@ -251,7 +254,10 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
// #1873 models-offered (A9): models() and modelsSet(), the owner's per-provider on/off lists.
...offeredControl({ dataRoot, riders, receipt, exclusive, rootAgents }),
unit: control.unit,
async status() {
async status({ worker: withWorker = true } = {}) {
// The Worker model view asks each organization's bus, so it starts first and runs alongside
// the reads below; a throw reads as null.
const workerRead = withWorker && workerModel ? Promise.resolve().then(() => workerModel()).catch(() => null) : null;
// The job is read before the release: a fetch that ends during the awaits below then reads
// "running" (and the page polls again), never "done" next to a release read before it ended.
const fetchJob = { ...job };
@@ -264,6 +270,7 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
try { organizations = await orgs(); } catch { organizations = null; }
const ride = await rideSummary();
const programs = await programsSummary();
const worker = workerRead ? await workerRead : null;
const next = nextSteps({ release, init, unitState, fence, lock });
// Ride all comes after the proxy runs with no fence, while an organization can still ride.
if (next[0] === 'accounts' && ride?.some(r => r.state === 'ready' || r.state === 'fixture')) next.push('ride-all');
@@ -288,6 +295,7 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
ownProxy: { available: false, state: 'not built', reason: 'This instance has one shared proxy. A separate proxy per organization is not built yet.' },
ride,
programs,
...(workerRead ? { worker } : {}),
next,
};
},
+2 -2
View File
@@ -277,9 +277,9 @@ export function unitName(instance) {
return instance && instance !== 'dev' ? `mosaic-proxy-${instance}.service` : 'mosaic-proxy.service';
}
export function defaultRun(command, args, { env, timeout = 60000 } = {}) {
export function defaultRun(command, args, { env, timeout = 60000, signal } = {}) {
return new Promise(resolve => {
execFile(command, args, { env, timeout, maxBuffer: 1024 * 1024, killSignal: 'SIGKILL' }, (error, stdout, stderr) => {
execFile(command, args, { env, timeout, signal, maxBuffer: 1024 * 1024, killSignal: 'SIGKILL' }, (error, stdout, stderr) => {
resolve({ code: error ? (typeof error.code === 'number' ? error.code : 1) : 0, stdout: String(stdout ?? ''), stderr: String(stderr ?? '') });
});
});
+14 -8
View File
@@ -157,20 +157,25 @@ export function offeredControl({ dataRoot, riders, receipt, exclusive, rootAgent
try { list = await riders(); } catch { throw failure('outcome-unknown'); }
const named = (Array.isArray(list) ? list : []).filter(r => typeof r?.id === 'string' && ORG_ID.test(r.id));
const riding = named.filter(r => r.cliproxy === true);
if (!riding.length) throw failure('not-configured');
const unanswered = new Set(named.filter(r => typeof r.cliproxy !== 'boolean').map(r => r.id));
if (!riding.length) throw failure(unanswered.size ? 'proxy-down' : 'not-configured');
let root;
try { root = typeof rootAgents === 'function' ? await rootAgents() : undefined; } catch { root = undefined; }
const answers = await Promise.all(riding.map(async r => {
let routes = null, named = null;
try { routes = await r.call('routes'); } catch { routes = null; }
try { named = typeof r.file === 'function' ? modelsNamed((await r.file())?.document) : null; } catch { named = null; }
let routes = null, named = null, noLaunchReply = false, document = null;
try { routes = await r.call('routes'); } catch (error) { noLaunchReply = error?.code === 'launch-not-configured'; }
try { document = typeof r.file === 'function' ? (await r.file())?.document : null; named = modelsNamed(document); } catch { named = null; }
// A separate file read can observe a newer launch than the bus did. Treat that race and
// unreadable files as unknown, never as proof that no model is in use.
const noLaunch = noLaunchReply && named !== null && !Object.hasOwn(document, 'launch');
const fromRoot = root === undefined ? null : rootModelsNamed(root, r.id);
return { id: r.id, routes, named: named && fromRoot ? [...named, ...fromRoot] : null };
return { id: r.id, routes, noLaunch, named: named && fromRoot ? [...named, ...fromRoot] : null };
}));
const providers = new Map(), usedBy = new Map();
let ready = false;
for (const { id, routes } of answers) {
for (const { id, routes, noLaunch } of answers) {
// routes explicitly refuses before reading a catalogue when the organization has no launch.
if (noLaunch) continue;
const c = routes?.catalogue;
if (c?.status !== 'ready' || !Array.isArray(c.models)) { unanswered.add(id); continue; }
ready = true;
@@ -181,8 +186,9 @@ export function offeredControl({ dataRoot, riders, receipt, exclusive, rootAgent
if (!providers.get(provider).includes(m.model)) providers.get(provider).push(m.model);
}
}
if (!ready) throw failure('proxy-down');
for (const { id, routes, named } of answers) {
if (!ready) throw failure(!unanswered.size && answers.every(a => a.noLaunch) ? 'not-configured' : 'proxy-down');
for (const { id, routes, named, noLaunch } of answers) {
if (noLaunch) continue;
if (!routes || typeof routes.instances !== 'object' || routes.instances === null || named === null) { unanswered.add(id); continue; }
const table = routes.providerModels && typeof routes.providerModels === 'object' ? routes.providerModels : {};
const legacy = routes.catalogue?.legacy && typeof routes.catalogue.legacy === 'object' ? routes.catalogue.legacy : {};
+5 -5
View File
@@ -3,8 +3,8 @@
//
// It is the redaction chat-fail landed for a failed turn's cause (packages/harness/src/failure-notice.mjs,
// its REDACT list), with Mosaic's own key prefixes added to the key-prefix line: mso- (an organization's
// model key), mst- (the Triage key) and msp-, beside msk- (the owner's). One line is added after it:
// those four Mosaic prefixes with no word boundary, so a key glued to a word (xmso-...) is caught
// model key), mst- (the Triage key), msw- (the worker key) and msp-, beside msk- (the owner's). One line is
// added after it: those five Mosaic prefixes with no word boundary, so a key glued to a word (xmso-...) is caught
// too. Over-redaction there is the safe side. failure-notice.mjs does not
// export its list, and packages/harness/ belongs to another lane now, so the list is copied here.
// tests/redact.test.mjs fails when a copied line no longer matches failure-notice.mjs.
@@ -20,7 +20,7 @@ export { secretValues } from '../../harness/src/failure-notice.mjs';
export const LINE_MAX = 300;
export const REDACTED = '[redacted]';
// Copied from failure-notice.mjs. The key-prefix line adds mso, mst and msp, and the line after
// Copied from failure-notice.mjs. The key-prefix line adds mso, mst, msp and msw, and the line after
// it is Mosaic's own; every other line is failure-notice's.
const KEY_WORD = "(?:\\w*[_-])?(?:api[_-]?key|key|token|secret|password|passwd|authorization|cookie|session)";
const REDACT = [
@@ -33,8 +33,8 @@ const REDACT = [
/\b(?:bearer|basic)\s+\S+/gi,
new RegExp(`\\b${KEY_WORD}["']?\\s*[:=]\\s*(?:"[^"]*"?|'[^']*'?|[^\\s,;}]+)`, "gi"),
new RegExp(`\\b${KEY_WORD}["']?\\s+(?:"[^"]*"?|'[^']*'?|(?=[^\\s"',;}]*\\d)[^\\s"',;}]+|[^\\s"',;}]{8,})`, "gi"),
/\b(?:sk|msk|mso|mst|msp|rk|pk|ghp|gho|ghs|glpat|xox[a-z])[-_]\S+/gi,
/ms[kotp][-_]\S+/gi, // Mosaic's prefixes glued to a word: added here, not in failure-notice.mjs
/\b(?:sk|msk|mso|mst|msp|msw|rk|pk|ghp|gho|ghs|glpat|xox[a-z])[-_]\S+/gi,
/ms[kotpw][-_]\S+/gi, // Mosaic's prefixes glued to a word: added here, not in failure-notice.mjs
/\beyJ\S+/g, // JWT
/[A-Za-z0-9_\-+/=.]{32,}/g,
/(?<![A-Za-z0-9_\-+/=.])(?=[A-Za-z0-9_\-+/=.]*\d)(?=[A-Za-z0-9_\-+/=.]*[A-Za-z])[A-Za-z0-9_\-+/=.]{16,}/g,
+2
View File
@@ -8,6 +8,8 @@ try {
const result = spawnSync(process.execPath, [path.join(repo, 'scripts/mosaic-config.mjs'), 'validate'], { encoding: 'utf8', env: process.env });
if (result.status !== 0) throw Error('proxy-config-invalid');
const config = JSON.parse(result.stdout);
// proxy-ride: a riding instance runs no proxy of its own; the owner's unit serves it.
if (config.proxy?.owner) delete config.proxy;
const child = startVerifiedProxy(config, {
onError: code => { process.stderr.write(code + '\n'); process.exitCode = 3; },
onExit: code => { process.exitCode = code; },
+55
View File
@@ -34,6 +34,59 @@ export function loadProxy(configDir, dataRoot) {
refuse('proxy-config-invalid');
}
}
// proxy-ride first slice (Jason 2026-11-21): <configDir>/proxy-ride.json names another instance
// of the same OS user whose model proxy this instance's worker runs use. It names an instance,
// never a path. scripts/mosaic-config.mjs resolves the name the way --instance does.
export const RIDE_FILE = 'proxy-ride.json';
export const RIDE_NAME = /^[a-z][a-z0-9]{1,15}$/;
// One owner line per refusal. packages/bus/src/refusals.mjs carries the same text.
export const RIDE_LINES = Object.freeze({
'proxy-ride-invalid': 'proxy-ride.json is not a valid ride file, or it names this instance. Run mosaic proxy ride --off, then mosaic proxy ride --instance NAME with the instance whose proxy this one should use.',
'proxy-ride-owner-missing': 'The instance named in proxy-ride.json has no valid model proxy, or it rides another instance itself. Set up the proxy on that instance, or run mosaic proxy ride --off.',
'proxy-ride-conflict': 'This instance has its own proxy.json and a proxy-ride.json. Run mosaic proxy ride --off to keep its own proxy.',
});
// undefined when the file is absent. Present beside proxy.json refuses proxy-ride-conflict, before
// the file is read. Otherwise it must be a regular 0600 file of the running user holding exactly
// {"version":1,"instance":NAME}; anything else refuses proxy-ride-invalid.
export function readRide(configDir) {
const file = path.join(configDir, RIDE_FILE);
let stat;
try { stat = fs.lstatSync(file); } catch (e) { if (e.code === 'ENOENT') return undefined; refuse('proxy-ride-invalid'); }
let own = true;
try { fs.lstatSync(path.join(configDir, 'proxy.json')); } catch (e) { if (e.code === 'ENOENT') own = false; }
if (own) refuse('proxy-ride-conflict');
if (!stat.isFile() || stat.isSymbolicLink() || stat.uid !== process.getuid() || (stat.mode & 0o777) !== 0o600 || stat.size > 4096) refuse('proxy-ride-invalid');
let value;
try { value = JSON.parse(regularBytes(file, 'proxy-ride-invalid')); } catch { refuse('proxy-ride-invalid'); }
if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).sort().join(',') !== 'instance,version'
|| value.version !== 1 || typeof value.instance !== 'string' || !RIDE_NAME.test(value.instance)) refuse('proxy-ride-invalid');
return { instance: value.instance };
}
// The verb's write: exclusive creation, 0600, so an existing file is never rewritten. Returns a
// function that removes the file only while it is still the one this call created.
export function createRide(configDir, instance) {
if (typeof instance !== 'string' || !RIDE_NAME.test(instance)) refuse('proxy-ride-invalid');
safePath(configDir, 'proxy-ride-invalid');
const file = path.join(configDir, RIDE_FILE);
let fd;
try { fd = fs.openSync(file, 'wx', 0o600); } catch { refuse('proxy-ride-invalid'); }
try { fs.writeFileSync(fd, JSON.stringify({ version: 1, instance }) + '\n'); }
finally { fs.closeSync(fd); }
const { ino, dev } = fs.lstatSync(file);
return () => {
try { const now = fs.lstatSync(file); if (now.ino === ino && now.dev === dev) fs.unlinkSync(file); } catch { /* already gone */ }
};
}
// The verb's --off: removes the ride file, never following a link. False when there was none.
export function removeRide(configDir) {
safePath(configDir, 'proxy-ride-invalid');
const file = path.join(configDir, RIDE_FILE);
let stat;
try { stat = fs.lstatSync(file); } catch (e) { if (e.code === 'ENOENT') return false; refuse('proxy-ride-invalid'); }
if (stat.isDirectory()) refuse('proxy-ride-invalid');
fs.unlinkSync(file);
return true;
}
export function verifyInstall(dataRoot, pin = PIN) {
const paths = installPaths(dataRoot, pin);
for (const [file, hash] of [[paths.binary, pin.binary.sha256], [paths.panel, pin.panel.sha256]]) {
@@ -74,6 +127,8 @@ export async function initializeProxy({ configDir, dataRoot, key, port, pin = PI
for (const file of [settingsFile, proxyDir, keyFile]) {
try { fs.lstatSync(file); refuse('proxy-already-initialized'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
}
// A riding instance uses another instance's proxy; its own init would make both files.
try { fs.lstatSync(path.join(configDir, RIDE_FILE)); refuse('proxy-ride-conflict'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
const lockFile = path.join(configDir, '.proxy-init.lock');
let lock;
try { lock = fs.openSync(lockFile, 'wx', 0o600); } catch { refuse('proxy-init-busy'); }
+67
View File
@@ -0,0 +1,67 @@
// Readiness only. Never calls offeredControl.models(), which seeds the offered-model file.
// The management client remains bus-only; account metadata comes from its existing owner read.
import { releaseState, initState, fenceState, createUnitControl, defaultRun } from './admin.mjs';
import { loadProxy } from './settings.mjs';
import { offeredProvider } from '../../plugins/model-provider/cliproxy/index.mjs';
export async function readService({ configDir, dataRoot, instance, unitDir = null, signal, run = defaultRun }) {
const release = releaseState(dataRoot), init = initState(configDir, dataRoot);
const fence = fenceState(dataRoot);
// An untouched instance needs no systemd call. A partial install still reports a failure.
if (!release.fetched && !init.initialized && !init.partial) return { release, init, unit: {}, fence: { set: !!fence } };
const unit = await createUnitControl({ instance, unitDir,
port: () => loadProxy(configDir, dataRoot)?.port ?? null,
run: (command, args, options) => run(command, args, { ...options, timeout: 1000, signal }),
}).status();
return { release, init, unit, fence: { set: !!fence } };
}
// The host can supply the instance account reader without requiring an organization.
export async function accountState(ctx) {
const read = ctx.accounts ? [ctx.accounts] : ctx.members.map(org => async () => {
const routes = await ctx.read(org, 'routes');
if (['unavailable', 'disabled'].includes(routes?.catalogue?.status)) return { catalogue: routes.catalogue };
const accounts = await ctx.read(org, 'accounts');
return { ...accounts, catalogue: routes?.catalogue };
});
if (!read.length) return { state: 'next', reason: 'Connect an account in Model setup so your agents can answer.' };
const answers = await Promise.allSettled(read.map(async reader => {
const a = await reader();
if (['unavailable', 'disabled'].includes(a?.catalogue?.status))
return { state: 'next', reason: 'Open Model setup to connect an account and load the models it offers.' };
if (!Array.isArray(a?.accounts)) throw Error('accounts unavailable');
const active = a.accounts.some(account => ['active', 'ready'].includes(account.status));
if (!a.accounts.length) return { state: 'next', reason: 'Connect an account in Model setup so your agents can answer.' };
if (!active) return { state: 'failed', reason: 'Your connected accounts are not ready. Open Model setup to sign in or add an account.' };
if (a.catalogue?.status !== 'ready' || !Array.isArray(a.catalogue.models)) throw Error('catalogue unavailable');
if (a.catalogue.models.some(model => model.available === true && typeof model.upstream === 'string' && a.accounts.some(account =>
['active', 'ready'].includes(account.status) && typeof account.provider === 'string' && account.provider === offeredProvider(model.upstream)))) return { state: 'done', reason: 'An account is connected and offers a model.' };
return { state: 'next', reason: 'The account is connected but has no model available. Choose a model in Model setup.' };
}));
const found = answers.filter(a => a.status === 'fulfilled').map(a => a.value);
return found.find(a => a.state === 'done') ?? found.find(a => a.state === 'failed') ?? found[0] ?? Promise.reject(Error('account read unavailable'));
}
export const setupState = Object.freeze([
{
id: 'model-service.running', title: 'Start the model service',
purpose: "The program on this computer that carries your agents' questions to the models you connect.",
fix: { page: '#/settings/model-setup', step: 1, label: 'Set up model service' }, order: { after: [] }, required: true,
async check(ctx) {
const s = await ctx.service();
if (!s || !s.release || !s.init || !s.unit) throw Error('service state unavailable');
if (!s.release.fetched && s.release.problem && s.release.problem !== 'proxy-install-missing') throw Error('release state unavailable');
if (s.release.verified && s.init.initialized && s.unit.active === 'active' && s.unit.healthy === true && s.unit.listening === true && !s.fence?.set)
return { state: 'done', reason: 'The model service is running and answers on this computer.' };
if (s.release.fetched || s.release.verified || s.init.initialized || s.init.partial || s.unit.active === 'failed' || s.fence?.set)
return { state: 'failed', reason: 'The model service is not ready. Open Model setup to check it and continue.' };
return { state: 'next', reason: "Mosaic downloads the model service, checks the download and starts it on this computer. You don't need an account for this step." };
},
},
{
id: 'model-account.connected', title: 'Connect a model account',
purpose: "Sign in to an account or add a model key in Model setup. This account pays for your agents' answers.",
fix: { page: '#/settings/model-setup', step: 2, label: 'Connect an account' },
order: { after: ['model-service.running'] }, required: true, check: accountState,
},
]);
@@ -628,3 +628,36 @@ test('accountImpact reads the accounts through the bus and each cliproxy organiz
const { admin: lost } = fixture(t, { extra: { riders: async () => { throw new Error('gone'); } } });
assert.equal((await lost.accountImpact(call, { accountId: 'a1b2c3d4e5f6' })).organizations, null);
});
// worker-model-setup: status carries the Worker model card's view only when one is given, and a
// view that throws reads as null, never as the error.
test('status carries worker only when a workerModel is given, and null when it throws', async t => {
assert.equal(Object.hasOwn(await fixture(t).admin.status(), 'worker'), false);
let calls = 0;
const view = { harness: 'pi', using: null, models: [] };
assert.deepEqual((await fixture(t, { extra: { workerModel: async () => { calls++; return view; } } }).admin.status()).worker, view);
assert.equal(calls, 1);
const thrown = await fixture(t, { extra: { workerModel: async () => { throw new Error(`${SENTINEL} at /secret`); } } }).admin.status();
assert.equal(thrown.worker, null);
assert.equal(JSON.stringify(thrown).includes(SENTINEL), false);
// A callback that throws before it returns a promise reads as null too.
assert.equal((await fixture(t, { extra: { workerModel: () => { throw new Error(SENTINEL); } } }).admin.status()).worker, null);
// status({ worker: false }) (the Triage snapshot) neither calls it nor carries the field.
const skipped = await fixture(t, { extra: { workerModel: async () => { calls++; return view; } } }).admin.status({ worker: false });
assert.equal(Object.hasOwn(skipped, 'worker'), false);
assert.equal(calls, 1);
});
test('the worker view starts before the ride read and runs alongside it, so a slow bus is not paid twice', async t => {
const order = [];
let release;
const gate = new Promise(r => { release = r; });
const admin = fixture(t, { extra: {
workerModel: async () => { order.push('worker-start'); await gate; order.push('worker-end'); return { harness: 'pi' }; },
riders: async () => { order.push('riders'); release(); return []; },
} }).admin;
const out = await admin.status();
assert.deepEqual(out.worker, { harness: 'pi' });
assert.equal(order[0], 'worker-start');
assert.ok(order.indexOf('riders') < order.indexOf('worker-end'), order.join(' '));
});
@@ -114,6 +114,38 @@ test('an organization whose usage is unknown (its file unreadable, its bus down,
}
});
test('a riding organization explicitly without launch is answered, while a bus failure remains unanswered', async t => {
const noLaunch = { ...rider('bolt', []), call: async () => { throw Object.assign(new Error('no launch'), { code: 'launch-not-configured' }); } };
const down = { ...rider('crux', []), call: async () => { throw new Error('down'); } };
const c = control(t, { riders: [rider('acme', [entry('zai', 'glm-5')]), noLaunch, down] });
assert.deepEqual((await c.models({ seed: true })).unanswered, ['crux']);
const onlyAnswered = control(t, { riders: [rider('acme', [entry('zai', 'glm-5')]), noLaunch] });
const view = await onlyAnswered.models({ seed: true });
assert.deepEqual(view.unanswered, []);
assert.equal(existsSync(offeredFile(onlyAnswered.dataRoot)), true);
await onlyAnswered.modelsSet({ provider: 'z.ai', on: [], revision: view.revision });
});
test('a no-launch response with a newer launch or unreadable file stays unanswered and blocks an unconfirmed switch off', async t => {
const noLaunch = async () => { throw Object.assign(new Error('no launch'), { code: 'launch-not-configured' }); };
for (const file of [async () => ({ document: { launch: { models: {} }, roles: {} } }), async () => { throw new Error('unreadable'); }, async () => ({ document: null })]) {
const c = control(t, { riders: [rider('acme', [entry('zai', 'glm-5')]), { ...rider('bolt', []), call: noLaunch, file }] });
const view = await c.models({ seed: true });
assert.deepEqual(view.unanswered, ['bolt']);
assert.equal(existsSync(offeredFile(c.dataRoot)), false);
await assert.rejects(c.modelsSet({ provider: 'z.ai', on: [], revision: view.revision }), { code: 'models-in-use' });
}
});
test('all riders confirmed without launch are not configured; a failed read is proxy-down', async t => {
const noLaunch = { ...rider('bolt', []), call: async () => { throw Object.assign(new Error('no launch'), { code: 'launch-not-configured' }); } };
const configured = control(t, { riders: [noLaunch] });
await assert.rejects(configured.models({ seed: true }), { code: 'not-configured' });
assert.equal(existsSync(offeredFile(configured.dataRoot)), false);
const down = control(t, { riders: [{ ...noLaunch, file: async () => { throw new Error('unreadable'); } }] });
await assert.rejects(down.models({ seed: true }), { code: 'proxy-down' });
});
test('in use counts every place the organization file names a model and the root agents it takes: project agents, root copies; not an excluded or assistant root agent, not launch.models itself', async t => {
const models = [entry('zai', 'glm-5'), entry('zai', 'glm-4.6'), entry('codex', 'gpt-5'), entry('codex', 'gpt-5-mini')];
const routes = { instances: { chief: { model: 'g5' } }, providerModels: { g5: entry('zai', 'glm-5'), g46: entry('zai', 'glm-4.6'), gpt: entry('codex', 'gpt-5'), mini: entry('codex', 'gpt-5-mini') } };
@@ -166,6 +198,9 @@ test(`a model kept on while its account is signed out counts against the ${MAX_O
// glm-0 is in use and this switches it off, but the list is too long first: no ask for a save that cannot be written.
await assert.rejects(c.modelsSet({ provider: 'z.ai', on: all.slice(1), revision: before.revision }), { code: 'models-invalid' }, `${all.length - 1} on plus 10 kept is past ${MAX_ON}`);
await assert.rejects(c.modelsSet({ provider: 'z.ai', on: all.slice(1), revision: before.revision, confirmInUse: true }), { code: 'models-invalid' });
// Exactly 257 including the 10 kept models must refuse before asking about glm-0.
assert.equal(all.slice(6).length + away.length, MAX_ON + 1);
await assert.rejects(c.modelsSet({ provider: 'z.ai', on: all.slice(6), revision: before.revision }), { code: 'models-invalid' });
assert.equal(readOffered(c.dataRoot).revision, before.revision);
assert.deepEqual(c.receipts, []);
const saved = await c.modelsSet({ provider: 'z.ai', on: all.slice(7), revision: before.revision, confirmInUse: true });
@@ -228,6 +263,7 @@ test('a damaged file: the read and the write answer outcome-unknown and write no
test('no organization riding is not-configured; none answering a ready catalogue is proxy-down; a busy lock refuses the write', async t => {
await assert.rejects(control(t, { riders: [] }).models(), { code: 'not-configured' });
await assert.rejects(control(t, { riders: [{ ...rider('acme', []), cliproxy: false }] }).models(), { code: 'not-configured' });
await assert.rejects(control(t, { riders: [{ ...rider('acme', []), cliproxy: undefined }] }).models(), { code: 'proxy-down' });
const down = { id: 'acme', cliproxy: true, call: async () => { throw new Error('down'); } };
await assert.rejects(control(t, { riders: [down, rider('bolt', [], { catalogue: { status: 'unavailable', models: [] } })] }).models(), { code: 'proxy-down' });
await assert.rejects(control(t, { riders: [rider('acme', [entry('zai', 'glm-5')])], busy: true }).modelsSet({ provider: 'z.ai', on: [], revision: 'a'.repeat(16) }), { code: 'proxy-admin-busy' });
@@ -0,0 +1,139 @@
// proxy-ride first slice (Jason 2026-11-21): the real config loader with two scratch instances of
// one made-up HOME. The owner is named instance "owner" with its own proxy.json; the rider is the
// default instance (or a second named one) with proxy-ride.json. Nothing reads a real config.
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { spawnSync } from 'node:child_process';
import { installPaths, RIDE_LINES } from '../src/settings.mjs';
const repo = fileURLToPath(new URL('../../../', import.meta.url));
const MANAGE = 'ride-config-management-key';
function instance(home, name, dataRoot) {
const configDir = join(home, '.config', `mosaic-${name}`), file = join(configDir, 'config.json');
mkdirSync(configDir, { recursive: true, mode: 0o700 }); mkdirSync(dataRoot, { recursive: true });
writeFileSync(file, JSON.stringify({ configVersion: 1, environment: 'development', dataRoot, execution: { backend: 'docker', provider: 'fixture', model: 'fixture' } }));
if (name !== 'dev') writeFileSync(join(dataRoot, 'instance.json'), JSON.stringify({ instanceVersion: 1, name }));
return { name, configDir, file, dataRoot };
}
function proxy(i, port = 18432) {
const v = { port, binary: installPaths(i.dataRoot).binary };
writeFileSync(join(i.configDir, 'proxy.json'), JSON.stringify(v));
mkdirSync(join(i.configDir, 'secrets'), { mode: 0o700 });
writeFileSync(join(i.configDir, 'secrets', 'proxy-management.key'), MANAGE, { mode: 0o600 });
return v;
}
const ride = (i, name) => writeFileSync(join(i.configDir, 'proxy-ride.json'), JSON.stringify({ version: 1, instance: name }) + '\n', { mode: 0o600 });
function fixture() {
const home = realpathSync(mkdtempSync(join(tmpdir(), 'rc-')));
const owner = instance(home, 'owner', join(home, 'od'));
const rider = instance(home, 'dev', join(home, 'rd'));
return { home, owner, rider, value: proxy(owner) };
}
function run(f, script, args = [], config = null) {
const env = { PATH: process.env.PATH, HOME: f.home, ...(config ? { MOSAIC_CONFIG: config } : {}) };
return spawnSync(script.endsWith('.mjs') ? process.execPath : 'bash', [join(repo, script), ...args], { env, encoding: 'utf8', timeout: 10000 });
}
const validate = (f, args = []) => run(f, 'scripts/mosaic-config.mjs', ['validate', ...args]);
function refuses(f, r, code) {
assert.equal(r.status, 2, r.stderr);
assert.equal(r.stderr.trim().split('\n').at(-1), `mosaic-config: ${code}: ${RIDE_LINES[code]}`);
assert.equal(r.stderr.includes(f.home), false, 'no path in the refusal');
assert.equal(r.stdout, '');
}
test('a rider resolves the owner proxy plus owner; its own data root and unit flag are unchanged', () => {
const f = fixture(); ride(f.rider, 'owner');
const before = readFileSync(f.rider.file);
let r = validate(f);
assert.equal(r.status, 0, r.stderr);
const out = JSON.parse(r.stdout);
assert.equal(out.dataRoot, f.rider.dataRoot);
assert.deepEqual(out.proxy, { ...f.value, owner: { instance: 'owner', configDir: f.owner.configDir, dataRoot: f.owner.dataRoot } });
assert.equal(r.stdout.includes(MANAGE), false, 'the management key never enters the output');
assert.deepEqual(readFileSync(f.rider.file), before);
r = run(f, 'scripts/mosaic-config.mjs', ['instance']);
assert.equal(r.status, 0, r.stderr);
assert.doesNotMatch(r.stdout, /MOSAIC_INSTANCE_PROXY_CONFIGURED/);
r = run(f, 'scripts/mosaic-config.mjs', ['instance', '--instance', 'owner']);
assert.match(r.stdout, /MOSAIC_INSTANCE_PROXY_CONFIGURED=1/, 'the owner keeps its flag');
r = run(f, 'scripts/bus-service.sh', ['render']);
assert.equal(r.status, 0, r.stderr);
assert.doesNotMatch(r.stdout, /Wants=mosaic-proxy/, "the rider's bus unit does not wait for a proxy unit");
for (const [script, args] of [['scripts/proxy-service.sh', ['render']], ['packages/model-proxy/src/run.mjs', []]]) {
r = run(f, script, args);
assert.equal(r.status, 3, `${script}: ${r.stderr}`);
assert.match(r.stderr, /proxy-not-configured/, 'a rider runs no proxy of its own');
}
});
test('a named instance rides the same way, selected with --instance', () => {
const f = fixture(); const second = instance(f.home, 'second', join(f.home, 'sd')); ride(second, 'owner');
const r = validate(f, ['--instance', 'second']);
assert.equal(r.status, 0, r.stderr);
assert.equal(JSON.parse(r.stdout).proxy.owner.instance, 'owner');
});
test('proxy-ride-invalid: a ride file naming its own instance, by name or by its config directory', () => {
let f = fixture(); ride(f.rider, 'dev'); refuses(f, validate(f), 'proxy-ride-invalid');
f = fixture(); ride(f.owner, 'owner'); rmSync(join(f.owner.configDir, 'proxy.json'));
refuses(f, validate(f, ['--instance', 'owner']), 'proxy-ride-invalid');
// MOSAIC_CONFIG names the owner's file by another path: the same directory still refuses.
f = fixture(); const alias = join(f.home, 'alias'); symlinkSync(f.rider.configDir, alias); ride(f.rider, 'dev');
refuses(f, run(f, 'scripts/mosaic-config.mjs', ['validate'], join(alias, 'config.json')), 'proxy-ride-invalid');
});
test('proxy-ride-invalid: an owner whose data root is the rider\'s', () => {
const f = fixture(); const twin = instance(f.home, 'twin', f.rider.dataRoot); proxy(twin, 18433);
ride(f.rider, 'twin');
// validate checks no claim of the rider's own, so the shared root is what refuses.
refuses(f, validate(f), 'proxy-ride-invalid');
});
test('proxy-ride-invalid: a malformed ride file, or one naming a path', () => {
for (const text of ['{', '{"version":1,"instance":"../mosaic-owner"}', '{"version":1,"instance":"owner","configDir":"/x"}']) {
const f = fixture(); writeFileSync(join(f.rider.configDir, 'proxy-ride.json'), text, { mode: 0o600 });
refuses(f, validate(f), 'proxy-ride-invalid');
}
});
test('proxy-ride-conflict: an instance with its own proxy.json and a ride file', () => {
const f = fixture(); proxy(f.rider, 18440); ride(f.rider, 'owner');
refuses(f, validate(f), 'proxy-ride-conflict');
refuses(f, run(f, 'scripts/mosaic-config.mjs', ['instance']), 'proxy-ride-conflict');
});
test('proxy-ride-owner-missing: no such instance, no proxy, a broken proxy, an unclaimed root, or a rider of a rider', () => {
const cases = {
absent: (f) => ride(f.rider, 'nobody'),
noProxy: (f) => { rmSync(join(f.owner.configDir, 'proxy.json')); ride(f.rider, 'owner'); },
badProxy: (f) => { writeFileSync(join(f.owner.configDir, 'proxy.json'), '{'); ride(f.rider, 'owner'); },
badConfig: (f) => { writeFileSync(f.owner.file, '{'); ride(f.rider, 'owner'); },
unclaimed: (f) => { rmSync(join(f.owner.dataRoot, 'instance.json')); ride(f.rider, 'owner'); },
riderOfRider: (f) => {
const middle = instance(f.home, 'middle', join(f.home, 'md')); ride(middle, 'owner'); ride(f.rider, 'middle');
assert.equal(validate(f, ['--instance', 'middle']).status, 0, 'the middle instance itself rides');
},
};
for (const [label, prep] of Object.entries(cases)) {
const f = fixture(); prep(f);
const r = validate(f);
assert.equal(r.status, 2, `${label}: ${r.stderr}`);
refuses(f, r, 'proxy-ride-owner-missing');
}
});
test('neither file still has no proxy; an instance with its own proxy.json is exactly as before', () => {
const f = fixture();
let r = validate(f);
assert.equal(r.status, 0, r.stderr);
assert.equal(Object.hasOwn(JSON.parse(r.stdout), 'proxy'), false);
r = run(f, 'packages/model-proxy/src/run.mjs');
assert.equal(r.status, 3); assert.match(r.stderr, /proxy-not-configured/);
r = validate(f, ['--instance', 'owner']);
assert.equal(r.status, 0, r.stderr);
assert.deepEqual(JSON.parse(r.stdout).proxy, f.value);
});
@@ -0,0 +1,96 @@
// proxy-ride first slice (Jason 2026-11-21): <configDir>/proxy-ride.json, its strict reader, the
// verb's exclusive write and --off removal, and proxy init refusing beside a ride file. Scratch
// directories only.
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, lstatSync, symlinkSync, chmodSync, existsSync, renameSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { RIDE_FILE, RIDE_LINES, readRide, createRide, removeRide, initializeProxy, installPaths } from '../src/settings.mjs';
import { REFUSALS } from '../../bus/src/refusals.mjs';
const digest = (x) => createHash('sha256').update(x).digest('hex');
const pin = { version: '8.0.42', binary: { sha256: digest('fake binary') }, panel: { sha256: digest('fake panel') } };
const code = (c) => (e) => e.code === c;
function dir() { const root = mkdtempSync(join(tmpdir(), 'rf-')); const configDir = join(root, 'config'), dataRoot = join(root, 'data'); mkdirSync(configDir, { mode: 0o700 }); mkdirSync(dataRoot); return { root, configDir, dataRoot, file: join(configDir, RIDE_FILE) }; }
const put = (f, text, mode = 0o600) => { writeFileSync(f.file, text, { mode }); chmodSync(f.file, mode); };
test('readRide: absent is undefined; a strict file names its instance', () => {
const f = dir();
assert.equal(readRide(f.configDir), undefined);
put(f, '{"version":1,"instance":"uiorg"}\n');
assert.deepEqual(readRide(f.configDir), { instance: 'uiorg' });
});
for (const [label, text] of Object.entries({
malformed: '{', array: '[]', null: 'null', extra: '{"version":1,"instance":"uiorg","path":"/x"}',
missing: '{"version":1}', version: '{"version":2,"instance":"uiorg"}', stringVersion: '{"version":"1","instance":"uiorg"}',
path: '{"version":1,"instance":"../mosaic-uiorg"}', absolute: '{"version":1,"instance":"/home/x/.config/mosaic-uiorg"}',
upper: '{"version":1,"instance":"UIorg"}', short: '{"version":1,"instance":"u"}', long: `{"version":1,"instance":"${'a'.repeat(17)}"}`,
number: '{"version":1,"instance":7}', big: `{"version":1,"instance":"uiorg","pad":"${'x'.repeat(5000)}"}`,
})) test(`proxy-ride-invalid: ${label}`, () => { const f = dir(); put(f, text); assert.throws(() => readRide(f.configDir), code('proxy-ride-invalid')); });
test('proxy-ride-invalid: wrong mode, symlink, directory', () => {
let f = dir(); put(f, '{"version":1,"instance":"uiorg"}', 0o644);
assert.throws(() => readRide(f.configDir), code('proxy-ride-invalid'));
f = dir(); const real = join(f.root, 'real.json'); writeFileSync(real, '{"version":1,"instance":"uiorg"}', { mode: 0o600 }); symlinkSync(real, f.file);
assert.throws(() => readRide(f.configDir), code('proxy-ride-invalid'));
f = dir(); mkdirSync(f.file);
assert.throws(() => readRide(f.configDir), code('proxy-ride-invalid'));
});
test('proxy-ride-conflict: a ride file beside proxy.json refuses before the ride file is read', () => {
const f = dir(); put(f, 'not json at all', 0o644); writeFileSync(join(f.configDir, 'proxy.json'), '{}');
assert.throws(() => readRide(f.configDir), code('proxy-ride-conflict'));
});
test('createRide writes 0600 exclusively, never rewrites, and its undo removes only its own file', () => {
const f = dir();
const undo = createRide(f.configDir, 'uiorg');
assert.equal(readFileSync(f.file, 'utf8'), '{"version":1,"instance":"uiorg"}\n');
assert.equal(lstatSync(f.file).mode & 0o777, 0o600);
assert.deepEqual(readRide(f.configDir), { instance: 'uiorg' });
assert.throws(() => createRide(f.configDir, 'other'), code('proxy-ride-invalid'));
assert.equal(readFileSync(f.file, 'utf8'), '{"version":1,"instance":"uiorg"}\n');
undo();
assert.equal(existsSync(f.file), false);
const again = createRide(f.configDir, 'uiorg');
// Moved aside, not unlinked, so the new file cannot reuse its inode number.
renameSync(f.file, join(f.root, 'aside.json')); put(f, '{"version":1,"instance":"later"}\n');
again();
assert.equal(readFileSync(f.file, 'utf8'), '{"version":1,"instance":"later"}\n', 'undo leaves a file it did not create');
for (const name of ['../x', 'UI', '', 'a', 'a/b']) assert.throws(() => createRide(dir().configDir, name), code('proxy-ride-invalid'));
});
test('removeRide removes the file, never a link target, and says when there was none', () => {
let f = dir();
assert.equal(removeRide(f.configDir), false);
createRide(f.configDir, 'uiorg');
assert.equal(removeRide(f.configDir), true);
assert.equal(existsSync(f.file), false);
f = dir(); const target = join(f.root, 'target.json'); writeFileSync(target, 'keep'); symlinkSync(target, f.file);
assert.equal(removeRide(f.configDir), true);
assert.equal(readFileSync(target, 'utf8'), 'keep');
f = dir(); mkdirSync(f.file);
assert.throws(() => removeRide(f.configDir), code('proxy-ride-invalid'));
});
test('proxy-ride-conflict: proxy init refuses beside a ride file and writes nothing', async () => {
const f = dir(); const p = installPaths(f.dataRoot, pin);
mkdirSync(join(p.dir, 'static'), { recursive: true }); writeFileSync(p.binary, 'fake binary', { mode: 0o755 }); writeFileSync(p.panel, 'fake panel');
createRide(f.configDir, 'uiorg');
await assert.rejects(initializeProxy({ configDir: f.configDir, dataRoot: f.dataRoot, pin, key: 'ride-management' }), code('proxy-ride-conflict'));
for (const name of ['proxy.json', 'secrets', '.proxy-init.lock']) assert.equal(existsSync(join(f.configDir, name)), false);
assert.equal(existsSync(join(f.dataRoot, 'proxy')), false);
});
test('each ride code has one plain owner line, and the bus refusal table carries the same line', () => {
assert.deepEqual(Object.keys(RIDE_LINES), ['proxy-ride-invalid', 'proxy-ride-owner-missing', 'proxy-ride-conflict']);
for (const [c, line] of Object.entries(RIDE_LINES)) {
assert.equal(line.includes('\n'), false);
assert.match(line, /mosaic proxy ride/);
assert.doesNotMatch(line, /\//, 'no path in the owner line');
assert.equal(REFUSALS[c]?.nextStep, line, c);
}
});
+6 -6
View File
@@ -8,7 +8,7 @@ import { redactLine, redactValue, holdsSecret, secretValues, REDACTED, LINE_MAX
// Short keys of each Mosaic prefix, letters only, so no long-run or mixed-run rule catches them:
// only the key-prefix line does.
const SHORT = { msk: 'msk-tag-qwertyu', mso: 'mso-tag-acme-qwerty', mst: 'mst-tag-triage-zxcvb', msp: 'msp-tag-plmokn' };
const SHORT = { msk: 'msk-tag-qwertyu', mso: 'mso-tag-acme-qwerty', mst: 'mst-tag-triage-zxcvb', msp: 'msp-tag-plmokn', msw: 'msw-tag-worker-asdfgh' };
for (const [prefix, key] of Object.entries(SHORT)) {
test(`a short ${prefix}- key is redacted from a line, and the rest of the line stays`, () => {
@@ -35,7 +35,7 @@ test('a short Mosaic key glued to a word or an underscore is redacted', () => {
test('the real key shapes of each prefix are redacted', () => {
const tail = 'Ab3_dE5-fG7hJ9kL1mN3pQ5rS7tU9vW1xY3zA5bC7dE';
for (const key of [`msk-k1a2-${tail}`, `mso-k1a2-acme-${tail}`, `mst-k1a2-triage-${tail}`, `msp-k1a2-${tail}`, `sk-ant-${tail}`]) {
for (const key of [`msk-k1a2-${tail}`, `mso-k1a2-acme-${tail}`, `mst-k1a2-triage-${tail}`, `msp-k1a2-${tail}`, `msw-k1a2-worker-${tail}`, `sk-ant-${tail}`]) {
const safe = redactLine(`key ${key} rejected by upstream`);
assert.equal(safe.text.includes(key.slice(0, 12)), false, key.slice(0, 4));
assert.equal(safe.text.includes(tail.slice(0, 8)), false, key.slice(0, 4));
@@ -103,7 +103,7 @@ test('a 128000-character run with no spaces is redacted in well under a second',
});
// The list is a copy of failure-notice.mjs's. Every line must still match it, except the key-prefix
// line, which is the original with mso, mst and msp added after msk, and Mosaic's glued-prefix line
// line, which is the original with mso, mst, msp and msw added after msk, and Mosaic's glued-prefix line
// right after it.
test('the redaction list matches failure-notice.mjs line for line, with only the prefixes and one line added', () => {
const source = file => fs.readFileSync(fileURLToPath(new URL(file, import.meta.url)), 'utf8');
@@ -114,9 +114,9 @@ test('the redaction list matches failure-notice.mjs line for line, with only the
const keyWord = text => /const KEY_WORD = (.*);/.exec(text)[1];
const theirs = source('../../harness/src/failure-notice.mjs'), ours = source('../src/redact.mjs');
assert.equal(keyWord(ours), keyWord(theirs));
const expected = block(theirs).map(line => line.replace('(?:sk|msk|rk|', '(?:sk|msk|mso|mst|msp|rk|'));
const expected = block(theirs).map(line => line.replace('(?:sk|msk|rk|', '(?:sk|msk|mso|mst|msp|msw|rk|'));
assert.equal(expected.filter((line, i) => line !== block(theirs)[i]).length, 1, 'the original has one key-prefix line');
const at = expected.findIndex(line => line.includes('(?:sk|msk|mso|mst|msp|rk|'));
expected.splice(at + 1, 0, '/ms[kotp][-_]\\S+/gi');
const at = expected.findIndex(line => line.includes('(?:sk|msk|mso|mst|msp|msw|rk|'));
expected.splice(at + 1, 0, '/ms[kotpw][-_]\\S+/gi');
assert.deepEqual(block(ours), expected);
});
@@ -0,0 +1,80 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { setupState } from '../src/setup-state.mjs';
const [service, accounts] = setupState;
const ready = { release: { verified: true }, init: { initialized: true }, unit: { active: 'active', healthy: true, listening: true }, fence: { set: false } };
test('service needs a verified install, active unit and answering port', async () => {
assert.equal((await service.check({ service: async () => ready })).state, 'done');
for (const unit of [{ active: 'inactive' }, { active: 'failed' }, { active: 'active', healthy: false }]) assert.equal((await service.check({ service: async () => ({ ...ready, unit }) })).state, 'failed');
assert.equal((await service.check({ service: async () => ({ release: {}, init: {}, unit: {} }) })).state, 'next');
assert.equal((await service.check({ service: async () => ({ release: {}, init: {}, unit: {}, fence: { set: true } }) })).state, 'failed');
await assert.rejects(service.check({ service: async () => ({ release: { problem: 'proxy-install-invalid' }, init: {}, unit: {} }) }));
await assert.rejects(service.check({ service: async () => { throw Error(); } }));
});
test('account completion uses existing account metadata and a usable model, with no new health probe', async () => {
const org = { id: 'team' };
const ctx = (a, models) => ({ members: [org], read: async (_, name) => name === 'accounts' ? { accounts: a } : { catalogue: { status: 'ready', models } } });
assert.equal((await accounts.check(ctx([{ provider: 'claude', status: 'active' }], [{ module: 'cliproxy', upstream: 'claude', available: true }]))).state, 'done');
assert.equal((await accounts.check(ctx([], []))).state, 'next');
assert.equal((await accounts.check(ctx([{ status: 'error' }], []))).state, 'failed');
assert.equal((await accounts.check(ctx([{ provider: 'claude', status: 'active' }], []))).state, 'next');
assert.equal((await accounts.check({ members: [], read: async () => { throw Error(); } })).state, 'next');
await assert.rejects(accounts.check({ members: [org], read: async () => { throw Error(); } }));
});
test('the account reader can answer without an organization and uses the catalogue provider identity', async () => {
const result = await accounts.check({ members: [], accounts: async () => ({ accounts: [{ provider: 'claude', status: 'active' }], catalogue: { status: 'ready', models: [{ module: 'another-provider', upstream: 'claude', available: true }] } }) });
assert.equal(result.state, 'done');
});
for (const status of ['unavailable', 'disabled']) test(`a ${status} catalogue gives a plain next step`, async () => {
for (const entries of [[], [{ provider: 'claude', status: 'active' }]]) {
const result = await accounts.check({ members: [], accounts: async () => ({ accounts: entries, catalogue: { status } }) });
assert.equal(result.state, 'next'); assert.doesNotMatch(result.reason, /Couldn't check/);
}
});
test('one account reader failing does not hide a usable account', async () => {
assert.equal((await accounts.check({ members: [{ id: 'bad' }, { id: 'good' }], read: async (org, name) => {
if (org.id === 'bad') throw Error();
return name === 'accounts' ? { accounts: [{ provider: 'claude', status: 'active' }] } : { catalogue: { status: 'ready', models: [{ upstream: 'claude', available: true }] } };
} })).state, 'done');
});
import { readService } from '../src/setup-state.mjs';
import { defaultRun } from '../src/admin.mjs';
import { mkdtemp, mkdir, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
test('service inspection makes one systemctl read and forwards cancellation', async t => {
const root = await mkdtemp(join(tmpdir(), 'setup-service-'));
t.after(() => rm(root, { recursive: true, force: true }));
const configDir = join(root, 'config'), dataRoot = join(root, 'data');
await mkdir(configDir); await mkdir(join(dataRoot, 'proxy'), { recursive: true });
let calls = 0;
const signal = new AbortController().signal;
await readService({ configDir, dataRoot, instance: 'test', signal, run: async (command, _args, options) => {
calls++; assert.equal(command, 'systemctl'); assert.equal(options.signal, signal); assert.equal(options.timeout, 1000);
return { code: 0, stdout: 'LoadState=loaded\nActiveState=inactive\n' };
} });
assert.equal(calls, 1);
});
test('the process reader stops promptly when its request is cancelled', async () => {
const controller = new AbortController(), start = Date.now();
const pending = defaultRun(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { timeout: 5000, signal: controller.signal });
setTimeout(() => controller.abort(), 30);
assert.notEqual((await pending).code, 0); assert.ok(Date.now() - start < 1500);
});
test('an unrelated available model does not stand in for an active account offering one', async () => {
assert.equal((await accounts.check({ members: [], accounts: async () => ({ accounts: [{ provider: 'claude', status: 'active' }], catalogue: { status: 'ready', models: [{ upstream: 'unrelated', available: true }] } }) })).state, 'next');
});
import { PROVIDERS, providerUpstream } from '../../plugins/model-provider/cliproxy/index.mjs';
test('account matching uses the plugin mapping when its provider and upstream names differ', async () => {
const provider = PROVIDERS.find(p => p.signIn === 'apikey');
const result = await accounts.check({ members: [], accounts: async () => ({ accounts: [{ provider: provider.id, status: 'active' }], catalogue: { status: 'ready', models: [{ upstream: providerUpstream(provider.id), available: true }] } }) });
assert.equal(result.state, 'done');
});
for (const status of ['unavailable', 'disabled']) test(`a ${status} organization catalogue does not depend on an account reader answering`, async () => {
let accountsRead = 0;
const result = await accounts.check({ members: [{ id: 'team' }], read: async (_, name) => {
if (name === 'accounts') { accountsRead++; throw Error('no proxy configured'); }
return { catalogue: { status, models: [] } };
} });
assert.equal(result.state, 'next'); assert.doesNotMatch(result.reason, /Couldn't check/); assert.equal(accountsRead, 0);
});
@@ -0,0 +1,33 @@
// Uses the host's recorded harness fit result, not a thinking-level list or a new preflight.
export const setupState = Object.freeze([
{
id: 'agents.routed', title: 'Choose models for your agents',
purpose: 'The model your assistant and your chief use. Filled in for you when the default fits.',
fix: { page: '#/settings/routing', label: 'Choose models' },
order: { after: ['organization.ready'] }, required: true,
async check(ctx) {
if (!ctx.members.length) return { state: 'next', reason: 'Create an organization first, then choose the models your agents use.' };
const answers = await Promise.allSettled(ctx.members.map(async org => {
const fix = { page: '#/settings/routing', label: 'Choose models' };
const r = await ctx.read(org, 'routes');
if (!r?.instances || !Array.isArray(r.harnesses) || !Array.isArray(r.models)) throw Error('routes unavailable');
const names = ['assistant', 'chief'].map(name => Object.entries(org.doc?.roles ?? {}).find(([, role]) => role.definition === (name === 'chief' ? 'chief-of-staff' : name))?.[0] ?? name);
const chosen = names.map(name => r.instances[name]);
if (chosen.some(c => !c?.harness || !c.model || !r.harnesses.includes(c.harness) || !r.models.includes(c.model)))
return { state: 'next', reason: 'Choose a model for both your assistant and your chief.', fix };
const p = await ctx.read(org, 'plugins');
if (!Array.isArray(p?.harnessWatch)) throw Error('fit unavailable');
const fits = chosen.map(c => p.harnessWatch.find(h => h.name === c.harness));
if (fits.some(h => ['red', 'missing'].includes(h?.state))) return { state: 'failed', reason: 'A chosen program did not pass its check. Choose models to check the program.', fix };
if (fits.some(h => h?.state === 'unknown')) return { state: 'failed', reason: 'The program check did not finish. Choose models to try again.', fix };
if (fits.some(h => !['fit', 'changed'].includes(h?.state) || !h?.checkedAt)) return { state: 'checking', reason: 'The chosen programs have not passed their check yet. Choose models to check them.', fix };
return { state: 'done', reason: 'Your assistant and chief have models, and their programs passed the recorded check.' };
}));
const found = answers.filter(a => a.status === 'fulfilled').map(a => a.value);
const missed = answers.length - found.length;
const result = found.find(a => a.state === 'failed') ?? found.find(a => a.state === 'next') ?? found.find(a => a.state === 'checking') ?? found[0];
if (!result) return { state: 'next', reason: "Your organizations couldn't be checked. Choose models to check their settings." };
return { ...result, reason: result.reason + (missed ? " Some organizations couldn't be checked. Choose models to check their settings." : '') };
},
},
]);
@@ -0,0 +1,22 @@
export const setupState = Object.freeze([
{
id: 'organization.ready', title: 'Create your organization',
purpose: 'One organization is one team of agents with its own projects and tasks.',
fix: { page: '#/settings/organizations?new=1', label: 'Create organization' },
order: { after: ['model-account.connected'] }, required: true,
async check(ctx) {
if (!ctx.members.length) return { state: 'next', reason: 'Give your team a name. Mosaic sets up its first project and agents.' };
const answers = await Promise.allSettled(ctx.members.map(async org => ctx.read(org, 'status')));
const states = answers.filter(a => a.status === 'fulfilled').map(a => a.value);
const missed = answers.length - states.length;
const unhealthy = states.some(s => s?.healthy !== true);
const fix = { page: '#/settings/organizations', label: 'Open Organizations' };
if (!states.length) return { state: 'next', reason: "Your organizations couldn't be checked. Open Organizations to check them.", fix };
if (!states.some(s => s?.healthy === true)) return { state: 'failed', reason: 'An organization is not answering. Open Organizations to check it and start it again.' + (missed ? " Some organizations couldn't be checked." : ''), fix };
return { state: 'done', reason: 'An organization is created and answering.' +
(unhealthy ? ' Some organizations are not answering. Open Organizations to check them.' : '') +
(missed ? " Some organizations couldn't be checked. Open Organizations to check them." : ''),
...(unhealthy || missed ? { fix } : {}) };
},
},
]);
@@ -0,0 +1,53 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { setupState } from '../src/setup/setup-state.mjs';
import { setupState as routes } from '../src/setup/setup-routing.mjs';
const org = { id: 'team', doc: { roles: { assistant: { definition: 'assistant' }, chief: { definition: 'chief' } } } };
test('organization readiness requires an owner organization and its bus answering', async () => {
assert.equal((await setupState[0].check({ members: [], read: async () => {} })).state, 'next');
for (const healthy of [true, false]) assert.equal((await setupState[0].check({ members: [org], read: async () => ({ healthy }) })).state, healthy ? 'done' : 'failed');
const unavailable = await setupState[0].check({ members: [org], read: async () => { throw Error('private'); } });
assert.equal(unavailable.state, 'next'); assert.match(unavailable.reason, /couldn't be checked/); assert.doesNotMatch(unavailable.reason, /private/);
});
test('one answering organization completes setup while another cannot be checked', async () => {
const result = await setupState[0].check({ members: [{ id: 'broken', invalid: true }, org], read: async member => {
if (member.invalid) throw Error('private');
return { healthy: true };
} });
assert.equal(result.state, 'done'); assert.match(result.reason, /Some organizations couldn't be checked/); assert.doesNotMatch(result.reason, /private/);
});
test('one answering organization is enough even when another reports an unhealthy bus', async () => {
const result = await setupState[0].check({ members: [{ id: 'stopped' }, org], read: async member => ({ healthy: member.id === org.id }) });
assert.equal(result.state, 'done'); assert.match(result.reason, /Some organizations are not answering/);
});
test('routing requires both agent routes and recorded fit, never levels alone', async () => {
const route = { harnesses: ['example'], models: ['model'], instances: { assistant: { harness: 'example', model: 'model' }, chief: { harness: 'example', model: 'model' } } };
const run = (r = route, state = 'fit') => routes[0].check({ members: [org], read: async (_, name) => name === 'routes' ? r : { harnessWatch: [{ name: 'example', state, checkedAt: '2026-11-21T01:00:00Z' }] } });
assert.equal((await run()).state, 'done');
assert.equal((await run(route, 'changed')).state, 'done');
assert.equal((await run(route, 'red')).state, 'failed');
assert.equal((await run(route, 'unchecked')).state, 'checking');
assert.equal((await run({ ...route, instances: { assistant: route.instances.assistant } })).state, 'next');
assert.equal((await routes[0].check({ members: [] })).state, 'next');
assert.match((await routes[0].check({ members: [org], read: async () => { throw Error(); } })).reason, /couldn't be checked/);
});
test('routing follows a renamed chief before another role with the conventional id', async () => {
const member = { id: 'team', doc: { roles: { assistant: { definition: 'assistant' }, chief: { definition: 'coder' }, lead: { definition: 'chief-of-staff' } } } };
const r = { harnesses: ['example'], models: ['model'], instances: { assistant: { harness: 'example', model: 'model' }, chief: { harness: 'example', model: 'model' } } };
const check = () => routes[0].check({ members: [member], read: async (_, name) => name === 'routes' ? r : { harnessWatch: [{ name: 'example', state: 'fit', checkedAt: '2026-11-21T01:00:00Z' }] } });
assert.equal((await check()).state, 'next');
r.instances.lead = r.instances.chief;
assert.equal((await check()).state, 'done');
});
test('routing keeps answering organizations and uses the worst answered state', async () => {
const route = { harnesses: ['example'], models: ['model'], instances: { assistant: { harness: 'example', model: 'model' }, chief: { harness: 'example', model: 'model' } } };
for (const state of ['fit', 'red', 'unknown']) {
const result = await routes[0].check({ members: [{ id: 'bad' }, org], read: async (member, name) => {
if (member.id === 'bad') throw Error('private');
return name === 'routes' ? route : { harnessWatch: [{ name: 'example', state, checkedAt: 'now' }] };
} });
assert.equal(result.state, state === 'fit' ? 'done' : 'failed'); assert.match(result.reason, /Some organizations couldn't be checked/); assert.doesNotMatch(result.reason, /private/);
}
const result = await routes[0].check({ members: [org, { ...org, id: 'empty' }], read: async (member, name) => name === 'routes' ? member.id === 'empty' ? { ...route, instances: {} } : route : { harnessWatch: [{ name: 'example', state: 'fit', checkedAt: 'now' }] } });
assert.equal(result.state, 'next');
});
+12
View File
@@ -158,3 +158,15 @@ export function checkPlugins(value, where, { kinds = KINDS, fail = raise } = {})
}
return value;
}
// Instance readiness contributions, independent of an organization's plugin switches. Each part
// owns its row definitions and read checks. The host loads these paths relative to packages/.
// No new kind, module interface, organization setting or capability is introduced here.
export const setupState = Object.freeze([
'model-proxy/src/setup-state.mjs',
'organization/src/setup/setup-state.mjs',
'organization/src/setup/setup-routing.mjs',
'bus/src/setup-reply.mjs',
'tasks/src/setup-state.mjs',
'processing-control/src/setup-state.mjs',
]);
@@ -255,15 +255,17 @@ test("W1 redaction covers real paths, trailing slashes, the user name, rule ids,
test("W1 no record holds a path under the scratch directory, the home directory or an env value", async (t) => {
const { pass, dir } = await watchTree(t, { h: {} });
const secret = "/srv/fixture-secret-place";
// A multi-segment loose path stays a path even when tmpdir() is just /tmp.
const looseTempPath = join(tmpdir(), "mosaic-redaction-probe");
const env = { WATCH_SECRET: secret, HOME: join(dir, "home"), MOSAIC_FIXTURE_BIN: "/opt/fixture/bin/tool", PATH: "/opt/fixture-tools/bin:/usr/bin:/bin" };
const answer = ok(DIGEST_A, {
source: dir, version: `9.9 at ${dir}/bin`,
line: `found at ${env.MOSAIC_FIXTURE_BIN} in ${secret}, home ${env.HOME}, ${realpathSync(dir)}`,
surfaces: { help: `usage: ${dir}/tool\nsee /opt/fixture-tools/bin/tool and ${tmpdir()}`, "Bad Name": dir },
surfaces: { help: `usage: ${dir}/tool\nsee /opt/fixture-tools/bin/tool and ${looseTempPath}`, "Bad Name": dir },
});
const out = await pass({ h: answer }, { env });
const text = JSON.stringify(out);
for (const p of [dir, realpathSync(dir), secret, env.MOSAIC_FIXTURE_BIN, "/opt/fixture-tools/bin", env.HOME, tmpdir()]) assert.ok(!text.includes(p), p);
for (const p of [dir, realpathSync(dir), secret, env.MOSAIC_FIXTURE_BIN, "/opt/fixture-tools/bin", env.HOME, looseTempPath]) assert.ok(!text.includes(p), p);
const rec = byName(out).h;
assert.equal(rec.source, null, "a source that is a path is dropped");
assert.deepEqual(Object.keys(rec.surfaces), ["help"]);
@@ -0,0 +1,282 @@
// worker-stop-reason: a run-task worker run is judged by pi's final stop reason, not only its exit
// status (adapters/pi/final-stop.mjs). The real pi runs through adapters/pi/adapter.sh against a
// fake Anthropic Messages provider on 127.0.0.1 port 0: no real model, no other network.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawn, spawnSync } from "node:child_process";
import http from "node:http";
import { chmodSync, cpSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { locate } from "../harness/pi/binary.mjs";
import { agentEndKeeper, judgeFinalStop } from "../../../adapters/pi/final-stop.mjs";
import { fixture, tmp } from "../../cli/tests/helpers.mjs";
const REPO = join(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const ADAPTER = join(REPO, "adapters", "pi", "adapter.sh");
const FINAL_STOP = join(REPO, "adapters", "pi", "final-stop.mjs");
const TASK = join(REPO, "scripts", "mosaic-task.mjs");
const scratch = (t) => {
const dir = mkdtempSync(join(tmpdir(), "fs"));
t.after(() => rmSync(dir, { recursive: true, force: true }));
return dir;
};
const agentEnd = (...messages) => JSON.stringify({ type: "agent_end", messages });
const assistant = (stopReason, extra = {}) => ({ role: "assistant", content: [{ type: "text", text: "partial" }], stopReason, ...extra });
// --- The judge -----------------------------------------------------------------------------------
test("final stop: a finished stop passes with the text parts as text mode prints them", () => {
const line = agentEnd({ role: "user", content: "hi" }, { role: "assistant", stopReason: "stop", content: [{ type: "thinking", thinking: "x" }, { type: "text", text: "OK" }, { type: "text", text: "two" }] });
assert.deepEqual(judgeFinalStop(line), { ok: true, answer: "OK\ntwo\n" });
assert.deepEqual(judgeFinalStop(agentEnd({ role: "assistant", stopReason: "stop", content: [] })), { ok: true, answer: "" });
});
test("final stop: an error, aborted or length stop fails with one plain line", () => {
assert.deepEqual(judgeFinalStop(agentEnd(assistant("error", { errorMessage: "401 invalid\nx-api-key" }))), { ok: false, message: "pi stopped on an error: 401 invalid x-api-key" });
assert.deepEqual(judgeFinalStop(agentEnd(assistant("aborted"))), { ok: false, message: "pi stopped on an error: request aborted" });
assert.deepEqual(judgeFinalStop(agentEnd(assistant("length"))), { ok: false, message: "pi stopped with no finished answer: the model reached its output limit" });
const long = judgeFinalStop(agentEnd(assistant("error", { errorMessage: "e".repeat(5000) })));
assert.equal(long.ok, false);
assert.ok(long.message.length < 2100);
});
test("final stop: no end of run, no reply or an unknown stop reason fails closed", () => {
assert.equal(judgeFinalStop(null).ok, false);
assert.equal(judgeFinalStop("{not json").ok, false);
assert.equal(judgeFinalStop(agentEnd({ role: "user", content: "hi" })).ok, false);
assert.match(judgeFinalStop(agentEnd(assistant("deferred"))).message, /unknown reason \(deferred\)/);
assert.equal(judgeFinalStop(agentEnd(assistant(undefined))).ok, false);
});
test("final stop: a tool that ends the run passes with nothing printed, as text mode does", () => {
const line = agentEnd(assistant("toolUse"), { role: "toolResult", content: [{ type: "text", text: "done" }] });
assert.deepEqual(judgeFinalStop(line), { ok: true, answer: "" });
});
test("final stop: the keeper keeps the last agent_end line across chunk splits and skips other lines", () => {
const k = agentEndKeeper();
const first = agentEnd(assistant("error", { errorMessage: "retrying" }));
const last = agentEnd({ role: "assistant", stopReason: "stop", content: [{ type: "text", text: "é OK" }] });
const stream = Buffer.from(`{"type":"session"}\n${first}\n{"type":"message_end","note":"{\\"type\\":\\"agent_end\\"}"}\n${last}\n{"type":"agent_start"}\n`);
for (let i = 0; i < stream.length; i += 3) k.write(stream.subarray(i, i + 3));
assert.deepEqual(k.end(), { line: last, tooBig: false });
const unterminated = agentEndKeeper();
unterminated.write(Buffer.from(last));
assert.equal(unterminated.end().line, last);
const none = agentEndKeeper();
none.write(Buffer.from('{"type":"agent_start"}\n'));
assert.deepEqual(none.end(), { line: null, tooBig: false });
});
test("final stop: an agent_end line over the cap is not kept, and a later one replaces it", () => {
const small = agentEnd(assistant("stop"));
const big = agentEnd(assistant("stop", { pad: "x".repeat(300) }));
assert.ok(small.length < 200 && big.length > 200);
const k = agentEndKeeper(200);
k.write(Buffer.from(`${small}\n${big}\n`));
assert.deepEqual(k.end(), { line: null, tooBig: true });
const again = agentEndKeeper(200);
again.write(Buffer.from(`${big}\n${small}\n`));
assert.deepEqual(again.end(), { line: small, tooBig: false });
});
// --- final-stop.mjs with a stand-in pi -----------------------------------------------------------
function standIn(dir, body) {
const file = join(dir, "pi-stand-in.mjs");
writeFileSync(file, `#!${process.execPath}\n${body}\n`, { mode: 0o755 });
return file;
}
const runFinalStop = (pi, args = []) => spawnSync(process.execPath, [FINAL_STOP, pi, ...args], { encoding: "utf8", timeout: 20000, input: "" });
test("final stop: an error stop that pi reports with exit 0 is a failed run", (t) => {
const dir = scratch(t);
const pi = standIn(dir, `process.stdout.write(${JSON.stringify(`{"type":"session"}\n${agentEnd(assistant("error", { errorMessage: "529 overloaded" }))}\n`)}); process.exit(0);`);
const r = runFinalStop(pi);
assert.equal(r.status, 1);
assert.equal(r.stdout, "");
assert.equal(r.stderr, "pi stopped on an error: 529 overloaded\n");
});
test("final stop: pi gets its arguments plus --mode json, and its own exit code and stderr pass through", (t) => {
const dir = scratch(t);
const pi = standIn(dir, `(await import("node:fs")).writeFileSync(${JSON.stringify(join(dir, "argv"))}, JSON.stringify(process.argv.slice(2))); process.stderr.write("pi said no\\n"); process.exit(3);`);
const r = runFinalStop(pi, ["-p", "two words"]);
assert.equal(r.status, 3);
assert.equal(r.stderr, "pi said no\n");
assert.equal(r.stdout, "");
assert.deepEqual(JSON.parse(readFileSync(join(dir, "argv"), "utf8")), ["-p", "two words", "--mode", "json"]);
});
test("final stop: a pi that can't start or dies on a signal fails with one line", (t) => {
const dir = scratch(t);
const missing = runFinalStop(join(dir, "no-such-pi"));
assert.equal(missing.status, 127);
assert.match(missing.stderr, /^pi final stop: pi could not be started \(ENOENT\)\n$/);
const killed = runFinalStop(standIn(dir, `process.kill(process.pid, "SIGKILL");`));
assert.equal(killed.status, 137);
assert.equal(killed.stderr, "pi final stop: pi was stopped by SIGKILL\n");
});
// --- The real pi through adapter.sh -------------------------------------------------------------
function sse(res, events) {
res.writeHead(200, { "content-type": "text/event-stream", connection: "close" });
for (const [type, data] of events) res.write(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`);
res.end();
}
const START = { message: { id: "m1", type: "message", role: "assistant", model: "probe-model", content: [], stop_reason: null, usage: { input_tokens: 1, output_tokens: 0 } } };
const reply = (text, stop) => [
["message_start", START],
...(text === null ? [] : [
["content_block_start", { index: 0, content_block: { type: "text", text: "" } }],
["content_block_delta", { index: 0, delta: { type: "text_delta", text } }],
["content_block_stop", { index: 0 }],
]),
["message_delta", { delta: { stop_reason: stop }, usage: { output_tokens: 1 } }],
["message_stop", {}],
];
const SCENARIOS = {
ok: (req, res) => sse(res, reply("OK", "end_turn")),
"status-401": (req, res) => { res.writeHead(401, { "content-type": "application/json" }); res.end(JSON.stringify({ type: "error", error: { type: "authentication_error", message: "invalid x-api-key" } })); },
"length-empty": (req, res) => sse(res, reply(null, "max_tokens")),
"length-text": (req, res) => sse(res, reply("partial", "max_tokens")),
refusal: (req, res) => sse(res, reply(null, "refusal")),
};
async function provider(t, scenario) {
const server = http.createServer((req, res) => { req.resume(); req.on("end", () => SCENARIOS[scenario](req, res)); });
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
t.after(() => { server.closeAllConnections(); server.close(); });
return `http://127.0.0.1:${server.address().port}`;
}
const PI = locate(process.env);
async function adapterRun(t, scenario, { finalStop }) {
assert.ok(PI.bin, PI.problem);
const dir = scratch(t);
const agentDir = join(dir, "agent");
mkdirSync(agentDir);
const url = await provider(t, scenario);
writeFileSync(join(agentDir, "models.json"), JSON.stringify({ providers: { probe: { baseUrl: url, api: "anthropic-messages", apiKey: "probe-dummy", models: [{ id: "probe-model" }] } } }));
writeFileSync(join(dir, "system.md"), "Answer briefly.\n");
const env = {
PATH: `${dirname(process.execPath)}:/usr/bin:/bin`,
HOME: dir,
TMPDIR: dir,
LANG: "C.UTF-8",
PI_CODING_AGENT_DIR: agentDir,
PI_OFFLINE: "1",
PI_SKIP_VERSION_CHECK: "1",
PI_PROVIDER: "probe",
PI_MODEL: "probe-model",
MOSAIC_PI_BIN: PI.bin,
MOSAIC_SYSTEM_PROMPT_FILE: join(dir, "system.md"),
MOSAIC_REQUEST: "Say OK",
MOSAIC_WORKSPACE: join(dir, "ws"),
...(finalStop ? { MOSAIC_PI_FINAL_STOP: "1" } : {}),
};
const child = spawn("/bin/sh", [ADAPTER], { env, stdio: ["ignore", "pipe", "pipe"] });
let stdout = "", stderr = "";
child.stdout.on("data", (d) => (stdout += d));
child.stderr.on("data", (d) => (stderr += d));
const status = await new Promise((resolve) => child.on("close", (code, signal) => resolve(signal ?? code)));
return { status, stdout, stderr };
}
test("real pi: without the final-stop check a length stop exits 0 with nothing printed (the gap)", async (t) => {
const r = await adapterRun(t, "length-empty", { finalStop: false });
assert.deepEqual({ status: r.status, stdout: r.stdout, stderr: r.stderr }, { status: 0, stdout: "", stderr: "" });
});
test("real pi: with the check a normal stop still passes and prints the answer as text mode does", async (t) => {
const plain = await adapterRun(t, "ok", { finalStop: false });
const judged = await adapterRun(t, "ok", { finalStop: true });
assert.deepEqual({ status: plain.status, stdout: plain.stdout }, { status: 0, stdout: "OK\n" });
assert.deepEqual({ status: judged.status, stdout: judged.stdout, stderr: judged.stderr }, { status: 0, stdout: "OK\n", stderr: "" });
});
test("real pi: with the check a length stop fails, with or without partial text", async (t) => {
for (const scenario of ["length-empty", "length-text"]) {
const r = await adapterRun(t, scenario, { finalStop: true });
assert.deepEqual({ status: r.status, stdout: r.stdout, stderr: r.stderr }, { status: 1, stdout: "", stderr: "pi stopped with no finished answer: the model reached its output limit\n" }, scenario);
}
});
test("real pi: with the check a provider error stop (pi exits 0 in JSON mode) fails with pi's error text", async (t) => {
const r = await adapterRun(t, "status-401", { finalStop: true });
assert.equal(r.status, 1);
assert.equal(r.stdout, "");
assert.match(r.stderr, /^pi stopped on an error: 401 .*authentication_error.*invalid x-api-key.*\n$/);
const refused = await adapterRun(t, "refusal", { finalStop: true });
assert.equal(refused.status, 1);
assert.equal(refused.stderr, "pi stopped on an error: The model refused to complete the request\n");
});
// --- src/run-agent.sh sets the check for pi only -------------------------------------------------
function entrypoint(t) {
const root = scratch(t);
const opt = join(root, "opt");
for (const d of ["src", "processing-control", "contracts", "adapters/pi"]) mkdirSync(join(opt, d), { recursive: true });
writeFileSync(join(opt, "src", "run-agent.sh"), readFileSync(join(REPO, "src", "run-agent.sh"), "utf8").replaceAll("/opt/mosaic", opt), { mode: 0o755 });
writeFileSync(join(opt, "src", "load-contracts.sh"), '#!/bin/sh\nprintf "prompt\\n" > "$2"\n', { mode: 0o755 });
writeFileSync(join(opt, "processing-control", "entrypoint.mjs"), "");
cpSync(join(REPO, "adapters", "mock"), join(opt, "adapters", "mock"), { recursive: true });
chmodSync(join(opt, "adapters", "mock", "adapter.sh"), 0o755);
writeFileSync(join(opt, "adapters", "pi", "adapter.sh"), '#!/bin/sh\n(env | grep "^MOSAIC_" | sort) >&2\necho pi-stand-in\n', { mode: 0o755 });
return (adapter, extra = {}) => spawnSync(join(opt, "src", "run-agent.sh"), ["the request"], {
encoding: "utf8",
timeout: 20000,
env: { PATH: `${dirname(process.execPath)}:/usr/bin:/bin`, HOME: root, MOSAIC_ADAPTER: adapter, MOSAIC_SYSTEM_PROMPT_FILE: join(root, "prompt.md"), MOSAIC_MOCK_RESPONSE: "MOCK_OK", ...extra },
});
}
test("run-agent.sh: a headless pi worker gets MOSAIC_PI_FINAL_STOP=1, and the mock sees exactly the variables it did", (t) => {
const run = entrypoint(t);
const pi = run("pi");
assert.equal(pi.status, 0, pi.stderr);
assert.match(pi.stderr, /^MOSAIC_PI_FINAL_STOP=1$/m);
const mock = run("mock");
assert.equal(mock.status, 0, mock.stderr);
assert.equal(mock.stdout, "MOCK_OK\n");
const vars = mock.stderr.split("\n").filter((l) => l.startsWith("MOSAIC_")).map((l) => l.split("=")[0]);
assert.deepEqual(vars, ["MOSAIC_ADAPTER", "MOSAIC_MOCK_RESPONSE", "MOSAIC_REQUEST", "MOSAIC_SYSTEM_PROMPT_FILE"]);
const interactive = run("pi", { MOSAIC_INTERACTIVE: "1" });
assert.equal(interactive.status, 0, interactive.stderr);
assert.doesNotMatch(interactive.stderr, /MOSAIC_PI_FINAL_STOP/);
});
// --- run-task prints the cause of a failed run ---------------------------------------------------
function runTask(t, dockerBody) {
const root = tmp(t), f = fixture(root), bin = join(root, "bin");
mkdirSync(bin);
writeFileSync(join(bin, "docker"), `#!${process.execPath}\n${dockerBody}\n`, { mode: 0o755 });
f.env.PATH = `${bin}:${f.env.PATH}`;
const task = join(root, "task.json");
writeFileSync(task, JSON.stringify({ taskVersion: 1, id: "one", prompt: "request" }));
const r = spawnSync(process.execPath, [TASK, "run", task], { env: f.env, encoding: "utf8", timeout: 20000 });
const runs = join(f.dataRoot, "runs");
const [id] = readdirSync(runs);
return { r, id, record: JSON.parse(readFileSync(join(runs, id, "result.json"), "utf8")) };
}
test("run-task: a worker that stops on an error is a failed run whose cause the terminal shows", (t) => {
const { r, id, record } = runTask(t, 'process.stderr.write("pi stopped on an error: 401 invalid x-api-key\\n"); process.exit(1);');
assert.equal(r.status, 1, r.stderr);
assert.equal(record.status, "failed");
assert.equal(record.reason, "exit-nonzero");
assert.equal(r.stdout, `run: ${id}\nstatus: failed (exit-nonzero)\nresponse: \ncause: worker said: pi stopped on an error: 401 invalid x-api-key\n`);
});
test("run-task: a succeeded run prints no cause line", (t) => {
const { r, id, record } = runTask(t, 'console.log("normal response");');
assert.equal(r.status, 0, r.stderr);
assert.equal(record.status, "succeeded");
assert.equal(r.stdout, `run: ${id}\nstatus: succeeded\nresponse: normal response\n`);
});
@@ -1,12 +1,30 @@
#!/usr/bin/env node
// The shell launchers already load/release/bootstrap. Resolve the selected config again here.
// worker-cutover: every container this starts goes through worker-proxy.mjs launchWorker, so a pi
// worker reaches its model only through the instance's model proxy with its own worker key.
// --check runs the processing guard only. A probe (hello.sh, verify.sh) reads the worker's output
// and writes it with the key removed; an interactive session (agent.sh --tty) keeps the terminal.
import { randomUUID } from 'node:crypto';
import { constants } from 'node:os';
import { loadSystem } from '../../cli/src/config.mjs';
import { runContainer } from './containers.mjs';
import { CliError } from '../../cli/src/errors.mjs';
import { RIDE_LINES } from '../../model-proxy/src/settings.mjs';
import { configDir } from '../../organization/src/organization-config.mjs';
import { launchWorker, redactSecrets, WorkerLaunchError } from './worker-proxy.mjs';
import { unregisterRuntime } from './runtimes.mjs';
import { initializeControl, assertProcessing } from './state.mjs';
function refusal(e) {
// proxy-ride: the config refusal keeps its code and fixed owner line. Other config problems read
// as they did.
const ride = e instanceof CliError ? /(?:^|: )(proxy-ride-(?:invalid|owner-missing|conflict)): /.exec(e.message) : null;
if (ride) return { line: `processing: ${ride[1]}: ${RIDE_LINES[ride[1]]}`, exitCode: 3 };
if (e instanceof WorkerLaunchError) return { line: e.code ? `processing: ${e.code}: ${e.line}` : e.line, exitCode: e.exitCode };
return { line: `processing: ${e?.code?.startsWith?.('processing-') ? e.code : 'container-launch-failed'}`, exitCode: 3 };
}
try {
const system = loadSystem();
const system = loadSystem({ ride: true });
initializeControl(system.dataRoot);
assertProcessing(system.dataRoot);
const args = process.argv.slice(2), tty = args[0] === '--tty';
@@ -20,12 +38,24 @@ try {
...(process.env.MOSAIC_AGENT_SOUL_SOURCE ? { agentSoulFile: process.env.MOSAIC_AGENT_SOUL_SOURCE } : {}),
...(process.env.MOSAIC_SKILL_SOURCES ? { skillDirs: process.env.MOSAIC_SKILL_SOURCES.split(',') } : {}),
} : { kind: 'probe' };
const result = runContainer(system.dataRoot, { kind: 'container', business: 'instance', run: randomUUID() },
{ tty, args, worker, env: process.env, stdio: 'inherit' });
const spawn = tty
? { env: process.env, stdio: 'inherit' }
: { env: process.env, stdio: ['inherit', 'pipe', 'pipe'], encoding: 'utf8', maxBuffer: 16 * 1024 * 1024 };
const result = await launchWorker({
system, configDir: configDir(process.env),
record: { kind: 'container', business: 'instance', run: randomUUID() },
worker, args, tty, spawn,
});
if (!tty) {
process.stdout.write(redactSecrets(result.proc.stdout, result.secrets));
process.stderr.write(redactSecrets(result.proc.stderr, result.secrets));
}
if (!result.haltGeneration && !result.proc.error && result.proc.status === 0) unregisterRuntime(system.dataRoot, result.managed.id);
if (result.proc.error) throw result.proc.error;
process.exit(result.proc.status ?? 1);
// A signal held while docker ran ends this process as that signal would have: 128 plus its number.
process.exitCode = result.signal ? 128 + (constants.signals[result.signal] ?? 0) : result.proc.status ?? 1;
} catch (e) {
process.stderr.write(`processing: ${e.code?.startsWith('processing-') ? e.code : 'container-launch-failed'}\n`);
process.exit(3);
const { line, exitCode } = refusal(e);
process.stderr.write(`${line}\n`);
process.exitCode = exitCode;
}
+18 -14
View File
@@ -1,5 +1,5 @@
import { randomUUID } from 'node:crypto';
import { spawnSync, execFile } from 'node:child_process';
import { execFile } from 'node:child_process';
import { existsSync, lstatSync, readdirSync, realpathSync } from 'node:fs';
import { isAbsolute, join, relative, resolve } from 'node:path';
import { promisify } from 'node:util';
@@ -9,8 +9,9 @@ const execute = promisify(execFile);
const pause = ms => new Promise(resolve => setTimeout(resolve, ms));
const WORKER_KEYS = new Set([
'kind', 'taskFile', 'missionFile', 'workspaceDir', 'sessionDir',
'sessionForkFile', 'agentSoulFile', 'skillDirs',
'sessionForkFile', 'agentSoulFile', 'skillDirs', 'proxyDir', 'proxyModelsFile',
]);
const PROXY_SLOTS = ['proxyDir', 'proxyModelsFile'];
const WORKER_ENV = Object.freeze({
MOSAIC_SYSTEM_PROMPT_FILE: '/var/lib/mosaic/private/system-prompt.md',
MOSAIC_TASK_FILE: '',
@@ -88,8 +89,22 @@ export function workerLaunch(dataRoot, input) {
environment.MOSAIC_USER_DIR = '/var/lib/mosaic/input/user';
}
// worker-cutover: the launch's model proxy view (worker-proxy.mjs), for any kind, both or neither.
// The directory holds only the relay's socket, so a socket path a Docker socket-file bind would
// pin is never mounted; models.json names the key as "$ANTHROPIC_API_KEY", never its value.
const proxyGiven = PROXY_SLOTS.filter(key => input[key] !== undefined);
if (proxyGiven.length === 1) throw workerError('proxyDir and proxyModelsFile are given together or not at all');
if (proxyGiven.length === 2) {
const proxy = exactParts(components(root, input.proxyDir, 'proxyDir', 'directory'), ['worker-proxy', null], 'proxyDir');
if (!/^[0-9a-f]{12}$/.test(proxy.parts[1])) throw workerError(`proxyDir is outside its allowed data-root path: ${proxy.source}`);
const models = exactParts(components(root, input.proxyModelsFile, 'proxyModelsFile', 'file'),
['worker-proxy', proxy.parts[1], 'home', '.pi', 'agent', 'models.json'], 'proxyModelsFile');
add(proxy.source, '/run/mosaic-proxy');
add(models.source, '/home/node/.pi/agent/models.json');
}
if (input.kind === 'probe') {
for (const key of WORKER_KEYS) if (key !== 'kind' && input[key] !== undefined) throw workerError(`${key} is unsupported for probe launches`);
for (const key of WORKER_KEYS) if (key !== 'kind' && !PROXY_SLOTS.includes(key) && input[key] !== undefined) throw workerError(`${key} is unsupported for probe launches`);
return Object.freeze({ mounts: Object.freeze(mounts), environment: Object.freeze(environment) });
}
@@ -175,17 +190,6 @@ export function containerArgs(record, { tty = false, args = [], worker } = {}) {
'--label', `mosaic.root=${record.root}`, '--label', `mosaic.runtime=${record.id}`,
'-e', `MOSAIC_PROCESSING_GENERATION=${record.generation}`, ...environment, ...volumes, 'mosaic-agent', ...args];
}
export function runContainer(dataRoot, record, { tty = false, args = [], worker, ...options } = {}) {
const launch = workerLaunch(dataRoot, worker);
const managed = prepareContainer(dataRoot, record);
assertProcessing(dataRoot, managed.generation);
// Keep this synchronous. The independent broker sweeps the prepublished name while this waits.
const proc = spawnSync('docker', containerArgs(managed, { tty, args, worker: launch }), options);
const current = runtime(dataRoot, managed.id);
// An interrupted Docker client is not proof its container exited. Keep that receipt for inspection.
if (!proc.error && proc.status === 0) updateRuntime(dataRoot, managed.id, { phase: 'ended' });
return { proc, managed, haltGeneration: current.haltGeneration ?? null };
}
export function createContainerStopper({ dataRoot = null, docker = async args => {
const r = await execute('docker', args, { encoding: 'utf8', timeout: 2000, maxBuffer: 131072 });
return r.stdout;
@@ -0,0 +1,201 @@
#!/usr/bin/env node
// Host relay for worker containers (worker-proxy-simple, slice S2).
//
// node proxy-relay.mjs <socket path> 127.0.0.1:<port>
//
// Listens on one Unix socket and copies bytes, both ways, to the one loopback
// port its caller names: the user-level model proxy. It parses nothing and
// holds no key. The only address it ever opens is 127.0.0.1:<port>, checked
// once at start; any other target refuses before anything listens.
//
// The caller makes the socket's directory first, mode 0700, owned by this
// user. The relay refuses a directory that is a symbolic link, is owned by
// another user or grants any group or other bit, a socket path that is
// relative or 108 bytes or longer (the Linux sun_path limit counts the NUL),
// and a path that already exists. The socket itself is created 0600.
//
// Lifetime. The caller spawns the relay with a pipe on stdin and keeps it
// open. The relay exits on EOF there, which is what the caller's exit gives,
// clean or not. It also exits on SIGTERM, SIGINT or SIGHUP, and when its
// parent process id changes. On exit it closes every connection and the
// listener, which removes the socket file; the caller removes the directory.
//
// Output. One line "ready" on stdout once the socket listens, then nothing
// more. A refusal is one line on stderr and exit 2. Nothing is written per
// connection: the caller may be blocked in spawnSync and not draining pipes.
// Not wired into any launch yet; the cutover slice (S4) starts it.
import { spawn } from 'node:child_process';
import { chmodSync, lstatSync, realpathSync } from 'node:fs';
import net from 'node:net';
import { dirname, isAbsolute } from 'node:path';
import { fileURLToPath } from 'node:url';
export const SOCKET_PATH_LIMIT = 108;
const SELF = fileURLToPath(import.meta.url);
function refusal(message) {
const error = new Error(message);
error.relayRefusal = true;
return error;
}
// Only the literal 127.0.0.1 and a decimal port 1 to 65535. No name lookup,
// no IPv6, no other loopback address.
export function relayTarget(target) {
const match = typeof target === 'string' ? /^127\.0\.0\.1:([1-9][0-9]{0,4})$/.exec(target) : null;
const port = match ? Number(match[1]) : 0;
if (!match || port > 65535) throw refusal('proxy relay: the target must be 127.0.0.1:<port>.');
return { host: '127.0.0.1', port };
}
export function relaySocketPath(socketPath) {
if (typeof socketPath !== 'string' || !isAbsolute(socketPath)) {
throw refusal('proxy relay: the socket path must be absolute.');
}
const bytes = Buffer.byteLength(socketPath, 'utf8');
if (bytes >= SOCKET_PATH_LIMIT) {
throw refusal(`proxy relay: the socket path is ${bytes} bytes; it must be under ${SOCKET_PATH_LIMIT}.`);
}
const dir = dirname(socketPath);
let stat;
try { stat = lstatSync(dir); }
catch { throw refusal('proxy relay: the socket directory is missing.'); }
if (!stat.isDirectory() || stat.uid !== process.getuid() || (stat.mode & 0o777) !== 0o700) {
throw refusal('proxy relay: the socket directory must be a 0700 directory owned by this user.');
}
let exists = true;
try { lstatSync(socketPath); } catch (error) { if (error.code === 'ENOENT') exists = false; }
if (exists) throw refusal('proxy relay: the socket path already exists.');
return socketPath;
}
// Copy bytes both ways. An error on either side ends both at once. A clean
// close of one side only ends the other, so bytes still queued for a slow
// reader are flushed first; that side is destroyed when its peer closes too,
// or LINGER_MS after its last byte was written. The same helper is in
// adapters/pi/proxy-forward.mjs, which the image carries without this package.
const LINGER_MS = 10000;
function join(a, b) {
const kill = () => { a.destroy(); b.destroy(); };
const finish = (other) => () => {
if (other.destroyed) return;
other.resume();
other.end();
const linger = () => {
const timer = setTimeout(() => other.destroy(), LINGER_MS).unref();
other.once('close', () => clearTimeout(timer));
};
if (other.writableFinished) linger(); else other.once('finish', linger);
};
a.on('error', kill); b.on('error', kill);
a.on('close', finish(b)); b.on('close', finish(a));
a.pipe(b); b.pipe(a);
}
function main(argv) {
let socketPath, target;
try {
if (argv.length !== 2) throw refusal('proxy relay: usage: proxy-relay.mjs <socket path> 127.0.0.1:<port>.');
target = relayTarget(argv[1]);
socketPath = relaySocketPath(argv[0]);
} catch (error) {
process.stderr.write(`${error.relayRefusal ? error.message : 'proxy relay: could not start.'}\n`);
process.exit(2);
}
const parent = process.ppid;
const live = new Set();
let stopping = false;
const server = net.createServer({ allowHalfOpen: true });
function stop(code = 0) {
if (stopping) return;
stopping = true;
for (const socket of live) socket.destroy();
// Closing a bound listener removes its socket file (libuv unlinks it).
server.close(() => process.exit(code));
setTimeout(() => process.exit(code), 1000).unref();
}
server.on('connection', (client) => {
if (stopping) { client.destroy(); return; }
// The one address this process opens. Fixed at start, never from the client.
const upstream = net.connect({ host: target.host, port: target.port, family: 4, allowHalfOpen: true });
for (const socket of [client, upstream]) { live.add(socket); socket.on('close', () => live.delete(socket)); }
join(client, upstream);
});
server.on('error', (error) => {
if (!stopping) {
process.stderr.write(error.code === 'EADDRINUSE'
? 'proxy relay: the socket path already exists.\n'
: 'proxy relay: could not listen on the socket.\n');
process.exit(2);
}
});
process.stdin.on('data', () => {});
process.stdin.on('end', () => stop(0));
process.stdin.on('close', () => stop(0));
process.stdin.on('error', () => stop(0));
for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) process.on(signal, () => stop(0));
setInterval(() => { if (process.ppid !== parent) stop(0); }, 500).unref();
const mask = process.umask(0o177);
server.listen(socketPath, () => {
process.umask(mask);
if (stopping) return;
try { chmodSync(socketPath, 0o600); }
catch {
process.stderr.write('proxy relay: could not set the socket mode.\n');
stop(2);
return;
}
process.stdout.write('ready\n');
});
}
// For the cutover slice and the tests: start the relay as a child of this
// process and wait for its "ready" line. Rejects with the relay's own refusal
// line when it exits first. The child stays in the caller's process group.
export function spawnRelay({ socketPath, target, node = process.execPath, timeoutMs = 5000 } = {}) {
return new Promise((resolveStart, rejectStart) => {
// An empty environment: the relay reads none, and must never carry the
// caller's keys or NODE_OPTIONS.
const child = spawn(node, [SELF, socketPath, target], { env: {}, stdio: ['pipe', 'pipe', 'pipe'] });
child.stdin.on('error', () => {});
let out = '', err = '', settled = false;
const settle = (fn, value) => { if (!settled) { settled = true; clearTimeout(timer); fn(value); } };
const timer = setTimeout(() => {
child.kill('SIGKILL');
settle(rejectStart, new Error('proxy relay: did not start in time.'));
}, timeoutMs);
child.stdout.on('data', (chunk) => {
out += chunk;
if (out.includes('ready\n')) {
settle(resolveStart, {
child,
// EOF on stdin, then SIGKILL after killAfterMs if the relay is wedged.
stop({ killAfterMs = 2000 } = {}) {
return new Promise((done) => {
if (child.exitCode !== null || child.signalCode !== null) { done(); return; }
const kill = setTimeout(() => child.kill('SIGKILL'), killAfterMs);
child.once('exit', () => { clearTimeout(kill); done(); });
child.stdin.end();
});
},
});
}
});
child.stderr.on('data', (chunk) => { err += chunk; });
child.on('error', (error) => settle(rejectStart, error));
child.on('exit', () => settle(rejectStart, new Error(err.trim().split('\n')[0] || 'proxy relay: exited before it was ready.')));
});
}
function isMain() {
try { return process.argv[1] !== undefined && realpathSync(process.argv[1]) === realpathSync(SELF); }
catch { return false; }
}
if (isMain()) main(process.argv.slice(2));
@@ -0,0 +1,30 @@
import { defaultRun } from '../../model-proxy/src/admin.mjs';
import { listRunIds, isRunId, readRunDocument } from '../../runs/src/runs.mjs';
import { accountState, setupState as modelSetupState } from '../../model-proxy/src/setup-state.mjs';
// Stop at the first result instead of parsing the whole run history.
export function hasRunResult(dataRoot, { ids = listRunIds, read = readRunDocument } = {}) {
if (!dataRoot) return false;
return ids(dataRoot).reverse().some(id => isRunId(id) && !!read(dataRoot, id, 'result.json'));
}
export async function dockerAnswers({ run = defaultRun, signal } = {}) {
try { return (await run('docker', ['info', '--format', '{{.ServerVersion}}'], { timeout: 800, env: process.env, signal })).code === 0; }
catch { return false; }
}
export const setupState = Object.freeze([
{
id: 'worker.ready', title: 'Run work in sealed containers',
purpose: 'Lets agents run code in a container on this computer. Not needed to start. Needs Docker.',
fix: null, order: { after: [] }, required: false,
async check(ctx) {
const [service, account, docker] = await Promise.allSettled([modelSetupState[0].check(ctx), accountState(ctx), (ctx.docker ?? (() => dockerAnswers({ signal: ctx.signal })))()]);
const ready = service.status === 'fulfilled' && service.value?.state === 'done';
const connected = account.status === 'fulfilled' ? account.value.state === 'done' ? 'connected' : 'not ready' : 'not checked';
const model = typeof ctx.system?.execution?.model === 'string' && !!ctx.system.execution.model.trim();
const answers = docker.status === 'fulfilled' && docker.value === true;
const ran = ctx.runs ? (await ctx.runs()).some(r => r?.result && typeof r.result === 'object') : hasRunResult(ctx.dataRoot);
return { state: ran && answers ? 'done' : 'optional',
reason: `A worker needs Docker, a model service, an account and a worker model. Model service: ${ready ? 'ready' : 'not ready'}; Account: ${connected}; Worker model: ${model ? 'chosen' : 'not chosen'}; Docker: ${answers ? 'answering' : 'not answering'}. ${ran ? 'A worker run is recorded.' : 'No worker run is recorded.'} Worker runs start outside the Console.` };
},
},
]);
@@ -0,0 +1,258 @@
// worker-model-setup (S6 of reports/worker-proxy-simple.md, Jason's ruling 2026-11-21): the
// instance owner's worker model choice, <dataRoot>/worker-model/choice.json.
//
// { version, revision, provider, model, updatedAt, updatedBy }
//
// The config wins: a worker run uses execution.provider and execution.model when the config names
// them, and this choice only when it names none (worker-proxy.mjs workerModel). Today the config
// must name both (scripts/mosaic-config.mjs), so this choice has no effect on a run yet.
//
// Names only: a provider and a model id in the config's own name shape. No key, no env value and
// no harness enter this file. A Claude model is refused at every write and every read, so a choice
// in this file can never send a Claude model to pi; the launcher checks the catalogue again.
//
// Shaped and guarded like the Triage route store (packages/webui/src/triage-route-store.mjs). Fail
// closed. The worker-model directory is 0700 and choice.json 0600, both owned by the running user;
// a link, a hard link, a non-file, a wrong mode or owner or an unknown field refuses. A write is
// compare-and-set on `revision` under an exclusive lock file, atomic (a 0600 temp file in the same
// directory, fsync, rename, fsync of the directory), and appends one line to
// choice-receipts.jsonl. A writer that dies leaves its lock; the next write refuses and names it,
// and the person removes it by hand.
//
// Errors are WorkerModelStoreError with a `reason` (unreadable, invalid, busy, stale, unrecorded)
// and no refusal code: nothing here reaches a person on its own. The launcher turns a store error
// into its existing processing-worker-launch-invalid line, and the Console shows it as a state.
import { closeSync, constants, fchmodSync, fstatSync, fsyncSync, lstatSync, mkdirSync, openSync, readFileSync, renameSync, unlinkSync, writeSync } from "node:fs";
import { randomBytes } from "node:crypto";
import { join } from "node:path";
import { processState } from "../../seat/src/process-state.mjs";
export const CHOICE_VERSION = 1;
export const CHOICE_FIELDS = Object.freeze(["version", "revision", "provider", "model", "updatedAt", "updatedBy"]);
// The config's own name shape for execution.provider and execution.model (scripts/mosaic-config.mjs
// NAME_PATTERN), so a choice is never something the config could not name.
const NAME = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,199}$/;
const ACTOR = /^[a-z][a-z0-9-]{0,31}$/;
const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
const MAX_BYTES = 64 * 1024;
const INPUT_FIELDS = Object.freeze(["provider", "model", "by", "revision"]);
// The one Claude rule for worker models, used here, by the launcher and by the Console's list.
export const claudeModel = (provider, model) => provider === "anthropic" || /claude/i.test(String(model ?? ""));
export class WorkerModelStoreError extends Error {
constructor(reason, message) {
super(message);
this.name = "WorkerModelStoreError";
this.reason = reason;
}
}
const unreadable = (why) => new WorkerModelStoreError("unreadable", `the worker model choice is unreadable: ${why}`);
const invalid = (why) => new WorkerModelStoreError("invalid", `the worker model choice is invalid: ${why}`);
// The file location, in one place.
export function choicePaths(dataRoot) {
const dir = join(dataRoot, "worker-model");
return { dir, file: join(dir, "choice.json"), receipts: join(dir, "choice-receipts.jsonl"), lock: join(dir, "choice.lock") };
}
const plain = (v) => v !== null && typeof v === "object" && !Array.isArray(v) && Object.getPrototypeOf(v) === Object.prototype;
const exactKeys = (v, keys) => plain(v) && Object.keys(v).length === keys.length && keys.every((k) => Object.hasOwn(v, k));
const canonicalTime = (s) => typeof s === "string" && ISO.test(s) && !Number.isNaN(Date.parse(s)) && new Date(s).toISOString() === s;
export function validChoice(doc) {
return exactKeys(doc, CHOICE_FIELDS)
&& doc.version === CHOICE_VERSION
&& Number.isSafeInteger(doc.revision) && doc.revision >= 1
&& typeof doc.provider === "string" && NAME.test(doc.provider)
&& typeof doc.model === "string" && NAME.test(doc.model)
&& !claudeModel(doc.provider, doc.model)
&& canonicalTime(doc.updatedAt)
&& typeof doc.updatedBy === "string" && ACTOR.test(doc.updatedBy);
}
const uidOf = (opts) => opts.uid ?? process.getuid();
// The owner the files are held to: the same user, unless a test checks the file owner alone.
const fileUidOf = (opts) => opts.fileUid ?? uidOf(opts);
// The worker-model directory: a real directory, the running user's, mode 0700. Returns false when
// it is missing and `create` is off; with `create` it makes it (never the data root above it).
function checkDir(dir, uid, { create }) {
let st;
try { st = lstatSync(dir); }
catch (e) {
if (e.code !== "ENOENT") throw unreadable("the worker-model directory cannot be read");
if (!create) return false;
try { mkdirSync(dir, { mode: 0o700 }); } catch { throw unreadable("the worker-model directory cannot be made"); }
// The umask may have narrowed the mode; set it on the directory just made, never through a link.
try {
const fd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW);
try { fchmodSync(fd, 0o700); } finally { closeSync(fd); }
} catch { throw unreadable("the worker-model directory cannot be made"); }
st = lstatSync(dir);
}
if (st.isSymbolicLink()) throw unreadable("the worker-model directory is a link");
if (!st.isDirectory()) throw unreadable("the worker-model directory is not a directory");
if (st.uid !== uid) throw unreadable("the worker-model directory is not owned by this user");
if ((st.mode & 0o777) !== 0o700) throw unreadable("the worker-model directory mode is not 0700");
return true;
}
// An opened file's own checks: a regular file, one link, the running user's, mode 0600.
function checkFile(fd, uid, what) {
const st = fstatSync(fd);
if (!st.isFile()) throw unreadable(`${what} is not a regular file`);
if (st.nlink !== 1) throw unreadable(`${what} has more than one link`);
if (st.uid !== uid) throw unreadable(`${what} is not owned by this user`);
if ((st.mode & 0o777) !== 0o600) throw unreadable(`${what} mode is not 0600`);
return st;
}
function readFile(file, uid) {
let fd;
try { fd = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); }
catch (e) {
if (e.code === "ENOENT") return null;
if (e.code === "ELOOP") throw unreadable("choice.json is a link");
throw unreadable("choice.json cannot be opened");
}
try {
const st = checkFile(fd, uid, "choice.json");
if (st.size > MAX_BYTES) throw unreadable("choice.json is too large");
let doc;
try { doc = JSON.parse(readFileSync(fd, "utf8")); } catch { throw unreadable("choice.json is not JSON"); }
if (!validChoice(doc)) throw unreadable("choice.json does not match the choice shape");
return doc;
} finally { closeSync(fd); }
}
// The stored choice, or null when none is set. Throws WorkerModelStoreError (unreadable) on
// anything unsafe.
export function readChoice(dataRoot, opts = {}) {
const { dir, file } = choicePaths(dataRoot);
if (!checkDir(dir, uidOf(opts), { create: false })) return null;
return readFile(file, fileUidOf(opts));
}
function checkInput(input) {
if (!plain(input)) throw invalid("the input is not an object");
for (const k of Object.keys(input)) if (!INPUT_FIELDS.includes(k)) throw invalid(`unknown field ${k}`);
if (typeof input.provider !== "string" || !NAME.test(input.provider)) throw invalid("provider must be a name the config could hold");
if (typeof input.model !== "string" || !NAME.test(input.model)) throw invalid("model must be a name the config could hold");
if (claudeModel(input.provider, input.model)) throw invalid("Claude models run only in the Claude harness");
if (typeof input.by !== "string" || !ACTOR.test(input.by)) throw invalid("the actor name is invalid");
if (input.revision !== undefined && (!Number.isSafeInteger(input.revision) || input.revision < 0)) throw invalid("revision must be a whole number, 0 or more");
}
function holderLine(lock) {
let holder = null;
try {
const fd = openSync(lock, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try { holder = JSON.parse(readFileSync(fd, "utf8")); } finally { closeSync(fd); }
} catch { holder = null; }
if (!Number.isSafeInteger(holder?.pid) || holder.pid <= 0) return `another write holds ${lock}, or a writer left it; remove it by hand once no write is running`;
let alive;
try { alive = processState(holder.pid).alive; } catch { alive = null; }
if (alive === false) return `a writer that is not running (pid ${holder.pid}) left ${lock}; remove it by hand`;
return `another write (pid ${holder.pid}) holds ${lock}; try again when it ends`;
}
function takeLock(lock) {
let fd;
try { fd = openSync(lock, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600); }
catch (e) {
if (e.code === "EEXIST") throw new WorkerModelStoreError("busy", `the worker model choice is busy: ${holderLine(lock)}`);
throw unreadable("the lock cannot be made");
}
try {
writeSync(fd, `${JSON.stringify({ pid: process.pid, at: new Date().toISOString() })}\n`);
return { fd, st: fstatSync(fd) };
} catch {
closeSync(fd);
try { unlinkSync(lock); } catch {}
throw unreadable("the lock cannot be written");
}
}
function dropLock(lock, held) {
try {
const now = lstatSync(lock);
if (now.dev === held.st.dev && now.ino === held.st.ino) unlinkSync(lock);
} catch {}
closeSync(held.fd);
}
// The receipts file, opened for append before the choice changes, so a bad one refuses first. An
// existing file is held to its checks as it is; only a file made here gets its mode set.
function openReceipts(receipts, uid) {
const flags = constants.O_WRONLY | constants.O_APPEND | constants.O_NOFOLLOW | constants.O_NONBLOCK;
let fd;
try { fd = openSync(receipts, flags); }
catch (e) {
if (e.code === "ELOOP") throw unreadable("choice-receipts.jsonl is a link");
if (e.code !== "ENOENT") throw unreadable("choice-receipts.jsonl cannot be opened");
try { fd = openSync(receipts, flags | constants.O_CREAT | constants.O_EXCL, 0o600); }
catch { throw unreadable("choice-receipts.jsonl cannot be made"); }
try { fchmodSync(fd, 0o600); } catch { closeSync(fd); throw unreadable("choice-receipts.jsonl cannot be made"); }
}
try { checkFile(fd, uid, "choice-receipts.jsonl"); } catch (e) { closeSync(fd); throw e; }
return fd;
}
const summary = (doc) => (doc ? { provider: doc.provider, model: doc.model } : null);
// Set the choice. input: { provider, model, by, revision? }. With `revision` the write happens only
// if the stored revision equals it (0: only when no choice is set yet); without it, the revision
// read under the lock. Returns the new choice. Nothing in the Console calls this yet: saving from
// Model setup needs a route Jason has not approved (reports/worker-model-setup-build.md).
// opts.now, opts.uid and opts.fileUid are for tests; opts.hook(stage) is a test seam at "write"
// (inside the temp write), "crash" (after the temp file is written), "before-rename" and
// "after-rename" (before the receipt line). A failure after the rename refuses as unrecorded: the
// choice did change, and its receipt line is missing.
export function writeChoice(dataRoot, input, opts = {}) {
checkInput(input);
const { dir, file, receipts, lock } = choicePaths(dataRoot);
const uid = uidOf(opts);
const fileUid = fileUidOf(opts);
const now = (opts.now ?? (() => new Date()))().toISOString();
const hook = opts.hook ?? (() => {});
checkDir(dir, uid, { create: true });
const held = takeLock(lock);
try {
const current = readFile(file, fileUid);
const stored = current?.revision ?? 0;
if (input.revision !== undefined && input.revision !== stored) {
throw new WorkerModelStoreError("stale", `the worker model choice changed: the stored revision is ${stored}, not ${input.revision}`);
}
const next = { version: CHOICE_VERSION, revision: stored + 1, provider: input.provider, model: input.model, updatedAt: now, updatedBy: input.by };
if (!validChoice(next)) throw invalid("the new choice does not match the choice shape");
const receiptFd = openReceipts(receipts, fileUid);
try {
const temp = join(dir, `.choice.json.${process.pid}.${randomBytes(6).toString("hex")}.tmp`);
let tfd;
try { tfd = openSync(temp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600); }
catch { throw unreadable("choice.json cannot be written"); }
try {
try {
fchmodSync(tfd, 0o600);
hook("write");
writeSync(tfd, `${JSON.stringify(next, null, 2)}\n`);
fsyncSync(tfd);
} finally { closeSync(tfd); }
} catch { try { unlinkSync(temp); } catch {} throw unreadable("choice.json cannot be written"); }
hook("crash");
hook("before-rename");
try { renameSync(temp, file); } catch { try { unlinkSync(temp); } catch {} throw unreadable("choice.json cannot be replaced"); }
try {
hook("after-rename");
const dfd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY);
try { fsyncSync(dfd); } finally { closeSync(dfd); }
writeSync(receiptFd, `${JSON.stringify({ at: now, by: input.by, revision: next.revision, from: summary(current), to: summary(next) })}\n`);
fsyncSync(receiptFd);
} catch {
throw new WorkerModelStoreError("unrecorded", `the worker model choice changed to revision ${next.revision}, but its receipt line was not written`);
}
} finally { closeSync(receiptFd); }
return next;
} finally { dropLock(lock, held); }
}
@@ -0,0 +1,344 @@
// worker-cutover (S4 of reports/worker-proxy-simple.md, Jason's ruling 2026-11-21): the one launch
// step every host worker launch takes (scripts/mosaic-task.mjs for run-task and the release health
// check, container-cli.mjs for verify.sh, hello.sh and agent.sh). A pi worker reaches its model
// only through the instance's model proxy, with the worker key Mosaic issues itself:
//
// 1. checks, before anything is written: the host user is uid 1000, the proxy is set up, the
// model is not a Claude model, the management key loads, and the catalogue has an account
// that serves execution.provider/execution.model for pi (the cliproxy preflight). The model is
// the config's, or the Settings > Model setup choice when the config names none (workerModel);
// 2. ensures the worker key (issue, re-add or rotate under the proxy lock when needed) and reads
// it, calling again when another launch rotated it in between (prepareWorker; a caller runs
// it before it makes a run directory, so every refusal leaves nothing behind);
// 3. registers the runtime record, writes <dataRoot>/worker-proxy/<id>/home/.pi/agent/models.json
// (pi's bundle: the key as "$ANTHROPIC_API_KEY", never the value), starts the relay at
// <id>/proxy.sock;
// 4. runs docker with the key only in ANTHROPIC_API_KEY, the dir mounted read-only at
// /run/mosaic-proxy and models.json read-only at /home/node/.pi/agent/models.json;
// 5. when docker returns: stops the relay, removes the dir, and marks the record ended. A docker
// client that died by a timeout, a buffer limit or a signal is not proof its container ended,
// so the container is stopped first, and the record stays open when that outcome is unknown.
//
// A launcher killed outright leaves its record open, which keeps old worker keys from retiring
// (triage-key.mjs runtimeLive). Each launch sweeps first: a record that is not ended, whose
// supervisor is proved dead and whose container docker no longer lists, is marked ended, and a
// launch dir whose launcher is gone is removed.
//
// The mock adapter takes no key, no catalogue and no relay. Every adapter gets ANTHROPIC_API_KEY
// emptied and ZAI_API_KEY removed from the docker environment, so a caller's own key never reaches
// a worker. Refusals are WorkerLaunchError: a reused code and one fixed line, never a value.
import { execFile, spawnSync } from 'node:child_process';
import { lstatSync, mkdirSync, readdirSync, rmSync } from 'node:fs';
import { join } from 'node:path';
import { promisify } from 'node:util';
import { containerArgs, createContainerStopper, prepareContainer, workerLaunch } from './containers.mjs';
import { listRuntimes, runtime, runtimeGroupState, unregisterRuntime, updateRuntime } from './runtimes.mjs';
import { assertProcessing } from './state.mjs';
import { spawnRelay as defaultSpawnRelay } from './proxy-relay.mjs';
import { ensureReady as defaultEnsureReady, readKey as defaultReadKey } from '../../bus/src/triage-key.mjs';
import { loadManagementClient } from '../../model-proxy/src/client.mjs';
import { createModelCatalogue } from '../../bus/src/model-catalogue.mjs';
import { preflight as cliproxyPreflight } from '../../plugins/model-provider/cliproxy/index.mjs';
import { checkedTransport } from '../../harness/src/bundle.mjs';
import { bundle as piBundle } from '../../plugins/harness/pi/index.mjs';
import { claudeModel, readChoice as defaultReadChoice } from './worker-model-store.mjs';
const execute = promisify(execFile);
export const WORKER_UID = 1000;
export const PROXY_DIR = 'worker-proxy';
export const CONTAINER_PROXY_DIR = '/run/mosaic-proxy';
export const CONTAINER_MODELS_FILE = '/home/node/.pi/agent/models.json';
const HELD_SIGNALS = ['SIGINT', 'SIGTERM', 'SIGHUP'];
const KEY_TRIES = 3;
export const UID_LINE = 'Worker containers run as user 1000. Run Mosaic as that user.';
const LINES = {
'proxy-not-configured': () => 'The model proxy is not set up. Set it up on Settings > Model setup.',
'proxy-down': () => 'The model proxy is not running. Start it on Settings > Model setup.',
'model-harness-incompatible': () => 'Claude models run only in the Claude harness. Pick another model in the config.',
'proxy-key-refused': () => 'The proxy did not accept the worker key. Restart the proxy on Settings > Model setup, then run verify again.',
'proxy-lock-busy': () => 'Another proxy key change is running. Run verify again in a minute.',
'proxy-uncertain': () => 'A proxy key change did not finish. Restart the proxy, then run `mosaic proxy fence clear --proxy-restarted`.',
'model-catalogue-unavailable': () => 'The model proxy did not give a usable model list. Check it on Settings > Model setup, then run verify again.',
};
export class WorkerLaunchError extends Error {
// code null: a plain line with no code (the uid rule).
constructor(code, line, exitCode = 3) {
super(code ? `${code}: ${line}` : line);
this.name = 'WorkerLaunchError';
this.code = code;
this.line = line;
this.exitCode = exitCode;
}
}
const refuse = (code, ...words) => new WorkerLaunchError(code, LINES[code](...words));
// A code this module did not word passes through as `code: message`, with the message only when it
// is a fixed sentence from triage-key.mjs or client.mjs (TriageKeyError or a proxy-client error).
function passThrough(error) {
if (error instanceof WorkerLaunchError) return error;
const code = typeof error?.code === 'string' ? error.code : '';
if (code === 'proxy-client-down' || code === 'proxy-down') return refuse('proxy-down');
if (Object.hasOwn(LINES, code)) return refuse(code);
if (/^(?:proxy|worker-key)-[a-z-]+$/.test(code)) {
const words = error?.name === 'TriageKeyError' && typeof error.message === 'string' ? error.message : 'the model proxy step failed';
return new WorkerLaunchError(code, words);
}
return null;
}
const spec = (provider, model) => `${provider}/${model}`;
// The worker's harness is the config's execution.adapter (scripts/mosaic-config.mjs always resolves
// it), so core code here names no harness (plugins/tests/harness.test.mjs row 65). Only the mock
// adapter takes no key; an adapter that is missing is not read as any harness and refuses.
const harnessOf = system => system?.execution?.adapter;
const keyed = system => harnessOf(system) !== 'mock';
// worker-model-setup: the model a worker run uses. The config wins: when it names a provider or a
// model, those are used as they are, and the choice on Settings > Model setup is not read. The
// choice (worker-model-store.mjs) fills in only when the config names neither. Today
// scripts/mosaic-config.mjs requires both, so a run never reaches the choice yet. Every check after
// this one (Claude, switched off, served for the harness) holds the choice to the same rules.
const CHOICE_UNREADABLE = 'The worker model chosen on Settings > Model setup cannot be read. Set execution.provider and execution.model in the config.';
const NO_MODEL = 'No worker model is set. Set execution.provider and execution.model in the config.';
export function workerModel(system, { readChoice = defaultReadChoice } = {}) {
const { provider, model } = system?.execution ?? {};
if (provider !== undefined || model !== undefined) return { provider, model, from: 'config' };
let choice;
try { choice = readChoice(system?.dataRoot); }
catch { throw new WorkerLaunchError('processing-worker-launch-invalid', CHOICE_UNREADABLE); }
if (!choice) throw new WorkerLaunchError('processing-worker-launch-invalid', NO_MODEL);
return { provider: choice.provider, model: choice.model, from: 'model-setup' };
}
// Step 1 for a pi worker. Throws WorkerLaunchError; returns the catalogue entry and its transport.
export async function checkModel({ system, configDir, uid = process.getuid(), client = null, readChoice } = {}) {
if (uid !== WORKER_UID) throw new WorkerLaunchError(null, UID_LINE);
if (!system?.proxy || !Number.isInteger(system.proxy.port)) throw refuse('proxy-not-configured');
const { provider, model } = workerModel(system, readChoice ? { readChoice } : {});
if (claudeModel(provider, model)) throw refuse('model-harness-incompatible');
const harness = harnessOf(system);
if (typeof harness !== 'string' || harness === '') throw new WorkerLaunchError('processing-worker-launch-invalid', 'The config names no worker adapter.');
let mgmt = client;
if (!mgmt) {
// proxy-ride: a riding instance reads the owner instance's management key (same OS user, same
// 0600 check) and the owner's model list. Its own worker key stays its own.
try { mgmt = await loadManagementClient({ configDir: system.proxy.owner?.configDir ?? configDir, proxy: system.proxy }); }
catch (e) { throw passThrough({ code: e?.code ?? 'proxy-management-key-invalid', name: 'TriageKeyError', message: 'the proxy management key file is missing or unsafe' }); }
}
// The catalogue swallows each client error into "unavailable"; keep the first code to tell a
// proxy that is not running from one that answered with something unusable.
let seen = null;
const wrap = name => async (...args) => {
try { return await mgmt[name](...args); }
catch (e) { seen ??= typeof e?.code === 'string' ? e.code : 'proxy-client-response'; throw e; }
};
const view = { getCredentials: wrap('getCredentials'), getCredentialModels: wrap('getCredentialModels'), getUpstreamGroups: wrap('getUpstreamGroups') };
const catalogue = await createModelCatalogue({ client: () => view, settings: () => ({ 'model-provider': ['cliproxy'] }), pluginDataRoot: system.proxy.owner?.dataRoot ?? system.dataRoot }).read('worker', { fresh: true });
if (catalogue.status !== 'ready') throw seen === 'proxy-client-down' ? refuse('proxy-down') : refuse('model-catalogue-unavailable');
const entry = catalogue.models.find(m => m.provider === provider && m.model === model) ?? null;
const usable = () => {
const list = catalogue.models.filter(m => m.available && m.harnesses.includes(harness)).map(m => spec(m.provider, m.model));
return list.length ? [...new Set(list)].sort().join(', ') : 'none';
};
if (entry?.reason === 'model-off') {
throw new WorkerLaunchError('model-provider-unavailable', `${spec(provider, model)} is switched off. Switch it on in Settings > Model setup.`);
}
const verdict = entry ? cliproxyPreflight({ selection: entry, harness, catalogue }) : { allow: false, code: 'model-provider-unavailable' };
if (!verdict.allow) {
if (verdict.code === 'model-harness-incompatible') throw refuse('model-harness-incompatible');
if (verdict.code === 'model-catalogue-unavailable') throw refuse('model-catalogue-unavailable');
throw new WorkerLaunchError('model-provider-unavailable',
`No account serves ${spec(provider, model)}. Models you can use: ${usable()}. Add an account on Settings > Model setup, or set execution.model in the config.`);
}
let transport;
try { transport = checkedTransport(entry, harness); }
catch (e) { throw e?.code === 'model-harness-incompatible' ? refuse('model-harness-incompatible') : new WorkerLaunchError(typeof e?.code === 'string' ? e.code : 'proxy-provider-unsupported', 'The model proxy has no route for this model.'); }
return { entry, transport, client: mgmt };
}
const dirName = id => id.replace(/-/g, '').slice(0, 12);
const DIR_NAME = /^[0-9a-f]{12}$/;
// A record whose supervisor is proved dead. Its own pid, when it has one, says nothing about a
// container, which only docker can answer for.
const supervisorDead = r => {
const s = r?.supervisor;
if (!s || typeof s !== 'object') return false;
try { return runtimeGroupState({ pid: s.pid, startTime: s.startTime, boot: s.boot }) === false; }
catch { return false; }
};
// The startup sweep. Never throws: it only marks what it proved, and a launch never waits on it.
export async function sweepStale(dataRoot, { dockerList = defaultDockerList, dead = supervisorDead } = {}) {
const ended = [], removed = [];
let records = [];
try { records = listRuntimes(dataRoot); } catch { records = null; }
if (records) {
for (const r of records) {
if (r.phase === 'ended' || r.containerName === undefined || !dead(r)) continue;
let listed;
try { listed = await dockerList(['ps', '--all', '--no-trunc', '--filter', `label=mosaic.runtime=${r.id}`, '--format', '{{.ID}}']); }
catch { continue; }
if (typeof listed !== 'string' || listed.trim() !== '') continue;
try { updateRuntime(dataRoot, r.id, { phase: 'ended' }); ended.push(r.id); } catch { /* raced with its owner */ }
}
}
const root = join(dataRoot, PROXY_DIR);
let names = [];
try { names = readdirSync(root); } catch { names = []; }
for (const name of names) {
if (!DIR_NAME.test(name)) continue;
let fresh = null;
try { fresh = listRuntimes(dataRoot); } catch { fresh = null; }
if (!fresh) break;
const owner = fresh.find(r => dirName(r.id) === name);
if (owner && !dead(owner)) continue;
const path = join(root, name);
try {
const stat = lstatSync(path);
if (!stat.isDirectory() || stat.isSymbolicLink() || stat.uid !== process.getuid()) continue;
rmSync(path, { recursive: true, force: true });
removed.push(name);
} catch { /* gone or not ours */ }
}
return { ended, removed };
}
async function defaultDockerList(args) {
const r = await execute('docker', args, { encoding: 'utf8', timeout: 5000, maxBuffer: 131072 });
return r.stdout;
}
// Holds the three terminal signals from registration until cleanup ends: the relay must outlive
// docker, and the record must be marked before this process leaves. A signal seen before docker
// starts stops the launch; one seen while docker runs is reported after cleanup.
function holdSignals() {
const got = [];
const handler = signal => { got.push(signal); };
for (const s of HELD_SIGNALS) process.on(s, handler);
return { got, release() { for (const s of HELD_SIGNALS) process.off(s, handler); } };
}
// `model` is the checked catalogue entry's model id: the config's execution.model, or the Model
// setup choice when the config names none (workerModel), so compose's PI_MODEL is the model checked.
function dockerEnv(base, { key = '', port = '', provider = null, model = null } = {}) {
const env = { ...(base ?? process.env) };
delete env.ZAI_API_KEY;
env.ANTHROPIC_API_KEY = key;
env.MOSAIC_PROXY_PORT = port === '' ? '' : String(port);
if (provider) env.MOSAIC_PROVIDER = provider;
if (model) env.MOSAIC_MODEL = model;
return env;
}
// Steps 1 and 2 before anything is written for the run: the checks, the sweep, then the key. A
// launch may read the key before it registers its runtime (triage-key.mjs RETIRE_LAUNCH_MARGIN_MS
// covers that gap), so every refusal in the table comes before a run directory exists. Returns
// null for the mock adapter, else { entry, transport, key, keyId }. The key stays in memory.
export async function prepareWorker({ system, configDir, deps = {} }) {
const dataRoot = system.dataRoot;
const needsKey = keyed(system);
const checked = needsKey ? await checkModel({ system, configDir, uid: deps.uid ?? process.getuid(), client: deps.client ?? null, readChoice: deps.readChoice }) : null;
await sweepStale(dataRoot, { dockerList: deps.dockerList ?? defaultDockerList, ...(deps.dead ? { dead: deps.dead } : {}) });
if (!needsKey) return null;
// keyOptions: test seams triage-key.mjs takes (uid, fileUid, now, hook, lock, retireGuard).
const seams = deps.keyOptions ?? {};
const ensure = deps.ensureReady ?? defaultEnsureReady, read = deps.readKey ?? defaultReadKey;
for (let i = 0; i < KEY_TRIES; i++) {
let ready, current;
try {
ready = await ensure({ ...seams, purpose: 'worker', configDir, system, client: checked.client, runtimes: () => listRuntimes(dataRoot) });
current = read(configDir, { purpose: 'worker', uid: seams.uid, fileUid: seams.fileUid });
} catch (e) { throw passThrough(e) ?? e; }
// Another launch may rotate between the two reads; that newer key was not probed for this one.
if (current && current.keyId === ready.keyId) return { entry: checked.entry, transport: checked.transport, key: current.key, keyId: current.keyId };
}
throw new WorkerLaunchError('worker-key-raced', 'The worker key changed during each of three reads. Run verify again.');
}
// Steps 3 to 5. `prepared` is prepareWorker's answer; without one, launchWorker runs it first.
// `spawn` holds spawnSync options (env, cwd, stdio, input, timeout, ...). Returns
// { proc, managed, haltGeneration, modelKeyId, secrets, signal }: secrets is what the caller must
// redact from everything it writes or prints (the key); modelKeyId is null for the mock adapter.
export async function launchWorker({ system, configDir, record, worker, args = [], tty = false, spawn = {}, prepared, deps = {} }) {
const dataRoot = system.dataRoot;
const ready = prepared === undefined ? await prepareWorker({ system, configDir, deps }) : prepared;
const needsKey = keyed(system);
if (needsKey && !ready?.key) throw new WorkerLaunchError('processing-worker-launch-invalid', 'A worker that takes a key needs the model proxy checks first.');
const signals = holdSignals();
let managed = null, dir = null, relay = null, proc = null, haltGeneration = null;
try {
managed = prepareContainer(dataRoot, record);
let env = dockerEnv(spawn.env);
let launch;
if (needsKey) {
const parent = join(dataRoot, PROXY_DIR);
mkdirSync(parent, { recursive: true, mode: 0o700 });
const ps = lstatSync(parent);
if (!ps.isDirectory() || ps.isSymbolicLink() || ps.uid !== process.getuid() || (ps.mode & 0o777) !== 0o700) {
throw new WorkerLaunchError('processing-worker-launch-invalid', `${PROXY_DIR} under the data root must be a 0700 directory owned by this user.`);
}
dir = join(parent, dirName(managed.id));
mkdirSync(dir, { mode: 0o700 });
const port = system.proxy.port;
const files = piBundle({ dir: join(dir, 'x'), models: { worker: ready.entry }, proxy: { baseUrl: `http://127.0.0.1:${port}`, key: ready.key }, transports: { worker: ready.transport } }).files;
try { relay = await (deps.spawnRelay ?? defaultSpawnRelay)({ socketPath: join(dir, 'proxy.sock'), target: `127.0.0.1:${port}` }); }
// No code: the table has none for this, so the relay's own fixed line goes out plain.
catch (e) { throw new WorkerLaunchError(null, typeof e?.message === 'string' && /^proxy relay: [^\n]{1,200}$/.test(e.message) ? e.message : 'proxy relay: could not start.'); }
launch = workerLaunch(dataRoot, { ...worker, proxyDir: dir, proxyModelsFile: files.proxyModels });
env = dockerEnv(spawn.env, { key: ready.key, port, provider: ready.transport.piProvider, model: ready.entry.model });
} else {
launch = workerLaunch(dataRoot, worker);
}
const argv = containerArgs(managed, { tty, args, worker: launch });
if (signals.got.length) throw new WorkerLaunchError(null, `worker: the launch stopped on ${signals.got[0]} before docker started.`);
assertProcessing(dataRoot, managed.generation);
// Keep this synchronous. The independent broker sweeps the prepublished name while this waits.
proc = (deps.docker ?? spawnSync)('docker', argv, { ...spawn, env });
haltGeneration = runtime(dataRoot, managed.id).haltGeneration ?? null;
} finally {
// The relay stops only after docker returned, and before the record is marked.
if (relay) { try { await relay.stop(); } catch { /* it exits on its own when this process does */ } }
if (dir) { try { rmSync(dir, { recursive: true, force: true }); } catch { /* the next sweep removes it */ } }
if (managed) {
if (!proc) { try { unregisterRuntime(dataRoot, managed.id); } catch { /* the sweep ends it */ } }
else await finish(dataRoot, managed, proc, deps);
}
signals.release();
}
return { proc, managed, haltGeneration, modelKeyId: ready?.keyId ?? null, secrets: ready?.key ? [ready.key] : [], signal: signals.got[0] ?? null };
}
// The record is ended when docker's own exit or a failed start proves the container is gone.
// A nonzero client exit is not that proof (the client can lose the daemon while the container
// runs), so docker is asked once by runtime label: none listed ends the record, a listed one is
// stopped first, and a failed query leaves the record for the sweep. A client stopped by a
// timeout, a buffer limit or a signal gets the container stopped first.
async function finish(dataRoot, managed, proc, deps) {
let cut = (proc.error && ['ETIMEDOUT', 'ENOBUFS'].includes(proc.error.code)) || (!proc.error && proc.signal);
let end = true;
if (!cut && !proc.error && proc.status !== 0) {
let listed = null;
try { listed = await (deps.dockerList ?? defaultDockerList)(['ps', '--all', '--no-trunc', '--filter', `label=mosaic.runtime=${managed.id}`, '--format', '{{.ID}}']); }
catch { listed = null; }
if (typeof listed !== 'string') end = false;
else if (listed.trim() !== '') cut = true;
}
if (cut) {
try {
const stop = deps.stopper ?? createContainerStopper({ dataRoot });
const outcome = await stop(runtime(dataRoot, managed.id));
end = outcome?.outcome !== 'unknown';
} catch { end = false; }
}
if (end) { try { updateRuntime(dataRoot, managed.id, { phase: 'ended' }); } catch { /* left for the sweep */ } }
}
// Redacts each secret's exact text from a string (stderr.txt, the response).
export function redactSecrets(text, secrets = []) {
let out = typeof text === 'string' ? text : '';
for (const s of secrets) if (typeof s === 'string' && s) out = out.split(s).join('[redacted]');
return out;
}
@@ -10,16 +10,14 @@ const { parse: parseYaml } = createRequire(import.meta.url)('yaml');
function fixture(t) { const root = mkdtempSync(join(tmpdir(), 'processing-container-')); initializeControl(root); t.after(() => rmSync(root, { recursive: true, force: true })); return root; }
const actor = { business: 'demo', human: 'jason', halted: true };
test('static Compose service has no data-root or control-directory mount', () => {
test('static Compose service has no data-root, control-directory or credential mount', () => {
const compose = parseYaml(readFileSync(new URL('../../../compose.yaml', import.meta.url), 'utf8'));
const volumes = compose.services['mosaic-agent'].volumes;
const targets = volumes.map(volume => {
if (typeof volume !== 'string') return volume.target;
const searchFrom = volume.startsWith('${') ? volume.indexOf('}') + 1 : 0;
const sourceEnd = volume.indexOf(':', searchFrom);
return volume.slice(sourceEnd + 1).split(':')[0];
});
assert.deepEqual(targets, ['/home/node/.pi/agent/auth.json']);
const service = compose.services['mosaic-agent'];
// worker-cutover: the pi auth.json mount and the ZAI_API_KEY pass-through are gone.
assert.equal(service.volumes, undefined);
assert.equal('ZAI_API_KEY' in service.environment, false);
assert.equal(service.environment.ANTHROPIC_API_KEY, '${ANTHROPIC_API_KEY:-}');
assert.equal(service.environment.MOSAIC_PROXY_PORT, '${MOSAIC_PROXY_PORT:-}');
});
test('source launch view maps the control directory so actual atomic state replacements stay path-visible', t => {
@@ -172,7 +170,8 @@ test('unsupported or incomplete worker launch refuses before runtime registratio
assert.throws(() => api.workerLaunch(root, { kind: 'probe', brokerSocket: files.socket }), /unsupported worker launch key/);
assert.throws(() => api.workerLaunch(root, { kind: 'interactive', workspaceDir: files.workspace }), /sessionDir/);
const runtimes = join(root, 'control', 'runtimes');
assert.throws(() => api.runContainer(root, { kind: 'task', business: 'instance', run: 'missing' }, {}), /worker launch/);
// worker-cutover: the only docker path is worker-proxy.mjs launchWorker, which checks the proxy first.
assert.equal(api.runContainer, undefined);
assert.equal(existsSync(runtimes) ? readdirSync(runtimes).length : 0, 0);
});
test('container admission publishes its generated name and root/runtime labels before dispatch', t => {
@@ -0,0 +1,23 @@
// Stand-in launch step for worker-proxy-task.test.mjs. WorkerLaunchError and redactSecrets are the
// real ones, so mosaic-task.mjs's refusal lines and redaction are what the test sees. The key is
// a constant here and never in the environment, so only redactSecrets can take it out.
import { registerRuntime } from '../../src/runtimes.mjs';
import { WorkerLaunchError, UID_LINE, redactSecrets } from '../../src/worker-proxy.mjs';
export { WorkerLaunchError, redactSecrets };
export const FAKE_KEY = 'msw-0123456789ab-worker-' + 'K'.repeat(43);
const mode = process.env.FAKE_WP;
export async function prepareWorker() {
if (mode === 'refuse') throw new WorkerLaunchError('proxy-down', 'The model proxy is not running. Start it on Settings > Model setup.');
if (mode === 'uid') throw new WorkerLaunchError(null, UID_LINE);
return { key: FAKE_KEY, keyId: 'a1b2c3d4e5f6' };
}
export async function launchWorker({ system, record }) {
const managed = registerRuntime(system.dataRoot, record);
// 'proxy' (proxy-ride-cli.test.mjs): the answer names the proxy the launch step was given.
const proc = mode === 'proxy'
? { status: 0, signal: null, stdout: `${system.proxy?.owner?.instance ?? 'no-owner'}:${system.proxy?.port ?? 'no-proxy'}`, stderr: '' }
: mode === 'signal'
? { status: 0, signal: null, stdout: 'done', stderr: '' }
: { status: 1, signal: null, stdout: `answer ${FAKE_KEY}`, stderr: `starting\nerror: the proxy rejected ${FAKE_KEY}\n` };
return { proc, managed, haltGeneration: null, modelKeyId: 'a1b2c3d4e5f6', secrets: [FAKE_KEY], signal: mode === 'signal' ? 'SIGTERM' : null };
}
@@ -0,0 +1,14 @@
// Test hook (worker-proxy-task.test.mjs): scripts/mosaic-task.mjs and src/container-cli.mjs get the
// stand-in launch step in fixtures/worker-proxy-fake.mjs in place of src/worker-proxy.mjs. Nothing
// else is redirected.
import { registerHooks } from 'node:module';
const fake = new URL('./worker-proxy-fake.mjs', import.meta.url).href;
const PARENTS = ['/scripts/mosaic-task.mjs', '/processing-control/src/container-cli.mjs'];
registerHooks({
resolve(specifier, context, next) {
const resolved = next(specifier, context);
// Matched on the resolved URL: container-cli imports it as './worker-proxy.mjs'.
if (resolved.url.endsWith('/processing-control/src/worker-proxy.mjs') && PARENTS.some(p => context.parentURL?.endsWith(p))) return { url: fake, shortCircuit: true };
return resolved;
},
});
@@ -0,0 +1,631 @@
// Slice S2 of worker-proxy-simple: the host relay (src/proxy-relay.mjs) and
// the in-container forwarder (adapters/pi/proxy-forward.mjs), against a fake
// proxy on port 0. Nothing here starts Docker. Run with a TMPDIR of 31 bytes
// or less so every socket path stays well under the 108-byte limit.
import test from 'node:test';
import assert from 'node:assert/strict';
import { spawn, spawnSync } from 'node:child_process';
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
import http from 'node:http';
import net from 'node:net';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { relayTarget, relaySocketPath, spawnRelay, SOCKET_PATH_LIMIT } from '../src/proxy-relay.mjs';
const RELAY = fileURLToPath(new URL('../src/proxy-relay.mjs', import.meta.url));
const FORWARD = fileURLToPath(new URL('../../../adapters/pi/proxy-forward.mjs', import.meta.url));
const LINE = 'worker: the container could not reach the model proxy socket.\n';
const pause = ms => new Promise(r => setTimeout(r, ms));
function scratch(t) {
const base = mkdtempSync(join(tmpdir(), 'r'));
const dir = join(base, 'd');
mkdirSync(dir, { mode: 0o700 });
t.after(() => rmSync(base, { recursive: true, force: true }));
return { base, dir, sock: join(dir, 'p.sock') };
}
function listen(server) {
return new Promise((r) => server.listen(0, '127.0.0.1', () => r(server.address().port)));
}
// A fake model proxy: counts TCP connections, answers HTTP, can stream.
async function fakeProxy(t) {
const seen = { connections: 0, requests: [] };
const server = http.createServer({ keepAliveTimeout: 5000 }, async (req, res) => {
let body = '';
for await (const chunk of req) body += chunk;
seen.requests.push({ method: req.method, url: req.url, body, socket: req.socket.remotePort });
if (req.url === '/stream') {
res.writeHead(200, { 'content-type': 'text/event-stream' });
for (let i = 0; i < 5; i++) { res.write(`data: ${i}\n\n`); await pause(30); }
res.end('data: done\n\n');
return;
}
res.writeHead(200, { 'content-type': 'text/plain' });
res.end(`ok ${req.method} ${req.url} ${body.length}`);
});
server.on('connection', () => { seen.connections++; });
const port = await listen(server);
t.after(() => { server.closeAllConnections(); server.close(); });
return { server, port, seen };
}
// A raw TCP echo listener, used as the proxy for binary round trips and as a
// decoy port that must never see a connection.
async function tcpListener(t, onSocket = s => s.pipe(s)) {
const seen = { connections: 0 };
const server = net.createServer({ allowHalfOpen: true }, (s) => { seen.connections++; onSocket(s); });
const port = await listen(server);
t.after(() => server.close());
return { port, seen };
}
async function freePort() {
const s = net.createServer();
const port = await listen(s);
await new Promise(r => s.close(r));
return port;
}
function run(file, args, { timeout = 15000 } = {}) {
return spawnSync(process.execPath, [file, ...args], { encoding: 'utf8', timeout, stdio: ['pipe', 'pipe', 'pipe'] });
}
// Start the forwarder and wait for its "ready" line, or its exit.
function startForwarder(t, sock, port) {
return new Promise((resolveStart) => {
const child = spawn(process.execPath, [FORWARD, sock, String(port)], { stdio: ['ignore', 'pipe', 'pipe'] });
let out = '', err = '', done = false;
const finish = (value) => { if (!done) { done = true; resolveStart(value); } };
child.stdout.on('data', (c) => { out += c; if (out.includes('ready\n')) finish({ child, ready: true, err: () => err, out: () => out }); });
child.stderr.on('data', (c) => { err += c; });
child.on('exit', (code) => finish({ child, ready: false, code, err: () => err, out: () => out }));
t.after(() => { if (child.exitCode === null && child.signalCode === null) child.kill('SIGKILL'); });
});
}
function request(agent, port, path, { method = 'GET', body } = {}) {
return new Promise((resolveReq, rejectReq) => {
const req = http.request({ host: '127.0.0.1', port, path, method, agent }, (res) => {
const chunks = [];
const times = [];
res.on('data', (c) => { chunks.push(String(c)); times.push(Date.now()); });
res.on('end', () => resolveReq({ status: res.statusCode, chunks, times, body: chunks.join('') }));
});
req.on('error', rejectReq);
if (body) req.write(body);
req.end();
});
}
function unixRoundTrip(sock, bytes) {
return new Promise((resolveTrip, rejectTrip) => {
const c = net.connect({ path: sock, allowHalfOpen: true });
const got = [];
c.on('connect', () => { c.end(bytes); });
c.on('data', (d) => got.push(d));
c.on('end', () => { c.end(); resolveTrip(Buffer.concat(got)); });
c.on('error', rejectTrip);
});
}
function groupAlive(pgid) {
try { process.kill(-pgid, 0); return true; } catch (error) { return error.code !== 'ESRCH'; }
}
// A client that sends its request, half-closes, then reads slowly.
function slowHalfClose(connectOpts, bytes) {
return new Promise((resolveTrip, rejectTrip) => {
const c = net.connect({ ...connectOpts, allowHalfOpen: true });
let got = 0;
c.on('connect', () => c.end(bytes));
c.on('data', (d) => { got += d.length; c.pause(); setTimeout(() => c.resume(), 2); });
c.on('end', () => { c.end(); resolveTrip(got); });
c.on('error', rejectTrip);
});
}
// A proxy that answers each connection with `size` bytes once the client half-closes.
function bigReply(size) {
return (s) => {
s.on('error', () => {});
s.once('data', (d) => { if (String(d).startsWith('HEAD ')) s.end('HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n'); });
s.on('end', () => { if (!s.writableEnded) s.end(Buffer.alloc(size, 0x61)); });
};
}
test('target: only 127.0.0.1 with a decimal port 1 to 65535', () => {
assert.deepEqual(relayTarget('127.0.0.1:8317'), { host: '127.0.0.1', port: 8317 });
assert.deepEqual(relayTarget('127.0.0.1:65535'), { host: '127.0.0.1', port: 65535 });
for (const bad of ['0.0.0.0:80', '127.0.0.2:80', '127.1:80', 'localhost:80', '[::1]:80', '::1:80', '10.0.0.1:80',
'127.0.0.1', '127.0.0.1:', '127.0.0.1:0', '127.0.0.1:65536', '127.0.0.1:080', '127.0.0.1:+80', '127.0.0.1:80x',
' 127.0.0.1:80', '127.0.0.1:80\n', '127.0.0.1:0x50', 'http://127.0.0.1:80', undefined, 8080]) {
assert.throws(() => relayTarget(bad), /target must be 127\.0\.0\.1:<port>/, String(bad));
}
});
test('relay CLI refuses a non-loopback target with one line, and listens nowhere', (t) => {
const { dir, sock } = scratch(t);
for (const bad of ['192.168.1.1:80', '0.0.0.0:443', 'localhost:8317', '[::1]:8317', '127.0.0.1:0']) {
const r = run(RELAY, [sock, bad]);
assert.equal(r.status, 2, bad);
assert.equal(r.stdout, '');
assert.equal(r.stderr, 'proxy relay: the target must be 127.0.0.1:<port>.\n');
assert.equal(existsSync(sock), false);
}
const usage = run(RELAY, [sock]);
assert.equal(usage.status, 2);
assert.equal(usage.stderr.split('\n').length, 2);
assert.ok(existsSync(dir));
});
test('relay refuses a socket path of 108 bytes or more, and accepts 107', async (t) => {
const { dir } = scratch(t);
const pad = (n) => join(dir, 'x'.repeat(n - dir.length - 1));
const long = pad(SOCKET_PATH_LIMIT);
assert.equal(Buffer.byteLength(long), 108);
const r = run(RELAY, [long, '127.0.0.1:9']);
assert.equal(r.status, 2);
assert.equal(r.stderr, 'proxy relay: the socket path is 108 bytes; it must be under 108.\n');
assert.equal(existsSync(long), false);
const longer = run(RELAY, [pad(200), '127.0.0.1:9']);
assert.equal(longer.stderr, 'proxy relay: the socket path is 200 bytes; it must be under 108.\n');
const fits = pad(107);
const relay = await spawnRelay({ socketPath: fits, target: '127.0.0.1:9' });
assert.ok(lstatSync(fits).isSocket());
await relay.stop();
});
test('relay refuses an unsafe socket directory or an existing path', (t) => {
const { base, dir, sock } = scratch(t);
assert.throws(() => relaySocketPath('rel/p.sock'), /must be absolute/);
assert.throws(() => relaySocketPath(join(base, 'none', 'p.sock')), /socket directory is missing/);
const open = join(base, 'o');
mkdirSync(open, { mode: 0o755 });
assert.throws(() => relaySocketPath(join(open, 'p.sock')), /0700 directory owned by this user/);
const link = join(base, 'l');
symlinkSync(dir, link);
assert.throws(() => relaySocketPath(join(link, 'p.sock')), /0700 directory owned by this user/);
writeFileSync(sock, '');
assert.throws(() => relaySocketPath(sock), /already exists/);
const r = run(RELAY, [sock, '127.0.0.1:9']);
assert.equal(r.status, 2);
assert.equal(r.stderr, 'proxy relay: the socket path already exists.\n');
assert.equal(lstatSync(sock).isFile(), true);
});
test('relay socket is 0600 and copies bytes both ways to the one port', async (t) => {
const { sock } = scratch(t);
const echo = await tcpListener(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${echo.port}` });
t.after(() => relay.stop());
assert.equal(lstatSync(sock).mode & 0o777, 0o600);
const bytes = Buffer.alloc(300000);
for (let i = 0; i < bytes.length; i++) bytes[i] = (i * 31 + 7) & 0xff;
const back = await unixRoundTrip(sock, bytes);
assert.ok(back.equals(bytes));
const [a, b] = await Promise.all([unixRoundTrip(sock, Buffer.from('one')), unixRoundTrip(sock, Buffer.from('two'))]);
assert.equal(String(a), 'one');
assert.equal(String(b), 'two');
assert.equal(echo.seen.connections, 3);
});
test('full path: streamed reply and kept-alive requests on one connection', async (t) => {
const { sock } = scratch(t);
const proxy = await fakeProxy(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
let fwd;
for (let i = 0; i < 5; i++) {
fwd = await startForwarder(t, sock, await freePort());
if (fwd.ready) break;
}
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
assert.equal(proxy.seen.requests[0].method, 'HEAD', 'the start check reaches the proxy');
const checkConnections = proxy.seen.connections;
const agent = new http.Agent({ keepAlive: true, maxSockets: 1 });
t.after(() => agent.destroy());
const one = await request(agent, port, '/v1/models');
const two = await request(agent, port, '/v1/messages', { method: 'POST', body: '{"model":"m","x":"' + 'y'.repeat(70000) + '"}' });
const three = await request(agent, port, '/stream');
assert.equal(one.body, 'ok GET /v1/models 0');
assert.equal(two.body, 'ok POST /v1/messages 70020');
assert.equal(three.body, 'data: 0\n\ndata: 1\n\ndata: 2\n\ndata: 3\n\ndata: 4\n\ndata: done\n\n');
assert.ok(three.chunks.length >= 3, `streamed in pieces: ${three.chunks.length}`);
assert.ok(three.times.at(-1) - three.times[0] >= 60, 'pieces arrived over time, not buffered');
const kept = proxy.seen.requests.slice(1);
assert.equal(kept.length, 3);
assert.equal(proxy.seen.connections - checkConnections, 1, 'three requests, one proxy connection');
assert.equal(new Set(kept.map(r => r.socket)).size, 1);
assert.equal(fwd.err(), '');
});
test('traffic reaches only the proxy port, whatever the bytes name', async (t) => {
const { sock } = scratch(t);
const decoy = await tcpListener(t);
const proxied = [];
const proxy = await tcpListener(t, (s) => {
s.on('data', (d) => proxied.push(String(d)));
s.on('end', () => s.end('HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n'));
});
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
const asks = [
`CONNECT 127.0.0.1:${decoy.port} HTTP/1.1\r\nHost: 127.0.0.1:${decoy.port}\r\n\r\n`,
`GET http://127.0.0.1:${decoy.port}/ HTTP/1.1\r\nHost: 127.0.0.1:${decoy.port}\r\n\r\n`,
`GET / HTTP/1.1\r\nHost: example.com\r\nX-Forwarded-Host: 10.0.0.1\r\n\r\n`,
];
for (const ask of asks) {
const back = await unixRoundTrip(sock, Buffer.from(ask));
assert.match(String(back), /^HTTP\/1\.1 200 OK/);
}
assert.equal(proxy.seen.connections, 3);
assert.equal(decoy.seen.connections, 0);
assert.deepEqual(proxied, asks);
});
test('relay exits on its parent\'s signal and on stdin EOF, removing the socket; the caller removes the directory', async (t) => {
const { dir, sock } = scratch(t);
const proxy = await fakeProxy(t);
for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) {
const a = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => a.stop());
const exited = new Promise(r => a.child.once('exit', (code, sig) => r({ code, signal: sig })));
a.child.kill(signal);
assert.deepEqual(await exited, { code: 0, signal: null }, signal);
assert.equal(existsSync(sock), false, signal);
}
const b = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => b.stop());
const held = net.connect({ path: sock });
await new Promise(r => held.once('connect', r));
const heldClosed = new Promise(r => held.once('close', r));
await b.stop();
await heldClosed;
assert.equal(b.child.exitCode, 0);
assert.equal(existsSync(sock), false);
rmSync(dir, { recursive: true });
assert.equal(existsSync(dir), false);
});
test('relay runs with an empty environment', async (t) => {
const { sock } = scratch(t);
process.env.RELAY_TEST_MARKER = 'must-not-pass';
t.after(() => { delete process.env.RELAY_TEST_MARKER; });
const relay = await spawnRelay({ socketPath: sock, target: '127.0.0.1:9' });
t.after(() => relay.stop());
assert.equal(readFileSync(`/proc/${relay.child.pid}/environ`).length, 0);
});
test('stop() kills a relay that does not exit on EOF', async (t) => {
const { sock } = scratch(t);
const relay = await spawnRelay({ socketPath: sock, target: '127.0.0.1:9' });
t.after(() => { try { relay.child.kill('SIGKILL'); } catch {} });
relay.child.kill('SIGSTOP');
const started = Date.now();
await relay.stop({ killAfterMs: 300 });
assert.equal(relay.child.signalCode, 'SIGKILL');
assert.ok(Date.now() - started < 3000);
});
test('relay exits when its parent changes, even while its stdin stays open', async (t) => {
const { base, sock } = scratch(t);
const proxy = await fakeProxy(t);
// The middle process gives the relay a stdin pipe whose write end a third
// process (sleep) holds, so only the parent check can end the relay once
// the middle dies. Node closes a child's own stdin pipe when it exits.
const middleFile = join(base, 'middle.mjs');
writeFileSync(middleFile, `import { spawn } from 'node:child_process';
const holder = spawn('sleep', ['30'], { stdio: ['ignore', 'pipe', 'ignore'] });
const relay = spawn(process.execPath, [${JSON.stringify(RELAY)}, process.argv[2], process.argv[3]], { env: {}, stdio: [holder.stdout, 'pipe', 'inherit'] });
relay.stdout.on('data', (c) => { if (String(c).includes('ready')) process.stdout.write('relay ' + relay.pid + ' ' + holder.pid + '\\n'); });
setInterval(() => {}, 1000);
`);
const middle = spawn(process.execPath, [middleFile, sock, `127.0.0.1:${proxy.port}`], { detached: true, stdio: ['ignore', 'pipe', 'pipe'] });
t.after(() => { try { process.kill(-middle.pid, 'SIGKILL'); } catch {} });
const [relayPid, holderPid] = await new Promise((resolvePid, rejectPid) => {
let out = '';
middle.stdout.on('data', (c) => { out += c; const m = /relay (\d+) (\d+)\n/.exec(out); if (m) resolvePid([Number(m[1]), Number(m[2])]); });
middle.on('exit', () => rejectPid(new Error('middle exited early')));
});
const middleGone = new Promise(r => middle.once('exit', r));
process.kill(middle.pid, 'SIGKILL');
await middleGone;
const deadline = Date.now() + 4000;
while (Date.now() < deadline) {
try { process.kill(relayPid, 0); } catch { break; }
await pause(50);
}
process.kill(holderPid, 0); // the stdin write end was still held when the relay exited
assert.throws(() => process.kill(relayPid, 0), { code: 'ESRCH' });
assert.equal(existsSync(sock), false);
});
test('half-closed client reading slowly gets the whole reply, through the relay and through the forwarder', async (t) => {
const { sock } = scratch(t);
const size = 3 * 1024 * 1024;
const proxy = await tcpListener(t, bigReply(size));
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
for (let i = 0; i < 3; i++) assert.equal(await slowHalfClose({ path: sock }, Buffer.from('ask')), size, `relay round ${i}`);
let fwd;
for (let i = 0; i < 5; i++) { fwd = await startForwarder(t, sock, await freePort()); if (fwd.ready) break; }
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
for (let i = 0; i < 3; i++) assert.equal(await slowHalfClose({ host: '127.0.0.1', port }, Buffer.from('ask')), size, `forwarder round ${i}`);
assert.equal(fwd.err(), '');
});
test('relay exits when its parent is killed and leaves no process in the group', async (t) => {
const { base, dir, sock } = scratch(t);
const proxy = await fakeProxy(t);
const parentFile = join(base, 'parent.mjs');
writeFileSync(parentFile, `import { spawnRelay } from ${JSON.stringify(RELAY)};
const relay = await spawnRelay({ socketPath: process.argv[2], target: process.argv[3] });
process.stdout.write('relay ' + relay.child.pid + '\\n');
setInterval(() => {}, 1000);
`);
const parent = spawn(process.execPath, [parentFile, sock, `127.0.0.1:${proxy.port}`], { detached: true, stdio: ['ignore', 'pipe', 'pipe'] });
const pgid = parent.pid;
t.after(() => { try { process.kill(-pgid, 'SIGKILL'); } catch {} });
const relayPid = await new Promise((resolvePid, rejectPid) => {
let out = '';
parent.stdout.on('data', (c) => { out += c; const m = /relay (\d+)\n/.exec(out); if (m) resolvePid(Number(m[1])); });
parent.on('exit', () => rejectPid(new Error('parent exited early')));
});
assert.ok(groupAlive(pgid));
process.kill(relayPid, 0);
// A live connection through the relay must not keep it alive.
const held = net.connect({ path: sock });
await new Promise(r => held.once('connect', r));
held.on('error', () => {});
const parentGone = new Promise(r => parent.once('exit', r));
process.kill(parent.pid, 'SIGKILL');
await parentGone;
const deadline = Date.now() + 5000;
while (groupAlive(pgid) && Date.now() < deadline) await pause(50);
assert.equal(groupAlive(pgid), false, 'no process left in the parent\'s group');
assert.throws(() => process.kill(relayPid, 0), { code: 'ESRCH' });
assert.equal(existsSync(sock), false);
held.destroy();
rmSync(dir, { recursive: true });
assert.equal(existsSync(dir), false);
});
test('forwarder: a missing socket gives the fixed line and exit 1', (t) => {
const { sock } = scratch(t);
const r = run(FORWARD, [sock, '9']);
assert.equal(r.status, 1);
assert.equal(r.stdout, '');
assert.equal(r.stderr, LINE);
});
test('forwarder: a dead socket file gives the fixed line and exit 1', async (t) => {
const { dir, sock } = scratch(t);
// A socket file with no listener behind it: the relay was SIGKILLed.
const relay = await spawnRelay({ socketPath: sock, target: '127.0.0.1:9' });
const gone = new Promise(r => relay.child.once('exit', r));
relay.child.kill('SIGKILL');
await gone;
assert.ok(lstatSync(sock).isSocket());
const r = run(FORWARD, [sock, '9']);
assert.equal(r.status, 1);
assert.equal(r.stdout, '');
assert.equal(r.stderr, LINE);
rmSync(dir, { recursive: true });
});
test('forwarder: a live relay with no proxy behind it gives the fixed line and exit 1', async (t) => {
const { sock } = scratch(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${await freePort()}` });
t.after(() => relay.stop());
const r = run(FORWARD, [sock, '9']);
assert.equal(r.status, 1);
assert.equal(r.stdout, '');
assert.equal(r.stderr, LINE);
});
test('forwarder: a proxy that accepts but never answers gives the fixed line after the check timeout', async (t) => {
const { sock } = scratch(t);
const silent = await tcpListener(t, (s) => { s.on('data', () => {}); });
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${silent.port}` });
t.after(() => relay.stop());
const started = Date.now();
const r = run(FORWARD, [sock, '9']);
assert.equal(r.status, 1);
assert.equal(r.stderr, LINE);
assert.ok(Date.now() - started >= 4500);
});
test('forwarder: bad arguments refuse with a usage line and exit 2', (t) => {
const { sock } = scratch(t);
for (const args of [[], [sock], ['rel.sock', '80'], [sock, '0'], [sock, '65536'], [sock, '80x'], [sock, '080']]) {
const r = run(FORWARD, args);
assert.equal(r.status, 2, JSON.stringify(args));
assert.equal(r.stderr, 'worker: proxy forwarder usage: proxy-forward.mjs <socket path> <port>.\n');
}
});
test('forwarder: a lost relay after start prints the fixed line once', async (t) => {
const { sock } = scratch(t);
const proxy = await fakeProxy(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
let fwd;
for (let i = 0; i < 5; i++) { fwd = await startForwarder(t, sock, await freePort()); if (fwd.ready) break; }
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
await relay.stop();
for (let i = 0; i < 3; i++) {
await assert.rejects(request(undefined, port, '/v1/models'));
}
await pause(100);
assert.equal(fwd.err(), LINE);
});
test('forwarder: a proxy gone after the start check prints the fixed line once', async (t) => {
const { sock } = scratch(t);
const proxy = await fakeProxy(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
let fwd;
for (let i = 0; i < 5; i++) { fwd = await startForwarder(t, sock, await freePort()); if (fwd.ready) break; }
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
proxy.server.closeAllConnections();
await new Promise(r => proxy.server.close(r));
for (let i = 0; i < 3; i++) await assert.rejects(request(undefined, port, '/v1/models'));
await pause(100);
assert.equal(fwd.err(), LINE);
assert.equal(fwd.child.exitCode, null, 'the forwarder keeps running');
});
test('forwarder: an idle connection the proxy closes is not a lost proxy, a request it hangs up on is', async (t) => {
const { sock } = scratch(t);
// Answers the start check, closes a connection idle for 200 ms, and hangs
// up on any other request with no reply.
const proxy = await tcpListener(t, (s) => {
s.on('error', () => {});
const idle = setTimeout(() => s.end(), 200);
s.once('data', (d) => {
clearTimeout(idle);
if (String(d).startsWith('HEAD ')) s.end('HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n');
else s.end();
});
});
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
let fwd;
for (let i = 0; i < 5; i++) { fwd = await startForwarder(t, sock, await freePort()); if (fwd.ready) break; }
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
for (let i = 0; i < 3; i++) {
const c = net.connect({ host: '127.0.0.1', port });
c.on('error', () => {});
c.resume();
await new Promise(r => c.once('close', r));
}
await pause(100);
assert.equal(fwd.err(), '', 'an idle close prints nothing');
await assert.rejects(request(undefined, port, '/v1/models'));
await pause(100);
assert.equal(fwd.err(), LINE);
});
// The copier's error path: a reset or a drop on one side must end the other
// side at once, so no socket is left behind in the relay or the forwarder.
test('dropped and reset connections mid-stream leave no socket in the relay or the forwarder', async (t) => {
const { sock } = scratch(t);
const open = new Set();
t.after(() => { for (const s of open) s.destroy(); });
const proxy = await tcpListener(t, (s) => {
open.add(s);
s.on('error', () => {});
let timer;
s.once('data', (d) => {
if (String(d).startsWith('HEAD ')) { s.end('HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n'); return; }
timer = setInterval(() => s.write(Buffer.alloc(1024, 0x62)), 20);
});
// Like a streaming HTTP reply, it keeps writing after the client's FIN
// and stops only when its socket closes.
s.on('close', () => { clearInterval(timer); open.delete(s); });
});
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
let fwd;
for (let i = 0; i < 5; i++) { fwd = await startForwarder(t, sock, await freePort()); if (fwd.ready) break; }
assert.ok(fwd.ready, fwd.err());
const port = Number(fwd.child.spawnargs.at(-1));
const fds = (pid) => readdirSync(`/proc/${pid}/fd`).length;
// Let the start check's own connection close in the relay before counting.
const settleBy = Date.now() + 2000;
while (Date.now() < settleBy && open.size > 0) await pause(20);
await pause(200);
const base = { relay: fds(relay.child.pid), fwd: fds(fwd.child.pid) };
const streamThenDrop = (opts, drop) => new Promise((resolveDrop) => {
const c = net.connect(opts);
c.on('error', () => {});
c.once('connect', () => c.write('GET /s HTTP/1.1\r\n\r\n'));
c.once('data', () => { drop(c); });
c.once('close', resolveDrop);
});
const waves = [
[{ host: '127.0.0.1', port }, c => c.resetAndDestroy()],
[{ host: '127.0.0.1', port }, c => c.destroy()],
[{ path: sock }, c => c.destroy()],
];
for (const [opts, drop] of waves) await Promise.all(Array.from({ length: 40 }, () => streamThenDrop(opts, drop)));
const deadline = Date.now() + 3000;
while (Date.now() < deadline && (fds(relay.child.pid) > base.relay || fds(fwd.child.pid) > base.fwd || open.size > 0)) await pause(50);
const now = { relay: fds(relay.child.pid), fwd: fds(fwd.child.pid), proxy: open.size };
assert.ok(now.relay <= base.relay && now.fwd <= base.fwd && now.proxy === 0, JSON.stringify({ base, now }));
assert.equal(fwd.err(), '');
});
test('forwarder: stdout carries only "ready" and closes, so a shell can capture it', async (t) => {
const { base, sock } = scratch(t);
const proxy = await fakeProxy(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
const port = await freePort();
const script = join(base, 'cap.sh');
writeFileSync(script, 'ready=$("$1" "$2" "$3" "$4" &)\nprintf "[%s]\\n" "$ready"\n');
const sh = spawn('sh', [script, process.execPath, FORWARD, sock, String(port)], { detached: true, stdio: ['ignore', 'pipe', 'pipe'] });
t.after(() => { try { process.kill(-sh.pid, 'SIGKILL'); } catch {} });
let out = '';
sh.stdout.on('data', (c) => { out += c; });
const code = await new Promise(r => sh.once('exit', r));
assert.equal(code, 0);
assert.equal(out, '[ready]\n');
const one = await request(undefined, port, '/after');
assert.equal(one.body, 'ok GET /after 0');
});
// The container shape without Docker: bwrap gives an empty network namespace
// (loopback only, like network_mode: none) and a read-only bind of the socket
// directory, as the cutover's /run/mosaic-proxy mount will be. Same uid
// inside and out, so this proves nothing about Docker's uid mapping.
test('forwarder in a network namespace reaches the proxy through a read-only bind', async (t) => {
const probe = spawnSync('bwrap', ['--ro-bind', '/', '/', '--dev', '/dev', '--unshare-net', 'true'], { encoding: 'utf8' });
if (probe.error || probe.status !== 0) { t.skip('bwrap with --unshare-net is not available here'); return; }
const { base, dir, sock } = scratch(t);
const proxy = await fakeProxy(t);
const relay = await spawnRelay({ socketPath: sock, target: `127.0.0.1:${proxy.port}` });
t.after(() => relay.stop());
const inside = join(base, 'inside.mjs');
writeFileSync(inside, `import { spawn } from 'node:child_process';
import http from 'node:http';
const [fwd, port] = process.argv.slice(2);
const child = spawn(process.execPath, [fwd, '/run/mosaic-proxy/p.sock', port], { stdio: ['ignore', 'pipe', 'inherit'] });
let out = '';
child.stdout.on('data', (c) => { out += c; if (out === 'ready\\n') go(); });
child.on('exit', (code) => { if (out !== 'ready\\n') { process.stdout.write('forwarder exit ' + code + '\\n'); process.exit(1); } });
function go() {
http.get({ host: '127.0.0.1', port }, (res) => {
let body = ''; res.on('data', (c) => { body += c; });
res.on('end', () => {
const direct = http.get({ host: '127.0.0.1', port: ${proxy.port} });
direct.on('error', (e) => { process.stdout.write(body + '|direct ' + e.code + '\\n'); child.kill(); process.exit(0); });
direct.on('response', () => { process.stdout.write(body + '|direct reached\\n'); child.kill(); process.exit(0); });
});
}).on('error', (e) => { process.stdout.write('error ' + e.code + '\\n'); process.exit(1); });
}
`);
// Async: the fake proxy lives in this process, so a spawnSync here would starve it.
const box = spawn('bwrap', ['--ro-bind', '/', '/', '--dev', '/dev', '--proc', '/proc', '--tmpfs', '/run',
'--ro-bind', dir, '/run/mosaic-proxy', '--unshare-net', '--unshare-pid', '--die-with-parent',
process.execPath, inside, FORWARD, '7999'], { stdio: ['ignore', 'pipe', 'pipe'] });
const r = { stdout: '', stderr: '' };
box.stdout.on('data', (c) => { r.stdout += c; });
box.stderr.on('data', (c) => { r.stderr += c; });
const killer = setTimeout(() => box.kill('SIGKILL'), 20000);
r.status = await new Promise(done => box.once('close', done));
clearTimeout(killer);
assert.equal(r.stderr, '');
assert.equal(r.stdout, 'ok GET / 0|direct ECONNREFUSED\n');
assert.equal(r.status, 0);
});
@@ -0,0 +1,74 @@
// proxy-ride first slice (Jason 2026-11-21): a riding instance's worker model check. The rider's
// system.proxy carries `owner`, as scripts/mosaic-config.mjs gives it. The rider holds no
// management key of its own; it reads the owner's, and the owner's model on/off list. A stand-in
// proxy on 127.0.0.1 port 0 in this process. Nothing here reads a real config, proxy or secret.
import test from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initializeControl } from '../src/state.mjs';
import { WORKER_UID, WorkerLaunchError, checkModel } from '../src/worker-proxy.mjs';
import { writeOffered } from '../../model-proxy/src/models-offered.mjs';
const MANAGE = 'fxride-worker-manage';
const UPSTREAM = [{ 'base-url': 'https://api.z.ai/api/anthropic', keys: [{}], models: [{ name: 'fixture-id' }, { name: 'fixture-two' }] }];
async function fixture(t) {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'rw-')));
const owner = { configDir: join(root, 'oc'), dataRoot: join(root, 'od') };
const rider = { configDir: join(root, 'rc'), dataRoot: join(root, 'rd') };
for (const d of [owner.configDir, join(owner.configDir, 'secrets'), owner.dataRoot, rider.configDir, rider.dataRoot]) mkdirSync(d, { recursive: true, mode: 0o700 });
writeFileSync(join(owner.configDir, 'secrets', 'proxy-management.key'), MANAGE, { mode: 0o600 });
initializeControl(rider.dataRoot);
const seen = [];
const server = http.createServer((req, res) => {
seen.push(req.url);
if (req.headers.authorization !== `Bearer ${MANAGE}`) { res.writeHead(401); res.end('{}'); return; }
const body = req.url === '/v8/management/config/api-keys/claude' ? UPSTREAM
: req.url === '/v8/management/credentials' ? { files: [] } : { models: [] };
res.end(JSON.stringify(body));
});
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
const port = server.address().port;
t.after(async () => { server.closeAllConnections(); await new Promise(r => server.close(r)); rmSync(root, { recursive: true, force: true }); });
const execution = { backend: 'docker', provider: 'zai', model: 'fixture-id', adapter: 'pi' };
rider.system = { dataRoot: rider.dataRoot, execution, proxy: { port, binary: '/fixture-unused', owner: { instance: 'owner', configDir: owner.configDir, dataRoot: owner.dataRoot } } };
return { root, owner, rider, seen };
}
const check = f => checkModel({ system: f.rider.system, configDir: f.rider.configDir, uid: WORKER_UID });
test("a rider's model check reads the owner's management key and passes with no key of its own", async t => {
const f = await fixture(t);
const out = await check(f);
assert.equal(out.entry.provider, 'zai'); assert.equal(out.entry.model, 'fixture-id');
assert.ok(f.seen.length > 0, 'the check asked the stand-in proxy');
});
test("the owner switching a model off refuses it for the rider; the rider's own data root has no say", async t => {
const f = await fixture(t);
writeOffered(f.owner.dataRoot, { version: 1, providers: { 'z.ai': { on: ['fixture-two'], seen: ['fixture-id', 'fixture-two'] } } });
// The rider's own list would switch it on: only the owner's counts.
writeOffered(f.rider.dataRoot, { version: 1, providers: { 'z.ai': { on: ['fixture-id', 'fixture-two'], seen: ['fixture-id', 'fixture-two'] } } });
await assert.rejects(check(f), e => {
assert.ok(e instanceof WorkerLaunchError, String(e));
assert.equal(e.code, 'model-provider-unavailable');
assert.equal(e.line, 'zai/fixture-id is switched off. Switch it on in Settings > Model setup.');
assert.equal(String(e.message).includes(MANAGE), false);
return true;
});
// The owner switches it back on: the rider passes again.
writeOffered(f.owner.dataRoot, { version: 1, providers: { 'z.ai': { on: ['fixture-id', 'fixture-two'], seen: ['fixture-id', 'fixture-two'] } } });
assert.equal((await check(f)).entry.model, 'fixture-id');
});
test("a rider whose owner has no management key refuses with the key's code and no request", async t => {
const f = await fixture(t);
rmSync(join(f.owner.configDir, 'secrets', 'proxy-management.key'));
// A key in the rider's own directory is not the one a rider uses.
mkdirSync(join(f.rider.configDir, 'secrets'), { mode: 0o700 });
writeFileSync(join(f.rider.configDir, 'secrets', 'proxy-management.key'), MANAGE, { mode: 0o600 });
await assert.rejects(check(f), e => e instanceof WorkerLaunchError && e.code === 'proxy-management-key-invalid');
assert.deepEqual(f.seen, []);
});
@@ -0,0 +1,29 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { setupState, dockerAnswers } from '../src/setup-state.mjs';
test('worker readiness is optional and names the requirements actually ready', async () => {
const ctx = { system: { execution: { model: 'chosen' } }, members: [], dataRoot: '/unused', service: async () => ({ release: { verified: true }, init: { initialized: true }, unit: { active: 'active', healthy: true, listening: true } }), docker: async () => false, runs: () => [] };
const row = await setupState[0].check(ctx);
assert.equal(setupState[0].required, false); assert.equal(row.state, 'optional');
assert.match(row.reason, /Model service: ready/); assert.match(row.reason, /Account: not ready/); assert.match(row.reason, /Worker model: chosen/); assert.match(row.reason, /Docker: not answering/);
const done = await setupState[0].check({ ...ctx, docker: async () => true, runs: () => [{ result: { status: 'succeeded' } }] });
assert.equal(done.state, 'done');
const fenced = await setupState[0].check({ ...ctx, service: async () => ({ ...(await ctx.service()), fence: { set: true } }) });
assert.match(fenced.reason, /Model service: not ready/);
await assert.rejects(setupState[0].check({ ...ctx, runs: () => { throw Error(); } }));
});
test('Docker read is short, never starts a container and returns false on failure', async () => {
assert.equal(await dockerAnswers({ run: async (command, args, options) => { assert.equal(command, 'docker'); assert.deepEqual(args, ['info', '--format', '{{.ServerVersion}}']); assert.ok(options.timeout <= 1000); return { code: 0 }; } }), true);
assert.equal(await dockerAnswers({ run: async () => { throw Error(); } }), false);
});
import { hasRunResult } from '../src/setup-state.mjs';
test('worker evidence stops reading run results after the first recorded result', () => {
const reads = [];
assert.equal(hasRunResult('/unused', { ids: () => ['r-1', 'r-2', 'r-3'], read: (_, id) => { reads.push(id); return id === 'r-2' ? {} : null; } }), true);
assert.deepEqual(reads, ['r-3', 'r-2']);
assert.equal(hasRunResult('/unused', { ids: () => [], read: () => { throw Error(); } }), false);
});
test('Docker receives the request cancellation signal', async () => {
const signal = new AbortController().signal;
assert.equal(await dockerAnswers({ signal, run: async (_command, _args, options) => { assert.equal(options.signal, signal); return { code: 0 }; } }), true);
});
@@ -0,0 +1,134 @@
// worker-model-setup (S6): the launch side reads the Settings > Model setup worker model choice
// only when the config names no model, and holds the choice to every rule the config's model
// meets (Claude refused, switched off refused, served for the harness). Scratch data roots, a
// stand-in management client and a stub docker; nothing here reads a real config or runs Docker.
import test from 'node:test';
import assert from 'node:assert/strict';
import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initializeControl } from '../src/state.mjs';
import { listRuntimes } from '../src/runtimes.mjs';
import { WORKER_UID, WorkerLaunchError, checkModel, launchWorker, workerModel } from '../src/worker-proxy.mjs';
import { choicePaths, writeChoice } from '../src/worker-model-store.mjs';
import { writeOffered } from '../../model-proxy/src/models-offered.mjs';
const KEY = 'msw-0123456789ab-worker-' + 'S'.repeat(43);
const KEY_ID = 'a1b2c3d4e5f6';
const UPSTREAM = [{ 'base-url': 'https://api.z.ai/api/anthropic', keys: [{}], models: [{ name: 'fixture-id' }, { name: 'off-id' }] }];
const CHOICE_UNREADABLE = 'The worker model chosen on Settings > Model setup cannot be read. Set execution.provider and execution.model in the config.';
const NO_MODEL = 'No worker model is set. Set execution.provider and execution.model in the config.';
// `config: false` leaves execution.provider and execution.model out, as a config that names no
// worker model would (scripts/mosaic-config.mjs does not allow that yet; see the build report).
function fixture(t, { config = true, provider = 'zai', model = 'fixture-id' } = {}) {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wc-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
const dataRoot = join(root, 'd'), configDir = join(root, 'c');
mkdirSync(dataRoot, { mode: 0o700 }); mkdirSync(configDir, { mode: 0o700 });
initializeControl(dataRoot);
const execution = { backend: 'docker', adapter: 'pi', ...(config ? { provider, model } : {}) };
return { dataRoot, configDir, system: { dataRoot, execution, proxy: { port: 1 } } };
}
const choose = (f, provider, model) => writeChoice(f.dataRoot, { provider, model, by: 'owner' });
function counted(groups = async () => UPSTREAM) {
const n = { asked: 0 };
return { n, client: { getCredentials: async () => ({ files: [] }), getCredentialModels: async () => ({ models: [] }), getUpstreamGroups: async () => { n.asked++; return groups(); } } };
}
const deps = (over = {}) => ({
uid: WORKER_UID, dockerList: async () => '', dead: () => false,
ensureReady: async () => ({ keyId: KEY_ID }), readKey: () => ({ key: KEY, keyId: KEY_ID }),
spawnRelay: async () => ({ stop: async () => {} }), stopper: async () => ({ outcome: 'stopped' }),
docker: () => ({ status: 0, signal: null, stdout: '', stderr: '' }),
...over,
});
const refusal = (code, line) => error => {
assert.ok(error instanceof WorkerLaunchError, String(error));
assert.equal(error.code, code); assert.equal(error.line, line);
return true;
};
const check = (f, client, extra = {}) => checkModel({ system: f.system, configDir: f.configDir, uid: WORKER_UID, client, ...extra });
test('the config wins: a stored choice is not read when the config names a model', async t => {
const f = fixture(t);
choose(f, 'zai', 'off-id');
let reads = 0;
const spy = () => { reads++; return { provider: 'zai', model: 'off-id' }; };
assert.deepEqual(workerModel(f.system, { readChoice: spy }), { provider: 'zai', model: 'fixture-id', from: 'config' });
const out = await check(f, counted().client, { readChoice: spy });
assert.equal(out.entry.model, 'fixture-id');
assert.equal(reads, 0);
// A config naming only one of the two still wins, and the other half is not filled from the choice.
f.system.execution = { backend: 'docker', adapter: 'pi', model: 'fixture-id' };
assert.deepEqual(workerModel(f.system, { readChoice: spy }), { provider: undefined, model: 'fixture-id', from: 'config' });
assert.equal(reads, 0);
});
test('a config that names no model uses the Model setup choice, and the launch passes that model to compose', async t => {
const f = fixture(t, { config: false });
choose(f, 'zai', 'fixture-id');
assert.deepEqual(workerModel(f.system), { provider: 'zai', model: 'fixture-id', from: 'model-setup' });
const { n, client } = counted();
let env = null;
const out = await launchWorker({
system: f.system, configDir: f.configDir, record: { kind: 'task', business: 'instance', run: 'r-choice' },
worker: { kind: 'probe' }, spawn: { env: { PATH: process.env.PATH, MOSAIC_MODEL: 'stale-from-caller', MOSAIC_PROVIDER: 'stale' } },
deps: deps({ client, docker: (cmd, argv, opts) => { env = opts.env; return { status: 0, signal: null, stdout: '', stderr: '' }; } }),
});
assert.equal(n.asked, 1);
assert.equal(env.MOSAIC_MODEL, 'fixture-id', 'compose gets the checked model, not what the caller had');
assert.notEqual(env.MOSAIC_PROVIDER, 'stale');
assert.equal(env.ANTHROPIC_API_KEY, KEY);
assert.equal(out.signal, null);
});
test('no config model and no choice refuses before the proxy is asked', async t => {
const f = fixture(t, { config: false });
const { n, client } = counted();
await assert.rejects(check(f, client), refusal('processing-worker-launch-invalid', NO_MODEL));
assert.equal(n.asked, 0);
// No system means no data root, so there is no choice to read either.
assert.throws(() => workerModel(undefined), refusal('processing-worker-launch-invalid', CHOICE_UNREADABLE));
});
test('a choice that cannot be read refuses with its own line, never the store message, before the proxy is asked', async t => {
const f = fixture(t, { config: false });
choose(f, 'zai', 'fixture-id');
chmodSync(choicePaths(f.dataRoot).file, 0o644);
const { n, client } = counted();
await assert.rejects(check(f, client), refusal('processing-worker-launch-invalid', CHOICE_UNREADABLE));
assert.equal(n.asked, 0);
assert.equal(listRuntimes(f.dataRoot).length, 0);
});
test('Claude never reaches pi through the choice: a Claude choice refuses model-harness-incompatible before the proxy is asked', async t => {
// The store refuses to hold a Claude model; a seam stands in for a store that let one through.
for (const [provider, model] of [['anthropic', 'fixture-id'], ['zai', 'claude-opus-5-5'], ['openrouter', 'anthropic/Claude-Sonnet']]) {
const f = fixture(t, { config: false });
const { n, client } = counted();
await assert.rejects(check(f, client, { readChoice: () => ({ provider, model }) }),
refusal('model-harness-incompatible', 'Claude models run only in the Claude harness. Pick another model in the config.'));
assert.equal(n.asked, 0, `${provider}/${model}`);
}
});
test('a stale choice for a model later switched off refuses with the switched-off line', async t => {
const f = fixture(t, { config: false });
choose(f, 'zai', 'off-id');
writeOffered(f.dataRoot, { version: 1, providers: { 'z.ai': { on: ['fixture-id'], seen: ['fixture-id', 'off-id'] } } });
await assert.rejects(check(f, counted().client),
refusal('model-provider-unavailable', 'zai/off-id is switched off. Switch it on in Settings > Model setup.'));
// The offered file is keyed by the offered provider id (z.ai), not the catalogue provider (zai).
// The same file leaves a model that is on usable through the choice.
const g = fixture(t, { config: false });
choose(g, 'zai', 'fixture-id');
writeOffered(g.dataRoot, { version: 1, providers: { 'z.ai': { on: ['fixture-id'], seen: ['fixture-id', 'off-id'] } } });
assert.equal((await check(g, counted().client)).entry.model, 'fixture-id');
});
test('a choice no account serves refuses model-provider-unavailable and lists what pi can use', async t => {
const f = fixture(t, { config: false });
choose(f, 'zai', 'glm-5.3-flash');
await assert.rejects(check(f, counted().client), error => error.code === 'model-provider-unavailable' && /^No account serves zai\/glm-5\.3-flash\. Models you can use: zai\/fixture-id, zai\/off-id\./.test(error.line));
});
@@ -0,0 +1,330 @@
// worker-model-setup (S6): the instance owner's worker model choice,
// <dataRoot>/worker-model/choice.json. Names only. The directory is 0700 and the file 0600, both
// the running user's; a link, a non-file, a wrong mode or owner refuses. A Claude model refuses at
// every write and every read. A write is compare-and-set on revision, atomic, and appends one
// receipt line. Shaped as the Triage route store's tests are.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { chmodSync, existsSync, linkSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, renameSync, rmSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import {
CHOICE_FIELDS, CHOICE_VERSION, WorkerModelStoreError, choicePaths, claudeModel, readChoice, validChoice, writeChoice,
} from "../src/worker-model-store.mjs";
const MODULE = new URL("../src/worker-model-store.mjs", import.meta.url).pathname;
const NOW = "2026-11-21T12:00:00.000Z";
const LATER = "2026-11-21T13:00:00.000Z";
const at = (iso) => () => new Date(iso);
const INPUT = { provider: "zai", model: "glm-5.3-flash", by: "owner" };
function root(t) {
const dir = mkdtempSync(join(process.env.TMPDIR, "wms-"));
t.after(() => rmSync(dir, { recursive: true, force: true }));
return dir;
}
const reason = (r) => (e) => e instanceof WorkerModelStoreError && e.reason === r && e.code === undefined;
const unreadableFor = (re) => (e) => reason("unreadable")(e) && re.test(e.message);
const receipts = (dataRoot) => readFileSync(choicePaths(dataRoot).receipts, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l));
const good = () => ({ version: 1, revision: 1, provider: "zai", model: "glm-5.3-flash", updatedAt: NOW, updatedBy: "owner" });
// Write a doc straight to choice.json at the right modes, to test the loader alone.
function plant(dataRoot, doc) {
const { dir, file } = choicePaths(dataRoot);
mkdirSync(dir, { recursive: true, mode: 0o700 });
chmodSync(dir, 0o700);
writeFileSync(file, typeof doc === "string" ? doc : JSON.stringify(doc), { mode: 0o600 });
chmodSync(file, 0o600);
}
test("paths and field names each live in one place", () => {
assert.deepEqual(choicePaths("/d"), {
dir: "/d/worker-model", file: "/d/worker-model/choice.json", receipts: "/d/worker-model/choice-receipts.jsonl", lock: "/d/worker-model/choice.lock",
});
assert.deepEqual([...CHOICE_FIELDS].sort(), ["model", "provider", "revision", "updatedAt", "updatedBy", "version"]);
assert.equal(CHOICE_VERSION, 1);
});
test("no file reads as not set; a write then reads back the same choice", (t) => {
const d = root(t);
assert.equal(readChoice(d), null);
assert.deepEqual(writeChoice(d, INPUT, { now: at(NOW) }), good());
assert.deepEqual(readChoice(d), good());
const { dir, file } = choicePaths(d);
assert.equal(lstatSync(dir).mode & 0o777, 0o700);
assert.equal(lstatSync(file).mode & 0o777, 0o600);
// Names only: nothing shaped like an env value or a key.
assert.doesNotMatch(readFileSync(file, "utf8"), /ANTHROPIC|_KEY|TOKEN|msw-|mst-|msk-|env/i);
const second = writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, { now: at(LATER) });
assert.equal(second.revision, 2);
assert.equal(second.model, "glm-5");
assert.equal(second.updatedAt, LATER);
assert.deepEqual(readChoice(d), second);
assert.deepEqual(readdirSync(dir).sort(), ["choice-receipts.jsonl", "choice.json"]);
});
test("each write appends one receipt line with the old and the new value; a refused write appends none", (t) => {
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
writeChoice(d, { ...INPUT, model: "glm-5" }, { now: at(LATER) });
const lines = receipts(d);
assert.equal(lines.length, 2);
assert.deepEqual(lines[0], { at: NOW, by: "owner", revision: 1, from: null, to: { provider: "zai", model: "glm-5.3-flash" } });
assert.deepEqual(lines[1], { at: LATER, by: "owner", revision: 2, from: { provider: "zai", model: "glm-5.3-flash" }, to: { provider: "zai", model: "glm-5" } });
assert.equal(lstatSync(choicePaths(d).receipts).mode & 0o777, 0o600);
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(LATER) }), reason("stale"));
assert.equal(receipts(d).length, 2);
});
test("a stale revision refuses and leaves the file alone; revision 0 means only when none is set", (t) => {
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
const before = readFileSync(choicePaths(d).file, "utf8");
assert.throws(() => writeChoice(d, { ...INPUT, model: "glm-5", revision: 2 }, { now: at(LATER) }), reason("stale"));
assert.throws(() => writeChoice(d, { ...INPUT, model: "glm-5", revision: 0 }, { now: at(LATER) }), reason("stale"));
assert.equal(readFileSync(choicePaths(d).file, "utf8"), before);
assert.equal(writeChoice(root(t), { ...INPUT, revision: 0 }, { now: at(NOW) }).revision, 1);
});
test("the Claude rule: a Claude model refuses on write and reads as unreadable", (t) => {
assert.equal(claudeModel("anthropic", "anything"), true);
assert.equal(claudeModel("openrouter", "anthropic/Claude-Sonnet"), true);
assert.equal(claudeModel("zai", "claude-opus-5-5"), true);
assert.equal(claudeModel("zai", "glm-5.3-flash"), false);
const d = root(t);
for (const [provider, model] of [["anthropic", "glm-5"], ["zai", "claude-opus-5-5"], ["openrouter", "anthropic/CLAUDE-x"]]) {
assert.throws(() => writeChoice(d, { ...INPUT, provider, model }, { now: at(NOW) }),
(e) => reason("invalid")(e) && /Claude models run only in the Claude harness/.test(e.message), `${provider}/${model}`);
}
assert.equal(readChoice(d), null);
// A hand-written file naming a Claude model never reads as a choice.
for (const doc of [{ ...good(), provider: "anthropic" }, { ...good(), model: "claude-sonnet-5-5" }]) {
const e = root(t);
plant(e, doc);
assert.equal(validChoice(doc), false);
assert.throws(() => readChoice(e), unreadableFor(/does not match the choice shape/));
}
});
test("refused input shapes on a write", (t) => {
const d = root(t);
const bad = [
[{ ...INPUT, provider: "" }, /provider must be/], [{ ...INPUT, provider: "has space" }, /provider must be/],
[{ ...INPUT, provider: 5 }, /provider must be/], [{ ...INPUT, provider: "x".repeat(201) }, /provider must be/],
[{ ...INPUT, model: "" }, /model must be/], [{ ...INPUT, model: "../x y" }, /model must be/], [{ ...INPUT, model: null }, /model must be/],
[{ ...INPUT, by: "" }, /actor name/], [{ ...INPUT, by: "Owner Person" }, /actor name/],
[{ ...INPUT, revision: -1 }, /revision must be/], [{ ...INPUT, revision: "1" }, /revision must be/],
[{ ...INPUT, harness: "pi" }, /unknown field harness/], [{ ...INPUT, key: "msw-x" }, /unknown field key/],
[null, /not an object/], [[INPUT], /not an object/],
];
for (const [input, message] of bad) {
assert.throws(() => writeChoice(d, input, { now: at(NOW) }), (e) => reason("invalid")(e) && message.test(e.message), JSON.stringify(input));
}
assert.equal(readChoice(d), null);
});
test("each refused stored shape reads as unreadable", (t) => {
const bad = [
"not json", "[]", "null",
{ ...good(), version: 2 }, { ...good(), revision: 0 }, { ...good(), revision: 1.5 },
{ ...good(), provider: "" }, { ...good(), model: "a b" }, { ...good(), updatedAt: "2026-11-21" },
{ ...good(), updatedBy: "Owner" }, { ...good(), harness: "pi" },
(() => { const { updatedBy, ...rest } = good(); return rest; })(),
];
for (const doc of bad) {
const d = root(t);
plant(d, doc);
assert.throws(() => readChoice(d), reason("unreadable"), JSON.stringify(doc));
}
const d = root(t);
plant(d, `${JSON.stringify(good())}${" ".repeat(64 * 1024)}`);
assert.throws(() => readChoice(d), unreadableFor(/too large/));
});
test("a wrong mode on the directory or the file refuses", (t) => {
for (const mode of [0o755, 0o750, 0o711, 0o500]) {
const d = root(t);
plant(d, good());
chmodSync(choicePaths(d).dir, mode);
assert.throws(() => readChoice(d), reason("unreadable"), `dir ${mode.toString(8)}`);
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(NOW) }), reason("unreadable"), `dir ${mode.toString(8)}`);
chmodSync(choicePaths(d).dir, 0o700);
}
for (const mode of [0o644, 0o640, 0o604, 0o400, 0o700]) {
const d = root(t);
plant(d, good());
chmodSync(choicePaths(d).file, mode);
assert.throws(() => readChoice(d), reason("unreadable"), `file ${mode.toString(8)}`);
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(NOW) }), reason("unreadable"), `file ${mode.toString(8)}`);
}
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
chmodSync(choicePaths(d).receipts, 0o644);
assert.throws(() => writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, { now: at(LATER) }), unreadableFor(/choice-receipts\.jsonl mode is not 0600/));
assert.equal(readChoice(d).revision, 1);
});
test("owner only: a directory or file another user owns refuses on read and write", (t) => {
const d = root(t);
plant(d, good());
const other = process.getuid() + 1;
assert.throws(() => readChoice(d, { fileUid: other }), unreadableFor(/choice\.json is not owned by this user/));
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(NOW), fileUid: other }), unreadableFor(/choice\.json is not owned by this user/));
assert.throws(() => readChoice(d, { uid: other }), unreadableFor(/the worker-model directory is not owned by this user/));
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(NOW), uid: other }), unreadableFor(/the worker-model directory is not owned by this user/));
const e = root(t);
writeChoice(e, INPUT, { now: at(NOW) });
unlinkSync(choicePaths(e).file);
assert.throws(() => writeChoice(e, INPUT, { now: at(NOW), fileUid: other }), unreadableFor(/choice-receipts\.jsonl is not owned by this user/));
assert.equal(existsSync(choicePaths(e).file), false);
assert.equal(readChoice(d).revision, 1);
});
test("a link at the file, the directory, the receipts or the lock refuses; so does a non-file", (t) => {
let d = root(t);
plant(d, good());
const elsewhere = join(d, "elsewhere.json");
renameSync(choicePaths(d).file, elsewhere);
symlinkSync(elsewhere, choicePaths(d).file);
assert.throws(() => readChoice(d), unreadableFor(/choice\.json is a link/));
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(NOW) }), reason("unreadable"));
d = root(t);
plant(d, good());
linkSync(choicePaths(d).file, join(d, "hard.json"));
assert.throws(() => readChoice(d), unreadableFor(/more than one link/));
d = root(t);
const real = join(d, "real");
mkdirSync(real, { mode: 0o700 });
writeFileSync(join(real, "choice.json"), JSON.stringify(good()), { mode: 0o600 });
symlinkSync(real, choicePaths(d).dir);
assert.throws(() => readChoice(d), unreadableFor(/the worker-model directory is a link/));
assert.throws(() => writeChoice(d, INPUT, { now: at(NOW) }), unreadableFor(/the worker-model directory is a link/));
d = root(t);
mkdirSync(choicePaths(d).file, { recursive: true, mode: 0o700 });
chmodSync(choicePaths(d).dir, 0o700);
assert.throws(() => readChoice(d), unreadableFor(/choice\.json is not a regular file/));
d = root(t);
writeFileSync(choicePaths(d).dir, "x", { mode: 0o700 });
chmodSync(choicePaths(d).dir, 0o700);
assert.throws(() => readChoice(d), unreadableFor(/the worker-model directory is not a directory/));
d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
const target = join(d, "target.jsonl");
writeFileSync(target, "", { mode: 0o600 });
unlinkSync(choicePaths(d).receipts);
symlinkSync(target, choicePaths(d).receipts);
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(LATER) }), unreadableFor(/choice-receipts\.jsonl is a link/));
assert.equal(readFileSync(target, "utf8"), "");
d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
symlinkSync(join(d, "nowhere"), choicePaths(d).lock);
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(LATER) }), reason("busy"));
assert.equal(existsSync(join(d, "nowhere")), false);
});
test("a write under a narrow umask still leaves 0700 and 0600; the data root is never made", (t) => {
const d = root(t);
const old = process.umask(0o277);
let written;
try { written = writeChoice(d, INPUT, { now: at(NOW) }); } finally { process.umask(old); }
const { dir, file, receipts: r } = choicePaths(d);
assert.equal(lstatSync(dir).mode & 0o777, 0o700);
assert.equal(lstatSync(file).mode & 0o777, 0o600);
assert.equal(lstatSync(r).mode & 0o777, 0o600);
assert.deepEqual(readChoice(d), written);
assert.throws(() => writeChoice(join(d, "missing"), INPUT, { now: at(NOW) }), reason("unreadable"));
assert.equal(existsSync(join(d, "missing")), false);
});
test("a failed temp write refuses and leaves the choice; a failed receipt after the rename says so", (t) => {
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
const before = readFileSync(choicePaths(d).file, "utf8");
const fail = (stage) => (s) => { if (s === stage) throw Object.assign(new Error("no space"), { code: "ENOSPC" }); };
assert.throws(() => writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, { now: at(LATER), hook: fail("write") }), unreadableFor(/choice\.json cannot be written/));
assert.equal(readFileSync(choicePaths(d).file, "utf8"), before);
assert.deepEqual(readdirSync(choicePaths(d).dir).sort(), ["choice-receipts.jsonl", "choice.json"]);
assert.throws(() => writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, { now: at(LATER), hook: fail("after-rename") }),
(e) => reason("unrecorded")(e) && /changed to revision 2, but its receipt line was not written/.test(e.message));
assert.equal(readChoice(d).revision, 2);
assert.equal(receipts(d).length, 1);
});
test("two writers in one process: the second, while the first holds the lock, refuses busy", (t) => {
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
let inner = null;
const won = writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, {
now: at(LATER),
hook: (stage) => {
if (stage !== "before-rename") return;
try { writeChoice(d, { ...INPUT, model: "glm-4", revision: 1 }, { now: at(LATER) }); } catch (e) { inner = e; }
},
});
assert.equal(won.model, "glm-5");
assert.ok(reason("busy")(inner), String(inner));
assert.equal(readChoice(d).model, "glm-5");
});
// A child that loads the store and writes once; prints {ok, reason}. `wait` makes it hold at a
// stage until the file `go` appears, so two children overlap for real.
const CHILD = `
import { writeChoice } from ${JSON.stringify(MODULE)};
import { existsSync, writeFileSync } from "node:fs";
const [dataRoot, model, revision, wait, go, ready] = process.argv.slice(1);
try {
const r = writeChoice(dataRoot, { provider: "zai", model, by: "owner", revision: Number(revision) }, {
hook: (stage) => {
if (stage === "crash") { if (wait === "crash") process.kill(process.pid, "SIGKILL"); return; }
if (stage !== wait) return;
writeFileSync(ready, "");
const end = Date.now() + 20000;
while (!existsSync(go) && Date.now() < end) Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 20);
},
});
console.log(JSON.stringify({ ok: true, model: r.model }));
} catch (e) { console.log(JSON.stringify({ ok: false, reason: e.reason ?? String(e) })); }
`;
function child(args) {
return new Promise((resolve, reject) => {
const p = spawn(process.execPath, ["--input-type=module", "-e", CHILD, ...args], { stdio: ["ignore", "pipe", "pipe"] });
let out = "", err = "";
p.stdout.setEncoding("utf8").on("data", (c) => { out += c; });
p.stderr.setEncoding("utf8").on("data", (c) => { err += c; });
p.on("error", reject);
p.on("close", (status, signal) => resolve({ status, signal, out: out.trim(), err }));
});
}
async function until(path) {
const end = Date.now() + 20000;
while (!existsSync(path)) {
if (Date.now() > end) throw new Error(`timed out waiting for ${path}`);
await new Promise((r) => setTimeout(r, 20));
}
}
test("two writer processes racing on one revision: one wins, one refuses", async (t) => {
const d = root(t);
writeChoice(d, INPUT, { now: at(NOW) });
const go = join(d, "go"), ready = join(d, "ready");
const first = child([d, "glm-a", "1", "before-rename", go, ready]);
await until(ready);
const second = await child([d, "glm-b", "1", "none", join(d, "u0"), join(d, "u1")]);
writeFileSync(go, "");
assert.deepEqual(JSON.parse((await first).out), { ok: true, model: "glm-a" });
assert.deepEqual(JSON.parse(second.out), { ok: false, reason: "busy" });
assert.deepEqual(JSON.parse((await child([d, "glm-b", "1", "none", join(d, "u2"), join(d, "u3")])).out), { ok: false, reason: "stale" });
assert.equal(readChoice(d).revision, 2);
});
test("a crash between the temp write and the rename leaves the old file and a lock that names the dead writer", async (t) => {
const d = root(t);
const old = writeChoice(d, INPUT, { now: at(NOW) });
const r = await child([d, "glm-z", "1", "crash", join(d, "go"), join(d, "ready")]);
assert.equal(r.signal, "SIGKILL", r.err);
assert.deepEqual(readChoice(d), old);
const { dir, lock } = choicePaths(d);
assert.ok(readdirSync(dir).some((n) => n.endsWith(".tmp")), readdirSync(dir).join(","));
assert.throws(() => writeChoice(d, { ...INPUT, revision: 1 }, { now: at(LATER) }), (e) => reason("busy")(e) && /not running/.test(e.message) && /choice\.lock/.test(e.message));
unlinkSync(lock);
assert.equal(writeChoice(d, { ...INPUT, model: "glm-5", revision: 1 }, { now: at(LATER) }).revision, 2);
});
@@ -0,0 +1,86 @@
// worker-cutover: scripts/mosaic-task.mjs around the launch step. fixtures/worker-proxy-hook.mjs
// swaps in a stand-in launch step, so these cases check what mosaic-task does with its answer:
// a refusal's exact line and exit before any run directory, the key out of every run file and
// out of the terminal, modelKeyId in result.json, and a held signal's exit code. The last case runs
// src/container-cli.mjs (a hello.sh or verify.sh probe) the same way.
import test from 'node:test';
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { FAKE_KEY } from './fixtures/worker-proxy-fake.mjs';
import { UID_LINE } from '../src/worker-proxy.mjs';
const ROOT = fileURLToPath(new URL('../../../', import.meta.url));
const HOOK = fileURLToPath(new URL('./fixtures/worker-proxy-hook.mjs', import.meta.url));
function run(t, mode) {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wpt-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
const dataRoot = join(root, 'd'); mkdirSync(dataRoot, { mode: 0o700 });
const config = join(root, 'config.json');
writeFileSync(config, JSON.stringify({ configVersion: 1, environment: 'development', dataRoot, execution: { backend: 'docker', provider: 'zai', model: 'fixture-id' } }));
const taskFile = join(root, 'task.json');
writeFileSync(taskFile, JSON.stringify({ taskVersion: 1, id: 't-wp', prompt: 'hi', workspace: ':run', timeoutSeconds: 30 }));
const proc = spawnSync(process.execPath, ['--import', HOOK, join(ROOT, 'scripts', 'mosaic-task.mjs'), 'run', taskFile], {
cwd: ROOT, encoding: 'utf8', timeout: 60000,
env: { PATH: process.env.PATH, HOME: root, TMPDIR: process.env.TMPDIR ?? tmpdir(), MOSAIC_CONFIG: config, FAKE_WP: mode },
});
const runs = existsSync(join(dataRoot, 'runs')) ? readdirSync(join(dataRoot, 'runs')) : [];
// Every regular file under the run directory, by its path inside it.
const files = {};
const walk = (dir, at) => { for (const e of readdirSync(dir, { withFileTypes: true })) {
if (e.isDirectory()) walk(join(dir, e.name), `${at}${e.name}/`); else if (e.isFile()) files[`${at}${e.name}`] = readFileSync(join(dir, e.name), 'utf8');
} };
if (runs.length) walk(join(dataRoot, 'runs', runs[0]), '');
return { ...proc, runs, files };
}
test('a launch refusal prints its code and fixed line, exits 3 and leaves no run directory', t => {
const r = run(t, 'refuse');
assert.equal(r.status, 3, r.stderr);
assert.equal(r.stderr, 'mosaic-task: proxy-down: The model proxy is not running. Start it on Settings > Model setup.\n');
assert.deepEqual(r.runs, []);
});
test('the uid refusal prints its plain line alone', t => {
const r = run(t, 'uid');
assert.equal(r.status, 3); assert.equal(r.stderr, `${UID_LINE}\n`); assert.deepEqual(r.runs, []);
});
test('the worker key is out of stderr.txt, the response, the cause and the terminal; result.json has only its id', t => {
const r = run(t, 'leak');
assert.equal(r.status, 1, r.stderr);
assert.equal(r.runs.length, 1);
for (const [name, text] of Object.entries(r.files)) assert.equal(text.includes(FAKE_KEY), false, name);
assert.equal(r.stdout.includes(FAKE_KEY) || r.stderr.includes(FAKE_KEY), false);
assert.match(r.files['stderr.txt'], /error: the proxy rejected \[redacted\]/);
const result = JSON.parse(r.files['result.json']);
assert.equal(result.response, 'answer [redacted]');
assert.equal(result.modelKeyId, 'a1b2c3d4e5f6');
assert.match(r.stdout, /^cause: worker said: error: the proxy rejected /m);
});
test('a signal held while docker ran exits 128 plus its number after the record is written', t => {
const r = run(t, 'signal');
assert.equal(r.status, 143, r.stderr);
assert.equal(JSON.parse(r.files['result.json']).status, 'succeeded');
assert.match(r.stdout, /^status: succeeded$/m);
});
test('a container-cli probe writes the worker output with the key taken out', t => {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wpc-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
const dataRoot = join(root, 'd'); mkdirSync(dataRoot, { mode: 0o700 });
const config = join(root, 'config.json');
writeFileSync(config, JSON.stringify({ configVersion: 1, environment: 'development', dataRoot, execution: { backend: 'docker', provider: 'zai', model: 'fixture-id' } }));
const r = spawnSync(process.execPath, ['--import', HOOK, join(ROOT, 'packages', 'processing-control', 'src', 'container-cli.mjs')], {
cwd: ROOT, encoding: 'utf8', timeout: 60000,
env: { PATH: process.env.PATH, HOME: root, TMPDIR: process.env.TMPDIR ?? tmpdir(), MOSAIC_CONFIG: config, FAKE_WP: 'leak' },
});
assert.equal(r.status, 1, r.stderr);
assert.equal(r.stdout, 'answer [redacted]');
assert.equal(r.stderr, 'starting\nerror: the proxy rejected [redacted]\n');
});
@@ -0,0 +1,411 @@
// worker-cutover (S4 of reports/worker-proxy-simple.md): the one launch step every host worker
// launch takes. Scratch data roots and config dirs, a stand-in management client, a stub docker
// (a function in place of spawnSync) and, for the relay case, a stand-in proxy on 127.0.0.1 port 0
// in a child process. Nothing here reads a real config, proxy, secret or pi file, or runs Docker.
import test from 'node:test';
import assert from 'node:assert/strict';
import { spawn, spawnSync } from 'node:child_process';
import { randomUUID } from 'node:crypto';
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initializeControl } from '../src/state.mjs';
import { listRuntimes, registerRuntime, runtime } from '../src/runtimes.mjs';
import {
CONTAINER_MODELS_FILE, CONTAINER_PROXY_DIR, PROXY_DIR, UID_LINE, WORKER_UID, WorkerLaunchError,
checkModel, launchWorker, prepareWorker, redactSecrets, sweepStale,
} from '../src/worker-proxy.mjs';
const KEY = 'msw-0123456789ab-worker-' + 'S'.repeat(43);
const KEY_ID = 'a1b2c3d4e5f6';
const UPSTREAM = [{ 'base-url': 'https://api.z.ai/api/anthropic', keys: [{}], models: [{ name: 'fixture-id' }] }];
const LINE = {
'proxy-not-configured': 'The model proxy is not set up. Set it up on Settings > Model setup.',
'proxy-down': 'The model proxy is not running. Start it on Settings > Model setup.',
'model-harness-incompatible': 'Claude models run only in the Claude harness. Pick another model in the config.',
'proxy-key-refused': 'The proxy did not accept the worker key. Restart the proxy on Settings > Model setup, then run verify again.',
'proxy-lock-busy': 'Another proxy key change is running. Run verify again in a minute.',
'proxy-uncertain': 'A proxy key change did not finish. Restart the proxy, then run `mosaic proxy fence clear --proxy-restarted`.',
'model-catalogue-unavailable': 'The model proxy did not give a usable model list. Check it on Settings > Model setup, then run verify again.',
};
function fixture(t, { adapter = 'pi', provider = 'zai', model = 'fixture-id', port = 1 } = {}) {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wp-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
const dataRoot = join(root, 'd'), configDir = join(root, 'c');
mkdirSync(dataRoot, { mode: 0o700 }); mkdirSync(configDir, { mode: 0o700 });
initializeControl(dataRoot);
const system = { dataRoot, execution: { backend: 'docker', provider, model, adapter }, ...(port === null ? {} : { proxy: { port } }) };
return { root, dataRoot, configDir, system };
}
const client = (groups = async () => UPSTREAM) => ({
getCredentials: async () => ({ files: [] }), getCredentialModels: async () => ({ models: [] }), getUpstreamGroups: groups,
});
// Every seam a pi launch takes, each one counting its calls.
function seams(over = {}) {
const calls = { ensure: 0, read: 0, relay: 0, stop: 0, docker: 0, list: [], stopper: 0 };
const deps = {
uid: WORKER_UID,
client: client(),
dockerList: async args => { calls.list.push(args); return ''; },
dead: () => false,
ensureReady: async () => { calls.ensure++; return { keyId: KEY_ID }; },
readKey: () => { calls.read++; return { key: KEY, keyId: KEY_ID }; },
spawnRelay: async () => { calls.relay++; return { stop: async () => { calls.stop++; } }; },
docker: () => { calls.docker++; return { status: 0, signal: null, stdout: 'ok', stderr: '' }; },
stopper: async () => { calls.stopper++; return { outcome: 'stopped' }; },
...over,
};
return { calls, deps };
}
const refusal = (code, line) => error => {
assert.ok(error instanceof WorkerLaunchError, String(error));
assert.equal(error.code, code); assert.equal(error.line, line); assert.equal(error.exitCode, 3);
assert.equal(error.message.includes(KEY), false);
return true;
};
const launch = (f, deps, extra = {}) => launchWorker({
system: f.system, configDir: f.configDir, record: { kind: 'task', business: 'instance', run: 'r-fixture' },
worker: { kind: 'probe' }, spawn: { env: { PATH: process.env.PATH, ZAI_API_KEY: 'CALLER-ZAI', ANTHROPIC_API_KEY: 'CALLER-ANTHROPIC' } },
deps, ...extra,
});
const cname = () => `mosaic-${'0'.repeat(12)}-${randomUUID()}`;
const proxyDirs = f => { try { return readdirSync(join(f.dataRoot, PROXY_DIR)); } catch { return []; } };
// ---------- the failure table, each before a run directory, the key or docker ----------
test('the uid rule comes first and is a plain line with no code', async t => {
const f = fixture(t, { port: null, provider: 'anthropic' });
const { calls, deps } = seams({ uid: 1001 });
await assert.rejects(prepareWorker({ system: f.system, configDir: f.configDir, deps }), refusal(null, UID_LINE));
assert.equal(UID_LINE, 'Worker containers run as user 1000. Run Mosaic as that user.');
assert.equal(calls.ensure + calls.read + calls.docker, 0);
});
test('no proxy block refuses proxy-not-configured', async t => {
const f = fixture(t, { port: null });
const { calls, deps } = seams();
await assert.rejects(prepareWorker({ system: f.system, configDir: f.configDir, deps }), refusal('proxy-not-configured', LINE['proxy-not-configured']));
assert.equal(calls.ensure + calls.docker, 0);
});
test('harness rule: a Claude model never runs on pi, by provider or by model name, before the proxy is asked', async t => {
for (const [provider, model] of [['anthropic', 'fixture-id'], ['zai', 'claude-opus-5-5'], ['openrouter', 'anthropic/Claude-Sonnet']]) {
const f = fixture(t, { provider, model });
let asked = 0;
const { calls, deps } = seams({ client: client(async () => { asked++; return UPSTREAM; }) });
await assert.rejects(launch(f, deps), refusal('model-harness-incompatible', LINE['model-harness-incompatible']));
assert.equal(asked + calls.ensure + calls.relay + calls.docker, 0, `${provider}/${model}`);
assert.equal(listRuntimes(f.dataRoot).length, 0);
}
});
test('a config with no worker adapter refuses before the proxy is asked, and is not read as any harness', async t => {
for (const adapter of [undefined, '', 7]) {
const f = fixture(t);
if (adapter === undefined) delete f.system.execution.adapter; else f.system.execution.adapter = adapter;
let asked = 0;
const { calls, deps } = seams({ client: client(async () => { asked++; return UPSTREAM; }) });
await assert.rejects(launch(f, deps), refusal('processing-worker-launch-invalid', 'The config names no worker adapter.'));
assert.equal(asked + calls.ensure + calls.relay + calls.docker, 0, String(adapter));
assert.equal(listRuntimes(f.dataRoot).length, 0);
}
});
test('a proxy that is not running refuses proxy-down; an unusable answer refuses model-catalogue-unavailable', async t => {
const f = fixture(t);
const down = Object.assign(new Error('connect ECONNREFUSED'), { code: 'proxy-client-down' });
await assert.rejects(checkModel({ system: f.system, configDir: f.configDir, uid: WORKER_UID, client: client(async () => { throw down; }) }),
refusal('proxy-down', LINE['proxy-down']));
const odd = Object.assign(new Error('bad'), { code: 'proxy-client-response' });
await assert.rejects(checkModel({ system: f.system, configDir: f.configDir, uid: WORKER_UID, client: client(async () => { throw odd; }) }),
refusal('model-catalogue-unavailable', LINE['model-catalogue-unavailable']));
});
test('a management key that does not load refuses with its own code before any request', async t => {
const f = fixture(t);
await assert.rejects(checkModel({ system: f.system, configDir: f.configDir, uid: WORKER_UID }),
refusal('proxy-management-key-invalid', 'the proxy management key file is missing or unsafe'));
});
test('no account for the model refuses model-provider-unavailable and lists the models pi can use', async t => {
const f = fixture(t, { model: 'glm-5.3-flash' });
const { calls, deps } = seams();
await assert.rejects(prepareWorker({ system: f.system, configDir: f.configDir, deps }),
refusal('model-provider-unavailable', 'No account serves zai/glm-5.3-flash. Models you can use: zai/fixture-id. Add an account on Settings > Model setup, or set execution.model in the config.'));
assert.equal(calls.ensure, 0);
});
test('a model that serves pi passes the checks and returns its entry and transport', async t => {
const f = fixture(t);
const out = await checkModel({ system: f.system, configDir: f.configDir, uid: WORKER_UID, client: client() });
assert.equal(out.entry.provider, 'zai'); assert.equal(out.entry.model, 'fixture-id');
assert.ok(out.transport.piProvider);
});
test('key refusals from the key module go out with the table line, never its message', async t => {
for (const code of ['proxy-key-refused', 'proxy-lock-busy', 'proxy-uncertain']) {
const f = fixture(t);
const thrown = Object.assign(new Error(`detail ${KEY}`), { code, name: 'TriageKeyError' });
const { calls, deps } = seams({ ensureReady: async () => { throw thrown; } });
await assert.rejects(launch(f, deps), refusal(code, LINE[code]));
assert.equal(calls.docker + calls.relay, 0);
assert.equal(listRuntimes(f.dataRoot).length, 0);
}
// A rarer worker-key-* code keeps the key module's own fixed sentence.
const f = fixture(t);
const rare = Object.assign(new Error('the worker key file is unsafe'), { code: 'worker-key-unsafe', name: 'TriageKeyError' });
await assert.rejects(prepareWorker({ system: f.system, configDir: f.configDir, deps: seams({ ensureReady: async () => { throw rare; } }).deps }),
refusal('worker-key-unsafe', 'the worker key file is unsafe'));
});
// ---------- the key: read again on a keyId mismatch ----------
test('a keyId mismatch reads again, and three in a row refuse worker-key-raced', async t => {
const f = fixture(t);
let n = 0;
const { calls, deps } = seams({ readKey: () => { n++; return { key: KEY, keyId: n < 3 ? 'ffffffffffff' : KEY_ID }; } });
const ready = await prepareWorker({ system: f.system, configDir: f.configDir, deps });
assert.equal(ready.keyId, KEY_ID); assert.equal(ready.key, KEY); assert.equal(calls.ensure, 3);
const always = seams({ readKey: () => ({ key: KEY, keyId: 'ffffffffffff' }) });
await assert.rejects(prepareWorker({ system: f.system, configDir: f.configDir, deps: always.deps }),
refusal('worker-key-raced', 'The worker key changed during each of three reads. Run verify again.'));
assert.equal(always.calls.ensure, 3);
});
// ---------- the launch: key only in ANTHROPIC_API_KEY, two read-only mounts, cleanup ----------
test('a pi launch passes the key only in ANTHROPIC_API_KEY, mounts the dir and models.json read-only, then cleans up', async t => {
const f = fixture(t);
let seen = null;
const { calls, deps } = seams({
docker: (cmd, argv, opts) => {
calls.docker++;
const dir = proxyDirs(f);
const models = readFileSync(join(f.dataRoot, PROXY_DIR, dir[0], 'home', '.pi', 'agent', 'models.json'), 'utf8');
const record = listRuntimes(f.dataRoot)[0];
seen = { cmd, argv, env: opts.env, dir, models, mode: lstatSync(join(f.dataRoot, PROXY_DIR, dir[0])).mode & 0o777, phase: record.phase };
return { status: 0, signal: null, stdout: 'ok', stderr: '' };
},
});
const out = await launch(f, deps);
assert.equal(seen.cmd, 'docker');
assert.equal(seen.env.ANTHROPIC_API_KEY, KEY);
assert.equal('ZAI_API_KEY' in seen.env, false);
assert.equal(seen.env.MOSAIC_PROXY_PORT, '1');
assert.equal(seen.argv.join(' ').includes(KEY), false, 'the key is never an argument');
assert.equal(seen.models.includes(KEY), false, 'models.json names the variable, never the key');
assert.match(seen.models, /\$ANTHROPIC_API_KEY|ANTHROPIC_API_KEY/);
assert.equal(seen.mode, 0o700);
const dir = join(f.dataRoot, PROXY_DIR, seen.dir[0]);
const vols = seen.argv.filter((a, i) => seen.argv[i - 1] === '--volume');
assert.ok(vols.includes(`${dir}:${CONTAINER_PROXY_DIR}:ro`), vols.join('\n'));
assert.ok(vols.includes(`${join(dir, 'home', '.pi', 'agent', 'models.json')}:${CONTAINER_MODELS_FILE}:ro`), vols.join('\n'));
assert.equal(vols.some(v => /auth\.json/.test(v)), false, 'no pi auth file is mounted');
assert.equal(calls.relay, 1); assert.equal(calls.stop, 1);
assert.deepEqual(proxyDirs(f), [], 'the launch dir goes when docker returns');
assert.equal(out.modelKeyId, KEY_ID); assert.deepEqual(out.secrets, [KEY]); assert.equal(out.signal, null);
assert.equal(runtime(f.dataRoot, out.managed.id).phase, 'ended');
});
test('a mock launch takes no key, no relay and no dir, and still drops the caller\'s keys', async t => {
const f = fixture(t, { adapter: 'mock', port: null });
let env = null;
const { calls, deps } = seams({ docker: (cmd, argv, opts) => { env = opts.env; return { status: 0, signal: null, stdout: '', stderr: '' }; } });
const out = await launch(f, deps);
assert.equal(env.ANTHROPIC_API_KEY, ''); assert.equal('ZAI_API_KEY' in env, false); assert.equal(env.MOSAIC_PROXY_PORT, '');
assert.equal(calls.ensure + calls.read + calls.relay, 0);
assert.equal(out.modelKeyId, null); assert.deepEqual(out.secrets, []);
assert.equal(existsSync(join(f.dataRoot, PROXY_DIR)), false);
});
test('launchWorker refuses a pi launch given no prepared key', async t => {
const f = fixture(t);
await assert.rejects(launch(f, seams().deps, { prepared: null }), error => error.code === 'processing-worker-launch-invalid');
assert.equal(listRuntimes(f.dataRoot).length, 0);
});
// ---------- the record is ended on success, failure and signal ----------
const exits = [
['exit 0', { status: 0, signal: null }, '', 'ended', 0],
['exit 1, docker lists no container', { status: 1, signal: null }, '', 'ended', 0],
['exit 1, docker still lists it (stopped first)', { status: 1, signal: null }, 'abc\n', 'ended', 1],
['exit 1, docker cannot be asked (left for the sweep)', { status: 1, signal: null }, null, 'starting', 0],
['timeout (stopped first)', { status: null, signal: 'SIGKILL', error: Object.assign(new Error('t'), { code: 'ETIMEDOUT' }) }, '', 'ended', 1],
['buffer limit (stopped first)', { status: null, signal: 'SIGTERM', error: Object.assign(new Error('b'), { code: 'ENOBUFS' }) }, '', 'ended', 1],
['client killed by a signal (stopped first)', { status: null, signal: 'SIGKILL' }, '', 'ended', 1],
['docker missing (never started)', { status: null, signal: null, error: Object.assign(new Error('e'), { code: 'ENOENT' }) }, '', 'ended', 0],
];
for (const [name, proc, listed, phase, stops] of exits) {
test(`record after ${name}: ${phase}`, async t => {
const f = fixture(t);
const { calls, deps } = seams({
docker: () => ({ stdout: '', stderr: '', ...proc }),
dockerList: async () => { if (listed === null) throw new Error('daemon gone'); return listed; },
});
const out = await launch(f, deps);
assert.equal(runtime(f.dataRoot, out.managed.id).phase, phase);
assert.equal(calls.stopper, stops);
assert.equal(calls.stop, 1, 'the relay stops on every exit path');
assert.deepEqual(proxyDirs(f), []);
});
}
test('a stop whose outcome is unknown leaves the record open for the sweep', async t => {
const f = fixture(t);
const { deps } = seams({ docker: () => ({ status: null, signal: 'SIGKILL', stdout: '', stderr: '' }), stopper: async () => ({ outcome: 'unknown' }) });
const out = await launch(f, deps);
assert.equal(runtime(f.dataRoot, out.managed.id).phase, 'starting');
});
test('a signal while docker runs is held: the relay outlives docker, the record is ended, and the signal is returned', async t => {
const f = fixture(t);
let stoppedDuringDocker = null;
const { calls, deps } = seams({
docker: () => { process.emit('SIGTERM', 'SIGTERM'); stoppedDuringDocker = calls.stop; return { status: 143, signal: null, stdout: '', stderr: '' }; },
});
const before = process.listenerCount('SIGTERM');
const out = await launch(f, deps);
assert.equal(out.signal, 'SIGTERM'); assert.equal(stoppedDuringDocker, 0); assert.equal(calls.stop, 1);
assert.equal(runtime(f.dataRoot, out.managed.id).phase, 'ended');
assert.equal(process.listenerCount('SIGTERM'), before, 'the hold is released');
});
test('a signal before docker starts stops the launch with a plain line and leaves nothing', async t => {
const f = fixture(t);
const { calls, deps } = seams({ spawnRelay: async () => { calls.relay++; process.emit('SIGINT', 'SIGINT'); return { stop: async () => { calls.stop++; } }; } });
await assert.rejects(launch(f, deps), refusal(null, 'worker: the launch stopped on SIGINT before docker started.'));
assert.equal(calls.docker, 0); assert.equal(calls.stop, 1);
assert.equal(listRuntimes(f.dataRoot).length, 0); assert.deepEqual(proxyDirs(f), []);
});
test('a relay that does not start refuses with its own plain line and leaves nothing', async t => {
const f = fixture(t);
const { calls, deps } = seams({ spawnRelay: async () => { throw new Error('proxy relay: the socket path already exists.'); } });
await assert.rejects(launch(f, deps), refusal(null, 'proxy relay: the socket path already exists.'));
const odd = seams({ spawnRelay: async () => { throw new Error(`boom ${KEY}`); } });
await assert.rejects(launch(f, odd.deps), refusal(null, 'proxy relay: could not start.'));
assert.equal(calls.docker + odd.calls.docker, 0);
assert.equal(listRuntimes(f.dataRoot).length, 0); assert.deepEqual(proxyDirs(f), []);
});
// ---------- the killed-launcher sweep ----------
test('the sweep ends a dead launcher\'s record only when docker lists no container, and removes orphan dirs', async t => {
const f = fixture(t);
const gone = registerRuntime(f.dataRoot, { kind: 'task', business: 'instance', run: 'r-a', containerName: cname() });
const running = registerRuntime(f.dataRoot, { kind: 'task', business: 'instance', run: 'r-b', containerName: cname() });
const unasked = registerRuntime(f.dataRoot, { kind: 'task', business: 'instance', run: 'r-c', containerName: cname() });
const live = registerRuntime(f.dataRoot, { kind: 'task', business: 'instance', run: 'r-d', containerName: cname() });
const parent = join(f.dataRoot, PROXY_DIR); mkdirSync(parent, { mode: 0o700 });
const hex = id => id.replace(/-/g, '').slice(0, 12);
for (const name of [hex(gone.id), hex(live.id), 'ffffffffffff', 'not-a-launch']) mkdirSync(join(parent, name));
const dead = r => r.id !== live.id;
const dockerList = async args => {
const label = args[args.indexOf('--filter') + 1];
if (label.endsWith(running.id)) return 'abc\n';
if (label.endsWith(unasked.id)) throw new Error('daemon gone');
return '';
};
const out = await sweepStale(f.dataRoot, { dockerList, dead });
assert.deepEqual(out.ended, [gone.id]);
assert.equal(runtime(f.dataRoot, gone.id).phase, 'ended');
for (const r of [running, unasked, live]) assert.equal(runtime(f.dataRoot, r.id).phase, 'starting');
assert.deepEqual(readdirSync(parent).sort(), [hex(live.id), 'not-a-launch'].sort());
assert.deepEqual(out.removed.sort(), [hex(gone.id), 'ffffffffffff'].sort());
});
test('every launch sweeps first, before the key is read', async t => {
const f = fixture(t);
const stale = registerRuntime(f.dataRoot, { kind: 'task', business: 'instance', run: 'r-a', containerName: cname() });
let endedBeforeKey = null;
const { deps } = seams({ dead: r => r.id === stale.id, ensureReady: async () => { endedBeforeKey = runtime(f.dataRoot, stale.id).phase; return { keyId: KEY_ID }; } });
await prepareWorker({ system: f.system, configDir: f.configDir, deps });
assert.equal(endedBeforeKey, 'ended');
});
test('the sweep never throws on a data root with no runtimes and no proxy dir', async t => {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wp-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
assert.deepEqual(await sweepStale(join(root, 'missing'), { dockerList: async () => '' }), { ended: [], removed: [] });
});
// ---------- redaction ----------
test('redactSecrets removes every copy of the key and leaves other text', () => {
assert.equal(redactSecrets(`a ${KEY} b ${KEY}`, [KEY]), 'a [redacted] b [redacted]');
assert.equal(redactSecrets('plain', [KEY]), 'plain');
assert.equal(redactSecrets('plain', ['']), 'plain');
assert.equal(redactSecrets(undefined, [KEY]), '');
});
// ---------- the real relay against a stand-in proxy on port 0 ----------
test('the real relay carries a request from the launch socket to the proxy port and is gone after docker returns', async t => {
const proxy = spawn(process.execPath, ['-e', `
const http = require('node:http');
const s = http.createServer((q, r) => { r.end('proxy:' + q.url); });
s.listen(0, '127.0.0.1', () => process.stdout.write(String(s.address().port) + '\\n'));
process.stdin.on('end', () => process.exit(0)); process.stdin.resume();
`], { env: {}, stdio: ['pipe', 'pipe', 'inherit'] });
t.after(() => { proxy.stdin.end(); proxy.kill(); });
const port = await new Promise(r => proxy.stdout.once('data', d => r(Number(String(d).trim()))));
const f = fixture(t, { port });
let answer = null, relayPid = null;
const { deps } = seams({
spawnRelay: async opts => {
const { spawnRelay } = await import('../src/proxy-relay.mjs');
const relay = await spawnRelay(opts); relayPid = relay.child.pid; return relay;
},
docker: () => {
const sock = join(f.dataRoot, PROXY_DIR, proxyDirs(f)[0], 'proxy.sock');
const r = spawnSync(process.execPath, ['-e', `
const http = require('node:http');
http.get({ socketPath: process.argv[1], path: '/v1/models' }, res => { let b = ''; res.on('data', c => b += c); res.on('end', () => process.stdout.write(b)); });
`, sock], { encoding: 'utf8', timeout: 5000, env: {} });
answer = r.stdout;
return { status: 0, signal: null, stdout: '', stderr: '' };
},
});
// The relay child answers only once this process's event loop runs again, so the request runs
// in a child: spawnSync blocks this loop but not the relay process.
await launch(f, deps);
assert.equal(answer, 'proxy:/v1/models');
assert.throws(() => process.kill(relayPid, 0), error => error.code === 'ESRCH', 'the relay exited');
assert.deepEqual(proxyDirs(f), []);
});
// ---------- the adapter waits for exactly "ready" ----------
test('adapter.sh runs pi only after the forwarder prints exactly "ready"', t => {
const root = realpathSync(mkdtempSync(join(tmpdir(), 'wp-')));
t.after(() => rmSync(root, { recursive: true, force: true }));
const bin = join(root, 'bin'); mkdirSync(bin);
// Stand-in node: records its arguments and prints $STUB_WORD, as the forwarder prints its one word.
writeFileSync(join(bin, 'node'), '#!/bin/sh\nprintf "%s\\n" "$*" > "$STUB_ARGS"\nprintf "$STUB_WORD"\n', { mode: 0o755 });
writeFileSync(join(bin, 'pi'), '#!/bin/sh\necho "pi ran"\n', { mode: 0o755 });
const prompt = join(root, 'prompt.md'); writeFileSync(prompt, 'p');
const adapter = new URL('../../../adapters/pi/adapter.sh', import.meta.url).pathname;
const run = (word, port = '4100') => {
const args = join(root, `args-${Math.random()}`);
const r = spawnSync('/bin/sh', [adapter], { encoding: 'utf8', timeout: 10000, env: {
PATH: `${bin}:/usr/bin:/bin`, PI_PROVIDER: 'zai', PI_MODEL: 'fixture-id', MOSAIC_SYSTEM_PROMPT_FILE: prompt,
MOSAIC_REQUEST: 'hi', MOSAIC_PI_BIN: join(bin, 'pi'), STUB_WORD: word, STUB_ARGS: args, ...(port ? { MOSAIC_PROXY_PORT: port } : {}),
} });
return { ...r, args: existsSync(args) ? readFileSync(args, 'utf8') : null };
};
const ok = run('ready\\n');
assert.equal(ok.status, 0, ok.stderr); assert.equal(ok.stdout, 'pi ran\n');
assert.match(ok.args, /proxy-forward\.mjs \/run\/mosaic-proxy\/proxy\.sock 4100\n$/);
for (const word of ['readyx', 'ready more', 'not ready', '', 'READY']) {
const r = run(word);
assert.equal(r.status, 1, JSON.stringify(word)); assert.equal(r.stdout, '', JSON.stringify(word));
}
const none = run('readyx', '');
assert.equal(none.status, 0); assert.equal(none.stdout, 'pi ran\n'); assert.equal(none.args, null, 'no port, no forwarder');
});
+2 -1
View File
@@ -5,6 +5,7 @@
export { RunsError } from "./errors.mjs";
export { RUNS_DIRNAME, STATE_DIRNAME } from "./paths.mjs";
export {
RUN_ID_PATTERN, RUN_DOCUMENTS, isRunId, listRunIds, readRunDocument, listRunRecords, readRunRecord,
RUN_ID_PATTERN, RUN_DOCUMENTS, RUN_TEXTS, RUN_TEXT_TAIL, isRunId, listRunIds, readRunDocument, listRunRecords,
readRunRecord, readRunText,
} from "./runs.mjs";
export { POINTER_FILE, ACTIVATION_LOG_FILE, readActivePointer, readActivationLog } from "./state.mjs";
+37
View File
@@ -70,3 +70,40 @@ export function readRunRecord(dataRoot, runId) {
artifacts,
};
}
export const RUN_TEXTS = ["stderr.txt"];
export const RUN_TEXT_TAIL = 64 * 1024;
// The last RUN_TEXT_TAIL bytes of one of RUN_TEXTS from a run, as UTF-8 text, or null when it is
// missing, not a regular file, unreadable or outside the data root. A cut that lands inside a
// line drops that partial first line.
export function readRunText(dataRoot, runId, name) {
requireRunId(runId);
if (!RUN_TEXTS.includes(name)) throw new RunsError(`unknown run text: ${JSON.stringify(name)}`);
let fd;
try {
const file = resolveInside(dataRoot, RUNS_DIRNAME, runId, name);
if (file === null) return null;
fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW);
const stat = fs.fstatSync(fd);
if (!stat.isFile()) return null;
const length = Math.min(stat.size, RUN_TEXT_TAIL);
const buffer = Buffer.alloc(length);
let read = 0;
while (read < length) {
const n = fs.readSync(fd, buffer, read, length - read, stat.size - length + read);
if (n === 0) break;
read += n;
}
let text = buffer.subarray(0, read).toString("utf8");
if (stat.size > length) {
const newline = text.indexOf("\n");
text = newline < 0 ? "" : text.slice(newline + 1);
}
return text;
} catch {
return null;
} finally {
if (fd !== undefined) fs.closeSync(fd);
}
}
+27 -1
View File
@@ -3,7 +3,7 @@ import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import {
RunsError, isRunId, listRunIds, listRunRecords, readRunDocument, readRunRecord,
RUN_TEXT_TAIL, RunsError, isRunId, listRunIds, listRunRecords, readRunDocument, readRunRecord, readRunText,
} from "../src/index.mjs";
import { RESULT, scratch, tree, write } from "./helpers.mjs";
@@ -241,3 +241,29 @@ test("the readers write nothing", (t) => {
readRunDocument(dataRoot, A, "mission.json");
assert.equal(tree(dataRoot), before);
});
test("readRunText reads a run's stderr.txt, its last RUN_TEXT_TAIL bytes from a line start", (t) => {
const { dataRoot } = scratch(t);
write(path.join(dataRoot, "runs", A, "stderr.txt"), "one\ntwo\n");
assert.equal(readRunText(dataRoot, A, "stderr.txt"), "one\ntwo\n");
const long = `${"x".repeat(RUN_TEXT_TAIL)}\nlast line\n`;
write(path.join(dataRoot, "runs", B, "stderr.txt"), long);
assert.equal(readRunText(dataRoot, B, "stderr.txt"), "last line\n");
});
test("readRunText is null for a missing run, a directory or a file linked out of the data root", (t) => {
const { dataRoot, outside } = scratch(t);
assert.equal(readRunText(dataRoot, A, "stderr.txt"), null);
fs.mkdirSync(path.join(dataRoot, "runs", A, "stderr.txt"), { recursive: true });
assert.equal(readRunText(dataRoot, A, "stderr.txt"), null);
write(path.join(outside, "secret.txt"), "outside\n");
fs.mkdirSync(path.join(dataRoot, "runs", B), { recursive: true });
fs.symlinkSync(path.join(outside, "secret.txt"), path.join(dataRoot, "runs", B, "stderr.txt"));
assert.equal(readRunText(dataRoot, B, "stderr.txt"), null);
});
test("readRunText reads only stderr.txt and refuses an invalid run id", (t) => {
const { dataRoot } = scratch(t);
assert.throws(() => readRunText(dataRoot, A, "result.json"), (e) => e instanceof RunsError && /unknown run text/.test(e.message));
assert.throws(() => readRunText(dataRoot, "../x", "stderr.txt"), (e) => e instanceof RunsError && /invalid run id/.test(e.message));
});
+16
View File
@@ -0,0 +1,16 @@
export const setupState = Object.freeze([
{
id: 'tasks.created', title: 'Give your team a first task',
purpose: 'Write down one piece of work so your chief has something to plan.',
fix: { page: '#/tasks', label: 'New task' },
order: { after: ['assistant.replied'] }, required: true,
async check(ctx) {
const answers = await Promise.allSettled(ctx.members.map(org => ctx.read(org, 'tasks')));
const tasks = answers.filter(a => a.status === 'fulfilled' && Array.isArray(a.value));
const missed = answers.length - tasks.length;
if (tasks.some(a => a.value.length)) return { state: 'done', reason: 'Your team has a task recorded.' + (missed ? " Some organizations couldn't be checked. Open Tasks to check them." : '') };
if (missed) return { state: 'next', reason: "Some organizations couldn't be checked. Open Tasks to check them or create your team's first task." };
return { state: 'next', reason: 'Create a task with one piece of work for your team.' };
},
},
]);
+14
View File
@@ -0,0 +1,14 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { setupState } from '../src/setup-state.mjs';
test('first task is evidence of work recorded, not work running', async () => {
const check = tasks => setupState[0].check({ members: [{ id: 'team' }], read: async () => tasks });
assert.equal((await check([])).state, 'next');
assert.equal((await check([{ id: 1, state: 'todo' }])).state, 'done');
assert.equal((await setupState[0].check({ members: [] })).state, 'next');
assert.match((await check(null)).reason, /couldn't be checked/);
assert.match((await setupState[0].check({ members: [{}], read: async () => { throw Error(); } })).reason, /couldn't be checked/);
});
test('a task in an answering organization counts when another cannot answer', async () => {
assert.equal((await setupState[0].check({ members: [{ id: 'bad' }, { id: 'good' }], read: async org => { if (org.id === 'bad') throw Error(); return [{}]; } })).state, 'done');
});
+23 -23
View File
@@ -1,4 +1,5 @@
#!/usr/bin/env node
import { setupRoutes } from './setup-http.mjs';
import { assertBuildStart, assertDevInstance, assertDeployed } from '../../cli/src/deployment.mjs';
// MOSAIC_TESTER_SOCKET_V1: private tester ingress, no tester TCP.
import { createChat } from './chat.mjs';
@@ -44,7 +45,8 @@ import { loadProxy } from '../../model-proxy/src/settings.mjs';
import { composeSnapshot, readSystemConfig, readOrgConfig } from './triage-snapshot.mjs';
import { busesStopped } from '../../cli/src/proxy-admin-host.mjs';
import { agentLevelsWriteRoutes } from './agent-levels-write.mjs';
import { triageRouteRoutes } from './triage-route-http.mjs';
import { installedHarnesses, triageRouteRoutes } from './triage-route-http.mjs';
import { readExecution, workerModelReader } from './worker-model-view.mjs';
import { keyState, ridingOrgs, triageKeyCall } from './triage-access.mjs';
import { readKey as readTriageKey } from '../../bus/src/triage-key-file.mjs';
import { RESTART_EXIT } from '../../cli/src/org-console.mjs';
@@ -251,6 +253,18 @@ try {
try { proxy = Number.isInteger(loadProxy(configDir(), system.dataRoot)?.port); } catch {}
return ownerModelsFrom({ proxy, riders: proxy ? await rideRiders() : [] });
} : null;
// Each active organization's bus `routes` catalogue, every one asked at once, so a hung bus costs
// one timeout, not one each. A bus that does not answer gives a catalogue that is not ready, so it
// confirms nothing. Triage's route checks and the Worker model card read the same list.
const routeCatalogues = async () => {
const read = async call => { try { return (await call('routes'))?.catalogue ?? { status: 'unknown' }; } catch { return { status: 'bus-unavailable' }; } };
if (!organizations) return ownerCall ? [await read(ownerCall)] : [];
const active = (await listOrgs({ configDir: configDir(), dataRoot: system.dataRoot, instance })).filter(org => org.state === 'active');
return Promise.all(active.map(org => {
let paths; try { paths = orgPaths(org.root); } catch { return { status: 'bus-unavailable' }; }
return read(busCaller({ dataRoot: org.root, socket: paths.socket, business: org.id, timeoutMs: 5000 }));
}));
};
const proxyAdmin = system && (devOwner || ssoOn) ? createProxyAdmin({
dataRoot: system.dataRoot, configDir: configDir(), instance, unitDir: options.proxyUnitDir ?? null,
buses: () => busesStopped({ dataRoot: system.dataRoot, configDir: configDir() }),
@@ -272,6 +286,9 @@ try {
// A stopped fixture organization moves through the organization service's whole-file update, the
// write `POST /api/orgs/ID/admin` makes, as the signed-in owner. The owner starts its bus after.
move: (id, models, principal) => moveFixtureOrg({ service: orgService, ctx: { configDir: configDir(), dataRoot: system.dataRoot, instance, principal }, id, models }),
// worker-model-setup: Settings > Model setup's Worker model card (read only), in the status read.
// The config is read again each time, as a worker launch reads it, so a config edit shows here.
workerModel: workerModelReader({ readExecution: () => readExecution({ instance }), dataRoot: system.dataRoot, catalogues: routeCatalogues, harnesses: () => installedHarnesses() }),
}) : null;
// #1873 triage-host: the instance owner's Triage session, hosted by the Console in no organization.
// Nothing starts here; the owner's first message starts it.
@@ -284,16 +301,7 @@ try {
};
const triageRoute = system && (devOwner || ssoOn) ? createTriageRoute({
dataRoot: system.dataRoot, configDir: configDir(), proxyPort: triageProxyPort,
catalogues: async () => {
// A bus that does not answer gives a catalogue that is not ready, so it confirms nothing.
const read = async call => { try { return (await call('routes'))?.catalogue ?? { status: 'unknown' }; } catch { return { status: 'bus-unavailable' }; } };
if (!organizations) return ownerCall ? [await read(ownerCall)] : [];
const active = (await listOrgs({ configDir: configDir(), dataRoot: system.dataRoot, instance })).filter(org => org.state === 'active');
return Promise.all(active.map(org => {
let paths; try { paths = orgPaths(org.root); } catch { return { status: 'bus-unavailable' }; }
return read(busCaller({ dataRoot: org.root, socket: paths.socket, business: org.id, timeoutMs: 5000 }));
}));
},
catalogues: routeCatalogues,
}) : null;
const triage = triageRoute ? createTriageHost({
dataRoot: system.dataRoot, route: triageRoute, proxyPort: triageProxyPort, tokenPattern: triageTokenPattern(configDir()),
@@ -309,7 +317,8 @@ try {
return busCaller({ dataRoot: org.root, socket: orgPaths(org.root).socket, business: id, timeoutMs: 5000 })('processing.status');
},
orgConfig: id => readOrgConfig(id),
proxy: () => proxyAdmin.status(),
// The snapshot keeps fixed proxy fields only, so it skips the Worker model card's view.
proxy: () => proxyAdmin.status({ worker: false }),
systemConfig: () => readSystemConfig(),
}),
}) : null;
@@ -349,19 +358,10 @@ try {
key: op => { const deadline = Date.now() + 30000; return triageKeyCall({ riders: () => rideRiders({ timeoutMs: 30000, deadline }), op }); },
stop: () => (triage ? triage.stop() : Promise.resolve()),
},
catalogues: async () => {
const read = async call => { try { return (await call('routes'))?.catalogue ?? { status: 'unknown' }; } catch { return { status: 'bus-unavailable' }; } };
if (!organizations) return ownerCall ? [await read(ownerCall)] : [];
// Every active organization is asked at once, so a hung bus costs one timeout, not one each.
const active = (await listOrgs({ configDir: configDir(), dataRoot: system.dataRoot, instance })).filter(org => org.state === 'active');
return Promise.all(active.map(org => {
let paths; try { paths = orgPaths(org.root); } catch { return { status: 'bus-unavailable' }; }
return read(busCaller({ dataRoot: org.root, socket: paths.socket, business: org.id, timeoutMs: 5000 }));
}));
},
catalogues: routeCatalogues,
})] : [];
try {
server = await startServer({ restart, trail, ...(lan ? { host: lan.bind, devAllowHosts: lan.allowHosts } : {}), board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, librarySync, libraryRevert, librarySave, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, proxyAdmin, triage, processing, decisionWrite, requestRead, requestWrite, userOptions, federation: system?.federation ?? null, organizations, agentLevels, ownerModels, rideRemoved: proxyAdmin ? id => proxyAdmin.rideRemoved(id) : null, routes: [...agentLevelRoutes, ...triageRoutes], preselect: options.business ?? null });
server = await startServer({ restart, trail, ...(lan ? { host: lan.bind, devAllowHosts: lan.allowHosts } : {}), board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, librarySync, libraryRevert, librarySave, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, proxyAdmin, triage, processing, decisionWrite, requestRead, requestWrite, userOptions, federation: system?.federation ?? null, organizations, agentLevels, ownerModels, rideRemoved: proxyAdmin ? id => proxyAdmin.rideRemoved(id) : null, routes: [setupRoutes({ organizations, system, dataRoot: system?.dataRoot, configDir: configDir(), instance, unitDir: options.proxyUnitDir ?? null }), ...agentLevelRoutes, ...triageRoutes], preselect: options.business ?? null });
} catch (err) { sso?.close(); throw err; }
if (testerSocket) {
const call = busCaller({ dataRoot: system.dataRoot, socket: socketPath(system.dataRoot), business: options.business ?? null, cli: TESTER_CLI });
+66 -15
View File
@@ -25,8 +25,27 @@
// E2E6 P1-1: a poll reloads this state; the last save's receipt and the open review list stay.
// A failed poll keeps the last good read only while the page holds the form (dirty, or a field has
// focus), so the held form is not dropped (review P3). A clean form shows the read error (re-check P3).
const held=org=>{const form=[...document.querySelectorAll('[data-autonomy-org]')].find(n=>n.dataset.autonomyOrg===org)?.querySelector('[data-autonomy-form]');return !!form&&(form.hasAttribute('data-live-dirty')||form.contains(document.activeElement));};
async function load(org) {const prev=states.get(org),keep={status:prev?.status??'',reviewOpen:prev?.reviewOpen===true};try{states.set(org,{...await call(org,''),...keep});}catch(error){states.set(org,prev&&!prev.error&&held(org)?{...prev,...keep}:{error:error.message,...keep});}}
// The settings section the page draws now for this organization (a decision notice carries the org too).
const live=org=>[...document.querySelectorAll('[data-autonomy-org]')].find(n=>n.dataset.autonomyOrg===org&&!n.hasAttribute('data-autonomy-decision'));
const held=org=>{const form=live(org)?.querySelector('[data-autonomy-form]');return !!form&&(form.hasAttribute('data-live-dirty')||form.contains(document.activeElement));};
// Each organization keeps one state object for the life of the page. A save or review keeps a reference
// to it across its own request, so a load that ends meanwhile writes its read into that object in place
// and never swaps it: the receipt lands where the next draw reads it. A load that began before the last
// applied read, or before a save answered, carries older values, so its answer is dropped.
const own=['status','reviewOpen','began','shown'];
function put(s,read) {for(const k of Object.keys(s))if(!own.includes(k))delete s[k];for(const [k,v]of Object.entries(read))if(!own.includes(k))s[k]=v;}
const stateOf=org=>{if(!states.has(org))states.set(org,{error:'Autonomy could not be read.',status:'',reviewOpen:false,began:0,shown:0});return states.get(org);};
async function load(org) {
const s=stateOf(org),began=++s.began;let read,failed=false;
try{read=await call(org,'');}catch(error){read={error:error.message};failed=true;}
if(began<s.shown)return;
s.shown=began;
if(failed&&!s.error&&held(org))return;
put(s,read);
}
// A result said on this organization's status line: kept in its state for the next draw, and written to
// the line the page draws now.
function say(org,text) {const s=stateOf(org);s.status=text;const line=live(org)?.querySelector('[data-autonomy-status]');if(line)line.textContent=text;}
function section(org) {
const s=states.get(org);
if(!s || s.error)return `<section class="panel" data-autonomy><h3>Autonomy</h3><p>${esc(s?.error??'Autonomy could not be read.')}</p></section>`;
@@ -42,22 +61,52 @@
<p data-autonomy-status role="status">${esc(s.status)}</p></form>
${s.writable?`<details data-autonomy-review-list${s.reviewOpen?' open':''}><summary${fk('review-list')}>Review reversals before moving out of the loop</summary><p>Read the decision trail and its reversals before recording your review.</p>${classes.map(k=>`<button class="btn" type="button" data-autonomy-review="${k}"${fk(`review-${k}`)}>Record review: ${labels[k]}</button>`).join('')}</details>`:''}</section>`;
}
// Decision notices on /inbox/<id>. The bus page draws that view again on every poll, and a decision the
// owner confirmed, returned or reversed leaves the inbox read, so its notice is gone from the next draw.
// The owner's result for each decision is kept here, in page memory, and the notice is drawn from it:
// the result line stays, a call in flight keeps its buttons off, and a notice whose decision has left
// the read is drawn with its result line only. A decision that comes back into the read after a draw
// saw it gone (an arbiter resolving it again) is a new question, so its old result is dropped.
const answers=new Map(),answerKey=(org,id)=>JSON.stringify([org,id]);
const noticeNode=(org,id)=>[...document.querySelectorAll('[data-autonomy-decision]')].find(n=>n.dataset.autonomyDecision===id&&n.dataset.autonomyOrg===org);
function notice(d) {
if(!d?.autonomy)return '';
const a=d.autonomy,org=d.business??window.MosaicOrg?.id;
return `<section class="panel" data-autonomy-decision="${esc(d.id)}" data-autonomy-org="${esc(org??'')}"><h2>${a.pending?'Confirm the arbiter resolution':'Decision near the loop'}</h2>
${a.pending?'<p>The arbiter resolved this decision. It has no effect until you confirm.</p><label>Reason for returning to the arbiter<input data-autonomy-reason maxlength="500"></label><p><button class="btn" data-autonomy-confirm="yes">Confirm resolution</button> <button class="btn" data-autonomy-confirm="no">Return to arbiter</button></p>':`<p>The agent resolved this decision. Reverse is available until ${esc(a.reverseUntil)}. A reverse cannot undo messages, work already done or code already shipped.</p><button class="btn" data-autonomy-reverse>Reverse decision</button>`}
<p role="status" data-autonomy-status></p></section>`;
if(!d)return '';
const org=d.business??window.MosaicOrg?.id??'',k=answerKey(org,d.id);
let kept=answers.get(k);
if(kept&&!kept.busy){if(!d.autonomy)kept.gone=true;else if(kept.gone){answers.delete(k);kept=undefined;}}
const line=`<p role="status" data-autonomy-status>${esc(kept?.text??'')}</p>`;
const open=`<section class="panel" data-autonomy-decision="${esc(d.id)}" data-autonomy-org="${esc(org)}">`;
if(!d.autonomy)return kept?`${open}<h2>${esc(kept.title)}</h2>${line}</section>`:'';
const a=d.autonomy,off=kept?.busy?' disabled':'';
return `${open}<h2>${a.pending?'Confirm the arbiter resolution':'Decision near the loop'}</h2>
${a.pending?`<p>The arbiter resolved this decision. It has no effect until you confirm.</p><label>Reason for returning to the arbiter<input data-autonomy-reason maxlength="500"></label><p><button class="btn" data-autonomy-confirm="yes"${off}>Confirm resolution</button> <button class="btn" data-autonomy-confirm="no"${off}>Return to arbiter</button></p>`:`<p>The agent resolved this decision. Reverse is available until ${esc(a.reverseUntil)}. A reverse cannot undo messages, work already done or code already shipped.</p><button class="btn" data-autonomy-reverse${off}>Reverse decision</button>`}
${line}</section>`;
}
// A notice's result: kept for the next draw, and written to the notice the page draws now.
function told(org,id,text) {
const k=answerKey(org,id);answers.set(k,{...answers.get(k),busy:false,text});
const node=noticeNode(org,id);if(!node)return;
const line=node.querySelector('[data-autonomy-status]');if(line)line.textContent=text;
for(const b of node.querySelectorAll('[data-autonomy-confirm],[data-autonomy-reverse]'))b.disabled=false;
}
document.addEventListener('submit',async e=>{
if(!e.target.matches('[data-autonomy-form]'))return;e.preventDefault();
const form=e.target,section=form.closest('[data-autonomy-org]'),org=section.dataset.autonomyOrg,s=states.get(org),fields=new FormData(form);
const form=e.target,org=form.closest('[data-autonomy-org]').dataset.autonomyOrg,s=states.get(org),fields=new FormData(form);
const autonomy=Object.fromEntries(classes.map(k=>[k,fields.get(k)]));autonomy.notices=fields.has('notices');autonomy.reverseWindowHours=Number(fields.get('reverseWindowHours'));
if(fields.get('authorityResolver'))autonomy.authorityResolver=fields.get('authorityResolver');
const button=form.querySelector('button[type="submit"]'),status=form.querySelector('[data-autonomy-status]');button.disabled=true;
const button=form.querySelector('button[type="submit"]');button.disabled=true;
let base=s.revision;try{if(form.dataset.revision)base=JSON.parse(form.dataset.revision);}catch{/* drawn without one */}
try{const r=await call(org,'',{revision:base,autonomy});s.revision=r.revision;s.autonomy=autonomy;s.levels=Object.fromEntries(classes.map(k=>[k,autonomy[k]]));s.status=`Saved. Receipt ${r.transaction}.`;form.dataset.revision=JSON.stringify(r.revision);delete form.dataset.liveDirty;status.textContent=s.status;}
catch(error){s.status=error.message;status.textContent=s.status;}finally{button.disabled=false;}
// A load that began before the save answered is dropped. A load that ended during the save read either
// the revision the save sent, from before it committed, or one at or after the save. One that is neither
// was written after the save by someone else: it stays, and the save's values do not replace it.
// A redraw during the save may have drawn a new form from an older read. A clean new form is drawn again
// from the state, which now holds the newest read known, so it shows that and saves against its revision.
// A new form the owner is editing stays: it saves against its own revision and fails closed.
try{const r=await call(org,'',{revision:base,autonomy}),sent=JSON.stringify(base);s.shown=++s.began;
if([sent,JSON.stringify(r.revision)].includes(JSON.stringify(s.revision))){s.revision=r.revision;s.autonomy=autonomy;s.levels=Object.fromEntries(classes.map(k=>[k,autonomy[k]]));}
form.dataset.revision=JSON.stringify(r.revision);delete form.dataset.liveDirty;s.status=`Saved. Receipt ${r.transaction}.`;
const el=live(org),drawn=el?.querySelector('[data-autonomy-form]');if(drawn&&drawn!==form&&!held(org))el.outerHTML=section(org);say(org,s.status);}
catch(error){say(org,error.message);}finally{button.disabled=false;}
});
// An edited form is dirty until it saves: holdLive keeps a dirty form through every poll.
const dirty=e=>{const form=e.target.closest?.('[data-autonomy-form]');if(form)form.dataset.liveDirty='1';};
@@ -67,12 +116,14 @@
document.addEventListener('toggle',e=>{if(!e.target.matches?.('[data-autonomy-review-list]'))return;const s=states.get(e.target.closest('[data-autonomy-org]')?.dataset.autonomyOrg);if(s)s.reviewOpen=e.target.open;},true);
document.addEventListener('click',async e=>{
const button=e.target.closest('[data-autonomy-confirm],[data-autonomy-reverse],[data-autonomy-review]');if(!button)return;
const section=button.closest('[data-autonomy-org]'),org=section.dataset.autonomyOrg,status=section.querySelector('[data-autonomy-status]');
const section=button.closest('[data-autonomy-org]'),org=section.dataset.autonomyOrg,id=section.dataset.autonomyDecision;
const review=button.dataset.autonomyReview,reverse=button.hasAttribute('data-autonomy-reverse');
const body=review?{decisionClass:review}:reverse?{id:section.dataset.autonomyDecision}:{id:section.dataset.autonomyDecision,confirm:button.dataset.autonomyConfirm==='yes',...(button.dataset.autonomyConfirm==='no'?{reason:section.querySelector('[data-autonomy-reason]').value}:{})};
const body=review?{decisionClass:review}:reverse?{id}:{id,confirm:button.dataset.autonomyConfirm==='yes',...(button.dataset.autonomyConfirm==='no'?{reason:section.querySelector('[data-autonomy-reason]').value}:{})};
button.disabled=true;
try{const r=await call(org,review?'review':reverse?'reverse':'confirm',body);status.textContent=review?'Trail review recorded.':reverse?(r.restart||r.trailPending?'Reversal committed. Completion needs recovery: inspect the decision trail and the stopped agent before requesting relaunch.':'Reversal recorded.'):(body.confirm?'Confirmation recorded. Refresh the inbox for current state.':'Returned to arbiter. Refresh the inbox for current state.');}
catch(error){status.textContent=error.message;}finally{button.disabled=false;}
// A notice takes one answer at a time: both of its buttons stay off until this one is told.
if(!review){const k=answerKey(org,id);answers.set(k,{text:'',...answers.get(k),title:section.querySelector('h2')?.textContent??'',busy:true});for(const b of section.querySelectorAll('[data-autonomy-confirm],[data-autonomy-reverse]'))b.disabled=true;}
try{const r=await call(org,review?'review':reverse?'reverse':'confirm',body);if(review)return say(org,'Trail review recorded.');told(org,id,reverse?(r.restart||r.trailPending?'Reversal committed. Completion needs recovery: inspect the decision trail and the stopped agent before requesting relaunch.':'Reversal recorded.'):(body.confirm?'Confirmation recorded. Refresh the inbox for current state.':'Returned to arbiter. Refresh the inbox for current state.'));}
catch(error){if(review)say(org,error.message);else told(org,id,error.message);}finally{button.disabled=false;}
});
window.MosaicAutonomy={load,section,notice};
})();
+3
View File
@@ -751,6 +751,9 @@
return `<li class="s1-opt"><p><b class="s1-key">${txt(o.key)}</b> ${txt(o.text)} ${approving ? badge(`approves ${d.authorization.action}`, 'attn') : ''}${o.key === d.recommendation ? badge('recommended', 'ok') : ''}${chosen ? badge('chosen', 'ok') : ''}</p>${copy}</li>`;
}).join('')}</ol>`;
const earlier = await chain(get, d);
// A newer draw may have begun while the earlier versions were read. This one is dropped, so stop
// here: the notice below keeps the owner's results, and only a draw that lands may change them.
if (!current()) return '';
const history = [...d.seen.map(s => `<li>Seen by ${txt(s.by)}${s.via ? ` via ${txt(s.via)}` : ''} ${when(s.at)}</li>`), ...(c ? [`<li>${esc(c.op)} by ${txt(c.by)}${c.via ? ` via ${txt(c.via)}` : ''} ${when(c.at)}</li>`] : [])];
// Without SSO, or under SSO without --send, the Console says why it has no buttons and never
// draws a disabled one (design section 5).
+106 -24
View File
@@ -597,6 +597,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
+ '<details id="proxy-setup-lines" class="proxy-setup-lines" hidden><summary>The service\'s last lines</summary><div class="inside" data-log></div></details>'
+ '<div id="proxy-setup-fixtures" class="banner mk-note" hidden></div>';
const accounts = '<p id="proxy-message" role="status" aria-live="polite">Reading accounts…</p>'
+ '<p class="small muted">Model setup has not checked whether a model answers.</p>'
+ '<p id="proxy-readonly" class="muted" hidden>This Console is read-only. Sign-in and account changes are unavailable.</p>'
+ '<div data-accounts-part><ul id="proxy-accounts" class="rows" aria-label="Accounts"></ul>'
// #1873 account-remove: the instance owner removes an account here; anyone else is told where.
@@ -610,8 +611,8 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
+ '<div class="form-foot"><button type="submit" id="proxy-login-go" hidden>Sign in</button></div></fieldset></form>'
+ '<form id="proxy-apikey" class="settings-form" hidden><fieldset data-write disabled><div class="field"><label for="proxy-apikey-key" id="proxy-apikey-label">API key</label>'
+ '<input id="proxy-apikey-key" name="key" type="password" autocomplete="off" spellcheck="false" maxlength="4096" required>'
+ '<p class="help small muted">Mosaic checks it with <span id="proxy-apikey-provider">the provider</span>, keeps it in your model service, and never shows it again.</p></div>'
+ '<div class="form-foot"><button type="submit">Check and add</button><button type="button" id="proxy-apikey-cancel">Cancel</button></div></fieldset><p id="proxy-apikey-message" role="status"></p></form>'
+ '<p class="help small muted">Mosaic saves the key for <span id="proxy-apikey-provider">the provider</span> in your model service and never shows it again. Saving does not check whether a model answers.</p></div>'
+ '<div class="form-foot"><button type="submit">Save account</button><button type="button" id="proxy-apikey-cancel">Cancel</button></div></fieldset><p id="proxy-apikey-message" role="status"></p></form>'
+ '</div></details>'
+ '<div id="proxy-signin" hidden><ol id="proxy-signin-progress" class="progress" aria-label="Sign-in progress"></ol>'
+ '<p class="form-foot"><a id="proxy-authorize" target="_blank" rel="noopener noreferrer" hidden>Open the sign-in tab again</a><button type="button" id="proxy-login-cancel" disabled>Cancel sign-in</button></p>'
@@ -627,6 +628,12 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
+ '<button type="button" id="proxy-ride-go" class="btn">Ride all</button><p id="proxy-ride-message" role="status" aria-live="polite"></p>'
+ '<ul id="proxy-ride-skips" class="proxy-ride-list"></ul><ul id="proxy-ride-fixtures" class="proxy-ride-list"></ul></div>';
const triage = '<p><a id="proxy-setup-triage" class="btn" href="#/settings/triage">Set up Triage</a></p>';
// worker-model-setup: card 5, the model worker runs use, from the status read's `worker` view
// (worker-model-view.mjs). It lists only models a worker can use. Saving a choice here needs an
// owner-only save route that is not built, so the card says how to change the model today.
const worker = '<p id="proxy-worker-using"></p><p id="proxy-worker-standing"></p><p id="proxy-worker-choice" class="small muted" hidden></p>'
+ '<h4 id="proxy-worker-list-title">Models worker runs can use</h4><ul id="proxy-worker-models" class="rows" aria-labelledby="proxy-worker-list-title"></ul>'
+ '<p id="proxy-worker-how" class="small muted"></p>';
const advanced = '<details id="proxy-setup-advanced" data-admin hidden><summary>Advanced: programs, port, timeouts</summary><div class="inside">'
// settings-s5c: the installed programs and which organizations have each one on, from the status
// read; one program can be turned off in every organization that can do without it.
@@ -667,19 +674,22 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
+ card('accounts', 2, 'Accounts and models', accounts, false)
+ card('organizations', 3, 'Organizations', organizations, true)
+ card('triage', 4, 'Triage', triage, true)
+ card('worker', 5, 'Worker model', worker, true)
+ advanced + tech + '</div>\n';
const STEP_KEYS = ['service', 'accounts', 'organizations', 'triage'];
const STEP_KEYS = ['service', 'accounts', 'organizations', 'triage', 'worker'];
const BADGE = { done: ['badge-ok', 'Done'], next: ['badge-attn', 'Next'], wait: ['badge-muted', 'Waiting'], work: ['badge-attn', 'Working'], bad: ['badge-danger', 'Needs you'], off: ['badge-muted', 'Not yet'] };
const CARD_CLASS = { done: 'done', next: 'next', work: 'next', bad: 'blocked', wait: 'waiting', off: 'waiting' };
// An account's health from its status in the accounts read. Anything else reads Unknown.
const HEALTH = { ready: ['badge-ok', 'Ready'], active: ['badge-ok', 'Ready'], error: ['badge-danger', 'Not working'], unavailable: ['badge-danger', 'Not available'], disabled: ['badge-muted', 'Off'] };
const health = status => HEALTH[status] ?? ['badge-muted', 'Unknown'];
// Account metadata does not prove that a model answered.
const HEALTH = { ready: ['badge-muted', 'Saved'], active: ['badge-muted', 'Saved'], error: ['badge-danger', 'Error'], unavailable: ['badge-danger', 'Not available'], disabled: ['badge-muted', 'Disabled'] };
const health = account => ['error', 'unavailable'].includes(account.status) && account.reason?.startsWith('The account needs to sign in again.')
? ['badge-danger', 'Sign-in needed'] : HEALTH[account.status] ?? ['badge-muted', 'Unknown'];
function mount() {
const $ = id => document.getElementById(id);
const root = $('proxy-setup');
let canWrite = false, login = null, poll = null, accountList = null, accountsDown = null;
const busy = new Set(), priorityReceipts = new Map(), opened = new Set();
const wanted = new URLSearchParams(location.hash.split('?')[1] ?? '').get('step');
const wantedParam = new URLSearchParams(location.hash.split('?')[1] ?? '').get('step');
const wanted = ({ 1: 'service', 2: 'accounts' })[wantedParam] ?? wantedParam;
if (STEP_KEYS.includes(wanted)) opened.add(wanted);
let scrolled = false;
const message = (text, loginOnly = false) => { $(loginOnly ? 'proxy-login-message' : 'proxy-message').textContent = text; };
@@ -713,7 +723,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
const shown = body.hidden; body.hidden = !shown; if (shown) opened.add(key); else opened.delete(key);
toggle.setAttribute('aria-expanded', String(shown)); toggle.textContent = shown ? 'Hide' : 'Show'; toggle.setAttribute('aria-label', `${shown ? 'Hide' : 'Show'} ${box.querySelector('h3').textContent}`);
}));
// ?step=service|accounts|organizations|triage opens that card and scrolls to it once.
// ?step=service|accounts|organizations|triage|worker opens that card and scrolls to it once.
function focusStep() {
if (scrolled || !STEP_KEYS.includes(wanted)) return;
const box = $(`setup-step-${wanted}`); if (!box || box.hidden || box.dataset.state === undefined) return;
@@ -739,6 +749,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
function paintAccountCard() {
const addBox = $('proxy-add-box');
addBox.hidden = !canWrite || Boolean(login);
$('proxy-accounts').querySelectorAll('[data-account-fix]').forEach(button => { button.disabled = Boolean(login); });
// The page has one primary button: here while no account is added and step 1 is done.
const first = canWrite && !login && accountList !== null && !accountList.length && !accountsDown && (!setup.enabled() || setup.serviceDone());
for (const b of [$('proxy-login-go'), $('proxy-apikey').querySelector('button[type=submit]')]) b.className = first ? 'btn primary' : 'btn';
@@ -746,10 +757,10 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
if (accountsDown) { setCard('accounts', 'off', accountsDown, { badge: ['badge-muted', 'Not available'] }); return focusStep(); }
if (accountList === null) { setCard('accounts', 'work', 'Reading accounts…', { badge: ['badge-muted', 'Reading'] }); return focusStep(); }
if (login) { setCard('accounts', 'work', `Signing in to ${providerShort(login.provider)}.`); return focusStep(); }
const bad = accountList.filter(a => ['error', 'unavailable'].includes(a.status));
const bad = accountList.filter(a => ['error', 'unavailable', 'disabled'].includes(a.status));
if (!accountList.length) { if (canWrite) addBox.open = true; setCard('accounts', 'next', 'Add an account once. Every organization that uses your models gets its models.'); }
else if (bad.length) setCard('accounts', 'bad', `${bad.map(a => `${providerLabel(a.provider)}: ${health(a.status)[1].toLowerCase()}`).join(' · ')}. Its models are not offered until it works. Add the account again under Add an account.`);
else setCard('accounts', 'done', accountList.map(a => `${providerLabel(a.provider)} ${health(a.status)[1].toLowerCase()}`).join(' · '));
else if (bad.length) setCard('accounts', 'bad', `${bad.map(a => `${providerLabel(a.provider)}: ${health(a)[1].toLowerCase()}`).join(' · ')}. Check the account rows below.`);
else setCard('accounts', 'done', accountList.map(a => `${providerLabel(a.provider)} ${health(a)[1].toLowerCase()}`).join(' · '));
focusStep();
}
// #1873 account-remove (Jason 2026-11-20): the owner's Remove asks in place and names what moves,
@@ -764,7 +775,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
'proxy-uncertain': 'Mosaic could not confirm the remove: the model service is between writes. Check the service and the list before trying again.',
'proxy-down': 'The model service stopped answering, so Mosaic could not confirm the remove. Check the service and the list before trying again.',
'proxy-admin-busy': 'Another setup action is running. Try again when it finishes.',
'invalid-request': 'The model service no longer lists this account. Read the accounts again.',
'invalid-request': 'Mosaic could not identify one account to remove. It may no longer be listed, have a duplicate ID, or share a credential with another account. Read the accounts again.',
'human-required': 'Only the instance owner removes an account.',
};
async function adminCall(path, args) {
@@ -809,8 +820,10 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
removeBusy = true; removeAsk.error = null; paintAccounts();
try {
await adminCall('accounts/remove', { accountId: account.accountId });
removeAsk = null; message(`${name} account removed.`);
await refresh(); await setup.reread();
let removalMessage = `${name} account removed.`;
removeAsk = null; message(removalMessage);
try { await refresh(); } catch (error) { removalMessage += ` The account list could not be refreshed: ${error.message}`; message(removalMessage); }
await setup.reread({ onError: error => message(`${removalMessage} The setup status could not be refreshed: ${error.message}`) });
} catch (error) {
if (removeAsk) removeAsk.error = error.message; message(error.message);
// The outcome may be unknown: read the list again so it shows what the service holds now.
@@ -843,7 +856,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
list.replaceChildren(...accountList.map(account => {
const li = el('li'), name = el('span', providerShort(account.provider)), kind = el('span', providerKind(account.provider) || '—');
kind.append(el('br'), el('small', account.displayName, 'muted'));
const [cls, word] = health(account.status), badge = el('span', word, `badge ${cls}`);
const [cls, word] = health(account), badge = el('span', word, `badge ${cls}`);
li.dataset.account = account.accountId; li.append(name, kind, badge);
if (proxyAdmin && canWrite) {
const remove = el('button', 'Remove'); remove.type = 'button'; remove.dataset.remove = account.accountId;
@@ -853,6 +866,19 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
}
// The accounts read says why an account is not working (account.reason, a7362a8fc and later). Absent, nothing shows.
if (typeof account.reason === 'string' && account.reason) li.append(el('p', account.reason, 'small muted account-reason'));
if (['error', 'unavailable', 'disabled'].includes(account.status)) {
if (canWrite && [...providerSelect.options].some(option => option.value === account.provider)) {
const fix = el('button', 'Add account'); fix.type = 'button'; fix.dataset.accountFix = '';
fix.disabled = Boolean(login);
fix.setAttribute('aria-label', `Add account for ${account.displayName}`);
fix.addEventListener('click', () => {
const add = $('proxy-add-box'); add.open = true;
providerSelect.value = account.provider;
paintProvider(); (keyed(providerSelect.value) ? $('proxy-apikey-key') : $('proxy-login-go')).focus();
});
li.append(fix);
} else li.append(el('p', canWrite ? 'This provider is not available here. Ask the owner to check Model setup.' : 'Ask the owner to add an account in Model setup.', 'small muted'));
}
if (removeAsk?.accountId === account.accountId && proxyAdmin && canWrite) li.append(removeBox(account));
return li;
}));
@@ -920,7 +946,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
'models-offered-conflict': 'The model list changed since this page read it. Read it again, then make your change.',
'models-invalid': 'The model service no longer lists one of these models. Read the list again.',
'proxy-down': 'No organization that uses your model service answered, so its models can\'t be listed now.',
'not-configured': 'No organization uses your model service yet. Its models show here once one does.',
'not-configured': 'No organization with a configured model launch uses your model service yet. Its models show here once one does.',
'proxy-admin-busy': 'Another change to the model service is running. Try again in a moment.',
'outcome-unknown': 'The outcome is unknown. Read the list again before you try again.',
};
@@ -937,8 +963,10 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
catch (error) { if (round === offeredRound) box.replaceChildren(el('p', offeredSays(error.code), 'small muted')); return; }
if (round !== offeredRound) return;
const providers = Array.isArray(view?.providers) ? view.providers : [];
if (!providers.length) { box.replaceChildren(el('p', 'Your model service lists no models yet.', 'small muted')); return; }
box.replaceChildren(...providers.map(p => offeredList(p, view.revision, open.has(p.provider), saved === p.provider)));
const quiet = Array.isArray(view?.unanswered) ? view.unanswered.filter(id => typeof id === 'string').map(id => window.MosaicOrg?.row?.(id)?.label || id) : [];
const notice = quiet.length ? [el('p', `Not answering now, so it is not known which models they use: ${quiet.join(', ')}. A model you switch off may stop working for them.`, 'offered-unanswered banner ref')] : [];
if (!providers.length) { box.replaceChildren(...notice, el('p', 'Your model service lists no models yet.', 'small muted')); return; }
box.replaceChildren(...notice, ...providers.map(p => offeredList(p, view.revision, open.has(p.provider), saved === p.provider)));
}
function offeredList(p, revision, open, saved) {
const name = String(p.label), models = Array.isArray(p.models) ? p.models : [];
@@ -1367,7 +1395,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
const PROGRESS = [['fetch', { ok: 'Downloaded and checked', run: 'Downloading and checking…', todo: 'Download and check', bad: 'Download and check: did not finish' }],
['init', { ok: 'Configured', run: 'Configuring…', todo: 'Configure', bad: 'Configure: did not finish' }],
['start', { ok: 'Started the service', run: 'Starting the service…', todo: 'Start the service', bad: 'Start the service: did not finish' }],
['answer', { ok: 'Answers on this machine', run: 'Checking it answers…', todo: 'Check it answers', bad: 'Check it answers: no answer yet' }]];
['answer', { ok: 'Service reachable on this machine', run: 'Checking the service…', todo: 'Check the service', bad: 'Service not reachable yet' }]];
const stageOf = step => step === 'fetch' || step === 'init' ? step : step === 'unit-install' || step === 'unit-start' ? 'start' : 'answer';
// Each line is done only when the status read says so.
function progressDone() {
@@ -1400,7 +1428,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
const next = state.next?.[0], box = $('proxy-setup-next'), lines = $('proxy-setup-lines');
const ok = progressDone(), some = Object.values(ok).some(Boolean), fetching = state.fetch?.state === 'running';
paintFixtures(next !== 'accounts');
if (next === 'accounts') { setCard('service', 'done', state.unit.port ? `Running. Answers on this machine, port ${state.unit.port}.` : 'Running. Answers on this machine.'); box.replaceChildren(); paintProgress(false); lines.hidden = true; return; }
if (next === 'accounts') { setCard('service', 'done', state.unit.port ? `Running. Service reachable on this machine, port ${state.unit.port}.` : 'Running. Service reachable on this machine.'); box.replaceChildren(); paintProgress(false); lines.hidden = true; return; }
if (chain?.running || (fetching && CHAIN.includes(next))) {
setCard('service', 'work', chain?.running ? 'Setting up. Keep this page open; if you leave, Continue setup picks up from here.' : 'Downloading the release. Keep this page open.');
box.replaceChildren(goButton('Working…', next, true, () => {})); paintProgress(true); lines.hidden = true; return;
@@ -1539,6 +1567,60 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
setCard('triage', nextUp ? 'next' : 'off', nextUp ? 'Next: choose the model Triage uses. Triage is the assistant that answers questions about this instance.' : 'Triage is the assistant that answers questions about this instance. Choose its model on the Triage page.');
if (nextUp) link.className = 'btn primary';
}
// Card 5, Worker model: the status read's `worker` view. Every name on it comes from that read.
// The config wins over a choice made here, and the config must name a model today (the schema
// requires execution.model), so the card never says a choice here takes effect.
function paintWorkerCard() {
const box = $('setup-step-worker'); if (!box) return;
const v = state?.worker;
const using = $('proxy-worker-using'), standing = $('proxy-worker-standing'), choice = $('proxy-worker-choice');
const list = $('proxy-worker-models'), how = $('proxy-worker-how');
choice.hidden = true; choice.textContent = '';
if (!v || typeof v !== 'object') {
using.textContent = 'The worker model could not be read.'; standing.textContent = ''; list.replaceChildren(); how.textContent = '';
setCard('worker', 'off', 'The worker model could not be read.', { badge: ['badge-muted', 'Unknown'] }); return;
}
const id = (p, m) => `${p ?? '(none)'}/${m ?? '(none)'}`;
const harness = v.harness ?? 'the worker harness';
// Why the list is not known: no organization to ask, the model service off in each one, or
// a list that could not be read (a bus or the proxy down).
const unread = v.catalogue === 'none' ? 'No organization is active' : v.catalogue === 'disabled' ? 'No organization has the model service on' : 'No organization\'s model list could be read';
how.textContent = v.canSave === true ? '' : 'Choosing the worker model on this page is not built yet. To change it, edit execution.provider and execution.model in the config.';
list.replaceChildren(...(Array.isArray(v.models) && v.models.length
? v.models.map(m => { const li = el('li'); li.append(el('strong', id(m.provider, m.model))); if (m.label && m.label !== id(m.provider, m.model)) li.append(' ', el('span', m.label, 'small muted')); return li; })
: [el('li', !v.takesModel ? 'None. This adapter takes no model.' : v.catalogue === 'ready' ? `None. No account serves a model for ${harness} that is switched on and is not a Claude model.` : `Not known. ${unread}.`, 'muted')]));
if (!v.takesModel && v.harness === 'mock') {
using.textContent = 'Worker runs use the mock adapter, which takes no model.'; standing.textContent = '';
setCard('worker', 'done', 'Worker runs use the mock adapter, which takes no model.'); return;
}
if (!v.using) {
using.textContent = 'No worker model is set.'; standing.textContent = 'Worker runs refuse to start until execution.provider and execution.model are set in the config.';
setCard('worker', 'bad', 'No worker model is set. Set execution.provider and execution.model in the config.'); return;
}
const name = id(v.using.provider, v.using.model);
using.replaceChildren(...(v.using.from === 'config'
? ['Worker runs use ', el('strong', name), '. The config names it (execution.provider and execution.model), and the config always wins.']
: ['Worker runs use ', el('strong', name), ', the choice saved for Model setup, because the config names no worker model.']));
// [card kind, the card's one line after the name, the line in the body]
const STANDING = {
usable: ['done', '', `An account serves it for ${harness}, and it is switched on.`],
off: ['bad', 'It is switched off.', 'It is switched off, so worker runs refuse to start. Switch it on under Accounts and models, Models offered, or name another model in the config.'],
'not-served': ['bad', `No account serves it for ${harness}.`, `No account serves it for ${harness} now, so worker runs refuse to start.`],
'claude-model': ['bad', 'It is a Claude model.', 'It is a Claude model. Claude models run only in the Claude harness, so worker runs refuse to start.'],
unknown: ['off', 'Whether it can run is not known.', `${unread}, so whether it can run now is not known.`],
unset: ['bad', 'The config names only half of it.', 'The config names only one of execution.provider and execution.model, so worker runs cannot start.'],
};
const [kind, short, line] = STANDING[v.using.standing] ?? ['off', 'Whether it can run is not known.', 'Whether it can run now is not known.'];
standing.textContent = line;
if (v.choice?.state === 'set' && v.using.from === 'config') {
choice.hidden = false;
choice.textContent = `A choice saved earlier (${id(v.choice.provider, v.choice.model)}) has no effect while the config names a worker model.`;
} else if (v.choice?.state === 'unreadable') {
choice.hidden = false;
choice.textContent = v.using.from === 'config' ? 'A choice saved earlier cannot be read. It has no effect while the config names a worker model.' : 'A choice saved earlier cannot be read.';
}
setCard('worker', kind, short ? `Worker runs use ${name}. ${short}` : `Worker runs use ${name}.`, kind === 'off' ? { badge: ['badge-muted', 'Unknown'] } : {});
}
function paintRestart() {
const box = $('proxy-setup-restart'), list = $('proxy-setup-restart-list');
for (const o of needRestart()) listed.add(o.id);
@@ -1872,7 +1954,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
};
$('proxy-ride-go').addEventListener('click', () => { if (!$('proxy-ride-go').disabled) rideAll(); });
function paint() {
paintState(); paintControls(); paintService(); paintRestart(); paintOrgsCard(); paintTriageCard(); paintAccountCard(); paintPrograms();
paintState(); paintControls(); paintService(); paintRestart(); paintOrgsCard(); paintTriageCard(); paintWorkerCard(); paintAccountCard(); paintPrograms();
const job = state.fetch;
if (job.state === 'running') say(`Downloading: ${STEPS[job.step] ?? job.step ?? 'starting'}…`);
else if (job.state === 'failed' && seenFetch !== 'failed') { say(`Fetch failed. ${explain(job.error)}`, 'error'); seenFetch = 'failed'; }
@@ -1884,10 +1966,10 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
focusStep();
}
// True once the answer is painted, so a caller can say the list on the page is the new one.
async function load() {
async function load({ onError } = {}) {
if (!enabled || loading) return false; loading = true; let painted = false;
try { state = await admin('status'); setProviders(state.providers); causeRound++; if (state.next?.[0] === 'accounts' && triageSet === null) await readTriage(); paint(); painted = true; if (state.init.initialized) await accountsReady(); }
catch (error) { say(error.message, 'error'); if (providerKey === null) setProviders(null); }
catch (error) { say(error.message, 'error'); if (providerKey === null) setProviders(null); onError?.(error); }
finally { loading = false; }
return painted;
}
@@ -1962,7 +2044,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
start(config) { enabled = true; sendOk = config.proxyAdminSend === true; return load(); },
accounts(n) { accountCount = n; if (state) { paintService(); paintRestart(); paintOrgsCard(); paintTriageCard(); } },
// A sign-in, a key or a restart can change the next step: read the whole status again.
async reread() { if (!enabled) return; triageSet = null; await load(); },
async reread(options) { if (!enabled) return; triageSet = null; await load(options); },
enabled: () => enabled,
serviceDone: () => running(),
// An organization whose model list is ready, for Models offered. The open one first.
+1
View File
@@ -7,6 +7,7 @@
<link rel="stylesheet" href="/processing.css"><script src="/processing.js" defer></script>
<link rel="stylesheet" href="/org-create.css"><script src="/org-create.js" defer></script>
<link rel="stylesheet" href="/onboarding.css"><script src="/onboarding.js" defer></script><link rel="stylesheet" href="/assistant.css">
<link rel="stylesheet" href="/setup-checklist.css"><script src="/setup-checklist.js" defer></script>
<script src="/workspace-data.js" defer></script><script src="/workspace.js" defer></script><script src="/tasks-board.js" defer></script><script src="/console-restart.js" defer></script><script src="/triage-settings.js" defer></script><script src="/display.js" defer></script><script src="/inbox.js" defer></script><script src="/files.js" defer></script><script src="/project-remote.js" defer></script><link rel="stylesheet" href="/autonomy.css"><script src="/autonomy.js" defer></script><link rel="stylesheet" href="/packs.css"><script src="/packs.js" defer></script><link rel="stylesheet" href="/workflows.css"><script src="/workflows.js" defer></script><script src="/federation.js" defer></script><script src="/brand.js" defer></script><script src="/app.js" defer></script><script src="/markdown-preview.js" defer></script><script src="/projects.js" defer></script><script src="/chat.js" defer></script><script src="/project-archive.js" defer></script><script src="/agents.js" defer></script><script src="/assistant.js" defer></script><script src="/bus.js" defer></script><script src="/requests.js" defer></script></head>
<body>
<div hidden aria-hidden="true"><select id="palette"></select><select id="mode"><option value="system">system</option><option value="light">light</option><option value="dim">dim</option><option value="dark">dark</option></select></div>
+17 -9
View File
@@ -62,21 +62,29 @@
}
// E2E6 P1-1, P2-2: a poll reloads the list but keeps the open preview, the project picked for each
// pack and the last receipt. A kept preview applies by its digest, so a stale one is refused by the bus.
// Each organization has one state object for the life of the page. A click keeps a reference to it
// across its own request, so a load that ends meanwhile updates the read fields in place and never
// swaps the object: the click's preview or receipt lands where the next draw reads it. A load that
// began before the last applied one carries older reads, so its answer is dropped.
async function load(org) {
const prev = states.get(org);
const s = { packs: null, applied: [], writable: false, error: null, projects: null, preview: prev?.preview ?? null, status: prev?.status ?? '', picks: prev?.picks ?? {} };
if (!states.has(org)) states.set(org, { packs: null, applied: [], writable: false, error: 'Starter packs could not be read.', projects: null, preview: null, status: '', picks: {}, began: 0, shown: 0 });
const s = states.get(org);
const began = ++s.began;
const read = { packs: null, applied: [], writable: false, error: null, projects: null };
try {
const value = await call(`${base(org)}/packs`);
s.packs = Array.isArray(value?.packs) ? value.packs.filter(p => p && typeof p === 'object' && p.pack && typeof p.pack === 'object') : [];
s.applied = Array.isArray(value?.applied) ? value.applied.filter(a => plainObj(a) && typeof a.id === 'string' && Array.isArray(a.hired) && Array.isArray(a.open)) : [];
s.writable = value?.writable === true;
} catch (error) { s.error = error.message; }
read.packs = Array.isArray(value?.packs) ? value.packs.filter(p => p && typeof p === 'object' && p.pack && typeof p.pack === 'object') : [];
read.applied = Array.isArray(value?.applied) ? value.applied.filter(a => plainObj(a) && typeof a.id === 'string' && Array.isArray(a.hired) && Array.isArray(a.open)) : [];
read.writable = value?.writable === true;
} catch (error) { read.error = error.message; }
// A project pack needs a code project. The workflow read lists them; when it fails, a project pack
// says so instead of guessing.
if (s.packs?.some(p => p.pack.scope === 'project')) {
try { const w = await call(`${base(org)}/workflows`); s.projects = w && typeof w.projects === 'object' ? Object.keys(w.projects).sort() : null; } catch { s.projects = null; }
if (read.packs?.some(p => p.pack.scope === 'project')) {
try { const w = await call(`${base(org)}/workflows`); read.projects = w && typeof w.projects === 'object' ? Object.keys(w.projects).sort() : null; } catch { read.projects = null; }
}
states.set(org, s);
if (began < s.shown) return;
s.shown = began;
Object.assign(s, read);
}
const markText = m => MARKS[m] ?? m;
function itemRow(item) {
@@ -0,0 +1,69 @@
/* State-derived setup checklist. Geometry follows the approved setup frames. */
.setup-card { max-width: 940px; background: var(--surface); border: 1px solid var(--line); border-radius: 10px; overflow: hidden; margin-bottom: 16px; }
.setup-head { padding: 16px 20px 14px; border-bottom: 1px solid var(--line); }
.setup-head .setup-where-am-i { font-size: 11px; letter-spacing: .06em; color: var(--action); font-weight: 700; margin: 0 0 4px; }
.setup-head h2 { margin: 0; font-size: 20px; color: var(--strong); line-height: 1.4; }
.setup-head p { margin: 6px 0 0; color: var(--muted); }
.setup-bar { display: flex; gap: 4px; margin-top: 12px; width: 320px; max-width: 100%; }
.setup-bar i { flex: 1; height: 6px; border-radius: 3px; background: var(--raised); }
.setup-bar .on { background: var(--success); }.setup-bar .cur { background: var(--action); }.setup-bar .bad { background: var(--danger); }
.checklist-steps { list-style: none; margin: 0; padding: 0; }
.checklist-step { display: grid; grid-template-columns: 32px minmax(0, 1fr) auto; column-gap: 14px; align-items: start; padding: 10px 20px; border-bottom: 1px solid color-mix(in srgb, var(--line) 70%, transparent); }
.checklist-step:last-child { border-bottom: 0; }
.setup-num { width: 28px; height: 28px; border-radius: 50%; border: 1.5px solid var(--line); display: grid; place-items: center; font-size: 13px; color: var(--muted); margin-top: 1px; }
.setup-done .setup-num { border-color: var(--success); color: var(--success); background: color-mix(in srgb, var(--success) 10%, transparent); }
.setup-open { background: color-mix(in srgb, var(--action) 6%, var(--surface)); box-shadow: inset 3px 0 0 var(--action); padding-top: 14px; padding-bottom: 15px; }
.setup-open .setup-num { border-color: var(--action); background: var(--action); color: var(--onAction); }
.setup-failed { background: color-mix(in srgb, var(--danger) 7%, var(--surface)); box-shadow: inset 3px 0 0 var(--danger); }
.setup-failed .setup-num { color: var(--danger); border-color: var(--danger); background: color-mix(in srgb, var(--danger) 12%, transparent); }
.setup-not-applicable .setup-num, .setup-optional .setup-num { border-style: dashed; }
.setup-title { color: var(--strong); font-size: 15px; font-weight: 700; line-height: 1.35; }
.setup-waiting .setup-title { color: var(--text); font-weight: 600; }
.setup-description { color: var(--muted); font-size: 13.5px; margin: 2px 0 0; }
.setup-right { display: flex; align-items: center; gap: 12px; margin-top: 2px; font-size: 13px; }
.setup-pill { display: inline-flex; align-items: center; white-space: nowrap; gap: 5px; border: 1px solid var(--line); border-radius: 999px; padding: 1px 9px; color: var(--muted); font-size: 12px; line-height: 1.5; }
.setup-pill::before { content: ''; width: 6px; height: 6px; border-radius: 50%; background: currentColor; }
.setup-next .setup-pill { color: var(--action); border-color: color-mix(in srgb, var(--action) 45%, var(--line)); }
.setup-done .setup-pill { color: var(--success); }.setup-failed .setup-pill { color: var(--danger); }.setup-checking .setup-pill { color: var(--warning); }
.setup-body { grid-column: 2 / 4; margin-top: 10px; }
.setup-body p { max-width: 640px; margin: 0 0 10px; }
.setup-actions { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; }
.setup-actions .btn { min-height: 40px; text-decoration: none; display: inline-flex; align-items: center; padding: 8px 16px; }
.setup-where { font-size: 12.5px; color: var(--muted); }
.setup-list-split { padding: 10px 20px 6px; font-size: 11px; line-height: 1.3; letter-spacing: .04em; color: var(--muted); border-bottom: 1px solid var(--line); margin: 0; }
.setup-needs { grid-column: 2 / 4; font-size: 13px; margin-top: 6px; }
.setup-needs summary { cursor: pointer; color: var(--action); }
.setup-needs p { max-width: 680px; margin: 8px 0 0; }
.setup-strip { display: flex; align-items: center; gap: 12px; border: 1px solid var(--line); background: var(--surface); border-radius: 10px; padding: 9px 14px; margin: 0 0 16px; font-size: 13.5px; }
.setup-ok { width: 20px; height: 20px; border-radius: 50%; display: grid; place-items: center; background: color-mix(in srgb, var(--success) 16%, transparent); color: var(--success); }
.setup-show { margin-left: auto; border: 0; color: var(--action); background: none; font: inherit; font-size: 13px; cursor: pointer; padding: 4px; }
.setup-alert { border: 1px solid color-mix(in srgb, var(--danger) 45%, var(--line)); border-radius: 10px; background: color-mix(in srgb, var(--danger) 7%, var(--surface)); margin-bottom: 16px; overflow: hidden; }
.setup-alert .checklist-step { padding: 14px 18px; }
.setup-foot-line { display: flex; gap: 10px; align-items: center; padding: 8px 18px 9px 64px; border-top: 1px solid var(--line); font-size: 12.5px; color: var(--muted); }
.setup-read-error { padding: 16px; }
.setup-status-link { display: inline-flex; align-items: center; gap: 6px; border: 1px solid var(--line); border-radius: 999px; padding: 1px 6px; white-space: nowrap; color: var(--strong); font-size: 12px; text-decoration: none; }
.setup-status-link::before { content: ''; width: 6px; height: 6px; border-radius: 50%; background: var(--action); }
.setup-status-link.setup-has-failure::before { background: var(--danger); }
.setup-status-replaced { display: none !important; }
.setup-status-link[hidden] { display: none; }
@media (max-width: 700px) {
.setup-head { padding: 16px; }.setup-head h2 { font-size: 19px; }
.checklist-step { grid-template-columns: 28px minmax(0, 1fr); column-gap: 12px; padding: 12px 16px; }
.setup-right { grid-column: 2; margin-top: 6px; }.setup-body, .setup-needs { grid-column: 1 / 3; }
.setup-body { margin-top: 12px; }.setup-actions { align-items: stretch; flex-direction: column; gap: 12px; }
.setup-actions .btn { justify-content: center; width: 100%; }.setup-strip { flex-wrap: wrap; gap: 8px; }
.setup-foot-line { padding: 8px 16px; flex-wrap: wrap; }.setup-status-link { max-width: 135px; white-space: normal; font-size: 11px; }
}
/* Override the workspace's generic list and section heading rules inside this card only. */
html[data-organizations] .ws .setup-steps { padding: 0; margin: 0; list-style: none; }
html[data-organizations] .ws .setup-head h2 { font-size: 20px; line-height: 1.4; margin: 0; }
html[data-organizations] .ws .setup-list-split { font-size: 11px; font-weight: 700; padding: 10px 20px 6px; margin: 0; }
html[data-organizations] .ws .setup-head .setup-where-am-i { font-size: 11px; line-height: 1.3; }
@media (max-width: 700px) { html[data-organizations] .ws .setup-head h2 { font-size: 19px; } }
@media (min-width: 701px) {
.checklist-step.setup-optional, .checklist-step:has(.setup-needs) { position: relative; }
.setup-needs { grid-column: 2 / 4; margin-top: 0; }
.setup-needs summary { position: absolute; right: 112px; top: 11px; list-style: none; }
.setup-needs summary::-webkit-details-marker { display: none; }
.setup-needs[open] p { margin-top: 12px; }
}
@@ -0,0 +1,125 @@
// Readiness presentation only. No step list, completion flag, provider list or write request.
(() => {
'use strict';
const esc = value => String(value ?? '').replace(/[&<>"']/g, c => ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' }[c]));
const satisfied = r => r && ['done', 'not-applicable'].includes(r.state);
function derive(input) {
const rows = input.map(r => ({ ...r, order: { after: [...(r.order?.after ?? [])] } }));
const byId = new Map(rows.map(r => [r.id, r])), ordered = [], seen = new Set(), visiting = new Set();
function visit(r) {
if (seen.has(r.id)) return;
if (visiting.has(r.id)) { for (const id of visiting) { const part = byId.get(id); part.state = 'failed'; part.invalidOrder = true; part.reason = "Couldn't check the order of these steps. Refresh to try again."; } return; }
visiting.add(r.id);
for (const id of r.order.after) {
if (byId.has(id)) visit(byId.get(id));
else { r.state = 'failed'; r.reason = "Couldn't check an earlier step. Refresh to try again."; }
}
visiting.delete(r.id); seen.add(r.id); ordered.push(r);
}
rows.forEach(visit);
const required = ordered.filter(r => r.required), optional = ordered.filter(r => !r.required);
required.forEach((r, i) => { r.number = i + 1; });
const current = required.find(r => !satisfied(r)) ?? null;
const alerts = required.filter(r => r.state === 'failed').map(r => ({ ...r }));
const alertsOnly = alerts.length > 0 && required.every(r => satisfied(r) || r.state === 'failed');
// Completed evidence stays completed when an earlier check fails later.
function blocker(r, trail = new Set()) {
if (trail.has(r.id)) return r;
trail.add(r.id);
for (const id of r.order.after) { const dep = byId.get(id); if (!satisfied(dep)) return dep ? blocker(dep, trail) ?? dep : r; }
return satisfied(r) ? null : r;
}
for (const r of required) {
if (satisfied(r) || r.invalidOrder) continue;
if (r === current) { if (!['failed', 'checking'].includes(r.state)) r.state = 'next'; continue; }
const dep = blocker(r);
if (dep && dep !== r) { r.state = 'waiting'; r.waitStep = Math.min(dep.number ?? current?.number, current?.number); }
else { r.state = 'waiting'; r.waitStep = current?.number; }
}
const complete = required.length > 0 && required.every(satisfied);
const failures = alerts;
return { rows: [...required, ...optional], required, optional, current, total: required.length, done: required.filter(satisfied).length, step: current?.number ?? required.length, complete,
alertsOnly, failures };
}
function target(fix) {
if (!fix || typeof fix.page !== 'string' || !/^#\/[A-Za-z0-9/?&=._%:-]*$/.test(fix.page)) return null;
return fix.step && [1, 2].includes(fix.step) ? `${fix.page}${fix.page.includes('?') ? '&' : '?'}step=${fix.step}` : fix.page;
}
function oneRow(r, current, alert = false) {
const open = current?.id === r.id || alert, failed = r.state === 'failed';
const words = { next: 'Next', checking: 'Checking', waiting: `After step ${r.waitStep ?? current?.number ?? 1}`, done: 'Done', failed: "Needs a fix", 'not-applicable': 'Not needed', optional: 'Optional' };
const href = target(r.fix), optional = !r.required;
return `<li class="checklist-step ${open ? 'setup-open ' : ''}setup-${esc(r.state)}">
<span class="setup-num" aria-hidden="true">${failed ? '!' : r.state === 'done' ? '✓' : r.number ?? '+'}</span>
<div class="setup-copy"><div class="setup-title">${esc(r.title)}</div><p class="setup-description">${esc(satisfied(r) || failed && open ? r.reason : r.purpose)}</p></div>
<div class="setup-right">${satisfied(r) && href ? `<a href="${esc(href)}">Open</a>` : ''}<span class="setup-pill">${esc(words[r.state] ?? 'Checking')}</span></div>
${open ? `<div class="setup-body">${!(failed || satisfied(r)) && r.reason ? `<p>${esc(r.reason)}</p>` : ''}${href ? `<div class="setup-actions"><a class="btn primary" href="${esc(href)}">${esc(r.fix.label)}</a><span class="setup-where">Opens the page for this step</span></div>` : ''}</div>` : ''}
${optional ? `<details class="setup-needs"><summary>What this needs</summary><p>${esc(r.reason)}</p></details>` : ''}
</li>`;
}
function card(s) {
const title = s.current ? `Step ${s.step} of ${s.total}: ${s.current.title.charAt(0).toLowerCase()}${s.current.title.slice(1)}` : 'Setup checks';
return `<section class="setup-card" aria-label="Set up Mosaic Stack"><header class="setup-head"><p class="setup-where-am-i">Set up Mosaic Stack</p><h2>${esc(title)}</h2><p>${s.complete ? 'Each check reads current state. Open a finished step to see its settings.' : 'One step at a time. The blue button is the thing to do now.'}</p><div class="setup-bar" aria-label="${s.done} of ${s.total} steps complete">${s.required.map(r => `<i class="${satisfied(r) ? 'on' : r === s.current ? r.state === 'failed' ? 'bad' : 'cur' : ''}"></i>`).join('')}</div></header><ol class="setup-steps">${s.required.map(r => oneRow(r, s.current)).join('')}</ol>${s.optional.length ? `<h3 class="setup-list-split">Later, when you need it</h3><ul class="setup-steps">${s.optional.map(r => oneRow(r, null)).join('')}</ul>` : ''}</section>`;
}
function render(s, expanded = false) {
if (s.complete) return `<section class="setup-strip" aria-label="Setup checks"><span class="setup-ok" aria-hidden="true">✓</span><strong>Setup done.</strong><span>All ${s.total} checks pass.</span><span class="muted">Checked just now.</span><button type="button" class="setup-show" data-setup-show aria-expanded="${expanded}">${expanded ? 'Hide checks' : 'Show checks'}</button></section>${expanded ? card(s) : ''}`;
if (s.alertsOnly && !expanded) return s.failures.map(r => `<section class="setup-alert" aria-label="Setup needs a fix"><ul class="setup-steps">${oneRow(r, r, true)}</ul><div class="setup-foot-line">This is step ${r.number} from setup. Its check needs a fix.<button type="button" class="setup-show" data-setup-show aria-expanded="false">Show checks</button></div></section>`).join('');
return card(s);
}
let last = null, expanded = false;
const fetchRead = window.fetch?.bind(window);
function paintShell(s = last?.state) {
if (!s || typeof document === 'undefined') return;
let pill = document.getElementById('setup-short');
const processing = document.getElementById('processing-link');
if (!pill && processing) { pill = document.createElement('a'); pill.id = 'setup-short'; pill.className = 'setup-status-link'; pill.href = '#/dashboard'; processing.before(pill); }
if (pill) { pill.hidden = s.complete; pill.textContent = `Setup: step ${s.step} of ${s.total}`; pill.classList.toggle('setup-has-failure', s.current?.state === 'failed'); }
if (processing) processing.classList.toggle('setup-status-replaced', !s.complete);
const hint = document.querySelector('#ws-org-list .ws-line');
const organization = s.required.find(r => target(r.fix)?.startsWith('#/settings/organizations'));
if (hint && !last?.organizations.length && organization) { const text = `None yet. Step ${organization.number} creates one.`; if (hint.textContent !== text) hint.textContent = text; }
}
let pendingRead = null, recentRead = null, reuseUntil = 0;
function read() {
// Shell startup and theme changes can render home again immediately. Share that read.
if (pendingRead) return pendingRead;
if (recentRead && Date.now() < reuseUntil) return Promise.resolve(recentRead);
pendingRead = load().then(result => {
recentRead = result; reuseUntil = Date.now() + 2000; return result;
}).finally(() => { pendingRead = null; });
return pendingRead;
}
async function load() {
try {
const response = await fetchRead('/api/setup', { cache: 'no-store', redirect: 'manual' });
if (!response.ok) throw Error();
const body = await response.json();
if (!Array.isArray(body.rows) || !Array.isArray(body.organizations) || !body.rows.length) throw Error();
const state = derive(body.rows);
last = { state, organizations: body.organizations }; paintShell(state);
const html = render(state, expanded), empty = !body.organizations.length && !state.complete;
return { html, empty, page: `<article class="ws ws-home" data-ws-home><div class="ws-vh"><h1 tabindex="-1">Dashboard</h1><p class="sub">${state.done ? `${state.done} of ${state.total} steps done.` : 'Nothing is set up yet.'} Work down this list. Each step checks itself, so it stays right even if you set something up elsewhere.</p></div><div data-setup-region>${html}</div></article>` };
} catch {
const html = '<section class="setup-alert setup-read-error" role="status"><strong>Couldn\'t check setup.</strong> Refresh to try again.</section>';
return { html, empty: false };
}
}
window.MosaicSetup = Object.freeze({ derive, render, read, paintShell });
if (typeof document === 'undefined' || !document.documentElement.hasAttribute('data-organizations')) return;
document.addEventListener('click', event => {
if (!event.target.closest?.('[data-setup-show]') || !last) return;
expanded = !expanded; recentRead = null;
const region = document.querySelector('[data-setup-region]');
if (region) { region.innerHTML = render(last.state, expanded); region.querySelector('[data-setup-show]')?.focus(); }
});
document.addEventListener('DOMContentLoaded', () => {
window.MosaicInitialOrganizations?.then(async rows => {
if (Array.isArray(rows) && !rows.length && ['', '#', '#/'].includes(location.hash)) {
const setup = await read();
if (setup.empty && ['', '#', '#/'].includes(location.hash)) location.hash = '#/dashboard';
}
});
const nav = document.getElementById('workspace-nav');
if (nav) new MutationObserver(() => paintShell()).observe(nav, { childList: true, subtree: true });
});
})();
+21 -7
View File
@@ -15,22 +15,35 @@
// draft holds the typed fields; base is the revision they were read at, which a save sends, so a
// change saved elsewhere since then answers as stale instead of being written over.
let data = null, readLine = '', draft = null, base = 0, busy = false, line = '', conflict = false;
// Reads answer in any order. began numbers each read as it starts. shown is the newest read or save
// the page has drawn its route from, and keyShown the newest read or key change it has drawn its key
// state from. A read older than shown is dropped, so a slow read cannot put back the route and
// revision a save or a newer read replaced. A read older than keyShown takes the key state the page
// holds instead of its own: a key change does not touch the route, so the read's route still stands.
// That key state is kept in key, apart from data, so a failed read that clears data keeps it.
let began = 0, shown = 0, keyShown = 0, key;
const call = init => window.fetch(PATH, { cache: 'no-store', redirect: 'manual', ...init });
// The stored route's fields, or empty ones: there is no default model, harness, cap or zone.
const fromRoute = r => ({ model: r?.model ?? '', harness: r?.harness ?? '', cap: r ? String(r.cap) : '', timeZone: r?.timeZone ?? '' });
// It says whether the page took this read (false when a newer read or write had drawn first).
async function read() {
readLine = '';
const n = ++began;
const failed = text => { if (n < shown) return false; shown = n; data = null; readLine = text; return true; };
let r, a = null;
try { r = await call({ method: 'GET' }); } catch { data = null; readLine = 'The Triage route could not be read. Reload the page to try again.'; return; }
try { r = await call({ method: 'GET' }); } catch { return failed('The Triage route could not be read. Reload the page to try again.'); }
try { a = await r.json(); } catch { /* refused below */ }
if (r.type === 'opaqueredirect') { data = null; readLine = 'Sign in again, then reload.'; return; }
if (!r.ok || !a || typeof a !== 'object') { data = null; readLine = typeof a?.message === 'string' ? a.message : 'The Triage route could not be read.'; return; }
if (r.type === 'opaqueredirect') return failed('Sign in again, then reload.');
if (!r.ok || !a || typeof a !== 'object') return failed(typeof a?.message === 'string' ? a.message : 'The Triage route could not be read.');
if (n < shown) return false;
shown = n; readLine = '';
// An untouched page follows the store; typed fields stay, with the revision they were read at.
const untouched = draft === null || JSON.stringify(draft) === JSON.stringify(fromRoute(data?.route ?? null));
if (n < keyShown) a = { ...a, access: key }; else { keyShown = n; key = a.access; }
data = a;
if (untouched) { draft = fromRoute(a.route); base = Number.isSafeInteger(a.revision) ? a.revision : 0; }
return true;
}
const models = () => (Array.isArray(data?.choices?.models) ? data.choices.models : []);
@@ -99,7 +112,7 @@
catch { busy = false; accessLine = 'Outcome unknown. Reload the page to see the key state.'; repaint(); return; }
try { a = await r.json(); } catch { /* refused below */ }
busy = false;
if (a?.access && data) data = { ...data, access: a.access };
if (a?.access && data) { data = { ...data, access: a.access }; key = a.access; keyShown = ++began; }
if (r.type === 'opaqueredirect') accessLine = 'Sign in again, then retry.';
else if (r.ok) accessLine = ACTION_LINE[a?.key?.action] ?? (op === 'off' ? 'Triage is off.' : 'Access was repaired.');
else accessLine = typeof a?.message === 'string' ? a.message : 'The Triage key was not changed.';
@@ -131,6 +144,7 @@
busy = false;
if (r.ok && a?.route) {
data = { ...data, route: a.route, revision: a.revision, routeMessage: undefined };
shown = ++began;
draft = fromRoute(a.route); base = a.revision;
line = `Saved ${savedAt(a.route.updatedAt)}.` + (a.key?.error ? ` ${a.key.message}` : ACTION_LINE[a.key?.action] ? ` ${ACTION_LINE[a.key.action]}` : '');
repaint();
@@ -147,8 +161,8 @@
async function reload() {
if (busy) return;
draft = null; conflict = false; line = '';
await read();
line = data ? 'Route read again. Your fields now show the stored route.' : '';
// A save that landed while this read was out has drawn the newer route and its receipt; keep both.
if (await read()) line = data ? 'Route read again. Your fields now show the stored route.' : '';
repaint();
}
+10 -2
View File
@@ -60,18 +60,26 @@
w.epoch++; w.inflight++;
try { return await fn(); } finally { w.inflight--; w.epoch++; }
}
// Two reads with no write between them can still answer out of order (the dashboard card and the page,
// or a poll and a navigation). Each organization numbers its reads as they start; a read that answers
// after a newer one has been kept is dropped, so an older answer cannot put back what a newer one replaced.
const reads = new Map();
const readOrder = org => { if (!reads.has(org)) reads.set(org, { began: 0, kept: 0 }); return reads.get(org); };
async function load(org) {
const prev = states.get(org), w = writeState(org), began = w.epoch;
const order = readOrder(org), n = ++order.began;
const s = { data: null, known: new Set(), error: null };
const raced = () => !!prev?.data && (w.inflight > 0 || w.epoch !== began);
try { await read(org, s); if (raced()) return; } catch (error) {
if (raced()) return;
const older = () => n < order.kept;
try { await read(org, s); if (raced() || older()) return; } catch (error) {
if (raced() || older()) return;
// Review P3: a failed poll while a change to this organization's workflows is held keeps the last good
// read, so the page still draws the change and its form, and graftLive puts the held form and focus back.
// SP17: the failure is still said; the page draws the last read under that line.
if (prev?.data && held(org).length) Object.assign(s, { data: prev.data, known: prev.known, readError: error.message });
else s.error = error.message;
}
order.kept = n;
states.set(org, s);
}
const scopes = s => [['', 'Organization', s.data.workflows], ...Object.keys(s.data.projects).sort().map(p => [p, `Project ${p}`, s.data.projects[p]])];
+3
View File
@@ -666,6 +666,8 @@
// ---- main dashboard (sample 02) ----
async function home() {
const setup = await window.MosaicSetup?.read();
if (setup?.empty) return setup.page;
let body;
let levels = null, listed = null;
// SP1: the agent counts come from the organization files (GET /api/agents/levels, owner only), so a
@@ -768,6 +770,7 @@
: `${body.summary?.read ?? o.rows.length - o.unread.length} of ${plural(o.rows.length, 'organization')} answered. Totals add up the ones that answered; ${andList(o.unread.map(stateWords))}.`;
return `<article class="ws ws-home" data-ws-home>
<div class="ws-vh"><h1 tabindex="-1">Dashboard</h1><p class="sub">${sub}</p></div>
<div data-setup-region>${setup?.html ?? ''}</div>
${tiles}
<div class="ws-split ws-home-split">
<section aria-label="Organizations, projects and tasks">
+2
View File
@@ -72,6 +72,8 @@ const files = new Map([
['/projects.css', ['projects.css', 'text/css; charset=utf-8']],
['/files.css', ['files.css', 'text/css; charset=utf-8']],
...['shared/app.css', 'tokens.css', 'console.css', 'live.css', 'bus.css', 'shell.css', 'chat.css', 'project-archive.css', 'processing.css', 'org-create.css', 'onboarding.css', 'tasks-board.css', 'assistant.css'].map(f => ['/' + f, [f, 'text/css; charset=utf-8']]),
['/setup-checklist.js', ['setup-checklist.js', 'text/javascript; charset=utf-8']],
['/setup-checklist.css', ['setup-checklist.css', 'text/css; charset=utf-8']],
['/display.js', ['display.js', 'text/javascript; charset=utf-8']],
['/inbox.js', ['inbox.js', 'text/javascript; charset=utf-8']],
['/inbox.css', ['inbox.css', 'text/css; charset=utf-8']],
+65
View File
@@ -0,0 +1,65 @@
// Owner-only instance readiness. Every dependency is a read; no seed, log, receipt or repair.
import { setupState } from '../../plugins/src/kinds.mjs';
import { readService } from '../../model-proxy/src/setup-state.mjs';
const states = new Set(['next', 'checking', 'waiting', 'done', 'failed', 'not-applicable', 'optional']);
const unavailable = { state: 'failed', reason: "Couldn't check. Open this step to try again.", code: 'setup-check-unavailable' };
const json = (res, status, body) => { res.writeHead(status, { 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-store' }); res.end(JSON.stringify(body)); };
export async function readSetup({ organizations, session, sources = setupState.map(path => () => import(new URL('../../' + path, import.meta.url)).then(m => m.setupState)), service, timeoutMs = 4000, ...deps } = {}) {
const controller = new AbortController(), deadline = Date.now() + timeoutMs;
let timer;
const expired = new Promise((_, reject) => { timer = setTimeout(() => { controller.abort(); reject(Error('setup deadline')); }, timeoutMs); });
// Keep this rejection observed even when all the checks complete before the deadline.
expired.catch(() => {});
const bounded = fn => Promise.race([Promise.resolve().then(fn), expired]);
const bound = { signal: controller.signal, deadline };
// Load row descriptions while admission runs, so a slow registry read cannot erase their fixes.
const loaded = sources.map(load => bounded(load).then(rows => Array.isArray(rows) ? rows : null, () => null));
const memo = new Map();
const once = (key, fn) => { if (!memo.has(key)) memo.set(key, bounded(fn)); return memo.get(key); };
const ctx = { ...deps, signal: controller.signal, service: () => once('service', service ?? (() => readService({ ...deps, signal: controller.signal }))) };
if (deps.accounts) ctx.accounts = () => once('accounts', () => deps.accounts(bound));
let members = [], memberError = null;
try { members = await bounded(() => organizations.members(session, { timeoutMs })); if (!Array.isArray(members)) throw Error(); }
catch (e) { memberError = e; }
Object.defineProperty(ctx, 'members', { get() { if (memberError) throw memberError; return members; } });
ctx.read = (org, name, args) => once(JSON.stringify([org.id, name, args]), async () => {
if (org.invalid) throw Error('organization unavailable');
if (name === 'status') return org.status(bound);
const options = await once(`options:${org.id}`, () => org.options(bound));
const reads = {
accounts: () => options.proxyCall('proxy.accounts', {}),
routes: () => options.bus.routes(),
plugins: () => options.proxyCall('plugins', {}),
tasks: () => options.bus.tasks(),
threads: () => options.chat.threads({}),
sessions: () => options.chat.sessions(args),
messages: () => options.chat.messages(args),
};
if (!Object.hasOwn(reads, name)) throw Error('unknown setup read');
return reads[name]();
});
try {
const groups = await Promise.all(loaded.map(async (pending, index) => {
const definitions = await pending;
if (!definitions) return [{ id: `setup.part-${index + 1}`, title: 'Check setup', purpose: 'One part could not report its setup state.', required: true, order: { after: [] }, fix: null, ...unavailable }];
return Promise.all(definitions.map(async ({ check, ...row }) => {
try {
const result = await bounded(() => check(ctx));
if (!states.has(result?.state) || typeof result.reason !== 'string') throw Error();
return { ...row, state: result.state, reason: result.reason, ...(result.fix ? { fix: result.fix } : {}) };
} catch { return { ...row, ...unavailable }; }
}));
}));
return { rows: groups.flat(), organizations: members.map(m => ({ id: m.id })) };
} finally { clearTimeout(timer); controller.abort(); }
}
export function setupRoutes(deps) {
return table => table.add('GET', '/api/setup', async (req, res, context) => {
if (!context.owner || !context.session) return json(res, 403, { error: 'owner-only', message: 'Setup checks are for the instance owner.' });
if (context.orgId !== null) return json(res, 404, { error: 'not found' });
return json(res, 200, await readSetup({ ...deps, session: context.session }));
});
}
+110
View File
@@ -0,0 +1,110 @@
// worker-model-setup (S6): what Settings > Model setup shows on its Worker model card. Read only.
//
// The model a worker run uses is the config's execution.provider and execution.model; the choice
// in <dataRoot>/worker-model/choice.json (processing-control/src/worker-model-store.mjs) is used
// only when the config names neither (worker-proxy.mjs workerModel). The card says which one is in
// use and whether it can run now, and lists the models a worker can use: switched on, served for
// the worker's harness, from the cliproxy module, and never a Claude model.
//
// The list comes from each active organization's bus `routes` catalogue, merged as Settings >
// Triage merges them (mergeCatalogues). The Console never loads the proxy management client.
// Those catalogues come from the same plugin and the same models-offered file the launch reads,
// so a model switched off here is switched off for a worker launch too.
//
// No key, path, env value or exception text goes into the view: names, states and a revision only.
import { execFile } from 'node:child_process';
import { join } from 'node:path';
import { promisify } from 'node:util';
import { claudeModel, readChoice as defaultReadChoice } from '../../processing-control/src/worker-model-store.mjs';
import { REPO } from '../../cli/src/config.mjs';
import { mergeCatalogues } from './triage-route-http.mjs';
// The config's provider and model pattern (scripts/mosaic-config.mjs NAME_PATTERN).
const NAME = /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,199}$/;
const HARNESS = /^[a-z][a-z0-9-]{0,31}$/;
const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
const named = v => (typeof v === 'string' && NAME.test(v) ? v : null);
// Saving a choice from this page needs an owner-only write route, which is not built (the build
// report's Ask B). Until it is, the card says so and offers no save.
export const CAN_SAVE = false;
// The system config's execution block as it is on disk now. A worker launch validates the config
// each time (scripts/mosaic-task.mjs, container-cli.mjs), and the card tells the owner to edit the
// config, so the card reads it each time too, not the copy the Console read when it started. It
// runs `mosaic-config.mjs validate`, the same check loadSystem runs, without blocking the server.
// Any failure throws, and the status read then carries `worker: null` ("could not be read").
const run = promisify(execFile);
export async function readExecution({ instance = null, exec = run, timeoutMs = 5000 } = {}) {
const argv = [join(REPO, 'scripts', 'mosaic-config.mjs'), 'validate', ...(instance === null ? [] : ['--instance', instance])];
const { stdout } = await exec(process.execPath, argv, { encoding: 'utf8', maxBuffer: 1024 * 1024, timeout: timeoutMs });
const execution = JSON.parse(stdout)?.execution;
if (!execution || typeof execution !== 'object' || Array.isArray(execution)) throw new Error('the config has no execution block');
return execution;
}
// The status read's `workerModel` callback: the config's execution block read fresh, the
// organizations' catalogues and the installed harnesses, each time it is called.
export function workerModelReader({ readExecution: read, dataRoot, catalogues, harnesses, readChoice }) {
return async () => {
const [execution, list] = await Promise.all([read(), catalogues()]);
return workerModelView({ execution, dataRoot, catalogues: list, harnesses: harnesses(), ...(readChoice ? { readChoice } : {}) });
};
}
// `execution` is the system config's execution block; `catalogues` the raw `routes` catalogues of
// the active organizations; `harnesses` the installed harness names.
//
// catalogue: 'ready'; 'none' when there is no active organization to ask; 'disabled' when every
// organization has the model service off; otherwise a status that says the list was not read.
//
// standing, for the model in use and for a stored choice:
// usable switched on and served for this harness
// off switched off on Models offered
// claude-model a Claude model, which runs only in the Claude harness
// not-served no account serves it for this harness now
// unknown the list is not known: no active organization, the model service off
// everywhere, or no organization's list could be read
// unset the config names only one of provider and model
// unused the adapter takes no model (mock)
export function workerModelView({ execution, dataRoot, catalogues, harnesses, readChoice = defaultReadChoice } = {}) {
const harness = typeof execution?.adapter === 'string' && HARNESS.test(execution.adapter) ? execution.adapter : null;
const takesModel = harness !== null && harness !== 'mock';
const fromConfig = !!execution && (execution.provider !== undefined || execution.model !== undefined);
let choice;
try {
const c = readChoice(dataRoot);
choice = c ? { state: 'set', provider: c.provider, model: c.model, revision: c.revision, updatedAt: ISO.test(c.updatedAt) ? c.updatedAt : null } : { state: 'none' };
} catch { choice = { state: 'unreadable' }; }
const merged = mergeCatalogues(catalogues, harnesses);
const ready = (Array.isArray(catalogues) ? catalogues : []).filter(c => c?.status === 'ready' && Array.isArray(c.models));
const seen = new Set();
const models = merged.status === 'ready' && takesModel ? merged.models.filter(m => {
if (m.module !== 'cliproxy' || m.available !== true || !m.harnesses.includes(harness)) return false;
if (!named(m.provider) || !named(m.model) || claudeModel(m.provider, m.model)) return false;
const id = `${m.provider}/${m.model}`;
if (seen.has(id)) return false;
seen.add(id); return true;
}).map(m => ({ provider: m.provider, model: m.model, label: m.label })) : [];
const off = (p, m) => ready.some(c => c.models.some(x => x?.module === 'cliproxy' && x.provider === p && x.model === m && x.reason === 'model-off'));
const standing = (p, m) => {
if (!named(p) || !named(m)) return 'unset';
if (claudeModel(p, m)) return 'claude-model';
if (merged.status !== 'ready') return 'unknown';
if (models.some(x => x.provider === p && x.model === m)) return 'usable';
if (off(p, m)) return 'off';
return 'not-served';
};
if (choice.state === 'set') choice.standing = takesModel ? standing(choice.provider, choice.model) : 'unused';
let using = null;
if (fromConfig) using = { from: 'config', provider: named(execution.provider), model: named(execution.model) };
else if (choice.state === 'set') using = { from: 'model-setup', provider: choice.provider, model: choice.model };
if (using) using.standing = takesModel ? standing(using.provider, using.model) : 'unused';
// 'disabled' only when every organization has the model service off; a mix reads as not read.
const listed = Array.isArray(catalogues) ? catalogues : [];
const catalogue = !listed.length ? 'none' : merged.status === 'ready' ? 'ready'
: listed.every(c => c?.status === 'disabled') ? 'disabled' : merged.status === 'disabled' ? 'unavailable' : merged.status;
return { harness, takesModel, using, choice, catalogue, models, canSave: CAN_SAVE };
}
@@ -51,3 +51,262 @@ for(const [width,height]of [[1440,900],[390,844]])for(const theme of ['light','d
await waitFor(b,`/Reversal recorded/.test(document.querySelector('[data-autonomy-status]').textContent)`);assert.equal(reverses,1);
await b.screenshot(join(shots,`reverse-${width}-${theme}.png`));
});
// The organization page reads autonomy again on a poll and after any write that changes the organization
// (mosaic-org-changed). A save or review receipt must survive a reload that overlaps it, in either order,
// and an older read that answers last must not put its older values back.
// hold(verb) makes the next call of that verb wait: its answer is computed when the call arrives, then held
// until release(). arrived resolves when the call has arrived.
function holds(){
const pending=new Map();
return {
hold(verb){const h={};h.arrived=new Promise(r=>{h.reached=r;});h.wait=new Promise(r=>{h.release=r;});pending.set(verb,[...(pending.get(verb)??[]),h]);return h;},
take:verb=>pending.get(verb)?.shift(),
};
}
async function raceFixture(t,gates){
const state={writes:0,reviews:0,setting:{merge:'out of the loop'},revision:'a'.repeat(64)};
const answer=(verb,args)=>{
if(verb==='autonomy.get.console')return {revision:state.revision,autonomy:state.setting,levels:{'cross-role':'out of the loop',merge:state.setting.merge,policy:'out of the loop',authority:'in the loop'}};
if(verb==='autonomy.set.console'){assert.equal(args.revision,state.revision);state.writes++;state.setting=args.autonomy;state.revision=String(state.writes+1).repeat(64);return {revision:state.revision,transaction:`10000000-0000-4000-8000-00000000000${state.writes}`};}
if(verb==='autonomy.review.console'){state.reviews++;return {recorded:true};}
throw Object.assign(new Error('not-configured'),{code:'not-configured'});
};
const call=async(verb,args)=>{const value=answer(verb,args),held=gates.take(verb);if(held){held.reached();await held.wait;}return value;};
const server=await startServer({port:0,sso:createDevOwner(),send:async()=>({}),organizations:{list:async()=>[{id:'acme',label:'Acme',busHealthy:true}],resolve:async()=>({busHealthy:true,decisionWrite:async()=>({}),requestRead:call,requestWrite:call,send:async()=>({}),reads:{business:async()=>({business:{id:'acme',projects:[],instances:[]}}),settings:async()=>({}),queue:async()=>({rows:[]})},bus:{instances:async()=>({instances:{}}),agents:async()=>[],'thread.claims':async()=>[],tasks:async()=>[],inbox:async()=>[],messages:async()=>({messages:[]})},chat:{projects:async()=>[],threads:async()=>[]}})},log:()=>{}});
t.after(()=>close(server));const b=await browser({machine:false});t.after(()=>b.close());
await b.navigate(`http://127.0.0.1:${server.address().port}/#/orgs/acme/organization`);
await waitFor(b,`document.querySelector('[data-autonomy-form] button[type=submit]')`);
return {b,state};
}
const reload=b=>b.evaluate(`document.dispatchEvent(new CustomEvent('mosaic-org-changed',{detail:{org:'acme'}}));true`);
const markDrawn=b=>b.evaluate(`document.querySelector('[data-autonomy]').dataset.drawn='before';true`);
const redrawn=b=>waitFor(b,`document.querySelector('[data-autonomy-form]') && !document.querySelector('[data-autonomy]').dataset.drawn`);
const statusText=b=>b.evaluate(`document.querySelector('[data-autonomy-form] [data-autonomy-status]').textContent`);
const submit=(b,level)=>b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value=${JSON.stringify(level)};document.querySelector('[data-autonomy-form]').requestSubmit();true`);
// Count each autonomy read by the revision it carries, once the reading load has used it: a task queued
// after the answer runs after the load's last step. A poll that reads meanwhile counts under its own revision.
const countReads=b=>b.evaluate(`{const f=window.fetch;window.__autonomyRead={};window.fetch=async(...a)=>{const r=await f(...a);if(String(a[0]).endsWith('/api/orgs/acme/autonomy')&&!a[1]?.method){const j=r.json.bind(r);r.json=()=>j().then(v=>{setTimeout(()=>{window.__autonomyRead[v.revision]=(window.__autonomyRead[v.revision]??0)+1});return v})}return r}}true`);
const readUsed=(b,revision)=>waitFor(b,`window.__autonomyRead[${JSON.stringify(revision)}]>=1`);
test('AU owner settings: a save that lands while a reload of autonomy is held keeps its receipt and revision',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
// The reload's read arrives first and answers after the save has been drawn.
const list=gates.hold('autonomy.get.console');
await reload(b);await list.arrived;
await submit(b,'in the loop');
await waitFor(b,`/^Saved\\. Receipt /.test(document.querySelector('[data-autonomy-status]').textContent)`);
assert.equal(state.writes,1);
await markDrawn(b);list.release();await redrawn(b);
assert.match(await statusText(b),/^Saved\. Receipt 10000000-0000-4000-8000-000000000001\.$/,'the reload kept the receipt');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form]').dataset.revision`),JSON.stringify(state.revision),'the form saves against the revision the save answered');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value`),'in the loop','the form shows what was saved');
// The next save is not a revision conflict.
await submit(b,'near the loop');
await waitFor(b,`/^Saved\\. Receipt 10000000-0000-4000-8000-000000000002\\.$/.test(document.querySelector('[data-autonomy-status]').textContent)`);
assert.equal(state.writes,2);
});
test('AU owner settings: a save that answers after a reload has drawn the page keeps its receipt',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
// The save arrives first and answers after a reload has drawn the page again.
const save=gates.hold('autonomy.set.console');
await submit(b,'in the loop');await save.arrived;
await markDrawn(b);await reload(b);await redrawn(b);
save.release();
await waitFor(b,`/^Saved\\. Receipt /.test(document.querySelector('[data-autonomy-form] [data-autonomy-status]').textContent)`);
await markDrawn(b);await reload(b);await redrawn(b);
assert.match(await statusText(b),/^Saved\. Receipt 10000000-0000-4000-8000-000000000001\.$/,'a later reload keeps the receipt');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form]').dataset.revision`),JSON.stringify(state.revision));
});
test('AU owner settings: a trail review that answers after a reload has drawn the page is said',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
await b.evaluate(`document.querySelector('[data-autonomy-review-list]').open=true;true`);
const review=gates.hold('autonomy.review.console');
await b.evaluate(`document.querySelector('[data-autonomy-review=merge]').click();true`);await review.arrived;
await markDrawn(b);await reload(b);await redrawn(b);
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-review-list]').open`),true,'the review list stays open');
review.release();
await waitFor(b,`document.querySelector('[data-autonomy-form] [data-autonomy-status]').textContent==='Trail review recorded.'`);
assert.equal(state.reviews,1);
await markDrawn(b);await reload(b);await redrawn(b);
assert.equal(await statusText(b),'Trail review recorded.','a later reload keeps the review receipt');
});
test('AU owner settings: an older read of autonomy that answers last does not replace a newer one',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
await countReads(b);
const older=gates.hold('autonomy.get.console');
await reload(b);await older.arrived;
// Another writer changes the organization; a newer read answers first.
state.setting={merge:'near the loop'};state.revision='c'.repeat(64);
await reload(b);
// The page draws after all of the dashboard's reads, so wait for the draw as well as the read.
await waitFor(b,`document.querySelector('[data-autonomy-form] select[name=merge]')?.value==='near the loop' && window.__autonomyRead[${JSON.stringify('c'.repeat(64))}]>=1`);
older.release();
await readUsed(b,'a'.repeat(64));
const drawn=await b.evaluate(`window.MosaicAutonomy.section('acme')`);
assert.match(drawn,/<option selected>near the loop<\/option>/,'the older values were not put back');
assert.match(drawn,new RegExp(`data-revision="&quot;${'c'.repeat(64)}&quot;"`),'the newer revision stays');
});
test('AU owner settings: a save that answers after a redraw from an older read draws what was saved',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
// The reload's read is served before the save commits. It draws a clean new form while the save is held.
const read=gates.hold('autonomy.get.console');
await reload(b);await read.arrived;
const save=gates.hold('autonomy.set.console');
await submit(b,'in the loop');await save.arrived;
assert.equal(state.writes,1);
await markDrawn(b);read.release();await redrawn(b);
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value`),'out of the loop','the older read was drawn');
save.release();
await waitFor(b,`/^Saved\\. Receipt /.test(document.querySelector('[data-autonomy-form] [data-autonomy-status]').textContent)`);
// Before any further reload, the drawn form shows the save and carries its revision.
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form]').dataset.revision`),JSON.stringify(state.revision),'the drawn form saves against the revision the save answered');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value`),'in the loop','the drawn form shows what was saved');
assert.match(await statusText(b),/^Saved\. Receipt 10000000-0000-4000-8000-000000000001\.$/);
await submit(b,'near the loop');
await waitFor(b,`/^Saved\\. Receipt 10000000-0000-4000-8000-000000000002\\.$/.test(document.querySelector('[data-autonomy-status]').textContent)`);
assert.equal(state.writes,2,'the next save is not a revision conflict');
});
test('AU owner settings: a save that answers after a newer read by another writer leaves the newer read',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await raceFixture(t,gates);
// The save commits, then another writer commits, and a reload draws that newer read before the save answers.
const save=gates.hold('autonomy.set.console');
await submit(b,'in the loop');await save.arrived;
assert.equal(state.writes,1);
state.setting={merge:'near the loop'};state.revision='c'.repeat(64);
await markDrawn(b);await reload(b);await redrawn(b);
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value`),'near the loop','the newer read was drawn');
save.release();
await waitFor(b,`/^Saved\\. Receipt 10000000-0000-4000-8000-000000000001\\.$/.test(document.querySelector('[data-autonomy-form] [data-autonomy-status]').textContent)`);
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form] select[name=merge]').value`),'near the loop','the drawn form keeps the newer read');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-form]').dataset.revision`),JSON.stringify('c'.repeat(64)));
const drawn=await b.evaluate(`window.MosaicAutonomy.section('acme')`);
assert.match(drawn,/<option selected>near the loop<\/option>/,'the state keeps the newer read');
assert.match(drawn,/data-autonomy-status role="status">Saved\. Receipt 10000000-0000-4000-8000-000000000001\./,'the state keeps the receipt');
// The next save goes against the newer revision and is not a conflict.
await submit(b,'in the loop');
await waitFor(b,`/^Saved\\. Receipt 10000000-0000-4000-8000-000000000002\\.$/.test(document.querySelector('[data-autonomy-status]').textContent)`);
assert.equal(state.writes,2);
});
// The decision notice on #/orgs/<org>/inbox/<id>. The bus page draws that view again on every poll (and on a
// fresh config, which is how these tests ask for a draw), and a decision the owner confirmed, returned or
// reversed leaves the inbox read, as the broker's inbox does. The owner's result must stay on the notice.
const decisionId='10000000-0000-4000-8000-0000000000d1';
const earlierId='10000000-0000-4000-8000-0000000000d0';
async function noticeFixture(t,gates,{autonomy,supersedes=false}){
const state={confirms:0,reverses:0,refuse:null,inbox:true};
const row=()=>({...(supersedes?{supersedes:earlierId}:{}),id:decisionId,business:'acme',at:'2026-11-11T12:00:00.000Z',class:'gated',action:'git.merge.protected',route_to:'cto',raised_by_role:'coder',raised_by_run:'coder-run',question:'Land the reviewed change?',options:[{key:'yes',text:'Land'},{key:'no',text:'Revise'}],recommendation:'yes',blocking:false,autonomy});
const answer=(verb,args)=>{
if(state.refuse){const code=state.refuse;state.refuse=null;throw Object.assign(new Error(code),{code});}
if(verb==='decision.confirm.console'){state.confirms++;state.inbox=false;return {op:args.confirm?'confirmed':'declined'};}
if(verb==='decision.reverse.console'){state.reverses++;state.inbox=false;return {reversed:true};}
throw Object.assign(new Error('not-configured'),{code:'not-configured'});
};
// The answer (or refusal) is settled when the call arrives; a held call gives it on release().
const call=async(verb,args)=>{let value,error;try{value=answer(verb,args);}catch(e){error=e;}const held=gates.take(verb);if(held){held.reached();await held.wait;}if(error)throw error;return value;};
// The earlier version's trail read, which a draw of a superseding decision waits on, can be held.
const earlier=async()=>{const held=gates.take('trail:earlier');if(held){held.reached();await held.wait;}return [{id:earlierId,business:'acme',at:'2026-11-11T11:00:00.000Z',class:'gated',action:'git.merge.protected',route_to:'cto',raised_by_role:'coder',raised_by_run:'coder-run',question:'Land the earlier change?',options:[{key:'yes',text:'Land'}],recommendation:'yes',blocking:false,table:'decisions'}];};
const trail=async id=>id===earlierId?earlier():[{...row(),table:'decisions'},{table:'events',kind:'action.allowed',body:{operation:'decision.raise',decision:id,approvalChoice:'yes',target:'vikunja:1/2'}}];
const server=await startServer({port:0,sso:createDevOwner(),send:async()=>({}),organizations:{list:async()=>[{id:'acme',label:'Acme',busHealthy:true}],resolve:async()=>({busHealthy:true,decisionWrite:async()=>({}),requestRead:call,requestWrite:call,send:async()=>({}),reads:{business:async()=>({business:{id:'acme',projects:[],instances:[]}}),settings:async()=>({}),queue:async()=>({rows:[]})},bus:{instances:async()=>({instances:{}}),agents:async()=>[],'thread.claims':async()=>[],tasks:async()=>[],inbox:async()=>state.inbox?[row()]:[],trail,messages:async()=>({messages:[]})},chat:{projects:async()=>[],threads:async()=>[]}})},log:()=>{}});
t.after(()=>close(server));const b=await browser({machine:false});t.after(()=>b.close());
await b.navigate(`http://127.0.0.1:${server.address().port}/#/orgs/acme/inbox/${decisionId}`);
await waitFor(b,`document.querySelector('[data-autonomy-decision] button')`);
return {b,state};
}
// Draw the decision view again, the way a bus restart's fresh config does, and wait until it has drawn.
const drawAgain=async b=>{
await b.evaluate(`document.querySelector('[aria-labelledby=b-q]').dataset.drawn='before';true`);
await b.evaluate(`fetch('/api/config',{cache:'no-store'}).then(r=>r.json()).then(c=>document.dispatchEvent(new CustomEvent('mosaic-config',{detail:c})));true`);
await waitFor(b,`document.querySelector('[aria-labelledby=b-q]') && !document.querySelector('[aria-labelledby=b-q]').dataset.drawn`);
};
const noticeLine=b=>b.evaluate(`document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent ?? null`);
const noticeButtons=b=>b.evaluate(`[...document.querySelectorAll('[data-autonomy-decision] button')].map(x=>[x.textContent,x.disabled])`);
test('AU decision notice: a confirm that answers after a redraw is said, and stays after the decision leaves the inbox',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await noticeFixture(t,gates,{autonomy:{pending:true}});
const confirm=gates.hold('decision.confirm.console');
await b.evaluate(`document.querySelector('[data-autonomy-confirm=yes]').click();true`);await confirm.arrived;
// The broker has recorded the confirm, so the inbox read no longer carries the decision's marker. A draw while
// the answer is still on its way keeps the notice, with no button to answer twice.
await drawAgain(b);
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-decision] h2')?.textContent ?? null`),'Confirm the arbiter resolution','a redraw in flight keeps the notice');
assert.deepEqual(await noticeButtons(b),[],'a redrawn notice takes no second answer while one is in flight');
confirm.release();
await waitFor(b,`/^Confirmation recorded/.test(document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent)`);
assert.equal(state.confirms,1);
// The confirmed decision has left the inbox read. The next draw keeps the notice and its result line.
await drawAgain(b);
assert.equal(await noticeLine(b),'Confirmation recorded. Refresh the inbox for current state.','the redraw keeps the result');
assert.equal(await b.evaluate(`document.querySelector('[data-autonomy-decision] h2').textContent`),'Confirm the arbiter resolution');
assert.deepEqual(await noticeButtons(b),[],'a decision that left the read offers no answer');
});
test('AU decision notice: a reverse result stays on the next draws',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await noticeFixture(t,gates,{autonomy:{notice:true,reverseUntil:'2099-01-01T00:00:00Z'}});
await b.evaluate(`document.querySelector('[data-autonomy-reverse]').click();true`);
await waitFor(b,`document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent==='Reversal recorded.'`);
assert.equal(state.reverses,1);
await drawAgain(b);
assert.equal(await noticeLine(b),'Reversal recorded.','the first redraw keeps the result');
await drawAgain(b);
assert.equal(await noticeLine(b),'Reversal recorded.','a later redraw keeps the result');
});
test('AU decision notice: a refused confirm is kept on the redrawn notice, with its buttons on',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await noticeFixture(t,gates,{autonomy:{pending:true}});
state.refuse='storage-refused';
const confirm=gates.hold('decision.confirm.console');
await b.evaluate(`document.querySelector('[data-autonomy-confirm=yes]').click();true`);await confirm.arrived;
assert.deepEqual(await noticeButtons(b),[['Confirm resolution',true],['Return to arbiter',true]],'the notice takes one answer at a time');
// A refusal leaves the decision in the inbox read. A draw while it is in flight keeps the buttons off.
await drawAgain(b);
assert.deepEqual(await noticeButtons(b),[['Confirm resolution',true],['Return to arbiter',true]],'a redrawn notice takes no second answer while one is in flight');
confirm.release();
await waitFor(b,`/trail write was refused/.test(document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent)`);
await drawAgain(b);
assert.equal(await noticeLine(b),'The trail write was refused. Inspect the decision trail before retrying.','the redraw keeps the refusal');
assert.deepEqual(await noticeButtons(b),[['Confirm resolution',false],['Return to arbiter',false]],'the decision is still open, so it can be answered again');
assert.equal(state.confirms,0);
});
test('AU decision notice: a returned decision that comes back to the inbox draws a fresh notice',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await noticeFixture(t,gates,{autonomy:{pending:true}});
await b.evaluate(`document.querySelector('[data-autonomy-confirm=no]').click();true`);
await waitFor(b,`/^Returned to arbiter/.test(document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent)`);
assert.equal(state.confirms,1);
// The returned decision has left the inbox read. The next draw keeps the result line.
await drawAgain(b);
assert.equal(await noticeLine(b),'Returned to arbiter. Refresh the inbox for current state.','the redraw keeps the result');
// The arbiter resolves it again, so it is back in the inbox and waits for a new answer.
state.inbox=true;
await drawAgain(b);
assert.equal(await noticeLine(b),'','the old result is not shown on the new question');
assert.deepEqual(await noticeButtons(b),[['Confirm resolution',false],['Return to arbiter',false]]);
});
test('AU decision notice: a draw that a newer draw replaced does not drop the result',{timeout:60000},async t=>{
const gates=holds();const {b,state}=await noticeFixture(t,gates,{autonomy:{pending:true},supersedes:true});
// Count the earlier-version trail answers the page has taken.
await b.evaluate(`(()=>{const prev=window.fetch;window.__earlier=0;window.fetch=(input,init)=>{const p=prev(input,init);if(!String(input).includes(${JSON.stringify(earlierId)}))return p;return p.then(r=>{const json=r.json.bind(r);r.json=()=>json().then(v=>{setTimeout(()=>{window.__earlier++;});return v;});return r;});};return true;})()`);
// A draw starts while the decision is still in the inbox, and waits on the earlier version's trail.
const slow=gates.hold('trail:earlier');
await b.evaluate(`fetch('/api/config',{cache:'no-store'}).then(r=>r.json()).then(c=>document.dispatchEvent(new CustomEvent('mosaic-config',{detail:c})));true`);
await slow.arrived;
await b.evaluate(`document.querySelector('[data-autonomy-confirm=yes]').click();true`);
await waitFor(b,`/^Confirmation recorded/.test(document.querySelector('[data-autonomy-decision] [data-autonomy-status]')?.textContent)`);
assert.equal(state.confirms,1);
// A newer draw reads the decision gone and draws the result. Then the replaced draw gets its answer.
await drawAgain(b);
assert.equal(await noticeLine(b),'Confirmation recorded. Refresh the inbox for current state.');
const taken=await b.evaluate('window.__earlier');
slow.release();
await waitFor(b,`window.__earlier===${taken+1}`);
await drawAgain(b);
assert.equal(await noticeLine(b),'Confirmation recorded. Refresh the inbox for current state.','the replaced draw does not drop the result');
});
+3 -3
View File
@@ -69,12 +69,12 @@ export async function browser({ machine = true, args = [] } = {}) {
}
const { targetId } = await send('Target.createTarget', { url: 'about:blank' });
const { sessionId } = await send('Target.attachToTarget', { targetId, flatten: true });
const call = (method, params) => send(method, params, sessionId);
const call = (method, params, ms) => send(method, params, sessionId, ms);
await call('Page.enable');
await call('Runtime.enable');
if (machine) await call('Page.addScriptToEvaluateOnNewDocument', { source: `try { localStorage.setItem('mosaic-console-board', '{"machine":true}'); } catch {}` });
const evaluate = async expression => {
const r = await call('Runtime.evaluate', { expression, returnByValue: true, awaitPromise: true });
const evaluate = async (expression, ms) => {
const r = await call('Runtime.evaluate', { expression, returnByValue: true, awaitPromise: true }, ms);
if (r.exceptionDetails) throw new Error(r.exceptionDetails.text + ': ' + r.result.description);
return r.result.value;
};
+1 -1
View File
@@ -19,7 +19,7 @@ test('served Console browser: real board fixtures, keyboard, drafts, receipts, t
await b.evaluate('document.querySelector("#hide-offline").click()');
assert.equal(await b.evaluate('document.querySelectorAll("table.sessions tbody tr").length'), 4);
// Rows 129, 154, 191, 194 and 294: chat.js, editor.js, processing.js, agents.js and federation.js are the scripts added, and #1839 adds requests.js (the intake panel under an owner's chat message), all same origin; the list is the pin, so another script is a visible change. #797 adds project-archive.js immediately after chat.js. F2 adds onboarding.js after org-create.js, and F3 adds console-restart.js after it. U2 (#1873) adds workspace-data.js and workspace.js before console-restart.js, and U4b adds tasks-board.js after workspace.js. triage-settings (#1873) adds triage-settings.js after console-restart.js. U5 (#1873) added panes.js (Side by side) just before bus.js; SP6 replaces it with assistant.js (the Assistant page), which bus.js mounts.
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("script")].map(s => s.getAttribute("src"))'), ['/editor.js', '/processing.js', '/org-create.js', '/onboarding.js', '/workspace-data.js', '/workspace.js', '/tasks-board.js', '/console-restart.js', '/triage-settings.js', '/display.js', '/inbox.js', '/files.js', '/project-remote.js', '/autonomy.js', '/packs.js', '/workflows.js', '/federation.js', '/brand.js', '/app.js', '/markdown-preview.js', '/projects.js', '/chat.js', '/project-archive.js', '/agents.js', '/assistant.js', '/bus.js', '/requests.js']);
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("script")].map(s => s.getAttribute("src"))'), ['/editor.js', '/processing.js', '/org-create.js', '/onboarding.js', '/setup-checklist.js', '/workspace-data.js', '/workspace.js', '/tasks-board.js', '/console-restart.js', '/triage-settings.js', '/display.js', '/inbox.js', '/files.js', '/project-remote.js', '/autonomy.js', '/packs.js', '/workflows.js', '/federation.js', '/brand.js', '/app.js', '/markdown-preview.js', '/projects.js', '/chat.js', '/project-archive.js', '/agents.js', '/assistant.js', '/bus.js', '/requests.js']);
assert.equal(await b.evaluate(`document.querySelectorAll('script[src="/processing.js"]').length`), 1);
assert.equal(await b.evaluate(`document.querySelectorAll('script[src="/chat.js"]').length`), 1);
assert.equal(await b.evaluate('!!window.injected'), false);
@@ -91,9 +91,10 @@ const homeReady = 'document.querySelector("[data-ws-home] [data-ws-tiles] [data-
test('The Dashboard tiles add both answering organizations\' numbers, one row, each tile a link to the page that lists it', async t => {
const { b, wait, text } = await setup(t);
await wait(homeReady);
// One row: no second tile group anywhere on the page, and the row leads the page under its heading.
// One tile row follows the setup checks under the dashboard heading.
assert.equal(await b.evaluate('document.querySelectorAll("[data-ws-home] [data-ws-tiles]").length'), 1);
assert.equal(await b.evaluate('document.querySelector("[data-ws-home] .ws-vh").nextElementSibling.hasAttribute("data-ws-tiles")'), true);
assert.equal(await b.evaluate('document.querySelector("[data-ws-home] .ws-vh").nextElementSibling.hasAttribute("data-setup-region")'), true);
assert.equal(await b.evaluate('document.querySelector("[data-ws-home] [data-setup-region]").nextElementSibling.hasAttribute("data-ws-tiles")'), true);
assert.deepEqual(await b.evaluate(tileKeys('[data-ws-home]')), ['agents', 'orgs', 'needs', 'tasks', 'done', 'projects']);
// Agents working of total: acme's coder works; beta's pm waits; chief and writer run nothing.
assert.equal(await b.evaluate(big('[data-ws-home]', 'agents')), '1 of 4');
@@ -6,6 +6,7 @@ import assert from 'node:assert/strict';
import { startServer } from '../src/serve.mjs';
import { createDevOwner } from '../src/dev-owner.mjs';
import { createOrgAdmin } from '../src/org-admin.mjs';
import { setupRoutes } from '../src/setup-http.mjs';
import { browser } from './browser.mjs';
import { close } from './fixture.mjs';
@@ -23,10 +24,12 @@ async function consoleWith(t, ids) {
const registry = { listOrgs: async () => [...docs.keys()].map(id => ({ id, state: 'active' })) };
const organizations = {
list: async () => [...docs.keys()].map(id => ({ id, label: id, busHealthy: true })),
members: async () => [],
resolve: async id => docs.has(id) ? { busHealthy: true, send: async () => ({}), reads: { business: async () => ({ business: { id, projects: ['work'], appearance: docs.get(id).appearance } }), busStatus: async () => ({ live: true }) }, bus: {} } : null,
};
const orgAdmin = createOrgAdmin({ service, registry, context: async () => ({ configDir: '/cfg', dataRoot: '/data', instance: 'test' }) });
const server = await startServer({ port: 0, sso: createDevOwner(), organizations, orgAdmin, send: async () => ({}), board: 'http://127.0.0.1:9', log: () => {} });
const routes = ids.length ? [] : [setupRoutes({ organizations, service: async () => ({ release: {}, init: {}, unit: {} }), docker: async () => false, runs: () => [] })];
const server = await startServer({ port: 0, sso: createDevOwner(), organizations, orgAdmin, routes, send: async () => ({}), board: 'http://127.0.0.1:9', log: () => {} });
t.after(() => close(server));
const b = await browser({ machine: false }); t.after(() => b.close());
const wait = (expression, label = expression) => b.evaluate(`(async()=>{for(let i=0;i<200;i++){if(${expression})return;await new Promise(r=>setTimeout(r,30))}throw Error('Timed out: '+${JSON.stringify(label)}+'\\n'+document.body.innerText.slice(0,800))})()`);
@@ -72,16 +75,16 @@ test('the sidebar Organizations group offers New organization', { timeout: 60000
await formFocused('#/orgs/alpha', 'sidebar');
});
test('the no-organization empty state offers New organization next to its Settings link', { timeout: 60000 }, async t => {
test('the empty dashboard explains organization setup and keeps its sidebar create link', { timeout: 60000 }, async t => {
const { base, b, wait, formFocused } = await consoleWith(t, []);
await b.viewport(1400, 900);
await b.navigate(`${base}#/`);
const button = `document.querySelector('#held a.btn[href="#/settings/organizations?new=1"]')`;
await wait(`document.querySelector('.setup-card ol .checklist-step:nth-child(3)')`, 'setup checklist');
assert.match(await b.evaluate(`document.querySelector('.setup-card ol .checklist-step:nth-child(3)').textContent`), /Create your organization/);
assert.match(await b.evaluate(`document.querySelector('#ws-org-list').textContent`), /None yet. Step 3 creates one/);
const button = `document.querySelector('#workspace-nav a.ws-add[href="#/settings/organizations?new=1"]')`;
await wait(button, 'empty state button');
const held = await b.evaluate(`document.querySelector('#held').textContent`);
assert.match(held, /No organizations yet/);
assert.equal(await b.evaluate(`!!document.querySelector('#held a[href="#/settings/organizations"]')`), true, 'the Settings link stays');
assert.equal(await b.evaluate(`${button}.textContent`), 'New organization');
assert.equal(await b.evaluate(`${button}.getAttribute('aria-label')`), 'New organization');
await b.evaluate(`${button}.click()`);
await formFocused('#', 'empty state');
});
@@ -31,7 +31,7 @@ const PREVIEW = {
// busy: the bus's apply committed but its hire loop did not run (hireError), and the packs read lists one line a
// member's pack hire left in the owner's inbox (fromPack), one raised some other way and one refused.
async function fixture(t, { busy = false } = {}) {
async function fixture(t, { busy = false, gates = null } = {}) {
const verbs = [], decisions = [];
const hired = new Map(); // name -> its pack provenance, as the bus's instances read reports it
let applied = null, route = false, instanceReads = 0;
@@ -46,19 +46,22 @@ async function fixture(t, { busy = false } = {}) {
const listed = () => applied ? [{ id: 'software-team', origin: 'shipped', project: null, hired: [...hired.keys()],
open: PREVIEW.hires.filter(h => !hired.has(h.name)).map(h => ({ name: h.name, reason: `Proposed by starter pack software-team for ${h.definition} work.`,
...(!busy || h.name === 'scout-1' ? { state: 'refused', code: 'pack-hire-route-missing' } : { state: 'awaiting-owner', decision: `inbox-${h.name}`, ...(h.name === 'coder-1' ? { fromPack: true } : {}) }) })) }] : [];
const read = async (verb, args) => {
const readAnswer = async (verb, args) => {
verbs.push([verb, args]);
if (verb === 'packs.console') return { packs: PACKS, applied: listed() };
if (verb === 'pack.preview.console') return PREVIEW;
if (verb === 'autonomy.get.console') return { revision: REV('e'), autonomy: {}, levels: {} };
throw Object.assign(new Error('not-configured'), { code: 'not-configured' });
};
const write = async (verb, args) => {
const writeAnswer = async (verb, args) => {
verbs.push([verb, args]);
if (verb === 'pack.apply.console') { applied = true; return { revision: REV('f'), added: ['board'], pack: { id: args.pack, version: '1.0.0' }, ...(busy ? { hires: [], hireError: 'organization-busy' } : { hires: hireLoop() }) }; }
if (verb === 'pack.hire.console') return { pack: args.pack, hires: hireLoop() };
throw Object.assign(new Error('not-configured'), { code: 'not-configured' });
};
// A held call is answered when it arrives, then waits for release() (holds() below).
const gated = answer => async (verb, args) => { const value = await answer(verb, args), held = gates?.take(verb); if (held) { held.reached(); await held.wait; } return value; };
const read = gated(readAnswer), write = gated(writeAnswer);
const decisionWrite = async (verb, args) => {
verbs.push([verb, args]);
if (verb === 'hire.start.console' && args.op === 'fire' && hired.has(args.name)) { hired.delete(args.name); return { id: 'fire-1', op: 'resolved', choice: 'yes' }; }
@@ -184,3 +187,80 @@ test('PD an apply whose hire loop did not run says so; the owner card counts a p
// coder-1 was left in the inbox by a pack hire, so the owner's pack hire approves it; reviewer-1 is the inbox's.
assert.equal(await b.evaluate(`document.querySelector('[data-pack-hire-button]').textContent`), 'Hire the 2 proposed agents');
});
// hold(verb) makes the next call of that verb wait: its answer is computed when the call arrives, then held
// until release(). arrived resolves when the call has arrived.
function holds() {
const pending = new Map();
return {
hold(verb) {
const h = {};
h.arrived = new Promise(r => { h.reached = r; });
h.wait = new Promise(r => { h.release = r; });
pending.set(verb, [...(pending.get(verb) ?? []), h]);
return h;
},
take: verb => pending.get(verb)?.shift(),
};
}
const reload = b => b.evaluate(`document.dispatchEvent(new CustomEvent('mosaic-org-changed',{detail:{org:'acme'}}));true`);
const markDrawn = b => b.evaluate(`document.querySelector('[data-packs]').dataset.drawn='before';true`);
const redrawn = (b, what) => waitFor(b, `document.querySelector('[data-packs]') && !document.querySelector('[data-packs]').dataset.drawn`, what);
// Count each packs answer once the reading load has used it. With no project pack listed the load has no
// further read, so a task queued after the answer runs after the load's last step.
const countPacksAnswers = b => b.evaluate(`{const f=window.fetch;window.__packsDone=0;window.fetch=async(...a)=>{const r=await f(...a);if(String(a[0]).endsWith('/api/orgs/acme/packs')){const j=r.json.bind(r);r.json=()=>j().finally(()=>setTimeout(()=>{window.__packsDone++}))}return r}}true`);
const HIRED = 'Hired scout-1. Already hired: coder-1, reviewer-1.';
// Apply the pack with no route, so scout-1 is left to hire, then set the route.
async function applied(t, gates) {
const f = await fixture(t, { gates });
await f.b.navigate(`http://127.0.0.1:${f.port}/#/orgs/acme/organization`);
await waitFor(f.b, `document.querySelector('[data-pack="software-team"] [data-pack-preview-button]')`, 'pack list');
await f.b.evaluate(`document.querySelector('[data-pack="software-team"] [data-pack-preview-button]').click();true`);
await waitFor(f.b, `document.querySelector('[data-pack-apply]')`, 'preview');
await f.b.evaluate(`document.querySelector('[data-pack-apply]').click();true`);
await waitFor(f.b, `/^Applied\\./.test(document.querySelector('[data-packs-status]')?.textContent??'') && document.querySelector('[data-pack-hire-button]')`, 'apply status and hire button');
f.setRoute(true);
return f;
}
// The page reads the packs again on a poll and after any write that changes the organization
// (mosaic-org-changed). A hire receipt must survive a reload that overlaps the hire, in either order.
test('PD a hire receipt that lands while an earlier read of the packs is held is kept', { timeout: 90000 }, async t => {
const gates = holds();
const { b, verbs } = await applied(t, gates);
await countPacksAnswers(b);
// A reload's packs read is held. The hire answers, reads the packs itself and asks the page to read
// everything again; both of those reads answer first. Then the held read answers last.
const earlier = gates.hold('packs.console');
await reload(b);
await earlier.arrived;
await b.evaluate(`document.querySelector('[data-pack-hire-button]').click();true`);
await waitFor(b, `document.querySelector('[data-packs-status]')?.textContent==='${HIRED}' && window.__packsDone>=2`, 'hire status');
earlier.release();
await waitFor(b, `window.__packsDone>=3`, 'held read answered');
await markDrawn(b);
await reload(b);
await redrawn(b, 'page drawn again after the held read');
assert.equal(await b.evaluate(`document.querySelector('[data-packs-status]').textContent`), HIRED, 'the hire receipt is kept');
assert.equal(await b.evaluate(`document.querySelector('[data-pack-hire-button]')`), null, 'nothing left to hire');
assert.deepEqual(verbs.filter(([v]) => v === 'pack.hire.console').map(([, a]) => a), [{ pack: 'software-team' }]);
});
test('PD a hire that answers after a reload has drawn the page keeps its receipt', { timeout: 90000 }, async t => {
const gates = holds();
const { b, verbs } = await applied(t, gates);
// The hire arrives first and answers after a reload has drawn the page again.
const hire = gates.hold('pack.hire.console');
await b.evaluate(`document.querySelector('[data-pack-hire-button]').click();true`);
await hire.arrived;
await markDrawn(b);
await reload(b);
await redrawn(b, 'page drawn again during the hire');
hire.release();
await waitFor(b, `document.querySelector('[data-packs-status]')?.textContent==='${HIRED}'`, 'hire status');
await markDrawn(b);
await reload(b);
await redrawn(b, 'page drawn again after the hire');
assert.equal(await b.evaluate(`document.querySelector('[data-packs-status]').textContent`), HIRED, 'a later reload keeps the hire receipt');
assert.deepEqual(verbs.filter(([v]) => v === 'pack.hire.console').map(([, a]) => a), [{ pack: 'software-team' }]);
});
@@ -35,7 +35,23 @@ const PREVIEW = {
skills: { notEnabled: ['lint-fix'] }, chief: true, plan: {}, digest: REV('d'),
};
async function fixture(t, { send = true } = {}) {
// hold(verb) makes the next read of that verb wait: its answer is computed when the read arrives, then
// held until release(). arrived resolves when the read has arrived.
function holds() {
const pending = new Map();
return {
hold(verb) {
const h = {};
h.arrived = new Promise(r => { h.reached = r; });
h.wait = new Promise(r => { h.release = r; });
pending.set(verb, [...(pending.get(verb) ?? []), h]);
return h;
},
take: verb => pending.get(verb)?.shift(),
};
}
async function fixture(t, { send = true, gates = null, packs = () => PACKS } = {}) {
const verbs = [];
let flows = {
revision: REV('a'),
@@ -48,13 +64,18 @@ async function fixture(t, { send = true } = {}) {
},
templates: ['coder', 'reviewer'],
};
const read = async (verb, args) => {
verbs.push([verb, args]);
if (verb === 'packs.console') return { packs: PACKS };
const answer = (verb, args) => {
if (verb === 'packs.console') return { packs: packs() };
if (verb === 'pack.preview.console') return { ...PREVIEW, project: args.project ?? null };
if (verb === 'autonomy.get.console') return { revision: REV('e'), autonomy: {}, levels: {} };
throw Object.assign(new Error('not-configured'), { code: 'not-configured' });
};
const read = async (verb, args) => {
verbs.push([verb, args]);
const value = answer(verb, args), held = gates?.take(verb);
if (held) { held.reached(); await held.wait; }
return value;
};
const write = async (verb, args) => {
verbs.push([verb, args]);
if (verb === 'pack.apply.console') return { revision: REV('f'), added: ['board', 'definitions.reviewer'], pack: { id: args.pack, version: '1.0.0' }, hires: [{ name: 'lead-reviewer', state: 'refused', code: 'definition-not-hireable' }] };
@@ -128,6 +149,85 @@ test('SP6 pack picker: list, preview with marks, flagged, credential and noHire
assert.deepEqual(verbs.filter(([v]) => v === 'pack.preview.console').at(-1)[1], { pack: 'web-project', project: 'site' });
});
// The page reads the packs again on a poll and after any write that changes the organization
// (mosaic-org-changed). A preview the owner asked for must survive a reload that overlaps it, in either
// order, and an older reload that answers last must not put its older list back.
const reload = b => b.evaluate(`document.dispatchEvent(new CustomEvent('mosaic-org-changed',{detail:{org:'acme'}}));true`);
const markDrawn = b => b.evaluate(`document.querySelector('[data-packs]').dataset.drawn='before';true`);
const redrawn = (b, what) => waitFor(b, `document.querySelector('[data-packs]') && !document.querySelector('[data-packs]').dataset.drawn`, what);
test('SP6 pack picker: a preview that overlaps a reload of the packs is kept, in either order', { timeout: 60000 }, async t => {
const gates = holds();
const { b, verbs } = await fixture(t, { gates });
await waitFor(b, `document.querySelector('[data-pack="software-team"] [data-pack-preview-button]')`, 'pack list');
// The reload's packs read starts first and answers after the preview has been drawn.
const list = gates.hold('packs.console');
await reload(b);
await list.arrived;
await b.evaluate(`document.querySelector('[data-pack="software-team"] [data-pack-preview-button]').click();true`);
await waitFor(b, `document.querySelector('[data-pack-preview="software-team"]')`, 'preview');
await markDrawn(b);
list.release();
await redrawn(b, 'page drawn again after the reload');
assert.ok(await b.evaluate(`!!document.querySelector('[data-pack-preview="software-team"]')`), 'the reload kept the preview');
// The preview's read starts first and answers after the reload has drawn the page.
const preview = gates.hold('pack.preview.console');
await b.evaluate(`document.querySelector('[data-pack="web-project"] [data-pack-preview-button]').click();true`);
await preview.arrived;
await markDrawn(b);
await reload(b);
await redrawn(b, 'page drawn again during the preview');
preview.release();
await waitFor(b, `/for project site/.test(document.querySelector('[data-pack-preview] h4')?.textContent??'')`, 'project preview after the reload');
assert.deepEqual(verbs.filter(([v]) => v === 'pack.preview.console').map(([, a]) => a), [{ pack: 'software-team' }, { pack: 'web-project', project: 'site' }]);
});
// Count each packs answer once the reading load has used it. With no project pack listed the load has
// no further read, so a task queued after the answer runs after the load's last step.
const countPacksAnswers = b => b.evaluate(`{const f=window.fetch;window.__packsDone=0;window.fetch=async(...a)=>{const r=await f(...a);if(String(a[0]).endsWith('/api/orgs/acme/packs')){const j=r.json.bind(r);r.json=()=>j().finally(()=>setTimeout(()=>{window.__packsDone++}))}return r}}true`);
test('SP6 pack picker: an older read of the packs that answers last does not replace a newer one', { timeout: 60000 }, async t => {
const gates = holds();
let packs = [PACKS[0]];
const { b } = await fixture(t, { gates, packs: () => packs });
await waitFor(b, `document.querySelector('[data-pack="software-team"] [data-pack-preview-button]')`, 'pack list');
await countPacksAnswers(b);
const older = gates.hold('packs.console');
await reload(b);
await older.arrived;
packs = [PACKS[0], { id: 'later', origin: 'installed', pack: { label: 'Later pack', description: 'Installed after the first read.', version: '0.1.0', scope: 'organization' } }];
await reload(b);
await waitFor(b, `document.querySelector('[data-pack="later"]') && window.__packsDone===1`, 'newer list drawn');
older.release();
await waitFor(b, `window.__packsDone===2`, 'older read answered');
assert.match(await b.evaluate(`window.MosaicPacks.section('acme')`), /data-pack="later"/, 'the older list was not put back');
await b.evaluate(`document.querySelector('[data-pack="software-team"] [data-pack-preview-button]').click();true`);
await waitFor(b, `document.querySelector('[data-pack-preview="software-team"]')`, 'preview');
assert.ok(await b.evaluate(`!!document.querySelector('[data-pack="later"]')`), 'the drawn list is the newer one');
});
test('SP6 pack picker: an apply receipt that lands while an earlier read of the packs is held is kept', { timeout: 60000 }, async t => {
const gates = holds();
const { b, verbs } = await fixture(t, { gates, packs: () => [PACKS[0]] });
await waitFor(b, `document.querySelector('[data-pack="software-team"] [data-pack-preview-button]')`, 'pack list');
await b.evaluate(`document.querySelector('[data-pack="software-team"] [data-pack-preview-button]').click();true`);
await waitFor(b, `document.querySelector('[data-pack-apply]')`, 'preview');
await countPacksAnswers(b);
// A reload's packs read is held. The apply answers, reads the packs itself and asks the page to read
// everything again; both of those reads answer first. Then the held read answers last.
const earlier = gates.hold('packs.console');
await reload(b);
await earlier.arrived;
await b.evaluate(`document.querySelector('[data-pack-apply]').click();true`);
await waitFor(b, `/^Applied\./.test(document.querySelector('[data-packs-status]')?.textContent??'') && window.__packsDone===2`, 'apply status');
assert.equal(verbs.filter(([v]) => v === 'pack.apply.console').length, 1);
earlier.release();
await waitFor(b, `window.__packsDone===3`, 'held read answered');
const drawn = await b.evaluate(`window.MosaicPacks.section('acme')`);
assert.match(drawn, /data-packs-status>Applied\. Added board, definitions\.reviewer\./, 'the receipt is kept');
assert.doesNotMatch(drawn, /data-pack-preview=/, 'the applied preview does not come back');
});
test('SP6 pack picker without send previews but offers no apply', { timeout: 60000 }, async t => {
const { b } = await fixture(t, { send: false });
await waitFor(b, `document.querySelector('[data-pack="software-team"] [data-pack-preview-button]')`, 'pack list');
@@ -667,3 +667,40 @@ test('models end to end: the first read seeds every present model on and seen; a
const all = [first.text, stale.text, inUse.text, saved.text, later.text, history.text, readFileSync(file, 'utf8'), JSON.stringify(lines), f.logs.join('\n')].join('\n');
for (const bit of [planted, 'msk-', root, '/home/']) assert.equal(all.includes(bit), false, bit);
});
// worker-model-setup: the Worker model card's view rides the status read, so only the instance
// owner gets it. A member, an organization owner and the tester listener are refused before the
// view is built.
test('worker model: only the owner gets the worker block on status; others are refused before the view is read', async t => {
const { createProxyAdmin } = await import('../../model-proxy/src/admin-control.mjs');
const root = mkdtempSync(join(tmpdir(), 'pa-wm-'));
t.after(() => rmSync(root, { recursive: true, force: true }));
const { mkdirSync } = await import('node:fs');
const configDir = join(root, 'config'), dataRoot = join(root, 'data');
mkdirSync(configDir, { mode: 0o700 }); mkdirSync(dataRoot, { mode: 0o700 });
let reads = 0;
const view = { harness: 'pi', takesModel: true, using: { from: 'config', provider: 'zai', model: 'glm-5.3', standing: 'usable' }, choice: { state: 'none' }, catalogue: 'ready', models: [], canSave: false };
const unit = { unit: 'mosaic-proxy-zz.service', status: async () => ({ unit: 'mosaic-proxy-zz.service', installed: false, active: 'inactive', healthy: false }) };
const make = () => createProxyAdmin({ dataRoot, configDir, unit, buses: async () => ({ stopped: true, running: [] }), workerModel: async () => { reads++; return view; } });
const owner = await devServer(t, { proxyAdmin: make() });
const r = await send(owner.port, '/api/proxy/admin/status', { headers: owner.headers() });
assert.equal(r.status, 200);
assert.deepEqual(r.json.worker, view);
assert.equal(reads, 1);
const organizations = { list: async () => [], resolve: async id => (id === 'x' ? { busHealthy: true } : null) };
for (const session of [{ name: 'Bob', provider: 'local-dev', iss: 'local-dev', sub: 'bob', rootOwner: false, instanceRole: 'member' },
{ name: 'Olive', provider: 'local-dev', iss: 'local-dev', sub: 'olive', rootOwner: false, instanceRole: 'operator', orgRole: 'owner' }]) {
const f = await devServer(t, { organizations, proxyAdmin: make(), sso: { ...createDevOwner(), session: () => session } });
for (const path of ['/api/proxy/admin/status', '/api/orgs/x/proxy/admin/status']) {
const refused = await send(f.port, path, { headers: f.headers() });
assert.equal(refused.status >= 400 && refused.status < 500, true, `${session.sub} ${path}`);
assert.equal(Object.hasOwn(refused.json ?? {}, 'worker'), false);
}
}
const { startTesterSocket } = await import('../src/tester-socket.mjs');
const { testerRequest } = await import('../../cli/src/tester-http.mjs');
const s = await startTesterSocket({ dataRoot, proxyAdmin: make(), proxyCall: async () => ({}), log: () => {} });
t.after(() => new Promise(r => s.close(r)));
assert.equal((await testerRequest({ dataRoot, path: '/api/proxy/admin/status' })).status, 403);
assert.equal(reads, 1, 'no refused caller reached the view');
});
@@ -35,4 +35,7 @@ test('signed-in Console buttons deliver the approved original request and downlo
assert.equal(await page.clock(), '25:00');
await page.advance(10000); assert.equal(await page.clock(), '25:00');
assert.deepEqual(await page.externalRequests(), []);
// A remote resource in the downloaded file must still be observed while the browser is offline.
await page.offline('<!doctype html><img src="https://external.invalid/tracker">');
assert.deepEqual(await page.externalRequests(), ['https://external.invalid/tracker']);
});
@@ -89,12 +89,53 @@ test('account-remove: Remove posts exactly { accountId }, reads the list again a
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
test('account-remove: a failed refresh after a successful remove keeps the success and the refresh error', { timeout: 90000 }, async t => {
const { b, wait } = await open(t, { extra: { [impactKey(A)]: { body: IMPACT_ALL },
'POST /api/proxy/admin/accounts/remove': { body: { accountId: A, provider: 'claude', removed: true } } } });
await b.evaluate(`document.querySelector('[data-remove="${A}"]').click(); true`);
await wait('!!document.querySelector("#proxy-remove-yes")');
await b.evaluate(`window.__scenario['GET /api/proxy/accounts'] = { status: 503, body: { error: 'proxy-down' } }; document.querySelector('#proxy-remove-yes').click(); true`);
await wait('document.querySelector("#proxy-message").textContent.includes("not answering")');
assert.equal(await b.evaluate('document.querySelector("#proxy-message").textContent'), 'Claude account removed. The account list could not be refreshed: The model service is not answering.');
assert.equal((await writes(b)).length, 1);
});
test('account-remove: a failed setup status refresh remains visible beside successful removal', { timeout: 90000 }, async t => {
const { b, wait } = await open(t, { extra: { [impactKey(A)]: { body: IMPACT_ALL },
'POST /api/proxy/admin/accounts/remove': { body: { accountId: A, provider: 'claude', removed: true } } } });
await b.evaluate(`document.querySelector('[data-remove="${A}"]').click(); true`);
await wait('!!document.querySelector("#proxy-remove-yes")');
await b.evaluate(`window.__scenario['GET /api/proxy/admin/status'] = { status: 503, body: { error: 'outcome-unknown' } }; document.querySelector('#proxy-remove-yes').click(); true`);
await wait('document.querySelector("#proxy-setup-message").textContent.length > 0');
assert.match(await b.evaluate('document.querySelector("#proxy-message").textContent'), /^Claude account removed\..*setup status could not be refreshed/i);
assert.equal((await writes(b)).length, 1);
});
test('account-remove: a late setup warning keeps the removal receipt when priority is saved meanwhile', { timeout: 90000 }, async t => {
const { b, wait } = await open(t, { extra: { [impactKey(A)]: { body: IMPACT_ALL },
'POST /api/proxy/admin/accounts/remove': { body: { accountId: A, provider: 'claude', removed: true } },
'POST /api/proxy/priority/set': { body: { accountId: B, priority: 0 } } } });
await b.evaluate(`document.querySelector('[data-remove="${A}"]').click(); true`);
await wait('!!document.querySelector("#proxy-remove-yes")');
await b.evaluate(`(() => { const real = window.fetch; window.__held = false; window.fetch = (input, init) => {
if (String(input) === '/api/proxy/admin/status') { window.__held = true; return new Promise(resolve => { window.__releaseStatus = () => resolve(new Response(JSON.stringify({ error: 'outcome-unknown' }), { status: 503, headers: { 'Content-Type': 'application/json' } })); }); }
return real(input, init); }; window.__scenario['GET /api/proxy/accounts'] = { body: { accounts: ${JSON.stringify(LIST.slice(1))} } }; document.querySelector('#proxy-remove-yes').click(); return true; })()`);
await wait('window.__held');
await b.evaluate('document.querySelector("#proxy-order form").requestSubmit(); true');
await wait('document.querySelector("#proxy-message").textContent === "Priority saved."');
await b.evaluate('window.__releaseStatus(); true');
await wait('document.querySelector("#proxy-message").textContent.includes("setup status could not be refreshed")');
assert.match(await b.evaluate('document.querySelector("#proxy-message").textContent'), /^Claude account removed\./);
assert.equal(await b.evaluate('document.querySelector("#proxy-order [role=status]").textContent'), 'Priority saved.');
});
test('account-remove: remove-write-refused and remove-unconfirmed are named in the ask and the account stays', { timeout: 90000 }, async t => {
// No refusal says nothing was removed: a refused or unanswered write may still have taken effect.
for (const [code, words] of [['remove-write-refused', 'The model service refused to remove the account. Check the list before trying again.'],
['remove-unconfirmed', 'Mosaic asked the model service to remove the account, but it is still listed. Read the accounts again before trying again.'],
['proxy-uncertain', 'Mosaic could not confirm the remove: the model service is between writes. Check the service and the list before trying again.'],
['proxy-down', 'The model service stopped answering, so Mosaic could not confirm the remove. Check the service and the list before trying again.'],
['invalid-request', 'Mosaic could not identify one account to remove. It may no longer be listed, have a duplicate ID, or share a credential with another account. Read the accounts again.'],
['some-new-code', 'The remove could not be confirmed. Nothing was retried. Read the accounts again before trying again.']]) {
const { b, wait } = await open(t, { extra: { [impactKey(A)]: { body: IMPACT_ALL }, 'POST /api/proxy/admin/accounts/remove': { status: 400, body: { error: code } } } });
await b.evaluate(`document.querySelector('[data-remove="${A}"]').click(); true`);
@@ -202,6 +202,14 @@ test('models-offered page: an organization that is not answering is named in the
assert.equal((await writes())[1].body.confirmInUse, true);
});
test('models-offered page: unanswered organizations are visible before saving', { timeout: 90000 }, async t => {
const { b, wait, writes } = await setup(t, { admin: { models: { ...VIEW, unanswered: ['lab', 'zeta'] } } });
await openOffered(b, wait);
assert.equal(await b.evaluate('document.querySelector("#proxy-offered-list .offered-unanswered")?.textContent'),
'Not answering now, so it is not known which models they use: Lab Bench, zeta. A model you switch off may stop working for them.');
assert.deepEqual(await writes(), []);
});
test('models-offered page: a save after another change is refused as a conflict, worded, with Read the list again; a busy or unknown answer is worded too', { timeout: 90000 }, async t => {
const conflict = { status: 409, body: { error: 'models-offered-conflict' } };
const busy = { status: 409, body: { error: 'proxy-admin-busy' } };
@@ -242,7 +250,7 @@ test('models-offered page: a page without proxyAdmin sends no /api/proxy/admin/
test('models-offered page: a read the model service cannot answer is worded, never an empty list', { timeout: 90000 }, async t => {
for (const [body, line] of [
[{ error: 'proxy-down' }, /No organization that uses your model service answered/],
[{ error: 'not-configured' }, /No organization uses your model service yet/],
[{ error: 'not-configured' }, /No organization with a configured model launch uses your model service yet/],
[{ error: 'outcome-unknown' }, /The outcome is unknown/],
]) {
const { b, wait } = await setup(t, { admin: { models: { status: 503, body } } });
@@ -261,6 +261,69 @@ const installed = { ...configured, unit: { ...configured.unit, installed: true }
const running = { ...installed, unit: { ...installed.unit, active: 'active', pid: 4242, port: 18432, listening: true }, next: ['accounts'] };
const config = { send: true, devOwner: true, proxyAdmin: true, proxyAdminSend: true };
const accounts = list => ({ 'GET /api/proxy/accounts': { body: { accounts: list } }, 'GET /api/proxy/keys': { body: { keys: [] } } });
test('setup states describe saved accounts and offer a repair without claiming a model answered', { timeout: 90000 }, async t => {
const list = [
{ accountId: 'a1', provider: 'claude', displayName: 'Claude', status: 'ready', priority: 0 },
{ accountId: 'a2', provider: 'z.ai', displayName: 'Key', status: 'active', priority: 0 },
{ accountId: 'a3', provider: 'codex', displayName: 'Sign in', status: 'error', priority: 0, reason: 'The account needs to sign in again. Sign in again, or add another account.' },
{ accountId: 'a4', provider: 'claude', displayName: 'Unavailable', status: 'unavailable', priority: 0 },
{ accountId: 'a5', provider: 'z.ai', displayName: 'Disabled', status: 'disabled', priority: 0 },
];
const { f, b, wait } = await w15Fixture(t, { scenario: { '/api/config': config, '/api/proxy/admin/status': running, ...accounts(list), '/api/triage/route': { route: null } } });
await b.navigate(f.base + '/#/settings/model-setup?step=accounts');
await wait('document.querySelectorAll("#proxy-accounts li[data-account]").length === 5');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-accounts li[data-account] .badge")].map(e=>e.textContent)'), ['Saved', 'Saved', 'Sign-in needed', 'Not available', 'Disabled']);
assert.equal(await b.evaluate('document.querySelector("#setup-step-accounts").dataset.state'), 'bad');
assert.match(await b.evaluate('document.querySelector("#setup-step-accounts").textContent'), /Model setup has not checked whether a model answers/);
assert.match(await b.evaluate('document.querySelector("#setup-step-service [data-step-summary]").textContent'), /Service reachable/);
assert.match(await b.evaluate('document.querySelector("#proxy-apikey .help").textContent'), /does not check whether a model answers/i);
assert.equal(await b.evaluate('document.querySelector("#proxy-apikey button[type=submit]").textContent'), 'Save account');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-accounts li[data-account]")].map(row=>row.querySelectorAll("[data-account-fix]").length)'), [0, 0, 1, 1, 1]);
await b.evaluate('document.querySelector("[data-account=a3] [data-account-fix]").click(); true');
assert.deepEqual(await b.evaluate('[document.querySelector("#proxy-add-box").open, document.querySelector("#proxy-provider").value, document.activeElement.id]'), [true, 'codex', 'proxy-login-go']);
await b.evaluate('document.querySelector("[data-account=a5] [data-account-fix]").click(); true');
assert.deepEqual(await b.evaluate('[document.querySelector("#proxy-provider").value, document.activeElement.id]'), ['z.ai', 'proxy-apikey-key']);
assert.deepEqual(await b.evaluate('window.__writes'), []);
});
test('bad account repair respects provider availability and active sign-in', { timeout: 90000 }, async t => {
const list = ['codex', 'z.ai', 'other', 'missing'].map((provider, i) => ({ accountId: `a${i}`, provider, displayName: `Account ${i}`, status: 'error', priority: 0 }));
for (const width of [1440, 390]) {
const { f, b, wait } = await w15Fixture(t, { scenario: { '/api/config': config, '/api/proxy/admin/status': running, ...accounts(list), '/api/triage/route': { route: null },
'POST /api/proxy/login/start': { body: { url: 'https://claude.ai/oauth/authorize?state=s4', state: 's4' } }, 'POST /api/proxy/login/status': { body: { status: 'wait' } }, 'POST /api/proxy/login/cancel': { body: { status: 'cancelled' } } } });
await b.call('Page.addScriptToEvaluateOnNewDocument', { source: 'window.open = () => null;' });
await b.viewport(width, width === 390 ? 844 : 900);
await b.navigate(f.base + '/#/settings/model-setup?step=accounts');
await wait('document.querySelectorAll("#proxy-accounts li[data-account]").length === 4');
assert.equal(await b.evaluate('[...document.querySelectorAll("[data-account=a2], [data-account=a3]")].every(row => !row.querySelector("[data-account-fix]") && row.textContent.includes("provider is not available"))'), true);
const ax = await b.call('Accessibility.getFullAXTree');
assert.ok(ax.nodes.some(n => n.role?.value === 'button' && n.name?.value === 'Add account for Account 0'));
await b.evaluate('document.querySelector("[data-account=a0] [data-account-fix]").click(); document.querySelector("#proxy-login-start").requestSubmit(); true');
await wait('!document.querySelector("#proxy-signin").hidden');
assert.equal(await b.evaluate('document.querySelector("[data-account=a1] [data-account-fix]").disabled'), true);
await b.evaluate('document.querySelector("#proxy-login-cancel").click(); true');
await wait('document.querySelector("#proxy-signin").hidden');
await b.evaluate('document.querySelector("[data-account=a1] [data-account-fix]").focus(); true');
await b.key('Enter');
assert.deepEqual(await b.evaluate('[document.querySelector("#proxy-provider").value, document.activeElement.id, document.activeElement.checkVisibility()]'), ['z.ai', 'proxy-apikey-key', true]);
assert.deepEqual(await b.evaluate('window.__writes.map(w => w.path)'), ['/api/proxy/login/start', '/api/proxy/login/cancel']);
await b.close();
}
});
test('bad account without a provider or write access gives guidance without a repair shortcut', { timeout: 90000 }, async t => {
const list = [{ accountId: 'a1', provider: 'codex', displayName: 'Account', status: 'error', priority: 0 }];
for (const [owner, providers] of [[true, []], [false, publicProviders()]]) {
const { f, b, wait } = await w15Fixture(t, { scenario: { '/api/config': { send: owner, devOwner: true, proxyAdmin: owner, proxyAdminSend: owner }, '/api/proxy/admin/status': { ...running, providers }, 'GET /api/proxy/accounts': { body: { accounts: list, providers } }, '/api/triage/route': { route: null } } });
await b.navigate(f.base + '/#/settings/model-setup?step=accounts');
await wait('document.querySelector("#proxy-accounts li[data-account]")');
assert.equal(await b.evaluate('document.querySelector("[data-account-fix]")'), null);
assert.match(await b.evaluate('document.querySelector("#proxy-accounts li").innerText'), owner ? /provider is not available/ : /Ask the owner to add an account in Model setup/);
assert.deepEqual(await b.evaluate('window.__writes'), []);
await b.close();
}
});
const admin = b => b.evaluate('window.__writes.filter(w => w.path.startsWith("/api/proxy/admin/")).map(w => w.path.slice(17))');
// A primary button is one drawn with the accent fill, whatever its class: a submit is drawn that way too.
const primaries = b => b.evaluate(`(() => { const host = document.querySelector(".model-setup"), probe = document.createElement("button"); probe.className = "btn primary"; host.append(probe);
@@ -284,13 +347,13 @@ test('settings-s4: one press runs fetch, init, install and start in that order,
await wait('document.querySelector("#proxy-setup-go")?.dataset.next === "fetch"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-go").textContent'), 'Set up model service');
assert.deepEqual(await primaries(b), ['proxy-setup-go'], 'one primary button on the page');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll(".setup-step h3")].map(h => h.textContent)'), ['Model service', 'Accounts and models', 'Organizations', 'Triage']);
assert.deepEqual(await b.evaluate('[...document.querySelectorAll(".setup-step h3")].map(h => h.textContent)'), ['Model service', 'Accounts and models', 'Organizations', 'Triage', 'Worker model']);
assert.equal(await b.evaluate('document.querySelector("#setup-step-accounts").dataset.state'), 'wait');
await b.evaluate('document.querySelector("#proxy-setup-go").click(); true');
await wait('document.querySelector("#setup-step-service").dataset.state === "done"');
assert.deepEqual(await admin(b), ['fetch', 'init', 'unit/install', 'unit/start']);
assert.deepEqual(await b.evaluate('JSON.parse(window.__writes.find(w => w.path === "/api/proxy/admin/init").body)'), { generate: true });
assert.match(await b.evaluate('document.querySelector("#setup-step-service [data-step-summary]").textContent'), /^Running\. Answers on this machine, port 18432\.$/);
assert.match(await b.evaluate('document.querySelector("#setup-step-service [data-step-summary]").textContent'), /^Running\. Service reachable on this machine, port 18432\.$/);
await wait('document.querySelector("#setup-step-accounts").dataset.state === "next"');
assert.deepEqual(await primaries(b), ['proxy-login-go'], 'the one primary moves to step 2');
assert.equal(await b.evaluate('document.querySelector("#proxy-login-go").textContent'), 'Sign in to Claude');
@@ -311,13 +374,43 @@ test('settings-s4: a step that fails stops the run with Try again, keeps the fin
assert.deepEqual(await admin(b), ['init', 'unit/install'], 'nothing is retried by itself');
assert.equal(await b.evaluate('document.querySelector("#setup-step-service [data-step-badge]").textContent'), 'Setup stopped');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-setup-progress li")].map(li => li.className)'), ['ok', 'ok', 'bad', 'todo']);
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-setup-progress li")].map(li => li.firstChild.textContent)'), ['Downloaded and checked', 'Configured', 'Start the service: did not finish', 'Check it answers']);
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-setup-progress li")].map(li => li.firstChild.textContent)'), ['Downloaded and checked', 'Configured', 'Start the service: did not finish', 'Check the service']);
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-lines").checkVisibility() && document.querySelector("#proxy-setup-lines summary").textContent'), "The service's last lines");
assert.deepEqual(await primaries(b), ['proxy-setup-go']);
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-advanced summary").textContent'), 'Advanced: programs, port, timeouts');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-tech summary").textContent'), 'Technical details: release, service, history');
});
test('service progress names the unreachable service after a failed start', { timeout: 90000 }, async t => {
const down = { ...running, unit: { ...running.unit, listening: false }, next: ['unit-restart'] };
const { f, b, wait } = await w15Fixture(t, { scenario: { '/api/config': config, 'POST /api/proxy/admin/unit/start': { body: { receipt: { op: 'unit-start', time: T } } }, ...accounts([]), '/api/triage/route': { route: null } } });
await b.call('Page.addScriptToEvaluateOnNewDocument', { source: `(() => { let reads = 0; Object.defineProperty(window.__scenario, '/api/proxy/admin/status', { configurable: true, get() {
if (!window.__writes.some(w => w.path === '/api/proxy/admin/unit/start')) return ${JSON.stringify(installed)};
return ++reads === 1 ? { ...${JSON.stringify(down)}, busy: 'starting' } : ${JSON.stringify(down)};
} }); })();` });
await b.navigate(f.base + '/#/settings/model-setup');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next === "unit-start"');
await b.evaluate('document.querySelector("#proxy-setup-go").click(); true');
await wait('document.querySelector("#proxy-setup-progress [data-stage=answer]")?.className === "run"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-progress [data-stage=answer]").firstChild.textContent'), 'Checking the service…');
await wait('document.querySelector("#proxy-setup-progress [data-stage=answer]")?.className === "bad"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-progress [data-stage=answer]").firstChild.textContent'), 'Service not reachable yet');
});
test('service progress names a reachable service during setup', { timeout: 90000 }, async t => {
const { f, b, wait } = await w15Fixture(t, { scenario: { '/api/config': config, 'POST /api/proxy/admin/unit/start': { body: { receipt: { op: 'unit-start', time: T } } }, ...accounts([]), '/api/triage/route': { route: null } } });
await b.call('Page.addScriptToEvaluateOnNewDocument', { source: `(() => { let reads = 0; Object.defineProperty(window.__scenario, '/api/proxy/admin/status', { configurable: true, get() {
if (!window.__writes.some(w => w.path === '/api/proxy/admin/unit/start')) return ${JSON.stringify(installed)};
return ++reads === 1 ? { ...${JSON.stringify(running)}, next: ['fence-clear'], busy: 'starting' } : ${JSON.stringify(running)};
} }); })();` });
await b.navigate(f.base + '/#/settings/model-setup');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next === "unit-start"');
await b.evaluate('document.querySelector("#proxy-setup-go").click(); true');
await wait('document.querySelector("#proxy-setup-progress [data-stage=answer]")?.className === "ok"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-progress [data-stage=answer]").firstChild.textContent'), 'Service reachable on this machine');
await wait('document.querySelector("#setup-step-service").dataset.state === "done"');
});
test('settings-s4: ?step= opens a done card; without it a done card folds to its line', { timeout: 90000 }, async t => {
const list = [{ accountId: 'a1', provider: 'claude', displayName: 'Claude work', status: 'active', priority: 0 }];
for (const [hash, shown] of [['#/settings/model-setup?step=accounts', true], ['#/settings/model-setup', false]]) {
@@ -344,15 +437,15 @@ test('settings-s4: accounts read as rows with kind and health, the provider pick
await wait('document.querySelectorAll("#proxy-accounts li[data-account]").length === 4');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-accounts li")].map(li => [...li.children].map(c => c.innerText.replace(/\\s+/g, " ").trim()))'), [
// #1873 account-remove: the instance owner's rows end in Remove.
['Claude', 'Subscription Claude work', 'Ready', 'Remove'], ['ChatGPT / Codex', 'Subscription Codex home', 'Not working', 'Remove'], ['z.ai', 'API key z.ai key', 'Off', 'Remove'], ['Claude', 'Subscription Claude old', 'Unknown', 'Remove']]);
['Claude', 'Subscription Claude work', 'Saved', 'Remove'], ['ChatGPT / Codex', 'Subscription Codex home', 'Error', 'Remove', 'Add account'], ['z.ai', 'API key z.ai key', 'Disabled', 'Remove', 'Add account'], ['Claude', 'Subscription Claude old', 'Unknown', 'Remove']]);
assert.equal(await b.evaluate('document.querySelector("#setup-step-accounts").dataset.state'), 'bad', 'an account that is not working needs the owner');
assert.match(await b.evaluate('document.querySelector("#setup-step-accounts [data-step-summary]").textContent'), /ChatGPT \/ Codex \(subscription\): not working/);
assert.match(await b.evaluate('document.querySelector("#setup-step-accounts [data-step-summary]").textContent'), /ChatGPT \/ Codex \(subscription\): error/);
// #1873 account-remove: removing is built; the owner sees no note, only each row's Remove.
assert.doesNotMatch(await b.evaluate('document.querySelector("#setup-step-accounts").textContent'), /Removing an account is not built yet/);
assert.equal(await b.evaluate('document.querySelector("#proxy-remove-note").hidden'), true);
// The provider picks how to add: a sign-in for a subscription, Check and add for an API key.
await b.evaluate('(() => { document.querySelector("#proxy-add-box").open = true; const s = document.querySelector("#proxy-provider"); s.value = "z.ai"; s.dispatchEvent(new Event("change")); return true; })()');
assert.deepEqual(await b.evaluate('[document.querySelector("#proxy-login-go").checkVisibility(), document.querySelector("#proxy-apikey").checkVisibility(), document.querySelector("#proxy-apikey button[type=submit]").textContent]'), [false, true, 'Check and add']);
assert.deepEqual(await b.evaluate('[document.querySelector("#proxy-login-go").checkVisibility(), document.querySelector("#proxy-apikey").checkVisibility(), document.querySelector("#proxy-apikey button[type=submit]").textContent]'), [false, true, 'Save account']);
assert.ok((await primaries(b)).length <= 1, 'one primary button with the key form open');
await b.evaluate('(() => { const s = document.querySelector("#proxy-provider"); s.value = "codex"; s.dispatchEvent(new Event("change")); return true; })()');
assert.equal(await b.evaluate('document.querySelector("#proxy-login-go").textContent'), 'Sign in to ChatGPT / Codex');
@@ -387,8 +480,8 @@ test('settings-s4 chat-fail F3: an account row shows the accounts read\'s reason
await b.navigate(f.base + '/#/settings/model-setup?step=accounts');
await wait('document.querySelectorAll("#proxy-accounts li[data-account]").length === 4');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-accounts li")].map(li => [...li.children].map(c => c.textContent.replace(/\\s+/g, " ").trim()))'), [
['Claude', 'SubscriptionClaude work', 'Not available', 'Remove', out], ['ChatGPT / Codex', 'SubscriptionCodex home', 'Not working', 'Remove', signin],
['Claude', 'SubscriptionClaude spare', 'Not working', 'Remove'], ['ChatGPT / Codex', 'SubscriptionCodex work', 'Ready', 'Remove']], `${width}: the reason follows the state; absent or empty shows nothing`);
['Claude', 'SubscriptionClaude work', 'Not available', 'Remove', out, 'Add account'], ['ChatGPT / Codex', 'SubscriptionCodex home', 'Sign-in needed', 'Remove', signin, 'Add account'],
['Claude', 'SubscriptionClaude spare', 'Error', 'Remove', 'Add account'], ['ChatGPT / Codex', 'SubscriptionCodex work', 'Saved', 'Remove']], `${width}: the reason follows the state; absent or empty shows nothing`);
assert.equal(await b.evaluate('document.querySelectorAll("#proxy-accounts img").length + (window.__xss ?? 0)'), 0, 'the reason is text, never markup');
assert.equal(await b.evaluate('[...document.querySelectorAll("#proxy-accounts .account-reason")].every(p => p.checkVisibility() && p.getBoundingClientRect().right <= document.documentElement.clientWidth)'), true);
assert.equal(await b.evaluate('document.documentElement.scrollWidth <= document.documentElement.clientWidth'), true, `${width}: nothing scrolls sideways`);
Loaded 100 of 120 files, more files were not shown because too many files have changed in this diff. Show more