343 lines
20 KiB
Markdown
343 lines
20 KiB
Markdown
# Request lifecycle acceptance
|
||
|
||
`request-lifecycle.mjs` is an opt-in submission/observation runner for a later
|
||
lead-authorized isolated instance. It uses the existing owner process proof and
|
||
the running production host. It does not start a host, launch a scripted role,
|
||
select resources or a mode, approve a PRD, mint credentials or provision services.
|
||
|
||
Prepare a named development instance with its matching `instance.json`, an
|
||
existing factory source, installed tracker board and scoped credentials, and
|
||
real PM, CTO, chief, coder, reviewer and assistant routes. Start that instance
|
||
through the supported entry points in the authorized window. The owner supplies
|
||
an existing assistant thread. The runner requires an explicit instance, config,
|
||
business, expected owner, request quotation and evidence output directory.
|
||
`--owner` checks the configured identity; it does not authenticate anybody.
|
||
|
||
Example with deliberate placeholders (replace every uppercase value):
|
||
|
||
```sh
|
||
node packages/cli/live/request-lifecycle.mjs \
|
||
--instance pomo797 --config /ABS/CONFIG.json --business BUSINESS --owner OWNER \
|
||
--thread THREAD_UUID --submit --authorize-isolated-run \
|
||
--request-file /ABS/pomodoro-request.json --evidence /ABS/EVIDENCE
|
||
```
|
||
|
||
The owner selects and confirms the exact displayed plan in the signed-in Console.
|
||
Autonomous PRD approval remains with the distinct technical arbiter. Guided mode
|
||
requires the owner's answers and exact-version approval. No owner relays messages,
|
||
creates a task manually, or runs worker shell commands. The quotation in
|
||
`../tests/fixtures/pomodoro-request.json` preserves the owner's original wording
|
||
and provenance; it is not a raw runtime event export. The new runtime event and
|
||
message IDs are retained separately.
|
||
|
||
The runner retains each observed request revision and final task/review trails.
|
||
On complete reviewed delivery it reports the owner-only Console download URL.
|
||
Download there, then rerun observation with `--request EVENT_UUID` and
|
||
`--artifact /ABS/DOWNLOADED.html` in place of `--submit --thread THREAD_UUID`.
|
||
The runner checks the exact reviewed artifact length and SHA-256 and keeps a copy.
|
||
Unknown outcomes and timeouts retain evidence and fail; they never trigger a
|
||
replacement submission or approval. Output files are created exclusively.
|
||
|
||
For the lead's live acceptance, retain the original input/message, owner plan
|
||
confirmation, active approved snapshot, tracker create/assignment, code commit,
|
||
independent reviewer fetch/test record and verdict, closure, delivery receipt,
|
||
chief/assistant notification and downloaded bytes. Check that their project,
|
||
snapshot, task, commit, review and verdict IDs form the same chain. In an offline
|
||
browser, check the approved proposal's work/break durations, start, pause/resume,
|
||
reset, phase transition and cue after a user gesture, and no network requests or
|
||
external assets. The defaults are agent proposals approved in the product PRD.
|
||
Inspect real independent role/test receipts before declaring live acceptance.
|
||
|
||
The repository fixture runs a separate runtime process over the same persistent
|
||
fake tracker and real Store/repositories after each crash. Its scripted models
|
||
use shipped role authority, launch admission/limits, session manifests, real typed
|
||
tool subprocesses and Git bundles; coder and reviewer run the committed Node
|
||
acceptance test independently. It exercises the signed-in HTTP listener and
|
||
Chromium. It does not demonstrate live model quality or deployment readiness.
|
||
|
||
Crash restart follows the existing [operator recovery rule](../../bus/README.md):
|
||
there is no automatic stale writer-lock reclaim. The fixture kills and reaps only
|
||
its own process group, verifies that exact PID is dead, records the stale lock's
|
||
identity and removes only its lock/socket before restart. The database, WAL and
|
||
SHM are preserved. A live operator must establish the same facts and preserve
|
||
evidence before recovery; this harness does not change that production rule.
|
||
|
||
## ui797 isolated acceptance support (#797)
|
||
|
||
These three owner-run tools support a separately provisioned **test** instance.
|
||
They do not provision a business, tracker, provider proxy or service units. Keep
|
||
all runtime/evidence directories outside this checkout, the config directory and
|
||
the instance data root. Use owner-only directories (0700) and campaign files
|
||
(0600). No command targets .53 or installs/starts production services implicitly.
|
||
The test instance uses its own proxy; there is no shared-proxy attachment here.
|
||
|
||
Requirements: the repository's pinned Node dependencies, Node with `node:sqlite`,
|
||
OpenSSL, Chromium, NSS `certutil`, and Linux Bubblewrap (`/usr/bin/bwrap`) with
|
||
usable user, mount, network and PID namespaces on every browser host. All
|
||
examples assume a provisioner already created/validated the named instance through `scripts/bootstrap.sh --instance
|
||
ui797`, with a matching `instance.json`. Preserve the owner shell HOME and agent
|
||
markers.
|
||
For temporary test directories set `TMPDIR` using
|
||
`mktemp -d /mnt/storage/scratch/tmp/XXXXXX` (never `/tmp`).
|
||
|
||
### Local identity and TLS
|
||
|
||
In an approved setup shell, set `CFG` to the **test** config's absolute path and
|
||
`RUNROOT` to an existing owner-only runtime/evidence parent. The config's parent
|
||
must be named `mosaic-ui797`. The following creates a new identity directory,
|
||
private two-day CA/server certificates, a private RS256 signing key and the
|
||
0600 slot `<configDir>/secrets/mosaic-stack/ui797-local`. Existing files refuse.
|
||
It prints the JSON block to insert into the test config, without changing that
|
||
config or printing the client secret:
|
||
|
||
```sh
|
||
node packages/cli/live/isolated-identity.mjs prepare \
|
||
--instance ui797 --config "$CFG" --owner ui797-owner \
|
||
--runtime "$RUNROOT/identity" --tls-port 17443
|
||
```
|
||
|
||
Add that exact `console.sso` block through the approved test-config setup step.
|
||
The public origin is `https://127.0.0.1:17443`, issuer
|
||
`https://127.0.0.1:17443/oidc/`, behind the TLS forwarder to Console port 17440.
|
||
There is no identity listener on port 17442. Only `ui797-agent-owner` can be
|
||
issued, displayed as `ui797-owner`. Authorization immediately redirects back with a random,
|
||
single-use 60-second code, state and S256 PKCE; signed ID tokens last 300 seconds.
|
||
There is no refresh grant, alternate subject or live identity-provider integration.
|
||
|
||
Run these as separate normally supervised **test** services, not Console children
|
||
of a browser agent. Preserve the real production entry point and ancestry proof:
|
||
|
||
```sh
|
||
node packages/cli/live/isolated-identity.mjs serve --runtime "$RUNROOT/identity"
|
||
```
|
||
|
||
```sh
|
||
MOSAIC_CONFIG="$CFG" NODE_EXTRA_CA_CERTS="$RUNROOT/identity/ca.pem" \
|
||
node packages/webui/src/cli.mjs serve --business mvp3-ui797 \
|
||
--port 17440 --board http://127.0.0.1:17439 --sso --send
|
||
```
|
||
|
||
The identity service binds only 127.0.0.1:17443 and forwards non-OIDC paths to
|
||
127.0.0.1:17440 with the original Host/Origin/cookies. It adds no subject headers.
|
||
The Console validates discovery, JWKS, PKCE, nonce, token signature and its normal
|
||
session/owner rules. A missing bus, business or ancestry proof still refuses.
|
||
No human/Console binding shortcut or TLS-verification disable flag is used.
|
||
Trust `ca.pem` via `NODE_EXTRA_CA_CERTS` only in these test services, never globally.
|
||
Prepare and serve require the literal `ui797`, a `mosaic-ui797` config directory
|
||
and the current matching data-root marker. Serve revalidates persisted metadata;
|
||
the budget commands enforce the same binding.
|
||
Certificates expire after two days; do not overwrite identity material on restart.
|
||
|
||
### Step-by-step real browser CLI for tester agents
|
||
|
||
`start` creates an exclusive evidence directory and fresh profile, then starts a
|
||
private Unix-socket daemon. Subsequent invocations reuse that daemon. Either set
|
||
`UI797_EVIDENCE` or pass `--evidence DIR` on each command. Only the exact loopback
|
||
origins `http://127.0.0.1:17440` and `https://127.0.0.1:17443` are accepted. Every
|
||
page/subresource redirect must stay on the selected origin. Other targets and
|
||
WebSocket connections outside that origin are blocked before the handshake by a
|
||
forced local HTTP proxy with Chromium's loopback bypass disabled. It opens sockets
|
||
only to the selected origin, refuses HTTP upgrades, and permits CONNECT only to
|
||
the selected HTTPS host/port. Chromium and its TCP relay run in a mandatory Bubblewrap
|
||
network namespace, with its own loopback up and no external interfaces or routes.
|
||
The relay accepts only the forced proxy's TCP port and carries bytes over a 0600
|
||
Unix socket to that origin-checking host proxy. All other TCP and UDP, including
|
||
IPv4/IPv6 STUN/TURN on other loopback addresses, stay inside the empty namespace.
|
||
Bubblewrap's PID namespace and parent-death handling also contain its children.
|
||
The WebRTC flag alone does not prevent UDP on Chromium 156; it remains defense
|
||
in depth, alongside disabling QUIC. Request interception also guards
|
||
redirects and subresources. A detected
|
||
blocked network attempt makes the session fail until
|
||
`stop`. The driver never contacts 7332 or mosaic.woltje.com.
|
||
|
||
The browser trusts the test CA through a private NSS database under its evidence
|
||
directory. Bubblewrap starts with an empty mount view, adds read-only `/usr`,
|
||
libraries and selected NSS/font/certificate configuration, a minimal private
|
||
`/dev` and `/proc`, and private tmpfs `/tmp`, `/run` and home. Only this browser's
|
||
evidence/profile/download directories and proxy/control sockets are writable
|
||
host mounts. Host home, runtime directories, pathname sockets and credential
|
||
paths are hidden. An explicit environment allowlist supplies a private HOME and
|
||
XDG directories; host D-Bus/SSH-agent addresses and secrets are not inherited.
|
||
The private NSS database is mounted at the sandbox HOME's `.pki/nssdb`; the
|
||
owner shell HOME and host trust files remain unchanged. Certificate
|
||
verification stays enabled; a different CA is rejected. Startup refuses when the
|
||
required trust setup or namespace is unavailable; it does not fall back to a
|
||
certificate-error exception.
|
||
|
||
Exact commands for a tester (replace discovered refs and button names with the
|
||
current `snapshot` output):
|
||
|
||
```sh
|
||
export UI797_EVIDENCE="$RUNROOT/browser-a"
|
||
node packages/cli/live/ui-browser.mjs start \
|
||
--evidence "$UI797_EVIDENCE" --origin https://127.0.0.1:17443 \
|
||
--ca "$RUNROOT/identity/ca.pem"
|
||
node packages/cli/live/ui-browser.mjs goto '/#chat'
|
||
node packages/cli/live/ui-browser.mjs snapshot
|
||
node packages/cli/live/ui-browser.mjs click --text Chat
|
||
node packages/cli/live/ui-browser.mjs type r1-123 'Build a Pomodoro timer.'
|
||
node packages/cli/live/ui-browser.mjs press Enter
|
||
node packages/cli/live/ui-browser.mjs wait --text 'Choose how to proceed' --timeout 120
|
||
node packages/cli/live/ui-browser.mjs screenshot request-plan
|
||
node packages/cli/live/ui-browser.mjs click r1-456
|
||
node packages/cli/live/ui-browser.mjs downloads
|
||
node packages/cli/live/ui-browser.mjs stop
|
||
```
|
||
|
||
With a sole provider, the initial page load signs in immediately via the normal
|
||
OIDC redirects. `snapshot` reports visible accessibility-tree text and interactive
|
||
roles/names/disabled states, with document-scoped stable refs. Snapshot again after
|
||
navigation; stale refs refuse. `click --text` needs exactly one enabled exact
|
||
accessible-name match. `type` focuses with a real mouse click and inserts text at
|
||
the current caret via CDP Input; it appends rather than clears. Supported keys:
|
||
Enter, Tab, Escape, Backspace, Delete, Home/End, PageUp/Down, arrows, Space and
|
||
letters/digits, with Ctrl/Control, Alt, Meta or Shift modifiers (for example
|
||
`press Ctrl+A`, `press Backspace` to clear a focused field). Clicks use
|
||
real CDP mouse input. No page-function calls or fetch/state mocks are injected.
|
||
`wait` is bounded to 120 seconds per invocation. No uncertain write is replayed.
|
||
A click that opens a native alert/confirm/prompt returns `{dialog:{type,message}}`;
|
||
answer it with `dialog accept` or `dialog dismiss` (for example the Console's
|
||
Resume confirmation). While a dialog is open every other command except `stop`
|
||
refuses, and no screenshot is taken because the page is blocked; the receipt
|
||
records the dialog type, message and answer.
|
||
|
||
Each accepted action, including failed commands, start and stop, writes
|
||
`NNNNN-before.png`, `NNNNN-after.png` and one `actions.jsonl` receipt with result
|
||
status. The journal omits URLs, query strings, text inputs, headers, bodies and
|
||
cookies. Screenshots show page contents, so keep all evidence private and avoid
|
||
entering secrets. Snapshots omit input values. Downloads are saved under browser
|
||
GUIDs and listed with absolute paths, byte lengths and SHA-256; compare these
|
||
with the product's delivery receipt. Partial downloads refuse. Profiles/evidence
|
||
are retained on stop. An interrupted daemon is not automatically replaced or its
|
||
commands replayed: inspect receipts and use a new evidence directory for a new
|
||
explicitly authorized session. `stop` remains available after a refused network
|
||
request. These tools do not yet open downloaded artifacts in an offline target.
|
||
|
||
### Durable metered campaign stop
|
||
|
||
An owner creates a single 0600 `campaign.json` **before launching inference**:
|
||
|
||
```json
|
||
{
|
||
"version": 1,
|
||
"instance": "ui797",
|
||
"config": "/ABS/CONFIG/mosaic-ui797/config.json",
|
||
"business": "mvp3-ui797",
|
||
"owner": "ui797-owner",
|
||
"start": "2026-11-06T02:00:00.000Z",
|
||
"end": "2026-11-06T03:00:00.000Z",
|
||
"caps": [
|
||
{"provider":"proxy","route":"glm","model":"glm-5.3-flash","tokens":100000000},
|
||
{"provider":"proxy","route":"anthropic","model":"claude-opus-5-5","tokens":10000000},
|
||
{"provider":"proxy","route":"anthropic","model":"claude-sonnet-5-5","tokens":10000000}
|
||
]
|
||
}
|
||
```
|
||
|
||
Use the approved actual absolute config path, canonical UTC campaign start/end
|
||
and **all** actual model routes; the dates above illustrate the schema. Caps are
|
||
positive safe integers: at most 100,000,000 for exactly
|
||
`proxy/glm/glm-5.3-flash`, and 10,000,000 for every other approved model. Lower
|
||
caps allow zero-spend tests. Only proxy GLM, Claude (`anthropic` route, `claude-*`)
|
||
and Codex (`codex` route, `gpt-*`) bindings are accepted. One provider/model has
|
||
one allowance across all routes; conflicting bindings refuse. The campaign must
|
||
end by **2026-11-07T06:00:00.000Z**; earlier windows narrow that absolute cutoff,
|
||
which checks also enforce independently.
|
||
The test bus store must exist before initialization. Run the supervisor outside
|
||
model-writable settings/data in a normally supervised owner service:
|
||
|
||
```sh
|
||
node packages/cli/live/budget-supervisor.mjs init \
|
||
--campaign "$RUNROOT/campaign.json" --ledger "$RUNROOT/campaign-ledger"
|
||
node packages/cli/live/budget-supervisor.mjs serve \
|
||
--campaign "$RUNROOT/campaign.json" --ledger "$RUNROOT/campaign-ledger" --interval 1
|
||
```
|
||
|
||
On-demand commands (safe for a tester to request; a triggered halt still requires
|
||
the normal owner-service ancestry):
|
||
|
||
```sh
|
||
node packages/cli/live/budget-supervisor.mjs check \
|
||
--campaign "$RUNROOT/campaign.json" --ledger "$RUNROOT/campaign-ledger"
|
||
node packages/cli/live/budget-supervisor.mjs status \
|
||
--campaign "$RUNROOT/campaign.json" --ledger "$RUNROOT/campaign-ledger"
|
||
```
|
||
|
||
`status` prints persisted totals from the last successful accounting check versus
|
||
each model's `tokens` cap, the end time and latch state. It does not refresh usage.
|
||
The supervisor opens only the pinned test bus SQLite store read-only, using the
|
||
same `turnRecords` accounting as token-limit (input + output + cacheWrite +
|
||
cacheRead). Proxy telemetry is excluded. A run's immutable bundle manifest supplies
|
||
its actual `proxyRoute`; fallback/drift/uncapped models refuse. Delivered/read
|
||
messages and ended runs detect missing completed usage. An in-flight begin waits
|
||
for its outcome; unknown/malformed completed usage stops the campaign.
|
||
|
||
Each provider/route/model has a separate append-only ledger, durable across
|
||
rounds/restarts. Campaign bytes, config bytes, root identity and store identity are pinned;
|
||
changed caps/epoch, rewritten/disappeared usage and damaged ledger files stop.
|
||
Checks recompute each journal entry's tokens and totals from authoritative usage;
|
||
a valid-JSON token mismatch durably latches and halts.
|
||
Initialization exclusively writes `campaign.json.binding.json` beside the owner
|
||
campaign, outside the ledger volume, before creating the ledger. Preserve this
|
||
0600 file and the campaign: they independently pin the target and ledger path
|
||
when accounting storage is unavailable. Checks never recreate that file or an
|
||
accounting epoch. The version-2 ledger manifest independently pins config bytes,
|
||
so missing, unreadable, malformed or wrong-mode sidecars stop and still attempt halt against
|
||
the verified target. Each report and halt receipt includes `evidenceErrors`;
|
||
damaged files are preserved. An intact sidecar must agree with both the owner
|
||
campaign bytes and ledger campaign/epoch. A running service retains its verified
|
||
binding and epoch while revalidating the actual target on every tick. Legacy
|
||
version-1 ledgers require their intact independent sidecar to verify config bytes;
|
||
a sidecar-less legacy ledger refuses targeting. Never reinitialize or reset an
|
||
existing campaign to work around missing evidence.
|
||
Initialization is exclusive. There is no reset, raise or resume command. A stale
|
||
`check.lock` after interruption requires owner diagnosis; never auto-reclaim it.
|
||
An existing or uncreatable lock prevents accounting and still asserts the sticky
|
||
halt against the verified ui797 binding on every check/serve tick. The lock is
|
||
preserved, including when a stop was already latched.
|
||
A changed target config refuses rather than halting an unverified data root.
|
||
On `campaign-instance-changed`, the owner must diagnose the campaign's pin,
|
||
binding (including root and store identity), config bytes and ledger target against
|
||
the original evidence. Restore the original files and target identities if they
|
||
can be verified, or end the campaign; there is no supported retarget or recovery
|
||
command. Never reinitialize, reset or raise the existing campaign, or auto-reclaim
|
||
its lock. A refused target is not proof of a halt or quiescence.
|
||
|
||
At any cap, deadline or accounting refusal, the supervisor fsyncs `stop.json`
|
||
before invoking the existing `mosaic halt --business ... --json` production CLI
|
||
with the selected test `MOSAIC_CONFIG`. The CLI's human-shell/ancestry proof stays
|
||
intact. A refusal/unknown stop is retained as a failed `halt-N.json` receipt and
|
||
`check` exits nonzero; the latch remains. Missing, renamed or nonprivate ledger
|
||
directories still trigger halt using the independently validated owner binding.
|
||
If storage cannot persist a latch or receipt, it explicitly reports
|
||
`stop-latch-not-durable` and/or `stop-receipt-not-durable` after attempting halt;
|
||
durable evidence then needs
|
||
owner repair before any further campaign work. Do not strip markers, direct-bind a principal,
|
||
write processing state, or treat that receipt as quiescence. A properly installed
|
||
owner service must perform/confirm halt. `serve` stays running on check/halt/storage
|
||
errors, reports each failed tick and reasserts halt on every tick, including after
|
||
a modelled Resume. It retains an in-memory stop if storage subsequently recovers.
|
||
Every attempt revalidates the target; invalid, changed or foreign bindings refuse
|
||
targeting. It never resumes. Manual Resume is outside the campaign protocol.
|
||
|
||
This is a **metered stop**, not strict inference admission or a hard provider
|
||
ceiling. Concurrent/in-flight work, transcript accounting and the 0.1–10-second
|
||
check interval can overshoot a cap. Install the native token-limit controls as
|
||
well, retain accounting evidence, and use only the lead-approved metered-stop
|
||
acceptance envelope. These tools add no model-dispatch hook or network quota gate.
|
||
|
||
Focused verification (no model spend, no remote access):
|
||
|
||
```sh
|
||
TMPDIR=$(mktemp -d /mnt/storage/scratch/tmp/XXXXXX) \
|
||
node --test --test-concurrency=1 packages/cli/live/*.test.mjs
|
||
```
|
||
|
||
Run these files sequentially: their integration fixtures exclusively bind the
|
||
fixed test ports 17440/17443 and fail if another process owns them. The browser
|
||
SSO test starts the unchanged production `cli.mjs serve --sso` without `--send`;
|
||
it proves genuine sign-in, not authenticated broker writes or real-model quality.
|
||
The budget tests inject a tiny cap and a fake halt at the supervisor boundary;
|
||
they also prove the real CLI refuses an agent environment. No live business is
|
||
halted by these tests.
|