mosaicstack/stack
1
0
Fork 1
Code Issues 287 Pull Requests 16 Actions Packages Projects Releases 15 Wiki Activity

docs: establish canonical documentation architecture (#1210)
ci/woodpecker/push/publish Pipeline failed
Details

Browse Source
This commit was merged in pull request #1210.
This commit is contained in:
mos-dt-0
2026-08-13 17:56:13 +00:00
parent f82307c4dc
commit 7a6fb024b4
241 changed files with 4722 additions and 1579 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guides/admin-guide.md
-376
View File
@@ -1,376 +0,0 @@
# Mosaic Stack — Admin Guide
## Table of Contents
1. [User Management](#user-management)
2. [System Health Monitoring](#system-health-monitoring)
3. [Provider Configuration](#provider-configuration)
4. [MCP Server Configuration](#mcp-server-configuration)
5. [Environment Variables Reference](#environment-variables-reference)
6. [Local Fleet Canary](./fleet-local-canary.md)
---
## User Management
Admins access user management at `/admin` in the web dashboard. All admin
endpoints require a session with `role = admin`.
### Creating a User
**Via the web admin panel:**
1. Navigate to `/admin`.
2. Click **Create User**.
3. Enter name, email, password, and role (`admin` or `member`).
4. Submit.
**Via the API:**
```http
POST /api/admin/users
Content-Type: application/json
{
"name": "Jane Doe",
"email": "jane@example.com",
"password": "securepassword",
"role": "member"
}
```
Passwords are hashed by BetterAuth before storage. Passwords are never stored in
plaintext.
### Roles
| Role | Permissions |
| -------- | --------------------------------------------------------------------- |
| `admin` | Full access: user management, health, all agent tools |
| `member` | Standard user access; agent tool set restricted by `AGENT_USER_TOOLS` |
### Updating a User's Role
```http
PATCH /api/admin/users/:id/role
Content-Type: application/json
{ "role": "admin" }
```
### Banning and Unbanning
Banned users cannot sign in. Provide an optional reason:
```http
POST /api/admin/users/:id/ban
Content-Type: application/json
{ "reason": "Violated terms of service" }
```
To lift a ban:
```http
POST /api/admin/users/:id/unban
```
### Deleting a User
```http
DELETE /api/admin/users/:id
```
This permanently deletes the user. Related data (sessions, accounts) is
cascade-deleted. Conversations and tasks reference the user via `owner_id`
which is set to `NULL` on delete (`set null`).
---
## System Health Monitoring
The health endpoint is available to admin users only.
```http
GET /api/admin/health
```
Sample response:
```json
{
"status": "ok",
"database": { "status": "ok", "latencyMs": 2 },
"cache": { "status": "ok", "latencyMs": 1 },
"agentPool": { "activeSessions": 3 },
"providers": [{ "id": "ollama", "name": "ollama", "available": true, "modelCount": 3 }],
"checkedAt": "2026-03-15T12:00:00.000Z"
}
```
`status` is `ok` when both database and cache pass. It is `degraded` when either
service fails.
The web admin panel at `/admin` polls this endpoint and renders the results in a
status dashboard.
---
## Provider Configuration
Providers are configured via environment variables and loaded at gateway startup.
No restart-free hot reload is supported; the gateway must be restarted after
changing provider env vars.
### Ollama
Set `OLLAMA_BASE_URL` (or the legacy `OLLAMA_HOST`) to the base URL of your
Ollama instance:
```env
OLLAMA_BASE_URL=http://localhost:11434
```
Specify which models to expose (comma-separated):
```env
OLLAMA_MODELS=llama3.2,codellama,mistral
```
Default when unset: `llama3.2,codellama,mistral`.
The gateway registers Ollama models using the OpenAI-compatible completions API
(`/v1/chat/completions`).
### Custom Providers (OpenAI-compatible APIs)
Any OpenAI-compatible API (LM Studio, llama.cpp HTTP server, etc.) can be
registered via `MOSAIC_CUSTOM_PROVIDERS`. The value is a JSON array:
```env
MOSAIC_CUSTOM_PROVIDERS='[
{
"id": "lmstudio",
"name": "LM Studio",
"baseUrl": "http://localhost:1234",
"models": ["mistral-7b-instruct"]
}
]'
```
Each entry must include:
| Field | Required | Description |
| --------- | -------- | ----------------------------------- |
| `id` | Yes | Unique provider identifier |
| `name` | Yes | Display name |
| `baseUrl` | Yes | API base URL (no trailing slash) |
| `models` | Yes | Array of model ID strings to expose |
| `apiKey` | No | API key if required by the endpoint |
### Testing Provider Connectivity
From the web admin panel or settings page, click **Test** next to a provider.
This calls:
```http
POST /api/agent/providers/:id/test
```
The response includes `reachable`, `latencyMs`, and optionally
`discoveredModels`.
---
## MCP Server Configuration
The gateway can connect to external MCP (Model Context Protocol) servers and
expose their tools to agent sessions.
Set `MCP_SERVERS` to a JSON array of server configurations:
```env
MCP_SERVERS='[
{
"name": "my-tools",
"url": "http://localhost:3001/mcp",
"headers": {
"Authorization": "Bearer my-token"
}
}
]'
```
Each entry:
| Field | Required | Description |
| --------- | -------- | ----------------------------------- |
| `name` | Yes | Unique server name |
| `url` | Yes | MCP server URL (`/mcp` endpoint) |
| `headers` | No | Additional HTTP headers (e.g. auth) |
On gateway startup, each configured server is connected and its tools are
discovered. Tools are bridged into the Pi SDK tool format and become available
in agent sessions.
The gateway itself also exposes an MCP server endpoint at `POST /mcp` for
external clients. Authentication requires a valid BetterAuth session (cookie or
`Authorization` header).
---
## Environment Variables Reference
### Required
| Variable | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------- |
| `BETTER_AUTH_SECRET` | Secret key for BetterAuth session signing. Must be set or gateway will not start. |
| `DATABASE_URL` | Runtime-only PostgreSQL connection injected from the dedicated deployment secret; no default or inline DSN. |
### Gateway
| Variable | Default | Description |
| --------------------- | ------------------------ | ---------------------------------------------- |
| `GATEWAY_PORT` | `14242` | Port the gateway listens on |
| `GATEWAY_CORS_ORIGIN` | `http://localhost:3000` | Allowed CORS origin for browser clients |
| `BETTER_AUTH_URL` | `http://localhost:14242` | Public URL of the gateway (used by BetterAuth) |
### SSO (Optional)
| Variable | Description |
| --------------------------- | ---------------------------------------------------------------------------- |
| `AUTHENTIK_CLIENT_ID` | Authentik OAuth2 client ID |
| `AUTHENTIK_CLIENT_SECRET` | Authentik OAuth2 client secret |
| `AUTHENTIK_ISSUER` | Authentik OIDC issuer URL |
| `AUTHENTIK_TEAM_SYNC_CLAIM` | Optional claim used to derive team sync data (defaults to `groups`) |
| `WORKOS_CLIENT_ID` | WorkOS OAuth client ID |
| `WORKOS_CLIENT_SECRET` | WorkOS OAuth client secret |
| `WORKOS_ISSUER` | WorkOS OIDC issuer URL |
| `WORKOS_TEAM_SYNC_CLAIM` | Optional claim used to derive team sync data (defaults to `organization_id`) |
| `KEYCLOAK_CLIENT_ID` | Keycloak OAuth client ID |
| `KEYCLOAK_CLIENT_SECRET` | Keycloak OAuth client secret |
| `KEYCLOAK_ISSUER` | Keycloak realm issuer URL |
| `KEYCLOAK_TEAM_SYNC_CLAIM` | Optional claim used to derive team sync data (defaults to `groups`) |
| `KEYCLOAK_SAML_LOGIN_URL` | Optional SAML login URL used when OIDC is unavailable |
Each OIDC provider requires its client ID, client secret, and issuer URL together. If only part of a provider configuration is set, gateway startup logs a warning and that provider is skipped. Keycloak can fall back to SAML when `KEYCLOAK_SAML_LOGIN_URL` is configured.
### Agent
| Variable | Default | Description |
| ------------------------ | --------------- | ------------------------------------------------------- |
| `AGENT_FILE_SANDBOX_DIR` | `process.cwd()` | Root directory for file/git/shell tool access |
| `AGENT_SYSTEM_PROMPT` | — | Platform-level system prompt injected into all sessions |
| `AGENT_USER_TOOLS` | all tools | Comma-separated allowlist of tools for non-admin users |
### Providers
| Variable | Default | Description |
| ------------------------- | ---------------------------- | ------------------------------------------------ |
| `OLLAMA_BASE_URL` | — | Ollama API base URL |
| `OLLAMA_HOST` | — | Alias for `OLLAMA_BASE_URL` (legacy) |
| `OLLAMA_MODELS` | `llama3.2,codellama,mistral` | Comma-separated Ollama model IDs |
| `MOSAIC_CUSTOM_PROVIDERS` | — | JSON array of custom OpenAI-compatible providers |
### Memory and Embeddings
| Variable | Default | Description |
| ----------------------- | --------------------------- | ---------------------------------------------------- |
| `OPENAI_API_KEY` | — | API key for OpenAI embedding and summarization calls |
| `EMBEDDING_API_URL` | `https://api.openai.com/v1` | Base URL for embedding API |
| `EMBEDDING_MODEL` | `text-embedding-3-small` | Embedding model ID |
| `SUMMARIZATION_API_URL` | `https://api.openai.com/v1` | Base URL for log summarization API |
| `SUMMARIZATION_MODEL` | `gpt-4o-mini` | Model used for log summarization |
| `SUMMARIZATION_CRON` | `0 */6 * * *` | Cron schedule for log summarization (every 6 hours) |
| `TIER_MANAGEMENT_CRON` | `0 3 * * *` | Cron schedule for log tier management (daily at 3am) |
### MCP
| Variable | Description |
| ------------- | ------------------------------------------------ |
| `MCP_SERVERS` | JSON array of external MCP server configurations |
### Plugins
| Variable | Description |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `DISCORD_BOT_TOKEN` | Discord bot token (enables Discord plugin) |
| `DISCORD_SERVICE_TOKEN` | Required high-entropy service credential used to authenticate and sign Discord ingress; inject through the approved secret mechanism only |
| `DISCORD_SERVICE_USER_ID` | Required Mosaic service-principal user ID that owns persisted Discord conversations; the original Discord user ID remains audit metadata |
| `DISCORD_GUILD_ID` | Discord guild/server ID |
| `DISCORD_GATEWAY_URL` | Gateway URL for Discord plugin to call (default: `http://localhost:14242`) |
| `DISCORD_ALLOWED_GUILD_IDS` | Required comma-separated Discord guild snowflake allowlist; default-deny |
| `DISCORD_ALLOWED_CHANNEL_IDS` | Required comma-separated Discord channel snowflake allowlist; default-deny |
| `DISCORD_ALLOWED_USER_IDS` | Required comma-separated Discord user snowflake allowlist; default-deny |
| `DISCORD_INTERACTION_BINDINGS` | Required JSON bindings from guild/channel to logical agent and paired Discord users with `viewer`, `operator`, or `admin` roles |
| `DISCORD_MESSAGE_RATE_LIMIT_PER_MINUTE` | Optional positive integer; authorized turns per guild/channel/user each minute (default: `30`) |
| `DISCORD_THREAD_RATE_LIMIT_PER_MINUTE` | Optional positive integer; mention-thread routes per guild/channel/user each minute (default: `5`) |
| `TELEGRAM_BOT_TOKEN` | Telegram bot token (enables Telegram plugin) |
| `TELEGRAM_GATEWAY_URL` | Gateway URL for Telegram plugin to call |
### Discord ingress security
When `DISCORD_BOT_TOKEN` is configured, `DISCORD_SERVICE_TOKEN`, `DISCORD_SERVICE_USER_ID`, and all three Discord allowlists are required. Gateway startup fails rather than enabling a broad or unauthenticated remote-control surface. The service user ID identifies a provisioned Mosaic service principal for persistence; the original Discord user ID is retained in ingress audit metadata. The service token is a secret supplied by the approved runtime secret mechanism and is never committed or logged.
Inbound Discord messages must originate from an allowed guild and configured parent channel, come from an allowed and paired user whose role permits sending, and carry a signed envelope containing the native Discord message ID and a generated correlation ID. Attachment references are limited in count, metadata size, field length, and declared size; only query-free HTTPS URLs without credentials or fragments are accepted, so bearer or presigned URLs never reach persistence or an agent prompt. The gateway validates the service identity, envelope signature, allowlists, pairing, and role again before dispatching. Replayed Discord message IDs are rejected during the bounded ingress replay window. Durable inbox/idempotency retention is introduced with Tess durable state.
Configured channels are dedicated agent interaction surfaces. An authorized untagged message routes to the bound logical agent and the response returns in that channel. Mentioning the bot on a normal channel message creates a public Discord thread, or reuses the thread already attached to that same message; the response and later thread messages stay in that thread without repeated mentions. A normal channel's category is not an authorization parent—only a Discord thread inherits authorization from its configured parent channel. Runtime control commands such as `/approve` and `/stop <approval>` remain on the current channel/thread because they target that durable session rather than opening a new topic.
Authorization and per-user/channel rate limits are evaluated before thread creation, so an unlisted guild/channel/user, unpaired user, `viewer`, or rate-limited sender cannot create bot threads or dispatch gateway work. The bot needs Discord permissions to view/send in configured channels and create/send in public threads. If thread creation fails, the turn is not dispatched because the requested response destination cannot be honored.
Conversation handles use the configured logical agent plus Discord channel/thread identity. They do not contain a Claude, Codex, Pi, OpenCode, model, process, or runtime-provider identifier; changing the runtime behind the logical session therefore does not require reconnecting the Discord bot.
#### Interaction binding format
`DISCORD_INTERACTION_BINDINGS` must be a non-empty JSON array. Each item requires `instanceId`, trusted `agentConfigId`, `guildId`, `channelId`, and a non-empty `pairedUsers` object. `instanceId` is the configured logical-agent name; its trusted database `agentConfigId` must resolve to an agent configuration with exactly that name, preserving provider/model/prompt/tool selection per binding. The guild/channel must also appear in their corresponding allowlists. IDs below are placeholders:
```json
[
{
"instanceId": "interaction-agent",
"agentConfigId": "agent-config-id",
"guildId": "guild-id",
"channelId": "channel-id",
"pairedUsers": {
"discord-user-id": {
"role": "operator",
"mosaicUserId": "mosaic-user-id"
}
}
}
]
```
| Pairing role | Send message | Create/continue thread | Approve | Stop |
| ------------ | ------------ | ---------------------- | ------- | ---- |
| `viewer` | No | No | No | No |
| `operator` | Yes | Yes | No | No |
| `admin` | Yes | Yes | Yes | Yes |
A legacy role-only value such as `"discord-user-id": "operator"` remains valid for non-privileged ingress. Approval and stop require the object form with a provisioned `mosaicUserId`; gateway policy checks that Mosaic identity and consumes one exact-action approval once. Do not make the Discord service principal an approving administrator.
After changing bindings or allowlists, restart the gateway/plugin through the normal service manager and verify both Discord and gateway connectivity. The adapter reports `connected` only when both links are ready, `degraded` when one is ready, and `disconnected` when neither is ready. Test one authorized untagged channel turn, one mention-created thread, one thread follow-up, and one unauthorized user denial without using production credential values in logs or evidence.
### Session retention and garbage collection
Session cleanup is scoped to one session identifier and only removes that session's Valkey keys and demotes that session's hot logs. Gateway startup and scheduled jobs do not perform global session cleanup; startup removes legacy repeatable `session-gc` schedules created by older deployments. The `/gc` command is intentionally disabled until a distinct global-retention job supplies explicit authorization and audit evidence. This prevents one tenant or session's cleanup from changing another's retained data.
### Observability
| Variable | Default | Description |
| ----------------------------- | ----------------------- | -------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OpenTelemetry collector endpoint |
| `OTEL_SERVICE_NAME` | `mosaic-gateway` | Service name in traces |
### Web App
| Variable | Default | Description |
| ------------------------- | ------------------------ | -------------------------------------- |
| `NEXT_PUBLIC_GATEWAY_URL` | `http://localhost:14242` | Gateway URL used by the Next.js client |
### Coordination
| Variable | Default | Description |
| ----------------------- | ----------------------------- | ------------------------------------------ |
| `MOSAIC_WORKSPACE_ROOT` | monorepo root (auto-detected) | Root path for mission workspace operations |
docs/guides/fleet-local-canary.md
+1 -1
View File
@@ -129,7 +129,7 @@ systemctl --user restart mosaic-agent@<name>
Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
[Upgrade Safety & Recovery](../ADMIN-GUIDE/operations/upgrade-safety-and-recovery.md).
## Release Preflight
docs/guides/lease-broker-operations.md
-44
View File
@@ -1,44 +0,0 @@
# Lease broker operations
Place the socket and state file in a dedicated directory with mode `0700`. Start the packaged daemon with:
```bash
python3 "$MOSAIC_HOME/tools/lease-broker/daemon.py" \
--socket /run/user/1000/mosaic-lease/broker.sock \
--state /run/user/1000/mosaic-lease/state.json
```
The broker refuses an existing parent directory whose mode is not exactly `0700`, an existing state file not at `0600`, corrupt/incompatible state, or an already-existing socket path. After bind it sets the socket to `0600`. It never silently unlinks a pre-existing socket. On normal termination it unlinks only the socket inode it created, so it does not remove a replacement path.
Before launching Claude, Claudex, or Pi, export the socket path; `mosaic` then runs the runtime through the packaged register-and-exec wrapper:
```bash
export MOSAIC_LEASE_BROKER_SOCKET=/run/user/1000/mosaic-lease/broker.sock
mosaic claude # or: mosaic claudex, mosaic yolo claudex, mosaic pi
```
The wrapper obtains a broker-minted session ID, creates a private `generation-<session>.state` file beside the socket, and `exec`s the runtime without changing its PID/starttime anchor. The all-tools Claude `PreToolUse` hook and Pi `tool_call` handler inherit that identity and read the current generation from the file. Claudex retains its isolated proxy environment and config directory; Mosaic merges the mandatory all-tools and compaction-lifecycle hooks into that isolated `settings.json` before invoking the same wrapper. PRDY init/update, QA remediation, coord, orchestrator, and fleet launchers also converge on this boundary. Broker registration failure, unsafe isolated settings, unsafe generation state, or missing identity denies launch/tool execution fail-closed; broker timeout/unavailability and malformed replies also block tools.
Claude `PreCompact` and `SessionStart(compact)` hooks and Pi pre-/post-compaction handlers invoke `revoke-lease.py`. Pi `session_start` reload/new/resume/fork and Claude resume/clear advance the locked generation before revocation, so a replacement session inherits no lease even when PID/starttime stay unchanged. Do not invoke the revoker manually as a way to restore authority; it only removes authority. If a lifecycle hook reports failure, stop consequential work and repair broker/generation-state availability before re-verification.
Run the permanent launch inventory locally with:
```bash
python3 packages/mosaic/framework/tools/lease-broker/check-runtime-launches.py --root .
```
The same check runs in the Mosaic package test suite and therefore in root CI. Any direct Claude/Pi binary launch must be replaced with `launch-runtime.py`, `execLeaseGatedRuntime`, or the gated `mosaic` runtime command; do not add static allowlist exceptions.
Clients must complete the request boundary before waiting for a reply. After sending the single JSON object and its terminating newline, the client **MUST half-close the socket's write side** (`shutdown(SHUT_WR)` in POSIX clients; `socket.end()` in Node) and only then await the response. Merely calling `write()` and waiting is invalid: the broker waits for EOF to enforce the exact-one-frame contract and fails closed at its one-second deadline. Do not replace `end()` with `write()` in client helpers. A delayed second frame remains malformed and is rejected.
`mosaic_context_recover` is the only unverified mutator class. Its durable `mosaic-context-refresh` skill is a thin wrapper over `tools/lease-broker/recover-context.py`: `begin` has the broker rebuild the validated `B_payload`/`H_payload`, revoke first, and mint a new `PENDING_DELIVERY` receipt challenge; `complete` accepts neither receipt text nor a challenge argument. Claude maps only the exact direct recovery executable/validated arguments to this exempt tool identity; ordinary `Bash` remains gated. Pi exposes only the `mosaic_context_recover` custom tool; ordinary `bash` and all other tools remain gated. A normal-path receipt cannot be replayed through recovery because each retry begins a distinct recovery cycle and recovery completion cannot receive caller-presented evidence.
Production daemon startup creates a separate private observer socket unless a test-only `--test-observer-file` fixture is selected. Claude's Stop hook sends its exact latest assistant entry and Pi's `message_end` handler sends only finalized assistant content to that authenticated transport; the broker public socket never accepts message text. This is byte-build and private out-of-process harness wiring only: do not activate it against a live daemon, live socket, systemd service, tmux session, or model-output stream outside the controlled integration procedure.
Receipt honesty is load-bearing: absent, malformed, prefix-truncated, and observable adapter-mutated terminal receipts do not promote. A tail-only case is non-promoting only where the concrete terminal payload is malformed or observably incomplete. A tail-preserving middle drop is **not receipt-detectable**; it is the disclosed T-C injection-contract residual deferred to WI-7 server-side evidence. The receipt remains a T-A delivery/liveness prerequisite, never a safety, obedience, or residency proof. The framework skill is source-resident and bridge-projected on install/upgrade; do not hand-create a live runtime symlink.
After a runtime exits, its `generation-<session>.state` file may be removed only after verifying that no process for that broker-minted session remains; stale files carry no lease authority but should be retained during incident analysis. After a broker crash, preserve the protected state file and restart only after verifying that no broker owns the socket. Restart intentionally clears all volatile VERIFIED leases. A leftover socket requires an operator to verify the owning service is stopped and remove that exact socket deliberately. Corrupt, oversized, symlinked, or non-regular state fails closed; do not overwrite it. Preserve it for incident review and establish new state only through an explicit operational decision, which invalidates prior sessions and tokens.
## Security posture
Directory `0700` plus socket/state `0600` is built-in same-principal hardening only: it excludes other UIDs but does **not** stop the same UID from unlinking and counterfeiting the socket. It therefore does not close T-C same-UID replacement. WI-1 does not provide a distinct-principal boundary. A stronger distinct-principal deployment requires an external protected proxy, ACL, or service boundary that clients cannot unlink or rebind and that preserves the authenticated client identity required by the broker's `SO_PEERCRED` and ancestry checks. Server-side branch protection remains the irreducible backstop.
docs/guides/mos-connector-lease-operations.md
-43
View File
@@ -1,43 +0,0 @@
# Mos Connector Lease Operations — M1
## Operational status
M1 installs the durable schema and gateway policy/adapter boundary. It does **not** activate a connector, expose a lease administration endpoint, or cut over a channel. The default gateway connector-lease policy is deny-all until a later work package supplies an authorized server-side policy and concrete adapter.
## Events to monitor
Use correlation IDs to follow `connector_lease_audit_log` events:
| Event | Meaning |
| ---------- | --------------------------------------------------------------------- |
| `acquire` | First holder inserted for an unused binding |
| `renew` | Current holder heartbeat extended the TTL |
| `takeover` | Authorized CAS replaced the holder and incremented epoch |
| `release` | Current holder explicitly relinquished authority |
| `expiry` | An expired current lease was observed |
| `reject` | Policy, CAS, expiry, scope, or fencing validation denied an operation |
Audit data is metadata-only. Raw grant objects, connector payloads, scopes, tokens, approval references, and credentials must never be added to audit output.
## Incident checks
For suspected duplicate/stale connector effects:
1. Correlate the attempted operation with its `reject`, `takeover`, or `expiry` event.
2. Compare the current row's connector ID, lease UUID, epoch, expiry, and release time with the adapter's normalized execution context.
3. Treat an old epoch, old lease UUID, expired lease, or released lease as non-authoritative. Do not retry it as the old holder.
4. Recovery uses the authorized takeover path with the observed expected epoch. Ordinary acquire is intentionally rejected for expired/released rows.
5. If an external effect may already have happened, preserve evidence and do not assume lease fencing provides exactly-once replay safety.
## Migration and rollback safety
Migration `0016_salty_morlocks.sql` is additive: it creates two new tables and indexes without modifying existing authorization/session tables. Before rollout, normal database backup and migration verification still apply. Rolling application code back leaves unused additive tables in place; dropping tables is not part of automated rollback because it would destroy lease/audit evidence.
## Security constraints
- Tenant comes from authenticated gateway context, never a connector request field.
- Logical agent, binding, connector, and scope identifiers use normalized constrained forms.
- Takeover requires explicit gateway policy authorization and an expected epoch.
- Default defense-in-depth TTL caps are 5 minutes for leases and 30 seconds for grants; policy may enforce stricter limits.
- Validation and rejection audit complete before adapter side effects.
- Existing authz and exact-action approval controls remain additional required gates; a valid connector lease does not bypass them.
docs/guides/upgrade-safety-and-recovery.md
-147
View File
@@ -1,147 +0,0 @@
# Upgrade Safety & Recovery
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
framework upgrades, and how to recover if a projection is ever lost.
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
this hardening addresses: an upgrade that silently overwrites or deletes a file
the operator owns — credentials, personas, a roster, or a generated agent env —
with no snapshot to fall back to.
Protection is layered. Each layer is independent; a later layer catches what an
earlier one misses.
## Layer 1 — Manifest-owned sync (prevention)
The single source of truth for ownership is
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
Both the bash installer and the TypeScript sync path resolve every path against
this one file (parity is enforced by test), so they can never drift.
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
`[framework]` glob matches and no `[operator]` carve-out overrides it.
- **Unknown paths default to operator** (fail-safe): a file the manifest never
anticipated is treated as operator-owned and is never pruned.
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
only ever iterates framework globs — operator and unknown paths are
structurally unreachable by the prune.
Result: a correct upgrade cannot touch operator config at all.
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
Before **any** mutation, the installer snapshots the operator-owned surface that
exists into:
```
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
```
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
outside `~/.config/mosaic` and outside any repo.
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
protects.
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
After the sync, a **verify net** compares each snapshot file against its target
and restores (with a loud warning) any operator file the upgrade diverged or
removed — a divergence means a manifest bug slipped through Layer 1.
Inspect and restore snapshots with the CLI:
```bash
mosaic restore --list # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
```
`mosaic restore` reports **counts and relative paths only** — it never emits file
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
closed rather than following out of the managed tree).
## Layer 3 — Regeneration from roster SSOT (recovery)
Some operator files are **derived** and do not need a byte-for-byte snapshot to
recover — they can be rebuilt from their source of truth. The fleet's per-agent
generated env projections are the prime case:
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
projection of `~/.config/mosaic/fleet/roster.yaml`.
- The launcher (`start-agent-session.sh`, invoked by
`mosaic-agent@<name>.service`) sources that generated projection to establish
each agent's identity, runtime, model, and working directory. If it is missing
or wrong, the agent cannot launch with its intended identity.
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
```bash
mosaic fleet regen # dry-run (default): show what would be rebuilt
mosaic fleet regen --json # same, machine-readable
mosaic fleet regen --write # rebuild the projections on disk
```
- **Dry-run by default.** Nothing is written until you pass `--write`.
- **Deterministic and idempotent** — the projection is a pure function of the
roster, so repeated `--write` runs produce byte-identical files.
- **Projection-only. It never restarts an agent.** Recovery order forbids
restart-before-verify; `regen` has no path to systemd lifecycle at all.
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
deletes the operator-owned `.env` / `.env.local` surface.
- It **validates the roster the same way `reconcile` does** (persona resolution
and protected-class tool-policy match), so a hand-edited or corrupt roster is
rejected rather than projected, and a `--write` takes the shared reconcile
lock so it cannot race a concurrent reconcile.
- Output is **paths and counts only** — the rendered `KEY=value` body is never
echoed.
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
recovered projection matches what a normal reconcile would have written.
## Recovery runbook — wiped `fleet/agents/*.env.generated`
If an upgrade (or a manual mistake) has left an agent without its generated
projection, **do not restart the unit first** — a launch against a missing
projection fails closed, and any stale state must be corrected before restart,
not after.
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp>
```
2. **Otherwise regenerate the derived projections from the roster SSOT:**
```bash
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
```
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
environment and `start-agent-session.sh` sources `.env.generated` itself, so
verify the generated file directly and confirm the launcher path:
```bash
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
# Confirm the unit launches the session script that reads it.
systemctl --user cat mosaic-agent@<name> | grep ExecStart
```
4. **Only then restart, one unit at a time:**
```bash
systemctl --user restart mosaic-agent@<name>
```
## See also
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)
docs/guides/user-guide.md
-621
View File
@@ -1,621 +0,0 @@
# Mosaic Stack — User Guide
## Table of Contents
1. [Getting Started](#getting-started)
2. [Chat Interface](#chat-interface)
3. [Projects](#projects)
4. [Tasks](#tasks)
5. [Settings](#settings)
6. [CLI Usage](#cli-usage)
7. [Sub-package Commands](#sub-package-commands)
8. [Telemetry](#telemetry)
9. [Local Fleet Canary](./fleet-local-canary.md)
---
## Getting Started
### Prerequisites
Mosaic Stack requires a running gateway. Your administrator provides the URL
(default: `http://localhost:14242`) and creates your account.
### Logging In (Web)
1. Navigate to the Mosaic web app (default: `http://localhost:3000`).
2. You are redirected to `/login` automatically.
3. Enter your email and password, then click **Sign in**.
4. On success you land on the **Chat** page.
### Registering an Account
If self-registration is enabled:
1. Go to `/register`.
2. Enter your name, email, and password.
3. Submit. You are signed in and redirected to Chat.
---
## Chat Interface
### Sending a Message
1. Type your message in the input bar at the bottom of the Chat page.
2. Press **Enter** to send.
3. The assistant response streams in real time. A spinner indicates the agent is
processing.
### Streaming Responses
Responses appear token by token as the model generates them. You can read the
response while it is still being produced. The streaming indicator clears when
the response is complete.
### Conversation Management
- **New conversation**: Navigate to `/chat` or click **New Chat** in the sidebar.
A new conversation ID is created automatically on your first message.
- **Resume a conversation**: Conversations are stored server-side. Refresh the
page or navigate away and back to continue where you left off. The current
conversation ID is shown in the URL.
- **Conversation list**: The sidebar shows recent conversations. Click any entry
to switch.
### Model and Provider
The current model and provider are displayed in the chat header. To change them,
use the Settings page (see [Provider Settings](#providers)) or the CLI
`/model` and `/provider` commands.
---
## Projects
Projects group related missions and tasks. Navigate to **Projects** in the
sidebar.
### Creating a Project
1. Go to `/projects`.
2. Click **New Project**.
3. Enter a name and optional description.
4. Select a status: `active`, `paused`, `completed`, or `archived`.
5. Save. The project appears in the list.
### Viewing a Project
Click a project card to open its detail view at `/projects/<id>`. From here you
can see the project's missions, tasks, and metadata.
### Managing Tasks within a Project
Tasks are linked to projects and optionally to missions. See [Tasks](#tasks) for
full details. On the project detail page, the task list is filtered to the
selected project.
---
## Tasks
Navigate to **Tasks** in the sidebar to see all tasks across all projects.
### Task Statuses
| Status | Meaning |
| ------------- | ------------------------ |
| `not-started` | Not yet started |
| `in-progress` | Actively being worked on |
| `blocked` | Waiting on something |
| `done` | Completed |
| `cancelled` | No longer needed |
### Creating a Task
1. Go to `/tasks`.
2. Click **New Task**.
3. Enter a title, optional description, and link to a project or mission.
4. Set the status and priority.
5. Save.
### Updating a Task
Click a task to open its detail panel. Edit the fields inline and save.
---
## Settings
Navigate to **Settings** in the sidebar (or `/settings`) to manage your profile,
appearance, and providers.
### Profile Tab
- **Name**: Display name shown in the UI.
- **Email**: Read-only; contact your administrator to change email.
- Changes save automatically when you click **Save Profile**.
### Appearance Tab
- **Theme**: Choose `light`, `dark`, or `system`.
- The theme preference is saved to your account and applies on all devices.
### Notifications Tab
Configure notification preferences (future feature; placeholder in the current
release).
### Providers Tab
View all configured LLM providers and their models.
- **Test Connection**: Click **Test** next to a provider to check reachability.
The result shows latency and discovered models.
- Provider configuration is managed by your administrator via environment
variables. See the [Admin Guide](./admin-guide.md) for setup.
---
## CLI Usage
The `mosaic` CLI provides a terminal interface to the same gateway API.
### Installation
Install via the Mosaic installer:
```bash
curl -fsSL https://mosaicstack.dev/install.sh | bash
```
Or use the direct URL:
```bash
bash <(curl -fsSL https://git.mosaicstack.dev/mosaicstack/stack/raw/branch/main/tools/install.sh)
```
The installer places the `mosaic` binary at `~/.npm-global/bin/mosaic`.
Install lanes:
| Lane | Command | Source |
| ------------------------ | ------------------------------------- | -------------------------------------------------------------------------------------------- |
| Stable | `bash tools/install.sh` | npm `@mosaicstack/mosaic@latest` + `main` |
| Prerelease integration | `bash tools/install.sh --next` | Fast npm `@mosaicstack/mosaic@next` + `@mosaicstack/gateway@next`; source fallback at `next` |
| Contributor/source build | `bash tools/install.sh --dev --ref X` | Build-from-source at the requested ref |
`--next` is fast-by-default from the Gitea npm `next` dist-tag and falls back to a source build at the permanent `next` branch if the dist-tag is missing or unreachable. Explicit `--ref` or `MOSAIC_REF` still wins and uses the source path.
Flags for non-interactive use:
```bash
--yes # Accept all defaults
--no-auto-launch # Skip auto-launch of wizard after install
```
Unrecognized flags or positional arguments fail before installation starts and print the supported-option usage.
Or if installed globally:
```bash
mosaic --help
```
### First-Run Wizard
After install the wizard launches automatically. You can re-run it at any time:
```bash
mosaic wizard
```
The wizard guides you through:
1. Gateway discovery or installation (`mosaic gateway install`)
2. Authentication (`mosaic gateway login`)
3. Post-install health check (`mosaic gateway verify`)
### Gateway Login and Token Recovery
```bash
# Authenticate with a gateway and save a session token
mosaic gateway login
# Verify the gateway is reachable and responding
mosaic gateway verify
# Rotate your current API token
mosaic gateway config rotate-token
# Recover a token via BetterAuth cookie (for accounts with no token)
mosaic gateway config recover-token
```
If you have an existing gateway account but lost your token (common after a
reinstall), use `mosaic gateway config recover-token` to retrieve a new one
without recreating your account.
### Configuration
```bash
# Print full config as JSON
mosaic config show
# Read a specific key
mosaic config get gateway.url
# Write a key
mosaic config set gateway.url http://localhost:14242
# Open config in $EDITOR
mosaic config edit
# Print config file path
mosaic config path
```
### Signing In (Legacy)
```bash
mosaic login --gateway http://localhost:14242 --email [email protected]
```
You are prompted for a password if `--password` is not supplied. The session
cookie is saved locally and reused on subsequent commands.
### Launching the TUI
```bash
mosaic tui
```
Options:
| Flag | Default | Description |
| ----------------------- | ------------------------ | ---------------------------------- |
| `--gateway <url>` | `http://localhost:14242` | Gateway URL |
| `--conversation <id>` | — | Resume a specific conversation |
| `--model <modelId>` | server default | Model to use (e.g. `llama3.2`) |
| `--provider <provider>` | server default | Provider (e.g. `ollama`, `openai`) |
If no valid session exists you are prompted to sign in before the TUI launches.
### TUI Slash Commands
Inside the TUI, type a `/` command and press Enter:
| Command | Description |
| ---------------------- | ------------------------------ |
| `/model <modelId>` | Switch to a different model |
| `/provider <provider>` | Switch to a different provider |
| `/models` | List available models |
| `/exit` or `/quit` | Exit the TUI |
### Session Management
```bash
# List saved sessions
mosaic sessions list
# Resume a session
mosaic sessions resume <sessionId>
# Destroy a session
mosaic sessions destroy <sessionId>
```
### Other Commands
```bash
# Run the Mosaic installation wizard
mosaic wizard
# PRD wizard (generate product requirement documents)
mosaic prdy
# Quality rails scaffolder
mosaic quality-rails
```
---
### Claude Code Skill Registration
Mosaic stores canonical skills under `~/.config/mosaic/skills/`. Claude Code scans
`~/.claude/skills/`, so Mosaic maintains one symlink per skill between those
directories.
```bash
mosaic skill list
mosaic skill register <name>
mosaic skill unregister <name>
```
- `register` is idempotent and repairs a dangling Mosaic-owned link. Names use
the safe grammar `[A-Za-z0-9][A-Za-z0-9._-]*`; files, directories, foreign
symlinks, path traversal, absolute paths, and names beginning with `-` are
refused.
- `unregister` is idempotent when no entry exists. It removes only symlinks that
point inside `~/.config/mosaic/skills/`; foreign entries are never removed.
- `list` reports `registered`, `unregistered`, `dangling`, `foreign`,
`foreign-dangling`, or `misdirected` for each canonical or Claude entry.
Install, wizard finalization, and `mosaic update` framework re-seeding reconcile
every canonical skill automatically. A skill directory added after initial
setup therefore receives its Claude bridge without a per-skill code change or
manual `ln -s`. If Claude Code is already running, use `/reload-skills` or start
a new session after registration so its in-process skill registry rescans.
This command group is Claude-only in M1. Pi can consume Mosaic's canonical skill
root through its Mosaic launcher configuration and does not need this Claude
bridge. Codex has a separate link path managed by the legacy full skill-sync
script; equivalent lifecycle management remains follow-up scope and is not
changed here.
## Sub-package Commands
Each Mosaic sub-package exposes its full API surface through the `mosaic` CLI.
All sub-package commands accept `--help` for usage details.
### `mosaic auth` — User & Authentication Management
Manage gateway users, SSO providers, and active sessions.
```bash
# List all users
mosaic auth users list
# Create a new user
mosaic auth users create --email [email protected] --name "Alice"
# Delete a user
mosaic auth users delete <userId>
# List configured SSO providers
mosaic auth sso
# List active sessions
mosaic auth sessions list
# Revoke a session
mosaic auth sessions revoke <sessionId>
```
### `mosaic brain` — Projects, Missions, Tasks, Conversations
Browse and manage the brain data layer (PostgreSQL-backed project/mission/task
store).
```bash
# List all projects
mosaic brain projects
# List missions for a project
mosaic brain missions --project <projectId>
# List tasks
mosaic brain tasks --status in-progress
# Browse conversations
mosaic brain conversations
mosaic brain conversations --project <projectId>
```
### `mosaic config` — CLI Configuration
Read and write the `mosaic` CLI configuration file.
```bash
# Show full config
mosaic config show
# Get a value
mosaic config get gateway.url
# Set a value
mosaic config set theme dark
# Open in editor
mosaic config edit
# Print file path
mosaic config path
```
### `mosaic forge` — AI Pipeline Management
Interact with the Forge multi-stage AI delivery pipeline (intake → board review
→ planning → coding → review → deploy).
```bash
# Start a new forge run for a brief
mosaic forge run --brief "Add dark mode toggle to settings"
# Check status of a running pipeline
mosaic forge status
mosaic forge status --run <runId>
# Resume a paused or interrupted run
mosaic forge resume --run <runId>
# List available personas (board review evaluators)
mosaic forge personas
```
### `mosaic gateway` — Gateway Lifecycle
Install, authenticate with, and verify the Mosaic gateway service.
```bash
# Install gateway (guided)
mosaic gateway install
# Verify gateway health post-install
mosaic gateway verify
# Log in and save token
mosaic gateway login
# Rotate API token
mosaic gateway config rotate-token
# Recover token via BetterAuth cookie (lost-token recovery)
mosaic gateway config recover-token
```
### `mosaic log` — Structured Log Access
Query and stream structured logs from the gateway.
```bash
# Stream live logs
mosaic log tail
mosaic log tail --level warn
# Search logs
mosaic log search "database connection"
mosaic log search --since 1h "error"
# Export logs to file
mosaic log export --output logs.json
mosaic log export --since 24h --level error --output errors.json
# Get/set log level
mosaic log level
mosaic log level debug
```
### `mosaic macp` — MACP Protocol
Interact with the MACP credential resolution, gate runner, and event bus.
```bash
# List MACP tasks
mosaic macp tasks
mosaic macp tasks --status pending
# Submit a new MACP task
mosaic macp submit --type credential-resolve --payload '{"key":"OPENAI_API_KEY"}'
# Run a gate check
mosaic macp gate --gate quality-check
# Stream MACP events
mosaic macp events
mosaic macp events --filter credential
```
### `mosaic memory` — Agent Memory
Query and inspect the agent memory layer.
```bash
# Semantic search over memory
mosaic memory search "previous decisions about auth"
# Show memory statistics
mosaic memory stats
# Generate memory insights report
mosaic memory insights
# View stored preferences
mosaic memory preferences
mosaic memory preferences --set editor=neovim
```
### `mosaic queue` — Task Queue (Valkey)
Manage the Valkey-backed task queue.
```bash
# List all queues
mosaic queue list
# Show queue statistics
mosaic queue stats
mosaic queue stats --queue agent-tasks
# Pause a queue
mosaic queue pause agent-tasks
# Resume a paused queue
mosaic queue resume agent-tasks
# List jobs in a queue
mosaic queue jobs agent-tasks
mosaic queue jobs agent-tasks --status failed
# Drain (empty) a queue
mosaic queue drain agent-tasks
```
### `mosaic storage` — Object Storage
Manage object storage tiers and data migrations.
```bash
# Show storage status and usage
mosaic storage status
# List available storage tiers
mosaic storage tier
# Export data from storage
mosaic storage export --bucket agent-artifacts --output ./artifacts.tar.gz
# Import data into storage
mosaic storage import --bucket agent-artifacts --input ./artifacts.tar.gz
# Schema migration is unavailable in this release. The current storage wrapper shells
# directly to `pnpm --filter @mosaicstack/db db:migrate`; it is legacy N-1,
# uncertified, and MUST NOT be invoked pending KBN-101-02/-03/-06/-08 activation.
# Future schema migration is non-operative: external bootstrap → TLS/roles → runner
# --run → runner --verify → readiness.
# Tier copy uses only the separately held secure migrate-tier route. Never use a legacy
# --from/--to storage-migrate command or pass a credential on argv.
```
---
## Telemetry
Mosaic includes an OpenTelemetry-based telemetry system. Local telemetry
(traces, metrics sent to Jaeger) is always available. Remote telemetry upload
requires explicit opt-in.
### Local Telemetry
```bash
# Show local OTEL collector / Jaeger status
mosaic telemetry local status
# Tail live OTEL spans
mosaic telemetry local tail
# Open Jaeger UI URL
mosaic telemetry local jaeger
```
### Remote Telemetry
Remote upload is a no-op (dry-run) until you opt in. Your consent state is
persisted in the config file.
```bash
# Show current consent state
mosaic telemetry status
# Opt in to remote telemetry
mosaic telemetry opt-in
# Opt out (data stays local)
mosaic telemetry opt-out
# Test telemetry pipeline without uploading
mosaic telemetry test
# Upload telemetry (requires opt-in; dry-run otherwise)
mosaic telemetry upload
```