docs: establish canonical documentation architecture (#1210)
ci/woodpecker/push/publish Pipeline failed
ci/woodpecker/push/publish Pipeline failed
This commit was merged in pull request #1210.
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
# Administrator Operations
|
||||
|
||||
> **Status:** Partially migrated. Procedures explicitly identify whether they are current or held.
|
||||
|
||||
## Current procedures
|
||||
|
||||
- [Upgrade safety and recovery](upgrade-safety-and-recovery.md) — installed-CLI and local-PGlite upgrade, rollback, and framework-configuration recovery.
|
||||
|
||||
## Held procedures
|
||||
|
||||
- [Mos connector lease operations](mos-connector-lease-operations.md) — non-operative M1 outline while the gateway policy remains deny-all and no connector is activated.
|
||||
|
||||
A held page is architecture and readiness context, not command authority. PostgreSQL, federated, bare-metal, Compose, Gateway/Web activation, and migration-runner procedures remain outside the current local route unless a later page explicitly removes the hold with verified evidence.
|
||||
|
||||
## Related
|
||||
|
||||
- [[ADMIN-GUIDE/README|Administrator guide]]
|
||||
- [[ADMIN-GUIDE/security/README|Administrator security]]
|
||||
- [[DEVELOPER-GUIDE/architecture/README|Architecture index]]
|
||||
@@ -0,0 +1,79 @@
|
||||
# Mos Connector Lease Operations — M1
|
||||
|
||||
> **Status:** Held / non-operative.
|
||||
> **Operational authority:** None. This page is a future runbook outline, not a current command, endpoint, migration, or activation procedure.
|
||||
> **Hold condition:** The gateway's `DenyConnectorLeasePolicy` remains the default policy and rejects every lease and grant operation. No connector is activated by this page.
|
||||
|
||||
## Current state
|
||||
|
||||
M1 provides an implemented lease, fencing, audit, and gateway policy boundary. It does not currently provide an operator-facing lease endpoint, activate a connector, cut over a channel, or connect an existing runtime provider to `ConnectorExecutionContext`. The default gateway module is deny-all, so operators must not treat the schema or internal service as an available lease-control surface.
|
||||
|
||||
There is no current operator command sequence to acquire, renew, take over, release, or grant connector authority. Do not attempt to operate the held procedure through direct database writes or by bypassing the gateway policy. Any future activation requires a separately approved server-side policy, concrete adapter, downstream fencing design, and an updated operational runbook.
|
||||
|
||||
## Held future procedure — not current command authority
|
||||
|
||||
The following records the intended shape of a later runbook. It is deliberately non-operative while deny-all remains:
|
||||
|
||||
### Events and evidence to monitor later
|
||||
|
||||
If an authorized policy and adapter are activated in a future work package, correlation IDs may be used to inspect `connector_lease_audit_log` events:
|
||||
|
||||
| Event | Meaning |
|
||||
| ---------- | ---------------------------------------------------------------------------------- |
|
||||
| `acquire` | First holder inserted for an unused binding |
|
||||
| `renew` | Current holder heartbeat extended the TTL |
|
||||
| `takeover` | Authorized compare-and-swap replaced the holder and incremented epoch |
|
||||
| `release` | Current holder explicitly relinquished authority |
|
||||
| `expiry` | An expired current lease was observed |
|
||||
| `reject` | Policy, compare-and-swap, expiry, scope, or fencing validation denied an operation |
|
||||
|
||||
Audit records are metadata-only. Raw grant objects, connector payloads, scopes, tokens, approval references, and credentials must never be added to audit output. The current schema does not independently enforce append-only storage; database-level protection remains a future hardening requirement.
|
||||
|
||||
### Held incident-review outline
|
||||
|
||||
For a future suspected duplicate or stale connector effect, an approved operator procedure would:
|
||||
|
||||
1. correlate the attempted operation with its `reject`, `takeover`, or `expiry` event;
|
||||
2. compare the durable row's connector ID, lease UUID, epoch, expiry, and release time with the adapter's normalized execution context;
|
||||
3. treat an old epoch, old lease UUID, expired lease, or released lease as non-authoritative rather than retrying it as the old holder;
|
||||
4. use only the authorized takeover path with the observed expected epoch, never ordinary acquire, for an expired or released row; and
|
||||
5. preserve evidence without assuming lease fencing provides exactly-once replay safety if an external effect may already have occurred.
|
||||
|
||||
These are held review requirements, not instructions to bypass the current deny-all policy.
|
||||
|
||||
### Held migration and rollback notes
|
||||
|
||||
The checked-in `0016_salty_morlocks.sql` artifact is additive: it creates the lease and audit tables and indexes without changing existing authorization/session tables. A future database rollout would still require the repository's approved migration, backup, verification, and rollback controls. Application rollback would leave additive lease/audit tables in place; dropping them would destroy evidence and is not an automatic rollback step.
|
||||
|
||||
This page does not authorize running migrations, connecting to PostgreSQL, initializing PGlite, or starting a connector. Those activities remain outside this held procedure and subject to repository/runtime gates.
|
||||
|
||||
### Held security prerequisites
|
||||
|
||||
Before this page could become operative, the activation work would need to demonstrate at least:
|
||||
|
||||
- tenant authority derived from authenticated gateway context, not connector request fields;
|
||||
- authorized policy decisions over normalized identity and scopes, with policy TTL handling based on the requested input and coordinator hard caps still applied;
|
||||
- explicit takeover authorization and expected-epoch compare-and-swap;
|
||||
- application-boundary normalization plus any separately approved database constraints or protections;
|
||||
- validation and rejection audit before adapter side effects;
|
||||
- a concrete adapter that consumes and propagates the normalized epoch/context; and
|
||||
- existing authorization and exact-action approval controls remaining in force.
|
||||
|
||||
M1 currently satisfies the boundary contract and default-deny posture, not these activation prerequisites.
|
||||
|
||||
## Explicit non-goals while held
|
||||
|
||||
This page does not authorize or claim:
|
||||
|
||||
1. lease administration from the dashboard, CLI, HTTP, SQL console, or a connector;
|
||||
2. production connector or channel activation;
|
||||
3. exactly-once delivery, side-effect journaling, checkpoint/handoff recovery, or replay safety;
|
||||
4. automatic failover, rollback, or stale-effect recovery across Claude, Pi, Codex, Matrix, tmux, or provider sessions; or
|
||||
5. database-enforced audit immutability or database-enforced application normalization.
|
||||
|
||||
The page may be promoted to an operative runbook only after deny-all is intentionally replaced, concrete adapters are reviewed, activation evidence exists, and this hold is explicitly removed by the owning work package.
|
||||
|
||||
## Related contract
|
||||
|
||||
- [M1 logical identity and fencing decision](../../DEVELOPER-GUIDE/architecture/decisions/mos-runtime-portability-m1.md)
|
||||
- [MOS-PORT requirements](../../PRD.md#mos-runtime-portability-workstream-mos-port)
|
||||
@@ -0,0 +1,248 @@
|
||||
# Upgrade safety and recovery
|
||||
|
||||
> **Supported route:** an already installed `mosaic` CLI using the local PGlite
|
||||
> configuration. This is a filesystem and CLI runbook; it does not activate a
|
||||
> service or connect to a database.
|
||||
|
||||
This page covers the supported upgrade, rollback, health, and recovery checks for
|
||||
Mosaic framework configuration under `MOSAIC_HOME`. It deliberately does not
|
||||
turn the repository's deployment or PostgreSQL material into an operative
|
||||
procedure.
|
||||
|
||||
## Support boundary
|
||||
|
||||
Use this runbook only when all of the following are true:
|
||||
|
||||
- `mosaic` resolves to the installed Mosaic CLI (`command -v mosaic`).
|
||||
- The active application configuration is the local tier: `tier: "local"`,
|
||||
`storage.type: "pglite"`, and `queue.type: "local"`.
|
||||
- `DATABASE_URL` is unset. An inherited PostgreSQL DSN is outside this route and
|
||||
must be removed before continuing.
|
||||
- `MOSAIC_STORAGE_TIER` is unset or `local`; standalone and federated overrides
|
||||
are outside this route.
|
||||
- No Gateway, Web, Compose, or other service activation is required.
|
||||
|
||||
The following routes are **held/non-operative** in this guide:
|
||||
|
||||
| Route or operation | Status |
|
||||
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| PostgreSQL-backed `standalone` storage | Held; do not activate or probe it here. |
|
||||
| Federated storage and peers | Held; do not activate or probe it here. |
|
||||
| Bare-metal deployment | Held; no service installation or lifecycle action is authorized. |
|
||||
| Gateway/Web activation or HTTP health checks | Held; `/health`, `/health/ready`, and `mosaic gateway ...` are not evidence for this route. |
|
||||
| Compose startup | Held; do not start a Compose profile or the full stack. |
|
||||
| Migration runners and tier migration | Held; do not run `mosaic-db-migrator`, `pnpm --filter @mosaicstack/db db:migrate`, `mosaic storage migrate`, or `mosaic storage migrate-tier`. |
|
||||
|
||||
A command being present in the CLI does not make a held route operative.
|
||||
|
||||
## Storage and path terminology
|
||||
|
||||
Keep these locations separate:
|
||||
|
||||
- **`MOSAIC_HOME`** — the framework/operator configuration directory. It
|
||||
defaults to `~/.config/mosaic` and can be overridden with `MOSAIC_HOME`.
|
||||
- **PGlite data** — the local, in-process database. The checked-in local config
|
||||
uses `.mosaic/storage-pglite` as its storage `dataDir`; `.mosaic/queue` is the
|
||||
local queue directory. These are project data, not framework configuration.
|
||||
- **Durable upgrade snapshots** — operator-file snapshots stored under
|
||||
`${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/`. They are outside
|
||||
`MOSAIC_HOME` and do not contain a PostgreSQL dump.
|
||||
|
||||
The configuration tier is named `local`; the storage CLI reports the backend as
|
||||
`pglite`. PGlite is not PostgreSQL and does not require a PostgreSQL server.
|
||||
Local PGlite schema setup is adapter-owned; this page does not authorize a
|
||||
separate migration runner.
|
||||
|
||||
## Pre-upgrade health gate
|
||||
|
||||
Run these checks from a shell that has no inherited database DSN:
|
||||
|
||||
```bash
|
||||
command -v mosaic
|
||||
mosaic --version
|
||||
mosaic config path
|
||||
test -d "$(mosaic config path)"
|
||||
|
||||
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
|
||||
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
|
||||
mosaic restore --list
|
||||
```
|
||||
|
||||
For the supported route, the storage checks should report the `pglite` backend
|
||||
and say that no network check is needed. `mosaic storage status` reports
|
||||
`PGLITE_DATA_DIR` when that variable is set; otherwise it reports the CLI's
|
||||
`:memory:` fallback. That output is an inspection of the CLI environment, not a
|
||||
claim that PGlite contents are healthy or durable.
|
||||
|
||||
If `DATABASE_URL` is set, stop. Do not point it at a local PostgreSQL instance
|
||||
to make the check pass. If the active configuration is not local/PGlite, stop;
|
||||
no operative route is documented here.
|
||||
|
||||
The supported health gate is intentionally limited to CLI resolution, framework
|
||||
configuration path, local storage selection, and available upgrade snapshots.
|
||||
It does not prove Gateway/Web readiness, provider connectivity, queue health, or
|
||||
PGlite data integrity.
|
||||
|
||||
## Safe upgrade procedure
|
||||
|
||||
1. **Record the baseline.** Save the output of `mosaic --version`,
|
||||
`mosaic config path`, and `mosaic restore --list`. Do not copy secrets into a
|
||||
ticket or report.
|
||||
2. **Check for updates without installing them:**
|
||||
|
||||
```bash
|
||||
mosaic update --check
|
||||
```
|
||||
|
||||
Exit status `0` means no update was reported; status `2` means an update is
|
||||
available. Other failures are not a successful health result. The check is a
|
||||
package-registry check and does not start Mosaic services or access storage.
|
||||
|
||||
3. **Run the installed-CLI upgrade when approved:**
|
||||
|
||||
```bash
|
||||
mosaic update
|
||||
```
|
||||
|
||||
The command updates the installed `@mosaicstack/*` packages through npm and,
|
||||
when the framework package changed or framework drift is detected, re-seeds
|
||||
framework files in keep mode. Leave the default re-seed enabled. Do not use
|
||||
`--relaunch` in this runbook: that option can restart durable fleet agents
|
||||
and is outside the supported no-activation route. `--no-reseed` is also not a
|
||||
normal upgrade path because it intentionally leaves framework files stale.
|
||||
|
||||
4. **Repeat the health gate.** Confirm the CLI version, the same
|
||||
`MOSAIC_HOME`, the local/PGlite storage selection, and the snapshot listing.
|
||||
A successful framework upgrade must not be used as evidence that a held
|
||||
Gateway, Web, PostgreSQL, or federated route is ready.
|
||||
|
||||
Do not replace this procedure with a direct edit of `~/.config/mosaic`, a
|
||||
repository `tools/install.sh` invocation, a Compose startup, or a migration
|
||||
command. The supported operator entry point for this page is the installed
|
||||
`mosaic update` command.
|
||||
|
||||
## What an upgrade protects
|
||||
|
||||
For an existing keep-mode installation, the framework installer takes two
|
||||
separate snapshots before the file sync:
|
||||
|
||||
1. An ephemeral whole-directory snapshot under `${TMPDIR:-/tmp}/` is used to
|
||||
restore the previous framework directory if the sync is interrupted or
|
||||
fails. If that restore cannot complete, the installer prints the retained
|
||||
temporary snapshot path for manual recovery.
|
||||
2. A durable snapshot captures the existing operator-owned files before the
|
||||
upgrade at:
|
||||
|
||||
```text
|
||||
${XDG_STATE_HOME:-$HOME/.local/state}/mosaic/backups/pre-update-<UTC timestamp>/
|
||||
```
|
||||
|
||||
Snapshot directories are mode `0700`; captured files are mode `0600`.
|
||||
Retention is five snapshots by default and can be changed with
|
||||
`MOSAIC_BACKUP_RETENTION`. A durable snapshot failure is a warning and does
|
||||
not replace the manifest and crash-rollback protections.
|
||||
|
||||
After the keep-mode sync, the installer compares each captured operator file
|
||||
with its target. If a file was changed or removed unexpectedly, it restores the
|
||||
snapshot copy and emits a warning. A symlinked parent is not followed; that
|
||||
case is left for manual recovery from the snapshot path.
|
||||
|
||||
Keep mode is manifest-driven and fail-safe for operator paths, but it does not
|
||||
mean every file under `MOSAIC_HOME` is user-owned. The framework contract files
|
||||
`CONSTITUTION.md`, `AGENTS.md`, and `STANDARDS.md` are refreshed by upgrades;
|
||||
keep local policy in the supported local overlays rather than editing those
|
||||
contract files as rollback data. A durable snapshot is for operator-owned
|
||||
configuration, not for the installed npm package, framework-owned contracts, or
|
||||
PGlite data.
|
||||
|
||||
## Rollback and recovery
|
||||
|
||||
### If the upgrade fails or is interrupted
|
||||
|
||||
The framework sync attempts an automatic rollback from its ephemeral snapshot.
|
||||
Do not delete `MOSAIC_HOME`, the PGlite data directory, or the temporary snapshot
|
||||
named in the error. Preserve the command output, then run the health gate.
|
||||
|
||||
If the installed framework reports that the automatic restore did not complete,
|
||||
use the durable snapshot procedure below. The durable snapshot is the recovery
|
||||
pointer that survives a successful upgrade and the removal of the temporary
|
||||
snapshot.
|
||||
|
||||
### Restore operator configuration from a durable snapshot
|
||||
|
||||
`mosaic restore` is confirmation-gated and reports counts and relative paths,
|
||||
not file contents. First list snapshots, then preview the selected timestamp:
|
||||
|
||||
```bash
|
||||
mosaic restore --list
|
||||
mosaic restore --from <UTC-timestamp> --dry-run
|
||||
```
|
||||
|
||||
If the preview is correct, run the interactive restore:
|
||||
|
||||
```bash
|
||||
mosaic restore --from <UTC-timestamp>
|
||||
```
|
||||
|
||||
Use `--yes` only when the overwrite has been explicitly approved:
|
||||
|
||||
```bash
|
||||
mosaic restore --from <UTC-timestamp> --yes
|
||||
```
|
||||
|
||||
The timestamp is the value printed by `mosaic restore --list`; the command also
|
||||
accepts the full `pre-update-<timestamp>` name. For a non-default configuration
|
||||
home, pass the same target explicitly:
|
||||
|
||||
```bash
|
||||
mosaic restore --mosaic-home "$MOSAIC_HOME" --from <UTC-timestamp>
|
||||
```
|
||||
|
||||
After restoring, repeat the health gate. Restore writes only the operator
|
||||
surface represented by that snapshot. It does not downgrade the installed CLI,
|
||||
restore framework-owned contract files, restore `.mosaic/storage-pglite`, or
|
||||
run any migration.
|
||||
|
||||
### If local PGlite data is missing or corrupt
|
||||
|
||||
Do not use a framework snapshot as a database backup. Do not run a migration
|
||||
runner, start PostgreSQL, start Compose, or activate Gateway/Web to investigate.
|
||||
The current CLI does not provide a wired PGlite export/import operation;
|
||||
`mosaic storage export` and `mosaic storage import` only print direct-copy
|
||||
guidance. Preserve the configured PGlite data directory and route data
|
||||
restoration through the held data-layer procedure rather than improvising a
|
||||
recursive delete or copy.
|
||||
|
||||
### Recovery decision tree
|
||||
|
||||
- **CLI or framework path is wrong:** stop, verify `command -v mosaic`,
|
||||
`mosaic --version`, `MOSAIC_HOME`, and `mosaic config path`; do not activate a
|
||||
held service route.
|
||||
- **Upgrade failed during framework sync:** use the installer's automatic
|
||||
rollback first; if it reports incomplete recovery, list and preview the
|
||||
durable snapshot, then restore it interactively.
|
||||
- **An operator file changed after a reported-successful upgrade:** select the
|
||||
pre-upgrade snapshot with `mosaic restore --list`, preview it with `--dry-run`,
|
||||
and restore only after reviewing the relative-path plan.
|
||||
- **PGlite data is affected:** preserve the data directory and stop at the
|
||||
boundary described above. Framework rollback cannot recover database rows.
|
||||
- **Output requests PostgreSQL, federated, bare-metal, Compose, Gateway/Web, or
|
||||
a migration runner:** stop; that is a held/non-operative route, not a next
|
||||
command for this runbook.
|
||||
|
||||
## Post-recovery verification
|
||||
|
||||
Run the non-mutating checks again:
|
||||
|
||||
```bash
|
||||
mosaic --version
|
||||
mosaic config path
|
||||
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage tier show
|
||||
env -u DATABASE_URL -u MOSAIC_STORAGE_TIER mosaic storage status
|
||||
mosaic restore --list
|
||||
```
|
||||
|
||||
A green result here means the installed CLI resolves, the framework path is
|
||||
present, the CLI selects local PGlite without a network probe, and snapshots can
|
||||
be enumerated. It does not certify database contents or any held deployment
|
||||
route.
|
||||
Reference in New Issue
Block a user