Framework upgrades must not destroy operator-owned config under ~/.config/mosaic #791

New Issue

jason.woltje · 2026-07-16T19:36:51Z

jason.woltje commented

2026-07-16 19:36:51 +00:00

Summary

Framework upgrades (mosaic self-update / install.sh re-run / framework-vN migration) must
never destroy operator-owned configuration under ~/.config/mosaic. A recent framework-v3
upgrade wiped operator-owned directories, silently degrading a running agent fleet. This issue
captures the failure class and tracks a durable fix.

Failure class

The updater does not distinguish framework-owned paths (which it is entitled to overwrite
verbatim on upgrade — e.g. CONSTITUTION.md, AGENTS.md, guides, tools) from operator-owned
paths (which it must treat as sacred). When an upgrade replaces/prunes subtrees that contain
operator files, operator state is lost with no snapshot and no warning.

Operator-owned paths that MUST be protected (audit for completeness — this list is a starting set):

agents/*.conf — per-agent runtime configuration
fleet/agents/*.env — per-agent fleet environment files (systemd EnvironmentFile sources)
memory/ — operator/framework memory notes
policy/*.md — operator policy overlays
*.local.md overlays — SOUL.local.md, USER.local.md, STANDARDS.local.md
any harvester / SOP artifacts and timers the operator installed
(audit) tools/_lib/credentials.json and any secret-bearing operator files

Observed evidence (one operator's incident — narrative only, not part of the requirement)

On a v3 upgrade: ~/.config/mosaic/agents/*.conf (whole dir gone) and
~/.config/mosaic/fleet/agents/*.env (dir gone); same event class previously removed a harvester

SOP (missed harvests). Live blast radius: mosaic-agent@ systemd services boot unit DEFAULTS
(runtime/workdir) on restart because EnvironmentFile=-... is absent-tolerant, so a restart
silently downgrades agent identity. Running agents survived only as pre-wipe processes. This is
evidence of the class; the fix must be operator-agnostic.

Options evaluated

(a) Upgrade-transactional protection — updater takes a pre-update snapshot of operator dirs,
then restores anything it did not intentionally migrate; snapshot retained for rollback. Fixes
blast radius; enables mosaic restore.
(b) Strict ownership separation — updater only ever writes an explicit framework-owned
manifest; operator dirs are declared off-limits and never touched. Fixes the root cause.
(c) Periodic backup timer — systemd user timer snapshots operator dirs on a cadence.
Defense-in-depth; also catches non-upgrade losses.
(d) Regeneration-from-SSOT — deterministic re-link/regenerate of derivable config (e.g.
jarvis-brain infra/fleet install.sh re-links confs). A recovery path, not prevention.

Recommendation (what works best — combination)

(b) Strict ownership separation as the primary structural fix. The updater must operate
from an explicit framework-owned path manifest and never write outside it. Everything under
the operator-owned list above is off-limits. This eliminates the failure class at the root.
(a) Upgrade-transactional pre-update snapshot as the safety net. Before ANY upgrade
mutation, snapshot operator dirs to a timestamped, non-world-readable backup
(~/.local/state/mosaic/backups/pre-update-<ts>/ or equivalent); on completion restore
anything touched-but-not-intentionally-migrated; keep N snapshots for rollback (mosaic restore). Guards against manifest bugs / unforeseen operator paths.
(d) Regeneration-from-SSOT as the documented recovery story (see below), and a mosaic doctor/mosaic fleet regen command that rebuilds derivable config deterministically.
(c) Periodic backup timer as optional defense-in-depth, later phase — belt-and-suspenders
for non-upgrade losses.

Rationale: (b) prevents the class; (a) bounds the blast radius of any miss and gives rollback;
(d) gives deterministic recovery; (c) is cheap insurance. (b)+(a) are the must-haves.

Recovery / regeneration story for a currently-wiped state

For the fleet EnvironmentFile losses specifically: DO NOT perform any service-level fleet
restart while fleet/agents/*.env is absent (a restart boots unit defaults and silently
downgrades identity). Recovery order: regenerate ~/.config/mosaic/fleet/agents/*.env from SSOT
(the fleet install/link script) → verify each unit's EnvironmentFile resolves and the intended
runtime/workdir is present → only then systemctl --user restart mosaic-agent@<name>. Provide a
mosaic fleet regen (or documented install.sh --relink) that performs this deterministically.

Acceptance criteria

Explicit framework-owned manifest; updater proven to never write operator-owned paths
(unit test asserting an upgrade run touches no path outside the manifest).
Pre-update snapshot + selective restore implemented; snapshot dir is not world-readable and
never contains logged secret values; mosaic restore (or equivalent rollback) works.
Secret-safety: if operator config that may contain secrets is snapshotted, backups are
mode-0600/dir-0700, excluded from any repo, and no secret value is emitted to logs/stdout.
mosaic fleet regen (or documented recovery command) regenerates derivable config from SSOT.
Docs updated (upgrade safety + recovery runbook).
Tests-first, ≥85% coverage on new code; independent review; secrev on the backup/restore
+ secret-handling surface.

Gates & scope

TDD tests-first ≥85%; trunk-based squash PR; independent review (author ≠ reviewer); secrev on the
backup/restore + secret-handling surface. Framework-PR firewall: operator-agnostic only — no
SOUL.md/USER.md/operator specifics in framework code; web1 details live in this issue narrative
as evidence only. Capacity-fill; must not preempt M5-001 or #790.

## Summary Framework upgrades (`mosaic` self-update / install.sh re-run / framework-vN migration) must **never** destroy operator-owned configuration under `~/.config/mosaic`. A recent framework-v3 upgrade wiped operator-owned directories, silently degrading a running agent fleet. This issue captures the failure **class** and tracks a durable fix. ## Failure class The updater does not distinguish **framework-owned** paths (which it is entitled to overwrite verbatim on upgrade — e.g. `CONSTITUTION.md`, `AGENTS.md`, guides, tools) from **operator-owned** paths (which it must treat as sacred). When an upgrade replaces/prunes subtrees that contain operator files, operator state is lost with no snapshot and no warning. Operator-owned paths that MUST be protected (audit for completeness — this list is a starting set): - `agents/*.conf` — per-agent runtime configuration - `fleet/agents/*.env` — per-agent fleet environment files (systemd `EnvironmentFile` sources) - `memory/` — operator/framework memory notes - `policy/*.md` — operator policy overlays - `*.local.md` overlays — `SOUL.local.md`, `USER.local.md`, `STANDARDS.local.md` - any harvester / SOP artifacts and timers the operator installed - (audit) `tools/_lib/credentials.json` and any secret-bearing operator files ## Observed evidence (one operator's incident — narrative only, not part of the requirement) On a v3 upgrade: `~/.config/mosaic/agents/*.conf` (whole dir gone) and `~/.config/mosaic/fleet/agents/*.env` (dir gone); same event class previously removed a harvester + SOP (missed harvests). Live blast radius: `mosaic-agent@` systemd services boot unit DEFAULTS (runtime/workdir) on restart because `EnvironmentFile=-...` is absent-tolerant, so a restart silently downgrades agent identity. Running agents survived only as pre-wipe processes. This is evidence of the class; the fix must be operator-agnostic. ## Options evaluated - **(a) Upgrade-transactional protection** — updater takes a pre-update snapshot of operator dirs, then restores anything it did not intentionally migrate; snapshot retained for rollback. Fixes blast radius; enables `mosaic restore`. - **(b) Strict ownership separation** — updater only ever writes an explicit **framework-owned manifest**; operator dirs are declared off-limits and never touched. Fixes the **root cause**. - **(c) Periodic backup timer** — systemd user timer snapshots operator dirs on a cadence. Defense-in-depth; also catches non-upgrade losses. - **(d) Regeneration-from-SSOT** — deterministic re-link/regenerate of derivable config (e.g. jarvis-brain `infra/fleet` install.sh re-links confs). A **recovery** path, not prevention. ## Recommendation (what works best — combination) 1. **(b) Strict ownership separation as the primary structural fix.** The updater must operate from an explicit framework-owned path manifest and **never** write outside it. Everything under the operator-owned list above is off-limits. This eliminates the failure class at the root. 2. **(a) Upgrade-transactional pre-update snapshot as the safety net.** Before ANY upgrade mutation, snapshot operator dirs to a timestamped, non-world-readable backup (`~/.local/state/mosaic/backups/pre-update-<ts>/` or equivalent); on completion restore anything touched-but-not-intentionally-migrated; keep N snapshots for rollback (`mosaic restore`). Guards against manifest bugs / unforeseen operator paths. 3. **(d) Regeneration-from-SSOT as the documented recovery story** (see below), and a `mosaic doctor`/`mosaic fleet regen` command that rebuilds derivable config deterministically. 4. **(c) Periodic backup timer as optional defense-in-depth**, later phase — belt-and-suspenders for non-upgrade losses. Rationale: (b) prevents the class; (a) bounds the blast radius of any miss and gives rollback; (d) gives deterministic recovery; (c) is cheap insurance. (b)+(a) are the must-haves. ## Recovery / regeneration story for a currently-wiped state For the fleet `EnvironmentFile` losses specifically: DO NOT perform any service-level fleet restart while `fleet/agents/*.env` is absent (a restart boots unit defaults and silently downgrades identity). Recovery order: regenerate `~/.config/mosaic/fleet/agents/*.env` from SSOT (the fleet install/link script) → verify each unit's `EnvironmentFile` resolves and the intended runtime/workdir is present → only then `systemctl --user restart mosaic-agent@<name>`. Provide a `mosaic fleet regen` (or documented install.sh `--relink`) that performs this deterministically. ## Acceptance criteria - [ ] Explicit framework-owned manifest; updater proven to never write operator-owned paths (unit test asserting an upgrade run touches no path outside the manifest). - [ ] Pre-update snapshot + selective restore implemented; snapshot dir is not world-readable and never contains logged secret values; `mosaic restore` (or equivalent rollback) works. - [ ] Secret-safety: if operator config that may contain secrets is snapshotted, backups are mode-0600/dir-0700, excluded from any repo, and no secret value is emitted to logs/stdout. - [ ] `mosaic fleet regen` (or documented recovery command) regenerates derivable config from SSOT. - [ ] Docs updated (upgrade safety + recovery runbook). - [ ] Tests-first, ≥85% coverage on new code; independent review; **secrev** on the backup/restore + secret-handling surface. ## Gates & scope TDD tests-first ≥85%; trunk-based squash PR; independent review (author ≠ reviewer); secrev on the backup/restore + secret-handling surface. **Framework-PR firewall:** operator-agnostic only — no `SOUL.md`/`USER.md`/operator specifics in framework code; web1 details live in this issue narrative as evidence only. Capacity-fill; must not preempt M5-001 or #790.

jason.woltje referenced this issue

2026-07-16 20:11:17 +00:00

EPIC: Agent Roster + webUI oversight + health watch #796

jason.woltje referenced this issue

2026-07-16 20:21:59 +00:00

Generic provider-agnostic PR/issue watcher service (pr-watch@<instance>) [epic #796] #801

jason.woltje referenced this issue from a commit

2026-07-16 20:48:31 +00:00

docs(791): Phase-1 design for upgrade config protection

jason.woltje referenced this issue from a commit

2026-07-16 20:48:31 +00:00

feat(mosaic): manifest-owned upgrade guard so updates never wipe operator config (#791)