Compare commits

..
Author SHA1 Message Date
mos-dt-0andMos 76eef39a29 docs(wake): #953 dead-letter retention recorded as load-bearing at the write site (#968)
ci/woodpecker/push/publish Pipeline was successful
ci/woodpecker/push/ci Pipeline was successful
Closes #953.

GATE RECORD: review CLEAR at this exact head (author != reviewer, pre-registered diff-blind checks) + terminal-green CI at this exact head + queue guard clear.
CI CAVEAT (#973): green on the wake suites is currently WEAKER THAN IT LOOKS, IN BOTH DIRECTIONS. grep error/spawn exit codes are read as absence across 257 assertion sites in six idiom forms; 36 inverted (&&-fail) sites — including 19 credential-security canaries — fail toward GREEN under load. These greens were obtained on solo reruns after load-correlated FALSE reds (2115/2116/2118; main itself was red). This merge's safety therefore rests on the CONTENT review, not on the green. Remediation charter fa551c2d0 is authored and in flight.

Co-authored-by: mos-dt-0 <[email protected]>
2026-07-30 23:25:38 +00:00
mos-dt-0andMos 47f8689231 fix(wake): #952 quarantine-audit clean sweep names BOTH unprovable residual classes (#967)
ci/woodpecker/push/publish Pipeline was canceled
ci/woodpecker/push/ci Pipeline was canceled
Closes #952.

GATE RECORD: review CLEAR at this exact head (author != reviewer, pre-registered diff-blind checks) + terminal-green CI at this exact head + queue guard clear.
CI CAVEAT (#973): green on the wake suites is currently WEAKER THAN IT LOOKS, IN BOTH DIRECTIONS. grep error/spawn exit codes are read as absence across 257 assertion sites in six idiom forms; 36 inverted (&&-fail) sites — including 19 credential-security canaries — fail toward GREEN under load. These greens were obtained on solo reruns after load-correlated FALSE reds (2115/2116/2118; main itself was red). This merge's safety therefore rests on the CONTENT review, not on the green. Remediation charter fa551c2d0 is authored and in flight.

Co-authored-by: mos-dt-0 <[email protected]>
2026-07-30 23:25:34 +00:00
mos-dt-0andMos 8d1d6e5e76 feat(wake): #958 A11 preimage.sh — durable provenance for the operator-side preimage definition (#964)
ci/woodpecker/push/publish Pipeline was successful
ci/woodpecker/push/ci Pipeline was successful
Closes #958.

The preimage definition (source-adapter.sh) is the single most consequential file in the wake pipeline — every observed_hash is a sha256 of what it emits — and it was UNVERSIONED: no git history, no backup. When it was edited at 07:27 on 2026-07-30, attribution was recoverable only because an agent transcript happened to still be on disk. A11 gives that file durable provenance (option (2) of #958: recorded content-addressed, not in-band).

DESIGN, per the pre-registration:
- Provenance is OUT-OF-BAND (never on the adapter's stdout) — an in-band record would advance observed_hash for every source at once and manufacture the re-baseline it exists to explain (B3).
- The obligation never depends on the provenance path: a missing/corrupt store cannot halt the detector or swallow a wake (#940 advisory-fields precedent; B4).
- DESC_FMT=d1 is NOT the provenance record — the tag versions the descriptor FORMAT; a behaviour change that keeps descriptor shape re-baselines every hash and leaves the tag unchanged (B6).

CREDENTIAL HARD GATE (rebuilt after the first verdict FAILED it): byte capture is now RECORD-ONLY BY DEFAULT (extras opt in via WAKE_PREIMAGE_CAPTURE), not allow-by-default-refuse-on-shape — because a shape list can only refuse the secrets someone already enumerated, and the tool's own usage text recommended adding detector.env (where HMAC material lives). Deny is evaluated on BOTH raw and resolved path forms with resolved anchors, ordered before allow — closing the realpath-before-deny ordering defect that let a renamed symlink target through.

VERIFICATION (reviewer, mos-dt, independent of the author's claims):
- Seven decoy cases by planted-marker-then-grep-whole-state-dir: known cred path / same-name symlink / RENAMED-target symlink / prefixed secret / prefixless secret / opted-in-symlink-to-DENIED-target all REFUSED; opted-in-symlink-to-ALLOWED-target CAPTURED (positive control that the harness can capture at all, and that C was not closed by breaking every symlink).
- Polarity-completeness self-test RE-RUN with the shape list stubbed always-allow AND both deny lists stubbed — case D still safe: the flip is complete, the shape list is not load-bearing. Each stub proven live first (a stub that silently fails to apply reports the dangerous state as safe).
- B11: rm-then-change fails LOUD (rc=1), refuses to re-baseline, leaves the ledger absent; absent-with-emptied-objects still first-installs cleanly (absent-is-not-corrupt not paid for by breaking first install).
- RED-first reproduced exactly P13-P16 pre-fix; each refusal corroborated three ways (loud stderr, ledger row captured:false WITH a hash so attribution survives refusal, objects/ holding only the adapter).

KNOWN RESIDUAL (filed #969, non-gating): the deny check is both-forms but the suite needles only the resolved form — a deny reduced to resolved-only survives 17/17 green and would leak a renamed-symlink case. No reachable leak at this head (shipped code correct on all seven decoys); it constrains a FUTURE edit. Doctrine: a both-forms fix needs a needle per form; a fixture that satisfies its assertion through a DIFFERENT rule is testing the rule it did not mean to test.

Authored by pepper (sb-it-1-dt); independently reviewed by mos-dt (sb-it-1-dt) under diff-blind pre-registration (7242688b1, predating first read) — NOT CLEAR on the first verdict (B2/B11 failed by decoy), CLEAR at 8aff7d8 after the polarity rebuild. Manifest version 0.7.0.

Co-authored-by: mos-dt-0 <[email protected]>
2026-07-30 21:48:06 +00:00
12 changed files with 1324 additions and 154 deletions
@@ -39,7 +39,6 @@ overwritten on upgrade. (Layer model: `constitution/LAYER-MODEL.md`.)
| TypeScript strict typing | `guides/TYPESCRIPT.md` |
| QA / test strategy | `guides/QA-TESTING.md` |
| Documentation (any code/API/auth/infra change) | `guides/DOCUMENTATION.md` |
| Writing style (docs, comms, any prose) | `guides/WRITING-STYLE.md` |
| Secrets / vault usage | `guides/VAULT-SECRETS.md` |
| Tool/credential reference (service CLIs, wrappers) | `guides/TOOLS-REFERENCE.md` |
| Memory protocol (OpenBrain capture/recall) | `guides/MEMORY.md` |
@@ -27,14 +27,6 @@ Master/slave model:
- Do not perform destructive git/file actions without explicit instruction.
- Browser automation (Playwright, Cypress, Puppeteer) MUST run in headless mode. Never launch a visible browser — it collides with the user's display and active session.
### Output standards (writing + code)
- Technical documentation follows **MOS-STE** (Mosaic Simplified Technical English — an adapted ASD-STE100 profile): short sentences, one instruction per sentence, active voice, one word per meaning, one term per concept. Full rules: `~/.config/mosaic/guides/WRITING-STYLE.md`.
- Apply MOS-STE **hardest to verification artifacts** (acceptance criteria, witness predicates, gate/alarm conditions). There an ambiguous term produces a false green, not just a confused reader.
- Source code follows the **Google Style Guide** for the language.
- User-facing comms follow the user's declared `communicationStyle` in `USER.md` "Communication Preferences" (`direct` | `friendly` | `formal`, default `direct`); `guides/WRITING-STYLE.md` §5 maps each value to output. The documentation standard does not change with user preference.
- **Carve-out:** MOS-STE does NOT apply to content that must carry a specific human voice (letters, personal or marketing prose, voice-matched output). A declared voice profile wins.
### Secrets handling (HARD RULE)
- Vault is the canonical source-of-truth for every secret in every environment. No exceptions.
@@ -1,134 +0,0 @@
# Writing Style Standard — MOS-STE (MANDATORY)
This guide defines how agents write. It sets one style standard per output type.
It is written in the standard it defines, as a worked example.
**Adapted, not compliant.** MOS-STE (Mosaic Simplified Technical English) is an
adapted profile of ASD-STE100. Mosaic does not license or certify against
ASD-STE100. Mosaic uses the load-bearing rules and fits them to agent work. This
is the same stance Mosaic takes toward DO-178B/C: use the rigor, do not claim the
certification.
## Scope — which standard governs which output
| Output type | Standard |
|---|---|
| Technical documentation (READMEs, runbooks, PRDs, procedures, ADRs, guides, acceptance criteria, design docs) | **MOS-STE** (this guide) |
| Source code and code comments | **Google Style Guide** for the language (§4) |
| Inter-agent comms | MOS-STE by default (concise, structured) |
| User-facing comms | **Per-user style choice** — read `USER.md` "Communication Preferences" (§5) |
| End-user prose the user owns (marketing, letters, personal writing, voice-matched content) | The user's declared voice. MOS-STE does NOT apply. |
**The user-voice carve-out is absolute.** Do not apply MOS-STE to content that
must carry a specific human voice (for example a cover letter, a personal
message, or marketing copy). That content needs the user's voice. MOS-STE would
damage it. When a project declares a voice profile, that profile wins.
## 1. Why one standard
Agent documentation drifts across projects. Different agents use different terms,
sentence styles, and structures for the same concept. Readers lose time.
Assumptions hide in ambiguous prose. One standard gives agents a clear target. It
gives reviewers a clear test.
## 2. Where MOS-STE matters most — verification artifacts
Apply MOS-STE hardest to acceptance criteria, witness predicates, gate
definitions, and alarm conditions. In prose, an ambiguous term produces a
confused reader. In a verification artifact, an ambiguous term produces a false
green — a check that passes without testing the claim.
The one-term-one-concept rule (rule 9) is the guard. When one word names two
concepts in one predicate, the check can test the wrong concept and still pass.
**Worked failure.** A rename used a witness predicate with three clauses: ref A
present, ref B absent, tip committed from this host. Every clause tested the git
*ref* (the channel). The claim under test was about a *field inside the payload*.
The word "beacon" named two concepts in one sentence. Deleting ref B was the next
scheduled step. That step flips the last clause green and certifies a state in
which the payload still names the wrong host. The predicate was one planned action
away from a false green on its normal path. The payload field was never tested.
Rule: when N failure modes share one observable, the observable is not a
diagnostic. In a verification artifact, that ambiguity does not confuse a reader —
it certifies the defect.
## 3. MOS-STE rules
### 3.1 Sentence rules
1. Keep sentences short. Use 20 words or fewer for a procedure. Use 25 words or
fewer for a description. (Reasoning and doctrine prose relaxes this limit —
see §3.4. A future lint enforces §3.1, not §3.4.)
2. Write one instruction per sentence. In a procedure, give one command per step.
3. Use the active voice. Write "Run the script." Do not write "The script should
be run."
4. Use the imperative for instructions. Start the sentence with the verb.
5. Use simple verb tenses. Prefer the present tense. Avoid the perfect and
progressive tenses when a simple tense works.
6. Do not use an `-ing` form when it makes the meaning unclear.
7. Write positive statements. State what to do, not only what to avoid.
### 3.2 Word rules
8. Use one word for one meaning. Do not use the same word in two senses.
9. Use one term for one concept. Do not use synonyms for variety. Example: choose
`secret`, `credential`, or `key` for each concept, and keep it.
10. Use articles (`a`, `the`). Do not drop words to save space.
11. Keep an approved-terms glossary per project. Add each domain noun and each
chosen verb. Technical names (for example `Vault`, `cgroup`, `systemd`) are
always allowed.
12. Define an abbreviation at its first use. Then use it consistently.
### 3.3 Structure rules
13. Use a list for parallel items or sequential steps. Do not put them in one long
sentence.
14. Use a table for data with more than two dimensions.
15. Use parallel structure in headings and steps.
16. Repeat the noun. Do not use a pronoun when the reference is unclear.
### 3.4 Adaptation notes (where MOS-STE deviates from ASD-STE100, and why)
- **No licensed dictionary.** ASD-STE100 ships a controlled dictionary under
copyright. MOS-STE uses per-project glossaries instead (rule 11).
- **Domain terms are allowed.** MOS-STE keeps every term the work needs.
- **Reasoning prose gets structure, not amputation.** Apply the sentence and word
rules to design and doctrine writing. Allow the length a subtle argument needs.
Readable-first beats rule-strict when the two conflict.
## 4. Code — Google Style Guide
Write source code to the Google Style Guide for the language (Python, TypeScript,
Shell, Go, and so on). Match the existing file when a local convention already
exists. Keep code comments to the MOS-STE sentence and word rules.
## 5. User-facing comms — a per-user choice
Mosaic is multi-user. Different users want different comms styles. The framework
already carries the selectable setting: `communicationStyle` (`direct` |
`friendly` | `formal`, default `direct`). `mosaic init` writes it, and the
builder renders it into the generated `USER.md` "Communication Preferences"
section. This guide adds the OUTPUT meaning of each value; do not invent new
values.
The builder renders the style as prose bullets, not the token name, so match on
the leading bullet the generated `USER.md` actually contains:
| `USER.md` leading bullet | Style | User-facing output |
|---|---|---|
| "Direct and concise" | `direct` (default) | MOS-STE structure — short, active, defined terms, tables for overview. |
| "Warm and conversational" | `friendly` | Warmer register. Full sentences, explain reasoning, fewer tables. |
| "Professional and structured" | `formal` | Professional and structured. Thorough, with explicit recommendations. |
This setting governs **user-facing comms only**. It does not change the
documentation standard (§3), which is always MOS-STE regardless of the value.
## 6. Enforcement
- **Now:** human review only. **No mechanical prose check exists today.** The
pre-push gate runs typecheck, lint, build, and tests; it inspects no prose.
Reviewers check output against the scope table and the MOS-STE rules by hand.
- **Future:** an MOS-STE lint check (built from the §3.1 sentence rules) and a
Google-style linter in the pre-push gate. A future linter enforces §3.1, not
§3.4 — see the note at rule 1.
@@ -414,6 +414,22 @@ cmd_poll_once() {
_wake_clean_stale_tmp "$STATE_DIR"
mkdir -p "$DET_DIR"
local failed=0
# #958 preimage provenance: check the OPERATOR-SIDE preimage definition (the
# source adapter file, the watch-list, operator-declared extras) BEFORE
# observing any source. A changed preimage re-baselines EVERY source at once;
# running the check first means its first-class cause line is enqueued at a
# LOWER observed_seq than the N per-source deltas it explains, so the digest
# shows the cause, not just the flood. An infrastructure failure of the check
# is LOUD and marks this pass failed (G2a discipline — never read as "no
# change"), but source observation still proceeds: provenance must not be
# able to starve wake delivery.
if ! "$SCRIPT_DIR/preimage.sh" check --enqueue; then
echo "detector.sh: FAIL LOUD — preimage provenance check failed (see preimage.sh above); source observation continues but this pass exits non-zero." >&2
failed=1
fi
# Iterate the DECLARED source-coverage inventory (§4/G3): only sources listed
# in watches[].sources[] are polled. An omitted source is not observed (and so
# cannot make anything pass vacuously); a referenced-but-undefined source is a
@@ -427,7 +443,7 @@ cmd_poll_once() {
exit 2
fi
local failed=0 kind id def class
local kind id def class
while IFS=$'\t' read -r kind id; do
[ -n "$kind" ] || continue
# Resolve the source definition from its top-level collection by id.
@@ -488,6 +488,19 @@ _dlq_alarm_offhost() {
# A dead-letter write failure is itself alarmed (both legs) but never aborts the
# drain — the good entries MUST still deliver (availability is the whole point of
# #920).
#
# RETENTION IS LOAD-BEARING (#953) — read BEFORE adding rotation/pruning/caps:
# this ledger is append-only history AND the SOLE evidence base for
# store.sh quarantine-audit's conviction predicate (#946): a false
# consumed-hashes witness row is provable ONLY while its matching entry
# survives HERE. Any rotation, pruning, or size cap — however locally correct
# — silently converts provable rows into unprovable ones, and the audit's
# clean sweep reads IDENTICALLY before and after the evidence disappears (no
# signal at either end; the audit never guesses, by design). If retention
# limits ever become genuinely necessary, they MUST ship with (a) a loud
# signal at prune time naming what evidence is being given up, and (b) the
# audit's residual-class reporting updated IN THE SAME CHANGE. Until then:
# nothing prunes this file, and that is a recorded decision, not an omission.
_quarantine_entry() {
local line="$1" seq="$2" dlq="$STATE_DIR/dead-letter.jsonl"
if [ ! -f "$dlq" ] || ! grep -qxF "$line" "$dlq" 2>/dev/null; then
@@ -374,8 +374,99 @@
# Changed: store.sh, digest.sh, ack.sh
# (+ test-wake-store-ack.sh T13-T16,
# test-wake-digest-quarantine.sh Q12-Q16).
# 0.7.0 #958 A11 preimage.sh — durable provenance for the OPERATOR-SIDE
# preimage definition (wake-pilot finding: source-adapter.sh was
# unversioned, so the operator-owned bytes every observed_hash is
# computed FROM had thinner provenance than any hash they feed;
# attribution required an agent transcript). NEW tool + two
# pre-step integrations, ADDITIVE: (1) preimage.sh derives the
# preimage set from the RUNTIME env (the resolved
# WAKE_DETECTOR_SOURCE_CMD file, WAKE_WATCH_LIST, optional
# WAKE_PREIMAGE_EXTRA paths) and, per file, records
# {ts,path,sha256,size,mtime,prev} to an append-only ledger +
# captures the bytes CONTENT-ADDRESSED under
# $STATE_DIR/preimage/objects/<sha256> — prior bytes + change
# time are answerable from durable state alone, no transcript
# (acceptance a). A git-repo-in-operator-dir design was REJECTED:
# its who/why claim is false under shared-author fleets, and
# .gitignore is pattern-based where acceptance (b) demands
# fail-closed. (2) CREDENTIAL HARD GATE (acceptance b), FOUR
# LAYERS (#964 re-verdict: B2 cases C+D): (i) POLARITY — extras
# (WAKE_PREIMAGE_EXTRA) are RECORD-ONLY by default (ledger row +
# cause line, NO bytes); byte capture for an extra requires the
# path listed in WAKE_PREIMAGE_CAPTURE (colon-separated opt-in);
# only the core set (adapter, watch-list) is capture-eligible by
# default — a shape list can only refuse the secrets someone
# already enumerated, so safety may not rest on one (#964-D).
# (ii) PATH DENY evaluated on BOTH the raw and realpath-resolved
# candidate forms against BOTH unresolved and resolved
# mosaic-home anchors (credentials.json,
# tools/_lib/credentials.json, credentials/**, any basename
# credentials.json) — a deny list written in unresolved paths
# cannot match a path resolved before it arrived (#964-C), and a
# symlinked opt-in entry cannot smuggle a denied target (deny
# runs FIRST, independent of what the opt-in matched). (iii)
# CONTENT DENY on the opt-in path only, defense-in-depth NOT the
# safety mechanism: digest.sh's six scrub shapes + a
# named-assignment probe (key/token/secret/passw/hmac/credential/
# bearer = unbroken value >=16 chars); probe ERROR fails TOWARD
# refusal (an error exit is not a negative result); false
# positives are acceptable — a false match only withholds byte
# capture, never tracking. (iv) size cap WAKE_PREIMAGE_MAX_BYTES
# (default 1 MiB). Deny ALWAYS wins over the opt-in. A refused
# file still gets its hash/size/mtime row (captured:false +
# refused reason) so change TIME survives even when content must
# not. (3) FIRST-CLASS CAUSE LINE
# (acceptance c): on a change (or deletion — ABSENT is a state,
# not an error), one class=actionable entry is enqueued via the
# store allocator with locators {kind:preimage, path (§2.1 hard
# locator — never quarantined), observed_hash, prev_hash,
# preimage:true, reason:preimage-definition-changed}. Both
# detector.sh cmd_poll_once and reconcile.sh cmd_reconcile run
# the check as a PRE-step, so the cause line lands at a LOWER
# observed_seq than the N per-source deltas/enumerations it
# explains — "preimage definition changed" reads first, not N
# UNACCOUNTED lines. (4) FAIL-LOUD discipline (D2/#955 class):
# an unresolvable adapter, unreadable watch-list, corrupt ledger
# (REFUSES to compare or re-baseline over corrupt history), failed
# object/ledger write, or failed enqueue is a loud non-zero —
# never read as "no change"; in both integrations the pass exits
# non-zero but source observation still proceeds (no starvation).
# An ABSENT/EMPTY ledger with objects/ NON-empty is a loud
# non-zero REFUSAL to re-baseline (#964-B11: objects with no
# ledger cannot be a first install — history was deleted; the
# first-install path must not silently absorb it). First-seen on
# a genuinely clean state dir is a SILENT baseline (detector
# first-poll idiom).
# The installer is UNCHANGED — Gate A auto-enumerates the new
# file from the filesystem; the recording site is the runtime
# tick, which is where the env-derived preimage set exists.
# store.sh/digest.sh/beacon.sh UNCHANGED; watch-list schema
# UNTOUCHED ([1,1]). Changed: preimage.sh (new), detector.sh,
# reconcile.sh (+ test-wake-preimage.sh P1-P17; P13-P17 are the
# #964 re-verdict regression needles: symlinked store, live
# secret value, polarity, ledger deletion, allowlist-symlink).
# 0.7.1 #952 quarantine-audit clean-sweep message named only ONE of the
# two unprovable residual classes ("rows without surviving
# dead-letter evidence"), so an operator reading the OK concluded
# NO evidence exists when evidence can exist and be UNUSABLE: a
# surviving dead-letter row whose locator extracts an empty
# observed_hash (live specimen: mos-dt seq 13,
# bench/malformed-locator-test, "deliberately non-conformant")
# can never satisfy the four-field conviction match, because
# _record_last_consumed only writes rows with a NON-empty hash.
# WORDING-ONLY fix (measurement defect, #951 review finding 1):
# THREE surfaces — the clean-sweep message, the PROVABILITY
# BOUND comment, and (second commit, author-side self-catch)
# the usage() help text — now name both classes; the
# conviction predicate is UNCHANGED.
# Test T17 drives the real writer flow and plants the verbatim
# live specimen (nested .locators.*, NO observed_hash key) —
# never a hand-built flat dead-letter row, which would make the
# audit's correct non-conviction look exactly like the defect
# under hunt (the #951 review's false-defect near-miss).
component=wake
version=0.6.15
version=0.7.1
# Watch-list schema this component consumes, and the INCLUSIVE range of
# schema_version values it supports. A wake-watch-list.json whose schema_version
@@ -446,6 +537,19 @@ schema_max=1
# seed) instead of a bare source error, and LINKS the unit into
# the user systemd search path + post-install-validates it
# resolves (#913). (W7, #913)
# preimage.sh A11 — operator-side PREIMAGE-DEFINITION provenance: derives the
# preimage set from the runtime env (resolved adapter file,
# watch-list, WAKE_PREIMAGE_EXTRA), appends
# {ts,path,sha256,size,mtime,prev} rows to an append-only
# ledger, captures bytes content-addressed under
# preimage/objects/<sha256>, and enqueues a FIRST-CLASS
# "preimage-definition-changed" actionable (path = §2.1 hard
# locator) BEFORE the deltas it explains (detector +
# reconcile pre-step). Credential HARD GATE: capture is
# REFUSED (hash/mtime still recorded) for credential-store
# paths, secret-shaped content, and oversized files —
# refusal, never redaction. Fail-loud on any infra failure;
# a corrupt ledger refuses re-baseline. (#958)
# Companion (framework subtree, not under tools/wake/): systemd/user/mosaic-wake.service
# — the long-lived detector daemon unit (per-class SLO lives in
# the daemon, NOT a systemd interval). (W7)
+571
View File
@@ -0,0 +1,571 @@
#!/usr/bin/env bash
# preimage.sh — A11 of the wake canon (#958): durable PROVENANCE for the
# OPERATOR-SIDE preimage definition.
#
# THE GAP THIS CLOSES (#958): every observed_hash in the lane store is
# sha256(adapter stdout) — the operator's source adapter (plus the watch-list
# and any operator-declared preimage files) IS the preimage definition for
# every hash the pipeline ever records. Those files live OUTSIDE the versioned
# wake component, so a byte change to them was attributable only through agent
# transcripts: provenance thinner than any hash it feeds. A changed preimage
# also re-baselines EVERY source at once, which previously surfaced only as N
# per-source deltas / UNACCOUNTED lines with no first-class cause.
#
# WHAT THIS TOOL DOES:
# - Derives the PREIMAGE SET generically (no operator paths baked in):
# * the resolved file behind WAKE_DETECTOR_SOURCE_CMD (first word),
# * the WAKE_WATCH_LIST file,
# * operator-declared extras via WAKE_PREIMAGE_EXTRA (colon-separated —
# e.g. sink/feed scripts). Extras are RECORD-ONLY by default (see the
# credential hard gate below); never list key-material files.
# - Records each file's (sha256, size, mtime, ts) as an APPEND-ONLY ledger
# row in <state>/preimage/preimage-ledger.jsonl and stores the full bytes
# CONTENT-ADDRESSED at <state>/preimage/objects/<sha256> — so a change is
# attributable (prior bytes + change time) from durable state alone, with
# no transcript required. Acceptance (a) of #958.
# - On a CHANGE (not first-seen), `check --enqueue` enqueues ONE first-class
# class=actionable entry per changed file via the store's single allocator
# (#908), carrying a §2.1 hard locator (`path`), BEFORE the per-source
# deltas it explains are observed — so the digest shows the cause line,
# not just N re-baselined sources. Acceptance (c).
#
# CREDENTIAL HARD GATE — acceptance (b): the mechanism must be UNABLE to
# capture credential material even by operator error. Layered, every layer
# fail-closed toward NOT capturing bytes (the hash/size/mtime row is still
# written — change-time attribution survives, content capture does not;
# a sha256 discloses nothing about the bytes):
# 1. POLARITY (#964 review, case D): byte capture is DENY-BY-DEFAULT for
# operator extras. WAKE_PREIMAGE_EXTRA paths are RECORD-ONLY (rows +
# first-class change entries, no bytes) unless explicitly listed in
# WAKE_PREIMAGE_CAPTURE — a shape list can only refuse the secrets
# someone already enumerated, so arbitrary operator-pointed files must
# not default to capture. The CORE set (the resolved adapter file, the
# watch-list) is capture-eligible: it IS the preimage definition this
# tool exists to snapshot, and the deny layers below still apply to it.
# 2. PATH deny: the mosaic credential store locations
# (<mosaic-home>/credentials.json, <mosaic-home>/tools/_lib/
# credentials.json, anything under <mosaic-home>/credentials/, and any
# file literally named credentials.json) are refused. Evaluated on BOTH
# the operator-supplied form AND the symlink-resolved form, against BOTH
# the unresolved and resolved forms of the mosaic-home anchor — a deny
# list written only in unresolved paths cannot match a path that was
# resolved before it arrived (#964 review, case C).
# 3. CONTENT deny: a candidate whose bytes match the well-known secret-token
# shapes (same conservative set digest.sh redacts: PAT/Slack/AKIA/JWT/PEM)
# OR a named-assignment secret shape (key/token/secret/passw/hmac/
# credential/bearer = long unbroken value — catches prefix-less
# high-entropy keys, e.g. an HMAC key in an env file) is refused —
# refusal, not redaction: a redacted preimage would be a false witness,
# and secret bytes must never land in the object store. Deliberately
# conservative toward REFUSAL: a false match only withholds byte capture,
# never tracking.
# Plus a size cap (WAKE_PREIMAGE_MAX_BYTES, default 1 MiB) so a stray extra
# pointing at a large binary cannot turn provenance into data hoovering.
# Deny ALWAYS wins over the WAKE_PREIMAGE_CAPTURE opt-in.
#
# FAIL-LOUD DISCIPLINE (the D2/#955 class, both layers): an infrastructure
# failure of THIS tool (unresolvable adapter, unreadable/corrupt ledger,
# failed durable write, failed enqueue) exits NON-ZERO and is never read as
# "no change". A corrupt ledger REFUSES to compare (and to re-baseline):
# silently restarting history would erase the very attribution this exists
# to provide. Likewise (#964 review, B11) an ABSENT/EMPTY ledger while
# objects/ still holds prior captures is DELETED HISTORY, not a first
# install — it refuses loudly instead of re-baselining, because a silent
# restart would absorb the next real change as first-seen.
#
# Operator-agnostic (framework firewall): all state via XDG/env; the deny
# list names only framework-defined credential locations, no operator hosts/
# names/secrets. This tool never prints file CONTENT to any stream.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=./_wake-common.sh disable=SC1091
. "$SCRIPT_DIR/_wake-common.sh"
STATE_DIR="$(wake_state_dir)"
STORE_SH="$SCRIPT_DIR/store.sh"
PRE_DIR="$STATE_DIR/preimage"
LEDGER="$PRE_DIR/preimage-ledger.jsonl"
OBJECTS="$PRE_DIR/objects"
_need_jq() {
command -v jq >/dev/null 2>&1 || {
echo "preimage.sh: jq is required" >&2
exit 3
}
}
# _hash_stdin — sha256 of stdin, first field only (portable, same fallback
# chain as detector.sh).
_hash_stdin() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum | awk '{print $1}'
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 | awk '{print $1}'
elif command -v openssl >/dev/null 2>&1; then
openssl dgst -sha256 | awk '{print $NF}'
else
echo "preimage.sh: no sha256 tool (sha256sum/shasum/openssl) available" >&2
exit 3
fi
}
usage() {
cat >&2 <<'EOF'
Usage: preimage.sh <command>
Commands:
check [--enqueue] Compare every preimage-set file against the ledger head.
A changed file gets a new ledger row (+ captured bytes
when capture-eligible and not denied — see Environment);
with --enqueue each change (not first-seen) also enqueues
ONE first-class class=actionable store entry (hard
locator: path). First-seen files baseline SILENTLY (row,
no enqueue — same idiom as the detector's first-seen).
Exit 0 whether or not changes were found; non-zero ONLY
on infrastructure failure (never read as "no change").
record Alias of `check` without enqueue (baseline/refresh).
set Print the derived preimage set, one resolved path per
line (diagnostic).
status Print the ledger head row for every tracked path.
Environment:
WAKE_DETECTOR_SOURCE_CMD Adapter command; its resolved file joins the set.
WAKE_WATCH_LIST Watch-list path; joins the set when set.
WAKE_PREIMAGE_EXTRA Colon-separated additional operator preimage files
(e.g. sink/feed scripts). Extras are RECORD-ONLY
by default: changes get a hash/size/mtime row and
a first-class change entry, but their BYTES are
never captured unless the path is also listed in
WAKE_PREIMAGE_CAPTURE. A listed path that does not
exist is tracked as ABSENT (deletion of a preimage
file is a change, not an error).
WAKE_PREIMAGE_CAPTURE Colon-separated allowlist of extras whose bytes
MAY be captured. The credential deny rules ALWAYS
win over this list. NEVER list files that can hold
key material (env files with keys/HMACs, credential
stores): the deny gate is a backstop, not a
license.
WAKE_PREIMAGE_MAX_BYTES Byte-capture cap (default 1048576). Larger files:
hash/size/mtime recorded, bytes refused.
WAKE_STATE_HOME/WAKE_AGENT store namespace (see store.sh).
EOF
}
# --- preimage-set derivation (generic; no operator paths baked in) ----------
# _realpath_or_self PATH — resolved form when resolvable, the input otherwise.
_realpath_or_self() {
local p="$1"
if command -v realpath >/dev/null 2>&1; then
realpath -- "$p" 2>/dev/null || printf '%s' "$p"
else
printf '%s' "$p"
fi
}
# _resolve_cmd_file CMD — resolve the FIRST WORD of an adapter command string
# to a real file; prints `raw<TAB>resolved`. Non-zero (loud) if it cannot be
# resolved: an adapter the detector will invoke but provenance cannot see is
# an infrastructure failure, not a smaller set. BOTH forms are kept: the deny
# gate must see the pre-resolution form too (#964 review, case C).
_resolve_cmd_file() {
local cmd="$1" word path
# shellcheck disable=SC2086
set -- $cmd
word="${1:-}"
[ -n "$word" ] || return 1
if [ -f "$word" ]; then
path="$word"
else
path="$(command -v -- "$word" 2>/dev/null)" || return 1
[ -f "$path" ] || return 1
fi
# realpath so the ledger keys on the actual file, not a symlink alias.
printf '%s\t%s' "$path" "$(_realpath_or_self "$path")"
}
# _preimage_set — print the derived set, one member per line as
# `origin<TAB>raw<TAB>resolved` (origin: core|extra). BOTH path forms travel
# with every member so the deny gate and the capture allowlist are evaluated
# on the SAME candidate in BOTH its forms — resolving before denying is the
# #964 case-C ordering defect. Paths from WAKE_PREIMAGE_EXTRA are printed
# EVEN IF ABSENT (deletion is a tracked state); the adapter and watch-list
# must resolve (loud failure otherwise — see _resolve_cmd_file rationale).
_preimage_set() {
local failed=0
if [ -n "${WAKE_DETECTOR_SOURCE_CMD:-}" ]; then
local af
if af="$(_resolve_cmd_file "$WAKE_DETECTOR_SOURCE_CMD")"; then
printf 'core\t%s\n' "$af"
else
echo "preimage.sh: FAIL LOUD — WAKE_DETECTOR_SOURCE_CMD ('$WAKE_DETECTOR_SOURCE_CMD') does not resolve to a file; the preimage definition cannot be observed (NOT treated as 'no change')." >&2
failed=1
fi
fi
if [ -n "${WAKE_WATCH_LIST:-}" ]; then
if [ -f "$WAKE_WATCH_LIST" ]; then
printf 'core\t%s\t%s\n' "$WAKE_WATCH_LIST" "$(_realpath_or_self "$WAKE_WATCH_LIST")"
else
echo "preimage.sh: FAIL LOUD — WAKE_WATCH_LIST ('$WAKE_WATCH_LIST') is not a file; the preimage definition cannot be observed." >&2
failed=1
fi
fi
if [ -n "${WAKE_PREIMAGE_EXTRA:-}" ]; then
local IFS=':' p
for p in $WAKE_PREIMAGE_EXTRA; do
[ -n "$p" ] || continue
# Extras are tracked even when absent (ABSENT is a state). Resolve the
# realpath only when the file exists; otherwise track the literal path.
if [ -e "$p" ]; then
printf 'extra\t%s\t%s\n' "$p" "$(_realpath_or_self "$p")"
else
printf 'extra\t%s\t%s\n' "$p" "$p"
fi
done
fi
return "$failed"
}
# --- credential hard gate (acceptance (b)) ----------------------------------
# _deny_path RAW RESOLVED — 0 (deny) iff EITHER form of the candidate names a
# known credential-store location. Framework-defined locations only
# (firewall): the mosaic credential store, the tools/_lib credential file the
# framework-manifest itself carves out of tools/**, the credentials/ operator
# subtree, and any file literally named credentials.json.
#
# Evaluated on BOTH the operator-supplied (raw) form and the symlink-resolved
# form, against BOTH the unresolved and resolved forms of the mosaic-home
# anchor: a deny list written only in unresolved paths cannot match a path
# that was resolved before it arrived (#964 review, case C — a symlink whose
# target was renamed slipped every path rule because the candidate had
# already been realpath'd but the anchors never were).
_deny_path() {
local raw="$1" resolved="$2"
local home="${MOSAIC_HOME:-$HOME/.config/mosaic}" rhome p a
rhome="$(_realpath_or_self "$home")"
for p in "$raw" "$resolved"; do
[ -n "$p" ] || continue
for a in "$home" "$rhome"; do
case "$p" in
"$a/credentials.json") return 0 ;;
"$a/tools/_lib/credentials.json") return 0 ;;
"$a/credentials/"*) return 0 ;;
esac
done
case "$(basename -- "$p")" in
credentials.json) return 0 ;;
esac
done
return 1
}
# _deny_content FILE — 0 (deny) iff FILE's bytes look secret-shaped.
# Two probes: (a) the well-known vendor-token shapes digest.sh redacts
# (PAT / Slack / AKIA / JWT / PEM — conservative: a 40-hex git sha never
# matches); (b) a named-assignment shape (key/token/secret/passw/hmac/
# credential/bearer = long unbroken value) that catches prefix-less
# high-entropy keys, e.g. an HMAC key in an env file (#964 review, case D).
#
# (b) is defense-in-depth for the OPT-IN path only, NOT the thing that keeps
# case D safe — polarity does that (extras are record-only unless allowlisted;
# a shape list can only refuse the secrets someone already enumerated). It is
# deliberately conservative toward REFUSAL: a false match (a checksum in a
# config, a path that happens to be long and unbroken) only withholds byte
# capture — the hash/size/mtime row and change entry still land. A variable
# REFERENCE (token="$GITEA_TOKEN") never matches: `$` is not in the value
# class — only literal secret values do.
#
# FAILS TOWARD REFUSAL: if a probe itself errors (unreadable file, grep
# failure — rc >= 2), the answer is DENY, not capture. An error exit is not a
# negative result.
_deny_content() {
local rc
LC_ALL=C grep -Eq \
-e 'gh[pousr]_[A-Za-z0-9]{16,}' \
-e 'github_pat_[A-Za-z0-9_]{16,}' \
-e 'xox[baprs]-[A-Za-z0-9-]{10,}' \
-e 'AKIA[0-9A-Z]{16}' \
-e 'eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}' \
-e '-----BEGIN[A-Z ]*PRIVATE KEY-----' \
-- "$1" 2>/dev/null
rc=$?
[ "$rc" -eq 1 ] || return 0
LC_ALL=C grep -Eiq \
-e "[A-Za-z0-9_]*(key|token|secret|passw|hmac|credential|bearer)[A-Za-z0-9_]*[[:space:]]*[=:][[:space:]]*[\"']?[A-Za-z0-9+/=_-]{16,}" \
-- "$1" 2>/dev/null
rc=$?
[ "$rc" -eq 1 ] || return 0
return 1
}
# _capture_allowed ORIGIN RAW RESOLVED — 0 iff byte capture may even be
# ATTEMPTED for this member (the deny gate still runs after, and wins).
# POLARITY (#964 review, case D): core members (the adapter file, the
# watch-list) are capture-eligible — they ARE the preimage definition this
# tool exists to snapshot (acceptance (a)). Extras are RECORD-ONLY unless
# listed in WAKE_PREIMAGE_CAPTURE.
#
# The allowlist is matched with the SAME both-forms discipline as the deny
# list (entry raw/resolved vs candidate raw/resolved): two lists keyed on
# different strings would let a symlink alias slip between them. "Deny wins
# over opt-in" is a precedence rule, not a guarantee both lists see the same
# path — the guarantee comes from _check_one running _deny_path on both forms
# FIRST, independent of anything matched here.
_capture_allowed() {
local origin="$1" raw="$2" resolved="$3"
[ "$origin" = "core" ] && return 0
[ -n "${WAKE_PREIMAGE_CAPTURE:-}" ] || return 1
local IFS=':' e er
for e in $WAKE_PREIMAGE_CAPTURE; do
[ -n "$e" ] || continue
er="$e"
[ -e "$e" ] && er="$(_realpath_or_self "$e")"
if [ "$e" = "$raw" ] || [ "$e" = "$resolved" ] ||
[ "$er" = "$raw" ] || [ "$er" = "$resolved" ]; then
return 0
fi
done
return 1
}
# --- ledger primitives ------------------------------------------------------
# _ledger_head PATH — print the LAST ledger row for PATH (empty if none).
# Non-zero (loud) if the ledger exists but is unparseable: a corrupt ledger
# must REFUSE to compare, never silently re-baseline (that would erase the
# attribution history this tool exists to keep).
_ledger_head() {
local path="$1"
[ -f "$LEDGER" ] || return 0
if [ -s "$LEDGER" ] && ! jq -cn 'inputs' <"$LEDGER" >/dev/null 2>&1; then
echo "preimage.sh: FAIL LOUD — preimage ledger $LEDGER is unparseable; REFUSING to compare or re-baseline over corrupt history. Investigate/restore the ledger." >&2
return 1
fi
jq -c --arg p "$path" 'select(.path == $p)' "$LEDGER" 2>/dev/null | tail -n 1
}
# _ledger_append ROW_JSON — append one row atomically (read + append + rename;
# rows are small and the writer is serialized by the preimage lock).
_ledger_append() {
local row="$1"
{
[ -f "$LEDGER" ] && cat "$LEDGER"
printf '%s\n' "$row"
} | _atomic_write "$LEDGER"
}
# --- the check itself -------------------------------------------------------
# _check_one ORIGIN RAW PATH ENQUEUE — compare PATH (the resolved form; the
# ledger keys on it) to its ledger head; on change record (+bytes only if
# capture-eligible and not denied) and optionally enqueue. Prints nothing on
# no-change. Returns: 0 ok (changed or not), 1 infrastructure failure.
_check_one() {
local origin="$1" raw="$2" path="$3" enqueue="$4"
local cur_sha size mtime denied="" refused=""
if [ -f "$path" ]; then
cur_sha="$(_hash_stdin <"$path")" || return 1
[ -n "$cur_sha" ] || {
echo "preimage.sh: FAIL LOUD — could not hash $path" >&2
return 1
}
size="$(wc -c <"$path" | tr -d '[:space:]')"
# Portable mtime (GNU stat -c / BSD stat -f).
mtime="$(stat -c %Y -- "$path" 2>/dev/null || stat -f %m -- "$path" 2>/dev/null || echo 0)"
local cap="${WAKE_PREIMAGE_MAX_BYTES:-1048576}"
case "$cap" in '' | *[!0-9]*) cap=1048576 ;; esac
# Gate order matters: path deny FIRST, on BOTH forms, INDEPENDENT of the
# capture allowlist — so an allowlisted symlink whose target is denied
# refuses no matter what string the allowlist matched (a known credential
# store is refused without a content probe; the hash read above is
# deliberate: a sha256 discloses nothing about the bytes). Then polarity
# (record-only extras never reach the content probe — case D must be safe
# WITHOUT the shape list), then size cap (never content-grep an oversized
# file), content shape last.
if _deny_path "$raw" "$path"; then
denied="credential-store path (deny list)"
elif ! _capture_allowed "$origin" "$raw" "$path"; then
denied="extra is record-only by default (byte capture requires WAKE_PREIMAGE_CAPTURE; deny rules still win)"
elif [ "$size" -gt "$cap" ]; then
denied="exceeds WAKE_PREIMAGE_MAX_BYTES ($size > $cap)"
elif _deny_content "$path"; then
denied="secret-shaped content"
fi
else
# ABSENT is a state (deletion of a preimage file is a change to record).
cur_sha="ABSENT"
size=0
mtime=0
fi
local head prev_sha=""
head="$(_ledger_head "$path")" || return 1
[ -n "$head" ] && prev_sha="$(jq -r '.sha256 // ""' <<<"$head")"
if [ "$cur_sha" = "$prev_sha" ]; then
return 0 # unchanged — no row, no enqueue (idempotent)
fi
# --- capture bytes (content-addressed) unless the hard gate refuses -------
local captured=true
if [ "$cur_sha" != "ABSENT" ]; then
if [ -n "$denied" ]; then
captured=false
refused="$denied"
echo "preimage.sh: byte capture WITHHELD for $path ($denied) — hash/size/mtime recorded, content NOT stored (#958)." >&2
else
mkdir -p "$OBJECTS" || return 1
if [ ! -f "$OBJECTS/$cur_sha" ]; then
if ! _atomic_write "$OBJECTS/$cur_sha" <"$path"; then
echo "preimage.sh: FAIL LOUD — durable object write failed for $path ($OBJECTS/$cur_sha); NO ledger row recorded (a row whose bytes were never durably kept would be a false witness)." >&2
return 1
fi
fi
fi
fi
local ts row
ts="$(date +%s)"
row="$(jq -cn \
--arg path "$path" \
--arg sha "$cur_sha" \
--arg prev "$prev_sha" \
--argjson size "$size" \
--argjson mtime "$mtime" \
--argjson ts "$ts" \
--argjson captured "$captured" \
--arg refused "$refused" \
'{ts:$ts, path:$path, sha256:$sha, size:$size, mtime:$mtime, captured:$captured}
+ (if $prev != "" then {prev:$prev} else {} end)
+ (if $refused != "" then {refused:$refused} else {} end)')" || return 1
if ! _ledger_append "$row"; then
echo "preimage.sh: FAIL LOUD — durable ledger append failed ($LEDGER)" >&2
return 1
fi
if [ -z "$head" ]; then
# First-seen: baseline SILENTLY (row only, no enqueue) — the same idiom as
# the detector's first-seen source baseline.
return 0
fi
echo "preimage.sh: preimage definition CHANGED — $path ($prev_sha -> $cur_sha); prior bytes: $OBJECTS/$prev_sha" >&2
if [ "$enqueue" = "1" ]; then
# First-class cause line (acceptance (c)): ONE class=actionable entry per
# changed preimage file, allocated by the store's single allocator (#908).
# `path` is a §2.1 hard locator, so the digest renders it — never
# quarantined. Callers run this BEFORE polling sources, so this seq is
# LOWER than the per-source deltas the change explains.
local locators
locators="$(jq -cn \
--arg path "$path" \
--arg sha "$cur_sha" \
--arg prev "$prev_sha" \
'{kind:"preimage", id:$path, path:$path, observed_hash:$sha, prev_hash:$prev,
preimage:true, reason:"preimage-definition-changed"}')"
if ! "$STORE_SH" enqueue --class actionable --locators "$locators" --emit-ts "$(date +%s)" >/dev/null; then
echo "preimage.sh: FAIL LOUD — store enqueue of the preimage-change entry FAILED for $path (the change IS recorded in the ledger; the first-class wake line is NOT — treat as infrastructure failure)." >&2
return 1
fi
fi
return 0
}
cmd_check() {
local enqueue=0
while [ $# -gt 0 ]; do
case "$1" in
--enqueue)
enqueue=1
shift
;;
*)
echo "preimage.sh check: unknown option '$1'" >&2
exit 2
;;
esac
done
_need_jq
local set_list
if ! set_list="$(_preimage_set)"; then
exit 1
fi
if [ -z "$set_list" ]; then
# Nothing declared (no adapter/watch-list/extras in env) — an empty set is
# a no-op, not an error: standalone invocations outside a wake runtime
# must not fail loud for lacking one.
return 0
fi
mkdir -p "$PRE_DIR" || exit 1
# Serialize concurrent checks (detector tick vs reconcile vs manual) — the
# ledger append is read+rewrite, so two writers must not interleave.
if ! _wake_lock_acquire "$PRE_DIR/preimage.lock"; then
echo "preimage.sh: FAIL LOUD — cannot acquire preimage lock" >&2
exit 1
fi
# #964 review (B11): an ABSENT/EMPTY ledger while objects/ still holds
# prior captures is DELETED HISTORY, not a first install — the asymmetry is
# locally detectable and is the whole detection. Re-baselining here would
# silently absorb the next real change as first-seen (the exact erasure
# this tool exists to prevent). ABSENT is not CORRUPT: it must not take the
# first-install path either.
if [ ! -s "$LEDGER" ] && [ -d "$OBJECTS" ] && [ -n "$(ls -A -- "$OBJECTS" 2>/dev/null)" ]; then
echo "preimage.sh: FAIL LOUD — preimage ledger $LEDGER is ABSENT/EMPTY but $OBJECTS still holds prior objects: history was deleted; REFUSING to re-baseline over it (a silent restart would absorb the next real change as first-seen). Restore the ledger, or move objects/ aside explicitly after investigation." >&2
_wake_lock_release
exit 1
fi
local failed=0 origin raw resolved
while IFS=$'\t' read -r origin raw resolved; do
[ -n "$resolved" ] || continue
_check_one "$origin" "$raw" "$resolved" "$enqueue" || failed=1
done <<EOF
$set_list
EOF
_wake_lock_release
[ "$failed" -eq 0 ] || exit 1
}
cmd_status() {
_need_jq
[ -f "$LEDGER" ] || {
echo "preimage.sh: no ledger yet ($LEDGER)" >&2
return 0
}
# Head row per path (last wins).
jq -cs 'group_by(.path) | map(last) | .[]' "$LEDGER"
}
cmd_set() {
local s
s="$(_preimage_set)" || exit 1
[ -n "$s" ] && printf '%s\n' "$s" | cut -f3
}
main() {
[ $# -ge 1 ] || {
usage
exit 2
}
local cmd="$1"
shift
case "$cmd" in
check) cmd_check "$@" ;;
record) cmd_check ;;
set) cmd_set "$@" ;;
status) cmd_status "$@" ;;
-h | --help | help) usage ;;
*)
echo "preimage.sh: unknown command '$cmd'" >&2
usage
exit 2
;;
esac
}
main "$@"
@@ -440,6 +440,17 @@ cmd_reconcile() {
: "$allow_enumerate"
local pairs kind id failed=0 unaccounted=0 enumerated=0
# #958 preimage provenance: same pre-step as the detector's poll tick, for
# the path where the preimage changed while the detector was down — without
# it, a changed adapter surfaces here only as N UNACCOUNTED enumerations
# with no first-class cause line. Run BEFORE observing any source so the
# cause entry's observed_seq precedes the enumerations it explains. Loud
# infrastructure failure marks the reconcile failed but does not stop it.
if ! "$SCRIPT_DIR/preimage.sh" check --enqueue; then
echo "reconcile.sh: FAIL LOUD — preimage provenance check failed (see preimage.sh above); reconcile continues but exits non-zero." >&2
failed=1
fi
pairs="$(printf '%s' "$json" | jq -r '[ .watches[].sources[] | "\(.kind)\t\(.id)" ] | unique | .[]')"
if [ -z "$pairs" ]; then
echo "reconcile.sh: watch-list declares no sources under watches[].sources[]" >&2
+19 -8
View File
@@ -110,9 +110,13 @@ Commands:
found); --repair removes exactly those
rows (atomic, loud). The dead-letter
ledger itself is NEVER modified (it is
history). Rows whose dead-letter evidence
was pruned are NOT provable and are
never touched.
history). TWO residual classes are
unprovable and never touched (#952):
rows whose dead-letter evidence was
pruned away, AND rows whose surviving
evidence extracts an empty
observed_hash (it can never satisfy
the four-field conviction match).
cursors Print observed_seq / consumed_seq / depth.
Environment:
@@ -563,10 +567,17 @@ cmd_quarantine_sync() {
# it on (kind, id, observed_seq, observed_hash) AND row.observed_seq <=
# consumed_seq: the per-key max_by merge means the surviving row's provenance IS
# that quarantined entry (a healed row differs in seq/hash and never matches).
# PROVABILITY BOUND: a row whose dead-letter evidence was pruned/rotated away is
# NOT provable and is never touched — this audit only ever removes what the
# ledger can convict. The dead-letter ledger itself is history and is NEVER
# modified here.
# PROVABILITY BOUND — TWO residual classes, both unprovable (#952): (1) a row
# whose dead-letter evidence was pruned/rotated away — no evidence to convict
# on; (2) a row whose dead-letter evidence SURVIVES but extracts an empty
# observed_hash (e.g. a deliberately non-conformant locator: nested
# .locators.* with no observed_hash key) — evidence exists but can never
# satisfy the four-field match, because _record_last_consumed only ever writes
# rows with a NON-empty hash. Neither class is touched — this audit only ever
# removes what the ledger can convict, and the clean-sweep message names BOTH
# classes: "no evidence" and "evidence unusable" are different operator
# conclusions. The dead-letter ledger itself is history and is NEVER modified
# here.
cmd_quarantine_audit() {
local repair=0
while [ $# -gt 0 ]; do
@@ -605,7 +616,7 @@ cmd_quarantine_audit() {
)) | length) > 0)
' "$rec" 2>/dev/null || true)"
if [ -z "$false_rows" ]; then
echo "store.sh quarantine-audit: OK — no provably-false consumed-hash rows (rows without surviving dead-letter evidence are not provable and were not judged)"
echo "store.sh quarantine-audit: OK — no provably-false consumed-hash rows. Two residual classes are unprovable and were NOT judged: rows whose dead-letter evidence was pruned/rotated away (no evidence to convict on), and rows whose surviving dead-letter evidence extracts an empty observed_hash (evidence exists but can never satisfy the four-field conviction match)."
return 0
fi
local n row
+534
View File
@@ -0,0 +1,534 @@
#!/usr/bin/env bash
# test-wake-preimage.sh — RED-FIRST invariant harness for A11 (#958): durable
# provenance for the OPERATOR-SIDE preimage definition (preimage.sh).
#
# Each test asserts ONE #958 invariant and is designed to go RED if it
# regresses:
# P1 first-seen -> SILENT baseline (ledger rows, NO enqueue) (detector idiom)
# P2 change -> attributable from durable state alone: new ledger row with
# prev, PRIOR BYTES retrievable content-addressed, first-class actionable
# enqueued with a §2.1 hard locator (path) (acceptance a+c)
# P3 detector ordering: the preimage cause line is enqueued at a LOWER
# observed_seq than the per-source delta it explains (acceptance c)
# P4 credential-store PATH is refused — bytes NEVER captured, even when the
# operator lists it by error (hash/mtime row still written) (acceptance b)
# P5 secret-SHAPED content is refused — refusal, not redaction (acceptance b)
# P6 deletion of a preimage file is a tracked CHANGE (ABSENT), not an error
# P7 no-change is idempotent (no row growth, no enqueue)
# P8 unresolvable adapter command -> FAIL LOUD, never "no change" (D2/#955 class)
# P9 corrupt ledger -> REFUSE to compare/re-baseline (loud, no append)
# P10 oversized file -> bytes refused, hash still recorded (no data hoovering)
# P11 reconcile pre-step: a preimage change surfaces as a first-class line
# BEFORE the UNACCOUNTED enumerations it explains (acceptance c)
# P12 detector: preimage infra failure is LOUD (pass exits non-zero) but does
# NOT starve source observation
#
# Regression needles from the #964 NOT-CLEAR review (decoy method: plant the
# bytes, grep the WHOLE state dir — the decoy does not know what the code
# believes):
# P13 case C: symlinked/renamed credential store (mosaic-home itself behind
# a symlink; a renamed target outside every path rule) — refused on BOTH
# path forms / by content, bytes grep-absent (acceptance b)
# P14 case D: prefix-less high-entropy key (detector.env class) — safe
# WITHOUT opt-in by POLARITY alone (must hold with the shape list
# deleted), and refused WITH opt-in by the named-assignment shape
# P15 polarity: extras are RECORD-ONLY by default (change row + enqueue, no
# bytes); byte capture requires explicit WAKE_PREIMAGE_CAPTURE opt-in
# P16 case B11: ledger deleted while objects/ survives — LOUD refusal to
# re-baseline; the v2->v3 change is NOT absorbed as first-seen
# P17 allowlist symmetry: an allowlisted symlink whose TARGET is denied
# refuses (deny runs first, on both forms — precedence is not path
# agreement); a symlink to an ALLOWED target still captures (the fix
# must not close case C by breaking every symlinked path)
#
# Uses FAKE/STUB sources only (no live network). Isolated per test.
# shellcheck disable=SC2030,SC2031
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PRE="$SCRIPT_DIR/preimage.sh"
STORE="$SCRIPT_DIR/store.sh"
DET="$SCRIPT_DIR/detector.sh"
RECON="$SCRIPT_DIR/reconcile.sh"
command -v jq >/dev/null 2>&1 || {
echo "SKIP: jq not available" >&2
exit 0
}
TMP_ROOT="$(mktemp -d)"
trap 'rm -rf "$TMP_ROOT"' EXIT
FAILFILE="$TMP_ROOT/failures"
: >"$FAILFILE"
pass=0
fail_msg() {
echo " FAIL: $*" >&2
echo "x" >>"$FAILFILE"
}
ok() { pass=$((pass + 1)); }
fresh_state() {
local d="$TMP_ROOT/$1"
rm -rf "$d"
mkdir -p "$d"
printf '%s' "$d"
}
depth() { "$STORE" cursors | sed -n 's/pending_depth=//p'; }
state_dir() { printf '%s/default' "$WAKE_STATE_HOME"; }
ledger_rows() {
local f
f="$(state_dir)/preimage/preimage-ledger.jsonl"
[ -f "$f" ] && wc -l <"$f" | tr -d '[:space:]' || echo 0
}
# make_stub DIR — a source adapter reading $DIR/<kind>_<id> (same shape as the
# reconcile harness stub).
make_stub() {
local dir="$1"
cat >"$dir/adapter.sh" <<EOF
#!/usr/bin/env bash
set -u
kind="\$1"; id="\$2"; base="$dir/\${kind}_\${id}"; rc=0
[ -f "\$base.rc" ] && rc="\$(cat "\$base.rc")"
[ -f "\$base" ] && cat "\$base"
exit "\$rc"
EOF
chmod +x "$dir/adapter.sh"
}
make_wl() { # make_wl FILE — one repo source r1
cat >"$1" <<'EOF'
{ "schema_version": 1,
"repos": [ { "id": "r1" } ],
"watches": [ { "lane": "L", "sources": [ { "kind": "repo", "id": "r1" } ] } ] }
EOF
}
echo "== P1: first-seen -> SILENT baseline (rows, no enqueue) =="
(
WAKE_STATE_HOME="$(fresh_state p1)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p1fx"; mkdir -p "$fx"
make_stub "$fx"; make_wl "$fx/wl.json"
printf 'extra v1\n' >"$fx/extra.env"
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh" WAKE_WATCH_LIST="$fx/wl.json" WAKE_PREIMAGE_EXTRA="$fx/extra.env"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P1: baseline check must exit 0 (rc=$rc) [$out]"
[ "$(ledger_rows)" -eq 3 ] || fail_msg "P1: expected 3 baseline rows (adapter, watch-list, extra), got $(ledger_rows)"
[ "$(depth)" = "0" ] || fail_msg "P1: first-seen must NOT enqueue (depth=$(depth))"
) && ok
echo "== P2: adapter change -> prior bytes + first-class hard-locator enqueue =="
(
WAKE_STATE_HOME="$(fresh_state p2)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p2fx"; mkdir -p "$fx"
make_stub "$fx"; make_wl "$fx/wl.json"
# Acceptance (a) names the OPERATOR ADAPTER — a core member. Core members
# capture by default (they ARE the preimage definition); extras do not (P15).
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh" WAKE_WATCH_LIST="$fx/wl.json"
unset WAKE_PREIMAGE_EXTRA WAKE_PREIMAGE_CAPTURE MOSAIC_HOME
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P2: baseline failed"
cp "$fx/adapter.sh" "$fx/adapter.v1"
old_sha="$(sha256sum "$fx/adapter.sh" | awk '{print $1}')"
printf '# adapter v2 CHANGED\n' >>"$fx/adapter.sh"
new_sha="$(sha256sum "$fx/adapter.sh" | awk '{print $1}')"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P2: change check must exit 0 (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(jq -c --arg p "$(realpath "$fx/adapter.sh")" 'select(.path == $p)' "$sd/preimage/preimage-ledger.jsonl" | tail -n1)"
[ "$(jq -r '.prev // ""' <<<"$row")" = "$old_sha" ] || fail_msg "P2: new row must carry prev=<old sha> [$row]"
[ "$(jq -r '.sha256' <<<"$row")" = "$new_sha" ] || fail_msg "P2: new row sha mismatch [$row]"
# Acceptance (a): the PRIOR BYTES are retrievable from durable state alone.
cmp -s "$sd/preimage/objects/$old_sha" "$fx/adapter.v1" || fail_msg "P2: prior bytes missing or not byte-exact"
cmp -s "$sd/preimage/objects/$new_sha" "$fx/adapter.sh" || fail_msg "P2: current bytes object missing or mismatched"
# Acceptance (c): ONE first-class actionable with a §2.1 hard locator (path).
[ "$(depth)" = "1" ] || fail_msg "P2: exactly one enqueue expected (depth=$(depth))"
ent="$(tail -n1 "$sd/pending.jsonl")"
[ "$(jq -r '.class' <<<"$ent")" = "actionable" ] || fail_msg "P2: entry class must be actionable [$ent]"
[ "$(jq -r '.locators.kind' <<<"$ent")" = "preimage" ] || fail_msg "P2: locator kind must be preimage [$ent]"
[ -n "$(jq -r '.locators.path // ""' <<<"$ent")" ] || fail_msg "P2: locator must carry path (hard locator) [$ent]"
[ "$(jq -r '.locators.observed_hash' <<<"$ent")" = "$new_sha" ] || fail_msg "P2: locator observed_hash mismatch [$ent]"
[ "$(jq -r '.locators.prev_hash' <<<"$ent")" = "$old_sha" ] || fail_msg "P2: locator prev_hash mismatch [$ent]"
) && ok
echo "== P3: detector orders the cause line BEFORE the delta it explains =="
(
WAKE_STATE_HOME="$(fresh_state p3)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p3fx"; mkdir -p "$fx"
make_stub "$fx"; make_wl "$fx/wl.json"
printf 'r1 state v1\n' >"$fx/repo_r1"
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh" WAKE_WATCH_LIST="$fx/wl.json"
unset WAKE_PREIMAGE_EXTRA
"$DET" poll-once >/dev/null 2>&1 || fail_msg "P3: baseline poll failed"
[ "$(depth)" = "0" ] || fail_msg "P3: baseline poll must enqueue nothing (depth=$(depth))"
# Change the ADAPTER (preimage) and the source state in the same window.
printf '# comment: adapter changed\n' >>"$fx/adapter.sh"
printf 'r1 state v2\n' >"$fx/repo_r1"
"$DET" poll-once >/dev/null 2>&1 || fail_msg "P3: second poll failed"
sd="$(state_dir)"
pre_seq="$(jq -r 'select(.locators.kind == "preimage") | .observed_seq' "$sd/pending.jsonl" | head -n1)"
src_seq="$(jq -r 'select(.locators.kind == "repo") | .observed_seq' "$sd/pending.jsonl" | head -n1)"
[ -n "$pre_seq" ] || fail_msg "P3: no preimage cause entry enqueued"
[ -n "$src_seq" ] || fail_msg "P3: no source delta entry enqueued"
if [ -n "$pre_seq" ] && [ -n "$src_seq" ]; then
[ "$pre_seq" -lt "$src_seq" ] || fail_msg "P3: cause line must precede the delta (preimage seq=$pre_seq, source seq=$src_seq)"
fi
) && ok
echo "== P4: credential-store PATH refused — bytes never captured (acceptance b) =="
(
WAKE_STATE_HOME="$(fresh_state p4)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p4fx"; mkdir -p "$fx"
export MOSAIC_HOME="$fx/mosaic-home"
mkdir -p "$MOSAIC_HOME"
secret='cred-value-P4-do-not-capture'
printf '{"svc":{"token":"%s"}}\n' "$secret" >"$MOSAIC_HOME/credentials.json"
# Operator error: the credential store listed as a preimage extra — AND
# explicitly opted in to capture. Deny must win over the opt-in (#964: a
# path both denied and allowlisted must refuse).
export WAKE_PREIMAGE_EXTRA="$MOSAIC_HOME/credentials.json"
export WAKE_PREIMAGE_CAPTURE="$MOSAIC_HOME/credentials.json"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P4: refusal is not an infra failure (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P4: row must record captured=false [$row]"
jq -r '.refused // ""' <<<"$row" | grep -qi 'path' || fail_msg "P4: refusal must name the path deny [$row]"
# THE invariant: the secret bytes exist NOWHERE under the wake state dir.
if grep -rq "$secret" "$sd" 2>/dev/null; then
fail_msg "P4: credential bytes leaked into the wake state dir"
fi
) && ok
echo "== P5: secret-SHAPED content refused (refusal, not redaction) =="
(
WAKE_STATE_HOME="$(fresh_state p5)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p5fx"; mkdir -p "$fx"
tok='ghp_abcdefghijklmnop0123456789ABCDEF'
printf 'export MY_TOKEN=%s\n' "$tok" >"$fx/leaky.env"
# Opted in, so the CONTENT gate (not record-only polarity) is what refuses.
export WAKE_PREIMAGE_EXTRA="$fx/leaky.env" WAKE_PREIMAGE_CAPTURE="$fx/leaky.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P5: refusal is not an infra failure (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P5: row must record captured=false [$row]"
jq -r '.refused // ""' <<<"$row" | grep -qi 'secret' || fail_msg "P5: refusal must name secret-shaped content [$row]"
if grep -rq "$tok" "$sd" 2>/dev/null; then
fail_msg "P5: secret-shaped bytes leaked into the wake state dir"
fi
) && ok
echo "== P6: deletion is a tracked CHANGE (ABSENT), not an error =="
(
WAKE_STATE_HOME="$(fresh_state p6)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p6fx"; mkdir -p "$fx"
printf 'to be deleted\n' >"$fx/gone.env"
export WAKE_PREIMAGE_EXTRA="$fx/gone.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P6: baseline failed"
rm -f "$fx/gone.env"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P6: deletion check must exit 0 (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.sha256' <<<"$row")" = "ABSENT" ] || fail_msg "P6: deletion must record sha256=ABSENT [$row]"
[ "$(depth)" = "1" ] || fail_msg "P6: deletion is a change and must enqueue (depth=$(depth))"
) && ok
echo "== P7: no-change is idempotent =="
(
WAKE_STATE_HOME="$(fresh_state p7)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p7fx"; mkdir -p "$fx"
printf 'stable\n' >"$fx/stable.env"
export WAKE_PREIMAGE_EXTRA="$fx/stable.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P7: baseline failed"
r1="$(ledger_rows)"
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P7: second check failed"
[ "$(ledger_rows)" = "$r1" ] || fail_msg "P7: no-change must not append rows ($r1 -> $(ledger_rows))"
[ "$(depth)" = "0" ] || fail_msg "P7: no-change must not enqueue (depth=$(depth))"
) && ok
echo "== P8: unresolvable adapter -> FAIL LOUD, never 'no change' (D2 class) =="
(
WAKE_STATE_HOME="$(fresh_state p8)"
export WAKE_STATE_HOME
unset WAKE_AGENT
export WAKE_DETECTOR_SOURCE_CMD="$TMP_ROOT/does-not-exist-adapter"
unset WAKE_WATCH_LIST WAKE_PREIMAGE_EXTRA MOSAIC_HOME
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -ne 0 ] || fail_msg "P8: an unobservable preimage definition must FAIL LOUD (rc=0)"
echo "$out" | grep -qi 'does not resolve' || fail_msg "P8: the failure must name the unresolvable adapter [$out]"
) && ok
echo "== P9: corrupt ledger -> REFUSE to compare/re-baseline =="
(
WAKE_STATE_HOME="$(fresh_state p9)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p9fx"; mkdir -p "$fx"
printf 'v1\n' >"$fx/f.env"
export WAKE_PREIMAGE_EXTRA="$fx/f.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P9: baseline failed"
sd="$(state_dir)"
printf 'NOT-JSON-GARBAGE{{{\n' >>"$sd/preimage/preimage-ledger.jsonl"
r1="$(wc -l <"$sd/preimage/preimage-ledger.jsonl")"
printf 'v2 changed\n' >"$fx/f.env"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -ne 0 ] || fail_msg "P9: a corrupt ledger must FAIL LOUD (rc=0)"
echo "$out" | grep -qi 'unparseable' || fail_msg "P9: the failure must name the corrupt ledger [$out]"
[ "$(wc -l <"$sd/preimage/preimage-ledger.jsonl")" = "$r1" ] || fail_msg "P9: nothing may be appended over corrupt history"
) && ok
echo "== P10: oversized file -> bytes refused, hash still recorded =="
(
WAKE_STATE_HOME="$(fresh_state p10)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p10fx"; mkdir -p "$fx"
head -c 200 /dev/zero | tr '\0' 'A' >"$fx/big.bin"
# Opted in, so the SIZE gate (not record-only polarity) is what refuses.
export WAKE_PREIMAGE_EXTRA="$fx/big.bin" WAKE_PREIMAGE_MAX_BYTES=64
export WAKE_PREIMAGE_CAPTURE="$fx/big.bin"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P10: size refusal is not an infra failure (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P10: row must record captured=false [$row]"
jq -r '.refused // ""' <<<"$row" | grep -qi 'MAX_BYTES' || fail_msg "P10: refusal must name the size cap [$row]"
sha="$(jq -r '.sha256' <<<"$row")"
[ "$sha" = "$(sha256sum "$fx/big.bin" | awk '{print $1}')" ] || fail_msg "P10: hash must still be recorded [$row]"
[ ! -f "$sd/preimage/objects/$sha" ] || fail_msg "P10: oversized bytes must NOT be stored"
) && ok
echo "== P11: reconcile surfaces the cause line before its enumerations =="
(
WAKE_STATE_HOME="$(fresh_state p11)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p11fx"; mkdir -p "$fx"
make_stub "$fx"; make_wl "$fx/wl.json"
printf 'r1 state\n' >"$fx/repo_r1"
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh" WAKE_WATCH_LIST="$fx/wl.json"
unset WAKE_PREIMAGE_EXTRA MOSAIC_HOME
# Baseline the preimage set, then change the adapter while "the detector is
# down" — the reconcile path must still produce the first-class cause line.
"$PRE" check >/dev/null 2>&1 || fail_msg "P11: preimage baseline failed"
printf '# adapter changed while detector down\n' >>"$fx/adapter.sh"
"$RECON" reconcile >/dev/null 2>&1 # rc 1 expected (unaccounted enumerated)
sd="$(state_dir)"
pre_seq="$(jq -r 'select(.locators.kind == "preimage") | .observed_seq' "$sd/pending.jsonl" | head -n1)"
enum_seq="$(jq -r 'select(.locators.reconciled == true) | .observed_seq' "$sd/pending.jsonl" | head -n1)"
[ -n "$pre_seq" ] || fail_msg "P11: reconcile must enqueue the preimage cause line"
[ -n "$enum_seq" ] || fail_msg "P11: reconcile must still enumerate the unaccounted source"
if [ -n "$pre_seq" ] && [ -n "$enum_seq" ]; then
[ "$pre_seq" -lt "$enum_seq" ] || fail_msg "P11: cause line must precede the enumeration (preimage seq=$pre_seq, enum seq=$enum_seq)"
fi
) && ok
echo "== P12: detector — preimage infra failure is LOUD but does not starve observation =="
(
WAKE_STATE_HOME="$(fresh_state p12)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p12fx"; mkdir -p "$fx"
make_stub "$fx"; make_wl "$fx/wl.json"
printf 'r1 state\n' >"$fx/repo_r1"
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh" WAKE_WATCH_LIST="$fx/wl.json"
unset WAKE_PREIMAGE_EXTRA MOSAIC_HOME
# Corrupt the ledger BEFORE the first poll: the preimage check fails loud,
# but the poll must still observe + baseline the source.
sd="$(state_dir)"
mkdir -p "$sd/preimage"
printf 'NOT-JSON-GARBAGE{{{\n' >"$sd/preimage/preimage-ledger.jsonl"
out="$("$DET" poll-once 2>&1)"; rc=$?
[ "$rc" -ne 0 ] || fail_msg "P12: the pass must exit non-zero on preimage infra failure"
echo "$out" | grep -qi 'preimage' || fail_msg "P12: the failure must name the preimage check [$out]"
n="$(find "$sd/detector" -name 'watch-*.hash' 2>/dev/null | wc -l | tr -d '[:space:]')"
[ "$n" -ge 1 ] || fail_msg "P12: source observation must still proceed (no watch hash baselined)"
) && ok
echo "== P13: symlinked/renamed credential store refused on BOTH path forms (#964 C) =="
(
WAKE_STATE_HOME="$(fresh_state p13)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p13fx"; mkdir -p "$fx"
# The mosaic home is ITSELF behind a symlink: a deny list written in
# unresolved paths cannot match a candidate that was realpath'd before the
# deny saw it — unless the anchors are resolved too.
mkdir -p "$fx/real-store/credentials"
ln -s "$fx/real-store" "$fx/mh"
export MOSAIC_HOME="$fx/mh"
s1='decoy-P13a-renamed-target-under-store'
printf 'x_token=%s\n' "$s1" >"$fx/mh/credentials/brain-creds.json"
# And a renamed target OUTSIDE every known store location, reached via a
# benign-looking symlink: no path rule can name it — the content gate must
# hold on the resolved file.
s2='0123456789abcdef0123456789abcdefP13b'
printf 'service_hmac_key=%s\n' "$s2" >"$fx/renamed-anywhere.cfg"
ln -s "$fx/renamed-anywhere.cfg" "$fx/link-b.env"
export WAKE_PREIMAGE_EXTRA="$fx/mh/credentials/brain-creds.json:$fx/link-b.env"
# Explicit opt-in for BOTH: deny must win over the allowlist.
export WAKE_PREIMAGE_CAPTURE="$fx/mh/credentials/brain-creds.json:$fx/link-b.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P13: refusal is not an infra failure (rc=$rc) [$out]"
sd="$(state_dir)"
n="$(jq -r 'select(.captured == false) | .path' "$sd/preimage/preimage-ledger.jsonl" | wc -l | tr -d '[:space:]')"
[ "$n" = "2" ] || fail_msg "P13: both decoys must record captured=false (got $n)"
# THE invariant (decoy method): the planted bytes exist NOWHERE in state.
if grep -rq "$s1" "$sd" 2>/dev/null; then
fail_msg "P13: renamed-target store bytes leaked into the wake state dir"
fi
if grep -rq "$s2" "$sd" 2>/dev/null; then
fail_msg "P13: prefix-less key bytes leaked into the wake state dir"
fi
) && ok
echo "== P14: detector.env-class key — safe WITHOUT opt-in by polarity, refused WITH opt-in by shape (#964 D) =="
(
WAKE_STATE_HOME="$(fresh_state p14)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p14fx"; mkdir -p "$fx"
hm='9f8e7d6c5b4a39281706f5e4d3c2b1a09f8e7d6c'
printf 'WAKE_HMAC_KEY=%s\n' "$hm" >"$fx/detector.env"
export WAKE_PREIMAGE_EXTRA="$fx/detector.env"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME WAKE_PREIMAGE_CAPTURE
# (a) NO opt-in: polarity alone keeps the bytes out. This leg must hold even
# with the content shape list deleted — the shape is defense-in-depth for
# the opt-in path, never the thing that makes case D read safe.
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P14a: record-only tracking must succeed (rc=$rc) [$out]"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P14a: extra must be record-only without opt-in [$row]"
if grep -rq "$hm" "$sd" 2>/dev/null; then
fail_msg "P14a: key bytes leaked without any opt-in"
fi
# (b) The operator opts the env file in (the exact error the old usage text
# invited): the named-assignment shape must still refuse the bytes.
export WAKE_PREIMAGE_CAPTURE="$fx/detector.env"
printf 'WAKE_HMAC_KEY=%s\nrotated=1\n' "$hm" >"$fx/detector.env"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P14b: refusal is not an infra failure (rc=$rc) [$out]"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P14b: opted-in key material must still refuse [$row]"
jq -r '.refused // ""' <<<"$row" | grep -qi 'secret' || fail_msg "P14b: refusal must name secret-shaped content [$row]"
if grep -rq "$hm" "$sd" 2>/dev/null; then
fail_msg "P14b: key bytes leaked despite refusal"
fi
) && ok
echo "== P15: extras are RECORD-ONLY by default; capture requires explicit opt-in =="
(
WAKE_STATE_HOME="$(fresh_state p15)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p15fx"; mkdir -p "$fx"
printf 'plain v1\n' >"$fx/notes.cfg"
export WAKE_PREIMAGE_EXTRA="$fx/notes.cfg"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST MOSAIC_HOME WAKE_PREIMAGE_CAPTURE
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P15: baseline failed"
printf 'plain v2\n' >"$fx/notes.cfg"
sha2="$(sha256sum "$fx/notes.cfg" | awk '{print $1}')"
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P15: change check failed"
sd="$(state_dir)"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "false" ] || fail_msg "P15: default must be record-only [$row]"
[ ! -f "$sd/preimage/objects/$sha2" ] || fail_msg "P15: bytes captured without opt-in"
[ "$(depth)" = "1" ] || fail_msg "P15: record-only must still track the change (depth=$(depth))"
# Opt in -> the next change captures.
export WAKE_PREIMAGE_CAPTURE="$fx/notes.cfg"
printf 'plain v3\n' >"$fx/notes.cfg"
sha3="$(sha256sum "$fx/notes.cfg" | awk '{print $1}')"
"$PRE" check --enqueue >/dev/null 2>&1 || fail_msg "P15: opted-in change check failed"
row="$(tail -n1 "$sd/preimage/preimage-ledger.jsonl")"
[ "$(jq -r '.captured' <<<"$row")" = "true" ] || fail_msg "P15: opt-in must capture [$row]"
cmp -s "$sd/preimage/objects/$sha3" "$fx/notes.cfg" || fail_msg "P15: opted-in bytes missing or mismatched"
) && ok
echo "== P16: deleted ledger over surviving objects/ — REFUSE to re-baseline (#964 B11) =="
(
WAKE_STATE_HOME="$(fresh_state p16)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p16fx"; mkdir -p "$fx"
make_stub "$fx"
export WAKE_DETECTOR_SOURCE_CMD="$fx/adapter.sh"
unset WAKE_WATCH_LIST WAKE_PREIMAGE_EXTRA WAKE_PREIMAGE_CAPTURE MOSAIC_HOME
"$PRE" check >/dev/null 2>&1 || fail_msg "P16: v1 baseline failed"
printf '# v2\n' >>"$fx/adapter.sh"
"$PRE" check >/dev/null 2>&1 || fail_msg "P16: v2 change failed"
sd="$(state_dir)"
nobj="$(ls "$sd/preimage/objects" | wc -l | tr -d '[:space:]')"
# Delete the ledger; objects/ survives. ABSENT is not CORRUPT — and it is
# not a first install either: that asymmetry is locally detectable.
rm -f "$sd/preimage/preimage-ledger.jsonl"
printf '# v3\n' >>"$fx/adapter.sh"
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -ne 0 ] || fail_msg "P16: absent ledger over prior objects must FAIL LOUD (rc=0) [$out]"
echo "$out" | grep -qi 'REFUSING to re-baseline' || fail_msg "P16: the failure must name the refusal [$out]"
[ ! -e "$sd/preimage/preimage-ledger.jsonl" ] || fail_msg "P16: the ledger must NOT be silently recreated over deleted history"
[ "$(ls "$sd/preimage/objects" | wc -l | tr -d '[:space:]')" = "$nobj" ] || fail_msg "P16: prior objects must remain untouched"
[ "$(depth)" = "0" ] || fail_msg "P16: the v2->v3 change must NOT be absorbed or enqueued from an unverifiable baseline (depth=$(depth))"
) && ok
echo "== P17: allowlist symmetry — symlink to a DENIED target refuses; symlink to an allowed target captures =="
(
WAKE_STATE_HOME="$(fresh_state p17)"
export WAKE_STATE_HOME
unset WAKE_AGENT
fx="$TMP_ROOT/p17fx"; mkdir -p "$fx"
export MOSAIC_HOME="$fx/mh"
mkdir -p "$MOSAIC_HOME"
s='decoy-P17-cred-behind-benign-alias'
printf '{"svc":{"token":"%s"}}\n' "$s" >"$MOSAIC_HOME/credentials.json"
# A benign-looking alias whose TARGET is denied: the allowlist matches the
# alias string, but deny runs FIRST and on BOTH forms — "deny wins over
# opt-in" is a precedence rule, path agreement is what makes it hold.
ln -s "$MOSAIC_HOME/credentials.json" "$fx/settings.json"
# And a symlink to an ALLOWED target: the fix must not close case C by
# breaking every symlinked path outright.
printf 'benign payload v1\n' >"$fx/target.cfg"
ln -s "$fx/target.cfg" "$fx/alias.cfg"
export WAKE_PREIMAGE_EXTRA="$fx/settings.json:$fx/alias.cfg"
export WAKE_PREIMAGE_CAPTURE="$fx/settings.json:$fx/alias.cfg"
unset WAKE_DETECTOR_SOURCE_CMD WAKE_WATCH_LIST
out="$("$PRE" check --enqueue 2>&1)"; rc=$?
[ "$rc" -eq 0 ] || fail_msg "P17: check must exit 0 (rc=$rc) [$out]"
sd="$(state_dir)"
crow="$(jq -c --arg p "$(realpath "$fx/settings.json")" 'select(.path == $p)' "$sd/preimage/preimage-ledger.jsonl" | tail -n1)"
[ "$(jq -r '.captured' <<<"$crow")" = "false" ] || fail_msg "P17: allowlisted symlink to a denied target must refuse [$crow]"
if grep -rq "$s" "$sd" 2>/dev/null; then
fail_msg "P17: credential bytes leaked via an allowlisted alias"
fi
brow="$(jq -c --arg p "$(realpath "$fx/alias.cfg")" 'select(.path == $p)' "$sd/preimage/preimage-ledger.jsonl" | tail -n1)"
[ "$(jq -r '.captured' <<<"$brow")" = "true" ] || fail_msg "P17: symlink to an allowed target must still capture [$brow]"
) && ok
echo
total=$((pass + $(wc -l <"$FAILFILE")))
echo "== test-wake-preimage: $pass/$total passed =="
[ -s "$FAILFILE" ] && exit 1
exit 0
@@ -28,6 +28,15 @@
# T16 #946 quarantine-audit: consumed-hashes rows provably false against the
# dead-letter ledger are reported (exit 1) and removed only under --repair;
# healed rows and the ledger itself are untouched
# T17 #952 quarantine-audit clean sweep names BOTH unprovable residual
# classes: evidence pruned (no evidence) AND evidence surviving with an
# empty observed_hash (evidence unusable). Fixture is the VERBATIM live
# specimen (mos-dt lane dead-letter seq 13, bench/malformed-locator-test)
# — a nested .locators.* row with NO observed_hash key. Do NOT hand-build
# a FLAT dead-letter fixture here: consumed-hashes rows are genuinely
# flat, dead-letter rows are genuinely nested, and a flat fixture makes
# the audit's CORRECT non-conviction look exactly like the defect under
# hunt (this false-defect near-miss actually happened in the #951 review)
#
# Isolated: every test runs against a fresh WAKE_STATE_HOME temp dir.
set -uo pipefail
@@ -680,6 +689,50 @@ echo "== T16: #946 — quarantine-audit: a consumed-hash row matching a dead-let
grep -qi 'OK' "$TMP_ROOT/t16.ok" || fail_msg "T16: a clean audit must say OK [$(cat "$TMP_ROOT/t16.ok")]"
) && ok
echo "== T17: #952 — the clean-sweep message names BOTH unprovable residual classes; a surviving-but-empty-hash dead-letter row is correctly NOT convicted =="
(
WAKE_STATE_HOME="$(fresh_state t17)"
export WAKE_STATE_HOME
unset WAKE_AGENT
STATE_DIR="$WAKE_STATE_HOME/default"
rec="$STATE_DIR/consumed-hashes.jsonl"
dl="$STATE_DIR/dead-letter.jsonl"
# Drive the REAL flow so the consumed-hashes row is produced by the actual
# writer (_record_last_consumed), not hand-built: 12 fillers, then the bench
# key lands at observed_seq 13 — the same seq as the live specimen below.
i=1
while [ "$i" -le 12 ]; do
"$STORE" enqueue --class actionable --locators "{\"kind\":\"filler\",\"id\":\"f$i\",\"observed_hash\":\"HF$i\"}" >/dev/null || fail_msg "T17: filler enqueue $i failed"
i=$((i + 1))
done
"$STORE" enqueue --class actionable --locators '{"kind":"bench","id":"malformed-locator-test","observed_hash":"H-REAL-CONSUMED"}' >/dev/null
"$STORE" consume --upto 13 >/dev/null 2>&1 || fail_msg "T17: baseline consume failed"
jq_any "$rec" '.kind=="bench" and .id=="malformed-locator-test" and .observed_seq==13 and .observed_hash=="H-REAL-CONSUMED"' ||
fail_msg "T17: fixture broken — the real writer did not record the bench@13 row"
# VERBATIM live specimen: mos-dt lane dead-letter.jsonl line 1 (2026-07-26).
# Copied byte-for-byte per the #952 fixture constraint — nested .locators.*
# with NO observed_hash key, so the audit's extraction yields "".
printf '%s\n' '{"observed_seq":13,"locators":{"kind":"bench","id":"malformed-locator-test","note":"deliberately non-conformant"},"class":"actionable","emit_ts":1785055610,"hmac":""}' >"$dl"
# Guard the near-miss trap: the specimen must be genuinely NESTED (no
# top-level kind) and must extract an EMPTY hash via the audit's own paths.
jq -e '(has("kind") | not) and .locators.kind=="bench" and ((.locators.observed_hash // "") == "")' "$dl" >/dev/null ||
fail_msg "T17: specimen fixture is not the genuine nested shape — rebuild it from the live lane, not the schema"
# Evidence SURVIVES (kind/id/seq all match the row) but extracts hash "";
# _record_last_consumed never writes an empty-hash row, so the four-field
# match can never fire: this is unprovable class 2, NOT a false witness.
rep="$TMP_ROOT/t17.rep"
"$STORE" quarantine-audit >"$rep" 2>&1 || fail_msg "T17: the audit must exit 0 — nothing here is provable [$(cat "$rep")]"
grep -q 'FALSE WITNESS' "$rep" && fail_msg "T17: the empty-hash evidence must NOT convict (the predicate is correct and must not change) [$(cat "$rep")]"
# The wording under test (#952): BOTH residual classes, named.
grep -qi 'pruned' "$rep" || fail_msg "T17: clean sweep must name residual class 1 — evidence pruned away [$(cat "$rep")]"
grep -qi 'empty observed_hash' "$rep" || fail_msg "T17: clean sweep must name residual class 2 — surviving evidence with an empty observed_hash [$(cat "$rep")]"
# --repair on a clean sweep removes nothing: the unprovable row survives.
"$STORE" quarantine-audit --repair >"$TMP_ROOT/t17.fix" 2>&1 || fail_msg "T17: --repair on a clean sweep must exit 0 [$(cat "$TMP_ROOT/t17.fix")]"
jq_any "$rec" '.kind=="bench" and .observed_seq==13 and .observed_hash=="H-REAL-CONSUMED"' ||
fail_msg "T17: --repair must NOT remove the unprovable bench@13 row — the audit only removes what the ledger can convict"
[ "$(grep -c . "$dl")" = "1" ] || fail_msg "T17: the dead-letter LEDGER must be untouched"
) && ok
echo
if [ -s "$FAILFILE" ]; then
echo "wake store/ack harness: FAILED ($(grep -c . "$FAILFILE") assertion(s))" >&2
+1 -1
View File
@@ -25,7 +25,7 @@
"lint": "eslint src",
"typecheck": "tsc --noEmit",
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
"test:framework-shell": "python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-install.sh"
"test:framework-shell": "python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
},
"dependencies": {
"@mosaicstack/brain": "workspace:*",