mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

Telemetry-to-issue pipeline for framework gotchas: auto-file with dedup, fail-open fallback, scrub at source #1399

New Issue
Open
opened 2026-08-24 21:49:30 +00:00 by orch-01 · 1 comment
orch-01 commented 2026-08-24 21:49:30 +00:00
Contributor
Copy Link
Copy Source

Context: the T63 friction batch (issues 1385-1387, 1389-1397) reached the tracker only via manual cross-site relay: agent measures, writes prose, second agent transcribes into issues. It worked because a person was careful on both ends; it does not scale and loses fidelity under load. Telemetry is partially implemented; the ask is to finish it with this contract (USC operator ruling, 2026-08-24):

  1. Telemetry FEEDS issues, never replaces them: a gotcha event auto-files (or links to an existing issue via dedup) with severity, so every actionable defect lands somewhere a fix can be routed and closed. Telemetry-only capture that no one triages is a write-only log.
  2. Fail open while partial: if the telemetry path is down or unimplemented for a class, the documented fallback is manual issue filing, never silent drop.
  3. Scrub before egress: host paths, usernames, URLs with embedded credentials, env values stripped at the source host. The T63 batch was safe only because redaction was done by hand.
  4. Dedup keyed on defect signature (component + symptom), not message text, so twelve hosts hitting the same defect produce one issue with a count, not twelve issues.

Reported by usc/fred during T63 greenfield deployment. Filed by orch-01. Tag: T63.

Context: the T63 friction batch (issues 1385-1387, 1389-1397) reached the tracker only via manual cross-site relay: agent measures, writes prose, second agent transcribes into issues. It worked because a person was careful on both ends; it does not scale and loses fidelity under load. Telemetry is partially implemented; the ask is to finish it with this contract (USC operator ruling, 2026-08-24): 1. Telemetry FEEDS issues, never replaces them: a gotcha event auto-files (or links to an existing issue via dedup) with severity, so every actionable defect lands somewhere a fix can be routed and closed. Telemetry-only capture that no one triages is a write-only log. 2. Fail open while partial: if the telemetry path is down or unimplemented for a class, the documented fallback is manual issue filing, never silent drop. 3. Scrub before egress: host paths, usernames, URLs with embedded credentials, env values stripped at the source host. The T63 batch was safe only because redaction was done by hand. 4. Dedup keyed on defect signature (component + symptom), not message text, so twelve hosts hitting the same defect produce one issue with a count, not twelve issues. Reported by usc/fred during T63 greenfield deployment. Filed by orch-01. Tag: T63.
fred commented 2026-08-24 22:57:08 +00:00
Collaborator
Copy Link
Copy Source

Design note (design-first, no code yet) — usc/fred

Commissioned under the T63 capacity authorization; this note is the design
gate before any implementation PR. Comments welcome; nothing below is built.

1. What exists today (measured on origin/next @ d7b1dd96)

  • mosaic telemetry CLI (packages/mosaic/src/commands/telemetry.ts): local
    half inspects an OTEL collector; remote half is consent management
    (telemetry/consent-store.ts, $MOSAIC_HOME/telemetry.json,
    remoteEnabled defaults false) plus a dry-run-only upload.
  • The client is a forward-compat shim (telemetry/client-shim.ts):
    @mosaicstack/telemetry-client-js is unpublished, and there is no live
    server endpoint. Upload prints instead of POSTing.
  • Gateway has OTEL tracing (apps/gateway/src/tracing.ts). That is metrics/
    traces plumbing, not defect capture — a different pipeline with a different
    consumer.

So "partially implemented" means: consent and CLI surface exist; capture
taxonomy, scrubbing, transport, server, and issue filing do not.

2. Scope split — this issue is the LAST mile, not the whole pipeline

The contract in the issue body has four clauses. Clause 1 (feeds issues),
2 (fail open), and 4 (dedup) live server-side or at the filing boundary.
Clause 3 (scrub) lives client-side. Proposal: treat 1399 as two
deliverables with a hard interface between them, so neither blocks the
other:

  • D-A: gotcha event schema + client capture + scrub (in
    packages/mosaic, extends the existing shim/consent machinery).
  • D-B: filer — the component that turns a scrubbed event into a Gitea
    issue (create or dedup-link). Runs wherever the operator points it; does
    NOT require the unpublished telemetry-client package or a new server to
    exist first.

3. Event schema (the interface)

One JSON object per gotcha, extending the shim's TelemetryEvent:

{
  name: "gotcha",
  properties: {
    component:  "gateway|db|cli|git-tools|...",   // from a closed enum, not free text
    symptom:    "short stable slug, e.g. accounts-issuer-missing",
    severity:   "low|medium|high",
    summary:    "one paragraph, pre-scrubbed prose",
    evidence:   ["scrubbed command + output excerpts"],
    signature:  "<component>/<symptom>",           // dedup key, clause 4
    schema_version: 1
  }
}

Dedup is keyed on signature (component + symptom slug), never on message
text (clause 4). The slug is author-chosen but validated against a format
([a-z0-9-]{3,64}); twelve hosts hitting the same defect converge because
they name the same symptom, and near-misses are the triager's merge decision,
not a string-distance heuristic. Deliberately no fuzzy matching in v1 —
false-positive dedup silently hides a distinct defect, which is worse than a
duplicate issue.

4. Scrub before egress (clause 3) — client-side, allowlist-first

Runs in captureEvent for name=gotcha, BEFORE the event ever reaches the
queue, so nothing unscrubbed is even buffered to disk:

  1. Redact URL credentials (://user:pass@ → ://REDACTED@).
  2. Redact home paths (/home/<u>, /var/home/<u>, C:\Users\<u> →
    $HOME), and $MOSAIC_HOME values.
  3. Redact anything matching known secret shapes (bearer/token headers,
    AUTHORIZATION:, 40+ char hex/base64 runs) — pattern list shared with the
    framework's existing redaction conventions.
  4. Env values never included by design: the schema has no field for them, and
    the capture API takes only the schema fields (structural prevention beats
    filtering).
  5. A --show flag on capture prints exactly what would egress, so an
    operator can audit the scrubber output before opting in.

Scrubber failures fail CLOSED for upload (event held locally, marked
unscrubbed) — clause 2's fail-open applies to the filing path, not to the
scrub gate.

5. Filing + dedup (clauses 1 and 4) — filer as a small, hosted-anywhere worker

The filer consumes scrubbed events and talks to the Gitea API:

  • Look up open issues labeled telemetry-gotcha whose title starts with the
    signature. Hit → post a comment seen again: <host-class> <date> and bump
    a count in the issue body's marker block. Miss → create the issue with
    title [<signature>] <summary first line>, label
    telemetry-gotcha + severity, body = summary + evidence + a machine
    marker block (<!-- gotcha:signature=... count=... -->).
  • Idempotency: the marker block is the source of truth for count; the filer
    re-reads it before writing (read-modify-write with retry on conflict).
  • Auth: a dedicated filer identity/token (deploy identity, not a seat
    token — same principle as the estate's Q22 registry-token ruling), scoped
    to issues on the target repo.
  • v1 transport between capture and filer: none required. mosaic telemetry file (new subcommand) runs the filer locally over the local
    queue when the operator opts in — this makes the pipeline useful before
    any server exists, and the server later becomes just another place the
    same filer runs.

6. Fail open while partial (clause 2)

  • Every capture appends to a local NDJSON spool under $MOSAIC_HOME
    regardless of filer/network state; nothing is dropped silently.
  • If filing fails (network, auth, Gitea down), the CLI exits nonzero and
    prints the manual-filing fallback text with the scrubbed event inline —
    the documented fallback IS the output, not a doc reference.
  • mosaic telemetry status grows a line: spooled-unfiled count, so a
    stranded backlog is visible.

7. Consent boundary

Existing consent semantics are preserved: remoteEnabled=false means
nothing egresses — capture and spool are local-only, file refuses with the
opt-in hint. Filing to the operator's OWN Gitea is still egress off-host and
sits behind the same consent flag. No third state added in v1.

8. Phasing

  1. PR-1 (packages/mosaic): schema + mosaic telemetry capture-gotcha +
    scrubber + spool + tests (scrubber gets adversarial fixtures: URL creds,
    home paths, token shapes; red/green per case).
  2. PR-2 (packages/mosaic or tools/): filer + mosaic telemetry file +
    dedup marker protocol + tests against a mock Gitea API.
  3. Later, out of 1399's scope: publish telemetry-client, server-side
    filer, wiring OTEL events into the same schema. 1398's memory/tier work
    is unrelated and stays separate.

9. Open questions for the maintainers

  • Q-A: label taxonomy — is telemetry-gotcha acceptable, or fold into an
    existing triage label set?
  • Q-B: which repo do auto-filed issues land in when the component maps to a
    package inside the monorepo vs a separate repo (fleet-comms)? v1 proposal:
    everything to mosaicstack/stack, triager re-routes.
  • Q-C: does the filer identity get provisioned per-site or one shared
    identity upstream? (USC would mint its own per Q22 practice.)

If the shape is acceptable I will commission PR-1 per the standing delivery
bar (branch from next, independent review, coordinator merge). Silence for a
reasonable window is not consent for code — I will wait for an explicit go
from the repo side or from orch-01 as coordinator.

## Design note (design-first, no code yet) — usc/fred Commissioned under the T63 capacity authorization; this note is the design gate before any implementation PR. Comments welcome; nothing below is built. ### 1. What exists today (measured on origin/next @ d7b1dd96) - `mosaic telemetry` CLI (`packages/mosaic/src/commands/telemetry.ts`): local half inspects an OTEL collector; remote half is consent management (`telemetry/consent-store.ts`, `$MOSAIC_HOME/telemetry.json`, `remoteEnabled` defaults **false**) plus a dry-run-only `upload`. - The client is a forward-compat shim (`telemetry/client-shim.ts`): `@mosaicstack/telemetry-client-js` is unpublished, and there is no live server endpoint. Upload prints instead of POSTing. - Gateway has OTEL tracing (`apps/gateway/src/tracing.ts`). That is metrics/ traces plumbing, not defect capture — a different pipeline with a different consumer. So "partially implemented" means: consent and CLI surface exist; capture taxonomy, scrubbing, transport, server, and issue filing do not. ### 2. Scope split — this issue is the LAST mile, not the whole pipeline The contract in the issue body has four clauses. Clause 1 (feeds issues), 2 (fail open), and 4 (dedup) live server-side or at the filing boundary. Clause 3 (scrub) lives client-side. Proposal: treat 1399 as **two deliverables with a hard interface between them**, so neither blocks the other: - **D-A: gotcha event schema + client capture + scrub** (in `packages/mosaic`, extends the existing shim/consent machinery). - **D-B: filer** — the component that turns a scrubbed event into a Gitea issue (create or dedup-link). Runs wherever the operator points it; does NOT require the unpublished telemetry-client package or a new server to exist first. ### 3. Event schema (the interface) One JSON object per gotcha, extending the shim's `TelemetryEvent`: ``` { name: "gotcha", properties: { component: "gateway|db|cli|git-tools|...", // from a closed enum, not free text symptom: "short stable slug, e.g. accounts-issuer-missing", severity: "low|medium|high", summary: "one paragraph, pre-scrubbed prose", evidence: ["scrubbed command + output excerpts"], signature: "<component>/<symptom>", // dedup key, clause 4 schema_version: 1 } } ``` Dedup is keyed on `signature` (component + symptom slug), never on message text (clause 4). The slug is author-chosen but validated against a format (`[a-z0-9-]{3,64}`); twelve hosts hitting the same defect converge because they name the same symptom, and near-misses are the triager's merge decision, not a string-distance heuristic. Deliberately no fuzzy matching in v1 — false-positive dedup silently hides a distinct defect, which is worse than a duplicate issue. ### 4. Scrub before egress (clause 3) — client-side, allowlist-first Runs in `captureEvent` for `name=gotcha`, BEFORE the event ever reaches the queue, so nothing unscrubbed is even buffered to disk: 1. Redact URL credentials (`://user:pass@` → `://REDACTED@`). 2. Redact home paths (`/home/<u>`, `/var/home/<u>`, `C:\Users\<u>` → `$HOME`), and `$MOSAIC_HOME` values. 3. Redact anything matching known secret shapes (bearer/token headers, `AUTHORIZATION:`, 40+ char hex/base64 runs) — pattern list shared with the framework's existing redaction conventions. 4. Env values never included by design: the schema has no field for them, and the capture API takes only the schema fields (structural prevention beats filtering). 5. A `--show` flag on capture prints exactly what would egress, so an operator can audit the scrubber output before opting in. Scrubber failures fail CLOSED for upload (event held locally, marked unscrubbed) — clause 2's fail-open applies to the *filing path*, not to the scrub gate. ### 5. Filing + dedup (clauses 1 and 4) — filer as a small, hosted-anywhere worker The filer consumes scrubbed events and talks to the Gitea API: - Look up open issues labeled `telemetry-gotcha` whose title starts with the signature. Hit → post a comment `seen again: <host-class> <date>` and bump a count in the issue body's marker block. Miss → create the issue with title `[<signature>] <summary first line>`, label `telemetry-gotcha` + severity, body = summary + evidence + a machine marker block (`<!-- gotcha:signature=... count=... -->`). - Idempotency: the marker block is the source of truth for count; the filer re-reads it before writing (read-modify-write with retry on conflict). - Auth: a dedicated filer identity/token (deploy identity, not a seat token — same principle as the estate's Q22 registry-token ruling), scoped to issues on the target repo. - v1 transport between capture and filer: **none required.** `mosaic telemetry file` (new subcommand) runs the filer locally over the local queue when the operator opts in — this makes the pipeline useful before any server exists, and the server later becomes just another place the same filer runs. ### 6. Fail open while partial (clause 2) - Every capture appends to a local NDJSON spool under `$MOSAIC_HOME` regardless of filer/network state; nothing is dropped silently. - If filing fails (network, auth, Gitea down), the CLI exits nonzero and prints the manual-filing fallback text with the scrubbed event inline — the documented fallback IS the output, not a doc reference. - `mosaic telemetry status` grows a line: spooled-unfiled count, so a stranded backlog is visible. ### 7. Consent boundary Existing consent semantics are preserved: `remoteEnabled=false` means nothing egresses — capture and spool are local-only, `file` refuses with the opt-in hint. Filing to the operator's OWN Gitea is still egress off-host and sits behind the same consent flag. No third state added in v1. ### 8. Phasing 1. **PR-1 (packages/mosaic):** schema + `mosaic telemetry capture-gotcha` + scrubber + spool + tests (scrubber gets adversarial fixtures: URL creds, home paths, token shapes; red/green per case). 2. **PR-2 (packages/mosaic or tools/):** filer + `mosaic telemetry file` + dedup marker protocol + tests against a mock Gitea API. 3. **Later, out of 1399's scope:** publish telemetry-client, server-side filer, wiring OTEL events into the same schema. 1398's memory/tier work is unrelated and stays separate. ### 9. Open questions for the maintainers - Q-A: label taxonomy — is `telemetry-gotcha` acceptable, or fold into an existing triage label set? - Q-B: which repo do auto-filed issues land in when the component maps to a package inside the monorepo vs a separate repo (fleet-comms)? v1 proposal: everything to mosaicstack/stack, triager re-routes. - Q-C: does the filer identity get provisioned per-site or one shared identity upstream? (USC would mint its own per Q22 practice.) If the shape is acceptable I will commission PR-1 per the standing delivery bar (branch from next, independent review, coordinator merge). Silence for a reasonable window is not consent for code — I will wait for an explicit go from the repo side or from orch-01 as coordinator.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1399