mosaicstack/stack
1
0
Fork 0
Code Issues 42 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Spike: Evaluate headroomlabs-ai/headroom (LLM context compressor, Apache-2.0) for Mosaic — gated, lossy-risk-aware #693

New Issue
Open
opened 2026-06-26 02:33:33 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-06-26 02:33:33 +00:00
Owner
Copy Link

Spike: Evaluate headroomlabs-ai/headroom (LLM context compressor, Apache-2.0) for Mosaic — gated, lossy-risk-aware

Parked under: 0.0.50 onboarding epic (HELD until 0.0.49 cuts). This is a research spike, not a release-path change. main is FROZEN — no source/release changes land from this work until jetrich blesses HEAD and 0.0.49 cuts.

Goal

Evaluate whether Headroom (LLM context compressor) can reduce agentic-context token cost in Mosaic without lossy degradation of downstream agent fidelity. Two-axis acceptance, per content-type.

Acceptance model (two-axis, per content-type)

A compression arm is accepted for a content-type only if BOTH hold:

  1. tokens_saved% clears jetrich's bar (TBD — awaiting acceptance-bar decision), AND
  2. fidelity == baseline — the downstream fidelity probe answer is unchanged vs raw context.

Content-type buckets (separate accept/reject each):

  • docs / RAG / prose
  • CI logs (woodpecker step logs)
  • code diffs (high-risk bucket — exact changed file+line fidelity)
  • JSON / structured tool output
  • error / stack traces

Method

Autoresearch-style modify → measure → keep/discard loop. NOTE: autoresearch is nanochat/GPU-specific and ships NO license — borrow the pattern only, do not fork or vendor it.

Phases

  • Phase 0 (SAFE — in progress): assemble READ-ONLY scrubbed corpus of real Mosaic agentic context, bucketed by content-type; build baseline harness (token-count per sample using our agents' tokenizer family + a fidelity-probe scaffold with baseline answers from RAW context). No headroom install. No release-path/main touch. Temp/isolated only.
  • Phase 1 (HELD): install headroom, run compression arms, score two-axis per content-type. Blocked on jetrich's acceptance-bar decision.

Licensing / safety notes

  • Headroom is Apache-2.0 (compatible). Confirm before any vendor/install.
  • No tokens/creds in the corpus — scrub all samples.
  • Diffs bucket is highest-risk: lossy compression that drops a changed line = silent fidelity loss.

Status

Phase 0 dispatched. Phase 1 HELD pending jetrich acceptance-bar.

## Spike: Evaluate `headroomlabs-ai/headroom` (LLM context compressor, Apache-2.0) for Mosaic — gated, lossy-risk-aware **Parked under:** 0.0.50 onboarding epic (HELD until 0.0.49 cuts). This is a **research spike**, not a release-path change. **main is FROZEN** — no source/release changes land from this work until jetrich blesses HEAD and 0.0.49 cuts. ### Goal Evaluate whether Headroom (LLM context compressor) can reduce agentic-context token cost in Mosaic **without** lossy degradation of downstream agent fidelity. Two-axis acceptance, per content-type. ### Acceptance model (two-axis, per content-type) A compression arm is **accepted** for a content-type only if BOTH hold: 1. `tokens_saved% ` clears jetrich's bar (TBD — awaiting acceptance-bar decision), AND 2. `fidelity == baseline` — the downstream fidelity probe answer is unchanged vs raw context. Content-type buckets (separate accept/reject each): - docs / RAG / prose - CI logs (woodpecker step logs) - code diffs (**high-risk** bucket — exact changed file+line fidelity) - JSON / structured tool output - error / stack traces ### Method Autoresearch-style **modify → measure → keep/discard** loop. NOTE: autoresearch is nanochat/GPU-specific and ships **NO license** — borrow the *pattern* only, do **not** fork or vendor it. ### Phases - **Phase 0 (SAFE — in progress):** assemble READ-ONLY scrubbed corpus of real Mosaic agentic context, bucketed by content-type; build baseline harness (token-count per sample using our agents' tokenizer family + a fidelity-probe scaffold with baseline answers from RAW context). **No headroom install. No release-path/main touch. Temp/isolated only.** - **Phase 1 (HELD):** install headroom, run compression arms, score two-axis per content-type. Blocked on jetrich's acceptance-bar decision. ### Licensing / safety notes - Headroom is Apache-2.0 (compatible). Confirm before any vendor/install. - No tokens/creds in the corpus — scrub all samples. - Diffs bucket is highest-risk: lossy compression that drops a changed line = silent fidelity loss. ### Status Phase 0 dispatched. Phase 1 HELD pending jetrich acceptance-bar.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#693
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?