TechResearch/Headroom_Token_Compression/headroom-compression-guide

Headroom Compression Guide

Context Compression Levels, Semantic Risk, and Enterprise Use

What This Document Covers

How much compression is safe — and where meaning starts to break
Two real scenarios: chat session history and RAG memory
What the HuggingFace (Kompress) model does and when it's worth using
Enterprise deployment options — with and without the HF model
Compliance and data residency
How we actually tested Headroom in this project

1. Headroom's Three Compressors and Their Safety Profiles

Headroom routes each content block to one of three compressors depending on what the content is. They carry very different semantic risk profiles.

SmartCrusher — JSON arrays and objects

What it does: Detects homogeneous arrays (same schema repeated across records), extracts a shared header, and writes each record as a compact diff against that header. Structural deduplication, entirely deterministic, fully reversible.

Typical reduction: 10–40% on repeated-schema JSON (user records, search results, API responses).

Semantic risk: Near zero. No information is removed. The compressed and original forms are semantically equivalent — only the representation changes. This is the safest compressor to run in any context.

Example from our experiment:

10 user records × 12 fields each → shared header + compact per-record diffs
343 tokens removed, 9.2% prompt token reduction, answer quality unchanged

Fail condition: Arrays with heterogeneous schemas (mixed shapes) — SmartCrusher falls back to passthrough rather than corrupting the structure.

CodeCompressor — Source code

What it does: Parses source into an AST (Python, JS, Go, Rust, Java, C++), then serializes only the structurally relevant parts: function signatures, class definitions, import statements, docstrings (optional). Body implementations are replaced with a single-line stub or removed.

Typical reduction: 15–50% on dense source files. Less on already-compact code.

Semantic risk: Low to moderate, context-dependent.

Usage	Risk
"What does this function signature accept?"	Very low — signatures preserved
"What does this function do step by step?"	High — body may be stubbed
"Find the bug in this loop"	High — implementation detail is the point

The protection that matters: Headroom's protect_analysis_context setting detects when the user message contains words like "analyze", "review", "debug", "explain this code" and skips CodeCompressor for that turn. This is the right behavior — when the model needs to read code carefully, the compressed version is the wrong input.

Fail condition: Requested code review/debugging where the implementation is the question. Headroom's analysis-intent detection reduces this risk but does not eliminate it.

Kompress (kompress-v2-base) — Prose text

What it does: A small ML model trained on agentic conversation traces. It performs extractive compression: scores each sentence for relevance, keeps the top-k sentences at the configured target_ratio, discards the rest. The result is still natural language but shorter.

Typical reduction: 40–85% depending on target_ratio.

`target_ratio`	% of text kept	Use case
`0.7`	70%	Conservative — documentation, specs, policies
`0.4`	40%	Moderate — general prose chunks, summaries
`0.2`	20%	Aggressive — logs, verbose tool output narratives

Semantic risk: This is where real meaning loss can occur.

At 0.7 the model reads something close to the original — the main risk is losing supporting sentences that make a conclusion credible. The answer may be correct but cite fewer sources.

At 0.4 you can lose minority cases, edge conditions, and qualifications. "Usually X, except when Y" may become "X" if Y-sentences scored low.

At 0.2 you're reading a headline summary. Fine for routing decisions, dangerous for detailed Q&A. Numerical specifics, proper nouns, and conditional logic are the first casualties.

Fail condition: Any question where the answer is a specific detail buried in prose — a date, a number, a named exception. Extractive compression cannot guarantee the survival of any specific sentence.

2. Chat Session — Where Compression Is Safe and Where It Isn't

A chat session builds up history turn by turn. By turn 20, you may have 10,000+ tokens of context the model drags forward on every call.

Turn 1:   System prompt + user question 1 + answer 1         ~500 tokens
Turn 5:   All of the above + turns 2-5                      ~2,500 tokens
Turn 20:  All of the above + turns 6-20                    ~12,000 tokens
Turn 50:  All history                                       ~30,000 tokens  ← context pressure begins

Safe compression zone

Early conversation history (turns 1–N-4 for a 4-message protection window) is the safe zone. The model still has access to the original via CCR if it needs to retrieve it. Key properties:

Turn-level decisions: "What did we agree on?" — safe to compress because the conclusion outlives the argument.
Tool outputs from early turns: A grep result from turn 5 is extremely compressible by turn 20. If the model needs it again, it will re-run the tool or call headroom_retrieve.
Boilerplate system message repetition: SmartCrusher can deduplicate JSON-heavy tool schemas in the system prompt.

Dangerous compression zone

The last 2–4 messages (Headroom's protect_recent default). These are the active working context. A partially-compressed instruction the user just gave is the single most common failure mode.
Messages containing commitments: "In the next step, do X, then Y, then Z" — extractive compression may drop Y.
Numerical state: "The retry count is currently 3" or "The threshold was set to 42" — these are facts, not background, and they do not survive aggressive Kompress ratios reliably.

Practical settings for chat

compress(
    messages,
    compress_user_messages=False,  # never compress live user turns
    protect_recent=4,              # keep last 4 turns intact
    target_ratio=0.5,              # moderate on older history
    kompress_model="disabled",     # if you want zero semantic risk: rule-based only
)

3. RAG Memory — Where Compression Earns Its Keep

RAG is where Headroom's payoff is clearest. A retriever doesn't know which of the 10 returned chunks actually answers the question — it returns all 10 to be safe. 80% of that context is noise.

The typical RAG bloat pattern

User query: "Which engineers have deploy permission?"

Retriever returns:
  Chunk 1: user records JSON (10 records, 12 fields each)     ~800 tokens
  Chunk 2: code search across 8 files                        ~600 tokens
  Chunk 3: architecture doc paragraph                        ~350 tokens
  Chunk 4: onboarding guide paragraph                        ~350 tokens
  Chunk 5–10: marginally relevant chunks                   ~1,800 tokens

Total context delivered to LLM:                            ~3,900 tokens
Token actually needed to answer:                             ~120 tokens
Waste ratio: ~97%

What Headroom does to this

SmartCrusher on the user records JSON removes the repeated schema — 10 records become 10 diffs against a shared header. 40% reduction on that chunk with zero semantic loss.

CodeCompressor on the code search removes function bodies and keeps only signatures and grep context lines. The model can see that compress() exists and what it accepts without reading every implementation.

Kompress on the prose chunks extracts the sentences most likely to answer a question and drops the rest. At target_ratio=0.5 a 350-token paragraph becomes ~175 tokens.

Combined: a 3,900-token RAG payload compresses to ~1,800–2,200 tokens depending on content mix.

When RAG compression breaks the answer

The failure mode is specificity: the question asks for something that appears once as a supporting detail, not as a main point.

"What is the error_protection_max_chars threshold?" — this is a single number in a doc that likely scored low in sentence importance. Kompress at 0.4 will probably drop it. The answer the model gives will be a hallucination or a "I don't have that information" — which is actually the better outcome.

Mitigation: Use the CCR retrieval tool. Headroom caches originals locally with a content_id. The model can call headroom_retrieve(content_id) when it detects it needs more detail. The retrievability is the safety net that justifies lossy compression.

Practical settings for RAG

compress(
    messages,
    compress_user_messages=True,   # tool results are in user messages
    protect_recent=0,              # compress all turns — it's all context
    target_ratio=0.5,              # moderate — balanced safety
    kompress_model="disabled",     # or set to HF model for prose-heavy retrieval
)

4. The HuggingFace Model — What It Does and When It's Worth It

What it is

chopratejas/kompress-v2-base is a small transformer model hosted on HuggingFace and downloaded once (~180 MB) to local disk. It runs via ONNX runtime — no GPU required, CPU-only inference. After the initial pull, it never calls home.

What it adds that rule-based compressors can't

SmartCrusher and CodeCompressor are deterministic — they operate on structure. They cannot compress prose because prose has no formal grammar to parse. Kompress is the only compressor in Headroom's pipeline that understands language well enough to reduce natural text.

Without Kompress:

JSON tool outputs → compressed (SmartCrusher)
Source code → compressed (CodeCompressor)
Prose documentation, log narratives, chat history, RAG chunks → unchanged

In a typical agentic workload where 40–60% of context is prose, disabling Kompress means you get half the compression potential.

The CPU overhead tradeoff

From our experiment: enabling Kompress added ~35% inference time overhead on CPU (compressor inference before the prompt even reaches the LLM). Disabling it gave 4.4% faster end-to-end time with 9% token reduction.

The crossover point is prompt length. The longer the prompt, the more LLM inference time the token savings buy back. At 3,500 tokens on a 7B model the savings didn't pay for the compressor overhead. At 15,000+ tokens on a slower machine or a 14B model, the math flips.

Prompt size	Recommendation
< 5,000 tokens	`kompress_model="disabled"` — overhead exceeds savings
5,000–15,000 tokens	Test both; depends on hardware
> 15,000 tokens	Enable Kompress — context pressure savings dominate
Context window near limit	Enable Kompress regardless — the alternative is truncation

5. Enterprise Deployment — With and Without the HF Model

Mode A: Rule-based only (`kompress_model="disabled"`)

Stack: Ollama (local LLM) + Headroom SmartCrusher + CodeCompressor only. No HuggingFace dependency.

What you get:

Deterministic, auditable compression — output is reproducible given the same input
No model download, no ONNX runtime overhead
10–40% token reduction on JSON and code-heavy payloads
Full transparency — you can inspect exactly what was removed and why

What you give up:

No prose compression — documentation chunks, chat history, and log narratives pass through at full size

When to choose this:

Regulated environments where every transformation touching content must be explainable and deterministic
Air-gapped deployments where external model downloads are blocked
Workloads that are predominantly structured data (JSON APIs, code repositories)
When you want to introduce compression incrementally and start with zero semantic risk

Enterprise use cases:

Internal API platforms with heavy JSON tool output (user records, config payloads, search results)
Code review agents that read many files — CodeCompressor reduces source bloat without touching logic
Financial data pipelines with repeated-schema JSON from market data feeds

Mode B: With Kompress (`kompress_model=None` or default)

Stack: Ollama + Headroom full pipeline including kompress-v2-base via ONNX (local).

What you get:

Full compression including prose — 40–85% on text-heavy content
Significant context window relief for long RAG workloads and extended chat sessions
Still fully local — the HF model runs on your hardware after one-time download

What you give up:

Non-deterministic output (ML models are stochastic; same input, slightly different compression at each run)
CPU overhead for ONNX inference
One-time model download (can be pre-provisioned offline)

When to choose this:

Long RAG sessions with prose-heavy documentation retrieval
Customer support agents with long conversation histories
Any workload where context window is the binding constraint, not latency

Enterprise use cases:

Legal document review agents (dense policy text compressed before LLM reads it)
HR/compliance knowledge bases (procedure manuals, policy docs)
Customer success agents with long multi-turn session histories

6. Is the HuggingFace Model Compliant?

License

kompress-v2-base is released under Apache 2.0. This is a permissive open-source license that allows commercial use, modification, and distribution without requiring source disclosure. It is compatible with most enterprise open-source policies.

Data residency

After the initial download, the model runs entirely on local hardware via ONNX. No content is sent to HuggingFace or any external endpoint during inference. The model is a static artifact that processes data in-process.

For air-gapped environments:

# Pre-download the model before going offline
huggingface-cli download chopratejas/kompress-v2-base
# Then lock the cache
export HF_HUB_OFFLINE=1

Concern	Status
Data leaves the network boundary	No — inference is local
Model trained on customer data	No — trained on public agentic traces
PII processed by external system	No — ONNX runs in-process
Audit trail of what was compressed	Partial — `transforms_applied` field in CompressResult; full originals in CCR local cache
Output determinism for audit	No — ML output varies slightly across runs

The gap: non-determinism. If your compliance requirement is that the same input must always produce the same compressed output (for audit trail reproducibility), Kompress is the wrong tool. Use kompress_model="disabled" and document that compression is structural-only.

The strength: the CCR (Content Compression Repository) cache stores originals locally with a TTL. For regulated workloads, this cache is itself a compliance asset — it is the record of what the LLM actually received vs. what was in the original document.

A note on enterprise TLS/SSL environments

Headroom fetches ONNX runtime and the Kompress model over HTTPS on first use. Corporate networks with SSL inspection (man-in-the-middle certificates) will break this unless the custom CA is trusted in the Python environment. Headroom documents a workaround via REQUESTS_CA_BUNDLE. Pre-provisioning the model artifacts before deployment avoids this entirely.

7. How We Tested Headroom in This Project

What we built

A LangGraph pipeline with two compiled graphs:

baseline_graph:   retrieve → generate
compressed_graph: retrieve → compress → generate

Both graphs are identical except for the compress node, which applies headroom.compress() to the RAG messages before they reach ChatOllama (Qwen 2.5:7b running locally via Ollama).

The payload in data/sample_rag_chunks.py was designed to exercise all three compressors:

A 10-record user JSON array (SmartCrusher target)
A grep + source code excerpt (CodeCompressor target)
Two prose documentation chunks (Kompress target)

What the experiments revealed

Run 1 — First attempt (wrong message format)

We initially built messages as plain user text (one big string). Headroom returned router:protected:user_message. Discovery: headroom protects plain user messages by default — it's designed for tool result content, not raw user input.

Run 2 — Added compress_user_messages=True (still wrong format)

Switching the flag didn't help — headroom returned router:noop. Reading the source revealed that headroom only processes tool_result blocks when block["content"] is a string, not a list of text blocks. Our "content": [{"type": "text", "text": "..."}] format fell to the small-content skip path.

Run 3 — Correct format: string content in tool_result + compress_user_messages=True + protect_recent=0

Compression fired. With the Kompress ML model enabled, results were:

25.6% fewer prompt tokens (3,496 → 2,600)
1,029 tokens saved
But +35% inference time overhead — ONNX on CPU

Run 4 — Disabled Kompress (kompress_model="disabled") ← current state

Baseline:     3,496 prompt tokens  |  11.57s inference
Compressed:   3,174 prompt tokens  |  11.06s inference
Reduction:    -9.2% tokens         |  -4.4% time
Transforms:   router:tool_result:smart_crusher

Only SmartCrusher ran (the JSON records). Code and prose passed through unmodified. Clean, transparent, no hidden model. The 9% token reduction and 4.4% speed gain came entirely from deterministic JSON restructuring.

Key findings

Message format matters more than you'd expect. Headroom's routing logic inspects content type down to individual block-level field types. String vs. list content in tool_result is the difference between compression and passthrough.
protect_recent defaults protect too aggressively for RAG. The default of 4 protects all messages in a short 2-message conversation. You must explicitly set protect_recent=0 for RAG payloads that are entirely context, not active dialogue.
The ML compressor CPU cost is real. At 3,500 tokens on a 7B model, Kompress overhead exceeds inference savings. The crossover is somewhere between 5,000–15,000 tokens depending on hardware.
SmartCrusher is the safest entry point. Zero semantic risk, deterministic output, no extra dependencies, immediate token reduction on any JSON-heavy payload. Start here before enabling the ML layer.

Files in this project

headroom/
  config.py                 model selection and query
  graph.py                  LangGraph state, nodes, and graph builders
  baseline.py               run without compression
  with_headroom.py          run with compress node
  compare.py                run both, print diff table
  data/
    sample_rag_chunks.py    tool_result messages (JSON + code + prose)
  results.json              last run output
  README.md                 setup and run instructions

Summary

Decision	Rule-based only	With Kompress
Semantic risk	Near zero	Low–moderate (tunable)
Token reduction	10–40%	40–85%
CPU overhead	None	Significant on CPU
Deterministic output	Yes	No
External model download	No	One-time (then local)
Apache 2.0 license	Yes	Yes
GDPR / air-gap safe	Yes	Yes (after pre-provision)
Best for	JSON/code workloads, regulated envs	Prose-heavy RAG, long sessions

The practical default for most local-first enterprise deployments: start with kompress_model="disabled", measure actual token savings on your real payloads, and add Kompress only when context window pressure genuinely requires it.