Raw/next_stage_in_agent_stack_series

Gap Analysis: Headroom's Non-Shell Coverage

Companion to: research-rtk.md · research-non-shell-token-compression.md Triggered by: empirical case study "60–95% vs. 9.2%"

1. 🧭 The Short Answer

No, Headroom does not cover all non-shell token scenarios. But the gap map has two separate dimensions that you must not conflate:

Dimension A — Content-type coverage gaps: scenarios where Headroom's compressors don't reach at all. Dimension B — Compliance-grade gaps: scenarios where Headroom reaches and compresses, but the output is architecturally inappropriate for a regulated workload.

Your case study revealed Dimension B precisely. The 9.2% vs. 25.6% split is not a performance gap — it is the boundary between what you can ship to a regulated client and what you cannot. That framing is more valuable than the raw numbers.

Applied to your stack:

Compressor	Content covered	Deterministic?	Audit-safe for Compliance AI?
SmartCrusher	Repeated-schema JSON	✅ Yes	✅ Yes
CodeCompressor	Source code (AST)	✅ Yes	✅ Conditionally
Kompress	Prose, history	⚠️ No guarantee	❌ No
CacheAligner	Prefix stabilization	✅ Yes	✅ Yes (orthogonal)

When you apply both filters — coverage and compliance-grade — the actual safe surface is smaller than it appears. The six gaps below fall on both dimensions.

2. 🔍 The Six Specific Gaps

Gap 1: Pre-Retrieval Relevance — Headroom's Hardest Boundary

What Headroom does: Compresses whatever arrives as message content.

What it cannot do: Decide which chunks should have arrived at all.

This is the most important architectural gap because it sits upstream of everything else. Headroom operates on the content layer — after retrieval. A reranker operates at the retrieval boundary — before content assembly. These are sequential responsibilities, not alternatives.

Your test payload was 10 user records × 12 fields = 3,496 tokens. SmartCrusher compressed that to 3,174 (−9.2%). But if a reranker had filtered your top-20 retrieved records to top-3 before they reached Headroom, you'd have started with ~1,050 tokens and SmartCrusher would work on a much smaller input.

The math that matters:

pgvector top-20 chunks → no rerank → Headroom SmartCrusher → 9.2% reduction
pgvector top-20 chunks → Cohere Rerank top-3 → Headroom SmartCrusher → ~85% reduction

The reranker's token elimination is categorical, not proportional. It removes whole chunks. Compression reduces what's left of each chunk. Reranking is not optional upstream of compression for RAG workloads.

What you need: Cohere Rerank is already in your stated stack. Confirm you're cutting aggressively to top-3 or top-5 post-rerank before the content reaches Headroom. FlashRank as a local fallback when Cohere latency is a constraint.

Gap 2: Heterogeneous JSON — SmartCrusher's Architectural Assumption

What SmartCrusher handles well: JSON arrays with a repeated schema — the 10-user-record case in your experiment. It extracts a shared header and emits compact per-record diffs. Deterministic, rule-based, near-zero semantic risk.

What it handles poorly: Deeply nested, heterogeneous JSON objects. AWS CloudFormation stack events, Kubernetes pod specs, FastAPI response envelopes with mixed nesting, pgvector result objects with metadata fields.

SmartCrusher's compression model requires structural regularity to find the shared header. A one-off deeply nested JSON response with mixed types — which is what AWS CLI, Kubernetes, and most REST APIs actually return outside of collection endpoints — doesn't offer that regularity. The router can classify it correctly, dispatch to SmartCrusher, and produce minimal savings because the structure doesn't compress cleanly.

The specific failure mode: SmartCrusher sends the full nested object through after finding no repeating structure. No warning, no error, the transforms_applied field shows SmartCrusher ran, but savings are <5%. You conclude it worked; it didn't compress.

What fills this gap: JSONPath projection / schema filtering as a pre-Headroom step. Before a response reaches Headroom, strip to only the keys the agent actually needs. For FastAPI tool results in Localz, this belongs in your tool result formatter — not in Headroom.

# In your tool result formatter, before returning to the agent:
import jq  # or use pydantic field selection

def format_business_result(raw_response: dict) -> dict:
    # Project to agent-relevant fields only
    return {
        "id": raw_response["id"],
        "name": raw_response["business_name"],
        "status": raw_response["verification_status"],
        # Drop: audit_trail, internal_metadata, payment_history, ...
    }
    # Then Headroom compresses what's left

For AWS CLI / Kubernetes, rtk aws and rtk kubectl filters in RTK already handle this at the shell boundary. The gap is specifically JSON that arrives as an in-process tool result.

Gap 3: Kompress on Regulatory Documents — A Systematic Bias, Not Just Non-Determinism

This is the critical gap for Compliance AI. It's more specific than the "non-determinism" label suggests, and the failure mode is worse.

Kompress uses extractive sentence scoring. It reads a document and scores each sentence by estimated information density, dropping the lowest-scoring sentences to hit the target ratio. That scoring model was trained on general prose. It has a systematic bias: it treats short, common-word sentences as low-signal and long, rare-word sentences as high-signal.

Regulatory documents have the opposite information distribution. The legally material content is often in qualifying clauses — short sentences containing "however," "except where," "unless," "notwithstanding," "subject to." These look like low-information sentences to an extractive scorer. They use common words, they're short, they don't carry domain-specific terminology. They are also the sentences where regulatory obligation hinges.

Concrete example:

OSFI B-10 Guideline, paraphrased:

"Federally regulated financial institutions must maintain audit records
of all third-party service provider relationships. [HIGH SCORE — long,
technical, domain-specific]

However, internal audit functions conducted under direct OSFI oversight
are exempt from this requirement. [LOW SCORE — short, common words,
gets dropped]"

Kompress producing 25.6% savings on that document and answering your test query correctly tells you nothing about whether it preserved that "however" clause. Your test query didn't require the qualifying clause. A real compliance agent query might.

This is not a theoretical concern. It's a systematic structural mismatch between Kompress's signal model and the information structure of regulatory text.

What fills this gap:

For Compliance AI: Do not use Kompress on regulatory source documents. SmartCrusher only (deterministic) plus reranking upstream.
For long regulatory documents where you need prose compression: use LongLLMLingua in question-aware mode — it scores token importance relative to the specific query, not globally. A query about OSFI audit requirements would weight the "however, exempt" clause correctly because the query terms would lift the relevance of the exception clause.
The framing from your case study still holds: Kompress = opt-in escalation when context pressure forces the trade and auditability isn't load-bearing. For your Localz chat history, this trade is fine. For Compliance AI source documents, it is not.

Gap 4: Cross-Session Persistent Memory — In-Session ≠ Cross-Session

What Headroom covers: Cross-agent memory within a session — deduplication across Claude, Codex, and Gemini calls in the same running session. RollingWindow compaction for accumulating history within a session.

What it doesn't cover: Persistent memory across sessions — "what did the agent learn or decide in past sessions?" This is a categorically different problem.

Headroom's cross-agent memory is a deduplication and compression store with a configurable TTL. It's designed for in-session context pressure, not for long-horizon procedural memory. When the session ends, the memory store expires.

For your Hermes ambient agent layer, this is the gap: Hermes needs to remember how to do things across restarts, VPS reboots, and sessions. That's procedural memory, not in-session compression.

What fills this gap: This is already in your architecture — Obsidian vault (declarative) + Hermes procedural memory layer. The memory boundary rule you've established (vault = what you know, Hermes = how to do things) is correct. Headroom's cross-agent memory doesn't replace either side of that boundary; it handles a third, narrower concern: in-session redundancy elimination.

For Compliance AI specifically, Mem0 is worth evaluating for agent memory of regulatory decisions across sessions (which regulations were determined applicable, which exemptions were found). Mem0 converts past agent turns into structured facts in a vector graph — it's a better fit than Headroom for cross-session compliance state.

Gap 5: LangGraph Checkpoint State — Never Hits the API

What Headroom covers: Everything that crosses the LLM API boundary (proxy mode) or is explicitly passed through the library (compress(messages)).

What it doesn't cover: LangGraph checkpoint state that lives between nodes but is never sent to the LLM directly. When your LangGraph compliance flow checkpoints intermediate state between the scraper node, the classification node, and the audit-trace node, that state object can be large — accumulated document content, intermediate classifications, citation references, retrieval metadata.

The Headroom proxy intercepts API calls. It doesn't intercept LangGraph's checkpoint writes to Postgres/SQLite/Redis. This state never goes through the proxy unless a node explicitly reads it from the checkpoint and re-inserts it into the next LLM call.

The compliance-specific version of this problem: Under your audit-grade traceability requirement, you want that checkpoint state to be: (a) compact enough for Postgres to store efficiently over long audit windows, (b) structured enough for retrieval by request ID, and (c) not dependent on lossy ML compression for its accuracy.

What fills this gap:

LangGraph's own state schema design — the primary lever. A well-designed state object that carries only what the next node needs, not accumulated document content. Documents stay in pgvector/R2; only their IDs and citation references live in checkpoint state.
Custom checkpoint reducers — LangGraph supports custom reducer functions that control how state accumulates across nodes. A reducer that compresses accumulated message lists on each checkpoint write is the right pattern.
This is not a Headroom integration gap to fix — it's a LangGraph schema design decision that you own.

Gap 6: The CCR Audit Pipeline — Data Exists, Wire-Up Doesn't

This is the gap you identified in your case study and it deserves precise scoping.

What currently exists:

transforms_applied is populated in every result — ["router:tool_result:smart_crusher"] for rule-based, ["router:tool_result:smart_crusher", "mixed", "text"] for Kompress
CCR cache stores originals with configurable TTL
headroom_retrieve is designed to pull the original from cache

What's missing:

transforms_applied lives in a local metrics dict, not surfaced as a retrievable artifact tied to a specific request ID
CCR retrieval is future work in the current repo state
No integration between the compression log and Langfuse traces

The gap is one integration layer. This is not a missing tool — it's a missing pipeline:

Current:
API call → Headroom compresses → LLM → response
                ↓
         [local metrics dict: transforms_applied]

Target (audit-grade):
API call → Headroom compresses → LLM → response
                ↓                              ↓
         Langfuse trace span            Langfuse trace span
         + transforms_applied           + request_id
         + CCR handle                   + output
         + original_tokens
         + compressed_tokens

Concrete implementation path:

# Headroom proxy mode + Langfuse instrumentation wrapper

from headroom import compress, get_last_compression_meta
from langfuse import Langfuse

langfuse = Langfuse()

def compress_with_audit_trace(messages: list, request_id: str) -> list:
    compressed = compress(messages)
    meta = get_last_compression_meta()  # transforms_applied, token counts, CCR handles

    # Write to Langfuse as a span on the current trace
    langfuse.trace(
        id=request_id,
        metadata={
            "compression.transforms": meta["transforms_applied"],
            "compression.tokens_in": meta["original_tokens"],
            "compression.tokens_out": meta["compressed_tokens"],
            "compression.ccr_handles": meta["ccr_handles"],  # retrieval keys
        }
    )
    return compressed

Once this wrapper exists, every production inference call has a Langfuse trace that carries the full provenance chain: what was compressed, how, by which algorithm, and where the original is stored. CCR handles in the trace let an auditor call headroom_retrieve(handle) to verify exactly what the agent read.

This closes the loop from "Headroom as cost optimization" to "Headroom as cost optimization + reproducibility mechanism" — the exact framing from your case study.

3. 📊 Full Coverage Map

Non-Shell Scenario	Headroom Covers?	Audit-Safe?	Gap	Fill With
Repeated-schema JSON (collections)	✅ SmartCrusher	✅ Yes	—	—
Heterogeneous nested JSON	⚠️ Partial	✅ If compressed	Schema projection before Headroom	JSONPath / Pydantic field selection
Source code in tool results	✅ CodeCompressor	✅ Conditionally	protect_analysis_context config	—
RAG chunk verbosity	✅ SmartCrusher/Kompress	⚠️ Kompress only for non-compliance	Relevance filtering upstream	Cohere Rerank → FlashRank
Regulatory prose documents	⚠️ Kompress	❌ Systematic clause bias	Query-aware compression	LongLLMLingua (question-aware mode)
Chat history accumulation	✅ RollingWindow	⚠️ Via Kompress	ML compaction not deterministic	SmartCrusher-equivalent rolling summary
Cross-session agent memory	❌ Not covered	N/A	Separate concern	Mem0 or Hermes procedural memory
LangGraph checkpoint state	❌ Not covered	N/A	Never hits API proxy	LangGraph state schema + reducers
Tool schema definitions	❌ Not covered	N/A	Sent on every call, static overhead	Tool registry + lazy schema injection
CCR audit integration	⚠️ Data exists	❌ Not wired	Pipeline gap	Langfuse wrapper (one layer)
Prefix KV-cache optimization	✅ CacheAligner	✅ Yes	Cloud-only benefit	Already covered

4. 🎯 Priority Order for Your Stack

Do now (zero new tools, configuration only):

Confirm Cohere Rerank is cutting to top-3 post-retrieval in Localz, not top-10. This is your highest-leverage token reduction, upstream of everything else.
Set protect_recent=0 and force string content in tool_result blocks. Your case study proved these are silent no-ops otherwise.
Keep Kompress disabled for Compliance AI document compression. SmartCrusher only on the compliance path.

Next sprint (one integration layer each): 4. Add JSONPath schema projection in Localz tool result formatters for heterogeneous API responses before Headroom receives them. 5. Build the Langfuse wrapper that elevates transforms_applied + CCR handles into trace spans. This closes the audit gap with ~50 lines of code.

When LangGraph compliance flow is scaffolded: 6. Design checkpoint state schema to carry IDs and references, not document content. Content stays in pgvector/R2. 7. Implement custom checkpoint reducer that compresses accumulated message lists on write.

When Hermes hits VPS: 8. Evaluate Mem0 for cross-session regulatory decision memory. This is orthogonal to Headroom's scope.

Later (lower leverage): 9. LongLLMLingua in question-aware mode for regulatory prose documents where context pressure forces a compression trade. This requires embedding a compressor model in your compliance pipeline, so the engineering cost is higher than the Headroom integration. 10. Tool schema lazy injection — worth doing at scale, low priority at current stage.

5. 🔑 The Reframe

Your case study ended with: "The default is the safe path. The escalation is the design decision that requires justification."

Applied to the full gap map:

Safe path (audit-grade, deterministic):
  RTK (shell) + SmartCrusher (JSON) + CodeCompressor (code) + CacheAligner (prefix)
  Upstream: Cohere Rerank (relevance gate)
  Coverage: ~55% of your token budget, fully reproducible

Opt-in escalation (higher savings, deliberate trade):
  + Kompress (prose/history) — for non-compliance workloads only
  + LongLLMLingua (regulatory prose) — when context pressure forces it, query-aware mode only
  Coverage: remaining ~45%, with stated auditability trade

Orthogonal concerns (different tools, different scope):
  Mem0 / Hermes — cross-session procedural memory
  LangGraph reducers — checkpoint state compression
  Langfuse wrapper — audit artifact pipeline

Headroom is comprehensive for the input-side content boundary. The gaps aren't about what Headroom does badly — they're about the four scenarios that exist outside the content boundary entirely (upstream relevance, cross-session memory, checkpoint state, audit wiring), plus one systematic bias in Kompress that matters specifically for regulatory text.

Filed under: Context Engineering · Compliance AI · Harness Engineering · RAG · LangGraph