Headroom: The Compression Layer Your AI Agent Pipeline Is Missing

Hook

Your AI coding agent just burned $47 processing a 200KB stack trace it compressed to 8KB—and still solved the bug. The LLM never saw the original, yet it retrieved exactly what it needed on-demand.

Context

If you're running Claude Code, Cursor, or Aider in production, you've watched your token bills spiral as agents dump entire log files, API responses, and test outputs into prompts. A single debugging session can burn through 500K tokens when the agent re-reads the same error logs across multiple tool calls. Anthropic and OpenAI charge by input token, so verbosity isn't just annoying—it's expensive.

The naive solution is manual prompt engineering: tell your agent to 'be concise' or truncate logs before passing them in. But this breaks context. The agent can't debug what it can't see, and you're now playing whack-a-mole trying to guess which parts of a 50KB JSON response matter. Headroom solves this by introducing a compression layer that sits between your agent and the LLM, automatically detecting content types and applying specialized compression strategies. JSON gets structurally reduced, code gets AST-parsed, and prose gets model-based compression. The clever bit: it's reversible. The LLM sees a compressed summary but can call a retrieval tool to fetch the original chunks it actually needs.

Technical Insight

Headroom's architecture is a three-stage pipeline that transforms content before it reaches the LLM. First is CacheAligner, which restructures prompts to maximize provider cache hits. Anthropic and OpenAI both offer prompt caching (they cache the KV pairs for identical prefixes to avoid recomputation), but they behave differently: Anthropic requires exact prefix matches down to the byte, while OpenAI is slightly more forgiving. CacheAligner pads and stabilizes your system prompt structure so that even when tool outputs change, the prefix stays cacheable. This alone can save 30-50% on costs before you even compress anything.

Next is ContentRouter, which dispatches content to specialized compressors. It uses regex patterns to detect JSON, tree-sitter grammars to parse code in six languages (Python, JavaScript, TypeScript, Go, Rust, Java), and falls back to a fine-tuned model for prose. Here's what the library integration looks like:

from headroom import Compressor, CompressionMode

compressor = Compressor(
    mode=CompressionMode.SMART,  # Auto-routes by content type
    cache_ttl=3600,  # Keep originals for 1 hour
    retrieval_enabled=True  # Give LLM a tool to fetch originals
)

# Your agent dumps a massive JSON response
tool_output = get_api_response()  # 180KB of nested JSON

# Compress before sending to LLM
compressed = compressor.compress(
    content=tool_output,
    context_id="debug-session-42",  # Links compressed chunks
    metadata={"source": "api_response", "turn": 3}
)

# compressed.text is now 12KB, compressed.savings = 0.93
# Original is cached with retrieval key
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"API returned:\n{compressed.text}"}
]

# Attach the retrieval tool
tools = compressor.get_retrieval_tools()  
response = client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    tools=tools  # LLM can call 'retrieve_original_chunk' if needed
)

The third layer is CCR (Compressed Context Retrieval), which stores the original content in a local cache keyed by content hash. When the LLM needs more detail, it calls the retrieve_original_chunk tool with a chunk ID from the compressed summary. Headroom looks up the original, retrieves the specific section, and injects it into the conversation. This sidesteps the lossiness problem: compression is aggressive because it's reversible.

For teams running multiple agents, the proxy mode is the killer feature. You don't rewrite your agent code—just point it at Headroom's local proxy server instead of the OpenAI/Anthropic API:

# Start the proxy
headroom proxy --upstream https://api.openai.com --port 8765

# Point your agent at localhost
export OPENAI_API_BASE=http://localhost:8765/v1

# Agent traffic now flows through Headroom automatically
cursor --api-key $OPENAI_API_KEY  # Cursor uses the proxied endpoint

The proxy uses ASGI middleware to intercept requests, rewrite message contents through the compression pipeline, and forward them upstream. It maintains the same streaming response format, so agents see no difference. The proxy also instruments the CacheAligner—it tracks which system prompt structures are hitting provider caches and auto-tunes padding to maximize hit rates across sessions.

SmartCrusher handles JSON by building a structural skeleton: it preserves keys, types, and array lengths but truncates values. A 50KB array of log entries becomes [{timestamp, level, message}, ... 847 more items]. IntelligentContext scoring (optional) can preserve high-importance fields by analyzing key names and value entropy, but it's not the default. For code, CodeCompressor uses tree-sitter to parse the AST and strips comments, docstrings, and redundant whitespace while preserving syntax. It understands scope, so it can compress helper functions more aggressively than the main logic.

The output token reduction feature is subtler: Headroom appends terse instructions to the system prompt (outside the cached prefix, so it doesn't break cache alignment) like 'Respond in bullet points' or 'Use abbreviations where unambiguous.' It dynamically adjusts these based on turn type—tool results get more aggressive compression than initial user queries because the agent is resuming work, not starting fresh.

Gotcha

Headroom's CCR cache is local-only with no distributed mode. If you're running multiple agent instances—common in CI pipelines or multi-user development environments—each instance maintains its own cache. The LLM might compress content on instance A, then get routed to instance B for the next turn, where the retrieval tool fails because the chunk isn't in B's cache. The workaround is sticky routing (pin sessions to instances), but that's infrastructure complexity you didn't sign up for.

The proxy's /admin/runtime-env endpoint is unauthenticated by default. Any process with localhost access can POST new environment variables, including HEADROOM_COMPRESSION_LEVEL or HEADROOM_CACHE_TTL, affecting all connected agents. In shared dev environments (cloud workstations, pair programming setups), this is a privilege escalation risk. You'll need to firewall the admin routes or run the proxy in a container with network isolation.

SmartCrusher's JSON compression is purely structural—it doesn't understand semantic importance. A 10KB array of debug timestamps gets the same treatment as a 10KB array of critical error codes. IntelligentContext scoring exists but isn't applied by default, and when enabled, it only looks at key names and value types, not domain-specific semantics. If your JSON has oddly named but critical fields, they'll get crushed. The CodeCompressor supports six languages with hardcoded tree-sitter grammars. If you're working in Elixir, Julia, or domain-specific languages, you're out of luck.

Verdict

Use if: You're running multiple AI coding agents (Claude Code, Cursor, Aider) in production and hitting context limits or burning >$500/month on repetitive tool outputs and log dumps. The proxy mode is zero-code integration, and CacheAligner's cache hit optimization often pays for itself before compression even kicks in. Also use if you're doing heavy RAG workloads where retrieved chunks are verbose—CCR's reversible compression lets you stuff more context in without sacrificing retrieval quality. Skip if: You're prototyping with a single agent or working in a sandboxed environment where running local proxies is blocked by security policy. Also skip if you need multi-instance deployments with shared state (the local-only cache will break you), or if you're compressing non-English or domain-specific content heavily (the Kompress model is tuned for English agentic traces). If your agents already have tight, well-engineered prompts, the marginal gains may not justify the operational overhead.