Context Viewer: Using GPT to Debug Why Your GPT Costs Too Much

Hook

The best way to understand why your AI chatbot uses 80,000 tokens per request? Ask another AI to analyze it. Welcome to recursive observability.

Context

As AI applications moved from demos to production, engineering teams discovered a new budget line item: context window costs. A customer support bot might burn through $50,000/month in API fees, but where are those tokens actually going? Traditional monitoring shows total token counts per request, but that's like optimizing a database by only looking at total query time—you need query-level breakdowns to fix anything.

The problem is semantic, not syntactic. Counting tokens is easy (tiktoken does that), but understanding whether those tokens represent system instructions, tool definitions, conversation history, or user content requires interpretation. You might have 20,000 tokens of documentation crammed into every request because someone copy-pasted your entire API reference into the system prompt. Or maybe your conversation history pruning broke and you're sending 50 turns of "hello"/"hi" exchanges. Token counters can't tell you this—they just report numbers. Context Viewer treats this as a classification problem: use an LLM to semantically segment another LLM's conversation log, then visualize the breakdown so you can actually optimize something.

Technical Insight

Context Viewer's architecture is deliberately minimal: a React frontend that orchestrates parsing, analysis, and visualization without touching a backend. The analysis pipeline has three stages, and the clever part is stage two.

First, parsers convert conversation logs into a normalized format. You paste in an OpenAI API request/response JSON (the kind you'd copy from network inspector or logs), and the parser extracts messages, function definitions, and metadata. The plugin architecture theoretically supports multiple providers, though only OpenAI is wired up:

// Simplified parser interface
interface ConversationParser {
  canParse(input: string): boolean;
  parse(input: string): ParsedConversation;
}

interface ParsedConversation {
  messages: Message[];
  tools?: ToolDefinition[];
  metadata: {
    model: string;
    timestamp?: string;
  };
}

Second, semantic decomposition happens via Vercel's AI SDK calling GPT-4o-mini. This is where it gets meta: the tool sends your conversation log to an LLM with a structured prompt asking it to classify each message component. The prompt defines categories like "system_instructions", "tool_definitions", "code_blocks", "conversation_history", and "user_query". The LLM returns a structured JSON mapping message parts to categories:

// The analysis LLM receives your conversation and returns
interface SemanticBreakdown {
  segments: Array<{
    category: 'system_instructions' | 'tool_definitions' | 
              'code_blocks' | 'conversation_history' | 'user_query' | 'other';
    content: string;
    tokenCount: number;
    messageIndex: number;
  }>;
}

This is the core innovation: you're paying tokens to understand tokens. A 50k-token conversation might cost $0.15 to analyze (input tokens) plus $0.05 for the structured classification output. The analysis cost is roughly 0.5-1% of the original conversation cost, which is acceptable for diagnostics but prohibitive for continuous monitoring.

Third, D3-based visualizations aggregate these segments into stacked bars, pie charts, and comparison views. The comparison mode is particularly useful—you can upload two versions of your prompt and instantly see that v2 added 15k tokens of tool definitions or that competitor X uses 3x more system instructions than you do.

The URL-based state persistence is smarter than it looks. Instead of base64-encoding entire conversation logs into URLs (which would exceed browser limits), it stores a hash of the input and uses localStorage for the actual data, then falls back to a compressed encoding for sharing:

// Simplified state serialization
function serializeState(analysis: Analysis): string {
  const hash = hashContent(analysis.rawInput);
  localStorage.setItem(`cv_${hash}`, JSON.stringify(analysis));
  
  // For sharing, create a compressed version
  const compressed = compressAnalysis(analysis);
  return `${window.location.origin}?analysis=${compressed}`;
}

This means your analysis links work immediately on your machine (localStorage hit) but also work when shared with teammates (decompresses from URL). It's stateless infrastructure that feels stateful.

The categorization taxonomy is where business logic meets observability. By distinguishing "tool_definitions" from "conversation_history", the tool reveals optimization targets: tool definitions are static and cacheable (with prompt caching), conversation history is prunable, system instructions are compressible through better prompt engineering. Raw token counts can't surface these insights—you need semantic understanding of what each token represents in the conversation's structure.

Gotcha

The "model and format agnostic" claim is aspirational. While Vercel AI SDK supports Anthropic, Google, and other providers, Context Viewer only implements OpenAI parsers. You can't analyze Claude or Gemini conversations without writing custom parsers, which means most teams on non-OpenAI stacks are excluded. The codebase has provider abstraction layers ready, but they're unpopulated—this is a one-provider tool pretending to be multi-provider.

Analysis cost scales badly. A 200k-token conversation (not uncommon for coding agents with multiple file contexts) costs $2-3 to analyze. If you're debugging why your context is bloated, that's fine. If you want to analyze every production conversation to track cost attribution trends, you'd spend more on analysis than you save from optimization. There's no batch mode, sampling strategy, or caching of analysis results. Every analysis is a fresh API call with fresh costs. For continuous observability, you'd need to fork this and add your own aggregation layer, at which point you're building a custom monitoring system, not using a tool.

Verdict

Use if: You're debugging specific prompt bloat issues (why does this agent use 80k tokens?), reverse-engineering competitor prompts to understand their context composition strategy, or comparing prompt versions during optimization sprints. The visualization immediately surfaces actionable insights that raw logs hide, and the client-side architecture means you can analyze sensitive conversations without data leaving your machine. Use if: You're on OpenAI and need a one-time diagnostic tool, not continuous monitoring. Skip if: You're on Anthropic/Google/Azure (provider support doesn't exist despite what the README implies), you need programmatic access or CI/CD integration (it's a manual UI tool), or you want continuous production monitoring (analysis costs make this prohibitive at scale). Skip if: You need multi-user collaboration beyond sharing URLs—there's no team features, audit trails, or access control. This is a scalpel for prompt autopsies, not a monitoring dashboard.