TOON: The Data Format That Makes LLMs Pay Attention to Your Tables

Hook

A production LLM application processing 10 million API calls monthly can save $15,000 in token costs by changing five characters in their data format—switching from JSON's square brackets to TOON's curly braces.

Context

Every developer sending structured data to LLMs faces the same tax: JSON verbosity. When you serialize an array of user objects to pass as context in a ChatGPT prompt, you repeat field names like "id", "email", and "created_at" for every single record. With GPT-4 charging per token, this repetition isn't just aesthetically displeasing—it's expensive.

The obvious solution is compression, but LLMs don't work like web servers. They need to parse and reason about your data, not just decompress it. Binary formats like Protocol Buffers save bytes but are opaque to language models. YAML saves ~30% tokens through indentation but still repeats keys in arrays. CSV is compact for flat tables but can't represent nested objects. TOON emerged from this gap: the need for a format that's simultaneously human-readable, LLM-parseable, and token-efficient for the most common LLM use case—arrays of uniform objects.

Technical Insight

TOON's core innovation is selective structure optimization. It looks at your JSON and asks: "Which parts would benefit from tabular representation?" For uniform arrays of objects—think database query results or API responses—it collapses them into CSV-style tables with field headers. For everything else, it uses YAML-style indentation.

Here's a concrete example. Standard JSON for a user list:

[
  {"id": 1, "name": "Alice", "role": "admin"},
  {"id": 2, "name": "Bob", "role": "user"},
  {"id": 3, "name": "Charlie", "role": "user"}
]

TOON's equivalent:

[3] {id, name, role}
1, Alice, admin
2, Bob, user
3, Charlie, user

The magic happens in that first line. [3] declares array length upfront—a structural guardrail that tells the LLM exactly how many rows to expect. {id, name, role} establishes the field schema once, eliminating per-object key repetition. What follows is pure CSV data. This representation uses ~60% fewer tokens while actually improving parsing reliability.

The TypeScript SDK handles conversion transparently. The core API is deliberately minimal:

import { toToon, fromToon } from 'toon-format';

const users = [
  { id: 1, name: 'Alice', role: 'admin' },
  { id: 2, name: 'Bob', role: 'user' }
];

// JSON → TOON
const toonString = toToon(users);

// TOON → JSON (lossless round-trip)
const reconstructed = fromToon(toonString);

Under the hood, the serializer performs a uniformity analysis. It scans array elements to determine if they share the same keys in the same order. If tabular eligibility exceeds a threshold (default 80%), it applies the CSV transformation. Otherwise, it falls back to indented notation:

users:
  [2] {id, name}
  1, Alice
  2, Bob
metadata:
  created: 2024-01-15
  version: 1.2

The parser is schema-aware without requiring external schema definitions. When it encounters [N] {fields}, it knows to expect exactly N rows with those specific fields. This declarative structure is why LLMs achieve 74% parsing accuracy with TOON versus 70% with JSON in mixed-structure benchmarks—the format itself provides hints about data boundaries.

For nested objects within tables, TOON uses a dot-notation escape hatch:

[2] {id, profile.email, profile.age}
1, alice@example.com, 28
2, bob@example.com, 32

This flattening preserves the full JSON data model while maintaining tabular compactness. During deserialization, the parser reconstructs nested objects from dotted paths. It's a clever balance—you get CSV efficiency for the outer structure without sacrificing JSON's expressiveness.

The most underrated architectural choice is deterministic serialization. TOON guarantees that JSON → TOON → JSON round-trips produce identical output, including field order. This makes it viable for production systems where data integrity is non-negotiable, unlike lossy compression schemes that sacrifice precision for size.

Gotcha

TOON's efficiency collapses when your data isn't tabular-friendly. If you're working with deeply nested configuration objects or highly variable structures (like polymorphic API responses where each array element has different keys), the tabular optimization never triggers. You end up with indented notation that's roughly equivalent to compact JSON, minus the ecosystem maturity.

Latency is another trap. Despite using 40% fewer tokens, TOON doesn't guarantee 40% faster responses. Some models—particularly local or quantized deployments like Ollama—have parsers heavily optimized for JSON. The cognitive overhead of interpreting TOON's format can offset token savings, especially for smaller payloads. Always benchmark end-to-end latency on your specific model and data shape. One team reported that switching to TOON reduced their token costs by 35% but increased p95 response time by 12% because their Llama 2 deployment processed compact JSON significantly faster.

The ecosystem is also young. While implementations exist for TypeScript, Python, Go, Rust, and .NET, you won't find TOON support in popular LLM frameworks like LangChain or LlamaIndex. Integration requires custom serialization layers. The 24K GitHub stars indicate strong interest, but production tooling—debuggers, syntax highlighters, schema validators—is sparse compared to JSON's decades-old ecosystem.

Verdict

Use if: You're making frequent LLM calls with structured data containing uniform arrays (database results, logs, API responses), token costs are a meaningful line item in your budget, and you control both serialization and deserialization. TOON shines in production systems with consistent data shapes where 30-40% token reduction translates to real savings. Skip if: Your data is deeply nested or highly variable (tabular eligibility <50%), you need maximum parsing speed and can't afford latency testing, you're working with pure flat tables (just use CSV), or you require mature ecosystem tooling and can't build custom integrations. Always benchmark on representative data—TOON's benefits are workload-dependent, not universal.