Teracrawl: Why the Fastest Web Scraper Beat 14 Competitors on LLM Benchmark Tests

Hook

When a TypeScript scraper beats established players like Firecrawl by 2.1 percentage points on a 1,000-URL benchmark, it's worth understanding what architectural choices made the difference.

Context

Building AI agents and RAG systems requires feeding LLMs clean, structured data from the web. But modern websites are hostile territory: JavaScript-heavy SPAs that don't render without execution, aggressive bot detection that blocks headless browsers, and DOM structures littered with navigation menus, cookie banners, and ads that dilute the actual content. Traditional scrapers like BeautifulSoup work fine for static HTML but fail on dynamic content. Browser automation tools like Playwright give you control but require managing browser infrastructure, implementing anti-bot evasion, and writing custom logic to identify main content areas.

The explosion of LLM applications created a specific scraping niche: converting arbitrary web pages into clean Markdown that maximizes signal and minimizes token waste. Firecrawl pioneered this category, but Teracrawl emerged as an alternative that prioritizes benchmark performance and intelligent mode-switching between fast static scraping and full JavaScript rendering. Rather than running browsers locally, it leverages Browser.cash's managed Chrome instances—real browsers that evade detection systems designed to block automation.

Technical Insight

Teracrawl's architecture revolves around a two-phase crawling strategy that optimizes the speed-vs-compatibility tradeoff. When you request a URL, it first attempts a fast static scrape that blocks images, videos, and fonts while reusing browser contexts across requests. If the resulting Markdown is too short (indicating the page needs JavaScript to render content), it automatically retries with dynamic mode, waiting for network idle and JavaScript hydration to complete.

The API surface is deliberately minimal. Here's a basic scrape request:

const response = await fetch('http://localhost:3000/scrape', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    url: 'https://example.com/article',
    options: {
      waitFor: 1000,  // Additional wait time in ms
      timeout: 30000   // Overall timeout
    }
  })
});

const { markdown, metadata } = await response.json();

The response includes cleaned Markdown with main content extracted and metadata like title, description, and language. Images are removed and replaced with alt text to reduce token count—a deliberate choice for LLM consumption rather than human reading.

What makes the /crawl endpoint powerful is its combination of search and parallel scraping. Instead of manually searching Google, parsing SERP results, and then scraping each URL sequentially, you get a single API call:

const response = await fetch('http://localhost:3000/crawl', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    query: 'best practices for vector databases',
    numResults: 5
  })
});

const results = await response.json();
// Returns array of { url, markdown, metadata } objects

Under the hood, this delegates to browser-serp for Google search execution (which itself uses Browser.cash to avoid CAPTCHA challenges), then scrapes all results concurrently. For RAG systems that need to gather context from multiple sources, this pattern eliminates 80% of the boilerplate.

The content extraction logic uses Mozilla's Readability library as a foundation but adds LLM-specific postprocessing. It strips script and style tags, removes navigation elements and sidebars, and converts the remaining HTML to Markdown using turndown. The key insight is that LLMs don't need pixel-perfect preservation of layout—they need semantic content hierarchy, which Markdown's heading structure provides naturally.

The session pooling architecture deserves attention. Rather than spawning a new browser for each request (expensive) or reusing a single browser serially (slow), Teracrawl maintains a pool of browser contexts. In fast mode, these contexts have aggressive resource blocking configured: images, media, fonts, and stylesheets never load, reducing bandwidth and rendering time by 60-80% for content-heavy pages. The tradeoff is that JavaScript expecting these resources might break, which is why the dynamic fallback mode exists.

Using Browser.cash instead of local Puppeteer/Playwright instances provides several advantages beyond anti-bot evasion. The remote browsers run in a data center with high bandwidth and low latency to most web servers. They're already warmed up with realistic browser fingerprints, plugin configurations, and cookies from normal browsing sessions. For sites with aggressive protection (Cloudflare, PerimeterX, DataDome), this makes the difference between successful scraping and endless CAPTCHA loops.

Gotcha

The Browser.cash dependency is both Teracrawl's strength and its Achilles' heel. You're paying per browser session, which means scraping costs scale linearly with volume. For prototyping or low-volume applications, the pricing is reasonable and the convenience is worth it. But if you're building a product that scrapes thousands of pages daily, those API costs become a line item that never goes away. You also inherit Browser.cash's rate limits and any service disruptions they experience. There's no fallback to local browsers—the entire architecture assumes remote execution.

The search functionality requires running browser-serp as a separate service. The documentation shows how to do this via Docker, but it's another moving part in your infrastructure. If the browser-serp instance goes down or gets overloaded, your /crawl endpoint breaks while /scrape continues working. For production deployments, you need to monitor and scale two services instead of one.

The Markdown output, while optimized for LLMs, isn't always ideal for human consumption. Images are stripped entirely rather than being preserved as links, which makes sense for token reduction but means you can't reconstruct visual context if needed. Complex tables sometimes get mangled in the HTML-to-Markdown conversion. And for pages with critical content in JavaScript-rendered elements (like React portals rendering outside the main content area), the Readability extraction occasionally misses sections that a human would consider important.

Verdict

Use if you're building LLM applications (RAG systems, AI agents, research tools) where scraping quality directly impacts answer accuracy and you need proven benchmark performance across diverse websites. The two-phase crawling handles both static and dynamic sites intelligently, and the combined search-and-scrape endpoint eliminates massive amounts of integration code. The Browser.cash dependency is a feature, not a bug—you're paying to avoid weeks of anti-bot engineering. Skip if you need complete control over your scraping infrastructure without external dependencies, operate at scale where per-request API costs become prohibitive, or primarily scrape simple static sites where the sophistication (and cost) isn't justified. Also skip if you need to preserve visual content like images and complex layouts rather than just extracting semantic text.