LLM Scraper: When AI Replaces CSS Selectors for Web Data Extraction

Hook

What if you could scrape a website without writing a single CSS selector or XPath expression—and have it keep working even when the site's HTML completely changes?

Context

Web scraping has always been a game of cat and mouse. You write CSS selectors like .product-price or XPath expressions to extract data, and everything works beautifully—until the website redesigns their HTML structure next Tuesday. Your selectors break, your pipeline fails, and you're back to spelunking through Chrome DevTools at 2 AM.

The fragility problem gets worse when you're scraping multiple sites with different structures, or when you need to extract semantic information that isn't cleanly wrapped in HTML tags. Want to grab "the author's main argument" or "all mentioned prices including currency"? Traditional scrapers require you to anticipate every possible HTML pattern and write defensive selector logic. Enter LLM Scraper, a TypeScript library that flips the paradigm: instead of telling the computer where to find data, you tell it what data you want, and let a Large Language Model figure out the rest. Built on Playwright and integrated with Vercel's AI SDK, it treats web scraping as a natural language understanding problem rather than a DOM traversal exercise.

Technical Insight

The architecture of LLM Scraper centers on a deceptively simple pipeline: extract page content, define your desired schema with Zod, send both to an LLM, and receive type-safe structured data. But the implementation reveals several clever decisions that make this approach practical.

Here's how you define and extract structured data in less than 20 lines of code:

import LLMScraper from 'llm-scraper';
import { z } from 'zod';
import { chromium } from 'playwright';
import { openai } from '@ai-sdk/openai';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://news.ycombinator.com');

const scraper = new LLMScraper(page, {
  model: openai('gpt-4o-mini')
});

const schema = z.object({
  stories: z.array(z.object({
    title: z.string(),
    points: z.number(),
    author: z.string()
  }))
});

const data = await scraper.run(schema, {
  format: 'html'
});

console.log(data.stories); // Fully typed array

Notice the Zod schema acts as both a type definition and a prompt component. LLM Scraper serializes this schema into a description the LLM can understand, essentially saying "extract an array of stories, each with a title (string), points (number), and author (string)." The Vercel AI SDK's generateObject function under the hood uses structured output modes—JSON schema for OpenAI, tool calling for Anthropic—ensuring the LLM response conforms to your schema rather than hoping the model returns valid JSON.

The library supports four content formatting modes, each with different tradeoffs. The default html mode preprocesses the page by removing scripts, styles, and hidden elements before sending cleaned HTML to the LLM. The markdown mode uses Mozilla's Readability.js to convert pages into clean markdown, which often produces better results for content-heavy sites since it strips navigation and boilerplate. The text mode extracts just visible text, useful when you want to minimize token usage. Most interesting is the image mode for vision models:

const scraper = new LLMScraper(page, {
  model: openai('gpt-4o'),
  mode: 'image'
});

const chartData = await scraper.run(z.object({
  dataPoints: z.array(z.object({
    label: z.string(),
    value: z.number()
  }))
}));

This takes a screenshot of the viewport and sends it to a vision model, enabling extraction from canvas charts, SVG graphics, or content rendered via JavaScript that's difficult to access from the DOM. It's the nuclear option for complex visual layouts.

One of the most pragmatic features is the built-in code generation capability. After you've prototyped a scrape and confirmed it works, you can generate a standalone Playwright script that extracts the same data using deterministic selectors:

const { code } = await scraper.generateScript(schema);
fs.writeFileSync('scraper.ts', code);

This generated script includes actual CSS selectors and extraction logic discovered during the LLM-powered scrape, giving you the best of both worlds: AI-assisted development with deterministic execution. For production pipelines where you'll run the same scrape thousands of times, this converts the expensive LLM-based approach into a traditional (free) selector-based scraper.

The streaming support deserves attention for real-time applications. Using scraper.stream() instead of scraper.run() returns a stream of partial objects as the LLM generates tokens:

const stream = await scraper.stream(schema);

for await (const partial of stream) {
  console.log(partial); // Incrementally populated object
  // Update UI, process partial data, etc.
}

This is particularly valuable for large scrapes where you want to show progress or start processing results before the LLM finishes generating the complete response. Under the hood, it uses the AI SDK's streaming object mode, which parses incomplete JSON and fills in typed placeholders until final values arrive.

Gotcha

The elephant in the room is cost and latency. Every scrape requires an LLM API call, and you're paying for both input tokens (the page content) and output tokens (the structured data). A typical product listing page might consume 2,000-5,000 input tokens, and at GPT-4 pricing that's $0.03-$0.075 per scrape. If you're extracting data from 10,000 pages, you're looking at $300-$750 in API costs compared to essentially zero for traditional scraping. The time penalty is even more stark—LLM calls take 2-10 seconds versus milliseconds for CSS selector extraction.

The accuracy story is murkier than deterministic selectors. While LLMs are remarkably good at understanding semantic intent, they can hallucinate data, especially for fields that look like they should exist but don't. If your schema asks for a rating field and the page doesn't have one, some models will invent a plausible value rather than returning null. The Zod integration provides runtime validation which catches type errors, but it can't detect plausible-but-wrong values. You also lose reproducibility—the same page scraped twice might return slightly different results due to LLM non-determinism, even with temperature set to zero. For compliance-critical applications where you need to prove exactly what data came from where, this uncertainty is a deal-breaker. The generated code feature helps, but you're still trusting the LLM got the selectors right during that initial discovery phase.

Verdict

Use if: You're scraping sites with frequently changing layouts where maintenance time exceeds API costs, you need to extract semantic information that isn't cleanly structured in HTML (like "all mentioned competitors" or "pros and cons"), you're prototyping and need results fast without DOM archaeology, or you're dealing with visual content that requires vision models. It's perfect for research projects, one-off data collection tasks, internal tools, and situations where developer time is more expensive than compute.

Skip if: You're building high-volume production scraping where you'll hit the same sites repeatedly (generate the script first, then use that), you need guaranteed deterministic extraction for compliance or auditing, your budget is tight and traditional selectors work reliably, or latency matters (real-time applications, user-facing features). Also skip if you're scraping simple, well-structured sites where CSS selectors are obvious—you're just burning money on API calls that add no value.