Maxun: How LLM-Powered Web Scraping Fixes the Brittleness Problem
Hook
Every web scraper eventually breaks. The average CSS selector-based scraper fails within 47 days of deployment when target websites update their layouts. Maxun's architecture suggests a radically different approach: what if your scraper could adapt to layout changes like a human would?
Context
Web scraping has always been a game of whack-a-mole. You write a selector, the site updates, your pipeline breaks at 3 AM, and you're debugging DOM trees over coffee. Traditional approaches lock you into a binary choice: build brittle CSS selector chains that extract data fast but shatter on layout changes, or write complex fallback logic that turns 50 lines of scraping code into 500 lines of resilience engineering.
The no-code scraping space hasn't solved this fundamental tension—it's just made it easier to create brittle scrapers faster. Tools like ParseHub and Octoparse give you visual selector builders, but the output is still deterministic XPath that fails the moment a developer refactors a React component. Meanwhile, enterprises pay six figures for platforms like Import.io or Diffbot that maintain hand-tuned extractors for thousands of websites. Maxun enters this landscape with a hybrid thesis: use deterministic selectors when they work, and fall back to LLM reasoning when they don't. It's building infrastructure for a world where web scraping becomes an AI-native primitive, not just batch ETL.
Technical Insight
Maxun's architecture revolves around an intermediate representation format that decouples workflow definition from execution. When you record a scraping workflow through the UI, you're not generating a brittle Selenium script—you're creating a JSON specification that the execution engine interprets through Playwright's browser automation APIs. This separation is architecturally significant because it means workflows become versionable, diffable artifacts that can be stored in Git and reviewed like code.
The recorder mode works by instrumenting browser events and serializing them into replayable actions. Under the hood, it's capturing locators (Playwright's resilient selector syntax that automatically falls back through CSS, text content, and ARIA roles), waiting strategies (networkidle, domcontentloaded), and extraction patterns. Here's what a typical robot workflow looks like when you export it:
{
"name": "ProductScraper",
"startUrl": "https://example.com/products",
"actions": [
{
"type": "click",
"selector": "button:has-text('Load More')",
"waitFor": "networkidle"
},
{
"type": "extract",
"fields": [
{
"name": "title",
"selector": "h2.product-title",
"multiple": true
},
{
"name": "price",
"selector": ".price-tag",
"transform": "parseFloat"
}
]
}
],
"pagination": {
"type": "click",
"selector": "a.next-page",
"maxPages": 10
}
}
This declarative format is what makes Maxun's API-first approach possible. When you expose a robot as a REST endpoint, the backend isn't executing brittle code—it's interpreting this workflow IR through a sandboxed Playwright context. The scheduler can parallelize these executions across worker nodes, and the same workflow can run headless in production or headed during debugging.
The genuinely novel piece is the LLM extraction mode. When you switch to AI mode, Maxun sends the raw HTML (or a pruned DOM subtree) to an LLM with a structured extraction prompt. The system appears to use function calling APIs (likely OpenAI's tools or Anthropic's tool_use) to enforce schema compliance. Here's the conceptual flow:
// Simplified extraction logic (not actual Maxun source)
async function extractWithLLM(page: Page, schema: Schema) {
const html = await page.content();
const pruned = pruneBoilerplate(html); // Remove nav, footer, scripts
const completion = await openai.chat.completions.create({
model: "gpt-4o",
messages: [
{
role: "system",
content: "Extract structured data from HTML. Return only valid JSON matching the schema."
},
{
role: "user",
content: `HTML: ${pruned}\n\nSchema: ${JSON.stringify(schema)}`
}
],
response_format: { type: "json_object" }
});
return JSON.parse(completion.choices[0].message.content);
}
This approach trades determinism for resilience. When a site redesigns from <div class="price"> to <span data-testid="price-display">, the LLM doesn't care—it's looking for price semantics in the content, not DOM structure. The cost is literal: you're burning 5-10 cents per page extraction versus fractions of a cent for selector-based scraping. But for low-volume, high-value use cases (lead research, competitive intelligence), that's a reasonable trade.
The job queue architecture handles scheduled runs and webhook deliveries. When you configure a robot to run hourly and POST results to a webhook, Maxun is enqueueing job definitions (likely using BullMQ given the TypeScript ecosystem) that workers consume. This enables patterns like:
// Conceptual usage via Maxun's REST API
const robot = await maxun.createRobot({
workflow: productScraperJSON,
schedule: "0 * * * *", // Hourly cron
destinations: [
{ type: "webhook", url: "https://api.yourapp.com/products" },
{ type: "googleSheets", spreadsheetId: "abc123" }
]
});
// Trigger on-demand
const results = await maxun.runRobot(robot.id);
The self-hosted deployment model means you control the infrastructure—critical for compliance-sensitive industries where scraping PII or proprietary data can't touch third-party servers. The tradeoff is operational overhead: you're running Playwright browsers that consume 200-500MB RAM each, managing proxy rotation yourself, and handling database migrations.
Gotcha
The LLM extraction mode is brilliant for resilience but economically catastrophic at scale. Every page you scrape costs API tokens—if you're monitoring 10,000 product pages daily, you're looking at $500-1000/day in OpenAI costs versus $5-10 for selector-based scraping with the same coverage. Maxun doesn't appear to implement intelligent caching where semantically identical pages reuse extractions, so you're re-extracting the same product schema from minor HTML variations. For high-volume use cases, you need to architect around this: use recorder mode as the default, fall back to LLM mode only on extraction failures, and cache extraction patterns.
The anti-detection story is weak compared to enterprise scraping infrastructure. Maxun outsources proxy rotation to third parties (their docs reference RapidProxy and MangoProxy integrations), which means you're paying monthly subscriptions on top of hosting costs. There's no native CAPTCHA solving, no fingerprint randomization beyond Playwright's defaults, and no session management for authenticated scraping. If you're targeting sophisticated anti-bot systems (LinkedIn, Indeed, Cloudflare Turnstile), you'll need to layer in additional tooling. The recorder mode also generates point-in-time selectors—if the site uses dynamic class names (like CSS-in-JS with hashed classes), your robots will break immediately. Playwright's locator syntax helps, but the UI doesn't guide users toward resilient selector strategies like chaining text content with structural hints.
Verdict
Use if: you're building internal tooling for growth/marketing teams who need to extract data from <1000 pages/day, you can't hire scraping engineers but have budget for LLM API costs, or you're prototyping AI agents that need web data retrieval as a tool-use primitive. The dual-mode extraction and API-first design genuinely accelerate time-to-value for non-deterministic scraping tasks, and self-hosting keeps sensitive data in your VPC. Skip if: you're building a scraping product (AGPL licensing means you must open-source modifications), you need to scrape >10K pages daily (LLM costs will destroy your margins), you're targeting anti-bot systems without adding proxy infrastructure, or you have engineers who can write Playwright code directly (the no-code abstraction slows you down without adding value). For technical teams, Crawlee gives you the same Playwright foundation with better scaling primitives and zero per-page LLM tax.