Magnitude: Browser Automation That Sees the Web Like You Do

Hook

What if your automation tests could work without a single CSS selector? Magnitude treats browser automation like a vision problem, not a DOM traversal exercise—and it's achieving 94% success rates on real-world web navigation tasks.

Context

Anyone who's maintained end-to-end tests knows the pain: a designer changes a button's class name, and suddenly 47 tests break. Traditional browser automation tools like Selenium and Playwright rely on DOM selectors—CSS classes, XPath expressions, data attributes—to identify and interact with elements. This works beautifully until it doesn't. Modern web applications with dynamic class names, shadow DOM, canvas-based interfaces, and complex JavaScript frameworks turn selector maintenance into a full-time job.

The underlying problem is that we've been solving browser automation the way computers see the web (as a tree of DOM nodes) rather than how humans see it (as a visual interface). We tell our tests to "click the button with class .submit-btn-primary-v2" when we really mean "click the blue Submit button in the bottom right." Magnitude inverts this model entirely. Built on TypeScript and Playwright, it uses visually grounded large language models to interpret browser interfaces through screenshots and specify actions via pixel coordinates. Instead of hunting for the right selector, you describe what you want in natural language, and the AI figures out where to click.

Technical Insight

Magnitude's architecture separates intent from execution through a vision-first pipeline. When you give it a command like "search for TypeScript tutorials," it captures a screenshot of the current page, sends it to a visually grounded LLM (Claude Sonnet 4 or Qwen-2.5VL 72B), and receives back a plan expressed as pixel coordinates and actions. The LLM doesn't just parse the page—it understands spatial relationships, visual hierarchies, and interface conventions the way a human would.

Here's what a basic Magnitude script looks like:

import { BrowserAgent } from '@magnitude/browser-agent';

const agent = new BrowserAgent({
  model: 'claude-sonnet-4',
  apiKey: process.env.ANTHROPIC_API_KEY
});

await agent.goto('https://github.com/search');
await agent.do('search for "browser automation" and click the first repository');

const stars = await agent.extract({
  stars: z.string().describe('the number of stars this repository has')
});

console.log(`Repository has ${stars.stars} stars`);

Notice what's missing: no page.locator(), no CSS selectors, no waiting for specific elements. The .do() method accepts natural language instructions, and the LLM handles the translation to concrete actions. Under the hood, Magnitude is orchestrating a multi-step process: screenshot capture, vision model inference, coordinate calculation, and Playwright action execution.

The data extraction capability is particularly elegant. Instead of writing brittle selectors to scrape content, you define a Zod schema describing what you want, and the LLM visually identifies and extracts the information. This works across wildly different page layouts because the model understands semantic meaning, not just DOM structure. If you're extracting product prices, it doesn't matter whether they're in a <span class="price">, a <div data-price>, or rendered in a canvas—the model sees "$49.99" and knows it's a price.

Magnitude also includes a test runner that embraces the vision-first philosophy:

import { test, expect } from '@magnitude/test';

test('user can complete checkout flow', async ({ agent }) => {
  await agent.goto('https://store.example.com');
  await agent.do('add the first item to cart');
  await agent.do('proceed to checkout');
  
  await expect(agent).toShow('order confirmation');
  
  const orderDetails = await agent.extract({
    orderNumber: z.string(),
    total: z.string(),
    email: z.string().email()
  });
  
  expect(orderDetails.email).toBe('test@example.com');
});

The toShow() assertion is a visual check—it uses the LLM to verify that something matching "order confirmation" appears on screen, without needing to know the exact text, layout, or DOM structure. This makes tests remarkably resilient to UI changes. Your designer can completely redesign the confirmation page, and as long as it still visually communicates "order confirmation," the test passes.

The framework supports multiple abstraction levels. For production workflows where cost and determinism matter, you can cache LLM responses or define custom actions that map high-level commands to specific sequences. This hybrid approach gives you natural language flexibility during development and exploration, with a path to optimization for production.

One architectural decision worth noting: Magnitude doesn't try to replace Playwright; it augments it. You still get access to the underlying Playwright page object for scenarios where explicit control makes sense. This pragmatic design means you can drop down to traditional selectors for performance-critical paths while using vision-based automation for the complex, brittle parts.

Gotcha

The elephant in the room is cost. Visually grounded LLMs aren't cheap—Claude Sonnet 4 charges per API call, and each action requires at least one screenshot analysis. A moderately complex automation flow might cost 10-50 cents to run, compared to fractions of a penny for traditional selector-based tools. If you're running thousands of test executions in CI/CD, this adds up fast. The economic model works best for scenarios where engineering time saved on selector maintenance outweighs compute costs, or for workflows that are impossible to automate reliably with traditional tools.

Speed is another consideration. Vision model inference adds latency—typically 2-5 seconds per action depending on the model and image complexity. Traditional Playwright scripts that execute in 10 seconds might take 30-60 seconds with Magnitude. This matters less for overnight automation jobs or occasional manual workflows, but it's a significant slowdown for rapid feedback loops during development. The framework's caching system (currently in progress) aims to address this for test runs by making LLM responses deterministic and reusable, but it's not fully baked yet.

There's also the inherent variability of LLM-based systems. Even with caching, you're dealing with models that can occasionally misinterpret visual interfaces or make unexpected decisions. A button that's partially obscured, unusual color schemes, or novel UI patterns might confuse the model. This non-determinism is acceptable for exploratory automation but potentially problematic for mission-critical workflows where absolute reliability is required.

Verdict

Use Magnitude if you're automating complex, dynamic web applications where DOM selectors are a maintenance nightmare—think modern SPAs with CSS-in-JS, shadow DOM, or frequently changing class names. It shines for cross-site workflows, competitive intelligence gathering, and scenarios where you need to extract structured data from visually presented information. If you're writing tests for customer-facing flows that must work despite constant UI tweaks, or building RPA solutions where describing tasks in natural language dramatically speeds development, the cost premium is worth it. Skip it if you're automating internal tools with stable, well-structured DOM trees where traditional selectors work fine, or if you're running high-volume automation where per-execution costs matter more than engineering time. For performance-critical paths, API-based alternatives, or scenarios where sub-second response times are required, stick with traditional Playwright or Selenium. The sweet spot is high-value, low-frequency automation where brittle selectors have been costing you hours of maintenance.