Testing Web Security Crawlers with Google's Security Crawl Maze

Hook

Most security crawlers miss 30-40% of URLs in a typical web application—not because the links are hidden, but because they're embedded using one of dozens of obscure HTML resource-linking methods that developers rarely test against.

Context

Web security crawlers serve a fundamentally different purpose than traditional web crawlers. While search engine bots prioritize content coverage and respect robots.txt, security crawlers need comprehensive code coverage to find every possible entry point where vulnerabilities might hide. A single missed endpoint—whether it's linked via an uncommon HTML attribute, embedded in a JavaScript framework's routing config, or referenced through a meta refresh tag—could be the one vulnerable path an attacker exploits.

The challenge is that HTML provides dozens of ways to link resources: anchor tags, form actions, script sources, iframe sources, meta redirects, link prefetch hints, and more. Modern JavaScript frameworks add another layer of complexity with client-side routing and dynamic imports. Before Security Crawl Maze, there was no standardized, comprehensive testbed for validating whether a security crawler could actually discover all these linking mechanisms. Developers building security tools were left creating ad-hoc test cases or, worse, assuming their crawlers worked correctly until a penetration test revealed blind spots.

Technical Insight

Security Crawl Maze uses a Flask blueprint architecture where the URL structure directly mirrors the HTML element hierarchy being tested. A test case at /html/body/a/href.html literally tests the <a href> linking mechanism, while /html/head/link/href.html tests <link href> elements. This self-documenting structure makes it trivial to understand what each test validates and to add new cases as web standards evolve.

The core contract is simple: any URL path ending with .found represents a resource that a competent security crawler should discover. The application serves an expected_results.json file cataloging all discoverable URLs, and provides an API endpoint at /fetch-expected-results that returns this canonical list. You can build automated validation by crawling the application, collecting discovered URLs, and comparing against the expected results:

import requests
import json
from your_crawler import SecurityCrawler

# Crawl the maze
crawler = SecurityCrawler(start_url='https://security-crawl-maze.app/')
discovered_urls = crawler.crawl()

# Fetch expected results
response = requests.get('https://security-crawl-maze.app/fetch-expected-results')
expected = set(response.json())

# Calculate coverage
found = set(discovered_urls)
coverage = len(found & expected) / len(expected) * 100
missed = expected - found

print(f"Coverage: {coverage:.1f}%")
if missed:
    print(f"Missed {len(missed)} URLs:")
    for url in sorted(missed)[:10]:
        print(f"  - {url}")

The test cases themselves draw from cure53's HTTPLeaks research, providing coverage of obscure linking mechanisms that many crawler developers never consider. For example, most crawlers handle standard anchor tags and form actions, but how many check <object data="/path.found">, <embed src="/path.found">, or <video poster="/path.found">? Security Crawl Maze tests all of these, along with meta refresh redirects, link prefetch hints, srcset attributes, and even JavaScript framework-specific patterns.

The Docker setup handles the complexity of testing modern frameworks. Angular and Polymer test cases require build steps that compile framework code into executable JavaScript. The Dockerfile orchestrates this multi-stage build process, installing Node.js dependencies and running framework-specific build commands before serving the compiled output alongside traditional HTML test cases. This means you can validate whether your crawler properly handles both server-rendered HTML and client-side routing in a single test run.

What makes the architecture particularly elegant is the separation of test case generation from validation logic. Each test case is a standalone HTML file that can be inspected, modified, or used independently. There's no complex test harness or framework-specific setup required—just serve the Flask app and start crawling. The /api/ endpoint structure also allows for programmatic test case discovery, meaning you could build a continuous integration pipeline that automatically tests your crawler against the latest version of the maze without manual updates.

Gotcha

The biggest limitation is scope: Security Crawl Maze only tests link discovery, not the full spectrum of security crawler challenges. Real-world applications require authentication, handle rate limiting, implement CSRF tokens, use WebSockets, trigger behavior based on user agents, and employ anti-bot mechanisms. The maze doesn't test any of these scenarios, so 100% coverage in Security Crawl Maze doesn't guarantee your crawler will succeed against production applications.

The JavaScript framework test cases only work in Docker, not with the standalone Flask development server. This creates friction for rapid iteration—if you're developing a crawler and want to quickly test against Angular routing patterns, you can't just flask run and go. You need to build the Docker image, which adds several minutes to your feedback loop. Additionally, the project's relatively low adoption (165 stars) suggests the community hasn't contributed many test cases for newer web technologies. There are no test cases for Web Components, Shadow DOM manipulation, or modern meta-frameworks like Next.js or SvelteKit, which means you'll need to extend the test suite yourself if you're targeting applications built with cutting-edge stacks.

Verdict

Use Security Crawl Maze if you're building or evaluating a web security crawler and need objective, reproducible benchmarks for link discovery capabilities. It's essential for ensuring your crawler won't miss obscure HTML resource-linking methods that could hide vulnerabilities, and the public instance makes it trivial to run quick validation tests. The expected results API enables easy CI/CD integration for regression testing as you add new crawler features. Skip it if you're working on general-purpose web scraping (not security-focused), need to test authentication flows or session handling, or require validation beyond resource discovery. Also skip if your target applications exclusively use modern JavaScript frameworks with complex client-side state management—the limited framework coverage means you'll spend more time extending the test suite than benefiting from existing cases.