Testing AI Security Tools Against 104 Zero-Contamination Vulnerabilities

Hook

Every AI security tool you've tested has likely already seen the vulnerabilities in its training data—making your benchmarks measure memorization, not capability. XBOW's validation-benchmarks solves this with 104 previously unreleased challenges containing canary strings to detect contamination.

Context

The explosion of AI-powered security tools promises autonomous vulnerability discovery and exploitation, but evaluating these tools has become an exercise in circular logic. Popular benchmarking datasets—OWASP's WebGoat, Damn Vulnerable Web Application, even challenge archives from CTF competitions—have been crawled, indexed, and likely incorporated into the training corpora of large language models. When you test an AI agent against DVWA's SQL injection challenges, you're not measuring its ability to discover vulnerabilities; you're measuring whether its training data included the DVWA repository.

This contamination problem extends beyond obvious sources. Security researchers share proof-of-concept exploits on GitHub, write-ups circulate on Medium and personal blogs, and vulnerability databases document thousands of real-world CVEs with exploitation details. An AI trained on "the internet" has effectively memorized the answer key. For offensive security tools specifically, this creates a perverse incentive: vendors optimize for performance on public benchmarks their models have already seen, while real-world capability remains unknown. XBOW Engineering built validation-benchmarks to cut through this fog—a set of 104 deliberately vulnerable challenges that were kept confidential until publication, designed explicitly to evaluate AI agents rather than human pentesters, with architectural safeguards to prevent the exact contamination problem plaguing existing benchmarks.

Technical Insight

The core architectural innovation in validation-benchmarks is its use of Docker build arguments to inject unique flags into each challenge instance. Rather than hardcoding secrets into Dockerfiles or source code (which would make them visible in the repository), each challenge accepts a flag as a build-time parameter. This approach serves two purposes: it minimizes rebuild times by allowing Docker layer caching for everything except the final flag injection, and it ensures that even if an AI model somehow trained on the repository structure, it couldn't memorize specific flag values.

Here's how a typical challenge structures its Dockerfile:

FROM php:8.1-apache

ARG FLAG
ENV APP_FLAG=${FLAG}

COPY src/ /var/www/html/
RUN chown -R www-data:www-data /var/www/html

# Flag gets written to filesystem only at build time
RUN echo "<?php define('SECRET_FLAG', '${APP_FLAG}'); ?>" > /var/www/html/config/flag.php

EXPOSE 80

The corresponding docker-compose.yml then injects the flag:

services:
  web:
    build:
      context: .
      args:
        FLAG: ${CHALLENGE_FLAG:-FLAG{default_for_testing}}
    ports:
      - "8080:80"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 5s
      timeout: 3s
      retries: 10

This architecture scales across 104 challenges with varying complexity. Some challenges expose single vulnerable web services; others orchestrate multi-container environments where an AI agent must chain vulnerabilities across services. For instance, a challenge might require exploiting an SSRF vulnerability in a frontend service to access an admin API on an internal network, then using that API to extract database credentials.

The benchmark suite includes sophisticated dependency management through Docker Compose's depends_on with health checks. This ensures services start in the correct order—databases initialize before web applications, authentication services become available before dependent APIs. Without this, AI agents would encounter race conditions where services appear available but aren't actually ready to process requests, contaminating benchmark results with spurious failures.

Each challenge includes metadata tags describing its characteristics:

{
  "id": "web-sqli-blind-002",
  "difficulty": 2,
  "category": "sql-injection",
  "subcategory": "blind-boolean",
  "win_condition": "flag_extraction",
  "estimated_time": "15-30min",
  "hints_available": false
}

The difficulty scale (1-3) reflects the complexity of exploitation chains rather than obscurity—a difficulty-3 challenge might require combining multiple vulnerability types or understanding subtle application logic. The win_condition field distinguishes between challenges where the AI must extract a specific flag string versus challenges where it must answer questions about application state ("How many admin users exist in the database?").

Crucially, the repository contains MAPS (Model-Authored Proof Strings)—unique canary strings embedded in challenge source code that shouldn't appear anywhere else on the internet. If these strings appear in an AI model's output without the model actively examining the challenge, it indicates the model trained on the benchmark data, invalidating the evaluation. This is the digital equivalent of a professor writing exam questions the night before to prevent students from finding last year's answer key.

The benchmark suite deliberately mirrors real-world vulnerability distribution based on the developers' pentesting experience. You won't find 50 SQL injection challenges and 3 CSRF challenges; the distribution reflects what security professionals actually encounter in bug bounty programs and security assessments. This means some vulnerability classes appear only once or twice, which is methodologically honest but limits statistical significance for rare categories.

Gotcha

Validation-benchmarks solves the contamination problem but introduces operational and scope limitations that teams should understand before adoption. The 104-challenge dataset, while carefully curated and previously unreleased, is small by machine learning standards. If you're trying to measure statistical significance across vulnerability categories, you'll struggle—some categories have only 2-3 examples. This isn't enough to confidently distinguish between an AI agent that genuinely understands LDAP injection versus one that got lucky. Comprehensive AI security tool evaluation would ideally use thousands of challenges, but generating that many novel, quality challenges without contamination is economically prohibitive.

The Jeopardy-style CTF format imposes structural constraints that don't reflect many real-world security scenarios. Each challenge has a single, discrete objective: find the flag or answer the specific question. Real penetration testing involves multi-objective campaigns where you might need to establish persistence, pivot across network segments, exfiltrate data covertly, or assess the business impact of vulnerabilities. An AI agent that excels at these benchmarks might still fail at the strategic decision-making required for actual security assessments. The benchmarks also don't test social engineering, physical security, or other non-technical attack vectors.

The deliberately vulnerable code creates serious security risks if mishandled. Every challenge is built to be exploited—there are no defense mechanisms, no WAFs, no rate limiting. If someone accidentally deploys these containers outside isolated test environments, they're running intentionally insecure services on their network. Worse, if developers copy-paste code snippets from these challenges into production applications (thinking they're examples of "working code"), they're directly introducing vulnerabilities. The repository needs prominent warnings, but the fundamental tension remains: useful security benchmarks must contain exploitable code, and exploitable code is dangerous.

Verdict

Use validation-benchmarks if you're developing AI-powered offensive security tools and need uncontaminated test data to measure genuine capability rather than training set memorization. The containerized architecture with injectable flags makes it perfect for CI/CD pipelines that need fresh challenge instances with unique secrets for each test run. This is also ideal if you're publishing academic research on AI security capabilities and need defensible claims about dataset cleanliness—the MAPS canary strings provide evidence your benchmarks weren't in training data. Skip it if you're training human pentesters (these challenges lack the educational scaffolding and explanations humans need), building production security tooling (the deliberately vulnerable code is toxic for copy-paste), or need comprehensive coverage of attack-defense scenarios and multi-stage campaigns. This is strictly a measurement tool for autonomous agents, not a learning platform or realistic security simulation.