CVE-Bench: Benchmarking AI Agents Against Real-World Web Vulnerabilities
Hook
As of early 2025, we can measure how well an LLM writes code or answers questions, but we still lack standardized ways to evaluate whether AI agents can exploit the same critical vulnerabilities that breach companies every year.
Context
The cybersecurity community faces an uncomfortable question: as AI agents become more capable of autonomous action, can they weaponize publicly disclosed vulnerabilities? While researchers have built numerous benchmarks for traditional AI tasks—code generation (HumanEval), mathematics (MATH), reasoning (MMLU)—the offensive security capabilities of AI agents remained largely unmeasured. Existing security benchmarks either use synthetic vulnerabilities divorced from real-world complexity or rely on manual evaluation that doesn’t scale.
CVE-Bench emerges from this gap as the first comprehensive benchmark designed to evaluate AI agents’ ability to exploit actual CVEs from the National Vulnerability Database. Developed by researchers at UIUC’s Kang Lab and recognized with an ICML 2025 spotlight designation, it containerizes 40 critical-severity web application vulnerabilities with automated grading infrastructure. The benchmark doesn’t just ask whether agents can find vulnerabilities in controlled environments—it measures whether they can successfully weaponize the same security flaws that have compromised real systems, from authentication bypasses in authentication servers to remote code execution in content management systems.
Technical Insight
CVE-Bench’s architecture centers on isolated Docker environments running vulnerable applications paired with automated grading containers. Each of the 40 CVEs gets its own containerized setup with a vulnerable web service, supporting infrastructure (databases, file systems), and a grader service that verifies exploit success without requiring human review. The benchmark builds on the Inspect AI evaluation framework, providing a standardized interface for running agent evaluations at scale.
The evaluation model offers two distinct variants that test different aspects of agent capability. The ‘zero_day’ variant provides minimal context—agents receive only the target URL and application type, simulating vulnerability discovery scenarios. The ‘one_day’ variant includes CVE identifiers and descriptions, testing pure exploitation ability when the vulnerability is known. This dual-mode approach lets researchers separate discovery capabilities from exploitation skills, answering whether agents can weaponize known vulnerabilities faster than human defenders can patch.
Setting up an evaluation involves installing the framework and running agents against specific CVE tasks. Here’s a practical example of how you’d evaluate an agent against CVE-2021-3129, a remote code execution vulnerability in Laravel applications:
from inspect_ai import Task, eval
from inspect_ai.solver import generate, system_message
from inspect_ai.model import get_model
# Load CVE-Bench task
task = Task.load("cve_bench/cve-2021-3129")
# Configure your agent with appropriate system prompting
agent = [
system_message(
"You are a security researcher testing a web application. "
"Your goal is to achieve remote code execution. "
"You can make HTTP requests and analyze responses."
),
generate() # Uses your configured LLM to generate actions
]
# Run evaluation with grading
results = eval(
task,
model=get_model("openai/gpt-4"),
solver=agent,
max_messages=50 # Limit agent interaction rounds
)
# Results include automated grading
print(f"Success: {results.success}")
print(f"Exploit type achieved: {results.metadata['exploit_type']}")The automated grading system represents CVE-Bench’s most sophisticated component. Rather than checking for simple flags or predefined outputs, graders verify actual exploit outcomes across eight categories: denial of service, local file read, remote code execution, SQL injection, authentication bypass, privilege escalation, and more. For a file read vulnerability, the grader might verify that the agent successfully retrieved /etc/passwd. For RCE, it confirms arbitrary code execution by checking for specific command outputs. This outcome-based verification means agents must achieve genuine exploitation, not just identify potential vulnerabilities.
The Docker isolation architecture serves dual purposes: security and reproducibility. Each evaluation runs in a fresh container instance, preventing any possibility of agents affecting the host system or other evaluations. The containers snapshot to known-good states between runs, ensuring consistent starting conditions. This isolation proved critical during development—early testing revealed agents occasionally attempting network scans or privilege escalation beyond the intended scope, actions safely contained by Docker boundaries.
CVE-Bench also implements a clever data contamination prevention strategy. Reference exploit solutions exist for only one CVE in the public repository, with the remaining 39 solutions withheld. This prevents AI models from memorizing correct exploitation sequences during training, ensuring the benchmark measures actual reasoning rather than regurgitation. The tradeoff is steeper learning curves for researchers debugging agent failures, but it preserves benchmark validity as models continue training on ever-larger internet corpora.
Gotcha
The architecture limitation hits immediately: CVE-Bench officially supports only amd64 systems, with arm64 marked experimental. If you’re developing on Apple Silicon or ARM-based cloud instances, expect Docker compatibility issues and performance degradation. The Docker images are built for x86_64 architectures, requiring emulation layers that dramatically slow evaluation times—what takes 2 minutes on amd64 might take 15+ minutes on ARM through emulation. This architectural constraint is particularly frustrating given the prevalence of M1/M2/M3 MacBooks in development workflows.
The withheld exploit solutions create a documentation gap that slows research iteration. When your agent fails a CVE task, you’re left guessing whether the failure stems from incorrect vulnerability understanding, wrong exploitation technique, or implementation bugs in your agent framework. The single public reference exploit helps, but debugging the other 39 requires either deep CVE analysis or trial-and-error. This opacity is intentional—the researchers prioritize benchmark integrity over convenience—but it means your first several evaluations will likely involve significant head-scratching. The scope limitation to web application vulnerabilities also means CVE-Bench doesn’t evaluate agents on binary exploitation, cryptographic attacks, or infrastructure vulnerabilities, leaving significant security domains unmeasured.
Verdict
Use CVE-Bench if you’re researching AI safety implications of autonomous agents, developing LLM-based penetration testing tools, or need rigorous evaluation of AI offensive security capabilities on realistic targets rather than toy problems. It’s the only benchmark that tests against actual NVD-listed CVEs with automated grading, making it invaluable for publishing credible security AI research. Skip it if you need ARM-compatible development environments, want training data with solution examples, require coverage of non-web vulnerability types, or need a production-ready penetration testing framework rather than a research evaluation tool. The benchmark excels at measuring what AI can do against real vulnerabilities but deliberately sacrifices convenience for scientific validity.
