VulnHunter: How Capital One Built a Self-Falsifying AI Security Scanner

Hook

Most static analysis tools find thousands of vulnerabilities. VulnHunter finds fewer—by forcing its AI to write working exploits, then argue against itself in a parallel adversarial pipeline that kills 80% of findings before they reach human analysts.

Context

Every security team knows the pain: your SAST tool flags 3,000 potential SQL injections, but 2,950 are false positives because the tool can't reason about whether the vulnerability is actually exploitable. It saw user input near a database query and panicked. Traditional static analysis works backwards from dangerous "sinks" (database calls, system commands, file operations) and traces dataflow to find user-controlled "sources." This sink-first approach is mathematically sound but pragmatically disastrous—it floods teams with theoretical vulnerabilities that may be protected by validation layers, authentication checks, or framework-level sanitization that the pattern matcher can't understand.

Capital One's VulnHunter inverts this entire paradigm. Released as an open-source Python framework, it orchestrates Claude Opus models to perform attacker-first analysis: start at entry points an adversary controls (API parameters, file uploads, HTTP headers), trace forward through the codebase until reaching a dangerous operation, then spawn a separate adversarial agent that actively tries to prove the vulnerability is unexploitable. Only findings that survive this gauntlet of self-critique reach your security team. It's expensive—forward analysis with multi-pass LLM reasoning burns tokens at 10-100x the rate of traditional SAST—but Capital One's thesis is that exploitability proofs are worth the compute cost when false positives destroy your team's ability to prioritize real threats.

Technical Insight

VulnHunter's architecture separates into three independent Claude "skills" implemented as structured prompt workflows in versioned Markdown files. The scanner (/vulnhunt) performs forward taint analysis starting from attacker-controlled entry points. Rather than searching for dangerous patterns, it reasons about dataflow from HTTP parameters, file inputs, or network data through the application logic until reaching security-sensitive operations. Here's what the entry point identification looks like in practice:

# VulnHunter starts here, not at the SQL query
@app.route('/user/profile', methods=['POST'])
def update_profile():
    user_id = request.form.get('user_id')  # Attacker-controlled source
    bio = request.form.get('bio')          # Another attack vector
    
    # Forward trace follows dataflow through helpers
    sanitized_bio = clean_html(bio)  # Does this actually prevent XSS?
    
    # Eventually reaches the sink
    db.execute(f"UPDATE users SET bio='{sanitized_bio}' WHERE id={user_id}")

Traditional SAST sees the f-string SQL query and immediately flags it. VulnHunter's scanner agent reasons forward: "user_id flows from request.form without sanitization into a SQL query—can I construct a working exploit?" The agent then attempts to generate a proof-of-concept, determining whether user_id='1 OR 1=1--' actually bypasses authentication or executes unintended queries.

The critical innovation arrives in the falsification pipeline. After the scanner identifies a potential vulnerability, VulnHunter spawns parallel reasoning chains with a single instruction: invalidate this finding. The adversarial agent searches for missing context—authentication middleware that restricts the endpoint, input validation in framework decorators, ORM protections that parameterize queries automatically. This structured self-critique is implemented through prompt engineering rather than formal methods:

# Simplified falsification prompt structure
You previously identified: SQL injection in update_profile via user_id parameter

Your task: Prove this vulnerability CANNOT be exploited by finding:
1. Authentication/authorization controls that prevent access
2. Input validation or sanitization that neutralizes the payload  
3. Framework protections (ORM escaping, prepared statements) between source and sink
4. Runtime protections (WAF rules, query monitors) that would block exploitation

If you find ANY of these, the vulnerability is invalid. Be maximally adversarial to the original finding.

Only vulnerabilities that survive this adversarial gauntlet proceed to the fixer skill (/vulnhunter-fix), which implements test-driven remediation. The fixer doesn't just patch the code—it first writes a failing exploit test that demonstrates the vulnerability, applies a fix, then verifies the test now passes. This forces concrete proof-of-concept generation rather than theoretical patching:

# Auto-generated exploit test before fix
def test_sql_injection_user_id():
    malicious_payload = "1 OR 1=1--"
    response = client.post('/user/profile', data={'user_id': malicious_payload})
    # This should fail before fix, pass after
    assert "admin@example.com" not in response.data  # Shouldn't leak other users

The verifier skill operates with intentionally neutered capabilities—no bash execution, no network access, read-only filesystem. This architectural constraint prevents automation bias where the verifier simply confirms what the fixer claimed because it has access to the same tools and context. The verifier receives only the original vulnerability report, the proposed fix, and the test results, forcing independent validation.

The headless runtime (vulnhunter-agent) wraps these prompt workflows in Python automation for CI/CD integration, while the harness provides batch orchestration across multiple repositories. Critically, VulnHunter implements "LLM-as-judge" benchmarking where Claude itself evaluates whether detected vulnerabilities match ground truth datasets—a pragmatic solution given the lack of standardized security benchmarks, though it creates circular dependencies where model quality determines both detection and validation accuracy.

The prompt-only architecture makes the core logic auditable and model-agnostic in theory. Vulnerability detection logic lives in SKILL.md files rather than buried in Python classes, so security teams can version control and review the exact reasoning chains the AI follows. In practice, Capital One's documentation explicitly warns that prompts are heavily optimized for Claude Opus—porting to GPT-4 or open-source models would require substantial re-tuning rather than plug-and-play substitution.

Gotcha

VulnHunter's hard dependency on Claude Opus through Anthropic's Cyber Verification Program creates a brutal cost barrier. Forward analysis with multi-pass reasoning consumes 10-100x more tokens than traditional SAST pattern matching. Scanning a 50K line codebase with dozens of API endpoints could burn $200-500 in API costs per run—feasible for Capital One's enterprise agreement, potentially unsustainable for startups or open-source projects. There's no graceful degradation to cheaper models; the prompts are tuned specifically for Opus-level reasoning, and the documentation explicitly states other models haven't been validated.

The prompt-based architecture means core vulnerability logic is untyped natural language subject to hallucination and edge case failures. Adversarial inputs in comments, variable names, or docstrings could potentially manipulate the analysis—imagine a developer comment saying "this function is safe from SQL injection" causing the falsification engine to incorrectly invalidate a real vulnerability. More fundamentally, the falsification engine can only reason about security controls it can statically analyze. Runtime protections like Web Application Firewalls, rate limiting, or cloud-provider security groups are invisible to the agent, leading to false positives where VulnHunter reports exploitable vulnerabilities that production environments actually block. The system also scales poorly to large attack surfaces—analyzing 200 API endpoints means exploring exponentially branching dataflow paths, and there's no clear batching or prioritization strategy documented for massive codebases.

Verdict

Use if: You're drowning in false positives from Semgrep/Bandit and need exploitability proofs to prioritize remediation, you already have a Claude Opus enterprise agreement and can absorb $500-2000/month in additional API costs, your codebase is medium-sized (10K-100K lines) with well-defined entry points, and your security team values lower false-positive rates over comprehensive coverage. VulnHunter genuinely advances LLM-based security tooling by forcing proof-of-concept generation rather than pattern matching—it's valuable when human analyst time is more expensive than compute costs. Skip if: You need open-source or self-hosted scanning for regulated industries, you're operating on tight budgets without enterprise LLM agreements, your codebase exceeds 500K lines with complex framework abstractions, you require integration with existing SAST/DAST toolchains or vulnerability databases, or you need formal verification guarantees rather than probabilistic LLM reasoning. The Anthropic lock-in and research-grade maturity make this impractical for most teams—you're paying premium costs for a prototype that assumes you trust adversarial AI reasoning over deterministic analysis.