> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

HoF-Bench: Testing Whether LLMs Can Spot Vulnerabilities When You Point Them at the Bug

[ View on GitHub ]

HoF-Bench: Testing Whether LLMs Can Spot Vulnerabilities When You Point Them at the Bug

Hook

Most vulnerability detection benchmarks ask 'can your scanner find the needle in the haystack?' HoF-Bench skips that question entirely—it hands you the exact haystack location and asks 'can you explain why this is sharp?'

Context

Evaluating automated vulnerability scanners has always been methodologically messy. Traditional benchmarks either use synthetic test cases that don't reflect real-world complexity (NIST's Juliet Suite) or scrape Git commits for CVE fixes without verifying the vulnerable code is actually reachable or correctly labeled. When researchers started experimenting with LLM-based security scanners in 2023-2024, the evaluation problem got worse: these tools operate on natural language representations of code, making their behavior sensitive to prompt engineering, context window limitations, and model non-determinism.

HoF-Bench emerged from AISLE's offensive security research as a controlled benchmark for testing whether language models can identify vulnerabilities when scope is pre-narrowed. Rather than measuring autonomous discovery across entire codebases, it freezes eight open-source repositories at vulnerable commits and provides scanners with exact file paths and line ranges where known CVEs exist. The benchmark includes 95 CVEs that AISLE researchers previously discovered in projects like OpenSSL, PHP, and various JavaScript libraries. This design choice—treating vulnerability detection as a rediscovery and explanation problem rather than a search problem—creates a fundamentally different evaluation than most existing benchmarks. It's testing model reasoning ability in a constrained scope, not real-world offensive security workflows where you don't know where bugs hide.

Technical Insight

Ground Truth

Evaluator

Scanner Pipeline

code snippets

candidate findings

validated findings

blame output

yes

no

repo snapshot

fix SHA

CVE Task Queue

8 repos × vulnerable commits

Context Generator

ripgrep keyword extraction

Detection Stage

LLM vulnerability scan

Skeptical Triage

adversarial LLM filter

Git-Blame Judge

line attribution check

Lines modified

in fix commit?

True Positive

False Positive

Frozen Repos

+ Fix Commits

System architecture — auto-generated

The benchmark's architecture implements a three-stage pipeline that mirrors AISLE's production scanner but reimplements it as a standalone harness. The first stage, context generation, uses ripgrep to extract relevant code snippets based on keyword searches. The second stage sends these snippets plus the target file range to an LLM with a detection prompt. The third stage applies adversarial triage—a second LLM call that challenges findings with counterarguments to filter false positives.

What makes the judge interesting is its use of git-blame for ground truth verification. Rather than relying on string matching or manual vulnerability annotations, it checks whether reported line numbers trace back to the commit that eventually fixed the CVE. Here's the core logic from the evaluation harness:

def judge_finding(repo_path, reported_lines, fix_commit_sha):
    """Verify finding by checking if reported lines were changed in the fix."""
    for line_num in reported_lines:
        blame_output = subprocess.run(
            ['git', 'blame', '-L', f'{line_num},{line_num}', 
             f'{fix_commit_sha}^', '--', target_file],
            capture_output=True, cwd=repo_path
        )
        blamed_commit = parse_commit_from_blame(blame_output.stdout)
        
        # Check if this line existed before fix and was modified in fix
        if blamed_commit != fix_commit_sha:
            diff_output = subprocess.run(
                ['git', 'diff', f'{fix_commit_sha}^', fix_commit_sha, 
                 '--', target_file],
                capture_output=True, cwd=repo_path
            )
            if line_num in extract_changed_lines(diff_output.stdout):
                return True
    return False

This approach sidesteps the labeling problem entirely—you never need humans to mark which specific lines contain vulnerabilities. The fix commit becomes the source of truth. If a scanner reports lines that the security patch later modified, it found something real.

The adversarial triage stage is architecturally distinct from confidence scoring. Instead of asking the model 'how sure are you?', it prompts with skeptical challenges:

triage_prompt = f"""
A scanner reported this potential vulnerability:
{finding_description}

Code context:
{code_snippet}

Skeptical review: Consider these alternative explanations:
- Could this be a false positive due to missing context?
- Are there input validations elsewhere that neutralize this?
- Does the surrounding code handle edge cases safely?

Provide a final verdict: CONFIRMED or FALSE_POSITIVE
"""

This mimics how offensive researchers stress-test findings before reporting them to vendors. The reference implementation shows that triage reduces false positives by 35-40% across models while only filtering 8-12% of true positives, suggesting the adversarial framing helps models engage System 2 reasoning rather than pattern matching.

The benchmark structure separates dataset from scanner implementation. The core artifact is a JSON file mapping CVE identifiers to repository snapshots, vulnerable file paths, line ranges, and fix commits:

{
  "CVE-2024-XXXX": {
    "repo": "openssl/openssl",
    "commit": "abc123...",
    "files": [
      {
        "path": "ssl/record/rec_layer_s3.c",
        "lines": [445, 467],
        "fix_commit": "def456..."
      }
    ],
    "cve_description": "Buffer over-read in DTLS record parsing"
  }
}

Third parties can evaluate their own tools against this ground truth without adopting the LLM-based scanner. You just need to output findings in the expected format and run the judge.

The reference results include 3,800 evaluation records (10 models × 95 CVEs × 4 repeated runs × 2 conditions with/without context generation), explicitly measuring variance from model non-determinism. Most benchmarks report single-run accuracy; HoF-Bench treats detection as a stochastic process and measures pass@k rates. A model that catches a CVE in 3 out of 4 runs at temperature 0.7 is behaving differently than one with deterministic detection, and the benchmark captures that distinction.

The context generation stage uses basic text search rather than static analysis. For a C vulnerability in SSL record handling, it might ripgrep for 'DTLS_record_layer' and 'buffer_read' then feed matching snippets to the LLM. This works across languages (the benchmark includes C, PHP, JavaScript) but misses dataflow relationships. A proper static analyzer would trace tainted data paths from input parsing to vulnerable operations; the reference scanner just hopes relevant code appears in grep results.

Gotcha

The most significant limitation is that scanners receive file paths and line ranges from ground truth—they're told exactly where to look. This transforms the task from 'find vulnerabilities in this codebase' to 'explain what's wrong with these 50 lines of code'. In real offensive security work, you analyze unfamiliar projects without knowing whether vulnerabilities exist or where to focus attention. Scope narrowing is the hard part; once you've identified a suspicious function, confirming exploitability is comparatively straightforward. HoF-Bench evaluates the second half while skipping the first.

The benchmark includes only 95 CVEs, all discovered by AISLE's own research. This creates selection bias—these are vulnerabilities their existing systems found well enough to disclose publicly. Edge cases that confused their scanners, or vulnerability classes their approach misses entirely, won't appear in the dataset. With fewer than 100 samples, statistical power for comparing models is limited. A scanner that scores 5 percentage points higher might just be overfitting to these specific bug patterns. The git-blame judge assumes vulnerabilities are fixed in single commits touching specific lines. Multi-component vulnerabilities spanning files fixed in different patches, or security issues requiring semantic understanding beyond line-level changes, won't be correctly evaluated. There's no discussion of how the benchmark handles logic bugs where the fix adds validation code in a different function rather than modifying vulnerable lines.

Verdict

Use HoF-Bench if you're researching prompt engineering for vulnerability detection, comparing LLM reasoning ability on security-relevant code, or developing hybrid scanners that combine automated scope reduction with LLM-based analysis. The git-blame judge and frozen snapshots provide reproducible ground truth that's rare in security benchmarks, and the explicit modeling of repeated runs addresses non-determinism that other evaluations ignore. The small size makes it practical for academic experiments with limited compute budgets. Skip if you're building autonomous scanners for offensive security work or red team engagements—the pre-narrowed scope doesn't test the discovery capabilities that distinguish useful tools from toys. Also skip if you need vulnerability detection across diverse CWE categories; eight repositories can't provide comprehensive coverage of bug classes. For production security tooling evaluation, combine this with whole-repository benchmarks that measure end-to-end discovery without hints, or consider OSS-Fuzz integration for runtime vulnerability validation.