Cyber Defense Benchmark: Testing Whether LLMs Can Hunt Threats with SQL
Hook
An AI agent that can write perfect Python but fails at basic SQL threat hunting is about as useful as a junior analyst who can't grep logs. Most security AI benchmarks test the wrong skills.
Context
The security industry has a dirty secret: most 'AI-powered threat detection' products don't let the AI do actual investigation. They pattern-match on prose indicators of compromise, call pre-built APIs, or operate entirely in natural language where humans still need to validate every finding against structured telemetry. When SecOps teams actually hunt threats, they're writing SQL (or SPL, or KQL) against gigabytes of process logs, network flows, and file events—looking for temporal correlations, statistical anomalies, and multi-stage attack chains that span multiple log sources.
The cyber_defense_benchmark repository from Simbian tackles this gap head-on. Instead of testing whether LLMs can answer multiple-choice questions about MITRE ATT&CK or generate Sigma rules from prose descriptions, it frames threat hunting as a reinforcement learning problem: here's an in-memory SQLite database full of security telemetry, here's the schema, now write queries to find the attacker. The action space is unconstrained SQL. The observation space is raw query results. No tool-use scaffolding, no retrieval APIs, no guardrails. Just the model, the data, and the question of whether language models can actually reason about structured investigations or if they've been coasting on framework code that does the heavy lifting.
Technical Insight
The architecture is refreshingly minimal. At the core is ThreatHuntEnv, a gymnasium-compatible environment that wraps SQLite. Each episode starts with reset(), which loads a dataset of security logs into an in-memory database with tables like process_events, network_connections, file_modifications, and registry_changes. The schema mimics real SIEM data—nullable fields, inconsistent timestamp formats, foreign-key-ish relationships between process IDs and parent processes that aren't actually enforced.
The agent interacts through step(action), where action is a SQL query string. The environment executes the query, serializes the results into text (because LLMs can't parse structured data natively), and returns it as the observation. The episode continues until the agent outputs a special 'done' signal or hits a step limit. Ground truth is a set of row IDs marked as 'flags'—these are log entries that participate in the actual attack chain, pre-labeled by security analysts. Scoring is simple set intersection: what percentage of flagged rows did your queries retrieve?
Here's what the reference harness looks like in practice:
from cyber_defense_benchmark import ThreatHuntEnv, ReActHarness
import litellm
# Initialize environment with a dataset
env = ThreatHuntEnv(dataset_path="./datasets/apt_simulation.db")
# The harness uses a prompt template with schema docs and few-shot examples
harness = ReActHarness(
model="gpt-4",
max_steps=20,
budget_dollars=5.0
)
# Run an episode
obs = env.reset()
query_trace = []
for step in range(harness.max_steps):
# Model generates next SQL query based on conversation history
action = harness.get_action(obs)
if action == "DONE":
break
# Execute query and get results
obs, reward, done, info = env.step(action)
query_trace.append((action, len(obs['rows'])))
if done:
break
# Scoring: did we retrieve the flagged rows?
flag_coverage = info['flags_retrieved'] / info['total_flags']
print(f"Coverage: {flag_coverage:.2%}")
print(f"Cost: ${info['total_cost']:.4f}")
The ReAct harness follows the 'reasoning and acting' pattern: the prompt includes the schema documentation, previous queries and their results, and asks the model to reason about what to check next. A typical trajectory might start with broad reconnaissance (SELECT COUNT(*) FROM process_events WHERE parent_name = 'powershell.exe'), then narrow down based on anomalies (SELECT * FROM network_connections WHERE process_id IN (...) AND remote_port = 4444), then correlate across tables to map out the attack chain.
What makes this interesting architecturally is what's missing. There's no retrieval-augmented generation pulling relevant hunting heuristics from a vector database. No tool-use API where the model calls search_for_lateral_movement() and a handcrafted function does the real work. No pre-chunked context windows with 'relevant' logs already filtered. The model has to construct joins, reason about temporal ordering (WHERE timestamp BETWEEN ... AND ...), and understand that process trees require recursive CTEs or self-joins. It's testing compositional reasoning about structured data, not memorized security patterns.
The benchmark runner orchestrates evaluation across multiple models with cost tracking:
from cyber_defense_benchmark import BenchmarkRunner
runner = BenchmarkRunner(
datasets=["apt_sim_01", "apt_sim_02", "ransomware_01"],
models=["gpt-4", "gpt-3.5-turbo", "claude-2"],
harness_config={"max_steps": 20},
budget_per_model=10.0,
output_dir="./results"
)
results = runner.run()
# Results include a Pareto frontier plot: coverage vs. cost
runner.plot_efficiency_frontier(results)
The output explicitly acknowledges that GPT-4 might achieve 85% flag coverage but cost $8 per episode, while GPT-3.5-turbo hits 65% for $0.50. That honesty—that not all performance gains are worth the API spend—is rare in ML benchmarks and valuable for practitioners making deployment decisions.
The schema design is deliberately realistic in its messiness. The process_events table has fields like command_line, parent_process_id, user, and timestamp, but parent_process_id might be NULL for orphaned processes, timestamps might be ISO8601 strings or Unix epochs depending on the log source, and there's no cascading foreign key to guarantee the parent actually exists in the table. This forces agents to write defensive SQL with COALESCE, type casts, and LEFT JOINs rather than assuming clean relational semantics.
Gotcha
The biggest limitation is the dataset: the sample included in the repo is tiny (probably a few hundred events based on file sizes), and the full benchmark datasets require contacting Simbian directly. You can't just clone and run comprehensive evals without negotiating data access, which limits reproducibility and makes it harder for academic researchers to iterate.
Scoring is also simplistic in ways that matter. Because it's pure retrieval coverage (did you touch the flagged rows?), an agent that writes SELECT * FROM process_events on every step would score perfectly despite being useless. There's no penalty for false positives, no measurement of query efficiency, and no credit for retrieving 'interesting' but unlabeled anomalies that a human analyst would want to investigate further. Real threat hunting cares about precision—surfacing 5 relevant events from 10,000 is the job, not dumping all 10,000 for someone else to triage.
The reference harness also lacks memory sophistication. It's a linear conversation with no explicit graph of findings, no persistent hypotheses about attack stages, and no reflection loop where the agent critiques its own queries. Human analysts don't just write sequential SQL—they build mental models ('this looks like lateral movement, so I should check for PsExec patterns next'), backtrack when leads go cold, and maintain a hypothesis tree. The benchmark doesn't test whether agents can do this meta-cognition.
Finally, the environment is completely static. There's no adversarial evaluation where attackers adapt their TTPs after the model learns patterns, no injection of realistic noise (benign admin activity that looks suspicious), and no query performance constraints. Production SIEM queries time out, hit cardinality limits, or crash the database if you join three large tables without indexes. None of that pressure exists here, so high benchmark scores might not transfer to real infrastructure.
Verdict
Use if: You're a security AI researcher who needs a reproducible testbed for evaluating whether LLMs can reason about structured telemetry rather than just pattern-match on prose indicators, or you're comparing agentic frameworks (AutoGPT vs. LangChain agents vs. custom loops) on a controlled security task where you want to measure actual query construction ability. The SQL-as-action-space design forces honest evaluation of compositional reasoning, and the reference architecture gives you a solid skeleton to adapt with your own datasets and scoring logic. Skip if: You're a practitioner trying to evaluate production threat-hunting agents—the toy dataset, binary scoring that rewards shotgun queries, and lack of adversarial drift make this a poor proxy for real deployments. Build a custom eval against your actual SIEM with known incidents instead, or wait for the authors to release production-scale datasets with adversarial scenarios and precision-aware metrics.