Why Your Security ML Benchmarks Are Lying: Inside ml-llm-ensembles' Leakage-Aware Evaluation
Hook
That phishing detector with 99% accuracy in the paper? It's probably just a spam filter that saw your test campaigns during training. Most published security ML results are methodologically bankrupt.
Context
The security ML landscape is drowning in false promises. Researchers claim LLMs revolutionize threat detection with near-perfect accuracy, while practitioners watch these same models collapse under temporal shift, generating thousands of false positives on live traffic. The disconnect isn't just hype—it's methodological malpractice. Academic evaluations use random train-test splits that leak attacker campaigns across folds, measure aggregate AUCPR that hides catastrophic precision collapse at operational thresholds, and evaluate at synthetic 50/50 prevalence instead of the 0.1% base rates security teams face.
The ml-llm-ensembles repository emerged from this frustration. It's not a deployment-ready detector—it's an indictment. Created as the evaluation harness for academic research on ML and LLM ensembles for security, it implements the discipline most papers ignore: flow-aware packet grouping that prevents related network traffic from splitting across train/test, campaign-disjoint evaluation that tests on unseen attacker infrastructure, temporal splits that measure drift, and hard-negative training that forces models to distinguish targeted phishing from bulk spam. The core question it answers: do expensive LLM calls actually outperform gradient boosting when you measure honestly?
Technical Insight
The architecture implements three model tiers with two ensemble strategies, orchestrated through a bash-scripted pipeline with aggressive caching. Traditional ML (XGBoost, TabPFN) forms the base tier, encoder transformers (ModernBERT with frozen and fine-tuned variants) provide the middle layer, and decoder LLMs (local Ollama models plus GPT-4/Claude via API) serve as the expensive top tier. The two ensemble approaches—confidence-gated routing and stacking meta-learners—represent fundamentally different production trade-offs.
The confidence-gating router is the architecturally interesting piece. It escalates predictions to expensive LLM calls only when base ML models express uncertainty, explicitly measuring cost in operational units. The implementation caches LLM predictions using a hash of input text, model identifier, and prompt template, with an ALLOW_LIVE_LLM_CALLS flag that lets you reproduce experiments without burning API credits:
import hashlib
import json
import os
from pathlib import Path
def get_llm_prediction(text, model_name, prompt_template, allow_live=False):
# Generate cache key from input components
cache_key = hashlib.sha256(
f"{text}|{model_name}|{prompt_template}".encode()
).hexdigest()
cache_path = Path(f"cache/llm_predictions/{cache_key}.json")
if cache_path.exists():
return json.loads(cache_path.read_text())
if not allow_live:
raise RuntimeError(
f"Cache miss for {model_name} but ALLOW_LIVE_LLM_CALLS=False. "
"Set to True to make API calls."
)
# Make expensive API call only if explicitly allowed
prediction = call_llm_api(text, model_name, prompt_template)
cache_path.parent.mkdir(parents=True, exist_ok=True)
cache_path.write_text(json.dumps(prediction))
return prediction
This cost discipline is rare in academic code. Most papers treat API calls as free, making their approaches uneconomical to reproduce. Here, you can rerun experiments using cached predictions from the original paper's evaluation.
The leakage-aware splitting is where methodological rigor shines. For network flow evaluation on the CTU-13 botnet dataset, the framework implements campaign-disjoint splits—ensuring training and test sets contain entirely different attacker infrastructure. For phishing, it enforces temporal splits where test data arrives chronologically after training data, exposing drift that random splits hide:
def campaign_disjoint_split(df, campaign_col='botnet_id', test_size=0.3):
"""Split by campaign/host, not individual samples."""
unique_campaigns = df[campaign_col].unique()
np.random.shuffle(unique_campaigns)
split_idx = int(len(unique_campaigns) * (1 - test_size))
train_campaigns = unique_campaigns[:split_idx]
test_campaigns = unique_campaigns[split_idx:]
train_mask = df[campaign_col].isin(train_campaigns)
return df[train_mask], df[~train_mask]
def temporal_split(df, time_col='timestamp', test_size=0.3):
"""Chronological split - test data arrives AFTER training data."""
df_sorted = df.sort_values(time_col)
split_idx = int(len(df_sorted) * (1 - test_size))
return df_sorted.iloc[:split_idx], df_sorted.iloc[split_idx:]
The multi-representation isolation on Mirai botnet traffic is clever experimental design. The same packet capture is evaluated at three granularities—raw packets, aggregated flows, and session-level summaries. This isolates feature representation as the experimental variable, proving that encoder transformers can recover signal from intermediate representations that zero-shot LLMs miss when you serialize network statistics to text. The result: fine-tuned ModernBERT matches LLM performance at 1/100th the inference cost.
For phishing detection, the framework decomposes evaluation by threat taxonomy: ham (legitimate email), bulk spam (mass campaigns), and targeted phishing. Most papers report pooled AUCPR across all classes, which hides that their "phishing detector" just filters bulk spam—the easy negatives. The hard-negative training protocol forces models to distinguish targeted attacks from sophisticated spam:
def compute_hard_negative_metrics(y_true, y_pred, y_text, spam_keywords):
"""Separate metrics for bulk spam vs targeted phishing."""
# Identify bulk spam by keyword heuristics
bulk_spam_mask = y_text.str.lower().str.contains('|'.join(spam_keywords))
metrics = {
'overall_aucpr': average_precision_score(y_true, y_pred),
'bulk_spam_aucpr': average_precision_score(
y_true[bulk_spam_mask], y_pred[bulk_spam_mask]
),
'targeted_aucpr': average_precision_score(
y_true[~bulk_spam_mask], y_pred[~bulk_spam_mask]
),
'fpr_at_operational_threshold': compute_fpr_at_threshold(
y_true, y_pred, threshold=0.5, benign_count=1000
)
}
return metrics
This operational metric—false positives per thousand benign samples—directly measures SOC burden. Academic papers optimize for AUCPR across all thresholds, but practitioners care about precision at the single threshold where they generate alerts. A model with 99% AUCPR can still produce 50 false alerts per thousand legitimate emails, rendering it useless.
The orchestration uses numbered bash scripts (01_preprocess.sh, 02_train_ml.sh, 03_evaluate_ensembles.sh) with three execution modes: synthetic generates tiny datasets for CI, smoke validates the pipeline on small samples, and paper reproduces full experiments. This trades modularity for reproducibility—you can't easily compose new experiments, but you can exactly replicate published results.
Gotcha
This is an evaluation harness, not deployable infrastructure. There's no model serving layer, no streaming ingestion, no A/B testing framework, no monitoring. Taking this to production requires rewriting everything. The hardcoded experiment scripts with sequential numbering make extension painful—adding a new model or dataset means duplicating boilerplate rather than composing modular components. If you need to evaluate a different architecture or dataset, expect to fork substantial code.
The network flow evaluation stops at historical datasets: Mirai from 2016 and CTU-13 from 2011. Modern C2 protocols—DNS tunneling, TLS 1.3 encrypted SNI, QUIC, cloud-native lateral movement—are unrepresented. Transferability claims about LLM performance on network traffic may not hold against contemporary attacks. More critically, there's zero adversarial robustness evaluation. No evasion attacks, no prompt injection testing on the LLM tier, no measurement of how attackers could game the confidence router with crafted inputs that trigger low ML confidence but remain malicious. The stacking meta-learner requires holding out folds for probability calibration, fragmenting already-small security datasets and making accuracy gains fragile to distribution shift.
Verdict
Use if: you're a security ML researcher who needs to evaluate LLM claims honestly, a practitioner building detectors who wants to understand why academic benchmarks don't transfer to production, or a red teamer studying how blue-team ML actually fails under realistic conditions. Fork the split strategies, hard-negative training, and operational metrics—they're immediately applicable. The confidence-gating router is worth prototyping if you face explicit cost constraints on model calls. Skip if: you need a deployment-ready detector (this is research code), you want to evaluate modern network protocols or adversarial robustness (datasets are dated and threat models incomplete), or you need modular components for experimentation (the bash-orchestrated numbered scripts resist composition). The value is methodological discipline that exposes how most security ML papers are wrong, not a tool you ship.