LLMmap: Fingerprinting Black-Box Language Models with Minimal Queries
Hook
What if you could identify which LLM powers a mysterious API with just a handful of queries, the same way nmap maps network topology? LLMmap does exactly that, treating language models as fingerprint-able black boxes.
Context
The proliferation of LLM-powered services has created a verification problem. Companies claim to use GPT-4 but might serve cheaper alternatives. Researchers need to verify model provenance for reproducibility. Traditional approaches fail here: you can’t inspect proprietary models, and exhaustive testing burns through API budgets.
LLMmap approaches this by treating LLM identification as a behavioral fingerprinting problem. Just as network scanners identify servers by probing response patterns, LLMmap queries models with carefully crafted prompts and matches their behavioral signatures against known templates. The tool has been rebuilt as a PyTorch-based system that includes behavioral templates for 52 LLMs across HuggingFace, OpenAI, and Anthropic—all without requiring model access beyond standard inference APIs.
Technical Insight
LLMmap’s architecture revolves around three core components: prompt configuration generation, behavioral template creation, and distance-based matching. Unlike traditional classification approaches that require retraining for each new model, LLMmap implements open-set learning where templates can be added dynamically.
The system generates diverse behavioral signatures by varying prompt configurations—combinations of formatting, instruction phrasing, and context that elicit model-specific responses. When you load the pretrained model, the workflow is elegantly simple:
from LLMmap.inference import load_LLMmap
# Load the model with 52 pre-trained templates
conf, llmmap = load_LLMmap('./data/pretrained_models/default/')
# Collect responses from your target model
answers = [
"Response to query 1",
"Response to query 2",
"Response to query 3",
]
# Get ranked predictions with distance metrics
llmmap.print_result(llmmap(answers))
# Output:
# [Distance: 32.96] --> LiquidAI/LFM2-1.2B <--
# [Distance: 40.79] microsoft/Phi-3-mini-128k-instructThe distance metric is the key innovation. Rather than training a closed classifier, LLMmap computes similarity between a target model’s response vector and stored templates. Lower distances indicate better matches. This approach means adding a new model template requires zero retraining of the core inference model:
python add_new_template.py gpt-4.1 1 \
--llmmap_path ./data/pretrained_models/default \
--num_prompt_confs 100The LLM_TYPE parameter (0 for HuggingFace, 1 for OpenAI, 2 for Anthropic) tells the system which client to use. The --num_prompt_confs flag controls template quality—more configurations mean more robust fingerprints but higher computational cost during template creation.
Under the hood, the PyTorch rebuild represents a significant engineering shift from the original implementation. The template matching uses distance metrics to compare behavioral patterns. The system maintains a PromptConfFactory that loads configuration templates from JSON files, allowing for diversity in how queries are presented to models.
For custom use cases, you can build datasets tailored to your model universe. The make_dataset.py script automates corpus generation by querying target LLMs with configurable prompt sets and writing results to JSONL format. This allows you to train domain-specific fingerprinting models when the default 52-model set doesn’t match your needs.
Gotcha
The v0.2 PyTorch rebuild explicitly warns it’s not a one-to-one conversion from the original implementation, meaning the models and procedures differ. If you need exact reproducibility with prior work, this gap matters. The differences in model architecture and training procedures aren’t fully documented.
Template creation currently only supports HuggingFace models via the add_new_template.py script. While the inference model works across HuggingFace, OpenAI, and Anthropic providers for fingerprinting, extending templates for OpenAI and Anthropic models isn’t yet supported. The README notes this “will be extended soon,” but right now you can fingerprint GPT-4 using existing templates but can’t easily add new GPT variants programmatically.
The approach assumes behavioral consistency—that models respond similarly to similar prompts across sessions. Query costs accumulate during template creation; the README notes “the higher —num_prompt_confs the better, but more resource demanding.” Even minimal queries to commercial APIs add up during the template creation process, which requires collecting responses across multiple prompt configurations.
Verdict
Use LLMmap if you need to identify which LLM is behind an API endpoint with minimal queries, want to verify model identity across sessions, or need to extend your fingerprinting capability as new models emerge. The open-set architecture shines when you can add templates incrementally rather than retraining classifiers. It’s particularly valuable when query budgets are constrained and you need identification with a limited number of API calls. Skip it if you have direct metadata access to your models, or if you need to programmatically extend templates for OpenAI/Anthropic models right now—the template extension currently only supports HuggingFace models, though inference works across all three providers.
