Building Statistically Robust LLM Rankings with Pairwise Comparisons

Hook

Ask an LLM to rank 100 documents and you'll get wildly different results each time—unless you fundamentally change how you're asking the question. That's exactly what raink does.

Context

Large language models are terrible at ranking documents consistently. Give GPT-4 a list of 50 code patches and ask it to rank them by relevance to a CVE advisory, and you'll face four brutal problems: nondeterministic outputs that change between runs, context windows too small for all documents, incomplete responses that cut off mid-ranking, and subjective scoring that makes reproducibility impossible.

This isn't just an academic problem. Security researchers at Bishop Fox encountered this while building 'Patch Perfect,' a system for mapping code diffs to CVE advisories during vulnerability research. When you're trying to identify which commits might address a security vulnerability, you need rankings that are both semantically intelligent and statistically reliable. Traditional vector similarity search gets you partway there, but it misses the nuanced semantic understanding LLMs provide. Prompting an LLM directly gives you that understanding but loses reliability. raink bridges this gap by transforming the ranking problem from subjective scoring into pairwise comparisons, then aggregating results across multiple randomized tournament rounds—a technique grounded in academic research on LLM ranking that produces statistically robust results.

Technical Insight

raink's core insight is deceptively simple: instead of asking an LLM to score or rank all documents at once, break the problem into many small pairwise comparisons and aggregate the results. The architecture implements a tournament-style ranking system that runs multiple rounds of randomized head-to-head matchups.

Here's how you'd use it to rank security advisories against code changes:

// Example: Ranking code diffs by relevance to a CVE
package main

import (
    "fmt"
    "os/exec"
)

func main() {
    // Your prompt defines the comparison criteria
    prompt := `You are analyzing security patches. 
    Given two code diffs, determine which one is more likely 
    to address CVE-2024-1234 (buffer overflow in authentication handler). 
    Output only 'A' or 'B'.`
    
    // raink handles batching, randomization, and aggregation
    cmd := exec.Command("raink",
        "-p", prompt,
        "-i", "patches.json",  // Your documents as JSON array
        "-r", "10",            // 10 ranking rounds for statistical reliability
        "-b", "5")             // Compare 5 items per batch
    
    output, _ := cmd.Output()
    fmt.Println(string(output)) // Returns ranked JSON
}

The input JSON is straightforward—an array of documents with content and optional metadata:

[
  {"id": "patch-1", "content": "diff --git a/auth.c...\n+ if (len > MAX_SIZE) return;"},
  {"id": "patch-2", "content": "diff --git a/logger.c...\n+ fix typo in comment"},
  {"id": "patch-3", "content": "diff --git a/auth.c...\n+ memcpy bounds check"}
]

Under the hood, raink's architecture solves each of the four LLM ranking problems systematically. For nondeterminism, it runs multiple independent ranking rounds (controlled by -r) and aggregates results statistically—similar to ensemble methods in machine learning. For context window limits, it uses batch sizes (via -b) that fit comfortably within token limits while maintaining comparison quality. For incomplete outputs, pairwise comparisons are inherently simple: the LLM only needs to output 'A' or 'B', drastically reducing the chance of truncation. For subjective scoring, relative comparisons eliminate the problem entirely—'which is better?' is more reliable than 'score this 1-10.'

The batch processing algorithm is particularly clever. raink doesn't just split your 100 documents into sequential groups. It randomizes document distribution across batches in each round, ensuring every document gets compared against a diverse set of peers. This randomization is critical—it prevents positional bias where documents always appear in the same context and eliminates ordering effects that plague traditional ranking approaches.

The tool uses hash-based IDs internally for efficient tracking across rounds. When aggregating results, raink counts pairwise wins across all rounds—if patch-1 beats patch-2 in 7 out of 10 rounds, that's statistically meaningful signal. The final ranking emerges from these accumulated comparisons, similar to how ELO ratings work in chess.

Parallelization is built-in but constrained by OpenAI's rate limits. raink makes concurrent API calls across different batches within a round, dramatically speeding up processing. A typical run with 100 documents, 10 rounds, and batch size 5 completes in under 2 minutes—impressive considering the number of LLM calls involved. The tool handles backoff and retry logic internally, though you'll want to monitor API costs as they scale with (num_items / batch_size) * num_rounds.

The output is clean JSON with documents sorted by aggregated ranking score:

{
  "ranked_documents": [
    {"id": "patch-3", "content": "...", "score": 0.89},
    {"id": "patch-1", "content": "...", "score": 0.72},
    {"id": "patch-2", "content": "...", "score": 0.31}
  ],
  "metadata": {
    "total_rounds": 10,
    "batch_size": 5,
    "total_comparisons": 450
  }
}

This makes integration into larger pipelines trivial—parse the JSON, take the top N results, and feed them into your next processing stage. For security researchers, that might mean automated patch analysis. For content teams, it could be relevance ranking of documentation. The architecture is domain-agnostic; the prompt defines the semantics.

Gotcha

The OpenAI hardcoding is raink's most significant limitation. Despite being written in Go and architecturally provider-agnostic, it currently only works with OpenAI's API. If you're using Anthropic's Claude, local models via Ollama, or any other provider, you're out of luck until the maintainers implement the roadmap item for multi-provider support. This is particularly frustrating for security-sensitive applications where you might need on-premises LLM deployment.

Cost and latency scale in ways that aren't immediately obvious. With 100 documents, 10 rounds, and batch size 5, you're making roughly 200 API calls. At current GPT-4 pricing, that's anywhere from $2-10 depending on document length and model choice. For one-off analyses, that's reasonable. For continuous ranking in a production pipeline, costs balloon quickly. There's no caching layer, so re-ranking the same documents costs the same as the first run—even if only one document changed. The roadmap mentions batch API support, but until that's implemented, you're paying full freight for every ranking operation. Similarly, those 200 API calls take time even with parallelization. If you need sub-second ranking for interactive applications, raink won't cut it.

The statistical robustness also has limits. While multiple rounds combat nondeterminism, raink doesn't expose confidence intervals or statistical significance metrics. You get aggregated scores, but no indication of whether a 0.72 versus 0.68 difference is meaningful or noise. For research applications, you might want to export raw pairwise comparison results and run your own statistical analysis—but that's not currently supported.

Verdict

Use raink if you're doing security research, vulnerability analysis, or any domain where you need semantically intelligent document ranking that's reproducible across runs, and you can tolerate OpenAI API costs and latency. It's particularly valuable when you have 50-500 documents to rank—small enough that API costs are manageable, large enough that manual ranking is infeasible. The pairwise comparison approach genuinely solves LLM reliability problems that naive prompting can't address. Skip it if you're working with small datasets (under 20 items) where manual review or simple prompting suffices, need real-time results for interactive applications, require non-OpenAI models for cost or compliance reasons, or are doing pure similarity search where vector databases would be faster and cheaper. Also skip if you need sub-dollar ranking operations—raink trades cost for reliability, and that's not always the right tradeoff.