DecBench: Measuring Decompilation Quality by Recompiling the Output
Hook
Most decompiler benchmarks compare source code similarity. DecBench does something more radical: it recompiles the decompiled output with the original compiler and measures how close the assembly gets to the original binary.
Context
Decompilation—recovering source code from compiled binaries—is still more art than science. Tools like Ghidra, IDA Pro, and Binary Ninja can transform raw assembly into C-like pseudocode, but measuring their accuracy has always been subjective. Does 'good' decompilation mean readable code? Semantically equivalent logic? Structurally similar control flow? The answer depends on who's asking.
Researchers face a measurement problem. Without ground truth, evaluating decompiler improvements requires manual inspection of thousands of functions or brittle semantic equivalence checks that fail on compiler optimizations. DecBench tackles this by introducing a three-dimensional scoring system where one metric—byte_match—takes a genuinely novel approach: it recompiles the decompiled code using the same compiler, optimization level, and architecture as the original binary, then performs assembly-level diffing with normalized operands. This measures something no other benchmark captures: whether a decompiler preserves enough semantic detail that recompilation can recreate functionally identical machine code.
Technical Insight
DecBench's architecture separates compilation, decompilation, and evaluation into distinct stages with checkpointing at project boundaries—a production-grade decision acknowledging that multi-hour benchmark runs crash. The compile stage uses gcc and MinGW cross-compilers to generate binaries at optimization levels O0 through O3, but crucially, it preserves preprocessed .i files and DWARF debug symbols. This isn't just bookkeeping: the GED (Graph Edit Distance) metric extracts source-side control flow graphs exclusively from preprocessed files, not original C, because parsing ambiguities in raw source would skew CFG comparisons.
The decompile stage wraps heterogeneous tools through a harness abstraction. Each decompiler (angr, Ghidra, IDA, Binary Ninja) gets normalized invocation and output collection. This design acknowledges reality: decompilers have wildly different interfaces—command-line scripts, GUI automation, Python APIs. The harness hides these details so evaluation can run uniformly.
Where DecBench gets interesting is the evaluate stage's three independent metrics. GED uses rust-joern to parse both source and decompiled code into control flow graphs, then cfgutils computes graph edit distance—how many node/edge insertions, deletions, or substitutions separate the two. This measures structural similarity: did the decompiler recover the same branching logic, loop nesting, and function call patterns?
The type_match metric compares recovered variable types against DWARF ground truth. If the original binary shows uint32_t* but the decompiler outputs int*, that's a mismatch. This is deceptively hard—pointer signedness, struct padding, typedef aliases all complicate exact matching.
But byte_match is the star. Here's the simplified pipeline:
# Pseudocode based on evaluation logic
def byte_match_score(original_binary, decompiled_source):
# Step 1: Apply fixup pass to make decompiled code compilable
fixed_source = apply_fixups(decompiled_source)
# Step 2: Recompile with EXACT original toolchain
recompiled_binary = compile_with_original_toolchain(
source=fixed_source,
compiler=original_binary.compiler, # gcc-9.4.0, etc.
opt_level=original_binary.opt_level, # -O2
arch=original_binary.arch, # x86_64, ARM
format=original_binary.format # ELF, PE
)
# Step 3: Disassemble both and normalize
original_asm = disassemble_and_normalize(original_binary)
recompiled_asm = disassemble_and_normalize(recompiled_binary)
# Step 4: Jaccard similarity on instruction sets
return jaccard_similarity(
set(original_asm.instructions),
set(recompiled_asm.instructions)
)
The normalization step is critical. It strips register allocations and linker-dependent addresses because those vary non-deterministically even with identical semantics. What remains is a set of assembly instructions representing the program's computational essence. Jaccard similarity—intersection over union—measures how much of that essence the decompiler preserved.
The fixup pass reveals pragmatic engineering. Raw decompiler output rarely compiles: missing headers, syntax quirks, undefined symbols. DecBench applies automated fixes (exact rules undocumented in the truncated README) to isolate semantic correctness from trivial formatting failures. This is philosophically contentious—aggressive fixups could artificially inflate scores—but reflects reality: you can't measure recompilation fidelity if the code won't compile.
Finally, the 'Union' metric scores 1.0 if any of the three dimensions achieves perfect match. This acknowledges that decompilation quality is multidimensional. A tool might recover perfect control flow (GED=0) but mangle types (type_match=0.4). Another might produce compilable, assembly-identical code (byte_match=1.0) with unreadable structure (GED=0.7). Union=1.0 means 'perfect on at least one axis,' a pragmatic definition of success.
Results flow into TOML scoreboards and JSON datasets embedded in a static HTML report. The benchmark supports an external competition model: contestants download binary kits, decompile offline, and submit only outputs for blind scoring. This lets commercial tools participate without revealing their proprietary decompiled code—a clever concession to industry involvement.
The architectural constraint that limits practical use is the preprocessed .i file requirement. GED metric's source-side CFG extraction depends on having build artifacts. This means DecBench can't evaluate decompilers on real-world stripped binaries found in the wild—you need the original compiler invocation and intermediate files. It's a research benchmark, not a malware analysis tool.
Gotcha
DecBench's dependency on preprocessed source files and DWARF debug info defeats the core use case of decompilation: analyzing unknown binaries without build artifacts. If you have the .i files and debug symbols, you already have most of the information decompilation tries to recover. The benchmark measures 'how close can we get with perfect ground truth' rather than 'how well does this work on stripped malware binaries.'
The fixup pass introduces unmeasured bias. What separates a 'trivial syntax fix' from semantic correction? If the fixup adds missing type casts or initializes uninitialized variables, it's compensating for decompiler failures that should lower the byte_match score. The opacity here—no documented fixup rules in the public README—makes scores hard to interpret. GED metric's rust-joern dependency creates brittleness: it requires building Rust toolchains from source, and evaluation silently skips functions where Joern's parser fails. You might think a decompiler scored 0.9 GED when it actually succeeded on only 60% of functions and the rest were unparseable. The function-level focus ignores modern compilation realities—whole-program optimization, inlining, and link-time code generation blur function boundaries. A stripped binary might have no clean function segmentation, but DecBench assumes symbols exist to delineate evaluation units.
Verdict
Use if: You're a decompiler researcher measuring incremental progress across multiple tools and need reproducible, multi-axis scoring. The byte_match metric's recompilation approach is genuinely novel and captures something no other benchmark does—compilation fidelity, not just source similarity. The external competition model is valuable if you're organizing a comparative evaluation involving commercial tools. Use if you're publishing academic work and need defensible ground truth with automated scoring at scale. Skip if: You're doing practical reverse engineering on stripped binaries, malware analysis, or firmware extraction. DecBench requires build artifacts that real-world scenarios never provide. Skip if you care about security-relevant semantics—buffer overflows, use-after-free, exploit primitives—because none of the three metrics measure those. Skip if you need quick evaluation: the setup overhead (cross-compilers, Rust Joern, toolchain matching) only pays off for systematic multi-tool studies. For single-tool assessment, manual inspection on your own binaries is faster and more informative than aggregate scores that obscure per-function details.