> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

DecBench: Measuring Decompilation Quality by Recompiling the Output

[ View on GitHub ]

DecBench: Measuring Decompilation Quality by Recompiling the Output

Hook

Most decompiler benchmarks compare source code similarity. DecBench does something more radical: it recompiles the decompiled output with the original compiler and measures how close the assembly gets to the original binary.

Context

Decompilation—recovering source code from compiled binaries—is still more art than science. Tools like Ghidra, IDA Pro, and Binary Ninja can transform raw assembly into C-like pseudocode, but measuring their accuracy has always been subjective. Does 'good' decompilation mean readable code? Semantically equivalent logic? Structurally similar control flow? The answer depends on who's asking.

Researchers face a measurement problem. Without ground truth, evaluating decompiler improvements requires manual inspection of thousands of functions or brittle semantic equivalence checks that fail on compiler optimizations. DecBench tackles this by introducing a three-dimensional scoring system where one metric—byte_match—takes a genuinely novel approach: it recompiles the decompiled code using the same compiler, optimization level, and architecture as the original binary, then performs assembly-level diffing with normalized operands. This measures something no other benchmark captures: whether a decompiler preserves enough semantic detail that recompilation can recreate functionally identical machine code.

Technical Insight

Evaluate Stage

Decompile Stage

Compile Stage

O0-O3 binaries

preserve

embed

input

normalize

raw output

compilable code

parse

parse

graph distance

extract types

ground truth

recompile

disassemble

Jaccard similarity

embed

Cross-Compilers

gcc/MinGW

Binary Kit

.i Preprocessed Files

DWARF Debug Info

Decompiler Harness

angr/Ghidra/IDA/BinjaⓇ

Fixup Pass

Decompiled Output

rust-joern CFG

GED Metric

type_match

Normalized Assembly

byte_match

TOML/JSON Results

Static HTML Report

System architecture — auto-generated

DecBench's architecture separates compilation, decompilation, and evaluation into distinct stages with checkpointing at project boundaries—a production-grade decision acknowledging that multi-hour benchmark runs crash. The compile stage uses gcc and MinGW cross-compilers to generate binaries at optimization levels O0 through O3, but crucially, it preserves preprocessed .i files and DWARF debug symbols. This isn't just bookkeeping: the GED (Graph Edit Distance) metric extracts source-side control flow graphs exclusively from preprocessed files, not original C, because parsing ambiguities in raw source would skew CFG comparisons.

The decompile stage wraps heterogeneous tools through a harness abstraction. Each decompiler (angr, Ghidra, IDA, Binary Ninja) gets normalized invocation and output collection. This design acknowledges reality: decompilers have wildly different interfaces—command-line scripts, GUI automation, Python APIs. The harness hides these details so evaluation can run uniformly.

Where DecBench gets interesting is the evaluate stage's three independent metrics. GED uses rust-joern to parse both source and decompiled code into control flow graphs, then cfgutils computes graph edit distance—how many node/edge insertions, deletions, or substitutions separate the two. This measures structural similarity: did the decompiler recover the same branching logic, loop nesting, and function call patterns?

The type_match metric compares recovered variable types against DWARF ground truth. If the original binary shows uint32_t* but the decompiler outputs int*, that's a mismatch. This is deceptively hard—pointer signedness, struct padding, typedef aliases all complicate exact matching.

But byte_match is the star. Here's the simplified pipeline:

# Pseudocode based on evaluation logic
def byte_match_score(original_binary, decompiled_source):
    # Step 1: Apply fixup pass to make decompiled code compilable
    fixed_source = apply_fixups(decompiled_source)
    
    # Step 2: Recompile with EXACT original toolchain
    recompiled_binary = compile_with_original_toolchain(
        source=fixed_source,
        compiler=original_binary.compiler,  # gcc-9.4.0, etc.
        opt_level=original_binary.opt_level,  # -O2
        arch=original_binary.arch,  # x86_64, ARM
        format=original_binary.format  # ELF, PE
    )
    
    # Step 3: Disassemble both and normalize
    original_asm = disassemble_and_normalize(original_binary)
    recompiled_asm = disassemble_and_normalize(recompiled_binary)
    
    # Step 4: Jaccard similarity on instruction sets
    return jaccard_similarity(
        set(original_asm.instructions),
        set(recompiled_asm.instructions)
    )

The normalization step is critical. It strips register allocations and linker-dependent addresses because those vary non-deterministically even with identical semantics. What remains is a set of assembly instructions representing the program's computational essence. Jaccard similarity—intersection over union—measures how much of that essence the decompiler preserved.

The fixup pass reveals pragmatic engineering. Raw decompiler output rarely compiles: missing headers, syntax quirks, undefined symbols. DecBench applies automated fixes (exact rules undocumented in the truncated README) to isolate semantic correctness from trivial formatting failures. This is philosophically contentious—aggressive fixups could artificially inflate scores—but reflects reality: you can't measure recompilation fidelity if the code won't compile.

Finally, the 'Union' metric scores 1.0 if any of the three dimensions achieves perfect match. This acknowledges that decompilation quality is multidimensional. A tool might recover perfect control flow (GED=0) but mangle types (type_match=0.4). Another might produce compilable, assembly-identical code (byte_match=1.0) with unreadable structure (GED=0.7). Union=1.0 means 'perfect on at least one axis,' a pragmatic definition of success.

Results flow into TOML scoreboards and JSON datasets embedded in a static HTML report. The benchmark supports an external competition model: contestants download binary kits, decompile offline, and submit only outputs for blind scoring. This lets commercial tools participate without revealing their proprietary decompiled code—a clever concession to industry involvement.

The architectural constraint that limits practical use is the preprocessed .i file requirement. GED metric's source-side CFG extraction depends on having build artifacts. This means DecBench can't evaluate decompilers on real-world stripped binaries found in the wild—you need the original compiler invocation and intermediate files. It's a research benchmark, not a malware analysis tool.

Gotcha

DecBench's dependency on preprocessed source files and DWARF debug info defeats the core use case of decompilation: analyzing unknown binaries without build artifacts. If you have the .i files and debug symbols, you already have most of the information decompilation tries to recover. The benchmark measures 'how close can we get with perfect ground truth' rather than 'how well does this work on stripped malware binaries.'

The fixup pass introduces unmeasured bias. What separates a 'trivial syntax fix' from semantic correction? If the fixup adds missing type casts or initializes uninitialized variables, it's compensating for decompiler failures that should lower the byte_match score. The opacity here—no documented fixup rules in the public README—makes scores hard to interpret. GED metric's rust-joern dependency creates brittleness: it requires building Rust toolchains from source, and evaluation silently skips functions where Joern's parser fails. You might think a decompiler scored 0.9 GED when it actually succeeded on only 60% of functions and the rest were unparseable. The function-level focus ignores modern compilation realities—whole-program optimization, inlining, and link-time code generation blur function boundaries. A stripped binary might have no clean function segmentation, but DecBench assumes symbols exist to delineate evaluation units.

Verdict

Use if: You're a decompiler researcher measuring incremental progress across multiple tools and need reproducible, multi-axis scoring. The byte_match metric's recompilation approach is genuinely novel and captures something no other benchmark does—compilation fidelity, not just source similarity. The external competition model is valuable if you're organizing a comparative evaluation involving commercial tools. Use if you're publishing academic work and need defensible ground truth with automated scoring at scale. Skip if: You're doing practical reverse engineering on stripped binaries, malware analysis, or firmware extraction. DecBench requires build artifacts that real-world scenarios never provide. Skip if you care about security-relevant semantics—buffer overflows, use-after-free, exploit primitives—because none of the three metrics measure those. Skip if you need quick evaluation: the setup overhead (cross-compilers, Rust Joern, toolchain matching) only pays off for systematic multi-tool studies. For single-tool assessment, manual inspection on your own binaries is faster and more informative than aggregate scores that obscure per-function details.