Streaming JVM Heap Dumps Without Running Out of Memory

Hook

Your Java application crashed with a 16GB heap dump, but your analysis machine only has 8GB of RAM. Most heap dump analyzers will fail before they even start—unless you parse the file as a stream.

Context

When a Java application experiences an OutOfMemoryError or you capture a heap dump for debugging, the JVM produces a binary hprof file containing a snapshot of every object in memory at that moment. For production applications, these files routinely exceed tens of gigabytes. Tools like Eclipse MAT (Memory Analyzer Tool) and VisualVM can analyze these dumps, but they have a fundamental constraint: they need to load significant portions of the dump into memory to build their object graphs and indexes. This creates a chicken-and-egg problem where analyzing a 20GB heap dump might require 30GB of RAM on your analysis machine.

The eaftan/hprof-parser project takes a different approach borrowed from XML parsing: treat the heap dump as a stream of records rather than a data structure to load. Just as SAX parsers revolutionized XML processing for large files by avoiding DOM tree construction, this library processes hprof files sequentially, invoking callbacks for each record type without maintaining global state. This architectural choice enables analysis of arbitrarily large heap dumps on modest hardware, trading convenience for memory efficiency. You won't get the rich query capabilities of full-featured analyzers, but you can answer specific questions about heap contents without worrying about RAM constraints.

Technical Insight

The parser implements a classic handler pattern with a clean separation between parsing logic and analysis logic. At its core, the HprofParser class reads the binary hprof format sequentially, dispatching to methods on the RecordHandler interface for each record type encountered. The hprof format defines numerous record types—heap dumps, stack traces, class loads, thread starts—and the handler receives each one as it's parsed.

The RecordHandler interface declares methods for every record type in the hprof specification, such as loadClass(), allocSite(), heapDump(), and cpuSample(). Rather than forcing implementers to provide implementations for dozens of methods, the library includes NullRecordHandler, a base class with no-op implementations of every method. This follows the null object pattern, allowing you to subclass and override only the record types you care about:

public class StringAnalyzer extends NullRecordHandler {
    private long stringCount = 0;
    private long totalStringBytes = 0;
    
    @Override
    public void heapDumpStringInstance(long id, long classId, 
                                       byte[] stringData) {
        stringCount++;
        totalStringBytes += stringData.length;
    }
    
    @Override
    public void finished() {
        System.out.println("Total Strings: " + stringCount);
        System.out.println("Avg Size: " + (totalStringBytes / stringCount));
        System.out.println("Total Memory: " + (totalStringBytes / 1024 / 1024) + "MB");
    }
}

This handler counts String instances and tracks their memory consumption without maintaining references to any objects. Once heapDumpStringInstance() returns, that record's data can be garbage collected—the parser never accumulates state beyond what your handler explicitly stores.

The entry point is equally straightforward. The Parse program accepts command-line arguments specifying the heap dump file and the fully-qualified name of your handler class:

java -cp hprof-parser.jar Parse --file heap.hprof --handler com.example.StringAnalyzer

The parser loads your handler via reflection and begins streaming through the dump file. This design makes it trivial to build a library of analysis handlers for different questions: "Which classes consume the most memory?" "How many instances of MyClass exist?" "What's the average size of HashMap instances?" Each analysis becomes a purpose-built handler class.

For more complex analysis requiring state across multiple record types, you can maintain data structures within your handler. For example, correlating object instances with their class definitions:

public class InstanceDistribution extends NullRecordHandler {
    private Map<Long, String> classIdToName = new HashMap<>();
    private Map<String, Long> classInstanceCounts = new HashMap<>();
    
    @Override
    public void loadClass(int serialNum, long classId, int stackTrace, 
                         String className) {
        classIdToName.put(classId, className);
    }
    
    @Override
    public void heapDumpInstance(long id, int stackTrace, long classId, 
                                byte[] instanceData) {
        String className = classIdToName.get(classId);
        classInstanceCounts.merge(className, 1L, Long::sum);
    }
    
    @Override
    public void finished() {
        classInstanceCounts.entrySet().stream()
            .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
            .limit(20)
            .forEach(e -> System.out.println(e.getKey() + ": " + e.getValue()));
    }
}

This handler builds a map from class IDs to class names, then counts instances per class. The memory footprint remains manageable because you're storing aggregate counts rather than individual object references. Even with millions of objects, your metadata stays small.

The streaming approach has another subtle advantage: speed. Sequential file I/O is dramatically faster than the random access patterns required for indexed lookup. When you only need to answer specific questions rather than support arbitrary queries, streaming can complete analysis in minutes rather than hours.

Gotcha

The biggest limitation is the outdated build system. The README references maven compile and maven jar, which are Maven 1.x commands deprecated since 2005. Maven 2 and later use mvn instead. The project includes a project.xml file (Maven 1 format) rather than pom.xml (Maven 2+). This means you'll likely need to manually set up a modern build configuration or compile via IDE rather than using the documented build process. For a library that hasn't seen updates in years, this isn't surprising, but it's a friction point for adoption.

The documentation is minimal beyond the basic example. The RecordHandler interface methods are tersely commented, and understanding what data each record type provides requires either reading the parser source code or consulting external hprof format documentation. For instance, heapDumpInstance() provides raw byte[] instanceData, but interpreting that byte array—understanding field offsets, parsing field values, handling primitive vs. reference types—is left entirely to the implementer. There's no convenience layer for common operations like "extract this field from this instance" or "follow this reference to another object." You're working at the level of raw binary parsing.

This also means the library provides no built-in analysis. The included PrintHandler is purely diagnostic, dumping record information to stdout. Every practical use case requires writing custom handler code. If you need standard heap dump analysis features like finding the largest objects, detecting memory leak candidates, or computing retained sizes, you'll implement those algorithms yourself. For quick one-off investigations, tools like Eclipse MAT are dramatically more productive.

Verdict

Use if: You're building custom heap dump analysis tools for specific production debugging scenarios, you need to analyze dumps larger than your available RAM, you have specialized analysis requirements not met by existing tools, or you're integrating heap dump analysis into automated pipelines where a streaming approach fits naturally. This is a power tool for tool builders. Skip if: You need interactive heap dump exploration with a GUI, you want ready-made analysis features like leak detection or dominator trees, the Maven 1.x build system is a blocker for your environment, or you're doing one-off debugging where Eclipse MAT or VisualVM would suffice. This library trades convenience and features for memory efficiency and extensibility—choose it when those tradeoffs align with your constraints.