Extracting Neural Network Weights Through Black-Box Queries: A Cryptanalytic Attack Framework

Hook

A neural network deployed behind an API isn't as safe as you think. With enough queries, attackers can extract every single weight, bias, and activation pattern—even without seeing a single gradient.

Context

When organizations deploy machine learning models as APIs, they assume the model's internal parameters remain secret. You send inputs, receive predictions, and the weights stay hidden on the server. This assumption underpins countless commercial ML services. But research from Carlini et al. and Canales-Martinez et al. shattered this illusion for ReLU-based networks, proving that black-box query access alone enables complete parameter extraction.

The cryptanalytical-extraction repository unifies these breakthrough techniques into a single framework. Before this work, researchers had to navigate separate codebases with different dependencies and incompatible approaches. Carlini's signature recovery used TensorFlow and focused on theoretical extraction bounds, while Canales-Martinez's neuron wiggle technique optimized for polynomial-time complexity. This repository bridges both worlds, migrating the computational pipeline to JAX for efficiency while providing a cohesive implementation of the full attack chain. It's a security researcher's toolkit for understanding—and demonstrating—fundamental vulnerabilities in deployed neural networks.

Technical Insight

The attack operates in two distinct phases that exploit the geometric structure of ReLU networks. Phase one—signature recovery—extracts weight ratios by finding critical points where the network's decision boundary changes. Phase two—sign recovery—determines which neurons activate for specific inputs, completing the parameter reconstruction.

Signature recovery works by following hyperplanes in the input space. When you query a ReLU network, you're essentially asking which side of a piecewise-linear decision boundary your input falls on. By carefully searching along lines in input space and detecting where predictions flip, you identify critical points. At each critical point, at least one neuron's activation changes from zero to positive (or vice versa). The framework implements multiple search strategies: binary search along random directions, targeted search toward known critical points, and hyperplane following that walks along decision boundaries.

Here's how the critical point search looks in practice:

# Simplified critical point search along a direction
def find_critical_point(model, start_point, direction, epsilon=1e-6):
    # Binary search to find where prediction changes
    low, high = 0.0, 1.0
    start_pred = model(start_point)
    
    while high - low > epsilon:
        mid = (low + high) / 2.0
        test_point = start_point + mid * direction
        test_pred = model(test_point)
        
        if jnp.allclose(test_pred, start_pred):
            low = mid  # No change yet, search further
        else:
            high = mid  # Found a boundary, narrow in
    
    critical_point = start_point + high * direction
    return critical_point

Once you've collected enough critical points, you can reconstruct weight ratios through linear algebra. Each critical point gives you a constraint on the weight space—you know a specific linear combination of weights equals the bias at that point. With sufficient critical points showing diverse activation patterns (high signature diversity), you solve for the weight ratios. The framework handles quantization precision (float16, float32, float64) to match the target model's numerical characteristics.

Sign recovery completes the extraction by determining activation signs. The repository implements both approaches: Carlini's exponential-time brute force method and Canales-Martinez's polynomial-time neuron wiggle technique. The neuron wiggle approach is particularly elegant—it searches for input perturbations that flip individual neuron activations while keeping others constant. By measuring how predictions change when you "wiggle" a single neuron's sign, you can deduce its activation pattern.

The JAX migration is architecturally significant. JAX's functional paradigm and automatic differentiation make the geometric computations faster and more numerically stable. The framework loads initial weights from TensorFlow (for compatibility with existing model checkpoints) but performs all attack computations in JAX:

# JAX-based layer extraction
@jax.jit
def compute_neuron_activation(weights, bias, input_point):
    # Pure functional computation for a single layer
    linear_output = jnp.dot(weights, input_point) + bias
    activated = jnp.maximum(0, linear_output)  # ReLU
    return activated

def extract_layer(model, layer_idx, critical_points, precision='float32'):
    # Convert to specified precision for matching target model
    dtype = getattr(jnp, precision)
    critical_points = jnp.array(critical_points, dtype=dtype)
    
    # Build constraint matrix from critical points
    constraints = build_constraints(critical_points, model, layer_idx)
    
    # Solve for weight ratios using least squares
    weight_ratios = jnp.linalg.lstsq(constraints.A, constraints.b)[0]
    
    return weight_ratios

The layer-by-layer extraction is crucial. You can't extract the entire network simultaneously because activation patterns in deeper layers depend on earlier layer outputs. The framework starts from the input layer and works forward, using extracted parameters from layer N to inform the search for layer N+1. This cascading approach means errors compound—if early layer extraction fails, subsequent layers become impossible to recover accurately.

Memory deduplication optimizations handle the massive query volumes required. Extracting a non-trivial network can require millions of queries, generating gigabytes of intermediate data. The framework deduplicates stored critical points and activation patterns to keep memory footprint manageable during extended attack runs.

Gotcha

The most significant limitation is dataset coverage: signature recovery only works for MNIST and random models. If you're targeting a CIFAR-10 model, you're stuck with sign recovery only, which dramatically reduces attack effectiveness on more complex vision tasks. This isn't a minor inconvenience—it means you can't fully reproduce the signature recovery results on realistic computer vision models.

Query volume makes stealth attacks impossible. Extracting even a small network requires thousands of queries, with larger models demanding millions. Any production API with basic rate limiting or query monitoring will detect this activity immediately. The attack assumes unlimited, unthrottled access to the target model—an assumption that doesn't hold for most real-world deployments. Furthermore, extraction success isn't guaranteed. The geometric search can get stuck in regions with low signature diversity, where critical points don't provide enough independent constraints to solve for weights. Network architectures with certain symmetries or unusual activation patterns may resist extraction, though the framework provides limited diagnostics when this occurs.

Verdict

Use if: You're a security researcher studying model extraction vulnerabilities, need to reproduce academic results on cryptanalytic attacks, or want to demonstrate extraction risks to stakeholders deploying ML APIs. The unified codebase saves significant time compared to wrangling multiple research prototypes. Skip if: You need production-ready extraction tools (this is research code with rough edges), want to attack non-ReLU architectures like transformers or batch-normalized networks, require stealth operations (query volumes are prohibitive), or need to work with CIFAR models beyond sign recovery. This is fundamentally an academic tool for understanding theoretical vulnerabilities, not a practical attack framework for stealing production models.