Natural Language Autoencoders: Reading Neural Activation Vectors as English Text

Hook

What if you could ask a language model to explain its own internal thoughts—not by analyzing outputs, but by translating the raw numerical vectors flowing through its hidden layers directly into English sentences?

Context

Mechanistic interpretability researchers face a fundamental translation problem: language models think in high-dimensional vectors (3584 dimensions for Qwen-7B, 8192 for Llama-70B), but humans reason in natural language. When you want to understand what a specific activation vector means—say, the residual stream at layer 20, token position 47—you're staring at thousands of floating-point numbers with no obvious semantic content.

Sparse Autoencoders (SAEs) became the dominant solution by decomposing these vectors into sparse combinations of interpretable features. Anthropic's Claude SAEs, for instance, might show that a vector is "30% feature_1847 (Python syntax) + 18% feature_9234 (technical documentation) + 12% feature_445 (code comments)." This works beautifully for smaller models, but SAE training costs scale catastrophically—training a production-quality SAE suite for a 70B model requires infrastructure most research labs simply don't have. More fundamentally, SAEs give you sparse feature IDs, not explanations. You still need a human to examine top-activating examples and interpret what "feature_1847" actually means. Natural Language Autoencoders take a radically different approach: train two LLMs to form a bidirectional translation system where one model reads activation vectors and writes English explanations, while the other reads those explanations and reconstructs the original vectors. If the reconstruction error is low, you know the English text actually captured the semantic content.

Technical Insight

The architecture bypasses tokenization entirely through direct embedding injection. The Activation Verbalizer (AV) doesn't convert your 3584-dimensional vector into tokens—it injects the normalized vector directly into the model's embedding space as if it were a synthetic token. Here's the key insight: when you normally feed text to a language model, the tokenizer converts "hello" → token_id_245 → embedding_lookup → 3584-dim vector. NLA reverses this: it takes your activation vector, L2-normalizes it, and inserts it into a prompt template at the embedding level, completely skipping the tokenization bottleneck.

The prompt structure is elegant in its simplicity: "The following is an activation vector: <INJECTED_VECTOR>. This vector represents:" The model then autoregressively generates an explanation like "a Python function definition with type hints, appearing in technical documentation." The Activation Reconstructor (AR) does the inverse—it's a truncated language model (just the first K+1 layers) with a linear projection head that maps from the final token's hidden state back to a 3584-dimensional vector. For Qwen-7B with 28 layers, they extract at layer 20 (71% through); for Llama-70B's 80 layers, extraction happens at layer 53 (66% through). This isn't arbitrary—layers near the end collapse toward unembedding logits optimized for next-token prediction, losing the rich semantic content in mid-layer residual streams.

The training pipeline has three distinct phases. First, AR supervised fine-tuning: generate random activation vectors from your target model's distribution (run inference on web text, cache activations), pair them with API-generated explanations from GPT-4, then train the AR with MSE loss to reconstruct the original vectors from those explanations. Second, AV supervised fine-tuning: train the verbalizer to produce explanations that the (now frozen) AR can successfully decode. Third, and most architecturally interesting, simultaneous GRPO reinforcement learning:

# Simplified training loop (actual implementation uses Miles' FSDP2 backend)
for batch in dataloader:
    # Sample random activation vectors from target model
    activation_vectors = sample_activations(target_model, web_text)
    
    # L2 normalize (MSE on normalized vectors = 2(1 - cosine_similarity))
    normalized_vecs = F.normalize(activation_vectors, p=2, dim=-1)
    
    # AV generates explanations via RL (GRPO actor)
    explanations = activation_verbalizer.generate(
        input_embeds=construct_prompt_with_injected_vector(normalized_vecs),
        max_new_tokens=128
    )
    
    # AR reconstructs vectors (continuing supervised learning, NOT frozen)
    reconstructed_vecs = activation_reconstructor(
        explanations,
        extract_layer=20  # for Qwen-7B
    )
    reconstructed_vecs = F.normalize(reconstructed_vecs, p=2, dim=-1)
    
    # GRPO reward: negative MSE (higher reward = better reconstruction)
    reward = -F.mse_loss(reconstructed_vecs, normalized_vecs, reduction='none').mean(-1)
    
    # Simultaneous updates
    grpo_loss = compute_grpo_loss(explanations, reward)  # RL for AV
    ar_supervised_loss = F.mse_loss(reconstructed_vecs, normalized_vecs)  # SFT for AR
    
    grpo_loss.backward()  # Update AV
    ar_supervised_loss.backward()  # Update AR in parallel

The simultaneous training schedule is critical. Standard RL setups freeze the value function (or critic) during policy optimization to prevent instability. NLA does the opposite: the AR continues supervised learning during the AV's GRPO phase. Why? Because the AR is your reward model. If it degrades, your reward signal becomes meaningless. By continuing supervised updates, the AR maintains calibration even as the AV explores novel explanation styles during RL.

The infrastructure integration is deceptively clean. NLA plugs into Miles (a production RL framework supporting FSDP2 and Megatron backends) via two extension points: --custom-rm-path for the AR reward model and --custom-generate-function-path for embedding injection during rollouts. The genius is that embedding construction happens trainer-side. The SGLang rollout server never sees raw activation vectors—it receives pre-constructed embedding sequences via the input_embeds transport. This means you can serve NLA rollouts on vanilla vLLM/SGLang with zero code changes. Future improvements like learned affine transformations (W·v + b instead of raw injection) require no serving infrastructure modifications.

Evaluation uses Fraction of Variance Explained (FVE), borrowed from neuroscience: if your reconstructed vector perfectly matches the original direction, FVE = 1.0; if it's orthogonal, FVE = 0. The Qwen-7B checkpoint achieves 75% FVE on layer 20 activations, meaning the English explanations genuinely compress the high-dimensional semantics. For a vocabulary-sized model (7B parameters) to encode 3584 continuous dimensions into discrete text with only 25% information loss is remarkable—it suggests natural language has far more representational bandwidth than we typically assume.

Gotcha

The most glaring limitation is context-blindness. NLA explains each activation vector in isolation, without access to the token sequence that produced it. Consider the word "bank" in two sentences: "The river bank was muddy" versus "The bank approved my loan." The activation vector at the "bank" token position will be radically different in each case, but the AV never sees surrounding tokens. It receives only the 3584-dimensional vector and must generate an explanation without knowing why that vector has those particular values. This makes NLA terrible at polysemy, contextual disambiguation, or any phenomenon where meaning depends on surrounding context. You'll get plausible-sounding explanations, but they might describe the vector's content without capturing the contextual reason it has that content.

The infrastructure requirements are prohibitive for most researchers. Training the 7B model requires 2×8×H100 GPUs (16 total) for RL. The 70B model mandates Megatron backend with tensor parallelism and pipeline parallelism—there's no single-node recipe. The repository provides no LoRA option, no gradient checkpointing tricks for memory reduction, no path for researchers with consumer hardware. This is production-scale tooling that assumes production-scale resources. Additionally, the Megatron mode enforces CP=1 (context parallelism disabled), limiting maximum sequence length even when you have the hardware for longer contexts.

The deeper epistemological problem is validation. The 75% FVE metric tells you the AR successfully reconstructs vector directions, but it says nothing about whether the AV's English explanations are causally faithful. The AV might generate "a Python function with type hints" because that explanation happens to produce a vector pointing the right direction, even if the activation vector is actually encoding something subtly different. You're trusting two fine-tuned LLMs to honestly translate between modalities with no ground truth. This is a hypothesis generation tool, not a source of interpretability ground truth. Use the explanations to form theories about what activations mean, then validate those theories with causal interventions (activation patching, ablations) before trusting them.

Verdict

Use if: you're doing mechanistic interpretability research on models >30B parameters where Sparse Autoencoders become computationally intractable, you have multi-GPU infrastructure (minimum 2 nodes with 8×H100 each for serious work), you need human-readable explanations for activation vectors rather than just sparse feature decompositions, or you're building tools that let non-experts explore model internals (the bidirectional design enables both probing and synthesis). Skip if: you need context-aware explanations that understand why a vector has certain values based on surrounding tokens, you're working on a laptop or single-GPU setup (no toy-scale path exists), you need formal guarantees about explanation faithfulness rather than plausible hypotheses, or you're doing automated interpretability at scale where SAE features' clean linear algebra beats fuzzy natural language interfaces. This is essential infrastructure for a specific niche—large-scale interpretability research—but it's not a universal SAE replacement.