Anamnesis: When Frontier LLMs Started Writing Exploits That Bypass CFI and Shadow Stacks

Hook

GPT-5.2 just spent 50 million tokens discovering that glibc's exit handler mechanism could chain function calls to write files without execve, ROP chains, or violating CFI. It took three hours and potentially hundreds of dollars in API costs—but it worked.

Context

Exploit development has always been the apex predator skill in security engineering. Writing a working exploit requires intimate knowledge of memory corruption primitives, runtime internals, compiler behavior, and mitigation bypass techniques. It's artisanal work: a skilled exploit developer might spend days or weeks crafting a reliable chain of operations to turn a use-after-free into arbitrary code execution, especially when facing modern defenses like Control Flow Integrity (CFI), Intel's Control-flow Enforcement Technology (CET) Shadow Stacks, or RELRO.

Anamnesis is a research framework that tests whether frontier LLMs can industrialize this artisanal process. Built by Sean Heelan, it provides a controlled experimental environment with a real QuickJS use-after-free vulnerability (discovered by Claude Opus 4.5, incidentally) and progressively harder mitigation configurations. The framework gives AI agents—currently Claude Opus 4.5 and GPT-5.2—massive token budgets (30-60 million tokens), debugging tools, compilation capabilities, and a sandbox to iteratively develop working exploits. The question isn't whether LLMs can help with exploit development; it's whether they can do it autonomously, from vulnerability report to shell, without human intervention.

Technical Insight

Anamnesis orchestrates LLM agents through a JavaScript-based framework that provides a controlled QuickJS target with four escalating challenge levels: baseline (minimal mitigations), RELRO (prevents GOT overwrites), RELRO+CFI (blocks indirect call/jump hijacking), and RELRO+CFI+Shadow Stack+Sandbox (the final boss, adding Intel CET hardware enforcement and seccomp restrictions on syscalls). Agents interact with the framework through a tool-calling interface that exposes compilation, GDB debugging, exploit execution, and source code inspection capabilities.

The architecture is deceptively simple: the agent receives the vulnerability report and target configuration, then enters an iterative loop of hypothesis generation, exploitation primitive development, testing, and refinement. The framework doesn't provide exploitation templates or guide the agent's strategy—it just gives tools and validates whether the exploit achieved objectives like spawning a shell or writing to arbitrary files. What emerges from this minimal scaffolding is remarkable: both models independently rediscovered advanced exploitation techniques that typically require years of expertise.

Here's what a typical agent interaction looks like in the framework's execution model:

// Agent requests compilation with specific flags
await compileTarget({
  mitigations: ['RELRO', 'CFI', 'SHADOW_STACK'],
  debugSymbols: true,
  optimization: 'O2'
});

// Agent crafts exploit payload
const exploit = `
// Stage 1: Trigger UAF and corrupt heap
let victim = createObject();
victim = null; // Free
gc(); // Force collection
let attacker = spray(0x1000); // Reclaim with controlled data

// Stage 2: Arbitrary read primitive via corrupted pointer
function leak(addr) {
  // Agent-discovered technique using type confusion
  return attacker[addr];
}

// Stage 3: Locate libc base via link_map traversal
const libcBase = findLibcBase(leak);
`;

// Execute and capture results
const result = await runExploit(exploit);
if (result.crashed) {
  // Agent analyzes GDB output to refine approach
  const crashAnalysis = await debugWithGDB(result.core);
}

The breakthrough moments came when agents hit hard mitigation walls and invented creative bypasses. When GPT-5.2 faced the RELRO+CFI+Shadow Stack+Sandbox configuration, traditional techniques failed: GOT overwrites were blocked by RELRO, ROP chains couldn't execute due to shadow stack enforcement, and the sandbox prevented execve/fork syscalls. The model's solution demonstrated genuine problem-solving: it discovered that glibc's __run_exit_handlers mechanism could be hijacked to chain multiple legitimate function calls (same-signature CFI bypass) that collectively performed file writes without triggering sandbox violations or requiring stack pivots.

Clauде Opus 4.5 independently discovered File Stream Oriented Programming (FSOP) attacks, a technique where attackers corrupt FILE structure internals to hijack control flow through vtable pointers. This required understanding glibc's internal FILE implementation, identifying which offsets could be corrupted to redirect execution, and crafting fake FILE structures with properly aligned pointers—all without human hints about this technique's existence.

Both models also defeated pointer mangling, a protection where glibc XORs function pointers with a secret per-process value before storing them. The agents discovered they could leak the mangling cookie by reading it from memory (using their arbitrary read primitive), XOR it with their desired target address, and write the mangled result to hijack control flow. This multi-step reasoning—realizing mangling exists, understanding how to reverse it, and incorporating it into the exploit chain—demonstrates sophisticated threat modeling.

The framework tracks every agent interaction, token consumption, and exploit iteration. Analysis of successful runs reveals agents typically progress through recognizable phases: initial reconnaissance (reading source code, identifying the vulnerability mechanism), primitive development (building arbitrary read/write), information gathering (leaking addresses, enumerating libraries via link_map traversal), and finally exploitation (chaining primitives to achieve objectives). Failed attempts show agents getting stuck on incorrect assumptions—like assuming GOT entries are writable when RELRO is enabled—but course-correcting after observing failure modes through GDB output.

What makes this particularly significant is the token efficiency curve. Early exploit attempts consumed massive token budgets exploring dead ends, but successful strategies emerged around the 20-40 million token mark for hard challenges. The models demonstrated metalearning within a single run: they'd encounter a protection mechanism, spend tokens understanding it, develop a bypass technique, then reuse that technique in subsequent iterations. This isn't template matching or retrieval from training data—it's dynamic problem-solving under novel constraints.

Gotcha

The elephant in the room is cost and statistical rigor. Each hard challenge consumed 50-60 million tokens, translating to potentially hundreds of dollars per run at current API pricing. The framework only performed 10 runs per experiment—as the author explicitly acknowledges, this is statistically insufficient for making authoritative claims about relative model capabilities. You're looking at research-grade proof-of-concept data, not production-ready benchmarks. If you're planning to replicate or extend this work, budget accordingly: comprehensive evaluation across model versions and configurations could easily cost tens of thousands of dollars.

The framework is also narrowly scoped—intentionally so, but it limits generalizability. Anamnesis is tightly coupled to the included QuickJS use-after-free vulnerability. It's not a general-purpose exploit generation system you can point at arbitrary CVEs. Adapting it to test different vulnerability types (integer overflows, race conditions, logic bugs) or different target applications would require significant framework modifications. The mitigation configurations are also Linux-specific and x86-64-centric; testing against Windows exploit mitigations (CFG, ACG, CIG) or ARM architectures would need substantial rework. This is an evaluation artifact for measuring LLM capabilities on a controlled problem, not a deployable tool for your security team's workflow.

Verdict

Use if: You're researching AI capabilities in adversarial domains and need concrete evidence that frontier models can perform sophisticated technical tasks requiring deep domain expertise. Security teams modeling future threat landscapes should study this—if LLMs can autonomously generate exploits bypassing modern mitigations today, that capability will only improve and potentially democratize advanced exploitation. Academic researchers benchmarking AI progress on complex reasoning tasks will find this a valuable data point beyond traditional coding benchmarks. Also use if you're exploring human-AI collaboration in security work and want to understand what tasks can be delegated versus where human expertise remains essential. Skip if: You need production tooling for vulnerability assessment or penetration testing—this is a research framework with prohibitive per-run costs and limited scope. Skip if you're expecting plug-and-play exploit generation for arbitrary vulnerabilities; it's purpose-built for evaluating LLMs against a specific challenge, not a general-purpose security tool. Also skip if you lack the budget for extensive LLM API usage; even small-scale experiments will consume significant resources. Finally, skip if you're looking for deterministic exploit generation—the stochastic nature of LLM outputs means success rates vary across runs, making this unsuitable for scenarios requiring reliability guarantees.