Mantis: Google's 15-Stage Pipeline for Teaching AI Agents to Hunt Vulnerabilities

Hook

Google built an AI security scanner where the primary safety mechanism is asking the AI nicely not to hack the host machine. It's both terrifying and instructive.

Context

Traditional static analysis tools like Semgrep and CodeQL excel at finding known vulnerability patterns with near-zero false positives, but they're fundamentally backward-looking—they can only detect what security researchers have already codified into rules. When a novel exploit class emerges, you wait for human researchers to analyze it, write detection rules, and push updates. LLMs promise something different: the ability to reason about code semantically and hypothesize vulnerabilities that don't match existing patterns.

But raw LLM-based code review produces an unusable flood of hallucinations. Ask GPT-4 to 'find all security bugs' in a codebase and you'll get confident assertions about SQL injections in files that don't touch databases, authentication bypasses in public endpoints, and memory corruption in memory-safe languages. Google's Mantis tackles this head-on by treating AI security analysis not as a single prompt, but as a 15-stage validation funnel where each stage filters out different categories of false positives. It's a prompt engineering blueprint for turning unreliable LLM hypotheses into validated, reproducible security findings.

Technical Insight

Mantis's core architectural insight is that vulnerability discovery should be structured like an attacker's workflow: broad reconnaissance, hypothesis generation, progressive validation, and practical exploitation. Each of the 15 stages is a discrete 'skill'—a specialized prompt template that reads from and writes to a shared filesystem workspace rather than passing data through function calls.

The pipeline starts with /mantis-architecture, which synthesizes the target repository into markdown documents describing build systems, authentication boundaries, and data flow patterns. This becomes the Knowledge Base (KB)—a filesystem-based context injection mechanism that sidesteps token limit constraints. Instead of cramming the entire codebase into context windows, later stages selectively inject relevant KB sections as few-shot examples:

# The KB lives as markdown files in workspace/kb/
/mantis-architecture --target /path/to/repo

# Generates files like:
# workspace/kb/architecture.md - System boundaries and trust zones
# workspace/kb/attack_surface.md - External input vectors
# workspace/kb/historical_vulns.md - Past CVEs in similar codebases

The /mantis-researcher stage then performs parallel scans, generating vulnerability hypotheses across multiple files. This stage runs hot—it's intentionally tuned for recall over precision, operating under the assumption that false positives are cheaper to filter later than false negatives are to recover:

# Conceptual structure of mantis-researcher output
# Each finding in workspace/findings/ contains:
{
  "hypothesis": "Potential TOCTOU race in file permission check",
  "file": "src/auth/validator.cpp",
  "line_range": [145, 167],
  "severity_estimate": "HIGH",
  "confidence": 0.4  # Deliberately low—expects validation
}

The filtering funnel then engages. /mantis-dedupe eliminates redundant findings by clustering similar vulnerability patterns. /mantis-review performs static validation—does this file actually compile? Do the referenced functions exist? Does the hypothesized data flow match static analysis? /mantis-critic applies adversarial prompting, asking a separate LLM instance to argue why each finding is not a vulnerability.

The critical validation stage is /mantis-reproduce, which generates proof-of-concept exploits and executes them in Docker containers with gVisor isolation. This is where Mantis diverges sharply from traditional SAST:

# Mantis instructs the LLM to generate PoC containers
# Example generated by /mantis-reproduce:
FROM ubuntu:22.04
COPY vulnerable_app /app/
RUN /app/setup.sh

# PoC script that demonstrates the vulnerability
COPY exploit.py /exploit.py
CMD python3 /exploit.py && echo 'VULN_CONFIRMED'

The LLM agent builds this container, executes it with docker run --runtime=runsc (gVisor for kernel isolation), and parses the output. If the exploit succeeds, the finding graduates to workspace/findings/validated/. If it fails, the system appends the failure trace to workspace/learnings.jsonl—a feedback mechanism that helps future iterations avoid similar hallucinations.

/mantis-chain attempts something particularly sophisticated: combining individual validated bugs into multi-step exploit chains. It reads all validated findings and reasons about how a low-severity information disclosure might chain with an authentication bypass to achieve remote code execution. This mirrors real attacker tradecraft in a way that isolated SAST findings never do.

The feedback loop closes with /mantis-reflect, which analyzes the entire execution trajectory—which hypotheses validated, which failed, what validation techniques worked—and extracts patterns back into learnings.jsonl. Over multiple runs, the system builds a corpus of empirically grounded patterns: 'Hypothesis X about framework Y always fails to reproduce in Docker environments with Z configuration.'

Critically, state sharing happens through the filesystem, not APIs. Each stage assumes it's being invoked by an LLM coding agent (like Claude with computer use or GPT-4 with code interpreter) that can execute bash commands and read files. This is both elegant and limiting—elegant because it's tool-agnostic and debuggable (you can inspect workspace/ at any stage), limiting because it's inherently sequential and can't exploit modern orchestration patterns like backpressure control or distributed execution.

Gotcha

The security model is fundamentally prompt-based. Mantis tells the LLM 'never run generated code on the host, always use Docker with gVisor,' but this is an instruction not a constraint. A sufficiently confused model—or one that's been adversarially prompted through malicious code comments in the target repository—could bypass this entirely. The README acknowledges this with warnings about running in isolated VMs, which is telling: even Google doesn't trust their own prompt boundaries enough to run this on developer workstations.

The sequential pipeline creates real throughput problems. All 15 stages must run serially, even when stages like /mantis-researcher are embarrassingly parallel across files. You can't easily distribute this across multiple machines or implement dynamic stage skipping based on intermediate results. If /mantis-reproduce fails for all findings, you still can't short-circuit back to /mantis-researcher with adjusted parameters—the architecture assumes manual human-in-the-loop adjustments between runs.

The 'continuous review loop' vision in the roadmap is aspirational. There's no built-in mechanism to prevent re-discovering the same bugs across multiple runs unless you manually archive validated findings. The system also lacks deduplication against existing issue trackers—it might spend compute validating a CVE that's already been patched and documented in your GitHub Issues.

Finally, this is a prompt library, not a product. You get 15 markdown files with prompt templates and a conceptual architecture. Building the actual orchestration layer—the code that invokes LLM APIs, manages workspace state, handles Docker execution, parses outputs, and implements retry logic—is left as an exercise for the reader. Teams expecting something they can pip install and point at a repository will be disappointed.

Verdict

Use if: You're a security engineering team at a tech company with existing AppSec maturity, already experimenting with LLM-assisted code review, and you need a structured framework to systematize your prompt engineering efforts. Mantis gives you a well-reasoned architecture for progressive validation and a template library that encodes genuine security expertise. It's particularly valuable for offensive security researchers studying adversarial code analysis patterns or teams working on novel vulnerability classes (hardware RTL, IaC, firmware) where traditional SAST tools have limited rules. Skip if: You need production-grade security scanning with compliance documentation and SLA guarantees—use Semgrep or CodeQL instead. Skip if you want turnkey automation without building custom orchestration infrastructure. Skip if you're uncomfortable with prompt-based security boundaries and don't have the expertise to harden the execution environment. Skip if you're hoping for Copilot-style IDE integration or vendor support. This is a research artifact that teaches you how to build LLM security pipelines, not a finished product that does the building for you.