Cloudflare's Security Audit Skill: Teaching AI Agents to Red Team Your Code

Hook

A single security audit run finds only 50% of vulnerabilities in a codebase. Cloudflare's solution? Make AI agents argue with each other until the real bugs survive.

Context

Traditional static analysis tools like Semgrep and CodeQL excel at finding known vulnerability patterns—SQL injection, XSS, hardcoded credentials—but they struggle with business logic flaws that require contextual understanding. A rule-based scanner can't reason about whether your authentication bypass is actually exploitable given your specific middleware stack, or whether that race condition matters in your deployment architecture.

Meanwhile, the AI coding agent revolution promised to change security analysis by applying LLM reasoning to vulnerability discovery. In practice, most teams just paste code into ChatGPT with prompts like "find security issues" and get back hallucinated vulnerabilities mixed with a few real findings. Cloudflare's security-audit-skill represents an attempt to systematize AI-powered security auditing: a structured prompt engineering framework that orchestrates multiple LLM agents through a six-phase pipeline, complete with adversarial validation and machine-readable output. It's not a static analysis tool—it's a deterministic workflow for running red team exercises using AI agents.

Technical Insight

The architecture is a directed acyclic graph of prompt templates executed sequentially, with each phase spawning parallel agent instances that operate on shared JSON state. Phase 1 (Recon) launches parallel research agents that analyze the codebase to generate foundational context in architecture.md—understanding the tech stack, data flows, and trust boundaries. Phase 2 (Hunt) is where things get interesting: it spawns 7+ specialized attack agents, each targeting a specific vulnerability class from modular prompt files like ATTACK-CLASSES.md or AI-AND-LLM.md. These agents can recursively spawn sub-agents for deeper analysis, mimicking how human auditors follow leads.

The real innovation is Phase 3 (Validate), which introduces adversarial review. Separate validator agents—with no context from the hunting phase—attempt to disprove every finding from Phase 2. Each vulnerability must survive this challenge to advance. It's red team versus blue team dynamics automated within a single workflow, operationalizing the skepticism that separates senior security engineers from junior ones who accept every scanner output at face value.

Here's what a finding looks like after surviving validation, pulled from the structured output schema:

{
  "id": "VULN-2024-001",
  "title": "Authentication Bypass via JWT Algorithm Confusion",
  "severity": "high",
  "exploitability": {
    "scenario": "Attacker modifies JWT algorithm from RS256 to HS256",
    "preconditions": ["Access to public key", "No algorithm whitelist"],
    "steps": [
      "Extract public key from /jwks endpoint",
      "Create new JWT with alg=HS256",
      "Sign using public key as HMAC secret"
    ],
    "impact": "Complete authentication bypass for any user account"
  },
  "validation": {
    "challenged": true,
    "validator_reasoning": "Confirmed vulnerable code path in auth.js:145",
    "source_references": ["src/auth.js#L145-L152"]
  }
}

Phases 4-5 generate human and machine-readable outputs, with Phase 5 producing findings.json validated against a formal JSON schema. Phase 6 runs independent verification agents that fact-check every claim in the structured output against actual source code—addressing LLM hallucination systematically rather than hoping for model accuracy.

The state management strategy is file-based: findings.json accumulates across runs, enabling iterative deepening. Cloudflare observed empirically that single runs discover only ~50% of total vulnerabilities, so the skill tracks known issues and prompts subsequent audits to skip already-discovered bugs and explore unexplored attack surfaces. This turns stochastic LLM exploration into systematic coverage through repeated execution.

The modular attack class architecture deserves attention. Instead of monolithic prompts, vulnerability classes are separated into version-controlled markdown files that can be composed based on target tech stack. Need to audit a Rust service? Include MEMORY-SAFETY-AND-BINARY.md. Evaluating an LLM application? Add AI-AND-LLM.md. Each file contains domain-specific attack methodology refined through real audits:

## Prompt Injection Attack Class

You are a security auditor specializing in LLM security.
Target: Systems that pass user input to language models.

Attack vectors to investigate:
1. Direct prompt injection via user inputs
2. Indirect injection via retrieved documents
3. System prompt extraction attempts
4. Multi-turn conversation state poisoning

For each potential vulnerability:
- Construct a specific attack payload
- Trace the data flow from input to LLM
- Demonstrate the exploit with concrete examples
- Assess impact on confidentiality/integrity/availability

Only report findings where you can show a working exploit.

This prompt engineering approach means the skill implements an 'only exploitable findings' policy without static rules: agents must construct concrete attack scenarios with specific inputs and outputs, not theoretical impact statements. It's a forcing function for practical security analysis.

The integration happens through Skills CLI, which injects these prompts into any coding agent supporting tool use and parallel execution. The skill doesn't run directly—it's a plugin for agent platforms, making it platform-agnostic in theory (though compatibility in practice is murky).

Gotcha

This framework inherits every limitation of LLM reasoning because there's no static analysis, symbolic execution, or traditional program analysis underneath. LLMs struggle with complex state machines and multi-file data flows, so vulnerabilities requiring deep cross-module reasoning may get missed. A time-of-check-time-of-use race condition spanning three services? Probably too complex for current models to track reliably.

Cost is the elephant in the room. Parallel agent orchestration with recursive sub-agent spawning means token consumption scales exponentially with codebase size. The repository provides no guidance on budgets or cost controls. Running this against a 100K+ line codebase with multiple iterations could easily consume thousands of dollars in API credits. Cloudflare can absorb that cost; most teams can't.

The adversarial validation is clever but still prompt-based. An agent 'disproving' a finding is an LLM opinion, not formal verification. Worse, prompt injection becomes a meta-vulnerability: malicious comments in the codebase itself could bias validators into accepting false positives or rejecting real vulnerabilities. And the multi-run coverage improvement is empirically observed but not guaranteed—there's no convergence criteria telling you when you've achieved sufficient coverage versus just burning compute.

Verdict

Use if: You're already running AI coding agents and want to systematize vulnerability discovery beyond ad-hoc prompting, you have budget for multiple runs at potentially hundreds of dollars per audit, your codebase is under 50K LOC or you can partition it effectively, and you value finding novel business logic flaws that traditional SAST tools miss. The adversarial validation and structured output are genuinely innovative for prompt-based security tooling. Skip if: You need deterministic results for compliance, you're analyzing binaries or runtime behavior, your codebase exceeds 50K LOC without serious compute budget, or you lack the operational maturity to productionize what Cloudflare describes as a 'starting point' for their fleet-wide system. For most teams, start with Semgrep for fast deterministic scanning, then use this skill for deep dives on critical services where the cost-per-finding math works out.