T3MP3ST: The Offensive Security Framework That Actually Shows Its Work

Hook

Most AI security tools publish trust-me numbers about exploitation success rates. T3MP3ST ships with npm run verify-claims that recomputes every headline metric from committed JSON artifacts—and fails your CI if you try to game the benchmarks.

Context

The offensive security industry has a reproducibility crisis. When a new AI-powered penetration testing tool claims "90% success rate on realistic targets," you have no way to verify those numbers. The evaluation datasets are proprietary, the prompts are secret, and the LLM responses are non-deterministic. Even worse, most tools require you to sign up for yet another API service, lock you into a specific model provider, and give you no visibility into whether the results came from genuine reasoning or memorized CVE descriptions.

T3MP3ST takes a different approach: it's an autonomous red teaming platform built as a meta-harness around your existing AI coding tools. Instead of competing with Claude Code or GitHub Copilot, it hijacks them as compute substrates through the Model Context Protocol (MCP). The framework ships with 35 offensive security tools (expandable to 83), an 8-operator agent architecture mapped to MITRE ATT&CK kill-chain phases, and—most importantly—a verify-claims system that lets anyone re-derive its benchmark results from committed artifacts. It's what happens when you build offensive automation with the discipline of academic research: show your work, hold out test sets, and make falsification trivial.

Technical Insight

T3MP3ST's architecture centers on a ReAct (Reasoning + Acting) loop engine that drives tool execution through a TypeScript-based Arsenal. Each "operator" in the 8-agent cell—RECON, INITIAL_ACCESS, EXECUTION, PERSISTENCE, PRIVILEGE_ESCALATION, DEFENSE_EVASION, CREDENTIAL_ACCESS, and EXFILTRATION—inherits the same tool-backed execution engine but receives specialized system prompts for different attack phases. Here's the critical insight: despite the multi-agent framing, the actual exploitation flow is currently single-agent ReAct. The framework's 90.1% pass rate on XBEN (a realistic web application penetration testing benchmark) came from one agent running a tool loop, not coordinated swarm behavior.

The keyless operation model is architecturally clever. Instead of requiring OpenAI/Anthropic API keys, T3MP3ST can hijack existing coding agents through MCP:

// T3MP3ST exposes itself as an MCP server
// Your existing coding agent (Claude Code, Cursor) calls it as a tool
{
  "name": "security_recon",
  "description": "Perform reconnaissance on target infrastructure",
  "parameters": {
    "target": "domain or IP",
    "scope": "CIDR blocks or domain patterns",
    "depth": "light | standard | deep"
  }
}

// Internally, T3MP3ST runs the ReAct loop:
const reconLoop = async (target: string, scope: ScopeDefinition) => {
  let thought = await llm.generate({
    system: RECON_OPERATOR_PROMPT,
    context: { target, previousFindings: [] }
  });
  
  while (!thought.includes('[MISSION_COMPLETE]')) {
    const tool = extractToolCall(thought);
    
    // Scope containment at tool layer
    if (!scope.permits(tool.params.target)) {
      throw new ScopeViolation('SCOPE DENIED');
    }
    
    const result = await arsenal.execute(tool);
    thought = await llm.generate({
      system: RECON_OPERATOR_PROMPT,
      context: { target, previousFindings: [...history, result] }
    });
  }
};

This means you're not paying for a new LLM API—you're treating your existing Cursor subscription as offensive security compute. The framework is provider-agnostic: it works with OpenRouter, Anthropic, OpenAI, or fully offline via Ollama/vLLM.

The scope containment mechanism is implemented at the tool layer, not the policy layer. Every Arsenal tool checks targets against the configured scope definition before executing:

// Example: DNS enumeration tool with scope checking
export const dnsEnumTool: ArsenalTool = {
  name: 'dns_enum',
  execute: async (params: { domain: string }, scope: ScopeDefinition) => {
    // Refuse off-scope execution by default
    if (!scope.domains.some(d => params.domain.endsWith(d))) {
      return { error: 'SCOPE DENIED', attempted: params.domain };
    }
    
    const subdomains = await performDNSBruteforce(params.domain);
    return { subdomains, timestamp: Date.now() };
  }
};

This is architecturally superior to prompt-based guardrails. You're not relying on the LLM to "follow rules"—the tools physically refuse to execute off-scope operations. It makes T3MP3ST safer-by-default than raw tool runners where an overeager prompt might scan the entire internet.

The verify-claims system is the framework's real innovation. Every benchmark result—XBEN pass rates, CVE-Zero detection counts, tool reliability scores—is re-computable from committed JSON artifacts:

$ npm run verify-claims

✓ XBEN: 90.1% pass@1 (derived from results/xben/runs/*.json)
✓ CVE-Zero: 8/10 exact matches (post-2026 held-out set)
✓ Tool reliability: 94.3% (35/35 tools, see results/arsenal-tests/)
✗ Multi-operator coordination: UNVERIFIED (experimental)

The CI pipeline enforces anti-fitting guards: if you modify prompts, you must re-run the held-out test set and commit new artifacts. You can't cherry-pick results or tune on the test data without leaving evidence. This is rare in AI security tools, where most results are marketing numbers with no audit trail.

The CVE-Zero validation demonstrates prompt hardening without memorization. The framework was tested on 10 CVEs published after the LLM's training cutoff (post-2026), covering 7 programming languages. It achieved 8/10 exact file/line/CWE matches—meaning the detection came from reasoning about code patterns, not regurgitating known vulnerability descriptions. The committed artifacts include the full LLM reasoning traces, so you can see exactly how it identified each vulnerability.

The coordinated-disclosure pipeline is production-ready infrastructure for responsible vulnerability hunting. When T3MP3ST finds a potential vulnerability, it runs an OSV novelty check (is this already public?), generates a live proof-of-concept, submits it to a refuter panel (LLM-based false positive filter), computes CVSS scores, and formats everything for vendor coordination. The maintainers currently have drafts in the disclosure queue, waiting for vendor response windows to close.

Gotcha

The multi-agent architecture is scaffolding, not reality. The 8-operator cell exists in the codebase—you can inspect RECON_OPERATOR_PROMPT and INITIAL_ACCESS_OPERATOR_PROMPT—but all benchmark results came from single-agent execution. The swarm's exploit coordination is explicitly documented as "experimental and unreliable." If you're expecting GPT-Researcher-style agent debate or AutoGPT-style task decomposition across the operator cell, you'll be disappointed. The current implementation is a single agent with a big tool library and phase-specific prompts, not true multi-agent orchestration.

White-box source analysis is limited to Python. Despite the CVE-Zero benchmark covering 7 languages (Python, JavaScript, Go, Rust, Java, C, C++), the general-purpose code ingestion pipeline uses regex patterns tuned for Python. The framework detected vulnerabilities in other languages during evaluation, but that was with hand-crafted ingest for each target. If you point T3MP3ST at a polyglot codebase and expect automatic analysis, it'll miss most non-Python files. The multi-model decomposition approach (parallelize analysis across code modules) also costs more tokens than direct analysis, which is backwards from the efficiency claims. Domain coverage is uneven: blockchain is reproduction-only (Damn Vulnerable DeFi), cloud/mobile/AD/binary exploitation are planned but stubbed out. The 'core' stability label only applies to web application black-box testing. Finally, scope containment is network-layer only—it won't prevent data exfiltration through LLM output encoding, steganography in generated reports, or side-channel leaks through tool usage patterns.

Verdict

Use T3MP3ST if you need a reproducible evaluation harness for offensive security LLM research, want to bolt real penetration testing tools onto your existing coding agent without new API contracts, or require verify-claims discipline for testing prompt engineering changes without memorization concerns. The keyless operation model and MCP integration make it valuable as offensive security middleware. Use it if you're doing proactive OSS vulnerability hunting and need the coordinated-disclosure pipeline infrastructure. Skip it if you need actual multi-agent swarm exploitation (that's vapor), production-grade coverage beyond web application black-box testing (cloud/mobile/binary domains are dark), white-box analysis for languages other than Python (the ingest won't cut it), or defense against a sophisticated operator (scope containment has obvious bypasses through output encoding). This is a research harness and single-agent automation layer with exceptional transparency about what works versus what's planned. Deploy the ReAct loop and Arsenal; ignore the 8-operator marketing until the coordination layer gets real benchmark validation.