InterCode-CTF: How Basic Prompts Beat Complex Agents at 95% Success in Security Testing

Hook

What if the reason LLMs struggle with security testing isn't capability—it's just bad prompting? A new benchmark suggests we've been overengineering our AI security agents while underestimating what simple approaches can achieve.

Context

The cybersecurity community has been cautiously optimistic about using large language models for penetration testing and vulnerability discovery. Early benchmarks like the original InterCode-CTF showed promising but limited results, with the best systems achieving around 72% success on standardized Capture The Flag challenges. The conventional wisdom emerged that LLMs needed complex agent architectures—multi-step reasoning frameworks, specialized tool chains, and sophisticated memory systems—to handle the intricate problem-solving required for security work.

PalisadeResearch's InterCode framework challenges this assumption head-on. By creating a reproducible evaluation harness using Docker-isolated CTF environments, the researchers discovered something counterintuitive: stripping away complexity and focusing on prompt engineering alone could push success rates to 95%. This isn't just an incremental improvement—it's a fundamental reassessment of where the bottleneck lies in AI-assisted security testing. The framework itself is designed for researchers who need to systematically evaluate LLM capabilities on security tasks, providing the infrastructure to run controlled experiments where models interact with realistic CTF challenges through command execution.

Technical Insight

InterCode's architecture separates three critical concerns: environment isolation, experiment orchestration, and result analysis. At its core, the system uses Docker containers to provide sandboxed CTF challenge environments. Each challenge runs in its own container with specific configurations, ensuring that LLM interactions don't pollute shared state and that experiments are perfectly reproducible across runs.

The orchestration layer manages the conversation loop between the LLM and the challenge environment. Here's a simplified version of how the interaction works:

# Simplified interaction loop from the framework
class CTFEnvironment:
    def __init__(self, challenge_config):
        self.docker_client = docker.from_env()
        self.container = self.setup_container(challenge_config)
        self.session_log = []
    
    def execute_command(self, command):
        """Execute command in Docker container and return output"""
        exec_result = self.container.exec_run(
            cmd=command,
            stdout=True,
            stderr=True
        )
        output = exec_result.output.decode('utf-8')
        self.session_log.append({
            'command': command,
            'output': output,
            'exit_code': exec_result.exit_code
        })
        return output
    
    def check_flag(self, submitted_flag):
        """Validate if submitted flag matches expected solution"""
        return submitted_flag.strip() == self.expected_flag

The framework then wraps this environment with an LLM agent that receives the challenge description, executes commands, observes outputs, and iteratively works toward the solution. The critical insight from the research is that the prompt structure matters far more than architectural complexity. Rather than implementing complex planning systems or tool-use frameworks, the researchers found that clearly structured prompts with explicit instructions about available tools and expected outputs dramatically improved performance.

The experiment configuration uses YAML files to define challenge parameters, model settings, and evaluation criteria. This declarative approach makes it trivial to run comparative experiments—testing the same challenge across different models, prompt strategies, or temperature settings. The statistics module then aggregates results across multiple runs, computing success rates, average attempts to solution, and command patterns.

One particularly clever design choice is the logging infrastructure. Every interaction—prompts sent, commands executed, outputs received, and model reasoning—gets captured in structured JSON logs. This enables post-hoc analysis of failure modes. Researchers can examine exactly where models get stuck, which types of commands they favor, and how they respond to error messages. This level of introspection is essential for understanding not just whether a model solved a challenge, but how it approached the problem.

The Docker integration deserves special attention. Each CTF challenge is defined as a Dockerfile with specific vulnerabilities, configuration files, and flag placement. When an experiment starts, the framework builds the image, spawns a container, and provides the LLM with network access to interact with services or shell access to execute commands. This mirrors real penetration testing workflows where attackers have limited initial access and must enumerate, exploit, and escalate privileges. The framework supports both network-based challenges (web vulnerabilities, service exploits) and system-level challenges (privilege escalation, file system enumeration).

What makes the 95% success rate remarkable is that it was achieved without reinforcement learning, without fine-tuning on security-specific datasets, and without complex agent frameworks. The models used were standard GPT-4 instances with carefully crafted system prompts that emphasized methodical exploration, tool usage patterns common in security testing, and clear formatting for command execution versus reasoning. This suggests that current frontier models already possess substantial security knowledge and reasoning capability—the challenge is elicitation, not fundamental capability.

Gotcha

The framework's dependency on Docker is both its strength and its Achilles heel. While containerization ensures perfect reproducibility and safety (critical when testing exploits), it creates a steep setup requirement. You need a properly configured Docker daemon with appropriate permissions, which can be problematic in corporate environments with restricted developer workstations or cloud development environments where Docker-in-Docker scenarios introduce additional complexity. The documentation assumes Docker familiarity, so expect friction if you're not already comfortable with container networking and volume management.

More limiting is the hardcoded OpenAI API integration. The codebase appears tightly coupled to OpenAI's API structure, with API keys configured in static files and response parsing tuned to OpenAI's specific JSON formats. If you want to test local models via Ollama, evaluate Claude or other providers, or experiment with custom fine-tuned models, you're looking at significant code modifications. For a research framework claiming to evaluate "LLM capabilities" broadly, this vendor lock-in is disappointing. The lack of abstraction around the model provider means comparative studies across different model families require forking and adapting the code rather than simple configuration changes. Additionally, the CTF challenges themselves aren't extensively documented—while the Docker definitions exist, there's limited explanation of difficulty levels, vulnerability types covered, or how to contribute new challenges to expand the benchmark.

Verdict

Use InterCode if you're conducting academic research on LLM security capabilities, need reproducible benchmarks for comparing prompt engineering strategies in cybersecurity contexts, or want to validate claims about AI-assisted penetration testing with controlled experiments. It's particularly valuable if you're investigating the gap between model capabilities and practical performance, since the comprehensive logging reveals exactly how models approach security challenges. Skip it if you need production penetration testing tools (this is purely an evaluation harness), want to test non-OpenAI models without significant code modifications, lack Docker infrastructure or the permissions to run it, or need ready-to-use automation rather than a research framework. This is a tool for people asking "how well can LLMs do security testing?" not "help me secure my application." The 95% success rate is the compelling headline, but the real contribution is the infrastructure for systematic evaluation that lets you ask nuanced questions about why certain approaches work and others don't.