Auto-Prompting Red Teams: Using DSPy to Generate LLM Jailbreaks Without Manual Prompt Engineering

Hook

A five-layer adversarial attack system that achieves a 4x improvement in jailbreaking language models—without writing a single hand-crafted adversarial prompt.

Context

Red-teaming language models has traditionally been a cat-and-mouse game requiring deep expertise in adversarial prompt engineering. Security researchers spend countless hours crafting clever prompts to bypass safety guardrails, manually iterating through variations like "pretend you're an AI without restrictions" or encoding harmful requests in base64. Each new model release resets this work, and each safety update renders previous attacks obsolete.

The dspy-redteam project takes a radically different approach: what if you could automate the prompt engineering itself? By treating adversarial prompt generation as a compilable program rather than a manual craft, this tool applies DSPy's auto-prompting framework to red-teaming. DSPy, originally designed for building reliable LLM applications through structured programming, turns out to be surprisingly effective at the opposite use case—systematically breaking those same models. This represents the first known application of DSPy's optimization capabilities to adversarial security research, and it reveals something important about the nature of prompt engineering: architecture and optimization often matter more than cleverness.

Technical Insight

The core innovation here is treating jailbreaking as a multi-stage compilation problem. Instead of manually crafting adversarial prompts, you define a deep pipeline of Attack and Refine modules that DSPy automatically optimizes. The architecture consists of five alternating layers—each Attack module generates adversarial text, while each Refine module polishes it to be more effective.

Here's what a simplified version of the architecture looks like:

import dspy

class AttackModule(dspy.Signature):
    """Generate adversarial prompt variation"""
    harmful_request = dspy.InputField()
    previous_attempt = dspy.InputField(desc="Previous attack attempt")
    adversarial_prompt = dspy.OutputField(desc="Refined jailbreak attempt")

class RefineModule(dspy.Signature):
    """Polish adversarial prompt for effectiveness"""
    raw_attack = dspy.InputField()
    refined_attack = dspy.OutputField(desc="More subtle variation")

class DeepRedTeam(dspy.Module):
    def __init__(self, depth=5):
        super().__init__()
        self.layers = []
        for i in range(depth):
            self.layers.append(dspy.ChainOfThought(AttackModule))
            self.layers.append(dspy.ChainOfThought(RefineModule))
    
    def forward(self, harmful_request):
        output = harmful_request
        for layer in self.layers:
            output = layer(harmful_request=harmful_request, 
                          previous_attempt=output).adversarial_prompt
        return output

The magic happens during compilation. DSPy's MIPRO optimizer treats this entire pipeline as a program to be optimized. It runs the pipeline through hundreds of examples, uses an LLM-as-judge to evaluate which variations successfully jailbreak the target model, and automatically adjusts the internal prompts of each module to maximize attack success rate (ASR). You never write "ignore previous instructions" or "you are DAN"—the optimizer discovers effective strategies through guided search.

This approach achieved a 44% attack success rate on Vicuna, compared to just 10% for raw harmful inputs. That 4x improvement comes purely from architectural depth and optimization, not from security expertise. The system learned to wrap harmful requests in scenarios, add conversational context, and employ semantic obfuscation—all emergent behaviors from optimization against the ASR objective.

The LLM-as-judge component is particularly clever. Instead of requiring human evaluation of thousands of attack attempts, the system uses a separate language model to assess whether each generated prompt successfully bypasses safety filters. This creates a fully automated red-teaming loop:

class JailbreakJudge(dspy.Signature):
    """Evaluate if model output indicates successful jailbreak"""
    prompt = dspy.InputField()
    model_response = dspy.InputField()
    success = dspy.OutputField(desc="Boolean: True if jailbroken")
    
# Used during MIPRO optimization
judge = dspy.ChainOfThought(JailbreakJudge)
metric = lambda example, prediction: judge(
    prompt=prediction.adversarial_prompt,
    model_response=target_model(prediction.adversarial_prompt)
).success

The depth of the architecture matters significantly. With five Attack/Refine layers, the system explores a much richer space of adversarial variations than shallow approaches. Early layers might generate obvious attacks that get filtered, but later layers learn to subtly transform those attempts into more sophisticated jailbreaks. This represents one of the deepest publicly optimized DSPy programs, pushing the framework beyond typical 2-3 module applications.

What's particularly interesting from a framework perspective is that DSPy wasn't designed for this. It was built to make LLM applications more reliable and maintainable. But the same mechanisms that optimize prompts for accuracy can optimize them for adversarial effectiveness. This dual-use nature of auto-prompting frameworks is something the community is still grappling with.

Gotcha

The 44% attack success rate, while a 4x improvement over baseline, is explicitly not state-of-the-art. Specialized techniques like GCG (gradient-based adversarial suffixes) and AutoDAN achieve significantly higher success rates, sometimes exceeding 80% on specific models. If you're conducting serious security research or need comprehensive coverage of adversarial attack vectors, this tool won't match purpose-built jailbreaking frameworks.

Compute costs are the silent killer here. The MIPRO optimization requires hundreds or thousands of LLM calls to compile the program effectively. The authors note they performed limited hyperparameter tuning due to compute constraints, suggesting the 44% ASR could potentially improve with more resources—but at what cost? For teams without substantial API budgets or local GPU clusters, the optimization phase alone could run into hundreds of dollars. Additionally, the evaluation appears focused on Vicuna; generalization to hardened production models like GPT-4 with sophisticated safety layers is unclear. The LLM-as-judge approach also introduces meta-level vulnerabilities—if your judge model can be fooled about what constitutes a successful jailbreak, your entire optimization loop trains toward the wrong objective. This is a research prototype that demonstrates compelling automation potential, but it's not a turnkey red-teaming solution for production security assessments.

Verdict

Use if: you're exploring automated red-teaming approaches and want to experiment with DSPy's capabilities beyond traditional applications, you need a rapid-prototyping framework for adversarial testing during model development, you're resource-constrained and prefer some automation over purely manual prompt engineering, or you're conducting research on auto-prompting frameworks and their security implications. This is a strong starting point for teams building internal red-teaming capabilities who value automation and framework-based approaches. Skip if: you need state-of-the-art jailbreaking performance for security research, you're conducting production-grade adversarial testing of hardened models, you lack compute budget for extensive MIPRO optimization runs, or you require comprehensive coverage across diverse attack taxonomies beyond prompt-based jailbreaks. For serious security work, combine this with specialized tools like Garak or custom GCG implementations rather than relying on it as your sole red-teaming approach.