How AI Worms Could Hijack Your RAG Pipeline: Inside the First Self-Replicating Prompt Attack

Hook

A single malicious prompt can turn your RAG-powered email assistant into a self-replicating virus that infects 20 other AI systems within three days—and most existing security tools won't even notice.

Context

As enterprises rush to deploy Retrieval-Augmented Generation systems for customer service, email automation, and internal tooling, they're inadvertently creating a new attack surface that traditional security models never anticipated. Unlike conventional applications where code execution boundaries are well-defined, RAG systems blur the line between data and instructions. When your AI assistant retrieves context from external sources—emails, documents, database entries—it treats that content as trusted input for generation. This architectural decision, fundamental to how RAG works, creates an opportunity for adversarial prompts to hijack the generation process.

The StavC/Here-Comes-the-AI-Worm repository presents the first academic demonstration of what happens when prompt injection meets self-replication: AI worms that propagate through GenAI ecosystems like biological viruses through a population. Unlike traditional prompt injection attacks that compromise a single session, these adversarial self-replicating prompts (ASRPs) embed instructions that cause the LLM to both execute malicious actions and propagate the attack payload through normal system operations. When a compromised AI email assistant processes an infected message, it doesn't just leak data or generate spam—it spreads the attack to every recipient it contacts, creating exponential propagation across interconnected AI systems.

Technical Insight

The core innovation behind RAGworm lies in crafting prompts that survive the RAG retrieval process and manipulate the LLM into two simultaneous behaviors: executing a malicious action and embedding the attack vector into generated output. The repository demonstrates this through what the researchers call "indirect prompt injection via RAG," where the attack payload exists in retrieved context rather than direct user input.

The attack architecture works like this: an attacker sends a seemingly benign email containing a carefully crafted prompt to a victim using a RAG-powered email assistant. When the victim's AI processes this email, the RAG system retrieves the malicious content as context. The LLM, unable to distinguish between legitimate system instructions and adversarial prompts in the retrieved context, follows both. For instance, a prompt might instruct the model to "include this exact phrase in any response" while simultaneously executing data exfiltration. The genius is that the exfiltration happens through normal email operations—the AI assistant sends replies that both answer the user's intent and forward sensitive information to attacker-controlled addresses.

The researchers tested propagation across simulated GenAI ecosystems and observed super-linear growth. A single infected client could compromise 20 new systems within 1-3 days, not through technical exploits but through normal AI-to-AI communication patterns. This isn't a vulnerability in the LLM itself—it's an emergent property of how RAG systems chain together.

The defensive component, DonkeyRail, takes a machine learning approach rather than pattern matching or keyword filtering. The guardrail sits between the RAG retrieval layer and the LLM inference, analyzing retrieved context for indicators of self-replicating behavior before passing it to generation. Here's the conceptual flow:

# Simplified DonkeyRail architecture concept
class DonkeyRailGuardrail:
    def __init__(self, classifier_model):
        self.classifier = classifier_model  # ML model trained on ASRP patterns
        self.latency_threshold_ms = 50
    
    def screen_rag_context(self, retrieved_documents, query):
        """
        Screens RAG-retrieved content before passing to LLM.
        Returns filtered context and threat assessment.
        """
        threat_scores = []
        safe_documents = []
        
        for doc in retrieved_documents:
            # Extract features: prompt injection indicators,
            # self-reference patterns, propagation instructions
            features = self.extract_asrp_features(doc)
            
            # Classify with minimal latency (7.6-38.3ms measured)
            score = self.classifier.predict_proba(features)
            
            if score < 0.5:  # Threshold tuned for 0.017 FPR
                safe_documents.append(doc)
            else:
                threat_scores.append({
                    'document': doc[:100],  # Log snippet
                    'confidence': score,
                    'features': features
                })
        
        return safe_documents, threat_scores
    
    def extract_asrp_features(self, document):
        """
        Feature engineering for self-replicating prompt detection.
        """
        return {
            'self_reference_density': self.count_self_references(document),
            'instruction_markers': self.detect_instruction_patterns(document),
            'propagation_indicators': self.detect_propagation_commands(document),
            'context_boundary_violations': self.check_role_confusion(document)
        }

The classifier achieves 1.0 true positive rate with only 0.017 false positive rate because it targets structural patterns specific to self-replicating prompts rather than generic prompt injection. The key insight is that ASRPs must contain both execution instructions and propagation logic, creating a detectable signature even when obfuscated.

What makes DonkeyRail particularly interesting is its robustness against out-of-distribution attacks. The researchers tested it against jailbreaking techniques not included in training data—variations of adversarial prompts using different encoding schemes, linguistic structures, and propagation mechanisms. The guardrail maintained effectiveness because it learned generalizable features of self-replication (recursive instructions, output manipulation directives, context poisoning patterns) rather than specific attack strings.

The latency characteristics deserve attention. Adding 7.6-38.3ms to RAG inference is negligible for most applications, especially considering that RAG retrieval and LLM generation typically take hundreds of milliseconds to seconds. This makes the defense practical for production deployment, unlike heavyweight content analysis systems that might double response times.

The repository includes Jupyter notebooks demonstrating both attack construction and defense evaluation, using datasets of benign prompts, adversarial prompts, and various LLM backends (GPT-4, Gemini Pro). The experimental setup simulates a GenAI ecosystem where multiple clients communicate through AI-mediated channels, allowing researchers to measure propagation dynamics and defense effectiveness under realistic conditions.

Gotcha

This is academic research code, not a hardened security product. The repository explicitly notes that some components are legacy code from the initial research phase and aren't actively maintained. If you're hoping to drop DonkeyRail into your production RAG pipeline, you'll need significant engineering work to adapt the proof-of-concept classifier into a production-grade system with proper error handling, monitoring, and performance optimization.

The effectiveness claims are based on specific threat models and LLM configurations tested in the research. Real-world RAG systems vary enormously in architecture—different embedding models, retrieval strategies, chunking approaches, and LLM backends all affect how adversarial prompts behave and how defenses perform. The 1.0 TPR and 0.017 FPR numbers come from controlled experiments with defined attack types. Your mileage will vary with different LLMs, especially newer models with different training regimes or architectural innovations. The defense also assumes you can intercept and analyze RAG context before generation, which may not fit all deployment architectures, particularly those using managed RAG services where you lack access to intermediate layers. Additionally, the attack demonstrations focus on email-based propagation scenarios; other RAG applications like chatbots, document analysis systems, or code assistants present different propagation dynamics that may require adapted defenses.

Verdict

Use if you're building or securing RAG-based applications and need to understand the emerging threat landscape of AI-to-AI attacks. This research is essential reading for security teams deploying GenAI in production, especially for applications where AI systems communicate with each other or process user-generated content through RAG. The DonkeyRail concept provides a solid foundation for implementing your own guardrails, and the attack demonstrations are invaluable for red team exercises and security testing. It's also perfect for researchers exploring adversarial ML in the GenAI era or anyone designing security architectures for interconnected AI systems. Skip if you're looking for a plug-and-play security solution or aren't working with RAG architectures. This is research code that requires significant adaptation for production use. If your AI systems don't retrieve and incorporate external content into generation, or if they operate in isolation without communicating with other AI systems, the specific threat model here doesn't apply to you. Also skip if you need immediate deployment—invest time understanding the concepts, then build production-grade implementations suited to your specific architecture and threat model.