RAGworm: The First Self-Replicating AI Prompt Attack on GenAI Ecosystems
Hook
A single malicious email can infect 20 GenAI-powered email assistants within three days, forcing them to spam, phish, and exfiltrate data autonomously. Welcome to the era of AI worms.
Context
As organizations rush to embed Large Language Models into productivity tools—email assistants, document processors, customer service bots—they’re creating an interconnected ecosystem that mirrors the early internet. And just like the early internet, this ecosystem has a worm problem.
Retrieval-Augmented Generation (RAG) powers most production GenAI applications because it grounds LLM responses in real data: your emails, company documents, customer records. When you ask an AI assistant a question, it retrieves relevant content from your inbox or knowledge base, feeds it to an LLM, and generates a contextual response. This architecture is powerful—but it also means these systems automatically ingest untrusted external content. Researchers from Technion, Cornell Tech, and Tel Aviv University discovered that this automatic retrieval creates an exploitable attack surface. Their proof-of-concept, RAGworm, demonstrates the first practical self-replicating prompt attack: a malicious email that forces GenAI assistants to propagate themselves, performing adversarial actions at scale. Unlike theoretical prompt injection vulnerabilities, RAGworm was demonstrated against real GenAI email assistant systems, exhibiting super-linear propagation where one infected client compromises 20 others within 1-3 days.
Technical Insight
RAGworm exploits a fundamental architectural assumption in RAG systems: that retrieved documents are informational rather than instructional. When a GenAI email assistant processes incoming mail, it retrieves message content to answer user queries like “summarize my unread emails” or “draft a reply to the latest message from finance.” RAGworm embeds adversarial self-replicating prompts within email bodies that override the LLM’s original instructions through indirect prompt injection.
The attack chain works like this: An attacker sends a crafted email containing a self-replicating prompt to a victim using a RAG-powered email assistant. When the victim queries their assistant, the system retrieves the malicious email, feeding its contents into the LLM context window. The adversarial prompt hijacks the generation process, forcing the assistant to compose and send emails containing copies of the same malicious prompt to the victim’s contacts. Each newly compromised assistant repeats the cycle, creating exponential propagation.
The researchers tested multiple payload variants available in their Datasets and Demos directories, including demonstrations of RAGworm with different payloads, retrieval document analysis from email assistants, and a full end-to-end demo on Gemini Workspace. The Self_Replicating_Test directory contains code validating that various LLMs successfully replicate these prompts when embedded in retrieved documents. The Worm_Evaluation code measures two critical metrics: retrieval rate (how often the RAG system actually fetches the malicious content) and success rate (how often retrieved prompts successfully hijack the LLM output). According to their findings, RAGworm achieves super-linear propagation—each compromised client infects approximately 20 new victims within the first 1-3 days depending on email volume.
To defend against this attack class, the researchers developed DonkeyRail, a machine learning-based guardrail that analyzes retrieved documents before they enter the LLM context. Rather than attempting to detect specific malicious keywords (easily evaded through obfuscation), DonkeyRail trains a classifier to recognize structural patterns common to self-replicating prompts: imperative instructions, context window manipulation attempts, instruction override language, and self-referential replication directives.
The DonkeyRail directory contains the complete implementation including training data, preprocessing pipelines, models, and evaluation scripts. At the end of DonkeyRail.ipynb, the researchers provide a pipeline showing how to integrate the guardrail in a real-world scenario. The guardrail processes each retrieved document before feeding it to the LLM, classifying whether the content exhibits self-replicating characteristics.
Their evaluation shows DonkeyRail achieves a true positive rate of 1.0 on their evaluation dataset with a false-positive rate of 0.017, meaning it detected every worm in testing while incorrectly flagging only 1.7% of legitimate documents. Critically, the latency overhead is minimal: 7.6-38.3ms depending on the number of retrieved documents. For a typical RAG query that takes 2-5 seconds to complete (retrieval + LLM generation), adding 10-40ms for security screening is operationally negligible.
The researchers also tested robustness against out-of-distribution attacks—worms using unseen jailbreaking prompts and various worm use cases not present in the training data. DonkeyRail maintained high detection rates, suggesting the classifier generalizes to structural patterns rather than memorizing specific attack strings. This is crucial because adversarial prompts evolve rapidly; defenses that rely on exact pattern matching become obsolete within weeks.
Gotcha
The repository provides research code and evaluation results rather than production-ready software. The DonkeyRail implementation includes models, preprocessing steps, training data, and evaluation scripts, with a pipeline demonstration at the end of DonkeyRail.ipynb showing real-world integration. However, deploying this in production environments will require additional engineering to handle operational concerns like continuous monitoring, retraining as attack patterns evolve, and integration with existing security infrastructure.
The research focuses heavily on email assistant use cases. The Demos directory shows RAGworm demonstrations including retrieval behavior on Copilot (work email assistant) and Gemini Workspace (work email assistant), plus a full end-to-end demo on Gemini Workspace. While this validates the attack against real systems, generalization to other RAG architectures (code assistants, document Q&A systems, customer service bots) receives less empirical validation. The attack surface differs across application types: a code assistant’s RAG retrieves from repositories and documentation rather than emails, potentially requiring different prompt injection techniques. DonkeyRail’s feature extraction and training data appear optimized for email-based worms; adapting it to other domains may require retraining with domain-specific attack samples.
Finally, this is fundamentally an arms race. DonkeyRail detects self-replicating prompts by recognizing structural patterns. Sophisticated adversaries will evolve evasion techniques: obfuscating replication logic through steganography, splitting malicious instructions across multiple retrieved documents, or using semantic manipulation rather than explicit commands. The strong detection performance reported reflects evaluation against the datasets included in the repository; maintaining that performance long-term requires continuous model updates as attackers adapt. The README notes that the Legacy_Arxiv_V1 directory contains code from the original ArXiv paper for reference, indicating the research has evolved beyond the initial publication.
Verdict
Use this research if you’re building or securing RAG-based GenAI applications, especially those that auto-retrieve user-generated or external content. The threat model is validated through demonstrations on real GenAI email assistant systems, not just theoretical vulnerabilities. Treat this repository as a blueprint: study the attack methodology in Worm_Evaluation for red-teaming your own systems, examine the DonkeyRail guardrail architecture and its pipeline integration example, and use their feature engineering approach as a starting point for building custom defenses. The Datasets directory contains evaluation data for both worm propagation and guardrail performance. Security teams should particularly focus on understanding the propagation dynamics—the super-linear spread where each infected client compromises 20 others within 1-3 days—to assess exposure risk in interconnected AI ecosystems. The Self_Replicating_Test code can help validate whether your LLMs are susceptible to this attack class. Skip if you’re expecting a plug-and-play security product with vendor support, monitoring dashboards, and automatic updates—this is academic research code requiring engineering effort to productionize. Also skip if your GenAI applications don’t use RAG or don’t retrieve untrusted external content; the attack requires automatic ingestion of adversarial documents. For those scenarios, focus instead on traditional prompt injection defenses targeting direct user input manipulation.
