Inside RAGET: How Giskard Automates RAG System Testing with Generated Test Cases

Hook

Most RAG applications fail silently in production—returning plausible-sounding nonsense because nobody thought to test whether the retrieval actually retrieved the right context. RAGET exists to prevent exactly that failure mode.

Context

Retrieval-Augmented Generation has become the de facto pattern for building LLM applications that need to reference proprietary or current information. You retrieve relevant documents from a vector store, inject them into a prompt, and let the LLM synthesize an answer. Simple in theory, catastrophically fragile in practice.

The problem isn't building RAG systems—it's validating they actually work. Traditional software testing assumes deterministic behavior. RAG systems combine three non-deterministic components: semantic search (which document chunks match?), LLM generation (what answer gets produced?), and the interaction between them (did the model actually use the retrieved context?). Manual testing doesn't scale when each query might surface different chunks and generate different responses. RAGET, part of the Giskard AI testing ecosystem, attempts to solve this by automatically generating test cases that probe for common RAG failure modes: irrelevant retrieval, context ignored by the generator, hallucinations despite correct context, and adversarial inputs that break the pipeline.

Technical Insight

The raget_demo repository demonstrates a test generation workflow that inverts the typical RAG evaluation approach. Instead of manually crafting question-answer pairs, RAGET analyzes your knowledge base and automatically generates evaluation datasets targeting specific failure scenarios.

The core workflow follows this pattern: ingest your document corpus, let RAGET generate questions based on document content, execute those questions through your RAG pipeline, then apply automated evaluation metrics to the results. What makes this interesting is the question generation strategy—RAGET creates different question types to stress-test different components. Simple factual questions test whether retrieval finds the right chunks. Multi-hop questions test whether the generator can synthesize across multiple retrieved documents. Adversarial questions test whether the system hallucinates when no relevant context exists.

A typical evaluation setup looks like this:

from giskard.rag import generate_testset, evaluate
from giskard.rag import QATestset

# Your RAG pipeline wrapped as callable functions
def retriever(question: str) -> list[str]:
    # Vector search returns relevant chunks
    embeddings = embed(question)
    chunks = vector_store.similarity_search(embeddings, k=5)
    return chunks

def generator(question: str, context: list[str]) -> str:
    # LLM generates answer from retrieved context
    prompt = f"Context: {context}\n\nQuestion: {question}\n\nAnswer:"
    return llm.generate(prompt)

# Generate test cases from your knowledge base
testset = generate_testset(
    knowledge_base=documents,
    num_questions=100,
    question_types=["simple", "complex", "distracting", "adversarial"]
)

# Run evaluation
results = evaluate(
    testset=testset,
    retriever=retriever,
    generator=generator
)

# Results include metrics for:
# - Retrieval precision/recall
# - Answer faithfulness (did it use the context?)
# - Answer correctness (vs. generated ground truth)
# - Hallucination detection

The evaluation metrics are where RAGET differentiates itself from simple accuracy checks. It employs an LLM-as-judge pattern to assess answer quality, but does so along multiple dimensions simultaneously. Faithfulness checks whether the generated answer can be verified using only the retrieved context—catching cases where your model ignored retrieval and relied on parametric knowledge. Correctness compares against reference answers, accounting for semantic equivalence rather than exact string matching. The retrieval metrics check whether the top-k chunks actually contained information necessary to answer the question.

What's particularly clever is how RAGET handles the bootstrapping problem. To evaluate answer correctness, you need ground truth answers. But manually creating hundreds of question-answer pairs defeats the automation purpose. RAGET generates reference answers using the same LLM against known-good context, then uses those synthetic references for evaluation. This works because you're not testing whether the LLM can answer questions—you're testing whether your RAG pipeline delivers the right context and whether your prompts reliably produce consistent outputs given that context.

The demo notebooks show evaluation results broken down by question type, revealing patterns like "90% accuracy on simple questions but 45% on multi-hop questions" or "retrieval works but generator ignores context in 30% of cases." These insights point directly to which component needs improvement—a significant advantage over end-to-end metrics that just report "system answers 73% of questions correctly" without diagnosing why.

Gotcha

The biggest limitation is one RAGET shares with all LLM-as-judge evaluation frameworks: you're using an LLM to evaluate LLM outputs, which means the evaluator inherits all the biases and failure modes of the model doing the judging. If your evaluation LLM has strong opinions about answer style or struggles with certain domains, those preferences contaminate your metrics. The demo doesn't address calibration—how often does the LLM judge disagree with human judgment on the same answer?

Another practical constraint is cost and latency. Generating 100 test questions, executing them through your RAG pipeline, and running multiple LLM-based evaluation metrics on each answer can consume thousands of tokens per question. For a comprehensive test suite, you're looking at potentially hundreds of thousands of tokens, which translates to meaningful API costs if using commercial models and significant runtime if using local models. The demo notebooks are single-run examples—they don't address how to maintain test suites over time as your knowledge base evolves, or how to run these evaluations in CI/CD pipelines where 10-minute test runs aren't acceptable. The HTML output suggests these are meant as exploratory analysis tools, not automated regression tests.

Verdict

Use if: You're building a RAG application and need to move beyond manual testing but don't have the resources to create comprehensive evaluation datasets by hand. RAGET shines when you need to quickly assess whether your RAG pipeline has fundamental problems—wrong chunks being retrieved, context being ignored, systematic hallucinations. It's particularly valuable during prototyping when you're comparing different embedding models, chunk sizes, or retrieval strategies and need quantitative feedback on what's actually working. The automated test generation gives you broad coverage across failure modes without the tedium of manual test case creation.

Skip if: You need production-grade evaluation infrastructure with deterministic tests, low latency requirements, or tight control over evaluation criteria. The demo nature of this repository means you're getting reference implementation ideas, not battle-tested code ready to drop into your CI pipeline. If your domain requires human-verified ground truth (medical, legal, financial applications where LLM-as-judge isn't acceptable), or if you already have established evaluation datasets and just need to run metrics against them, look at more mature frameworks like RAGAS or the full Giskard library. Also skip if you're cost-sensitive—the LLM-based evaluation approach can get expensive at scale, and the demo doesn't optimize for token efficiency.