Building an Adversarial AI Judge: Dual-LLM Architecture for Prompt-Injection-Proof Hackathon Scoring

Hook

At NEBULA:FOG 2026, participants tried to hijack the AI judge with prompt injections embedded in their demo slides. None of them succeeded. Here's the architecture that stopped them.

Context

Traditional hackathon judging is slow, subjective, and doesn't scale. Human judges get fatigued after watching dozens of five-minute demos, applying rubrics inconsistently, and struggling to compare teams they saw hours apart. The obvious solution—letting an LLM watch demos and score them—introduces a catastrophic attack vector: participants can embed prompt injections in their slides, speech, or UI to manipulate their scores.

Arbiter was built for NEBULA:FOG 2026, a security-focused hackathon where adversarial behavior was expected and encouraged. The system needed to process live audio/video streams, generate real-time commentary for audience entertainment, score demos against a multi-dimensional rubric using multiple LLMs, and do all of this without being exploited by participants who knew exactly how it worked. This isn't just a hackathon judging tool—it's a reference architecture for building LLM systems that operate in adversarial environments where users actively try to manipulate model outputs.

Technical Insight

The core innovation in Arbiter is its dual-LLM privilege separation model, which treats untrusted user input like an operating system treats untrusted code. A "capture layer" LLM (powered by Gemini Live API) watches the demo stream and generates observations—essentially converting audio/video into structured text summaries. Critically, this LLM has no scoring authority. Its outputs feed into a four-layer defense pipeline before reaching the "privileged" judging LLM.

The defense pipeline implements regex-based detection for obvious injection patterns ("ignore previous instructions"), semantic classification to identify manipulation attempts in natural language, multi-language detection to catch attempts that switch languages mid-injection, and structural analysis to validate JSON outputs haven't been corrupted. Only sanitized observations reach the judging layer. Here's a simplified version of the sanitization flow:

class DefensePipeline:
    def sanitize(self, raw_observation: str) -> SanitizedObservation:
        # Layer 1: Regex-based pattern matching
        if self.contains_injection_keywords(raw_observation):
            return self.quarantine(raw_observation, reason="keyword_match")
        
        # Layer 2: Semantic classification
        injection_score = self.classifier.predict(raw_observation)
        if injection_score > 0.85:
            return self.quarantine(raw_observation, reason="semantic_anomaly")
        
        # Layer 3: Multi-language detection
        if self.detects_language_switching(raw_observation):
            return self.quarantine(raw_observation, reason="language_switch")
        
        # Layer 4: Structural validation
        parsed = self.extract_structured_data(raw_observation)
        if not self.validate_schema(parsed):
            return self.quarantine(raw_observation, reason="schema_violation")
        
        return SanitizedObservation(content=parsed, trusted=True)

The judging layer uses a Mixture of Experts approach across three LLM providers: Gemini, Claude (via Anthropic), and Groq. Each model independently scores the demo against the rubric, producing numerical scores for categories like innovation, technical execution, and presentation quality. Arbiter then applies outlier detection—if one model's score deviates significantly from the others, it's flagged and weighted down in the final aggregate. This prevents a single model's hallucination or bias from dominating the result.

The system's theatrical presentation layer is surprisingly sophisticated for a hackathon tool. Commentary generation uses a separate prompt chain that synthesizes observations into British-accented critique, delivered via Cartesia TTS with sub-500ms latency. The React-based operator dashboard lets staff trigger score reveals with animated transitions, while the audience display shows streaming commentary synchronized with the live demo. WebSocket connections handle real-time bidirectional communication between all components.

Arbiter's deliberation engine implements a structured memory system that persists observations across demos. After all teams present, it can run cross-team comparisons—asking the judging LLM to directly contrast approaches between specific teams rather than scoring in isolation. This addresses the "recency bias" problem where later demos are remembered more vividly than earlier ones.

The repository includes a comprehensive testing infrastructure with 1,451 tests, including a "rehearsal mode" that generates synthetic demo events with simulated prompt injection attempts. Post-event, the team conducted red-team validation (documented in included slides) where security researchers tried to bypass the defense layers. The documented failure cases—which include sophisticated multi-turn injections that slowly shifted context—informed additional safeguards added post-hackathon.

Gotcha

The elephant in the room is cost. Running three concurrent LLM APIs (Gemini, Claude, Groq) plus Cartesia TTS for every demo burns through API credits fast. For NEBULA:FOG's 25 teams, this was justified; for a college hackathon with 100 teams and no budget, it's a non-starter. There's no graceful "cheap mode" that maintains the security guarantees—the dual-LLM architecture and MoE scoring are the security model.

The hardware capture pipeline is brittle. The system expects specific audio/video devices (configured via device index) and doesn't handle dynamic device changes well. If your venue has flaky HDMI connections or participants want to demo on their own laptops rather than a standardized setup, you'll spend more time debugging capture issues than watching demos. The codebase is also tightly coupled to NEBULA:FOG's specific rubric and track bonus structure—adapting it to your event's scoring criteria requires non-trivial refactoring of prompt templates and validation schemas across multiple modules. This isn't a "configure your rubric in YAML" situation; it's a "modify hardcoded prompts in three different files" situation.

Verdict

Use if: You're running a security-focused or high-stakes hackathon where participants might attempt adversarial attacks, you have budget for multiple premium LLM APIs, you want theatrical AI-driven judging as part of the event experience, or you're researching adversarial-robust LLM architectures and need a real-world reference implementation. The dual-LLM security model and MoE scoring are genuinely innovative and battle-tested. Skip if: You're organizing a standard hackathon with limited budget, you need simple drop-in judging software without extensive customization, you don't have stable hardware capture infrastructure, or your event doesn't expect adversarial behavior. The complexity, API costs, and configuration overhead only make sense when security and spectacle are core requirements—not nice-to-haves.