Why Your RLVR System Fails: What Noisy Labels Do to Reinforcement Learning

Hook

Reinforcement learning from verifiable rewards sounds like the perfect AI training paradigm—until you discover that a single mislabeled 'correct' answer can poison your entire policy gradient. Unlike supervised learning where noisy labels merely confuse the model, RLVR creates feedback loops where the model learns to confidently produce wrong answers.

Context

Reinforcement Learning with Verifiable Rewards (RLVR) promised to solve AI's reliance on expensive human feedback. Instead of asking humans "is this good?", you execute code, run SQL queries, or check mathematical equivalence—deterministic verification that should eliminate subjective judgment. The pitch was compelling: train a model to generate SQL queries, execute them against a database, and reward correct results. No reward model uncertainty, no human preference disagreements, just ground truth.

But UIUC's Kang Lab discovered a catastrophic failure mode. When your training dataset contains annotation errors—where incorrect solutions are labeled as correct—RLVR doesn't just underperform. It actively learns to be wrong. The researchers audited the BIRD text-to-SQL benchmark and found systematic labeling errors even in curated datasets. Their corrected subset (BIRD-Corrected) and noise-mitigation experiments reveal why production RLVR systems mysteriously degrade: mislabeled positives create hallucinated feedback loops where models optimize toward incorrect behavior while believing they're being rewarded for correctness.

Technical Insight

The repository implements RLVR pipelines across two domains—math problem-solving and text-to-SQL—to isolate how annotation noise propagates through policy gradients. The architecture choice is revealing: rather than building a unified framework, they extend domain-specific tools (SkyRL for math, Tinker SDK for SQL) because verifiable rewards require deep integration with execution environments.

The math pipeline wraps symbolic equivalence checking as the reward signal. After generating a solution, the system parses the mathematical expression and compares it against the ground truth using symbolic manipulation (likely SymPy-based). The critical insight is in the data preprocessing—noise injection happens pre-training by flipping labels in parquet files:

# Simplified noise injection pattern from the repository structure
def inject_label_noise(dataset, noise_rate=0.2):
    """
    Flip correct/incorrect labels with uniform probability.
    This simulates real-world annotation errors.
    """
    noisy_dataset = dataset.copy()
    num_flips = int(len(dataset) * noise_rate)
    flip_indices = random.sample(range(len(dataset)), num_flips)
    
    for idx in flip_indices:
        # Flip the label: correct -> incorrect or vice versa
        noisy_dataset[idx]['is_correct'] = not dataset[idx]['is_correct']
    
    return noisy_dataset

This pre-baking approach means each noise level (10%, 20%, 30%) requires separate dataset artifacts. While less flexible than dynamic noise injection, it ensures reproducibility—critical for academic experiments but impractical for production systems where noise rates vary across data segments.

The SQL pipeline uses BIRD database execution as verification. When the model generates a query, the system executes it against the actual database and compares results with ground truth. But here's where things get interesting: the researchers manually audited 600 examples where models failed, separating "legitimately incorrect" from "falsely labeled incorrect." This revealed that annotation errors cluster around complex queries with joins, subqueries, and edge cases—exactly where RLVR training signal matters most.

PGFC (Pessimistic Group Feedback Correction) implements noise correction by scaling rewards based on estimated noise rate:

# Core PGFC reward scaling logic
def pgfc_reward_scaling(reward, estimated_noise_rate):
    """
    Scale rewards to correct for label noise bias.
    Theorem: This recovers unbiased gradients under uniform noise.
    """
    if reward > 0:
        # Down-weight positive rewards more aggressively
        corrected = reward * (1 - 2 * estimated_noise_rate)
    else:
        # Negative rewards assumed less noisy
        corrected = reward
    
    return corrected

# In practice during GRPO update:
for trajectory in batch:
    raw_reward = verify_solution(trajectory.output)
    corrected_reward = pgfc_reward_scaling(raw_reward, noise_rate=0.2)
    policy_gradient = compute_gradient(trajectory, corrected_reward)

The (1 - 2 * noise_rate) scaling factor comes from theoretical analysis: under uniform label flipping, positive rewards are overestimated by exactly this amount. At 20% noise, a reward of 1.0 gets scaled to 0.6. This is elegant but brittle—it assumes you know the true noise rate, which requires the same manual auditing that RLVR was supposed to eliminate.

The repository also implements four comparison algorithms (DAPO, SAPO, TIS, Dr. GRPO), but provides minimal analysis of their relative effectiveness. The code structure suggests these are exploratory implementations rather than production-ready alternatives. The format-only reward baseline is more interesting—it isolates syntactic validity from semantic correctness, revealing that models can achieve 95%+ format compliance while producing completely wrong SQL semantics. This matters because many production systems use parsing success as a proxy for correctness, which this research proves is dangerously misleading.

The verifiable reward architecture sidesteps reward model uncertainty entirely, but introduces different brittleness. SQL has semantic equivalence issues: queries with different JOIN orders or WHERE clause arrangements can produce identical results. The symbolic math checker handles algebraic equivalence but struggles with notational variations (fractions vs decimals, implicit vs explicit multiplication). The repository doesn't address these edge cases, which means real-world deployment would require extensive verifier engineering beyond the core RLVR algorithm.

Gotcha

PGFC's fatal flaw is circular dependency: you need to know the noise rate to correct for noise, but estimating noise rate accurately requires manual auditing of your dataset. The repository provides no methodology for noise rate estimation beyond "have humans check everything," which defeats RLVR's entire value proposition. If you're manually auditing annotations anyway, why not just train on clean data from the start?

The 600-sample BIRD-Corrected dataset is too small for production use. Text-to-SQL systems need to generalize across thousands of database schemas, query patterns, and domain-specific conventions. The corrected subset represents a single benchmark under controlled conditions—expect significant distribution shift in real applications. The uniform noise injection methodology also misses how annotation errors actually occur: systematic patterns where hard examples get mislabeled and easy ones stay correct. Real-world noise is adversarial (concentrated on ambiguous cases) rather than random, which means PGFC's theoretical guarantees won't hold. Finally, the research assumes verifiers are perfect oracles, but SQL execution has NULL handling ambiguities, floating-point precision issues, and timeout edge cases that aren't addressed in the codebase.

Verdict

Use if: You're building RLVR systems for math or code generation and experiencing mysterious performance degradation despite clean verifiers—this research explains the failure mode and provides diagnostic tools. Also valuable if you're deciding between RLVR and supervised learning for a new project with known annotation quality issues; the comparative experiments quantify exactly how much worse noise affects policy optimization. The BIRD-Corrected dataset is useful for benchmarking SQL generation models against a cleaner baseline. Skip if: You need production-ready noise mitigation (PGFC requires knowing noise rates you can't estimate), you're doing standard supervised learning (noise matters less), your domain lacks verifiable rewards (can't apply RLVR anyway), or you want plug-and-play code (this is an experimental research artifact requiring deep framework modifications). Consider alternatives like DPO for preference learning robust to label noise, or inference-time verification with best-of-N sampling if you need correctness guarantees without poisoning training data.