Self-Refine: When Your LLM Becomes Its Own Critic, Editor, and Judge
Hook
What if the same neural network that made a mistake could also identify why it's wrong and fix it—without any fine-tuning, external verifiers, or reward models?
Context
Large language models hallucinate, make arithmetic errors, and commit logical fallacies. The traditional solution involves training separate critic models, building verifiers with labeled data, or implementing reward models for reinforcement learning. But what if you don't have training compute, can't fine-tune the base model, or need a solution that works out-of-the-box?
This is where test-time compute scaling enters the picture. Instead of making models smarter through training, you spend more inference budget on better reasoning. OpenAI's o1 model reportedly uses a similar principle, and the self-refine paradigm asks: can a single LLM wear three hats—generator, critic, and corrector—differentiated only by prompt engineering? The AhsanBilal7/self_refine repository implements exactly this: a multi-stage refinement framework where one model critiques and corrects its own reasoning traces through iterative loops, aggregating final answers through plurality voting.
Technical Insight
The architecture operates on two dimensions: breadth (N rollouts) and depth (D refinement stages). Unlike beam search, which prunes poor candidates incrementally, this framework maintains N completely independent solution paths through all D depths. Each depth executes three sequential generation passes per rollout: reasoning continuation, self-critique, and self-correction.
Here's the core loop structure from the codebase:
for depth in range(1, max_depth + 1):
# Stage 1: Generate reasoning continuation
continuations = model.generate(
prompts=[build_continuation_prompt(trace) for trace in traces],
temperature=0.0 # Greedy for refinement
)
# Stage 2: Critique the continuation
critiques = model.generate(
prompts=[build_critique_prompt(trace, cont)
for trace, cont in zip(traces, continuations)],
max_tokens=384
)
# Stage 3: Correct based on critique
corrections = model.generate(
prompts=[build_correction_prompt(trace, cont, crit)
for trace, cont, crit in zip(traces, continuations, critiques)],
max_tokens=1024
)
# Checkpoint: save (g, c, r) triplets
checkpoint_state(problem_id, depth, continuations, critiques, corrections)
The magic lives in the prompts. The critic sees: "Review the reasoning trace above. Identify specific errors in calculations, logic, or mathematical principles." The corrector gets both the original trace and the critique: "Given the identified issues, provide a corrected reasoning trace." Same model, different instructions—no training required.
What makes this architecturally interesting is the persistent state store. After every (rollout, depth) step, the system checkpoints all three generation outputs to cimr_full_results_*.json. If a generation crashes at depth 3 of 5, you don't restart from scratch—you resume from the last checkpoint. For researchers, this creates a gold mine: you can analyze exactly how reasoning traces evolve, measure answer stability across depths, and track recovery rates when the corrector fixes mistakes.
The answer extraction pipeline reveals practical compromises:
def extract_answer(text):
# Try boxed LaTeX first
if match := re.search(r'\\boxed{([^}]+)}', text):
return normalize(match.group(1))
# Fall back to last number in text
numbers = re.findall(r'-?\d+\.?\d*', text)
if numbers:
return numbers[-1]
# If corrector produced nothing, use previous trace
return extract_answer(previous_trace)
This fallback chain exposes a critical brittleness: sometimes the corrector generates nothing useful, and the system silently falls back to the uncorrected continuation. When this happens across multiple rollouts, it pollutes the plurality vote with unrefineed answers.
The voting mechanism aggregates N final answers per depth, strips whitespace, unwraps one layer of braces for normalization, then selects the most common answer. Ties get broken by PRM (Process Reward Model) scores—but here's the twist: the PRM only affects tie-breaking, not the refinement loop itself. This keeps the framework truly verifier-free while still providing observability.
Token budget management happens at three layers: prompt truncation at 2048 tokens, character-based head+tail preservation at 4000 characters for traces, and per-role generation limits (2048 for continuation, 384 for critique, 1024 for correction). This creates friction: deep reasoning chains exceed context windows, forcing the system to truncate earlier logic that might be relevant for correction.
One subtle indexing hack: depth 1 treats the initial rollout as both r⁰ and g¹, meaning the first critique sees raw samples while later critiques see corrected traces. This asymmetry could affect critique quality—early critiques judge unrefined thinking while later critiques judge already-corrected reasoning.
Gotcha
The sequential depth execution creates a brutal synchronization barrier. All N rollouts must complete depth d before any can start depth d+1. If one rollout generates slowly, the other N-1 sit idle. This makes the framework unsuitable for production inference where throughput matters. A single problem with N=8 rollouts and D=4 depths requires 96 generation calls (8 × 4 × 3), and they can't be parallelized across depths.
Memory scaling gets ugly fast. Storing full traces at every depth means those 96 generation calls produce 96 stored text blobs. With 2048-token generations, a single problem can accumulate 200K tokens before you even load the model. The checkpointing system is great for research but devastating for inference servers.
The framework has no learned stopping mechanism. D is fixed per run, so some problems over-refine (wasting compute on already-converged solutions) while others under-refine (stopping before correction takes effect). Finding optimal D requires expensive grid search across problems. The role prompts are hardcoded for mathematical reasoning—mentions of 'calculations' and 'mathematical errors' make this non-transferable to code generation or strategic planning without significant prompt reengineering. And plurality voting with simple normalization is fragile: mathematically equivalent expressions like '2^3' versus '8' will vote separately unless caught by answer normalization, potentially causing correct answers to lose to consistently wrong ones.
Verdict
Use if: You're researching test-time compute scaling and need infrastructure to analyze how reasoning traces evolve through critique-correction loops. The checkpointing system and depth-wise metrics extraction make this superior to ad-hoc self-refinement scripts for studying refinement dynamics on math benchmarks. It's a solid baseline for papers exploring breadth+depth tradeoffs without training custom verifiers. Skip if: You need production-ready inference (the 3D synchronization overhead kills throughput), want to apply refinement to non-math domains (the prompts are math-specific), or lack compute budget for grid search over N and D hyperparameters. For production, choose adaptive methods like Tree-of-Thought with learned pruning or streaming self-correction systems that don't require depth synchronization barriers.