AI-Scientist-v2: Autonomous Research Through Tree Search Code Generation
Hook
What if your research assistant didn't just write code—it explored thousands of experimental variations through backtracking tree search, debugged itself when tests failed, wrote the paper, cited related work, and peer-reviewed its own findings? You'd get a 30% success rate and a $25 API bill.
Context
Scientific research follows a tedious loop: read literature, propose hypothesis, implement experiments, debug failures, analyze results, write up findings, respond to peer review. Machine learning research amplifies this pain—every architectural tweak requires training runs, hyperparameter sweeps, and ablation studies. The first AI-Scientist (v1) automated this workflow but relied on human-authored templates defining experiment scaffolds. Researchers still needed to encode domain knowledge about what experiments to run and how code should be structured.
AI-Scientist-v2 removes these guardrails entirely. Instead of filling in templates, it explores the space of possible experiments through best-first tree search, where each node represents a complete experimental codebase and edges represent LLM-proposed modifications. The system generates research ideas by querying Semantic Scholar for novelty, implements experiments through parallel tree exploration with automatic debugging, writes papers using O1-preview, gathers citations, and simulates peer review—all without human intervention beyond the initial research area prompt. It's a bet that unrestricted exploration with 70% failure rates will discover ideas templates would never encode.
Technical Insight
The architecture decouples ideation from execution through a three-stage pipeline. First, launch_scientist_ideation.py runs a separate LLM agent that generates structured research proposals. Unlike v1's template selection, this phase queries Semantic Scholar's API to filter out ideas matching existing paper titles, producing JSON proposals with motivation, methods, and expected contributions:
# Generated idea structure (simplified)
{
"title": "Adaptive Learning Rate Schedules via Meta-Gradient Descent",
"motivation": "Current schedulers require manual tuning...",
"methods": "Learn schedule parameters through meta-gradients...",
"novelty_check": {
"query": "adaptive learning rate meta-learning",
"similar_papers": 3,
"passes_threshold": true
}
}
This separation creates a reusable hypothesis bank—you can generate 50 ideas once, then run experiments on the promising subset without re-invoking ideation costs.
The core innovation happens in launch_scientist_bfts.py, which implements best-first tree search over experimental code states. Unlike simple prompt-and-execute systems, each node in the search tree contains a complete codebase snapshot. The system maintains a priority queue of nodes scored by validation metrics (accuracy, loss convergence, etc.). At each iteration, workers select the highest-scoring unexpanded node, prompt an LLM to propose code modifications, execute the modified code in isolated subprocesses, and add successful children back to the queue:
# Conceptual tree search (actual implementation in AIDE codebase)
while not budget_exhausted():
node = priority_queue.pop() # Highest validation score
# LLM proposes 3 variations
proposals = model.generate_code_modifications(
current_code=node.codebase,
experiment_results=node.metrics,
instruction="Improve validation accuracy"
)
for proposal in proposals:
result = execute_in_subprocess(
code=proposal,
timeout=3600,
max_debug_attempts=3 # Configurable backtracking
)
if result.success:
child = Node(
codebase=proposal,
metrics=result.metrics,
score=result.val_accuracy
)
priority_queue.push(child)
The max_debug_depth parameter controls how many times the system attempts to fix runtime errors before abandoning a branch. When code crashes, the error traceback gets fed back to the LLM with a "fix this bug" prompt. The debug_prob parameter adds stochasticity—only debugging 50% of failures prevents infinite repair loops on fundamentally broken ideas.
Parallelization happens at two levels. The num_workers parameter runs independent tree search threads on the same idea, each exploring different modification sequences. The num_drafts parameter launches completely separate search trees for the same research proposal, essentially running ensemble experiments. This embarrassingly parallel design is critical for cost management—you can horizontally scale across cheaper models or spot instances rather than waiting hours for sequential exploration.
After tree search converges (either hitting a score threshold or exhausting the node budget), separate LLM agents handle downstream tasks. The paper writing agent (model_writeup) receives the best-performing experiment code plus its metrics, generating LaTeX through O1-preview's extended context window. The citation agent (model_citation) extracts key claims from each paper section, queries Semantic Scholar for supporting references, and inserts \cite{} commands. Finally, a reviewer agent (model_review) scores the paper on novelty, rigor, and clarity, producing the same rubric used in actual ML conference reviews.
The system generates tree visualizations showing the entire search trajectory—you can literally see which experimental branches were explored and where debugging failed. This transparency is crucial for understanding why success rates hover around 30%: most failures happen when the LLM proposes code that's syntactically valid but semantically nonsensical (training loops that never update parameters, metrics computed on wrong tensors, etc.).
Gotcha
The 70-80% failure rate isn't a bug—it's the fundamental tradeoff of template-free exploration. When v1 provided experiment scaffolds, it constrained the search space to known-good patterns. V2's unrestricted code generation means LLMs frequently propose experiments that are coherent in isolation but fail when integrated. The FAQ acknowledges that even Claude 3.5 Sonnet, the best-performing model, only completes about 20-30% of runs successfully. Budget $100 to get three complete papers, and expect multi-hour runtimes even with parallelization.
Security is entirely punted to users. The repository executes arbitrary LLM-generated Python code in subprocesses without sandboxing enforcement. The README suggests using Docker but provides no containerization configs or capability restrictions. An LLM could theoretically generate code that exfiltrates data, mines cryptocurrency, or consumes unbounded memory. The system won't stop it—you need to implement isolation yourself. Worse, there's no resource monitoring: if generated code spawns 100 CUDA processes and deadlocks your GPUs, the tree search continues queueing nodes until you manually kill it. The CUDA memory error mentioned in FAQs requires editing the ideation prompt to suggest smaller models, revealing there's no automatic resource management.
Semantic Scholar API dependency creates brittle failure modes. Novelty checking, citation retrieval, and related work synthesis all require API access. Rate limiting or service outages will silently degrade results—the system might generate ideas already published or produce papers with zero citations. The FAQ mentions "you may be able to skip" novelty checking, but the codebase doesn't document how to disable this gracefully or what happens when API calls timeout during citation insertion.
Verdict
Use if: You're researching autonomous agent capabilities and need to benchmark end-to-end scientific reasoning, you're exploring meta-learning for scientific discovery and can tolerate 70% failure rates as part of the experimental design, or you need to generate workshop papers as existence proofs where the methodology itself is the contribution. The tree search architecture is genuinely novel for code generation research. Skip if: You need reliable experiment automation for actual research (v1's template-based approach gives 60-80% higher success rates), you can't deploy proper containerized sandboxing for arbitrary code execution, you have budget constraints (each run costs $20-25 with no incremental checkpointing on failures), or you need reproducible results for production ML workflows. This is a research prototype demonstrating what's possible with agentic systems, not infrastructure for doing science at scale.