Weco CLI: Tree Search for Code Optimization When Manual Iteration Isn't Enough

Hook

Most developers use LLMs to generate code once. Weco uses them to explore thousands of variations in a search tree, backtracking from dead ends like AlphaGo exploring move sequences—except the game is making your CUDA kernel 30% faster.

Context

If you've ever spent a week manually tweaking a GPU kernel, trying different memory access patterns and comparing benchmark results in a spreadsheet, you've experienced the frustration Weco targets. The loop is always the same: change some code, run an evaluation script, check if the metric improved, repeat. Tools like GitHub Copilot accelerate the 'change some code' step, but they don't close the loop—you still manually decide what to try next based on results.

The problem gets worse when the solution space is large but structured. A CUDA kernel might have dozens of knobs (thread block size, shared memory usage, loop unrolling factors) that interact in non-obvious ways. A prompt template might need adjustments across multiple chain-of-thought steps. Traditional hyperparameter tools like Optuna can optimize numeric parameters, but they can't modify code structure. Manual LLM prompting can generate variations, but without systematic exploration you're doing random search with expensive evaluations. Weco sits in this gap: automated enough to explore more thoroughly than a human, but flexible enough to optimize arbitrary code against arbitrary metrics.

Technical Insight

Weco's core insight is treating code optimization as a tree search problem where each node is a version of your code and edges are LLM-proposed modifications. This isn't just a metaphor—the system literally maintains parent-child relationships between code states, enabling it to backtrack when a modification path stops improving metrics.

The execution model works like this: You point Weco at a source file (or directory of files) and provide an evaluation command that outputs a metric to stdout. Weco runs your eval command on the initial code to establish a baseline, then enters a loop: (1) prompt an LLM with the current code, evaluation results from the search tree, and your optimization objective, (2) parse the proposed code modification, (3) execute your eval command on the new code in isolation, (4) compare the metric to parent node results, (5) use improvement signals to decide which tree branch to expand next.

Here's what a minimal optimization run looks like:

# Optimize a Python function that prints execution time
weco optimize trainer.py \
  --eval "python trainer.py" \
  --metric "loss" \
  --steps 20 \
  --objective "minimize validation loss"

Your trainer.py might print loss: 0.342 to stdout. Weco parses that value, stores it in the search tree node, and uses it to judge whether the LLM's proposed changes (maybe switching from SGD to AdamW, or adjusting data augmentation) represent progress. After 20 iterations, it shows you the best-performing code variant and the full tree of attempted modifications.

The multi-file support is where this gets interesting for real projects. Most LLM code tools treat files independently, but production ML code spans modules—your model architecture, data loading, and training loop are separate files with tight coupling. Weco handles this by treating file collections as atomic state nodes:

weco optimize src/ \
  --eval "pytest tests/test_accuracy.py -v" \
  --metric "accuracy" \
  --steps 15 \
  --files "src/model.py,src/preprocessing.py"

Now the LLM can propose synchronized changes: "Let's add batch normalization to model.py AND adjust the normalization in preprocessing.py to match." Both files get modified together, evaluated together, and succeed or fail as a unit. This enables refactorings that cross module boundaries—exactly the kind of systematic changes that are tedious to do manually but too complex for isolated file editing.

The 'review mode' shows production awareness that most research tools lack. When you're optimizing a CUDA kernel where each evaluation burns 10 minutes of A100 time, you don't want Weco trying obviously broken code:

weco optimize kernel.cu \
  --eval "./benchmark" \
  --metric "throughput" \
  --review \
  --steps 30

With --review, Weco pauses before each evaluation and shows you the proposed diff. You can skip garbage suggestions (like the LLM hallucinating a CUDA intrinsic that doesn't exist) without wasting compute. This human-in-the-loop design means you get automation benefits—the LLM explores the space and proposes variations you wouldn't think of—while maintaining quality control on expensive evaluations.

The observe mode cleverly repositions Weco from optimizer to optimization platform. Sometimes you're already running experiments (training multiple model variants, A/B testing prompts) and just want telemetry:

# In your existing training script
import weco

for epoch in range(100):
    loss = train_one_epoch()
    weco.observe(
        code_path="model.py",
        metric={"loss": loss, "epoch": epoch},
        metadata={"learning_rate": current_lr}
    )

The observe API guarantees exit code 0 even if the backend is unreachable, so instrumentation won't crash your training run. Your metrics flow to Weco's dashboard where you can compare runs, but the optimization loop is entirely manual. This matters because many researchers won't trust autonomous code modification for critical experiments, but they'll gladly instrument manual iterations if logging is trivial.

Gotcha

The 'no backticks in code' limitation is a red flag about parsing robustness. This suggests Weco extracts code from LLM responses using fragile pattern matching rather than structured formats or AST parsing. Hit a Python f-string with backticks in your docstring and the extraction likely breaks, forcing you to sanitize your codebase before optimization—exactly the kind of friction that kills adoption for real projects.

Metric parsing from stdout/stderr is similarly brittle. The system assumes your evaluation command prints clean output like loss: 0.342 that regex can extract. But real evaluation scripts are messy—they print progress bars (tqdm), warnings (deprecated TensorFlow APIs), debug logs. Weco has no documented way to enforce structured output like JSON, so you'll likely need to write wrapper scripts that silence everything except the metric line. This adds friction and makes the 'arbitrary evaluation command' promise less flexible than it sounds.

The credit-based pricing and mandatory authentication reveal SaaS lock-in. Despite being open source Python code, you can't run Weco with local LLMs or offline. All inference goes through their hosted service, and the BYOK (bring your own key) option still routes through their API for credit tracking. If you work in finance, healthcare, or defense where air-gapped deployment is mandatory, Weco is a non-starter. Even if you just want cost control by running Llama locally via Ollama, the architecture doesn't support it.

Verdict

Use if: You have expensive, automated evaluation (GPU benchmarks, ML training runs, A/B tests with real metrics) and the solution space is too large for exhaustive manual exploration. The tree search genuinely finds optimizations humans miss by systematically exploring branches, and multi-file support enables refactorings that span modules. Best for CUDA kernel optimization, prompt template engineering for production LLM apps, feature engineering pipelines where you're chasing percentage points, or any scenario where evaluation is reliable but slow enough that systematic exploration justifies LLM costs. Skip if: Your problem is 'make this work' rather than 'make this optimal'—the overhead of writing evaluation scripts and configuring tree search isn't worth it for prototyping or web development. Also skip for air-gapped environments (finance, defense, healthcare) since the SaaS architecture requires internet access and external API calls. Finally, skip if your evaluation is flaky or non-deterministic—tree search depends on metric reliability, and noisy signals will cause thrashing between branches that aren't actually different.