OSWorld-V2: The GUI Agent Benchmark That Hides Its Answers

Hook

Most benchmarks leak their answers into foundation model training data the moment they're published on GitHub. OSWorld-V2 solves this by hiding task logic in a gated HuggingFace dataset—agents can't Google their way to high scores.

Context

The explosion of vision-language models like GPT-4V and Claude created a new problem: how do you benchmark computer-use agents when the models have likely seen your test cases during pre-training? Existing GUI benchmarks like WebArena publish everything openly—task definitions, evaluation code, ground-truth trajectories. This transparency is great for reproducibility but terrible for benchmark integrity when models can pattern-match against solutions scraped from GitHub.

OSWorld-V2 takes a different approach. Built by the XLang research group, it evaluates multimodal agents on long-horizon desktop tasks—managing files, operating terminals, navigating websites, manipulating Git repositories—inside real Ubuntu virtual machines. The critical innovation isn't the VM harness or the task variety; it's the gated distribution model. Task classes and evaluator logic live in a restricted HuggingFace dataset, separate from the open-source runner code. You can build agents against the framework without ever seeing how tasks are scored or what the ground-truth solutions look like. This architectural split between public infrastructure and private task definitions represents a pragmatic middle ground between fully open benchmarks (vulnerable to contamination) and fully closed corporate evals (impossible to reproduce).

Technical Insight

OSWorld-V2's architecture splits into three layers that work together to create reproducible, isolated evaluation environments. At the foundation sits a provider abstraction supporting Docker with KVM acceleration and AWS EC2 instances. The provider layer handles the gnarly details of spawning headless or VNC-enabled Ubuntu VMs, configuring network ports for task services (3000 for mocked websites, 8000 for GitLab), and cleaning up resources after evaluation runs.

The task execution layer is where the gated distribution model becomes concrete. When you launch an evaluation, the system downloads Python task classes from a restricted HuggingFace dataset—you need explicit access permissions, preventing casual scraping. Each task class defines initial filesystem state, required web services, and evaluation criteria. The benchmark injects these assets into the VM, spins up mocked websites and self-hosted GitLab instances, then instruments the environment with a custom DesktopEnv API.

Here's what the agent interaction loop looks like:

from desktop_env import DesktopEnv

# Initialize environment with task configuration
env = DesktopEnv(
    provider="docker",  # or "aws" for EC2
    task_config=task_config,
    observation_type="screenshot+a11y_tree"  # dual modality
)

# Agent receives natural language task description
task_desc = "Clone the repository at https://gitlab.local/project, "\
            "create a new branch 'feature-x', and open the README in gedit"

for step in range(max_steps):
    # Get multimodal observation
    obs = env.observe()
    # obs["screenshot"] is a PIL Image
    # obs["accessibility_tree"] is parsed UI hierarchy
    
    # Agent (VLM) decides next action
    action = agent.predict(
        task=task_desc,
        screenshot=obs["screenshot"],
        a11y_tree=obs["accessibility_tree"]
    )
    
    # Execute grounded action: click, type, key press, shell command
    env.step(action)  # e.g., {"type": "click", "x": 450, "y": 120}
    
    # Task-specific evaluator checks progress
    result = env.evaluate()
    if result["done"]:
        break

The dual observation modality—screenshot pixels plus accessibility tree—is more sophisticated than it appears. Most GUI automation tools force you to choose between pixel-based vision (like humans see) or structured DOM parsing (like traditional automation). OSWorld-V2 exposes both simultaneously, letting agents leverage OCR for visual grounding while using the accessibility tree for precise element targeting. A smart agent might use vision to understand layout context, then switch to the accessibility tree to click the exact button without fragile pixel coordinate heuristics.

The reproducibility mechanism deserves special attention. OSWorld-V2 uses release manifests that pin together:

• Task class versions (git tags in the gated dataset)
• Asset snapshots (file trees, database dumps)
• Website deployment versions (mocked service containers)
• VM base images (Ubuntu configuration with pre-installed software)
• Evaluation harness code (the public GitHub repo tag)

When you report results on the "0624" release, anyone with dataset access can reproduce your exact evaluation environment—same task logic, same evaluator, same initial state. This solves the nightmare scenario where researchers unknowingly compare agents across floating dependencies, getting different scores from the same codebase months apart.

The parallel evaluation infrastructure shows this isn't a toy research artifact. The AWS provider automatically provisions subnets and security groups for multi-VM workloads. The bash runner scripts accept task assignment slices, letting you distribute 100 tasks across 10 EC2 instances with simple array indexing. This first-class parallelization support means you can evaluate frontier models at scale without manually orchestrating infrastructure—critical when each task takes 5-15 minutes of agent decision-making.

Task evaluators run inside the gated dataset as black-box Python functions. They have full VM access: parsing screenshots with OCR, checking filesystem state with subprocess calls, scraping web page DOM through Selenium, querying task-specific SQLite databases. An evaluator might verify that a Git branch exists with the correct commit message, that a spreadsheet contains formula results in specific cells, or that a configuration file was edited with precise YAML structure. This rich evaluation goes far beyond simple text matching—it's checking actual computational outcomes in real software.

Gotcha

The gated distribution model creates immediate friction for anyone trying to debug task failures or develop new agents. Evaluator logic is completely opaque—you can't inspect the Python functions that score your agent without HuggingFace dataset permissions. When your agent gets a score of 0.3 on a task, you have no local way to understand which subtasks failed or what the evaluation criteria actually check. The public documentation provides no evaluator schemas, metric definitions, or example scoring logic. You're essentially flying blind unless you have insider access or can manually reverse-engineer requirements by examining filesystem changes across runs.

The infrastructure burden is significant. While the provider abstraction theoretically supports multiple clouds, the 0624 release only ships VM images for Docker and AWS EC2. If you want to run evaluations on GCP, Azure, or Aliyun, you're manually updating OSWorld 1.0 base images—undocumented work that could take days of VM debugging. Worse, the mocked web services and GitLab setup is fragmented. Team-hosted websites work automatically (they use the web.hku.icu domain), but GitLab requires you to self-host an instance and manage authentication tokens manually. This asymmetric deployment means you can't just spin up the benchmark—you need to operate persistent services with their own configuration complexity.

The benchmark enforces uv for Python dependency management but provides no container image for the evaluation harness itself. Reproducing the exact environment across different research groups requires trusting uv's lockfile resolution rather than pulling a frozen Docker image with known hashes. Small version skews in dependencies could silently change agent behavior, and you won't know until you compare results with another lab.

Proxy configuration is relegated to optional documentation, yet it directly impacts task success for geo-restricted websites or rate-limited APIs. This should be a first-class provider setting, not something users discover after debugging mysterious network timeouts.

Verdict

Use if: You're publishing academic research on multimodal agents and need reproducible benchmarks with protected task definitions, you have infrastructure budget for multi-VM AWS workloads or KVM-enabled Linux servers, you're training foundation models that need evaluation environments resistant to data contamination, or you're building agents that must handle cross-application workflows (terminal + browser + file manager) beyond single-domain demos. Skip if: You're prototyping GUI automation and need fast iteration with transparent evaluation logic, you lack the infrastructure to run persistent GitLab services and parallel VM workloads, you need pre-built support for non-AWS cloud providers, or your use case only requires web browser automation—WebArena is simpler and more accessible for pure web tasks. OSWorld-V2 is the right benchmark when solution leakage is a bigger threat than evaluation opacity.