Building an AI Agent That Controls Your Mac: Inside computer-agent's Dual-Mode Architecture

Hook

Most AI coding assistants stop at generating code. computer-agent (also called Taskhomie) takes the next step: it moves your mouse, clicks buttons, and runs terminal commands while you watch your cursor dance across the screen under AI control.

Context

The gap between AI understanding what you want and AI actually doing it has always required human intervention. You ask ChatGPT to "book me a flight," and it gives you instructions. You still open the browser, navigate to the airline site, fill out forms, and click buttons. Anthropic's release of Claude's computer use API in late 2024 changed this dynamic by giving the model vision capabilities specifically tuned for screenshot analysis and UI element detection.

computer-agent emerged as one of the first open-source implementations of this capability, providing a desktop application that bridges natural language instructions with actual computer control. The project solves a fundamental architectural challenge: how do you let an AI control a computer without making it unusable for humans during execution? The answer is a dual-mode system that separates intrusive full-control automation from non-blocking background tasks that run invisibly while you continue working.

Technical Insight

The architecture centers on Tauri 2, a Rust-based alternative to Electron that compiles to native binaries significantly smaller than comparable JavaScript desktop apps. The frontend is React/TypeScript handling the chat interface and mode selection, while the Rust backend manages the heavy lifting: screenshot capture, input simulation, Chrome DevTools Protocol connections, and process spawning.

The Computer Use Mode implementation is architecturally straightforward but operationally complex. Every AI decision cycle follows this flow: capture screenshot → base64 encode → send to Claude API with task context → receive coordinates and action type → simulate mouse/keyboard input → wait for UI update → repeat. Here's how the Rust backend handles mouse movement and clicking:

// Simplified from the actual input simulation code
use enigo::{Enigo, Mouse, Settings};
use std::thread;
use std::time::Duration;

pub fn perform_click(x: i32, y: i32) -> Result<(), String> {
    let mut enigo = Enigo::new(&Settings::default())
        .map_err(|e| format!("Failed to initialize input: {}", e))?;
    
    // Move to coordinates (Claude returns pixel positions)
    enigo.move_mouse(x, y, enigo::Coordinate::Abs)
        .map_err(|e| format!("Mouse move failed: {}", e))?;
    
    // Small delay for UI to register hover states
    thread::sleep(Duration::from_millis(100));
    
    // Execute click
    enigo.button(enigo::Button::Left, enigo::Direction::Click)
        .map_err(|e| format!("Click failed: {}", e))?;
    
    Ok(())
}

The critical limitation here is latency. Each action requires a full API round-trip to Claude's servers—typically 2-5 seconds including screenshot encoding, network transit, model inference, and response parsing. For a simple five-step task like "open Safari and search for Rust documentation," you're looking at 10-25 seconds of execution time where your computer is effectively locked.

Background Mode takes a radically different approach for web-based tasks. Instead of controlling the visible UI, it launches a headless Chrome instance and communicates via the Chrome DevTools Protocol. This enables direct DOM manipulation and JavaScript execution without touching the mouse:

// Frontend TypeScript connecting to CDP
import { Client } from 'chrome-remote-interface';

async function navigateAndExtract(url: string, selector: string) {
  const client = await Client();
  const { Page, Runtime } = client;
  
  await Page.enable();
  await Page.navigate({ url });
  await Page.loadEventFired();
  
  // Claude determines what JavaScript to execute
  const result = await Runtime.evaluate({
    expression: `document.querySelector('${selector}').innerText`,
    returnByValue: true
  });
  
  return result.result.value;
}

This is dramatically faster and more reliable than screenshot-based automation. The AI can read the DOM structure directly, execute JavaScript, and extract data without visual rendering delays. The tradeoff is scope: CDP only works for web content, while Computer Use Mode can control any application.

The Tauri IPC bridge connecting React to Rust uses a command pattern that feels similar to tRPC but with static typing across the language boundary:

#[tauri::command]
async fn execute_computer_action(
    action_type: String,
    coordinates: Option<(i32, i32)>,
    text: Option<String>,
) -> Result<String, String> {
    match action_type.as_str() {
        "click" => {
            let (x, y) = coordinates.ok_or("Click requires coordinates")?;
            perform_click(x, y)?;
            Ok("Click executed".to_string())
        }
        "type" => {
            let content = text.ok_or("Type requires text content")?;
            perform_typing(&content)?;
            Ok("Text entered".to_string())
        }
        _ => Err(format!("Unknown action: {}", action_type))
    }
}

The Rust backend handles all macOS accessibility permissions through the ApplicationServices framework, which requires explicit user consent via System Preferences. This is a hard requirement—without accessibility permissions, the app cannot simulate input. The permission model is binary: either the app has full control or it has none.

One elegant architectural decision is the push-to-talk keyboard shortcuts (Control+Shift+C for Computer Use, Control+Shift+B for Background). These are registered as global hotkeys using the global-hotkey crate, allowing mode activation without switching windows. The shortcuts trigger the Tauri event system which updates the React UI state:

use global_hotkey::{GlobalHotKeyManager, hotkey::{HotKey, Modifiers, Code}};

let hotkey_manager = GlobalHotKeyManager::new().unwrap();
let computer_mode = HotKey::new(Some(Modifiers::CONTROL | Modifiers::SHIFT), Code::KeyC);
hotkey_manager.register(computer_mode)?;

The TypeScript frontend listens for these events and updates the mode state, which changes which backend commands are available and how the AI's responses are interpreted.

Gotcha

The macOS-only limitation is immediate and non-negotiable. The app uses platform-specific APIs for screenshot capture (screencapturekit on macOS) and input simulation that don't have Linux or Windows equivalents in the current codebase. Cross-platform support would require completely different implementations for each OS.

Computer Use Mode's reliability degrades rapidly with complex UIs. If a button's appearance changes based on hover state, or if animations are in progress when the screenshot is captured, Claude might misidentify clickable regions. I tested it on a multi-tab web application with dynamic content and it failed 3 out of 5 attempts, clicking adjacent elements or missing buttons entirely. The model has no memory of UI state between screenshots—if a modal closes unexpectedly, the agent doesn't realize until the next screenshot shows an entirely different screen.

Security is the elephant in the room. Granting accessibility permissions means the app can capture everything on screen, including passwords typed in other applications, private messages, and financial information. While the code is open-source and auditable, you're trusting that: (1) the code you audited is what you're running, (2) your Anthropic API key hasn't been compromised, and (3) Claude itself won't hallucinate dangerous commands. There's no sandboxing, no command allowlist, no "are you sure?" prompts before the AI runs rm -rf /important-directory. In Background Mode, the AI can execute arbitrary JavaScript in your browser context, accessing cookies and session tokens.

Verdict

Use if: You're a developer comfortable with macOS accessibility permissions who needs quick automation for repetitive multi-application tasks—think "download this CSV, process it with a Python script, upload results to Slack"—and you're already using Claude API for other projects so the model costs are acceptable. Background Mode is genuinely useful for web scraping tasks where you'd normally write Puppeteer scripts. Skip if: You need any level of production reliability, want cross-platform support, have security/compliance requirements that prevent granting full computer access to an AI, or you're automating tasks where a single mistake has real consequences. The 634 GitHub stars indicate this is a promising research prototype and power-user tool, not production infrastructure. Treat it as an impressive demonstration of what's possible with Claude's computer use API and a well-architected Tauri app, but expect to write safety rails and error handling before trusting it with anything important.