> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

Building a Universal Agent Control Panel by Scraping tmux Panes

[ View on GitHub ]

Building a Universal Agent Control Panel by Scraping tmux Panes

Hook

What if you could control Claude Code, Codex, and Gemini CLI from your phone by reading their terminal output instead of waiting for each vendor to build an API? That's exactly what happens when you treat the terminal buffer as ground truth.

Context

AI coding agents have proliferated faster than their control interfaces. You're running Claude Code in one tmux pane, Codex in another, maybe a Gemini CLI session in a third—each with its own output format, its own API (if any), its own way of signaling that it's blocked waiting for your input. The moment you step away from your desk, these agents sit idle, waiting for confirmation on file overwrites or answers to ambiguous requirements. SSH from your phone works, but typing complex responses on a touchscreen keyboard is misery, and you can't even tell which pane needs attention without manually checking each one.

The traditional solution would be to integrate with each vendor's API, building custom clients for Anthropic's Claude API, OpenAI's Codex interface, and Google's Gemini endpoints. But that approach fragments immediately: Bedrock-hosted Claude works differently than direct API access, self-hosted models have no API at all, and any new agent means another integration project. tmux-rc takes a radically different approach—it watches the terminal itself, treating whatever gets printed to stdout as the canonical state. If a human can read the terminal and understand that an agent is waiting for input, an LLM can too.

Technical Insight

React PWA

FastAPI Server

Python Daemon

Tmux Server

tmux capture-pane -p

Text buffer

Changed text

idle/running/waiting

Pane states

Real-time updates

Send keystrokes

tmux send-keys

Session JSON

Tmux Panes

Polling Loop

Every few seconds

capture-pane

Buffer Diff

Regex Heuristics

+ Gemini LLM

In-Memory

State Dict

HTTP Endpoints

Server-Sent Events

Phone UI

System architecture — auto-generated

The architecture centers on a polling daemon that runs tmux capture-pane every few seconds for each active pane, comparing the output against the previous snapshot to detect changes. Here's the core state capture logic:

def capture_pane_state(pane_id: str) -> PaneState:
    # Get visible terminal buffer
    result = subprocess.run(
        ['tmux', 'capture-pane', '-p', '-t', pane_id],
        capture_output=True,
        text=True
    )
    current_text = result.stdout
    
    # Fast heuristics for obvious states
    if re.match(r'^[\w@]+.*[$#]\s*$', current_text.split('\n')[-1]):
        return PaneState(status='idle', prompt=None)
    
    if '...' in current_text[-50:] or current_text.count('\n') > prev_count:
        return PaneState(status='streaming', prompt=None)
    
    # Uncertain? Ask the LLM
    if needs_classification(current_text, prev_text):
        classification = classify_with_gemini(current_text)
        return PaneState(
            status=classification['status'],
            prompt=classification.get('question'),
            pane_id=pane_id
        )

The two-stage classification is critical for cost and latency. Simple cases—an idle bash prompt, obviously streaming output where line count keeps increasing—get resolved with regex in microseconds. Only ambiguous states trigger a Gemini Flash Lite API call, and even then, the prompt is carefully constrained: "Is this terminal waiting for user input? If yes, extract the question being asked." This keeps tokens low and responses fast, typically under 200ms even with API roundtrip.

The mobile UI receives updates via Server-Sent Events rather than WebSocket, which matters for battery life and connection stability. SSE automatically reconnects on dropped connections without client-side code, and the unidirectional flow (server pushes state, client sends commands via POST) maps cleanly to the polling architecture:

const eventSource = new EventSource('/api/stream');
eventSource.addEventListener('pane_update', (event) => {
    const pane = JSON.parse(event.data);
    if (pane.status === 'waiting' && pane.prompt) {
        showNotification(pane.pane_id, pane.prompt);
        updatePaneUI(pane);
    }
});

Sending input back to the pane uses tmux send-keys with careful timing. The 'enter settle delay' isn't premature optimization—it's essential because agent TUIs distinguish 'submit message' from 'add newline to message' based on how quickly Enter follows the last character. Send text and immediately send Return, and you'll often compose a message without submitting it:

def send_to_pane(pane_id: str, text: str, submit: bool = True):
    subprocess.run(['tmux', 'send-keys', '-t', pane_id, '-l', text])
    
    if submit:
        time.sleep(0.3)  # Let the agent TUI register completion
        subprocess.run(['tmux', 'send-keys', '-t', pane_id, 'Enter'])

Session restoration gets complex because tmux layouts and agent conversations are orthogonal concepts. tmux-rc serializes both separately—the tmux topology (split percentages, window indices, pane sizes) goes into one JSON structure using tmux list-panes -F format strings, while agent state (conversation IDs, model configs) gets extracted by matching pane working directories against known agent patterns. Reconstruction happens in phases: first restore the tmux layout with split-window commands and precise sizing, then inject agent resume commands with sufficient delays for each TUI to boot:

def restore_session(session_data: dict):
    # Phase 1: Rebuild tmux topology
    subprocess.run(['tmux', 'new-session', '-d', '-s', session_data['name']])
    
    for window in session_data['windows']:
        for pane in window['panes'][1:]:  # First pane exists from new-session
            subprocess.run(['tmux', 'split-window', '-t', window['id'],
                          '-p', str(pane['height_percent'])])
    
    # Phase 2: Restart agents (with timing)
    for pane in session_data['agent_panes']:
        cmd = f"{pane['agent_cmd']} --resume {pane['conversation_id']}"
        subprocess.run(['tmux', 'send-keys', '-t', pane['tmux_id'], cmd, 'Enter'])
        time.sleep(2)  # Agent startup time

The decision to use service-account keys instead of Application Default Credentials is operationally motivated. ADC tokens expire and require interactive reauth via browser, which breaks the 'start the daemon and forget it' model. A service account key sitting in a JSON file on disk is less secure in principle but more reliable in practice for a daemon that needs to run unattended for days during multi-agent orchestration sessions.

Gotcha

The elephant in the room is complete absence of authentication. The FastAPI server binds to localhost by default, but that only protects you on single-user machines. The moment you expose it for remote access—which is the entire point of mobile control—you need to layer something like Tailscale or Cloudflare Tunnel in front, because tmux-rc itself does nothing to verify who's sending commands. Any process that can reach the HTTP port can inject arbitrary keystrokes into any pane, which means full terminal control.

Polling-based state detection is inherently laggy and lossy. If an agent prompts for input and you respond within the same polling interval (default few seconds), tmux-rc might never record that the prompt happened. Rapid back-and-forth interactions won't be captured accurately, and transient error messages that appear and disappear between polls are invisible. The tool is designed for human-timescale oversight—catching agents that have been blocked for minutes—not sub-second monitoring. Terminal output parsing is also brittle by nature. The regex patterns are tuned for current Claude Code and Codex prompt formats. When Anthropic updates their TUI styling or changes how they render questions, detection breaks until you update the patterns. There's no structured protocol here, no agent SDK—just string matching against whatever got dumped to stdout. Finally, the Gemini classification dependency means you cannot use this tool without Google Cloud credentials and you'll incur Vertex AI charges (tiny per-call, but continuous). There's no offline fallback that maintains full functionality.

Verdict

Use if: You're a tmux power user running multiple AI coding agents (Claude Code, Codex, Gemini CLI) across long-lived sessions, you need mobile oversight without SSH-ing from your phone, and you want universal agent control without integrating each vendor's API. Perfect for solo developers or small teams who can layer their own auth (Tailscale, WireGuard) and accept eventual-consistency state updates in exchange for tool-agnostic terminal scraping. Skip if: You need audit logs or multi-user access control (there's zero auth), you require sub-second latency (polling guarantees stale state), you're running agents in containers with structured logging (just parse those logs instead), or you're uncomfortable with service-account keys on disk. Also skip if you're not already a tmux user—this is a power tool for existing tmux workflows, not a gateway to terminal multiplexing.