> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

curl/curl: The Multi-Protocol Transfer Engine Running Billions of Devices

[ View on GitHub ]

curl/curl: The Multi-Protocol Transfer Engine Running Billions of Devices

Hook

Every time you run 'apt-get', your iPhone syncs photos, or your car checks for updates, there's a good chance curl is handling the transfer. This 26-year-old C library powers everything from Alpine Linux containers to NASA's Ingenuity helicopter—yet most developers have never looked under its hood.

Context

Before curl, each protocol required its own client library with incompatible APIs. Need to download via HTTP? Use one library. Add FTP support? Vendor another. Want HTTPS? Now you're managing SSL for each protocol separately. In 1998, Daniel Stenberg started httpget (later curl) to solve his own problem: a currency exchange rate fetcher that needed reliable HTTP in C without massive dependencies.

What differentiated curl from contemporaries like libwww wasn't feature richness—it was architectural restraint. No threads, no global state, minimal allocations. This made it embeddable anywhere: firmware with 64KB RAM, kernel modules, language runtimes needing C bindings. The protocol abstraction came later, growing organically as users needed FTP, then SMTP, then LDAP. Each addition reinforced the core design: protocols are plugins to a transfer state machine. Today, curl is preinstalled on 10+ billion devices, yet its core architecture remains unchanged since the early 2000s because the original constraints—memory safety through discipline, zero-copy I/O, synchronous and async modes from the same API—turned out to be timeless requirements.

Technical Insight

Protocol Stack

Core Engine

Application Control

Data flow

Socket events

CLI Tool

curl binary

Easy Handle

CURL*

Multi Handle

CURLM*

Transfer Engine

State Machine

Protocol Handlers

HTTP/FTP/etc

SSL/TLS Layer

vtable abstraction

Socket Multiplexer

select/poll/epoll

Connection Cache

Persistent connections

App Callbacks

WRITEFUNCTION

System architecture — auto-generated

curl's architecture centers on two handle types that expose fundamentally different programming models. The easy interface wraps a single synchronous transfer—you configure options, call curl_easy_perform(), and block until completion. The multi interface enables asynchronous I/O by letting applications drive transfers through socket multiplexing:

CURLM *multi_handle = curl_multi_init();
CURL *easy_handle = curl_easy_init();

curl_easy_setopt(easy_handle, CURLOPT_URL, "https://api.example.com/data");
curl_multi_add_handle(multi_handle, easy_handle);

int still_running = 1;
while(still_running) {
  CURLMcode mc = curl_multi_perform(multi_handle, &still_running);
  
  if(still_running) {
    // Get file descriptors to wait on
    fd_set fdread, fdwrite, fdexcep;
    int maxfd = -1;
    
    FD_ZERO(&fdread); FD_ZERO(&fdwrite); FD_ZERO(&fdexcep);
    curl_multi_fdset(multi_handle, &fdread, &fdwrite, &fdexcep, &maxfd);
    
    // Application controls the event loop
    select(maxfd + 1, &fdread, &fdwrite, &fdexcep, &timeout);
  }
}

This API design is why curl dominates language runtimes. Python's pycurl, PHP's curl_multi_exec(), and Node.js bindings all leverage multi handles to integrate with their event loops. The application retains control of select()/poll()/epoll(), treating curl as a state machine that advances when sockets are ready. Compare this to threaded HTTP libraries that force you into their concurrency model—curl's multi interface composes with any async architecture.

Under the hood, protocol handlers implement a standardized vtable (Curl_handler struct) with function pointers for lifecycle stages: connect, do, done, disconnect. The HTTP handler (lib/http.c) implements HTTP/1.1 pipelining and HTTP/2 multiplexing through this interface. The FTP handler manages control and data connections. WebSocket support reuses the HTTP handler's connection establishment, then switches to a different do callback for frame processing. Shared infrastructure like SSL wrapping (the vtable system supports OpenSSL, mbedTLS, NSS, BearSSL, and others) and proxy tunneling sits above the protocol layer, invoked automatically based on URL scheme and configuration.

The connection cache (lib/conncache.c) is where curl's efficiency becomes apparent. It maintains a pool of persistent connections keyed by protocol, hostname, port, and credentials. When you request https://api.example.com/users followed by https://api.example.com/posts, curl reuses the TCP connection and TLS session. With HTTP/2, multiple easy handles can share a single connection through stream multiplexing—the multi handle automatically detects this and routes requests over existing connections. This pooling even works across protocols when semantically valid: an HTTP connection can be upgraded to WebSocket, reusing the negotiated TLS session.

The URL parser (lib/urlapi.c) demonstrates curl's performance obsessions. Rather than allocating strings for each URL component, it returns struct Curl_URL with pointers and lengths into the original buffer:

CURLU *url_handle = curl_url();
curl_url_set(url_handle, CURLUPART_URL, 
             "https://user:pass@host.com:8080/path?query#fragment", 0);

char *scheme, *host;
curl_url_get(url_handle, CURLUPART_SCHEME, &scheme, 0); // "https"
curl_url_get(url_handle, CURLUPART_HOST, &host, 0);     // "host.com"

// Returns allocated strings, but internally uses zero-copy views
// until you extract components
curl_free(scheme);
curl_free(host);
curl_url_cleanup(url_handle);

This API handles edge cases that break naive parsers: IPv6 literals in URLs (https://[::1]:8080/), URL-encoded credentials with special characters, scheme-relative URLs, and normalization for comparison. The implementation uses a state machine with 40+ states to parse according to WHATWG URL specification while maintaining backward compatibility with older RFC interpretations. It's this attention to RFC compliance across all 20+ protocols that makes curl irreplaceable—your application doesn't need to know that FTPS uses implicit TLS on port 990 versus explicit TLS on port 21, or that LDAP URLs support filter extensions. curl handles it.

The callback architecture deserves special attention because it inverts control in ways that surprise developers used to streaming APIs. Instead of reading from a response object, you provide callbacks that curl invokes with chunks of data:

size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata) {
  size_t real_size = size * nmemb;
  struct MemoryStruct *mem = (struct MemoryStruct *)userdata;
  
  // Reallocate buffer, append data
  mem->memory = realloc(mem->memory, mem->size + real_size + 1);
  memcpy(&(mem->memory[mem->size]), ptr, real_size);
  mem->size += real_size;
  
  return real_size; // Must return bytes processed
}

curl_easy_setopt(easy_handle, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(easy_handle, CURLOPT_WRITEDATA, (void *)&chunk);

This design enables zero-copy in the common case—if you're streaming to disk or another socket, you can process data as it arrives without intermediate buffers. The callback must return the number of bytes processed; returning a different value signals backpressure or error, pausing the transfer. There are separate callbacks for headers (CURLOPT_HEADERFUNCTION), progress (CURLOPT_PROGRESSFUNCTION), and debug tracing. This model maps cleanly to async I/O patterns but feels alien to developers expecting synchronous read operations.

Gotcha

curl's C heritage means error handling is verbose and easy to mishandle. Return codes like CURLE_OK, CURLE_COULDNT_CONNECT, and CURLE_OPERATION_TIMEDOUT require checking after every operation, but some failures manifest only through callbacks returning error counts. The CURLINFO_RESPONSE_CODE option retrieves HTTP status codes, but only after a successful transfer—a DNS failure means no response code, yet curl_easy_perform() returns CURLE_COULDNT_RESOLVE_HOST instead. Distinguishing between network failures, protocol errors, and application-level failures requires reading documentation for each protocol's error semantics. The callback-based architecture compounds this: if your write callback crashes, curl can't unwind the stack safely, often leaving connections in inconsistent states.

The build system is notoriously complex due to supporting every Unix variant, Windows, and embedded platforms across three decades. autoconf generates 40,000+ line configure scripts that probe for hundreds of optional dependencies. Disabling features with --without-ssl seems safe until you discover that some protocol handlers assume SSL is always present, causing link errors. CMake support is improving but trails autoconf in platform coverage. Cross-compiling for ARM or MIPS requires carefully specifying --host, and pkg-config integration breaks on systems with libraries in non-standard paths. HTTP/3 support requires choosing between quiche (Rust-based, requires cargo) and ngtcp2 (C, requires OpenSSL with QUIC patches)—each has different performance characteristics and maturity levels, and you can't easily switch between them after building. For production deployments, plan to spend time understanding exactly which features compiled into your binary via curl_version_info().

Verdict

Use if: You need multi-protocol support beyond HTTP/HTTPS (FTP, SMTP, LDAP, MQTT, WebSocket) in a single library; you're embedding in C/C++ firmware or systems without high-level language runtimes; you require proven RFC compliance and security response (curl's CVE disclosure and patching process is exemplary); you're building language bindings or integrating with existing event loops where curl's multi interface composes cleanly; or you need maximum portability across Unix, Windows, and embedded platforms with long-term ABI stability guarantees.

Skip if: You're building pure HTTP/HTTPS services in Rust, Go, or modern languages where native clients (reqwest, hyper, net/http) provide better ergonomics and memory safety; you need HTTP/3 as a primary transport today—curl's QUIC support works but remains experimental with dependency complexity; you're prototyping REST APIs and want human-friendly CLI tools (HTTPie, Postman) over curl's terse syntax; or you're unfamiliar with C memory management and can't invest time learning callback lifecycles and error handling patterns. curl is infrastructure-grade software, but the learning curve is real and the API footguns can cause subtle bugs if you don't read documentation carefully.