Building Production Web Scrapers in Go: Inside Colly's Event-Driven Architecture

Hook

While most Go developers reach for net/http and goquery to build scrapers, they end up reimplementing the same rate limiting, cookie management, and concurrency patterns every time. Colly baked all of that into a framework that's scraped billions of pages.

Context

Web scraping in Go historically meant stitching together disparate libraries: net/http for requests, goquery for HTML parsing, custom semaphores for concurrency, homegrown rate limiters to avoid getting blocked, and manual cookie jar management. Every project started from scratch, and production scrapers became tangled messes of goroutines, channels, and error-handling boilerplate.

Colly emerged in 2017 to solve this fragmentation. Built by developers who needed to crawl millions of pages reliably, it consolidated the scraping stack into a single framework with sensible defaults. The key insight was treating scraping as an event-driven problem: instead of imperative "fetch, parse, store" loops, you register callbacks for different stages of the request lifecycle. This architectural choice, borrowed from Node.js frameworks like Cheerio and Puppeteer, turned out to be perfect for Go's concurrency model. Today, Colly powers everything from e-commerce price monitors to search engine crawlers, processing terabytes of HTML daily.

Technical Insight

Colly's architecture centers on the Collector type, which acts as a request orchestrator. Unlike traditional HTTP clients where you manually manage each request, the Collector lets you register handler functions that fire at specific lifecycle events: before a request goes out, when HTML arrives, after response processing, or on errors. This callback model composes beautifully for complex scraping workflows.

Here's how you'd scrape Hacker News, following comment threads recursively:

package main

import (
    "fmt"
    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector(
        colly.AllowedDomains("news.ycombinator.com"),
        colly.MaxDepth(2),
    )

    // Rate limit: 1 request per second per domain
    c.Limit(&colly.LimitRule{
        DomainGlob:  "*",
        Parallelism: 2,
        Delay:       1 * time.Second,
    })

    // Extract story titles and links
    c.OnHTML(".titleline > a", func(e *colly.HTMLElement) {
        title := e.Text
        link := e.Attr("href")
        fmt.Printf("Story: %s\n", title)
        
        // Follow internal links only
        if strings.HasPrefix(link, "item?id=") {
            e.Request.Visit(link)
        }
    })

    // Extract comment text on detail pages
    c.OnHTML(".comment-tree .commtext", func(e *colly.HTMLElement) {
        comment := e.Text
        fmt.Printf("  Comment: %.80s...\n", comment)
    })

    // Log errors
    c.OnError(func(r *colly.Response, err error) {
        fmt.Printf("Error: %s\n", err)
    })

    c.Visit("https://news.ycombinator.com/")
}

Notice what you didn't write: no goroutine management, no rate limiter implementation, no cookie handling, no retry logic. Colly handles it all. The Limit configuration automatically queues requests per domain, respecting the delay while maintaining parallelism. The MaxDepth prevents infinite recursion. The AllowedDomains whitelist ensures you don't accidentally crawl the entire internet.

Under the hood, Colly uses a domain-keyed request queue. When you call Visit() or e.Request.Visit(), requests go into per-domain buckets. A scheduler pulls from these buckets based on your LimitRule, spawning goroutines that respect both parallelism and delay constraints. This architecture is why Colly can sustain 1000+ requests/sec on a single core while never overwhelming a target server.

The framework's extension system adds powerful capabilities without cluttering the core API. Need to cache responses to avoid re-fetching? Add two lines:

import "github.com/gocolly/colly/v2/extensions"

c := colly.NewCollector()
extensions.RandomUserAgent(c)  // Rotate user agents
extensions.Referer(c)           // Set realistic referer headers

For distributed scraping across multiple machines, Colly supports pluggable storage backends through the storage interface. Swap in Redis or MongoDB to share visited URL state:

import "github.com/gocolly/redisstorage"

storage := &redisstorage.Storage{
    Address:  "127.0.0.1:6379",
    Password: "",
    DB:       0,
    Prefix:   "colly_crawler",
}

c := colly.NewCollector()
c.SetStorage(storage)

Now multiple Colly instances can crawl cooperatively, deduplicating URLs across processes or servers. This is production-grade distributed systems engineering, abstracted behind a clean interface.

The callback architecture also enables sophisticated error handling and retry logic. The OnError callback fires for HTTP errors, but you can implement exponential backoff by re-queuing failed requests:

c.OnError(func(r *colly.Response, err error) {
    if r.StatusCode == 429 {  // Rate limited
        time.Sleep(60 * time.Second)
        r.Request.Retry()
    }
})

Colly's design shines when scraping structured data from traditional server-rendered sites. The goquery-based HTML selector API is intuitive for anyone who's used jQuery, and the automatic handling of cookies, redirects, and encoding means you spend time writing business logic, not debugging HTTP quirks.

Gotcha

Colly's biggest limitation is its reliance on static HTML. Modern single-page applications built with React, Vue, or Angular render content client-side via JavaScript. Since Colly only makes HTTP requests and parses the initial HTML response, it sees empty divs and skeleton screens—not the data you need. If you're scraping Twitter, LinkedIn, or any JavaScript-heavy site, Colly won't work. You need a headless browser like chromedp or rod that executes JavaScript and waits for dynamic content to load. These tools are 10-100x slower and more resource-intensive, but they're your only option for modern SPAs.

The callback-based API, while powerful, leads to deeply nested code for complex workflows. If your scraper needs to make decisions based on parsed data before deciding what to scrape next, you end up with callback pyramids reminiscent of JavaScript's pre-Promise era. For example, scraping a site that requires login, then pagination through search results, then detail page extraction creates three levels of nested OnHTML handlers. Some developers prefer the linear, imperative style of manually calling HTTP functions and parsing responses sequentially, even if it means more boilerplate.

Colly also doesn't include advanced anti-detection features. While it handles user-agent rotation and basic header manipulation through extensions, sophisticated sites with bot detection (Cloudflare, Akamai, PerimeterX) will still block you. You'll need to integrate third-party proxy services, implement request fingerprint randomization, or add browser-like request timing yourself. Scrapy's ecosystem has mature plugins for this; Colly's is nascent.

Verdict

Use Colly if you're building production scrapers in Go for server-rendered websites and want automatic rate limiting, distributed crawling support, and concurrency management without writing infrastructure code. It's the right choice for e-commerce monitoring, content aggregation, SEO analysis, or any high-volume scraping where targets serve traditional HTML. The framework's maturity (25k+ stars, active maintenance, extensive documentation) means you're not betting on abandonware. Skip Colly if your targets are JavaScript-heavy SPAs that require browser rendering—reach for chromedp or rod instead. Also skip it if you prefer imperative, linear code over callback-driven architectures, or if you need cutting-edge anti-bot evasion out of the box. For those use cases, Scrapy's Python ecosystem or custom solutions with residential proxies make more sense.