Building a Web Directory Bruteforcer: A Case Study in Go Concurrency

Hook

The author built this entire tool specifically to learn Go's concurrency model—and the result is one of the clearest examples of goroutines and channels in production security tooling.

Context

Web directory enumeration is foundational reconnaissance in security testing. Before tools like dirsearch, penetration testers manually crawled websites or used slow, single-threaded scripts to discover hidden files, backup directories, admin panels, and forgotten endpoints. The problem is scale: testing thousands of potential paths against a target server is IO-bound work that begs for parallelization.

The original Python dirsearch by Mauro Soria became a go-to tool, but Python's Global Interpreter Lock limits true parallelism. Enter evilsocket's Go reimplementation—not built to compete with feature-rich alternatives, but explicitly as a learning exercise in Go's concurrency primitives. What makes this repository valuable isn't its feature completeness (it's intentionally minimal), but its clarity as a teaching tool. It demonstrates how to build a production-ready concurrent application in Go with fewer than 300 lines of code, making it an ideal case study for developers moving from synchronous to concurrent programming paradigms.

Technical Insight

At its core, dirsearch implements the producer-consumer pattern using Go's channels and goroutines. The architecture is refreshingly simple: one goroutine reads wordlist entries and pushes them into a buffered channel, while a configurable pool of consumer goroutines pulls from that channel and performs HTTP HEAD requests concurrently.

Here's the simplified consumer pattern from the codebase:

for i := 0; i < consumers; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for path := range paths {
            target := baseURL + path
            resp, err := client.Head(target)
            if err != nil {
                atomic.AddInt32(&errors, 1)
                if errors >= maxErrors {
                    close(paths)
                    return
                }
                continue
            }
            
            if !filter200 || resp.StatusCode == 200 {
                fmt.Printf("%d %s\n", resp.StatusCode, target)
            }
        }
    }()
}

The beauty here is in what Go handles for you. Each go func() spawns a lightweight goroutine (not an OS thread), allowing you to run hundreds of concurrent workers without the overhead of traditional threading models. The paths channel acts as a synchronized queue—goroutines block on the channel read until data is available, eliminating the need for manual locking or semaphore management.

The producer side is equally elegant. It reads the wordlist file line-by-line and, if an extension is specified, appends it to each entry before pushing to the channel:

go func() {
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        path := scanner.Text()
        if extension != "" {
            path = path + "." + extension
        }
        paths <- path
    }
    close(paths)
}()

Closing the channel after the producer finishes is critical—it signals to all consumers that no more work is coming. When a consumer reads from a closed channel with no remaining buffered data, the range loop exits cleanly, triggering the deferred wg.Done() call. The sync.WaitGroup then ensures the main goroutine waits for all consumers to finish before the program exits.

The tool uses HEAD requests instead of GET for efficiency—HEAD returns only headers, not the response body, reducing bandwidth and speeding up enumeration. This is a practical optimization often overlooked in naive implementations. For a wordlist with 10,000 entries against a server with large HTML responses, this can mean the difference between a 10-minute scan and an hour-long one.

Error handling demonstrates atomic operations for thread-safe counter increments. The atomic.AddInt32(&errors, 1) ensures that multiple goroutines can safely increment the error counter without race conditions. When errors exceed the threshold, the producer channel is closed early, gracefully terminating all workers. This prevents runaway execution against dead servers or firewall-blocked targets.

The configurable consumer pool size (default 8) reflects a key concurrency principle: more goroutines doesn't always mean better performance. Too many concurrent requests can trigger rate limiting, overwhelm the target server, or saturate your network connection. Eight is a reasonable default that balances discovery speed with polite scanning, though aggressive assessments might crank this to 50 or 100 depending on the target's capacity.

Gotcha

The biggest limitation is the single-extension constraint. If you're testing a web application that uses multiple languages (PHP for the blog, JSP for the admin panel, Python for the API), you'll need to run dirsearch multiple times with different extension flags. This isn't just inconvenient—it multiplies scan time linearly with the number of extensions you want to test. Modern alternatives like ffuf allow pattern-based fuzzing where you can test multiple extensions simultaneously.

Dependency management is another pain point. The repository uses Glide, a dependency management tool that predates Go modules and has been effectively abandoned. On Go 1.16+, you'll need to either convert the project to modules manually or use an older Go toolchain. This makes the 'just clone and run' experience frustrating for developers accustomed to modern Go workflows. The filtering options are also primitive—you can filter for 200 status codes only, but you can't create complex rules like 'show me 200-299 and 403, but hide 404 and 500.' You also can't analyze response sizes, which is crucial for detecting wildcard responses where a server returns 200 for every request with identical content. Without content-length or body hash filtering, you'll get false positives on misconfigured servers.

Verdict

Use if: You're learning Go and want a crystal-clear example of channels, goroutines, and the producer-consumer pattern in a real-world context, or you need an educational codebase to teach concurrent programming concepts. It's also viable for quick, one-off directory enumeration tasks where you don't need advanced features and want something lightweight with minimal dependencies. Skip if: You're doing professional penetration testing or security research that demands recursive scanning, multiple extension testing, authentication support, or sophisticated response filtering. In production scenarios, gobuster offers better performance with more features, ffuf provides extensive filtering and fuzzing capabilities, and feroxbuster includes modern conveniences like auto-tuning and wildcard detection. This tool's value is pedagogical, not practical—treat it as a learning resource, not your daily driver.