Building a High-Performance Proxy Rotator in Go: Inside mubeng’s Concurrency Model
Hook
Most proxy rotation tools add 200-500ms of latency per request. mubeng adds virtually none, processing proxy validation at thousands per second using Go’s concurrency primitives.
Context
Web scrapers, security researchers, and API automation engineers face a perpetual arms race: rate limits, IP blocks, and geographic restrictions. Traditional solutions involve either expensive rotating proxy services ($300-500/month) or clunky tools like ProxyChains that force-proxy entire applications without health checking or intelligent rotation.
The problem intensifies when you’re managing proxy pools of hundreds or thousands of IPs scraped from various sources. You need to validate them continuously (proxies die constantly), rotate them intelligently (not just round-robin), and handle failures gracefully (retry with a different proxy). Existing tools either excel at one dimension or require complex configuration. mubeng emerged from this frustration: a single Go binary that does both validation and rotation with zero configuration beyond providing a text file of proxies.
Technical Insight
mubeng’s architecture revolves around two core modes, both leveraging Go’s goroutine-based concurrency model for exceptional performance.
Concurrent Proxy Validation
The checker mode spawns goroutines to test proxies in parallel. By default, it limits concurrency to 50 goroutines (configurable via --goroutine), preventing resource exhaustion while maintaining high throughput:
▶ mubeng -c -f proxies.txt --goroutine 100 --timeout 10s --only-cc US,GBEach goroutine attempts an HTTP request through a proxy within the timeout window. The country code filtering (--only-cc) happens post-validation by inspecting the IP’s geolocation, allowing you to build geographically-targeted proxy pools. This concurrent checking model can validate 1,000+ proxies in under 30 seconds on modest hardware—something that would take minutes with sequential checking.
Rotation Engine Architecture
The proxy server mode is where mubeng’s design gets interesting. It acts as a man-in-the-middle HTTP/SOCKS proxy that maintains an in-memory pool of upstream proxies:
▶ mubeng -a 0.0.0.0:8080 -f proxies.txt --rotate 5 --rotate-on-error --remove-on-errorThis configuration creates a local proxy on port 8080 that rotates to a new upstream proxy every 5 requests. The rotation counter increments even on failed requests by default—a crucial detail. If you set --rotate 1, each request uses a different proxy, but failed requests still consume that “slot.”
The --rotate-on-error flag introduces retry logic: if an upstream proxy fails (connection refused, timeout, HTTP error), mubeng automatically retries with the next proxy in the pool. Pair this with --max-errors 3 to limit retry attempts. Set it to -1 for infinite retries, useful when you have a massive proxy pool and can tolerate latency spikes.
The --remove-on-error flag is aggressive: it permanently removes failing proxies from the in-memory pool. This is powerful for maintaining pool health but means your proxy file becomes a “startup configuration” rather than a live truth source—unless you use --watch to reload the file when it changes on disk.
Protocol Flexibility
mubeng’s proxy format parser supports multiple protocols in a single file:
http://proxy1.example.com:8080
socks5://user:pass@proxy2.example.com:1080
socks4a://proxy3.example.com:9050
https://api-gateway-id.execute-api.region.amazonaws.com/stageThe Amazon API Gateway support is particularly clever for users with AWS-based proxy infrastructure. You can template credentials dynamically:
http://user-<<RANDOM>>:password@proxy.example.com:8080The <<RANDOM>> placeholder generates a unique string per request, useful for systems that track sessions by username.
Integration with Security Tools
For penetration testers, mubeng integrates seamlessly as an upstream proxy for Burp Suite or OWASP ZAP. In Burp, navigate to Settings → Network → Connections → Upstream Proxy Servers, add 127.0.0.1:8080, and all Burp traffic flows through your rotating proxy pool. This is invaluable for testing rate limiting or IP-based blocking mechanisms without triggering defensive countermeasures against your real IP.
Daemon Mode for Production
The --daemon flag backgrounds the process and installs mubeng as a system service on Linux/OSX systems (setting up callbacks on Windows), enabling persistent rotation infrastructure. The authentication flag --auth user:pass adds basic authentication to your proxy server, preventing unauthorized access if you expose it beyond localhost.
Gotcha
The rotation semantics can surprise you. Because rotation happens based on request count (not successful requests), a proxy pool with many dead proxies will rotate faster than expected. If you have 10 proxies, set --rotate 1, and 5 proxies are dead, you’ll cycle through all proxies in roughly 10 requests, not 10 successful requests. The --rotate-on-error flag mitigates this but adds retry latency.
The proxy health model appears to be binary: alive or dead based on the concurrent checking behavior. The --sync flag is necessary if you need strict rotation guarantees, as the proxy server runs asynchronously by default. The README notes explicitly that async mode “doesn’t guarantee that your requests after N (which is N+1 and so on) will rotate the proxy IP.” Most production use cases involving thousands of requests will see unpredictable rotation timing in async mode, especially under load, unless you use --sync to wait for previous requests to complete (which reduces throughput).
Verdict
Use mubeng if you need a zero-config proxy rotator for web scraping, penetration testing, or API automation where IP rotation is more important than latency optimization. It excels when you have large proxy pools (100+ proxies) that need continuous validation and automatic failure handling. The Burp Suite/ZAP integration makes it perfect for security researchers testing IP-based defenses. The ability to install as a daemon and authenticate the proxy server makes it production-ready for internal infrastructure. Skip it if you need guaranteed sequential rotation semantics under high load—the async nature and error-based rotation make behavior unpredictable when many proxies are failing, unless you use --sync mode which reduces throughput. For scenarios requiring strict rotation guarantees with high performance, invest in HAProxy with custom Lua scripts or a commercial rotating proxy service with SLA guarantees.
