Building a GitHub Trending Bot in Go: Lessons from a Pre-API Era Project
Hook
Before GitHub had a proper API for trending repositories, developers had to scrape HTML. This Go bot did exactly that—and included a clever growth hack that analyzed follower networks to find new audiences.
Context
In the mid-2010s, GitHub’s trending page was prime real estate for developers looking to discover hot new projects, but there was no easy way to stay updated without manually checking the site. The trending algorithm surfaced repositories gaining traction, but for busy developers, keeping tabs on this firehose of innovation meant yet another tab to check daily. Andy Grunwald built TrendingGithub to solve this friction by bringing GitHub’s trending content directly to Twitter, where developers already spent their time.
The project represents a particular moment in developer tooling history: scraping was often the only option when official APIs didn’t exist or didn’t expose certain features. While modern GitHub provides comprehensive REST and GraphQL APIs, TrendingGithub’s architecture offers valuable lessons about building resilient bots, managing state with time-based blacklists, and growing social media audiences programmatically. It’s a window into how developers solve integration problems when the front door isn’t available, and the patterns it employs remain relevant even as the specific implementation has aged.
Technical Insight
TrendingGithub’s architecture revolves around two concurrent goroutines running on different schedules. The main loop fetches trending repositories every 30 minutes and tweets them, while a secondary loop refreshes Twitter’s URL shortener configuration every 24 hours. This dual-timer approach is surprisingly elegant for a bot that needs both frequent updates and periodic maintenance tasks.
The storage abstraction is where the project shines architecturally. Rather than hardcoding Redis dependencies throughout, it defines a simple interface that both in-memory and Redis implementations satisfy:
type StorageBackend interface {
Get(key string) (string, error)
Set(key string, value string, ttl time.Duration) error
Exists(key string) (bool, error)
}
type MemoryStorage struct {
data map[string]storageItem
mu sync.RWMutex
}
type RedisStorage struct {
client *redis.Client
}This lets developers run the bot locally without Redis, using an in-memory map protected by a read-write mutex for concurrency safety. In production, swapping to Redis is just a configuration change. The 30-day TTL on blacklist entries is particularly clever—it acknowledges that trending content is ephemeral. A repository might trend again months later, and that’s fine. Permanent deduplication would be overkill for this use case.
The tweet composition logic demonstrates thoughtful engineering around Twitter’s constraints. Since Twitter automatically shortens URLs to t.co links of variable length, the bot makes an API call to fetch the current shortener configuration before composing tweets:
func (t *Twitter) GetTCoLength() (int, error) {
config, _, err := t.client.Help.Configuration()
if err != nil {
return 0, err
}
return config.ShortURLLength, nil
}
func (t *Twitter) ComposeTweet(repo TrendingRepo, tcoLength int) string {
// Calculate actual length accounting for t.co replacement
baseText := fmt.Sprintf("%s: %s", repo.Name, repo.Description)
urlLength := tcoLength + 1 // +1 for space
maxTextLength := 140 - urlLength
if len(baseText) > maxTextLength {
baseText = baseText[:maxTextLength-3] + "..."
}
return fmt.Sprintf("%s %s", baseText, repo.URL)
}This dynamic calculation ensures tweets don’t get truncated unpredictably. Instead of assuming a fixed URL length, the bot adapts to Twitter’s actual configuration at runtime.
The most interesting feature is the “growth hack” for building followers. Every hour, the bot analyzes its existing followers’ social graphs to find potential new followers. The algorithm picks a random follower, fetches their friend list, then follows someone from that list who isn’t already following back. This exploits Twitter’s reciprocity norms—people often follow back accounts that follow them, especially within the same community:
func (t *Twitter) GrowFollowers() error {
// Get our followers
followers, err := t.getFollowers()
if err != nil {
return err
}
// Pick a random follower
randomFollower := followers[rand.Intn(len(followers))]
// Get their friends (people they follow)
friends, err := t.getFriends(randomFollower.ID)
if err != nil {
return err
}
// Find someone we don't already follow
for _, friend := range friends {
if !friend.Following && !friend.FollowedBy {
return t.follow(friend.ID)
}
}
return nil
}This organic discovery mechanism targets people already interested in developer content (since they follow someone who follows a GitHub trending bot), making them likely to engage with the account. It’s a programmatic version of the “follow friends of friends” strategy that many accounts use manually.
Gotcha
The elephant in the room: this project almost certainly doesn’t work anymore without significant updates. It scrapes GitHub’s trending page by parsing HTML structure, which is extraordinarily brittle. GitHub has no obligation to keep their HTML consistent, and any redesign breaks the scraper completely. Moreover, web scraping at scale may violate GitHub’s terms of service, potentially leading to IP bans or legal issues.
Twitter’s API has also evolved dramatically since this bot was built. The code references 140-character limits (Twitter expanded to 280 characters in 2017), uses authentication patterns from the v1.1 API era, and doesn’t account for Twitter’s current API pricing tiers. As of 2023, Twitter’s API access requires approval and costs money for most use cases, making this approach economically non-viable for a hobby project. The growth hack feature that automatically follows users would likely trigger Twitter’s automation detection, potentially resulting in account suspension under current rules. There’s no rate limiting or exponential backoff visible in the codebase, which means hitting API limits would cause crashes rather than graceful degradation. For anyone considering building something similar today, you’d need to rewrite using GitHub’s official API, implement proper error handling and retries, and carefully navigate Twitter/X’s current API policies and pricing.
Verdict
Use if: You’re studying bot architecture patterns in Go and want to see clean examples of pluggable storage backends, timer-based scheduling with goroutines, and external API integration. This codebase demonstrates solid engineering fundamentals—interface-driven design, separation of concerns, and thoughtful handling of third-party API constraints. It’s also valuable if you’re researching social media growth strategies or need inspiration for building similar monitoring tools for other platforms. Skip if: You’re looking for production-ready code to deploy. This project is a historical artifact that requires substantial modernization. The scraping approach is fragile and potentially against terms of service, Twitter’s API has changed fundamentally, and there’s no active maintenance. For actual GitHub trend monitoring, use GitHub’s official APIs with webhooks, or explore modern notification tools like Octobox. If you want to build a similar bot today, treat this as architectural inspiration only—you’ll need to rewrite the integration layers entirely.
