> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

SearXNG: Building a Privacy-Respecting Metasearch Engine That Scrapes the Scrapers

[ View on GitHub ]

SearXNG: Building a Privacy-Respecting Metasearch Engine That Scrapes the Scrapers

Hook

Most search engines track you. SearXNG tracks them instead—dispatching parallel requests across 80+ sources, scraping results with deliberately randomized headers, and treating bot-detection evasion as a first-class architectural feature rather than something to whisper about in closed-door meetings.

Context

The search engine privacy problem isn't new, but it's gotten worse. Google processes 8.5 billion searches daily, building psychological profiles that follow you across the web. DuckDuckGo promised privacy but got caught allowing Microsoft trackers. Brave Search requires trusting a crypto company's claims about data handling. The common thread: centralized control means trusting someone else's promises.

SearXNG emerged from this trust vacuum as a self-hosted metasearch engine—a proxy server that aggregates results from dozens of search engines without storing queries or building profiles. It's the continuation of the original Searx project, forked when development stagnated. The value proposition is radical transparency: you run the code, you control the logs, you verify the privacy claims by inspecting the source. This isn't a service you subscribe to; it's infrastructure you operate, trading Google's convenience for absolute certainty about who sees your queries.

Technical Insight

optional

configure

User Query

Flask Router

AsyncIO Engine Pool

Engine Module 1

request + response

Engine Module 2

request + response

Engine Module N

request + response

Result Aggregator

Deduplication

Weighted Ranking

Plugin Hooks

filters/modifications

Template Engine

UI themes

Redis Cache

rate limiting

YAML Settings

timeouts/priorities

System architecture — auto-generated

SearXNG's architecture centers on an elegant abstraction: every search engine is a Python module implementing a standardized interface. Each engine module exports configuration metadata and implements two core functions—request() prepares the HTTP call, and response() parses results into normalized dictionaries.

Here's a simplified engine implementation:

# searx/engines/example_engine.py
categories = ['general']
paging = True
time_range_support = True

def request(query, params):
    params['url'] = 'https://example.com/search'
    params['method'] = 'GET'
    params['data'] = {
        'q': query,
        'page': params['pageno']
    }
    # Anti-detection: randomize headers per request
    params['headers']['User-Agent'] = random.choice(USER_AGENTS)
    params['headers']['Accept-Language'] = 'en-US,en;q=0.9'
    return params

def response(resp):
    results = []
    dom = html.fromstring(resp.text)
    
    for result in dom.xpath('//div[@class="result"]'):
        results.append({
            'url': result.xpath('.//a/@href')[0],
            'title': result.xpath('.//h3/text()')[0],
            'content': result.xpath('.//p[@class="snippet"]/text()')[0],
            'engine': 'example'
        })
    
    return results

The beauty is in the duck-typed flexibility. Engines returning API JSON look different from HTML scrapers, but both produce the same normalized output schema. The core doesn't care about implementation details—it only cares that you return a list of result dictionaries.

Query processing flows through an asyncio-based engine pool. When a search hits the Flask route, the dispatcher fans out requests to all enabled engines simultaneously, each with its own timeout (configurable per-engine in YAML). Results aggregate in a shared pool as engines complete:

# Simplified dispatch logic
async def search(query, engines):
    tasks = []
    for engine in engines:
        task = asyncio.create_task(
            engine_request(engine, query, timeout=engine.timeout)
        )
        tasks.append(task)
    
    # Gather with timeout, don't fail if some engines timeout
    results = await asyncio.gather(*tasks, return_exceptions=True)
    
    # Pool results, filter exceptions
    pooled = [r for r in results if not isinstance(r, Exception)]
    return merge_and_rank(pooled)

The ranking algorithm is deliberately simple—no machine learning, no personalization, no behavioral tracking. Each result gets a score based on: engine weight (configured priority), position in source results (first result scores higher), and keyword match quality. Results from multiple engines are deduplicated by URL normalization, and scores are summed. This naive approach actually works well because diversity of sources compensates for lack of sophistication in individual ranking.

The anti-detection mechanisms are surprisingly sophisticated. Per-engine configuration includes User-Agent pools, custom headers, and cookie handling strategies. Some engines require specific quirks:

# settings.yml excerpt
engines:
  - name: google
    engine: google
    shortcut: go
    # Rotate through mobile and desktop UAs
    enable_http2: true
    use_mobile_ui: false
    # Required for non-captcha results
    cookies:
      - 'CONSENT=YES+'
  
  - name: startpage
    engine: startpage
    # Startpage requires specific session handling
    additional_headers:
      'Accept': 'text/html,application/xhtml+xml'
      'Cache-Control': 'max-age=0'

The plugin system uses decorator-based hooks throughout the request/response lifecycle. Want to strip tracking parameters from all URLs? Write a result finalizer plugin:

# searx/plugins/tracker_removal.py
from flask_babel import gettext

name = gettext('Tracker URL remover')
description = gettext('Remove tracking parameters from URLs')

default_on = True

TRACKING_PARAMS = [
    'utm_source', 'utm_medium', 'utm_campaign',
    'fbclid', 'gclid', 'msclkid'
]

def post_search(request, search):
    for result in search.result_container.get_ordered_results():
        if 'url' in result:
            result['url'] = remove_tracking(result['url'])
    return True

def remove_tracking(url):
    parsed = urlparse(url)
    params = parse_qs(parsed.query)
    cleaned = {k: v for k, v in params.items() 
               if k not in TRACKING_PARAMS}
    return urlunparse(parsed._replace(
        query=urlencode(cleaned, doseq=True)
    ))

Plugins register at module import time via decorator introspection. The core pipeline discovers them automatically and invokes hooks based on naming conventions—post_search, pre_request, on_result. This is cleaner than explicit plugin registration APIs because there's no central registry to maintain.

The Redis integration provides optional result caching and granular rate limiting. Crucially, rate limits apply per (IP + engine) combination, not globally. This recognizes that scraping 80 engines simultaneously requires different backoff strategies—you might need to slow requests to Google while maintaining full speed against DuckDuckGo. The token bucket implementation tracks consumption per user per engine, allowing responsive UX while avoiding detection.

Gotcha

The fundamental limitation is engine maintenance. Search engines change HTML structure constantly, breaking XPath selectors with no warning. At any given time, expect 10-20% of engines to be silently failing. The community fixes these quickly, but you're committing to regular updates and occasional manual fixes. If you enable an engine and it returns zero results, you'll be debugging CSS selectors and reverse-engineering DOM changes.

Scalability hits a ceiling fast. SearXNG runs as a single-process Flask app with asyncio for concurrent engine requests, but there's no horizontal scaling support. Running a public instance means external load balancing, distributed Redis caching, and monitoring infrastructure that SearXNG doesn't help with. The architecture assumes you're serving yourself or a small organization, not building the next DuckDuckGo. Result quality also degrades on obscure queries—the naive ranking can't do query expansion or semantic understanding, so you get literal keyword matching and nothing more. And legally, this entire approach exists in a gray area. Major search engines prohibit automated access in their ToS. You're betting they won't bother with legal action against small self-hosters, which is probably correct but not guaranteed.

Verdict

Use if: You're self-hosting for personal use or a small team, you have the technical chops to fix broken scrapers occasionally, and your threat model is "avoid corporate tracking" rather than "hide from nation-states." SearXNG excels when you control the infrastructure and accept maintenance burden as the price of verifiable privacy. The AGPL license is fine for internal tools, and the architecture handles hundreds of concurrent users gracefully. It's the pragmatic choice when you want Google-quality results without Google surveillance. Skip if: You need production SaaS reliability, can't tolerate 20% of engines breaking between updates, or want zero-touch maintenance. Also skip if you're building a commercial search product—the AGPL requires open-sourcing modifications, and the legal ambiguity of scraping makes this unsuitable for anything revenue-dependent. Choose Whoogle for simpler Google-only proxying, or just pay for Kagi if operational burden outweighs privacy absolutism.