Web Sight: Building Enterprise-Scale Reconnaissance with Docker and Celery

Hook

Most security teams waste 60% of their penetration testing time on manual reconnaissance. Web Sight automates the entire attack surface enumeration pipeline—but at what operational cost?

Context

Before automated reconnaissance platforms, security professionals manually chained together dozens of tools: nmap for port scanning, sublist3r for DNS enumeration, EyeWitness for screenshots, custom scripts for SSL certificate inspection. Each tool required different runtime environments, produced incompatible output formats, and couldn't scale beyond single-host execution. For a Fortune 500 company with thousands of domains and constantly shifting infrastructure, this manual approach meant reconnaissance data was stale before analysts could act on it.

Web Sight emerged from this operational pain point as a comprehensive attack surface mapping platform. Presented at Black Hat Arsenal 2017, it consolidates the entire reconnaissance workflow—DNS enumeration, network scanning, SSL inspection, web crawling, and visual documentation—into a single orchestrated system. The ws-docker-community repository packages this N-tier architecture into a Docker Compose deployment, theoretically simplifying what would otherwise require expertise across Django, Celery, RabbitMQ, PostgreSQL, Elasticsearch, Redis, and Angular to deploy manually.

Technical Insight

Web Sight's architecture centers on Celery's distributed task queue pattern to achieve horizontal scalability. When you submit a domain for reconnaissance, Django Rest Framework receives the request and decomposes it into atomic tasks: DNS record enumeration, subdomain discovery, IP range identification, port scanning per host, HTTP service fingerprinting, SSL certificate extraction, and screenshot capture. Each task type is routed through RabbitMQ to specialized Celery workers.

The task flow demonstrates classic queue-based fan-out: a single domain generates hundreds of child tasks. For example, DNS enumeration might discover 50 subdomains, each spawning network scan tasks for associated IP addresses, which then spawn web crawling tasks for each discovered HTTP service. This cascading work distribution is where Celery shines—you can horizontally scale by adding worker containers that subscribe to the RabbitMQ queues.

Here's how you'd configure a custom Celery task in Web Sight's architecture:

from celery import shared_task
from lib.sqlalchemy import get_sa_session
from wselasticsearch import bootstrap_index_model

@shared_task
def scan_network_range(org_uuid, ip_range, port_list):
    """
    Distributed network scanning task that can be
    parallelized across multiple Celery workers
    """
    results = []
    for ip in expand_cidr(ip_range):
        for port in port_list:
            # Actual scanning logic here
            scan_result = perform_port_scan(ip, port)
            if scan_result.is_open:
                results.append({
                    'ip': ip,
                    'port': port,
                    'service': scan_result.service
                })
                # Chain additional tasks for open ports
                if port in [80, 443, 8080, 8443]:
                    screenshot_web_service.delay(
                        org_uuid, ip, port
                    )
    
    # Bulk index to Elasticsearch for fast querying
    es_model = bootstrap_index_model(org_uuid)
    es_model.bulk_create(results)
    
    # Persist summary to PostgreSQL
    db_session = get_sa_session()
    create_network_scan_record(
        db_session, org_uuid, ip_range, len(results)
    )
    return len(results)

The data storage strategy reveals thoughtful architectural decisions. Elasticsearch handles the high-volume, write-heavy reconnaissance results where queries need full-text search across service banners, SSL certificate subject names, and HTTP headers. PostgreSQL stores relational metadata: organizations, scan configurations, user accounts, and task orchestration state. Redis provides caching for frequently accessed organization settings and API rate limiting. This polyglot persistence pattern optimizes each datastore for its strengths rather than forcing a single database to handle incompatible workloads.

The Docker Compose orchestration in ws-docker-community connects these services through named networks and environment-based configuration. The compose file defines service dependencies ensuring RabbitMQ starts before Celery workers, and database migrations run before the Django application accepts requests. Volume mounts persist PostgreSQL data and Elasticsearch indices across container restarts, critical for maintaining reconnaissance history.

What's particularly clever is how Web Sight handles credential management for distributed scanning. Rather than embedding API keys in task code, credentials are stored per-organization in PostgreSQL and injected into task context at runtime. This means a single Celery worker pool can execute reconnaissance tasks for multiple tenants without cross-contamination, enabling a SaaS deployment model from the same codebase that powers the community edition.

Gotcha

The documentation's 'work in progress' warning isn't just modesty—Web Sight's setup requires substantial infrastructure beyond the Docker containers. You must provision PostgreSQL and Elasticsearch separately, configure AWS S3 buckets with appropriate IAM policies, obtain Stripe API keys even if you're not accepting payments (it's hardcoded into the application initialization), and optionally integrate Farsight DNSDB for passive DNS lookups. The Docker deployment only orchestrates the application tier; you're responsible for production-grade database hosting, backup strategies, and Elasticsearch cluster management.

The third-party service dependencies create operational lock-in that contradicts the 'community edition' framing. AWS S3 isn't optional—the application expects S3 for storing screenshots and crawl results. Stripe integration is mandatory in the codebase even though the community edition doesn't offer subscription features, suggesting incomplete extraction from a commercial version. These aren't simple feature flags; they're architectural assumptions baked into the data models. You'll need to fork and refactor significant portions to eliminate these dependencies, at which point you might question whether starting from scratch with modern alternatives would be simpler. The project's activity trailing off after 2017 means you're also inheriting technical debt from Django 1.x patterns and Angular 2 before the framework's stabilization.

Verdict

Use if: You're managing attack surface reconnaissance for enterprise-scale infrastructure (1000+ domains), have security engineering resources to maintain an N-tier distributed system, already run PostgreSQL and Elasticsearch in production, can justify AWS/Stripe API costs, and need horizontally scalable task processing to handle continuous monitoring rather than point-in-time assessments. The Celery architecture genuinely shines when you're processing thousands of concurrent scanning tasks across global infrastructure. Skip if: You need reconnaissance for individual projects or small networks where Amass or Recon-ng would suffice, want a maintained actively-developed tool (check the commit history first), lack infrastructure for running seven interconnected services in production, can't justify third-party API costs for internal security tooling, or prefer modern Go-based tools like Nuclei and httpx that offer similar capabilities with minimal dependencies. The operational complexity and abandonment risk outweigh the benefits unless you're operating at enterprise scale.