Crossfeed: How CISA Built a Serverless Attack Surface Monitor on AWS

Hook

The US Cybersecurity and Infrastructure Security Agency scans millions of government assets continuously—and they've open-sourced the entire platform that does it.

Context

Before Crossfeed, federal agencies faced a fragmented landscape of security scanning tools. IT teams would manually run Amass for subdomain enumeration one week, fire up nmap for port scanning the next, and maybe remember to check Shodan for exposed services quarterly. Each tool produced its own output format, stored results in different locations, and required separate scheduling mechanisms. For large organizations managing thousands of domains and IP addresses across dozens of departments, this ad-hoc approach created massive blind spots.

CISA built Crossfeed in 2020 to solve this orchestration nightmare for federal agencies, but the problem extends far beyond government. Any organization with a sprawling digital footprint—multi-cloud deployments, acquisitions that brought legacy infrastructure, shadow IT projects—needs continuous visibility into what assets are actually exposed to the internet. Crossfeed transforms disparate security tools into a unified monitoring engine that runs automatically, stores historical data for trend analysis, and presents findings through a self-service portal where different teams can configure their own scanning profiles without bothering the security operations center.

Technical Insight

Crossfeed's architecture centers on decoupling scan orchestration from scan execution. The core is a TypeScript backend API running on Lambda, backed by PostgreSQL on RDS for storing scan configurations, results, and organizational metadata. When a scan schedule triggers, the system doesn't directly invoke tools like Amass or nmap—instead, it publishes messages to SQS queues, and separate Lambda workers consume those messages to execute actual scans.

This queue-based design provides two critical advantages: fault tolerance and cost optimization. If a Nuclei scan against 10,000 endpoints takes three hours and fails at hour two, the message remains in the queue for retry rather than losing all progress. For cost, Lambda functions only run when processing queue messages, so you're not paying for idle compute between scheduled scans. The system can burst to hundreds of concurrent workers during intensive scanning periods, then scale to zero.

The scan worker architecture follows a consistent pattern across different tools. Here's a simplified version of how a domain scan gets orchestrated:

// backend/src/tasks/scanExecution.ts
export const handler = async (event: SQSEvent) => {
  for (const record of event.Records) {
    const scan = JSON.parse(record.body) as Scan;
    
    // Fetch organization configuration
    const org = await Organization.findOne(scan.organizationId);
    const domains = await Domain.find({ organization: org });
    
    // Publish individual scan jobs to tool-specific queues
    for (const domain of domains) {
      if (scan.type.includes('amass')) {
        await sqs.sendMessage({
          QueueUrl: process.env.AMASS_QUEUE_URL,
          MessageBody: JSON.stringify({
            domainId: domain.id,
            scanId: scan.id,
            options: org.scanConfig.amassOptions
          })
        });
      }
      
      if (scan.type.includes('portscanner')) {
        await sqs.sendMessage({
          QueueUrl: process.env.PORTSCAN_QUEUE_URL,
          MessageBody: JSON.stringify({
            domainId: domain.id,
            scanId: scan.id,
            ports: org.scanConfig.ports || '80,443,8080,8443'
          })
        });
      }
    }
  }
};

Each tool-specific worker (Amass, nmap, Nuclei) runs in its own Lambda function with appropriate timeouts and memory allocations. The Amass worker might get 15 minutes and 3GB RAM for deep subdomain enumeration, while a simple HTTP screenshot worker needs only 1 minute and 512MB. This granular resource allocation prevents the "one size fits all" problem that plagues monolithic scanning platforms.

The workers themselves use Docker container images as Lambda deployment packages, which solves the "how do we install native tools like nmap in Lambda" problem. The Dockerfile for the port scanner worker looks roughly like this:

FROM public.ecr.aws/lambda/nodejs:18

# Install nmap binary
RUN yum install -y nmap nmap-ncat

# Copy function code
COPY package*.json ./
RUN npm ci --production
COPY src/ ./src/

CMD ["src/portscanner.handler"]

The worker then shells out to invoke nmap, parses the XML output, and writes structured results back to PostgreSQL. This hybrid approach—managed TypeScript for orchestration, native tools for specialized scanning—provides both developer velocity and security tool maturity.

One architectural decision that distinguishes Crossfeed from simpler scanning tools is its multi-tenant data model. Organizations don't just get separate accounts; they get complete isolation of scan configurations, results, and even the ability to define custom vulnerability severity thresholds. The database schema uses row-level security policies in PostgreSQL to ensure Organization A literally cannot query Organization B's data, even if application logic fails. This makes Crossfeed suitable for scenarios where a central security team operates the platform but individual business units need autonomous control over their monitoring without seeing each other's results.

The React frontend communicates with the backend exclusively through a REST API, and this API implements pagination, filtering, and sorting at the database level rather than in application memory. When viewing 50,000 discovered services, the query uses PostgreSQL cursors to stream results efficiently:

// backend/src/api/services.ts
export const listServices = async (req: Request, res: Response) => {
  const { organizationId } = req.user;
  const { page = 1, pageSize = 50, filters = {} } = req.query;
  
  const query = Service.createQueryBuilder('service')
    .where('service.organizationId = :organizationId', { organizationId })
    .skip((page - 1) * pageSize)
    .take(pageSize);
  
  // Dynamic filtering based on user input
  if (filters.port) {
    query.andWhere('service.port = :port', { port: filters.port });
  }
  
  const [services, total] = await query.getManyAndCount();
  
  res.json({
    services,
    pagination: {
      page,
      pageSize,
      total,
      pages: Math.ceil(total / pageSize)
    }
  });
};

This pagination strategy keeps the API responsive even as scan data accumulates over months. Organizations running Crossfeed for a year might have millions of historical service records, and the interface remains snappy because it never attempts to load everything at once.

Gotcha

The AWS dependency runs deeper than most open-source projects. Crossfeed doesn't just prefer AWS—it assumes Lambda for compute, SQS for messaging, RDS for databases, and S3 for artifact storage. There's no abstraction layer that would make switching to GCP Cloud Run or Azure Container Instances straightforward. If you're in a multi-cloud environment or have regulatory requirements for on-premise deployment, expect substantial reengineering. The Terraform configuration in the repository is tightly coupled to AWS-specific resources, and the application code directly imports AWS SDK clients rather than using provider-agnostic interfaces.

Active scanning introduces legal and operational risks that the documentation doesn't emphasize enough. When you configure Crossfeed to run nmap port scans or Nuclei vulnerability checks against your infrastructure, you're generating network traffic that intrusion detection systems will flag. If you misconfigure the target scope and accidentally scan third-party services or cloud provider infrastructure, you could violate terms of service or even face legal issues. The tool doesn't include circuit breakers or rate limiting by default—it will happily fire off 10,000 concurrent scans if you tell it to. Organizations need mature change management and scanning approval workflows before deploying Crossfeed in production. The "move fast and scan everything" approach can quickly become "explain to legal why we're banned from AWS Marketplace."

Verdict

Use Crossfeed if you're managing external attack surface for multiple organizations or business units that need self-service scanning capabilities, you're already standardized on AWS infrastructure, and you have the engineering resources to operate and customize a complex distributed system. It shines for federal agencies, large enterprises with dedicated security engineering teams, and managed security service providers who need to offer continuous monitoring to clients. The multi-tenant architecture and historical tracking make it ideal for demonstrating security posture improvements over time. Skip it if you're a small team looking for quick point-in-time assessments (just run the individual tools directly), you need multi-cloud or on-premise support, you lack experience operating serverless applications at scale, or you want a commercial product with vendor support and SLAs. The operational overhead of managing Lambda functions, SQS queues, RDS databases, and scan workers is substantial—only invest that effort if continuous automated monitoring delivers proportional value for your organization.