Building a JavaScript Surveillance System: Inside FileChangeMonitor's Architecture

Hook

Every time a company pushes updated JavaScript to production, they're inadvertently publishing their API roadmap. FileChangeMonitor turns this reality into actionable intelligence.

Context

Modern web applications are JavaScript-heavy beasts. Single-page applications bundle their routing logic, API endpoints, and business rules directly into client-side code that anyone can download. When developers push updates, they often expose new features, API endpoints, or security changes before official announcements. Security researchers need to track these changes to discover vulnerabilities. Competitive analysts want to know when rivals ship new features. Integration developers must stay current with third-party APIs they depend on.

Manually tracking JavaScript changes across multiple websites is tedious and error-prone. You'd need to periodically download files, run diffs, parse the results, and somehow handle versioned filenames like 'app.a3f82b9c.js' that change with every build. FileChangeMonitor automates this entire pipeline by combining web scraping, intelligent file matching, cloud storage, and email notifications into a single system. Built by cablej with 222 GitHub stars, it demonstrates how to architect a practical surveillance tool using Node.js, MongoDB, AWS S3, and SendGrid.

Technical Insight

FileChangeMonitor's architecture follows a classic scraper-differ-notifier pattern, but its implementation reveals several clever design decisions. At its core, the system maintains a MongoDB collection of target domains and periodically fetches JavaScript files from each. Rather than requiring users to specify exact file URLs, it scrapes the domain to auto-discover linked JavaScript files, lowering the barrier to monitoring new targets.

The most interesting architectural challenge is handling dynamically-named files. Modern build tools like webpack generate filenames with content hashes: 'bundle.a3f82b9c.js' becomes 'bundle.f9d12e4a.js' after any code change. Naive URL-based monitoring would see these as entirely different files. FileChangeMonitor solves this by storing file metadata that allows matching across filename changes. When comparing files, it looks beyond just the URL to identify conceptually identical files that have been renamed.

Here's a simplified example of how the file comparison logic might work:

const compareFiles = async (domain, newFiles) => {
  const storedFiles = await db.collection('files')
    .find({ domain: domain.url })
    .toArray();
  
  for (const newFile of newFiles) {
    // Match by URL first
    let match = storedFiles.find(f => f.url === newFile.url);
    
    // If no URL match, try pattern matching for versioned files
    if (!match) {
      const basePattern = newFile.url.replace(/\.[a-f0-9]{8}\./, '.*.') 
      match = storedFiles.find(f => 
        new RegExp(basePattern).test(f.url)
      );
    }
    
    if (match) {
      // Download from S3 and compare content
      const oldContent = await s3.getObject({
        Bucket: 'filechangemonitor',
        Key: match.s3Key
      }).promise();
      
      if (oldContent.Body.toString() !== newFile.content) {
        await detectEndpointChanges(oldContent.Body, newFile.content);
      }
    }
  }
};

The S3 integration is particularly smart. Rather than storing entire file contents in MongoDB (which has a 16MB document limit and would balloon database size), FileChangeMonitor uploads each file version to S3 and stores only metadata and S3 keys in MongoDB. This separation allows unlimited file sizes and keeps the database lean for fast queries.

The notification system focuses specifically on API endpoint detection. When diffing files, the system doesn't just report 'file changed' – it parses JavaScript looking for new URL patterns that suggest API endpoints. This filtering dramatically reduces noise. Instead of getting alerted about every minification difference, you're notified when new routes like '/api/v2/payments' appear in the code:

const detectEndpointChanges = (oldContent, newContent) => {
  const urlPattern = /['"`](\/api\/[a-z0-9\/\-_]+)['"`]/gi;
  
  const oldEndpoints = new Set(
    (oldContent.match(urlPattern) || []).map(s => s.slice(1, -1))
  );
  const newEndpoints = new Set(
    (newContent.match(urlPattern) || []).map(s => s.slice(1, -1))
  );
  
  const added = [...newEndpoints].filter(e => !oldEndpoints.has(e));
  
  if (added.length > 0) {
    sendEmail({
      subject: 'New API endpoints detected',
      endpoints: added
    });
  }
};

The web UI uses Bower for front-end dependency management, which reveals the codebase's age – Bower was deprecated in 2017 in favor of npm and yarn. The authentication system relies on JWT tokens, a solid choice that allows stateless API authentication. Users can add domains through the UI, which triggers the scraping pipeline.

Scheduling is handled through a cron-like system that periodically triggers the scraping job for all monitored domains. The interval configuration allows balancing between freshness of detection and server load. For high-value targets, you might scrape hourly; for less critical ones, daily checks suffice.

Gotcha

FileChangeMonitor's reliance on multiple external services creates operational complexity that shouldn't be underestimated. Self-hosting requires standing up MongoDB, configuring AWS S3 with proper IAM credentials, and setting up SendGrid for email delivery. Each service adds configuration overhead, potential failure points, and recurring costs. The MongoDB dependency means you can't simply deploy to serverless platforms without additional database hosting.

The codebase uses Bower, a package manager deprecated since 2017, which signals that this project may not follow modern JavaScript practices. You'll likely encounter outdated dependencies with known vulnerabilities, and the code probably doesn't use ES6+ features or modern frameworks. While the core scraping logic remains valid, integrating this into a modern development workflow would require significant refactoring. The project also monitors only JavaScript files – if you need comprehensive website change detection including CSS, HTML, or image assets, you'll need to extend the codebase significantly or look elsewhere.

Verdict

Use FileChangeMonitor if you're conducting security research or competitive analysis that requires tracking JavaScript changes across multiple domains, especially when those files use versioned filenames. The hosted version at filechangemonitor.io is worth trying before self-hosting, as it eliminates operational overhead. The focused approach of detecting API endpoint additions makes it particularly valuable for security researchers looking for newly exposed attack surfaces or developers tracking third-party services they integrate with. Skip it if you need to monitor only a few static files (simple diff tools or GitHub notifications suffice), require modern tooling and active maintenance, need comprehensive website monitoring beyond JavaScript, or want to avoid managing MongoDB and S3 infrastructure. The 222 stars and deprecated dependencies suggest this is more of a proof-of-concept than a production-ready tool, so evaluate whether the architectural patterns are worth extracting into your own implementation rather than running this code directly.