Testing Secret Scanners With Fake Committed Credentials: A Controlled Chaos Approach

Hook

Every security tool vendor claims 99% detection accuracy for committed secrets, but when you test them against real-world repositories, the false positive rates tell a different story—and you can't use production repos to find out which tools actually work.

Context

Secret scanning has become table stakes for modern development pipelines. GitHub has its own secret scanning, GitLab offers it natively, and third-party tools like TruffleHog, GitLeaks, and git-secrets have become standard components of CI/CD security gates. But there's a chicken-and-egg problem: how do you validate that your secret scanner actually works before trusting it with your production repositories?

The traditional approach is either hoping the vendor's claims are accurate or waiting until an actual breach reveals gaps in your detection coverage. Some teams test their scanners against their own repositories, but this creates operational noise—security teams get flooded with alerts, developers receive false positive notifications, and you can't deliberately commit test secrets without triggering the very systems you're trying to validate. You need a controlled environment with known bad patterns, realistic code contexts, and zero risk of exposing actual credentials. That's the gap fake_commited_secrets fills: a deliberately vulnerable repository that lets you break things safely.

Technical Insight

The repository's architecture is deceptively simple but strategically designed. Rather than creating a single file with a list of fake API keys, it embeds secrets throughout realistic code structures across multiple programming languages. You'll find AWS credentials in Python scripts, database connection strings in configuration files, and API tokens in Swift code—exactly how developers accidentally commit them in the wild.

Here's what a typical pattern looks like in the repository:

// Example from a Swift file
let awsAccessKey = "AKIAIOSFODNN7EXAMPLE"
let awsSecretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

class AWSClient {
    func authenticate() {
        let credentials = AWSCredentials(
            accessKey: awsAccessKey,
            secretKey: awsSecretKey
        )
        // Connection logic here
    }
}

This isn't just a string literal dump—it's embedded in functional code patterns. The AWS keys follow Amazon's actual format (AKIA prefix for access keys, specific character lengths), which tests whether your scanner validates format patterns or just greps for the string "aws_access_key". Many naive implementations would miss this because there's no comment saying "this is a secret" and the variable names could plausibly be placeholders.

The real power emerges when you use this repository as a benchmark suite. Clone it locally, run your secret scanner against it, and document what gets detected versus what slips through. Here's a typical testing workflow:

# Clone the test repository
git clone https://github.com/carnal0wnage/fake_commited_secrets.git
cd fake_commited_secrets

# Test with TruffleHog
trufflehog filesystem . --json > trufflehog_results.json

# Test with GitLeaks
gitleaks detect --source . --report-path gitleaks_results.json

# Test with git-secrets
git secrets --scan

# Compare detection rates
jq '.[] | .DetectorName' trufflehog_results.json | sort | uniq -c
jq '.[] | .RuleID' gitleaks_results.json | sort | uniq -c

What makes this repository particularly valuable is its coverage of edge cases that separate good scanners from great ones. It includes secrets in binary files, base64-encoded credentials, environment variable assignments, and multi-line configuration blocks. Consider this PostgreSQL connection string pattern:

# Embedded in a Python Django settings file
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'production_db',
        'USER': 'admin',
        'PASSWORD': 'super_secret_password_123!',
        'HOST': 'prod-db.company.internal',
        'PORT': '5432',
    }
}

Simple regex-based scanners might miss this because the password isn't labeled with obvious keywords like "api_key" or "secret". Entropy-based detection should catch it (high randomness in the password string), but context-aware scanners need to understand that this is a database configuration block where the PASSWORD field matters.

The repository also exposes weaknesses in how tools handle historical commits versus current state. Some scanners only check the current HEAD, missing secrets that were committed and then removed in later commits—a common developer mistake that still leaves credentials exposed in Git history. Testing against this repository with its full commit history reveals whether your tool actually scans the entire object database or just the working tree.

For teams building custom detection rules, this repository serves as a training set. You can fork it, add your organization-specific secret patterns (like internal API token formats), and use it to validate custom regex patterns or entropy thresholds before deploying them to your actual CI/CD pipeline. This prevents the common scenario where a new rule generates thousands of false positives across your production repositories because it wasn't properly calibrated.

Gotcha

The repository's biggest limitation is staleness. It was created in 2018 and hasn't seen significant updates to cover newer authentication mechanisms. Modern cloud providers have moved toward more complex token formats—GitHub's fine-grained personal access tokens follow different patterns than the older tokens in this repository, and newer services like Vercel, Netlify, and Cloudflare Workers have their own distinctive credential formats that aren't represented here.

More importantly, the repository doesn't include a ground truth manifest—a documented list of exactly what secrets exist, where they are, and what tool should detect them. This means you're doing manual analysis to determine if a scanner missed something or if what looks like a secret is actually just a random string. For rigorous benchmarking, you'd need to audit the repository yourself and create that manifest, which defeats some of the time-saving purpose. The lack of active maintenance also means no issues or pull requests are being processed, so if you discover coverage gaps, you're on your own to fork and extend it.

Verdict

Use if: You're evaluating secret scanning tools before procurement, building a security training program that needs realistic examples of committed credentials, or setting up CI/CD pipelines and need a safe test target before deploying to production repositories. It's perfect for security engineers who need to justify tool selection with concrete detection metrics, or DevOps teams who want to validate their secret scanning configuration without triggering real security incidents. Skip if: You need comprehensive coverage of modern credential formats from services that emerged after 2018, require an actively maintained benchmark suite with documented test cases, or are building detection rules for highly specialized internal systems where generic cloud provider credentials aren't relevant. In those cases, invest time in creating your own test fixture repository with your organization's specific secret patterns and maintain it as your authentication landscape evolves.