Project Wycheproof: How Google’s Test Vectors Catch Crypto Bugs Before Hackers Do
Hook
When AWS-LC, OpenSSL, and Go’s crypto library all test against the same attack vectors, those vectors matter. Project Wycheproof has become the cryptographic testing standard you’ve probably never heard of—but definitely depend on.
Context
Cryptographic implementations fail in spectacularly subtle ways. You can implement AES or RSA exactly according to specification and still create exploitable vulnerabilities through timing channels, padding oracle attacks, or edge cases the spec never explicitly addressed. The 1998 Bleichenbacher attack against RSA-PKCS#1 v1.5 worked perfectly against correct implementations—the problem wasn’t algorithm choice, but how libraries handled malformed padding.
Google created Project Wycheproof in 2016 to systematize testing against known cryptographic attacks. Named after the lowest mountain in the world (Mount Wycheproof in Australia, at 43 meters), the project embodies its philosophy: if you can’t pass the lowest bar, you’ve got serious problems. Rather than proving cryptographic strength, Wycheproof tests whether implementations fall to already-published attacks. It started as language-specific test harnesses but evolved into pure JSON test vectors that crypto libraries integrate into their own test suites. Today it’s managed by C2SP (Crunchy Cryptography Specifications Project) and represents critical infrastructure—when OpenSSL, BoringSSL, AWS-LC, NSS, and RustCrypto all test against the same vectors, those tests protect a massive swath of internet traffic.
Technical Insight
Wycheproof’s architecture is deliberately minimal: JSON files containing inputs, expected outputs, and flags describing what each test vector targets. Each cryptographic algorithm gets a dedicated JSON file with dozens to thousands of test cases. The genius is in what those test cases encode—not random inputs, but carefully crafted attack attempts.
Consider how it tests ECDSA signature verification. A naive implementation might only test valid signatures with standard test vectors. Wycheproof includes vectors for invalid curve attacks, where an attacker provides a public key that doesn’t actually lie on the specified elliptic curve. Here’s what integration looks like in Go:
package crypto_test
import (
"crypto/ecdsa"
"encoding/json"
"os"
"testing"
)
type WycheproofTest struct {
TcID int `json:"tcId"`
Comment string `json:"comment"`
Msg string `json:"msg"`
Sig string `json:"sig"`
Result string `json:"result"` // "valid", "invalid", or "acceptable"
Flags []string `json:"flags"`
}
func TestECDSAWycheproof(t *testing.T) {
data, err := os.ReadFile("testvectors/ecdsa_secp256r1_sha256_test.json")
if err != nil {
t.Fatal(err)
}
var suite struct {
TestGroups []struct {
Tests []WycheproofTest `json:"tests"`
} `json:"testGroups"`
}
json.Unmarshal(data, &suite)
for _, group := range suite.TestGroups {
for _, tc := range group.Tests {
result := verifySignature(tc.Msg, tc.Sig)
switch tc.Result {
case "invalid":
if result {
t.Errorf("Test %d: accepted invalid signature (%s): %s",
tc.TcID, tc.Comment, tc.Flags)
}
case "valid":
if !result {
t.Errorf("Test %d: rejected valid signature", tc.TcID)
}
}
}
}
}The flags field is particularly powerful—it categorizes what attack or edge case each vector represents. Flags like "EdgeCase", "InvalidCurve", "SmallResidue", or "ModifiedSignature" let you track exactly which attack types your implementation resists. When a test fails, you know precisely what vulnerability you have.
The shift from language-specific harnesses to pure data was strategic. Initially, Wycheproof maintained Java, Python, and Go test harnesses. But maintaining multiple implementations meant bugs could hide in the test code itself, and library authors had to wait for Wycheproof updates. By providing only JSON vectors with accompanying JSON schemas, the project distributed maintenance burden while enabling crypto libraries to test themselves continuously during development.
JSON schemas serve as contracts for vector format. They define required fields, value types, and valid enumerations. This enables automated tooling—validators that check new vectors conform to schema, code generators that produce type-safe parsers, and documentation generators that explain what each field means. The schema for ECDSA vectors specifies that signatures must be hex-encoded strings, results must be one of three enum values, and certain flag combinations are semantically meaningful.
For post-quantum algorithms, Wycheproof has expanded to cover ML-KEM (the standardized version of Kyber) and ML-DSA (Dilithium). These vectors test decapsulation failures, malformed ciphertexts, and the subtle constant-time requirements that prevent timing side-channels. As organizations implement NIST’s post-quantum standards, having battle-tested vectors prevents repeating the mistakes made with classical cryptography.
The test vector coverage spans over 80 specific test cases across 30+ algorithms: ECDSA with multiple curves, RSA signatures (PKCS#1 v1.5, PSS), RSA encryption (OAEP, PKCS#1 v1.5), AES-GCM, AES-EAX, ChaCha20-Poly1305, HKDF, X25519, Ed25519, and more. Each test case targets known attacks—Bleichenbacher’s million message attack against RSA, Small subgroup attacks on Diffie-Hellman, padding oracle attacks on CBC mode, and timing attacks through biased nonces in DSA.
Gotcha
Wycheproof doesn’t provide test harnesses anymore—you must write integration code yourself. For large crypto libraries with existing test infrastructure, this is trivial. For smaller projects or developers unfamiliar with test frameworks, it creates an initial barrier. The JSON parsing, test result interpretation, and CI integration all fall on you. This is intentional design, but it means Wycheproof isn’t a download-and-run solution.
Documentation quality varies significantly. Some algorithms have complete JSON schemas with detailed field descriptions. Others (AES-FF1, certain RSA-PSS variants, PBKDF2) lack schemas entirely, forcing you to reverse-engineer format from examples. The project explicitly warns that documentation in the doc/ directory may be outdated and recommends preferring schemas, but this creates confusion for newcomers trying to understand what specific flags mean or how to interpret “acceptable” versus “valid” results. The “acceptable” result category covers implementation-defined behavior—like whether to accept signatures with non-minimal DER encoding—which requires understanding cryptographic nuance to handle correctly. Finally, Wycheproof tests known attacks, not comprehensive correctness. A library can pass all Wycheproof tests and still have bugs in untested code paths or novel vulnerabilities. It’s necessary but not sufficient validation.
Verdict
Use if: You’re developing or maintaining a cryptographic library and need comprehensive attack vectors covering known vulnerabilities. Wycheproof is industry standard—if major projects like OpenSSL, AWS-LC, and Go crypto depend on it, you should too. The integration effort pays for itself the first time it catches a padding oracle or invalid curve bug. Also use it if you’re implementing post-quantum algorithms and want existing test coverage for ML-KEM or ML-DSA. Skip if: You’re building applications that consume crypto libraries rather than implementing primitives—just use a well-tested library that already integrates Wycheproof. Also skip if you need official NIST validation for compliance purposes (use CAVP instead) or if you want ready-made test harnesses without writing integration code. For research into novel attacks or cutting-edge algorithms without existing vectors, Wycheproof won’t have coverage yet—you’re on your own.
