SecureForge: MCMC Prompt Exploration for Finding Adversarial Boundaries in Code LLMs

Hook

Most adversarial testing for code generation models asks 'does this prompt produce vulnerable code?' SecureForge asks a far more interesting question: 'what's the smallest change to this prompt that flips the model from safe to unsafe?'

Context

As code generation models like Copilot and CodeLlama move from novelty to production tooling, a critical question emerges: how do we systematically test whether these models can be tricked into generating vulnerable code? Traditional benchmarks like HumanEval measure functional correctness, but say nothing about security. Early security work (Pearce et al., 2021) demonstrated that prompted models could generate CWE-specific vulnerabilities, but relied on hand-crafted scenarios that don't scale.

The deeper problem is one of search topology. The space of possible prompts is infinite, and vulnerabilities hide in unexpected corners. Random fuzzing wastes compute on obviously safe or obviously dangerous prompts. Gradient-based attacks require white-box access that production APIs don't provide. What's missing is a principled exploration strategy that can walk the boundary between safe and unsafe prompts—finding the subtle phrasings where models fail. SecureForge implements this using MCMC: treating an LLM as a Markov kernel that proposes prompt variations, guided by static analysis oracles that judge the resulting code.

Technical Insight

SecureForge's architecture separates trust domains: a 'trusted' model generates test scenarios and evaluation code, while the 'model under test' only produces code rollouts. This two-model pattern prevents the circular logic where a model evaluates its own outputs. The workflow splits into three commands: generate, amplify, and propose.

The generate stage bootstraps from Pearce 2021's hand-written CWE prompts using few-shot learning. It samples the code model at temperature 0.8 for diversity, then passes outputs to Semgrep or CodeQL as external oracles:

# Simplified generate flow
for cwe_category in ['CWE-78', 'CWE-79', 'CWE-89']:  # Command injection, XSS, SQL injection
    few_shot_examples = load_pearce_prompts(cwe_category)
    
    # Trusted model generates new scenarios
    scenario_prompt = build_few_shot(few_shot_examples, count=5)
    new_scenarios = trusted_model.generate(scenario_prompt, n=10)
    
    for scenario in new_scenarios:
        # Model under test generates code
        rollouts = code_model.generate(scenario, temperature=0.8, n=20)
        
        # External oracle judges vulnerability
        for code in rollouts:
            findings = semgrep.scan(code, rules=cwe_ruleset)
            results.append({
                'scenario': scenario,
                'code': code,
                'vulnerable': len(findings) > 0,
                'cwe': findings[0].rule_id if findings else None
            })

The truly novel piece is the amplify command, which implements MCMC exploration of prompt space. Starting from a seed prompt that produces vulnerable code, it uses the LLM itself as a proposal distribution—asking it to rephrase the prompt while preserving intent. The Metropolis-Hastings acceptance criterion is binary: if the rephrased prompt still triggers vulnerabilities (according to the static analysis oracle), accept the move; otherwise, reject and stay at the current prompt. This walks the decision boundary:

def amplify_mcmc(seed_prompt, code_model, oracle, steps=100):
    current_prompt = seed_prompt
    chain = [current_prompt]
    
    for _ in range(steps):
        # LLM proposes variation
        rephrase_prompt = f"""Rephrase this coding task preserving the core intent:
        
        Original: {current_prompt}
        
        Rephrased:"""
        proposed = trusted_model.generate(rephrase_prompt, temperature=0.7)
        
        # Test if proposed prompt still triggers vulnerability
        code = code_model.generate(proposed, temperature=0.8)
        findings = oracle.scan(code)
        
        if len(findings) > 0:  # Still vulnerable - accept move
            current_prompt = proposed
            chain.append(proposed)
        # else: reject, stay at current_prompt
    
    return chain  # Collection of prompts near decision boundary

This discovers subtle variations that maintain adversarial properties—for example, finding that 'sanitize user input' in a prompt prevents SQL injection, but 'clean user input' doesn't, because the model associates 'sanitize' with security libraries but treats 'clean' as cosmetic string manipulation.

The propose command targets safety fine-tuning workflows. It generates paired examples: a nominal prompt that should produce safe code, and a perturbed version designed to trigger failures. Critically, it uses Beta distribution variance as a stopping criterion. Each time you sample the model, you update a Beta(α, β) posterior where α counts safe outputs and β counts vulnerable ones. When the variance drops below a threshold, you've collected enough samples to reliably estimate the model's failure probability:

from scipy.stats import beta

def adaptive_sampling(prompt, model, oracle, variance_threshold=0.01):
    alpha, beta_param = 1, 1  # Uniform prior
    
    while beta.var(alpha, beta_param) > variance_threshold:
        code = model.generate(prompt)
        is_vulnerable = len(oracle.scan(code)) > 0
        
        if is_vulnerable:
            beta_param += 1
        else:
            alpha += 1
    
    failure_rate = beta_param / (alpha + beta_param)
    return failure_rate, alpha + beta_param  # Rate and confidence

This adaptive approach is statistically principled—models with consistent behavior need fewer samples, while models near the decision boundary (where variance stays high) get sampled more until confidence stabilizes. All commands checkpoint to JSONL, so interrupted runs resume without wasting compute.

The architecture's key insight is treating static analyzers as black-box oracles rather than building custom vulnerability detectors. This makes SecureForge model-agnostic and leverages battle-tested tools like Semgrep's 2000+ security rules. The two-model trust separation ensures you're not asking the fox to guard the henhouse.

Gotcha

SecureForge inherits all the false positive and false negative rates from its static analysis backends. Semgrep and CodeQL are industry-standard tools, but they miss vulnerabilities that require runtime context (like authentication state or database schema constraints), and flag secure code that uses patterns resembling vulnerabilities. There's no mechanism to validate oracle reliability or handle ambiguous cases—if Semgrep says code is vulnerable, SecureForge treats that as ground truth.

The MCMC rephrasing kernel has no convergence guarantees. In theory, it should explore prompt space and find decision boundaries. In practice, it might get stuck rephrasing the same concept repeatedly ('validate input' → 'check input' → 'verify input') without discovering fundamentally different attack vectors. The paper provides no guidance on mixing time, burn-in periods, or chain diagnostics like Gelman-Rubin statistics that would tell you if your exploration is actually working. Temperature 0.7 for rephrasing and 0.8 for code generation are hardcoded magic numbers with no ablation studies.

Cost scaling is linear with API calls: (scenarios × rollouts × MCMC steps). A thorough CWE-89 evaluation might need 50 scenarios × 20 rollouts × 100 MCMC steps × $0.002 per call = $200 for a single vulnerability class. There's no batching, embedding-based caching of similar prompts, or distillation to cheaper models. For comprehensive sweeps across model checkpoints during fine-tuning, this gets expensive fast.

Verdict

Use if: You're doing offensive security research on code LLMs and need to discover novel failure modes rather than just run fixed benchmarks. The MCMC boundary exploration finds subtle prompt variations that standard fuzzing misses, making it valuable for red-teaming before deployment. Also use if you're safety fine-tuning instruction models and need contrastive examples—the propose command's paired nominal/adversarial prompts with adaptive sampling gives you high-quality training data with confidence estimates. Perfect for research teams with API budgets who need CWE-specific scenarios.

Skip if: You need dynamic runtime vulnerability detection (SecureForge only does static analysis), non-Python language support, or production-scale evaluation on a budget. The linear API cost scaling makes comprehensive sweeps expensive. Also skip if you need provable coverage guarantees or can't tolerate static analysis false positives—the oracle reliability problem is unaddressed. For standardized leaderboards, CyberSecEval's fixed benchmarks are more comparable across papers. For traditional software fuzzing of specific functions, Atheris or Hypothesis are cheaper and faster.