Violin: Making LLM-Driven Penetration Testing Auditable Through State Machines

Hook

Your LLM-powered security agent just ran 300 commands overnight, then lost its entire conversation history to context window limits. Can you prove what it tested, explain its reasoning, or even resume where it left off?

Context

The promise of LLM-driven penetration testing is seductive: natural language tasking, automated hypothesis generation, and 24/7 security assessments without burning out your red team. Tools like PentestGPT emerged quickly, wrapping ChatGPT with system prompts that encourage security thinking. But prompt engineering hits a wall when engagements span weeks instead of hours.

The core problem isn't model capability—it's architectural. Real penetration tests generate thousands of commands across reconnaissance, vulnerability research, and exploitation phases. Each command produces logs, screenshots, and packet captures that inform the next hypothesis. When your agent's context window fills up after 200 interactions, you lose the entire decision tree. Worse, without explicit state management, there's no audit trail explaining why the agent tested certain endpoints, no way to resume after a crash, and no mechanism preventing an LLM hallucination from running destructive commands against production systems. Violin tackles these problems not through better prompts, but through a state-machine architecture that externalizes engagement state to the filesystem and enforces phase-based execution gates.

Technical Insight

Violin implements penetration testing as a finite state machine with six phases: SCOPING → RECON → VULN_RESEARCH → EXPLOITATION → REPORTING → RETROSPECTIVE. Each phase transition requires creating a PTT (Pentest Task Tracker) entry—a markdown file that records the hypothesis, required skills, and approval status. This isn't documentation theater; the phase gates are enforced at the execution boundary:

# From violin_guard plugin - simplified for clarity
def validate_execution(command: str, engagement_state: dict) -> ValidationResult:
    current_phase = engagement_state['phase']
    active_tasks = load_ptt_files(engagement_state['eng_dir'])
    
    # Block execution if no active task in current phase
    if not any(task['phase'] == current_phase and task['status'] == 'ACTIVE' 
               for task in active_tasks):
        return ValidationResult(
            allowed=False,
            reason=f"No active PTT task for {current_phase} phase"
        )
    
    # Validate command against scope definition
    scope = load_scope_yaml(engagement_state['eng_dir'])
    if not command_matches_scope(command, scope):
        return ValidationResult(
            allowed=False,
            reason="Command targets out-of-scope assets"
        )
    
    return ValidationResult(allowed=True)

The genius is in what gets persisted. Instead of trying to keep everything in the LLM's context window, Violin writes checkpoint.json files that capture phase state, skill bindings, and command history. The PTT files become the durable decision log—each entry explains what you're testing and why, with explicit skill declarations that trigger lazy-loading of specialist playbooks.

Skill lazy-loading solves a critical token economy problem. Violin ships with 35 playbooks covering web application testing, API security, identity/auth exploitation, and even LLM-specific attacks. Loading all of them upfront would consume 50,000+ tokens before the agent runs a single command. Instead, the violin_record_ptt tool implements a two-phase commit:

# Example PTT file: tasks/002-api-enumeration.md
---
phase: RECON
status: ACTIVE
skills: [api-recon, openapi-analysis]
hypothesis: Target exposes REST API at /api/v1 based on JS bundle analysis
---

## Approach
1. Enumerate API endpoints using discovered OpenAPI spec
2. Test authentication requirements per endpoint
3. Map parameter schemas and data types

## Evidence Trail
- evidence/recon/openapi-spec.json
- evidence/recon/endpoint-matrix.csv

When the agent declares skills: [api-recon], Violin doesn't immediately inject the playbook content. It returns a skill_prepared status and waits for explicit confirmation through the Hermes content delivery mechanism. This prevents phantom skill bindings where marker files claim capabilities that were never actually loaded into context.

The execution model provides four boundaries: single commands (immediate execution with output capture), burst mode (multiple related commands with aggregated output), background processes (long-running tools like network scanners), and batch review (queued commands that require explicit approval). All four share the same validation pipeline and write to the same evidence directory structure:

$ENG_DIR/
├── scope.yaml              # Target definitions, IP ranges, exclusions
├── checkpoint.json         # Current phase, skill state, command counter
├── state/                  # Engagement metadata
│   └── session.log        # Command history (secrets excluded)
├── tasks/                  # PTT markdown files
│   ├── 001-initial-recon.md
│   └── 002-api-enumeration.md
├── evidence/
│   ├── recon/             # Phase-segregated raw artifacts
│   ├── vuln_research/
│   └── exploitation/
└── reporting/
    ├── findings/           # FIND-001.md format with proof
    └── final-report.md

Evidence segregation matters for compliance. Raw artifacts (nmap scans, Burp logs, screenshots) go in evidence// directories. Reportable findings with sanitized proof go in reporting/findings/. The state/ directory explicitly excludes secrets through .gitignore patterns, preventing credential leakage in version-controlled engagement files.

The guarded execution boundary is what separates Violin from a shell script that logs commands. Every execution flows through violin_exec, which validates scope boundaries, checks phase alignment, verifies hypothesis linkage to active PTT tasks, and inspects command history for duplicates or dangerous patterns. This creates a typed interface around arbitrary subprocess invocations—you get access control without writing tool-specific wrappers for nmap, sqlmap, and 50 other security utilities.

Gotcha

Violin's architecture assumes you trust the host environment. There's no network-level containment, no cgroups limiting resource consumption, and the 'raw-terminal hook' fallback is explicitly documented as 'best-effort.' If the LLM generates a command that the guards don't catch, it runs with your user's full permissions. The tool prevents accidental scope violations and phase-skipping, but won't stop a determined jailbreak attempt or logic bug from executing unauthorized operations.

State persistence through JSON and markdown files provides no transactional guarantees. If Hermes crashes mid-batch or two sessions run concurrently (nothing prevents this), you can corrupt engagement state with no rollback mechanism. The checkpoint.json format has no schema versioning, so upgrading Violin mid-engagement could break state deserialization. For regulated environments requiring tamper-evident audit logs, you'll need to wrap Violin's filesystem operations in your own transaction layer or use filesystem snapshots as poor-man's commits.

The Hermes Agent dependency is both advantage and limitation. You get native tool registration and typed message passing, but you're locked into Nous Research's agent framework with its small ecosystem and uncertain long-term support. Violin can't easily port to AutoGPT, LangChain, or other agent frameworks because the plugin architecture, content delivery, and tool registry are Hermes-specific. If Hermes development stalls or you need to integrate with existing agent infrastructure, you're rewriting significant portions of the codebase.

Verdict

Use Violin if you're running multi-week penetration tests where context window exhaustion is killing your LLM-driven automation, you need auditable evidence trails that satisfy compliance requirements for financial or healthcare clients, you already have Hermes Agent infrastructure deployed and want native integration rather than API-wrapped tools, or you're tired of autonomous agents skipping reconnaissance to jump straight to exploitation without methodological discipline. Skip Violin if you need fully autonomous security scanning without human supervision (use Nuclei or OWASP ZAP instead), require runtime sandboxing and multi-tenant isolation for SaaS deployment, don't want to bet on Hermes Agent's ecosystem longevity when LangChain and AutoGPT have broader adoption, or your engagements complete in hours rather than weeks so context window persistence doesn't matter. This is a workflow harness for supervised AI-assisted testing, not a autonomous red team agent—treat it accordingly and you'll appreciate the architectural discipline it enforces.