AutoHarness: Teaching Claude Code to Remember What It Learned Yesterday
Hook
Every time Claude Code figures out your deployment script or testing pattern, that knowledge evaporates when the session ends. AutoHarness treats your coding sessions as training data, automatically extracting and maintaining a self-pruning library of executable skills.
Context
Claude Code's skill system lets you define reusable patterns as MCP tools—write a skill once, and Claude can invoke it across sessions. The problem is authorship: someone has to notice recurring patterns, distill them into skills, and maintain the library as projects evolve. Most teams never bother, letting hard-won knowledge like 'how we run integration tests' or 'our deployment checklist' exist only in chat history.
AutoHarness flips the script by treating skill creation as a byproduct of normal work. Every 50 tool calls during a coding session triggers a background reflection where a separate Claude instance analyzes the recent conversation window, decides whether it contains a reusable pattern, and proposes a new skill or update to an existing one. Skills that stop getting used get archived. Near-duplicates get merged. The system runs entirely as hooks into Claude Code's lifecycle—no daemon process, no separate evaluation loop, just filesystem operations and subagent calls.
Technical Insight
The architecture is surprisingly simple: Python hooks that intercept Claude Code at session start, every N tool calls, and session end. The capture hook maintains a redacted conversation window (personally identifiable info stripped) and counts tool invocations. When the count hits 50, it bundles the window into a fresh Claude API call with a reflection prompt asking 'does this episode warrant creating or updating a skill?'
Here's the core of the reflection trigger:
def on_tool_call(tool_name: str, args: dict, result: any):
window.append({"tool": tool_name, "args": redact(args), "result": redact(result)})
state.tool_count += 1
if state.tool_count % 50 == 0:
candidate = reflect_on_window(
window=window[-100:], # Last 100 events
existing_skills=load_skills(),
mode="bundle" # Fresh subagent, not forked context
)
if candidate and validate_structure(candidate):
atomic_write(f".claude/skills/{candidate.name}.md", candidate.body)
append_ledger(candidate.name, "created", evidence=window[-20:])
The reflection runs in a stateless subagent rather than forking the parent session. This trades potential cache hits (a forked session could reuse embeddings and context) for determinism—the same conversation window always produces the same reflection opportunity, regardless of what happened earlier in the parent session. The validation step enforces a 25-line body cap to prevent the subagent from dumping entire transcripts as 'skills.'
Lifecycle management happens at session start, not on a schedule. When you launch Claude Code, AutoHarness scans its ledgers to compute usage rates for every skill it manages. The critical insight: usage rate is loads-per-request-opportunity, not loads-per-day. A skill that fired 5 times out of 1000 total tool calls has a 0.5% rate, regardless of whether those 1000 calls happened in one afternoon or three months. This makes the system laptop-close-resistant—your skill library doesn't decay just because you took a vacation.
def compute_usage_rate(skill_name: str) -> float:
ledger = read_ledger(skill_name)
loads = sum(1 for event in ledger if event.type == "load")
opportunities = state.total_tool_calls_since_graduation
return loads / opportunities if opportunities >= 100 else None # Still in probation
Skills graduate from probation after the project accumulates 100 tool calls. Once mature, they face capacity contention: if the project skill pool exceeds 50 entries (or the global pool hits 20), the lowest-usage-rate skills get archived to .claude/archive/ with a snapshot of their final state. The archive isn't a soft delete—AutoHarness never touches those files again unless you manually restore them.
The session-start index injection is architecturally clever. Instead of hoping Claude notices your skills organically, AutoHarness prepends every conversation with a categorized table of contents:
## AutoHarness Skills Available
### Testing (3 skills)
- `run_integration_suite`: Executes full test suite with Docker dependencies
- `check_coverage`: Generates and validates coverage report against 80% threshold
- `run_affected_tests`: Uses git diff to run only tests for changed files
### Deployment (2 skills)
- `deploy_staging`: Builds, tags, and pushes to staging with health checks
- `rollback_deployment`: Reverts to previous Docker image and restarts services
This burns context on every session regardless of whether skills fire, but it upgrades Claude's skill awareness from 'probabilistic recall based on semantic similarity' to 'explicit menu presented upfront.' The index is dynamically generated from skill frontmatter, so categorization happens at skill-creation time during reflection.
Provenance tracking is ledger-based: every skill gets an append-only log of creation, loads, updates, and archival decisions, with snapshots of the conversation windows that triggered each event. This builds the raw material for post-hoc benchmarking—you could theoretically replay archived sessions with and without skills to measure impact—but AutoHarness ships with zero evaluation tooling. The ledgers are just text files:
2024-01-15T10:23:11Z CREATE evidence_hash=a3f2b1c reason="Recurring pattern: Docker Compose health checks before deployment"
2024-01-18T14:05:33Z LOAD context="User requested staging deployment"
2024-01-22T09:12:44Z UPDATE reason="Added rollback step" evidence_hash=d4e5f6a
The curator runs every 250 tool calls to find near-duplicates across the entire library. The README doesn't specify the algorithm—could be embedding cosine similarity, keyword Jaccard index, or LLM-based comparison—but merges are irreversible except for the pre-merge snapshot. This is a sharp edge: a bad consolidation pass could collapse genuinely distinct skills, and the only recovery is manual restoration from snapshots.
Gotcha
The biggest limitation is unmeasured impact. AutoHarness assumes that having more skills automatically improves outcomes, but provides no evidence that learned skills actually reduce tokens, speed up tasks, or improve success rates. The README cites 42%→78% on CORE-Bench from HAL's harness, but that's a different system with human-curated skills and an evaluation loop. Without A/B testing—and the daemon-free architecture makes that impossible—you're trusting that automatic distillation produces quality comparable to manual curation.
The usage-rate eviction policy has a fatal flaw for low-activity projects: skills must survive 100 project-level tool calls to graduate from probation, but once graduated, they're immortal until the pool hits 50 entries. If your project sees 10 tool calls per week, skills created in month one will still be in probation a year later, never facing capacity pressure. Meanwhile, high-activity projects churn through the 50-skill cap quickly, potentially evicting valuable niche skills (like 'handle rate limit in API client') that have low load counts but high value when they do fire.
Global skills create tight coupling. A skill learned from Project A's deployment process lands in the global pool and fires for Project B, potentially with wrong assumptions about directory structure or tooling. The 20-skill global cap means this pool fills fast, and eviction is purely usage-rate-driven—a skill that's critical for one project but rarely used elsewhere gets archived, breaking the original project's workflow.
Finally, reflection quality is a black box. The subagent gets a redacted conversation window and a prompt, but there's no validation that proposed skills encode reusable patterns rather than session-specific transcripts. The 25-line body cap rejects obvious dumps, but can't catch subtler overfitting—like a skill that hardcodes a Git branch name or assumes a specific database schema that only existed during the learning session.
Verdict
Use AutoHarness if you're doing iterative coding sessions in Claude Code (not just Q&A), work on projects with recurring patterns like deployment scripts or test workflows, and trust LLM-generated code enough to let it persist automatically without approval gates. The value proposition is strongest for teams scaling Claude Code across multiple developers—institutional memory accumulates organically instead of living in one person's head. Skip it if you need hermetic builds (skills evolve based on session history, making outputs non-deterministic), work in regulated environments requiring human review before code persistence, or primarily use Claude for stateless conversations that don't generate tool calls. Also skip if you need proof that skills improve measurable outcomes—without benchmarks or A/B testing, you're betting that automatic distillation beats manual curation, a bet that only pays off if your workflow involves enough repeated patterns that recall probability outweighs individual skill quality.