> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

AutoHarness: Teaching Claude Code to Remember What It Learned Yesterday

[ View on GitHub ]

AutoHarness: Teaching Claude Code to Remember What It Learned Yesterday

Hook

Every time Claude Code figures out your deployment script or testing pattern, that knowledge evaporates when the session ends. AutoHarness treats your coding sessions as training data, automatically extracting and maintaining a self-pruning library of executable skills.

Context

Claude Code's skill system lets you define reusable patterns as MCP tools—write a skill once, and Claude can invoke it across sessions. The problem is authorship: someone has to notice recurring patterns, distill them into skills, and maintain the library as projects evolve. Most teams never bother, letting hard-won knowledge like 'how we run integration tests' or 'our deployment checklist' exist only in chat history.

AutoHarness flips the script by treating skill creation as a byproduct of normal work. Every 50 tool calls during a coding session triggers a background reflection where a separate Claude instance analyzes the recent conversation window, decides whether it contains a reusable pattern, and proposes a new skill or update to an existing one. Skills that stop getting used get archived. Near-duplicates get merged. The system runs entirely as hooks into Claude Code's lifecycle—no daemon process, no separate evaluation loop, just filesystem operations and subagent calls.

Technical Insight

Every tool call

Redact & append

Every 50 calls

Context

Existing skills

Skill candidate

Atomic rename

Record creation

Session start

Read & inject index

Session start

Track loads/views/patches

Archive low-usage

Every 250 calls

Merge duplicates

Record merge

Claude Code Session

Capture Hook

Redacted Window

Last 100 Events

Reflection Subagent

Validator

.claude/skills/

Session Start Hook

Lifecycle Manager

Curator

Append-Only Ledger

System architecture — auto-generated

The architecture is surprisingly simple: Python hooks that intercept Claude Code at session start, every N tool calls, and session end. The capture hook maintains a redacted conversation window (personally identifiable info stripped) and counts tool invocations. When the count hits 50, it bundles the window into a fresh Claude API call with a reflection prompt asking 'does this episode warrant creating or updating a skill?'

Here's the core of the reflection trigger:

def on_tool_call(tool_name: str, args: dict, result: any):
    window.append({"tool": tool_name, "args": redact(args), "result": redact(result)})
    state.tool_count += 1
    
    if state.tool_count % 50 == 0:
        candidate = reflect_on_window(
            window=window[-100:],  # Last 100 events
            existing_skills=load_skills(),
            mode="bundle"  # Fresh subagent, not forked context
        )
        
        if candidate and validate_structure(candidate):
            atomic_write(f".claude/skills/{candidate.name}.md", candidate.body)
            append_ledger(candidate.name, "created", evidence=window[-20:])

The reflection runs in a stateless subagent rather than forking the parent session. This trades potential cache hits (a forked session could reuse embeddings and context) for determinism—the same conversation window always produces the same reflection opportunity, regardless of what happened earlier in the parent session. The validation step enforces a 25-line body cap to prevent the subagent from dumping entire transcripts as 'skills.'

Lifecycle management happens at session start, not on a schedule. When you launch Claude Code, AutoHarness scans its ledgers to compute usage rates for every skill it manages. The critical insight: usage rate is loads-per-request-opportunity, not loads-per-day. A skill that fired 5 times out of 1000 total tool calls has a 0.5% rate, regardless of whether those 1000 calls happened in one afternoon or three months. This makes the system laptop-close-resistant—your skill library doesn't decay just because you took a vacation.

def compute_usage_rate(skill_name: str) -> float:
    ledger = read_ledger(skill_name)
    loads = sum(1 for event in ledger if event.type == "load")
    opportunities = state.total_tool_calls_since_graduation
    return loads / opportunities if opportunities >= 100 else None  # Still in probation

Skills graduate from probation after the project accumulates 100 tool calls. Once mature, they face capacity contention: if the project skill pool exceeds 50 entries (or the global pool hits 20), the lowest-usage-rate skills get archived to .claude/archive/ with a snapshot of their final state. The archive isn't a soft delete—AutoHarness never touches those files again unless you manually restore them.

The session-start index injection is architecturally clever. Instead of hoping Claude notices your skills organically, AutoHarness prepends every conversation with a categorized table of contents:

## AutoHarness Skills Available

### Testing (3 skills)
- `run_integration_suite`: Executes full test suite with Docker dependencies
- `check_coverage`: Generates and validates coverage report against 80% threshold
- `run_affected_tests`: Uses git diff to run only tests for changed files

### Deployment (2 skills)
- `deploy_staging`: Builds, tags, and pushes to staging with health checks
- `rollback_deployment`: Reverts to previous Docker image and restarts services

This burns context on every session regardless of whether skills fire, but it upgrades Claude's skill awareness from 'probabilistic recall based on semantic similarity' to 'explicit menu presented upfront.' The index is dynamically generated from skill frontmatter, so categorization happens at skill-creation time during reflection.

Provenance tracking is ledger-based: every skill gets an append-only log of creation, loads, updates, and archival decisions, with snapshots of the conversation windows that triggered each event. This builds the raw material for post-hoc benchmarking—you could theoretically replay archived sessions with and without skills to measure impact—but AutoHarness ships with zero evaluation tooling. The ledgers are just text files:

2024-01-15T10:23:11Z CREATE evidence_hash=a3f2b1c reason="Recurring pattern: Docker Compose health checks before deployment"
2024-01-18T14:05:33Z LOAD context="User requested staging deployment"
2024-01-22T09:12:44Z UPDATE reason="Added rollback step" evidence_hash=d4e5f6a

The curator runs every 250 tool calls to find near-duplicates across the entire library. The README doesn't specify the algorithm—could be embedding cosine similarity, keyword Jaccard index, or LLM-based comparison—but merges are irreversible except for the pre-merge snapshot. This is a sharp edge: a bad consolidation pass could collapse genuinely distinct skills, and the only recovery is manual restoration from snapshots.

Gotcha

The biggest limitation is unmeasured impact. AutoHarness assumes that having more skills automatically improves outcomes, but provides no evidence that learned skills actually reduce tokens, speed up tasks, or improve success rates. The README cites 42%→78% on CORE-Bench from HAL's harness, but that's a different system with human-curated skills and an evaluation loop. Without A/B testing—and the daemon-free architecture makes that impossible—you're trusting that automatic distillation produces quality comparable to manual curation.

The usage-rate eviction policy has a fatal flaw for low-activity projects: skills must survive 100 project-level tool calls to graduate from probation, but once graduated, they're immortal until the pool hits 50 entries. If your project sees 10 tool calls per week, skills created in month one will still be in probation a year later, never facing capacity pressure. Meanwhile, high-activity projects churn through the 50-skill cap quickly, potentially evicting valuable niche skills (like 'handle rate limit in API client') that have low load counts but high value when they do fire.

Global skills create tight coupling. A skill learned from Project A's deployment process lands in the global pool and fires for Project B, potentially with wrong assumptions about directory structure or tooling. The 20-skill global cap means this pool fills fast, and eviction is purely usage-rate-driven—a skill that's critical for one project but rarely used elsewhere gets archived, breaking the original project's workflow.

Finally, reflection quality is a black box. The subagent gets a redacted conversation window and a prompt, but there's no validation that proposed skills encode reusable patterns rather than session-specific transcripts. The 25-line body cap rejects obvious dumps, but can't catch subtler overfitting—like a skill that hardcodes a Git branch name or assumes a specific database schema that only existed during the learning session.

Verdict

Use AutoHarness if you're doing iterative coding sessions in Claude Code (not just Q&A), work on projects with recurring patterns like deployment scripts or test workflows, and trust LLM-generated code enough to let it persist automatically without approval gates. The value proposition is strongest for teams scaling Claude Code across multiple developers—institutional memory accumulates organically instead of living in one person's head. Skip it if you need hermetic builds (skills evolve based on session history, making outputs non-deterministic), work in regulated environments requiring human review before code persistence, or primarily use Claude for stateless conversations that don't generate tool calls. Also skip if you need proof that skills improve measurable outcomes—without benchmarks or A/B testing, you're betting that automatic distillation beats manual curation, a bet that only pays off if your workflow involves enough repeated patterns that recall probability outweighs individual skill quality.