> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

mu: A Two-Tier Memory System That Makes Coding Agents 10x Cheaper

[ View on GitHub ]

mu: A Two-Tier Memory System That Makes Coding Agents 10x Cheaper

Hook

Your coding agent just spent $3 reading a 10,000-line test failure log—twice. Only 40 lines were relevant. mu fixes this with a judge that decides what your expensive LLM actually needs to see.

Context

Coding agents have a token economics problem. When you point GPT-4 or Claude at a failing test suite, it reads everything: stack traces, passing tests, verbose logging, the works. A single Jest failure can dump 15,000 lines into context. Do this across five tool calls in one session and you've burned through 200K tokens—most of it noise. The agent re-reads the same compiler errors, re-processes identical stack frames, and still asks for permission to run the exact command you already approved three turns ago.

The standard solution is retrieval-augmented generation: chunk everything, embed it, query semantically. But RAG adds latency, requires vector infrastructure, and forces you to pre-decide chunk boundaries. It's also lossy by design—once you've embedded and discarded the original, you can't go back for the full artifact. mu takes a different approach: stream tool output through a small, fast classifier that makes binary admission decisions chunk-by-chunk. Relevant segments enter the main model's context. Everything else gets archived behind a pointer. The expensive LLM sees a curated slice. The cheap judge does triage. Built as a layer on top of pi (a TypeScript coding agent framework), mu intercepts 41 decision points across the agent loop—not just context management, but also prompt injection screening, command risk assessment, and turn orchestration.

Technical Insight

Two-Tier Memory

Judge Kernel

Intent Classification

Invokes Tools

Raw Output Stream

50-line Segments

Relevant Chunks

Irrelevant Chunks

Pointers Only

Retrieval Decisions

Relevant Items

Risk/Injection Check

Response

Drift/Completion Check

User Message

Input Preflight Judge

New Task or Refinement?

Main LLM Agent

Tool Executor

Output Chunker

Admission Judge

Active Context

Archive Storage

Context Manager

Relevance Judge

Skills & Lessons DB

Safety Judge

Turn Orchestration Judge

Response to User

System architecture — auto-generated

The architecture centers on a swappable judge kernel that operates at five interception stages. At input preflight, the judge classifies incoming messages (is this a refinement of the previous task or a completely new request?). During context management, it makes admission decisions on tool output chunks and relevance assessments for retrieving skills or lessons from memory. At safety checkpoints, it evaluates command risk (is rm -rf on these paths dangerous?) and screens for prompt injection in external content. During turn orchestration, it detects goal drift and validates whether a run actually completed the stated objective. Finally, for multi-agent workflows, it routes subtask delegation and validates subagent patches.

The killer feature is chunked output admission. Here's how it works in practice:

// Tool executor wraps command output in a streaming admission pipeline
const testOutput = await exec('npm test');
const chunks = chunkByLine(testOutput, 50); // 50-line segments
const admitted = [];
const archived = [];

for (const chunk of chunks) {
  const verdict = await judge({
    decision: 'context.tool_admit',
    chunk,
    taskContext: currentGoal,
    priorContext: conversationSummary
  });
  
  if (verdict.admit && verdict.confidence > 0.7) {
    admitted.push(chunk);
  } else {
    archived.push({ hash: sha256(chunk), location: archivePath });
  }
}

// Main LLM sees only admitted chunks plus pointers
context.toolOutput = {
  relevant: admitted.join('\n'),
  archived: archived.map(a => `[${a.hash.slice(0,8)}]: ${a.location}`)
};

This creates a two-tier memory system. The main model's context stays lean—maybe 2,000 lines instead of 15,000. But the archived chunks aren't lost. If the agent later says "show me the full output for test X," the pointer resolves to lossless retrieval. No embeddings, no semantic approximation.

Each decision point has three modes: active (verdict is enforced), shadow (logged but ignored—the agent proceeds as if the judge wasn't there), and off. Shadow mode is the pragmatic stroke of genius here. Before trusting a judge with production workloads, you run it in shadow across real tasks. The ledger captures every verdict, probability distribution, and latency:

{
  "decision": "context.tool_admit",
  "mode": "shadow",
  "verdict": false,
  "confidence": 0.83,
  "latency_ms": 47,
  "would_have_blocked": "lines 150-200 of pytest output",
  "actual_outcome": "main LLM read it anyway"
}

After a week of shadow logs, you can compare task success rates, token usage, and failure modes. Did the judge correctly identify irrelevant chunks? Did it ever block something the main LLM needed later? This observability is rare in agent frameworks—most force you to trust the abstraction or fork the code.

The judge itself defaults to Jev (a hosted TypeSafe endpoint trained on coding agent decision patterns), but the interface is pluggable. You can swap in a local model, a HuggingFace classifier, or even call another LLM if you're optimizing for accuracy over speed. The contract is simple: accept a decision type and context object, return a boolean verdict with confidence.

Beyond context management, mu addresses approval fatigue. Coding agents often pause mid-task to ask, "Should I run tests now?" even when you've already said yes to the overall goal. The turn.continue decision point detects these redundant confirmation-seeking patterns and auto-continues up to two times per message. It's heuristic—capped retries prevent infinite loops—but it solves a real UX pathology.

Prompt injection screening happens at tool boundaries. When the agent scrapes a web page, fetches from an MCP server, or runs curl, the output passes through safety.injection_detect before entering context. This is architecturally cleaner than trying to make the LLM injection-proof—you're screening untrusted content at the perimeter rather than hoping the model resists adversarial prompts embedded in API responses.

Gotcha

The entire system has a single point of failure: judge accuracy. If the classifier's false negative rate exceeds 5% on context admission, you're silently starving the main LLM of information it needs. There's no automated ground truth collection. The ledger shows you what the judge decided, but not whether it was correct. You're flying blind unless you manually audit verdicts against task outcomes.

Chunked admission also has a fundamental blindness problem. The judge assesses each chunk without full conversation history—by design, because passing all context would negate the efficiency gains. But relevance is contextual. A log line that seems irrelevant at turn three might become critical at turn ten when the agent realizes the earlier failure was related. There's no mechanism to say, "I was wrong about that archived chunk; retrieve it now." You'd need to re-run the tool, wasting the original execution.

The 41 decision points are hardcoded. If you're building a security-focused agent that needs CVE severity triage or a data agent that needs schema validation, you can't extend the decision taxonomy without forking. The architecture is pluggable at the judge layer but rigid at the decision point layer.

Finally, the hosted Jev dependency means the default free tier is a single point of failure. Latency numbers aren't published, so it's unclear whether 50ms judge calls on every chunk actually save time compared to just feeding everything to a fast regional LLM endpoint. If your main model responds in 200ms and the judge adds 50ms per chunk across 20 chunks, you've added a full second of overhead.

Verdict

Use mu if: you're running high-volume coding agents on expensive frontier models (GPT-4, Claude Opus) where token costs are a line item in your budget; your tasks routinely generate multi-thousand-line tool outputs (test suites, build logs, web scrapes); you're experiencing approval fatigue from agents that ask permission for already-authorized work; or you need auditable decision-making with shadow mode testing before production deployment. The chunked admission architecture is genuinely novel and the observability is production-grade.

Skip mu if: you're using local or cheap models where the main LLM is already fast and token costs are negligible; your tasks produce small, focused outputs that fit comfortably in context; you require deterministic agent behavior and can't tolerate probabilistic judge errors introducing non-reproducible failures; you need to extend the framework with custom decision points for domain-specific workflows; or you're in a research phase where you want full LLM visibility into all data rather than optimizing costs. In those cases, stick with pi directly or use a simpler agent framework like Aider.