mu: A Two-Tier Memory System That Makes Coding Agents 10x Cheaper
Hook
Your coding agent just spent $3 reading a 10,000-line test failure log—twice. Only 40 lines were relevant. mu fixes this with a judge that decides what your expensive LLM actually needs to see.
Context
Coding agents have a token economics problem. When you point GPT-4 or Claude at a failing test suite, it reads everything: stack traces, passing tests, verbose logging, the works. A single Jest failure can dump 15,000 lines into context. Do this across five tool calls in one session and you've burned through 200K tokens—most of it noise. The agent re-reads the same compiler errors, re-processes identical stack frames, and still asks for permission to run the exact command you already approved three turns ago.
The standard solution is retrieval-augmented generation: chunk everything, embed it, query semantically. But RAG adds latency, requires vector infrastructure, and forces you to pre-decide chunk boundaries. It's also lossy by design—once you've embedded and discarded the original, you can't go back for the full artifact. mu takes a different approach: stream tool output through a small, fast classifier that makes binary admission decisions chunk-by-chunk. Relevant segments enter the main model's context. Everything else gets archived behind a pointer. The expensive LLM sees a curated slice. The cheap judge does triage. Built as a layer on top of pi (a TypeScript coding agent framework), mu intercepts 41 decision points across the agent loop—not just context management, but also prompt injection screening, command risk assessment, and turn orchestration.
Technical Insight
The architecture centers on a swappable judge kernel that operates at five interception stages. At input preflight, the judge classifies incoming messages (is this a refinement of the previous task or a completely new request?). During context management, it makes admission decisions on tool output chunks and relevance assessments for retrieving skills or lessons from memory. At safety checkpoints, it evaluates command risk (is rm -rf on these paths dangerous?) and screens for prompt injection in external content. During turn orchestration, it detects goal drift and validates whether a run actually completed the stated objective. Finally, for multi-agent workflows, it routes subtask delegation and validates subagent patches.
The killer feature is chunked output admission. Here's how it works in practice:
// Tool executor wraps command output in a streaming admission pipeline
const testOutput = await exec('npm test');
const chunks = chunkByLine(testOutput, 50); // 50-line segments
const admitted = [];
const archived = [];
for (const chunk of chunks) {
const verdict = await judge({
decision: 'context.tool_admit',
chunk,
taskContext: currentGoal,
priorContext: conversationSummary
});
if (verdict.admit && verdict.confidence > 0.7) {
admitted.push(chunk);
} else {
archived.push({ hash: sha256(chunk), location: archivePath });
}
}
// Main LLM sees only admitted chunks plus pointers
context.toolOutput = {
relevant: admitted.join('\n'),
archived: archived.map(a => `[${a.hash.slice(0,8)}]: ${a.location}`)
};
This creates a two-tier memory system. The main model's context stays lean—maybe 2,000 lines instead of 15,000. But the archived chunks aren't lost. If the agent later says "show me the full output for test X," the pointer resolves to lossless retrieval. No embeddings, no semantic approximation.
Each decision point has three modes: active (verdict is enforced), shadow (logged but ignored—the agent proceeds as if the judge wasn't there), and off. Shadow mode is the pragmatic stroke of genius here. Before trusting a judge with production workloads, you run it in shadow across real tasks. The ledger captures every verdict, probability distribution, and latency:
{
"decision": "context.tool_admit",
"mode": "shadow",
"verdict": false,
"confidence": 0.83,
"latency_ms": 47,
"would_have_blocked": "lines 150-200 of pytest output",
"actual_outcome": "main LLM read it anyway"
}
After a week of shadow logs, you can compare task success rates, token usage, and failure modes. Did the judge correctly identify irrelevant chunks? Did it ever block something the main LLM needed later? This observability is rare in agent frameworks—most force you to trust the abstraction or fork the code.
The judge itself defaults to Jev (a hosted TypeSafe endpoint trained on coding agent decision patterns), but the interface is pluggable. You can swap in a local model, a HuggingFace classifier, or even call another LLM if you're optimizing for accuracy over speed. The contract is simple: accept a decision type and context object, return a boolean verdict with confidence.
Beyond context management, mu addresses approval fatigue. Coding agents often pause mid-task to ask, "Should I run tests now?" even when you've already said yes to the overall goal. The turn.continue decision point detects these redundant confirmation-seeking patterns and auto-continues up to two times per message. It's heuristic—capped retries prevent infinite loops—but it solves a real UX pathology.
Prompt injection screening happens at tool boundaries. When the agent scrapes a web page, fetches from an MCP server, or runs curl, the output passes through safety.injection_detect before entering context. This is architecturally cleaner than trying to make the LLM injection-proof—you're screening untrusted content at the perimeter rather than hoping the model resists adversarial prompts embedded in API responses.
Gotcha
The entire system has a single point of failure: judge accuracy. If the classifier's false negative rate exceeds 5% on context admission, you're silently starving the main LLM of information it needs. There's no automated ground truth collection. The ledger shows you what the judge decided, but not whether it was correct. You're flying blind unless you manually audit verdicts against task outcomes.
Chunked admission also has a fundamental blindness problem. The judge assesses each chunk without full conversation history—by design, because passing all context would negate the efficiency gains. But relevance is contextual. A log line that seems irrelevant at turn three might become critical at turn ten when the agent realizes the earlier failure was related. There's no mechanism to say, "I was wrong about that archived chunk; retrieve it now." You'd need to re-run the tool, wasting the original execution.
The 41 decision points are hardcoded. If you're building a security-focused agent that needs CVE severity triage or a data agent that needs schema validation, you can't extend the decision taxonomy without forking. The architecture is pluggable at the judge layer but rigid at the decision point layer.
Finally, the hosted Jev dependency means the default free tier is a single point of failure. Latency numbers aren't published, so it's unclear whether 50ms judge calls on every chunk actually save time compared to just feeding everything to a fast regional LLM endpoint. If your main model responds in 200ms and the judge adds 50ms per chunk across 20 chunks, you've added a full second of overhead.
Verdict
Use mu if: you're running high-volume coding agents on expensive frontier models (GPT-4, Claude Opus) where token costs are a line item in your budget; your tasks routinely generate multi-thousand-line tool outputs (test suites, build logs, web scrapes); you're experiencing approval fatigue from agents that ask permission for already-authorized work; or you need auditable decision-making with shadow mode testing before production deployment. The chunked admission architecture is genuinely novel and the observability is production-grade.
Skip mu if: you're using local or cheap models where the main LLM is already fast and token costs are negligible; your tasks produce small, focused outputs that fit comfortably in context; you require deterministic agent behavior and can't tolerate probabilistic judge errors introducing non-reproducible failures; you need to extend the framework with custom decision points for domain-specific workflows; or you're in a research phase where you want full LLM visibility into all data rather than optimizing costs. In those cases, stick with pi directly or use a simpler agent framework like Aider.