Memoria: Why AI Agents Need More Than RAG to Remember
Hook
Your AI agent forgets everything between conversations. It rediscovers the same solutions, relearns your preferences, and repeats solved problems like a memoryless goldfish swimming in circles.
Context
Traditional RAG systems treat memory like a search engine: embed documents, retrieve semantically similar chunks, stuff them into context. This works for lookup tasks—finding relevant documentation or fetching facts from a corpus. But AI agents need more than semantic similarity. They need to remember why a decision was made, how a problem was solved last time, what preferences the user has expressed, and which methods failed in previous attempts.
Memoria addresses this by implementing a hybrid architecture that stores agent artifacts—decisions, methods, preferences, contextual relationships—in both vector and graph databases. Vector search handles semantic recall (“find similar situations”), while the graph structure maintains provenance and relationships (“what led to this decision” and “what other choices were considered”). Crucially, Memoria implements agent-aware write-back, allowing agents to store reasoning outcomes so knowledge compounds over time instead of evaporating after each session. The project is currently under active development with an MVP expected in 2-3 weeks, but the architectural approach is worth examining now.
Technical Insight
Memoria’s architecture rests on three pillars: Milvus for vector embeddings, Neo4j for graph relationships, and a FastAPI backend that orchestrates both. The repository structure reveals this separation clearly—backend/milvus/ contains collection schemas for vector storage, backend/neo4j/ defines graph schemas and relationship mappings, and backend/src/ implements the business logic that queries both simultaneously.
The key insight is in the memory schemas themselves. Rather than storing generic “documents,” Memoria defines agent-native constructs: requests (what the agent was asked), answers (what it concluded), actions (what it executed), and preferences (learned user choices). Each memory item lives in both databases—embedded vectors enable similarity search while graph nodes preserve contextual links. When an agent recalls “similar requests,” it doesn’t just get semantically close text chunks; it gets the full decision tree: what alternatives were considered, why one was chosen, what the outcome was, and what follow-up actions resulted.
The frontend, while currently labeled a “toy or demo,” provides visibility into this dual structure. You can deploy it locally to inspect memory contents:
# clone the repo
git clone https://github.com/TheBuddyDave/Memoria.git
cd memoria/frontend
# install dependencies
npm install
# run the demo UI
npm run devThis won’t give you a working memory system yet—the backend pipeline isn’t complete—but it illustrates the intended mental model: memory isn’t a flat list of embeddings but a navigable graph of agent experiences.
The write-back mechanism is where Memoria diverges most sharply from static RAG. In typical RAG, the knowledge base is read-only: documents go in during indexing, embeddings come out during retrieval. Memoria inverts this. After an agent completes a task, it can write back: “I chose approach X because of constraint Y, and it produced outcome Z.” That decision becomes a first-class memory node, linked to the original request, related to similar past decisions, and retrievable when similar constraints appear again. Over time, the agent builds a corpus of its own reasoning—not just facts about the world, but knowledge about itself.
The backend structure in backend/src/ is organized around API routes, business logic, and database interactions. Memory writes must update both Milvus collections (adding vector embeddings) and Neo4j nodes (creating relationship edges). Memory reads query both in parallel: vector search returns candidate memories by semantic similarity, then graph traversal enriches results with provenance and context. The FastAPI entry point in main.py exposes these operations as HTTP endpoints, allowing agent frameworks to integrate Memoria as an external memory service rather than tightly coupling it to specific agent implementations.
The roadmap reveals ambitious plans for memory evolution. Basic pruning logic is in progress—deciding which memories to keep when storage fills. Planned features include temporal weighting (recent memories matter more), decay strategies (old memories fade unless reinforced), and multi-agent shared memory (team knowledge pools). These aren’t solved problems. Human memory is messy, contextual, and imperfect; replicating even crude versions of forgetting, reinforcement, and consolidation in software is genuinely hard. Memoria is tackling these challenges in the open, which means early adopters will be navigating rough edges alongside the maintainers.
Gotcha
Memoria is pre-MVP and under active development. The repository contains folder structure, schemas, and architectural foundations, but the core memory pipeline is not yet complete. The backend isn’t deployable in production. The core write and recall logic is listed as “Near-term (MVP)” on the roadmap, expected in approximately 2-3 weeks. The frontend runs as a demonstration UI, but has limited functionality without the backend pipeline. If you clone this today expecting to integrate it into your agent, you’ll be working with incomplete components.
Even once the MVP arrives, significant questions remain unanswered. How do you avoid polluting the memory graph with low-value interactions? What’s the API surface for writing memories—does the agent explicitly call write-back endpoints, or does Memoria infer what to store from observing agent behavior? How does pruning decide which memories to forget when the graph grows large? The README mentions “human-controllable” memory with inspection and editing, but doesn’t specify whether that’s a UI feature, an API, or manual database access. Production deployment guidance is still being developed. The project states that “Once the MVP is complete and stabilized we will begin opening up for contributions,” so external collaboration is currently limited while core functionality is being built. This is a project to watch and potentially contribute to once stable, not a project to deploy in production today.
Verdict
Use if: You’re researching agent memory architectures and want to follow a promising hybrid approach as it develops. Memoria’s combination of vector search for similarity and graph structure for relationships is architecturally sound, addressing real limitations in pure RAG systems. If you’re building an agent framework and anticipating the need for write-back memory in the coming months, star the repo and join the Discord to track progress. The explicit focus on agent-native schemas (decisions, methods, preferences) suggests the maintainers understand agent workflows beyond basic chatbot RAG.
Skip if: You need functional agent memory today. Memoria’s core pipeline is under development with an MVP expected in 2-3 weeks. For immediate production use, evaluate mature alternatives or build a custom solution using existing frameworks with vector and graph database integration—you’ll have a working system sooner than waiting for Memoria to reach production readiness. Come back when the MVP is released, the core features are stabilized, and there’s documentation for production deployment. The architectural approach is promising, but the implementation is still being built. Until core functionality ships, this is an interesting design to monitor rather than a tool to deploy.
