An ICML 2025 Best Paper: Can xMemory Fully Solve RAG’s Memory Predicament?
As large-model agents continue to flourish, enabling AI to remember conversation history across dozens of sessions has become crucial to agents’ real-world usefulness. When we spend months chatting with an AI companion, we expect it to remember the project-related preferences we discussed earlier, understand our work habits, and even recall key details from a past conversation. Yet standard retrieval-augmented generation (RAG) methods expose striking limitations in this “agent memory” setting.
The Overlooked Root Problem: RAG and Agent Memory Are “Genetically Incompatible”
Before diving into xMemory, we need to understand the core context behind this shift. What was traditional RAG designed to do? It was built to retrieve from massive collections of internet documents, enterprise knowledge bases, or hundreds of research papers. In these settings, retrieved passages come from different sources and often cover disparate topics and perspectives. Precisely for that reason, traditional “top-k similarity retrieval” can effectively select different pieces of information most relevant to the current query from a large pool of heterogeneous candidates.
But the agent-memory setting is entirely different. When a user has a multi-turn conversation with an agent over weeks or months, the stored memories have a distinctive structure: highly relevant, highly redundant, and tightly linked in time. For example, if a user discusses the same project repeatedly over a month, all conversation fragments about that project form a dense “cloud” in semantic space: they are extremely similar to one another, while the core information relevant to the user’s current query may occupy only a few points within that dense cloud.
This creates an awkward situation: when standard top-5 or top-10 retrieval is used, the results often consist of near-duplicate fragments on the same topic. In other words, the “most relevant” context that the RAG system works so hard to find may simply keep restating the same thing, while the temporally connected chain of evidence that is actually needed is instead treated as redundant and “optimized away” by the algorithm.
The central insight of the xMemory paper is precisely this: standard RAG assumes a “heterogeneous text corpus,” whereas agent memory is essentially a “homogeneous memory stream.” This fundamental mismatch cannot be solved by simple parameter tuning.
xMemory’s Core Idea: Decoupling and Aggregation
Facing this challenge, xMemory proposes an elegant and insightful solution—“Decouple, Then Aggregate” (Decoupling-then-Aggregation). Its central idea is simple: rather than struggling at the level of raw conversation fragments, first organize memories into a structured form, then invert that structure to drive retrieval.
Specifically, xMemory builds a four-level hierarchical memory structure:
- Raw Messages (Messages): the actual conversation turns between the user and agent
- Episodes: consecutive multi-turn conversations condensed into general summaries that capture a complete unit of discussion
- Semantic Units (Semantics): reusable long-term facts extracted from episodes, such as the user’s name, employer, preferences, and habits
- Themes: related semantic units grouped into higher-level conceptual aggregates
The beauty of this hierarchy is that it “decouples” information previously mixed into the temporal stream: semantic units are extracted on their own and no longer tied to the original conversation turns. At the same time, the theme layer “aggregates” them, establishing higher-level relationships between semantic units. It is like letting a pool of muddy riverwater settle into distinct layers, then drawing from it as needed.
Two-Stage Adaptive Retrieval: Representative Selection and Uncertainty Awareness
Building the hierarchy is not enough; the key is retrieving efficiently from it. xMemory uses a two-stage adaptive retrieval strategy—the part of the paper I find most compelling.
Stage 1: Query-Aware Representative Selection on a kNN Graph
Across the theme and semantic-unit layers, xMemory maintains a k-nearest neighbors (kNN) graph. When a user submits a query, the system does not immediately search the raw conversations for the most similar fragments. Instead, it first performs “representative selection” at the theme layer, choosing representative nodes from multiple relevant themes that together best cover different areas of knowledge.
The key innovation here is a trade-off formula:
i* = argmax [α × 覆盖增益 + (1-α) × 查询相关性]
The system balances two objectives: selecting content that is semantically relevant to the current query while ensuring that the selected content covers different areas of the memory, rather than clustering around one dense region. The process is iterative: each time a node is selected, the coverage status of its neighbors is updated until a coverage threshold is reached.
Stage 2: Uncertainty-Adaptive Evidence Selection
After selecting relevant semantic units, the next step is to determine which specific episodes to include as “evidence.” xMemory does this cleverly: instead of stuffing every relevant fragment into the context window, it filters them according to the principle of “uncertainty reduction.”
Specifically, for each candidate episode, the system evaluates whether including it would reduce the language model’s predictive uncertainty about the answer. An episode is added to the final context only when including it significantly lowers that uncertainty. This naturally delivers a “retrieve on demand” effect: details directly relevant to the answer are retained, while redundant near-duplicates are filtered out.
Experimental Results: A Win–Win for Efficiency and Quality
xMemory was thoroughly evaluated on two benchmark datasets: LoCoMo (a multi-session dialogue dataset with an average of roughly 300 conversation turns) and PerLTQA (question answering on personal long-term memory).
The results are impressive:
- On Qwen3-8B, xMemory raised average BLEU from 28.51 to 34.48 and F1 from 40.45 to 43.98
- On GPT-5 nano, it similarly improved BLEU from 36.65 to 38.71 and F1 from 48.17 to 50.00
- Most astonishing was its token efficiency: with Qwen3-8B, token consumption fell from 9103 to 4711, cutting usage by nearly half!
This means xMemory improves answer quality while reducing costs at the same time. For agent applications that need to run long term and are sensitive to cost, that is undoubtedly an appealing feature.
Deeper Implications
After reading this paper, I can’t help reflecting on its deeper implications for the agent field as a whole.
First, it reveals an important design principle: “How memories are organized determines how efficiently they can be retrieved.” When designing an agent’s memory system, we should focus not only on how to store and index information, but also on how to “decouple” knowledge—separating durable, general facts from short-lived conversational context, and core concepts from minor details. This separation is itself a form of intelligence.
Second, xMemory’s “two-stage retrieval” also suggests a broader design pattern: use coarse-grained representative selection to constrain the search space, then use fine-grained uncertainty evaluation to filter the content. This “coarse-to-fine” strategy is broadly applicable in many settings where limited resources must be used efficiently.
Finally, the paper reminds us to reconsider the boundaries of RAG. RAG is a powerful paradigm, but it is not a universal solution. When an application’s characteristics do not align with RAG’s original assumptions, we need to be willing to customize or even restructure the approach rather than forcing it to fit at all costs.
Conclusion
This ICML 2025 paper has breathed new life into the frontier field of agent memory. With its hierarchical “decoupling–aggregation” approach and two-stage “representative selection–uncertainty awareness” retrieval, xMemory successfully resolves the difficulties standard RAG faces in homogeneous memory-stream settings. Even more impressive, it significantly reduces token costs while improving answer quality—an advantage no engineer focused on practical deployment can overlook.
If you are building AI agents that need long-term memory, or are simply more deeply interested in retrieval-augmented techniques, this paper is well worth reading. It offers not only a practical technical solution, but, more importantly, a way of thinking about problems: when you find that an existing methodology does not match the true nature of the problem, it may be worth going back to the source and redesigning the entire system.