ICML 2025 Best Paper: Can xMemory Completely Solve RAG's Memory Problem?
With the booming development of Large Language Model Agents, how to make AI remember dialogue history spanning dozens of sessions has become a key factor determining agent practicality. When we interact with an AI assistant that has accompanied us for months, we want it to remember project preferences we discussed before, understand our work habits, and even recall key details mentioned in a particular chat. However, standard Retrieval Augmented Generation (RAG) methods reveal surprising limitations when facing this "agent memory" scenario.
The Overlooked Fundamental Problem: RAG and Agent Memory are "Genetically Incompatible"
Before diving into xMemory, it's worth understanding the core background of this transformation. What was the original purpose of traditional RAG? It was built for retrieving from massive internet documents, enterprise knowledge bases, or hundreds of research papers. In these scenarios, the retrieved text segments come from different sources, often with varied topics and diverse perspectives. That's exactly why traditional "top-k similarity retrieval" can effectively filter the most relevant information from a large pool of heterogeneous candidates.
But the agent memory scenario is entirely different. When a user engages in multi-round conversations with an Agent spanning weeks or even months, the stored memories present a unique structure: highly relevant, highly redundant, and tightly connected in the temporal dimension. For example, if a user discusses the same project multiple times within a month, all dialogue segments about that project will form a dense "cloud" in the semantic space—their similarity to each other is extremely high, but the core information relevant to the user's current query might only occupy a few points in this dense cloud.
This creates an awkward situation: when using standard top-5 or top-10 retrieval, the returned results are often near-duplicate segments from the same topic. In other words, what the RAG system painstakingly finds as "most relevant" context is actually just repeatedly talking about the same thing, while the truly needed temporally-connected evidence chain is "optimized away" by the algorithm as redundant content.
The core insight of the xMemory paper is precisely this: standard RAG assumes a "heterogeneous text corpus," while agent memory is essentially a "homogeneous memory stream." This fundamental mismatch cannot be solved by simple parameter tuning.
xMemory's Core Approach: Decoupling and Aggregation
Facing this challenge, xMemory proposes an elegant and profound solution—"Decoupling-then-Aggregation." The core idea is: instead of struggling at the level of raw dialogue segments, it's better to first organize memory structurally, then reverse this structure to drive retrieval.
Specifically, xMemory builds a four-level hierarchical memory structure:
- Messages: Actual dialogue turns between user and Agent
- Episodes: Compressed summaries of continuous multi-turn dialogues, capturing complete topic units
- Semantics: Reusable long-term facts extracted from episode segments, such as user's name, workplace, preferences, habits, etc.
- Themes: Related semantic units organized together to form higher-level conceptual aggregates
The brilliance of this hierarchical structure lies in its "decoupling" of information that was originally mixed in the temporal flow—semantic units are extracted separately and no longer bound to original dialogue turns; meanwhile, through the "aggregation" of the theme layer, high-level associations between semantic units are established. It's like first settling and layering muddy river water, then drawing from it as needed.
Two-Stage Adaptive Retrieval: Representative Selection and Uncertainty Awareness
Simply building a hierarchical structure isn't enough—the key is efficient retrieval within this structure. xMemory employs a two-stage adaptive retrieval strategy, which I consider the most brilliant part of this paper.
Stage 1: Query-Aware Representative Selection on kNN Graph
At the theme and semantic unit layers, xMemory maintains a k-Nearest Neighbors (kNN) graph structure. When a user poses a query, the system doesn't directly search for the most similar segments in raw dialogues. Instead, it first performs "representative selection" at the theme layer—selecting representative nodes that best cover different knowledge directions from multiple relevant themes.
The key innovation here is a trade-off formula:
i* = argmax [α × coverage_gain + (1-α) × query_relevance]
The system balances two objectives: on one hand, selecting content semantically relevant to the current query; on the other hand, ensuring the selected content can cover different knowledge areas in memory, avoiding always clustering in some dense region. This process is iterative—after selecting each node, it updates the coverage status of its neighbors until reaching some coverage threshold.
Stage 2: Uncertainty-Adaptive Evidence Inclusion
After selecting relevant semantic units, the next step is deciding which specific dialogue segments to include as "evidence." xMemory's approach is clever: rather than cramming all relevant segments into the context window, it filters based on the principle of "uncertainty reduction.
The specific implementation: for each candidate dialogue segment, the system evaluates whether it can reduce the language model's prediction uncertainty about the answer. Only when additionally including a segment can significantly reduce answer uncertainty will it be included in the final context. This method naturally achieves a "on-demand retrieval" effect—details directly related to the answer are preserved, while redundant near-duplicate content is filtered out.
Experimental Results: A Win-Win for Efficiency and Quality
xMemory was thoroughly validated on two benchmark datasets: LoCoMo (multi-session dialogue dataset with approximately 300 turns on average) and PerLTQA (Personal Long-Term Memory Question Answering).
The results are impressive:
- On Qwen3-8B model, xMemory improved average BLEU from 28.51 to 34.48, and F1 from 40.45 to 43.98
- On GPT-5 nano, it also achieved improvements from 36.65 to 38.71 (BLEU) and 48.17 to 50.00 (F1)
- Most stunning is token efficiency: on Qwen3-8B, token consumption dropped from 9103 to 4711, a reduction of nearly 50%!
This means xMemory not only improves answer quality but also reduces costs simultaneously. For long-running, cost-sensitive Agent applications, this is undoubtedly an attractive feature.
Deeper Implications
After reading this paper, I can't help but contemplate its deeper implications for the entire Agent field.
First, it reveals an important design principle: "The way memory is organized determines retrieval efficiency." When designing Agent memory systems, we shouldn't only focus on how to store and index, but also think about how to "decouple" knowledge—separating long-term valid general facts from short-term valid dialogue context, separating core concepts from minor details. This separation itself is a form of intelligence.
Second, xMemory's "two-stage retrieval" also hints at a broader design pattern: using coarse-grained representative selection to control the search space, then using fine-grained uncertainty assessment to filter content. This "coarse-to-fine" strategy has universal value in many scenarios requiring efficient use of limited resources.
Finally, this paper reminds us to re-examine RAG's boundaries. RAG is a powerful paradigm, but it's not omnipotent. When application scenario characteristics don't match RAG's original assumptions, we need courage to customize or even rebuild, rather than forcing a fit.
Conclusion
This ICML 2025 paper injects a strong dose of motivation into the cutting-edge field of agent memory. xMemory, with its "decoupling-aggregation" hierarchical approach and "representative selection-uncertainty awareness" two-stage retrieval, successfully resolves the困境 of standard RAG in homogeneous memory stream scenarios. What's even more commendable is that while improving answer quality, it also significantly reduces token costs—an irresistible temptation for any engineer concerned with practical deployment.
If you're building AI Agents requiring long-term memory, or have deeper interest in retrieval augmentation technology, this paper is definitely worth reading. It not only provides a practical technical solution but, more importantly, demonstrates a way of thinking—when you find existing methodologies don't match the problem's essence, return to the problem's origin and redesign the entire system.