arXiv 2409.055912025-04-09cs.CLcs.AI

MemoRAG：通过全局记忆增强的检索增强生成提升长上下文处理能力

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation

Peking University（北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Hong Kong Polytechnic University（香港理工大学）

Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, Tiejun Huang

备注：: theWebConf 2025. Codes and models are in https://github.com/qhjqhj00/MemoRAG

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MemoRAG框架，通过轻量长程系统构建全局记忆并生成草稿答案以辅助检索，再由强表达力系统生成最终答案，从而在多种长上下文任务中超越传统RAG方法。

摘要：

处理长上下文对大型语言模型（LLM）而言是一项重大挑战。尽管近期进展使LLM能够处理比以往更长的上下文（例如32K或128K个token），但计算开销高昂，且在许多应用中仍可能不足。检索增强生成（RAG）被认为是解决这一问题的有前景策略。然而，传统RAG方法因两个内在要求而面临根本性局限：1）明确陈述的查询，以及2）结构良好的知识。但在一般的长上下文处理任务中，这些条件并不成立。在本文中，我们提出了MemoRAG，一种由全局记忆增强检索驱动的新型RAG框架。MemoRAG采用双系统架构。首先，它使用一个轻量但长程的系统来构建长上下文的全局记忆。一旦任务被提出，该系统会生成草稿答案，为检索工具在长上下文中定位相关信息提供有用线索。其次，它利用一个昂贵但表达力强的系统，基于检索到的信息生成最终答案。在此基础框架之上，我们以KV压缩的形式实现记忆模块，并通过生成质量反馈（又称RLGF）增强其记忆和线索提供能力。在我们的实验中，MemoRAG在多种长上下文评估任务上取得了优越性能，不仅在传统RAG方法难以应对的复杂场景中表现突出，在通常应用RAG的简单场景中同样表现优异。

英文摘要：

Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 32K or 128K tokens), it is computationally expensive and can still be insufficient for many applications. Retrieval-Augmented Generation (RAG) is considered a promising strategy to address this problem. However, conventional RAG methods face inherent limitations because of two underlying requirements: 1) explicitly stated queries, and 2) well-structured knowledge. These conditions, however, do not hold in general long-context processing tasks. In this work, we propose MemoRAG, a novel RAG framework empowered by global memory-augmented retrieval. MemoRAG features a dual-system architecture. First, it employs a light but long-range system to create a global memory of the long context. Once a task is presented, it generates draft answers, providing useful clues for the retrieval tools to locate relevant information within the long context. Second, it leverages an expensive but expressive system, which generates the final answer based on the retrieved information. Building upon this fundamental framework, we realize the memory module in the form of KV compression, and reinforce its memorization and cluing capacity from the Generation quality's Feedback (a.k.a. RLGF). In our experiments, MemoRAG achieves superior performances across a variety of long-context evaluation tasks, not only complex scenarios where traditional RAG methods struggle, but also simpler ones where RAG is typically applied.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。