GEM-RAG:用于检索增强生成的图特征记忆
GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation
AI总结:
受人类记忆机制启发,提出GEM-RAG方法,通过LLM生成效用问题标记文本块、构建相似度图并利用特征分解生成高层摘要节点,在标准QA任务上优于现有RAG方法。
摘要:
形成、检索记忆并对记忆进行推理以响应刺激的能力,是通用智能的基石——它塑造了具备学习、适应和直觉洞察力的实体。大型语言模型(LLMs)已证明,在给定适当记忆或上下文的情况下,它们能够对刺激进行推理并作出有意义的响应。然而,它们仍无法以最优方式编码、存储和检索记忆——实现这一能力将解锁其作为AI智能体运作以及在细分领域进行专业化的全部潜力。为解决此问题,一个极具前景的研究方向是检索增强生成(RAG),旨在通过提供丰富的上下文示例和信息来增强LLMs。在问答(QA)应用中,RAG方法将目标文本分块嵌入,并使用文本嵌入为提示检索最相关的文本块。受人类记忆编码和检索机制的启发,我们旨在通过生成和编码更高层次的信息,并根据文本块回答问题的效用对其进行标记,来改进标准RAG方法。我们提出了用于检索增强生成的图特征记忆(GEM-RAG)。GEM-RAG的工作原理是:使用LLM生成的“效用”问题为给定文本语料库中的每个文本块添加标记,基于文本和效用问题的相似性在图中连接这些文本块,然后利用记忆图的特征分解构建能够捕捉文本主题的高层摘要节点。我们使用UnifiedQA和GPT-3.5 Turbo作为LLMs,结合SBERT和OpenAI的文本编码器,在两个标准QA任务上对GEM-RAG进行了评估,结果表明GEM-RAG在这些任务上优于其他最先进的RAG方法。我们还讨论了拥有强大RAG系统的影响及未来方向。
英文摘要:
The ability to form, retrieve, and reason about memories in response to stimuli serves as the cornerstone for general intelligence - shaping entities capable of learning, adaptation, and intuitive insight. Large Language Models (LLMs) have proven their ability, given the proper memories or context, to reason and respond meaningfully to stimuli. However, they are still unable to optimally encode, store, and retrieve memories - the ability to do this would unlock their full ability to operate as AI agents, and to specialize to niche domains. To remedy this, one promising area of research is Retrieval Augmented Generation (RAG), which aims to augment LLMs by providing them with rich in-context examples and information. In question-answering (QA) applications, RAG methods embed the text of interest in chunks, and retrieve the most relevant chunks for a prompt using text embeddings. Motivated by human memory encoding and retrieval, we aim to improve over standard RAG methods by generating and encoding higher-level information and tagging the chunks by their utility to answer questions. We introduce Graphical Eigen Memories For Retrieval Augmented Generation (GEM-RAG). GEM-RAG works by tagging each chunk of text in a given text corpus with LLM generated ``utility'' questions, connecting chunks in a graph based on the similarity of both their text and utility questions, and then using the eigendecomposition of the memory graph to build higher level summary nodes that capture the main themes of the text. We evaluate GEM-RAG, using both UnifiedQA and GPT-3.5 Turbo as the LLMs, with SBERT, and OpenAI's text encoders on two standard QA tasks, showing that GEM-RAG outperforms other state-of-the-art RAG methods on these tasks. We also discuss the implications of having a robust RAG system and future directions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。