基于AMR概念蒸馏的上下文压缩以增强RAG
Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation
AI总结：
本文提出一种基于AMR概念蒸馏的RAG框架，将检索到的长文档压缩为关键概念以过滤噪声，实验表明其能提升开放域问答性能并具备跨模型鲁棒性。
摘要：
大型语言模型（LLMs）在信息获取方面取得了显著进展。然而，它们过度依赖可能存在缺陷的参数化知识，导致幻觉和不准确，尤其是在处理长尾、领域特定查询时。检索增强生成（RAG）通过引入外部非参数化知识来解决这一局限。然而，检索到的长上下文文档通常包含噪声、无关信息以及关键知识，从而负面稀释了LLMs的注意力。受基本概念在个体阅读理解中的支持作用启发，我们提出了一种新颖的基于概念的RAG框架，并采用基于抽象意义表示（AMR）的概念蒸馏算法。该算法通过参考可靠的语言特征，将杂乱的原始检索文档压缩为一组从AMR信息节点中蒸馏出的关键概念。这些概念明确约束LLMs在推理过程中仅关注重要信息。我们在开放域问答数据集上进行了广泛实验，以实证评估所提方法的有效性。结果表明，基于概念的RAG框架优于其他基线方法，特别是在支持文档数量增加时表现尤为突出，同时在不同骨干LLMs上展现出鲁棒性。这强调了蒸馏出的概念通过过滤干扰信息对增强RAG过程具有信息价值。据我们所知，这是首次将AMR引入以增强RAG的工作，为通过基于语义的上下文压缩提升推理性能提供了一种潜在解决方案。
英文摘要：
Large Language Models (LLMs) have made significant strides in information acquisition. However, their overreliance on potentially flawed parametric knowledge leads to hallucinations and inaccuracies, particularly when handling long-tail, domain-specific queries. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. Nevertheless, the retrieved long-context documents often contain noisy, irrelevant information alongside vital knowledge, negatively diluting LLMs' attention. Inspired by the supportive role of essential concepts in individuals' reading comprehension, we propose a novel concept-based RAG framework with the Abstract Meaning Representation (AMR)-based concept distillation algorithm. The proposed algorithm compresses the cluttered raw retrieved documents into a compact set of crucial concepts distilled from the informative nodes of AMR by referring to reliable linguistic features. The concepts explicitly constrain LLMs to focus solely on vital information in the inference process. We conduct extensive experiments on open-domain question-answering datasets to empirically evaluate the proposed method's effectiveness. The results indicate that the concept-based RAG framework outperforms other baseline methods, particularly as the number of supporting documents increases, while also exhibiting robustness across various backbone LLMs. This emphasizes the distilled concepts are informative for augmenting the RAG process by filtering out interference information. To the best of our knowledge, this is the first work introducing AMR to enhance the RAG, presenting a potential solution to augment inference performance with semantic-based context compression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
