BEE-RAG：面向检索增强生成的平衡熵工程
BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
AI总结：
本文提出 BEE-RAG 框架，通过熵不变性原理和平衡上下文熵重构注意力动态，缓解长检索上下文中的熵增长与注意力稀释问题，并引入零样本推理与参数高效微调机制，在多个 RAG 任务上验证了有效性。
摘要：
随着大语言模型（LLMs）的快速发展，检索增强生成（RAG）已成为补充大语言模型固有知识局限的关键方法。然而，由于检索到的信息通常体量庞大，RAG 往往需要处理较长的上下文长度。从熵工程的角度来看，我们发现长检索上下文所导致的熵无约束增长和注意力稀释是影响 RAG 性能的重要因素。本文提出了平衡熵工程化的 RAG（BEE-RAG）框架，该框架通过熵不变性原理提升 RAG 系统对不同上下文长度的适应性。BEE-RAG 利用平衡上下文熵重新构建注意力动态，将注意力敏感性与上下文长度解耦，从而确保稳定的熵水平。在此基础上，我们引入了一种用于多重重要性估计的零样本推理策略，以及一种参数高效的自适应微调机制，以针对不同设置获得最优平衡因子。在多个 RAG 任务上的大量实验证明了 BEE-RAG 的有效性。
英文摘要：
With the rapid advancement of large language models (LLMs), retrieval-augmented generation (RAG) has emerged as a critical approach to supplement the inherent knowledge limitations of LLMs. However, due to the typically large volume of retrieved information, RAG tends to operate with long context lengths. From the perspective of entropy engineering, we identify unconstrained entropy growth and attention dilution due to long retrieval context as significant factors affecting RAG performance. In this paper, we propose the balanced entropy-engineered RAG (BEE-RAG) framework, which improves the adaptability of RAG systems to varying context lengths through the principle of entropy invariance. By leveraging balanced context entropy to reformulate attention dynamics, BEE-RAG separates attention sensitivity from context length, ensuring a stable entropy level. Building upon this, we introduce a zero-shot inference strategy for multi-importance estimation and a parameter-efficient adaptive fine-tuning mechanism to obtain the optimal balancing factor for different settings. Extensive experiments across multiple RAG tasks demonstrate the effectiveness of BEE-RAG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
