arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.15438cs.AI
收藏

揭示并咨询检索增强型基于 MoE 的大语言模型中的核心专家

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs

School of Computer Science, Fudan University(复旦大学计算机科学技术学院);LightSpeed Studios, Tencent(腾讯光速工作室);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学智能信息处理重点实验室);School of Informatics and ELLIS, University of Edinburgh(爱丁堡大学信息学院及欧洲学习与智能系统实验室)

Xin Zhou, Ping Nie, Yiwen Guo, Haojie Wei, Zhanqiu Zhang, Pasquale Minervini, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

本文研究基于 MoE 的大语言模型中影响检索增强生成效果的核心专家激活机制,识别出可判断内部知识充分性、评估文档质量和增强上下文利用能力的核心专家,并据此提出改进 RAG 效率与效果的策略。

摘要:

检索增强生成(RAG)显著提升了大语言模型(LLMs)解决知识密集型任务的能力。尽管现有研究试图通过检索更高质量的文档或设计面向 RAG 的专用大语言模型来提升 RAG 性能,但大语言模型内部有助于 RAG 系统有效性的机制仍未得到充分探索。本文旨在研究当前流行的基于专家混合(Mixture-of-Expert,MoE)的大语言模型中的这些内部机制,并通过考察这些大语言模型中的专家激活情况来展示如何改进 RAG。我们的对照实验揭示,若干核心专家组主要负责与 RAG 相关的行为。这些核心专家的激活可以表明模型对外部/内部知识的倾向,并调节其行为。例如,我们识别出能够(1)指示模型内部知识是否充分、(2)评估检索文档质量以及(3)增强模型利用上下文能力的核心专家。基于这些发现,我们提出了若干通过专家激活来提升 RAG 效率和有效性的策略。在多个数据集和基于 MoE 的大语言模型上的实验结果表明了我们方法的有效性。

英文摘要:

Retrieval-Augmented Generation (RAG) significantly improved the ability of Large Language Models (LLMs) to solve knowledge-intensive tasks. While existing research seeks to enhance RAG performance by retrieving higher-quality documents or designing RAG-specific LLMs, the internal mechanisms within LLMs that contribute to the effectiveness of RAG systems remain underexplored. In this paper, we aim to investigate these internal mechanisms within the popular Mixture-of-Expert (MoE)-based LLMs and demonstrate how to improve RAG by examining expert activations in these LLMs. Our controlled experiments reveal that several core groups of experts are primarily responsible for RAG-related behaviors. The activation of these core experts can signify the model's inclination towards external/internal knowledge and adjust its behavior. For instance, we identify core experts that can (1) indicate the sufficiency of the model's internal knowledge, (2) assess the quality of retrieved documents, and (3) enhance the model's ability to utilize context. Based on these findings, we propose several strategies to enhance RAG's efficiency and effectiveness through expert activation. Experimental results across various datasets and MoE-based LLMs show the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑