arXiv 2410.017822024-10-02cs.CLcs.AIcs.LG

Open-RAG：利用开源大型语言模型增强检索增强推理

Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models

Bangladesh University of Engineering and Technology（孟加拉工程技术大学）；Cohere For AI（Cohere AI研究团队）；University of North Texas（北得克萨斯大学）；York University（约克大学）；Salesforce Research（Salesforce研究院）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）

Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, Md Rizwan Parvez

备注：: Accepted to EMNLP 2024 Findings. Website: https://openragmoe.github.io/. 14 pages, 7 figures, 5 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 Open-RAG 框架，将开源稠密 LLM 转为稀疏 MoE 以增强复杂推理与抗干扰检索，并通过混合自适应检索平衡效果和速度，在多项知识密集型任务上超过 ChatGPT 等模型。

摘要：

检索增强生成（Retrieval-Augmented Generation，RAG）已被证明能够提升大型语言模型（LLM）的事实准确性，但现有方法往往在有效利用检索到的证据方面存在推理能力有限的问题，尤其是在使用开源 LLM 时。为弥补这一差距，我们提出了一个新颖框架 Open-RAG，旨在利用开源 LLM 增强 RAG 中的推理能力。我们的框架将任意稠密 LLM 转化为参数高效的稀疏混合专家（mixture of experts，MoE）模型，使其能够处理复杂推理任务，包括单跳和多跳查询。Open-RAG 的独特之处在于训练模型应对看似相关但具有误导性的困难干扰项。因此，Open-RAG 利用潜在学习，动态选择相关专家并有效整合外部知识，从而生成更准确、更符合上下文的回答。此外，我们提出一种混合自适应检索方法，用于判断是否需要检索，并在性能提升与推理速度之间进行权衡。实验结果表明，基于 Llama2-7B 的 Open-RAG 在多种知识密集型任务上优于 ChatGPT、Self-RAG 和 Command R+ 等最先进的 LLM 和 RAG 模型。我们在 https://openragmoe.github.io/ 开源了代码和模型。

英文摘要：

Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs), but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs. To mitigate this gap, we introduce a novel framework, Open-RAG, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading. As a result, Open-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses. In addition, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed. Experimental results show that the Llama2-7B-based Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT, Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source our code and models at https://openragmoe.github.io/

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。