arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.15228cs.CLcs.IR
收藏

通过多智能体强化学习改进检索增强生成

Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning

Renmin University of China(中国人民大学);Baidu Inc.(百度公司);Carnegie Mellon University(卡内基梅隆大学)

Yiqun Chen, Lingyong Yan, Weiwei Sun, Xinyu Ma, Yi Zhang, Shuaiqiang Wang, Dawei Yin, Yiming Yang, Jiaxin Mao

AI总结:

针对RAG组件独立优化导致目标错位的问题,提出多智能体强化学习算法MMOA-RAG,将各组件视为协作智能体并统一优化最终答案的F1分数,在多个问答基准上优于现有基线。

摘要:

检索增强生成(RAG)被广泛用于将外部知识融入大型语言模型,从而增强问答(QA)任务的事实性并减少幻觉。标准的RAG流程包含多个组件,如查询重写、文档检索、文档过滤和答案生成。然而,这些组件通常通过监督微调进行单独优化,这可能导致各组件目标与生成准确答案这一总体目标之间出现错位。尽管最近有研究尝试使用强化学习(RL)来优化特定的RAG组件,但这些方法往往只关注仅含两个组件的简单流程,或者未能充分处理模块之间复杂的相互依赖和协作交互。为克服这些局限,我们提出将包含多个组件的复杂RAG流程视为一个多智能体协作任务,其中每个组件可被视为一个RL智能体。具体而言,我们提出了MMOA-RAG(多模块联合优化算法),该算法采用多智能体强化学习将所有智能体的目标协调至统一的奖励,例如最终答案的F1分数。在多个问答基准上的实验表明,MMOA-RAG有效提升了流程的整体性能,并优于现有基线。此外,全面的消融研究验证了各组件的贡献,并证明MMOA-RAG可适配不同的RAG流程和基准。

英文摘要:

Retrieval-augmented generation (RAG) is widely utilized to incorporate external knowledge into large language models, thereby enhancing factuality and reducing hallucinations in question-answering (QA) tasks. A standard RAG pipeline consists of several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically optimized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual components and the overarching aim of generating accurate answers. Although recent efforts have explored using reinforcement learning (RL) to optimize specific RAG components, these approaches often focus on simple pipelines with only two components or do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these limitations, we propose treating the complex RAG pipeline with multiple components as a multi-agent cooperative task, in which each component can be regarded as an RL agent. Specifically, we present MMOA-RAG, Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent reinforcement learning to harmonize all agents' goals toward a unified reward, such as the F1 score of the final answer. Experiments conducted on various QA benchmarks demonstrate that MMOA-RAG effectively boost the overall performance of the pipeline and outperforms existing baselines. Furthermore, comprehensive ablation studies validate the contributions of individual components and demonstrate MMOA-RAG can be adapted to different RAG pipelines and benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑