R3-RAG：通过强化学习让大语言模型逐步推理与检索
R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning
AI总结：
R3-RAG通过强化学习让大语言模型逐步推理与检索，以解决RAG中稠密检索器的瓶颈，提升答案正确性并迁移至不同检索器。
摘要：
检索增强生成（RAG）将外部知识与大语言模型（LLMs）相结合，以增强事实正确性并缓解幻觉。然而，由于稠密检索器的参数相比LLMs有限，且无法进行逐步推理，它们往往成为RAG系统的瓶颈。虽然基于提示的迭代式RAG试图解决这些局限，但其受限于人工设计的流程。为解决这些问题，我们提出R3-RAG，它使用强化学习使LLM学会如何逐步推理和检索，从而获取全面的外部知识并得出正确答案。R3-RAG分为两个阶段。首先，我们使用冷启动使模型学会迭代交错推理和检索的方式。然后，我们使用强化学习进一步增强其探索外部检索环境的能力。具体而言，我们为R3-RAG提出两种奖励：1）用于结果奖励的答案正确性，判断轨迹是否导向正确答案；2）用于过程奖励的基于相关性的文档验证，鼓励模型检索与用户问题相关的文档，通过这种方式让模型学会如何迭代推理并检索相关文档以获得正确答案。实验结果表明，R3-RAG显著优于基线，并能很好地迁移到不同的检索器。我们在https://github.com/Yuan-Li-FNLP/R3-RAG发布R3-RAG。
英文摘要：
Retrieval-Augmented Generation (RAG) integrates external knowledge with Large Language Models (LLMs) to enhance factual correctness and mitigate hallucination. However, dense retrievers often become the bottleneck of RAG systems due to their limited parameters compared to LLMs and their inability to perform step-by-step reasoning. While prompt-based iterative RAG attempts to address these limitations, it is constrained by human-designed workflows. To address these limitations, we propose $\textbf{R3-RAG}$, which uses $\textbf{R}$einforcement learning to make the LLM learn how to $\textbf{R}$eason and $\textbf{R}$etrieve step by step, thus retrieving comprehensive external knowledge and leading to correct answers. R3-RAG is divided into two stages. We first use cold start to make the model learn the manner of iteratively interleaving reasoning and retrieval. Then we use reinforcement learning to further harness its ability to better explore the external retrieval environment. Specifically, we propose two rewards for R3-RAG: 1) answer correctness for outcome reward, which judges whether the trajectory leads to a correct answer; 2) relevance-based document verification for process reward, encouraging the model to retrieve documents that are relevant to the user question, through which we can let the model learn how to iteratively reason and retrieve relevant documents to get the correct answer. Experimental results show that R3-RAG significantly outperforms baselines and can transfer well to different retrievers. We release R3-RAG at https://github.com/Yuan-Li-FNLP/R3-RAG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
