arXiv 2507.227162025-08-06cs.CL

从充分性到反思：检索增强推理中强化引导的思维质量提升

From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs

School of Informatics, University of Edinburgh, UK（爱丁堡大学信息学院）；School of Computer Science and Informatics, Cardiff University, UK（卡迪夫大学计算机科学与信息学院）

Jie He, Victor Gutiérrez-Basulto, Jeff Z. Pan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对RAG推理中信息不足、推理错误和答案不一致问题，提出TIRESRAG-R1框架，通过思考-检索-反思过程和多维奖励系统提升推理质量，在四个多跳问答数据集上优于现有方法。

摘要：

基于强化学习的检索增强生成（RAG）方法增强了大型语言模型（LLMs）的推理能力。然而，大多数方法仅依赖最终答案奖励，忽视了中间推理质量。本文分析了现有的RAG推理模型，识别出三种主要失败模式：（1）信息不足，即模型未能检索到足够的支持信息；（2）推理错误，即在信息充分的情况下出现逻辑或内容层面的缺陷；（3）答案与推理不一致，即有效的推理链导致不匹配的最终答案。我们提出了TIRESRAG-R1，一个新颖的框架，采用思考-检索-反思过程和多维奖励系统来提升推理能力和稳定性。TIRESRAG-R1引入了：（1）充分性奖励，鼓励彻底检索；（2）推理质量奖励，评估推理链的合理性和准确性；（3）反思奖励，用于检测和修正错误。它还采用了难度感知的重加权策略和训练样本过滤，以提升在复杂任务上的表现。在四个多跳问答数据集上的实验表明，TIRESRAG-R1优于先前的RAG方法，并能很好地泛化到单跳任务。代码和数据可在以下网址获取：https://github.com/probe2/TIRESRAG-R1。

英文摘要：

Reinforcement learning-based retrieval-augmented generation (RAG) methods enhance the reasoning abilities of large language models (LLMs). However, most rely only on final-answer rewards, overlooking intermediate reasoning quality. This paper analyzes existing RAG reasoning models and identifies three main failure patterns: (1) information insufficiency, meaning the model fails to retrieve adequate support; (2) faulty reasoning, where logical or content-level flaws appear despite sufficient information; and (3) answer-reasoning inconsistency, where a valid reasoning chain leads to a mismatched final answer. We propose TIRESRAG-R1, a novel framework using a think-retrieve-reflect process and a multi-dimensional reward system to improve reasoning and stability. TIRESRAG-R1 introduces: (1) a sufficiency reward to encourage thorough retrieval; (2) a reasoning quality reward to assess the rationality and accuracy of the reasoning chain; and (3) a reflection reward to detect and revise errors. It also employs a difficulty-aware reweighting strategy and training sample filtering to boost performance on complex tasks. Experiments on four multi-hop QA datasets show that TIRESRAG-R1 outperforms prior RAG methods and generalizes well to single-hop tasks. The code and data are available at: https://github.com/probe2/TIRESRAG-R1.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。