arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.09109cs.CLcs.AIcs.IR
收藏

正向与反向思考:用于检索增强推理的多目标强化学习

Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning

Wenda Wei, Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Lixin Su, Shuaiqiang Wang, Dawei Yin, Maarten de Rijke, Xueqi Cheng

AI总结:

Bi-RAR通过双向信息距离和多目标强化学习框架,提升检索增强推理在复杂多步骤任务中的表现。

摘要:

检索增强生成(RAG)已被证明在缓解大语言模型中的幻觉问题上是有效的,但在复杂的多步骤推理场景中其效果仍然有限。最近的努力将搜索基于的交互引入RAG,使能够进行迭代推理和实时检索。大多数方法依赖于基于结果的监督,没有为中间步骤提供明确的指导。这通常导致奖励黑客和响应质量下降。我们提出了Bi-RAR,一种新的检索增强推理框架,它在正向和反向方向上共同评估每个中间步骤。为了评估每个步骤的信息完整性,我们引入了基于柯尔莫哥洛夫复杂度的双向信息距离,通过语言模型生成概率进行近似。这种量化方法衡量了当前推理与答案的距离以及其对问题的解答程度。为了在这些双向信号下优化推理,我们采用了一种多目标强化学习框架,具有级联奖励结构,强调早期轨迹的一致性。在七个问答基准上的实验证明,Bi-RAR优于先前的方法,并在训练和推理过程中实现了与搜索引擎的高效交互和推理。

英文摘要:

Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models, yet its effectiveness remains limited in complex, multi-step reasoning scenarios. Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval. Most approaches rely on outcome-based supervision, offering no explicit guidance for intermediate steps. This often leads to reward hacking and degraded response quality. We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions. To assess the information completeness of each step, we introduce a bidirectional information distance grounded in Kolmogorov complexity, approximated via language model generation probabilities. This quantification measures both how far the current reasoning is from the answer and how well it addresses the question. To optimize reasoning under these bidirectional signals, we adopt a multi-objective reinforcement learning framework with a cascading reward structure that emphasizes early trajectory alignment. Empirical results on seven question answering benchmarks demonstrate that Bi-RAR surpasses previous methods and enables efficient interaction and reasoning with the search engine during training and inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑