超越正确性：在检索增强生成中奖励忠实推理
Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
AI总结：
本文提出VERITAS框架，通过细粒度轮次级忠实性奖励强化学习，提升检索增强生成中推理步骤的忠实性，同时改善任务性能。
摘要：
受强化学习在数学和代码等领域的大语言模型训练中取得成功的启发，近期工作开始训练LLMs动态规划、查询并使用搜索引擎作为工具进行推理——这种范式日益被称为智能体搜索。尽管这些方法在流行的短问答基准上取得了性能提升，但许多方法优先考虑最终答案的正确性，而忽略了中间推理步骤的质量，这可能导致思维链不忠实。本文首先引入了一个全面的智能体搜索评估框架，涵盖三种不同的忠实性指标：思考-搜索忠实性、信息-思考忠实性和思考-答案忠实性。我们的评估表明，通过基于回合级结果奖励的可验证奖励强化学习训练的典型智能体搜索系统（包括Search-R1和ReSearch）在这些忠实性维度上有显著的改进空间。为了促进智能体搜索中的忠实推理，我们引入了VERITAS（通过智能体搜索中的中间可追溯性验证蕴含推理），这是一个新颖的框架，将细粒度的轮次级忠实性奖励整合到强化学习过程中。我们的实验表明，使用VERITAS训练的模型不仅显著提高了推理忠实性，而且与基于回合级结果奖励训练的基线相比，还实现了更好的任务性能。
英文摘要：
Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent work has begun training LLMs to dynamically plan, query, and reason with search engines as tools -- a paradigm increasingly referred to as agentic search. Although these methods achieve performance improvement across popular short-form QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for agentic search, covering three distinct faithfulness metrics: Think-Search faithfulness, Information-Think faithfulness, and Think-Answer faithfulness. Our evaluations reveal that canonical agentic search systems trained through Reinforcement Learning from Verifiable Reward (RLVR) using episode-level outcome-based reward -- including Search-R1 and ReSearch -- have significant room for improvement on these faithfulness dimensions. To foster faithful reasoning in agentic search, we introduce VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained turn-level faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with \ours not only significantly improve reasoning faithfulness, but also achieve better task performance compared to baselines trained against episode-level outcome-based reward.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
