FaithRL: 通过步骤级忠实性最大化学习合理推理
FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization
AI总结：
FaithRL通过步骤级忠实性最大化优化，减少大型语言模型的幻觉并提高推理准确性。
摘要：
可验证奖励的强化学习（RLVR）显著提高了大型语言模型（LLMs）在需要多步推理的任务上的性能。然而，大多数RLVR流程依赖于稀疏的基于结果的奖励，对中间步骤提供很少的监督，从而鼓励过度自信和虚假推理，进而增加幻觉。为了解决这个问题，我们提出了FaithRL，一个通用的强化学习框架，直接优化推理的忠实性。我们正式化了一个忠实性最大化的目标，并理论上证明了优化它可以缓解过度自信。为了实现这个目标，我们引入了一种几何奖励设计和一种忠实性感知的优势调节机制，通过惩罚未支持的步骤来分配步骤级信用，同时保留有效的部分推导。在多样化的骨干网络和基准测试中，FaithRL一致地降低了幻觉率，同时保持（并经常提高）答案的正确性。进一步分析证实，FaithRL增加了逐步推理的忠实性，并具有良好的鲁棒性。我们的代码可在https://github.com/aintdoin/FaithRL上找到。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has markedly improved the performance of Large Language Models (LLMs) on tasks requiring multi-step reasoning. However, most RLVR pipelines rely on sparse outcome-based rewards, providing little supervision over intermediate steps and thus encouraging over-confidence and spurious reasoning, which in turn increases hallucinations. To address this, we propose FaithRL, a general reinforcement learning framework that directly optimizes reasoning faithfulness. We formalize a faithfulness-maximization objective and theoretically show that optimizing it mitigates over-confidence. To instantiate this objective, we introduce a geometric reward design and a faithfulness-aware advantage modulation mechanism that assigns step-level credit by penalizing unsupported steps while preserving valid partial derivations. Across diverse backbones and benchmarks, FaithRL consistently reduces hallucination rates while maintaining (and often improving) answer correctness. Further analysis confirms that FaithRL increases step-wise reasoning faithfulness and generalizes robustly. Our code is available at https://github.com/aintdoin/FaithRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
