arXiv 2602.058972026-05-27cs.CL

停止奖励幻觉步骤：面向小型推理模型的忠实感知步骤级强化学习

Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models

Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳研究院）；Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd（中国电信人工智能研究院）；College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China（浙江大学集成电路学院）；Zhongguancun Academy, Beijing, China（中关村学院）

Shuo Nie, Hexuan Deng, Chao Wang, Ruiyu Fang, Xuebo Liu, Shuangyong Song, Yu Li, Min Zhang, Xuelong Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对小型推理模型在中间推理步骤中容易产生忠实性幻觉的问题，提出忠实感知步骤级强化学习（FaithRL），通过过程奖励模型提供步骤级监督和隐式截断重采样策略，减少幻觉并提高推理可靠性。

摘要：

随着大型语言模型变得更小更高效，小型推理模型（SRM）在资源受限环境中实现思维链（CoT）推理至关重要。然而，它们容易产生忠实性幻觉，尤其是在中间推理步骤中。现有的基于在线强化学习的缓解方法依赖于结果奖励或粗粒度的CoT评估，这可能在最终答案正确时无意中强化不忠实的推理。为了解决这些局限性，我们提出了忠实感知步骤级强化学习（FaithRL），通过来自过程奖励模型的显式忠实奖励引入步骤级监督，以及一种隐式截断重采样策略，该策略从忠实前缀生成对比信号，同时减轻步骤级奖励的奖励黑客攻击。在多个SRM和开放书籍QA基准上的实验表明，FaithRL持续减少CoT和最终答案中的幻觉，从而实现更忠实和可靠的推理。代码可在 https://github.com/Easy195/FaithRL 获取。

英文摘要：

As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings. However, they are prone to faithfulness hallucinations, especially in intermediate reasoning steps. Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained CoT evaluation, which can inadvertently reinforce unfaithful reasoning when the final answer is correct. To address these limitations, we propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model, together with an implicit truncated resampling strategy that generates contrastive signals from faithful prefixes, while also mitigating reward hacking from step-level rewards. Experiments across multiple SRMs and Open-Book QA benchmarks demonstrate that FaithRL consistently reduces hallucinations in both the CoT and final answers, leading to more faithful and reliable reasoning. Code is available at https://github.com/Easy195/FaithRL.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。