发表机构
Korea University（高丽大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
RL-ARC是一种校准感知训练框架，联合利用推理与答案置信度，可改善推理模型校准效果，缓解分布偏移下的过度自信问题，且不大幅牺牲推理性能。
AI 中文摘要
语言模型（LMs）通常采用带可验证奖励的强化学习（RLVR）进行训练，以提升其推理能力。但由于RLVR在训练过程中未明确考虑校准问题，可能导致严重的校准退化，包括过度自信。近期针对语言模型的校准感知训练方法，将不确定性估计目标融入训练，虽改善了校准效果，但在分布偏移下仍存在过度自信问题，且会牺牲推理性能。为此，我们提出RL-ARC，这是一种校准感知训练框架，联合利用推理置信度与答案置信度。具体而言，RL-ARC将推理置信度作为校准答案置信度的辅助信号，对正确案例应用推理引导正则化，对错误案例应用过度自信惩罚。在分布内（ID）和分布外（OOD）设置下的综合结果表明，除改善校准外，RL-ARC还能让推理模型基于给定问题自适应估计置信度，且不会大幅牺牲推理性能，凸显了推理置信度对训练可靠推理模型的重要性。
英文摘要
Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.
CommentsAACL-IJCNLP 2026
