CARE-RFT：基于置信度的强化微调用于大语言模型中的可靠推理
CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models
AI总结：
CARE-RFT通过引入置信度锚定的正则化方法，在保持大语言模型推理能力的同时提升其可信度和校准性能。
摘要：
强化微调（RFT）已逐渐成为解锁大语言模型推理能力的强大范式。然而，我们发现了一个关键的权衡：虽然无约束RFT在推理性能上表现强劲，但会严重损害模型的可信度，通过放大幻觉并恶化校准；相反，RKL约束的RFT保持了可信度，但因对其探索偏差的无界惩罚而限制了推理收益。为解决这一矛盾，我们引入了CARE-RFT（置信度锚定的正则化强化微调），一种新的方法，将标准反KL正则化替换为偏斜反KL散度。CARE-RFT提供了一种置信度敏感的惩罚：它对自信、一致奖励的探索具有边界，以促进推理，而在其他地方则无界以保持校准。在多个模型规模和RFT算法上的广泛实验表明，CARE-RFT实现了更优的平衡，其推理性能与无约束RFT相当，同时恢复了基础模型的可信度和校准。我们的工作确立了精心设计的、置信度感知的正则化是构建既强大又可信推理模型的关键。
英文摘要：
Reinforcement finetuning (RFT) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models. However, we identify a critical trade-off: while unconstrained RFT achieves strong reasoning performance, it severely compromises model trustworthiness by amplifying hallucination and worsening calibration; conversely, RKL-constrained RFT preserves trustworthiness but limits reasoning gains due to its unbounded penalty on exploratory deviations. To resolve this tension, we introduce CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), a novel method that replaces standard reverse KL regularization with a skew reverse KL divergence. CARE-RFT provides a confidence-sensitive penalty: it is bounded for confident, consistently rewarded explorations to enable reasoning, while unbounded elsewhere to preserve calibration. Extensive experiments across multiple model scales and RFT algorithms show that CARE-RFT achieves a superior balance, matching the reasoning performance of unconstrained RFT while recovering the trustworthiness and calibration of the base model. Our work establishes that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
