PACR：用于大语言模型推理的渐进上升置信度奖励
PACR: Progressively Ascending Confidence Reward for LLM Reasoning
AI总结：
针对RLVR稀疏奖励缺乏中间指导的问题，提出PACR密集内在奖励，基于模型对正确答案信念的上升趋势加速探索，提升多基准推理性能。
摘要：
基于可验证奖励的强化学习（RLVR）显著提升了大语言模型的推理能力，但其稀疏的、基于结果的奖励无法为中间步骤提供指导，从而减缓了探索过程。我们提出了渐进上升置信度奖励（PACR），这是一种密集的、模型内在的奖励，直接根据模型对正确答案不断演化的信念来计算。PACR 编码了一种归纳偏置，即沿着一条结构良好的推理轨迹，真实答案的概率应呈现总体上升的趋势。我们提供了实证和理论分析，验证了这种归纳偏置将探索搜索空间限制在逻辑推理更丰富的区域。我们证明，PACR 加速了探索，用更少的轨迹达到奖励饱和，并在多个基准上带来了改进。我们的结果表明，密集的、模型内在的塑形信号可以使 RLVR 训练更加有效和可靠。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
