GRPO-VPS：通过可验证的过程监督增强组相对策略优化以实现有效的推理
GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
AI总结：
本文提出GRPO-VPS，通过在推理轨迹中探测模型对正确答案的信心，改进GRPO的轨迹级反馈，实现更精准高效的策略更新，实验显示在数学和通用领域任务中均取得显著提升。
摘要：
可验证奖励的强化学习（RLVR）通过直接结果验证而非学习奖励模型，提升了大语言模型（LLMs）的推理能力。基于此范式，组相对策略优化（GRPO）消除了对批评模型的依赖，但其对中间步骤的无差别信用分配限制了其识别有效推理策略的能力，并导致过度思考。本文引入一种无模型且可验证的过程监督方法，通过在推理轨迹中探测模型对正确答案的信心，将生成过程分割为离散步骤，并跟踪每个步骤边界处正确答案的条件概率，从而高效计算可解释的分段进展度量，以优化GRPO的轨迹级反馈。此方法实现了更精准且样本效率更高的策略更新，同时避免了从昂贵的蒙特卡洛回放或辅助模型中衍生的中间监督。在数学和通用领域基准测试中，实验结果表明在多样化的模型上，GRPO-VPS在数学任务中准确率提升达2.6个百分点，推理长度减少13.7%，在通用领域任务中准确率提升达2.4个百分点，推理长度减少4%，展示了强大的泛化能力。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
