arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.30451cs.LG
收藏

VeriGate: 验证器门控的步骤级监督用于GRPO

VeriGate: Verifier-Gated Step-Level Supervision for GRPO

University of Maryland, College Park(马里兰大学学院公园分校)

Aakriti Agrawal, Minghui Liu, Furong Huang

AI总结:

提出VeriGate方法,通过验证器门控的步骤级监督扩展GRPO,解决稀疏奖励和信用分配问题,在多个推理基准上显著提升准确率。

摘要:

组相对策略优化(GRPO)是一种有效的训练推理模型的方法,使用基于验证器的结果奖励,但其监督是稀疏的:当针对某个提示的所有采样轨迹获得相同的验证器奖励时,组相对优势会坍缩为零,学习停滞。仅结果奖励也不提供步骤级信用分配,限制了探索,使得学习稳健推理更加困难。我们提出了VeriGate(验证器门控步骤级GRPO),这是GRPO的一种验证器门控扩展,通过三个设计选择解决了这些限制。首先,每当验证器奖励在采样轨迹之间诱导出有意义的偏好时,VeriGate让验证器负责,并且仅在验证器奖励退化时使用过程监督。其次,VeriGate不将过程奖励模型(PRM)的步骤分数坍缩为单个轨迹奖励,而是将其转换为未来累积奖励,以分配延续感知的信用。第三,VeriGate将这些奖励转换为组归一化的令牌级优势,恢复信息丰富的梯度和细粒度的信用分配,同时相比优化聚合PRM分数的方法,对奖励黑客攻击的敏感性更低。实验上,在MATH上使用1.5B和7B Qwen2.5-Instruct模型进行训练,并在六个推理基准上评估,VeriGate将1.5B和7B模型的平均准确率分别提高了约20%和12%,显著减少了零梯度失败,降低了奖励黑客行为,并相对于仅结果GRPO和PRM作为结果的基线提高了推理质量。

英文摘要:

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑