arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20516cs.CLcs.AIcs.LG
收藏

VASPO:结合特权价值估计的自蒸馏方法

VASPO: Self-Distillation with Privileged Value Estimation

University of Science and Technology of China(中国科学技术大学);IFLYTEK Research(讯飞研究院)

Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du

AI总结:

本文提出结合特权价值估计的自蒸馏方法VASPO,在PPO框架内整合自教师与评论者的互补动态,在六个数学推理基准上提升了Qwen3-4B和OLMo3-7B的准确率,在Knights and Knaves任务中表现最优。

摘要:

在线策略自蒸馏通过特权自教师为推理模型提供密集监督,但该监督反映的是局部标记偏好,而非无引导学生延续的期望回报。现有方法将该信号与GRPO结合,用GRPO确定更新方向、教师进行信号缩放。本文证明在PPO框架内可实现更有效的整合,具体识别出互补的位置动态:特权信息可提升评论者对响应末尾的价值预测准确性,而自教师指导在生成初期较强、中期下降、末期回升。基于此提出价值增强自蒸馏策略优化(VASPO),其中自教师与评论者在学生生成的前缀处评估相同特权信息,评论者从预测的延续回报计算标记级PPO优势,自教师通过有界概率比率调节选定的正优势。在六个数学推理基准测试中,VASPO使Qwen3-4B变体的平均准确率较PPO提升2.59个百分点,OLMo3-7B变体提升1.81个百分点,且在Knights and Knaves任务中达到所有评估方法中的最高平均准确率。

英文摘要:

On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑