本文提出结合特权价值估计的自蒸馏方法VASPO,在PPO框架内整合自教师与评论者的互补动态,在六个数学推理基准上提升了Qwen3-4B和OLMo3-7B的准确率,在Knights and Knaves任务中表现最优。
摘要:
在线策略自蒸馏通过特权自教师为推理模型提供密集监督,但该监督反映的是局部标记偏好,而非无引导学生延续的期望回报。现有方法将该信号与GRPO结合,用GRPO确定更新方向、教师进行信号缩放。本文证明在PPO框架内可实现更有效的整合,具体识别出互补的位置动态:特权信息可提升评论者对响应末尾的价值预测准确性,而自教师指导在生成初期较强、中期下降、末期回升。基于此提出价值增强自蒸馏策略优化(VASPO),其中自教师与评论者在学生生成的前缀处评估相同特权信息,评论者从预测的延续回报计算标记级PPO优势,自教师通过有界概率比率调节选定的正优势。在六个数学推理基准测试中,VASPO使Qwen3-4B变体的平均准确率较PPO提升2.59个百分点,OLMo3-7B变体提升1.81个百分点,且在Knights and Knaves任务中达到所有评估方法中的最高平均准确率。
英文摘要:
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.
$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Meituan(美团);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation
Anhui Robot Technology Standard Innovation Base, School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院安徽机器人技术标准创新基地)
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences(中国科学院大学杭州高等研究院);The Chinese University of Hong Kong(香港中文大学);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学);vivo AI Lab(vivo AI实验室)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Shanghai Jiao Tong University(上海交通大学);Xiaohongshu Inc.(小红书公司);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);Southeast University(东南大学)
Learning from the Self-future: On-policy Self-distillation for dLLMs
Tsinghua University(清华大学);Technical University of Munich(慕尼黑工业大学);Nanyang Technological University(南洋理工大学);University of British Columbia(不列颠哥伦比亚大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);ELLIS Institute Tubingen(ELLIS蒂宾根研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tubingen AI Center(蒂宾根人工智能中心)