arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.25973cs.LGcs.AIcs.CL
收藏

用于自蒸馏策略优化的依赖正确性的非对称目标分布

Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

Korea University(高丽大学);Chung-Ang University(中央大学)

Suwan Yoon, Seongryong Jung, Changhee Lee

AI总结:

针对自蒸馏策略优化中直接匹配自教师的缺陷,提出 APSD 方法,通过混合学生与自教师分布构建依赖正确性的目标,在 Qwen3 模型上以更少训练步骤达到与 SDPO 相当的性能。

摘要:

自蒸馏策略优化通过反馈条件自教师为大语言模型(LLM)的后训练提供密集的 token 级监督。然而,直接匹配自教师会破坏正确 rollout 上的成功行为,同时对错误行为提供的修正不足。我们提出了通过自蒸馏的非对称策略优化(APSD),该方法通过混合学生和自教师分布来构建依赖正确性的目标:插值保留正确 rollout 上的成功行为,而超出自教师的外推则加强对错误行为的修正。我们通过散度分解、反向 KL 重心几何和梯度分析对 APSD 进行分析,强调了通过依赖学生的目标的梯度流的重要性。在具有可验证奖励和丰富反馈的强化学习实验中,APSD 相对于强化学习和自蒸馏基线实现了强劲的整体性能,并且在 Qwen3 模型规模上,平均仅需 2.6 倍更少的训练步骤即可达到与 SDPO 相当的性能。

英文摘要:

Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑