arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.04428cs.CLcs.AI
收藏

dOPSD:扩散语言模型的策略内自蒸馏

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

National University of Singapore(新加坡国立大学)

Phuong Tuan Dat, Qi Li, Xinchao Wang

AI总结:

研究扩散语言模型强化推理难的问题,提出dOPSD,通过从学生去噪轨迹直接推导教师特权信息,实现策略内自蒸馏,提升模型在数学推理和代码生成等任务表现。

摘要:

扩散大语言模型通过迭代去噪生成文本,但强化推理难。策略内自蒸馏有潜力,但依赖教师特权信息,效果不佳。我们引入dOPSD,直接从学生去噪轨迹推导教师特权,用同一轨迹后续步骤评估掩码位置,在Dream和LLaDA上提升了数学推理和代码生成能力。

英文摘要:

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑