arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.09536cs.CLcs.AI
收藏

TAD: 时空感知轨迹自蒸馏用于快速准确的扩散大语言模型

TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Ant Group(蚂蚁集团)

Haoyang Zhou, Li Kong, Shijie Ren, Xiting Wang, Shuang Liang, Guowei Wang, Zhenxuan Pan

AI总结:

TAD通过时空感知轨迹自蒸馏方法,在提升生成速度的同时保持准确性,实验表明在LLaDA上质量模型将平均准确率提升至51.6%,速度模型将AUP提升至257.1。

摘要:

扩散大语言模型(dLLMs)为并行文本生成提供了有前景的范式,但在实践中面临准确性和并行性之间的权衡,增加每个前向步骤的令牌数(TPF)通常会降低生成质量。现有加速方法往往以牺牲准确性为代价来提高速度。为了解决这一限制,我们提出了TAD,一种时空感知轨迹自蒸馏框架。在数据构建过程中,我们使教师模型在提示和真实响应的基础上生成解码轨迹,记录整个过程中中间的掩码状态。基于每个掩码令牌被揭示前剩余的解码步骤数,我们将掩码位置分为近和远子集。对于近令牌,我们使用教师轨迹令牌作为标签,通过硬交叉熵损失训练学生模型,鼓励对即将被解码的令牌做出自信预测。对于远令牌,我们应用教师和学生令牌分布之间的软KL散度损失,提供更柔和的监督并保留未来的规划知识。这种时空感知的划分自然导致两种部署配置:优先准确性的质量模型和更激进加速的速度模型。实验表明,TAD在准确性和并行性之间取得了持续的改进。在LLaDA上,质量模型将平均准确率从46.2%提升到51.6%,速度模型将平均AUP从46.2提升到257.1。我们的代码可在:https://github.com/BHmingyang/TAD获取。

英文摘要:

Diffusion large language models (dLLMs) offer a promising paradigm for parallel text generation, but in practice they face an accuracy-parallelism trade-off, where increasing tokens per forward (TPF) often degrades generation quality. Existing acceleration methods often gain speed at the cost of accuracy. To address this limitation, we propose TAD, a Temporal-Aware trajectory self-Distillation framework. During data construction, we condition a teacher model on both the prompt and the ground-truth response to generate decoding trajectories, recording the intermediate masked states throughout the process. Based on how many decoding steps remain before each masked token is revealed, we partition masked positions into near and distant subsets. For near tokens, we train the student with a hard cross-entropy loss using the teacher trajectory tokens as labels, encouraging confident predictions for tokens that are about to be decoded. For distant tokens, we apply a soft KL divergence loss between the teacher and student token distributions, providing softer supervision and preserving future planning knowledge. This temporal-aware partition naturally gives rise to two deployment configurations: a Quality model that prioritizes accuracy and a Speed model that favors more aggressive acceleration. Experiments show that TAD consistently improves the accuracy-parallelism trade-off. On LLaDA, it raises average accuracy from 46.2\% to 51.6\% with the Quality model and average AUP from 46.2 to 257.1 with the Speed model. Our code is available at: https://github.com/BHmingyang/TAD

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑