arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.06133cs.CLcs.AI
收藏

CreditDecoding: 利用轨迹信用加速扩散大语言模型中的并行解码

CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

Shanghai Jiao Tong University(上海交通大学);Ant Group(蚂蚁集团);Westlake University(西湖大学)

Kangyu Wang, Zhiyun Jiang, Haibo Feng, Weijia Zhao, Lin Liu, Jianguo Li, Zhenzhong Lan, Weiyao Lin

AI总结:

针对扩散大语言模型并行解码中正确令牌被反复重掩导致冗余迭代的问题,提出基于轨迹信用的无训练并行解码方法CreditDecoding,融合历史证据与当前logits提升低置信度正确令牌的置信度,实现高达5.48倍加速并提升准确性。

摘要:

扩散大语言模型(dLLMs)通过迭代去噪生成文本。在普遍采用的并行解码方案中,每一步仅确认高置信度位置,而重掩其他位置。通过分析dLLM去噪轨迹,我们发现一个关键的低效问题:模型通常在目标令牌的置信度足够高以被解码之前的几个步骤就预测出正确令牌。这种早期预测与后期解码之间的差距导致已正确的令牌被反复重掩,造成冗余迭代并限制加速。为利用这种时间冗余,我们引入轨迹信用(Trace Credit),通过累积历史证据来量化令牌的解码潜力。基于此,我们提出CreditDecoding,一种无训练的并行解码方法,将轨迹信用与当前logits融合,以提升正确但低置信度令牌的置信度,从而加速去噪并提高鲁棒性。在八个基准测试上,CreditDecoding在LLaDA-8B上实现了高达5.48倍的加速和+0.48的准确率提升,并在多种dLLM架构和参数规模上持续改进性能。它还能扩展到长上下文,并与主流推理优化方法正交,使其成为一种实用且广泛适用的解决方案。

英文摘要:

Diffusion large language models (dLLMs) generate text through iterative denoising. In commonly adopted parallel decoding schemes, each step confirms only high-confidence positions while remasking the others. By analyzing dLLM denoising traces, we uncover a key inefficiency: models often predict the correct target token several steps before its confidence becomes high enough to be decoded. This gap between early prediction and late decoding forces repeated remasking of already-correct tokens, causing redundant iterations and limiting acceleration. To exploit this temporal redundancy, we introduce Trace Credit to quantify a token's decoding potential by accumulating historical evidence. Building on this, we propose CreditDecoding, a training-free parallel decoding method that fuses Trace Credit with current logits to boost the confidence of correct but underconfident tokens, thereby accelerating denoising and improving robustness. On eight benchmarks, CreditDecoding achieves up to 5.48 times speedup with +0.48 accuracy on LLaDA-8B and consistently improves performance across diverse dLLM architectures and parameter scales. It further scales to long contexts and remains orthogonal to mainstream inference optimizations, making it a practical and widely applicable solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑