arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.06330cs.CL
收藏

STDec:基于时空稳定性的解码方法用于dLLMs

STDec: Spatio-Temporal Stability Guided Decoding for dLLMs

Tianjin University(天津大学);Sichuan University(四川大学);Chongqing University(重庆大学)

Yuzhe Chen, Jiale Cao, Xuyang Liu, Jin Xie, Aiping Yang, Yanwei Pang

AI总结:

STDec通过结合空间和时间稳定性,改进dLLM解码,提升吞吐量并保持性能。

摘要:

扩散大语言模型(dLLMs)在快速进展中被视为自回归范式的替代品。然而,大多数dLLM解码器仍采用全局置信度阈值,未显式建模邻近解码状态的局部上下文或预测token ID的时序一致性。为了解决这个问题,我们提出了一种简单的时空稳定性引导解码方法,称为STDec。我们观察到dLLM解码中的强时空稳定性:新解码的token倾向于靠近已解码的邻居,且其预测的ID在多个去噪步骤中往往保持一致。受此稳定性启发,STDec包含空间感知解码和时间感知解码。空间感知解码通过聚合邻近token的解码状态动态生成token自适应阈值。时间感知解码则放松了那些在去噪步骤中预测token ID保持一致的token的解码阈值。STDec是训练免费的,并且与基于缓存的加速方法保持兼容。在文本推理和多模态理解基准上,STDec显著提高了吞吐量,同时保持可比的任务性能得分。值得注意的是,在MBPP与LLaDA上,STDec在保持同等得分的情况下实现了14.17倍的速度提升。主页:https://yzchen02.github.io/STDec.

英文摘要:

Diffusion Large Language Models (dLLMs) have achieved rapid progress, viewed as a promising alternative to the autoregressive paradigm. However, most dLLM decoders still adopt a global confidence threshold, and do not explicitly model local context from neighboring decoded states or temporal consistency of predicted token IDs across steps. To address this issue, we propose a spatio-temporal stability guided decoding approach, named STDec. We observe strong spatio-temporal stability in dLLM decoding: newly decoded tokens tend to lie near decoded neighbors, and their predicted IDs often remain consistent across several denoising steps. Inspired by this stability, our STDec introduces spatial-aware decoding and temporal-aware decoding. The spatial-aware decoding dynamically generates the token-adaptive thresholds by aggregating the decoded states of nearby tokens. The temporal-aware decoding relaxes the decoding thresholds for tokens whose predicted token IDs remain consistent over denoising steps. Our STDec is training-free and remains compatible with cache-based acceleration methods. Across textual reasoning and multimodal understanding benchmarks, STDec achieves improved inference throughput while maintaining comparable generation score. Notably, on MBPP with LLaDA, STDec achieves up to 14.17$\times$ speedup with a comparable score. The homepage is available at https://yzchen02.github.io/STDec.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑