arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.26432cs.LGcs.AI
收藏

AdaBlock-dLLM: 通过自适应块大小实现语义感知的扩散语言模型推理

AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size

Imperial College London(帝国理工学院);Institute of Science Tokyo(东京科学大学)

Guanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, Hongxiang Fan

AI总结:

AdaBlock-dLLM通过自适应块大小实现语义感知的扩散语言模型推理,提升解码准确性与效率。

摘要:

基于扩散的大型语言模型(dLLM)因其固有的并行解码能力而受到关注,为自回归语言模型提供了有吸引力的替代方案。在各种解码策略中,基于块的半自回归(半AR)方法因其支持键值缓存和良好的准确率-速度权衡而被广泛采用。然而,本文指出了传统半AR解码方法中应用固定块大小的两个根本性局限:i)晚期解码开销,即在当前块外高置信度标记的解屏蔽被不必要的延迟;ii)早期解码错误,即当前块内低置信度标记过早被提交,导致错误标记。本文首次系统地探讨了半AR解码中固定块大小设置的挑战。通过在去噪过程中对置信度动态的统计分析,我们识别出一个波动带(VB)区域,该区域在dLLM解码中编码了局部语义结构,可用于指导自适应块大小。利用这些见解,我们引入了AdaBlock-dLLM,一种无需训练的即插即用调度器,通过在运行时调整块大小来自适应地对齐块边界与语义步骤。在多样化的基准测试中,AdaBlock-dLLM在相同吞吐量预算下实现了高达5.3%的准确性提升。除了推理时间优化外,我们希望我们的语义感知自适应调度方法和基于置信度的分析将启发未来dLLM的训练策略。我们的代码可在https://github.com/lgxi24/AdaBlock-dLLM上获得。

英文摘要:

Diffusion-based large language models (dLLMs) are gaining attention for their inherent capacity for parallel decoding, offering a compelling alternative to autoregressive LLMs. Among various decoding strategies, block-wise semi-autoregressive (semi-AR) approaches are widely adopted due to their support for KV caching and their favorable accuracy-speed trade-off. However, this paper identifies two fundamental limitations in the conventional semi-AR decoding approach that applies a fixed block size: i) late decoding overhead, where the unmasking of high-confidence tokens outside the current block is unnecessarily delayed, and ii) premature decoding error, where low-confidence tokens inside the current block are committed too early, leading to incorrect tokens. This paper presents the first systematic investigation challenging the fixed block size setting in semi-AR decoding. Through a statistical analysis of confidence dynamics during the denoising process, we identify a volatility band (VB) region during dLLM decoding, which encodes local semantic structure and can be used to guide adaptive block sizing. Leveraging these insights, we introduce AdaBlock-dLLM, a training-free, plug-and-play scheduler that adaptively aligns block boundaries with semantic steps by adjusting block size during runtime. Extensive experiments across diverse benchmarks show that AdaBlock-dLLM achieves up to 5.3% accuracy improvement under the same throughput budget. Beyond inference-time optimization, we hope our semantics-aware adaptive scheduling approach and confidence-based analysis will inspire future training strategies for dLLMs. Our code is available at https://github.com/lgxi24/AdaBlock-dLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑