用于扩散语言模型的轨迹级推测解码

中文摘要 AI 辅助

基于扩散的语言模型（dLLMs）通过迭代去噪实现并行令牌生成，但现有解码策略在低置信度下会退化为单令牌生成，严重限制了吞吐量。与自回归模型中推测解码按固定从左到右顺序作用于令牌序列不同，dLLMs需要对去噪轨迹（带有显式位置和解蔽顺序的多令牌更新序列）进行推测。我们开发了一种轨迹级推测框架，该框架通过置信度分层树探索构建草稿去噪轨迹，并通过带双向注意力掩码的分块并行评估对其进行验证。我们的方法进一步引入了块间推测，利用扩散模型的双向结构执行跨块前瞻。我们正式确定了该方法精确的条件，并将轨迹漂移识别为并行度提升的基本代价。基于Fast-dLLM的双缓存基础设施，我们的框架将去噪迭代减少了30%-40%，令牌每步从2.6提升至4.3，在推理和代码基准上实现了比普通dLLMs 7-14倍的加速，比Fast-dLLM提升1.3倍，且准确率变化小于1%。

英文摘要

Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike autoregressive models where speculative decoding operates on token sequences in a fixed left-to-right order, dLLMs require speculating over denoising trajectories-sequences of multi-token updates with explicit positions and unmasking orders. We develop a trajectory-level speculative framework that constructs draft denoising trajectories via confidence-stratified tree exploration and verifies them through blockwise parallel evaluation with bidirectional attention masking. Our method further introduces inter-block speculation, exploiting diffusion models' bidirectional structure to perform cross-block lookahead. We formally characterize when this approach is exact and identify trajectory drift as the fundamental cost of increased parallelism. Building on Fast-dLLM's dual-cache infrastructure, our framework reduces denoising iterations by 30-40% and increases tokens-per-step from 2.6 to 4.3, achieving 7-14x speedup over vanilla dLLMs and 1.3x over Fast-dLLM with less than 1% accuracy change across reasoning and code benchmarks.

发表机构

• Li Auto Inc.（理想汽车）

机构由 AI 辅助整理，请以论文原文为准。