ParallelBench: 理解扩散大语言模型中并行解码的权衡
ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
AI总结：
针对扩散LLM并行解码导致生成质量下降的问题，通过信息论分析和合成实验揭示其根本限制，并提出首个专门基准ParallelBench，系统评估发现并行解码在现实任务中质量损失严重，且现有策略无法根据任务难度调整并行度。
摘要：
尽管大多数自回归LLM受限于逐词解码，扩散LLM（dLLM）因其通过并行解码大幅加速推理的潜力而日益受到关注。然而，dLLM中的条件独立性假设导致并行解码忽略词元依赖，当这些依赖较强时不可避免地降低生成质量。现有工作大多忽视这些固有挑战，且标准基准（如数学和编程）的评估不足以捕捉并行解码导致的质量下降。为填补这一空白，我们首先对并行解码进行信息论分析。然后，从数据分布和解码策略两个角度，对可解析的合成列表操作进行案例研究，提供量化见解以突出并行解码的根本局限性。基于这些见解，我们提出ParallelBench，这是首个专门为dLLM设计的基准，包含对人类和自回归LLM简单但对dLLM在并行解码下极具挑战的现实任务。利用ParallelBench，我们系统分析了dLLM和自回归LLM，揭示：（i）并行解码下的dLLM在现实场景中可能遭受严重的质量下降，（ii）当前并行解码策略难以根据任务难度调整并行度，从而无法在不牺牲质量的情况下实现有意义的加速。我们的发现强调了迫切需要能够克服当前速度-质量权衡的创新解码方法。我们发布基准以帮助加速真正高效dLLM的开发。
英文摘要：
While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to dramatically accelerate inference through parallel decoding. Despite this promise, the conditional independence assumption in dLLMs causes parallel decoding to ignore token dependencies, inevitably degrading generation quality when these dependencies are strong. However, existing works largely overlook these inherent challenges, and evaluations on standard benchmarks (e.g., math and coding) are not sufficient to capture the quality degradation caused by parallel decoding. To address this gap, we first provide an information-theoretic analysis of parallel decoding. We then conduct case studies on analytically tractable synthetic list operations from both data distribution and decoding strategy perspectives, offering quantitative insights that highlight the fundamental limitations of parallel decoding. Building on these insights, we propose ParallelBench, the first benchmark specifically designed for dLLMs, featuring realistic tasks that are trivial for humans and autoregressive LLMs yet exceptionally challenging for dLLMs under parallel decoding. Using ParallelBench, we systematically analyze both dLLMs and autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can suffer dramatic quality degradation in real-world scenarios, and (ii) current parallel decoding strategies struggle to adapt their degree of parallelism based on task difficulty, thus failing to achieve meaningful speedup without compromising quality. Our findings underscore the pressing need for innovative decoding methods that can overcome the current speed-quality trade-off. We release our benchmark to help accelerate the development of truly efficient dLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
