arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.09573cs.LGcs.AI
收藏

Block Diffusion:在自回归语言模型与扩散语言模型之间插值

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

Cornell Tech(康奈尔科技学院);Stanford University(斯坦福大学);Cohere(Cohere公司)

Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov

AI总结:

针对扩散语言模型似然建模弱、仅能固定长度生成的问题,提出可在离散去噪扩散与自回归模型间插值的Block Diffusion模型,支持灵活长度生成、推理效率更高,刷新扩散模型语言建模基准SOTA。

摘要:

扩散语言模型因具备并行生成潜力与可控性,相比自回归模型有独特优势,但在似然建模上表现滞后,且仅能进行固定长度生成。本研究提出一类Block Diffusion(块扩散)语言模型,可在离散去噪扩散模型与自回归模型之间插值。块扩散模型支持灵活长度生成,还通过KV缓存与并行令牌采样提升推理效率,克服了两类方法的核心局限。我们提出了构建高效块扩散模型的方案,包含高效训练算法、梯度方差估计器,以及最小化方差的数据驱动噪声调度。块扩散模型在语言建模基准上刷新了扩散模型的最优性能,还可生成任意长度序列。我们在项目页面https://m-arriola.com/bd3lms提供了代码、模型权重与博客文章。

英文摘要:

Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑