Set Diffusion: 在自回归与扩散之间插值令牌顺序以实现快速灵活的解码
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
AI总结:
提出Set Diffusion模型,通过将似然参数化为灵活位置和长度的令牌集,并设计集因果扩散架构,支持任意顺序解码和KV缓存更新,在数学推理、摘要和无条件生成上优于先前的扩散语言模型。
摘要:
离散扩散模型在质量上相对于自回归(AR)模型稳步提升。然而,这些模型通常受限于固定长度生成,且不支持键值(KV)缓存。块扩散通过从左到右生成令牌块部分地弥合了扩散与AR之间的差距,但其固定大小的顺序块限制了解码灵活性和并行性。在此,我们提出了一类新的语言模型——集扩散,它包含(i)一种在灵活位置、灵活长度的令牌集上分解的似然参数化,以及(ii)一种支持每次推理步骤后KV缓存更新的集因果扩散架构。通过分解为令牌集而非固定大小的块,令牌可以以任意顺序的集合解码,包括滑动窗口集,从而实现更快的推理并支持任意顺序解码。与先前的扩散语言模型相比,集扩散在数学推理、摘要和无条件生成上实现了更好的速度-质量权衡,同时提供了比块扩散更强的填充性能。我们在项目页面上提供了代码、模型权重和博客文章:此 https URL。
英文摘要:
Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by generating token blocks left-to-right, but its fixed-size sequential blocks limit decoding flexibility and parallelism. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/setdlms/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。