arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.00161cs.LG
收藏

一致性扩散语言模型

Consistent Diffusion Language Models

University of California, Berkeley(加州大学伯克利分校)

Hasan Amin, Yuan Gao, Yaser Souri, Subhojit Som, Ming Yin, Rajiv Khanna, Xia Song

AI总结:

提出多路径离散一致性(MPDC)原则,通过训练去噪器在随机桥上的路径不变性,实现单阶段训练的一致性扩散语言模型(CDLM),在文本生成中达到最先进性能。

摘要:

扩散语言模型(DLM)是自回归模型的一个有吸引力的替代方案,因为它们承诺亚线性时间、并行生成,但实际收益仍然难以捉摸,因为高质量样本仍需要数百个细化步骤。在连续域中,沿着概率流ODE的一致性训练是加速扩散的流行方法。对于离散扩散,不存在类似的样本空间ODE,使得直接适应不明确。我们认为正确的离散替代是精确后验桥,即连接任意两个噪声水平的闭式条件分布,这对于包括掩码扩散和均匀扩散在内的广泛损坏是可用的。基于这一观察,我们引入了多路径离散一致性(MPDC),这是一个新原则,它训练去噪器在这些随机桥上期望路径不变,并将其实例化为一致性扩散语言模型(CDLM),这是一个不需要预训练教师模型的单阶段训练框架。我们的CDLM目标将掩码扩散、连续一致性模型以及渐进或离散蒸馏恢复为同一观点的分析极限或经验近似。实验上,CDLM在条件和非条件文本生成上建立了新的最先进水平,在采样预算下始终优于强基线的离散扩散模型,并且通常甚至优于多阶段蒸馏基线,在少步数情况下增益最大。总之,这些结果将CDLM定位为下一代快速、高保真离散生成建模的原则性和可扩展基础。

英文摘要:

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑