arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.06190cs.LG
收藏

关于强大生成方式的研究:自回归、扩散及其他

On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond

Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Massachusetts Institute of Technology(麻省理工学院)

Chenxiao Yang, Cai Zhou, David Wipf, Zhiyuan Li

AI总结:

本文研究掩码扩散模型(MDM)的非自回归生成能力,提出任意过程生成扩展MDM以支持自校正等功能,理论与实证表明其能解决ARM和普通MDM难以处理的复杂推理问题,助力LLM扩展至编码等领域。

摘要:

扩散语言模型最近成为自回归语言模型的有力替代方案。除了下一个token生成外,它们通过支持并行和任意顺序的token生成,效率更高且更灵活。然而,尽管取得了实证成功,其计算能力和基本局限性仍未被充分理解。在本文中,我们正式研究了掩码扩散模型(MDM)中的非自回归生成是否能够解决自回归模型(ARM)无法解决的问题。我们的结果表明,具有足够大上下文长度的MDM在计算上是通用的,其解码步骤匹配PRAM中的最优并行时间复杂度。但是,当控制其他因素时,MDM任意顺序生成的灵活性并未扩展ARM已能解决的问题范围。为解决这一问题,我们提出了一种新的生成形式,称为任意过程生成,它通过重新掩码、插入和删除token的能力扩展MDM,允许自校正、长度可变编辑和自适应并行。从理论和实证上,我们证明这些能力使其能够扩展到难度显著更高的推理问题,而这些问题对于ARM和普通MDM来说是难以处理的。此外,它们被证明对于对象通过非顺序过程自然演化的生成任务至关重要,这对于将当前大语言模型(LLM)从自然语言扩展到编码和科学等领域至关重要。

英文摘要:

Diffusion language models have recently emerged as a competitive alternative to autoregressive language models. Beyond next-token generation, they are more efficient and flexible by enabling parallel and any-order token generation. However, despite empirical successes, their computational power and fundamental limitations remain poorly understood. In this paper, we formally study whether non-autoregressive generation in Masked Diffusion Models (MDM) enables solving problems beyond the reach of Auto-Regressive Models (ARM). Our results show that MDM with sufficiently large context length is computationally universal with decoding steps matching the optimal parallel time complexity in PRAM. However, when controlling for other factors, MDM's flexibility to generate in any-order does not expand what ARM can already solve. To address this, we propose a new form of generation called any-process generation, which extends MDM with capabilities to remask, insert and delete tokens, allowing self-correction, length-variable editing, and adaptive parallelism. Theoretically and empirically, we demonstrate these capabilities enable scalability to significantly harder reasoning problems that are otherwise intractable for ARM and vanilla MDM. Additionally, they prove essential for generation tasks where objects naturally evolve through non-sequential processes, crucial for extending current LLMs beyond natural language to domains such as coding and science.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑