arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.03540cs.LGcs.AI
收藏

基于掩码扩散模型采样的路径规划

Path Planning for Masked Diffusion Model Sampling

Duke University(杜克大学);Atom Bioworks(Atom生物工坊);Mila – Québec AI Institute(魁北克AI研究院);Université de Montréal(蒙特利尔大学);The University of Oxford(牛津大学);Aithyra(Aithyra公司);University of Pennsylvania(宾夕法尼亚大学)

Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Avishek Joey Bose, Alexander Tong, Pranam Chatterjee

AI总结:

本文提出路径规划策略,通过规划和去噪两个阶段提升掩码扩散模型的生成质量,实验表明在多个领域均取得显著改进。

摘要:

任何使用掩码扩散模型(MDMs)生成离散数据的顺序生成方法都为传统自回归模型提供了一个有吸引力的替代方案,特别是在数据缺乏自然因果顺序的领域。然而,当前流行的MDMs在简化掩码推理方面偏离了其成功的连续扩散模型对应物,其中未掩码的token不能被迭代细化——即使存在错误。在本文中,我们通过引入一种新的推理采样策略,即路径规划(P2),来提取MDMs的全部潜力。P2将每个生成步骤分解为两个子阶段:规划和去噪。在P2中,每一步的规划器会选择适当的token进行更新,这些token随后可以使用去噪器进行采样。我们证明P2可以概括现有的所有MDMs采样策略,并通过新的能力(即细化和更新现有未掩码token)显著提高生成质量。我们理论证明P2在数据的对数边际似然上建立了新的扩展证据下界(ELBO)。我们通过一组规划器家族实例化P2,包括:1)自规划,2)BERT-规划,3)训练-规划,其中具有学习规划器的训练-规划在多个领域中实现了MDMs的最先进生成性能。具体而言,仅使用P2推理,我们观察到在蛋白质序列折叠性方面相对提高了22%,在RNA序列pLDDT方面提高了8%,在数学推理方面提高了4%,在故事生成(ROUGE得分)方面提高了68%,在代码生成方面(pass@1指标)提高了33%。

英文摘要:

Any order generation of discrete data using masked diffusion models (MDMs) offers a compelling alternative to traditional autoregressive models, especially in domains that lack a natural causal ordering of data. However, current popular MDMs depart from their successful continuous diffusion model counterparts with simplified masked inference wherein unmasked tokens cannot be iteratively refined -- even if there is a mistake. In this paper, we extract the full power of MDMs by introducing a novel inference sampling strategy termed Path Planning (P2) that decomposes each generation step into two sub-stages: planning and denoising. Under P2, the planner at every step selects appropriate tokens that are marked to be updated, which can then be sampled using the denoiser. We demonstrate that P2 generalizes all existing sampling strategies for MDMs and critically enhances generative quality through the new capability of refining and updating existing unmasked tokens. We theoretically prove that P2 establishes a (new) expanded evidence lower bound (ELBO) on the log marginal likelihood of data. We instantiate P2 with a family of planners including: 1.) Self-Planning, 2.) BERT-Planning, and 3.) Trained-Planning with a learned planner leading to SOTA generative performance for MDMs on a suite of domains. Specifically, solely using P2 inference, we observe relative improvements of 22% in protein sequence foldability, 8% in RNA sequence pLDDT, 4% in math reasoning, 68% in story generation (ROUGE score), and 33% in code generation for the challenging pass@1 metric.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑