arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.25544stat.MLcs.ITcs.LGmath.ITstat.CO
收藏

掩码扩散模型的误差界与最优调度方案:基于因子化近似

Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations

Bocconi University, Department of Decision Sciences and BIDSA(博科尼大学决策科学系和BIDSA)

Hugo Lavenant, Giacomo Zanella

AI总结:

本文研究了掩码扩散模型中计算成本与准确性之间的权衡,通过因子化近似提供误差界,并提出基于数据分布信息轮廓的最优调度方案。

摘要:

最近提出的用于离散数据生成的模型,如掩码扩散模型(MDMs),利用条件独立性近似来减少流行自回归模型(ARMs)的计算成本,但以采样分布中的一些偏差为代价。我们研究由此产生的计算成本与准确性之间的权衡,提供了一般性的误差界(在相对熵中),这些误差界仅依赖于每个迭代生成的平均token数量,并且与数据维度(即序列长度)无关,从而支持了MDMs的实证成功。然后,我们研究通过使用非恒定的调度大小(即在生成过程中变化未掩码token的数量)所获得的增益,并将最优调度确定为数据分布所谓信息轮廓函数的函数,从而允许对调度大小进行原则性的优化。我们直接将方法定义为采样算法,而不使用经典的推导作为时间反转扩散过程,这使我们能够得到简单而透明的证明。

英文摘要:

Recently proposed generative models for discrete data, such as Masked Diffusion Models (MDMs), exploit conditional independence approximations to reduce the computational cost of popular Auto-Regressive Models (ARMs), at the price of some bias in the sampling distribution. We study the resulting computation-vs-accuracy trade-off, providing general error bounds (in relative entropy) that depend only on the average number of tokens generated per iteration and are independent of the data dimensionality (i.e. sequence length), thus supporting the empirical success of MDMs. We then investigate the gain obtained by using non-constant schedule sizes (i.e. varying the number of unmasked tokens during the generation process) and identify the optimal schedule as a function of a so-called information profile of the data distribution, thus allowing for a principled optimization of schedule sizes. We define methods directly as sampling algorithms and do not use classical derivations as time-reversed diffusion processes, leading us to simple and transparent proofs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑