扩散中的置信度极限

中文摘要 AI 辅助

离散扩散，包括重新掩蔽和均匀状态采样器，通过每步写入多个令牌位置来生成序列，每个位置从各自的分布中抽取，并选择从这些相同分布中写入哪些位置。对于一般感兴趣的领域（像素、音素或单词），令牌之间存在固有的依赖关系。我们证明，只有当写入的位置在给定已固定令牌的条件下条件独立时，一步才匹配训练分布；任何逐位置分布的乘积都无法匹配一个依赖组；并且逐位置分布并不能决定一个组是否依赖：两个联合分布可以具有相同的逐位置边际，但在哪些值组合出现上有所不同。在ScanAndAdd上，这是一个联合分布以闭式形式可用的合成任务，我们验证了置信度排序写入的每两个或更多未确定位置的组都是依赖的，并测量生成的分布为采样噪声下限总变差的29倍，而每个样本的指标为1.0。

英文摘要

Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution and choosing which positions to write from those same distributions. For domains of general interest (pixels, phonemes, or words) there are inherent dependencies between tokens. We show that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that per-position distributions do not determine whether a group is dependent: two joint distributions can have identical per-position marginals while differing in which combinations of values occur. On ScanAndAdd, a synthetic task whose joint distribution is available in closed form, we verify that every group of two or more undetermined positions a confidence ranking writes is dependent, and measure the generated distribution to be $29\times$ the sampling-noise floor total variation while per-sample metrics are $1.0$.

发表机构

• Apple（苹果公司）

机构由 AI 辅助整理，请以论文原文为准。