离散扩散的单纯形松弛

中文摘要 AI 辅助

用于类别生成的离散扩散模型由一个损坏核定义，该损坏核决定了中间状态空间及相关的反向预测问题。我们研究均匀离散扩散，探讨能否在不改变基础类别损坏过程的前提下，丰富其训练目标和反向转移。我们提出Simplax，一种精确的狄利克雷-类别增强方法，该方法将每个损坏的类别状态与一个辅助单纯形值变量耦合，同时保留原始均匀扩散过程作为其类别边际分布。此增强方法产生了易处理的Rao-Blackwellized反向桥目标及相应的随机反向采样器，同时将损坏的类别状态保留为去噪器输入。实验表明，Simplax在无条件OpenWebText生成任务中改善了生成困惑度-熵的权衡；在数独任务中，仅在30线索谜题上训练的模型，在所有评估的线索密度（包括可唯一求解的最少17线索场景）下，均达到了对比方法中的最高准确率，且在无条件生成中也达到了最高有效性。

英文摘要

Discrete diffusion models for categorical generation are defined by a corruption kernel, which determines the intermediate state space and the associated reverse prediction problem. Within this family, uniform diffusion has been extensively developed, with recent work connecting its categorical corruption process to continuous representations and dynamics. Motivated by this view, we ask whether uniform discrete diffusion can be augmented with an explicit continuous state while leaving its categorical corruption process unchanged. We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the uniform diffusion process as its categorical marginal. This augmentation yields a tractable Rao--Blackwellized reverse-bridge objective and a stochastic reverse sampler, while retaining the corrupted categorical state as the denoiser input. Empirically, Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation. On Sudoku, a model trained exclusively on $30$-clue puzzles achieves the highest accuracy among the compared methods across all evaluated clue densities, including the minimum uniquely solvable $17$-clue regime, and also achieves the highest validity in unconditional generation.

发表机构

• NTU Singapore（新加坡南洋理工大学）
• The University of Tokyo（东京大学）
• Purdue University（普渡大学）
• Institute for Advanced Intelligence and Computing (IAIC), A*STAR（新加坡科技研究局高级智能与计算研究所）
• Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）

机构由 AI 辅助整理，请以论文原文为准。