扩散对偶性，第二章：Ψ-采样器
The Diffusion Duality, Chapter II: $Ψ$-Samplers
AI总结：
本文提出了一种通用的预测-校正采样器，用于离散扩散模型，提升了语言和图像建模的生成质量，并展示了其在训练效率上的优势。
摘要：
离散扩散模型因其自我校正能力在少量步骤生成和引导中表现出色，使其在这些场景中优于自回归或遮蔽扩散模型。然而，随着采样步骤的增加，其采样质量会趋于平缓。我们引入了一类预测-校正（PC）采样器，用于离散扩散，这些方法能够推广先前的方法并适用于任意噪声过程。当与均匀状态扩散结合时，我们的采样器在语言和图像建模上均优于祖先采样，实现了在OpenWebText上的生成困惑度更低，在CIFAR10上的FID/IS分数更优。关键的是，与传统采样器不同，我们的PC方法随着更多采样步骤的增加而持续改进。这些发现质疑了遮蔽扩散是扩散语言模型不可避免未来的假设。除了采样外，我们还开发了一种内存高效的课程学习方法，用于高斯松弛训练阶段，将训练时间减少25%，内存减少33%，同时保持在OpenWebText和LM1B上的困惑度相当，并在下游任务中表现强劲。我们发布了代码、检查点和视频教程：https://s-sahoo.com/duo-ch2
英文摘要：
Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
