Denoising diffusion models have become ubiquitous for generative modeling. The core idea is to transport the data distribution to a Gaussian by using a diffusion. Approximate samples from the data distribution are then obtained by estimating the time-reversal of this diffusion using score matching ideas. We follow here a similar strategy to sample from unnormalized probability densities and compute their normalizing constants. However, the time-reversed diffusion is here simulated by using an original iterative particle scheme relying on a novel score matching loss. Contrary to standard denoising diffusion models, the resulting Particle Denoising Diffusion Sampler (PDDS) provides asymptotically consistent estimates under mild assumptions. We demonstrate PDDS on multimodal and high dimensional sampling tasks.
Department of Computer Science, Aalto University, Finland（阿姆斯特丹大学计算机科学系，芬兰）；Department of Engineering, University of Cambridge, United Kingdom（剑桥大学工程系，英国）；Department of Empirical Inference, Max Planck Institute for Intelligent Systems, Tübingen, Germany（智能系统研究所经验推断系，德国）
Diffusion models have emerged as an expressive family of generative models rivaling GANs in sample quality and autoregressive models in likelihood scores. Standard diffusion models typically require hundreds of forward passes through the model to generate a single high-fidelity sample. We introduce Differentiable Diffusion Sampler Search (DDSS): a method that optimizes fast samplers for any pre-trained diffusion model by differentiating through sample quality scores. We also present Generalized Gaussian Diffusion Models (GGDM), a family of flexible non-Markovian samplers for diffusion models. …
Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers
School of Mathematical Science, Peking University（北京大学数学科学学院）；Department of Computer Science, University of California, Los Angeles（加利福尼亚大学洛杉矶分校计算机科学系）
A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that…
Improved off-policy training of diffusion samplers
Mila, Université de Montréal（米拉-蒙特利尔大学）；Jagiellonian University（雅盖隆大学）；KAIST（韩国科学技术院）；Ciela Institute（塞拉研究所）；Dreamfold（德里姆福尔德公司）；CIFAR（加拿大高级研究院）；University of Edinburgh（爱丁堡大学）
University of Oxford（牛津大学）；CNRS ENS Ulm（法国国家科学研究中心 巴黎高等师范学院）
We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques developed in chemical physics and exploit our continuous time framework to derive high performance samplers that we show can outperform discrete time methods for discrete data. The continuous time treatment…
Asia University（亚细亚大学）；PKSHA Technology, Inc.（PKSHA科技公司）
Diffusion generative models have emerged as a new challenger to popular deep neural generative models such as GANs, but have the drawback that they often require a huge number of neural function evaluations (NFEs) during synthesis unless some sophisticated sampling strategies are employed. This paper proposes new efficient samplers based on the numerical schemes derived by the familiar Taylor expansion, which directly solves the ODE/SDE of interest. In general, it is not easy to compute the derivatives that are required in higher-order Taylor schemes, but in the case of diffusion models, this …
MaRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Laboratory（湖畔实验室）
On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
Institute of Data Science, The University of Hong Kong（数据科学研究所，香港大学）；School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通讯学院，华中科技大学）；University of California, Berkeley（加州大学伯克利分校）；Simons Institute for the Theory of Computing（Simons计算理论研究所）；Institute of Data Science & School of Computing and Data Science, The University of Hong Kong（数据科学研究所及计算与数据科学学院，香港大学）
Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models. We additionally use precision and recall to compare how well DDPMs…
