Diffusion models have emerged as an expressive family of generative models rivaling GANs in sample quality and autoregressive models in likelihood scores. Standard diffusion models typically require hundreds of forward passes through the model to generate a single high-fidelity sample. We introduce Differentiable Diffusion Sampler Search (DDSS): a method that optimizes fast samplers for any pre-trained diffusion model by differentiating through sample quality scores. We also present Generalized Gaussian Diffusion Models (GGDM), a family of flexible non-Markovian samplers for diffusion models. We show that optimizing the degrees of freedom of GGDM samplers by maximizing sample quality scores via gradient descent leads to improved sample quality. Our optimization procedure backpropagates through the sampling process using the reparametrization trick and gradient rematerialization. DDSS achieves strong results on unconditional image generation across various datasets (e.g., FID scores on LSUN church 128x128 of 11.6 with only 10 inference steps, and 4.82 with 20 steps, compared to 51.1 and 14.9 with strongest DDPM/DDIM baselines). Our method is compatible with any pre-trained diffusion model without fine-tuning or re-training required.
MaRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE Solvers
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Laboratory（湖畔实验室）
Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations
Gaoling School of AI, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Ant Group（蚂蚁集团）
Diffusion models have recently shown great promise for generative modeling, outperforming GANs on perceptual quality and autoregressive models at density estimation. A remaining downside is their slow sampling time: generating high quality samples takes many hundreds or thousands of model evaluations. Here we make two contributions to help eliminate this downside: First, we present new parameterizations of diffusion models that provide increased stability when using few sampling steps. Second, we present a method to distill a trained deterministic diffusion sampler, using many steps, into a ne…
Differentiating Metropolis-Hastings to Optimize Intractable Densities
Massachusetts Institute of Technology（麻省理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）；University of Oxford（牛津大学）；JuliaHub；Pumas-AI
Much effort has been put into developing samplers with specific properties, such as producing blue noise, low-discrepancy, lattice or Poisson disk samples. These samplers can be slow if they rely on optimization processes, may rely on a wide range of numerical methods, are not always differentiable. The success of recent diffusion models for image generation suggests that these models could be appropriate for learning how to generate point sets from examples. However, their convolutional nature makes these methods impractical for dealing with scattered data such as point sets. We propose a gen…
Any explicit functional representation $f$ of a density is hampered by two main obstacles when we wish to use it as a generative model: designing $f$ so that sampling is fast, and estimating $Z = \int f$ so that $Z^{-1}f$ integrates to 1. This becomes increasingly complicated as $f$ itself becomes complicated. In this paper, we show that when modeling one-dimensional conditional densities with a neural network, $Z$ can be exactly and efficiently computed by letting the network represent the cumulative distribution function of a target density, and applying a generalized fundamental theorem of …
A fundamental task in machine learning and related fields is to perform inference on Bayesian networks. Since exact inference takes exponential time in general, a variety of approximate methods are used. Gibbs sampling is one of the most accurate approaches and provides unbiased samples from the posterior but it has historically been too expensive for large models. In this paper, we present an optimized, parallel Gibbs sampler augmented with state replication (SAME or State Augmented Marginal Estimation) to decrease convergence time. We find that SAME can improve the quality of parameter estim…
From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training
California Institute of Technology（加州理工学院）；NVIDIA（英伟达）；Zuse Institute Berlin（柏林泽尼克研究所）；dida Datenschmiede GmbH（dida数据隐私公司）；Jagiellonian University（雅盖隆大学）；Mila, Université de Montréal（蒙特利尔大学机器学习研究所）；University of Edinburgh（爱丁堡大学）；CIFAR Fellow, Learning in Machines and Brains（CIFAR Fellow, 机器学习与大脑学习）
CNRS（法国国家科学研究中心）；Univ Grenoble-Alpes（格勒诺布尔阿尔卑斯大学）；Gipsa-lab（Gipsa实验室（格勒诺布尔自动化、信号与图像处理实验室））；France（法国（注：根据上下文，此处“France”是国家，非机构，原句“Thanks: All three authors are with CNRS, Univ Grenoble-Alpes, Gipsa-lab, France.”中“France”是国家，不应作为机构提取，正确机构应为前三个）}]}修正后：{）
Discrete Determinantal Point Processes (DPPs) have a wide array of potential applications for subsampling datasets. They are however held back in some cases by the high cost of sampling. In the worst-case scenario, the sampling cost scales as O(n^3) where n is the number of elements of the ground set. A popular workaround to this prohibitive cost is to sample DPPs defined by low-rank kernels. In such cases, the cost of standard sampling algorithms scales as O(np^2 + nm^2) where m is the (average) number of samples of the DPP (usually m << n) and p the rank of the kernel used to define the DPP …
INRIA Lille — Nord Europe（法国国家信息与自动化研究所里尔-北欧中心）；Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；CRIStAL（里尔信号与自动化信息研究中心）
Determinantal point processes (DPPs) are distributions over sets of items that model diversity using kernels. Their applications in machine learning include summary extraction and recommendation systems. Yet, the cost of sampling from a DPP is prohibitive in large-scale applications, which has triggered an effort towards efficient approximate samplers. We build a novel MCMC sampler that combines ideas from combinatorial geometry, linear programming, and Monte Carlo methods to sample from DPPs with a fixed sample cardinality, also called projection DPPs. Our sampler leverages the ability of the…
University of Science and Technology of China（中国科学技术大学）
Sampling is an important process in many GNN structures in order to train larger datasets with a smaller computational complexity. However, compared to other processes in GNN (such as aggregate, backward propagation), the sampling process still costs tremendous time, which limits the speed of training. To reduce the time of sampling, hardware acceleration is an ideal choice. However, state of the art GNN acceleration proposal did not specify how to accelerate the sampling process. What's more, directly accelerating traditional sampling algorithms will make the structure of the accelerator very…
Iterated Denoising Energy Matching for Sampling from Boltzmann Densities
Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Dreamfold；Université de Montréal（蒙特利尔大学）；University of Oxford（牛津大学）；Ciela Institute（Ciela研究所）；Center for Computational Astrophysics, Flatiron Institute（弗拉铁隆研究所计算天体物理中心）；Jagiellonian University（雅盖隆大学）；CIFAR（加拿大高级研究所）
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab（计算机科学与技术系，人工智能研究院，BNRist中心，THBI实验室）；Tsinghua-Bosch Joint ML Center, Tsinghua University, Beijing, 100084 China（清华大学-博世联合机器学习中心，清华大学，北京，100084中国）；Gaoling School of Artificial Intelligence, Renmin University of China（北京人工智能学院，中国人民大学）；Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China（北京大数据管理与分析方法重点实验室，北京，中国）
