While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level. Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels. Our method generates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets. Our implementation is available at https://github.com/ruiqigao/recovery_likelihood.
Stanford University（斯坦福大学）；University of Edinburgh（爱丁堡大学）
Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion…
Diffusion models achieve state-of-the-art performance in various generation tasks. However, their theoretical foundations fall far behind. This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. Our result provides sample complexity bounds for distribution estimation using diffusion models. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. Furthermore, the generated distribution based o…
Fast and Robust Likelihood-Guided Diffusion Posterior Sampling with Amortized Variational Inference
Institute of Foundation Models, MBZUAI（基础模型研究所，MBZUAI）；Huawei Lagrange Mathematics and Computing Research Center, Paris, France（华为拉格朗日数学与计算研究中心，法国巴黎）；Mohamed Bin Zayed University of AI, Chair of the ML Department, EPITA, Laboratoire Recherche de l'EPITA（穆罕默德·本·扎耶德人工智能大学，机器学习系主任，EPITA，EPITA研究实验室）
We study the information-theoretic lower bound of the sample complexity of the correct recovery of diffusion network structures. We introduce a discrete-time diffusion model based on the Independent Cascade model for which we obtain a lower bound of order $Ω(k \log p)$, for directed graphs of $p$ nodes, and at most $k$ parents per node. Next, we introduce a continuous-time diffusion model, for which a similar lower bound of order $Ω(k \log p)$ is obtained. Our results show that the algorithm of Pouget-Abadie et al. is statistically optimal for the discrete-time regime. Our work also opens the …
Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts
Saarland University（萨尔兰大学）；Helmholtz AI（亥姆霍兹人工智能研究所）；Technical University of Munich（慕尼黑技术大学）；CISPA Helmholtz Center for Information Security（亥姆霍兹信息安全部分研究所）；MPI for Intelligent Systems, Tübingen（图宾根智能系统研究所）
The generalized partial credit model (GPCM) is a popular polytomous IRT model that has been widely used in large-scale educational surveys and health care services. Same as other IRT models, GPCM can be estimated via marginal maximum likelihood estimation (MMLE) and Markov chain Monte Carlo (MCMC) methods. While studies comparing MMLE and MCMC as estimation methods for other polytomous IRT models such as the nominal response model and the graded response model exist in literature, no studies have compared how well MMLE and MCMC recover the model parameters of GPCM. In the current study, a comp…
Center for Data Science（数据科学中心）；Peking University（北京大学）；Guanghua School of Management（光华管理学院）；Department of Statistics and Data Science（统计与数据科学系）；Tsinghua University（清华大学）
University of Montreal（蒙特利尔大学）；McGill University（麦吉尔大学）；Mila（米拉研究所）
Diffusion models are recent state-of-the-art methods for image generation and likelihood estimation. In this work, we generalize continuous-time diffusion models to arbitrary Riemannian manifolds and derive a variational framework for likelihood estimation. Computationally, we propose new methods for computing the Riemannian divergence which is needed in the likelihood estimation. Moreover, in generalizing the Euclidean case, we prove that maximizing this variational lower-bound is equivalent to Riemannian score matching. Empirically, we demonstrate the expressive power of Riemannian diffusion…
Whereas diverse variations of diffusion models exist, extending the linear diffusion into a nonlinear diffusion process is investigated by very few works. The nonlinearity effect has been hardly understood, but intuitively, there would be promising diffusion patterns to efficiently train the generative distribution towards the data distribution. This paper introduces a data-adaptive nonlinear diffusion process for score-based diffusion models. The proposed Implicit Nonlinear Diffusion Model (INDM) learns by combining a normalizing flow and a diffusion process. Specifically, INDM implicitly con…
Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusin…
We address the problem of likelihood based inference for correlated diffusion processes using Markov chain Monte Carlo (MCMC) techniques. Such a task presents two interesting problems. First, the construction of the MCMC scheme should ensure that the correlation coefficients are updated subject to the positive definite constraints of the diffusion matrix. Second, a diffusion may only be observed at a finite set of points and the marginal likelihood for the parameters based on these observations is generally not available. We overcome the first issue by using the Cholesky factorisation on the d…
This paper provides closed-form expansions for the log-likelihood function of multivariate diffusions sampled at discrete time intervals. The coefficients of the expansion are calculated explicitly by exploiting the special structure afforded by the diffusion model. Examples of interest in financial statistics and Monte Carlo evidence are included, along with the convergence of the expansion to the true likelihood function.
Statistical inference for discretely observed jump-diffusion processes is a complex problem which motivates new methodological challenges. Thus existing approaches invariably resort to time-discretisations which inevitably lead to approximations in inference. In this paper, we give the first general collection of methodologies for exact (in this context meaning discretisation-free) likelihood-based inference for discretely observed finite activity jump-diffusions. The only sources of error involved are Monte Carlo error and convergence of EM or MCMC algorithms. We shall introduce both frequent…
LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models
Shenzhen University（深圳大学）；Hong Kong University of Science（香港科学大学）；Guangdong Laboratory of Artificial Intelligence（广东省人工智能与数字经济实验室）；University of Science（科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Carleton University（卡尔顿大学）；Southeast University（东南大学）
Detection and Imputation based Two-Stage Denoising Diffusion Power System Measurement Recovery under Cyber-Physical Uncertainties
State Key Laboratory of Advanced Electromagnetic Technology, School of Electrical and Electronic Engineering, Huazhong University of Science and Technology（华中科技大学电气与电子工程学院先进电磁技术国家重点实验室）；Department of Electrical Engineering and Computer Science, Lassonde School of Engineering, York University（约克大学拉森德工程学院电气工程与计算机科学系）
This paper concerns the use of the expectation-maximisation (EM) algorithm for inference in partially observed diffusion processes. In this context, a well known problem is that all except a few diffusion processes lack closed-form expressions of the transition densities. Thus, in order to estimate efficiently the EM intermediate quantity we construct, using novel techniques for unbiased estimation of diffusion transition densities, a random weight fixed-lag auxiliary particle smoother, which avoids the well known problem of particle trajectory degeneracy in the smoothing mode. The estimator i…
Toyota Technologicical Institute at Chicago (TTIC)（芝加哥丰田技术研究院（TTIC））
This paper gives direct derivations of the differential equations and likelihood formulas of diffusion models assuming only knowledge of Gaussian distributions. A VAE analysis derives both forward and backward stochastic differential equations (SDEs) as well as non-variational integral expressions for likelihood formulas. A score-matching analysis derives the reverse diffusion ordinary differential equation (ODE) and a family of reverse-diffusion SDEs parameterized by noise level. The paper presents the mathematics directly with attributions saved for a final section.
ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Department of Computer Science, Oklahoma University（俄克拉荷马大学计算机科学系）；School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA（普渡大学电气与计算机工程学院）
In this study, we develop a deterministic nonlinear filtering algorithm based on a high-dimensional version of Kitagawa (1987) to evaluate the likelihood function of models that allow for stochastic volatility and jumps whose arrival intensity is also stochastic. We show numerically that the deterministic filtering method is precise and much faster than the particle filter, in addition to yielding a smooth function over the parameter space. We then find the maximum likelihood estimates of various models that include stochastic volatility, jumps in the returns and variance, and also stochastic …
We have observed an interesting, yet unexplained, phenomenon: Semidefinite programming (SDP) based relaxations of maximum likelihood estimators (MLE) tend to be tight in recovery problems with noisy data, even when MLE cannot exactly recover the ground truth. Several results establish tightness of SDP based relaxations in the regime where exact recovery from MLE is possible. However, to the best of our knowledge, their tightness is not understood beyond this regime. As an illustrative example, we focus on the generalized Procrustes problem.
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；Renmin University of China（中国人民大学）
Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and…
This paper describes a fast algorithm for recovering low-rank matrices from their linear measurements contaminated with Poisson noise: the Poisson noise Maximum Likelihood Singular Value thresholding (PMLSV) algorithm. We propose a convex optimization formulation with a cost function consisting of the sum of a likelihood function and a regularization function which the nuclear norm of the matrix. Instead of solving the optimization problem directly by semi-definite program (SDP), we derive an iterative singular value thresholding algorithm by expanding the likelihood function. We demonstrate t…
Our work is focused on the joint sparsity recovery problem where the common sparsity pattern is corrupted by Poisson noise. We formulate the confidence-constrained optimization problem in both least squares (LS) and maximum likelihood (ML) frameworks and study the conditions for perfect reconstruction of the original row sparsity and row sparsity pattern. However, the confidence-constrained optimization problem is non-convex. Using convex relaxation, an alternative convex reformulation of the problem is proposed. We evaluate the performance of the proposed approach using simulation results on …
Ecole Polytechnique Federale de Lausanne (EPFL)（洛桑联邦理工学院）
We cast the problem of source localization on graphs as the simultaneous problem of sparse recovery and diffusion kernel learning. An l1 regularization term enforces the sparsity constraint while we recover the sources of diffusion from a single snapshot of the diffusion process. The diffusion kernel is estimated by assuming the process to be as generic as the standard heat diffusion. We show with synthetic data that we can concomitantly learn the diffusion kernel and the sources, given an estimated initialization. We validate our model with cholera mortality and atmospheric tracer diffusion d…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We introduce the {\it diffusion $K$-means} clustering method on Riemannian submanifolds, which maximizes the within-cluster connectedness based on the diffusion distance. The diffusion $K$-means constructs a random walk on the similarity graph with vertices as data points randomly sampled on the manifolds and edges as similarities given by a kernel that captures the local geometry of manifolds. The diffusion $K$-means is a multi-scale clustering tool that is suitable for data with non-linear and non-Euclidean geometric features in mixed dimensions. Given the number of clusters, we propose a po…
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering（计算机科学与工程学院）；South China University of Technology（华南理工大学）；School of Future Technology（未来技术学院）；University of Oxford（牛津大学）；Pengcheng Laboratory（鹏城实验室）；Department of Computer Science（计算机科学系）
Data \& Decision Sciences Technion-Israel Institute of Technology Haifa 3200003, Israel orcidID: 0009-0008-8304-1188；Data \& Decision Sciences Technion-Israel Institute of Technology Haifa 3200003, Israel orcidID: 0000-0002-7028-661X；School of Information Technology
Consider the problem of learning the drift coefficient of a $p$-dimensional stochastic differential equation from a sample path of length $T$. We assume that the drift is parametrized by a high-dimensional vector, and study the support recovery problem when both $p$ and $T$ can tend to infinity. In particular, we prove a general lower bound on the sample-complexity $T$ by using a characterization of mutual information as a time integral of conditional variance, due to Kadota, Zakai, and Ziv. For linear stochastic differential equations, the drift coefficient is parametrized by a $p\times p$ ma…
ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；The Chinese University of Hong Kong（香港中文大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学深圳分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
RAEDiff: Denoising Diffusion Probabilistic Models Based Reversible Adversarial Examples Self-Generation and Self-Recovery
The School of Cyberspace Security, Hainan University（海南大学网络空间安全学院）；Science and Technology Communication Department of Hainan Provincial Public Security Department（海南省公安厅科技通信处）
