The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient due to the intractability of the normalizing constant. For this reason an interest to score matching has increased being independent on the normalizing constant. However, such estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data which is called Denoising Score Matching. In this work we derive analytical expression for the Denoising Score matching using the Kernel Exponential Family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To tackle the computational complexity we use Random Fourier Features based approximation of the kernel function. The analytical expression allows to drop additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the model was tested on various synthetic distributions to study the behaviour of the model in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.
Score-based model research in the last few years has produced state of the art generative models by employing Gaussian denoising score-matching (DSM). However, the Gaussian noise assumption has several high-dimensional limitations, motivating a more concrete route toward even higher dimension PDF estimation in future. We outline this limitation, before extending the theory to a broader family of noising distributions -- namely, the generalised normal distribution. To theoretically ground this, we relax a key assumption in (denoising) score matching theory, demonstrating that distributions whic…
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；Renmin University of China（中国人民大学）
Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and…
Redwood Center for Theoretical Neuroscience（雷德伍德理论神经科学中心）；University of California Berkeley（加州大学伯克利分校）；Berkeley AI Research（伯克利人工智能研究院）；Helen Wills Neuroscience Institute（海伦·威尔斯神经科学研究所）
Energy-Based Models (EBMs) assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score matching \citep{vincent2011connection} has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these …
The first order derivative of a data density can be estimated efficiently by denoising score matching, and has become an important component in many applications, such as image generation and audio synthesis. Higher order derivatives provide additional local information about the data distribution and enable new applications. Although they can be estimated via automatic differentiation of a learned density model, this can amplify estimation errors and is expensive in high dimensional settings. To overcome these limitations, we propose a method to directly estimate high order derivatives (score…
National Tsing Hua University（国立清华大学）；Mediatek Inc.（联发科股份有限公司）
Many existing conditional score-based data generation methods utilize Bayes' theorem to decompose the gradients of a log posterior density into a mixture of scores. These methods facilitate the training procedure of conditional score models, as a mixture of scores can be separately estimated using a score model and a classifier. However, our analysis indicates that the training objectives for the classifier in these methods may lead to a serious score mismatch issue, which corresponds to the situation that the estimated scores deviate from the true ones. Such an issue causes the samples to be …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Tweedie distributions are a special case of exponential dispersion models, which are often used in classical statistics as distributions for generalized linear models. Here, we reveal that Tweedie distributions also play key roles in modern deep learning era, leading to a distribution independent self-supervised image denoising formula without clean reference images. Specifically, by combining with the recent Noise2Score self-supervised image denoising approach and the saddle point approximation of Tweedie distribution, we can provide a general closed-form denoising formula that can be used fo…
Iterated Denoising Energy Matching for Sampling from Boltzmann Densities
Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Dreamfold；Université de Montréal（蒙特利尔大学）；University of Oxford（牛津大学）；Ciela Institute（Ciela研究所）；Center for Computational Astrophysics, Flatiron Institute（弗拉铁隆研究所计算天体物理中心）；Jagiellonian University（雅盖隆大学）；CIFAR（加拿大高级研究所）
TOOPLOOX（TOOPLOOX 公司）；Wrocław University of Science and Technology（弗罗茨瓦夫科技大学）
Image quality is a nebulous concept with different meanings to different people. To quantify image quality a relative difference is typically calculated between a corrupted image and a ground truth image. But what metric should we use for measuring this difference? Ideally, the metric should perform well for both natural and scientific images. The structural similarity index (SSIM) is a good measure for how humans perceive image similarities, but is not sensitive to differences that are scientifically meaningful in microscopy. In electron and super-resolution microscopy, the Fourier Ring Corre…
Nonlinear denoising score matching for enhanced learning of structured distributions
Department of Mathematics（数学系）；Texas State University（德克萨斯州立大学）；Department of Mathematics and Statistics（数学与统计学系）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）；Division of Applied Mathematics（应用数学系）；Brown University（布朗大学）；School of Mathematics（数学学院）；Georgia Institute of Technology（佐治亚理工学院）
CEA（法国原子能和替代能源委员会）；Inria（法国国家信息与自动化研究所）；AIM；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎-萨克雷大学）；Université Paris Diderot（巴黎狄德罗大学）；Sorbonne Paris Cité（索邦巴黎西岱大学）
Deep neural networks have proven extremely efficient at solving a wide rangeof inverse problems, but most often the uncertainty on the solution they provideis hard to quantify. In this work, we propose a generic Bayesian framework forsolving inverse problems, in which we limit the use of deep neural networks tolearning a prior distribution on the signals to recover. We adopt recent denoisingscore matching techniques to learn this prior from data, and subsequently use it aspart of an annealed Hamiltonian Monte-Carlo scheme to sample the full posteriorof image inverse problems. We apply this fra…
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；National Research Council Canada（加拿大国家研究委员会）；University of Ottawa（渥太华大学）；HEC Montréal（蒙特利尔高等商学院）
Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space fo…
Score-based generative models provide state-of-the-art quality for image and audio synthesis. Sampling from these models is performed iteratively, typically employing a discretized series of noise levels and a predefined scheme. In this note, we first overview three common sampling schemes for models trained with denoising score matching. Next, we focus on one of them, consistent annealed sampling, and study its hyper-parameter boundaries. We then highlight a possible formulation of such hyper-parameter that explicitly considers those boundaries and facilitates tuning when using few or a varia…
The Generation Phases of Flow Matching: a Denoising Perspective
ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, Inria, LIP, UMR 5668（里昂大学、国家科学研究中心、克莱尔伯恩大学 Lyon 1、Inria、LIP、UMR 5668）；Technische Universität Berlin（柏林技术大学）；Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP, UMR 5668（Inria、里昂大学、国家科学研究中心、克莱尔伯恩大学 Lyon 1、LIP、UMR 5668）
University of Montreal（蒙特利尔大学）；Microsoft Research Lab（微软研究院实验室）
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin sam…
We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score…
Binomial flows: Denoising and flow matching for discrete ordinal data
Division of Applied Mathematics, Brown University, Providence, RI, USA 02912（应用数学系，布朗大学，普罗维德恩，罗德岛州，美国 02912）；Department of Statistical Sciences, University of Toronto, Toronto, ON, Canada M5G 1X6（统计科学系，多伦多大学，多伦多，安大略省，加拿大 M5G 1X6）
Hierarchical Discrete Flow Matching for Graph Generation
Department of Computer Science, University of Geneva, Switzerland（瑞士日内瓦大学计算机科学系）；DMML Group, Geneva School for Business administration HES-SO, Switzerland（瑞士日内瓦高等商学院DMML研究组）
Mila（米拉（学习算法研究所））；Université de Montréal（蒙特利尔大学）；ETH Zürich（苏黎世联邦理工学院）；Helmholtz AI（亥姆霍兹人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
MIT（麻省理工学院）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and th…
University Hospital Bonn（波恩大学医院）；Graz University of Technology（格拉茨技术大学）
In this paper, we propose a unified framework of denoising score-based models in the context of graduated non-convex energy minimization. We show that for sufficiently large noise variance, the associated negative log density -- the energy -- becomes convex. Consequently, denoising score-based models essentially follow a graduated non-convexity heuristic. We apply this framework to learning generalized Fields of Experts image priors that approximate the joint density of noisy images and their associated variances. These priors can be easily incorporated into existing optimization algorithms fo…
From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems
College of Science, National University of Defense Technology（国防科技大学理学院）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Image denoising can be described as the problem of mapping from a noisy image to a noise-free image. The best currently available denoising methods approximate this mapping with cleverly engineered algorithms. In this work we attempt to learn this mapping directly with plain multi layer perceptrons (MLP) applied to image patches. We will show that by training on large image databases we are able to outperform the current state-of-the-art image denoising methods. In addition, our method achieves results that are superior to one type of theoretical bound and goes a large way toward closing the g…
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experi…
