在设计学习定义在流形上的分布的方法时，一个重要关注点是减少隐式学习流形的需要，从而使学习能够专注于流形内的数据分布。然而，实现这一目标往往导致计算密集型的解决方案。在本文中，我们提出了一种简单的修改，用于在环境空间中进行去噪分数匹配，以隐式地考虑流形，从而减少学习流形的负担，同时保持计算效率。具体来说，我们提出将分数函数分解为已知部分$ s^{base} $和余数部分$ s - s^{base} $（学习目标），其中前者隐含包含数据流形所在位置的信息。我们推导出已知部分$ s^{base} $的解析形式，包括旋转矩阵上的分布和离散分布等重要情况，并利用这些结果在这些情况下展示了该方法的实用性。
英文摘要：
A major focus in designing methods for learning distributions defined on manifolds is to alleviate the need to implicitly learn the manifold so that learning can concentrate on the data distribution within the manifold. However, accomplishing this often leads to compute-intensive solutions. In this work, we propose a simple modification to denoising score-matching in the ambient space to implicitly account for the manifold, thereby reducing the burden of learning the manifold while maintaining computational efficiency. Specifically, we propose a simple decomposition of the score function into a known component $s^{base}$ and a remainder component $s-s^{base}$ (the learning target), with the former implicitly including information on where the data manifold resides. We derive known components $s^{base}$ in analytical form for several important cases, including distributions over rotation matrices and discrete distributions, and use them to demonstrate the utility of this approach in those cases.
Score-based model research in the last few years has produced state of the art generative models by employing Gaussian denoising score-matching (DSM). However, the Gaussian noise assumption has several high-dimensional limitations, motivating a more concrete route toward even higher dimension PDF estimation in future. We outline this limitation, before extending the theory to a broader family of noising distributions -- namely, the generalised normal distribution. To theoretically ground this, we relax a key assumption in (denoising) score matching theory, demonstrating that distributions whic…
Redwood Center for Theoretical Neuroscience（雷德伍德理论神经科学中心）；University of California Berkeley（加州大学伯克利分校）；Berkeley AI Research（伯克利人工智能研究院）；Helen Wills Neuroscience Institute（海伦·威尔斯神经科学研究所）
Energy-Based Models (EBMs) assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score matching \citep{vincent2011connection} has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these …
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；Renmin University of China（中国人民大学）
Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient due to the intractability of the normalizing constant. For this reason an interest to score matching has increased being independent on the normalizing constant. However, such estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data which is called Denoising Score Matching. In this work we derive analytical expression for the Denoisin…
The Generation Phases of Flow Matching: a Denoising Perspective
ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, Inria, LIP, UMR 5668（里昂大学、国家科学研究中心、克莱尔伯恩大学 Lyon 1、Inria、LIP、UMR 5668）；Technische Universität Berlin（柏林技术大学）；Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP, UMR 5668（Inria、里昂大学、国家科学研究中心、克莱尔伯恩大学 Lyon 1、LIP、UMR 5668）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Tweedie distributions are a special case of exponential dispersion models, which are often used in classical statistics as distributions for generalized linear models. Here, we reveal that Tweedie distributions also play key roles in modern deep learning era, leading to a distribution independent self-supervised image denoising formula without clean reference images. Specifically, by combining with the recent Noise2Score self-supervised image denoising approach and the saddle point approximation of Tweedie distribution, we can provide a general closed-form denoising formula that can be used fo…
Score-based generative models provide state-of-the-art quality for image and audio synthesis. Sampling from these models is performed iteratively, typically employing a discretized series of noise levels and a predefined scheme. In this note, we first overview three common sampling schemes for models trained with denoising score matching. Next, we focus on one of them, consistent annealed sampling, and study its hyper-parameter boundaries. We then highlight a possible formulation of such hyper-parameter that explicitly considers those boundaries and facilitates tuning when using few or a varia…
The first order derivative of a data density can be estimated efficiently by denoising score matching, and has become an important component in many applications, such as image generation and audio synthesis. Higher order derivatives provide additional local information about the data distribution and enable new applications. Although they can be estimated via automatic differentiation of a learned density model, this can amplify estimation errors and is expensive in high dimensional settings. To overcome these limitations, we propose a method to directly estimate high order derivatives (score…
National Tsing Hua University（国立清华大学）；Mediatek Inc.（联发科股份有限公司）
Many existing conditional score-based data generation methods utilize Bayes' theorem to decompose the gradients of a log posterior density into a mixture of scores. These methods facilitate the training procedure of conditional score models, as a mixture of scores can be separately estimated using a score model and a classifier. However, our analysis indicates that the training objectives for the classifier in these methods may lead to a serious score mismatch issue, which corresponds to the situation that the estimated scores deviate from the true ones. Such an issue causes the samples to be …
CEA（法国原子能和替代能源委员会）；Inria（法国国家信息与自动化研究所）；AIM；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎-萨克雷大学）；Université Paris Diderot（巴黎狄德罗大学）；Sorbonne Paris Cité（索邦巴黎西岱大学）
Deep neural networks have proven extremely efficient at solving a wide rangeof inverse problems, but most often the uncertainty on the solution they provideis hard to quantify. In this work, we propose a generic Bayesian framework forsolving inverse problems, in which we limit the use of deep neural networks tolearning a prior distribution on the signals to recover. We adopt recent denoisingscore matching techniques to learn this prior from data, and subsequently use it aspart of an annealed Hamiltonian Monte-Carlo scheme to sample the full posteriorof image inverse problems. We apply this fra…
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；National Research Council Canada（加拿大国家研究委员会）；University of Ottawa（渥太华大学）；HEC Montréal（蒙特利尔高等商学院）
Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space fo…
Iterated Denoising Energy Matching for Sampling from Boltzmann Densities
Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Dreamfold；Université de Montréal（蒙特利尔大学）；University of Oxford（牛津大学）；Ciela Institute（Ciela研究所）；Center for Computational Astrophysics, Flatiron Institute（弗拉铁隆研究所计算天体物理中心）；Jagiellonian University（雅盖隆大学）；CIFAR（加拿大高级研究所）
University of Montreal（蒙特利尔大学）；Microsoft Research Lab（微软研究院实验室）
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin sam…
Nonlinear denoising score matching for enhanced learning of structured distributions
Department of Mathematics（数学系）；Texas State University（德克萨斯州立大学）；Department of Mathematics and Statistics（数学与统计学系）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）；Division of Applied Mathematics（应用数学系）；Brown University（布朗大学）；School of Mathematics（数学学院）；Georgia Institute of Technology（佐治亚理工学院）
Binomial flows: Denoising and flow matching for discrete ordinal data
Division of Applied Mathematics, Brown University, Providence, RI, USA 02912（应用数学系，布朗大学，普罗维德恩，罗德岛州，美国 02912）；Department of Statistical Sciences, University of Toronto, Toronto, ON, Canada M5G 1X6（统计科学系，多伦多大学，多伦多，安大略省，加拿大 M5G 1X6）
Mila（米拉（学习算法研究所））；Université de Montréal（蒙特利尔大学）；ETH Zürich（苏黎世联邦理工学院）；Helmholtz AI（亥姆霍兹人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score…
Hierarchical Discrete Flow Matching for Graph Generation
Department of Computer Science, University of Geneva, Switzerland（瑞士日内瓦大学计算机科学系）；DMML Group, Geneva School for Business administration HES-SO, Switzerland（瑞士日内瓦高等商学院DMML研究组）
University Hospital Bonn（波恩大学医院）；Graz University of Technology（格拉茨技术大学）
In this paper, we propose a unified framework of denoising score-based models in the context of graduated non-convex energy minimization. We show that for sufficiently large noise variance, the associated negative log density -- the energy -- becomes convex. Consequently, denoising score-based models essentially follow a graduated non-convexity heuristic. We apply this framework to learning generalized Fields of Experts image priors that approximate the joint density of noisy images and their associated variances. These priors can be easily incorporated into existing optimization algorithms fo…
Simple and Critical Iterative Denoising: A Recasting of Discrete Diffusion in Graph Generation
Data Mining and Machine Learning group, School for Business Administration (HEG-Geneva), University of Applied Sciences and Arts of Western Switzerland（应用科学与艺术大学西瑞士分校数据挖掘与机器学习小组）；Computer Science Department, University of Geneva, Switzerland（日内瓦大学计算机科学系）
MIT（麻省理工学院）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and th…
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experi…
With the advent of advances in self-supervised learning, paired clean-noisy data are no longer required in deep learning-based image denoising. However, existing blind denoising methods still require the assumption with regard to noise characteristics, such as zero-mean noise distribution and pixel-wise noise-signal independence; this hinders wide adaptation of the method in the medical domain. On the other hand, unpaired learning can overcome limitations related to the assumption on noise characteristics, which makes it more feasible for collecting the training data in real-world scenarios. I…
