Regularization by denoising: Bayesian model and Langevin-within-split Gibbs sampling
Institut Denis Poisson, University of Orléans（德尼·泊松研究所，奥尔良大学）；University of Toulouse, IRIT/INP-ENSEEIHT, CNRS（图卢兹大学，法国国家信息与自动化研究所/国立高等电子、电子技术、计算机与机械学院，法国国家科学研究中心）
Elhadji C. Faye, Mame Diarra Fall, Nicolas Dobigeon
本文提出一种基于去噪正则化（RED）的图像反演贝叶斯框架，并设计了内含Langevin Monte Carlo步骤的分裂Gibbs采样算法，通过去模糊等任务验证了其有效性。
摘要：
本文通过推导去噪正则化（RED）范式的概率对应物，引入了一个用于图像反演的贝叶斯框架。此外，它实现了一种专为从所得后验分布中采样而量身定制的Monte Carlo算法，该算法基于渐近精确数据增强（AXDA）。所提出的算法是分裂Gibbs采样（SGS）的一个近似实例，其中嵌入了一个Langevin Monte Carlo步骤。所提方法被应用于常见的成像任务，如去模糊、图像修复和超分辨率，并通过广泛的数值实验证明了其有效性。这些贡献通过在概率框架内利用数据驱动的正则化策略，推进了成像中的贝叶斯推断。
英文摘要：
This paper introduces a Bayesian framework for image inversion by deriving a probabilistic counterpart to the regularization-by-denoising (RED) paradigm. It additionally implements a Monte Carlo algorithm specifically tailored for sampling from the resulting posterior distribution, based on an asymptotically exact data augmentation (AXDA). The proposed algorithm is an approximate instance of split Gibbs sampling (SGS) which embeds one Langevin Monte Carlo step. The proposed method is applied to common imaging tasks such as deblurring, inpainting and super-resolution, demonstrating its efficacy through extensive numerical experiments. These contributions advance Bayesian inference in imaging by leveraging data-driven regularization strategies within a probabilistic framework.
Lagrange Mathematics and Computing Research Center（拉格朗日数学与计算研究中心）；Ecole Polytechnique（巴黎综合理工学院）；Université Paris-Saclay（巴黎萨克雷大学）；Ecole Normale Supérieure Paris-Saclay（巴黎-萨克雷高等师范学院）
Performing reliable Bayesian inference on a big data scale is becoming a keystone in the modern era of machine learning. A workhorse class of methods to achieve this task are Markov chain Monte Carlo (MCMC) algorithms and their design to handle distributed datasets has been the subject of many works. However, existing methods are not completely either reliable or computationally efficient. In this paper, we propose to fill this gap in the case where the dataset is partitioned and stored on computing nodes within a cluster under a master/slaves architecture. We derive a user-friendly centralise…
Harvard University（哈佛大学）；UC Irvine（加州大学欧文分校）；UBC（不列颠哥伦比亚大学）；Facebook（脸书）；University of Amsterdam（阿姆斯特丹大学）
The Gibbs sampler is one of the most popular algorithms for inference in statistical models. In this paper, we introduce a herding variant of this algorithm, called herded Gibbs, that is entirely deterministic. We prove that herded Gibbs has an $O(1/T)$ convergence rate for models with independent variables and for fully connected probabilistic graphical models. Herded Gibbs is shown to outperform Gibbs in the tasks of image denoising with MRFs and named entity recognition with CRFs. However, the convergence for herded Gibbs for sparsely connected probabilistic graphical models is still an ope…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform [0,1] value, u, and to then update u non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration. This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions. It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories. This is of significance when some variables are updated by…
We propose discrete Langevin proposal (DLP), a simple and scalable gradient-based proposal for sampling complex high-dimensional discrete distributions. In contrast to Gibbs sampling-based methods, DLP is able to update all coordinates in parallel in a single step and the magnitude of changes is controlled by a stepsize. This allows a cheap and efficient exploration in the space of high-dimensional and strongly correlated variables. We prove the efficiency of DLP by showing that the asymptotic bias of its stationary distribution is zero for log-quadratic distributions, and is small for distrib…
National University of Singapore（新加坡国立大学）；Nanjing University（南京大学）
Given a target function $H$ to minimize or a target Gibbs distribution $π_β^0 \propto e^{-βH}$ to sample from in the low temperature, in this paper we propose and analyze Langevin Monte Carlo (LMC) algorithms that run on an alternative landscape as specified by $H^f_{β,c,1}$ and target a modified Gibbs distribution $π^f_{β,c,1} \propto e^{-βH^f_{β,c,1}}$, where the landscape of $H^f_{β,c,1}$ is a transformed version of that of $H$ which depends on the parameters $f,β$ and $c$. While the original Log-Sobolev constant affiliated with $π^0_β$ exhibits exponential dependence on both $β$ and the en…
The problem of Bayesian reduced rank regression is considered in this paper. We propose, for the first time, to use Langevin Monte Carlo method in this problem. A spectral scaled Student prior distrbution is used to exploit the underlying low-rank structure of the coefficient matrix. We show that our algorithms are significantly faster than the Gibbs sampler in high-dimensional setting. Simulation results show that our proposed algorithms for Bayesian reduced rank regression are comparable to the state-of-the-art method where the rank is chosen by cross validation.
Bregman geometry-aware split Gibbs sampling for Bayesian Poisson inverse problems
Institut Denis Poisson（德尼·泊松研究所）；CNRS（法国国家科学研究中心）；University of Tours（图尔大学）；Univ Rouen Normandie（鲁昂诺曼底大学）；INSA Rouen Normandie（鲁昂诺曼底国立应用科学学院）；Normandie Univ（诺曼底大学）；LITIS（信息、信号与图像处理实验室）；University of Toulouse（图卢兹大学）；IRIT（图卢兹计算机研究所）；INP-ENSEEIHT（图卢兹国立综合理工学院-高等电力、电子、计算机、水力与电信学院）；CEA（法国原子能和替代能源委员会）；University of Paris-Saclay（巴黎萨克雷大学）
The Technion - Israel Institute of Technology（以色列理工学院）
Image denoising is a well-known and well studied problem, commonly targeting a minimization of the mean squared error (MSE) between the outcome and the original image. Unfortunately, especially for severe noise levels, such Minimum MSE (MMSE) solutions may lead to blurry output images. In this work we propose a novel stochastic denoising approach that produces viable and high perceptual quality results, while maintaining a small MSE. Our method employs Langevin dynamics that relies on a repeated application of any given MMSE denoiser, obtaining the reconstructed image by effectively sampling f…
Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie
University of Oxford（牛津大学）；Institut Universitaire de France（法国大学研究院）；Heriot-Watt University（赫瑞瓦特大学）；Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
本文针对贝叶斯成像中 Plug & Play (PnP) 先验方法缺乏收敛证明的问题，提出了两种可证明收敛的算法 PnP-ULA 和 PnP-SGD，并建立了详细的收敛保证，适用于图像去模糊、修复和去噪等任务。
MIT（麻省理工学院）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and th…
University of California, Irvine（加州大学尔湾分校）；Google（谷歌）；Yahoo Labs（雅虎实验室）；University of Amsterdam（阿姆斯特丹大学）
Despite having various attractive qualities such as high prediction accuracy and the ability to quantify uncertainty and avoid over-fitting, Bayesian Matrix Factorization has not been widely adopted because of the prohibitive cost of inference. In this paper, we propose a scalable distributed Bayesian matrix factorization algorithm using stochastic gradient MCMC. Our algorithm, based on Distributed Stochastic Gradient Langevin Dynamics, can not only match the prediction accuracy of standard MCMC methods like Gibbs sampling, but at the same time is as fast and simple as stochastic gradient desc…
University of British Columbia（不列颠哥伦比亚大学）；Google Research Brain Team（谷歌研究院大脑团队）；New York University（纽约大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling. We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise. This enables direct comparison of GRBMs with deep generative models, improving evaluation protocols in the RBM literature. Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning…
University of Montreal（蒙特利尔大学）；Microsoft Research Lab（微软研究院实验室）
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Fréchet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin sam…
This paper studies a new and highly efficient Markov chain Monte Carlo (MCMC) methodology to perform Bayesian inference in low-photon imaging problems, with particular attention to situations involving observation noise processes that deviate significantly from Gaussian noise, such as binomial, geometric and low-intensity Poisson noise. These problems are challenging for many reasons. From an inferential viewpoint, low-photon numbers lead to severe identifiability issues, poor stability and high uncertainty about the solution. Moreover, low-photon models often exhibit poor regularity propertie…
NVIDIA（英伟达）；University of Waterloo（滑铁卢大学）；Vector Institute（矢量研究所）
Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually perturbs the data towards a tractable distribution, while the generative model learns to denoise. The complexity of this denoising task is, apart from the data distribution itself, uniquely determined by the diffusion process. We argue that current SGMs employ overly simplistic diffusions, leading to unnecessarily complex denoising processes, which limit generative modeling performance. Based on connections to statistical mechanics, we propose a novel critically-…
Rapid mixing for Gibbs measures in Riemannian manifolds
University of Cambridge（剑桥大学）；Universität Tübingen（蒂宾根大学）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Universitat de Barcelona（巴塞罗那大学）；Politecnico di Milano（米兰理工大学）；Instituto de Ciencias Matemáticas（数学科学研究所）
We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score…
Penalized Nonreversible Langevin for Constrained Sampling
Department of Mathematics, Florida State University, Tallahassee, Florida, United States of America（美利坚合众国佛罗里达州塔拉哈西市佛罗里达州立大学数学系）；FinTech Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, People’s Republic of China（中华人民共和国广东省广州市香港科技大学（广州）金融科技方向）
We develop an efficient sampling method by simulating Langevin dynamics with an artificial force rather than a natural force by using the gradient of the potential energy. The standard technique for sampling following the predetermined distribution such as the Gibbs-Boltzmann one is performed under the detailed balance condition. In the present study, we propose a modified Langevin dynamics violating the detailed balance condition on the transition-probability formulation. We confirm that the numerical implementation of the proposed method actually demonstrates two major beneficial improvement…
The first order derivative of a data density can be estimated efficiently by denoising score matching, and has become an important component in many applications, such as image generation and audio synthesis. Higher order derivatives provide additional local information about the data distribution and enable new applications. Although they can be estimated via automatic differentiation of a learned density model, this can amplify estimation errors and is expensive in high dimensional settings. To overcome these limitations, we propose a method to directly estimate high order derivatives (score…
Norwegian University of Science and Technology（挪威科技大学）
We study the problem of matrix completion in this paper. A spectral scaled Student prior is exploited to favour the underlying low-rank structure of the data matrix. We provide a thorough theoretical investigation for our approach through PAC-Bayesian bounds. More precisely, our PAC-Bayesian approach enjoys a minimax-optimal oracle inequality which guarantees that our method works well under model misspecification and under general sampling distribution. Interestingly, we also provide efficient gradient-based sampling implementations for our approach by using Langevin Monte Carlo. More specifi…
In this paper, we investigate the problem of computing Bayesian estimators using Langevin Monte-Carlo type approximation. The novelty of this paper is to consider together the statistical and numerical counterparts (in a general log-concave setting). More precisely, we address the following question: given $n$ observations in $\mathbb{R}^q$ distributed under an unknown probability $\mathbb{P}_{θ^\star}$ with $θ^\star \in \mathbb{R}^d$ , what is the optimal numerical strategy and its cost for the approximation of $θ^\star$ with the Bayesian posterior mean? To answer this question, we establish …
Department of Mathematics, University of California, Los Angeles, CA 90095（加州大学洛杉矶分校数学系）；Department of Mathematics, University of South Carolina, Columbia, SC 29208（南卡罗来纳大学数学系）
