arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.04567cs.CVcs.LGcs.NEeess.IVstat.ML
收藏

基于深度生成模型的贝叶斯图像重建

Bayesian Image Reconstruction using Deep Generative Models

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

Razvan V Marinescu, Daniel Moyer, Polina Golland

摘要:

机器学习模型通常使用成对的(输入,输出)数据进行端到端和监督设置的训练。例子包括最近的超分辨率方法,训练对(低分辨率,高分辨率)图像。然而,这些端到端方法需要在每次输入中存在分布偏移时重新训练(例如,夜间图像对日光)或相关潜变量(例如,照相机模糊或手部运动)。在这项工作中,我们利用最先进的(SOTA)生成模型(这里是StyleGAN 2)来构建强大的图像先验,从而将贝叶斯定理应用于许多下游重建任务。我们的方法,通过生成模型的贝叶斯重建(BRGM),使用单个预训练的生成器模型来解决不同的图像复原任务,即,超分辨率和绘画,通过将其与不同的前向腐败模型相结合。我们保持生成器模型的权重固定,并通过估计生成重建图像的输入潜在向量的贝叶斯最大后验概率(MAP)估计来重建图像。我们进一步使用变分推理来近似隐向量的后验分布,从中我们采样多个解决方案。我们在三个大型且不同的数据集上展示了BRGM:(i)来自Flick Faces高质量数据集的60,000张图像;(ii)来自MIMIC III的240,000张胸部X光片;(iii)5个脑部MRI数据集的组合集合,其中包含7,329次扫描。在所有三个数据集上,并且没有任何特定于数据集的超参数调整,我们的简单方法产生的性能与当前特定于任务的最先进的超分辨率和图像修复方法相比具有竞争力,同时更具有一般性,并且不需要任何训练。我们的源代码和预训练模型可在线获取:https://razvanmarinescu.github.io/brgm/。

英文摘要:

Machine learning models are commonly trained end-to-end and in a supervised setting, using paired (input, output) data. Examples include recent super-resolution methods that train on pairs of (low-resolution, high-resolution) images. However, these end-to-end approaches require re-training every time there is a distribution shift in the inputs (e.g., night images vs daylight) or relevant latent variables (e.g., camera blur or hand motion). In this work, we leverage state-of-the-art (SOTA) generative models (here StyleGAN2) for building powerful image priors, which enable application of Bayes' theorem for many downstream reconstruction tasks. Our method, Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks, i.e., super-resolution and in-painting, by combining it with different forward corruption models. We keep the weights of the generator model fixed, and reconstruct the image by estimating the Bayesian maximum a-posteriori (MAP) estimate over the input latent vector that generated the reconstructed image. We further use variational inference to approximate the posterior distribution over the latent vectors, from which we sample multiple solutions. We demonstrate BRGM on three large and diverse datasets: (i) 60,000 images from the Flick Faces High Quality dataset (ii) 240,000 chest X-rays from MIMIC III and (iii) a combined collection of 5 brain MRI datasets with 7,329 scans. Across all three datasets and without any dataset-specific hyperparameter tuning, our simple approach yields performance competitive with current task-specific state-of-the-art methods on super-resolution and in-painting, while being more generalisable and without requiring any training. Our source code and pre-trained models are available online: https://razvanmarinescu.github.io/brgm/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑