Plug-and-play (PnP) methods offer an iterative strategy for solving image restoration (IR) problems in a zero-shot manner, using a learned \textit{discriminative denoiser} as the implicit prior. More recently, a sampling-based variant of this approach, which utilizes a pre-trained \textit{generative diffusion model}, has gained great popularity for solving IR problems through stochastic sampling. The IR results using PnP with a pre-trained diffusion model demonstrate distinct advantages compared to those using discriminative denoisers, \ie improved perceptual quality while sacrificing the data fidelity. The unsatisfactory results are due to the lack of integration of these strategies in the IR tasks. In this work, we propose a novel zero-shot IR scheme, dubbed Reconciling Diffusion Model in Dual (RDMD), which leverages only a \textbf{single} pre-trained diffusion model to construct \textbf{two} complementary regularizers. Specifically, the diffusion model in RDMD will iteratively perform deterministic denoising and stochastic sampling, aiming to achieve high-fidelity image restoration with appealing perceptual quality. RDMD also allows users to customize the distortion-perception tradeoff with a single hyperparameter, enhancing the adaptability of the restoration process in different practical scenarios. Extensive experiments on several IR tasks demonstrate that our proposed method could achieve superior results compared to existing approaches on both the FFHQ and ImageNet datasets.
Equivariant Sampling for Improving Diffusion Model-based Image Restoration
School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC（生物医学工程学院，生命科学与医学系，中国科学技术大学）；MIRACLE Center, Suzhou Institute for Advance Research, USTC（MIRACLE中心，苏州市先进研究院，中国科学技术大学）；Key Laboratory of Intelligent Information Processing of CAS, ICT, CAS（中国科学院智能信息处理重点实验室，ICT，中国科学院）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（中国科学技术大学精密与智能化学国家重点实验室）
Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration
Biomedical Engineering Department, College of Future Technology, Peking University（北京大学未来技术学院生物医学工程系）；Institute of Medical Technology, Peking University Health Science Center（北京大学医学部医学技术研究所）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）
Unifying Heterogeneous Degradations: Uncertainty-Aware Diffusion Bridge Model for All-in-One Image Restoration
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, Guangdong 518107, China（中山大学智能系统工程学院深圳校区）；Department of Frontier Research, Peng Cheng Laboratory, Shenzhen, Guangdong 518055, China（鹏城实验室前沿研究部）；Guangdong Provincial Key Laboratory of Fire Science and Technology, Guangzhou 510006, China（广东省消防科学与技术重点实验室）
University of Maryland（马里兰大学）；New York University（纽约大学）
Standard diffusion models involve an image transform -- adding Gaussian noise -- and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully…
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanyang Technological University（南洋理工大学）
Deep convolutional neural network has demonstrated its capability of learning a deterministic mapping for the desired imagery effect. However, the large variety of user flavors motivates the possibility of continuous transition among different output effects. Unlike existing methods that require a specific design to achieve one particular transition (e.g., style transfer), we propose a simple yet universal approach to attain a smooth control of diverse imagery effects in many low-level vision tasks, including image restoration, image-to-image translation, and style transfer. Specifically, our …
Tsinghua University（清华大学）；Chinese University of Hong Kong（香港中文大学）；University of Rochester（罗切斯特大学）
Deep learning methods have witnessed the great progress in image restoration with specific metrics (e.g., PSNR, SSIM). However, the perceptual quality of the restored image is relatively subjective, and it is necessary for users to control the reconstruction result according to personal preferences or image characteristics, which cannot be done using existing deterministic networks. This motivates us to exquisitely design a unified interactive framework for general image restoration tasks. Under this framework, users can control continuous transition of different objectives, e.g., the percepti…
IRPO: Boosting Image Restoration via Post-training GRPO
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；Technical University of Munich（慕尼黑技术大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）
Nanyang Technological University（南洋理工大学）；Michigan State University（密歇根州立大学）
Image restoration schemes based on the pre-trained deep models have received great attention due to their unique flexibility for solving various inverse problems. In particular, the Plug-and-Play (PnP) framework is a popular and powerful tool that can integrate an off-the-shelf deep denoiser for different image restoration tasks with known observation models. However, obtaining the observation model that exactly matches the actual one can be challenging in practice. Thus, the PnP schemes with conventional deep denoisers may fail to generate satisfying results in some real-world image restorati…
Huya Inc（虎牙公司）；City University of Hong Kong（香港城市大学）；Hunan University（湖南大学）；Tencent AI Lab（腾讯AI实验室）
We propose PD-GAN, a probabilistic diverse GAN for image inpainting. Given an input image with arbitrary hole regions, PD-GAN produces multiple inpainting results with diverse and visually realistic content. Our PD-GAN is built upon a vanilla GAN which generates images based on random noise. During image generation, we modulate deep features of input random noise from coarse-to-fine by injecting an initially restored image and the hole regions in multiple scales. We argue that during hole filling, the pixels near the hole boundary should be more deterministic (i.e., with higher probability tru…
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（多模态人工智能系统国家重点实验室（MAIS），中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Ant Group（蚂蚁集团）；PeopleAI Inc.（PeopleAI公司）
Physically-Grounded Manifold Projection Model for Generalizable Metal Artifact Reduction in Dental CBCT
School of Cyberspace, Hangzhou Dianzi University（电子科技大学信息学院）；Innovation Center for Electronic Design Automation Technology, Hangzhou Dianzi University（电子科技大学电子设计自动化创新中心）；Hangzhou Geriatric Stomatology Hospital, Hangzhou Dental Hospital Group（杭州老年口腔医院，杭州口腔医院集团）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）
Universität Innsbruck（因斯布鲁克大学）；Universität Graz（格拉茨大学）；Research Center for Non Destructive Testing GmbH（非破坏性测试研究中心 GmbH）；Prospective Instruments LK OG（前瞻性仪器 LK OG）
Nanyang Technological University（南洋理工大学）；Delft University of Technology（代尔夫特理工大学）
State-of-the-art image denoisers exploit various types of deep neural networks via deterministic training. Alternatively, very recent works utilize deep reinforcement learning for restoring images with diverse or unknown corruptions. Though deep reinforcement learning can generate effective policy networks for operator selection or architecture search in image restoration, how it is connected to the classic deterministic training in solving inverse problems remains unclear. In this work, we propose a novel image denoising scheme via Residual Recovery using Reinforcement Learning, dubbed R3L. W…
Reprojection-Guided 3D Gaussian Splatting Diffusion for Weakly Supervised Single-Image Normal Estimation
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China（江南大学人工智能与计算机科学学院，中国无锡）；School of Data Science, Lingnan University, Hong Kong, China（岭南大学数据科学学院，中国香港）
MRI super-resolution in ten sampling steps using a diffusion bridge model
The University of Chicago（芝加哥大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）；Mayo Clinic（梅奥诊所）
