Diffusion models have been used as priors for solving inverse problems. However, existing approaches typically overlook side information that could significantly improve reconstruction quality, especially in severely ill-posed settings. In this work, we propose a novel framework that incorporates side information into existing diffusion-based inverse problem solvers via inference-time search, in a plug-and-play, training-free manner. Through extensive experiments across a range of inverse problems, including inpainting, super-resolution, and several deblurring tasks, and across multiple diffusion-based inverse problem solvers (DPS, DAPS, and MPGD), we show that augmenting each solver with our framework consistently improves the quality of the reconstructions over the corresponding original method. To demonstrate the generality of our approach, we consider diverse forms of side information, including reference images, textual descriptions, and anatomical MRI scans. The code is available at this \href{https://github.com/mahdi-farahbakhsh/DISS}{repository}\footnote{https://github.com/mahdi-farahbakhsh/DISS}.
Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation
Department of Electrical and Electronic Engineering, Southern University of Science and Technology(南方科技大学电子与电气工程系);Undergraduate School of Artificial Intelligence, Shenzhen Polytechnic University(深圳理工大学人工智能本科生学院);Huawei Technologies Co., Ltd.(华为技术有限公司);Pengcheng Lab(鹏城实验室)
Bayesian Diffusion Models for 3D Shape Reconstruction
University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);University of California, San Diego(加利福尼亚大学圣迭戈分校);Tsinghua University(清华大学)
Gradient-Guided Conditional Diffusion Models for Private Image Reconstruction: Analyzing Adversarial Impacts of Differential Privacy and Denoising
School of Computer and Big Data, Minjiang University(闽江学院计算机与大数据学院);School of Information, Renmin University of China(中国人民大学信息学院);RMIT University(皇家墨尔本理工大学);Victoria University(维多利亚大学)
Dual-domain Multi-path Self-supervised Diffusion Model for Accelerated MRI Reconstruction
Department of Radiology, Northwestern University(放射科,西北大学);Department of Biomedical Engineering, Huazhong University of Science and Technology(生物医学工程系,华中科技大学)
Conditional Diffusion Posterior Alignment for Sparse-View CT Reconstruction
Swiss Data Science Center (SDSC) in Paul Scherrer Institute (PSI)(瑞士数据科学中心(SDSC)在保罗·舍勒研究所(PSI));Swiss Data Science Center (SDSC) and ETH Zurich(瑞士数据科学中心(SDSC)和苏黎世联邦理工学院);Swiss Data Science Center (SDSC) and Paul Scherrer Institute (PSI)(瑞士数据科学中心(SDSC)和保罗·舍勒研究所(PSI))
eBIRD: Event-based Intensity Image Reconstruction Using Controllable Diffusion Models
Institute of Engineering Sciences, Universidad de O’Higgins(奥伊金斯大学工程科学学院);The Iniciativa de Datos e Inteligencia Artificial, University of Chile(智利大学数据与人工智能倡议)
Product-of-Gaussian-Mixture Diffusion Models for Joint Nonlinear MRI Reconstruction
Graz University of Technology(格拉茨技术大学);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);Biomedical Imaging Group and Center for Biomedical Imaging(生物医学成像组和生物医学成像中心)
Fast Controllable Diffusion Models for Undersampled MRI Reconstruction
School of Electrical Engineering and Computer Science, University of Queensland(昆士兰大学电气工程与计算机科学学院);School of Mathematics and Physics, University of Queensland(昆士兰大学数学与物理学院)
Iterative Diffusion-Refined Neural Attenuation Fields for Multi-Source Stationary CT Reconstruction: NAF Meets Diffusion Model
School of Information Engineering, Nanchang University(南昌大学信息工程学院);School of Mathematics and Computer Sciences, Nanchang University(南昌大学数学与计算机科学学院);School of Biomedical Engineering, Sun Yat-sen University(中山大学生物医学工程学院);Laboratory of Image Science and Technology, School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Southeast University(东南大学计算机科学与工程学院图像科学与技术实验室,以及教育部新一代人工智能技术及其交叉应用重点实验室)
DVG-Diffusion: Dual-View Guided Diffusion Model for CT Reconstruction from X-Rays
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学)
SeisFusion: Constrained Diffusion Model with Input Guidance for 3D Seismic Data Interpolation and Reconstruction
Key Laboratory of Earth Exploration and Information Techniques of Education Ministry, College of Geophysics, Chengdu University of Technology(成都理工大学地球物理学院 教育部地球勘探与信息技术重点实验室);College of Computer Science and Cyber Security, Chengdu University of Technology(成都理工大学计算机与网络安全学院)
Noise-Inspired Diffusion Model for Generalizable Low-Dose CT Reconstruction
Institute of Science and Technology for Brain-inspired Intelligence(脑启发智能科学与技术研究院);Fudan University(复旦大学);School of Biomedical Engineering(生物医学工程学院);Southern Medical University(南方医科大学);School of Computer Science(计算机科学学院);Xi’an Jiaotong University(西安交通大学);School of Life Science and Technology(生命科学与技术学院);MOE Frontiers Center for Brain Science(教育部脑科学前沿中心);Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence (Ministry of Education)(教育部计算神经科学与脑启发智能重点实验室);State Key Laboratory of Brain Function and Disorders(脑功能与疾病国家重点实验室)