Learning-based single image super-resolution (SISR) methods are continuously showing superior effectiveness and efficiency over traditional model-based methods, largely due to the end-to-end training. However, different from model-based methods that can handle the SISR problem with different scale factors, blur kernels and noise levels under a unified MAP (maximum a posteriori) framework, learning-based methods generally lack such flexibility. To address this issue, this paper proposes an end-to-end trainable unfolding network which leverages both learning-based methods and model-based methods. Specifically, by unfolding the MAP inference via a half-quadratic splitting algorithm, a fixed number of iterations consisting of alternately solving a data subproblem and a prior subproblem can be obtained. The two subproblems then can be solved with neural modules, resulting in an end-to-end trainable, iterative network. As a result, the proposed network inherits the flexibility of model-based methods to super-resolve blurry, noisy images for different scale factors via a single model, while maintaining the advantages of learning-based methods. Extensive experiments demonstrate the superiority of the proposed deep unfolding network in terms of flexibility, effectiveness and also generalizability.
University of Science and Technology of China（中国科学技术大学）；Institute of Advanced Technology（先进技术研究院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Magnetic resonance imaging (MRI) with high resolution (HR) provides more detailed information for accurate diagnosis and quantitative image analysis. Despite the significant advances, most existing super-resolution (SR) reconstruction network for medical images has two flaws: 1) All of them are designed in a black-box principle, thus lacking sufficient interpretability and further limiting their practical applications. Interpretable neural network models are of significant interest since they enhance the trustworthiness required in clinical practice when dealing with medical images. 2) most ex…
Deep learning methods have been successfully applied to various computer vision tasks. However, existing neural network architectures do not per se incorporate domain knowledge about the addressed problem, thus, understanding what the model has learned is an open research topic. In this paper, we rely on the unfolding of an iterative algorithm for sparse approximation with side information, and design a deep learning architecture for multimodal image super-resolution that incorporates sparse priors and effectively utilizes information from another image modality. We develop two deep models per…
Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution
School of Computer Science and Technology and the Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University（计算机科学与技术学院和嵌入式系统与服务计算重点实验室，同济大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，城市大学）；Pengcheng Laboratory（鹏城实验室）；School of Data Science, Lingnan University（数据科学学院，岭大大学）
Hefei Institute of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Nanjing University of Science and Technology（南京理工大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Xi’an Jiaotong University（西安交通大学）
Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-based methods mainly takes the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image, simply ignoring many non-local common characteristics between them. To alleviate this issue, we firstly propose a maximal a posterior (MAP) estimation model for GISR with two types of prior on the HR target image, i.e., local implicit …
Bundesanstalt für Materialforschung und -prüfung (BAM)（德国联邦材料研究与测试研究所）；Technical University of Berlin（柏林工业大学）
This paper presents deep unfolding neural networks to handle inverse problems in photothermal radiometry enabling super resolution (SR) imaging. Photothermal imaging is a well-known technique in active thermography for nondestructive inspection of defects in materials such as metals or composites. A grand challenge of active thermography is to overcome the spatial resolution limitation imposed by heat diffusion in order to accurately resolve each defect. The photothermal SR approach enables to extract high-frequency spatial components based on the deconvolution with the thermal point spread fu…
Blind image super-resolution (BISR) aims to reconstruct a high-resolution image from its low-resolution counterpart degraded by unknown blur kernel and noise. Many deep neural network based methods have been proposed to tackle this challenging problem without considering the image degradation model. However, they largely rely on the training sets and often fail to handle images with unseen blur kernels during inference. Deep unfolding methods have also been proposed to perform BISR by utilizing the degradation model. Nonetheless, the existing deep unfolding methods cannot explicitly solve the …
Xidian University（西安电子科技大学）；West Virginia University（西弗吉尼亚大学）
Deep neural networks (DNNs) based methods have achieved great success in single image super-resolution (SISR). However, existing state-of-the-art SISR techniques are designed like black boxes lacking transparency and interpretability. Moreover, the improvement in visual quality is often at the price of increased model complexity due to black-box design. In this paper, we present and advocate an explainable approach toward SISR named model-guided deep unfolding network (MoG-DUN). Targeting at breaking the coherence barrier, we opt to work with a well-established image prior named nonlocal auto-…
Physics-Guided Deep Unfolding for Blind Cross-Sensor Spectral Super-Resolution via Learning the Spectral Transformation Function
Center for Geo-Spatial Information, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（地理信息中心，深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Engineering Laboratory of Ocean Environmental Big Data Analysis and Application（深圳海洋环境大数据分析与应用工程实验室）
National Laboratory of Pattern Recognition（模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Chinese Academy of Sciences, Artificial Intelligence Research（中国科学院人工智能创新研究院）
Previous methods decompose blind super resolution (SR) problem into two sequential steps: \textit{i}) estimating blur kernel from given low-resolution (LR) image and \textit{ii}) restoring SR image based on estimated kernel. This two-step solution involves two independently trained models, which may not be well compatible with each other. Small estimation error of the first step could cause severe performance drop of the second one. While on the other hand, the first step can only utilize limited information from LR image, which makes it difficult to predict highly accurate blur kernel. Toward…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Deep neural networks (DNN) have achieved great success in image restoration. However, most DNN methods are designed as a black box, lacking transparency and interpretability. Although some methods are proposed to combine traditional optimization algorithms with DNN, they usually demand pre-defined degradation processes or handcrafted assumptions, making it difficult to deal with complex and real-world applications. In this paper, we propose a Deep Generalized Unfolding Network (DGUNet) for image restoration. Concretely, without loss of interpretability, we integrate a gradient estimation strat…
Real-world low-light images suffer from two main degradations, namely, inevitable noise and poor visibility. Since the noise exhibits different levels, its estimation has been implemented in recent works when enhancing low-light images from raw Bayer space. When it comes to sRGB color space, the noise estimation becomes more complicated due to the effect of the image processing pipeline. Nevertheless, most existing enhancing algorithms in sRGB space only focus on the low visibility problem or suppress the noise under a hypothetical noise level, leading them impractical due to the lack of robus…
Combined Dictionary Unfolding Network with Gradient-Adaptive Fidelity for Transferable Multi-Source Fusion
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；College of Surveying and Geoinformatics, Tongji University（同济大学测绘与地理信息学院）；School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）
Hyperspectral images are crucial for many research works. Spectral super-resolution (SSR) is a method used to obtain high spatial resolution (HR) hyperspectral images from HR multispectral images. Traditional SSR methods include model-driven algorithms and deep learning. By unfolding a variational method, this paper proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior, resulting in physically interpretable networks. Unlike the fully data-driven CNN, auxiliary spectral response function (SRF) is utilized to guide CNNs to group the bands with spec…
SlowFast-SCI: Slow-Fast Deep Unfolding Learning for Spectral Compressive Imaging
Harvard University（哈佛大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shanghai Jiaotong University（上海交通大学）；City University of Hong Kong（香港城市大学）；Harbin Institute of Technology, Weihai（哈尔滨工业大学（威海））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Pengcheng Laboratory（鹏城实验室）；Dalian University of Technology（大连理工大学）
Deep learning for image super-resolution (SR) has been investigated by numerous researchers in recent years. Most of the works concentrate on effective block designs and improve the network representation but lack interpretation. There are also iterative optimization-inspired networks for image SR, which take the solution step as a whole without giving an explicit optimization step. This paper proposes an unfolding iterative shrinkage thresholding algorithm (ISTA) inspired network for interpretable image SR. Specifically, we analyze the problem of image SR and propose a solution based on the I…
Beyond Unfolding: 60x Faster One-Stage Unmixing for Closely-Spaced Infrared Small Targets
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（中国科学院西安光学精密机械研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Astronautical Systems Engineering（北京航天自动控制研究所）；VCIP, College of Computer Science, Nankai University（南开大学计算机科学学院可视化与计算机图像处理实验室）；NKIARI（未提及具体中文名称）
Deep LoRA-Unfolding Networks for Image Restoration
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术学院，哈尔滨工业大学（深圳））；Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（图像通信与网络工程院，上海交通大学）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen)（电子与信息工程学院，哈尔滨工业大学（深圳））
Deep Unfolding Multi-modal Image Fusion Network via Attribution Analysis
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；ETH Zürich（苏黎世联邦理工学院）；School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；University of Maryland, College Park（马里兰大学帕克分校）
We present a solution for the goal of extracting a video from a single motion blurred image to sequentially reconstruct the clear views of a scene as beheld by the camera during the time of exposure. We first learn motion representation from sharp videos in an unsupervised manner through training of a convolutional recurrent video autoencoder network that performs a surrogate task of video reconstruction. Once trained, it is employed for guided training of a motion encoder for blurred images. This network extracts embedded motion information from the blurred image to generate a sharp video in …
Model-Guided Network with Cluster-Based Operators for Spatio-Spectral Super-Resolution
Institute of Applied Computing and Community Code (IAC3)（应用计算与社区代码研究所）；Dept. of Mathematics and Computer Science, Universitat de les Illes Balears（数学与计算机科学系，巴利阿里大学）
Although current deep learning-based methods have gained promising performance in the blind single image super-resolution (SISR) task, most of them mainly focus on heuristically constructing diverse network architectures and put less emphasis on the explicit embedding of the physical generation mechanism between blur kernels and high-resolution (HR) images. To alleviate this issue, we propose a model-driven deep neural network, called KXNet, for blind SISR. Specifically, to solve the classical SISR model, we propose a simple-yet-effective iterative algorithm. Then by unfolding the involved ite…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）
Advances in image super-resolution (SR) have recently benefited significantly from rapid developments in deep neural networks. Inspired by these recent discoveries, we note that many state-of-the-art deep SR architectures can be reformulated as a single-state recurrent neural network (RNN) with finite unfoldings. In this paper, we explore new structures for SR based on this compact RNN view, leading us to a dual-state design, the Dual-State Recurrent Network (DSRN). Compared to its single state counterparts that operate at a fixed spatial resolution, DSRN exploits both low-resolution (LR) and …
Phy-CoSF: Physics-Guided Continuous Spectral Fields Reconstruction and Super-Resolution for Snapshot Compressive Imaging
College of Communication Engineering, Jilin University, Changchun 130012, China.；School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, China.；School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 639798.；Department of Computer；Information Science, University of Macau, Macau 999078, China.；College of Computer Science；Technology, Jilin University, Changchun 130012, China.；College of Artificial Intelligence, Jilin University, Changchun 130012, China.；School of Information；Communication Engineering, University of Electronic Science
Spectral Super-Resolution via Adversarial Unfolding and Data-Driven Spectrum Regularization: From Multispectral Satellite Data to NASA Hyperspectral Image
Xi’an Jiaotong University（西安交通大学）；Harbin Institute of Technology（哈尔滨工业大学）
Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spec…
Huazhong University of Science & Technology（华中科技大学）；Wuhan Jingce Electronic Group（武汉精测电子集团）
We present a simple, efficient, and scalable unfolding network, SAUNet, to simplify the network design with an adaptive alternate optimization framework for hyperspectral image (HSI) reconstruction. SAUNet customizes a Residual Adaptive ADMM Framework (R2ADMM) to connect each stage of the network via a group of learnable parameters to promote the usage of mask prior, which greatly stabilizes training and solves the accuracy degradation issue. Additionally, we introduce a simple convolutional modulation block (CMB), which leads to efficient training, easy scale-up, and less computation. Couplin…
Re-Visible Dual-Domain Self-Supervised Deep Unfolding Network for MRI Reconstruction
Shanghai University（上海大学）；Shanghai Institute of Applied Mathematics and Mechanics（上海应用数学和力学研究所）；School of Mechanics and Engineering Science（机械与工程科学学院）；Xi’an Jiaotong University（西安交通大学）；School of Communication and Information Engineering（通信与信息工程学院）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）
While deep neural networks have achieved impressive success in image compressive sensing (CS), most of them lack flexibility when dealing with multi-ratio tasks and multi-scene images in practical applications. To tackle these challenges, we propose a novel end-to-end flexible ISTA-unfolding deep network, dubbed ISTA-Net++, with superior performance and strong flexibility. Specifically, by developing a dynamic unfolding strategy, our model enjoys the adaptability of handling CS problems with different ratios, i.e., multi-ratio tasks, through a single model. A cross-block strategy is further ut…
Latent Diffusion Prior Enhanced Deep Unfolding for Snapshot Spectral Compressive Imaging
Zhejiang University（浙江大学）；Westlake University（西湖大学）；School of Engineering, Westlake University（西湖大学工程学院）；Xidian University（西安电子科技大学）；School of Cyber Engineering, Xidian University（西安电子科技大学网络与工程学院）；Beijing Institute of Technology（北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
Tencent Jarvis Lab（腾讯天衍实验室）；Beijing Information Science and Technology University（北京信息科技大学）；Xi’an Jiaotong University（西安交通大学）；Peng Cheng Laboratory（鹏城实验室）；Macau University of Science and Technology（澳门科技大学）
During the computed tomography (CT) imaging process, metallic implants within patients often cause harmful artifacts, which adversely degrade the visual quality of reconstructed CT images and negatively affect the subsequent clinical diagnosis. For the metal artifact reduction (MAR) task, current deep learning based methods have achieved promising performance. However, most of them share two main common limitations: 1) the CT physical imaging geometry constraint is not comprehensively incorporated into deep network structures; 2) the entire framework has weak interpretability for the specific …
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Snapshot compressive imaging (SCI) aims to record three-dimensional signals via a two-dimensional camera. For the sake of building a fast and accurate SCI recovery algorithm, we incorporate the interpretability of model-based methods and the speed of learning-based ones and present a novel dense deep unfolding network (DUN) with 3D-CNN prior for SCI, where each phase is unrolled from an iteration of Half-Quadratic Splitting (HQS). To better exploit the spatial-temporal correlation among frames and address the problem of information loss between adjacent phases in existing DUNs, we propose to a…
MVMS-RCN: A Dual-Domain Unfolding CT Reconstruction with Multi-sparse-view and Multi-scale Refinement-correction
Xiangtan University（湘潭大学）；Zhejiang Normal University（浙江师范大学）；The University of Liverpool（利物浦大学）；University of Strathclyde（斯特拉斯克莱德大学）；Hunan National Applied Mathematics Center（湖南国家应用数学中心）
The non-uniform photoelectric response of infrared imaging systems results in fixed-pattern stripe noise being superimposed on infrared images, which severely reduces image quality. As the applications of degraded infrared images are limited, it is crucial to effectively preserve original details. Existing image destriping methods struggle to concurrently remove all stripe noise artifacts, preserve image details and structures, and balance real-time performance. In this paper we propose a novel algorithm for destriping degraded images, which takes advantage of neighbouring column signal correl…
Snapshot compressive imaging (SCI) can record the 3D information by a 2D measurement and from this 2D measurement to reconstruct the original 3D information by reconstruction algorithm. As we can see, the reconstruction algorithm plays a vital role in SCI. Recently, deep learning algorithm show its outstanding ability, outperforming the traditional algorithm. Therefore, to improve deep learning algorithm reconstruction accuracy is an inevitable topic for SCI. Besides, deep learning algorithms are usually limited by scalability, and a well trained model in general can not be applied to new syst…
Deep Unfolding Network with Spatial Alignment for multi-modal MRI reconstruction
Shanghai University（上海大学）；School of Science, Shanghai University（上海大学理学院）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；Shanghai Institute of Applied Mathematics and Mechanics, Shanghai University（上海大学上海应用数学与力学研究所）；School of Mechanics and Engineering Science, Shanghai University（上海大学力学与工程科学学院）
Low-Rank Representations Meets Deep Unfolding: A Generalized and Interpretable Network for Hyperspectral Anomaly Detection
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Mathematics and Statistics, Southeast University（东南大学数学学院与统计学院）；College of Resources and Environment, University of Chinese Academy of Sciences（中国科学院大学资源与环境学院）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔理工学院、约瑟夫·傅里叶大学联合实验室）
