Recent advancements in video restoration have focused on recovering high-quality video frames from low-quality inputs. Compared with static images, the performance of video restoration significantly depends on efficient exploitation of temporal correlations among successive video frames. The numerous techniques make use of temporal information via flow-based strategies or recurrent architectures. However, these methods often encounter difficulties in preserving temporal consistency as they utilize degraded input video frames. To resolve this issue, we propose a novel video restoration framework named Joint Flow and Feature Refinement using Attention (JFFRA). The proposed JFFRA is based on key philosophy of iteratively enhancing data through the synergistic collaboration of flow (alignment) and restoration. By leveraging previously enhanced features to refine flow and vice versa, JFFRA enables efficient feature enhancement using temporal information. This interplay between flow and restoration is executed at multiple scales, reducing the dependence on precise flow estimation. Moreover, we incorporate an occlusion-aware temporal loss function to enhance the network's capability in eliminating flickering artifacts. Comprehensive experiments validate the versatility of JFFRA across various restoration tasks such as denoising, deblurring, and super-resolution. Our method demonstrates a remarkable performance improvement of up to 1.62 dB compared to state-of-the-art approaches.
Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）
This paper presents a novel strategy for high-fidelity image restoration by characterizing both local smoothness and nonlocal self-similarity of natural images in a unified statistical manner. The main contributions are three-folds. First, from the perspective of image statistics, a joint statistical modeling (JSM) in an adaptive hybrid space-transform domain is established, which offers a powerful mechanism of combining local smoothness and nonlocal self-similarity simultaneously to ensure a more reliable and robust estimation. Second, a new form of minimization functional for solving image i…
University of Science and Technology of China（中国科学技术大学）
We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a te…
Joint multi-dimensional dynamic attention and transformer for general image restoration
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education（教育部光子技术集成感知与通信重点实验室）；Guangdong Provincial Key Laboratory of Information Photonics Technology（广东省信息光子技术重点实验室）；Institute of Advanced Photonics Technology, School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院先进光子技术研究所）；School of Electronics and Communication Engineering, Sun Yat-Sen University（中山大学电子与通信工程学院）
Civil Aviation Flight University of China（中国民用航空飞行学院）；University of Electronic Science and Technology of China（电子科技大学）；Institute of North Electronic Equipment（北方电子设备研究所）；Pensees Ptd Ltd（澎思科技有限公司）；Baidu Research（百度研究院）；Tencent AI Lab（腾讯人工智能实验室）；National University of Singapore（新加坡国立大学）
In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise. To address these challenges, previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. However, most of these methods are stage-wise, which is sub-optimal and deviates from the reality. In this paper, we address all these challenges jointly for unconstrained face recognition. We propose an Multi-Degradation Face Restoration (MDFR) model to restore frontalized high-quality faces from the given l…
UHDRes: Ultra-High-Definition Image Restoration via Dual-Domain Decoupled Spectral Modulation
College of Computer Science and Artificial Intelligence, Wenzhou University（计算机科学与人工智能学院，温州大学）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；College of Engineering and Computer Science, Australian National University（工程与计算机科学学院，澳大利亚国立大学）
The Chinese University of Hong Kong（香港中文大学）；Smartmore Corporation（思谋科技）
Long-range temporal alignment is critical yet challenging for video restoration tasks. Recently, some works attempt to divide the long-range alignment into several sub-alignments and handle them progressively. Although this operation is helpful in modeling distant correspondences, error accumulation is inevitable due to the propagation mechanism. In this work, we present a novel, generic iterative alignment module which employs a gradual refinement scheme for sub-alignments, yielding more accurate motion compensation. To further enhance the alignment accuracy and temporal consistency, we devel…
Infrared-Assisted Single-Stage Framework for Joint Restoration and Fusion of Visible and Infrared Images under Hazy Conditions
Faculty of Information Engineering and Automation（信息工程与自动化学院）；Key Laboratory of Artificial Intelligence in Yunnan Province（云南省人工智能重点实验室）；School of Instrument Science and Opto-Electronic Engineering（仪器科学与光电工程学院）
WMFormer++: Nested Transformer for Visible Watermark Removal via Implict Joint Learning
School of Software and Engineering, South China University of Technology（华南理工大学软件学院）；Peng Cheng Laboratory（鹏城实验室）；Tencent Wechat Department（腾讯微信事业群）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机学院）；Xidian University（西安电子科技大学）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lingnan University（岭南大学）；Hong Kong Metropolitan University（香港都会大学）
Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question - if the combination of image restoration and object detection, can boost the performance of cutting-edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet.…
National University of Singapore（新加坡国立大学）；Yale-NUS College（耶鲁-新加坡国立大学学院）
Most deraining works focus on rain streaks removal but they cannot deal adequately with heavy rain images. In heavy rain, streaks are strongly visible, dense rain accumulation or rain veiling effect significantly washes out the image, further scenes are relatively more blurry, etc. In this paper, we propose a novel method to address these problems. We put forth a 2-stage network: a physics-based backbone followed by a depth-guided GAN refinement. The first stage estimates the rain streaks, the transmission, and the atmospheric light governed by the underlying physics. To tease out these compon…
Boosting Image Restoration via Priors from Pre-trained Models
Zhejiang Lab（之江实验室）；CUHK（香港中文大学）；RealityEdge（睿迪智芯）；University of Macau（澳门大学）；Institute of Collaborative Innovation（协同创新研究院）；Texas A&M University（德克萨斯农工大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）
We leverage deep learning techniques to jointly denoise and super-resolve biomedical images acquired with fluorescence microscopy. We develop a deep learning algorithm based on the networks and method described in the recent W2S paper to solve a joint denoising and super-resolution problem. Specifically, we address the restoration of SIM images from widefield images. Our TensorFlow model is trained on the W2S dataset of cell images and is made accessible online in this repository: https://github.com/mchatton/w2s-tensorflow. On test images, the model shows a visually-convincing denoising and in…
V-Bridge: Bridging Video Generative Priors to Versatile Few-shot Image Restoration
The Hong Kong University of Science and Technology（香港科学与技术大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
Small compression noises, despite being transparent to human eyes, can adversely affect the results of many image restoration processes, if left unaccounted for. Especially, compression noises are highly detrimental to inverse operators of high-boosting (sharpening) nature, such as deblurring and superresolution against a convolution kernel. By incorporating the non-linear DCT quantization mechanism into the formulation for image restoration, we propose a new sparsity-based convex programming approach for joint compression noise removal and image restoration. Experimental results demonstrate s…
TIDE: Two-Stage Inverse Degradation Estimation with Guided Prior Disentanglement for Underwater Image Restoration
Mohamed bin Zayed University of AI（莫扎德·本·泽亚德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Nagasaki University（长崎大学）
From Zero to Detail: A Progressive Spectral Decoupling Paradigm for UHD Image Restoration with New Benchmark
State Key Laboratory of Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Department of Computer Science and Engineering, Michigan State University（计算机科学与工程系，密歇根州立大学）；School of Computer Science and Technology, Dalian University of Technology（计算机科学与技术学院，大连理工大学）
Non-local self-similarity and sparsity principles have proven to be powerful priors for natural image modeling. We propose a novel differentiable relaxation of joint sparsity that exploits both principles and leads to a general framework for image restoration which is (1) trainable end to end, (2) fully interpretable, and (3) much more compact than competing deep learning architectures. We apply this approach to denoising, jpeg deblocking, and demosaicking, and show that, with as few as 100K parameters, its performance on several standard benchmarks is on par or better than state-of-the-art me…
Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）
Blind super-resolution (SR) aims to recover high-quality visual textures from a low-resolution (LR) image, which is usually degraded by down-sampling blur kernels and additive noises. This task is extremely difficult due to the challenges of complicated image degradations in the real-world. Existing SR approaches either assume a predefined blur kernel or a fixed noise, which limits these approaches in challenging cases. In this paper, we propose a Degradation-guided Meta-restoration network for blind Super-Resolution (DMSR) that facilitates image restoration for real cases. DMSR consists of a …
City University of Hong Kong（香港城市大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）
Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. Directly adjusting exposure time and ISO ratings cannot obtain sharp and noise-free images at the same time in low-light conditions. Though many methods have been proposed to enhance noisy or blurry night images, their performances on real-world night photos are still unsatisfactory due to two main reasons: 1) Limited information in a single image and 2) Domain gap between synthetic training images and real-world photos (e.g., differences in blur area and resolution)…
We present a novel variational approach to image restoration (e.g., denoising, inpainting, labeling) that enables to complement established variational approaches with a histogram-based prior enforcing closeness of the solution to some given empirical measure. By minimizing a single objective function, the approach utilizes simultaneously two quite different sources of information for restoration: spatial context in terms of some smoothness prior and non-spatial statistics in terms of the novel prior utilizing the Wasserstein distance between probability measures. We study the combination of t…
