Modulating image restoration level aims to generate a restored image by altering a factor that represents the restoration strength. Previous works mainly focused on optimizing the mean squared reconstruction error, which brings high reconstruction accuracy but lacks finer texture details. This paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate high-frequency textures in the modulation tasks. CUGAN consists of two modules -- base networks and condition networks. The base networks comprise a generator and a discriminator. In the generator, we realize the interactive control of restoration levels by tuning the weights of different features from different scales in the Unet architecture. Moreover, we adaptively modulate the intermediate features in the discriminator according to the severity of degradations. The condition networks accept the condition vector (encoded degradation information) as input, then generate modulation parameters for both the generator and the discriminator. During testing, users can control the output effects by tweaking the condition vector. We also provide a smooth transition between GAN and MSE effects by a simple transition method. Extensive experiments demonstrate that the proposed CUGAN achieves excellent performance on image restoration modulation tasks.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Interactive image restoration aims to generate restored images by adjusting a controlling coefficient which determines the restoration level. Previous works are restricted in modulating image with a single coefficient. However, real images always contain multiple types of degradation, which cannot be well determined by one coefficient. To make a step forward, this paper presents a new problem setup, called multi-dimension (MD) modulation, which aims at modulating output effects across multiple degradation types and levels. Compared with the previous single-dimension (SD) modulation, the MD is …
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；Tencent PCG（腾讯平台与内容事业群）；Shanghai AI Laboratory（上海人工智能实验室）
Interactive image restoration aims to restore images by adjusting several controlling coefficients, which determine the restoration strength. Existing methods are restricted in learning the controllable functions under the supervision of known degradation types and levels. They usually suffer from a severe performance drop when the real degradation is different from their assumptions. Such a limitation is due to the complexity of real-world degradations, which can not provide explicit supervision to the interactive modulation during training. However, how to realize the interactive modulation …
Digital cameras and mobile phones enable us to conveniently record precious moments. While digital image quality is constantly being improved, taking high-quality photos of digital screens still remains challenging because the photos are often contaminated with moiré patterns, a result of the interference between the pixel grids of the camera sensor and the device screen. Moiré patterns can severely damage the visual quality of photos. However, few studies have aimed to solve this problem. In this paper, we introduce a novel multiresolution fully convolutional network for automatically removin…
Arizona State University（亚利桑那州立大学）；SenseBrain Technology（商汤科技（SenseBrain Technology））
The lensless pinhole camera is perhaps the earliest and simplest form of an imaging system using only a pinhole-sized aperture in place of a lens. They can capture an infinite depth-of-field and offer greater freedom from optical distortion over their lens-based counterparts. However, the inherent limitations of a pinhole system result in lower sharpness from blur caused by optical diffraction and higher noise levels due to low light throughput of the small aperture, requiring very long exposure times to capture well-exposed images. In this paper, we explore an image restoration pipeline using…
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Chinese University of Hong Kong（香港中文大学）
Tsinghua University（清华大学）；Chinese University of Hong Kong（香港中文大学）；University of Rochester（罗切斯特大学）
Deep learning methods have witnessed the great progress in image restoration with specific metrics (e.g., PSNR, SSIM). However, the perceptual quality of the restored image is relatively subjective, and it is necessary for users to control the reconstruction result according to personal preferences or image characteristics, which cannot be done using existing deterministic networks. This motivates us to exquisitely design a unified interactive framework for general image restoration tasks. Under this framework, users can control continuous transition of different objectives, e.g., the percepti…
Nanyang Technological University（南洋理工大学）；SenseBrain；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). To bridge the gap, we introduce the first MIPI challenge including five tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Under-Display…
Digital scans of analogue photographic film typically contain artefacts such as dust and scratches. Automated removal of these is an important part of preservation and dissemination of photographs of historical and cultural importance. While state-of-the-art deep learning models have shown impressive results in general image inpainting and denoising, film artefact removal is an understudied problem. It has particularly challenging requirements, due to the complex nature of analogue damage, the high resolution of film scans, and potential ambiguities in the restoration. There are no publicly av…
City University of Hong Kong（香港城市大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）
Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. Directly adjusting exposure time and ISO ratings cannot obtain sharp and noise-free images at the same time in low-light conditions. Though many methods have been proposed to enhance noisy or blurry night images, their performances on real-world night photos are still unsatisfactory due to two main reasons: 1) Limited information in a single image and 2) Domain gap between synthetic training images and real-world photos (e.g., differences in blur area and resolution)…
Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition
Xi’an University of Architecture and Technology（西安建筑科技大学）；Northwestern Polytechnical University（西北工业大学）；University of Macau（澳门大学）；Xi’an Institute of Optics and Precision Mechanics of CAS（中国科学院西安光学精密机械研究所）
Southern University of Science and Technology（南方科技大学）；A*STAR（新加坡科技研究局）；New York University（纽约大学）；Chinese Academy of Sciences（中国科学院）；Shenzhen People’s Hospital（深圳市人民医院）
Fundus photography is a routine examination in clinics to diagnose and monitor ocular diseases. However, for cataract patients, the fundus image always suffers quality degradation caused by the clouding lens. The degradation prevents reliable diagnosis by ophthalmologists or computer-aided systems. To improve the certainty in clinical diagnosis, restoration algorithms have been proposed to enhance the quality of fundus images. Unfortunately, challenges remain in the deployment of these algorithms, such as collecting sufficient training data and preserving retinal structures. In this paper, to …
Efficient Terrain Stochastic Differential Efficient Terrain Stochastic Differential Equations for Multipurpose Digital Elevation Model Restoration
School of Aeronautics and Astronautics, Shanghai Jiao Tong University（上海交通大学航空航天学院）；Land Satellite Remote Sensing Application Center（陆地卫星遥感应用中心）；Ministry of Natural Resources of the People’s Republic of China（中华人民共和国自然资源部）；China-ASEAN Satellite Remote Sensing Applications（中国-东盟卫星遥感应用（中心））
Data61-Commonwealth Scientific Industrial Research Organization (CSIRO)（Data61-联邦科学与工业研究组织）；Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this paper proposes a novel single-stage blind real image restoration network (R$^2$Net) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthe…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we c…
CRNet: A Detail-Preserving Network for Unified Image Restoration and Enhancement Task
Northwestern Polytechnical University（西北工业大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）；Xi’an Institute of Optics and Precision Mechanics of CAS（中国科学院西安光学精密机械研究所）
Inception Institute of Artificial Intelligence（起源人工智能研究院）；Mohammed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of California at Merced（加州大学默塞德分校）；Google（谷歌）；Terminus Group（特斯联集团）
Given a degraded input image, image restoration aims to recover the missing high-quality image content. Numerous applications demand effective image restoration, e.g., computational photography, surveillance, autonomous vehicles, and remote sensing. Significant advances in image restoration have been made in recent years, dominated by convolutional neural networks (CNNs). The widely-used CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatial details are preserved but the contextual information cannot be prec…
HDR is an important part of computational photography technology. In this paper, we propose a lightweight neural network called Efficient Attention-and-alignment-guided Progressive Network (EAPNet) for the challenge NTIRE 2022 HDR Track 1 and Track 2. We introduce a multi-dimensional lightweight encoding module to extract features. Besides, we propose Progressive Dilated U-shape Block (PDUB) that can be a progressive plug-and-play module for dynamically tuning MAccs and PSNR. Finally, we use fast and low-power feature-align module to deal with misalignment problem in place of the time-consumin…
Couger Inc（库格公司）；Sethu Institute of Technology（塞图理工学院）；Thiagarajar college of Engineering（蒂亚加拉贾工程学院）
A moire pattern in the images is resulting from high frequency patterns captured by the image sensor (colour filter array) that appear after demosaicing. These Moire patterns would appear in natural images of scenes with high frequency content. The Moire pattern can also vary intensely due to a minimal change in the camera direction/positioning. Thus the Moire pattern depreciates the quality of photographs. An important issue in demoireing pattern is that the Moireing patterns have dynamic structure with varying colors and forms. These challenges makes the demoireing more difficult than many o…
Alternating direction method of multiplier (ADMM) is a widely used algorithm for solving constrained optimization problems in image restoration. Among many useful features, one critical feature of the ADMM algorithm is its modular structure which allows one to plug in any off-the-shelf image denoising algorithm for a subproblem in the ADMM algorithm. Because of the plug-in nature, this type of ADMM algorithms is coined the name "Plug-and-Play ADMM". Plug-and-Play ADMM has demonstrated promising empirical results in a number of recent papers. However, it is unclear under what conditions and by …
University of Washington（华盛顿大学）；Adobe Inc.（Adobe公司）；Google Research（谷歌研究院）
Many historical people were only ever captured by old, faded, black and white photos, that are distorted due to the limitations of early cameras and the passage of time. This paper simulates traveling back in time with a modern camera to rephotograph famous subjects. Unlike conventional image restoration filters which apply independent operations like denoising, colorization, and superresolution, we leverage the StyleGAN2 framework to project old photos into the space of modern high-resolution photos, achieving all of these effects in a unified framework. A unique challenge with this approach …
University of Notre Dame（圣母大学）；University of Ljubljana（卢布尔雅那大学）
Advances in image restoration and enhancement techniques have led to discussion about how such algorithmscan be applied as a pre-processing step to improve automatic visual recognition. In principle, techniques like deblurring and super-resolution should yield improvements by de-emphasizing noise and increasing signal in an input image. But the historically divergent goals of the computational photography and visual recognition communities have created a significant need for more work in this direction. To facilitate new research, we introduce a new benchmark dataset called UG^2, which contain…
We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans -- all corrupted by different proces…
In plug-and-play image restoration, the regularization is performed using powerful denoisers such as nonlocal means (NLM) or BM3D. This is done within the framework of alternating direction method of multipliers (ADMM), where the regularization step is formally replaced by an off-the-shelf denoiser. Each plug-and-play iteration involves the inversion of the forward model followed by a denoising step. In this paper, we present a couple of ideas for improving the efficiency of the inversion and denoising steps. First, we propose to use linearized ADMM, which generally allows us to perform the in…
DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；ByteDance, Inc（字节跳动公司）；University of Science and Technology of China（中国科学技术大学）
Denoising diffusion models for high-resolution microscopy image restoration
Frankfurt Institute for Advanced Studies (FIAS)（法兰克福高级研究所）；Goethe University Frankfurt（法兰克福大学）；Institute of Physical and Theoretical Chemistry（物理与理论化学研究所）
VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Fujian Normal University（福建师范大学）
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）
When taking photos under an environment with insufficient light, the exposure time and the sensor gain usually require to be carefully chosen to obtain images with satisfying visual quality. For example, the images with high ISO usually have inescapable noise, while the long-exposure ones may be blurry due to camera shake or object motion. Existing solutions generally suggest to seek a balance between noise and blur, and learn denoising or deblurring models under either full- or self-supervision. However, the real-world training pairs are difficult to collect, and the self-supervised methods m…
Inception Institute of Artificial Intelligence（inception人工智能研究院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）
With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, sem…
Universität Innsbruck（因斯布鲁克大学）；Universität Graz（格拉茨大学）；Research Center for Non Destructive Testing GmbH（非破坏性测试研究中心 GmbH）；Prospective Instruments LK OG（前瞻性仪器 LK OG）
A CNN-based interactive contrast enhancement algorithm, called IceNet, is proposed in this work, which enables a user to adjust image contrast easily according to his or her preference. Specifically, a user provides a parameter for controlling the global brightness and two types of scribbles to darken or brighten local regions in an image. Then, given these annotations, IceNet estimates a gamma map for the pixel-wise gamma correction. Finally, through color restoration, an enhanced image is obtained. The user may provide annotations iteratively to obtain a satisfactory image. IceNet is also ca…
AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；vivo Mobile Communication Co., Ltd（vivo移动通信有限公司）；Nanyang Technological University（南洋理工大学）
City University of Hong Kong（香港城市大学）；Microsoft Research（微软研究院）；Microsoft Cloud+AI（微软云与人工智能）；University of Science and Technology of China（中国科学技术大学）
We propose to restore old photos that suffer from severe degradation through a deep learning approach. Unlike conventional restoration tasks that can be solved through supervised learning, the degradation in real photos is complex and the domain gap between synthetic images and real old photos makes the network fail to generalize. Therefore, we propose a novel triplet domain translation network by leveraging real photos along with massive synthetic image pairs. Specifically, we train two variational autoencoders (VAEs) to respectively transform old photos and clean photos into two latent space…
Photon-limited images are often seen in fields such as medical imaging. Although the number of collected photons on an image sensor statistically follows Poisson distribution, this type of noise is intractable, unlike Gaussian noise. In this study, we propose a Bayesian restoration method of Poisson corrupted image using Integrated Nested Laplace Approximation (INLA), which is a computational method to evaluate marginalized posterior distributions of latent Gaussian models (LGMs). When the original image can be regarded as ICAR (intrinsic conditional auto-regressive) model reasonably, our meth…
Renovating the memories in old photos is an intriguing research topic in computer vision fields. These legacy images often suffer from severe and commingled degradations such as cracks, noise, and color-fading, while lack of large-scale paired old photo datasets makes this restoration task very challenging. In this work, we present a novel reference-based end-to-end learning framework that can jointly repair and colorize the degraded legacy pictures. Specifically, the proposed framework consists of three modules: a restoration sub-network for degradation restoration, a similarity sub-network f…
