PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；SenseTime Research（商汤科技研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, Chao Dong
备注：
This paper has been accepted for publication at ECCV2020
Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent IR methods based on Generative Adversarial Networks (GANs) have achieved significant improvement in visual performance, but also presented great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. Then we raise two questions: (1) Can existing IQA methods objectively evaluate recent IR algorithms? (2) When focus on beating current benchmarks, are we getting better IR algorithms? To answer these questions and promote the development of IQA methods, we contribute a large-scale IQA dataset, called Perceptual Image Processing Algorithms (PIPAL) dataset. Especially, this dataset includes the results of GAN-based methods, which are missing in previous datasets. We collect more than 1.13 million human judgments to assign subjective scores for PIPAL images using the more reliable "Elo system". Based on PIPAL, we present new benchmarks for both IQA and super-resolution methods. Our results indicate that existing IQA methods cannot fairly evaluate GAN-based IR algorithms. While using appropriate evaluation methods is important, IQA methods should also be updated along with the development of IR algorithms. At last, we improve the performance of IQA networks on GAN-based distortions by introducing anti-aliasing pooling. Experiments show the effectiveness of the proposed method.
The University of Sydney（悉尼大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent perceptual IR algorithms based on generative adversarial networks (GANs) have brought in significant improvement on visual performance, but also pose great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. We present two questions: Can existing IQA methods objectively evaluate recent IR algorithms? With the focus on beating current benchmarks, are we getting better IR algorith…
This paper reports on the NTIRE 2022 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2022. This challenge is held to address the emerging challenge of IQA by perceptual image processing algorithms. The output images of these algorithms have completely different characteristics from traditional distortions and are included in the PIPAL dataset used in this challenge. This challenge is divided into two tracks, a full-reference IQA track similar to the previous NTIRE IQA challeng…
Ningbo University（宁波大学）；Ningbo Institute of Industrial Technology, Chinese Academy of Sciences（中国科学院宁波工业技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
With the development of image recovery models,especially those based on adversarial and perceptual losses,the detailed texture portions of images are being recovered more naturally.However,these restored images are similar but not identical in detail texture to their reference images.With traditional image quality assessment methods,results with better subjective perceived quality often score lower in objective scoring.Assessment methods suffer from subjective and objective inconsistencies.This paper proposes a regional differential information entropy (RDIE) method for image quality assessmen…
Perceptual image restoration seeks for high-fidelity images that most likely degrade to given images. For better visual quality, previous work proposed to search for solutions within the natural image manifold, by exploiting the latent space of a generative model. However, the quality of generated images are only guaranteed when latent embedding lies close to the prior distribution. In this work, we propose to restrict the feasible region within the prior manifold. This is accomplished with a non-parametric metric for two distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model th…
Many super-resolution (SR) models are optimized for high performance only and therefore lack efficiency due to large model complexity. As large models are often not practical in real-world applications, we investigate and propose novel loss functions, to enable SR with high perceptual quality from much more efficient models. The representative power for a given low-complexity generator network can only be fully leveraged by strong guidance towards the optimal set of parameters. We show that it is possible to improve the performance of a recently introduced efficient generator architecture sole…
Can Nano Banana 2 Replace Traditional Image Restoration Models? An Evaluation of Its Performance on Image Restoration Tasks
Shenzhen University of Advanced Technology（深圳先进技术大学）；Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）
Development of perceptual image quality assessment (IQA) metrics has been of significant interest to computer vision community. The aim of these metrics is to model quality of an image as perceived by humans. Recent works in Full-reference IQA research perform pixelwise comparison between deep features corresponding to query and reference images for quality prediction. However, pixelwise feature comparison may not be meaningful if distortion present in query image is severe. In this context, we explore utilization of no-reference features in Full-reference IQA task. Our model consists of both …
AdaQual-Diff: Diffusion-Based Image Restoration via Adaptive Quality Prompting
Fuzhou University（福州大学）；University of Science and Technology of China（中国科学技术大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Shandong Normal University（山东师范大学）；Sun Yat-sen University（中山大学）
Tsinghua University（清华大学）；Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy of Science（中国科学院深圳先进技术研究院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；University College London（伦敦大学学院）
Most state-of-the-art semantic segmentation approaches only achieve high accuracy in good conditions. In practically-common but less-discussed adverse environmental conditions, their performance can decrease enormously. Existing studies usually cast the handling of segmentation in adverse conditions as a separate post-processing step after signal restoration, making the segmentation performance largely depend on the quality of restoration. In this paper, we propose a novel deep-learning framework to tackle semantic segmentation and image restoration in adverse environmental conditions in a hol…
Rethinking RGB Color Representation for Image Restoration Models
Seoul National University（首尔大学）；Hanyang University（汉阳大学）；ASRI（ASRI（高级科学技术研究所））；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能跨学科项目）
Technion – Israel Institute of Technology（以色列理工学院）
Image restoration algorithms are typically evaluated by some distortion measure (e.g. PSNR, SSIM, IFC, VIF) or by human opinion scores that quantify perceived perceptual quality. In this paper, we prove mathematically that distortion and perceptual quality are at odds with each other. Specifically, we study the optimal probability for correctly discriminating the outputs of an image restoration algorithm from real images. We show that as the mean distortion decreases, this probability must increase (indicating worse perceptual quality). As opposed to the common belief, this result holds true f…
The appearance of objects in underwater images is degraded by the selective attenuation of light, which reduces contrast and causes a colour cast. This degradation depends on the water environment, and increases with depth and with the distance of the object from the camera. Despite an increasing volume of works in underwater image enhancement and restoration, the lack of a commonly accepted evaluation measure is hindering the progress as it is difficult to compare methods. In this paper, we review commonly used colour accuracy measures, such as colour reproduction error and CIEDE2000, and no-…
Optimal Transport Based Unsupervised Restoration Learning Exploiting Degradation Sparsity
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, China（信息科学与电子工程学院，上海交通大学）；School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China（通信工程学院，杭州电子科技大学）
Northwestern University（西北大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）
Computed tomography is widely used to examine internal structures in a non-destructive manner. To obtain high-quality reconstructions, one typically has to acquire a densely sampled trajectory to avoid angular undersampling. However, many scenarios require a sparse-view measurement leading to streak-artifacts if unaccounted for. Current methods do not make full use of the domain-specific information, and hence fail to provide reliable reconstructions for highly undersampled data. We present a novel framework for sparse-view tomography by decoupling the reconstruction into two steps: First, we …
We report the possibility of using a simple neural network for effortless restoration of low-light images inspired by the retina model, which mimics the neurophysiological principles and dynamics of various types of optical neurons. The proposed neural network model saves the cost of computational overhead in contrast with traditional signal-processing models, and generates results comparable with complicated deep learning models from the subjective perceptual perspective. This work shows that to directly simulate the functionalities of retinal neurons using neural networks not only avoids the…
University of Würzburg（维尔茨堡大学）；Institute of Computer Science（计算机科学研究所）
Generative models for image restoration, enhancement, and generation have significantly improved the quality of the generated images. Surprisingly, these models produce more pleasant images to the human eye than other methods, yet, they may get a lower perceptual quality score using traditional perceptual quality metrics such as PSNR or SSIM. Therefore, it is necessary to develop a quantitative metric to reflect the performance of new algorithms, which should be well-aligned with the person's mean opinion score (MOS). Learning-based approaches for perceptual image quality assessment (IQA) usua…
Image degradation due to atmospheric turbulence is common while capturing images at long ranges. To mitigate the degradation due to turbulence which includes deformation and blur, we propose a generative single frame restoration algorithm which disentangles the blur and deformation due to turbulence and reconstructs a restored image. The disentanglement is achieved by decomposing the distortion due to turbulence into blur and deformation components using deblur generator and deformation correction generator respectively. Two paths of restoration are implemented to regularize the disentanglemen…
Technion–Israel Institute of Technology（以色列理工学院）；ETH Zurich（苏黎世联邦理工学院）
This paper reports on the 2018 PIRM challenge on perceptual super-resolution (SR), held in conjunction with the Perceptual Image Restoration and Manipulation (PIRM) workshop at ECCV 2018. In contrast to previous SR challenges, our evaluation methodology jointly quantifies accuracy and perceptual quality, therefore enabling perceptual-driven methods to compete alongside algorithms that target PSNR maximization. Twenty-one participating teams introduced algorithms which well-improved upon the existing state-of-the-art methods in perceptual SR, as confirmed by a human opinion study. We also analy…
Shanghai Jiao Tong University（上海交通大学）；Texas A&M University（得克萨斯农工大学）；University of Macau（澳门大学）
Liquify is a common technique for image editing, which can be used for image distortion. Due to the uncertainty in the distortion variation, restoring distorted images caused by liquify filter is a challenging task. To edit images in an efficient way, distorted images are expected to be restored automatically. This paper aims at the distorted image restoration, which is characterized by seeking the appropriate warping and completion of a distorted image. Existing methods focus on the hardware assistance or the geometric principle to solve the specific regular deformation caused by natural phen…
RELD: Regularization by Latent Diffusion Models for Image Restoration
University of Bologna（博洛尼亚大学）；University of Macerata（马切拉塔大学）；University of Modena and Reggio Emilia（摩德纳大学与雷焦艾米利亚大学）；University of Milan（米兰大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
How far have we gone in Generative Image Restoration? A study on its capability, limitations and evaluation practices
Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of the Chinese Academy of Sciences（中国科学院大学）；Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Digital scans of analogue photographic film typically contain artefacts such as dust and scratches. Automated removal of these is an important part of preservation and dissemination of photographs of historical and cultural importance. While state-of-the-art deep learning models have shown impressive results in general image inpainting and denoising, film artefact removal is an understudied problem. It has particularly challenging requirements, due to the complex nature of analogue damage, the high resolution of film scans, and potential ambiguities in the restoration. There are no publicly av…
Harbin Institute of Technology（哈尔滨工业大学）；Dalian Maritime University（大连海事大学）；Peng Cheng Laboratory（鹏城实验室）
Full-reference (FR) image quality assessment (IQA) evaluates the visual quality of a distorted image by measuring its perceptual difference with pristine-quality reference, and has been widely used in low-level vision tasks. Pairwise labeled data with mean opinion score (MOS) are required in training FR-IQA model, but is time-consuming and cumbersome to collect. In contrast, unlabeled data can be easily collected from an image degradation or restoration process, making it encouraging to exploit unlabeled training data to boost FR-IQA performance. Moreover, due to the distribution inconsistency…
Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sa…
This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021. As a new type of image processing technology, perceptual image processing algorithms based on Generative Adversarial Networks (GAN) have produced images with more realistic textures. These output images have completely different characteristics from traditional distortions, thus pose a new challenge for IQA methods to evaluate their visual quality. In comparison with previous IQA challen…
Pathology Image Restoration via Mixture of Prompts
ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；Nanjing University of Information Science and Technology（南京信息工程大学）
Universität Bremen（不来梅大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Berlin Institute of Technology（柏林工业大学）
In most practical situations, the compression or transmission of images and videos creates distortions that will eventually be perceived by a human observer. Vice versa, image and video restoration techniques, such as inpainting or denoising, aim to enhance the quality of experience of human viewers. Correctly assessing the similarity between an image and an undistorted reference image as subjectively experienced by a human viewer can thus lead to significant improvements in any transmission, compression, or restoration system. This paper introduces the Haar wavelet-based perceptual similarity…
Real-world image super-resolution is a practical image restoration problem that aims to obtain high-quality images from in-the-wild input, has recently received considerable attention with regard to its tremendous application potentials. Although deep learning-based methods have achieved promising restoration quality on real-world image super-resolution datasets, they ignore the relationship between L1- and perceptual- minimization and roughly adopt auxiliary large-scale datasets for pre-training. In this paper, we discuss the image types within a corrupted image and the property of perceptual…
Image Restoration Learning via Noisy Supervision in the Fourier Domain
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Key Laboratory of Image Processing and Intelligent Control, Ministry of Education（教育部图像处理与智能控制重点实验室；华中科技大学人工智能与自动化学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology
