Despite the growing popularity of video super-resolution (VSR), there is still no good way to assess the quality of the restored details in upscaled frames. Some SR methods may produce the wrong digit or an entirely different face. Whether a method's results are trustworthy depends on how well it restores truthful details. Image super-resolution can use natural distributions to produce a high-resolution image that is only somewhat similar to the real one. VSR enables exploration of additional information in neighboring frames to restore details from the original scene. The ERQA metric, which we propose in this paper, aims to estimate a model's ability to restore real details using VSR. On the assumption that edges are significant for detail and character recognition, we chose edge fidelity as the foundation for this metric. Experimental validation of our work is based on the MSU Video Super-Resolution Benchmark, which includes the most difficult patterns for detail restoration and verifies the fidelity of details from the original frame. Code for the proposed metric is publicly available at https://github.com/msu-video-group/ERQA.
Ningbo University（宁波大学）；Ningbo Institute of Industrial Technology, Chinese Academy of Sciences（中国科学院宁波工业技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
With the development of image recovery models,especially those based on adversarial and perceptual losses,the detailed texture portions of images are being recovered more naturally.However,these restored images are similar but not identical in detail texture to their reference images.With traditional image quality assessment methods,results with better subjective perceived quality often score lower in objective scoring.Assessment methods suffer from subjective and objective inconsistencies.This paper proposes a regional differential information entropy (RDIE) method for image quality assessmen…
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；SenseTime Research（商汤科技研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent IR methods based on Generative Adversarial Networks (GANs) have achieved significant improvement in visual performance, but also presented great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. Then we raise two questions: (1) Can existing IQA methods objectively evaluate recent IR algorithms? (2) When focus on beating current benchmarks, are we getting better IR algorithms? …
The University of Sydney（悉尼大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent perceptual IR algorithms based on generative adversarial networks (GANs) have brought in significant improvement on visual performance, but also pose great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. We present two questions: Can existing IQA methods objectively evaluate recent IR algorithms? With the focus on beating current benchmarks, are we getting better IR algorith…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；University of Rochester（罗切斯特大学）
An important scenario for image quality assessment (IQA) is to evaluate image restoration (IR) algorithms. The state-of-the-art approaches adopt a full-reference paradigm that compares restored images with their corresponding pristine-quality images. However, pristine-quality images are usually unavailable in blind image restoration tasks and real-world scenarios. In this paper, we propose a practical solution named degraded-reference IQA (DR-IQA), which exploits the inputs of IR models, degraded images, as references. Specifically, we extract reference information from degraded images by dist…
Perceptual image restoration seeks for high-fidelity images that most likely degrade to given images. For better visual quality, previous work proposed to search for solutions within the natural image manifold, by exploiting the latent space of a generative model. However, the quality of generated images are only guaranteed when latent embedding lies close to the prior distribution. In this work, we propose to restrict the feasible region within the prior manifold. This is accomplished with a non-parametric metric for two distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model th…
Super-resolution (SR) has become a widely researched topic in recent years. SR methods can improve overall image and video quality and create new possibilities for further content analysis. But the SR mainstream focuses primarily on increasing the naturalness of the resulting image despite potentially losing context accuracy. Such methods may produce an incorrect digit, character, face, or other structural object even though they otherwise yield good visual quality. Incorrect detail restoration can cause errors when detecting and identifying objects both manually and automatically. To analyze …
School of Engineering, Universidad Autonoma de Madrid（马德里自治大学工程学院）
In this paper we develop FaceQgen, a No-Reference Quality Assessment approach for face images based on a Generative Adversarial Network that generates a scalar quality measure related with the face recognition accuracy. FaceQgen does not require labelled quality measures for training. It is trained from scratch using the SCface database. FaceQgen applies image restoration to a face image of unknown quality, transforming it into a canonical high quality image, i.e., frontal pose, homogeneous background, etc. The quality estimation is built as the similarity between the original and the restored…
This paper reports on the NTIRE 2022 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2022. This challenge is held to address the emerging challenge of IQA by perceptual image processing algorithms. The output images of these algorithms have completely different characteristics from traditional distortions and are included in the PIPAL dataset used in this challenge. This challenge is divided into two tracks, a full-reference IQA track similar to the previous NTIRE IQA challeng…
Cooperative Medianet Innovation Center（协同媒体创新中心）；Key Laboratory of Machine Perception (MoE), School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院机器感知重点实验室（教育部））；Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS)（中国科学院智能信息处理重点实验室）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Inspired by the free-energy brain theory, which implies that human visual system (HVS) tends to reduce uncertainty and restore perceptual details upon seeing a distorted image, we propose restorative adversarial net (RAN), a GAN-based model for no-reference image quality assessment (NR-IQA). RAN, which mimics the process of HVS, consists of three components: a restorator, a discriminator and an evaluator. The restorator restores and reconstructs input distorted image patches, while the discriminator distinguishes the reconstructed patches from the pristine distortion-free patches. After restor…
University of Würzburg（维尔茨堡大学）；Institute of Computer Science（计算机科学研究所）
Generative models for image restoration, enhancement, and generation have significantly improved the quality of the generated images. Surprisingly, these models produce more pleasant images to the human eye than other methods, yet, they may get a lower perceptual quality score using traditional perceptual quality metrics such as PSNR or SSIM. Therefore, it is necessary to develop a quantitative metric to reflect the performance of new algorithms, which should be well-aligned with the person's mean opinion score (MOS). Learning-based approaches for perceptual image quality assessment (IQA) usua…
Universität Bremen（不来梅大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Berlin Institute of Technology（柏林工业大学）
In most practical situations, the compression or transmission of images and videos creates distortions that will eventually be perceived by a human observer. Vice versa, image and video restoration techniques, such as inpainting or denoising, aim to enhance the quality of experience of human viewers. Correctly assessing the similarity between an image and an undistorted reference image as subjectively experienced by a human viewer can thus lead to significant improvements in any transmission, compression, or restoration system. This paper introduces the Haar wavelet-based perceptual similarity…
Restore, Assess, Repeat: A Unified Framework for Iterative Image Restoration
Samsung AI Center Cambridge（三星剑桥人工智能中心）；National Taiwan University（国立台湾大学）；Technical University of Iasi（雅西科技大学）；Chang Gung University（长庚大学）；University of Wurzburg（维尔茨堡大学）；Queen Mary University of London（伦敦玛丽女王大学）
Qingdao University（青岛大学）；Ocean University of China（中国海洋大学）
Achieving subjective and objective quality assessment of underwater images is of high significance in underwater visual perception and image/video processing. However, the development of underwater image quality assessment (UIQA) is limited for the lack of comprehensive human subjective user study with publicly available dataset and reliable objective UIQA metric. To address this issue, we establish a large-scale underwater image dataset, dubbed UID2021, for evaluating no-reference UIQA metrics. The constructed dataset contains 60 multiply degraded underwater images collected from various sour…
Bridging the Gap Between Image Restoration and Navigational Safety in Hazy Conditions: A New Visibility Estimation Metric for Maritime Surveillance
Sanya Science and Education Innovation Park, Wuhan University of Technology, Sanya 572000, China（武汉理工大学三亚科学教育创新园，三亚572000，中国）；School of Navigation, Wuhan University of Technology, Wuhan 430063, China（武汉理工大学航海学院，武汉430063，中国）；State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China（武汉理工大学海洋技术与安全国家重点实验室，武汉430063，中国）；Department of Mechanics and Maritime Sciences, Chalmers University of Technology, Göteborg 41296, Sweden（楚科奇大学机械与航海科学系，哥德堡41296，瑞典）
AdaQual-Diff: Diffusion-Based Image Restoration via Adaptive Quality Prompting
Fuzhou University（福州大学）；University of Science and Technology of China（中国科学技术大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Shandong Normal University（山东师范大学）；Sun Yat-sen University（中山大学）
Image quality assessment (IQA) aims to assess the perceptual quality of images. The outputs of the IQA algorithms are expected to be consistent with human subjective perception. In image restoration and enhancement tasks, images generated by generative adversarial networks (GAN) can achieve better visual performance than traditional CNN-generated images, although they have spatial shift and texture noise. Unfortunately, the existing IQA methods have unsatisfactory performance on the GAN-based distortion partially because of their low tolerance to spatial misalignment. To this end, we propose t…
Towards Real-world Video Face Restoration: A New Benchmark
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021. As a new type of image processing technology, perceptual image processing algorithms based on Generative Adversarial Networks (GAN) have produced images with more realistic textures. These output images have completely different characteristics from traditional distortions, thus pose a new challenge for IQA methods to evaluate their visual quality. In comparison with previous IQA challen…
Harbin Institute of Technology（哈尔滨工业大学）；Dalian Maritime University（大连海事大学）；Peng Cheng Laboratory（鹏城实验室）
Full-reference (FR) image quality assessment (IQA) evaluates the visual quality of a distorted image by measuring its perceptual difference with pristine-quality reference, and has been widely used in low-level vision tasks. Pairwise labeled data with mean opinion score (MOS) are required in training FR-IQA model, but is time-consuming and cumbersome to collect. In contrast, unlabeled data can be easily collected from an image degradation or restoration process, making it encouraging to exploit unlabeled training data to boost FR-IQA performance. Moreover, due to the distribution inconsistency…
HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restoration
Hawkes Institute, UCL（UCL哈维斯研究所）；Dept. of Medical Physics and Biomedical Engineering, UCL（UCL医学物理与生物医学工程系）；Dept. of Computer Science, UCL（UCL计算机科学系）；Centre for AI, DS&AI, AstraZeneca, UK（阿斯利康AI中心）
McMaster University（麦克马斯特大学）；Sharif University of Technology（谢里夫理工大学）
Recently, deep convolutional neural networks (DCNN) that leverage the adversarial training framework for image restoration and enhancement have significantly improved the processed images' sharpness. Surprisingly, although these DCNNs produced crispier images than other methods visually, they may get a lower quality score when popular measures are employed for evaluating them. Therefore it is necessary to develop a quantitative metric to reflect their performances, which is well-aligned with the perceived quality of an image. Famous quantitative metrics such as Peak signal-to-noise ratio (PSNR…
Centre for Intelligent Sensing, Queen Mary University of London（伦敦玛丽女王大学智能感知中心）
Underwater images are degraded by the selective attenuation of light that distorts colours and reduces contrast. The degradation extent depends on the water type, the distance between an object and the camera, and the depth under the water surface the object is at. Underwater image filtering aims to restore or to enhance the appearance of objects captured in an underwater image. Restoration methods compensate for the actual degradation, whereas enhancement methods improve either the perceived image quality or the performance of computer vision algorithms. The growing interest in underwater ima…
BiRQA: Bidirectional Robust Quality Assessment for Images
MSU Institute for Artificial Intelligence（莫斯科人工智能研究所）；Lomonosov Moscow State University（洛蒙诺索夫莫斯科国立大学）；ISP RAS Research Center for Trusted Artificial Intelligence（可信人工智能俄罗斯科学院ISP研究中心）；Innopolis University（英诺波利斯大学）
Dual High-Order Total Variation Model for Underwater Image Restoration
College of Computer Science and Technology, Qingdao University（青岛大学计算机科学与技术学院）；Key Laboratory of Knowledge Automation for Industrial Processes, Ministry of Education, the School of Automation and Electrical Engineering, University of Science and Technology Beijing（北京科技大学自动化与电气工程学院 工业过程知识自动化教育部重点实验室）
Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）
Surveillance Facial Image Quality Assessment: A Multi-dimensional Dataset and Lightweight Model
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；School of Communication and Electronic Engineering, East China Normal University（东华大学通信与电子工程学院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai AI Lab（上海人工智能实验室）
The Chinese University of Hong Kong（香港中文大学）；SIAT CAS（中国科学院深圳先进技术研究院）
As a popular entertainment art form, manga enriches the line drawings details with bitonal screentones. However, manga resources over the Internet usually show screentone artifacts because of inappropriate scanning/rescaling resolution. In this paper, we propose an innovative two-stage method to restore quality bitonal manga from degraded ones. Our key observation is that the aliasing induced by downsampling bitonal screentones can be utilized as informative clues to infer the original resolution and screentones. First, we predict the target resolution from the degraded manga via the Scale Est…
