Super-resolution results are usually measured by full-reference image quality metrics or human rating scores. However, these evaluation methods are general image quality measurement, and do not account for the nature of the super-resolution problem. In this work, we analyze the evaluation problem based on the one-to-many mapping nature of super-resolution, and propose a novel distribution-based metric for super-resolution. Starting from the distribution distance, we derive the proposed metric to make it accessible and easy to compute. Through a human subject study on super-resolution, we show that the proposed metric is highly correlated with the human perceptual quality, and better than most existing metrics. Moreover, the proposed metric has a higher correlation with the fidelity measure compared to the perception-based metrics. To understand the properties of the proposed metric, we conduct extensive evaluation in terms of its design choices, and show that the metric is robust to its design choices. Finally, we show that the metric can be used to train super-resolution networks for better perceptual quality.
We extensively study how to combine Generative Adversarial Networks and learned compression to obtain a state-of-the-art generative lossy compression system. In particular, we investigate normalization layers, generator and discriminator architectures, training strategies, as well as perceptual losses. In contrast to previous work, i) we obtain visually pleasing reconstructions that are perceptually similar to the input, ii) we operate in a broad range of bitrates, and iii) our approach can be applied to high-resolution images. We bridge the gap between rate-distortion-perception theory and pr…
Shandong Normal University（山东师范大学）；Nanyang Technological University（南洋理工大学）
Recently, learned image compression schemes have achieved remarkable improvements in image fidelity (e.g., PSNR and MS-SSIM) compared to conventional hybrid image coding ones due to their high-efficiency non-linear transform, end-to-end optimization frameworks, etc. However, few of them take the Just Noticeable Difference (JND) characteristic of the Human Visual System (HVS) into account and optimize learned image compression towards perceptual quality. To address this issue, a JND-based perceptual quality loss is proposed. Considering that the amounts of distortion in the compressed image at …
The Unanticipated Asymmetry Between Perceptual Optimization and Assessment
Institute of Microelectronics of the Chinese Academy of Sciences（中国科学院微电子研究所）；Beihang University（北航）；The Hong Kong Polytechnic University（香港理工大学）；City University of Hong Kong（香港城市大学）
This paper proposes an explicit way to optimize the super-resolution network for generating visually pleasing images. The previous approaches use several loss functions which is hard to interpret and has the implicit relationships to improve the perceptual score. We show how to exploit the machine learning based model which is directly trained to provide the perceptual score on generated images. It is believed that these models can be used to optimizes the super-resolution network which is easier to interpret. We further analyze the characteristic of the existing loss and our proposed explicit…
Perceptual image restoration seeks for high-fidelity images that most likely degrade to given images. For better visual quality, previous work proposed to search for solutions within the natural image manifold, by exploiting the latent space of a generative model. However, the quality of generated images are only guaranteed when latent embedding lies close to the prior distribution. In this work, we propose to restrict the feasible region within the prior manifold. This is accomplished with a non-parametric metric for two distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model th…
Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China（信息与通信工程学院和媒体融合与通信国家重点实验室，中国通信大学）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（信息科学与电子工程学院，上海交通大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；State Key Laboratory of Multimedia information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）
Enhancing Perceptual Loss with Adversarial Feature Matching for Super-Resolution
Carnegie Mellon University（卡内基梅隆大学）；Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；International Institute of Information Technology Bhubaneswar（布巴内斯瓦尔国际信息技术学院）
Samsung AI Center（三星人工智能中心）；University of Warwick（华威大学）；University of Cambridge（剑桥大学）
Recent works in single-image perceptual super-resolution (SR) have demonstrated unprecedented performance in generating realistic textures by means of deep convolutional networks. However, these convolutional models are excessively large and expensive, hindering their effective deployment to end devices. In this work, we propose a neural architecture search (NAS) approach that integrates NAS and generative adversarial networks (GANs) with recent advances in perceptual SR and pushes the efficiency of small perceptual SR models to facilitate on-device execution. Specifically, we search over the …
We introduce Frequency Domain Perceptual Loss (FDPL), a loss function for single image super resolution (SR). Unlike previous loss functions used to train SR models, which are all calculated in the pixel (spatial) domain, FDPL is computed in the frequency domain. By working in the frequency domain we can encourage a given model to learn a mapping that prioritizes those frequencies most related to human perception. While the goal of FDPL is not to maximize the Peak Signal to Noise Ratio (PSNR), we found that there is a correlation between decreasing FDPL and increasing PSNR. Training a model wi…
Image dehazing without paired haze-free images is of immense importance, as acquiring paired images often entails significant cost. However, we observe that previous unpaired image dehazing approaches tend to suffer from performance degradation near depth borders, where depth tends to vary abruptly. Hence, we propose to anneal the depth border degradation in unpaired image dehazing with cyclic perceptual-depth supervision. Coupled with the dual-path feature re-using backbones of the generators and discriminators, our model achieves $\mathbf{20.36}$ Peak Signal-to-Noise Ratio (PSNR) on NYU Dept…
Liaoning University of Technology（辽宁工业大学）；Dalian University（大连大学）
The proposal of perceptual loss solves the problem that per-pixel difference loss function causes the reconstructed image to be overly-smooth, which acquires a significant progress in the field of single image super-resolution reconstruction. Furthermore, the generative adversarial networks (GAN) is applied to the super-resolution field, which effectively improves the visual quality of the reconstructed image. However, under the condtion of high upscaling factors, the excessive abnormal reasoning of the network produces some distorted structures, so that there is a certain deviation between th…
National University of Singapore（新加坡国立大学）；HiSilicon Technologies（海思半导体）
In image super-resolution, both pixel-wise accuracy and perceptual fidelity are desirable. However, most deep learning methods only achieve high performance in one aspect due to the perception-distortion trade-off, and works that successfully balance the trade-off rely on fusing results from separately trained models with ad-hoc post-processing. In this paper, we propose a novel super-resolution model with a low-frequency constraint (LFc-SR), which balances the objective and perceptual quality through a single model and yields super-resolved images with high PSNR and perceptual scores. We furt…
Perceptually Regularized Diffusion Model for Image Super-Resolution
University of North Carolina（北卡罗来纳大学）；Texas Tech University（德克萨斯理工大学）；Case Western Reserve University（凯斯西储大学）；University of Houston（休斯顿大学）；University of Kentucky（肯塔基大学）
Many super-resolution (SR) models are optimized for high performance only and therefore lack efficiency due to large model complexity. As large models are often not practical in real-world applications, we investigate and propose novel loss functions, to enable SR with high perceptual quality from much more efficient models. The representative power for a given low-complexity generator network can only be fully leveraged by strong guidance towards the optimal set of parameters. We show that it is possible to improve the performance of a recently introduced efficient generator architecture sole…
Textural-Perceptual Joint Learning for No-Reference Super-Resolution Image Quality Assessment
School of Software, Dalian University of Technology（大连理工大学软件学院）；International School of Information Science and Engineering, Dalian University of Technology（大连理工大学国际信息与工程学院）
Perceptual Quality Optimization of Image Super-Resolution
School of Computer Science and Informatics, Cardiff University, UK（计算机科学与信息学学院，卡迪夫大学）；Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria（Christian Doppler实验室ATHENA，阿尔卑斯-亚得里亚大学）；XiaomiEV, China（小米EV，中国）；Centre for Vision, Speech and Signal Processing, University of Surrey, UK（视觉、语音和信号处理中心，萨里大学）
Can Nano Banana 2 Replace Traditional Image Restoration Models? An Evaluation of Its Performance on Image Restoration Tasks
Shenzhen University of Advanced Technology（深圳先进技术大学）；Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）
West Virginia University（西弗吉尼亚大学）；Case Western Reserve University（凯斯西储大学）
Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this paper, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supporte…
Spatial resolution adaptation is a technique which has often been employed in video compression to enhance coding efficiency. This approach encodes a lower resolution version of the input video and reconstructs the original resolution during decoding. Instead of using conventional up-sampling filters, recent work has employed advanced super-resolution methods based on convolutional neural networks (CNNs) to further improve reconstruction quality. These approaches are usually trained to minimise pixel-based losses such as Mean-Squared Error (MSE), despite the fact that this type of loss metric …
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Whirlpool（惠而浦公司）
Capturing high-resolution magnetic resonance (MR) images is a time consuming process, which makes it unsuitable for medical emergencies and pediatric patients. Low-resolution MR imaging, by contrast, is faster than its high-resolution counterpart, but it compromises on fine details necessary for a more precise diagnosis. Super-resolution (SR), when applied to low-resolution MR images, can help increase their utility by synthetically generating high-resolution images with little additional time. In this paper, we present a SR technique for MR images that is based on generative adversarial netwo…
Goldsmiths, University of London（伦敦大学金史密斯学院）；Paul Math School（保罗数学学校）
Recently, there has been discussions on the ill-posed nature of super-resolution that multiple possible reconstructions exist for a given low-resolution image. Using normalizing flows, SRflow[23] achieves state-of-the-art perceptual quality by learning the distribution of the output instead of a deterministic output to one estimate. In this paper, we adapt the concepts of SRFlow to improve GAN-based super-resolution by properly implementing the one-to-many property. We modify the generator to estimate a distribution as a mapping from random noise. We improve the content loss that hampers the p…
Flow-based generative super-resolution (SR) models learn to produce a diverse set of feasible SR solutions, called the SR space. Diversity of SR solutions increases with the temperature ($τ$) of latent variables, which introduces random variations of texture among sample solutions, resulting in visual artifacts and low fidelity. In this paper, we present a simple but effective image ensembling/fusion approach to obtain a single SR image eliminating random artifacts and improving fidelity without significantly compromising perceptual quality. We achieve this by benefiting from a diverse set of …
Signal Processing Lab (LTS5) EPFL（信号处理实验室（LTS5）洛桑联邦理工学院）
Existing reference (RF)-based super-resolution (SR) models try to improve perceptual quality in SR under the assumption of the availability of high-resolution RF images paired with low-resolution (LR) inputs at testing. As the RF images should be similar in terms of content, colors, contrast, etc. to the test image, this hinders the applicability in a real scenario. Other approaches to increase the perceptual quality of images, including perceptual loss and adversarial losses, tend to dramatically decrease fidelity to the ground-truth through significant decreases in PSNR/SSIM. Addressing both…
