While humans can effortlessly transform complex visual scenes into simple words and the other way around by leveraging their high-level understanding of the content, conventional or the more recent learned image compression codecs do not seem to utilize the semantic meanings of visual content to their full potential. Moreover, they focus mostly on rate-distortion and tend to underperform in perception quality especially in low bitrate regime, and often disregard the performance of downstream computer vision algorithms, which is a fast-growing consumer group of compressed images in addition to human viewers. In this paper, we (1) present a generic framework that can enable any image codec to leverage high-level semantics and (2) study the joint optimization of perception quality and distortion. Our idea is that given any codec, we utilize high-level semantics to augment the low-level visual features extracted by it and produce essentially a new, semantic-aware codec. We propose a three-phase training scheme that teaches semantic-aware codecs to leverage the power of semantic to jointly optimize rate-perception-distortion (R-PD) performance. As an additional benefit, semantic-aware codecs also boost the performance of downstream computer vision algorithms. To validate our claim, we perform extensive empirical evaluations and provide both quantitative and qualitative results.
The lower the distortion of an estimator, the more the distribution of its outputs generally deviates from the distribution of the signals it attempts to estimate. This phenomenon, known as the perception-distortion tradeoff, has captured significant attention in image restoration, where it implies that fidelity to ground truth images comes at the expense of perceptual quality (deviation from statistics of natural images). However, despite the increasing popularity of performing comparisons on the perception-distortion plane, there remains an important open question: what is the minimal distor…
National University of Singapore（新加坡国立大学）；HiSilicon Technologies（海思半导体）
In image super-resolution, both pixel-wise accuracy and perceptual fidelity are desirable. However, most deep learning methods only achieve high performance in one aspect due to the perception-distortion trade-off, and works that successfully balance the trade-off rely on fusing results from separately trained models with ad-hoc post-processing. In this paper, we propose a novel super-resolution model with a low-frequency constraint (LFc-SR), which balances the objective and perceptual quality through a single model and yields super-resolved images with high PSNR and perceptual scores. We furt…
Technion – Israel Institute of Technology（以色列理工学院）
Image restoration algorithms are typically evaluated by some distortion measure (e.g. PSNR, SSIM, IFC, VIF) or by human opinion scores that quantify perceived perceptual quality. In this paper, we prove mathematically that distortion and perceptual quality are at odds with each other. Specifically, we study the optimal probability for correctly discriminating the outputs of an image restoration algorithm from real images. We show that as the mean distortion decreases, this probability must increase (indicating worse perceptual quality). As opposed to the common belief, this result holds true f…
Flow-based generative super-resolution (SR) models learn to produce a diverse set of feasible SR solutions, called the SR space. Diversity of SR solutions increases with the temperature ($τ$) of latent variables, which introduces random variations of texture among sample solutions, resulting in visual artifacts and low fidelity. In this paper, we present a simple but effective image ensembling/fusion approach to obtain a single SR image eliminating random artifacts and improving fidelity without significantly compromising perceptual quality. We achieve this by benefiting from a diverse set of …
PIGUIQA: A Physical Imaging Guided Perceptual Framework for Underwater Image Quality Assessment
Chongqing Research Institute of Harbin Institute of Technology（哈尔滨工业大学重庆研究所）；Harbin Institute of Technology（哈尔滨工业大学）；Faculty of Computing（计算机学院）；School of Computer Science（计算机科学学院）；University of Macau（澳门大学）；Institute for Infocomm Research（信息与通信研究所以）
Perceptual quality assessment of the videos acquired in the wilds is of vital importance for quality assurance of video services. The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task. Although model-based transfer learning is an effective and efficient paradigm for the BVQA task, it remains to be a challenge to explore what and how to bridge the domain shifts for better video representation. In this work, we propose to transfer knowledge from image quality asse…
Perception-oriented Bidirectional Attention Network for Image Super-resolution Quality Assessment
School of Mathematical Sciences, Beihang University（北京航空航天大学数学学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）；CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, University of Science and Technology of China（中国科学技术大学地球空间信息处理与应用系统国家重点实验室）；School of Information Science and Engineering, Ningbo University（宁波大学信息科学与工程学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
Seoul National University（首尔国立大学）；Samsung Electronics（三星电子）
Single-image super-resolution (SISR) networks trained with perceptual and adversarial losses provide high-contrast outputs compared to those of networks trained with distortion-oriented losses, such as L1 or L2. However, it has been shown that using a single perceptual loss is insufficient for accurately restoring locally varying diverse shapes in images, often generating undesirable artifacts or unnatural details. For this reason, combinations of various losses, such as perceptual, adversarial, and distortion losses, have been attempted, yet it remains challenging to find optimal combinations…
Gustave Eiffel University（古斯塔夫·埃菲尔大学）；Cardiff University（卡迪夫大学）；Xi'an Jiaotong University（西安交通大学）；Zhongkai University of Agriculture and Engineering（仲恺农业工程学院）；University of Bath（巴斯大学）；Xidian University（西安电子科技大学）
Research on image quality assessment (IQA) remains limited mainly due to our incomplete knowledge about human visual perception. Existing IQA algorithms have been designed or trained with insufficient subjective data with a small degree of stimulus variability. This has led to challenges for those algorithms to handle complexity and diversity of real-world digital content. Perceptual evidence from human subjects serves as a grounding for the development of advanced IQA algorithms. It is thus critical to acquire reliable subjective data with controlled perception experiments that faithfully ref…
Objective image quality assessment deals with the prediction of digital images' perceptual quality. No-reference image quality assessment predicts the quality of a given input image without any knowledge or information about its pristine (distortion free) counterpart. Machine learning algorithms are heavily used in no-reference image quality assessment because it is very complicated to model the human visual system's quality perception. Moreover, no-reference image quality assessment algorithms are evaluated on publicly available benchmark databases. These databases contain images with their c…
Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China（信息与通信工程学院和媒体融合与通信国家重点实验室，中国通信大学）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（信息科学与电子工程学院，上海交通大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；State Key Laboratory of Multimedia information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）
BPCLIP: A Bottom-up Image Quality Assessment from Distortion to Semantics Based on CLIP
Harbin Institute of Technology（哈尔滨工业大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Sichuan University of Science & Engineering（四川工程职业技术学院）
University of Science and Technology of China（中国科学技术大学）
Deep neural networks (DNNs) have shown great potential in non-reference image quality assessment (NR-IQA). However, the annotation of NR-IQA is labor-intensive and time-consuming, which severely limits their application especially for authentic images. To relieve the dependence on quality annotation, some works have applied unsupervised domain adaptation (UDA) to NR-IQA. However, the above methods ignore that the alignment space used in classification is sub-optimal, since the space is not elaborately designed for perception. To solve this challenge, we propose an effective perception-oriented…
Mila, Quebec AI Institute（Mila，魁北克人工智能研究所）；Independent Robotics；Google Brain Inc.（谷歌大脑公司）；Polytechnique Montréal（蒙特利尔理工学院）；ElementAI；Service Now
Recent work has shown that learned image compression strategies can outperform standard hand-crafted compression algorithms that have been developed over decades of intensive research on the rate-distortion trade-off. With growing applications of computer vision, high quality image reconstruction from a compressible representation is often a secondary objective. Compression that ensures high accuracy on computer vision tasks such as image segmentation, classification, and detection therefore has the potential for significant impact across a wide variety of settings. In this work, we develop a …
Beijing Jiaotong University（北京交通大学）；Institute of Information Science（信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京先进信息科学与网络技术重点实验室）
Fisheye lens gains increasing applications in computational photography and assisted driving because of its wide field of view (FoV). However, the fisheye image generally contains invalid black regions induced by its imaging model. In this paper, we present a FisheyeEX method that extends the FoV of the fisheye lens by outpainting the invalid regions, improving the integrity of captured scenes. Compared with the rectangle and undistorted image, there are two challenges for fisheye image outpainting: irregular painting regions and distortion synthesis. Observing the radial symmetry of the fishe…
Stimulating Diffusion Model for Image Denoising via Adaptive Embedding and Ensembling
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
Rate-distortion optimization (RDO) of codecs, where distortion is quantified by the mean-square error, has been a standard practice in image/video compression over the years. RDO serves well for optimization of codec performance for evaluation of the results in terms of PSNR. However, it is well known that the PSNR does not correlate well with perceptual evaluation of images; hence, RDO is not well suited for perceptual optimization of codecs. Recently, rate-distortion-perception trade-off has been formalized by taking the Kullback-Leibner (KL) divergence between the distributions of the origi…
Content-Distortion High-Order Interaction for Blind Image Quality Assessment
College of Applied Technology, Shenzhen University（深圳大学应用技术学院）；College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）；College of Electronic and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；School of Electronics and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学未来媒体网络协同创新中心）；Nanjing University（南京大学）
In this paper, we propose a new distortion quantification method for point clouds, the multiscale potential energy discrepancy (MPED). Currently, there is a lack of effective distortion quantification for a variety of point cloud perception tasks. Specifically, for dense point clouds, a distortion quantification method is used to predict human subjective scores and optimize the selection of human perception tasks parameters, such as compression and enhancement. For sparse point clouds, a distortion quantification methods is work as a loss function to guide the training of deep neural networks …
School of Electrical and Computer Engineering Cornell University（康奈尔大学电气与计算机工程学院）；Cornell University（康奈尔大学）
A rate-distortion-perception (RDP) tradeoff has recently been proposed by Blau and Michaeli and also Matsumoto. Focusing on the case of perfect realism, which coincides with the problem of distribution-preserving lossy compression studied by Li et al., a coding theorem for the RDP tradeoff that allows for a specified amount of common randomness between the encoder and decoder is provided. The existing RDP tradeoff is recovered by allowing for the amount of common randomness to be infinite. The quadratic Gaussian case is examined in detail.
Indian Institute of Information Technology, Design and Manufacturing Kancheepuram（印度坎奇普拉姆信息技术设计与制造学院）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Numerous stereoscopic 3D movies are released every year to theaters and created large revenues. Despite the improvement in stereo capturing and 3D video post-production technology, stereoscopic artifacts which cause viewer discomfort continue to appear even in high-budget films. Existing automatic 3D video quality measurement tools can detect distortions in stereoscopic images or videos, but they fail to consider the viewer's subjective perception of those artifacts, and how these distortions affect their choices. In this paper, we introduce a novel recommendation system for stereoscopic 3D mo…
End-to-End PET Image Reconstruction via a Posterior-Mean Diffusion Model
Department of Electrical and Computer Engineering, Rice University, Houston, USA（电气与计算机工程系，里士大学，美国休斯顿）；Department of Imaging Physics, University of Texas MD Anderson Cancer Center, Houston, USA（影像物理系，德克萨斯大学MD安德森癌症中心，美国休斯顿）
This paper describes a quality assessment model for perceptual video compression applications (PVM), which stimulates visual masking and distortion-artefact perception using an adaptive combination of noticeable distortions and blurring artefacts. The method shows significant improvement over existing quality metrics based on the VQEG database, and provides compatibility with in-loop rate-quality optimisation for next generation video codecs due to its latency and complexity attributes. Performance comparison are validated against a range of different distortion types.
