Recently, learned image compression schemes have achieved remarkable improvements in image fidelity (e.g., PSNR and MS-SSIM) compared to conventional hybrid image coding ones due to their high-efficiency non-linear transform, end-to-end optimization frameworks, etc. However, few of them take the Just Noticeable Difference (JND) characteristic of the Human Visual System (HVS) into account and optimize learned image compression towards perceptual quality. To address this issue, a JND-based perceptual quality loss is proposed. Considering that the amounts of distortion in the compressed image at different training epochs under different Quantization Parameters (QPs) are different, we develop a distortion-aware adjustor. After combining them together, we can better assign the distortion in the compressed image with the guidance of JND to preserve the high perceptual quality. All these designs enable the proposed method to be flexibly applied to various learned image compression schemes with high scalability and plug-and-play advantages. Experimental results on the Kodak dataset demonstrate that the proposed method has led to better perceptual quality than the baseline model under the same bit rate.
Ningbo University（宁波大学）；City University of Hong Kong（香港城市大学）；Nanyang Technological University（南洋理工大学）
Just noticeable difference (JND) of natural images refers to the maximum pixel intensity change magnitude that typical human visual system (HVS) cannot perceive. Existing efforts on JND estimation mainly dedicate to modeling the diverse masking effects in either/both spatial or/and frequency domains, and then fusing them into an overall JND estimate. In this work, we turn to a dramatically different way to address this problem with a top-down design philosophy. Instead of explicitly formulating and fusing different masking effects in a bottom-up way, the proposed JND estimation model dedicates…
Recently, due to the strength of deep convolutional neural networks (CNN), many CNN-based image quality assessment (IQA) models have been studied. However, previous CNN-based IQA models likely have yet to utilize the characteristics of the human visual system (HVS) fully for IQA problems when they simply entrust everything to the CNN, expecting it to learn from a training dataset. However, in this paper, we propose a novel saliency-channel attention residual network based on the just-noticeable-difference (JND) concept for full-reference image quality assessments (FR-IQA). It is referred to as…
Lossy Image compression is necessary for efficient storage and transfer of data. Typically the trade-off between bit-rate and quality determines the optimal compression level. This makes the image quality metric an integral part of any imaging system. While the existing full-reference metrics such as PSNR and SSIM may be less sensitive to perceptual quality, the recently introduced learning methods may fail to generalize to unseen data. In this paper we propose the largest image compression quality dataset to date with human perceptual preferences, enabling the use of deep learning, and we dev…
Czech Technical University in Prague（布拉格捷克理工大学）；Amazon Web Services（亚马逊云科技）
This paper proposes a new end-to-end trainable model for lossy image compression, which includes several novel components. The method incorporates 1) an adequate perceptual similarity metric; 2) saliency in the images; 3) a hierarchical auto-regressive model. This paper demonstrates that the popularly used evaluations metrics such as MS-SSIM and PSNR are inadequate for judging the performance of image compression techniques as they do not align with the human perception of similarity. Alternatively, a new metric is proposed, which is learned on perceptual similarity data specific to image comp…
Center for Machine Perception, Czech Technical University（捷克技术大学机器感知中心）；Amazon（亚马逊）
Recently, there has been much interest in deep learning techniques to do image compression and there have been claims that several of these produce better results than engineered compression schemes (such as JPEG, JPEG2000 or BPG). A standard way of comparing image compression schemes today is to use perceptual similarity metrics such as PSNR or MS-SSIM (multi-scale structural similarity). This has led to some deep learning techniques which directly optimize for MS-SSIM by choosing it as a loss function. While this leads to a higher MS-SSIM for such techniques, we demonstrate using user studie…
Several deep learned lossy compression techniques have been proposed in the recent literature. Most of these are optimized by using either MS-SSIM (multi-scale structural similarity) or MSE (mean squared error) as a loss function. Unfortunately, neither of these correlate well with human perception and this is clearly visible from the resulting compressed images. In several cases, the MS-SSIM for deep learned techniques is higher than say a conventional, non-deep learned codec such as JPEG-2000 or BPG. However, the images produced by these deep learned techniques are in many cases clearly wors…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Netflix Inc.（Netflix公司）
Mean squared error (MSE) and $\ell_p$ norms have largely dominated the measurement of loss in neural networks due to their simplicity and analytical properties. However, when used to assess visual information loss, these simple norms are not highly consistent with human perception. Here, we propose a different proxy approach to optimize image analysis networks against quantitative perceptual models. Specifically, we construct a proxy network, which mimics the perceptual model while serving as a loss layer of the network.We experimentally demonstrate how this optimization framework can be appli…
In subjective full-reference image quality assessment, differences between perceptual image qualities of the reference image and its distorted versions are evaluated, often using degradation category ratings (DCR). However, the DCR has been criticized since differences between rating categories on this ordinal scale might not be perceptually equidistant, and observers may have different understandings of the categories. Pair comparisons (PC) of distorted images, followed by Thurstonian reconstruction of scale values, overcome these problems. In addition, PC is more sensitive than DCR, and it c…
Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Shandong Normal University（山东师范大学）；Beijing Jiao Tong University（北京交通大学）
Significant improvement has been made on just noticeable difference (JND) modelling due to the development of deep neural networks, especially for the recently developed unsupervised-JND generation models. However, they have a major drawback that the generated JND is assessed in the real-world signal domain instead of in the perceptual domain in the human brain. There is an obvious difference when JND is assessed in such two domains since the visual signal in the real world is encoded before it is delivered into the brain with the human visual system (HVS). Hence, we propose an HVS-inspired si…
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）
Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Tsinghua University（清华大学）；Beijing Jiao Tong University（北京交通大学）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京市先进信息科学与网络技术重点实验室）；Alibaba Cloud Business Group（阿里巴巴云业务集团）
As an important perceptual characteristic of the Human Visual System (HVS), the Just Noticeable Difference (JND) has been studied for decades with image and video processing (e.g., perceptual visual signal compression). However, there is little exploration on the existence of JND for the Deep Machine Vision (DMV), although the DMV has made great strides in many machine vision tasks. In this paper, we take an initial attempt, and demonstrate that the DMV has the JND, termed as the DMV-JND. We then propose a JND model for the image classification task in the DMV. It has been discovered that the …
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Netflix Inc.（奈飞公司）
The use of $\ell_p$ $(p=1,2)$ norms has largely dominated the measurement of loss in neural networks due to their simplicity and analytical properties. However, when used to assess the loss of visual information, these simple norms are not very consistent with human perception. Here, we describe a different "proximal" approach to optimize image analysis networks against quantitative perceptual models. Specifically, we construct a proxy network, broadly termed ProxIQA, which mimics the perceptual model while serving as a loss layer of the network. We experimentally demonstrate how this optimiza…
Institute for AI Industry Research, Tsinghua University（清华人工智能产业研究院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；Hong Kong University of Science and Technology（香港科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Beijing University of Posts and Telecommunications（北京邮电大学）
Conventional image compression methods typically aim at pixel-level consistency while ignoring the performance of downstream AI tasks.To solve this problem, this paper proposes a Semantic-Assisted Image Compression method (SAIC), which can maintain semantic-level consistency to enable high performance of downstream AI tasks.To this end, we train the compression network using semantic-level loss function. In particular, semantic-level loss is measured using gradient-based semantic weights mechanism (GSW). GSW directly consider downstream AI tasks' perceptual results. Then, this paper proposes a…
We propose a deep learning system for attention-guided dual-layer image compression (AGDL). In the AGDL compression system, an image is encoded into two layers, a base layer and an attention-guided refinement layer. Unlike the existing ROI image compression methods that spend an extra bit budget equally on all pixels in ROI, AGDL employs a CNN module to predict those pixels on and near a saliency sketch within ROI that are critical to perceptual quality. Only the critical pixels are further sampled by compressive sensing (CS) to form a very compact refinement layer. Another novel CNN method is…
Typically, metadata of images are stored in a specific data segment of the image file. However, to securely detect changes, data can also be embedded within images. This follows the goal to invisibly and robustly embed as much information as possible to, ideally, even survive compression. This work searches for embedding principles which allow to distinguish between unintended changes by lossy image compression and malicious manipulation of the embedded message based on the change of its perceptual or robust hash. Different embedding and compression algorithms are compared. The study shows tha…
We extensively study how to combine Generative Adversarial Networks and learned compression to obtain a state-of-the-art generative lossy compression system. In particular, we investigate normalization layers, generator and discriminator architectures, training strategies, as well as perceptual losses. In contrast to previous work, i) we obtain visually pleasing reconstructions that are perceptually similar to the input, ii) we operate in a broad range of bitrates, and iii) our approach can be applied to high-resolution images. We bridge the gap between rate-distortion-perception theory and pr…
Perceptual Image Compression with Cooperative Cross-Modal Side Information
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能重点实验室）；Huawei Technologies Company Ltd（华为技术有限公司）；Shenzhen University（深圳大学）
Compact Latent Representation for Image Compression (CLRIC)
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所）；Department of Physics, Faculty of Science, Sohag University（科学学院物理系，索哈格大学）；Friedrich-Alexander University at Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学）
本文提出 CLRIC，利用预训练模型潜在变量和过拟合可学习函数实现任意质量级别的感知图像压缩，兼具低复杂度、模型与分辨率无关性及接近 SOTA 的质量。
In many professional fields, such as medicine, remote sensing and sciences, users often demand image compression methods to be mathematically lossless. But lossless image coding has a rather low compression ratio (around 2:1 for natural images). The only known technique to achieve significant compression while meeting the stringent fidelity requirements is the methodology of $\ell_\infty$-constrained coding that was developed and standardized in nineties. We make a major progress in $\ell_\infty$-constrained image coding after two decades, by developing a novel CNN-based soft $\ell_\infty$-con…
JPEG AIC2026: A large-scale dataset for fine-grained assessment of image coding
University of Konstanz（康斯坦茨大学）；Cloudinary（Cloudinary公司）；IST-IT（IST-IT机构）；Fraunhofer IIS（弗劳恩霍夫应用固体物理研究所）；Huawei（华为公司）；Takushoku University（拓殖大学）；IT-UBI（IT-UBI机构）；EPFL（洛桑联邦理工学院）
Recent years have witnessed the rapid development of image storage and transmission systems, in which image compression plays an important role. Generally speaking, image compression algorithms are developed to ensure good visual quality at limited bit rates. However, due to the different compression optimization methods, the compressed images may have different levels of quality, which needs to be evaluated quantificationally. Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of co…
