Conducting pairwise comparisons is a widely used approach in curating human perceptual preference data. Typically raters are instructed to make their choices according to a specific set of rules that address certain dimensions of image quality and aesthetics. The outcome of this process is a dataset of sampled image pairs with their associated empirical preference probabilities. Training a model on these pairwise preferences is a common deep learning approach. However, optimizing by gradient descent through mini-batch learning means that the "global" ranking of the images is not explicitly taken into account. In other words, each step of the gradient descent relies only on a limited number of pairwise comparisons. In this work, we demonstrate that regularizing the pairwise empirical probabilities with aggregated rankwise probabilities leads to a more reliable training loss. We show that training a deep image quality assessment model with our rank-smoothed loss consistently improves the accuracy of predicting human preferences.
The Gaussian-smoothed optimal transport (GOT) framework, recently proposed by Goldfeld et al., scales to high dimensions in estimation and provides an alternative to entropy regularization. This paper provides convergence guarantees for estimating the GOT distance under more general settings. For the Gaussian-smoothed $p$-Wasserstein distance in $d$ dimensions, our results require only the existence of a moment greater than $d + 2p$. For the special case of sub-gamma distributions, we quantify the dependence on the dimension $d$ and establish a phase transition with respect to the scale parame…
University of Electronic Science and Technology of China（电子科技大学）；Chinese University of Hong Kong（香港中文大学）
In the field of objective image quality assessment (IQA), the Spearman's $ρ$ and Kendall's $τ$ are two most popular rank correlation indicators, which straightforwardly assign uniform weight to all quality levels and assume each pair of images are sortable. They are successful for measuring the average accuracy of an IQA metric in ranking multiple processed images. However, two important perceptual properties are ignored by them as well. Firstly, the sorting accuracy (SA) of high quality images are usually more important than the poor quality ones in many real world applications, where only th…
Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data
School of Engineering, University of Galway（Galway大学工程学院）；State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University（浙江大学极端光信息获取国家重点实验室）
The Hong Kong Polytechnic University（香港理工大学）；OPPO Research Institute（OPPO研究院）；Harbin Institute of Technology（哈尔滨工业大学）
How to generate the ground-truth (GT) image is a critical issue for training realistic image super-resolution (Real-ISR) models. Existing methods mostly take a set of high-resolution (HR) images as GTs and apply various degradations to simulate their low-resolution (LR) counterparts. Though great progress has been achieved, such an LR-HR pair generation scheme has several limitations. First, the perceptual quality of HR images may not be high enough, limiting the quality of Real-ISR outputs. Second, existing schemes do not consider much human perception in GT generation, and the trained models…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
The ability to estimate the perceptual error between images is an important problem in computer vision with many applications. Although it has been studied extensively, however, no method currently exists that can robustly predict visual differences like humans. Some previous approaches used hand-coded models, but they fail to model the complexity of the human visual system. Others used machine learning to train models on human-labeled datasets, but creating large, high-quality datasets is difficult because people are unable to assign consistent error labels to distorted images. In this paper,…
Image-generating machine learning models are typically trained with loss functions based on distance in the image space. This often leads to over-smoothed results. We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), that mitigate this problem. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric better reflects perceptually similarity of images and thus leads to better results. We show three applications: autoencoder training, a modification of a variational au…
PG-ControlNet: A Physics-Guided ControlNet for Generative Spatially Varying Image Deblurring
1 Computer Science Department, University of Rochester；2 Goergen Institute for Data Science \& AIS, University of Rochester；Computer Engineering Department, University of Rochester
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack high-frequency textures and do not look natural despite yielding high PSNR values. We propose a novel application of automated texture synthesis in c…
Spectral embedding provides a framework for solving perceptual organization problems, including image segmentation and figure/ground organization. From an affinity matrix describing pairwise relationships between pixels, it clusters pixels into regions, and, using a complex-valued extension, orders pixels according to layer. We train a convolutional neural network (CNN) to directly predict the pairwise relationships that define this affinity matrix. Spectral embedding then resolves these predictions into a globally-consistent segmentation and figure/ground organization of the scene. Experiment…
HAPI: A Model for Learning Robot Facial Expressions from Human Preferences
Graduate School of Informatics, Kyoto University（京都大学信息学研究生院）；Psychological Process Research Team, Guardian Robot Project, RIKEN（守护机器人项目心理过程研究组，RIKEN）；Field Science Education and Research Center, Kyoto University（京都大学田野科学教育研究中心）；Interactive Robot Research Team, Guardian Robot Project, RIKEN（守护机器人项目交互机器人研究组，RIKEN）；Research and Development Center for Large Language Models, National Institute of Informatics（信息处理研究所大型语言模型研究开发中心）
Xidian University（西安电子科技大学）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Northwestern Polytechnical University（西北工业大学）
In this paper, we propose a deformable convolution-based generative adversarial network (DCNGAN) for perceptual quality enhancement of compressed videos. DCNGAN is also adaptive to the quantization parameters (QPs). Compared with optical flows, deformable convolutions are more effective and efficient to align frames. Deformable convolutions can operate on multiple frames, thus leveraging more temporal information, which is beneficial for enhancing the perceptual quality of compressed videos. Instead of aligning frames in a pairwise manner, the deformable convolution can process multiple frames…
VersusQ: Pairwise Margin Reasoning for Generalizable Video Quality Assessment
The Chinese University of Hong Kong（香港中文大学）；WeChat Vision, Tencent Inc.（腾讯公司视觉部门）；Beijing Normal University（北京师范大学）；Peking University（北京大学）；The University of Hong Kong（香港大学）
RefReward-SR: LR-Conditioned Reward Modeling for Preference-Aligned Super-Resolution
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；OPPO AI Center, OPPO Inc.（OPPO人工智能中心）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Bokeh effect is used in photography to capture images where the closer objects look sharp and every-thing else stays out-of-focus. Bokeh photos are generally captured using Single Lens Reflex cameras using shallow depth-of-field. Most of the modern smartphones can take bokeh images by leveraging dual rear cameras or a good auto-focus hardware. However, for smartphones with single-rear camera without a good auto-focus hardware, we have to rely on software to generate bokeh images. This kind of system is also useful to generate bokeh effect in already captured images. In this paper, an end-to-en…
ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality
Simon Fraser University（西蒙弗雷泽大学）；University of British Columbia（不列颠哥伦比亚大学）；Eastern Institute of Technology（东部技术学院）；Xi’an Jiaotong University（西安交通大学）；Google Inc（谷歌公司）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Inc.（谷歌公司）
We consider the problem of obtaining image quality representations in a self-supervised manner. We use prediction of distortion type and degree as an auxiliary task to learn features from an unlabeled image dataset containing a mixture of synthetic and realistic distortions. We then train a deep Convolutional Neural Network (CNN) using a contrastive pairwise objective to solve the auxiliary problem. We refer to the proposed training framework and resulting deep IQA model as the CONTRastive Image QUality Evaluator (CONTRIQUE). During evaluation, the CNN weights are frozen and a linear regressor…
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University（上海交通大学 USC-文化创意产业学院）；Institut Universitaire de France (IUF)（法国大学研究院）；University of Nantes（南特大学）
Evaluating Perceptual Distance Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data
School of Engineering Mathematics University of Bristol（工程数学学院英国布里斯托尔大学）；Instituto de Optica Consejo Superior de Investigaciones Científicas (CSIC)（光学研究所（西班牙国家科学研究委员会CSIC））
University of Waterloo（滑铁卢大学）；UBTech Sydney Artificial Intelligence Institute（优必选悉尼人工智能研究院）；The University of Sydney（悉尼大学）
Objective assessment of image quality is fundamentally important in many image processing tasks. In this work, we focus on learning blind image quality assessment (BIQA) models which predict the quality of a digital image with no access to its original pristine-quality counterpart as reference. One of the biggest challenges in learning BIQA models is the conflict between the gigantic image space (which is in the dimension of the number of image pixels) and the extremely limited reliable ground truth data for training. Such data are typically collected via subjective testing, which is cumbersom…
Harbin Institute of Technology（哈尔滨工业大学）；Dalian Maritime University（大连海事大学）；Peng Cheng Laboratory（鹏城实验室）
Full-reference (FR) image quality assessment (IQA) evaluates the visual quality of a distorted image by measuring its perceptual difference with pristine-quality reference, and has been widely used in low-level vision tasks. Pairwise labeled data with mean opinion score (MOS) are required in training FR-IQA model, but is time-consuming and cumbersome to collect. In contrast, unlabeled data can be easily collected from an image degradation or restoration process, making it encouraging to exploit unlabeled training data to boost FR-IQA performance. Moreover, due to the distribution inconsistency…
Wuhan University of Technology（武汉理工大学）；Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks（湖北省宽带无线通信与传感器网络重点实验室）；School of Civil Engineering & Architecture（土木工程与建筑学院）；School of Information Engineering（信息工程学院）；State Key Laboratory of Magnetic Resonance and Atomic Molecular Phys（磁共振与原子分子物理国家重点实验室）
The presence of residual and dense neural networks which greatly promotes the development of image Super-Resolution(SR) have witnessed a lot of impressive results. Depending on our observation, although more layers and connections could always improve performance, the increase of model parameters is not conducive to launch application of SR algorithms. Furthermore, algorithms supervised by L1/L2 loss can achieve considerable performance on traditional metrics such as PSNR and SSIM, yet resulting in blurry and over-smoothed outputs without sufficient high-frequency details, namely low perceptua…
Indian Institute of Technology Delhi（印度理工学院德里分校）；Virginia Tech（弗吉尼亚理工大学）；MIT Media Lab（麻省理工学院媒体实验室）
Computer vision methods that quantify the perception of urban environment are increasingly being used to study the relationship between a city's physical appearance and the behavior and health of its residents. Yet, the throughput of current methods is too limited to quantify the perception of cities across the world. To tackle this challenge, we introduce a new crowdsourced dataset containing 110,988 images from 56 cities, and 1,170,000 pairwise comparisons provided by 81,630 online volunteers along six perceptual attributes: safe, lively, boring, wealthy, depressing, and beautiful. Using thi…
End-to-End PET Image Reconstruction via a Posterior-Mean Diffusion Model
Department of Electrical and Computer Engineering, Rice University, Houston, USA（电气与计算机工程系，里士大学，美国休斯顿）；Department of Imaging Physics, University of Texas MD Anderson Cancer Center, Houston, USA（影像物理系，德克萨斯大学MD安德森癌症中心，美国休斯顿）
Data61 - CSIRO（联邦科学与工业研究组织数据61中心）；ANU（澳大利亚国立大学）；University of Canberra（堪培拉大学）
Variational auto-encoders (VAEs) provide an attractive solution to image generation problem. However, they tend to produce blurred and over-smoothed images due to their dependence on pixel-wise reconstruction loss. This paper introduces a new approach to alleviate this problem in the VAE based generative models. Our model simultaneously learns to match the data, reconstruction loss and the latent distributions of real and fake images to improve the quality of generated samples. To compute the loss distributions, we introduce an auto-encoder based discriminator model which allows an adversarial…
Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation
Electronics and Telecommunications Research Institute（电子和电信研究所）；Dept. of Electronics and Informatics, Vrije Universiteit Brussel（布鲁塞尔自由大学电子与信息系）；IMEC（imec公司）；Department of Computer Science, Texas State University（德克萨斯州立大学计算机科学系）；Embedded Computing Lab., Dept. of Computer Science and Statistics, Federal University of Santa Catarina（圣卡塔琳娜联邦大学计算机科学与统计系嵌入式计算实验室）；Licks Attorneys（利克斯律师事务所）；Federal University of Rio de Janeiro (UFRJ)（里约热内卢联邦大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东理工大学宁波数字孪生研究院）；Instituto Militar de Engenharia-IME, PGEE/PGED（军事工程学院-IME，PGEE/PGED）
