The explosive growth of image data facilitates the fast development of image processing and computer vision methods for emerging visual applications, meanwhile introducing novel distortions to the processed images. This poses a grand challenge to existing blind image quality assessment (BIQA) models, failing to continually adapt to such subpopulation shift. Recent work suggests training BIQA methods on the combination of all available human-rated IQA datasets. However, this type of approach is not scalable to a large number of datasets, and is cumbersome to incorporate a newly created dataset as well. In this paper, we formulate continual learning for BIQA, where a model learns continually from a stream of IQA datasets, building on what was learned from previously seen data. We first identify five desiderata in the new setting with a measure to quantify the plasticity-stability trade-off. We then propose a simple yet effective method for learning BIQA models continually. Specifically, based on a shared backbone network, we add a prediction head for a new dataset, and enforce a regularizer to allow all prediction heads to evolve with new data while being resistant to catastrophic forgetting of old data. We compute the quality score by an adaptive weighted summation of estimates from all prediction heads. Extensive experiments demonstrate the promise of the proposed continual learning method in comparison to standard training techniques for BIQA. We made the code publicly available at https://github.com/zwx8981/BIQA_CL.
Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）
Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization
Department of Psychological and Cognitive Sciences, Tsinghua University（心理与认知科学系，清华大学）；Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University（人工智能研究院、北京人工智能研究所、清华大学-博世联合机器学习中心、THBI实验室，清华大学）；Department of Computer Science, Durham University（计算机科学系，杜尔姆大学）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；Zhongguancun Laboratory, Beijing（中关村实验室，北京）
LaTIM UMR 1101, Inserm（法国国家健康与医学研究院LaTIM UMR 1101实验室）；University of Western Brittany（西布列塔尼大学）；University of Electronic Science and Technology of China（电子科技大学）；IMT Atlantique（大西洋国立高等矿业电信学院）
Diabetic Retinopathy (DR) is a severe complication of diabetes that can cause blindness. Although effective treatments exist (notably laser) to slow the progression of the disease and prevent blindness, the best treatment remains prevention through regular check-ups (at least once a year) with an ophthalmologist. Optical Coherence Tomography Angiography (OCTA) allows for the visualization of the retinal vascularization, and the choroid at the microvascular level in great detail. This allows doctors to diagnose DR with more precision. In recent years, algorithms for DR diagnosis have emerged al…
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）
Large-scale datasets play a vital role in computer vision. But current datasets are annotated blindly without differentiation to samples, making the data collection inefficient and unscalable. The open question is how to build a mega-scale dataset actively. Although advanced active learning algorithms might be the answer, we experimentally found that they are lame in the realistic annotation scenario where out-of-distribution data is extensive. This work thus proposes a novel active learning framework for realistic dataset annotation. Equipped with this framework, we build a high-quality visio…
CL-VISTA: Benchmarking Continual Learning in Video Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学前沿交叉科学学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院福建省模式识别与图像理解重点实验室）
Blind Image Quality Assessment Using Multi-Stream Architecture with Spatial and Channel Attention
Xi’an Eurasia University（西安欧亚学院）；Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences（中国科学院长春光学精密机械与物理研究所）；University of Engineering and Technology Lahore（拉合尔工程技术大学）
University of Oulu（奥卢大学）；Nanyang Technological University（南洋理工大学）；ByteDance（字节跳动）；National University of Defense Technology（国防科技大学）
The goal of Few-Shot Continual Learning (FSCL) is to incrementally learn novel tasks with limited labeled samples and preserve previous capabilities simultaneously, while current FSCL methods are all for the class-incremental purpose. Moreover, the evaluation of FSCL solutions is only the cumulative performance of all encountered tasks, but there is no work on exploring the domain generalization ability. Domain generalization is a challenging yet practical task that aims to generalize beyond training domains. In this paper, we set up a Generalized FSCL (GFSCL) protocol involving both class- an…
Extracting discriminative features plays a crucial role in the fine-grained visual classification task. Most of the existing methods focus on developing attention or augmentation mechanisms to achieve this goal. However, addressing the ambiguity in the top-k prediction classes is not fully investigated. In this paper, we introduce a Self Assessment Classifier, which simultaneously leverages the representation of the image and top-k prediction classes to reassess the classification results. Our method is inspired by continual learning with coarse-grained and fine-grained classifiers to increase…
University of Würzburg（维尔茨堡大学）；Institute of Computer Science（计算机科学研究所）
Generative models for image restoration, enhancement, and generation have significantly improved the quality of the generated images. Surprisingly, these models produce more pleasant images to the human eye than other methods, yet, they may get a lower perceptual quality score using traditional perceptual quality metrics such as PSNR or SSIM. Therefore, it is necessary to develop a quantitative metric to reflect the performance of new algorithms, which should be well-aligned with the person's mean opinion score (MOS). Learning-based approaches for perceptual image quality assessment (IQA) usua…
University of Engineering and Technology Lahore（拉合尔工程技术大学）
The assessment of the perceptual quality of digital images is becoming increasingly important as a result of the widespread use of digital multimedia devices. Smartphones and high-speed internet are just two examples of technologies that have multiplied the amount of multimedia content available. Thus, obtaining a representative dataset, which is required for objective quality assessment training, is a significant challenge. The Blind Image Quality Assessment Database, BIQ2021, is presented in this article. By selecting images with naturally occurring distortions and reliable labeling, the dat…
This short paper presents a perspective plan to build a null reference image quality assessment. Its main goal is to deliver both the objective score and the distortion map for a given distorted image without the knowledge of its reference image.
City University of Hong Kong（香港城市大学）；Peng Cheng Laboratory（鹏城实验室）
Ensemble methods are generally regarded to be better than a single model if the base learners are deemed to be "accurate" and "diverse." Here we investigate a semi-supervised ensemble learning strategy to produce generalizable blind image quality assessment models. We train a multi-head convolutional network for quality prediction by maximizing the accuracy of the ensemble (as well as the base learners) on labeled data, and the disagreement (i.e., diversity) among them on unlabeled data, both implemented by the fidelity loss. We conduct extensive experiments to demonstrate the advantages of em…
This paper addresses the problem of blind stereoscopic image quality assessment (NR-SIQA) using a new multi-task deep learning based-method. In the field of stereoscopic vision, the information is fairly distributed between the left and right views as well as the binocular phenomenon. In this work, we propose to integrate these characteristics to estimate the quality of stereoscopic images without reference through a convolutional neural network. Our method is based on two main tasks: the first task predicts naturalness analysis based features adapted to stereo images, while the second task pr…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook Reality Labs（Facebook现实实验室）；NVIDIA（英伟达）
Previous blind or No Reference (NR) video quality assessment (VQA) models largely rely on features drawn from natural scene statistics (NSS), but under the assumption that the image statistics are stationary in the spatial domain. Several of these models are quite successful on standard pictures. However, in Virtual Reality (VR) applications, foveated video compression is regaining attention, and the concept of space-variant quality assessment is of interest, given the availability of increasingly high spatial and temporal resolution contents and practical ways of measuring gaze direction. Dis…
Wuhan University（武汉大学）；New York University（纽约大学）；University of Waterloo（滑铁卢大学）
We propose a deep bilinear model for blind image quality assessment (BIQA) that handles both synthetic and authentic distortions. Our model consists of two convolutional neural networks (CNN), each of which specializes in one distortion scenario. For synthetic distortions, we pre-train a CNN to classify image distortion type and level, where we enjoy large-scale training data. For authentic distortions, we adopt a pre-trained CNN for image classification. The features from the two CNNs are pooled bilinearly into a unified representation for final quality prediction. We then fine-tune the entir…
The research in image quality assessment (IQA) has a long history, and significant progress has been made by leveraging recent advances in deep neural networks (DNNs). Despite high correlation numbers on existing IQA datasets, DNN-based models may be easily falsified in the group maximum differentiation (gMAD) competition with strong counterexamples being identified. Here we show that gMAD examples can be used to improve blind IQA (BIQA) methods. Specifically, we first pre-train a DNN-based BIQA model using multiple noisy annotators, and fine-tune it on multiple subject-rated databases of synt…
Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The University of Tokyo（东京大学）；University of Maryland, College Park（马里兰大学帕克分校）；Pingan Group（平安集团）
Stochastic BIQA: Median Randomized Smoothing for Certified Blind Image Quality Assessment
MSU Institute for Artificial Intelligence（莫斯科国立大学人工智能研究所）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院信息学与自动化研究所可信人工智能研究中心）；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；Artificial Intelligence Research Institute（人工智能研究所）；Innopolis University（因诺波利斯大学）
A key problem in blind image quality assessment (BIQA) is how to effectively model the properties of human visual system in a data-driven manner. In this paper, we propose a simple and efficient BIQA model based on a novel framework which consists of a fully convolutional neural network (FCNN) and a pooling network to solve this problem. In principle, FCNN is capable of predicting a pixel-by-pixel similar quality map only from a distorted image by using the intermediate similarity maps derived from conventional full-reference image quality assessment methods. The predicted pixel-by-pixel quali…
University of Science and Technology of China（中国科学技术大学）；CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application System（中国科学院地理空间信息处理与应用系统技术重点实验室）
Existing learning-based methods for blind image quality assessment (BIQA) are heavily dependent on large amounts of annotated training data, and usually suffer from a severe performance degradation when encountering the domain/distribution shift problem. Thanks to the development of unsupervised domain adaptation (UDA), some works attempt to transfer the knowledge from a label-sufficient source domain to a label-free target domain under domain shift with UDA. However, it requires the coexistence of source and target data, which might be impractical for source data due to the privacy or storage…
Collaborative Auto-encoding for Blind Image Quality Assessment
Shenzhen University（深圳大学）；Peng Cheng Laboratory（鹏城实验室）；Shenzhen Institute for Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；University of Nottingham（诺丁汉大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google LLC（谷歌有限责任公司）
Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC content are unpredictable, complicated, and often commingled. Here we contrib…
From Images to Point Clouds: An Efficient Solution for Cross-media Blind Quality Assessment without Annotated Training
Cooperative Medianet Innovation Center, Shanghai Jiaotong University（上海交通大学合作Medianet创新中心）；University of Missouri-Kansas City（密苏里大学堪萨斯分校）；University of Canterbury（坎特伯雷大学）
MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment
Beihang University（北京航空航天大学）；Durham University（杜伦大学）；Tsinghua University（清华大学）；Children’s Hospital of Capital Institute of Pediatrics（首都儿科研究所附属儿童医院）；Zhongguancun Laboratory（中关村实验室）
Omnidirectional image quality assessment (OIQA) aims to predict the perceptual quality of omnidirectional images that cover the whole 180$\times$360$^{\circ}$ viewing range of the visual environment. Here we propose a blind/no-reference OIQA method named S$^2$ that bridges the gap between low-level statistics and high-level semantics of omnidirectional images. Specifically, statistic and semantic features are extracted in separate paths from multiple local viewports and the hallucinated global omnidirectional image, respectively. A quality regression along with a weighting process is then foll…
Perceptual quality assessment of the videos acquired in the wilds is of vital importance for quality assurance of video services. The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task. Although model-based transfer learning is an effective and efficient paradigm for the BVQA task, it remains to be a challenge to explore what and how to bridge the domain shifts for better video representation. In this work, we propose to transfer knowledge from image quality asse…
In this work we investigate the use of deep learning for distortion-generic blind image quality assessment. We report on different design choices, ranging from the use of features extracted from pre-trained Convolutional Neural Networks (CNNs) as a generic image description, to the use of features extracted from a CNN fine-tuned for the image quality task. Our best proposal, named DeepBIQ, estimates the image quality by average pooling the scores predicted on multiple sub-regions of the original image. The score of each sub-region is computed using a Support Vector Regression (SVR) machine tak…
Numerous image superresolution (SR) algorithms have been proposed for reconstructing high-resolution (HR) images from input images with lower spatial resolutions. However, effectively evaluating the perceptual quality of SR images remains a challenging research problem. In this paper, we propose a no-reference/blind deep neural network-based SR image quality assessor (DeepSRQ). To learn more discriminative feature representations of various distorted SR images, the proposed DeepSRQ is a two-stream convolutional network including two subcomponents for distorted structure and texture SR images. …
The enormous space and diversity of natural images is usually represented by a few small-scale human-rated image quality assessment (IQA) datasets. This casts great challenges to deep neural network (DNN) based blind IQA (BIQA), which requires large-scale training data that is representative of the natural image distribution. It is extremely difficult to create human-rated IQA datasets composed of millions of images due to constraints of subjective testing. While a number of efforts have focused on design innovations to enhance the performance of DNN based BIQA, attempts to address the scarcit…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Inc.（谷歌公司）
Many objective video quality assessment (VQA) algorithms include a key step of temporal pooling of frame-level quality scores. However, less attention has been paid to studying the relative efficiencies of different pooling methods on no-reference (blind) VQA. Here we conduct a large-scale comparative evaluation to assess the capabilities and limitations of multiple temporal pooling strategies on blind VQA of user-generated videos. The study yields insights and general guidance regarding the application and selection of temporal pooling models. In addition, we also propose an ensemble pooling …
DEFNet: Multitasks-based Deep Evidential Fusion Network for Blind Image Quality Assessment
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术重点实验室（北京大学）教育部）；School of Computer Science, Peking University（计算机学院（北京大学））
Shanghai Jiao Tong University（上海交通大学）；City University of Hong Kong（香港城市大学）
Performance of blind image quality assessment (BIQA) models has been significantly boosted by end-to-end optimization of feature engineering and quality regression. Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa). To confront the cross-distortion-scenario challenge, we develop a \textit{unified} BIQA model and an approach of training it for both synthetic and realistic distortions. We first sample p…
Blind image quality assessment (BIQA) aims to automatically evaluate the perceived quality of a single image, whose performance has been improved by deep learning-based methods in recent years. However, the paucity of labeled data somewhat restrains deep learning-based BIQA methods from unleashing their full potential. In this paper, we propose to solve the problem by a pretext task customized for BIQA in a self-supervised learning manner, which enables learning representations from orders of magnitude more data. To constrain the learning process, we propose a quality-aware contrastive loss ba…
School of Electronic Information, Wuhan University（武汉大学电子信息学院）
The goal in a blind image quality assessment (BIQA) model is to simulate the process of evaluating images by human eyes and accurately assess the quality of the image. Although many approaches effectively identify degradation, they do not fully consider the semantic content in images resulting in distortion. In order to fill this gap, we propose a deep adaptive superpixel-based network, namely DSN-IQA, to assess the quality of image based on multi-scale and superpixel segmentation. The DSN-IQA can adaptively accept arbitrary scale images as input images, making the assessment process similar t…
