Deep Convolutional Neural Network (CNN) features have been demonstrated to be effective perceptual quality features. The perceptual loss, based on feature maps of pre-trained CNN's has proven to be remarkably effective for CNN based perceptual image restoration problems. In this work, taking inspiration from the the Human Visual System (HVS) and visual perception, we propose a spatial attention mechanism based on the dependency human contrast sensitivity on spatial frequency. We identify regions in input images, based on the underlying spatial frequency, which are not generally well reconstructed during Super-Resolution but are most important in terms of visual sensitivity. Based on this prior, we design a spatial attention map that is applied to feature maps in the perceptual loss and its variants, helping them to identify regions that are of more perceptual importance. The results demonstrate the our technique improves the ability of the perceptual loss and contextual loss to deliver more natural images in CNN based super-resolution.
Low-light image enhancement is a promising solution to tackle the problem of insufficient sensitivity of human vision system (HVS) to perceive information in low light environments. Previous Retinex-based works always accomplish enhancement task by estimating light intensity. Unfortunately, single light intensity modelling is hard to accurately simulate visual perception information, leading to the problems of imbalanced visual photosensitivity and weak adaptivity. To solve these problems, we explore the precise relationship between light source and visual perception and then propose the visua…
University of Science and Technology of China（中国科学技术大学）
Deep neural networks (DNNs) have shown great potential in non-reference image quality assessment (NR-IQA). However, the annotation of NR-IQA is labor-intensive and time-consuming, which severely limits their application especially for authentic images. To relieve the dependence on quality annotation, some works have applied unsupervised domain adaptation (UDA) to NR-IQA. However, the above methods ignore that the alignment space used in classification is sub-optimal, since the space is not elaborately designed for perception. To solve this challenge, we propose an effective perception-oriented…
Recent studies of deep learning based stereo image super-resolution (StereoSR) have promoted the development of StereoSR. However, existing StereoSR models mainly concentrate on improving quantitative evaluation metrics and neglect the visual quality of super-resolved stereo images. To improve the perceptual performance, this paper proposes the first perception-oriented stereo image super-resolution approach by exploiting the feedback, provided by the evaluation on the perceptual quality of StereoSR results. To provide accurate guidance for the StereoSR model, we develop the first special ster…
Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China（信息与通信工程学院和媒体融合与通信国家重点实验室，中国通信大学）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（信息科学与电子工程学院，上海交通大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；State Key Laboratory of Multimedia information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）
Perception-oriented Bidirectional Attention Network for Image Super-resolution Quality Assessment
School of Mathematical Sciences, Beihang University（北京航空航天大学数学学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）；CAS Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, University of Science and Technology of China（中国科学技术大学地球空间信息处理与应用系统国家重点实验室）；School of Information Science and Engineering, Ningbo University（宁波大学信息科学与工程学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
There exists a widely accepted set of assertions in the digital image and video coding literature, which are as follows: the Human Visual System (HVS) is more sensitive to luminance (often confused with brightness) than photon energies (often confused with chromaticity and chrominance). Passages similar to the following occur with high frequency in the peer reviewed literature and academic text books: "the HVS is much more sensitive to brightness than colour" and/or "the HVS is much more sensitive to luma than chroma". In this discussion paper, a Visible Light-Based Human Visual System (VL-HVS…
Seoul National University（首尔国立大学）；Samsung Electronics（三星电子）
Single-image super-resolution (SISR) networks trained with perceptual and adversarial losses provide high-contrast outputs compared to those of networks trained with distortion-oriented losses, such as L1 or L2. However, it has been shown that using a single perceptual loss is insufficient for accurately restoring locally varying diverse shapes in images, often generating undesirable artifacts or unnatural details. For this reason, combinations of various losses, such as perceptual, adversarial, and distortion losses, have been attempted, yet it remains challenging to find optimal combinations…
Full-reference image quality assessment (FR-IQA) techniques compare a reference and a distorted/test image and predict the perceptual quality of the test image in terms of a scalar value representing an objective score. The evaluation of FR-IQA techniques is carried out by comparing the objective scores from the techniques with the subjective scores (obtained from human observers) provided in the image databases used for the IQA. Hence, we reasonably assume that the goal of a human observer is to rate the distortion present in the test image. The goal oriented tasks are processed by the human …
Blind Stereoscopic Omnidirectional Image Quality Assessment Using Predictive Coding Hierarchy
School of Computational and Mathematical Sciences, Cardiff University（卡迪夫大学计算与数学科学学院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
University of Science and Technology of China（中国科学技术大学）
Image quality assessment (IQA) aims to estimate human perception based image visual quality. Although existing deep neural networks (DNNs) have shown significant effectiveness for tackling the IQA problem, it still needs to improve the DNN-based quality assessment models by exploiting efficient multi-scale features. In this paper, motivated by the human visual system (HVS) combining multi-scale features for perception, we propose to use pyramid features learning to build a DNN with hierarchical multi-scale features for distorted image quality prediction. Our model is based on both residual map…
Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models
School of Electronics and Communication Engineering, Sun Yat-sen University（电子与通信工程学院，中山大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）；University of Michigan（密歇根大学）
The use of 3D and stereo imaging is rapidly increasing. Compression, transmission, and processing could degrade the quality of stereo images. Quality assessment of such images is different than their 2D counterparts. Metrics that represent 3D perception by human visual system (HVS) are expected to assess stereoscopic quality more accurately. In this paper, inspired by brain sensory/motor fusion process, two stereo images are fused together. Then from every fused image two synthesized images are extracted. Effects of different distortions on statistical distributions of the synthesized images a…
Xidian University（西安电子科技大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Recently, it has been demonstrated that deep neural networks can significantly improve the performance of single image super-resolution (SISR). Numerous studies have concentrated on raising the quantitative quality of super-resolved (SR) images. However, these methods that target PSNR maximization usually produce blurred images at large upscaling factor. The introduction of generative adversarial networks (GANs) can mitigate this issue and show impressive results with synthetic high-frequency textures. Nevertheless, these GAN-based approaches always have a tendency to add fake textures and eve…
NORCE Norwegian Research Centre（NORCE挪威研究中心）；Hong Kong University of Science and Technology（香港科技大学）
Visual (image, video) quality assessments can be modelled by visual features in different domains, e.g., spatial, frequency, and temporal domains. Perceptual mechanisms in the human visual system (HVS) play a crucial role in generation of quality perception. This paper proposes a general framework for no-reference visual quality assessment using efficient windowed transformer architectures. A lightweight module for multi-stage channel attention is integrated into Swin (shifted window) Transformer. Such module can represent appropriate perceptual mechanisms in image quality assessment (IQA) to …
Guangzhou University（广州大学）；Xidian University（西安电子科技大学）；City University of Hong Kong（香港城市大学）
Video quality is a primary concern for video service providers. In recent years, the techniques of video quality assessment (VQA) based on deep convolutional neural networks (CNNs) have been developed rapidly. Although existing works attempt to introduce the knowledge of the human visual system (HVS) into VQA, there still exhibit limitations that prevent the full exploitation of HVS, including an incomplete model by few characteristics and insufficient connections among these characteristics. To overcome these limitations, this paper revisits HVS with five representative characteristics, and f…
Qingdao University（青岛大学）；New York University（纽约大学）；Shandong University（山东大学）；De Montfort University（德蒙福特大学）
With the wide applications of colored point cloud in many fields, point cloud perceptual quality assessment plays a vital role in the visual communication systems owing to the existence of quality degradations introduced in various stages. However, the existing point cloud quality assessments ignore the mechanism of human visual system (HVS) which has an important impact on the accuracy of the perceptual quality assessment. In this paper, a progressive knowledge transfer based on human visual perception mechanism for perceptual quality assessment of point clouds (PKT-PCQA) is proposed. The PKT…
Fuzhou University（福州大学）；Peng Cheng Laboratory（鹏城实验室）；Beijing University of Technology（北京工业大学）；Beijing Artificial Intelligence Institute（北京人工智能研究院）；Université de Nantes（南特大学）；Laboratoire des Sciences du Numérique de Nantes（南特数字科学实验室）
The widespread image applications have greatly promoted the vision-based tasks, in which the Image Quality Assessment (IQA) technique has become an increasingly significant issue. For user enjoyment in multimedia systems, the IQA exploits image fidelity and aesthetics to characterize user experience; while for other tasks such as popular object recognition, there exists a low correlation between utilities and perceptions. In such cases, the fidelity-based and aesthetics-based IQA methods cannot be directly applied. To address this issue, this paper proposes a utility-oriented IQA in object rec…
Department of Computer Science, University of Warwick（华威大学计算机科学系）
There exists an intrinsic relationship between the spectral sensitivity of the Human Visual System (HVS) and colour perception; these intertwined phenomena are often overlooked in perceptual compression research. In general, most previously proposed visually lossless compression techniques exploit luminance (luma) masking including luma spatiotemporal masking, luma contrast masking and luma texture/edge masking. The perceptual relevance of color in a picture is often overlooked, which constitutes a gap in the literature. With regard to the spectral sensitivity phenomenon of the HVS, the color …
CORP: A Multi-Modal Dataset for Campus-Oriented Roadside Perception Tasks
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）；University of Science and Technology of China（中国科学技术大学）
Perceptual Depth Quality Assessment of Stereoscopic Omnidirectional Images
School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；Department of Electrical and Computer Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）
Perception-Oriented Video Frame Interpolation via Asymmetric Blending
Shanghai Jiao Tong University（上海交通大学）；Kuaishou Technology（快手科技）；SeeKoo（视库）；University of Electronic Science and Technology of China（电子科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
MMUSV-Sim: A Perception-Oriented Simulation and Data-Generation Platform for Multi-USV Cooperative Perception
School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能工程学院）；School of Intelligent Imagery Engineering, Beijing Film Academy（北京电影学院智能影像工程学院）
Self-supervised Multi-task Learning Framework for Safety and Health-Oriented Connected Driving Environment Perception using Onboard Camera
The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）
It is believed that human vision system (HVS) consists of pre-attentive process and attention process when performing salient object detection (SOD). Based on this fact, we propose a four-stage framework for SOD, in which the first two stages match the \textbf{P}re-\textbf{A}ttentive process consisting of general feature extraction (GFE) and feature preprocessing (FP), and the last two stages are corresponding to \textbf{A}ttention process containing saliency feature extraction (SFE) and the feature aggregation (FA), namely \textbf{PAANet}. According to the pre-attentive process, the GFE stage…
University of Michigan（密歇根大学）；Robotics Institute（机器人研究所）
In order to enable robust operation in unstructured environments, robots should be able to generalize manipulation actions to novel object instances. For example, to pour and serve a drink, a robot should be able to recognize novel containers which afford the task. Most importantly, robots should be able to manipulate these novel containers to fulfill the task. To achieve this, we aim to provide robust and generalized perception of object affordances and their associated manipulation poses for reliable manipulation. In this work, we combine the notions of affordance and category-level pose, an…
Safety-Oriented Trajectory Ranking from BEV Perception Outputs
University of the Bundeswehr Munich（德国联邦国防军慕尼黑大学）；Ludwig-Maximilians-Universität München（慕尼黑大学）；Research Institute CODE, University of the Bundeswehr Munich（德国联邦国防军慕尼黑大学CODE研究院）
Wuhan University of Technology（武汉理工大学）；Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks（湖北省宽带无线通信与传感器网络重点实验室）；School of Civil Engineering & Architecture（土木工程与建筑学院）；School of Information Engineering（信息工程学院）；State Key Laboratory of Magnetic Resonance and Atomic Molecular Phys（磁共振与原子分子物理国家重点实验室）
The presence of residual and dense neural networks which greatly promotes the development of image Super-Resolution(SR) have witnessed a lot of impressive results. Depending on our observation, although more layers and connections could always improve performance, the increase of model parameters is not conducive to launch application of SR algorithms. Furthermore, algorithms supervised by L1/L2 loss can achieve considerable performance on traditional metrics such as PSNR and SSIM, yet resulting in blurry and over-smoothed outputs without sufficient high-frequency details, namely low perceptua…
RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Computer Science and Technology, Harbin University of Science and Technology（哈尔滨理工大学计算机科学与技术学院）
Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；JITRI（江苏省产业技术研究院）；Nantong University（南通大学）
The vigorous developments of Internet of Things make it possible to extend its computing and storage capabilities to computing tasks in the aerial system with collaboration of cloud and edge, especially for artificial intelligence (AI) tasks based on deep learning (DL). Collecting a large amount of image/video data, Unmanned aerial vehicles (UAVs) can only handover intelligent analysis tasks to the back-end mobile edge computing (MEC) server due to their limited storage and computing capabilities. How to efficiently transmit the most correlated information for the AI model is a challenging top…
INRIA（法国国家信息与自动化研究所）；ENS（巴黎高等师范学院）；PSL（巴黎文理研究大学）；Human-Machine Art（人机艺术）；Zuse Institute, Berlin（柏林祖斯研究所）
We propose visual creations that put differences in algorithms and humans \emph{perceptions} into perspective. We exploit saliency maps of neural networks and visual focus of humans to create diptychs that are reinterpretations of an original image according to both machine and human attentions. Using those diptychs as a qualitative evaluation of perception, we discuss some crucial issues of current \textit{task-oriented} artificial intelligence.
Beijing Institute of Technology（北京理工大学）；University of Delaware（特拉华大学）；The Aviation Industry Corporation of China, Ltd.（中国航空工业集团有限公司）；Luoyang Institute of Electro-Optical Equipment（洛阳电光设备研究所）
A few lightweight convolutional neural network (CNN) models have been recently designed for remote sensing object detection (RSOD). However, most of them simply replace vanilla convolutions with stacked separable convolutions, which may not be efficient due to a lot of precision losses and may not be able to detect oriented bounding boxes (OBB). Also, the existing OBB detection methods are difficult to constrain the shape of objects predicted by CNNs accurately. In this paper, we propose an effective lightweight oriented object detector (LO-Det). Specifically, a channel separation-aggregation …
Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences (UCAS)（中国科学院大学先进交叉科学学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Ant Group（蚂蚁集团）；Sangfor Technologies Inc.（深信服科技股份有限公司）
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；East China Normal University（华东师范大学）
Pursuing realistic results according to human visual perception is the central concern in the image transformation tasks. Perceptual learning approaches like perceptual loss are empirically powerful for such tasks but they usually rely on the pre-trained classification network to provide features, which are not necessarily optimal in terms of visual perception of image transformation. In this paper, we argue that, among the features representation from the pre-trained classification network, only limited dimensions are related to human visual perception, while others are irrelevant, although b…
SenseTime Research（商汤科技研究院）；Wuhan University（武汉大学）；Shanghai AI Laboratory（上海人工智能实验室）
Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction, and planning. In order to perform a wide diversity of tasks and achieve advanced-level intelligence, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from accumulative errors or deficient task coordination. Instead, we argue that a favorable framework should be devised and optimized in pursuit of the ultimate goal, i.e., planning of the self-driving car. Oriented at this…
