In networked video applications, the frame rate (FR) and quantization stepsize (QS) of a compressed video are often adapted in response to the changes of the available bandwidth. It is important to understand how do the variation of FR and QS and their variation pattern affect the video quality. In this paper, we investigate the impact of temporal variation of FR and QS on the perceptual video quality. Among all possible variation patterns, we focus on videos in which two FR's (or QS's) alternate over a fixed interval. We explore the human responses to such variation by conducting subjective evaluation of test videos with different variation magnitudes and frequencies. We further analyze statistical significance of the impact of variation magnitude, variation frequency, video content, and their interactions. By analyzing the subjective ratings, we propose two models for predicting the quality of video with alternating FR and QS, respectively, The proposed models have simple mathematical forms with a few content-dependent parameters. The models fit the measured data very well using parameters determined by least square fitting with the measured data. We further propose some guidelines for adaptation of FR and QS based on trends observed from subjective test results.
In this paper, we investigate the impact of spatial, temporal and amplitude resolution (STAR) on the perceptual quality of a compressed video. Subjective quality tests were carried out on a mobile device. Seven source sequences are included in the tests and for each source sequence we have 27 test configurations generated by JSVM encoder (3 QP levels, 3 spatial resolutions, and 3 temporal resolutions), resulting a total of 189 processed video sequences (PVSs). Videos coded at different spatial resolutions are displayed at the full screen size of the mobile platform. Subjective data reveal that…
Due to the spectral sensitivity phenomenon of the Human Visual System (HVS), the color channels of raw RGB 4:4:4 sequences contain significant psychovisual redundancies; these redundancies can be perceptually quantized. The default quantization systems in the HEVC standard are known as Uniform Reconstruction Quantization (URQ) and Rate Distortion Optimized Quantization (RDOQ); URQ and RDOQ are not perceptually optimized for the coding of RGB 4:4:4 video data. In this paper, we propose a novel spatiotemporal perceptual quantization technique named SPAQ. With application for RGB 4:4:4 video data…
FORCE Technology（FORCE科技）；Meta Reality Labs（Meta现实实验室）
To open up new possibilities to assess the multimodal perceptual quality of omnidirectional media formats, we proposed a novel open source 360 audiovisual (AV) quality dataset. The dataset consists of high-quality 360 video clips in equirectangular (ERP) format and higher-order ambisonic (4th order) along with the subjective scores. Three subjective quality experiments were conducted for audio, video, and AV with the procedures detailed in this paper. Using the data from subjective tests, we demonstrated that this dataset can be used to quantify perceived audio, video, and audiovisual quality.…
This paper applies energy conservation principles to the Daala video codec using gain-shape vector quantization to encode a vector of AC coefficients as a length (gain) and direction (shape). The technique originates from the CELT mode of the Opus audio codec, where it is used to conserve the spectral envelope of an audio signal. Conserving energy in video has the potential to preserve textures rather than low-passing them. Explicitly quantizing a gain allows a simple contrast masking model with no signaling cost. Vector quantizing the shape keeps the number of degrees of freedom the same as s…
There is an increasing consumer demand for high bit-depth 4:4:4 HD video data playback due to its superior perceptual visual quality compared with standard 8-bit subsampled 4:2:0 video data. Due to vast file sizes and associated bitrates, it is desirable to compress raw high bit-depth 4:4:4 HD video sequences as much as possible without incurring a discernible decrease in visual quality. In this paper, we propose a Coding Block (CB)-level perceptual video coding technique for HEVC named Full Color Perceptual Quantization (FCPQ). FCPQ is designed to adjust the Quantization Parameter (QP) at the…
In this paper, the perceptual quality difference between scalable and single-layer videos coded at the same spatial, temporal and amplitude resolution (STAR) is investigated through a subjective test using a mobile platform. Three source videos are considered and for each source video single-layer and scalable video are compared at 9 different STARs. We utilize paired comparison methods with and without tie option. Results collected from 10 subjects in the without "tie" option and 6 subjects in the with "tie" option show that there is no significant quality difference between scalable and sing…
Conventional images/videos are often rendered within the central vision area of the human visual system (HVS) with uniform quality. Recent virtual reality (VR) device with head mounted display (HMD) extends the field of view (FoV) significantly to include both central and peripheral vision areas. It exhibits the unequal image quality sensation among these areas because of the non-uniform distribution of photoreceptors on our retina. We propose to study the sensation impact on the image subjective quality with respect to the eccentric angle $θ$ across different vision areas. Often times, image …
The JCT-VC standardized Screen Content Coding (SCC) extension in the HEVC HM RExt + SCM reference codec offers an impressive coding efficiency performance when compared with HM RExt alone; however, it is not significantly perceptually optimized. For instance, it does not include advanced HVS-based perceptual coding methods, such as JND-based spatiotemporal masking schemes. In this paper, we propose a novel JND-based perceptual video coding technique for HM RExt + SCM. The proposed method is designed to further improve the compression performance of HM RExt + SCM when applied to YCbCr 4:4:4 SC …
The default quantisation algorithms in the state-of-the-art High Efficiency Video Coding (HEVC) standard, namely Uniform Reconstruction Quantisation (URQ) and Rate-Distortion Optimised Quantisation (RDOQ), do not take into account the perceptual relevance of individual transform coefficients. In this paper, a Frequency-Dependent Perceptual Quantisation (FDPQ) technique for HEVC is proposed. FDPQ exploits the well-established Modulation Transfer Function (MTF) characteristics of the linear transformation basis functions by taking into account the Euclidean distance of an AC transform coefficien…
Immersive video offers the freedom to navigate inside virtualized environment. Instead of streaming the bulky immersive videos entirely, a viewport (also referred to as field of view, FoV) adaptive streaming is preferred. We often stream the high-quality content within current viewport, while reducing the quality of representation elsewhere to save the network bandwidth consumption. Consider that we could refine the quality when focusing on a new FoV, in this paper, we model the perceptual impact of the quality variations (through adapting the quantization stepsize and spatial resolution) with…
Recently, we have observed an exponential increase of user-generated content (UGC) videos. The distinguished characteristic of UGC videos originates from the video production and delivery chain, as they are usually acquired and processed by non-professional users before uploading to the hosting platforms for sharing. As such, these videos usually undergo multiple distortion stages that may affect visual quality before ultimately being viewed. Inspired by the increasing consensus that the optimization of the video coding and processing shall be fully driven by the perceptual quality, in this pa…
High dynamic range (HDR) imaging has been attracting much attention as a technology that can provide immersive experience. Its ultimate goal is to provide better quality of experience (QoE) via enhanced contrast. In this paper, we analyze perceptual experience of tone-mapped HDR videos both explicitly by conducting a subjective questionnaire assessment and implicitly by using EEG and peripheral physiological signals. From the results of the subjective assessment, it is revealed that tone-mapped HDR videos are more interesting and more natural, and give better quality than low dynamic range (LD…
In video coding, it is expected that the encoder could adaptively select the encoding parameters (e.g., quantization parameter) to optimize the bit allocation to different sources under the given constraint. However, in hybrid video coding, the dependency between sources brings high complexity for the bit allocation optimization, especially in the block-level, and existing optimization methods mostly focus on frame-level bit allocation. In this paper, we propose a macroblock (MB) level bit allocation method based on the minimum maximum (MINMAX) criterion, which has acceptable encoding complexi…
This paper deals with the issue of the perceptual quality evaluation of user-generated videos shared online, which is an important step toward designing video-sharing services that maximize users' satisfaction in terms of quality. We first analyze viewers' quality perception patterns by applying graph analysis techniques to subjective rating data. We then examine the performance of existing state-of-the-art objective metrics for the quality estimation of user-generated videos. In addition, we investigate the feasibility of metadata accompanied with videos in online video-sharing services for q…
HTTP-based video streaming technologies allow for flexible rate selection strategies that account for time-varying network conditions. Such rate changes may adversely affect the user's Quality of Experience; hence online prediction of the time varying subjective quality can lead to perceptually optimised bitrate allocation policies. Recent studies have proposed to use dynamic network approaches for continuous-time prediction; yet they do not consider multiple video quality models as inputs nor consider forecasting ensembles. Here we address the problem of predicting continuous-time subjective …
Recent years have witnessed an ever-expandingvolume of user-generated content (UGC) videos available on the Internet. Nevertheless, progress on perceptual quality assessmentof UGC videos still remains quite limited. There are many distinguished characteristics of UGC videos in the complete video production and delivery chain, and one important property closely relevant to video quality is that there does not exist the pristine source after they are uploaded to the hosting platform,such that they often undergo multiple compression stages before ultimately viewed. To facilitate the UGC video qua…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI（Facebook人工智能）
Blind or no-reference (NR) perceptual picture quality prediction is a difficult, unsolved problem of great consequence to the social and streaming media industries that impacts billions of viewers daily. Unfortunately, popular NR prediction models perform poorly on real-world distorted pictures. To advance progress on this problem, we introduce the largest (by far) subjective picture quality database, containing about 40000 real-world distorted pictures and 120000 patches, on which we collected about 4M human judgments of picture quality. Using these picture and patch quality labels, we built …
To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG) and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this paper, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem fo…
Staircase-like contours introduced to a video by quantization in flat areas, commonly known as banding, have been a long-standing problem in both video processing and quality assessment communities. The fact that even a relatively small change of the original pixel values can result in a strong impact on perceived quality makes banding especially difficult to be detected by objective quality metrics. In this paper, we study how banding annoyance compares to more commonly studied scaling and compression artifacts with respect to the overall perceptual quality. We further propose a simple combin…
Gustave Eiffel University（古斯塔夫·埃菲尔大学）；Cardiff University（卡迪夫大学）；Xi'an Jiaotong University（西安交通大学）；Zhongkai University of Agriculture and Engineering（仲恺农业工程学院）；University of Bath（巴斯大学）；Xidian University（西安电子科技大学）
Research on image quality assessment (IQA) remains limited mainly due to our incomplete knowledge about human visual perception. Existing IQA algorithms have been designed or trained with insufficient subjective data with a small degree of stimulus variability. This has led to challenges for those algorithms to handle complexity and diversity of real-world digital content. Perceptual evidence from human subjects serves as a grounding for the development of advanced IQA algorithms. It is thus critical to acquire reliable subjective data with controlled perception experiments that faithfully ref…
Due to the strong correlation between visual attention and perceptual quality, many methods attempt to use human saliency information for image quality assessment. Although this mechanism can get good performance, the networks require human saliency labels, which is not easily accessible for omnidirectional images (ODI). To alleviate this issue, we propose a spatial attention-based perceptual quality prediction network for non-reference quality assessment on ODIs (SAP-net). To drive our SAP-net, we establish a large-scale IQA dataset of ODIs (IQA-ODI), which is composed of subjective scores of…
Xidian University（西安电子科技大学）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Northwestern Polytechnical University（西北工业大学）
In this paper, we propose a deformable convolution-based generative adversarial network (DCNGAN) for perceptual quality enhancement of compressed videos. DCNGAN is also adaptive to the quantization parameters (QPs). Compared with optical flows, deformable convolutions are more effective and efficient to align frames. Deformable convolutions can operate on multiple frames, thus leveraging more temporal information, which is beneficial for enhancing the perceptual quality of compressed videos. Instead of aligning frames in a pairwise manner, the deformable convolution can process multiple frames…
Perceptual Quality Optimization of Image Super-Resolution
School of Computer Science and Informatics, Cardiff University, UK（计算机科学与信息学学院，卡迪夫大学）；Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria（Christian Doppler实验室ATHENA，阿尔卑斯-亚得里亚大学）；XiaomiEV, China（小米EV，中国）；Centre for Vision, Speech and Signal Processing, University of Surrey, UK（视觉、语音和信号处理中心，萨里大学）
FORCE Technology（FORCE Technology 公司）；Technical University of Denmark（丹麦技术大学）
In an earlier study, we gathered perceptual evaluations of the audio, video, and audiovisual quality for 360 audiovisual content. This paper investigates perceived audiovisual quality prediction based on objective quality metrics and subjective scores of 360 video and spatial audio content. Thirteen objective video quality metrics and three objective audio quality metrics were evaluated for five stimuli for each coding parameter. Four regression-based machine learning models were trained and tested here, i.e., multiple linear regression, decision tree, random forest, and support vector machine…
Computer-generated holographic (CGH) displays show great potential and are emerging as the next-generation displays for augmented and virtual reality, and automotive heads-up displays. One of the critical problems harming the wide adoption of such displays is the presence of speckle noise inherent to holography, that compromises its quality by introducing perceptible artifacts. Although speckle noise suppression has been an active research area, the previous works have not considered the perceptual characteristics of the Human Visual System (HVS), which receives the final displayed imagery. Ho…
With the rapid growth of in-the-wild videos taken by non-specialists, blind video quality assessment (VQA) has become a challenging and demanding problem. Although lots of efforts have been made to solve this problem, it remains unclear how the human visual system (HVS) relates to the temporal quality of videos. Meanwhile, recent work has found that the frames of natural video transformed into the perceptual domain of the HVS tend to form a straight trajectory of the representations. With the obtained insight that distortion impairs the perceived video quality and results in a curved trajector…
