Dynamic adaptive streaming over HTTP provides the work of most multimedia services, however, the nature of this technology further complicates the assessment of the QoE (Quality of Experience). In this paper, the influence of various objective factors on the subjective estimation of the QoE of streaming video is studied. The paper presents standard and handcrafted features, shows their correlation and p-Value of significance. VQA (Video Quality Assessment) models based on regression and gradient boosting with SRCC reaching up to 0.9647 on the validation subsample are proposed. The proposed regression models are adapted for applied applications (both with and without a reference video); the Gradient Boosting Regressor model is perspective for further improvement of the quality estimation model. We take SQoE-III database, so far the largest and most realistic of its kind. The VQA (video quality assessment) models are available at https://github.com/AleksandrIvchenko/QoE-assesment
University of Waterloo（滑铁卢大学）；Shanghai Jiao Tong University（上海交通大学）；Ningbo University（宁波大学）
Quality of experience (QoE) assessment for adaptive video streaming plays a significant role in advanced network management systems. It is especially challenging in case of dynamic adaptive streaming schemes over HTTP (DASH) which has increasingly complex characteristics including additional playback issues. In this paper, we provide a brief overview of adaptive video streaming quality assessment. Upon our review of related works, we analyze and compare different variations of objective QoE assessment models with or without using machine learning techniques for adaptive video streaming. Throug…
Live multimedia streaming from mobile devices is rapidly gaining popularity but little is known about the QoE they provide. In this paper, we examine the Periscope service. We first crawl the service in order to understand its usage patterns. Then, we study the protocols used, the typical quality of experience indicators, such as playback smoothness and latency, video quality, and the energy consumption of the Android application.
Recently, multi-screen cloud social TV is invented to transform TV into social experience. People watching the same content on social TV may come from different locations, while freely interact with each other through text, image, audio and video. This crucial virtual living-room experience adds social aspects into existing performance metrics. In this paper, we parse social TV user experience into three elements (i.e., inter-user delay, video quality of experience (QoE), and resource efficiency), and provide a joint analytical framework to enhance user experience. Specifically, we propose a c…
The fundamental conflict between the enormous space of adaptive streaming videos and the limited capacity for subjective experiment casts significant challenges to objective Quality-of-Experience (QoE) prediction. Existing objective QoE models exhibit complex functional form, failing to generalize well in diverse streaming environments. In this study, we propose an objective QoE model namely knowledge-driven streaming quality index (KSQI) to integrate prior knowledge on the human visual system and human annotated data in a principled way. By analyzing the subjective characteristics towards str…
Dynamic Adaptive Streaming over HTTP (DASH) is a video streaming technique largely used. One key point is the adaptation mechanism which resides at the client's side. This mechanism impacts greatly on the overall Quality of Experience (QoE) of the video streaming. In this paper, we propose a new adaptation algorithm for DASH, namely SSIM Based Adaptation (SBA). This mechanism is user-experience driven: it uses the Structural Similarity Index Measurement (SSIM) as main video perceptual quality indicator; moreover, the adaptation is based on a joint consideration of SSIM indicator and the physic…
Video streaming via TCP networks has become a popular and highly demanded service, but its quality assessment in both objective and subjective terms has not been properly addressed. In this paper, based on statistical analysis a full analytic model of a no-reference objective metric, namely Pause Intensity, for video quality assessment is presented. The model characterizes the video playout buffer behavior in connection with the network performance (throughput) and the video playout rate. This allows for instant quality measurement and control without requiring a reference video. Pause intensi…
The commercialization of Virtual Reality (VR) headsets has made immersive and 360-degree video streaming the subject of intense interest in the industry and research communities. While the basic principles of video streaming are the same, immersive video presents a set of specific challenges that need to be addressed. In this survey, we present the latest developments in the relevant literature on four of the most important ones: (i) omnidirectional video coding and compression, (ii) subjective and objective Quality of Experience (QoE) and the factors that can affect it, (iii) saliency measure…
Over the last few years, the evolution of network and user handsets' technologies, have challenged the telecom industry and the Internet ecosystem. Especially, the unprecedented progress of multimedia streaming services like YouTube, Vimeo and DailyMotion resulted in an impressive demand growth and a significant need of Quality of Service (QoS) (e.g., high data rate, low latency/jitter, etc.). Mainly, numerous difficulties are to be considered while delivering a specific service, such as a strict QoS, human-centric features, massive number of devices, heterogeneous devices and networks, and un…
Scalable video coding has drawn great interest in content delivery in many multimedia services thanks to its capability to handle terminal heterogeneity and network conditions variation. In our previous work, and under the umbrella of ENVISION, we have proposed a playout smoothing mechanism to ensure the uniform delivery of the layered stream, by reducing the quality changes that the stream undergoes when adapting to changing network conditions. In this paper we study the resulting video quality, from the final user perception under different network conditions of loss and delays. For that we …
In today's Internet, HTTP Adaptive Streaming (HAS) is the mainstream standard for video streaming, which switches the bitrate of the video content based on an Adaptive BitRate (ABR) algorithm. An effective Quality of Experience (QoE) assessment metric can provide crucial feedback to an ABR algorithm. However, predicting such real-time QoE on the client side is challenging. The QoE prediction requires high consistency with the Human Visual System (HVS), low latency, and blind assessment, which are difficult to realize together. To address this challenge, we analyzed various characteristics of H…
HTTP Adaptive Streaming (HAS) is nowadays a popular solution for multimedia delivery. The novelty of HAS lies in the possibility of continuously adapting the streaming session to current network conditions, facilitated by Adaptive Bitrate (ABR) algorithms. Various popular streaming and Video on Demand services such as Netflix, Amazon Prime Video, and Twitch use this method. Given this broad consumer base, ABR algorithms continuously improve to increase user satisfaction. The insights for these improvements are, among others, gathered within the research area of Quality of Experience (QoE). Wit…
HTTP Adaptive Streaming (HAS) has become a cost-effective means for multimedia delivery nowadays. However, how the quality of experience (QoE) is jointly affected by 1) varying perceptual quality and 2) interruptions is not well-understood. In this paper, we present the first attempt to quantitatively quantify the relative impacts of these factors on the QoE of streaming sessions. To achieve this purpose, we first model the impacts of the factors using histograms, which represent the frequency distributions of the individual factors in a session. By using a large dataset, various insights into…
The diversity of video delivery pipeline poses a grand challenge to the evaluation of adaptive bitrate (ABR) streaming algorithms and objective quality-of-experience (QoE) models. Here we introduce so-far the largest subject-rated database of its kind, namely WaterlooSQoE-IV, consisting of 1350 adaptive streaming videos created from diverse source contents, video encoders, network traces, ABR algorithms, and viewing devices. We collect human opinions for each video with a series of carefully designed subjective experiments. Subsequent data analysis and testing/comparison of ABR algorithms and …
Recent improvements in Extended Reality (XR) technology created an increase in XR products and solutions in the industry, while raising new requirements for new or improved architectural concepts. This need can be particularly complex as XR applications often relate both to 3D geometric rendering and multimedia paradigms. This paper outlines the main concepts relevant to XR, both from a game engineering and multimedia streaming system perspective. XR requires new metadata and media/game orchestration to allow complex interaction between users, objects, and (volumetric) multimedia content, whic…
Streaming services gradually support high-quality videos for better user experience. However, streaming high-quality video on mobile devices consumes a considerable amount of energy. This paper presents the design and prototype of EVSO, which achieves power saving by applying adaptive frame rates to parts of videos with a little degradation of the user experience. EVSO utilizes a novel perceptual similarity measurement method based on human visual perception specialized for a video encoder. We also extend the media presentation description, in which the video content is selected based only on …
Consumer Virtual Reality (VR) has been widely used in various application areas, such as entertainment and medicine. In spite of the superb immersion experience, to enable high-quality VR on untethered mobile devices remains an extremely challenging task. The high bandwidth demands of VR streaming generally overburden a conventional wireless connection, which affects the user experience and in turn limits the usability of VR in practice. In this paper, we propose FoVR, attention-based hierarchical VR streaming through bandwidth-limited wireless networks. The design of FoVR stems from the insig…
The growing popularity of virtual and augmented reality communications and 360° video streaming is moving video communication systems into much more dynamic and resource-limited operating settings. The enormous data volume of 360° videos requires an efficient use of network bandwidth to maintain the desired quality of experience for the end user. To this end, we propose a framework for viewport-driven rate-distortion optimized 360° video streaming that integrates the user view navigation pattern and the spatiotemporal rate-distortion characteristics of the 360° video content to maximize the de…
In this paper, we propose a novel algorithm for video rate adaptation in HTTP Adaptive Streaming (HAS), based on online learning. The proposed algorithm, named Learn2Adapt (L2A), is shown to provide a robust rate adaptation strategy which, unlike most of the state-of-the-art techniques, does not require parameter tuning, channel model assumptions or application-specific adjustments. These properties make it very suitable for mobile users, who typically experience fast variations in channel characteristics. Simulations show that L2A improves on the overall Quality of Experience (QoE) and in par…
360-degree video provides an immersive 360-degree viewing experience and has been widely used in many areas. The 360-degree video live streaming systems involve capturing, compression, uplink (camera to video server) and downlink (video server to user) transmissions. However, few studies have jointly investigated such complex systems, especially the rate adaptation for the coupled uplink and downlink in the 360-degree video streaming under limited bandwidth constraints. In this letter, we propose a quality of experience (QoE)-driven 360-degree video live streaming system, in which a video serv…
The increasing demand for video streaming services with high Quality of Experience (QoE) has prompted a lot of research on client-side adaptation logic approaches. However, most algorithms use the client's previous download experience and do not use a crowd knowledge database generated by users of a professional service. We propose a new crowd algorithm that maximizes the QoE. Additionally, we show how crowd information can be integrated into existing algorithms and illustrate this with two state-of-the-art algorithms. We evaluate our algorithm and state-of-the-art algorithms (including our mo…
Adaptive bitrate streaming (ABR) has become thede factotechnique for videostreaming over the Internet. Despite a flurry of techniques, achieving high quality ABRstreaming over cellular networks remains a tremendous challenge. First, the design ofan ABR scheme needs to balance conflicting Quality of Experience (QoE) metrics suchas video quality, quality changes, stalls and startup performance, which is even harderunder highly dynamic bandwidth in cellular network. Second, streaming providers havebeen moving towards using Variable Bitrate (VBR) encodings for the video content,which introduces ne…
The recent rise of interest in Virtual Reality (VR) came with the availability of commodity commercial VR prod- ucts, such as the Head Mounted Displays (HMD) created by Oculus and other vendors. To accelerate the user adoption of VR headsets, content providers should focus on producing high quality immersive content for these devices. Similarly, multimedia streaming service providers should enable the means to stream 360 VR content on their platforms. In this study, we try to cover different aspects related to VR content representation, streaming, and quality assessment that will help establis…
Point cloud video is the most popular representation of hologram, which is the medium to precedent natural content in VR/AR/MR and is expected to be the next generation video. Point cloud video system provides users immersive viewing experience with six degrees of freedom and has wide applications in many fields such as online education, entertainment. To further enhance these applications, point cloud video streaming is in critical demand. The inherent challenges lie in the large size by the necessity of recording the three-dimensional coordinates besides color information, and the associated…
The panoramic video is widely used to build virtual reality (VR) and is expected to be one of the next generation Killer-Apps. Transmitting panoramic VR videos is a challenging task because of two problems: 1) panoramic VR videos are typically much larger than normal videos but they need to be transmitted with limited bandwidth in mobile networks. 2) high-resolution and fluent views should be provided to guarantee a superior user experience and avoid side-effects such as dizziness and nausea. To address these two problems, we propose a novel interactive streaming technology, namely Focus-based…
Point cloud video has been widely used by augmented reality (AR) and virtual reality (VR) applications as it allows users to have an immersive experience of six degrees of freedom (6DoFs). Yet there is still a lack of research on quality of experience (QoE) model of point cloud video streaming, which cannot provide optimization metric for streaming systems. Besides, position and color information contained in each pixel of point cloud video, and viewport distance effect caused by 6DoFs viewing procedure make the traditional objective quality evaluation metric cannot be directly used in point c…
Among the various means to evaluate the quality of video streams, No-Reference (NR) methods have low computation and may be executed on thin clients. Thus, NR algorithms would be perfect candidates in cases of real-time quality assessment, automated quality control and, particularly, in adaptive mobile streaming. Yet, existing NR approaches are often inaccurate, in comparison to Full-Reference (FR) algorithms, especially under lossy network conditions. In this work, we present an NR method that combines machine learning with simple NR metrics to achieve a quality index comparably as accurate a…
We consider a multi-user video streaming service optimization problem over a time-varying and mutually interfering multi-cell wireless network. The key research challenge is to appropriately adapt each user's video streaming rate according to the radio frequency environment (e.g., channel fading and interference level) and service demands (e.g., play request), so that the users' long-term experience for watching videos can be optimized. To address the above challenge, we propose a novel two-level cross-layer optimization framework for multiuser adaptive video streaming over wireless networks. …
Point cloud video transmission is challenging due to high encoding/decoding complexity, high video bitrate, and low latency requirement. Consequently, conventional adaptive streaming methodologies often find themselves unsatisfactory to meet the requirements in threefold: 1) current algorithms reuse existing quality of experience (QoE) definitions while overlooking the unique features of point cloud video thus failing to provide optimal user experience, 2) most deep learning approaches require long-span data collections to learn sufficiently varied network conditions and result in long trainin…
Good user experience with interactive cloud-based multimedia applications, such as cloud gaming and cloud-based VR, requires low end-to-end latency and large amounts of downstream network bandwidth at the same time. In this paper, we present a foveated video streaming system for cloud gaming. The system adapts video stream quality by adjusting the encoding parameters on the fly to match the player's gaze position. We conduct measurements with a prototype that we developed for a cloud gaming system in conjunction with eye tracker hardware. Evaluation results suggest that such foveated streaming…
Recently, HTTP-based adaptive streaming has become the de facto standard for video streaming over the Internet. It allows clients to dynamically adapt media characteristics to network conditions in order to ensure a high quality of experience, that is, minimize playback interruptions, while maximizing video quality at a reasonable level of quality changes. In the case of live streaming, this task becomes particularly challenging due to the latency constraints. The challenge further increases if a client uses a wireless network, where the throughput is subject to considerable fluctuations. Cons…
As an important application form of immersive multimedia services, free-viewpoint video(FVV) enables users with great immersive experience by strong interaction. However, the computational complexity of virtual view synthesis algorithms poses a significant challenge to the real-time performance of an FVV system. Furthermore, the individuality of user interaction makes it difficult to serve multiple users simultaneously for a system with conventional architecture. In this paper, we novelly introduce a CNN-based view interpolation algorithm to synthesis dense virtual views in real time. Based on…
Multimedia streaming to mobile devices is challenging for two reasons. First, the way content is delivered to a client must ensure that the user does not experience a long initial playback delay or a distorted playback in the middle of a streaming session. Second, multimedia streaming applications are among the most energy hungry applications in smartphones. The energy consumption mostly depends on the delivery techniques and on the power management techniques of wireless access technologies (Wi-Fi, 3G, and 4G). In order to provide insights on what kind of streaming techniques exist, how they …
Inferring the quality of network services is the vital basis of optimization for network operators. However, prevailing real-time video streaming applications adopt encryption for security, leaving it a problem to extract Quality of Service (QoS) indicators of real-time video. In this paper, we propose DaI, a traffic-based real-time video quality estimator. DaI can partially decrypt the encrypted real-time video data and applies machine learning methods to estimate key objective Quality of Experience (QoE) metrics of real-time video. According to the experimental results, DaI can estimate obje…
The live streaming is more challenging than on-demand streaming, because the low latency is also a strong requirement in addition to the trade-off between video quality and jitters in playback. To balance several inherently conflicting performance metrics and improve the overall quality of experience (QoE), many adaptation schemes have been proposed. Bitrate adaptation is one of the major solutions for video streaming under time-varying network conditions, which works even better combining with some latency control methods, such as adaptive playback rate control and frame dropping. However, it…
The video quality assessment (VQA) technology has attracted a lot of attention in recent years due to an increasing demand of video streaming services. Existing VQA methods are designed to predict video quality in terms of the mean opinion score (MOS) calibrated by humans in subjective experiments. However, they cannot predict the satisfied user ratio (SUR) of an aggregated viewer group. Furthermore, they provide little guidance to video coding parameter selection, e.g. the Quantization Parameter (QP) of a set of consecutive frames, in practical video streaming services. To overcome these shor…
In mobile networks, users may lose coverage when entering a building due to the high signal attenuation at windows and walls. Under such conditions, services with minimum bit-rate requirements, such as video streaming, often show poor Quality-of-Experience (QoE). We will present a Bayesian detector that combines measurements from two Smartphone sensors to decide if a user is inside a building or not. Based on this coverage classification, we will propose an HTTP adaptive streaming (HAS) algorithm to increase playback stability at a high average bitrate. Measurements in a typical office buildin…
Adaptive 3D Gaussian Splatting Video Streaming: Visual Saliency-Aware Tiling and Meta-Learning-Based Bitrate Adaptation
School of Electrical Engineering and Automation, Hefei University of Technology（电子工程与自动化学院，合肥工业大学）；Engineering Technology Research Center of Industrial Automation of Anhui Province（安徽省工业自动化工程技术研究中心）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）；The University of Electro-Communications（电子通信大学）
360° video is attracting an increasing amount of attention in the context of Virtual Reality (VR). Owing to its very high-resolution requirements, existing professional streaming services for 360° video suffer from severe drawbacks. This paper introduces a novel end-to-end streaming system from encoding to displaying, to transmit 8K resolution 360° video and to provide an enhanced VR experience using Head Mounted Displays (HMDs). The main contributions of the proposed system are about tiling, integration of the MPEG-Dynamic Adaptive Streaming over HTTP (DASH) standard, and viewport-aware bitra…
In the tile-based 360-degree video streaming, predicting user's future viewpoints and developing adaptive bitrate (ABR) algorithms are essential for optimizing user's quality of experience (QoE). Traditional single-user based viewpoint prediction methods fail to achieve good performance in long-term prediction, and the recently proposed reinforcement learning (RL) based ABR schemes applied in traditional video streaming can not be directly applied in the tile-based 360-degree video streaming due to the exponential action space. Therefore, we propose a sequential reinforced 360-degree video str…
ESVQA: Perceptual Quality Assessment of Egocentric Spatial Videos
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；Institut Universitaire de France (IUF), University of Nantes（法国南特大学机构大学（IUF））
As the video streaming traffic in mobile networks is increasing, improving the content delivery process becomes crucial, e.g., by utilizing edge computing support. At an edge node, we can deploy adaptive bitrate (ABR) algorithms with a better understanding of network behavior and access to radio and player metrics. In this work, we present ECAS-ML, Edge Assisted Adaptation Scheme for HTTP Adaptive Streaming with Machine Learning. ECAS-ML focuses on managing the tradeoff among bitrate, segment switches, and stalls to achieve a higher quality of experience (QoE). For that purpose, we use machine…
Technische Universität Berlin（柏林工业大学）；DFKI Projektbüro Berlin（德国人工智能研究中心柏林项目办公室）
Video gaming streaming services are growing rapidly due to new services such as passive video streaming, e.g. Twitch.tv, and cloud gaming, e.g. Nvidia Geforce Now. In contrast to traditional video content, gaming content has special characteristics such as extremely high motion for some games, special motion patterns, synthetic content and repetitive content, which makes the state-of-the-art video and image quality metrics perform weaker for this special computer generated content. In this paper, we outline our plan to build a deep learningbased quality metric for video gaming quality assessme…
Nowadays video streaming is growing over the social clouds, where end-users always want to share High Definition (HD) videos among friends. Mostly videos were recorded via smartphones and other HD devices and short time videos have a big file size. The big file size of videos required high bandwidth to upload and download on the Internet and also required more time to load in a web page for play. So avoiding this problem social cloud compress videos during the upload for smooth play and fast loading in a web page. Compression decreases the video quality which also decreases the quality of expe…
