In video streaming services, predicting the continuous user's quality of experience (QoE) plays a crucial role in delivering high quality streaming contents to the user. However, the complexity caused by the temporal dependencies in QoE data and the non-linear relationships among QoE influence factors has introduced challenges to continuous QoE prediction. To deal with that, existing studies have utilized the Long Short-Term Memory model (LSTM) to effectively capture such complex dependencies, resulting in excellent QoE prediction accuracy. However, the high computational complexity of LSTM, caused by the sequential processing characteristic in its architecture, raises a serious question about its performance on devices with limited computational power. Meanwhile, Temporal Convolutional Network (TCN), a variation of convolutional neural networks, has recently been proposed for sequence modeling tasks (e.g., speech enhancement), providing a superior prediction performance over baseline methods including LSTM in terms of prediction accuracy and computational complexity. Being inspired of that, in this paper, an improved TCN-based model, namely CNN-QoE, is proposed for continuously predicting the QoE, which poses characteristics of sequential data. The proposed model leverages the advantages of TCN to overcome the computational complexity drawbacks of LSTM-based QoE models, while at the same time introducing the improvements to its architecture to improve QoE prediction accuracy. Based on a comprehensive evaluation, we demonstrate that the proposed CNN-QoE model can reach the state-of-the-art performance on both personal computers and mobile devices, outperforming the existing approaches.
Point cloud video has been widely used by augmented reality (AR) and virtual reality (VR) applications as it allows users to have an immersive experience of six degrees of freedom (6DoFs). Yet there is still a lack of research on quality of experience (QoE) model of point cloud video streaming, which cannot provide optimization metric for streaming systems. Besides, position and color information contained in each pixel of point cloud video, and viewport distance effect caused by 6DoFs viewing procedure make the traditional objective quality evaluation metric cannot be directly used in point c…
HTTP based adaptive video streaming has become a popular choice of streaming due to the reliable transmission and the flexibility offered to adapt to varying network conditions. However, due to rate adaptation in adaptive streaming, the quality of the videos at the client keeps varying with time depending on the end-to-end network conditions. Further, varying network conditions can lead to the video client running out of playback content resulting in rebuffering events. These factors affect the user satisfaction and cause degradation of the user quality of experience (QoE). It is important to …
The diversity of video delivery pipeline poses a grand challenge to the evaluation of adaptive bitrate (ABR) streaming algorithms and objective quality-of-experience (QoE) models. Here we introduce so-far the largest subject-rated database of its kind, namely WaterlooSQoE-IV, consisting of 1350 adaptive streaming videos created from diverse source contents, video encoders, network traces, ABR algorithms, and viewing devices. We collect human opinions for each video with a series of carefully designed subjective experiments. Subsequent data analysis and testing/comparison of ABR algorithms and …
In today's Internet, HTTP Adaptive Streaming (HAS) is the mainstream standard for video streaming, which switches the bitrate of the video content based on an Adaptive BitRate (ABR) algorithm. An effective Quality of Experience (QoE) assessment metric can provide crucial feedback to an ABR algorithm. However, predicting such real-time QoE on the client side is challenging. The QoE prediction requires high consistency with the Human Visual System (HVS), low latency, and blind assessment, which are difficult to realize together. To address this challenge, we analyzed various characteristics of H…
Mobile streaming video data accounts for a large and increasing percentage of wireless network traffic. The available bandwidths of modern wireless networks are often unstable, leading to difficulties in delivering smooth, high-quality video. Streaming service providers such as Netflix and YouTube attempt to adapt their systems to adjust in response to these bandwidth limitations by changing the video bitrate or, failing that, allowing playback interruptions (rebuffering). Being able to predict end user' quality of experience (QoE) resulting from these adjustments could lead to perceptually-dr…
360-degree video provides an immersive 360-degree viewing experience and has been widely used in many areas. The 360-degree video live streaming systems involve capturing, compression, uplink (camera to video server) and downlink (video server to user) transmissions. However, few studies have jointly investigated such complex systems, especially the rate adaptation for the coupled uplink and downlink in the 360-degree video streaming under limited bandwidth constraints. In this letter, we propose a quality of experience (QoE)-driven 360-degree video live streaming system, in which a video serv…
We provide in this paper a tutorial and a comprehensive survey of QoE management solutions in current and future networks. We start with a high level description of QoE management for multimedia services, which integrates QoE modelling, monitoring, and optimization. This followed by a discussion of HTTP Adaptive Streaming (HAS) solutions as the dominant technique for streaming videos over the best-effort Internet. We then summarize the key elements in SDN/NFV along with an overview of ongoing research projects, standardization activities and use cases related to SDN, NFV, and other emerging ap…
Point cloud video is the most popular representation of hologram, which is the medium to precedent natural content in VR/AR/MR and is expected to be the next generation video. Point cloud video system provides users immersive viewing experience with six degrees of freedom and has wide applications in many fields such as online education, entertainment. To further enhance these applications, point cloud video streaming is in critical demand. The inherent challenges lie in the large size by the necessity of recording the three-dimensional coordinates besides color information, and the associated…
The fundamental conflict between the enormous space of adaptive streaming videos and the limited capacity for subjective experiment casts significant challenges to objective Quality-of-Experience (QoE) prediction. Existing objective QoE models exhibit complex functional form, failing to generalize well in diverse streaming environments. In this study, we propose an objective QoE model namely knowledge-driven streaming quality index (KSQI) to integrate prior knowledge on the human visual system and human annotated data in a principled way. By analyzing the subjective characteristics towards str…
Proactive tile-based virtual reality (VR) video streaming employs the current tracking data of a user to predict future requested tiles, then renders and delivers the predicted tiles before playback. Very recently, privacy protection in proactive VR video streaming starts to raise concerns. However, existing privacy protection may fail even with privacy-preserve federated learning. This is because when the future requested tiles can be predicted accurately, the user-behavior-related data can still be recovered from the predicted tiles. In this paper, we consider how to protect privacy even wit…
Cardiff Metropolitan University（卡迪夫城市大学）；University of Western Australia（西澳大利亚大学）；Dublin City University（都柏林城市大学）；University College Dublin（都柏林大学学院）；University of Cagliari（卡利亚里大学）
The Quality of Experience (QoE) based service management remains key for successful provisioning of multimedia services in next-generation networks such as 5G/6G, which requires proper tools for quality monitoring, prediction and resource management where machine learning (ML) can play a crucial role. In this paper, we provide a tutorial on the development and deployment of the QoE measurement and prediction solutions for video streaming services based on supervised learning ML models. Firstly, we provide a detailed pipeline for developing and deploying supervised learning-based video streamin…
We take an analytical approach to study Quality of user Experience (QoE) for video streaming applications. First, we show that random linear network coding applied to blocks of video frames can significantly simplify the packet requests at the network layer and save resources by avoiding duplicate packet reception. Network coding allows us to model the receiver's buffer as a queue with Poisson arrivals and deterministic departures. We consider the probability of interruption in video playback as well as the number of initially buffered packets (initial waiting time) as the QoE metrics. We char…
HTTP Adaptive Streaming (HAS) is nowadays a popular solution for multimedia delivery. The novelty of HAS lies in the possibility of continuously adapting the streaming session to current network conditions, facilitated by Adaptive Bitrate (ABR) algorithms. Various popular streaming and Video on Demand services such as Netflix, Amazon Prime Video, and Twitch use this method. Given this broad consumer base, ABR algorithms continuously improve to increase user satisfaction. The insights for these improvements are, among others, gathered within the research area of Quality of Experience (QoE). Wit…
MPEG has recently proposed Server and Network Assisted Dynamic Adaptive Streaming over HTTP (SAND-DASH) for video streaming over the Internet. In contrast to the purely client-based video streaming in which each client makes its own decision to adjust its bitrate, SAND-DASH enables a group of simultaneous clients to select their bitrates in a coordinated fashion in order to improve resource utilization and quality of experience. In this paper, we study the performance of such an adaptation strategy compared to the traditional approach with large number of clients having mobile Internet access.…
With the merit of containing full panoramic content in one camera, Virtual Reality (VR) and 360-degree videos have attracted more and more attention in the field of industrial cloud manufacturing and training. Industrial Internet of Things (IoT), where many VR terminals needed to be online at the same time, can hardly guarantee VR's bandwidth requirement. However, by making use of users' quality of experience (QoE) awareness factors, including the relative moving speed and depth difference between the viewpoint and other content, bandwidth consumption can be reduced. In this paper, we propose …
University of Waterloo（滑铁卢大学）；Shanghai Jiao Tong University（上海交通大学）；Ningbo University（宁波大学）
Quality of experience (QoE) assessment for adaptive video streaming plays a significant role in advanced network management systems. It is especially challenging in case of dynamic adaptive streaming schemes over HTTP (DASH) which has increasingly complex characteristics including additional playback issues. In this paper, we provide a brief overview of adaptive video streaming quality assessment. Upon our review of related works, we analyze and compare different variations of objective QoE assessment models with or without using machine learning techniques for adaptive video streaming. Throug…
Scalable video coding has drawn great interest in content delivery in many multimedia services thanks to its capability to handle terminal heterogeneity and network conditions variation. In our previous work, and under the umbrella of ENVISION, we have proposed a playout smoothing mechanism to ensure the uniform delivery of the layered stream, by reducing the quality changes that the stream undergoes when adapting to changing network conditions. In this paper we study the resulting video quality, from the final user perception under different network conditions of loss and delays. For that we …
Continuous QoE prediction is crucial in the purpose of maximizing viewer satisfaction, by which video service providers could improve the revenue. Continuously predicting QoE is challenging since it requires QoE models that are capable of capturing the complex dependencies among QoE influence factors. The existing approaches that utilize Long-Short-Term-Memory (LSTM) network successfully model such long-term dependencies, providing the superior QoE prediction performance. However, the inherent drawback of sequential computing of LSTM will result in high computational cost in training and predi…
In this work, a new approach to the definition of the quality of experience is presented. By considering the quality of service as a baseline, that portion of the QoE that can be inferred from the QoS is excluded, and then the rest of the QoE is approached with the notion of QoE at a Boundary (QoEaaB). With the QoEaaB as the core of the proposed approach, various potential boundaries, and their associated unseen opportunities to improve the QoE are discussed. In particular, property, contract, SLA, and content are explored in terms of their boundaries and also their associated QoEaaB. With an …
Proactive tile-based virtual reality (VR) video streaming employs the current tracking data of a user to predict future requested tiles, then renders and delivers the predicted tiles to be requested before playback. The quality of experience (QoE) depends on the overall performance of prediction, computing (i.e., rendering) and communication. All prior works neglect that users may have privacy requirement, i.e., not all the current tracking data are allowed to be uploaded. In this paper, we investigate the privacy-aware VR streaming. We first establish a dataset that collects the privacy requi…
The live streaming is more challenging than on-demand streaming, because the low latency is also a strong requirement in addition to the trade-off between video quality and jitters in playback. To balance several inherently conflicting performance metrics and improve the overall quality of experience (QoE), many adaptation schemes have been proposed. Bitrate adaptation is one of the major solutions for video streaming under time-varying network conditions, which works even better combining with some latency control methods, such as adaptive playback rate control and frame dropping. However, it…
The knowledge of future throughput variations in mobile networks becomes more and more possible today thanks to the rich contextual information provided by mobile applications and services and smartphone sensors. It is even likely that such contextual information, which may include traffic, mobility and radio conditions will lead to a novel agile resource management not yet thought of. In this paper, we propose an framework (called NEWCAST) that anticipates the throughput variations to deliver video streaming content. We develop an optimization problem that realizes a fundamental trade-off amo…
This paper presents a brief review of some existing correlation models which attempt to map Quality of Service (QoS) to Quality of Experience (QoE) for multimedia services. The term QoS refers to deterministic network behaviour, so that data can be transported with a minimum of packet loss, delay and maximum bandwidth. QoE is a subjective measure that involves human dimensions; it ties together user perception, expectations, and experience of the application and network performance. The Holy Grail of subjective measurement is to predict it from the objective measurements; in other words predic…
Proactive tile-based virtual reality (VR) video streaming can use the trace of FoV and eye movement to predict future requested tiles, then renders and delivers the predicted tiles before playback. The quality of experience (QoE) depends on the combined effect of tile prediction and consumed resources. Recently, it has been found that with the FoV and eye movement data collected for a user, one can infer the identity and preference of the user. Existing works investigate the privacy protection for eye movement, but never address how to protect the privacy in terms of FoV and how the privacy pr…
HTTP Adaptive Streaming (HAS) has become a cost-effective means for multimedia delivery nowadays. However, how the quality of experience (QoE) is jointly affected by 1) varying perceptual quality and 2) interruptions is not well-understood. In this paper, we present the first attempt to quantitatively quantify the relative impacts of these factors on the QoE of streaming sessions. To achieve this purpose, we first model the impacts of the factors using histograms, which represent the frequency distributions of the individual factors in a session. By using a large dataset, various insights into…
Our purpose in this paper is to characterize buffer starvations for streaming services. The buffer is modeled as an M/M/1 queue, plus the consideration of bursty arrivals. When the buffer is empty, the service restarts after a certain amount of packets are \emph{prefetched}. With this goal, we propose two approaches to obtain the \emph{exact distribution} of the number of buffer starvations, one of which is based on \emph{Ballot theorem}, and the other uses recursive equations. The Ballot theorem approach gives an explicit result. We extend this approach to the scenario with a constant playbac…
Driven by the tremendous technological advancement of personal devices and the prevalence of wireless mobile network accesses, the world has witnessed an explosion in crowdsourced live streaming. Ensuring a better viewers quality of experience (QoE) is the key to maximize the audiences number and increase streaming providers' profits. This can be achieved by advocating a geo-distributed cloud infrastructure to allocate the multimedia resources as close as possible to viewers, in order to minimize the access delay and video stalls. Moreover, allocating the exact needed resources beforehand avoi…
Although HTTP-based video streaming can easily penetrate firewalls and profit from Web caches, the underlying TCP may introduce large delays in case of a sudden capacity loss. To avoid an interruption of the video stream in such cases we propose the Backward-Shifted Coding (BSC). Based on Scalable Video Coding (SVC), BSC adds a time-shifted layer of redundancy to the video stream such that future frames are downloaded at any instant. This pre-fetched content maintains a fluent video stream even under highly variant network conditions and leads to high Quality of Experience (QoE). We characteri…
Multimedia streaming to mobile devices is challenging for two reasons. First, the way content is delivered to a client must ensure that the user does not experience a long initial playback delay or a distorted playback in the middle of a streaming session. Second, multimedia streaming applications are among the most energy hungry applications in smartphones. The energy consumption mostly depends on the delivery techniques and on the power management techniques of wireless access technologies (Wi-Fi, 3G, and 4G). In order to provide insights on what kind of streaming techniques exist, how they …
Existing reinforcement learning~(RL)-based adaptive bitrate~(ABR) approaches outperform the previous fixed control rules based methods by improving the Quality of Experience~(QoE) score, as the QoE metric can hardly provide clear guidance for optimization, finally resulting in the unexpected strategies. In this paper, we propose \emph{Tiyuntsong}, a self-play reinforcement learning approach with generative adversarial network~(GAN)-based method for ABR video streaming. Tiyuntsong learns strategies automatically by training two agents who are competing against each other. Note that the competit…
We propose a robust scheme for streaming 360-degree immersive videos to maximize the quality of experience (QoE). Our streaming approach introduces a holistic analytical framework built upon the formal method of stochastic optimization. We propose a robust algorithm which provides a streaming rate such that the video quality degrades below that rate with very low probability even in presence of uncertain head movement, and bandwidth. It assumes the knowledge of the viewing probability of different portions (tiles) of a panoramic scene. Such probabilities can be easily derived from crowdsourced…
HTTP Adaptive Streaming (HAS) techniques are now the dominant solution for video delivery in mobile networks. Over the past few years, several HAS algorithms have been introduced in order to improve user quality-of-experience (QoE) by bit-rate adaptation. Their difference is mainly the required input information, ranging from network characteristics to application-layer parameters such as the playback buffer. Interestingly, despite the recent outburst in scientific papers on the topic, a comprehensive comparative study of the main algorithm classes is still missing. In this paper we provide su…
Immersive media streaming, especially virtual reality (VR)/360-degree video streaming which is very bandwidth demanding, has become more and more popular due to the rapid growth of the multimedia and networking deployments. To better explore the usage of resource and achieve better quality of experience (QoE) perceived by users, this paper develops an application-layer scheme to jointly exploit the available bandwidth from the LTE and Wi-Fi networks in 360-degree video streaming. This newly proposed scheme and the corresponding solution algorithms utilize the saliency of video, prediction of u…
Inferring the quality of network services is the vital basis of optimization for network operators. However, prevailing real-time video streaming applications adopt encryption for security, leaving it a problem to extract Quality of Service (QoS) indicators of real-time video. In this paper, we propose DaI, a traffic-based real-time video quality estimator. DaI can partially decrypt the encrypted real-time video data and applies machine learning methods to estimate key objective Quality of Experience (QoE) metrics of real-time video. According to the experimental results, DaI can estimate obje…
Moscow Institute of Physics and Technology（莫斯科物理技术学院）
Dynamic adaptive streaming over HTTP provides the work of most multimedia services, however, the nature of this technology further complicates the assessment of the QoE (Quality of Experience). In this paper, the influence of various objective factors on the subjective estimation of the QoE of streaming video is studied. The paper presents standard and handcrafted features, shows their correlation and p-Value of significance. VQA (Video Quality Assessment) models based on regression and gradient boosting with SRCC reaching up to 0.9647 on the validation subsample are proposed. The proposed reg…
Shandong University（山东大学）；City University of Hong Kong（香港城市大学）
Rate adaptation is one of the most important issues in dynamic adaptive streaming over HTTP (DASH). Due to the frequent fluctuations of the network bandwidth and complex variations of video content, it is difficult to deal with the varying network conditions and video content perfectly by using a single rate adaptation method. In this paper, we propose an ensemble rate adaptation framework for DASH, which aims to leverage the advantages of multiple methods involved in the framework to improve the quality of experience (QoE) of users. The proposed framework is simple yet very effective. Specifi…
Point cloud video transmission is challenging due to high encoding/decoding complexity, high video bitrate, and low latency requirement. Consequently, conventional adaptive streaming methodologies often find themselves unsatisfactory to meet the requirements in threefold: 1) current algorithms reuse existing quality of experience (QoE) definitions while overlooking the unique features of point cloud video thus failing to provide optimal user experience, 2) most deep learning approaches require long-span data collections to learn sufficiently varied network conditions and result in long trainin…
The increasing demand for video streaming services with high Quality of Experience (QoE) has prompted a lot of research on client-side adaptation logic approaches. However, most algorithms use the client's previous download experience and do not use a crowd knowledge database generated by users of a professional service. We propose a new crowd algorithm that maximizes the QoE. Additionally, we show how crowd information can be integrated into existing algorithms and illustrate this with two state-of-the-art algorithms. We evaluate our algorithm and state-of-the-art algorithms (including our mo…
In mobile networks, users may lose coverage when entering a building due to the high signal attenuation at windows and walls. Under such conditions, services with minimum bit-rate requirements, such as video streaming, often show poor Quality-of-Experience (QoE). We will present a Bayesian detector that combines measurements from two Smartphone sensors to decide if a user is inside a building or not. Based on this coverage classification, we will propose an HTTP adaptive streaming (HAS) algorithm to increase playback stability at a high average bitrate. Measurements in a typical office buildin…
The rapid development of 5G communication technology has given birth to various real-time broadband communication services, such as augmented reality (AR), virtual reality (VR) and cloud games. Compared with traditional services, consumers tend to focus more on their subjective experience when utilizing these services. In the meantime, the problem of power consumption is particularly prominent in 5G and beyond. The traditional design of physical layer (PHY) receiver is based on maximizing spectrum efficiency or minimizing error, but this will no longer be the best after considering energy effi…
The cache and transcoding of the multi-access edge computing (MEC) server and wireless resource allocation in eNodeB interact and determine the quality of experience (QoE) of dynamic adaptive streaming over HTTP (DASH) clients in MEC networks. However, the relationship among the three factors has not been explored, which has led to limited improvement in clients' QoE. Therefore, we propose a joint optimization framework of video segment cache and transcoding in MEC servers and resource allocation to improve the QoE of DASH clients. Based on the established framework, we develop a MEC cache man…
