In recent years, live video streaming has gained widespread popularity across various social media platforms. Quality of experience (QoE), which reflects end-users' satisfaction and overall experience, plays a critical role for media service providers to optimize large-scale live compression and transmission strategies to achieve perceptually optimal rate-distortion trade-off. Although many QoE metrics for video-on-demand (VoD) have been proposed, there remain significant challenges in developing QoE metrics for live video streaming. To bridge this gap, we conduct a comprehensive study of subjective and objective QoE evaluations for live video streaming. For the subjective QoE study, we introduce the first live video streaming QoE dataset, TaoLive QoE, which consists of $42$ source videos collected from real live broadcasts and $1,155$ corresponding distorted ones degraded due to a variety of streaming distortions, including conventional streaming distortions such as compression, stalling, as well as live streaming-specific distortions like frame skipping, variable frame rate, etc. Subsequently, a human study was conducted to derive subjective QoE scores of videos in the TaoLive QoE dataset. For the objective QoE study, we benchmark existing QoE models on the TaoLive QoE dataset as well as publicly available QoE datasets for VoD scenarios, highlighting that current models struggle to accurately assess video QoE, particularly for live content. Hence, we propose an end-to-end QoE evaluation model, Tao-QoE, which integrates multi-scale semantic features and optical flow-based motion features to predicting a retrospective QoE score, eliminating reliance on statistical quality of service (QoS) features.
Point cloud video has been widely used by augmented reality (AR) and virtual reality (VR) applications as it allows users to have an immersive experience of six degrees of freedom (6DoFs). Yet there is still a lack of research on quality of experience (QoE) model of point cloud video streaming, which cannot provide optimization metric for streaming systems. Besides, position and color information contained in each pixel of point cloud video, and viewport distance effect caused by 6DoFs viewing procedure make the traditional objective quality evaluation metric cannot be directly used in point c…
Multimedia content delivery over the Internet is predominantly using the Hypertext Transfer Protocol (HTTP) as its primary protocol and multiple proprietary solutions exits. The MPEG standard Dynamic Adaptive Streaming over HTTP (DASH) provides an interoperable solution and in recent years various adaptation logics/algorithms have been proposed. However, to the best of our knowledge, there is no comprehensive evaluation of the various logics/algorithms. Therefore, this paper provides a comprehensive evaluation of ten different adaptation logics/algorithms, which have been proposed in the past …
This paper proposes a method for generating bullet comments for live-streaming games based on highlights (i.e., the exciting parts of video clips) extracted from the game content and evaluate the effect of mental health promotion. Game live streaming is becoming a popular theme for academic research. Compared to traditional online video sharing platforms, such as Youtube and Vimeo, video live streaming platform has the benefits of communicating with other viewers in real-time. In sports broadcasting, the commentator plays an essential role as mood maker by making matches more exciting. The enj…
Renmin University of China（中国人民大学）；State Key Laboratory of Internet Architecture, Tsinghua University（清华大学网络与交换技术全国重点实验室）；Tencent Technologies（腾讯科技）；Shenzhen University（深圳大学）
The fundamental conflict between the enormous space of adaptive streaming videos and the limited capacity for subjective experiment casts significant challenges to objective Quality-of-Experience (QoE) prediction. Existing objective QoE models exhibit complex functional form, failing to generalize well in diverse streaming environments. In this study, we propose an objective QoE model namely knowledge-driven streaming quality index (KSQI) to integrate prior knowledge on the human visual system and human annotated data in a principled way. By analyzing the subjective characteristics towards str…
University of Macau（澳门大学）；vivo AI Lab（vivo人工智能实验室）
With the prevailing of live video streaming, establishing an online pixelation method for privacy-sensitive objects is an urgency. Caused by the inaccurate detection of privacy-sensitive objects, simply migrating the tracking-by-detection structure into the online form will incur problems in target initialization, drifting, and over-pixelation. To cope with the inevitable but impacting detection issue, we propose a novel Privacy-sensitive Objects Pixelation (PsOP) framework for automatic personal privacy filtering during live video streaming. Leveraging pre-trained detection networks, our PsOP…
Recent years have witnessed a dramatic increase of user-generated video services. In such user-generated video services, crowdsourced live streaming (e.g., Periscope, Twitch) has significantly challenged today's edge network infrastructure: today's edge networks (e.g., 4G, Wi-Fi) have limited uplink capacity support, making high-bitrate live streaming over such links fundamentally impossible. In this paper, we propose to let broadcasters (i.e., users who generate the video) upload crowdsourced video streams using aggregated network resources from multiple edge networks. There are several chall…
The demand for streaming media and live video conferencing is at peak and expected to grow further, thereby the need for low-cost streaming services with better quality and lower latency is essential. Therefore, in this paper, we propose a novel peer-to-peer (P2P) live streaming platform, called fybrrStream, where a logical mesh and physical tree i.e., hybrid topology-based approach is leveraged for low latency streaming. fybrrStream distributes the load on participating peers in a hierarchical manner by considering their network bandwidth, network latency, and node stability. fybrrStream cost…
University of Tsukuba（筑波大学）；University of Tokyo（东京大学）；Gunosy Inc.（Gunosy公司）
Billions of live-streaming viewers share their opinions on scenes they are watching in real-time and interact with the event, commentators as well as other viewers via text comments. Thus, there is necessary to explore viewers' comments with scenes in E-sport live-streaming events. In this paper, we developed CS-lol, a new large-scale dataset containing comments from viewers paired with descriptions of game scenes in E-sports live-streaming. Moreover, we propose a task, namely viewer comment retrieval, to retrieve the viewer comments for the scene of the live-streaming event. Results on a seri…
In live streaming applications, a fixed set of bitrate-resolution pairs (known as bitrate ladder) is generally used to avoid additional pre-processing run-time to analyze the complexity of every video content and determine the optimized bitrate ladder. Furthermore, live encoders use the fastest available preset for encoding to ensure the minimum possible latency in streaming. For live encoders, it is expected that the encoding speed is equal to the video framerate. An optimized encoding preset may result in (i) increased Quality of Experience (QoE) and (ii) improved CPU utilization while encod…
Empowered by today's rich tools for media generation and collaborative production, the multimedia service paradigm is shifting from the conventional single source, to multi-source, to many sources, and now toward {\em crowdsource}. Such crowdsourced live streaming platforms as Twitch.tv allow general users to broadcast their content to massive viewers, thereby greatly expanding the content and user bases. The resources available for these non-professional broadcasters however are limited and unstable, which potentially impair the streaming quality and viewers' experience. The diverse live inte…
Live multimedia streaming from mobile devices is rapidly gaining popularity but little is known about the QoE they provide. In this paper, we examine the Periscope service. We first crawl the service in order to understand its usage patterns. Then, we study the protocols used, the typical quality of experience indicators, such as playback smoothness and latency, video quality, and the energy consumption of the Android application.
The live streaming is more challenging than on-demand streaming, because the low latency is also a strong requirement in addition to the trade-off between video quality and jitters in playback. To balance several inherently conflicting performance metrics and improve the overall quality of experience (QoE), many adaptation schemes have been proposed. Bitrate adaptation is one of the major solutions for video streaming under time-varying network conditions, which works even better combining with some latency control methods, such as adaptive playback rate control and frame dropping. However, it…
Empowered by today's rich tools for media generation and distribution, and the convenient Internet access, crowdsourced streaming generalizes the single-source streaming paradigm by including massive contributors for a video channel. It calls a joint optimization along the path from crowdsourcers, through streaming servers, to the end-users to minimize the overall latency. The dynamics of the video sources, together with the globalized request demands and the high computation demand from each sourcer, make crowdsourced live streaming challenging even with powerful support from modern cloud com…
The diversity of video delivery pipeline poses a grand challenge to the evaluation of adaptive bitrate (ABR) streaming algorithms and objective quality-of-experience (QoE) models. Here we introduce so-far the largest subject-rated database of its kind, namely WaterlooSQoE-IV, consisting of 1350 adaptive streaming videos created from diverse source contents, video encoders, network traces, ABR algorithms, and viewing devices. We collect human opinions for each video with a series of carefully designed subjective experiments. Subsequent data analysis and testing/comparison of ABR algorithms and …
360-degree video provides an immersive 360-degree viewing experience and has been widely used in many areas. The 360-degree video live streaming systems involve capturing, compression, uplink (camera to video server) and downlink (video server to user) transmissions. However, few studies have jointly investigated such complex systems, especially the rate adaptation for the coupled uplink and downlink in the 360-degree video streaming under limited bandwidth constraints. In this letter, we propose a quality of experience (QoE)-driven 360-degree video live streaming system, in which a video serv…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google（谷歌）
High frame rate (HFR) videos are becoming increasingly common with the tremendous popularity of live, high-action streaming content such as sports. Although HFR contents are generally of very high quality, high bandwidth requirements make them challenging to deliver efficiently, while simultaneously maintaining their quality. To optimize trade-offs between bandwidth requirements and video quality, in terms of frame rate adaptation, it is imperative to understand the intricate relationship between frame rate and perceptual video quality. Towards advancing progression in this direction we design…
Peer-to-Peer Live streaming and Video on Demand is the most popular media applications over the Internet in recent years. These systems reduce the load on the server and provide a scalable content distribution. A new paradigm of P2P network collaborates to build large distributed video applications on existing networks .But, the problem of designing the system are at par with the P2P media streaming, live and Video on demand systems. Hence a comprehensive design comparison is needed to build such kind of system architecture. Therefore, in this paper we elaborately studied the traditional appro…
HTTP Adaptive Streaming (HAS) is nowadays a popular solution for multimedia delivery. The novelty of HAS lies in the possibility of continuously adapting the streaming session to current network conditions, facilitated by Adaptive Bitrate (ABR) algorithms. Various popular streaming and Video on Demand services such as Netflix, Amazon Prime Video, and Twitch use this method. Given this broad consumer base, ABR algorithms continuously improve to increase user satisfaction. The insights for these improvements are, among others, gathered within the research area of Quality of Experience (QoE). Wit…
Newly developed HTTP-based video streaming technologies enable flexible rate-adaptation under varying channel conditions. Accurately predicting the users' Quality of Experience (QoE) for rate-adaptive HTTP video streams is thus critical to achieve efficiency. An important aspect of understanding and modeling QoE is predicting the up-to-the-moment subjective quality of a video as it is played, which is difficult due to hysteresis effects and nonlinearities in human behavioral responses. This paper presents a Hammerstein-Wiener model for predicting the time-varying subjective quality (TVSQ) of r…
Crowdsourced Live Streaming (CLS), most notably Twitch.tv, has seen explosive growth in its popularity in the past few years. In such systems, any user can lively broadcast video content of interest to others, e.g., from a game player to many online viewers. To fulfill the demands from both massive and heterogeneous broadcasters and viewers, expensive server clusters have been deployed to provide video ingesting and transcoding services. Despite the existence of highly popular channels, a significant portion of the channels is indeed unpopular. Yet as our measurement shows, these broadcasters …
University of Tsukuba（筑波大学）；University of British Columbia（不列颠哥伦比亚大学）；Ritsumeikan University（立命馆大学）
To solve the existing sentence punctuation problem for collaborative commentary generation in Esports live-streaming, this paper presents two strategies for sentence punctuation for text sequences of game commentary, that is, punctuating sentences by two or three text sequence(s) originally punctuated by Youtube to obtain a complete sentence of commentary. We conducted comparative experiments utilizing and fine-tuning a state-of-the-art pre-trained generative language model among two strategies and the baseline to generate collaborative commentary. Both objective evaluations by automatic metri…
In recent years, research efforts tried to exploit peer-to-peer (P2P) systems in order to provide Live Streaming (LS) and Video-on-Demand (VoD) services. Most of these research efforts focus on the development of distributed P2P block schedulers for content exchange among the participating peers and on the characteristics of the overlay graph (P2P overlay) that interconnects the set of these peers. Currently, researchers try to combine peer-to-peer systems with cloud infrastructures. They developed monitoring and control architectures that use resources from the cloud in order to enhance QoS a…
A Multimodal Transformer for Live Streaming Highlight Prediction
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；KuaiShou Inc.（快手公司）；CAIR, HK Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
This paper studies the statistical nature of TV channels streaming variable bit rate distribution and allocation. The goal of the paper is to derive the best-fit rate distribution to describe TV streaming bandwidth allocation, which can reveal traffic demands of users. Our analysis uses multiplexers channel bandwidth allocation (PID) data of 13 TV live channels. We apply 17 continuous and 3 discrete distributions to determine the best-fit distribution function for each individual channel and for the whole set of channels. We found that the generalized extreme distribution fitting most of our c…
YTLive: A Dataset of Real-World YouTube Live Streaming Sessions
School of Electrical and Computer Engineering, College of Engineering, University of Tehran（德黑兰大学电气与计算机工程学院）；Department of Computer Engineering, Sharif University of Technology（谢里夫理工大学计算机工程系）；Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität Klagenfurt（克拉根福阿尔卑斯-亚得里亚大学ATHENA克里斯蒂安·多普勒实验室）
Initial offset placement in p2p streaming systems is studied in this paper. Proportional placement (PP) scheme is proposed. In this scheme, peer places the initial offset as the offset reported by other reference peer with a shift proportional to the buffer width or offset lag of this reference peer. This will introduce a stable placement that supports larger buffer width for peers and small buffer width for tracker. Real deployed placement method in PPLive is studied through measurement. It shows that, instead of based on offset lag, the placement is based on buffer width of the reference pee…
With the advancement in personal smart devices and pervasive network connectivity, users are no longer passive content consumers, but also contributors in producing new contents. This expansion in live services requires a detailed analysis of broadcasters' and viewers' behavior to maximize users' Quality of Experience (QoE). In this paper, we present a dataset gathered from one of the popular live streaming platforms: Facebook. In this dataset, we stored more than 1,500,000 live stream records collected in June and July 2018. These data include public live videos from all over the world. Howev…
Multiview video supports observing a scene from different viewpoints. The Joint Video Team (JVT) developed H.264/MVC to enhance the compression efficiency for multiview video, however, MVC encoded multiview video (MVC video) still requires high bitrates for transmission. This paper investigates live MVC video streaming over Peer-to-Peer (P2P) networks. The goal is to minimize the server bandwidth costs whist ensuring high streaming quality to peers. MVC employs intra-view and inter-view prediction structures, which leads to a complicated layer dependency relationship. As the peers' outbound ba…
Hamad Bin Khalifa University（哈马德·本·哈利法大学）；Qatar University（卡塔尔大学）；Comsats Institute of Information Technology（COMSATS信息技术学院）
Crowdsourced live video streaming (livecast) services such as Facebook Live, YouNow, Douyu and Twitch are gaining more momentum recently. Allocating the limited resources in a cost-effective manner while maximizing the Quality of Service (QoS) through real-time delivery and the provision of the appropriate representations for all viewers is a challenging problem. In our paper, we introduce a machine-learning based predictive resource allocation framework for geo-distributed cloud sites, considering the delay and quality constraints to guarantee the maximum QoS for viewers and the minimum cost …
As an important application form of immersive multimedia services, free-viewpoint video(FVV) enables users with great immersive experience by strong interaction. However, the computational complexity of virtual view synthesis algorithms poses a significant challenge to the real-time performance of an FVV system. Furthermore, the individuality of user interaction makes it difficult to serve multiple users simultaneously for a system with conventional architecture. In this paper, we novelly introduce a CNN-based view interpolation algorithm to synthesis dense virtual views in real time. Based on…
Experience of live video streaming can be improved if the video uploader has more accurate knowledge about the future available bandwidth. Because with such knowledge, one is able to know what sizes should he encode the frames to be in an ever-changing network. Researchers have developed some algorithms to predict throughputs in the literature, from where some are simple hence practical. However, limitation remains as most current bandwidth prediction methods are predicting a value, or a point estimate, of future bandwidth. Because in many practical scenarios, it is desirable to control the pe…
Recently, HTTP-based adaptive streaming has become the de facto standard for video streaming over the Internet. It allows clients to dynamically adapt media characteristics to network conditions in order to ensure a high quality of experience, that is, minimize playback interruptions, while maximizing video quality at a reasonable level of quality changes. In the case of live streaming, this task becomes particularly challenging due to the latency constraints. The challenge further increases if a client uses a wireless network, where the throughput is subject to considerable fluctuations. Cons…
Several overlay-based live multimedia streaming platforms have been proposed in the recent peer-to-peer streaming literature. In most of the cases, the overlay neighbors are chosen randomly for robustness of the overlay. However, this causes nodes that are distant in terms of proximity in the underlying physical network to become neighbors, and thus data travels unnecessary distances before reaching the destination. For efficiency of bulk data transmission like multimedia streaming, the overlay neighborhood should resemble the proximity in the underlying network. In this paper, we exploit the …
The commercialization of Virtual Reality (VR) headsets has made immersive and 360-degree video streaming the subject of intense interest in the industry and research communities. While the basic principles of video streaming are the same, immersive video presents a set of specific challenges that need to be addressed. In this survey, we present the latest developments in the relevant literature on four of the most important ones: (i) omnidirectional video coding and compression, (ii) subjective and objective Quality of Experience (QoE) and the factors that can affect it, (iii) saliency measure…
Video streaming via TCP networks has become a popular and highly demanded service, but its quality assessment in both objective and subjective terms has not been properly addressed. In this paper, based on statistical analysis a full analytic model of a no-reference objective metric, namely Pause Intensity, for video quality assessment is presented. The model characterizes the video playout buffer behavior in connection with the network performance (throughput) and the video playout rate. This allows for instant quality measurement and control without requiring a reference video. Pause intensi…
