Motivated by emerging big streaming data processing paradigms (e.g., Twitter Storm, Streaming MapReduce), we investigate the problem of scheduling graphs over a large cluster of servers. Each graph is a job, where nodes represent compute tasks and edges indicate data-flows between these compute tasks. Jobs (graphs) arrive randomly over time, and upon completion, leave the system. When a job arrives, the scheduler needs to partition the graph and distribute it over the servers to satisfy load balancing and cost considerations. Specifically, neighboring compute tasks in the graph that are mapped to different servers incur load on the network; thus a mapping of the jobs among the servers incurs a cost that is proportional to the number of "broken edges". We propose a low complexity randomized scheduling algorithm that, without service preemptions, stabilizes the system with graph arrivals/departures; more importantly, it allows a smooth trade-off between minimizing average partitioning cost and average queue lengths. Interestingly, to avoid service preemptions, our approach does not rely on a Gibbs sampler; instead, we show that the corresponding limiting invariant measure has an interpretation stemming from a loss system.
The era of big data has led to the emergence of new systems for real-time distributed stream processing, e.g., Apache Storm is one of the most popular stream processing systems in industry today. However, Storm, like many other stream processing systems lacks an intelligent scheduling mechanism. The default round-robin scheduling currently deployed in Storm disregards resource demands and availability, and can therefore be inefficient at times. We present R-Storm (Resource-Aware Storm), a system that implements resource-aware scheduling within Storm. R-Storm is designed to increase overall thr…
We propose and analyze centralized and distributed algorithms for device-to-device video scheduling and streaming. The proposed algorithms address jointly the problems of device-to-device link scheduling and video quality adaptation in streaming. Our simulations show that the proposed algorithms significantly outperform conventional separated approaches that treat these two problems independently.
During the last decade, we witnessed a rapid growth in deployment of pull-based P2P streaming applications. In these applications, each node selects some other nodes as its neighbors and requests streaming data from them. This scheme allows eliminating data redundancy and recovering from data loss, but it pushes the complexity to the receiver node side. In this paper, we theoretically study the scheduling problem in Pull-based P2P video streaming and we model it as an assignment problem. Then, we propose AsSched, new scheduling algorithm for layered streaming, in order to optimize the throughp…
Most online service providers deploy their own data stream processing systems in the cloud to conduct large-scale and real-time data analytics. However, such systems, e.g., Apache Heron, often adopt naive scheduling schemes to distribute data streams (in the units of tuples) among processing instances, which may result in workload imbalance and system disruption. Hence, there still exists a mismatch between the temporal variations of data streams and such inflexible scheduling scheme designs. Besides, the fundamental benefits of predictive scheduling to data stream processing systems also rema…
We consider the jointly optimal design of a transmission scheduling and admission control policy for adaptive video streaming over small cell networks. We formulate the problem as a dynamic network utility maximization and observe that it naturally decomposes into two subproblems: admission control and transmission scheduling. The resulting algorithms are simple and suitable for distributed implementation. The admission control decisions involve each user choosing the quality of the video chunk asked for download, based on the network congestion in its neighborhood. This form of admission cont…
Quality of Service (QoS) is now regarded as a requirement for all networks in managing resources like bandwidth and avoidance of network impairments like packet loss, jitter, and delay. Media transfer or streaming would be virtually impossible if QoS parameters were not used even if the streaming protocols were perfectly designed. QoS Scheduling classes help in network traffic optimization and the priority management of packets. This paper presents an analysis of QoS scheduling classes using video traffic in a MANET. The main objective was to identify a scheduling class that provides better Qo…
In this paper, we consider the multi-user scheduling problem in millimeter wave (mmWave) video streaming networks, which comprise a streaming server and several users, each requesting a video stream with a different resolution. The main objective is to optimize the long-term average quality of experience (QoE) for all users. We tackle this problem by considering the physical layer characteristics of the mmWave network, including the beam alignment overhead due to pencil-beams. To develop an efficient scheduling policy, we leverage the contextual multi-armed bandit (MAB) models to propose a bea…
Virtual Reality (VR) applications require high data rate for a high-quality immersive experience, in addition to low latency to avoid dizziness and motion sickness. One of the key wireless VR challenges is providing seamless connectivity and meeting the stringent latency and bandwidth requirements. This work proposes a proactive wireless VR system that utilizes information about the user's future orientation for proactive scheduling and caching. This is achieved by leveraging deep neural networks to predict users' orientation trained on a real dataset. The 360° scene is then partitioned using …
This paper presents a Brownian-approximation framework to optimize the quality of experience (QoE) for real-time video streaming in wireless networks. In real-time video streaming, one major challenge is to tackle the natural tension between the two most critical QoE metrics: playback latency and video interruption. To study this trade-off, we first propose an analytical model that precisely captures all aspects of the playback process of a real-time video stream, including playback latency, video interruptions, and packet dropping. Built on this model, we show that the playback process of a r…
As video-streaming services have expanded and improved, cloud-based video has evolved into a necessary feature of any successful business for reaching internal and external audiences. In this paper, video streaming over distributed storage is considered where the video segments are encoded using an erasure code for better reliability. There are multiple parallel streams between each server and the edge router. For each client request, we need to determine the subset of servers to get the data, as well as one of the parallel stream from each chosen server. In order to have this scheduling, this…
This paper studies the problem of serving multiple live video streams to several different clients from a single access point over unreliable wireless links, which is expected to be major a consumer of future wireless capacity. This problem involves two characteristics. On the streaming side, different video streams may generate variable-bit-rate traffic with different traffic patterns. On the network side, the wireless transmissions are unreliable, and the link qualities differ from client to client. In order to alleviate the above stochastic aspects of both video streams and link unreliabili…
Layered video streaming in peer-to-peer (P2P) networks has drawn great interest, since it can not only accommodate large numbers of users, but also handle peer heterogeneity. However, there's still a lack of comprehensive studies on chunk scheduling for the smooth playout of layered streams in P2P networks. In these situations, a playout smoothing mechanism can be used to ensure the uniform delivery of the layered stream. This can be achieved by reducing the quality changes that the stream undergoes when adapting to changing network conditions. This paper complements previous efforts in throug…
Environmental and economic concerns promote research on designing energy-efficient Time and Wavelength Division Multiplexed Ethernet Passive Optical Network (TWDM-EPON), which is the future extension to TDM-EPON. In TDM-EPON, a plethora of research is already present to achieve energy savings at Optical Network Units (ONUs) which can easily be applied for TWDM-EPON ONUs. However, TWDM-EPON provides an additional opportunity for saving energy at the Optical Line Terminal (OLT). All existing protocols have primarily been designed for saving energy at the OLT receivers. The protocols to save ener…
This paper studies the problem of mobile video delivery in heterogenous wireless networks from a server to multihomed device. Most existing works only consider delivering video streaming on single path which bandwidth is limited causing ultimate video transmission rate. To solve this live video streaming transmission bottleneck problem, we propose a novel solution named Joint Data Allocation and Fountain Coding (JDAFC) method that contain below characters: (1) path selection, (2) dynamic data allocation, and (3) fountain coding. We evaluate the performance of JDAFC by simulation experiments us…
In this paper, we consider the problem of optimal scalable video delivery to mobile users in wireless networks given arbitrary Quality Adaptation (QA) mechanisms. In current practical systems, QA and scheduling are performed independently by the content provider and network operator, respectively. While most research has been focused on jointly optimizing these two tasks, the high complexity that comes with a joint approach makes the implementation impractical. Therefore, we present a scheduling mechanism that takes the QA logic of each user as input and optimizes the scheduling accordingly. H…
The emergence of novel interactive multimedia applications with high rate and low latency requirements has led to a drastic increase in the video data traffic over wireless cellular networks. Endowing the small base stations of a macro-cell with caches that can store some of the content is a promising technology to cope with the increasing pressure on the backhaul connections, and to reduce the delay for demanding video applications. In this work, delivery of an interactive multiview video to a set of wireless users is studied in an heterogeneous cellular network. Differently from existing wor…
Palantir: Towards Efficient Super Resolution for Ultra-high-definition Live Streaming
School of Software, Tsinghua University（清华大学软件学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Global Innovation Exchange, Tsinghua University（清华大学全球创新交流中心）；School of Computing Science, Simon Fraser University（Simon Fraser大学计算机科学系）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学系）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Pengcheng Laboratory, China（鹏城实验室）；RISC-V International Open Source Laboratory, China（RISC-V国际开源实验室）
Data communication in cloud-based distributed stream data analytics often involves a collection of parallel and pipelined TCP flows. As the standard TCP congestion control mechanism is designed for achieving "fairness" among competing flows and is agnostic to the application layer contexts, the bandwidth allocation among a set of TCP flows traversing bottleneck links often leads to sub-optimal application-layer performance measures, e.g., stream processing throughput or average tuple complete latency. Motivated by this and enabled by the rapid development of the Software-Defined Networking (SD…
Multiview video supports observing a scene from different viewpoints. The Joint Video Team (JVT) developed H.264/MVC to enhance the compression efficiency for multiview video, however, MVC encoded multiview video (MVC video) still requires high bitrates for transmission. This paper investigates live MVC video streaming over Peer-to-Peer (P2P) networks. The goal is to minimize the server bandwidth costs whist ensuring high streaming quality to peers. MVC employs intra-view and inter-view prediction structures, which leads to a complicated layer dependency relationship. As the peers' outbound ba…
Proportional fair scheduling (PFS) has been adopted as a standard solution for fair resource allocation in modern wireless cellular networks. With the emergence of heterogeneous networks with widely varying user loads, it is of great importance to characterize the performance of PFS under bursty traffic, which is the case in most wireless streaming and data transfer services. In this letter, we provide the first analytical solution to the performance of PFS under bursty on-off traffic load. We use the Gaussian approximation model to derive a closed-form expression of the achievable user data r…
In this paper, we deal with the problem of jointly determining the optimal coding strategy and the scheduling decisions when receivers obtain layered data from multiple servers. The layered data is encoded by means of Prioritized Random Linear Coding (PRLC) in order to be resilient to channel loss while respecting the unequal levels of importance in the data, and data blocks are transmitted simultaneously in order to reduce decoding delays and improve the delivery performance. We formulate the optimal coding and scheduling decisions problem in our novel framework with the help of Markov Decisi…
