Many emerging distributed applications, including big data analytics, generate a number of flows that concurrently transport data across data center networks. To improve their performance, it is required to account for the behavior of a collection of flows, i.e., coflows, rather than individual. State-of-the-art solutions allow for a near-optimal completion time by continuously reordering the unfinished coflows at the end-host, using network priorities. This paper shows that dynamically changing flow priorities at the end host, without taking into account in-flight packets, can cause high-degrees of packet re-ordering, thus imposing pressure on the congestion control and potentially harming network performance in the presence of switches with shallow buffers. We present pCoflow, a new solution that integrates end-host based coflow ordering with in-network scheduling based on packet history. Our evaluation shows that pCoflow improves in CCT upon state-of-the-art solutions by up to 34% for varying load.
A coflow is a collection of parallel flows belonging to the same job. It has the all-or-nothing property: a coflow is not complete until the completion of all its constituent flows. In this paper, we focus on optimizing \emph{coflow-level delay}, i.e., the time to complete all the flows in a coflow, in the context of an $N\times N$ input-queued switch. In particular, we develop a throughput-optimal scheduling policy that achieves the best scaling of coflow-level delay as $N\rightarrow\infty$. We first derive lower bounds on the coflow-level delay that can be achieved by any scheduling policy. …
In distributed computing frameworks like MapReduce, Spark, and Dyrad, a coflow is a set of flows transferring data between two stages of a job. The job cannot start its next stage unless all flows in the coflow finish. To improve the execution performance of such a job, it is crucial to reduce the completion time of a coflow which can contribute more than 50% of the job completion time. While several schedulers have been proposed, we observe that routing, as a factor greatly impacting the Coflow Completion Time (CCT), has not been well considered. In this paper, we focus on the coflow scheduli…
The coflow scheduling problem is considered: given an input/output switch with each port having a fixed capacity, find a scheduling algorithm that minimizes the weighted sum of the coflow completion times respecting the port capacities, where each flow of a coflow has a demand per input/output port, and coflow completion time is the finishing time of the last flow of the coflow. The objective of this paper is to present theoretical guarantees on approximating the sum of coflow completion time in the non-clairvoyant setting, where on a coflow arrival, only the number of flows, and their input-o…
Datacenter networks routinely support the data transfers of distributed computing frameworks in the form of coflows, i.e., sets of concurrent flows related to a common task. The vast majority of the literature has focused on the problem of scheduling coflows for completion time minimization, i.e., to maximize the average rate at which coflows are dispatched in the network fabric. Modern applications, though, may generate coflows dedicated to online services and mission-critical computing tasks which have to comply with specific completion deadlines. In this paper, we introduce $\mathtt{DCoflow…
Modern data centers face new scheduling challenges in optimizing job-level performance objectives, where a significant challenge is the scheduling of highly parallel data flows with a common performance goal (e.g., the shuffle operations in MapReduce applications). Chowdhury and Stoica introduced the coflow abstraction to capture these parallel communication patterns, and Chowdhury et al. proposed effective heuristics to schedule coflows efficiently. In our previous paper, we considered the strongly NP-hard problem of minimizing the total weighted completion time of coflows with release dates,…
Applications in data-parallel computing typically consist of multiple stages. In each stage, a set of intermediate parallel data flows (Coflow) is produced and transferred between servers to enable starting of next stage. While there has been much research on scheduling isolated coflows, the dependency between coflows in multi-stage jobs has been largely ignored. In this paper, we consider scheduling coflows of multi-stage jobs represented by general DAGs (Directed Acyclic Graphs) in a shared data center network, so as to minimize the total weighted completion time of jobs. This problem is sig…
The average coflow completion time (CCT) is the standard performance metric in coflow scheduling. However, standard CCT minimization may introduce unfairness between the data transfer phase of different computing jobs. Thus, while progress guarantees have been introduced in the literature to mitigate this fairness issue, the trade-off between fairness and efficiency of data transfer is hard to control. This paper introduces a fairness framework for coflow scheduling based on the concept of slowdown, i.e., the performance loss of a coflow compared to isolation. By controlling the slowdown it is…
Technische Universität Berlin（柏林工业大学）；University of Pittsburgh（匹兹堡大学）；Arizona State University（亚利桑那州立大学）；Khoury College of Computer Sciences, Northeastern University（东北大学科尔里计算机学院）
Coflow scheduling improves data-intensive application performance by improving their networking performance. State-of-the-art Coflow schedulers in essence approximate the classic online Shortest-Job-First (SJF) scheduling, designed for a single CPU, in a distributed setting, with no coordination among how the flows of a Coflow at individual ports are scheduled, and as a result suffer two performance drawbacks: (1) The flows of a Coflow may suffer the out-of-sync problem -- they may be scheduled at different times and become drifting apart, negatively affecting the Coflow completion time (CCT);…
In the past decade, increasingly network scheduling techniques have been proposed to boost the distributed application performance. Flow-level metrics, such as flow completion time (FCT), are based on the abstraction of flows yet they cannot capture the semantics of communication in a cluster application. Being aware of this problem, coflow is proposed as a new network abstraction. However, it is insufficient to reveal the dependencies between computation and communication. As a result, the real application performance can be hurt, especially in the absence of hard barriers. Based on the compu…
Distributed applications, such as database queries and distributed training, consist of both compute and network tasks. DAG-based abstraction primarily targets compute tasks and has no explicit network-level scheduling. In contrast, Coflow abstraction collectively schedules network flows among compute tasks but lacks the end-to-end view of the application DAG. Because of the dependencies and interactions between these two types of tasks, it is sub-optimal to only consider one of them. We argue that co-scheduling of both compute and network tasks can help applications towards the globally optim…
We consider optimal/efficient power allocation policies in a single/multihop wireless network in the presence of hard end-to-end deadline delay constraints on the transmitted packets. Such constraints can be useful for real time voice and video. Power is consumed in only transmission of the data. We consider the case when the power used in transmission is a convex function of the data transmitted. We develop a computationally efficient online algorithm, which minimizes the average power for the single hop. We model this problem as dynamic program (DP) and obtain the optimal solution. Next, we …
Supporting quality of service (QoS) guarantees for diverse multimedia services are the primary concerns for WiMAX (IEEE 802.16) networks. A scheduling scheme that satisfies QoS requirements has become more important for wireless communications. We propose a downlink scheduling scheme called adaptive priority-based downlink scheduling (APDS) for providing QoS guarantees in IEEE 802.16 networks. APDS comprises two major components: priority assignment and resource allocation. Different service-type connections primarily depend on their QoS requirements to adjust priority assignments and dispatch…
Along with the rapid growth of Industrial Internet-of-Things (IIoT) applications and their penetration into many industry sectors, real-time wireless networks (RTWNs) have been playing a more critical role in providing real-time, reliable and secure communication services for such applications. A key challenge in RTWN management is how to ensure real-time Quality of Services (QoS) especially in the presence of unexpected disturbances and lossy wireless links. Most prior work takes centralized approaches for handling disturbances, which are slow and subject to single-point failure, and do not s…
In this paper, we study the performance of greedy scheduling in multihop wireless networks, where the objective is aggregate utility maximization. Following standard approaches, we consider the dual of the original optimization problem. The dual can be solved optimally, only with the knowledge of the maximal independent sets in the network. But computation of maximal independent sets is known to be NP-hard. Motivated by this, we propose a distributed greedy heuristic to address the problem of link scheduling. We evaluate the effect of the distributed greedy heuristic on aggregate utility maxim…
Control of multihop Wireless networks in a distributed manner while providing end-to-end delay requirements for different flows, is a challenging problem. Using the notions of Draining Time and Discrete Review from the theory of fluid limits of queues, an algorithm that meets delay requirements to various flows in a network is constructed. The algorithm involves an optimization which is implemented in a cyclic distributed manner across nodes by using the technique of iterative gradient ascent, with minimal information exchange between nodes. The algorithm uses time varying weights to give prio…
Today's data centers face extreme challenges in providing low latency. However, fair sharing, a principle commonly adopted in current congestion control protocols, is far from optimal for satisfying latency requirements. We propose Preemptive Distributed Quick (PDQ) flow scheduling, a protocol designed to complete flows quickly and meet flow deadlines. PDQ enables flow preemption to approximate a range of scheduling disciplines. For example, PDQ can emulate a shortest job first algorithm to give priority to the short flows by pausing the contending flows. PDQ borrows ideas from centralized sch…
In this work we consider the problem of downlink resource allocation for proportional fairness of long term received rates of data users and quality of service for real time sessions in an OFDMA-based wireless system. The base station allocates available power and bandwidth to individual users based on long term average received rates, QoS based rate constraints and channel conditions. We solve the underlying constrained optimization problem and propose an algorithm that achieves the optimal allocation. Numerical evaluation results show that the proposed algorithm provides better QoS to voice …
The evolution of 5G and Beyond networks has enabled new applications with stringent end-to-end latency requirements, but providing reliable low-latency service with high throughput over public wireless networks is still a significant challenge. One of the possible ways to solve this is to exploit path diversity, encoding the information flow over multiple streams across parallel links. The challenge presented by this approach is the design of joint coding and scheduling algorithms that adapt to the state of links to take full advantage of path diversity. In this paper, we address this problem …
Millimeter wave (mmWave) communication has recently attracted significant attention from both industrial and academic communities. The large bandwidth availability as well as low interference nature of mmWave spectrum is particularly attractive for industrial communication. However, inherent challenges such as coverage and blockage of mmWave communication cause highly fluctuated channel quality. This paper explores wireless medium access control (MAC) schedulers for mmWave-based industrial wireless applications. Our objective is to design a high-performance and enhanced fairness MAC scheduling…
With the advent of 5G and the evolution of Internet protocols, industrial applications are moving from vertical solutions to general purpose IP-based infrastructures that need to meet deterministic Quality of Service (QoS) requirements. The IETF DetNet working group aims at providing an answer to this need with support for (i) deterministic worst-case latency and jitter, and (ii) zero packet loss for time-sensitive traffic. In this paper we focus on the joint routing and scheduling problem in large scale deterministic networks using Cycle Specified Queuing and Forwarding (CSQF), an extension o…
In this paper, we develop algorithms for joint user scheduling and three types of mmWave link configuration: relay selection, codebook optimization, and beam tracking in millimeter wave (mmWave) networks. Our goal is to design an online controller that dynamically schedules users and configures their links to minimize the system delay. To solve this complex scheduling problem, we model it as a dynamic decision-making process and develop two reinforcement learning-based solutions. The first solution is based on deep reinforcement learning (DRL), which leverages the proximal policy optimization …
In this paper, we consider the multi-user scheduling problem in millimeter wave (mmWave) video streaming networks, which comprise a streaming server and several users, each requesting a video stream with a different resolution. The main objective is to optimize the long-term average quality of experience (QoE) for all users. We tackle this problem by considering the physical layer characteristics of the mmWave network, including the beam alignment overhead due to pencil-beams. To develop an efficient scheduling policy, we leverage the contextual multi-armed bandit (MAB) models to propose a bea…
The IEEE 802.11e standard introduced by IEEE 802.11 Task Group E (TGe) enhances the Quality of Service (QoS) by means of HCF Controlled Channel Access (HCCA). The scheduler of HCCA allocates Transmission Opportunities (TXOPs) to QoS-enabled Station (QSTA) based on their TS Specifications (TSPECs) negotiated at the traffic setup time so that it is only efficient for Constant Bit Rate (CBR) applications. However, Variable Bit Rate (VBR) traffics are not efficiently supported as they exhibit nondeterministic profile during the time. In this paper, we present a dynamic TXOP assignment Scheduling A…
Millimeter wave (mmWave) will play a significant role as a 5G candidate in facing the growing demand of enormous data rate in the near future. The conventional mmWave standard, IEEE 802.11ad, considers establishing only one mmWave link in wireless local area network (WLAN) to provide multi Gbps data rate. But, mmWave has a tenuous channel which hinders it from providing such rate. Hence, it's necessary to establish multiple mmWave links simultaneously by deploying a multiple number of mmWave access points (APs) in 5G networks. Unfortunately, applying conventional standard without any modificat…
In the first part of this paper we present the first steps towards providing low-risk and economically feasible network data transfer services. We introduce three types of data transfer services and present general guidelines and algorithms for managing service prices, risks and schedules. In the second part of the paper we solve two packet scheduling cost optimization problems and present efficient algorithms for identifying maximum weight (k-level-) caterpillar subtrees in tree networks.
