We consider general large-scale service systems with multiple customer classes and multiple server (agent) pools, mean service times depend both on the customer class and server pool. It is assumed that the allowed activities (routing choices) form a tree (in the graph with vertices being both customer classes and server pools). We study the behavior of the system under a natural (load balancing) routing/scheduling rule, Longest-Queue Freest-Server (LQFS-LB), in the many-server asymptotic regime, such that the exogenous arrival rates of the customer classes, as well as the number of agents in each pool, grow to infinity in proportion to some scaling parameter $r$. Equilibrium point of the system under LQBS-LB is the desired operating point, with server pool loads minimized and perfectly balanced. Our main results are as follows. (a) We show that, quite surprisingly (given the tree assumption), for certain parameter ranges, the fluid limit of the system may be unstable in the vicinity of the equilibrium point; such instability may occur if the activity graph is not "too small." (b) Using (a), we demonstrate that the sequence of stationary distributions of diffusion-scaled processes [measuring $O(\sqrt{r})$ deviations from the equilibrium point] may be nontight, and in fact may escape to infinity. (c) In one special case of interest, however, we show that the sequence of stationary distributions of diffusion-scaled processes is tight, and the limit of stationary distributions is the stationary distribution of the limiting diffusion process.
We study the load balancing system operating under Join the Shortest Queue (JSQ) in the many-server heavy-traffic regime. If $N$ is the number of servers, we let the difference between the total service rate and the total arrival rate be $N^{1-α}$ with $α>0$. We show that for $α>4$ the average queue length behaves similarly to the classical heavy-traffic regime. Specifically, we prove that the distribution of the average queue length multiplied by $N^{1-α}$ converges to an exponential random variable. Moreover, we show a result analogous to state space collapse. We provide two proofs for our r…
We consider a system of N queues with decentralized load balancing such as power-of-D strategies(where D may depend on N) and generic scheduling disciplines. To measure the dependence of the queues, we use the clan of ancestors, a technique coming from interacting particle systems. Relying in that analysis we prove quantitative estimates on the queues correlations implying propagation of chaos for systems with Markovian arrivals and general service time distribution. This solves the conjecture posed by Bramsom et. al. in [*] concerning the asymptotic independence of the servers in the case of …
We analyse Join-the-Shortest-Queue in a contemporary scaling regime known as the Non-Degenerate Slowdown regime. Join-the-Shortest-Queue (JSQ) is a classical load balancing policy for queueing systems with multiple parallel servers. Parallel server queueing systems are regularly analysed and dimensioned by diffusion approximations achieved in the Halfin-Whitt scaling regime. However, when jobs must be dispatched to a server upon arrival, we advocate the Non-Degenerate Slowdown regime (NDS) to compare different load-balancing rules. In this paper we identify novel diffusion approximation and ti…
This paper considers the steady-state performance of load balancing algorithms in a many-server system with distributed queues. The system has $N$ servers, and each server maintains a local queue with buffer size $b-1,$ i.e. a server can hold at most one job in service and $b-1$ jobs in the queue. Jobs in the same queue are served according to the first-in-first-out (FIFO) order. The system is operated in a heavy-traffic regime such that the workload per server is $λ= 1 - N^{-α}$ for $0.5\leq α<1.$ We identify a set of algorithms such that the steady-state queues have the following universal s…
It is well-known that the power-of-d choices routing algorithm maximizes throughput and is heavy-traffic optimal in load balancing systems with homogeneous servers. However, if the servers are heterogeneous, throughput optimality does not hold in general. We find necessary and sufficient conditions for throughput optimality of power-of-d choices when the servers are heterogeneous, and we prove that almost the same conditions are sufficient to show heavy-traffic optimality. Additionally, we generalize the sufficient condition for throughput optimality to a larger class of routing policies.
To reduce carbon emission in the transportation sector, there is currently a steady move taking place to an electrified transportation system. This brings about various issues for which a promising solution involves the construction and operation of a battery swapping infrastructure rather than in-vehicle charging of batteries. In this paper, we study a closed Markovian queueing network that allows for spare batteries under a dynamic arrival policy. We propose a provisioning rule for the capacity levels and show that these lead to near-optimal resource utilization, while guaranteeing good qual…
This paper studies a class of load balancing algorithms for many-server ($N$ servers) systems assuming finite buffer with size $b-1$ (i.e. a server can have at most one job in service and $b-1$ jobs in queue). We focus on steady-state performance of load balancing algorithms in the heavy traffic regime such that the load of system is $λ= 1 - N^{-α}$ for $0<α<0.5,$ which we call sub-Halfin-Whitt regime ($α=0.5$ is the so-called the Halfin-Whitt regime). We establish a sufficient condition under which the probability that an incoming job is routed to an idle server is one asymptotically. The cla…
We consider a system of $N$ parallel queues with identical exponential service rates and a single dispatcher where tasks arrive as a Poisson process. When a task arrives, the dispatcher always assigns it to an idle server, if there is any, and to a server with the shortest queue among $d$ randomly selected servers otherwise $(1 \leq d \leq N)$. This load balancing scheme subsumes the so-called Join-the-Idle Queue (JIQ) policy $(d = 1)$ and the celebrated Join-the-Shortest Queue (JSQ) policy $(d = N)$ as two crucial special cases. We develop a stochastic coupling construction to obtain the diff…
We consider a large-scale flexible service system with two large server pools and two types of customers. Servers in pool 1 can only serve type 1 customers, while server in pool 2 are flexible -- they can serve both types 1 and 2. (This is a so-called "N-system." Our results hold for a more general class of systems as well.) The service rate of a customer depends both on its type and the pool where it is served. We study a priority service discipline, where type 2 has priority in pool 2, and type 1 prefers pool 1. We consider the Halfin-Whitt asymptotic regime, where the arrival rate of custom…
We consider load balancing in service systems with affinity relations between jobs and servers. Specifically, an arriving job can be allocated to a fast, primary server from a particular selection associated with this job or to a secondary server to be processed at a slower rate. Such job-server affinity relations can model network topologies based on geographical proximity, or data locality in cloud scenarios. We introduce load balancing schemes that allocate jobs to primary servers if available, and otherwise to secondary servers. A novel coupling construction is developed to obtain stabilit…
Efficiently exploiting servers in data centers requires performance analysis methods that account not only for the stochastic nature of demand but also for server heterogeneity. Although several recent works proved optimality results for heterogeneity-aware variants of classical load-balancing algorithms in the many-server regime, we still lack a fundamental understanding of the impact of heterogeneity on performance in finite-size systems. In this paper, we consider a load-balancing algorithm that leads to a product-form queueing model and can therefore be analyzed exactly even when the numbe…
AM/M/N+Mqueueingnetworkisconsideredwithdindependentcustomerclasses and d server pools in Halfin-Whitt regime. Class i customers has priority for service in pool i for i = 1, . . . , d, and may access some other pool if the pool has an idle server and all the servers in pool i are busy. We formulate an ergodic control problem where the running cost is given by a non- negative convex function with polynomial growth. We show that the limiting controlled diffusion is modeled by an action space which depends on the state variable. We provide a complete analysis for the limiting ergodic control prob…
In large storage systems, files are often coded across several servers to improve reliability and retrieval speed. We study load balancing under the Batch Sampling routing scheme for a network of $n$ servers storing a set of files using the Maximum Distance Separable (MDS) code (cf. Li, Ramamoorthy, and Srikant (2016)). Specifically, each file is stored in equally sized pieces across $L$ servers such that any $k$ pieces can reconstruct the original file. When a request for a file is received, the dispatcher routes the job into the $k$-shortest queues among the $L$ for which the corresponding s…
This paper studies load balancing for many-server ($N$ servers) systems. Each server has a buffer of size $b-1,$ and can have at most one job in service and $b-1$ jobs in the buffer. The service time of a job follows the Coxian-2 distribution. We focus on steady-state performance of load balancing policies in the heavy traffic regime such that the normalized load of system is $λ= 1 - N^{-α}$ for $0<α<0.5.$ We identify a set of policies that achieve asymptotic zero waiting. The set of policies include several classical policies such as join-the-shortest-queue (JSQ), join-the-idle-queue (JIQ), i…
We consider a system of $N$ parallel single-server queues with unit exponential service rates and a single dispatcher where tasks arrive as a Poisson process of rate $λ(N)$. When a task arrives, the dispatcher assigns it to a server with the shortest queue among $d(N)$ randomly selected servers ($1 \leq d(N) \leq N$). This load balancing strategy is referred to as a JSQ($d(N)$) scheme, marking that it subsumes the celebrated Join-the-Shortest Queue (JSQ) policy as a crucial special case for $d(N) = N$. We construct a stochastic coupling to bound the difference in the queue length processes bet…
Applications in cloud platforms motivate the study of efficient load balancing under job-server constraints and server heterogeneity. In this paper, we study load balancing on a bipartite graph where left nodes correspond to job types and right nodes correspond to servers, with each edge indicating that a job type can be served by a server. Thus edges represent locality constraints, i.e., each job can only be served at servers which contained certain data and/or machine learning (ML) models. Servers in this system can have heterogeneous service rates. In this setting, we investigate the perfor…
We consider a system with several job types and two parallel server pools. Within the pools the servers are homogeneous, but across pools possibly not in the sense that the service speed of a job may depend on its type as well as the server pool. Immediately upon arrival, jobs are assigned to a server pool. This could be based on (partial) knowledge of their type, but such knowledge might not be available. Information about the job type can however be obtained while the job is in service; as the service progresses, the likelihood that the service speed of this job type is low increases, creati…
A fundamental challenge in large-scale networked systems viz., data centers and cloud networks is to distribute tasks to a pool of servers, using minimal instantaneous state information, while providing excellent delay performance. In this thesis we design and analyze load balancing algorithms that aim to achieve a highly efficient distribution of tasks, optimize server utilization, and minimize communication overhead.
The analysis of large-scale, parallel-server load balancing systems has relied heavily on mean-field analysis. A pivotal assumption for this framework is that the servers are exchangeable. However, modern data-centers have data locality constraints, where tasks of a particular type can only be routed to a small subset of servers. An emerging line of research, therefore, considers load balancing algorithms on bipartite graphs where vertices in the two partitions represent the task types and servers, respectively, and an edge represents the server's ability to process the corresponding task type…
The model is a service system, consisting of several large server pools. A server processing speed and buffer size (which may be finite or infinite) depend on the pool. The input flow of customers is split equally among a fixed number of routers, which must assign customers to the servers immediately upon arrival. We consider an asymptotic regime in which the customer total arrival rate and pool sizes scale to infinity simultaneously, in proportion to a scaling parameter $n$, while the number of routers remains fixed. We define and study a multi-router generalization of the pull-based customer…
In large storage systems, files are often coded across several servers to improve reliability and retrieval speed. We consider a system of $n$ servers storing files using a Maximum Distance Separable code (cf. \cite{li2016mean}). Specifically, each file is stored in equally sized pieces across $L$ servers such that any $k$ pieces can reconstruct the original file. File requests are routed using the Batch Sampling routing scheme. I.e. when a request for a file is received, a centralized dispatcher routes the job into the $k$-shortest queues among the $L$ for which the corresponding server conta…
In this note, we apply Stein's method to analyze the performance of general load balancing schemes in the many-server heavy-traffic regime. In particular, consider a load balancing system of $N$ servers and the distance of arrival rate to the capacity region is given by $N^{1-α}$ with $α> 1$. We are interested in the performance as $N$ goes to infinity under a large class of policies. We establish different asymptotics under different scalings and conditions. Specifically, (i) If the second moments linearly increase with $N$ with coefficients $σ_a^2$ and $ν_s^2$, then for any $α> 4$, the distr…
We analyze randomized dynamic load balancing schemes for multi-server processor sharing systems when the number of servers in the system is large and the servers have heterogeneous service rates. In particular, we focus on the classical power-of-two load balancing scheme and a variant of it in which a newly arrived job is assigned to the server having the least instantaneous Lagrange shadow cost among two randomly chosen servers. The instantaneous Lagrange shadow cost at a server is given by the ratio of the number of unfinished jobs at the server to the capacity of the server. Two different a…
The Persistent-Idle (PI) load distribution policy was recently introduced as an appealing alternative to current low-communication load balancing techniques. In PI, servers only update the dispatcher when they become idle, and the dispatcher always sends jobs to the last server that reported being idle. PI is unique in that it does not seek to push the server queue lengths towards equalization greedily. Rather, it aggressively pulls the servers away from starvation. As a result, PI's analysis requires different tools than other load balancing approaches. So far, PI was proven to achieve the st…
We consider large-scale service systems with multiple customer classes and multiple server pools; interarrival and service times are exponentially distributed, and mean service times depend both on the customer class and server pool. It is assumed that the allowed activities (routing choices) form a tree (in the graph with vertices being both customer classes and server pools). We study the behavior of the system under a Leaf Activity Priority (LAP) policy, which assigns static priorities to the activities in the order of sequential "elimination" of the tree leaves. We consider the scaling lim…
We consider a system of $N$ identical server pools and a single dispatcher where tasks arrive as a Poisson process of rate $λ(N)$. Arriving tasks cannot be queued, and must immediately be assigned to one of the server pools to start execution, or discarded. The execution times are assumed to be exponentially distributed with unit mean, and do not depend on the number of other tasks receiving service. However, the experienced performance (e.g. in terms of received throughput) does degrade with an increasing number of concurrent tasks at the same server pool. The dispatcher therefore aims to eve…
The model is motivated by the problem of load distribution in large-scale cloud-based data processing systems. We consider a heterogeneous service system, consisting of multiple large server pools. The pools are different in that their servers may have different processing speed and/or different buffer sizes (which may be finite or infinite). We study an asymptotic regime in which the customer arrival rate and pool sizes scale to infinity simultaneously, in proportion to some scaling parameter $n$. Arriving customers are assigned to the servers by a "router", according to a {\em pull-based} al…
The basic load balancing scenario involves a single dispatcher where tasks arrive that must immediately be forwarded to one of $N$ single-server queues. We discuss recent advances on scalable load balancing schemes which provide favorable delay performance when $N$ grows large, and yet only require minimal implementation overhead. Join-the-Shortest-Queue (JSQ) yields vanishing delays as $N$ grows large, as in a centralized queueing arrangement, but involves a prohibitive communication burden. In contrast, power-of-$d$ or JSQ($d$) schemes that assign an incoming task to a server with the shorte…
We consider the following distributed service model: jobs with unit mean, general distribution, and independent processing times arrive as a renewal process of rate $λn$, with $0<λ<1$, and are immediately dispatched to one of several queues associated with $n$ identical servers with unit processing rate. We assume that the dispatching decisions are made by a central dispatcher endowed with a finite memory, and with the ability to exchange messages with the servers. We study the fundamental resource requirements (memory bits and message exchange rate), in order to drive the expected queueing de…
Load balancing algorithms play a crucial role in delivering robust application performance in data centers and cloud networks. Recently, strong interest has emerged in Join-the-Idle-Queue (JIQ) algorithms, which rely on tokens issued by idle servers in dispatching tasks and outperform power-of-$d$ policies. Specifically, JIQ strategies involve minimal information exchange, and yet achieve zero blocking and wait in the many-server limit. The latter property prevails in a multiple-dispatcher scenario when the loads are strictly equal among dispatchers. For various reasons it is not uncommon howe…
A large-scale service system with multiple customer classes and multiple server pools is considered, with the mean service time depending both on the customer class and server pool. The allowed activities (routing choices) form a tree (in the graph with vertices being both customer classes and server pools). We study the behavior of the system under a {\em Leaf Activity Priority} (LAP) policy, introduced in [17]. An asymptotic regime is considered, where the arrival rate of customers and number of servers in each pool tend to infinity in proportion to a scaling parameter $r$, while the overall…
We study large-scale systems operating under the JSQ$(d)$ policy in the presence of stringent task-server compatibility constraints. Consider a system with $N$ identical single-server queues and $M(N)$ task types, where each server is able to process only a small subset of possible task types. Each arriving task selects $d\geq 2$ random servers compatible to its type, and joins the shortest queue among them. The compatibility constraint is naturally captured by a fixed bipartite graph $G_N$ between the servers and the task types. When $G_N$ is complete bipartite, the meanfield approximation is…
Load balancing in parallel infinite-server queues with action delay via phase representation
Graduate School of Science and Technology, University of Tsukuba（筑波大学理工学研究科）；Institute of Systems and Information Engineering, University of Tsukuba（筑波大学系统信息工学研究院）
Load balancing plays a critical role in efficiently dispatching jobs in parallel-server systems such as cloud networks and data centers. A fundamental challenge in the design of load balancing algorithms is to achieve an optimal trade-off between delay performance and implementation overhead (e.g. communication or memory usage). This trade-off has primarily been studied so far from the angle of the amount of overhead required to achieve asymptotically optimal performance, particularly vanishing delay in large-scale systems. In contrast, in the present paper, we focus on an arbitrarily sparse c…
In this thesis, we study the queueing systems with heterogeneous servers and service rate uncertainty under the Halfin-Whitt heavy traffic regime. First, we analyse many server queues with abandonments when service rates are i.i.d. random variables. We derive a diffusion approximation using a novel method. The diffusion has a random drift, and hence depending on the realisations of service rates, the system can be in Quality Driven (QD), Efficiency Driven (ED) or Quality-Efficiency-Driven (QED) regime. When the system is under QD or QED regime, the abandonments are negligible in the fluid limi…
We consider the load balancing problem in large-scale heterogeneous systems with multiple dispatchers. We introduce a general framework called Local-Estimation-Driven (LED). Under this framework, each dispatcher keeps local (possibly outdated) estimates of queue lengths for all the servers, and the dispatching decision is made purely based on these local estimates. The local estimates are updated via infrequent communications between dispatchers and servers. We derive sufficient conditions for LED policies to achieve throughput optimality and delay optimality in heavy-traffic, respectively. Th…
In previous papers we developed a deterministic fluid approximation for an overloaded Markovian queueing system having two customer classes and two service pools, known in the call-center literature as the X model. The system uses the fixed-queue-ratio-with-thresholds (FQR-T) control, which we proposed in a recent paper as a way for one service system to help another in face of an unexpected overload. Under FQR-T, customers are served by their own service pool until a threshold is exceeded. Then, one-way sharing is activated with customers from one class allowed to be served in both pools. The…
Randomized load-balancing algorithms play an important role in improving performance in large-scale networks at relatively low computational cost. A common model of such a system is a network of $N$ parallel queues in which incoming jobs with independent and identically distributed service times are routed on arrival using the join-the-shortest-of-$d$-queues routing algorithm. Under fairly general conditions, it was shown by Aghajani and Ramanan that as $N\rightarrow\infty$, the state dynamics converges to the unique solution of a countable system of coupled deterministic measure-valued equati…
Mean field models are a popular tool used to analyse load balancing policies. In some cases the waiting time distribution of the mean field limit has an explicit form. In other cases it can be computed as the solution of a set of differential equations. Here we study the limit of the mean waiting time $E[W_λ]$ as the arrival rate $λ$ approaches $1$ for a number of load balancing policies when job sizes are exponential with mean $1$ (i.e. the system gets close to instability). As $E[W_λ]$ diverges to infinity, we scale with $-\log(1-λ)$ and present a method to compute the limit $\lim_{λ\rightar…
Load balancing algorithms play a vital role in enhancing performance in data centers and cloud networks. Due to the massive size of these systems, scalability challenges, and especially the communication overhead associated with load balancing mechanisms, have emerged as major concerns. Motivated by these issues, we introduce and analyze a novel class of load balancing schemes where the various servers provide occasional queue updates to guide the load assignment. We show that the proposed schemes strongly outperform JSQ($d$) strategies with comparable communication overhead per job, and can a…
We present an overview of scalable load balancing algorithms which provide favorable delay performance in large-scale systems, and yet only require minimal implementation overhead. Aimed at a broad audience, the paper starts with an introduction to the basic load balancing scenario, consisting of a single dispatcher where tasks arrive that must immediately be forwarded to one of $N$ single-server queues. A popular class of load balancing algorithms are so-called power-of-$d$ or JSQ($d$) policies, where an incoming task is assigned to a server with the shortest queue among $d$ servers selected …
We consider a system of $N$ servers inter-connected by some underlying graph topology $G_N$. Tasks arrive at the various servers as independent Poisson processes of rate $λ$. Each incoming task is irrevocably assigned to whichever server has the smallest number of tasks among the one where it appears and its neighbors in $G_N$. Tasks have unit-mean exponential service times and leave the system upon service completion. The above model has been extensively investigated in the case $G_N$ is a clique. Since the servers are exchangeable in that case, the queue length process is quite tractable, an…
