The fundamental problem in the study of parallel-server systems is that of finding and analyzing `good' routing policies of arriving jobs to the servers. It is well known that, if full information regarding the workload process is available to a central dispatcher, then the {\em join the shortest workload} (JSW) policy, which assigns jobs to the server with the least workload, is the optimal assignment policy, in that it maximizes server utilization, and thus minimizes sojourn times. The {\em join the shortest queue} (JSQ) policy is an efficient dispatching policy when information is available only on the number of jobs with each of the servers, but not on their service requirements. If information on the state of the system is not available, other dispatching policies need to be employed, such as the power-of-$d$ routing policy, in which each arriving job joins the shortest among $d \ge 1$ queues sampled uniformly at random. (Under this latter policy, the system is known as {\em the supermarket model}.) In this paper we study the stability question of parallel server systems assuming that routing errors occur, so that arrivals may be routed to the `wrong' (not to the smallest) queue with a positive probability. We show that, even if a `non-idling' dispatching policy is employed, under which new arrivals are always routed to an idle server, if any is available, the performance of the system can be much worse than under the policy that chooses one of the servers uniformly at random. More specifically, we prove that the usual traffic intensity $ρ< 1$ does not guarantee that the system is stable.
Motivated by the growing interest in today's massive parallel computing capabilities we analyze a queueing network with many servers in parallel to which jobs arrive a according to a Poisson process. Each job, upon arrival, is split into several pieces which are randomly routed to specific servers in the network, without centralized information about the status of the servers' individual queues. The main feature of this system is that the different pieces of a job must initiate their service in a synchronized fashion. Moreover, the system operates in a FCFS basis. The synchronization and servi…
We consider the problem of minimizing queue-length costs in a system with heterogenous parallel servers, operating in a many-server heavy-traffic regime with nondegenerate slowdown. This regime is distinct from the well-studied heavy traffic diffusion regimes, namely the (single server) conventional regime and the (many-server) Halfin-Whitt regime. It has the distinguishing property that waiting times and service times are of comparable magnitudes. We establish an asymptotic lower bound on the cost and devise a sequence of policies that asymptotically attain this bound. As in the conventional …
We investigate, under general stationary ergodic assumptions, the stability of systems of $S$ parallel queues in which any incoming customer joins the queue of the server having the $p+1$-th shortest workload ($p < S$), or a free server if any. This change in the allocation policy makes the analysis much more challenging with respect to the classical FCFS model with $S$ servers, as it leads to the non-monotonicity of the underlying stochastic recursion. We provide sufficient conditions of existence of a stationary workload, which indicate a "splitting" of the system in heavy traffic, into a lo…
We consider Poisson streams of exponentially distributed jobs arriving at each edge of a hypergraph of queues. Upon arrival, an incoming job is rooted to the shortest queue among the corresponding vertices. This generalizes many known models such as power-of-d load balancing and JSQ (join the shortest queue) on generic graphs. We provide a generic condition for stability of this model. We show that some graph topologies lead to a loss of capacity, implying more restrictive stability conditions than in, e.g., complete graphs.
The model is a service system, consisting of several large server pools. A server processing speed and buffer size (which may be finite or infinite) depend on the pool. The input flow of customers is split equally among a fixed number of routers, which must assign customers to the servers immediately upon arrival. We consider an asymptotic regime in which the customer total arrival rate and pool sizes scale to infinity simultaneously, in proportion to a scaling parameter $n$, while the number of routers remains fixed. We define and study a multi-router generalization of the pull-based customer…
We consider three parallel service models in which customers of several types are served by several types of servers subject to a bipartite compatibility graph, and the service policy is first come first served. Two of the models have a fixed set of servers. The first is a queueing model in which arriving customers are assigned to the longest idling compatible server if available, or else queue up in a single queue, and servers that become available pick the longest waiting compatible customer, as studied by Adan and Weiss, 2014. The second is a redundancy service model where arriving customer…
We study a parallel queueing system with multiple types of servers and customers. A bipartite graph describes which pairs of customer-server types are compatible. We consider the service policy that always assigns servers to the first, longest waiting compatible customer, and that always assigns customers to the longest idle compatible server if on arrival, multiple compatible servers are available. For a general renewal stream of arriving customers and general service time distributions, the behavior of such systems is very complicated. In particular, the calculation of matching rates, the fr…
We consider a system of N queues with decentralized load balancing such as power-of-D strategies(where D may depend on N) and generic scheduling disciplines. To measure the dependence of the queues, we use the clan of ancestors, a technique coming from interacting particle systems. Relying in that analysis we prove quantitative estimates on the queues correlations implying propagation of chaos for systems with Markovian arrivals and general service time distribution. This solves the conjecture posed by Bramsom et. al. in [*] concerning the asymptotic independence of the servers in the case of …
A broad class of parallel server systems is considered, for which we prove the steady-state asymptotic independence of server workloads, as the number of servers goes to infinity, while the system load remains sub-critical. Arriving jobs consist of multiple components. There are multiple job classes, and each class may be of one of two types, which determines the rule according to which the job components add workloads to the servers. The model is broad enough to include as special cases some popular queueing models with redundancy, such as cancel-on-start and cancel-on-completion redundancy. …
We study a system, where a random flow of customers is served by servers (called agents) invited on-demand. Each invited agent arrives into the system after a random time; after each service completion, an agent returns to the system or leaves it with some fixed probabilities. Customers and/or agents may be impatient, that is, while waiting in queue, they leave the system at a certain rate (which may be zero). We consider the queue-length-based feedback scheme, which controls the number of pending agent invitations, depending on the customer and agent queue lengths and their changes. The basic…
Processing computation-intensive jobs at multiple processing cores in parallel is essential in many real-world applications. In this paper, we consider an idealised model for job parallelism in which a job can be served simultaneously by $d$ distinct servers. The job is considered complete when the total amount of work done on it by the $d$ servers equals its size. We study the effect of parallelism on the average delay of jobs. Specifically, we analyze a system consisting of $n$ parallel processor sharing servers in which jobs arrive according to a Poisson process of rate $n λ$ ($λ<1$) and ea…
We investigate the stability condition for redundancy-d systems where each of the servers follows a processor-sharing (PS) discipline. We allow for generally distributed job sizes, with possible dependence among the d replica sizes being governed by an arbitrary joint distribution. We establish that the stability condition is characterized by the expectation of the minimum of d replica sizes being less than the mean interarrival time per server. In the special case of identical replicas, the stability condition is insensitive to the job size distribution given its mean, and the stability condi…
The standard setting for studying parallel server systems (PSS) at the diffusion scale is based on the heavy traffic condition (HTC), which assumes that the underlying static allocation linear program (LP) is critical and has a unique solution. This solution determines the graph of basic activities, which identifies the set of activities (i.e., class-server pairs) that are operational. In this paper we explore the extended HTC, where the LP is merely assumed to be critical. Because multiple solutions are allowed, multiple sets of operational activities, referred to as modes, are available. For…
Consider a modified, stable, two node Jackson network where server 2 helps server 1 when server 2 is idle. The probability of a large deviation of the number of customers at node one can be calculated using the flat boundary theory of Schwartz and Weiss [Large Deviations Performance Analysis (1994), Chapman and Hall, New York]. Surprisingly, however, these calculations show that the proportion of time spent on the boundary, where server 2 is idle, may be zero. This is in sharp contrast to the unmodified Jackson network which spends a nonzero proportion of time on this boundary.
We investigate the stability condition of redundancy-$d$ multi-server systems. Each server has its own queue and implements popular scheduling disciplines such as First-Come-First-Serve (FCFS), Processor Sharing (PS), and Random Order of Service (ROS). New jobs arrive according to a Poisson process and copies of each job are sent to $d$ servers chosen uniformly at random. The service times of jobs are assumed to be exponentially distributed. A job departs as soon as one of its copies finishes service. Under the assumption that all $d$ copies are i.i.d., we show that for PS and ROS (for FCFS it…
We introduce the appropriate framework for pathwise comparison of multiple server queues under general stationary ergodic assumptions. We show in what sense it is better to have more servers for a system under FCFS ('First Come, First Served') or equivalently, more queues in a system of parallel queues under the JSW ('Join the Shortest Workload') allocation policy. This comparison result is based on the recursive representation of Kiefer and Wolfowitz, and on a non-mass conservative generalization of the Schur-Convex semi-ordering. We also show that the latter result does not hold true in gene…
We consider GI/Ph/n+M parallel-server systems with a renewal arrival process, a phase-type service time distribution, n homogenous servers, and an exponential patience time distribution with positive rate. We show that in the Halfin-Whitt regime, the sequence of stationary distributions corresponding to the normalized state processes is tight. As a consequence, we establish an interchange of heavy traffic and steady state limits for GI/Ph/n+M queues.
We consider a parallel server system with so-called cancel-on-completion redundancy. There are $n$ servers and multiple job classes $j$. An arriving class $j$ job consists of $d_j$ components, placed on a randomly selected subset of servers; the job service is complete as soon as $k_j$ components out of $d_j$ (with $k_j \le d_j$) complete their service, at which point the unfinished service of all remaining $d_j-k_j$ components is canceled. The system is in general non-work-conserving, in the sense that the average amount of new workload added to the system by an arriving class $j$ job is not …
This paper studies a 2-class, 2-server parallel server system under the recently introduced extended heavy traffic condition, which states that the underlying 'static allocation' linear program (LP) is critical, but does not require that it has a unique solution. The main result is the construction of policies that asymptotically achieve a lower bound, proved in [1], on an expected discounted linear combination of diffusion-scaled queue lengths, and are therefore asymptotically optimal (AO). Each extreme point solution to the LP determines a control mode, i.e., a set of activities (class--serv…
This paper studies a spatial queueing system on a circle, polled at random locations by a myopic server that can only observe customers in a bounded neighborhood. The server operates according to a greedy policy, always serving the nearest customer in its neighborhood, and leaving the system unchanged at polling instants where the neighborhood is empty. This system is modeled as a measure-valued random process, which is shown to be positive recurrent under a natural stability condition that does not depend on the server's scan range. When the interpolling times are light-tailed, the stable sys…
In this paper we present a stability criterion for finite measure-valued stochastic recursions, generalizing Loynes's Theorem to spaces of measures. This result provides conditions for the reach of a "total stationary state" for the queue with an infinity of servers and the single-server SRPT queue. Indeed, we give in both cases a condition of existence of a stationary measure-valued recursive sequence characterizing the queueing system exhaustively.
We consider the bipartite matching model of customers and servers introduced by Caldentey, Kaplan, and Weiss (Adv. Appl. Probab., 2009). Customers and servers play symmetrical roles. There is a finite set C resp. S, of customer, resp. server, classes. Time is discrete and at each time step, one customer and one server arrive in the system according to a joint probability measure on CxS, independently of the past. Also, at each time step, pairs of matched customer and server, if they exist, depart from the system. Authorized matchings are given by a fixed bipartite graph. A matching policy is c…
Infinite server queues have ultimate processing power to accommodate explosive demand surges. We provide a new stability criterion based on the Borel-Cantelli lemma to judge whether the infinite server safely accommodates heavy-tailed demands. We illustrate the battles between heavy-tailed demand and infinite servers in detail. In particular, we show some cases where the explosive demand overwhelms the infinite server queue. The medical demand caused by pandemics such as the COVID- 19 creates huge stress to the healthcare system. This framework indicates that healthcare systems need to account…
We consider a single-server system with service stations in each point of the circle. Customers arrive after exponential times at uniformly-distributed locations. The server moves at finite speed and adopts a greedy routing mechanism. It was conjectured by Coffman and Gilbert in~1987 that the service rate exceeding the arrival rate is a sufficient condition for the system to be positive recurrent, for any value of the speed. In this paper we show that the conjecture holds true.
Networks in which the processing of jobs occurs both sequentially and in parallel are prevalent in many application domains, such as computer systems, healthcare, manufacturing, and project management. The parallel processing of jobs gives rise to synchronization constraints that can be a main reason for job delay. In comparison with feedforward queueing networks that have only sequential processing of jobs, the approximation and control of networks that have synchronization constraints is less understood. One well-known modeling framework in which synchronization constraints are prominent is …
We consider a system with $N$ parallel servers where incoming jobs are immediately replicated to, say, $d$ servers. Each of the $N$ servers has its own queue and follows a FCFS discipline. As soon as the first job replica is completed, the remaining replicas are abandoned. We investigate the achievable stability region for a quite general workload model with different job types and heterogeneous servers, reflecting job-server affinity relations which may arise from data locality issues and soft compatibility constraints. Under the assumption that job types are known beforehand we show for New-…
A parallel server system is a stochastic processing network with applications in manufacturing, supply chain, ride-hailing, call centers, etc. Heterogeneous customers arrive in the system, and only a subset of servers can serve any customer type given by the flexibility graph. The goal of the system operator is to minimize the delay that depends on the scheduling policy and the flexibility graph. A long line of literature focuses on designing near-optimal scheduling policies given a flexibility graph. On the contrary, we fix the scheduling policy to be the so-called MaxWeight scheduling given …
We analyse Join-the-Shortest-Queue in a contemporary scaling regime known as the Non-Degenerate Slowdown regime. Join-the-Shortest-Queue (JSQ) is a classical load balancing policy for queueing systems with multiple parallel servers. Parallel server queueing systems are regularly analysed and dimensioned by diffusion approximations achieved in the Halfin-Whitt scaling regime. However, when jobs must be dispatched to a server upon arrival, we advocate the Non-Degenerate Slowdown regime (NDS) to compare different load-balancing rules. In this paper we identify novel diffusion approximation and ti…
We consider a single-server cyclic polling system with three queues where the server follows an adaptive rule: if it finds one of queues empty in a given cycle, it decides not to visit that queue in the next cycle. In the case of limited service policies, we prove stability and instability results under some conditions which are sufficient but not necessary, in general. Then we discuss open problems with identifying the exact stability region for models with limited service disciplines: we conjecture that a necessary and sufficient condition for the stability may depend on the whole distributi…
The Persistent-Idle (PI) load distribution policy was recently introduced as an appealing alternative to current low-communication load balancing techniques. In PI, servers only update the dispatcher when they become idle, and the dispatcher always sends jobs to the last server that reported being idle. PI is unique in that it does not seek to push the server queue lengths towards equalization greedily. Rather, it aggressively pulls the servers away from starvation. As a result, PI's analysis requires different tools than other load balancing approaches. So far, PI was proven to achieve the st…
We are concerned with an $M/M$-type join the shortest queue ($M/M$-JSQ for short) with $k$ parallel queues for an arbitrary positive integer $k$, where the servers may be heterogeneous. We are interested in the tail asymptotic of the stationary distribution of this queueing model, provided the system is stable. We prove that this asymptotic for the minimum queue length is exactly geometric, and its decay rate is the $k$-th power of the traffic intensity of the corresponding $k$ server queues with a single waiting line. For this, we use two formulations, a quasi-birth-and-death (QBD for short) …
Redundancy scheduling has emerged as a powerful strategy for improving response times in parallel-server systems. The key feature in redundancy scheduling is replication of a job upon arrival by dispatching replicas to different servers. Redundant copies are abandoned as soon as the first of these replicas finishes service. By creating multiple service opportunities, redundancy scheduling increases the chance of a fast response from a server that is quick to provide service, and mitigates the risk of a long delay incurred when a single selected server turns out to be slow. The diversity enable…
Load balancing in parallel infinite-server queues with action delay via phase representation
Graduate School of Science and Technology, University of Tsukuba（筑波大学理工学研究科）；Institute of Systems and Information Engineering, University of Tsukuba（筑波大学系统信息工学研究院）
In this paper we study the uniform stability properties of two classes of parallel server networks with multiple classes of jobs and multiple server pools of a tree topology. These include a class of networks with a single non-leaf server pool, such as the 'N' and 'M' models, and networks of any tree topology with class-dependent service rates. We show that with $\sqrt{n}$ safety staffing, and no abandonment, in the Halfin--Whitt regime, the diffusion-scaled controlled queueing processes are exponentially ergodic and their invariant probability distributions are tight, uniformly over all stati…
Policy iteration techniques for multiple-server dispatching rely on the computation of value functions. In this context, we consider the continuous-space M/G/1-FCFS queue endowed with an arbitrarily-designed cost function for the waiting times of the incoming jobs. The associated relative value function is a solution of Poisson's equation for Markov chains, which in this work we solve in the Laplace transform domain by considering an ancillary, underlying stochastic process extended to (imaginary) negative backlog states. This construction enables us to issue closed-form relative value functio…
Cloud computing today is dominated by multi-server jobs. These are jobs that request multiple servers simultaneously and hold onto all of these servers for the duration of the job. Multi-server jobs add a lot of complexity to the traditional one-job-per-server model: an arrival might not "fit" into the available servers and might have to queue, blocking later arrivals and leaving servers idle. From a queueing perspective, almost nothing is understood about multi-server job queueing systems; even understanding the exact stability region is a very hard problem. In this paper, we investigate a mu…
We consider load balancing in service systems with affinity relations between jobs and servers. Specifically, an arriving job can be allocated to a fast, primary server from a particular selection associated with this job or to a secondary server to be processed at a slower rate. Such job-server affinity relations can model network topologies based on geographical proximity, or data locality in cloud scenarios. We introduce load balancing schemes that allocate jobs to primary servers if available, and otherwise to secondary servers. A novel coupling construction is developed to obtain stabilit…
We analyze the performance of redundancy in a multi-type job and multi-type server system. We assume the job dispatcher is unaware of the servers' capacities, and we set out to study under which circumstances redundancy improves the performance. With redundancy an arriving job dispatches redundant copies to all its compatible servers, and departs as soon as one of its copies completes service. As a benchmark comparison, we take the non-redundant system in which a job arrival is routed to only one randomly selected compatible server. Service times are generally distributed and all copies of a j…
We introduce a "system-wide safety staffing" (SWSS) parameter for multiclass multi-pool networks of any tree topology, Markovian or non-Markovian, in the Halfin-Whitt regime. This parameter can be regarded as the optimal reallocation of the capacity fluctuations (positive or negative) of order $\sqrt{n}$ when each server pool employs a square-root staffing rule. We provide an explicit form of the SWSS as a function of the system parameters, which is derived using a graph theoretic approach based on Gaussian elimination. For Markovian networks, we give an equivalent characterization of the SWSS…
We construct a generic, simple, and efficient scheduling policy for stochastic processing networks, and provide a general framework to establish its stability. Our policy is randomized and prioritized: with high probability it prioritizes jobs which have been least routed through the network. We show that the network is globally stable under this policy if there exists an appropriate quadratic local Lyapunov function that provides a negative drift with respect to nominal loads at servers. Applying this generic framework, we obtain stability results for our policy in many important examples of …
We consider a heterogeneous distributed service system, consisting of $n$ servers with unknown and possibly different processing rates. Jobs with unit mean and independent processing times arrive as a renewal process of rate $λn$, with $0<λ<1$, to the system. Incoming jobs are immediately dispatched to one of several queues associated with the $n$ servers. We assume that the dispatching decisions are made by a central dispatcher endowed with a finite memory, and with the ability to exchange messages with the servers. We study the fundamental resource requirements (memory bits and message excha…
We introduce the \textit{generalized join the shortest queue model with retrials} and two infinite capacity orbit queues. Three independent Poisson streams of jobs, namely a \textit{smart}, and two \textit{dedicated} streams, flow into a single server system, which can hold at most one job. Arriving jobs that find the server occupied are routed to the orbits as follows: Blocked jobs from the \textit{smart} stream are routed to the shortest orbit queue, and in case of a tie, they choose an orbit randomly. Blocked jobs from the \textit{dedicated} streams are routed directly to their orbits. Orbi…
