We study multiclass many-server queues for which the arrival, service and abandonment rates are all modulated by a common finite-state Markov process. We assume that the system operates in the "averaged" Halfin-Whitt regime, which means that it is critically loaded in the average sense, although not necessarily in each state of the Markov process. We show that under any static priority policy, the Markov-modulated diffusion-scaled queueing process is geometrically ergodic. This is accomplished by employing a solution to an associated Poisson equation in order to construct a suitable Lyapunov function. We establish a functional central limit theorem for the diffusion-scaled queueing process and show that the limiting process is a controlled diffusion with piecewise linear drift and constant covariance matrix. We address the infinite-horizon discounted and long-run average (ergodic) optimal control problems and establish asymptotic optimality.
In this paper, we present counter-intuitive examples for the multiclass queueing network system. In the system, each station may serve more than one job class with differentiated service priority, and each job may require service sequentially by more than one service station. In our examples, the network performance is improved even when more workloads are admitted for service.
We consider a multidimentional Brownian control problem (BCP) with model uncertainty that formally emerges from a multiclass M/M/1 queueing control problem under heavy-traffic with model uncertainty. The BCP is formulated as a multidimensional stochastic differential game with two players: a minimizer that has an equivalent role to the decision maker in the queueing control problem and a maximizer whose role is to set up the uncertainty of the model. The dynamics are driven by a Brownian motion. We show that a state-space collapse propery holds. That is, the multidimensional BCP can be reduced…
AM/M/N+Mqueueingnetworkisconsideredwithdindependentcustomerclasses and d server pools in Halfin-Whitt regime. Class i customers has priority for service in pool i for i = 1, . . . , d, and may access some other pool if the pool has an idle server and all the servers in pool i are busy. We formulate an ergodic control problem where the running cost is given by a non- negative convex function with polynomial growth. We show that the limiting controlled diffusion is modeled by an action space which depends on the state variable. We provide a complete analysis for the limiting ergodic control prob…
In this paper, we study optimal control problems for multiclass GI/M/n+M queues in an alternating renewal (up-down) random environment in the Halfin-Whitt regime. Assuming that the downtimes are asymptotically negligible and only the service processes are affected, we show that the limits of the diffusion-scaled state processes under non-anticipative, preemptive, work-conserving scheduling policies, are controlled jump diffusions driven by a compound Poisson jump process. We establish the asymptotic optimality of the infinite-horizon discounted and long-run average (ergodic) problems for the q…
Tail asymptotics of the solution $R$ to a fixpoint problem of type $R =_{st} Q + \sum_1^N R_m$ is derived under heavy-tailed conditions allowing both dependence between $Q$ and $N$ and the tails to be of the same order of magnitude. Similar results are derived for a $K$-class version with applications to multitype branching processes and busy periods in multiclass queues.
This paper studies the heavy-traffic asymptotics for the multiclass FIFO M${}^X$/G/1 queue. We first derive the probability generating function of the joint queue length distribution. Using the probability generating function, we then present heavy-traffic asymptotic formulas for the joint queue length distribution and its joint moments (i.e., the joint queue length moments). These formulas are proved under weaker conditions on the service time distributions, compared to the ones reported in the literature. This fact leads us to conjectures that some of the conditions made in the literature ar…
A multi-class single-server system with general service time distributions is studied in a moderate deviation heavy traffic regime. In the scaling limit, an optimal control problem associated with the model is shown to be governed by a differential game that can be explicitly solved. While the characterization of the limit by a differential game is akin to results at the large deviation scale, the analysis of the problem is closely related to the much studied area of control in heavy traffic at the diffusion scale.
We consider the stability of robust scheduling policies for multiclass queueing networks. These are open networks with arbitrary routing matrix and several disjoint groups of queues in which at most one queue can be served at a time. The arrival and potential service processes and routing decisions at the queues are independent, stationary and ergodic. A scheduling policy is called robust if it does not depend on the arrival and service rates nor on the routing probabilities. A policy is called throughput-optimal if it makes the system stable whenever the parameters are such that the system ca…
We consider multiclass feedforward queueing networks with first in first out and priority service disciplines at the nodes, and class dependent deterministic routing between nodes. The random behavior of the network is constructed from cumulative arrival and service time processes which are assumed to satisfy an appropriate sample path large deviation principle. We establish logarithmic asymptotics of large deviations for waiting time, idle time, queue length, departure and sojourn-time processes in critical loading. This transfers similar results from Puhalskii about single class queueing net…
A classical result for the steady-state queue-length distribution of single-class queueing systems is the following: the distribution of the queue length just before an arrival epoch equals the distribution of the queue length just after a departure epoch. The constraint for this result to be valid is that arrivals, and also service completions, with probability one occur individually, i.e., not in batches. We show that it is easy to write down somewhat similar balance equations for {\em multidimensional} queue-length processes for a quite general network of multiclass multiserver queues. We f…
This paper addresses the question when the underlying Markov process of a multiclass queueing network is positive Harris recurrent. It is well-known that stability of the fluid limit model is a sufficient condition for this. Hence, stability of fluid (limit) models is of vital interest. Recently, it has been shown that if the fluid model satisfies certain properties, it is stable if and only if there exists a Lyapunov function. In this paper a new method is provided to conclude that the underlying Markov process is positive Harris recurrent if the fluid model is stable by using explicitly the …
One of the basic properties of a queueing network is stability. Roughly speaking, it is the property that the total number of jobs in the network remains bounded as a function of time. One of the key questions related to the stability issue is how to determine the exact conditions under which a given queueing network operating under a given scheduling policy remains stable. While there was much initial progress in addressing this question, most of the results obtained were partial at best and so the complete characterization of stable queueing networks is still lacking. In this paper, we resol…
For a multiclass G/G/1 queue with finite buffers, admission and scheduling control, and holding and rejection costs, we construct a policy that is asymptotically optimal in the heavy traffic limit. The policy is specified in terms of a single parameter which constitutes the free boundary point from the Harrison-Taksar free boundary problem, but otherwise depends "explicitly" on the problem data. The c mu priority rule is also used by the policy, but in a way that is novel, and, in particular, different than that used in problems with infinite buffers. We also address an analogous problem where…
We introduce a multiclass single-server queueing system in which the arrival rates depend on the current job in service. The system is characterized by a matrix of arrival rates in lieu of a vector of arrival rates. Our proposed model departs from existing state-dependent queueing models in which the parameters depend primarily on the number of jobs in the system rather than on the job in service. We formulate the queueing model and its corresponding fluid model and proceed to obtain the necessary and sufficient conditions for stability via fluid models. Utilizing the natural connection with t…
We study a differential game that governs the moderate-deviation heavy-traffic asymptotics of a multiclass single-server queueing control problem with a risk-sensitive cost. We consider a cost set on a finite but sufficiently large time horizon, and show that this formulation leads to stationary feedback policies for the game. Several aspects of the game are explored, including its characterization via a (one-dimensional) free boundary problem, the semi-explicit solution of an optimal strategy, and the specification of a saddle point. We emphasize the analogy to the well-known Harrison-Taksar …
This paper studies a scheduling control problem for a single-server multiclass queueing network in heavy traffic, operating in a changing environment. The changing environment is modeled as a finite state Markov process that modulates the arrival and service rates in the system. Various cases are considered: fast changing environment, fixed environment and slow changing environment. In each of the cases, using weak convergence analysis, in particular functional limit theorems for renewal processes and ergodic Markov processes, it is shown that an appropriate "averaged" version of the classical…
One of the key performance measures in queueing systems is the exponential decay rate of the steady-state tail probabilities of the queue lengths. It is known that if a corresponding fluid model is stable and the stochastic primitives have finite moments, then the queue lengths also have finite moments, so that the tail probability \pr(\cdot >s) decays faster than s^{-n} for any n. It is natural to conjecture that the decay rate is in fact exponential. In this paper an example is constructed to demonstrate that this conjecture is false. For a specific stationary policy applied to a network wit…
We study a multiclass M/M/1 queueing control problem with finite buffers under heavy-traffic where the decision maker is uncertain about the rates of arrivals and service of the system and by scheduling and admission/rejection decisions acts to minimize a discounted cost that accounts for the uncertainty. The main result is the asymptotic optimality of a $cμ$-type of policy derived via underlying stochastic differential games studied in [16]. Under this policy, with high probability, rejections are not performed when the workload lies below some cut-off that depends on the ambiguity level. Whe…
A multi-class single-server queueing model with finite buffers, in which scheduling and admission of customers are subject to control, is studied in the moderate deviation heavy traffic regime. A risk-sensitive cost set over a finite time horizon $[0,T]$ is considered. The main result is the asymptotic optimality of a control policy derived via an underlying differential game. The result is the first to address a queueing control problem at the moderate deviation regime that goes beyond models having the so called pathwise minimality property. Moreover, despite the well known fact that an opti…
We consider an overloaded multiclass nonidling first-in-first-out single-server queue with abandonment. The interarrival times, service times, and deadline times are sequences of independent and identically, but generally distributed random variables. In prior work, Jennings and Reed studied the workload process associated with this queue. Under mild conditions, they establish both a functional law of large numbers and a functional central limit theorem for this process. We build on that work here. For this, we consider a more detailed description of the system state given by $K$ finite, nonne…
We study the ergodic properties of a class of controlled stochastic differential equations (SDEs) driven by $α$-stable processes which arise as the limiting equations of multiclass queueing models in the Halfin-Whitt regime that have heavy-tailed arrival processes. When the safety staffing parameter is positive, we show that the SDEs are uniformly ergodic and enjoy a polynomial rate of convergence to the invariant probability measure in total variation, which is uniform over all stationary Markov controls resulting in a locally Lipschitz continuous drift. We also derive a matching lower bound …
We study the ergodic properties of a class of multidimensional piecewise Ornstein-Uhlenbeck processes with jumps, which contains the limit of the queueing processes arising in multiclass many-server queues with heavy-tailed arrivals and/or asymptotically negligible service interruptions in the Halfin-Whitt regime as special cases. In these queueing models, the Itô equations have a piecewise linear drift, and are driven by either (1) a Brownian motion and a pure-jump Lévy process, or (2) an anisotropic Lévy process with independent one-dimensional symmetric $α$-stable components, or (3) an anis…
We consider the discrete Hammersley-Aldous-Diaconis process (HAD) and the totally asymmetric simple exclusion process (TASEP) in Z. The basic coupling induces a multiclass process which is useful in discussing shock measures and other important properties of the processes. The invariant measures of the multiclass systems are the same for both processes, and can be constructed as the law of the output process of a system of multiclass queues in tandem; the arrival and service processes of the queueing system are a collection of independent Bernoulli product measures. The proof of invariance inv…
In the Hammersley-Aldous-Diaconis process infinitely many particles sit in R and at most one particle is allowed at each position. A particle at x$ whose nearest neighbor to the right is at y, jumps at rate y-x to a position uniformly distributed in the interval (x,y). The basic coupling between trajectories with different initial configuration induces a process with different classes of particles. We show that the invariant measures for the two-class process can be obtained as follows. First, a stationary M/M/1 queue is constructed as a function of two homogeneous Poisson processes, the arriv…
The Multi-class Queueing Network (McQN) arises as a natural multi-class extension of the traditional (single-class) Jackson network. In a single-class network subcriticality (i.e. subunitary nominal workload at every station) entails stability, but this is no longer sufficient when jobs/customers of different classes (i.e. with different service requirements and/or routing scheme) visit the same server; therefore, analytical conditions for stability of McQNs are lacking, in general. In this note we design a numerical (simulation-based) method for determining the stability region of a McQN, in …
We consider a heterogeneous queueing system consisting of one large pool of $O(r)$ identical servers, where $r\to\infty$ is the scaling parameter. The arriving customers belong to one of several classes which determines the service times in the distributional sense. The system is heavily loaded in the Halfin-Whitt sense, namely the nominal utilization is $1-a/\sqrt{r}$ where $a>0$ is the spare capacity parameter. Our goal is to obtain bounds on the steady state performance metrics such as the number of customers waiting in the queue $Q^r(\infty)$. While there is a rich literature on deriving p…
We analyze asymptotically a differential-difference equation, that arises in a Markov-modulated fluid model. We use singular perturbation methods to analyze the problem with appropriate scalings of the two state variables. In particular, the ray method and asymptotic matching are used.
Ramaswami showed recently that standard Brownian motion arises as the limit of a family of Markov-modulated linear fluid processes. We pursue this analysis with a fluid approximation for Markov-modulated Brownian motion. Furthermore, we prove that the stationary distribution of a Markov-modulated Brownian motion reflected at zero is the limit from the well-analyzed stationary distribution of approximating linear fluid processes. Key matrices in the limiting stationary distribution are shown to be solutions of a new quadratic equation, and we describe how this equation can be efficiently solved…
This paper considers a BMAP/M/$\infty$ queue with a batch Markovian arrival process (BMAP) and an exponential service time distribution. We first prove that the BMAP/M/$\infty$ queue is stable if and only if the expectation of the logarithm of the batch-size distribution is finite. Using this result, we also present the stability condition for an infinite-server queue with a multiclass batch Markovian arrival process and class-dependent exponential service times.
This paper deals with a single-server queue with modulated arrivals, service requirements and service capacity. In our first result, we derive the mean of the total workload assuming generally distributed service requirements and any service discipline which does not depend on the modulating environment. We then show that the workload is exponentially distributed under heavy-traffic scaling. In our second result, we focus on the discriminatory processor sharing (DPS) discipline. Assuming exponential, class-dependent service requirements, we show that the joint distribution of the queue lengths…
This paper studies the diffusion limit for a network of infinite-server queues operating under Markov modulation (meaning that the system's parameters depend on an autonomously evolving background process). In previous papers on (primarily single-node) queues with Markov modulation, two variants were distinguished: one in which the server speed is modulated, and one in which the service requirement is modulated (i.e., depends on the state of the background process upon arrival). The setup of the present paper, however, is more general, as we allow both the server speed and the service requirem…
We consider the serve-the-longest-queue discipline for a multiclass queue with buffers of equal size, operating under (i) the conventional and (ii) the Halfin-Whitt heavy traffic regimes, and show that while the queue length process' scaling limits are fully determined by the first and second order data in case (i), they depend on finer properties in case (ii). The proof of the latter relies on the construction of a {\it deterministic} arrival pattern.
The production of molecules in a chemical reaction network is modelled as a Poisson process with a Markov-modulated arrival rate and an exponential decay rate. We analyze the distributional properties of $M$, the number of molecules, under specific time-scaling; the background process is sped up by $N^α$, the arrival rates are scaled by $N$, for $N$ large. A functional central limit theorem is derived for $M$, which after centering and scaling, converges to an Ornstein-Uhlenbeck process. A dichotomy depending on $α$ is observed. For $α\leq1$ the parameters of the limiting process contain the d…
In this paper we study a reflected Markov-modulated Brownian motion with a two sided reflection in which the drift, diffusion coefficient and the two boundaries are (jointly) modulated by a finite state space irreducible continuous time Markov chain. The goal is to compute the stationary distribution of this Markov process, which in addition to the complication of having a stochastic boundary can also include jumps at state change epochs of the underlying Markov chain because of the boundary changes. We give the general theory and then specialize to the case where the underlying Markov chain h…
This paper focuses on an infinite-server queue modulated by an independently evolving finite-state Markovian background process, with transition rate matrix $Q\equiv(q_{ij})_{i,j=1}^d$. Both arrival rates and service rates are depending on the state of the background process. The main contribution concerns the derivation of central limit theorems for the number of customers in the system at time $t\ge 0$, in the asymptotic regime in which the arrival rates $λ_i$ are scaled by a factor $N$, and the transition rates $q_{ij}$ by a factor $N^α$, with $α\in \mathbb R^+$. The specific value of $α$ h…
In this work, we focus on the stationary analysis of a specific class of continuous time Markov-modulated reflected random walks in the quarter plane with applications in the modelling of two-node Markov-modulated queueing networks with coupled queues. The transition rates of the two-dimensional process depend on the state of a finite state Markovian background process. Such a modulation is space homogeneous in the set of inner states of the two-dimensional lattice but may be different in the set of states at its boundaries. To obtain the stationary distribution, we apply the power series appr…
This paper considers a parallel system of queues fed by independent arrival streams, where the service rate of each queue depends on the number of customers in all of the queues. Necessary and sufficient conditions for the stability of the system are derived, based on stochastic monotonicity and marginal drift properties of multiclass birth and death processes. These conditions yield a sharp characterization of stability for systems, where the service rate of each queue is decreasing in the number of customers in other queues, and has uniform limits as the queue lengths tend to infinity. The r…
Multiclass open queueing networks find wide applications in communication, computer and fabrication networks. Often one is interested in steady-state performance measures associated with these networks. Conceptually, under mild conditions, a regenerative structure exists in multiclass networks, making them amenable to regenerative simulation for estimating the steady-state performance measures. However, typically, identification of a regenerative structure in these networks is difficult. A well known exception is when all the interarrival times are exponentially distributed, where the instants…
This paper examines a discrete-time queuing system with applications to telecommunications traffic. The arrival process is a particular Markov modulated process which belongs to the class of discrete batched Markovian arrival processes. The server process is a single server deterministic queue. A closed form exact solution is given for the expected queue length and delay. A simple system of equations is given for the probability of the queue exceeding a given length.
Let $\{\boldsymbol{X}_n\}$ be a discrete-time $d$-dimensional process on $\mathbb{Z}_+^d$ with a supplemental (background) process $\{J_n\}$ on a finite set and assume the joint process $\{\boldsymbol{Y}_n\}=\{(\boldsymbol{X}_n,J_n)\}$ to be Markovian. Then, the process $\{\boldsymbol{X}_n\}$ can be regarded as a kind of reflecting random walk (RRW for short) in which the transition probabilities of the RRW are modulated according to the state of the background process $\{J_n\}$; we assume this modulation is space-homogeneous inside $\mathbb{Z}_+^d$ and on each boundary face of $\mathbb{Z}_+^d…
This paper studies an infinite-server queue in a random environment, meaning that the arrival rate, the service requirements and the server work rate are modulated by a general càdlàg stochastic background process. To prove a large deviations principle, the concept of attainable parameters is introduced. Scaling both the arrival rates and the background process, a large deviations principle for the number of jobs in the system is derived using attainable parameters. Finally, some known results about Markov-modulated infinite-server queues are generalized and new results for several background …
We analyze the stationary distribution of regulated Markov modulated Brownian motions (MMBM) modified so that their evolution is slowed down when the process reaches level zero --- level zero is said to be {\em sticky}. To determine the stationary distribution, we extend to MMBMs a construction of Brownian motion with sticky boundary, and we follow a Markov-regenerative approach similar to the one developed in past years in the context of quasi-birth-and-death processes and fluid queues. We also rely on recent work showing that Markov-modulated Brownian motions may be analyzed as limits of a p…
Power law distributions have been repeatedly observed in a wide variety of socioeconomic, biological and technological areas. In many of the observations, e.g., city populations and sizes of living organisms, the objects of interest evolve due to the replication of their many independent components, e.g., births-deaths of individuals and replications of cells. Furthermore, the rates of the replication are often controlled by exogenous parameters causing periods of expansion and contraction, e.g., baby booms and busts, economic booms and recessions, etc. In addition, the sizes of these objects …
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Cornell University（康奈尔大学）
Novel advanced policy gradient (APG) methods, such as Trust Region policy optimization and Proximal policy optimization (PPO), have become the dominant reinforcement learning algorithms because of their ease of implementation and good practical performance. A conventional setup for notoriously difficult queueing network control problems is a Markov decision problem (MDP) that has three features: infinite state space, unbounded costs, and long-run average cost objective. We extend the theoretical framework of these APG methods for such MDP problems. The resulting PPO algorithm is tested on a pa…
We consider an infinite-server queue into which customers arrive according to a Cox process and have independent service times with a general distribution. We prove a functional large deviations principle for the equilibrium queue length process. The model is motivated by a linear feed-forward gene regulatory network, in which the rate of protein synthesis is modulated by the number of RNA molecules present in a cell. The system can be modelled as a tandem of infinite-server queues, in which the number of customers present in a queue modulates the arrival rate into the next queue in the tandem…
Exact analysis of transient behavior of finite-capacity MAP-driven queues
Mathematical Institute, Leiden University（莱顿大学数学研究所）；Eurandom, Eindhoven University of Technology（埃因霍温理工大学Eurandom）；Amsterdam Business School, Faculty of Economics and Business, University of Amsterdam（阿姆斯特丹大学经济商学院阿姆斯特丹商学院）；Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（特文特大学电气工程学院、数学与计算机科学系）
This paper develops tools to obtain robust probabilistic estimates for queueing models at the large deviations (LD) scale. These tools are based on the recently introduced robust Rényi bounds, which provide LD estimates (and more generally risk-sensitive (RS) cost estimates) that hold uniformly over an uncertainty class of models, provided that the class is defined in terms of Rényi divergence with respect to a reference model and that estimates are available for the reference model. One very attractive quality of the approach is that the class to which the estimates apply may consist of hard …
