We consider a class of open stochastic processing networks, with feedback routing and overlapping server capabilities, in heavy traffic. The networks we consider satisfy the so-called complete resource pooling condition and therefore have one-dimensional approximating Brownian control problems.
We propose a simple discrete review policy for controlling such networks.
Assuming 2+εmoments on the interarrival times and processing times, we provide a conceptually simple proof of asymptotic optimality of the proposed policy.
We consider a class of stochastic processing networks. Assume that the networks satisfy a complete resource pooling condition. We prove that each maximum pressure policy asymptotically minimizes the workload process in a stochastic processing network in heavy traffic. We also show that, under each quadratic holding cost structure, there is a maximum pressure policy that asymptotically minimizes the holding cost. A key to the optimality proofs is to prove a state space collapse result and a heavy traffic limit theorem for the network processes under a maximum pressure policy. We extend a framew…
This paper develops near-optimal sustainable harvesting strategies for the predator in a predator-prey system. The objective function is of long-run average per unit time type. To date, ecological systems under environmental noise are usually modeled as stochastic differential equations driven by a Brownian motion. Recognizing that the formulation using a Brownian motion is only an idealization, in this paper, it is assumed that the environment is subject to disturbances characterized by a jump process with rapid jump rates. Under broad conditions, it is shown that the systems under considerat…
Lost sales inventory models with large lead times, which arise in many practical settings, are notoriously difficult to optimize due to the curse of dimensionality. In this paper we show that when lead times are large, a very simple constant-order policy, first studied by Reiman (\cite{Reiman04}), performs nearly optimally. The main insight of our work is that when the lead time is very large, such a significant amount of randomness is injected into the system between when an order for more inventory is placed and when that order is received, that "being smart" algorithmically provides almost …
Inventory models with lost sales and large lead times have traditionally been considered intractable due to the curse of dimensionality. Recently, Goldberg and co-authors laid the foundations for a new approach to solving these models, by proving that as the lead time grows large, a simple constant-order policy is asymptotically optimal. However, the bounds proven there require the lead time to be very large before the constant-order policy becomes effective, in contrast to the good numerical performance demonstrated by Zipkin even for small lead time values. In this work, we prove that for th…
This paper studies a 2-class, 2-server parallel server system under the recently introduced extended heavy traffic condition, which states that the underlying 'static allocation' linear program (LP) is critical, but does not require that it has a unique solution. The main result is the construction of policies that asymptotically achieve a lower bound, proved in [1], on an expected discounted linear combination of diffusion-scaled queue lengths, and are therefore asymptotically optimal (AO). Each extreme point solution to the LP determines a control mode, i.e., a set of activities (class--serv…
We consider finite-horizon restless bandits with multiple pulls per period, which play an important role in recommender systems, active learning, revenue management, and many other areas. While an optimal policy can be computed, in principle, using dynamic programming, the computation required scales exponentially in the number of arms $N$. Thus, there is substantial value in understanding the performance of index policies and other policies that can be computed efficiently for large $N$. We study the growth of the optimality gap, i.e., the loss in expected performance compared to an optimal p…
We study the asymptotic optimal control of multi-class restless bandits. A restless bandit is a controllable stochastic process whose state evolution depends on whether or not the bandit is made active. Since finding the optimal control is typically intractable, we propose a class of priority policies that are proved to be asymptotically optimal under a global attractor property and a technical condition. We consider both a fixed population of bandits as well as a dynamic population where bandits can depart and arrive. As an example of a dynamic population of bandits, we analyze a multi-class …
We consider the load balancing problem in large-scale heterogeneous systems with multiple dispatchers. We introduce a general framework called Local-Estimation-Driven (LED). Under this framework, each dispatcher keeps local (possibly outdated) estimates of queue lengths for all the servers, and the dispatching decision is made purely based on these local estimates. The local estimates are updated via infrequent communications between dispatchers and servers. We derive sufficient conditions for LED policies to achieve throughput optimality and delay optimality in heavy-traffic, respectively. Th…
Dual-sourcing inventory systems, in which one supplier is faster (i.e. express) and more costly, while the other is slower (i.e. regular) and cheaper, arise naturally in many real-world supply chains. These systems are notoriously difficult to optimize due to the complex structure of the optimal solution and the curse of dimensionality, having resisted solution for over 40 years. Recently, so-called Tailored Base-Surge (TBS) policies have been proposed as a heuristic for the dual-sourcing problem. Under such a policy, a constant order is placed at the regular source in each period, while the o…
We consider a single-server queue with renewal arrivals and i.i.d. service times, in which the server employs either the preemptive Shortest Remaining Processing Time (SRPT) policy, or its non-preemptive variant, Shortest Job First (SJF). We show that for given stochastic primitives (initial condition, arrival and service processes), the model has the same fluid limit under either policy. In particular, we conclude that the well-known queue length optimality of preemptive SRPT is also achieved, asymptotically on fluid scale, by the simpler-to-implement SJF policy. We also conclude that on flui…
A multiclass queueing system is considered, with heterogeneous service stations, each consisting of many servers with identical capabilities. An optimal control problem is formulated, where the control corresponds to scheduling and routing, and the cost is a cumulative discounted functional of the system's state. We examine two versions of the problem: ``nonpreemptive,'' where service is uninterruptible, and ``preemptive,'' where service to a customer can be interrupted and then resumed, possibly at a different station. We study the problem in the asymptotic heavy traffic regime proposed by Ha…
This paper is devoted to studying the average optimality in continuous-time Markov decision processes with fairly general state and action spaces. The criterion to be maximized is expected average rewards. The transition rates of underlying continuous-time jump Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We first provide two optimality inequalities with opposed directions, and also give suitable conditions under which the existence of solutions to the two optimality inequalities is ensured. Then, from the two optimality inequaliti…
We consider the problem of minimizing queue-length costs in a system with heterogenous parallel servers, operating in a many-server heavy-traffic regime with nondegenerate slowdown. This regime is distinct from the well-studied heavy traffic diffusion regimes, namely the (single server) conventional regime and the (many-server) Halfin-Whitt regime. It has the distinguishing property that waiting times and service times are of comparable magnitudes. We establish an asymptotic lower bound on the cost and devise a sequence of policies that asymptotically attain this bound. As in the conventional …
This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/c…
We consider a queueing network in which there are constraints on which queues may be served simultaneously; such networks may be used to model input-queued switches and wireless networks. The scheduling policy for such a network specifies which queues to serve at any point in time. We consider a family of scheduling policies, related to the maximum-weight policy of Tassiulas and Ephremides [IEEE Trans. Automat. Control 37 (1992) 1936--1948], for single-hop and multihop networks. We specify a fluid model and show that fluid-scaled performance processes can be approximated by fluid model solutio…
All-optical switching networks have been considered a promising candidate for the next generation data center networks thanks to its scalability in data bandwidth and power efficiency. However, the bufferless nature and the nonzero recon- figuration delay of optical switches remain great challenges in deploying all-optical networks. This paper considers the end-to- end scheduling for all-optical data center networks with no in- network buffer and nonzero reconfiguration delay. A framework is proposed to deal with the nonzero reconfiguration delay. The proposed approach constructs an adaptive v…
Achieving $\tilde{\mathcal{O}}(1/N)$ Optimality Gap in Restless Bandits through Gaussian Approximation
Electrical Engineering and Computer Science Department, University of Michigan, Ann Arbor（密歇根大学电气工程与计算机科学系）；Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）
We study the design of dynamic scheduling controls in closed queueing networks with a fixed number of jobs. Each time a server becomes available, the controller has (limited) flexibility in choosing the buffer from which to serve a job. If no jobs are available at any compatible buffer, the server idles. If the job is served, it relocates to a ``destination'' buffer. We study how to maximize throughput in steady state via a large deviations analysis. We propose a family of simple state-dependent policies called Scaled MaxWeight (SMW) policies that dynamically manage the distribution of jobs in…
The stochastic knapsack has been used as a model in wide ranging applications from dynamic resource allocation to admission control in telecommunication. In recent years, a variation of the model has become a basic tool in studying problems that arise in revenue management and dynamic/flexible pricing; and it is in this context that our study is undertaken. Based on a dynamic programming formulation and associated properties of the value function, we study in this paper a class of control that we call switch-over policies -- start from accepting only orders of the highest price, and switch to …
We consider a single-hop switched queueing network. Amongst a plethora of applications, these networks have been used to model wireless networks and input queued switches. The MaxWeight scheduling policies have proved popular, chiefly, because they are throughput optimal and do not require explicit estimation of arrival rates. In this article, we prove the same throughput optimality property for a generalization of the MaxWeight policy called the MaxWeight-(α,g) policy. These throughput optimal, myopic scheduling policies allow for scheduling choices similar to those found in bandwidth sharing…
