This paper presents a general class of dynamic stochastic optimization problems we refer to as Stochastic Depletion Problems. A number of challenging dynamic optimization problems of practical interest are stochastic depletion problems. Optimal solutions for such problems are difficult to obtain, both from a pragmatic computational perspective as also from a theoretical perspective. As such, simple heuristics are highly desirable. We isolate two simple properties that, if satisfied by a problem within this class, guarantee that a myopic policy incurs a performance loss of at most 50 % relative to the optimal adaptive control policy for that problem. We are able to verify that these two properties are satisfied for several interesting families of stochastic depletion problems and as a consequence identify efficient near-optimal control policies for a number of interesting dynamic stochastic optimization problems.
This paper provides a relaxation of the sufficient conditions, and also an extension of the structural results for Partially Observed Markov Decision Processes (POMDPs) given in Lovejoy (1987). Sufficient conditions are provided so that the optimal policy can be upper and lower bounded by judiciously chosen myopic policies. These myopic policy bounds are constructed to maximize the volume of belief states where they coincide with the optimal policy. Numerical examples illustrate these myopic bounds for both continuous and discrete observation sets.
Lookahead, also known as non-myopic, Bayesian optimization (BO) aims to find optimal sampling policies through solving a dynamic program (DP) that maximizes a long-term reward over a rolling horizon. Though promising, lookahead BO faces the risk of error propagation through its increased dependence on a possibly mis-specified model. In this work we focus on the rollout approximation for solving the intractable DP. We first prove the improving nature of rollout in tackling lookahead BO and provide a sufficient condition for the used heuristic to be rollout improving. We then provide both a theo…
We study the problem of maximizing a stochastic monotone submodular function with respect to a matroid constraint. Due to the presence of diminishing marginal values in real-world problems, our model can capture the effect of stochasticity in a wide range of applications. We show that the adaptivity gap -- the ratio between the values of optimal adaptive and optimal non-adaptive policies -- is bounded and is equal to e/(e-1). We propose a polynomial-time non-adaptive policy that achieves this bound. We also present an adaptive myopic policy that obtains at least half of the optimal value. Furt…
In this paper, we bring the celebrated max-weight features (myopic and discrete actions) to mainstream convex optimization. Myopic actions are important in control because decisions need to be made in an online manner and without knowledge of future events, and discrete actions because many systems have a finite (so non-convex) number of control decisions. For example, whether to transmit a packet or not in communication networks. Our results show that these two features can be encompassed in the subgradient method for the Lagrange dual problem by the use of stochastic and $ε$-subgradients. On…
We propose a formulation of the stochastic cutting stock problem as a discounted infinite-horizon Markov decision process. At each decision epoch, given current inventory of items, an agent chooses in which patterns to cut objects in stock in anticipation of the unknown demand. An optimal solution corresponds to a policy that associates each state with a decision and minimizes the expected total cost. Since exact algorithms scale exponentially with the state-space dimension, we develop a heuristic solution approach based on reinforcement learning. We propose an approximate policy iteration alg…
We consider a single-hop switched queueing network. Amongst a plethora of applications, these networks have been used to model wireless networks and input queued switches. The MaxWeight scheduling policies have proved popular, chiefly, because they are throughput optimal and do not require explicit estimation of arrival rates. In this article, we prove the same throughput optimality property for a generalization of the MaxWeight policy called the MaxWeight-(α,g) policy. These throughput optimal, myopic scheduling policies allow for scheduling choices similar to those found in bandwidth sharing…
Optimal and Order-optimal Gated Priority-based Greedy Policies for Two-layer Multi-item Order Fulfillment
Leonard N. Stern School of Business, New York University（纽约大学 Leonard N. Stern 商学院）；Qiuzhen College, Tsinghua University（清华大学邱泽学院）；Yau Mathematical Sciences Center & Department of Mathematical Sciences, Tsinghua University（清华大学姚数学科学中心及数学科学系）
We consider a periodic review inventory control problem having an underlying modulation process that affects demand and that is partially observed by the uncensored demand process and a novel additional observation data (AOD) process. We present an attainability condition, AC, that guarantees the existence of an optimal myopic base stock policy if the reorder cost $K=0$ and the existence of an optimal $(s, S)$ policy if $K>0$, where both policies depend on the belief function of the modulation process. Assuming AC holds, we show that (i) when $K=0$, the value of the optimal base stock level is…
We consider a dynamic server allocation problem over parallel queues with randomly varying connectivity and server switchover delay between the queues. At each time slot the server decides either to stay with the current queue or switch to another queue based on the current connectivity and the queue length information. Switchover delay occurs in many telecommunications applications and is a new modeling component of this problem that has not been previously addressed. We show that the simultaneous presence of randomly varying connectivity and switchover delay changes the system stability regi…
The problem of enhancing Quality-of-Service (QoS) in power constrained, mobile relay beamforming networks, by optimally and dynamically controlling the motion of the relaying nodes, is considered, in a dynamic channel environment. We assume a time slotted system, where the relays update their positions before the beginning of each time slot. Modeling the wireless channel as a Gaussian spatiotemporal stochastic field, we propose a novel $2$-stage stochastic programming problem formulation for optimally specifying the positions of the relays at each time slot, such that the expected QoS of the n…
Of paramount importance in both ecological systems and economic policies are the problems of harvesting of natural resources. A paradigmatic situation where this question is raised is that of fishing strategies. Indeed, overfishing is a well-known problem in the management of live-stocks, as being too greedy may lead to an overall dramatic depletion of the population we are harvesting. A closely related topic is that of Nash equilibria in the context of fishing policies. Namely, two players being in competition for the same pool of resources, is it possible for them to find an equilibrium situ…
