In this paper, we study the optimal stopping problem in the so-called exploratory framework, in which the agent takes actions randomly conditioning on current state and an entropy-regularized term is added to the reward functional. Such a transformation reduces the optimal stopping problem to a standard optimal control problem. We derive the related HJB equation and prove its solvability. Furthermore, we give a convergence rate of policy iteration and the comparison to classical optimal stopping problem. Based on the theoretical analysis, a reinforcement learning algorithm is designed and numerical results are demonstrated for several models.
The purpose of this paper is two-fold: We extend the well-known relation between optimal stopping and randomized stopping of a given stochastic process to a situation where the available information flow is a filtration with no a priori assumed relation to the filtration of the process. We call these problems optimal stopping and randomized stopping with general information. Following an idea of Krylov [K] we introduce a special singular stochastic control problem with general information and show that this is also equivalent to the partial information optimal stopping and randomized stopping …
We study two-player zero-sum stopping games in continuous time and infinite horizon. We prove that the value in randomized stopping times exists as soon as the payoff processes are right-continuous. In particular, as opposed to existing literature, we do not assume any conditions on the relations between the payoff processes. We also show that both players have simple epsilon- optimal randomized stopping times; namely, randomized stopping times which are small perturbations of non-randomized stopping times.
A general result on the method of randomized stopping is proved. It is applied to optimal stopping of controlled diffusion processes with unbounded coefficients to reduce it to an optimal control problem without stopping. This is motivated by recent results of Krylov on numerical solutions to the Bellman equation.
In this paper we study randomized optimal stopping problems and consider corresponding forward and backward Monte Carlo based optimisation algorithms. In particular we prove the convergence of the proposed algorithms and derive the corresponding convergence rates.
UCLA Anderson School of Management, University of California, Los Angeles（加州大学洛杉矶分校安德森管理学院）
Optimal stopping is the problem of determining when to stop a stochastic system in order to maximize reward, which is of practical importance in domains such as finance, operations management and healthcare. Existing methods for high-dimensional optimal stopping that are popular in practice produce deterministic linear policies -- policies that deterministically stop based on the sign of a weighted sum of basis functions -- but are not guaranteed to find the optimal policy within this policy class given a fixed basis function architecture. In this paper, we propose a new methodology for optima…
In this article we study and classify optimal martingales in the dual formulation of optimal stopping problems. In this respect we distinguish between weakly optimal and surely optimal martingales. It is shown that the family of weakly optimal and surely optimal martingales may be quite large. On the other hand it is shown that the Doob-martingale, that is, the martingale part of the Snell envelope, is in a certain sense the most robust surely optimal martingale under random perturbations. This new insight leads to a novel randomized dual martingale minimization algorithm that doesn't require …
Given an initial (resp., terminal) probability measure $μ$ (resp., $ν$) on $\mathbb{R}^d$, we characterize those optimal stopping times $τ$ that maximize or minimize the functional $\mathbb{E} |B_0 - B_τ|^α$, $α> 0$, where $(B_t)_t$ is Brownian motion with initial law $B_0\sim μ$ and with final distribution --once stopped at $τ$-- equal to $B_τ\sim ν$. The existence of such stopping times is guaranteed by Skorohod-type embeddings of probability measures in "subharmoic order" into Brownian motion. This problem is equivalent to an optimal mass transport problem with certain constraints, namely t…
We consider the game-theoretic approach to time-inconsistent stopping of a one-dimensional diffusion where the time-inconsistency is due to the presence of a non-exponential (weighted) discount function. In particular, we study (weak) equilibria for this problem in a novel class of mixed (i.e., randomized) stopping times based on a local time construction of the stopping intensity. For a general formulation of the problem we provide a verification theorem giving sufficient conditions for mixed (and pure) equilibria in terms of a set of variational inequalities, including a smooth fit condition…
We study the problem of stopping a Brownian motion at a given distribution $ν$ while optimizing a reward function that depends on the (possibly randomized) stopping time and the Brownian motion. Our first result establishes that the set $\mathcal{T}(ν)$ of stopping times embedding $ν$ is weakly dense in the set $\mathcal{R}(ν)$ of randomized embeddings. In particular, the optimal Skorokhod embedding problem over $\mathcal{T}(ν)$ has the same value as the relaxed one over $\mathcal{R}(ν)$ when the reward function is semicontinuous, which parallels a fundamental result about Monge maps and Kanto…
In tabular multi-agent reinforcement learning with average-cost criterion, a team of agents sequentially interacts with the environment and observes local incentives. We focus on the case that the global reward is a sum of local rewards, the joint policy factorizes into agents' marginals, and full state observability. To date, few global optimality guarantees exist even for this simple setting, as most results yield convergence to stationarity for parameterized policies in large/possibly continuous spaces. To solidify the foundations of MARL, we build upon linear programming (LP) reformulation…
We study a class of optimal stopping games (Dynkin games) of preemption type, with uncertainty about the existence of competitors. The set-up is well-suited to model, for example, real options in the context of investors who do not want to publicly reveal their interest in a certain business opportunity. We show that there exists a Nash equilibrium in randomized stopping times which is described explicitly in terms of the corresponding one-player game.
We prove the dynamic programming principe for uniformly nondegenerate stochastic differential games in the framework of time-homogeneous diffusion processes considered up to the first exit time from a domain. The zeroth-order "coefficient" and the "free" term are only assumed to be measurable. In contrast with previous results established for constant stopping times we allow arbitrary stopping times and randomized ones as well. The main assumption, which will be removed in a subsequent article, is that there exists a sufficiently regular solution of the Isaacs equation.
We consider a zero-sum stochastic differential game over elementary mixed feed-back strategies. These are strategies based only on the knowledge of the past state, randomized continuously in time from a sampling distribution which is kept constant in between some stopping rules. Once both players choose such strategies, the state equation admits a unique solution in the sense of the martingale problem of Stroock and Varadhan. We show that the game defined over martingale solutions has a value, which is the unique continuous viscosity solution of the randomized Isaacs equation.
We prove the dynamic programming principe for uniformly nondegenerate stochastic differential games in the framework of time-homogeneous diffusion processes considered up to the first exit time from a domain. In contrast with previous results established for constant stopping times we allow arbitrary stopping times and randomized ones as well. There is no assumption about solvability of the the Isaacs equation in any sense (classical or viscosity). The zeroth-order "coefficient" and the "free" term are only assumed to be measurable in the space variable. We also prove that value functions are …
Value iteration (VI) is a ubiquitous algorithm for optimal control, planning, and reinforcement learning schemes. Under the right assumptions, VI is a vital tool to generate inputs with desirable properties for the controlled system, like optimality and Lyapunov stability. As VI usually requires an infinite number of iterations to solve general nonlinear optimal control problems, a key question is when to terminate the algorithm to produce a "good" solution, with a measurable impact on optimality and stability guarantees. By carefully analysing VI under general stabilizability and detectabilit…
We combine the fields of heuristic optimization and optimal stopping. We propose a strategy for benchmarking randomized optimization algorithms that minimizes the expected total cost for obtaining a good solution with an optimal number of calls to the solver. To do so, rather than letting the objective function alone define a cost to be minimized, we introduce a further cost-per-call of the algorithm. We show that this problem can be formulated using optimal stopping theory. The expected cost is a flexible figure of merit for benchmarking probabilistic solvers that can be computed when the opt…
Zhejiang University（浙江大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We propose and study kernel conjugate gradient methods (KCGM) with random projections for least-squares regression over a separable Hilbert space. Considering two types of random projections generated by randomized sketches and Nyström subsampling, we prove optimal statistical results with respect to variants of norms for the algorithms under a suitable stopping rule. Particularly, our results show that if the projection dimension is proportional to the effective dimension of the problem, KCGM with randomized sketches can generalize optimally, while achieving a computational advantage. As a co…
This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy addresses the exploration-exploitation dilemma in linear control systems that evolve according to unknown stochastic differential equations and their operating cost is quadratic. More precisely, we establish square-root of time regret bounds, indicating that randomized certainty equivalent policy learns optimal control actions fast from a single state trajectory. Further, linear…
Reinforcement learning with function approximation has recently achieved tremendous results in applications with large state spaces. This empirical success has motivated a growing body of theoretical work proposing necessary and sufficient conditions under which efficient reinforcement learning is possible. From this line of work, a remarkably simple minimal sufficient condition has emerged for sample efficient reinforcement learning: MDPs with optimal value function $V^*$ and $Q^*$ linear in some known low-dimensional features. In this setting, recent works have designed sample efficient algo…
In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequences. Accordingly, the objective of this paper is to present efficient reinforcement learning algorithms for risk-constrained Markov decision processes (MDPs), where risk is represented via a chance constraint or a constraint on the conditional value-at-risk (CVaR) of the cumulative cost. We collectively refer to such problems as percentile risk-constrained MDPs. Specifically, we …
