A new definition of continuous-time equilibrium controls is introduced. As opposed to the standard definition, which involves a derivative-type operation, the new definition parallels how a discrete-time equilibrium is defined, and allows for unambiguous economic interpretation. The terms "strong equilibria" and "weak equilibria" are coined for controls under the new and the standard definitions, respectively. When the state process is a time-homogeneous continuous-time Markov chain, a careful asymptotic analysis gives complete characterizations of weak and strong equilibria. Thanks to Kakutani-Fan's fixed-point theorem, general existence of weak and strong equilibria is also established, under additional compactness assumption. Our theoretic results are applied to a two-state model under non-exponential discounting. In particular, we demonstrate explicitly that there can be incentive to deviate from a weak equilibrium, which justifies the need for strong equilibria. Our analysis also provides new results for the existence and characterization of discrete-time equilibria under infinite horizon.
We consider three equilibrium concepts proposed in the literature for time-inconsistent stopping problems, including mild equilibria, weak equilibria and strong equilibria. The discount function is assumed to be log sub-additive and the underlying process is one-dimensional diffusion. We first provide necessary and sufficient conditions for the characterization of weak equilibria. The smooth-fit condition is obtained as a by-product. Next, based on the characterization of weak equilibria, we show that an optimal mild equilibrium is also weak. Then we provide conditions under which a weak equil…
We study an optimal stopping problem under non-exponential discounting, where the state process is a multi-dimensional continuous strong Markov process. The discount function is taken to be log sub-additive, capturing decreasing impatience in behavioral economics. On strength of probabilistic potential theory, we establish the existence of an optimal equilibrium among a sufficiently large collection of equilibria, consisting of finely closed equilibria satisfying a boundary condition. This generalizes the existence of optimal equilibria for one-dimensional stopping problems in prior literature…
This paper studies a nonzero-sum Dynkin game in discrete time under non-exponential discounting. For both players, there are two levels of game-theoretic reasoning intertwined. First, each player looks for an intra-personal equilibrium among her current and future selves, so as to resolve time inconsistency triggered by non-exponential discounting. Next, given the other player's chosen stopping policy, each player selects a best response among her intra-personal equilibria. A resulting inter-personal equilibrium is then a Nash equilibrium between the two players, each of whom employs her best …
We extend the construction of equilibria for linear-quadratic and mean-variance portfolio problems available in the literature to a large class of mean-field time-inconsistent stochastic control problems in continuous time. Our approach relies on a time discretization of the control problem via n-person games, which are characterized via the maximum principle using Backward Stochastic Differential Equations (BSDEs). The existence of equilibria is proved by applying weak convergence arguments to the solutions of n-person games. A numerical implementation is provided by approximating n-person ga…
We investigate the stability of equilibrium-induced optimal values with respect to (w.r.t.) reward functions $f$ and transition kernels $Q$ for time-inconsistent stopping problems under nonexponential discounting in discrete time. First, with locally uniform convergence of $f$ and $Q$ equipped with total variation distance, we show that the optimal value is semi-continuous w.r.t. $(f,Q)$. We provide examples showing that continuity may fail in general, and the convergence for $Q$ in total variation cannot be replaced by weak convergence. Next we show that with the uniform convergence of $f$ an…
We consider the game-theoretic approach to time-inconsistent stopping of a one-dimensional diffusion where the time-inconsistency is due to the presence of a non-exponential (weighted) discount function. In particular, we study (weak) equilibria for this problem in a novel class of mixed (i.e., randomized) stopping times based on a local time construction of the stopping intensity. For a general formulation of the problem we provide a verification theorem giving sufficient conditions for mixed (and pure) equilibria in terms of a set of variational inequalities, including a smooth fit condition…
We investigate the stability of the equilibrium-induced optimal value in one-dimensional diffusion setting for a time-inconsistent stopping problem under non-exponential discounting. We show that the optimal value is semi-continuous with respect to the drift, volatility, and reward function. An example is provided showing that the exact continuity may fail. With equilibria extended to $\varepsilon$-equilibria, we establish the relaxed continuity of the optimal value.
In this paper, we consider constrained discounted stochastic games with a countably generated state space and norm continuous transition probability having a density function. We prove existence of approximate stationary equilibria and stationary weak correlated equilibria. Our results imply the existence of stationary Nash equilibrium in $ARAT$ stochastic games.
Equilibrium for Time-inconsistent Mean Field Games: A Systematic Analysis by Entropy Regularization
University of Michigan（密歇根大学）；Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；The Hong Kong Polytechnic University（香港理工大学）；Tsinghua University（清华大学）
We study an infinite-horizon discrete-time optimal stopping problem under non-exponential discounting. A new method, which we call the iterative approach, is developed to find subgame perfect Nash equilibria. When the discount function induces decreasing impatience, we establish the existence of an equilibrium through fixed-point iterations. Moreover, we show that there exists a unique optimal equilibrium, which generates larger value than any other equilibrium does at all times. To the best of our knowledge, this is the first time a dominating subgame perfect Nash equilibrium is shown to exis…
We study a class of dynamic decision problems of mean field type with time inconsistent cost functionals, and derive a stochastic maximum principle to characterize subgame perfect Nash equilibrium points. Subsequently, this approach is extended to a mean field game to construct decentralized strategies and obtain an estimate of their performance.
The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independent transitions, and stochastic games with mixtures of constant transition kernels as special cases. A remarkably simple proof is provided via establishing a new connection between stochastic games and conditional expectations of correspondences. New applications of stochastic gam…
This paper characterizes differentiable and subgame Markov perfect equilibria in a continuous time intertemporal decision problem with non-constant discounting. Capturing the idea of non commitment by letting the commitment period being infinitesimally small, we characterize the equilibrium strategies by a value function, which must satisfy a certain equation. The equilibrium equation is reminiscent of the classical Hamilton-Jacobi-Bellman equation of optimal control, but with a non-local term leading to differences in qualitative behavior. As an application, we formulate an overlapping genera…
In this paper we study a class of time-inconsistent terminal Markovian control problems in discrete time subject to model uncertainty. We combine the concept of the sub-game perfect strategies with the adaptive robust stochastic to tackle the theoretical aspects of the considered stochastic control problem. Consequently, as an important application of the theoretical results, by applying a machine learning algorithm we solve numerically the mean-variance portfolio selection problem under the model uncertainty.
In this paper, we study a time-inconsistent stochastic optimal control problem with a recursive cost functional by a multi-person hierarchical differential game approach. An equilibrium strategy of this problem is constructed and a corresponding equilibrium Hamilton-Jacobi-Bellman (HJB, for short) equation is established to characterize the associated equilibrium value function. Moreover, a well-posedness result of the equilibrium HJB equation is established under certain conditions.
In this paper, we consider a linear quadratic stochastic two-person nonzero-sum differential game. Open-loop and closed-loop Nash equilibria are introduced. The existence of the former is characterized by the solvability of a system of forward-backward stochastic differential equations, and that of the latter is characterized by the solvability of a system of coupled symmetric Riccati differential equations. Sometimes, open-loop Nash equilibria admit a closed-loop representation, via the solution to a system of non-symmetric Riccati equations, which is different from the outcome of the closed-…
We construct an approximate public-signal correlated equilibrium for a nonzero-sum differential game in the class of stochastic strategies with memory. The construction is based on a solution of an auxiliary nonzero-sum continuous-time stochastic game. This class of games includes stochastic differential games and continuous-time Markov games. Moreover, we study the limit of approximate equilibrium outcomes in the case when the auxiliary stochastic games tend to the original deterministic one. We show that it lies in the convex hull of the set of equilibrium values provided by deterministic pu…
We introduce a notion of subgames for stochastic timing games and the related notion of subgame-perfect equilibrium in possibly mixed strategies. While a good notion of subgame-perfect equilibrium for continuous-time games is not available in general, we argue that our model is the appropriate version for timing games. We show that the notion coincides with the usual one for discrete-time games. Many timing games in continuous time have only equilibria in mixed strategies -- in particular preemption games, which often occur in the strategic real option literature. We provide a sound foundation…
This paper studies open-loop equilibriums for a general class of time-inconsistent stochastic control problems under jump-diffusion SDEs with deterministic coefficients. Inspired by the idea of Four-Step-Scheme for forward-backward stochastic differential equations with jumps (FBSDEJs, for short), we derive two systems of integro-partial differential equations (IPDEs, for short). Then, we rigorously prove a verification theorem which provides a sufficient condition for open-loop equilibrium strategies. As an illustration of the general theory, we discuss a mean-variance portfolio selection pro…
Ridesharing markets are complex: drivers are strategic, rider demand and driver availability are stochastic, and complex city-scale phenomena like weather induce large scale correlation across space and time. At the same time, past work has focused on a subset of these challenges. We propose a model of ridesharing networks with strategic drivers, spatiotemporal dynamics, and stochasticity. Supporting both computational tractability and better modeling flexibility than classical fluid limits, we use a two-level stochastic model that allows correlated shocks caused by weather or large public eve…
The distributed computation of Nash equilibria is assuming growing relevance in engineering where such problems emerge in the context of distributed control. Accordingly, we present schemes for computing equilibria of two classes of static stochastic convex games complicated by a parametric misspecification, a natural concern in the control of large-scale networked engineered system. In both schemes, players learn the equilibrium strategy while resolving the misspecification: (1) Monotone stochastic Nash games: We present a set of coupled stochastic approximation schemes distributed across age…
This paper deals with a class of time inconsistent stochastic linear quadratic (SLQ) optimal control problems in Markovian framework. Three notions, i.e., closed-loop equilibrium controls/strategies, open-loop equilibrium controls and their closed-loop representations, are characterized in unified manners. These results indicate clearer and deeper distinctions among these notions. For example, in particular time consistent setting, the open-loop equilibrium controls are fully characterized by first-order, second-order necessary optimality conditions, and become needlessly optimal, while the cl…
This paper considers mean field games in a multi-agent Markov decision process (MDP) framework. Each player has a continuum state and binary action, and benefits from the improvement of the condition of the overall population. Based on an infinite horizon discounted individual cost, we show existence of a stationary equilibrium, and prove its uniqueness under a positive externality condition. We further analyze comparative statics of the stationary equilibrium by quantitatively determining the impact of the effort cost.
Time-Inconsistent MDPs with Entropy Regularization: Equilibrium Existence and Policy Iteration
Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；Shandong Province Key Laboratory of Financial Risk（山东省金融风险重点实验室）
In this paper, we study extended backward stochastic Volterra integral equations (EBSVIEs, for short). We establish the well-posedness under weaker assumptions than the literature, and prove a new kind of regularity property for the solutions. As an application, we investigate, in the open-loop framework, a time-inconsistent stochastic recursive control problem where the cost functional is defined by the solution to a backward stochastic Volterra integral equation (BSVIE, for short). We show that the corresponding adjoint equations become EBSVIEs, and provide a necessary and sufficient conditi…
Inspired by recent work of P.-L. Lions on conditional optimal control, we introduce a problem of optimal stopping under bounded rationality: the objective is the expected payoff at the time of stopping, conditioned on another event. For instance, an agent may care only about states where she is still alive at the time of stopping, or a company may condition on not being bankrupt. We observe that conditional optimization is time-inconsistent due to the dynamic change of the conditioning probability and develop an equilibrium approach in the spirit of R. H. Strotz' work for sophisticated agents …
An optimal control problem is considered for a stochastic differential equation containing a state-dependent regime switching, with a recursive cost functional. Due to the non-exponential discounting in the cost functional, the problem is time-inconsistent in general. Therefore, instead of finding a global optimal control (which is not possible), we look for a time-consistent (approximately) locally optimal equilibrium strategy. Such a strategy can be represented through the solution to a system of partial differential equations, called an equilibrium Hamilton-Jacob-Bellman (HJB, for short) eq…
In this paper, a class of time inconsistent linear quadratic optimal control problems of mean-field stochastic differential equations (SDEs) is considered under Markovian framework. Open-loop equilibrium controls and their particular closed-loop representations are introduced and characterized via variational ideas. Several interesting features are revealed and a system of coupled Riccati equations is derived. In contrast with the analogue optimal control problems of SDEs, the mean-field terms in state equation, which is another reason of time inconsistency, prompts us to define above two noti…
In this paper, we study a class of stochastic time-inconsistent linear-quadratic (LQ) control problems with control input constraints. These problems are investigated within the more general framework associated with random coefficients. This paper aims to further develop a new methodology, which fundamentally differs from those in the standard control (without constraints) theory in the literature, to cope with the mathematical difficulties raised due to the presence of input constraints. We first prove that the existence of an equilibrium solution is equivalent to the existence of a solution…
Standard Markovian optimal stopping problems are consistent in the sense that the first entrance time into the stopping set is optimal for each initial state of the process. Clearly, the usual concept of optimality cannot in a straightforward way be applied to non-standard stopping problems without this time-consistent structure. This paper is devoted to the solution of time-inconsistent stopping problems with the reward depending on the initial state using an adaptation of Strotz's consistent planning. More precisely, we give a precise equilibrium definition --- of the type subgame perfect Na…
We consider two classes of constrained finite state-action stochastic games. First, we consider a two player nonzero sum single controller constrained stochastic game with both average and discounted cost criterion. We consider the same type of constraints as in [1], i.e., player 1 has subscription based constraints and player 2, who controls the transition probabilities, has realization based constraints which can also depend on the strategies of player 1. Next, we consider a N -player nonzero sum constrained stochastic game with independent state processes where each player has average cost …
In this paper, which is a continuation of the previously published discrete time paper we develop a theory for continuous time stochastic control problems which, in various ways, are time inconsistent in the sense that they do not admit a Bellman optimality principle. We study these problems within a game theoretic framework, and we look for Nash subgame perfect equilibrium points. For a general controlled continuous time Markov process and a fairly general objective functional we derive an extension of the standard Hamilton-Jacobi-Bellman equation, in the form of a system of non-linear equati…
The modelling of modern power markets requires the representation of the following main features: (i) a stochastic dynamic decision process, with uncertainties related to renewable production and fuel costs, among others; and (ii) a game-theoretic framework that represents the strategic behavior of multiple agents, for example in daily price bids. These features can be in theory represented as a stochastic dynamic programming recursion, where we have a Nash equilibrium for multiple agents. However, the resulting problem is very challenging to solve. This work presents an iterative process to s…
This paper studies stochastic games on large graphs and their graphon limits. We propose a new formulation of graphon games based on a single typical player's label-state distribution. In contrast, other recently proposed models of graphon games work directly with a continuum of players, which involves serious measure-theoretic technicalities. In fact, by viewing the label as a component of the state process, we show in our formulation that graphon games are a special case of mean field games, albeit with certain inevitable degeneracies and discontinuities that make most existing results on me…
We present a subjective equilibrium notion (called "subjective equilibrium under beliefs of exogenous uncertainty (SEBEU)" for stochastic dynamic games in which each player chooses her decisions under the (incorrect) belief that a stochastic environment process driving the system is exogenous whereas in actuality this process is a solution of closed-loop dynamics affected by each individual player. Players observe past realizations of the environment variables and their local information. At equilibrium, if players are given the full distribution of the stochastic environment process as if it …
For a discrete time Markov chain and in line with Strotz' consistent planning we develop a framework for problems of optimal stopping that are time-inconsistent due to the consideration of a non-linear function of an expected reward. We consider pure and mixed stopping strategies and a (subgame perfect Nash) equilibrium. We provide different necessary and sufficient equilibrium conditions including a verification theorem. Using a fixed point argument we provide equilibrium existence results. We adapt and study the notion of the myopic adjustment process and introduce different kinds of equilib…
Motivated by uncertain parameters encountered in Markov decision processes (MDPs) and stochastic games, we study the effect of parameter uncertainty on Bellman operator-based algorithms under a set-based framework. Specifically, we first consider a family of MDPs where the cost parameters are in a given compact set; we then define a Bellman operator acting on a set of value functions to produce a new set of value functions as the output under all possible variations in the cost parameter. We prove the existence of a fixed point of this set-based Bellman operator by showing that it is contracti…
In this paper, the open-loop and closed-loop local and remote stochastic nonzero-sum game (LRSNG) problem is investigated. Different from previous works, the stochastic nonzero-sum game problem under consideration is essentially a special class of two-person nonzero-sum game problem, in which the information sets accessed by the two players are inconsistent. More specifically, both the local player and the remote player are involved in the system dynamics, and the information sets obtained by the two players are different, and each player is designed to minimize its own cost function. For the …
