In this paper, we propose a novel equilibrium solution notion for the time-inconsistent stochastic linear-quadratic optimal control problem. This notion is called the mixed equilibrium solution, which consists of two parts: a pure-feedback-strategy part and an open-loop-control part. When the pure-feedback-strategy part is zero or the open-loop-control part does not depend on the initial state, the mixed equilibrium solution reduces to the open-loop equilibrium control and the feedback equilibrium strategy, respectively. Using a maximum-principle-like methodology with forward-backward stochastic difference equations, a necessary and sufficient condition is established to characterize the existence of a mixed equilibrium solution. Then, by decoupling the forward-backward stochastic difference equations, three sets of difference equations, which together portray the existence of a mixed equilibrium solution, are obtained. Moreover, the case with a fixed time-state initial pair and the case with all the initial pairs are separately investigated. Furthermore, an example is constructed to show that the mixed equilibrium solution exists for all the initial pairs, although neither the open-loop equilibrium control nor the feedback equilibrium strategy exists for some initial pairs.
The paper deals with a class of time-inconsistent control problems for McKean-Vlasov dynamics. By solving a backward time-inconsistent Hamilton-Jacobi-Bellman (HJB for short) equation coupled with a forward distribution-dependent stochastic differential equation, we investigate the existence and uniqueness of a closed-loop equilibrium for such time-inconsistent distribution-dependent control problem. Moreover, a special case of semi-linear McKean-Vlasov dynamics with a quadratic-type cost functional is considered due to its special structure.
A game-theoretic framework for time-inconsistent stopping problems where the time-inconsistency is due to the consideration of a non-linear function of an expected reward is developed. A class of mixed strategy stopping times that allows the agents in the game to jointly choose the intensity function of a Cox process is introduced and motivated. A subgame perfect Nash equilibrium is defined. The equilibrium is characterized and other necessary and sufficient equilibrium conditions including a smooth fit result are proved. Existence and uniqueness are investigated. A mean-variance and a varianc…
This paper deals with a class of time inconsistent stochastic linear quadratic (SLQ) optimal control problems in Markovian framework. Three notions, i.e., closed-loop equilibrium controls/strategies, open-loop equilibrium controls and their closed-loop representations, are characterized in unified manners. These results indicate clearer and deeper distinctions among these notions. For example, in particular time consistent setting, the open-loop equilibrium controls are fully characterized by first-order, second-order necessary optimality conditions, and become needlessly optimal, while the cl…
This paper studies open-loop equilibriums for a general class of time-inconsistent stochastic control problems under jump-diffusion SDEs with deterministic coefficients. Inspired by the idea of Four-Step-Scheme for forward-backward stochastic differential equations with jumps (FBSDEJs, for short), we derive two systems of integro-partial differential equations (IPDEs, for short). Then, we rigorously prove a verification theorem which provides a sufficient condition for open-loop equilibrium strategies. As an illustration of the general theory, we discuss a mean-variance portfolio selection pro…
In this paper, we study a time-inconsistent stochastic optimal control problem with a recursive cost functional by a multi-person hierarchical differential game approach. An equilibrium strategy of this problem is constructed and a corresponding equilibrium Hamilton-Jacobi-Bellman (HJB, for short) equation is established to characterize the associated equilibrium value function. Moreover, a well-posedness result of the equilibrium HJB equation is established under certain conditions.
An optimal control problem is considered for a stochastic differential equation containing a state-dependent regime switching, with a recursive cost functional. Due to the non-exponential discounting in the cost functional, the problem is time-inconsistent in general. Therefore, instead of finding a global optimal control (which is not possible), we look for a time-consistent (approximately) locally optimal equilibrium strategy. Such a strategy can be represented through the solution to a system of partial differential equations, called an equilibrium Hamilton-Jacob-Bellman (HJB, for short) eq…
In this paper, a class of time inconsistent linear quadratic optimal control problems of mean-field stochastic differential equations (SDEs) is considered under Markovian framework. Open-loop equilibrium controls and their particular closed-loop representations are introduced and characterized via variational ideas. Several interesting features are revealed and a system of coupled Riccati equations is derived. In contrast with the analogue optimal control problems of SDEs, the mean-field terms in state equation, which is another reason of time inconsistency, prompts us to define above two noti…
In this paper, we study a class of stochastic time-inconsistent linear-quadratic (LQ) control problems with control input constraints. These problems are investigated within the more general framework associated with random coefficients. This paper aims to further develop a new methodology, which fundamentally differs from those in the standard control (without constraints) theory in the literature, to cope with the mathematical difficulties raised due to the presence of input constraints. We first prove that the existence of an equilibrium solution is equivalent to the existence of a solution…
This paper focuses on a class of continuous-time controlled Markov chains with time-inconsistent and distribution-dependent cost functional (in some appropriate sense). A new definition of time-inconsistent distribution-dependent equilibrium in closed-loop sense is given and its existence and uniqueness have been established. Because of the time-inconsistency, it is proved that the equilibrium is locally optimal in an appropriate sense. Moreover, it has been shown that our problem is essentially equivalent to an infinite-player mean-field game with time-inconsistent cost.
We consider the game-theoretic approach to time-inconsistent stopping of a one-dimensional diffusion where the time-inconsistency is due to the presence of a non-exponential (weighted) discount function. In particular, we study (weak) equilibria for this problem in a novel class of mixed (i.e., randomized) stopping times based on a local time construction of the stopping intensity. For a general formulation of the problem we provide a verification theorem giving sufficient conditions for mixed (and pure) equilibria in terms of a set of variational inequalities, including a smooth fit condition…
In this paper, we investigate a class of time-inconsistent discrete-time stochastic linear-quadratic optimal control problems, whose time-consistent solutions consist of an open-loop equilibrium control and a linear feedback equilibrium strategy. The open-loop equilibrium control is defined for a given initial pair, while the linear feedback equilibrium strategy is defined for all the initial pairs. Maximum-principle-type necessary and sufficient conditions containing stationary and convexity are derived for the existence of these two time-consistent solutions, respectively. Furthermore, for t…
This paper is devoted to solving a time-inconsistent risk-sensitive control problem with parameter $\e$ and its limit case ($\e\rightarrow0^+$) for countable-stated Markov decision processes (MDPs for short). Since the cost functional is time-inconsistent, it is impossible to find a global optimal strategy for both cases. Instead, for each case, we will prove the existence of time-inconstant equilibrium strategies which verify the so-called step-optimality. Moreover, we prove the convergence of $\e$-equilibriums and the corresponding value functions as $\e\rightarrow0^+$.
In this paper, which is a continuation of the previously published discrete time paper we develop a theory for continuous time stochastic control problems which, in various ways, are time inconsistent in the sense that they do not admit a Bellman optimality principle. We study these problems within a game theoretic framework, and we look for Nash subgame perfect equilibrium points. For a general controlled continuous time Markov process and a fairly general objective functional we derive an extension of the standard Hamilton-Jacobi-Bellman equation, in the form of a system of non-linear equati…
This paper characterizes differentiable and subgame Markov perfect equilibria in a continuous time intertemporal decision problem with non-constant discounting. Capturing the idea of non commitment by letting the commitment period being infinitesimally small, we characterize the equilibrium strategies by a value function, which must satisfy a certain equation. The equilibrium equation is reminiscent of the classical Hamilton-Jacobi-Bellman equation of optimal control, but with a non-local term leading to differences in qualitative behavior. As an application, we formulate an overlapping genera…
Equilibrium for Time-inconsistent Mean Field Games: A Systematic Analysis by Entropy Regularization
University of Michigan（密歇根大学）；Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；The Hong Kong Polytechnic University（香港理工大学）；Tsinghua University（清华大学）
A time-inconsistent optimal control problem is formulated and studied for a controlled linear ordinary differential equation with quadratic cost functional. A notion of equilibrium control is introduced, which can be regarded as a time-consistent solution to the original time-inconsistent problem. Under certain conditions, we constructively prove the existence of such an equilibrium control which is represented via a forward ordinary differential equation coupled with a backward Riccati--Volterra integral equation. Our constructive approach is based on the introduction of a family of $N$-perso…
A general time-inconsistent optimal control problem is considered for stochastic differential equations with deterministic coefficients. Under suitable conditions, a Hamilton-Jacobi-Bellman type equation is derived for the equilibrium value function of the problem. Well-posedness and some properties of such an equation is studied, and time-consistent equilibrium strategies are constructed. As special cases, the linear-quadratic problem and a generalized Merton's portfolio problem are investigated.
Control problems not admitting the dynamic programming principle are known as time-inconsistent. The game-theoretic approach is to interpret such problems as intrapersonal dynamic games and look for subgame perfect Nash equilibria. A fundamental result of time-inconsistent stochastic control is a verification theorem saying that solving the extended HJB system is a sufficient condition for equilibrium. We show that solving the extended HJB system is a necessary condition for equilibrium, under regularity assumptions. The controlled process is a general Itô diffusion.
Existence of Relaxed Equilibrium for Time-Inconsistent Mean Field Games: A Set-Valued Fixed Point Approach
Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰金融研究院）；School of Mathematics and Statistics, University of Sydney（悉尼大学数学与统计学院）
We study an infinite-horizon discrete-time optimal stopping problem under non-exponential discounting. A new method, which we call the iterative approach, is developed to find subgame perfect Nash equilibria. When the discount function induces decreasing impatience, we establish the existence of an equilibrium through fixed-point iterations. Moreover, we show that there exists a unique optimal equilibrium, which generates larger value than any other equilibrium does at all times. To the best of our knowledge, this is the first time a dominating subgame perfect Nash equilibrium is shown to exis…
A \emph{new} notion of equilibrium, which we call \emph{strong equilibrium}, is introduced for time-inconsistent stopping problems in continuous time. Compared to the existing notions introduced in ArXiv: 1502.03998 and ArXiv: 1709.05181, which in this paper are called \emph{mild equilibrium} and \emph{weak equilibrium} respectively, a strong equilibrium captures the idea of subgame perfect Nash equilibrium more accurately. When the state process is a continuous-time Markov chain and the discount function is log sub-additive, we show that an optimal mild equilibrium is always a strong equilibr…
For a discrete time Markov chain and in line with Strotz' consistent planning we develop a framework for problems of optimal stopping that are time-inconsistent due to the consideration of a non-linear function of an expected reward. We consider pure and mixed stopping strategies and a (subgame perfect Nash) equilibrium. We provide different necessary and sufficient equilibrium conditions including a verification theorem. Using a fixed point argument we provide equilibrium existence results. We adapt and study the notion of the myopic adjustment process and introduce different kinds of equilib…
We consider the dividend maximization problem including a ruin penalty in a diffusion environment. The additional penalty term is motivated by a constraint on dividend strategies. Intentionally, we use different discount rates for the dividends and the penalty, which causes time-inconsistency. This allows to study different types of constraints. For the diffusion approximation of the classical surplus process we derive an explicit equilibrium dividend strategy and the associated value function. Inspired by duality arguments, we can identify a particular equilibrium strategy such that for a giv…
In this technical note, we consider the linear-quadratic time-inconsistent mean-field type leader-follower Stackelberg differential game with an adapted open-loop information structure. The objective functionals of the leader and the follower include conditional expectations of state and control (mean field) variables, and the cost parameters could be general nonexponential discounting depending on the initial time. As stated in the existing literature, these two general settings of the objective functionals induce time inconsistency in the optimal solutions. Given an arbitrary control of the …
This is a companion paper of [Mixed equilibrium solution of time-inconsistent stochastic LQ problem, arXiv:1802.03032], where general theory has been established to characterize the open-loop equilibrium control, feedback equilibrium strategy and mixed equilibrium solution for a time-inconsistent stochastic linear-quadratic problem. This note is, on the one hand to test the developed theory of that paper, and on the other hand to push the solvability of multi-period mean-variance portfolio selection. A nondegenerate assumption has been removed in this note, which is popular in existing literat…
This paper is concerned with the open-loop time-consistent solution of time-inconsistent mean-field stochastic linear-quadratic optimal control. Different from standard stochastic linear-quadratic problems, both the system matrices and the weighting matrices are dependent on the initial times, and the conditional expectations of the control and state enter quadratically into the cost functional. Such features will ruin Bellman's principle of optimality and result in the time-inconsistency of the optimal control. Based on the dynamical nature of the systems involved, a kind of open-loop time-co…
Time-Inconsistent MDPs with Entropy Regularization: Equilibrium Existence and Policy Iteration
Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；Shandong Province Key Laboratory of Financial Risk（山东省金融风险重点实验室）
In this paper, the open-loop and closed-loop local and remote stochastic nonzero-sum game (LRSNG) problem is investigated. Different from previous works, the stochastic nonzero-sum game problem under consideration is essentially a special class of two-person nonzero-sum game problem, in which the information sets accessed by the two players are inconsistent. More specifically, both the local player and the remote player are involved in the system dynamics, and the information sets obtained by the two players are different, and each player is designed to minimize its own cost function. For the …
We consider three equilibrium concepts proposed in the literature for time-inconsistent stopping problems, including mild equilibria, weak equilibria and strong equilibria. The discount function is assumed to be log sub-additive and the underlying process is one-dimensional diffusion. We first provide necessary and sufficient conditions for the characterization of weak equilibria. The smooth-fit condition is obtained as a by-product. Next, based on the characterization of weak equilibria, we show that an optimal mild equilibrium is also weak. Then we provide conditions under which a weak equil…
In this paper, we consider a general time-inconsistent optimal control problem for a non homogeneous linear system, in which its state evolves according to a stochastic differential equation with deterministic coefficients, when the noise is driven by a Brownian motion and an independent Poisson point process. The running and the terminal costs in the objective functional, are explicitly dependent on some general discounting coefficients which cover the non-exponential and the hyperbolic discounting situations. Furthermore, the presence of some quadratic terms of the conditional expectation of…
An optimal control problem is considered for a stochastic differential equation with the cost functional determined by a backward stochastic Volterra integral equation (BSVIE, for short). This kind of cost functional can cover the general discounting (including exponential and non-exponential) situation with a recursive feature. It is known that such a problem is time-inconsistent in general. Therefore, instead of finding a global optimal control, we look for a time-consistent locally near optimal equilibrium strategy. With the idea of multi-person differential games, a family of approximate e…
Standard Markovian optimal stopping problems are consistent in the sense that the first entrance time into the stopping set is optimal for each initial state of the process. Clearly, the usual concept of optimality cannot in a straightforward way be applied to non-standard stopping problems without this time-consistent structure. This paper is devoted to the solution of time-inconsistent stopping problems with the reward depending on the initial state using an adaptation of Strotz's consistent planning. More precisely, we give a precise equilibrium definition --- of the type subgame perfect Na…
In practical applications, decision-makers with heterogeneous dynamics may be engaged in the same decision-making process. This motivates us to study distributed Nash equilibrium seeking for games in which players are mixed-order (first- and second-order) integrators influenced by unknown dynamics and external disturbances in this paper. To solve this problem, we employ an adaptive neural network to manage unknown dynamics and disturbances, based on which a distributed Nash equilibrium seeking algorithm is developed by further adapting concepts from gradient-based optimization and multi-agent …
In this paper, we establish a subgame perfect equilibrium reinforcement learning (SPERL) framework for time-inconsistent (TIC) problems. In the context of RL, TIC problems are known to face two main challenges: the non-existence of natural recursive relationships between value functions at different time points and the violation of Bellman's principle of optimality that raises questions on the applicability of standard policy iteration algorithms for unprovable policy improvement theorems. We adapt an extended dynamic programming theory and propose a new class of algorithms, called backward po…
This paper studies a nonzero-sum Dynkin game in discrete time under non-exponential discounting. For both players, there are two levels of game-theoretic reasoning intertwined. First, each player looks for an intra-personal equilibrium among her current and future selves, so as to resolve time inconsistency triggered by non-exponential discounting. Next, given the other player's chosen stopping policy, each player selects a best response among her intra-personal equilibria. A resulting inter-personal equilibrium is then a Nash equilibrium between the two players, each of whom employs her best …
Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems
Department of Industrial Engineering and Operations Research, University of California, Berkeley, USA（加州大学伯克利分校工业工程与运筹学系）；Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong（香港理工大学应用数学系）
In this paper, a Nash-type fictitious game framework is introduced to handle a time-inconsistent linear-quadratic optimal control. The Nash-type game in this framework is called fictitious as it is between the decision maker (called real player) and an auxiliary control variable (called fictitious player) with the real player and fictitious player looking for time-consistent policy and precomitted optimal policy, respectively. Namely, the fictitious-game framework is actually an auxiliary-variable-based mechanism where the fictitious player is our particular design. Noting that the real player…
In this paper, we formulate a general time-inconsistent stochastic linear--quadratic (LQ) control problem. The time-inconsistency arises from the presence of a quadratic term of the expected state as well as a state-dependent term in the objective functional. We define an equilibrium, instead of optimal, solution within the class of open-loop controls, and derive a sufficient condition for equilibrium controls via a flow of forward--backward stochastic differential equations. When the state is one dimensional and the coefficients in the problem are all deterministic, we find an explicit equili…
We investigate the stability of equilibrium-induced optimal values with respect to (w.r.t.) reward functions $f$ and transition kernels $Q$ for time-inconsistent stopping problems under nonexponential discounting in discrete time. First, with locally uniform convergence of $f$ and $Q$ equipped with total variation distance, we show that the optimal value is semi-continuous w.r.t. $(f,Q)$. We provide examples showing that continuity may fail in general, and the convergence for $Q$ in total variation cannot be replaced by weak convergence. Next we show that with the uniform convergence of $f$ an…
We investigate the stability of the equilibrium-induced optimal value in one-dimensional diffusion setting for a time-inconsistent stopping problem under non-exponential discounting. We show that the optimal value is semi-continuous with respect to the drift, volatility, and reward function. An example is provided showing that the exact continuity may fail. With equilibria extended to $\varepsilon$-equilibria, we establish the relaxed continuity of the optimal value.
A new definition of continuous-time equilibrium controls is introduced. As opposed to the standard definition, which involves a derivative-type operation, the new definition parallels how a discrete-time equilibrium is defined, and allows for unambiguous economic interpretation. The terms "strong equilibria" and "weak equilibria" are coined for controls under the new and the standard definitions, respectively. When the state process is a time-homogeneous continuous-time Markov chain, a careful asymptotic analysis gives complete characterizations of weak and strong equilibria. Thanks to Kakutan…
We study a generic family of two-player continuous-time nonzero-sum stopping games modeling a war of attrition with symmetric information and stochastic payoffs that depend on an homogeneous linear diffusion. We first show that any Markovian mixed strategy for player $i$ can be represented by a pair $(μ^i,S^i)$, where $μ^i$ is a measure over the state space representing player $i$'s stopping intensity, and $S^i$ is a subset of the state space over which player $i$ stops with probability $1$. We then prove that, if players are asymmetric, then, in all mixed-strategy Markov-perfect equilibria, t…
