We present a new, tractable method for solving and analyzing risk-aware control problems over finite and infinite, discounted time-horizons where the dynamics of the controlled process are described as a martingale problem. Supposing general Polish state and action spaces, and using generalized, relaxed controls, we state a risk-aware dynamic optimal control problem of minimizing risk of costs described by a generic risk function. We then construct an alternative formulation that takes the form of a nonlinear programming problem, constrained by the dynamic, {i.e.} time-dependent, and linear Kolmogorov forward equation describing the distribution of the state and accumulated costs. We show that the formulations are equivalent, and that the optimal control process can be taken to be Markov in the controlled process state, running costs, and time. We further prove that under additional conditions, the optimal value is attained. An example numeric problem is presented and solved.
We describe an abstract control-theoretic framework in which the validity of the dynamic programming principle can be established in continuous time by a verification of a small number of structural properties. As an application we treat several cases of interest, most notably the lower-hedging and utility-maximization problems of financial mathematics both of which are naturally posed over ``sets of martingale measures''.
Mean field games are studied in the framework of controlled martingale problems, and general existence theorems are proven in which the equilibrium control is Markovian. The framework is flexible enough to include degenerate volatility, which may depend on both the control and the mean field. The objectives need not be strictly convex, and the mean field interactions considered are nonlocal and Wasserstein-continuous. When the volatility is nondegenerate, continuity assumptions may be weakened considerably. The proofs first use relaxed controls to establish existence. Then, using a convexity a…
We consider controlled martingales with bounded steps where the controller is allowed at each step to choose the distribution of the next step, and where the goal is to hit a fixed ball at the origin at time $n$. We show that the algebraic rate of decay (as $n$ increases to infinity) of the value function in the discrete setup coincides with its continuous counterpart, provided a reachability assumption is satisfied. We also study in some detail the uniformly elliptic case and obtain explicit bounds on the rate of decay. This generalizes and improves upon several recent studies of the one dime…
We construct an abstract framework in which the dynamic programming principle (DPP) can be readily proven. It encompasses a broad range of common stochastic control problems in the weak formulation, and deals with problems in the "martingale formulation" with particular ease. We give two illustrations; first, we establish the DPP for general controlled diffusions and show that their value functions are viscosity solutions of the associated Hamilton-Jacobi-Bellman equations under minimal conditions. After that, we show how to treat singular control on the example of the classical monotone-follo…
The ergodic control problem for a non-degenerate controlled diffusion controlled through its drift is considered under a uniform stability condition that ensures the well-posedness of the associated Hamilton-Jacobi-Bellman (HJB) equation. A nonlinear parabolic evolution equation is then proposed as a continuous time continuous state space analog of White's `relative value iteration' algorithm for solving the ergodic dynamic programming equation for the finite state finite action case. Its convergence to the solution of the HJB equation is established using the theory of monotone dynamical syst…
We introduce a Benamou-Brenier formulation for the continuous-time martingale optimal transport problem as a weak length relaxation of its discrete-time counterpart. By the correspondence between classical martingale problems and Fokker-Planck equations, we obtain an equivalent PDE formulation for which basic properties such as existence, duality and geodesic equations can be analytically studied, yielding corresponding results for the stochastic formulation. In the one dimensional case, sufficient conditions for finiteness of the cost are also given and a link between geodesics and porous med…
This paper rigorously connects the problem of optimal control of McKean-Vlasov dynamics with large systems of interacting controlled state processes. Precisely, the empirical distributions of near-optimal control-state pairs for the $n$-state systems, as $n$ tends to infinity, admit limit points in distribution (if the objective functions are suitably coercive), and every such limit is supported on the set of optimal control-state pairs for the McKean-Vlasov problem. Conversely, any distribution on the set of optimal control-state pairs for the McKean-Vlasov problem can be realized as a limit …
This paper considers the problem of controlled invariance of involutive regular distribution, both for smooth and real analytic cases. After a review of some existing work, a precise formulation of the problem of local and global controlled invariance of involutive regular distributions for both affine control systems and affine distributions is introduced. A complete characterization for local controlled invariance of involutive regular distributions for affine control systems is presented. A geometric interpretation for this characterization is provided. A result on local controlled invarian…
Let $G$ be a semimartingale, and $S$ its Snell envelope. Under the assumption that $G\in\mathcal{H}^1$, we show that the finite-variation part of $S$ is absolutely continuous with respect to the decreasing part of the finite-variation part of $G$. In the Markovian setting, this enables us to identify sufficient conditions for the value function of the optimal stopping problem to belong to the domain of the extended (martingale) generator of the underlying Markov process. We then show that the \textit{dual} of the optimal stopping problem is a stochastic control problem for a controlled Markov …
Consider a multi-agent network comprised of risk averse social sensors and a controller that jointly seek to estimate an unknown state of nature, given noisy measurements. The network of social sensors perform Bayesian social learning - each sensor fuses the information revealed by previous social sensors along with its private valuation using Bayes' rule - to optimize a local cost function. The controller sequentially modifies the cost function of the sensors by discriminatory pricing (control inputs) to realize long term global objectives. We formulate the stochastic control problem faced by…
The problem of robust hedging requires to solve the problem of superhedging under a nondominated family of singular measures. Recent progress was achieved by [9,11]. We show that the dual formulation of this problem is valid in a context suitable for martingale optimal transportation or, more generally, for optimal transportation under controlled stochastic dynamics.
We develop the linear programming approach to mean-field games in a general setting. This relaxed control approach allows to prove existence results under weak assumptions, and lends itself well to numerical implementation. We consider mean-field game problems where the representative agent chooses both the optimal control and the optimal time to exit the game, where the instantaneous reward function and the coefficients of the state process may depend on the distribution of the other agents. Furthermore, we establish the equivalence between mean-field games equilibria obtained by the linear p…
Research in psychology and neuroscience has successfully modeled decision making as a process of noisy evidence accumulation to a decision bound. While there are several variants and implementations of this idea, the majority of these models make use of a noisy accumulation between two absorbing boundaries. A common assumption of these models is that decision parameters, e.g., the rate of accumulation (drift rate), remain fixed over the course of a decision, allowing the derivation of analytic formulas for the probabilities of hitting the upper or lower decision threshold, and the mean decisio…
By the classical Martingale Representation Theorem, replication of random vectors can be achieved via stochastic integrals or solutions of stochastic differential equations. We introduce a new approach to replication of random vectors via adapted differentiable processes generated by a controlled ordinary differential equation. We found that the solution of this replication problem exists and is not unique. This leads to a new optimal control problem: find a replicating process that is minimal in an integral norm. We found an explicit solution of this problem. Possible applications to portfoli…
This paper proves the existence of optimal stopping times via elementary functional analytic arguments. The problem is first relaxed into a convex optimization problem over a closed convex subset of the unit ball of the dual of a Banach space. The existence of optimal solutions then follows from the Banach--Alaoglu compactness theorem and the Krein--Millman theorem on extreme points of convex sets. This approach seems to give the most general existence results known to date. Applying convex duality to the relaxed problem gives a dual problem and optimality conditions in terms of martingales th…
