In this paper, we consider a risk-averse control problem for diffusion processes, in which there is a partition of the admissible control strategy into two decision-making groups (namely, the {\it leader} and {\it follower}) with different cost functionals and risk-averse satisfactions. Our approach, based on a hierarchical optimization framework, requires that a certain level of risk-averse satisfaction be achieved for the {\it leader} as a priority over that of the {\it follower's} risk-averseness. In particular, we formulate such a risk-averse control problem involving a family of time-consistent dynamic convex risk measures induced by conditional $g$-expectations (i.e., filtration-consistent nonlinear expectations associated with the generators of certain backward stochastic differential equations). Moreover, under suitable conditions, we establish the existence of optimal risk-averse solutions, in the sense of viscosity solutions, for the corresponding risk-averse dynamic programming equations. Finally, we briefly comment on the implication of our results.
Risk-averse model predictive control (MPC) offers a control framework that allows one to account for ambiguity in the knowledge of the underlying probability distribution and unifies stochastic and worst-case MPC. In this paper we study risk-averse MPC problems for constrained nonlinear Markovian switching systems using generic cost functions, and derive Lyapunov-type risk-averse stability conditions by leveraging the properties of risk-averse dynamic programming operators. We propose a controller design procedure to design risk-averse stabilizing terminal conditions for constrained nonlinear …
This paper features expectiles in dynamic and stochastic optimization. Expectiles are a family of risk functionals characterized as minimizers of optimization problems. For this reason, they enjoy various unique stability properties, which can be exploited in risk averse management, in stochastic optimization and in optimal control. The paper provides tight relates of expectiles to other risk functionals and addresses their properties in regression. Further, we extend expectiles to a dynamic framework. As such, they allow incorporating a risk averse aspect in continuous-time dynamic optimizati…
This paper studies risk in a stochastic auction which facilitates the integration of renewable generation in electricity markets. We model market participants who are risk averse and reflect their risk aversion through coherent risk measures. We uncover a closed form characterization of a risk-averse generator's optimal pre-commitment behaviour for a given real-time policy, both with and without risk trading.
The aim of this paper is to show that in some cases risk averse multistage stochastic programming problems can be reformulated in a form of risk neutral setting. This is achieved by a change of the reference probability measure making ``bad" (extreme) scenarios more frequent. As a numerical example we demonstrate advantages of such change-of-measure approach applied to the Brazilian Interconnected Power System operation planning problem.
Multistage risk-averse optimal control problems with nested conditional risk mappings are gaining popularity in various application domains. Risk-averse formulations interpolate between the classical expectation-based stochastic and minimax optimal control. This way, risk-averse problems aim at hedging against extreme low-probability events without being overly conservative. At the same time, risk-based constraints may be employed either as surrogates for chance (probabilistic) constraints or as a robustification of expectation-based constraints. Such multistage problems, however, have been id…
Two approaches to time consistency of risk averse multistage stochastic problems were discussed in the recent literature. In one approach certain properties of the cor-responding risk measure are postulated which imply its decomposability. The other approach deals directly with conditional optimality of solutions of the considered problem. The aim of this paper is to discuss a relation between these two approaches.
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which multiple risk-averse agents choose their decisions in such a way to minimize their individual accumulated risk-costs over a finite-time horizon. In particular, we introduce multi-structure dynamic risk measures induced from conditional $g$-expectations, where the latter are associated with the generator functionals of certain BSDEs that implicitly take into account the risk-cost functionals of the risk-averse agents. Here, we also require that such solutions of the…
The multi-armed bandit (MAB) is a classical online optimization model for the trade-off between exploration and exploitation. The traditional MAB is concerned with finding the arm that minimizes the mean cost. However, minimizing the mean does not take the risk of the problem into account. We now want to accommodate risk-averse decision makers. In this work, we introduce a coherent risk measure as the criterion to form a risk-averse MAB. In particular, we derive an index-based online sampling framework for the risk-averse MAB. We develop this framework in detail for three specific risk measure…
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which there are two risk-averse decision makers (i.e., {\it leader} and {\it follower}) with different risk-averse related responsibilities and information. Moreover, we assume that there are two objectives that these decision makers are expected to achieve. That is, the first objective being of {\it stochastic controllability} type that describes an acceptable risk-exposure set vis-á-vis some uncertain future payoff, and while the {\it second one} is making sure the sol…
We propose and investigate a discrete-time mean field game model involving risk-averse agents. The model under study is a coupled system of dynamic programming equations with a Kolmogorov equation. The agents' risk aversion is modeled by composite risk measures. The existence of a solution to the coupled system is obtained with a fixed point approach. The corresponding feedback control allows to construct an approximate Nash equilibrium for a related dynamic game with finitely many players.
We discuss risked competitive partial equilibrium in a setting in which agents are endowed with coherent risk measures. In contrast to socialplanning models, we show by example that risked equilibria are not unique, even when agents' objective functions are strictly concave. We also show that standard computational methods find only a subset of the equilibria, even with multiple starting points.
We study statistical properties of the optimal value and optimal solutions of the Sample Average Approximation of risk averse stochastic problems. Central Limit Theorem type results are derived for the optimal value and optimal solutions when the stochastic program is expressed in terms of a law invariant coherent risk measure. The obtained results are applied to hypotheses testing problems aiming at comparing the optimal values of several risk averse convex stochastic programs on the basis of samples of the underlying random vectors. We also consider non-asymptotic tests based on confidence i…
In this paper we present a framework for risk-averse model predictive control (MPC) of linear systems affected by multiplicative uncertainty. Our key innovation is to consider time-consistent, dynamic risk metrics as objective functions to be minimized. This framework is axiomatically justified in terms of time-consistency of risk preferences, is amenable to dynamic optimization, and is unifying in the sense that it captures a full range of risk assessments from risk-neutral to worst case. Within this framework, we propose and analyze an online risk-averse MPC algorithm that is provably stabil…
We introduce adaptive sampling methods for stochastic programs with deterministic constraints. First, we propose and analyze a variant of the stochastic projected gradient method where the sample size used to approximate the reduced gradient is determined on-the-fly and updated adaptively. This method is applicable to a broad class of expectation-based risk measures and leads to a significant reduction in the individual gradient evaluations used to estimate the objective function gradient. Numerical experiments with expected risk minimization and conditional value-at-risk minimization support …
Risk-averse optimal control problems have gained a lot of attention in the last decade, mostly due to their attractive mathematical properties and practical importance. They can be seen as an interpolation between stochastic and robust optimal control approaches, allowing the designer to trade-off performance for robustness and vice-versa. Due to their stochastic nature, risk-averse problems are of a very large scale, involving millions of decision variables, which poses a challenge in terms of efficient computation. In this work, we propose a splitting for general risk-averse problems and sho…
Storage is expected to be a critical source of firming in low-carbon grids. A common concern raised from ex-post assessments is that storage resources can fail to respond to strong price signals during times of scarcity. While commonly attributed to forecast error or failures in operations, we posit that this behavior can be explained from the perspective of risk-averse scheduling. Using a stochastic self-scheduling framework and real-world data harvested from the Australian National Electricity Market, we demonstrate that risk-averse storage resources tend to have a myopic operational perspec…
Bellman formulated a vague principle for optimization over time, which characterizes optimal policies by stating that a decision maker should not regret previous decisions retrospectively. This paper addresses time consistency in stochastic optimization. The problem is stated in generality first. The paper discusses time consistent decision-making by addressing risk measures which are recursive, nested, dynamically or time consistent and introduces stopping time risk measures. It turns out that the paradigm of time consistency is in conflict with various desirable, classical properties of gene…
This paper addresses risk awareness of stochastic optimization problems. Nested risk measures appear naturally in this context, as they allow beneficial reformulations for algorithmic treatments. The reformulations presented extend usual Hamilton-Jacobi-Bellman equations in dynamic optimization by involving risk awareness in the problem formulation. Nested risk measures are built on risk measures, which originate by conditioning on the history of a stochastic process. We derive martingale properties of these risk measures and use them to prove continuity. It is demonstrated that stochastic opt…
We consider risk-averse convex stochastic programs expressed in terms of extended polyhedral risk measures. We derive computable confidence intervals on the optimal value of such stochastic programs using the Robust Stochastic Approximation and the Stochastic Mirror Descent (SMD) algorithms. When the objective functions are uniformly convex, we also propose a multistep extension of the Stochastic Mirror Descent algorithm and obtain confidence intervals on both the optimal values and optimal solutions. Numerical simulations show that our confidence intervals are much less conservative and are q…
In this paper we present a risk-averse model predictive control (MPC) scheme for the operation of islanded microgrids with very high share of renewable energy sources. The proposed scheme mitigates the effect of errors in the determination of the probability distribution of renewable infeed and load. This allows to use less complex and less accurate forecasting methods and to formulate low-dimensional scenario-based optimisation problems which are suitable for control applications. Additionally, the designer may trade performance for safety by interpolating between the conventional stochastic …
We consider optimal control problems for diffusion processes, where the objective functional is defined by a time-consistent dynamic risk measure. We focus on coherent risk measures defined by $g$-evaluations. For such problems, we construct a family of time and space perturbed systems with piecewise-constant control functions. We obtain a regularized optimal value function by a special mollification procedure. This allows us to establish a bound on the difference between the optimal value functions of the original problem and of the problem with piecewise-constant controls.
We consider a class of sampling-based decomposition methods to solve risk-averse multistage stochastic convex programs. We prove a formula for the computation of the cuts necessary to build the outer linearizations of the recourse functions. This formula can be used to obtain an efficient implementation of Stochastic Dual Dynamic Programming applied to convex nonlinear problems. We prove the almost sure convergence of these decomposition methods when the relatively complete recourse assumption holds. We also prove the almost sure convergence of these algorithms when applied to risk-averse mult…
The vast majority of the literature on stochastic semidefinite programs (stochastic SDPs) with recourse is concerned with risk-neutral models. In this paper, we introduce mean-risk models for stochastic SDPs and study structural properties as convexity and (Lipschitz) continuity. Special emphasis is placed on stability with respect to changes of the underlying probability distribution. Perturbations of the true distribution may arise from incomplete information or working with (finite discrete) approximations for the sake of computational efficiency. We discuss extended formulations for stocha…
We consider reinforcement learning with performance evaluated by a dynamic risk measure. We construct a projected risk-averse dynamic programming equation and study its properties. Then we propose risk-averse counterparts of the methods of temporal differences and we prove their convergence with probability one. We also perform an empirical study on a complex transportation problem.
The Ohio State University（俄亥俄州立大学）；University of Michigan（密歇根大学）
Traditional reinforcement learning (RL) aims to maximize the expected total reward, while the risk of uncertain outcomes needs to be controlled to ensure reliable performance in a risk-averse setting. In this paper, we consider the problem of maximizing dynamic risk of a sequence of rewards in infinite-horizon Markov Decision Processes (MDPs). We adapt the Expected Conditional Risk Measures (ECRMs) to the infinite-horizon risk-averse MDP and prove its time consistency. Using a convex combination of expectation and conditional value-at-risk (CVaR) as a special one-step conditional risk measure,…
We consider bilevel linear problems, where some parameters are stochastic, and the leader has to decide in a here-and-now fashion, while the follower has complete information. In this setting, the leader's outcome can be modeled by a random variable, which we evaluate based on some law-invariant convex risk measure. A qualitative stability result under perturbations of the underlying probability distribution is presented. Moreover, for the expectation, the expected excess, and the upper semideviation, we establish Lipschitz continuity as well as sufficient conditions for differentiability. Fin…
Developing safe automated vehicles that can be proactive, safe, and comfortable in mixed traffic requires improved planning methods that are risk-averse and that account for predictions of the motion of other road users. To consider these criteria, in this article, we propose a non-linear model-predictive trajectory generator scheme, which couples the longitudinal and lateral motion of the vehicle to steer the vehicle with minimal risk, while progressing towards the goal state. The proposed method takes into account the infrastructure, surrounding objects, and predictions of the objects' state…
Increasing penetration of distribution generation (DG) and electric vehicles (EVs) calls for an effective way to estimate the achievable capacity connected to the distribution systems, but the exogenous uncertainties of DG outputs and EV charging loads make it challengeable. This study provides a joint capacity evaluation method with a risk threshold setting function for photovoltaic (PV) generation and EV charging stations (EVCSs). The method is mathematically formulated as a distributionally robust joint chance constrained programming model. And the worst-case conditional value at risk (WC-C…
This paper generalizes results concerning strong convexity of two-stage mean-risk models with linear recourse to distortion risk measures. Introducing the concept of (restricted) partial strong convexity, we conduct an in-depth analysis of the expected excess functional with respect to the decision variable and the threshold parameter. These results allow to derive sufficient conditions for strong convexity of models building on the conditional value-at-risk due to its variational representation. Via Kusuoka representation these carry over to comonotonic and distortion risk measures, where we …
We consider a multiperiod stochastic capacitated facility location problem under uncertain demand and budget in each period. Using a scenario tree representation of the uncertainties, we formulate a multistage stochastic integer program to dynamically locate facilities in each period and compare it with a two-stage approach that determines the facility locations up front. In the multistage model, in each stage, a decision maker optimizes facility locations and recourse flows from open facilities to demand sites, to minimize certain risk measures of the cost associated with current facility loc…
Stochastic dominance is a preference relation of uncertain prospect defined over a class of utility functions. While this utility class represents basic properties of risk aversion, it includes some extreme utility functions rarely characterizing a rational decision maker's preference. In this paper we introduce reference-based almost stochastic dominance (RSD) rules which well balance the general representation of risk aversion and the individualization of the decision maker's risk preference. The key idea is that, in the general utility class, we construct a neighborhood of the decision make…
In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computation…
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（决策与控制系统系，电气工程与计算机科学学院，皇家理工学院）；Chair of Information-oriented Control (ITR), Technical University of Munich（信息导向控制系（ITR），慕尼黑技术大学）
We present a method for optimal control of systems governed by partial differential equations (PDEs) with uncertain parameter fields. We consider an objective function that involves the mean and variance of the control objective, leading to a risk-averse optimal control problem. To make the problem tractable, we invoke a quadratic Taylor series approximation of the control objective with respect to the uncertain parameter. This enables deriving explicit expressions for the mean and variance of the control objective in terms of its gradients and Hessians with respect to the uncertain parameter.…
The paper deals with a risk averse dynamic programming problem with infinite horizon. First, the required assumptions are formulated to have the problem well defined. Then the Bellman equation is derived, which may be also seen as a standalone reinforcement learning problem. The fact that the Bellman operator is contraction is proved, guaranteeing convergence of various solution algorithms used for dynamic programming as well as reinforcement learning problems, which we demonstrate on the value iteration algorithm.
