This paper investigates the problem of designing data-driven stochastic Model Predictive Control (MPC) for linear time-invariant systems under additive stochastic disturbance, whose probability distribution is unknown but can be partially inferred from data. We propose a novel online learning based risk-averse stochastic MPC framework in which Conditional Value-at-Risk (CVaR) constraints on system states are required to hold for a family of distributions called an ambiguity set. The ambiguity set is constructed from disturbance data by leveraging a Dirichlet process mixture model that is self-adaptive to the underlying data structure and complexity. Specifically, the structural property of multimodality is exploit-ed, so that the first- and second-order moment information of each mixture component is incorporated into the ambiguity set. A novel constraint tightening strategy is then developed based on an equivalent reformulation of distributionally ro-bust CVaR constraints over the proposed ambiguity set. As more data are gathered during the runtime of the controller, the ambiguity set is updated online using real-time disturbance data, which enables the risk-averse stochastic MPC to cope with time-varying disturbance distributions. The online variational inference algorithm employed does not require all collected data be learned from scratch, and therefore the proposed MPC is endowed with the guaranteed computational complexity of online learning. The guarantees on recursive feasibility and closed-loop stability of the proposed MPC are established via a safe update scheme. Numerical examples are used to illustrate the effectiveness and advantages of the proposed MPC.
Risk-averse model predictive control (MPC) offers a control framework that allows one to account for ambiguity in the knowledge of the underlying probability distribution and unifies stochastic and worst-case MPC. In this paper we study risk-averse MPC problems for constrained nonlinear Markovian switching systems using generic cost functions, and derive Lyapunov-type risk-averse stability conditions by leveraging the properties of risk-averse dynamic programming operators. We propose a controller design procedure to design risk-averse stabilizing terminal conditions for constrained nonlinear …
In this paper we present a framework for risk-averse model predictive control (MPC) of linear systems affected by multiplicative uncertainty. Our key innovation is to consider time-consistent, dynamic risk metrics as objective functions to be minimized. This framework is axiomatically justified in terms of time-consistency of risk preferences, is amenable to dynamic optimization, and is unifying in the sense that it captures a full range of risk assessments from risk-neutral to worst case. Within this framework, we propose and analyze an online risk-averse MPC algorithm that is provably stabil…
In this paper we present a risk-averse model predictive control (MPC) scheme for the operation of islanded microgrids with very high share of renewable energy sources. The proposed scheme mitigates the effect of errors in the determination of the probability distribution of renewable infeed and load. This allows to use less complex and less accurate forecasting methods and to formulate low-dimensional scenario-based optimisation problems which are suitable for control applications. Additionally, the designer may trade performance for safety by interpolating between the conventional stochastic …
This paper studies risk in a stochastic auction which facilitates the integration of renewable generation in electricity markets. We model market participants who are risk averse and reflect their risk aversion through coherent risk measures. We uncover a closed form characterization of a risk-averse generator's optimal pre-commitment behaviour for a given real-time policy, both with and without risk trading.
Developing safe automated vehicles that can be proactive, safe, and comfortable in mixed traffic requires improved planning methods that are risk-averse and that account for predictions of the motion of other road users. To consider these criteria, in this article, we propose a non-linear model-predictive trajectory generator scheme, which couples the longitudinal and lateral motion of the vehicle to steer the vehicle with minimal risk, while progressing towards the goal state. The proposed method takes into account the infrastructure, surrounding objects, and predictions of the objects' state…
Optimizing management practices and selecting the best cultivar for planting play a significant role in increasing agricultural food production and decreasing environmental footprint. In this study, we develop optimization frameworks under uncertainty using conditional value-at-risk in the stochastic programming objective function. We integrate the crop model, APSIM, and a parallel Bayesian optimization algorithm to optimize the management practices and select the best cultivar at different levels of risk aversion. This approach integrates the power of optimization in determining the best deci…
California Institute of Technology（加州理工学院）；NASA Jet Propulsion Laboratory（美国国家航空航天局喷气推进实验室）
We consider the problem of designing policies for partially observable Markov decision processes (POMDPs) with dynamic coherent risk objectives. Synthesizing risk-averse optimal policies for POMDPs requires infinite memory and thus undecidable. To overcome this difficulty, we propose a method based on bounded policy iteration for designing stochastic but finite state (memory) controllers, which takes advantage of standard convex optimization methods. Given a memory budget and optimality criterion, the proposed method modifies the stochastic finite state controller leading to sub-optimal soluti…
On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measures
Department of Integrated Systems Engineering, The Ohio State University（俄亥俄州立大学整合系统工程系）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）
We consider a Markov decision process subject to model uncertainty in a Bayesian framework, where we assume that the state process is observed but its law is unknown to the observer. In addition, while the state process and the controls are observed at time $t$, the actual cost that may depend on the unknown parameter is not known at time $t$. The controller optimizes total cost by using a family of special risk measures, that we call risk filters and that are appropriately defined to take into account the model uncertainty of the controlled system. These key features lead to non-standard and …
We consider a continuous time Principal-Agent model on a finite time horizon, where we look for the existence of an optimal contract both parties agreed on. Contrary to the main stream, where the principal is modelled as risk-neutral, we assume that both the principal and the agent have exponential utility, and are risk averse with same risk awareness level. Moreover, the agent's quality is unknown and modelled as a filtering term in the problem, which is revealed as time passes by. The principal can not observe the agent's real action, but can only recommend action levels to the agent. Hence,…
Day-ahead scheduling of electricity generation or unit commitment is an important and challenging optimization problem in power systems. Variability in net load arising from the increasing penetration of renewable technologies have motivated study of various classes of stochastic unit commitment models. In two-stage models, the generation schedule for the entire day is fixed while the dispatch is adapted to the uncertainty, whereas in multi-stage models the generation schedule is also allowed to dynamically adapt to the uncertainty realization. Multi-stage models provide more flexibility in th…
We present a method for optimal control of systems governed by partial differential equations (PDEs) with uncertain parameter fields. We consider an objective function that involves the mean and variance of the control objective, leading to a risk-averse optimal control problem. To make the problem tractable, we invoke a quadratic Taylor series approximation of the control objective with respect to the uncertain parameter. This enables deriving explicit expressions for the mean and variance of the control objective in terms of its gradients and Hessians with respect to the uncertain parameter.…
The Ohio State University（俄亥俄州立大学）；University of Michigan（密歇根大学）
Traditional reinforcement learning (RL) aims to maximize the expected total reward, while the risk of uncertain outcomes needs to be controlled to ensure reliable performance in a risk-averse setting. In this paper, we consider the problem of maximizing dynamic risk of a sequence of rewards in infinite-horizon Markov Decision Processes (MDPs). We adapt the Expected Conditional Risk Measures (ECRMs) to the infinite-horizon risk-averse MDP and prove its time consistency. Using a convex combination of expectation and conditional value-at-risk (CVaR) as a special one-step conditional risk measure,…
We introduce adaptive sampling methods for stochastic programs with deterministic constraints. First, we propose and analyze a variant of the stochastic projected gradient method where the sample size used to approximate the reduced gradient is determined on-the-fly and updated adaptively. This method is applicable to a broad class of expectation-based risk measures and leads to a significant reduction in the individual gradient evaluations used to estimate the objective function gradient. Numerical experiments with expected risk minimization and conditional value-at-risk minimization support …
In this paper, we consider the optimal controller design problem against data injection attacks on actuators for an uncertain control system. We consider attacks that aim at maximizing the attack impact while remaining stealthy in the finite horizon. To this end, we use the Conditional Value-at-Risk to characterize the risk associated with the impact of attacks. The worst-case attack impact is characterized using the recently proposed output-to-output $\ell_2$-gain (OOG). We formulate the design problem and observe that it is non-convex and hard to solve. Using the framework of scenario-based …
Politecnico di Milano（米兰理工大学）；Institute for Scientific Interchange Foundation（科学交流基金会）；Banca IMI（意大利投资银行）
In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper,…
We consider two market designs for a network of prosumers, trading energy: (i) a centralized design which acts as a benchmark, and (ii) a peer-to-peer market design. High renewable energy penetration requires that the energy market design properly handles uncertainty. To that purpose, we consider risk neutral models for market designs (i), (ii), and their risk-averse interpretations in which prosumers are endowed with coherent risk measures reflecting heterogeneity in their risk attitudes. We characterize analytically risk-neutral and risk-averse equilibrium in terms of existence and uniquenes…
This paper studies the communication complexity of convex risk-averse optimization over a network. The problem generalizes the well-studied risk-neutral finite-sum distributed optimization problem and its importance stems from the need to handle risk in an uncertain environment. For algorithms in the literature, there exists a gap in communication complexities for solving risk-averse and risk-neutral problems. We propose two distributed algorithms, namely the distributed risk averse optimization (DRAO) method and the distributed risk averse optimization with sliding (DRAO-S) method, to close t…
Multistage risk-averse optimal control problems with nested conditional risk mappings are gaining popularity in various application domains. Risk-averse formulations interpolate between the classical expectation-based stochastic and minimax optimal control. This way, risk-averse problems aim at hedging against extreme low-probability events without being overly conservative. At the same time, risk-based constraints may be employed either as surrogates for chance (probabilistic) constraints or as a robustification of expectation-based constraints. Such multistage problems, however, have been id…
We consider a multiperiod stochastic capacitated facility location problem under uncertain demand and budget in each period. Using a scenario tree representation of the uncertainties, we formulate a multistage stochastic integer program to dynamically locate facilities in each period and compare it with a two-stage approach that determines the facility locations up front. In the multistage model, in each stage, a decision maker optimizes facility locations and recourse flows from open facilities to demand sites, to minimize certain risk measures of the cost associated with current facility loc…
We propose a distributionally robust return-risk model for Markov decision processes (MDPs) under risk and reward ambiguity. The proposed model optimizes the weighted average of mean and percentile performances, and it covers the distributionally robust MDPs and the distributionally robust chance-constrained MDPs (both under reward ambiguity) as special cases. By considering that the unknown reward distribution lies in a Wasserstein ambiguity set, we derive the tractable reformulation for our model. In particular, we show that that the return-risk model can also account for risk from uncertain…
California Institute of Technology（加州理工学院）；NASA Jet Propulsion Laboratory（NASA喷气推进实验室）
A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviat…
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which multiple risk-averse agents choose their decisions in such a way to minimize their individual accumulated risk-costs over a finite-time horizon. In particular, we introduce multi-structure dynamic risk measures induced from conditional $g$-expectations, where the latter are associated with the generator functionals of certain BSDEs that implicitly take into account the risk-cost functionals of the risk-averse agents. Here, we also require that such solutions of the…
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which there are two risk-averse decision makers (i.e., {\it leader} and {\it follower}) with different risk-averse related responsibilities and information. Moreover, we assume that there are two objectives that these decision makers are expected to achieve. That is, the first objective being of {\it stochastic controllability} type that describes an acceptable risk-exposure set vis-á-vis some uncertain future payoff, and while the {\it second one} is making sure the sol…
Increasing penetration of distribution generation (DG) and electric vehicles (EVs) calls for an effective way to estimate the achievable capacity connected to the distribution systems, but the exogenous uncertainties of DG outputs and EV charging loads make it challengeable. This study provides a joint capacity evaluation method with a risk threshold setting function for photovoltaic (PV) generation and EV charging stations (EVCSs). The method is mathematically formulated as a distributionally robust joint chance constrained programming model. And the worst-case conditional value at risk (WC-C…
Having a perfect model to compute the optimal policy is often infeasible in reinforcement learning. It is important in high-stakes domains to quantify and manage risk induced by model uncertainties. Entropic risk measure is an exponential utility-based convex risk measure that satisfies many reasonable properties. In this paper, we propose an entropic risk constrained policy gradient and actor-critic algorithms that are risk-averse to the model uncertainty. We demonstrate the usefulness of our algorithms on several problem domains.
The real challenge in drone-logistics is to develop an economically-feasible Unmanned Aerial Mobility Network (UAMN). In this paper, we propose an integrated airport location (strategic decision) and routes planning (operational decision) optimization framework to minimize the total cost of the network, while guaranteeing flow constraints, capacity constraints, and electricity constraints. To facility expensive long-term infrastructure planning facing demand uncertainty, we develop a data-driven risk-averse two-stage stochastic optimization model based on the Wasserstein distance. We develop a…
Stochastic dominance is a preference relation of uncertain prospect defined over a class of utility functions. While this utility class represents basic properties of risk aversion, it includes some extreme utility functions rarely characterizing a rational decision maker's preference. In this paper we introduce reference-based almost stochastic dominance (RSD) rules which well balance the general representation of risk aversion and the individualization of the decision maker's risk preference. The key idea is that, in the general utility class, we construct a neighborhood of the decision make…
Enforcing safety in the presence of stochastic uncertainty is a challenging problem. Traditionally, researchers have proposed safety in the statistical mean as a safety measure in this case. However, ensuring safety in the statistical mean is only reasonable if system's safe behavior in the large number of runs is of interest, which precludes the use of mean safety in practical scenarios. In this paper, we propose a risk sensitive notion of safety called conditional-value-at-risk (CVaR) safety, which is concerned with safe performance in the worst case realizations. We introduce CVaR barrier f…
For many real-world decision-making problems subject to uncertainty, it may be essential to deal with multiple and often conflicting objectives while taking the decision-makers' risk preferences into account. Conditional value-at-risk (CVaR) is a widely applied risk measure to address risk-averseness of the decision-makers. In this paper, we use the subset-based polyhedral representation of the CVaR to reformulate the bi-objective two-stage stochastic facility location problem presented in Nazemi et al. (2021). We propose an approximate cutting-plane method to deal with this more computational…
This article develops a new algorithm named TTRISK to solve high-dimensional risk-averse optimization problems governed by differential equations (ODEs and/or PDEs) under uncertainty. As an example, we focus on the so-called Conditional Value at Risk (CVaR), but the approach is equally applicable to other coherent risk measures. Both the full and reduced space formulations are considered. The algorithm is based on low rank tensor approximations of random fields discretized using stochastic collocation. To avoid non-smoothness of the objective function underpinning the CVaR, we propose an adapt…
Conditional Value at Risk (CVaR) is widely used to account for the preferences of a risk-averse agent in the extreme loss scenarios. To study the effectiveness of randomization in interdiction games with an interdictor that is both risk and ambiguity averse, we introduce a distributionally robust network interdiction game where the interdictor randomizes over the feasible interdiction plans in order to minimize the worst-case CVaR of the flow with respect to both the unknown distribution of the capacity of the arcs and his mixed strategy over interdicted arcs. The flow player, on the contrary,…
We propose an Adaptive MPC framework for uncertain linear systems to achieve robust satisfaction of state and input constraints. The uncertainty in the system is assumed additive, state dependent, and globally Lipschitz with a known Lipschitz constant. We use a non-parametric technique for online identification of the system uncertainty by approximating its graph via envelopes defined by quadratic constraints. At any given time, by solving a set of convex optimization problems, the MPC controller guarantees robust constraint satisfaction for the closed loop system for all possible values of sy…
Shape Derivative-Informed Neural Operators with Application to Risk-Averse Shape Optimization
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas, USA（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia（数学科学学院，昆士兰科技大学，布里斯班，澳大利亚）；Centre for Data Science, Queensland University of Technology, Brisbane, Australia（数据科学中心，昆士兰科技大学，布里斯班，澳大利亚）；Department of Mathematics, The Ohio State University, Columbus, Ohio, USA（数学系，俄亥俄州立大学，哥伦布，俄亥俄，美国）；Department of Engineering Science and Biomedical Engineering, University of Auckland, Auckland, New Zealand（工程科学与生物医学工程系，奥克兰大学，奥克兰，新西兰）；Walker Department of Mechanical Engineering, The University of Texas at Austin, Austin, Texas, USA（沃克机械工程系，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）
This paper presents a Distributed Stochastic Model Predictive Control algorithm for networks of linear systems with multiplicative uncertainties and local chance constraints on the states and control inputs. The chance constraints are approximated via Cantelli's inequality by means of expected value and covariance. The cooperative control algorithm is based on the distributed Alternating Direction Method of Multipliers, which renders the controller fully distributedly implementable, recursively feasible and ensures point-wise convergence of the states. The aforementioned properties are guarant…
An unconventional approach for optimal stopping under model ambiguity is introduced. Besides ambiguity itself, we take into account how ambiguity-averse an agent is. This inclusion of ambiguity attitude, via an $α$-maxmin nonlinear expectation, renders the stopping problem time-inconsistent. We look for subgame perfect equilibrium stopping policies, formulated as fixed points of an operator. For a one-dimensional diffusion with drift and volatility uncertainty, we show that every equilibrium can be obtained through a fixed-point iteration. This allows us to capture much more diverse behavior, …
