We consider reinforcement learning with performance evaluated by a dynamic risk measure. We construct a projected risk-averse dynamic programming equation and study its properties. Then we propose risk-averse counterparts of the methods of temporal differences and we prove their convergence with probability one. We also perform an empirical study on a complex transportation problem.
The Ohio State University（俄亥俄州立大学）；University of Michigan（密歇根大学）
Traditional reinforcement learning (RL) aims to maximize the expected total reward, while the risk of uncertain outcomes needs to be controlled to ensure reliable performance in a risk-averse setting. In this paper, we consider the problem of maximizing dynamic risk of a sequence of rewards in infinite-horizon Markov Decision Processes (MDPs). We adapt the Expected Conditional Risk Measures (ECRMs) to the infinite-horizon risk-averse MDP and prove its time consistency. Using a convex combination of expectation and conditional value-at-risk (CVaR) as a special one-step conditional risk measure,…
On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measures
Department of Integrated Systems Engineering, The Ohio State University（俄亥俄州立大学整合系统工程系）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）
Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criteria, namely the CVaR. In particular, we present the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns policies with higher CVaR than risk-neutr…
We propose a distributionally robust return-risk model for Markov decision processes (MDPs) under risk and reward ambiguity. The proposed model optimizes the weighted average of mean and percentile performances, and it covers the distributionally robust MDPs and the distributionally robust chance-constrained MDPs (both under reward ambiguity) as special cases. By considering that the unknown reward distribution lies in a Wasserstein ambiguity set, we derive the tractable reformulation for our model. In particular, we show that that the return-risk model can also account for risk from uncertain…
In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. A risk measure often focuses on the worst returns out of the agent's experience. As a result, standard methods for risk-averse RL often ignore high-return strategies. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We also devise a novel Cross Entropy module for risk sampling, which (1) preserves risk aversion despite the soft risk; (2) independently improves sample efficiency. By separating the risk avers…
University of Oxford（牛津大学）；Auburn University（奥本大学）
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperfo…
Ecole polytechnique（巴黎综合理工学院）；INRIA（法国国家信息与自动化研究所）；INSERM（法国国家健康与医学研究院）；Sorbonne Université（索邦大学）；Université Paris Cité（巴黎西岱大学）
Robust Reinforcement Learning tries to make predictions more robust to changes in the dynamics or rewards of the system. This problem is particularly important when the dynamics and rewards of the environment are estimated from the data. In this paper, we approximate the Robust Reinforcement Learning constrained with a $Φ$-divergence using an approximate Risk-Averse formulation. We show that the classical Reinforcement Learning formulation can be robustified using standard deviation penalization of the objective. Two algorithms based on Distributional Reinforcement Learning, one for discrete a…
This paper tackles the risk averse multi-armed bandits problem when incurred losses are non-stationary. The conditional value-at-risk (CVaR) is used as the objective function. Two estimation methods are proposed for this objective function in the presence of non-stationary losses, one relying on a weighted empirical distribution of losses and another on the dual representation of the CVaR. Such estimates can then be embedded into classic arm selection methods such as epsilon-greedy policies. Simulation experiments assess the performance of the arm selection algorithms based on the two novel es…
In this work, we address risk-averse Bayes-adaptive reinforcement learning. We pose the problem of optimising the conditional value at risk (CVaR) of the total return in Bayes-adaptive Markov decision processes (MDPs). We show that a policy optimising CVaR in this setting is risk-averse to both the parametric uncertainty due to the prior distribution over MDPs, and the internal uncertainty due to the inherent stochasticity of MDPs. We reformulate the problem as a two-player stochastic game and propose an approximate algorithm based on Monte Carlo tree search and Bayesian optimisation. Our expe…
Department of Computer Science University of New Hampshire（计算机科学系新罕布什尔大学）；Department of Computing & Mathematical Sciences California Institute of Technology（计算与数学科学系加州理工学院）；Tencent AI Lab（腾讯AI实验室）
KTH Royal Institute of Technology（KTH皇家理工学院）；Duke University（杜克大学）
We consider an online stochastic game with risk-averse agents whose goal is to learn optimal decisions that minimize the risk of incurring significantly high costs. Specifically, we use the Conditional Value at Risk (CVaR) as a risk measure that the agents can estimate using bandit feedback in the form of the cost values of only their selected actions. Since the distributions of the cost functions depend on the actions of all agents that are generally unobservable, they are themselves unknown and, therefore, the CVaR values of the costs are difficult to compute. To address this challenge, we p…
Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
Data 61, CSIRO, Australia（数据61，CSIRO，澳大利亚）；School of Computing, Macquarie University, Australia（计算学院，麦考瑞大学，澳大利亚）；Adobe Research, USA（Adobe研究，美国）；School of Computer Science and Engineering, University of New South Wales, Australia（计算机科学与工程学院，新南威尔士大学，澳大利亚）
One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distrib…
The multi-armed bandit problems have been studied mainly under the measure of expected total reward accrued over a horizon of length $T$. In this paper, we address the issue of risk in multi-armed bandit problems and develop parallel results under the measure of mean-variance, a commonly adopted risk measure in economics and mathematical finance. We show that the model-specific regret and the model-independent regret in terms of the mean-variance of the reward process are lower bounded by $Ω(\log T)$ and $Ω(T^{2/3})$, respectively. We then show that variations of the UCB policy and the DSEE po…
We introduce a distributional method for learning the optimal policy in risk averse Markov decision process with finite state action spaces, latent costs, and stationary dynamics. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures constructed from nested Kusuoka-type conditional risk mappings. For such performance criteria, randomized policies may outperform deterministic policies, therefore, the candidate policies lie in the d-dimensional simplex where d is the cardinality of the action space. Existing risk averse…
Politecnico di Milano（米兰理工大学）；Institute for Scientific Interchange Foundation（科学交流基金会）；Banca IMI（意大利投资银行）
In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper,…
CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）；LRI（计算机研究实验室）；Université Paris Sud（巴黎第十一大学）
Motivated by applications in energy management, this paper presents the Multi-Armed Risk-Aware Bandit (MARAB) algorithm. With the goal of limiting the exploration of risky arms, MARAB takes as arm quality its conditional value at risk. When the user-supplied risk level goes to 0, the arm quality tends toward the essential infimum of the arm distribution density, and MARAB tends toward the MIN multi-armed bandit algorithm, aimed at the arm with maximal minimal value. As a first contribution, this paper presents a theoretical analysis of the MIN algorithm under mild assumptions, establishing its…
Intesa Sanpaolo（联合圣保罗银行）；Politecnico di Milano（米兰理工大学）
In this paper we show how risk-averse reinforcement learning can be used to hedge options. We apply a state-of-the-art risk-averse algorithm: Trust Region Volatility Optimization (TRVO) to a vanilla option hedging environment, considering realistic factors such as discrete time and transaction costs. Realism makes the problem twofold: the agent must both minimize volatility and contain transaction costs, these tasks usually being in competition. We use the algorithm to train a sheaf of agents each characterized by a different risk aversion, so to be able to span an efficient frontier on the vo…
In this paper, we consider a risk-averse multi-armed bandit (MAB) problem where the goal is to learn a policy that minimizes the risk of low expected return, as opposed to maximizing the expected return itself, which is the objective in the usual approach to risk-neutral MAB. Specifically, we formulate this problem as a transfer learning problem between an expert and a learner agent in the presence of contexts that are only observable by the expert but not by the learner. Thus, such contexts are unobserved confounders (UCs) from the learner's perspective. Given a dataset generated by the exper…
This paper unifies the design and the analysis of risk-averse Thompson sampling algorithms for the multi-armed bandit problem for a class of risk functionals $ρ$ that are continuous and dominant. We prove generalised concentration bounds for these continuous and dominant risk functionals and show that a wide class of popular risk functionals belong to this class. Using our newly developed analytical toolkits, we analyse the algorithm $ρ$-MTS (for multinomial distributions) and prove that they admit asymptotically optimal regret bounds of risk-averse algorithms under CVaR, proportional hazard, …
Deep reinforcement learning has recently made significant progress in solving computer games and robotic control tasks. A known problem, though, is that policies overfit to the training environment and may not avoid rare, catastrophic events such as automotive accidents. A classical technique for improving the robustness of reinforcement learning algorithms is to train on a set of randomized environments, but this approach only guards against common situations. Recently, robust adversarial reinforcement learning (RARL) was developed, which allows efficient applications of random and systematic…
KTH Royal Institute of Technology（KTH皇家理工学院）；Digital Futures（数字未来中心）；Duke University（杜克大学）；Air Force Research Laboratory（美国空军研究实验室）
We consider risk-averse learning in repeated unknown games where the goal of the agents is to minimize their individual risk of incurring significantly high cost. Specifically, the agents use the conditional value at risk (CVaR) as a risk measure and rely on bandit feedback in the form of the cost values of the selected actions at every episode to estimate their CVaR values and update their actions. A major challenge in using bandit feedback to estimate CVaR is that the agents can only access their own cost values, which, however, depend on the actions of all agents. To address this challenge,…
Measures of Variability for Risk-averse Policy Gradient
School of Computer Science, University of Waterloo（滑铁卢大学计算机科学学院）；Vector Institute（矢量研究所）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；School of Computing Science, Simon Fraser University（西蒙菲莎大学计算科学学院）
In this paper we consider the contextual multi-armed bandit problem for linear payoffs under a risk-averse criterion. At each round, contexts are revealed for each arm, and the decision maker chooses one arm to pull and receives the corresponding reward. In particular, we consider mean-variance as the risk criterion, and the best arm is the one with the largest mean-variance reward. We apply the Thompson Sampling algorithm for the disjoint model, and provide a comprehensive regret analysis for a variant of the proposed algorithm. For $T$ rounds, $K$ actions, and $d$-dimensional feature vectors…
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（决策与控制系统系，电气工程与计算机科学学院，皇家理工学院）；Chair of Information-oriented Control (ITR), Technical University of Munich（信息导向控制系（ITR），慕尼黑技术大学）
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（决策与控制系统系，电气工程与计算机科学学院，皇家理工学院）
Risk-averse model predictive control (MPC) offers a control framework that allows one to account for ambiguity in the knowledge of the underlying probability distribution and unifies stochastic and worst-case MPC. In this paper we study risk-averse MPC problems for constrained nonlinear Markovian switching systems using generic cost functions, and derive Lyapunov-type risk-averse stability conditions by leveraging the properties of risk-averse dynamic programming operators. We propose a controller design procedure to design risk-averse stabilizing terminal conditions for constrained nonlinear …
We use Markov risk measures to formulate a risk-averse version of the undiscounted total cost problem for a transient controlled Markov process. We derive risk-averse dynamic programming equations and we show that a randomized policy may be strictly better than deterministic policies, when risk measures are employed. We illustrate the results on an optimal stopping problem and an organ transplant problem.
In this paper, we consider a risk-averse control problem for diffusion processes, in which there is a partition of the admissible control strategy into two decision-making groups (namely, the {\it leader} and {\it follower}) with different cost functionals and risk-averse satisfactions. Our approach, based on a hierarchical optimization framework, requires that a certain level of risk-averse satisfaction be achieved for the {\it leader} as a priority over that of the {\it follower's} risk-averseness. In particular, we formulate such a risk-averse control problem involving a family of time-cons…
This paper features expectiles in dynamic and stochastic optimization. Expectiles are a family of risk functionals characterized as minimizers of optimization problems. For this reason, they enjoy various unique stability properties, which can be exploited in risk averse management, in stochastic optimization and in optimal control. The paper provides tight relates of expectiles to other risk functionals and addresses their properties in regression. Further, we extend expectiles to a dynamic framework. As such, they allow incorporating a risk averse aspect in continuous-time dynamic optimizati…
This paper studies risk in a stochastic auction which facilitates the integration of renewable generation in electricity markets. We model market participants who are risk averse and reflect their risk aversion through coherent risk measures. We uncover a closed form characterization of a risk-averse generator's optimal pre-commitment behaviour for a given real-time policy, both with and without risk trading.
This paper investigates the use of risk measures and theories of choice for modeling risk-averse route choice and traffic network equilibrium with random travel times. We interpret the postulates of these theories in the context of routing, and we identify additive consistency as a plausible and relevant condition that allows to reduce risk-averse route choice to a standard shortest path problem. Within the classical theories of choice under risk, we show that the only preferences that satisfy this consistency property are the ones induced by the entropic risk measures.
This paper studies the communication complexity of convex risk-averse optimization over a network. The problem generalizes the well-studied risk-neutral finite-sum distributed optimization problem and its importance stems from the need to handle risk in an uncertain environment. For algorithms in the literature, there exists a gap in communication complexities for solving risk-averse and risk-neutral problems. We propose two distributed algorithms, namely the distributed risk averse optimization (DRAO) method and the distributed risk averse optimization with sliding (DRAO-S) method, to close t…
The aim of this paper is to show that in some cases risk averse multistage stochastic programming problems can be reformulated in a form of risk neutral setting. This is achieved by a change of the reference probability measure making ``bad" (extreme) scenarios more frequent. As a numerical example we demonstrate advantages of such change-of-measure approach applied to the Brazilian Interconnected Power System operation planning problem.
Multistage risk-averse optimal control problems with nested conditional risk mappings are gaining popularity in various application domains. Risk-averse formulations interpolate between the classical expectation-based stochastic and minimax optimal control. This way, risk-averse problems aim at hedging against extreme low-probability events without being overly conservative. At the same time, risk-based constraints may be employed either as surrogates for chance (probabilistic) constraints or as a robustification of expectation-based constraints. Such multistage problems, however, have been id…
We develop a new approach to solving classification problems, which is bases on the theory of coherent measures of risk and risk sharing ideas. The proposed approach aims at designing a risk-averse classifier. The new approach allows for associating distinct risk functional to each classes. The risk may be measured by different (non-linear in probability) measures, We analyze the structure of the new classifier design problem and establish its theoretical relation to known risk-neutral design problems. In particular, we show that the risk-sharing classification problem is equivalent to an impl…
Two approaches to time consistency of risk averse multistage stochastic problems were discussed in the recent literature. In one approach certain properties of the cor-responding risk measure are postulated which imply its decomposability. The other approach deals directly with conditional optimality of solutions of the considered problem. The aim of this paper is to discuss a relation between these two approaches.
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which multiple risk-averse agents choose their decisions in such a way to minimize their individual accumulated risk-costs over a finite-time horizon. In particular, we introduce multi-structure dynamic risk measures induced from conditional $g$-expectations, where the latter are associated with the generator functionals of certain BSDEs that implicitly take into account the risk-cost functionals of the risk-averse agents. Here, we also require that such solutions of the…
The multi-armed bandit (MAB) is a classical online optimization model for the trade-off between exploration and exploitation. The traditional MAB is concerned with finding the arm that minimizes the mean cost. However, minimizing the mean does not take the risk of the problem into account. We now want to accommodate risk-averse decision makers. In this work, we introduce a coherent risk measure as the criterion to form a risk-averse MAB. In particular, we derive an index-based online sampling framework for the risk-averse MAB. We develop this framework in detail for three specific risk measure…
Optimizing management practices and selecting the best cultivar for planting play a significant role in increasing agricultural food production and decreasing environmental footprint. In this study, we develop optimization frameworks under uncertainty using conditional value-at-risk in the stochastic programming objective function. We integrate the crop model, APSIM, and a parallel Bayesian optimization algorithm to optimize the management practices and select the best cultivar at different levels of risk aversion. This approach integrates the power of optimization in determining the best deci…
In this paper, we consider a risk-averse decision problem for controlled-diffusion processes, with dynamic risk measures, in which there are two risk-averse decision makers (i.e., {\it leader} and {\it follower}) with different risk-averse related responsibilities and information. Moreover, we assume that there are two objectives that these decision makers are expected to achieve. That is, the first objective being of {\it stochastic controllability} type that describes an acceptable risk-exposure set vis-á-vis some uncertain future payoff, and while the {\it second one} is making sure the sol…
We propose and investigate a discrete-time mean field game model involving risk-averse agents. The model under study is a coupled system of dynamic programming equations with a Kolmogorov equation. The agents' risk aversion is modeled by composite risk measures. The existence of a solution to the coupled system is obtained with a fixed point approach. The corresponding feedback control allows to construct an approximate Nash equilibrium for a related dynamic game with finitely many players.
We discuss risked competitive partial equilibrium in a setting in which agents are endowed with coherent risk measures. In contrast to socialplanning models, we show by example that risked equilibria are not unique, even when agents' objective functions are strictly concave. We also show that standard computational methods find only a subset of the equilibria, even with multiple starting points.
We study statistical properties of the optimal value and optimal solutions of the Sample Average Approximation of risk averse stochastic problems. Central Limit Theorem type results are derived for the optimal value and optimal solutions when the stochastic program is expressed in terms of a law invariant coherent risk measure. The obtained results are applied to hypotheses testing problems aiming at comparing the optimal values of several risk averse convex stochastic programs on the basis of samples of the underlying random vectors. We also consider non-asymptotic tests based on confidence i…
