On the relative value iteration with a risk-sensitive criterion

Discrete-time Zero-Sum Games for Markov chains with risk-sensitive average cost criterion

We study zero-sum stochastic games for controlled discrete time Markov chains with risk-sensitive average cost criterion with countable state space and Borel action spaces. The payoff function is nonnegative and possibly unbounded. Under a certain Lyapunov stability assumption on the dynamics, we establish the existence of a value and saddle point equilibrium. Further we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. Finally, we present and analyze an illustrative example.

Risk-sensitive discounted cost criterion for Continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton-Jacobi-Bellman (HJB) equation. Also we prove the existence of optimal risk-sensitive control in the class of Markov control.

迈向高效的风险敏感策略梯度：迭代复杂度分析

Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

针对风险敏感策略梯度迭代复杂度研究不足的问题，对采用指数效用函数的REINFORCE算法开展严格分析，证明其达ε-近似一阶平稳点的迭代复杂度为O(ε⁻²)，且相比风险中性算法具备更快收敛潜力，实验验证了该结论。

Zero-Sum Games for Continuous-time Markov Decision Processes with Risk-Sensitive Average Cost Criterion

We consider zero-sum stochastic games for continuous time Markov decision processes with risk-sensitive average cost criterion. Here the transition and cost rates may be unbounded. We prove the existence of the value of the game and a saddle-point equilibrium in the class of all stationary strategies under a Lyapunov stability condition. This is accomplished by establishing the existence of a principal eigenpair for the corresponding Hamilton-Jacobi-Isaacs (HJI) equation. This in turn is established by using the nonlinear version of Krein-Rutman theorem. We then obtain a characterization of th…

On the convergence of a Risk Sensitive like Filter

In this paper, we analyze the convergence of a risk sensitive like filter where the risk sensitivity parameter is time varying. Such filter has a Kalman like structure and its gain matrix is updated according to a Riccati like iteration. We show that the iteration converges to a fixed point by using the contraction analysis.

Stochastic Primal-Dual Method for Empirical Risk Minimization with $\mathcal{O}(1)$ Per-Iteration Complexity

Regularized empirical risk minimization problem with linear predictor appears frequently in machine learning. In this paper, we propose a new stochastic primal-dual method to solve this class of problems. Different from existing methods, our proposed methods only require O(1) operations in each iteration. We also develop a variance-reduction variant of the algorithm that converges linearly. Numerical experiments suggest that our methods are faster than existing ones such as proximal SGD, SVRG and SAGA on high-dimensional problems.

近似价值迭代用于风险感知马尔可夫决策过程

Approximate Value Iteration for Risk-aware Markov Decision Processes

本文提出基于近似动态规划的算法，用于解决大规模风险感知马尔可夫决策过程，同时推导了算法的样本复杂度界。

Nonzero-sum risk-sensitive stochastic differential games

We study two person nonzero-sum stochastic differential games with risk-sensitive discounted and ergodic cost criteria. Under certain conditions we establish a Nash equilibrium in Markov strategies for the discounted cost criterion and a Nash equilibrium in stationary strategies for the ergodic cost criterion. We achieve our results by studying the relevant systems of coupled HJB equations.

具有风险敏感平均准则的零和随机博弈算法

Algorithms for zero-sum stochastic games with the risk-sensitive average criterion

本文针对零和风险敏感平均随机博弈，引入不可约系数并建立其与不可约条件的等价性，提出两种迭代算法分别计算值的ε-近似和有限步内获得ε-鞍点，并通过智能电网能量管理数值例子验证了结果。

迭代风险约束模型预测控制：一种数据驱动的分布鲁棒方法

Iterative risk-constrained model predictive control: A data-driven distributionally robust approach

本文提出迭代分布鲁棒MPC方案求解带风险约束的无限时域最优控制问题，通过累积数据逐步优化轨迹代价与安全性，证明了递归可行性与收敛性，并给出可计算重构与仿真验证。

Zero-sum stochastic differential game with risk-sensitive cost

Zero sum games with risk-sensitive cost criterion are considered with underlying dynamics being given by controlled stochastic differential equations. Under the assumption of geometric stability on the dynamics , we completely characterize all possible saddle point strategies in the class of stationary Markov controls. In addition, we also establish existence-uniqueness result for the value function of the Hamilton-Jacobi-Isaacs equation.

用于运动规划的Wasserstein分布鲁棒风险约束迭代MPC：计算高效的近似方法

Wasserstein distributionally robust risk-constrained iterative MPC for motion planning: computationally efficient approximations

针对迭代MPC结合分布鲁棒风险约束的运动规划问题计算量随迭代升高的问题，提出数据聚类调整模糊集和预确定DR安全区域两种高效近似方法，保证安全性的同时降低实时计算负担。

Optimal investment and reinsurance policies for the Cram{é}r-Lundberg risk model under monotone mean-variance preference

In this paper, an optimization problem for the monotone mean-variance(MMV) criterion is considered in the perspective of the insurance company. The MMV criterion is an amended version of the classical mean-variance(MV) criterion which guarantees the monotonicity of the utility function. With this criterion we study the optimal investment and reinsurance problem which is formulated as a zero-sum game between the insurance company and an imaginary player. We apply the dynamic programming principle to obtain the corresponding Hamilton-Jacobi-Bellman-Isaacs(HJBI) equation. As the main conclusion o…

Data-driven distributionally robust iterative risk-constrained model predictive control

This paper considers a risk-constrained infinite-horizon optimal control problem and proposes to solve it in an iterative manner. Each iteration of the algorithm generates a trajectory from the starting point to the target equilibrium state by implementing a distributionally robust risk-constrained model predictive control (MPC) scheme. At each iteration, a set of safe states (that satisfy the risk-constraint with high probability) and a certain number of independent and identically distributed samples of the uncertainty governing the risk constraint are available. These states and samples are…

一个风险敏感的遍历奇异随机控制问题

A risk-sensitive ergodic singular stochastic control problem

本文研究风险敏感遍历准则下的双边奇异随机控制问题，通过HJB方程的C2解和Sturm-Liouville特征值问题，给出完整解，应用于央行汇率目标区间控制。

遍历风险准则用于随机稳定化策略优化

Ergodic-risk Criterion for Stochastically Stabilizing Policy Optimization

本文提出遍历风险准则用于随机稳定化策略优化，通过理论分析和模拟验证了该方法在处理重尾分布噪声下的有效性。

Partially Observable Risk-Sensitive Stopping Problems in Discrete Time

In this paper we consider stopping problems with partial observation under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. Our aim is to maximize the certainty equivalent of the stopping reward. We develop a general theory and discuss the Bayesian risk-sensitive house selling problem as a special example. In particular we are able to study the influence of the attitude towards risk of the decision maker on the optimal stopping rule.

长期风险敏感随机控制的Blackwell最优性与策略稳定性

Blackwell optimality and policy stability for long-run risk sensitive stochastic control

本文研究有限MDP长期平均风险敏感准则下最优策略对风险厌恶参数扰动的稳定性，揭示Blackwell最优性及其与消失折扣近似的联系，并通过实例阐明框架的复杂性。

风险敏感随机控制中的Blackwell最优性

Blackwell optimality in risk-sensitive stochastic control

本文研究了风险敏感随机控制中Blackwell最优性的概念，探讨了其与终极平稳性的关系，并通过示例说明了两者之间的结构性差异。

Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterion

CVaR (Conditional Value at Risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult since it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite-horizon discrete-time MDP with a long-run CVaR criterion, from the view of sensitivity-based optimization. By introducing a pseudo CVaR metric, we derive a CVaR difference formula which quantifies the difference of long-run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so-called Bellma…

离散时间均场博弈在折扣成本准则下的鲁棒性与近似

Robustness and Approximation of Discrete-time Mean-field Games under Discounted Cost Criterion

本文研究了在模型不确定性下离散时间均场均衡的鲁棒性，通过价值迭代算法的收敛条件，证明了系统动态误标情况下均场均衡的鲁棒性，并展示状态空间量化足够精细时有限模型的均场均衡近似于名义解。

Nonzero-Sum Risk Sensitive Stochastic Games for Continuous Time Markov Chains

We study nonzero-sum stochastic games for continuous time Markov chains on a denumerable state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we first show that the corresponding system of coupled HJB equations has an appropriate solution. Then under an additional additive structure on the transition rate matrix and payoff functions, we establish the existence of a Nash equilibrium in Markov strategies. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show that…

Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Risk

In this paper we present an algorithm to compute risk averse policies in Markov Decision Processes (MDP) when the total cost criterion is used together with the average value at risk (AVaR) metric. Risk averse policies are needed when large deviations from the expected behavior may have detrimental effects, and conventional MDP algorithms usually ignore this aspect. We provide conditions for the structure of the underlying MDP ensuring that approximations for the exact problem can be derived and solved efficiently. Our findings are novel inasmuch as average value at risk has not previously bee…

Howard策略迭代的概率框架：BML评估与鲁棒收敛性分析

Probabilistic Framework of Howard's Policy Iteration: BML Evaluation and Robust Convergence Analysis

本文用FBSDEs构建Howard策略迭代的概率框架，提出BML准则以支持on-policy/off-policy评估，并证明其在理想与实际条件下的收敛及鲁棒性。

Risk-sensitive Markov decision problems under model uncertainty: finite time horizon case

In this paper we study a class of risk-sensitive Markovian control problems in discrete time subject to model uncertainty. We consider a risk-sensitive discounted cost criterion with finite time horizon. The used methodology is the one of adaptive robust control combined with machine learning.

有限时域部分可观测半马尔可夫博弈的风险概率准则

Finite-Horizon Partially Observable Semi-Markov Games with Risk Probability Criteria

针对系统状态部分可观测的两人零和半马尔可夫博弈，本文基于状态空间扩充技术分析终端累积收益低于目标的风险概率，推导沙普利方程并证明值函数唯一存在及纳什均衡存在。

基于风险价值准则的马尔可夫决策过程

Markov Decision Processes with Value-at-Risk Criterion

针对马尔可夫决策过程中VaR因非可加性难以用传统动态规划优化的问题，本文通过建立VaR最大化与概率最小化MDP的等价性，将其转化为双层优化问题，提出对应求解算法并证明收敛性，为分位数类指标优化提供新思路。

Iteration-Complexity of a Generalized Forward Backward Splitting Algorithm

In this paper, we analyze the iteration-complexity of Generalized Forward--Backward (GFB) splitting algorithm, as proposed in \cite{gfb2011}, for minimizing a large class of composite objectives $f + \sum_{i=1}^n h_i$ on a Hilbert space, where $f$ has a Lipschitz-continuous gradient and the $h_i$'s are simple (\ie their proximity operators are easy to compute). We derive iteration-complexity bounds (pointwise and ergodic) for the inexact version of GFB to obtain an approximate solution based on an easily verifiable termination criterion. Along the way, we prove complexity bounds for relaxed an…

Zero and Non-zero Sum Risk-sensitive Semi-Markov Games

In this article we consider zero and non-zero sum risk-sensitive average criterion games for semi-Markov processes with a finite state space. For the zero-sum case, under suitable assumptions we show that the game has a value. We also establish the existence of a stationary saddle point equilibrium. For the non-zero sum case, under suitable assumptions we establish the existence of a stationary Nash equilibrium.

Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.

Nonzero-Sum Risk-Sensitive Stochastic Differential Games: A Multi-parameter Eigenvalue Problem Approach

We study nonzero-sum stochastic differential games with risk-sensitive ergodic cost criterion. Under certain conditions, using multi-parameter eigenvalue approach, we establish the existence of a Nash equilibrium in the space of stationary Markov strategies. We achieve our results by studying the relevant systems of coupled HJB equations. Exploiting the stochastic representation of the principal eigenfunctions we completely characterize Nash equilibrium points in the space of stationary Markov strategies.

Contingency-Risk Informed Power System Design

We consider the problem of designing (or augmenting) an electric power system at a minimum cost such that it satisfies the N-k-e survivability criterion. This survivability criterion is a generalization of the well-known N-k criterion, and it requires that at least (1- e_j) fraction of the total demand to be met after failures of up to j components, for j=1,...,k. The network design problem adds another level of complexity to the notoriously hard contingency analysis problem, since the contingency analysis is only one of the requirements for the design optimization problem. We present a mixed-…

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterise the value function via HJB equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.

A Partially Inexact Alternating Direction Method of Multipliers and its Iteration-Complexity Analysis

This paper proposes a partially inexact alternating direction method of multipliers for computing approximate solution of a linearly constrained convex optimization problem. This method allows its first subproblem to be solved inexactly using a relative approximate criterion, whereas a proximal term is added to its second subproblem in order to simplify it. A stepsize parameter is included in the updating rule of the Lagrangian multiplier to improve its computational performance. Pointwise and ergodic interation-complexity bounds for the proposed method are established. To the best of our know…

Point-Based Value Iteration and Approximately Optimal Dynamic Sensor Selection for Linear-Gaussian Processes

The problem of synthesizing an optimal sensor selection policy is pertinent to a variety of engineering applications ranging from event detection to autonomous navigation. We consider such a synthesis problem over an infinite time horizon with a discounted cost criterion. We formulate this problem in terms of a value iteration over the continuous space of covariance matrices. To obtain a computationally tractable solution, we subsequently formulate an approximate sensor selection problem, which is solvable through a point-based value iteration over a finite "mesh" of covariance matrices with a…

Contractivity of Bellman Operator in Risk Averse Dynamic Programming with Infinite Horizon

The paper deals with a risk averse dynamic programming problem with infinite horizon. First, the required assumptions are formulated to have the problem well defined. Then the Bellman equation is derived, which may be also seen as a standalone reinforcement learning problem. The fact that the Bellman operator is contraction is proved, guaranteeing convergence of various solution algorithms used for dynamic programming as well as reinforcement learning problems, which we demonstrate on the value iteration algorithm.

Long run risk sensitive portfolio with general factors

In the paper portfolio optimization over long run risk sensitive criterion is considered. It is assumed that economic factors which stimulate asset prices are ergodic but non necessarily uniformly ergodic. Solution to suitable Bellman equation using local span contraction with weighted norms is shown. The form of optimal strategy is presented and examples of market models satisfying imposed assumptions are shown.

Affine Monotonic and Risk-Sensitive Models in Dynamic Programming

In this paper we consider a broad class of infinite horizon discrete-time optimal control models that involve a nonnegative cost function and an affine mapping in their dynamic programming equation. They include as special cases classical models such as stochastic undiscounted nonnegative cost problems, stochastic multiplicative cost problems, and risk-sensitive problems with exponential cost. We focus on the case where the state space is finite and the control space has some compactness properties. We assume that the affine mapping has a semicontractive character, whereby for some policies it…

一种用于风险规避优化的自适应重要性采样算法

An adaptive importance sampling algorithm for risk-averse optimization

针对风险规避优化中自适应采样算法因尾部采样困难导致样本量增长过快的问题，提出一种同时调整样本量和采样分布的自适应重要性采样算法，通过降阶模型构建偏置分布过采样风险区域以降低梯度方差，在保留收敛速度的同时减少样本量，实现显著计算节省。

线性二次优化下的稳定股息

Stable Dividends under Linear-Quadratic Optimization

本文针对风险业务的股息优化，提出在线性二次准则下为一般莱维过程推导最优仿射股息策略，通过HJB方程求解，实现明确要求的股息稳定性。

Simulation Optimization of Risk Measures with Adaptive Risk Levels

Optimizing risk measures such as Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR) of a general loss distribution is usually difficult, because 1) the loss function might lack structural properties such as convexity or differentiability since it is often generated via black-box simulation of a stochastic system; 2) evaluation of risk measures often requires rare-event simulation, which is computationally expensive. In this paper, we study the extension of the recently proposed gradient-based adaptive stochastic search (GASS) to the optimization of risk measures VaR and CVaR. Instead of …

Nonzero-sum Risk-sensitive Stochastic Games on a Countable State Space

The infinite horizon risk-sensitive discounted-cost and ergodic-cost nonzero-sum stochastic games for controlled Markov chains with countably many states are analyzed. For the discounted-cost game, we prove the existence of Nash equilibrium strategies in the class of Markov strategies under fairly general conditions. Under an additional geometric ergodicity condition and a small cost criterion, the existence of Nash equilibrium strategies in the class of stationary Markov strategies is proved for the ergodic-cost game.

Risk-Averse No-Regret Learning in Online Convex Games

We consider an online stochastic game with risk-averse agents whose goal is to learn optimal decisions that minimize the risk of incurring significantly high costs. Specifically, we use the Conditional Value at Risk (CVaR) as a risk measure that the agents can estimate using bandit feedback in the form of the cost values of only their selected actions. Since the distributions of the cost functions depend on the actions of all agents that are generally unobservable, they are themselves unknown and, therefore, the CVaR values of the costs are difficult to compute. To address this challenge, we p…

分布鲁棒马尔可夫决策过程及其与风险度量的联系

Distributionally Robust Markov Decision Processes and their Connection to Risk Measures

本文研究具有Borel状态空间、无界成本和有限时间范围的分布鲁棒马尔可夫决策过程，将其建模为Stackelberg博弈，推导出鲁棒成本迭代和值迭代，证明确定性最优策略的存在性，并探讨其与一致风险度量的联系及应用。

Never Go Full Batch (in Stochastic Convex Optimization)

We study the generalization performance of $\text{full-batch}$ optimization algorithms for stochastic convex optimization: these are first-order methods that only access the exact gradient of the empirical risk (rather than gradients with respect to individual data points), that include a wide range of algorithms such as gradient descent, mirror descent, and their regularized and/or accelerated variants. We provide a new separation result showing that, while algorithms such as stochastic gradient descent can generalize and optimize the population risk to within $ε$ after $O(1/ε^2)$ iterations,…

基于优化确定性等价的可证明高效风险敏感离线强化学习

Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents

本研究针对风险敏感离线强化学习的理论空白，以优化确定性等价为风险度量，开发了表格设置和线性函数近似下的悲观算法，建立了相关次优性界，填补了该领域的理论与算法空白。

学习牢固非扩张算子

Learning Firmly Nonexpansive Operators

本文提出一种数据驱动方法学习牢固非扩张算子，用于即插即用图像去噪，并证明经验风险最小化收敛于期望风险最小化，且算子收敛到最优经验解。

Risk-sensitive Markov control processes

We introduce a general framework for measuring risk in the context of Markov control processes with risk maps on general Borel spaces that generalize known concepts of risk measures in mathematical finance, operations research and behavioral economics. Within the framework, applying weighted norm spaces to incorporate also unbounded costs, we study two types of infinite-horizon risk-sensitive criteria, discounted total risk and average risk, and solve the associated optimization problems by dynamic programming. For the discounted case, we propose a new discount scheme, which is different from …