arXiv 1202.37552012-02-14cs.GTcs.AIq-fin.RM

Iterated risk measures for risk-sensitive Markov decision processes with discounted cost

IBM Research - Tokyo（IBM东京研究院）

Takayuki Osogami

英文摘要：

We demonstrate a limitation of discounted expected utility, a standard approach for representing the preference to risk when future cost is discounted. Specifically, we provide an example of the preference of a decision maker that appears to be rational but cannot be represented with any discounted expected utility. A straightforward modification to discounted expected utility leads to inconsistent decision making over time. We will show that an iterated risk measure can represent the preference that cannot be represented by any discounted expected utility and that the decisions based on the iterated risk measure are consistent over time.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

折扣马尔可夫博弈中的独立强化学习

Independent Reinforcement Learning in Discounted Markov Games

University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

针对折扣一般和马尔可夫博弈，在假设ETH与PPAD等价下证明独立学习无多项式时间算法，提出首个完全解耦的分层乐观镜像下降算法，保证次指数收敛并开发全反馈与部分反馈版本。

2026-09-02cs.AIcs.GTcs.LG

A unified algorithm framework for mean-variance optimization in discounted Markov decision processes

Sun Yat-sen University（中山大学）；Tsinghua University（清华大学）

This paper studies the risk-averse mean-variance optimization in infinite-horizon discounted Markov decision processes (MDPs). The involved variance metric concerns reward variability during the whole process, and future deviations are discounted to their present values. This discounted mean-variance optimization yields a reward function dependent on a discounted mean, and this dependency renders traditional dynamic programming methods inapplicable since it suppresses a crucial property -- time consistency. To deal with this unorthodox problem, we introduce a pseudo mean to transform the untre…

2022-01-15cs.AImath.OC

On the Use of Non-Stationary Policies for Infinite-Horizon Discounted Markov Decision Processes

We consider infinite-horizon $γ$-discounted Markov Decision Processes, for which it is known that there exists a stationary optimal policy. We consider the algorithm Value Iteration and the sequence of policies $π_1,...,π_k$ it implicitely generates until some iteration $k$. We provide performance bounds for non-stationary policies involving the last $m$ generated policies that reduce the state-of-the-art bound for the last stationary policy $π_k$ by a factor $\frac{1-γ}{1-γ^m}$. In particular, the use of non-stationary policies allows to reduce the usual asymptotic performance bounds of Value…

2012-03-30cs.AI

自然策略梯度算法在无限时域折扣奖励马尔可夫决策过程中一般参数化下的改进样本复杂度分析

Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processes

Purdue University（普渡大学）

针对无限时域折扣奖励马尔可夫决策过程，提出加速自然策略梯度（ANPG）算法，在一般参数化下实现$\mathcal{O}(ε^{-2})$样本复杂度和$\mathcal{O}(ε^{-1})$迭代复杂度，改进现有最优样本复杂度，且无需不可验证的IS权重方差假设。

2024-02-05cs.AIcs.LG

Robust Average-Reward Markov Decision Processes

University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Central Florida（中佛罗里达大学）；Air Force Research Laboratory（空军研究实验室）

In robust Markov decision processes (MDPs), the uncertainty in the transition kernel is addressed by finding a policy that optimizes the worst-case performance over an uncertainty set of MDPs. While much of the literature has focused on discounted MDPs, robust average-reward MDPs remain largely unexplored. In this paper, we focus on robust average-reward MDPs, where the goal is to find a policy that optimizes the worst-case average reward over an uncertainty set. We first take an approach that approximates average-reward MDPs using discounted MDPs. We prove that the robust discounted value fun…

2023-03-01cs.AIcs.LG

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Masaryk University（马萨里克大学）；Institute of Science and Technology Austria（奥地利科学技术研究所）

Markov decision processes (MDPs) are the defacto frame-work for sequential decision making in the presence ofstochastic uncertainty. A classical optimization criterion forMDPs is to maximize the expected discounted-sum pay-off, which ignores low probability catastrophic events withhighly negative impact on the system. On the other hand,risk-averse policies require the probability of undesirableevents to be below a given threshold, but they do not accountfor optimization of the expected payoff. We consider MDPswith discounted-sum payoff with failure states which repre-sent catastrophic outcomes…

2020-02-27cs.AIcs.LG

更多相关论文收起更多论文

Expectation Optimization with Probabilistic Guarantees in POMDPs with Discounted-sum Objectives

Institute of Science and Technology Austria（奥地利科学技术研究所）；École Normale Supérieure de Rennes（雷恩高等师范学院）

Partially-observable Markov decision processes (POMDPs) with discounted-sum payoff are a standard framework to model a wide range of problems related to decision making under uncertainty. Traditionally, the goal has been to obtain policies that optimize the expectation of the discounted-sum payoff. A key drawback of the expectation measure is that even low probability events with extreme payoff can significantly affect the expectation, and thus the obtained policies are not necessarily risk-averse. An alternate approach is to optimize the probability that the payoff is above a certain threshol…

2018-04-30cs.AI

Metrics for Markov Decision Processes with Infinite State Spaces

McGill University（麦吉尔大学）

We present metrics for measuring state similarity in Markov decision processes (MDPs) with infinitely many states, including MDPs with continuous state spaces. Such metrics provide a stable quantitative analogue of the notion of bisimulation for MDPs, and are suitable for use in MDP approximation. We show that the optimal value function associated with a discounted infinite horizon planning task varies continuously with respect to our metric distances.

2012-07-04cs.AI

Metrics for Finite Markov Decision Processes

We present metrics for measuring the similarity of states in a finite Markov decision process (MDP). The formulation of our metrics is based on the notion of bisimulation for MDPs, with an aim towards solving discounted infinite horizon reinforcement learning tasks. Such metrics can be used to aggregate states, as well as to better structure other value function approximators (e.g., memory-based or nearest-neighbor approximators). We provide bounds that relate our metric distances to the optimal values of states in the given MDP.

2012-07-11cs.AI

Gaussian Process Bandits for Tree Search: Theory and Application to Planning in Discounted MDPs

University College London（伦敦大学学院）

We motivate and analyse a new Tree Search algorithm, GPTS, based on recent theoretical advances in the use of Gaussian Processes for Bandit problems. We consider tree paths as arms and we assume the target/reward function is drawn from a GP distribution. The posterior mean and variance, after observing data, are used to define confidence intervals for the function values, and we sequentially play arms with highest upper confidence bounds. We give an efficient implementation of GPTS and we adapt previous regret bounds by determining the decay rate of the eigenvalues of the kernel matrix on the …

2011-01-15cs.AIcs.LG

Geometric Policy Iteration for Markov Decision Processes

University of California, Davis（加州大学戴维斯分校）

Recently discovered polyhedral structures of the value function for finite state-action discounted Markov decision processes (MDP) shed light on understanding the success of reinforcement learning. We investigate the value function polytope in greater detail and characterize the polytope boundary using a hyperplane arrangement. We further show that the value space is a union of finitely many cells of the same hyperplane arrangement and relate it to the polytope of the classical linear programming formulation for MDPs. Inspired by these geometric properties, we propose a new algorithm, Geometri…

2022-06-24cs.AIcs.LG

用于折扣和收益的分类器自动机

Categorizer Automata for Discounted-Sum Payoffs

Univ Rennes（雷恩大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；IRISA（雷恩信息与自动化研究所）；University of Sydney（悉尼大学）；Université Libre de Bruxelles（布鲁塞尔自由大学）；Rice University（莱斯大学）

本文提出用于折扣和收益的分类器自动机，将其应用于马尔可夫决策过程，可综合出针对不连续效用函数的最优或ε-最优策略，且证明该综合问题是PSPACE难的。

2026-08-28cs.AIcs.FL

Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mapping

University of California, Los Angeles（加州大学洛杉矶分校）

Modern tasks in reinforcement learning have large state and action spaces. To deal with them efficiently, one often uses predefined feature mapping to represent states and actions in a low-dimensional space. In this paper, we study reinforcement learning for discounted Markov Decision Processes (MDPs), where the transition kernel can be parameterized as a linear function of certain feature mapping. We propose a novel algorithm that makes use of the feature mapping and obtains a $\tilde O(d\sqrt{T}/(1-γ)^2)$ regret, where $d$ is the dimension of the feature space, $T$ is the time horizon and $γ…

2021-02-22cs.AIcs.LGmath.OC

Globalized Robust Markov Perfect Equilibrium for Discounted Stochastic Games and its Application on Intrusion Detection in Wireless Sensor Networks: Part I -- Theory

In this article, we study a discounted stochastic game to model resource optimal intrusion detection in wireless sensor networks. To address the problem of uncertainties in various network parameters, we propose a globalized robust game theoretic framework for discounted robust stochastic games. A robust solution to the considered problem is an optimal point that is feasible for all realizations of data from a given uncertainty set. To allow a controlled violation of the constraints when the parameters move out of the uncertainty set, the concept of globalized robust framework comes into view.…

2019-10-28cs.GTmath.OC

超越折扣回报：具备平均最优性与Blackwell最优性的鲁棒马尔可夫决策过程

Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality

本文研究折扣回报之外的鲁棒马尔可夫决策过程，证明sa-矩形与s-矩形RMDP在平均最优性、Blackwell最优性上的差异结论，给出策略存在性条件与计算算法，凸显sa-矩形模型的实用优势。

2025-01-14cs.GTmath.OC

Budget Allocation in Binary Opinion Dynamics

In this article we study the allocation of a budget to promote an opinion in a group of agents. We assume that their opinion dynamics are based on the well-known voter model. We are interested in finding the most efficient use of a budget over time in order to manipulate a social network. We address the problem using the theory of discounted Markov decision processes. Our contributions can be summarized as follows: (i) we introduce the discounted Markov decision process in our cases, (ii) we present the corresponding Bellman equations, and, (iii) we solve the Bellman equations via backward pro…

2017-11-22cs.GTcs.SI

具有少量参与者的演化博弈的马尔可夫链

Markov Chains of Evolutionary Games with a Small Number of Players

本文构建并研究了有限参与者演化博弈的马尔可夫链转移概率矩阵，基于Sandholm的简化种群博弈框架，通过迭代囚徒困境等示例及多种修正协议，显式构建MC转移概率矩阵并分析其性质。

2025-06-29cs.GTmath.OC

通过行为经济学实现无限期一般和马尔可夫博弈的收敛Q学习

Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economics

本文研究双人正则形式博弈和折扣无限期马尔可夫博弈中风险厌恶量化响应均衡（RQE）的计算，通过定义收缩的风险厌恶量化响应Bellman算子，提出了一种适用于所有无限期一般和马尔可夫博弈且具有收敛保证的Q学习算法。

2025-08-12cs.GT

Discounting in Games across Time Scales

We introduce two-level discounted games played by two players on a perfect-information stochastic game graph. The upper level game is a discounted game and the lower level game is an undiscounted reachability game. Two-level games model hierarchical and sequential decision making under uncertainty across different time scales. We show the existence of pure memoryless optimal strategies for both players and an ordered field property for such games. We show that if there is only one player (Markov decision processes), then the values can be computed in polynomial time. It follows that whether th…

2010-06-08cs.GTcs.LO