Shaan Ul Haque, Vivek Borkar
We derive a concentration bound for a Q-learning algorithm for average cost Markov decision processes based on an equivalent shortest path problem, and compare it numerically with the alternative scheme based on relative value iteration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We investigate online Markov Decision Processes (MDPs) with adversarially changing loss functions and known transitions. We choose dynamic regret as the performance measure, defined as the performance difference between the learner and any sequence of feasible changing policies. The measure is strictly stronger than the standard static regret that benchmarks the learner's performance with a fixed compared policy. We consider three foundational models of online MDPs, including episodic loop-free Stochastic Shortest Path (SSP), episodic SSP, and infinite-horizon MDPs. For these three models, we …
Policy Gradients for CVaR-Constrained MDPs
我们研究的风险约束版本的随机最短路径（SSP）问题，其中考虑的风险度量是条件风险价值（CVaR）。我们提出了两个算法，获得一个局部风险最优的政策，采用四种工具：随机逼近，小批量，政策梯度和重要性抽样。这两种算法都结合了CVaR估计程序，沿着Bardou等人的路线。[2009]，这反过来又基于Rockafellar-Uryasev的CVaR表示，并利用似然比原理估计一个成本函数（SSP的目标）之和的梯度和另一个成本函数（在SSP的约束下）之和的CVaR梯度。这些算法在近似CVaR估计值/必要梯度的方式上有所不同-第一种算法使用随机逼近，而第二种算法则根据蒙特卡罗方法的精神采用小批量。我们建立了这两个算法的渐近收敛性。此外，由于估计CVaR是与稀有事件模拟，我们将一个重要性抽样为基础的方差减少计划到我们提出的算法。
Solving general Markov decision processes (MDPs) is a computationally hard problem. Solving finite-horizon MDPs, on the other hand, is highly tractable with well known polynomial-time algorithms. What drives this extreme disparity, and do problems exist that lie between these diametrically opposed complexities? In this paper we identify and analyse a sub-class of stochastic shortest path problems (SSPs) for general state-action spaces whose dynamics satisfy a particular drift condition. This construction generalises the traditional, temporal notion of a horizon via decreasing reachability: a p…
Previous work on planning as active inference addresses finite horizon problems and solutions valid for online planning. We propose solving the general Stochastic Shortest-Path Markov Decision Process (SSP MDP) as probabilistic inference. Furthermore, we discuss online and offline methods for planning under uncertainty. In an SSP MDP, the horizon is indefinite and unknown a priori. SSP MDPs generalize finite and infinite horizon MDPs and are widely used in the artificial intelligence community. Additionally, we highlight some of the differences between solving an MDP using dynamic programming …
Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
在强化学习中，鲁棒MDP（RMDP）可用于计算具有可证明的最坏情况保证的策略。RMDP解决方案的质量和鲁棒性由模糊集-合理转移概率集-决定，模糊集通常构造为多维置信区域。现有的方法构造模糊集的置信区域使用浓度不等式，导致过于保守的解决方案。本文提出了一种新的范例，可以实现更好的解决方案，具有相同的鲁棒性保证，而不使用置信区域作为模糊集。为了结合先验知识，我们的算法使用贝叶斯推理优化歧义集的大小和位置。我们的理论分析表明，所提出的方法的安全性，和实证结果表明其实际的承诺。
We address the problem of computing reliable policies in reinforcement learning problems with limited data. In particular, we compute policies that achieve good returns with high confidence when deployed. This objective, known as the \emph{percentile criterion}, can be optimized using Robust MDPs~(RMDPs). RMDPs generalize MDPs to allow for uncertain transition probabilities chosen adversarially from given ambiguity sets. We show that the RMDP solution's sub-optimality depends on the spans of the ambiguity sets along the value function. We then propose new algorithms that minimize the span of a…
Concentration of Cumulative Reward in Markov Decision Processes
本文研究了马尔可夫决策过程中累积奖励的集中性质，提出统一方法分析无限和有限时间 horizon情形，并探讨了奖励差异和regret定义的等价性。
Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approach
本文提出了一种数据驱动的鲁棒马尔可夫决策过程方法，通过公理化方法在博雷尔空间上提供性能保证，建立了鲁棒最优值函数与样本外值函数的收敛性及概率收敛速率。
Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition
本文研究老虎机反馈下带未知转移和对抗损失的线性混合MDP强化学习，提出利用全状态访问信息的最小二乘估计器与适配的自归一化浓度方法，得到更优的高概率遗憾界。
Recently, several studies (Zhou et al., 2021a; Zhang et al., 2021b; Kim et al., 2021; Zhou and Gu, 2022) have provided variance-dependent regret bounds for linear contextual bandits, which interpolates the regret for the worst-case regime and the deterministic reward regime. However, these algorithms are either computationally intractable or unable to handle unknown variance of the noise. In this paper, we present a novel solution to this open problem by proposing the first computationally efficient algorithm for linear bandits with heteroscedastic noise. Our algorithm is adaptive to the unkno…
Mathematical methods of reinforcement learning
综述梳理强化学习的数学结构，从马尔可夫决策过程和贝尔曼算子出发，阐述优化视角、函数逼近等内容，涵盖多种相关学习与方法，通过常见视角统一算法模板，为多领域研究人员提供强化学习的数学切入点。
Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs
本文基于平均回报准则，重新审视未知离散MDP中的强化学习问题，通过引入局部方差代替MDP直径，改进KL-UCRL算法的后悔界，提供更优的性能保证。
An Improved Model-Free Decision-Estimation Coefficient with Applications in Adversarial MDPs
本文提出Dig-DEC，一种无模型的决策-估计系数，通过信息增益驱动探索，解决了对抗性MDP中的再生问题，并改进了函数估计过程的性能。
Entropic Risk-Aware Monte Carlo Tree Search
本文提出了一种基于熵风险度量的风险感知蒙特卡洛树搜索算法，通过非渐近分析证明其正确性与多项式后悔集中性，并通过实验验证其有效性。
Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chains
本文首次为无限时间折扣马尔可夫决策过程中的经典在线Q学习提供了后悔界，不依赖乐观或奖励项。通过分析退火Q学习并结合平滑ε_n-贪心探索方案，证明了近似O(N^{9/10})的后悔界，并获得样本复杂性保证。
We study reinforcement learning (RL) with linear function approximation where the underlying transition probability kernel of the Markov decision process (MDP) is a linear mixture model (Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2020) and the learning agent has access to either an integration or a sampling oracle of the individual basis kernels. We propose a new Bernstein-type concentration inequality for self-normalized martingales for linear bandit problems with bounded noise. Based on the new inequality, we propose a new, computationally efficient algorithm with linear function app…
Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints in Non-Convex Feature Spaces
本文针对非凸特征空间中线性MDP的瞬时安全约束问题，提出可证明高效的强化学习算法，在星凸与非星凸情形下均达到次线性遗憾界且零约束违反，并通过自动驾驶仿真验证有效性。
Prior-dependent analysis of posterior sampling reinforcement learning with function approximation
针对线性混合MDP的函数逼近RL，提出先验相关贝叶斯遗憾界，通过解耦与方差缩减优化PSRL上界至O(d√(H³T logT))。
Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noise
本文研究带任意范数压缩与马尔可夫噪声的非线性双时间尺度随机逼近，利用广义Moreau包络和Poisson方程给出有限时间均方界，并应用于Q-Learning与GNE学习。
Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation
针对一般函数逼近下的长视界强化学习，提出UCRL-WVTR算法，通过加权价值目标回归、高阶矩估计及精细分析实现无视界、依赖实例且匹配下界的锐利遗憾保证。
Minimax Regret Bounds for Reinforcement Learning
我们考虑有限视野MDP强化学习中可证明的最优探索问题。我们发现，一个乐观的修改值迭代实现了遗憾界的$\tilde{O}( \sqrt{HSAT} + H^2S^2A+H\sqrt{T})$，其中$H$是时间范围，$S$的状态数，$A$的动作数和$T$的时间步数。该结果改进了Jaksch等人的UCRL 2算法所实现的最佳先前已知边界$\tilde{O}(HS \sqrt{AT})$，2010.我们的新结果的关键意义是，当$T\geq H^3S^3A$和$SA\geq H$，它会导致一个遗憾的$\tilde{O}(\sqrt{HSAT})$匹配的$Ω(\sqrt{HSAT})$的建立下限的对数因子。我们的分析包含两个关键的见解。我们仔细地将浓度不等式应用于最优值函数作为一个整体，而不是转移概率（以改善$S$中的缩放），并且我们定义了基于伯恩斯坦的“探索奖金”，该奖金使用下一个状态的估计值的经验方差（以改善$H$中的缩放）。
What Are the Odds? Improving the foundations of Statistical Model Checking
针对统计模型检查（SMC）处理马尔可夫决策过程（MDPs）时统计方法朴素的问题，通过调研更好的集中不等式并提出利用MDP知识的专门方法，大幅减少样本量，提升SMC的基础性能。
We study the Stochastic Shortest Path (SSP) problem in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent has no prior knowledge about the costs and dynamics of the model. She repeatedly interacts with the model for $K$ episodes, and has to minimize her regret. In this work we show that the minimax regret for this setting is $\widetilde O(\sqrt{ (B_\star^2 + B_\star) |S| |A| K})$ where $B_\star$ is a bound on the expected cost of the optimal policy from any state, $S$ is the state space, and $A$ is the action space. Th…
Achieving Constant Regret in Linear Markov Decision Processes
本研究针对强化学习恒定遗憾问题，提出Cert-LSVI-UCB算法，通过创新的多阶段值目标回归集中分析，在带误设的线性MDP中实现了与轮次数无关的高概率恒定遗憾界，为首个无需先验分布假设的同类算法。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。