Philip S. Thomas, Scott M. Jordan, Yash Chandak, Chris Nota, James Kostas
We propose a new objective function for finite-horizon episodic Markov decision processes that better captures Bellman's principle of optimality, and provide an expression for the gradient of the objective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Empowerment is an information-theoretic method that can be used to intrinsically motivate learning agents. It attempts to maximize an agent's control over the environment by encouraging visiting states with a large number of reachable next states. Empowered learning has been shown to lead to complex behaviors, without requiring an explicit reward signal. In this paper, we investigate the use of empowerment in the presence of an extrinsic reward signal. We hypothesize that empowerment can guide reinforcement learning (RL) agents to find good early behavioral solutions by encouraging highly empo…
Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
针对连续动作空间RL算法样本效率低的问题，提出一种从Bellman最优算子退火过渡到Bellman算子的退火方法，结合TD3和SAC，在加速学习的同时减轻过估计偏差，显著提升了性能与超参数鲁棒性。
Towards Blackwell Optimality: Bellman Optimality Is All You Can Get
本文研究马尔可夫决策过程中从平均增益到 Blackwell 最优性的策略识别问题，提出错误概率趋于零的学习算法，刻画可有限时间停止的 MDP 类别，并提供可计算的停止规则。
A Bellman Optimality Equation for Plasticity
本文针对持续强化学习中的可塑性优化问题，基于Abel等人对可塑性的新定义，证明了在马尔可夫决策过程中存在可塑性的贝尔曼最优性方程，为可塑性优化提供了理论基础。
Policy evaluation is a key process in Reinforcement Learning (RL). It assesses a given policy by estimating the corresponding value function. When using parameterized value functions, common approaches minimize the sum of squared Bellman temporal-difference errors and receive a point-estimate for the parameters. Kalman-based and Gaussian-processes based frameworks were suggested to evaluate the policy by treating the value as a random variable. These frameworks can learn uncertainties over the value parameters and exploit them for policy exploration. When adopting these frameworks to solve dee…
Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
本文提出一种基于混合一步与两步贝尔曼残差及自适应评论家表示的离线策略评估方法，通过数据依赖的核表示和样本分割控制过拟合，在模拟与MetaWorld任务中验证了中间残差组合对值估计的改进效果。
Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
本文提出双重平滑策略迭代框架，将自然策略梯度等方法统一为贝尔曼算子形式，证明其分布无关的几何收敛性，且无需修改MDP或引入额外正则化。
Symmetric solution of the Bellman optimality equation for repeated harmony game
本研究求解重复和谐博弈中贝尔曼最优性方程的对称解，发现三类解分别对应All-C、赢留输变策略及非平凡策略，并通过数值实验考察了强化学习智能体的实际学习结果。
Recently, \citet{SuttonMW15} introduced the emphatic temporal differences (ETD) algorithm for off-policy evaluation in Markov decision processes. In this short note, we show that the projected fixed-point equation that underlies ETD involves a contraction operator, with a $\sqrtγ$-contraction modulus (where $γ$ is the discount factor). This allows us to provide error bounds on the approximation error of ETD. To our knowledge, these are the first error bounds for an off-policy evaluation algorithm under general target and behavior policies.
Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
本文分析了在无Bellman完备性条件下软拟合Q迭代的稳定性机制，提出静态重加权软拟合Q迭代方法，证明其在近似可实现性和受控加权误差下具有有限样本局部线性收敛性。
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximize the mean value, and derive a method that minimizes the residual $\|T_* v_π- v_π\|_{1,ν}$ over policies. A theoretical analysis shows how good this proxy is to policy optimization, and notably that it is better than its value-based counterpart. We also propose experiments on ra…
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
针对离线强化学习中固定正则化强度难以平衡外推误差与策略性能的问题，提出选择性状态自适应正则化方法，通过状态级信任与选择性约束高质量动作，在D4RL基准上显著优于现有方法。
Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation
本文提出深度加权贝尔曼残差最小化框架，通过密度比加权整合专家演示与行为数据，解决离线强化学习中的分布偏移和Q值高估问题，并显著提升数值性能与策略泛化。
The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
本文研究带线性函数逼近的离线强化学习问题，基于低固有Bellman误差假设提出高效算法，在单策略覆盖下有性能保证，次优误差随固有Bellman误差的平方根缩放且该下界无法改进。
Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain
本文研究了鲁棒马尔可夫决策过程中的平均奖励问题，通过分析常增益设置，探讨了鲁棒贝尔曼方程的解存在性及其与最优平均奖励的关系，为长期平均标准下的鲁棒动态决策提供了理论基础。
We propose and analyze a reinforcement learning principle that approximates the Bellman equations by enforcing their validity only along an user-defined space of test functions. Focusing on applications to model-free offline RL with function approximation, we exploit this principle to derive confidence intervals for off-policy evaluation, as well as to optimize over policies within a prescribed policy class. We prove an oracle inequality on our policy optimization procedure in terms of a trade-off between the value and uncertainty of an arbitrary comparator policy. Different choices of test fu…
On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process
针对安全约束 CMDP 中多链结构导致 Bellman 最优性原理失效的问题，本文将多目标优化转化为零和博弈，提出异步值迭代与改进 $Q$-learning，并给出迭代下界和误差界。
When is Realizability Sufficient for Off-Policy Reinforcement Learning?
本文研究仅满足可实现性时的离策略强化学习，提出由度量熵、集中系数和 Bellman 完备性违反程度共同决定的有限样本保证。
Bellman Policy Optimization
本文提出贝尔曼策略优化（BPO），一种基于策略镜像下降的无评论家强化学习方法，通过贝尔曼方程将目标转化为轨迹级形式，避免中间状态值估计，并在数学推理基准上验证了其有效性。
Fitted Occupancy-Ratio Evaluation without Bellman Completeness
针对离线强化学习分布偏移下的离策略评估问题，提出拟合占用比评估方法FORE，通过伴随贝尔曼递归估计占用比，仅需占用比可实现性，无需贝尔曼完备性等条件，保障离线策略评估收敛性。
We study representation learning for Offline Reinforcement Learning (RL), focusing on the important task of Offline Policy Evaluation (OPE). Recent work shows that, in contrast to supervised learning, realizability of the Q-function is not enough for learning it. Two sufficient conditions for sample-efficient OPE are Bellman completeness and coverage. Prior work often assumes that representations satisfying these conditions are given, with results being mostly theoretical in nature. In this work, we propose BCRL, which directly learns from data an approximately linear Bellman complete represen…
Bellman-Certified Rounding for Sparse Policy Deployment in MDPs
针对MDP中稀疏策略部署，提出Bellman认证舍入方法，利用包络和加权曲率积分提供候选特定保证，显著提升认证覆盖率并降低界限与损失比。
We study the problem of policy optimization for infinite-horizon discounted Markov Decision Processes with softmax policy and nonlinear function approximation trained with policy gradient algorithms. We concentrate on the training dynamics in the mean-field regime, modeling e.g., the behavior of wide single hidden layer neural networks, when exploration is encouraged through entropy regularization. The dynamics of these models is established as a Wasserstein gradient flow of distributions in parameter space. We further prove global optimality of the fixed points of this dynamics under mild con…
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
本文通过利用熵正则化强化学习的Bellman结构，证明了Wasserstein策略梯度（WPG）方法的全局收敛性，并建立了分布Polyak-Łojasiewicz条件。
We develop a parameterized Primal-Dual $π$ Learning method based on deep neural networks for Markov decision process with large state space and off-policy reinforcement learning. In contrast to the popular Q-learning and actor-critic methods that are based on successive approximations to the nonlinear Bellman equation, our method makes primal-dual updates to the policy and value functions utilizing the fundamental linear Bellman duality. Naive parametrization of the primal-dual $π$ learning method using deep neural networks would encounter two major challenges: (1) each update requires computi…
On solutions of the distributional Bellman equation
本文研究一般分布型贝尔曼方程解的存在性、唯一性及尾部性质，通过将其与多元仿射分布方程建立联系，使相关一般理论可应用于分布型强化学习。
Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
本研究提出策略一致性假设，证明了存在与Bellman最优策略一致的确定性平稳最优鲁棒策略，指出需用L∞范数最小化Bellman误差，据此构建的CAR-DQN在多基准上取得顶尖性能。
Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
本文针对离线强化学习中边缘化重要性加权的残留占用平衡违反问题，提出等渗贝尔曼校准方法，可减少此类违反并保证下游策略价值估计等任务的性能。
Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence
本文研究了控制中贝尔曼残差最小化方法的几何特性、站定性及收敛性，探讨其在策略优化中的基础理论与应用价值。
Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
本文提出了一种基于目标域贝尔曼目标对齐的跨域离线强化学习方法，旨在通过评估源域过渡与目标域贝尔曼目标的一致性来提升策略学习性能。
Distributional Soft Bellman Operator under the Cramér Geometry
研究分布软策略迭代中策略评估步骤里的固定策略分布软贝尔曼算子，基于克拉默几何，制定CDF级算子并证明其收缩性质，获得唯一不动点，还通过共轭得到谱域表示，为DSPI式算法研究提供参考。
Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach
本文提出了一种基于希尔伯特空间嵌入的多维分布式强化学习框架，通过核均值嵌入估计价值分布，解决了多维状态-动作空间中瓦瑟斯坦距离计算的计算挑战。
In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving very low cumulative regret during the learning process. Our learning algorithm, Adaptive Value-function Elimination (AVE), is inspired by the policy elimination algorithm proposed in (Jiang et al., 2017), known as OLIVE. One of our key technical contributions in AVE is to formulate…
Off-Policy Reinforcement Learning with High Dimensional Reward
该研究为分布强化学习建立了支持无限维可分巴拿赫奖励空间的理论基础，证明其贝尔曼算子的压缩性与低维近似可行性，提出可解决传统RL难处理问题的新型DRL算法。
Learning Chance-Constrained MDPs with Bellman Distributional Certificates
本文提出贝尔曼分布证书方法，用于解决机会约束马尔可夫决策过程，通过构建约束违反概率的贝尔曼递归，实现模型基上下界匹配，并提供无模型策略梯度算法，在合成和IEEE 14节点基准上验证安全性。
Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
针对LLM推理中基于策略的RL方法占主导、基于价值的方法未被充分探索的问题，提出轨迹贝尔曼残差最小化（TBRM）算法，无需评论家等组件，在数学推理基准上优于PPO等基线且开销低，为LLM推理提供高效替代方案。
A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
本文提出广义投影贝尔曼误差，将线性 MSPBE 扩展到非线性函数逼近，统一已有方法并给出价值误差新界，所提算法更稳定且对超参数不敏感，在四个控制领域表现优异。
Future-Dependent Value-Based Off-Policy Evaluation in POMDPs
针对POMDP中现有OPE方法受维度灾难限制的问题，提出引入未来依赖价值函数的新无模型OPE方法，推导新贝尔曼方程并采用极小极大学习，在信息充分和贝尔曼完备条件下获得PAC一致性保证，并扩展到动力学学习。该研究为部分可观测环境下的离线评估提供了新途径。
We consider off-policy temporal-difference (TD) learning in discounted Markov decision processes, where the goal is to evaluate a policy in a model-free way by using observations of a state process generated without executing the policy. To curb the high variance issue in off-policy TD learning, we propose a new scheme of setting the $λ$-parameters of TD, based on generalized Bellman equations. Our scheme is to set $λ$ according to the eligibility trace iterates calculated in TD, thereby easily keeping these traces in a desired bounded range. Compared with prior work, this scheme is more direc…
Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problem
本文提出基于加权Lp范数的软贝尔曼残差最小化方法，通过调整p值对齐贝尔曼算子的收缩几何特性，推导出性能误差界，改进误差传播控制并保持梯度优化兼容性。
BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
BRACE通过锚定贝尔曼残差校正，解决异步RL中评论家偏差问题，提升BrowseComp-Plus性能2.4%，速度提升2.46倍。
Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variant of Bellman operator that is not necessarily a contraction. As a result, they may easily lose convergence guarantees, as can be observed in practice. In this paper, we propose a novel loss function, which can be optimized using standard gradient-based methods without risking divergence. The key advantage is that its gradient can be easily approximated using sampled t…
We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, thereby extending the potential exploratory benefit of a policy beyond individual time-steps. We prove that the unique fixed point of the UBE yields an upper bound on the variance of the posteri…
Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at all. We answer both the questions affirmatively in the overparameterized regime. In detail, we prove that neural natural policy gradient converges to a globally optimal policy at a sublinear rate. Also, we show that neural vanilla policy gradient converges sublinearly to a statio…
Bellman Error Centering
本文揭示基于价值的奖励中心化本质上是贝尔曼误差中心化，提出中心化不动点及 CTD、CTDC 算法并证明收敛性，实验验证其稳定性。
Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning
本文提出了一种基于多级蒙特卡洛技术的鲁棒策略评估方法，实现了最优样本复杂度，解决了鲁棒平均奖励强化学习中的样本复杂度问题。
Lifted Bellman Linear Programming for Offline Reinforcement Learning
本文提出提升的贝尔曼线性规划（LBLP）及其近似算法ALBUM，通过不等式约束实现样本内贝尔曼最优性，无需目标网络，在OGBench上以最少参数和内存达到与FQL相当的性能。
Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we pre…
MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
针对离线强化学习的分布偏移问题，提出带保守贝尔曼算子的基于模型的算法MICRO，通过在状态不确定集选最小Q值兼顾性能与鲁棒性，降低计算成本，在基准测试中表现更优且抗扰动能力强。
On Bellman equations for continuous-time policy evaluation I: discretization and approximation
针对连续时间扩散过程的离散观测策略评估，提出兼容函数逼近型离散时间 RL 的易实现数值算法，证明其高阶精度与误差保证，并借助椭圆结构实现不随有效时域发散的有界逼近因子。
Bellman Calibration for $V$-Learning in Offline Reinforcement Learning
本文提出贝尔曼校准方法，用于解决离线强化学习中长期价值预测的可靠性问题，通过校准误差评估和迭代贝尔曼校准方法提升价值预测性能。
Prospect-Theory Behavior from Bellman Optimality in MDPs with Catastrophic States
研究具有吸收灾难状态的马尔可夫决策过程中的风险中性控制，发现标准贝尔曼最优性产生前景理论特征：S形值函数、内生损失敏感系数和反射效应策略反转，并推导出渐近损失厌恶平台的闭式表达式。
Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics
本文提出QAvatar方法，通过域间Bellman一致性与混合批评者解决跨领域RL中的可转移性问题，实现可靠的知识迁移。
In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the propos…
Revisiting Bellman Errors for Offline Model Selection
本文重新审视基于贝尔曼误差的离线模型选择，解释先前悲观结果的原因，提出更准确的MSBE估计器，并在Atari等离散控制任务上取得优异性能。
Robust Markov decision processes (MDPs) allow to compute reliable solutions for dynamic decision problems whose evolution is modeled by rewards and partially-known transition probabilities. Unfortunately, accounting for uncertainty in the transition probabilities significantly increases the computational complexity of solving robust MDPs, which severely limits their scalability. This paper describes new efficient algorithms for solving the common class of robust MDPs with s- and sa-rectangular ambiguity sets defined by weighted $L_1$ norms. We propose partial policy iteration, a new, efficient…
This paper develops an inverse reinforcement learning algorithm aimed at recovering a reward function from the observed actions of an agent. We introduce a strategy to flexibly handle different types of actions with two approximations of the Bellman Optimality Equation, and a Bellman Gradient Iteration method to compute the gradient of the Q-value with respect to the reward function. These methods allow us to build a differentiable relation between the Q-value and the reward function and learn an approximately optimal reward function with gradient methods. We test the proposed method in two si…
Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions
针对线性函数逼近的在线强化学习中Bellman完备性下现有统计算法需解非凸优化、缺乏计算高效方法的问题，本文首次提出动作数为任意常数时满足线性Bellman完备性的多项式时间RL算法。
A Unified Bellman Operator for Safety-Critical Reinforcement Learning
本研究提出统一贝尔曼算子，将安全关键强化学习的性能与安全目标整合，经理论证明收敛，在连续控制任务上实现近零安全违规的稳定收敛。
End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions
针对线性Bellman完备MDP，提出一种在确定性转移下计算高效的算法，对有限动作空间端到端高效，对无限动作空间仅需标准argmax oracle，实现多项式样本与计算复杂度。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。