发表机构
Princeton University（普林斯顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出统一贝尔曼算子，将安全关键强化学习的性能与安全目标整合，经理论证明收敛，在连续控制任务上实现近零安全违规的稳定收敛。
AI 中文摘要
安全关键领域的强化学习需要在最大化任务性能的同时严格遵守安全约束。现有安全强化学习范式通常需要权衡：要么需要先验知识以提供严格的安全保证（例如安全过滤器），要么支持联合学习但仅平均满足安全约束。在本研究中，我们提出了一种新颖的贝尔曼算子，将性能和安全目标统一为联合价值函数。我们证明，使用该联合贝尔曼算子的时序差分学习在双时间尺度随机近似框架下收敛：在快时间尺度上估计学习到的联合策略的安全价值，在慢时间尺度上估计联合价值。通过将极限动力学公式化为占用平均微分包含，并证明其渐近收敛到一组极限最优安全约束任务价值函数，从而确保收敛。理论上，一旦收敛，所得最优策略可最大化任务回报并始终保持安全。在使用神经近似的连续控制任务上的实证评估表明，测试时的安全违规接近零，且收敛稳定。
英文摘要
Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints. Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average. In this work, we propose a novel Bellman operator that unifies performance and safety objectives into a joint value function. We show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework. On the fast timescale, the safety value of the learning joint policy is estimated, while the joint value is estimated on the slow timescale. Convergence is ensured by formulating the limiting dynamics as an occupation-averaged differential inclusion, and showing that it asymptotically converges to a set of limiting optimal safety-constrained task value functions. Theoretically, once converged, the resulting optimal policy maximizes task return while maintaining safety at all times. Empirical evaluations on continuous control tasks with neural approximations demonstrate stable convergence with near-zero safety violations at test time.
