基于分层自适应机会约束防护的安全强化学习
Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards
AI总结：
针对约束马尔可夫决策过程下强化学习难以平衡安全性与最优性的问题，提出基于安全恢复率的自适应机会约束防护算法，理论上保证安全集前向不变性，实验中实现近乎零违规且最优性提升23.8%。
摘要：
在强化学习（RL）中保障安全性（通常被建模为约束马尔可夫决策过程（CMDP））对现实世界的探索应用至关重要。当前处理CMDP的方法难以平衡最优性与可行性：直接优化方法无法保证训练过程中逐状态的安全性，而基于投影的方法需通过冗长迭代修正动作，效率低下。为应对这些挑战，我们提出自适应机会约束防护（ACS）——一种自适应的无模型安全RL算法，以安全恢复率作为代理机会约束，在探索过程中及收敛后迭代式保障安全性。理论分析表明，该松弛概率约束足以充分保证安全集的前向不变性。在模拟与现实世界安全关键任务上开展的大量实验证明，该方法可有效保障安全性（违规率近乎为零），同时在随机现实场景中保留最优性（提升23.8%）、鲁棒性与快速响应能力。
英文摘要：
Ensuring safety in Reinforcement Learning (RL), typically framed as a Constrained Markov Decision Process (CMDP), is crucial for real-world exploration applications. Current approaches in handling CMDP struggle to balance optimality and feasibility, as direct optimization methods cannot ensure state-wise in-training safety, and projection-based methods correct actions inefficiently through lengthy iterations. To address these challenges, we propose Adaptive Chance-constrained Safeguards (ACS), an adaptive, model-free safe RL algorithm using the safety recovery rate as a surrogate chance constraint to iteratively ensure safety during exploration and after achieving convergence. Theoretical analysis indicates that the relaxed probabilistic constraint sufficiently guarantees forward invariance to the safe set. And extensive experiments conducted on both simulated and real-world safety-critical tasks demonstrate its effectiveness in enforcing safety (nearly zero-violation) while preserving optimality (+23.8%), robustness, and fast response in stochastic real-world settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
