鲁棒概率防护用于安全的离线强化学习
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
AI总结：
本文提出一种鲁棒概率防护方法，结合安全策略改进与防护机制，在离线强化学习中确保策略性能与安全，实验表明其在低数据环境下表现更优。
摘要：
在离线强化学习（RL）中，我们从固定数据集学习策略而不进行环境交互。主要挑战是确保生成策略的性能和安全性。安全策略改进（SPI）提供性能保证：以高概率，新策略优于给定的基线策略，该基线假设安全。同时，在安全RL中，防护机制通过限制动作空间到可证明安全的动作来提供安全保证。本文通过将防护扩展到离线RL中，仅依赖可用数据集和对安全与不安全状态的知识，然后对策略改进步骤进行防护，以高概率保证安全策略。实验结果表明，防护的SPI优于未防护的版本，在平均和最坏情况性能上均有提升，特别是在低数据环境下。
英文摘要：
In offline reinforcement learning (RL), we learn policies from fixed datasets without environment interaction. The major challenges are to provide guarantees on the (1) performance and (2) safety of the resulting policy. A technique called safe policy improvement (SPI) provides a performance guarantee: with high probability, the new policy outperforms a given baseline policy, which is assumed to be safe. Orthogonally, in the context of safe RL, a shield provides a safety guarantee by restricting the action space to those actions that are provably safe with respect to a given safety-relevant model. We integrate these paradigms by extending shielding to offline RL, relying solely on the available dataset and knowledge of safe and unsafe states. Then, we shield the policy improvement steps, guaranteeing, with high probability, a safe policy. Experimental results demonstrate that shielded SPI outperforms its unshielded counterpart, improving both average and worst-case performance, particularly in low-data regimes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
