超越硬约束：用于安全离线强化学习的预算条件可达性
Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
AI总结：
本文提出一种预算条件可达性方法，用于安全离线强化学习，通过解耦奖励最大化与累积安全成本约束，实现安全策略学习。
摘要：
马尔可夫决策过程支撑了许多现实应用。基于模型和无模型方法在这些设置中均取得了显著成果。然而，现实任务必须在奖励最大化与安全约束之间取得平衡，这些往往冲突的目标可能导致不稳定min/max、对抗性优化。一种有前途的替代方法是安全可达性分析，它预计算一个向前不变的安全状态、动作集，确保代理在该集合内启动后可保持安全。然而，大多数基于可达性的方法仅处理硬安全约束，很少有工作将可达性扩展到累积成本约束。为此，首先定义了一个安全条件可达集，将奖励最大化与累积安全成本约束解耦。其次，展示如何通过该集强制安全约束而无需不稳定min/max或拉格朗日优化，从而获得一种新颖的离线安全RL算法，从固定数据集中学习安全策略而无需环境交互。最后，实验表明，在标准离线安全RL基准和现实世界航海导航任务中，本文方法在保持安全的同时，匹配或优于现有最先进基线。
英文摘要：
Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
