离线安全强化学习使用轨迹分类
Offline Safe Reinforcement Learning Using Trajectory Classification
AI总结：
本文提出一种基于轨迹分类的离线安全强化学习方法，通过将轨迹分为理想与非理想集合并学习分类器来指导策略生成，绕过了极小极大目标的复杂性，在DSRL基准上取得更高奖励和更好约束满足。
摘要：
离线安全强化学习（RL）已成为一种有前景的方法，用于在不与环境进行风险在线交互的情况下学习安全行为。现有的大多数离线安全RL方法依赖于每个时间步的成本约束（从全局成本约束推导而来），这可能导致策略过于保守或违反安全约束。在本文中，我们提出学习一种生成理想轨迹并避免非理想轨迹的策略。具体来说，我们首先将预收集的状态-动作轨迹数据集划分为理想和非理想子集。直观上，理想集包含高奖励和安全轨迹，而非理想集包含不安全轨迹和低奖励的安全轨迹。其次，我们学习一种生成理想轨迹并避免非理想轨迹的策略，其中（非）理想性分数由从理想和非理想轨迹数据集学习到的分类器提供。这种方法绕过了现有方法中采用的极小极大目标所带来的计算复杂性和稳定性问题。理论上，我们还展示了我们的方法与涉及人类反馈的现有学习范式之间的强联系。最后，我们使用DSRL基准对离线安全RL进行了广泛评估。实验上，我们的方法优于竞争基线，在多种基准任务中实现了更高的奖励和更好的约束满足。
英文摘要：
Offline safe reinforcement learning (RL) has emerged as a promising approach for learning safe behaviors without engaging in risky online interactions with the environment. Most existing methods in offline safe RL rely on cost constraints at each time step (derived from global cost constraints) and this can result in either overly conservative policies or violation of safety constraints. In this paper, we propose to learn a policy that generates desirable trajectories and avoids undesirable trajectories. To be specific, we first partition the pre-collected dataset of state-action trajectories into desirable and undesirable subsets. Intuitively, the desirable set contains high reward and safe trajectories, and undesirable set contains unsafe trajectories and low-reward safe trajectories. Second, we learn a policy that generates desirable trajectories and avoids undesirable trajectories, where (un)desirability scores are provided by a classifier learnt from the dataset of desirable and undesirable trajectories. This approach bypasses the computational complexity and stability issues of a min-max objective that is employed in existing methods. Theoretically, we also show our approach's strong connections to existing learning paradigms involving human feedback. Finally, we extensively evaluate our method using the DSRL benchmark for offline safe RL. Empirically, our method outperforms competitive baselines, achieving higher rewards and better constraint satisfaction across a wide variety of benchmark tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
