OSIL: 通过非偏好轨迹推断学习离线安全模仿策略
OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
AI总结：
OSIL通过非偏好轨迹推断安全策略，从离线演示中学习安全且奖励最大化的策略，优于基线方法。
摘要：
本工作解决了离线安全模仿学习（IL）的问题，目标是从没有每时间步安全成本或奖励信息的演示中学习安全且奖励最大化的策略。在许多现实世界领域中，环境中的在线学习可能有风险，准确指定安全成本可能困难。然而，收集反映不良或不安全行为的轨迹通常是可行的，这些轨迹隐含地传达了智能体应避免的内容。我们将这些称为非偏好轨迹。我们提出了一种新的离线安全IL算法OSIL，该算法从非偏好演示中推断安全。我们将安全策略学习公式化为约束马尔可夫决策过程（CMDP）。与依赖显式安全成本和奖励注释不同，OSIL通过推导奖励最大化目标的下界，并学习一个成本模型来估计非偏好行为的可能性来重新公式化CMDP问题。我们的方法允许智能体完全从离线演示中学习安全且奖励最大化的行为。我们实证表明，我们的方法能够学习满足成本约束的更安全策略，而不会降低奖励性能，从而优于几种基线方法。
英文摘要：
This work addresses the problem of offline safe imitation learning (IL), where the goal is to learn safe and reward-maximizing policies from demonstrations that do not have per-timestep safety cost or reward information. In many real-world domains, online learning in the environment can be risky, and specifying accurate safety costs can be difficult. However, it is often feasible to collect trajectories that reflect undesirable or unsafe behavior, implicitly conveying what the agent should avoid. We refer to these as non-preferred trajectories. We propose a novel offline safe IL algorithm, OSIL, that infers safety from non-preferred demonstrations. We formulate safe policy learning as a Constrained Markov Decision Process (CMDP). Instead of relying on explicit safety cost and reward annotations, OSIL reformulates the CMDP problem by deriving a lower bound on reward maximizing objective and learning a cost model that estimates the likelihood of non-preferred behavior. Our approach allows agents to learn safe and reward-maximizing behavior entirely from offline demonstrations. We empirically demonstrate that our approach can learn safer policies that satisfy cost constraints without degrading the reward performance, thus outperforming several baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
