模仿细心的专家避免灾难性事件
Imitating careful experts to avoid catastrophic events
摘要:
RL越来越多地用于控制与人类密切互动的机器人系统。这种交互提出了安全RL的问题:如何确保RL控制的机器人系统永远不会伤害人类。这个问题在丰富、现实的环境中尤其具有挑战性,在这些环境中,甚至不可能清楚地写下包含这些结果的奖励函数。在这种情况下,也许唯一可行的方法是基于IRL,它从人类演示中推断奖励。然而,由于许多不同的奖励可能导致相同的最优策略,因此IRL是严重欠定的;我们表明,这使得很难区分灾难性的结果(例如伤害人类)和仅仅是不理想的结果。我们的关键见解是,当灾难性结果可能发生时,人类确实会表现出不同的行为:他们变得更加谨慎。我们将谨慎信号纳入IRL,并发现它们确实允许IRL从灾难性结果中消除不希望的歧义,这对于确保未来现实世界中人机交互的安全至关重要。
英文摘要:
RL is increasingly being used to control robotic systems that interact closely with humans. This interaction raises the problem of safe RL: how to ensure that a RL-controlled robotic system never, for instance, injures a human. This problem is especially challenging in rich, realistic settings where it is not even possible to clearly write down a reward function which incorporates these outcomes. In these circumstances, perhaps the only viable approach is based on IRL, which infers rewards from human demonstrations. However, IRL is massively underdetermined as many different rewards can lead to the same optimal policies; we show that this makes it difficult to distinguish catastrophic outcomes (such as injuring a human) from merely undesirable outcomes. Our key insight is that humans do display different behaviour when catastrophic outcomes are possible: they become much more careful. We incorporate carefulness signals into IRL, and find that they do indeed allow IRL to disambiguate undesirable from catastrophic outcomes, which is critical to ensuring safety in future real-world human-robot interactions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。