arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.05150cs.CLcs.AIcs.LGcs.RO
收藏

基于自然语言约束的安全强化学习

Safe Reinforcement Learning with Natural Language Constraints

Princeton University(普林斯顿大学);Google Research(谷歌研究院)

Tsung-Yen Yang, Michael Hu, Yinlam Chow, Peter J. Ramadge, Karthik Narasimhan

摘要:

虽然安全的强化学习(RL)在机器人或自动驾驶汽车等许多实际应用中具有很大的前景,但目前的方法需要以数学形式指定约束。这些规范需要领域专业知识,限制了安全RL的采用。在本文中,我们提出了学习解释自然语言的安全RL的约束。为此,我们首先介绍了HazardWorld,一个新的多任务基准,需要一个代理优化奖励,同时不违反自由形式的文本中指定的约束。然后,我们开发了一个代理的模块化架构,可以解释和坚持这样的文本约束,同时学习新的任务。我们的模型包括(1)一个约束解释器,它将文本约束编码为禁止状态的空间和时间表示,以及(2)一个策略网络,它使用这些表示来产生一个策略,在训练过程中实现最小的约束违反。在HazardWorld的不同领域中,我们表明,与现有方法相比,我们的方法实现了更高的奖励(高达11倍)和更少的约束违反(1.8倍)。然而,就绝对性能而言,HazardWorld仍然对智能体有效学习提出了重大挑战,激发了未来工作的需求。

英文摘要:

While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HazardWorld, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HazardWorld, we show that our method achieves higher rewards (up to11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HazardWorld still poses significant challenges for agents to learn efficiently, motivating the need for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑