最小监督下的安全强化学习
Safe Reinforcement Learning with Minimal Supervision
AI总结：
针对安全强化学习依赖示范数据的问题，提出基于无监督RL的离线数据收集流程和乐观遗忘方法，以在极少监督下实现安全在线探索。
摘要：
现实世界中的强化学习（RL）需要开发相应流程，使智能体能够在探索时不伤害自身或他人。安全RL问题最成功的解决方案利用离线数据学习安全集，从而实现安全的在线探索。然而，这种安全学习方法往往受到可用于学习的示范数据限制。本文研究离线训练初始安全学习问题所用数据的数量和质量，对在线学习安全RL策略能力的影响。具体而言，我们关注具有空间扩展目标状态且几乎没有或完全没有可用示范的任务。传统上，该问题通过使用手工设计的控制器生成数据，或收集用户生成的示范来解决。但这些方法通常成本高昂，且无法扩展到更复杂的任务和环境。为克服这一局限，我们提出一种基于无监督RL的离线数据收集流程，无需手工设计的控制器或用户示范即可学习复杂且可扩展的策略。研究表明，为智能体提供足够示范以在线学习最优安全RL策略具有重要意义；因此，我们提出乐观遗忘（optimistic forgetting），这是一种适用于数据有限场景的新型在线安全RL方法。此外，我们的无监督数据收集方法强调，需要在安全在线探索中平衡多样性与最优性。
英文摘要：
Reinforcement learning (RL) in the real world necessitates the development of procedures that enable agents to explore without causing harm to themselves or others. The most successful solutions to the problem of safe RL leverage offline data to learn a safe-set, enabling safe online exploration. However, this approach to safe-learning is often constrained by the demonstrations that are available for learning. In this paper we investigate the influence of the quantity and quality of data used to train the initial safe learning problem offline on the ability to learn safe-RL policies online. Specifically, we focus on tasks with spatially extended goal states where we have few or no demonstrations available. Classically this problem is addressed either by using hand-designed controllers to generate data or by collecting user-generated demonstrations. However, these methods are often expensive and do not scale to more complex tasks and environments. To address this limitation we propose an unsupervised RL-based offline data collection procedure, to learn complex and scalable policies without the need for hand-designed controllers or user demonstrations. Our research demonstrates the significance of providing sufficient demonstrations for agents to learn optimal safe-RL policies online, and as a result, we propose optimistic forgetting, a novel online safe-RL approach that is practical for scenarios with limited data. Further, our unsupervised data collection approach highlights the need to balance diversity and optimality for safe online exploration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
