arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.17600cs.CVcs.RO
收藏

当学习遥不可及时,重置:自主视觉运动强化学习中的泛化

When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learning

Allen Institute for AI(艾伦人工智能研究所)

Zichen Zhang, Luca Weihs

AI总结:

针对无重置强化学习中的不可逆转换和泛化问题,提出最小化重置的视觉运动RL方法,构建新基准和无监督指标,在多任务上以更少重置实现更高成功率。

摘要:

情景式训练,即在每次成功或失败后重置智能体的环境,是训练具身强化学习(RL)智能体的事实标准。环境可以轻松重置这一潜在假设在实践中和理论上都具有局限性:在实践中,重置通常需要现实世界中的人力投入,且在仿真中计算成本高昂;在理论上,我们期望智能体能够在没有干预的情况下持续学习。无需任何重置的学习工作,即无重置强化学习(RF-RL),虽然前景广阔,但受到不可逆转换问题(例如物体破碎)的困扰,这会阻碍学习。此外,RF-RL期间遇到的状态多样性有限和仪器设置问题,意味着研究RF-RL的工作在很大程度上不要求其模型泛化到新环境。在这项工作中,我们转而寻求最小化而非完全消除重置,同时构建能够有意义地泛化的视觉智能体。由于研究泛化此前并非为RF-RL设计的基准的重点,我们提出了一个新的Stretch Pick-and-Place基准,旨在评估跨目标、外观变化和结构变化的泛化能力。此外,为了构建高性能的最小化重置RL智能体,我们提出了用于检测不可逆转换的无监督指标,以及一种实现泛化的单策略训练机制。我们提出的方法显著优于先前的情景式、无重置和最小化重置方法,在Stretch-P&P及另一个流行的RF-RL基准中,以更少的重置次数实现了更高的成功率。最后,我们发现我们提出的方法可以大幅减少训练其他具身任务所需的重置次数,特别是在RoboTHOR ObjectNav中,我们获得了比情景式方法更高的成功率,且使用的重置次数减少了99.97%。

英文摘要:

Episodic training, where an agent's environment is reset after every success or failure, is the de facto standard when training embodied reinforcement learning (RL) agents. The underlying assumption that the environment can be easily reset is limiting both practically, as resets generally require human effort in the real world and can be computationally expensive in simulation, and philosophically, as we'd expect intelligent agents to be able to continuously learn without intervention. Work in learning without any resets, i.e{.} Reset-Free RL (RF-RL), is promising but is plagued by the problem of irreversible transitions (e.g{.} an object breaking) which halt learning. Moreover, the limited state diversity and instrument setup encountered during RF-RL means that works studying RF-RL largely do not require their models to generalize to new environments. In this work, we instead look to minimize, rather than completely eliminate, resets while building visual agents that can meaningfully generalize. As studying generalization has previously not been a focus of benchmarks designed for RF-RL, we propose a new Stretch Pick-and-Place benchmark designed for evaluating generalizations across goals, cosmetic variations, and structural changes. Moreover, towards building performant reset-minimizing RL agents, we propose unsupervised metrics to detect irreversible transitions and a single-policy training mechanism to enable generalization. Our proposed approach significantly outperforms prior episodic, reset-free, and reset-minimizing approaches achieving higher success rates with fewer resets in Stretch-P\&P and another popular RF-RL benchmark. Finally, we find that our proposed approach can dramatically reduce the number of resets required for training other embodied tasks, in particular for RoboTHOR ObjectNav we obtain higher success rates than episodic approaches using 99.97\% fewer resets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑