利用非专家数据通过离线强化学习增强模仿学习的鲁棒性
Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
AI总结：
本研究提出利用离线强化学习算法修改，有效整合非专家数据以增强模仿学习策略的鲁棒性，显著提升操作任务的恢复与泛化能力。
摘要：
模仿学习已被证明能有效训练机器人从专家人类演示中执行复杂任务。然而，它仍然受限于对高质量、任务特定数据的依赖，这限制了其对现实世界中多样化物体配置和场景的适应性。相比之下，非专家数据——如玩耍数据、次优演示、部分任务完成或次优策略的轨迹——可以提供更广泛的覆盖范围和更低的收集成本。然而，传统的模仿学习方法无法有效利用这些数据。为了解决这些挑战，我们提出，通过正确的设计决策，离线强化学习可以作为工具，利用非专家数据来增强模仿学习策略的性能。我们表明，在现实世界中常见的稀疏数据覆盖设置下，标准的离线强化学习方法可能无法有效利用非专家数据，但简单的算法修改可以在没有显著额外假设的情况下实现对这些数据的利用。我们的方法表明，拓宽策略分布的支撑可以使由离线强化学习增强的模仿算法稳健地解决任务，展现出显著增强的恢复和泛化行为。在操作任务中，这些创新显著增加了在纳入非专家数据时学习策略成功的初始条件范围。此外，我们表明这些方法能够利用所有收集的数据，包括部分或次优演示，以增强任务导向的策略性能。这强调了在机器人学中利用非专家数据进行稳健策略学习的算法技术的重要性。网站：https://uwrobotlearning.github.io/RISE-offline/
英文摘要：
Imitation learning has proven effective for training robots to perform complex tasks from expert human demonstrations. However, it remains limited by its reliance on high-quality, task-specific data, restricting adaptability to the diverse range of real-world object configurations and scenarios. In contrast, non-expert data -- such as play data, suboptimal demonstrations, partial task completions, or rollouts from suboptimal policies -- can offer broader coverage and lower collection costs. However, conventional imitation learning approaches fail to utilize this data effectively. To address these challenges, we posit that with right design decisions, offline reinforcement learning can be used as a tool to harness non-expert data to enhance the performance of imitation learning policies. We show that while standard offline RL approaches can be ineffective at actually leveraging non-expert data under the sparse data coverage settings typically encountered in the real world, simple algorithmic modifications can allow for the utilization of this data, without significant additional assumptions. Our approach shows that broadening the support of the policy distribution can allow imitation algorithms augmented by offline RL to solve tasks robustly, showing considerably enhanced recovery and generalization behavior. In manipulation tasks, these innovations significantly increase the range of initial conditions where learned policies are successful when non-expert data is incorporated. Moreover, we show that these methods are able to leverage all collected data, including partial or suboptimal demonstrations, to bolster task-directed policy performance. This underscores the importance of algorithmic techniques for using non-expert data for robust policy learning in robotics. Website: https://uwrobotlearning.github.io/RISE-offline/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
