用于离线强化学习与模仿学习的引导式数据增强
Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning
AI总结：
本文提出引导式数据增强框架GuDA，通过人类对轨迹进展的简单判断自动筛选专家质量增强数据，在少量次优数据下提升离线强化学习与模仿学习性能，并优于随机和基于模型的数据增强策略。
摘要：
在离线强化学习（RL）中，RL智能体仅使用先前收集的固定数据集来学习解决任务。虽然离线RL已在学习真实世界机器人控制策略方面取得成功，但它通常需要大量专家质量的数据，才能学习出能够泛化到分布外状态的有效策略。遗憾的是，在真实世界任务中获取此类数据往往困难且成本高昂。近期一些工作利用数据增强（DA）以低成本生成额外数据，但大多数DA工作以随机方式应用增强，最终产生高度次优的增强经验。在本工作中，我们提出引导式数据增强（GuDA），这是一种由人类引导的DA框架，能够生成专家质量的增强数据。GuDA背后的关键洞察是：虽然演示生成专家数据所需的动作序列可能很困难，但用户通常可以容易地刻画一条增强轨迹片段是否代表朝向任务完成的进展。因此，用户可以限制可能增强的空间，从而自动拒绝次优的增强数据。为了从GuDA中提取策略，我们使用现成的离线强化学习和行为克隆算法。我们在一个物理机器人足球任务以及模拟的D4RL导航任务、一个模拟自动驾驶任务和一个模拟足球任务上评估了GuDA。实验结果表明，GuDA能够在仅有少量可能次优的初始经验数据集的情况下实现学习，并且优于随机DA策略以及基于模型的DA策略。
英文摘要：
In offline reinforcement learning (RL), an RL agent learns to solve a task using only a fixed dataset of previously collected data. While offline RL has been successful in learning real-world robot control policies, it typically requires large amounts of expert-quality data to learn effective policies that generalize to out-of-distribution states. Unfortunately, such data is often difficult and expensive to acquire in real-world tasks. Several recent works have leveraged data augmentation (DA) to inexpensively generate additional data, but most DA works apply augmentations in a random fashion and ultimately produce highly suboptimal augmented experience. In this work, we propose Guided Data Augmentation (GuDA), a human-guided DA framework that generates expert-quality augmented data. The key insight behind GuDA is that while it may be difficult to demonstrate the sequence of actions required to produce expert data, a user can often easily characterize when an augmented trajectory segment represents progress toward task completion. Thus, a user can restrict the space of possible augmentations to automatically reject suboptimal augmented data. To extract a policy from GuDA, we use off-the-shelf offline reinforcement learning and behavior cloning algorithms. We evaluate GuDA on a physical robot soccer task as well as simulated D4RL navigation tasks, a simulated autonomous driving task, and a simulated soccer task. Empirically, GuDA enables learning given a small initial dataset of potentially suboptimal experience and outperforms a random DA strategy as well as a model-based DA strategy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
