EXTRACT：从离线数据中提取可迁移机器人技能的高效策略学习
EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
AI总结：
EXTRACT利用预训练视觉语言模型从离线数据中提取语义技能，无需人工监督，使机器人通过选择技能和调整参数快速学习新任务，在稀疏奖励图像操作环境中显著提升样本效率和性能。
摘要：
大多数强化学习（RL）方法侧重于在低层动作空间上学习最优策略。虽然这些方法在训练环境中表现良好，但它们缺乏迁移到新任务的灵活性。相反，能够在有用的、时间上扩展的技能而非低层动作上进行操作的强化学习智能体可以更容易地学习新任务。先前基于技能的强化学习工作要么需要专家监督来定义有用的技能，这难以扩展，要么使用启发式方法从离线数据中学习技能空间，这限制了技能的适应性，使其在下游强化学习过程中难以迁移。我们的方法EXTRACT利用预训练的视觉语言模型从离线数据中提取一组离散的、语义上有意义的技能，每个技能由连续参数参数化，无需人工监督。这种技能参数化使机器人只需学习何时选择特定技能以及如何针对特定任务修改其参数，即可学习新任务。我们通过在稀疏奖励、基于图像的机器人操作环境中的实验证明，EXTRACT可以比先前工作更快地学习新任务，在样本效率和性能上相比先前基于技能的强化学习有显著提升。网站：https://www.jessezhang.net/projects/extract/。
英文摘要：
Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces. While these methods can perform well in their training environments, they lack the flexibility to transfer to new tasks. Instead, RL agents that can act over useful, temporally extended skills rather than low-level actions can learn new tasks more easily. Prior work in skill-based RL either requires expert supervision to define useful skills, which is hard to scale, or learns a skill-space from offline data with heuristics that limit the adaptability of the skills, making them difficult to transfer during downstream RL. Our approach, EXTRACT, instead utilizes pre-trained vision language models to extract a discrete set of semantically meaningful skills from offline data, each of which is parameterized by continuous arguments, without human supervision. This skill parameterization allows robots to learn new tasks by only needing to learn when to select a specific skill and how to modify its arguments for the specific task. We demonstrate through experiments in sparse-reward, image-based, robot manipulation environments that EXTRACT can more quickly learn new tasks than prior works, with major gains in sample efficiency and performance over prior skill-based RL. Website at https://www.jessezhang.net/projects/extract/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
