通过价值函数预训练从互联网视频进行机器人离线强化学习
Robotic Offline RL from Internet Videos via Value-Function Pre-Training
AI总结：
本文提出V-PTR系统，通过时序差分学习在互联网视频上预训练价值函数，再结合机器人离线RL，显著提升真实机器人操作任务的性能与泛化能力。
摘要：
在互联网数据上进行预训练已被证明是许多现代机器学习系统实现广泛泛化的关键要素。要在机器人强化学习（RL）中实现这种能力需要什么条件？离线RL方法从机器人经验数据集中学习，为将先验数据引入机器人学习流程提供了一种途径。然而，这些方法与视频数据（如Ego4D）——机器人领域可用的最大规模先验数据集——之间存在“类型不匹配”，因为视频只提供仅含观测的经验，缺少RL方法所需的动作或奖励标注。在本文中，我们开发了一个系统，用于在机器人离线RL中利用大规模人类视频数据集，该系统完全基于通过时序差分学习来学习价值函数。我们表明，在视频数据集上进行价值学习所学习到的表示，比其他从视频数据中学习的方法更有利于下游机器人离线RL。我们的系统称为V-PTR，结合了在视频数据上预训练的优势与在多样化机器人数据上训练的机器人离线RL方法，从而得到在操作任务中表现更好、行动稳健且泛化广泛的价值函数和策略。在真实WidowX机器人上的多个操作任务中，我们的框架产生的策略相比先前方法有大幅提升。我们的视频和更多细节可在 https://dibyaghosh.com/vptr/ 找到。
英文摘要：
Pre-training on Internet data has proven to be a key ingredient for broad generalization in many modern ML systems. What would it take to enable such capabilities in robotic reinforcement learning (RL)? Offline RL methods, which learn from datasets of robot experience, offer one way to leverage prior data into the robotic learning pipeline. However, these methods have a "type mismatch" with video data (such as Ego4D), the largest prior datasets available for robotics, since video offers observation-only experience without the action or reward annotations needed for RL methods. In this paper, we develop a system for leveraging large-scale human video datasets in robotic offline RL, based entirely on learning value functions via temporal-difference learning. We show that value learning on video datasets learns representations that are more conducive to downstream robotic offline RL than other approaches for learning from video data. Our system, called V-PTR, combines the benefits of pre-training on video data with robotic offline RL approaches that train on diverse robot data, resulting in value functions and policies for manipulation tasks that perform better, act robustly, and generalize broadly. On several manipulation tasks on a real WidowX robot, our framework produces policies that greatly improve over prior methods. Our video and additional details can be found at https://dibyaghosh.com/vptr/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
