arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.19664cs.ROcs.LG
收藏

迈向从互联网视频中学习的通用机器人:综述

Towards Generalist Robot Learning from Internet Video: A Survey

University College London(伦敦大学学院);Massachusetts Institute of Technology(麻省理工学院)

Robert McCarthy, Daniel C. H. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G. Thuruthel, Zhibin Li

AI总结:

本文综述从互联网视频中学习(LfV)以缓解机器人数据稀缺的方法,梳理分布偏移、动作标签缺失等挑战与现有技术,并展望可扩展基础模型如何促进通用机器人学习。

摘要:

将深度学习扩展到海量且多样化的互联网数据,已推动了视频生成和自然语言处理等领域的显著突破。然而,机器人学习迄今未能复制这一成功,仍受限于可用数据的稀缺。从视频中学习(LfV)方法旨在通过利用大规模互联网视频扩充传统机器人数据,以解决这一数据瓶颈。这些视频数据提供了关于物理动力学、行为和任务的基础信息,对通用机器人具有很高的参考价值。本综述系统考察了新兴的 LfV 领域。我们首先概述关键概念,包括详细说明 LfV 的基本挑战,例如视频数据中的分布偏移和动作标签缺失。随后,我们全面综述当前用于从大规模互联网视频中提取知识、克服 LfV 挑战以及通过视频信息训练改进机器人学习的方法。综述最后对未来机遇进行了批判性讨论。在此,我们强调需要可扩展的基础模型方法,这些方法能够利用全部可用互联网视频,并增强机器人策略和动力学模型的学习。总体而言,本综述旨在为未来的 LfV 研究提供参考并加以推动,从而促进通用机器人的发展。

英文摘要:

Scaling deep learning to massive and diverse internet data has driven remarkable breakthroughs in domains such as video generation and natural language processing. Robot learning, however, has thus far failed to replicate this success and remains constrained by a scarcity of available data. Learning from videos (LfV) methods aim to address this data bottleneck by augmenting traditional robot data with large-scale internet video. This video data provides foundational information regarding physical dynamics, behaviours, and tasks, and can be highly informative for general-purpose robots. This survey systematically examines the emerging field of LfV. We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data. Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training. The survey concludes with a critical discussion of future opportunities. Here, we emphasize the need for scalable foundation model approaches that can leverage the full range of available internet video and enhance the learning of robot policies and dynamics models. Overall, the survey aims to inform and catalyse future LfV research, driving progress towards general-purpose robots.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑