arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.18240cs.CVcs.AIcs.LGcs.RO
收藏

在面向具身智能的人工视觉皮层探索中,我们处于什么阶段?

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

Georgia Tech(佐治亚理工学院);Meta AI;UPenn(宾夕法尼亚大学);Stanford University(斯坦福大学);UC Berkeley(加州大学伯克利分校)
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhr… 展开作者

Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, Franziska Meier

AI总结:

该研究构建含17项任务的CortexBench基准,系统评估预训练视觉表征,训练出VC-1模型并验证适配提升效果,通过真实硬件实验验证性能,为具身智能视觉皮层研究提供全面实证与开源资源。

摘要:

我们开展了迄今为止规模最大、最全面的针对具身人工智能的预训练视觉表征(PVRs,又称视觉“基础模型”)的实证研究。首先,我们构建了CortexBench基准,涵盖17项不同任务,涉及运动、导航、灵巧操作以及移动操作领域。随后,我们对现有PVRs进行了系统性评估,发现没有任何一种模型具备普遍优势。\n为研究预训练数据规模与多样性的影响,我们整合了来自7个不同来源的超4000小时第一人称视角视频(对应超430万张图像)与ImageNet数据集,采用掩码自编码器(MAE)在该数据的不同子集上训练不同规模的视觉Transformer。与既往研究的推论相反,我们发现扩大数据集规模与多样性并不能带来普遍的性能提升,但平均性能会有所提高。我们的最大模型VC-1平均表现优于所有现有PVRs,但同样不具备普遍优势。\n接下来我们证实,对VC-1进行任务或领域特定的适配可带来显著性能提升:适配后的VC-1在CortexBench的所有基准上,性能均达到与已知最优结果相当甚至更优的水平。最后我们开展了真实硬件实验,VC-1与适配后的VC-1表现均优于此前性能最强的PVR。总体而言,本文未提出新技术,而是提供了严谨的系统性评估、一系列关于PVRs的广泛发现(部分结论反驳了既往窄领域研究的推论),并开源了训练耗时超10000 GPU小时的代码与模型,以助力研究社区发展。

英文摘要:

We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑