arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.00899cs.RO
收藏

RoboVQA:面向机器人的多模态长程推理

RoboVQA: Multimodal Long-Horizon Reasoning for Robotics

Google DeepMind(谷歌DeepMind)
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhi… 展开作者

Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, Pete Florence, Wei Han, Robert Baruch, Yao Lu, Suvir Mirchandani, Peng Xu, Pannag Sanketi, Karol Hausman, Izhak Shafran, Brian Ichter, Yuan Cao

AI总结:

本文提出RoboVQA数据集与可扩展多具身数据收集方案,并训练RoboVQA-VideoCoCa,显著降低真实机器人长程任务的认知干预率。

摘要:

我们提出了一种可扩展、自下而上且本质上多样化的数据收集方案,可用于长程和中程的高层推理,与传统狭窄的自上而下逐步收集相比,其吞吐量提高2.2倍。我们通过在全部3栋办公楼内执行任意用户请求,并使用多种机器人和人类具身形态来收集真实数据。利用这些数据,我们表明,在所有具身形态上训练的模型表现优于仅在机器人数据上训练的模型,即使仅在机器人片段上评估也是如此。我们发现,在固定收集预算下,同时利用成本更低的人类收集和机器人收集是有益的。我们发布了一个大型且高度多样化的数据集,包含29,520条独特指令,名为RoboVQA,其中含有829,502个用于面向机器人的视觉问答的(视频,文本)对。我们还展示了如何用干预机制评估真实机器人实验,使任务能够执行至完成,从而即使模型不完美也能在人类监督下部署,同时提供单一性能指标。我们展示了一个在我们的数据集上训练、名为RoboVQA-VideoCoCa的单一视频条件模型,它能够在广泛的真实环境中执行各种接地的高层推理任务,认知干预率比零样本最先进视觉语言模型(VLM)基线低46%,并能够引导真实机器人完成长程任务。与零样本最先进模型之间的性能差距表明,要实现真实世界部署,仍有大量接地数据有待收集,这凸显了可扩展数据收集方法的关键需求。最后,我们表明视频VLM显著优于单图像VLM,在所有VQA任务上平均错误率降低19%。数据和视频可在https://robovqa.github.io获取。

英文摘要:

We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiments. With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We find that for a fixed collection budget it is beneficial to take advantage of cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos available at https://robovqa.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑