We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiments. With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We find that for a fixed collection budget it is beneficial to take advantage of cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos available at https://robovqa.github.io
Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya, Aichi, Japan(Nagoya大学微纳米机械科学与工程系);Institutes of Innovation for Future Society, Nagoya University, Nagoya, Aichi, Japan(Nagoya大学未来社会创新研究所)
RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Engineering, Great Bay University(大湾区大学工程学院);School of Computing and Information Technology, Great Bay University(大湾区大学计算与信息技术学院)
OpenRC: An Open-Source Robotic Colonoscopy Framework for Multimodal Data Acquisition and Autonomy Research
Walker Department of Mechanical Engineering, The University of Texas at Austin(德克萨斯大学奥斯汀分校沃克机械工程系);Department of Surgical Oncology, Division of Surgery, The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心外科肿瘤学系);School of Medicine and Health, Technical University of Munich(慕尼黑工业大学医学与健康学院)
M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
School of Software Northwestern Polytechnical University Xi'an, China(软件学院 西安理工大学 西安);Laboratoire L2Tl University Sorbonne Paris Nord Paris, France(L2Tl实验室 索邦巴黎北大学 巴黎)
Preventing Robotic Jailbreaking via Multimodal Domain Adaptation
University of Padova(帕多瓦大学);Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学);University of Pennsylvania(宾夕法尼亚大学);Örebro University(奥雷布罗大学);NVIDIA Research(NVIDIA研究)
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University(机器人与自动信息系统研究所,人工智能学院,南开大学);Tianjin Key Laboratory of Intelligent Robotics, Nankai University(智能机器人重点实验室,南开大学);College of Artificial Intelligence, Nankai University(人工智能学院,南开大学);Department of Computing, The Hong Kong Polytechnic University(computing 部门,香港理工大学)
BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
WheelArm-Sim: A Manipulation and Navigation Combined Multimodal Synthetic Data Generation Simulator for Unified Control in Assistive Robotics
Aerospace and Mechanical Engineering Department, Saint Louis University(圣路易斯大学航空航天与机械工程系);Computer Science Department, Saint Louis University(圣路易斯大学计算机科学系)
AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly
Fudan University(复旦大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信);Tianjin University(天津大学);Northwestern Polytechnical University(西北工业大学);Tsinghua University(清华大学);City University of Hong Kong(香港城市大学)
STREAMS: An Assistive Multimodal AI Framework for Empowering Biosignal Based Robotic Controls
Dept. of Electrical Engineering University of Rhode Island Kingston, RI, USA(电气工程系 罗德岛大学 罗得岛州金斯顿);Computer Engineering Northeastern University Boston, Massachusetts, USA(计算机工程 美国东北大学 波士顿)
Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));College of Engineering, Peking University(北京大学工学院);Sensetime Research(商汤科技研究院);University of Toronto(多伦多大学);University College London(伦敦大学学院);Shanghai AI Lab(上海人工智能实验室)
MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Beijing Innovation Center of Humanoid Robotics (X-Humanoid)(北京类人机器人创新中心(X-Humanoid));The Chinese University of Hong Kong (CUHK)(香港中文大学(CUHK))
REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly
Autonomous Systems Lab, Institute of Computer Technology, TU Wien(自主系统实验室,计算机技术学院,维也纳技术大学);Institute of Robotics and Mechatronics (DLR), German Aerospace Center(机器人与机电系统研究所(DLR),德国航空航天中心)
M2R2: MultiModal Robotic Representation for Temporal Action Segmentation
Autonomous Systems Lab, Technische Universität Wien (TU Wien)(自主系统实验室,维也纳技术大学);Institute of Robotics and Mechatronics, German Aerospace Center (DLR)(机器人与机电研究所,德国航空航天中心)
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis(广东大数据分析与处理重点实验室);X-Era AI Lab(X-Era人工智能实验室);Guangdong University of Technology(广东工业大学)
Advantages of Multimodal versus Verbal-Only Robot-to-Human Communication with an Anthropomorphic Robotic Mock Driver
Centre for Applied Autonomous Sensor Systems (AASS)(应用自主传感器系统中心(AASS));Örebro University(厄勒布鲁大学);Robert Bosch GmbH(罗伯特·博世有限公司);University of Skövde(舍夫德大学);TU Munich(慕尼黑工业大学)
PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability
School of Traffic and Transportation, Beijing Jiaotong University(北京交通大学交通运输学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ObjectEye Inc.(奥森科技(ObjectEye公司));Guangdong Polytechnic Normal University(广东技术师范大学)
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);ETH Zurich(苏黎世联邦理工学院);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
Xiaomi EV(小米电动车);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学 Gallagher人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院)