Prompt-based learning has been demonstrated as a compelling paradigm contributing to large language models' tremendous success (LLMs). Inspired by their success in language tasks, existing research has leveraged LLMs in embodied instruction following and task planning. In this work, we tackle the problem of training a robot to understand multimodal prompts, interleaving vision signals with text descriptions. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. In this work, we introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts from multi-task expert trajectories. Our methods consist of a two-stage training pipeline that performs inverse dynamics pretraining and multi-task finetuning. To facilitate multimodal understanding, we design our multimodal prompt encoder by augmenting a pretrained LM with a residual connection to the visual input and model the dependencies among action dimensions. Empirically, we evaluate the efficacy of our method on the VIMA-BENCH and establish a new state-of-the-art (10% improvement in success rate). Moreover, we demonstrate that our model exhibits remarkable in-context learning ability. Project page: \url{https://midas-icml.github.io/}.
SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
Institute of Artificial Intelligence, China Telecom(中国电信人工智能研究院);Technical University of Munich(慕尼黑工业大学);Harbin Institute of Technology(哈尔滨工业大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Zhejiang University(浙江大学)
ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen(移动网络与移动多媒体技术国家重点实验室(深圳));ZTE Corporation(中兴通讯);Shanghai Jiao Tong University(上海交通大学);Central South University(中南大学)
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);National Engineering Research Center for Visual Information and Applications(视觉信息与应用国家工程研究中心);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Xi’an Jiaotong University(西安交通大学)
College of Engineering and Physical Sciences, Aston University(工程与物理科学学院,阿斯顿大学);Department of Computer Science Department, Loughborough University(计算机科学系,洛桑大学)
Move-Then-Operate: Behavioral Phasing for Human-Like Robotic Manipulation
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China.(杭州高等研究院,中国科学院大学,中国杭州);University of Science and Technology of China, Hefei, China.(中国科学技术大学,中国合肥);School of Aritificial intelligence, Institute of Artificial Intelligence, University of Science and Technology Beijing, Beijing, China(人工智能学院,人工智能研究院,北京科技大学,中国北京)
Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));MMLab, The Chinese University of Hong Kong(香港中文大学MMLab);X-Humanoid Robots(X-Humanoid机器人);The University of Hong Kong(香港大学);Tsinghua University(清华大学)
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(北京大学计算机科学学院PKU-Agibot实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学)
RoboSynChallenge: Mastering Real-World Dexterity via Generalizing Synthesized Manipulation Skills
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));DexForce(德克斯力公司);The Chinese University of Hong Kong(香港中文大学);The Hong Kong University of Science and Technology(香港科技大学);University of Colorado Anschutz(科罗拉多大学安舒茨医学中心);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Vector Institute(向量研究所);University of Waterloo(滑铁卢大学);Fudan University(复旦大学);Sun Yat-sen University(中山大学);Shenzhen Loop Area Institute (SLAI)(深圳环区研究所)
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Innovation and Research and Development Department, BoardWare Information System Company Ltd.(创新与研发部,BoardWare信息系统有限公司);School of Artificial Intelligence, Nanjing Agricultural University(人工智能学院,南京农业大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学);School of Electronic Engineering and Automation, Guilin University of Electronic Technology(电子工程与自动化学院,桂林电子科技大学)
AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院)
Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua University(清华大学);Wuhan University(武汉大学);Central South University(中南大学);Microsoft Research(微软研究院)
Object-Centric Instruction Augmentation for Robotic Manipulation
School of Computer Science, East China Normal University(华东师范大学计算机科学学院);Midea Group(美的集团);Department of Mathematics, School of Science, Shanghai University(上海大学理学院数学系)
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis(广东大数据分析与处理重点实验室);X-Era AI Lab(X-Era人工智能实验室);Guangdong University of Technology(广东工业大学)
GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learning
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School(国家一般人工智能重点实验室,北京大学深圳研究生院);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究所,中国科学院);State Key Laboratory of Internet of Things for Smart City, University of Macau(智慧城市物联网重点实验室,澳门大学)
AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
Wangxuan Institute of Computer Technology, Peking University(北京大学计算机技术研究院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室)
Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and Shanghai Key Laboratory of Navigation and Location Based Services(自动化与智能感知学院,上海交通大学,导航与基于位置的服务重点实验室);Department of Engineering, Cambridge University(工程系,剑桥大学)