Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\times$ task success rate given the same training data. With $10\times$ less training data, VIMA still performs $2.7\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/
RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Engineering, Great Bay University（大湾区大学工程学院）；School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）
VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation
Humanoid Robot (Shanghai) Co., Ltd.（人形机器人（上海）有限公司）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；School of Astronautics, Harbin Institute of Technology（航天学院，哈尔滨工程大学）
DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院）；School of Computation, Information and Technology, Technical University of Munich, Garching, Germany（慕尼黑技术大学计算与信息学院）；Department of Shenyang Institute of Computing Technology, University of Chinese Academy of Sciences, Beijing, China（中国科学院沈阳计算技术研究所部门）；State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China（新型软件技术国家重点实验室）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）
Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation
Embodied AI and Robotics (AIR) Lab, New York University, New York, USA and NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学Embodied AI和机器人实验室及纽约大学阿布扎赫尔人工智能与机器人中心）；Harvard AI and Robotics Lab, Harvard University, Boston, USA（哈佛大学人工智能与机器人实验室）；Department of Computer Science, University College London, London, UK（伦敦大学学院计算机科学系）；Department of Computer Science, University of Liverpool, UK（利物浦大学计算机科学系）；NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE（纽约大学阿布扎赫尔人工智能与机器人中心）
Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Lab of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究院）；State Key Lab for General Artificial Intelligence（一般人工智能国家重点实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学武汉人工智能研究院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Multimodal Anomaly Detection based on Deep Auto-Encoder for Object Slip Perception of Mobile Manipulation Robots
Artificial Intelligence Institute, Seoul National University（首尔国立大学人工智能研究所）；Dept. of Computer Science and Engineering, Seoul National University（首尔国立大学计算机科学与工程系）；Interdisciplinary Program in Neuroscience, Seoul National University（首尔国立大学神经科学跨学科项目）；Surromind Inc.（Surromind公司）
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University（里士大学）；U.S. National Institute of Standards and Technology（美国国家标准与技术研究院）；Massachusetts Institute of Technology（麻省理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄技术大学）；Autodesk Research（Autodesk研究）；University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（皇家理工学院）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）；ASTM International（美国材料与试验协会）；Carnegie Mellon University（卡内基梅隆大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA Research（NVIDIA研究）
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（机器人与自动信息系统研究所，人工智能学院，南开大学）；Tianjin Key Laboratory of Intelligent Robotics, Nankai University（智能机器人重点实验室，南开大学）；College of Artificial Intelligence, Nankai University（人工智能学院，南开大学）；Department of Computing, The Hong Kong Polytechnic University（computing 部门，香港理工大学）
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis（广东大数据分析与处理重点实验室）；X-Era AI Lab（X-Era人工智能实验室）；Guangdong University of Technology（广东工业大学）
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Midea Group（美的集团）；The Hong Kong University of Science and Technology（香港科技大学）
MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Beijing Innovation Center of Humanoid Robotics (X-Humanoid)（北京类人机器人创新中心（X-Humanoid））；The Chinese University of Hong Kong (CUHK)（香港中文大学（CUHK））
RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
Xiaomi EV（小米电动车）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallagher人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）
Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation
Department of Computer Science（计算机科学系）；The Chinese University of Hong Kong（香港中文大学）；Gaoling School of Artificial Intelligence（九龙人工智能学院）；Renmin University of China（中国人民大学）；AWS AI Amazon（AWS人工智能亚马逊）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Singapore Management University（新加坡管理大学）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身智能研究院）
