Imitation learning has emerged as a promising approach towards building generalist robots. However, scaling imitation learning for large robot foundation models remains challenging due to its reliance on high-quality expert demonstrations. Meanwhile, large amounts of video data depicting a wide range of environments and diverse behaviors are readily available. This data provides a rich source of information about real-world dynamics and agent-environment interactions. Leveraging this data directly for imitation learning, however, has proven difficult due to the lack of action annotation. In this work, we present Unified World Models (UWM), a framework that allows for leveraging both video and action data for policy learning. Specifically, a UWM integrates an action diffusion process and a video diffusion process within a unified transformer architecture, where independent diffusion timesteps govern each modality. By controlling each diffusion timestep, UWM can flexibly represent a policy, a forward dynamics, an inverse dynamics, and a video generator. Through simulated and real-world experiments, we show that: (1) UWM enables effective pretraining on large-scale multitask robot datasets with both dynamics and action predictions, resulting in more generalizable and robust policies than imitation learning, (2) UWM naturally facilitates learning from action-free video data through independent control of modality-specific diffusion timesteps, further improving the performance of finetuned policies. Our results suggest that UWM offers a promising step toward harnessing large, heterogeneous datasets for scalable robot learning, and provides a simple unification between the often disparate paradigms of imitation learning and world modeling. Videos and code are available at https://weirdlabuw.github.io/uwm/.
The Chinese University of Hong Kong（香港中文大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Deep visuomotor policy learning, which aims to map raw visual observation to action, achieves promising results in control tasks such as robotic manipulation and autonomous driving. However, it requires a huge number of online interactions with the training environment, which limits its real-world application. Compared to the popular unsupervised feature learning for visual recognition, feature pretraining for visuomotor control tasks is much less explored. In this work, we aim to pretrain policy representations for driving tasks by watching hours-long uncurated YouTube videos. Specifically, w…
We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a …
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）
GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learning
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（国家一般人工智能重点实验室，北京大学深圳研究生院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；State Key Laboratory of Internet of Things for Smart City, University of Macau（智慧城市物联网重点实验室，澳门大学）
Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video capti…
UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning
Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China.（清华大学交叉信息研究院）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院）；Peking University, Beijing, China（北京大学）；Shanghai AI Lab, Shanghai, China（上海人工智能实验室）
ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen（移动网络与移动多媒体技术国家重点实验室（深圳））；ZTE Corporation（中兴通讯）；Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）
Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息处理国家重点实验室，北京大学计算机学院）；AI 2 Robotics（AI与机器人）；Hong Kong University of Science and Technology（香港科学与技术大学）
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis（广东大数据分析与处理重点实验室）；X-Era AI Lab（X-Era人工智能实验室）；Guangdong University of Technology（广东工业大学）
Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly
Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）；Technical University of Munich（慕尼黑技术大学）；Aalto University（艾尔沃斯大学）；Shanghai University（上海大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Learning an accurate model of the environment is essential for model-based control tasks. Existing methods in robotic visuomotor control usually learn from data with heavily labelled actions, object entities or locations, which can be demanding in many cases. To cope with this limitation, we propose a method, dubbed DMotion, that trains a forward model from video data only, via disentangling the motion of controllable agent to model the transition dynamics. An object extractor and an interaction learner are trained in an end-to-end manner without supervision. The agent's motions are explicitly…
Istituto Italiano di Tecnologia（意大利理工学院）；Australian Centre for Robotic Vision, The University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential informati…
DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos
Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Freiburg（弗赖堡大学）；Technical University of Munich（慕尼黑技术大学）；Huawei Heisenberg Research Center (Munich)（华为海森堡研究中心（慕尼黑））
Istituto Italiano di Tecnologia（意大利理工学院）；The University of Manchester（曼彻斯特大学）
We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our n…
