Distilling knowledge from human demonstrations is a promising way for robots to learn and act. Existing methods, which often rely on coarsely-aligned video pairs, are typically constrained to learning global or task-level features. As a result, they tend to neglect the fine-grained frame-level dynamics required for complex manipulation and generalization to novel tasks. We posit that this limitation stems from a vicious circle of inadequate datasets and the methods they inspire. To break this cycle, we propose a paradigm shift that treats fine-grained human-robot alignment as a conditional video generation problem. To this end, we first introduce H&R, a novel third-person dataset containing 2,600 episodes of precisely synchronized human and robot motions, collected using a VR teleoperation system. We then present Human2Robot, a framework designed to leverage this data. Human2Robot employs a Video Prediction Model to learn a rich and implicit representation of robot dynamics by generating robot videos from human input, which in turn guides a decoupled action decoder. Our real-world experiments demonstrate that this approach not only achieves high performance on seen tasks but also exhibits significant one-shot generalization to novel positions, objects, instances, and even new task categories.
Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences(中国科学院机器人与智能制造创新研究院);Westlake University(西湖大学);University of Chinese Academy of Sciences(中国科学院大学)
Co-training with Ego-centric Video and Demonstration for Robot Navigation Task
Department of Informatics, Graduate School of Informatics, Kyoto University(信息学系,京都大学研究生院);Spatial Robotics Research Center, Fujitsu Limited(空间机器人研究中心,富士通有限公司)
EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
School of Interactive Computing, Georgia Institute of Technology(交互计算学院,佐治亚理工学院);InnoCORE AI-Transformed Aerospace Research Center, KAIST(AI转型航空航天研究中心,韩国成均馆大学)
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis(广东大数据分析与处理重点实验室);X-Era AI Lab(X-Era人工智能实验室);Guangdong University of Technology(广东工业大学)
PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);University of Liverpool(利物浦大学);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系)
Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,北京大学计算机学院);AI 2 Robotics(AI与机器人);Hong Kong University of Science and Technology(香港科学与技术大学)
Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly
Munich Institute of Robotics and Machine Intelligence (MIRMI)(慕尼黑机器人与机器智能研究所);Technical University of Munich(慕尼黑技术大学);Aalto University(艾尔沃斯大学);Shanghai University(上海大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Self-Supervised Motion Retargeting with Safety Guarantee
School of Artificial Intelligence, Korea University(高丽大学人工智能学院);Courant Institute of Mathematical Sciences, New York University(纽约大学柯朗数学科学研究所);Technical University of Munich(慕尼黑工业大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)