Learning dexterous manipulation from human videos remains an open challenge, particularly in the context of transferring a single demonstration into a spatially generalizable robotic skill and maintaining physically plausible human-like interactions. Unlike prior works that emphasize hand-motion imitation, we introduce an object-centric framework for acquiring dexterous skills from egocentric human RGB-D videos. Within our framework, we formulate the reinforcement learning (RL) objective in terms of object motion to decouple the underlying skill from the specific hand trajectory and impose constraints on hand-object interaction states to promote more anatomically natural hand configurations. Extensive experiments in simulation and on real robots demonstrate that our method enables efficient learning from human videos and generalizable skill transfer across novel spatial configurations.
EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets
School of Astronautics, Harbin Institute of Technology(哈尔滨工业大学航天学院);Lumos Robotic;Suzhou Research Institute, Harbin Institute of Technology(哈尔滨工业大学苏州研究院);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);Nanjing University(南京大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);Fudan University(复旦大学)
DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations
The Hong Kong University of Science and Technology(香港科技大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机学院多媒体信息处理国家重点实验室);Beihang University(北京航空航天大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation
The University of Tokyo(东京大学);ByteDance Seed(字节跳动 Seed);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学)
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology(智能制造装备与技术国家重点实验室);Huazhong University of Science and Technology(华中科技大学);Wuhan Huaweike Intelligent Technology Co., Ltd.(武汉华为凯科技有限公司)
Structural Action Transformer for 3D Dexterous Manipulation
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发智能感知与认知国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心)
Kinematic-Tactile Aligned Dexterous Retargeting with Contact Wrench Space Alignment
School of Mechanical Engineering, Shanghai Jiao Tong University(上海交通大学机械工程学院);College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);International Collaboration for Creative Research Teams, Shanghai Jiao Tong University(上海交通大学国际创意研究团队合作中心);School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)人工智能学院)
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·扎耶德人工智能大学);Technical University of Munich(慕尼黑技术大学);National University of Singapore(新加坡国立大学);Westlake University(西湖大学)
Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration
University of California at Berkeley(加州大学伯克利分校);Technische Universität Berlin(柏林工业大学)
灵巧的多指手可以完成精细的操作行为,这是不可行的简单的机器人抓手。然而,复杂的多指手往往是昂贵和脆弱的。低成本的软手为更传统的设备提供了一种有吸引力的替代方案,但在传感和致动方面存在相当大的挑战,使得它们难以应用于更复杂的操作任务。在本文中,我们描述了一种方法,从示范,可用于训练软机器人手执行灵巧的操作任务的学习。我们的方法使用以对象为中心的演示,其中人类用自己的手演示所需的操作对象的运动,机器人自主学习模仿这些演示使用强化学习。我们提出了一种新的算法,使我们能够混合和选择一个子集的最可行的示范学习模仿的硬件,我们使用的指导政策搜索框架的扩展,使用多个示范学习概括的神经网络政策。我们在RBO Hand 2上展示了我们的方法,学习了转动阀门,操纵算盘和抓握的运动技能。
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);The University of Hong Kong(香港大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Dexterous Grasping
School of Artificial Intelligence and Robotics, Hunan University(人工智能与机器人学院,湖南大学);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(机器人视觉感知与控制技术国家工程研究中心,湖南大学)