In-hand manipulation of tools using dexterous hands in real-world is an underexplored problem in the literature. In addition to more complex geometry and larger size of the tools compared to more commonly used objects like cubes or cylinders, task oriented in-hand tool manipulation involves many sub-tasks to be performed sequentially. This may involve reaching to the tool, picking it up, reorienting it in hand with or without regrasping to reach to a desired final grasp appropriate for the tool usage, and carrying the tool to the desired pose. Research on long-horizon manipulation using dexterous hands is rather limited and the existing work focus on learning the individual sub-tasks using a method like reinforcement learning (RL) and combine the policies for different subtasks to perform a long horizon task. However, in general a single method may not be the best for all the sub-tasks, and this can be more pronounced when dealing with multi-fingered hands manipulating objects with complex geometry like tools. In this paper, we investigate the use of a mixed-method approach to solve for the long-horizon task of tool usage and we use imitation learning, reinforcement learning and model based control. We also discuss a new RL-based teacher-student framework that combines real world data into offline training. We show that our proposed approach for each subtask outperforms the commonly adopted reinforcement learning approach across different subtasks and in performing the long horizon task in simulation. Finally we show the successful transferability to real world.
DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo
NLPR & MAIS, CASIA(中国科学院南京自动化研究所与模式识别与人工智能实验室);SJTU(上海交通大学);MBZUAI(马克斯·普朗克研究所);Beijing Institute of Basic Medical Sciences(北京基础医学研究所);PKU & Galbot(北京大学与Galbot);CUHK(香港大学)
PartDexTOG: Generating Dexterous Task-Oriented Grasping via Language-driven Part Analysis
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
DexTOG: Learning Task-Oriented Dexterous Grasp with Language
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);AI Institute, Shanghai Jiao Tong University(上海交通大学AI研究院);Southeast University(东南大学)
Sampling-Based Model Predictive Control for Dexterous Manipulation on a Biomimetic Tendon-Driven Hand
Soft Robotics Lab, Department of Mechanical and Process Engineering, ETH Zurich(苏黎世联邦理工学院机械与过程工程系软机器人实验室);School of Computer Science, University of Leeds(利兹大学计算机科学学院)
Scenario Convex Programs for Dexterous Manipulation under Modeling Uncertainties
Université Paris-Saclay(巴黎萨克雷大学);CNRS(法国国家科学研究中心);CentraleSupélec(中央高等电力学院);ESME Sudria Engineering School(ESME苏德里亚工程学校);College of Computing, University Mohammed VI Polytechnic(穆罕默德六世理工大学计算机学院);ESME Research Lab.(ESME研究实验室)
NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation
Robotics Research Center (RRC), International Institute of Information Technology (IIIT), Hyderabad, India(印度海得拉巴国际信息技术研究所机器人研究中心);Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系)
Sequential Multi-Object Grasping with One Dexterous Hand
Thomas Lord Department of Computer Science, University of Southern California, USA(美国南加州大学计算机科学系);Fudan University, China(复旦大学);INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria(保加利亚索菲亚大学“圣克莱孟·奥赫里茨基”学院)
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·扎耶德人工智能大学);Technical University of Munich(慕尼黑技术大学);National University of Singapore(新加坡国立大学);Westlake University(西湖大学)
ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
CFCS, School of Computer Science, Peking University(计算机学院,北京大学);PKU-AgiBot Lab(北京大学阿吉机器人实验室);PKU-PsiBot Lab(北京大学Psi机器人实验室);Princeton University(普林斯顿大学)
Dexterous Control of an 11-DOF Redundant Robot for CT-Guided Needle Insertion With Task-Oriented Weighted Policies
Jacobs School of Engineering, University of California San Diego(加州大学圣地亚哥分校 Jacobs 工程学院);School of Medicine, University of California San Diego(加州大学圣地亚哥分校医学院);School of Medicine, University of Missouri-Kansas City(密苏里大学堪萨斯城分校医学院)