Dexterous grasping of a novel object given a single view is an open problem. This paper makes several contributions to its solution. First, we present a simulator for generating and testing dexterous grasps. Second we present a data set, generated by this simulator, of 2.4 million simulated dexterous grasps of variations of 294 base objects drawn from 20 categories. Third, we present a basic architecture for generation and evaluation of dexterous grasps that may be trained in a supervised manner. Fourth, we present three different evaluative architectures, employing ResNet-50 or VGG16 as their visual backbone. Fifth, we train, and evaluate seventeen variants of generative-evaluative architectures on this simulated data set, showing improvement from 69.53% grasp success rate to 90.49%. Finally, we present a real robot implementation and evaluate the four most promising variants, executing 196 real robot grasps in total. We show that our best architectural variant achieves a grasp success rate of 87.8% on real novel objects seen from a single view, improving on a baseline of 57.1%.
FunGrasp: Functional Grasping for Diverse Dexterous Hands
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；ETH Zürich（苏黎世联邦理工学院）；The Hong Kong University of Science and Technology（香港科技大学）
Peking University（北京大学）；Beijing Institute for General AI（北京通用人工智能研究院）；Stanford University（斯坦福大学）；Tsinghua University（清华大学）
In this work, we tackle the problem of learning universal robotic dexterous grasping from a point cloud observation under a table-top setting. The goal is to grasp and lift up objects in high-quality and diverse ways and generalize across hundreds of categories and even the unseen. Inspired by successful pipelines used in parallel gripper grasping, we split the task into two stages: 1) grasp proposal (pose) generation and 2) goal-conditioned grasp execution. For the first stage, we propose a novel probabilistic model of grasp pose conditioned on the point cloud observation that factorizes rota…
Beijing Institute of General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；Tsinghua University（清华大学）；Peking University（北京大学）
Generating dexterous grasping has been a long-standing and challenging robotic task. Despite recent progress, existing methods primarily suffer from two issues. First, most prior arts focus on a specific type of robot hand, lacking the generalizable capability of handling unseen ones. Second, prior arts oftentimes fail to rapidly generate diverse grasps with a high success rate. To jointly tackle these challenges with a unified solution, we propose GenDexGrasp, a novel hand-agnostic grasping algorithm for generalizable grasping. GenDexGrasp is trained on our proposed large-scale multi-hand gra…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）
Dexterous robotic hands are appealing for their agility and human-like morphology, yet their high degree of freedom makes learning to manipulate challenging. We introduce an approach for learning dexterous grasping. Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people. Unlike traditional approaches that learn from human demonstration trajectories (e.g., hand joint sequences captured with a glove), the proposed prior is object-centric and image-based, allowing …
The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Facebook AI Research（脸书人工智能研究院）
Dexterous multi-fingered robotic hands have a formidable action space, yet their morphological similarity to the human hand holds immense potential to accelerate robot learning. We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interactions present in in-the-wild YouTube videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose when learning to grasp with deep reinforcement learning. A key advantage of our method is that the learned policy is able to leverage free-form in-the-wild visual d…
How should a robot direct active vision so as to ensure reliable grasping? We answer this question for the case of dexterous grasping of unfamiliar objects. By dexterous grasping we simply mean grasping by any hand with more than two fingers, such that the robot has some choice about where to place each finger. Such grasps typically fail in one of two ways, either unmodeled objects in the scene cause collisions or object reconstruction is insufficient to ensure that the grasp points provide a stable force closure. These problems can be solved more easily if active sensing is guided by the anti…
Multi-Keypoint Affordance Representation for Functional Dexterous Grasping
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；College of Computer Science and Electronic Engineering, Hunan University（计算机科学与电子工程学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
Reinforcement learning is a promising method for robotic grasping as it can learn effective reaching and grasping policies in difficult scenarios. However, achieving human-like manipulation capabilities with sophisticated robotic hands is challenging because of the problem's high dimensionality. Although remedies such as reward shaping or expert demonstrations can be employed to overcome this issue, they often lead to oversimplified and biased policies. We present Dext-Gen, a reinforcement learning framework for Dexterous Grasping in sparse reward ENvironments that is applicable to a variety o…
GRASPLAT: Enabling dexterous grasping through novel view synthesis
TeV, Fondazione Bruno Kessler（TeV，布鲁诺·凯瑟尔基金会）；PAVIS, Fondazione Istituto Italiano di Tecnologia（PAVIS，意大利技术研究院基金会）；ISR, Instituto Superior Técnico, Universidade de Lisboa（ISR，里斯本大学理工学院）；University of Trento（特伦托大学）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；ByteDance Inc.（字节跳动公司）；Pixocial Technology（Pixocial科技公司）；Dolby Laboratories（杜比实验室）；Wormpex AI Research（Wormpex人工智能研究院）
CADGrasp: Learning Contact and Collision Aware General Dexterous Grasping in Cluttered Scenes
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机学院，北京大学）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家级重点实验室，计算机学院，北京大学）
DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（人工智能学院，香港中文大学（深圳））；School of Computing, University of Portsmouth（计算学院，朴茨茅斯大学）
DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Key Lab of CS&AUS of Zhejiang Province（浙江省计算机与应用系统安全重点实验室）；Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）
Dexterous robotic hands have the capability to interact with a wide variety of household objects to perform tasks like grasping. However, learning robust real world grasping policies for arbitrary objects has proven challenging due to the difficulty of generating high quality training data. In this work, we propose a learning system (ISAGrasp) for leveraging a small number of human demonstrations to bootstrap the generation of a much larger dataset containing successful grasps on a variety of novel objects. Our key insight is to use a correspondence-aware implicit generative model to deform ob…
East China Normal University（华东师范大学）；Midea Group（美的集团）；Guangdong Polytechnic Normal University（广东技术师范大学）
In this paper, we propose a novel representation for grasping using contacts between multi-finger robotic hands and objects to be manipulated. This representation significantly reduces the prediction dimensions and accelerates the learning process. We present an effective end-to-end network, CMG-Net, for grasping unknown objects in a cluttered environment by efficiently predicting multi-finger grasp poses and hand configurations from a single-shot point cloud. Moreover, we create a synthetic grasp dataset that consists of five thousand cluttered scenes, 80 object categories, and 20 million ann…
University of Birmingham（伯明翰大学）；Poznan University of Technology（波兹南工业大学）
This paper concerns the problem of how to learn to grasp dexterously, so as to be able to then grasp novel objects seen only from a single view-point. Recently, progress has been made in data-efficient learning of generative grasp models which transfer well to novel objects. These generative grasp models are learned from demonstration (LfD). One weakness is that, as this paper shall show, grasp transfer under challenging single view conditions is unreliable. Second, the number of generative model elements rises linearly in the number of training examples. This, in turn, limits the potential of…
PartDexTOG: Generating Dexterous Task-Oriented Grasping via Language-driven Part Analysis
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）
Spirit AeroSystems（势必锐航空系统公司）；Wichita State University（威奇托州立大学）
Humans can determine a proper strategy to grasp an object according to the measured physical attributes or the prior knowledge of the object. This paper proposes an approach to determining the strategy of dexterous grasping by using an anthropomorphic robotic hand simply based on a label or a description of an object. Object attributes are parsed from natural-language descriptions and augmented with an object knowledge base that is scraped from retailer websites. A novel metric named joint probability distance is defined to measure distance between object attributes. The probability distributi…
Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）
Robotic dexterous grasping is the first step to enable human-like dexterous object manipulation and thus a crucial robotic technology. However, dexterous grasping is much more under-explored than object grasping with parallel grippers, partially due to the lack of a large-scale dataset. In this work, we present a large-scale robotic dexterous grasp dataset, DexGraspNet, generated by our proposed highly efficient synthesis method that can be generally applied to any dexterous hand. Our method leverages a deeply accelerated differentiable force closure estimator and thus can efficiently and robu…
ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
CFCS, School of Computer Science, Peking University（计算机学院，北京大学）；PKU-AgiBot Lab（北京大学阿吉机器人实验室）；PKU-PsiBot Lab（北京大学Psi机器人实验室）；Princeton University（普林斯顿大学）
T(R,O) Grasp: Efficient Graph Diffusion of Robot-Object Spatial Transformation for Cross-Embodiment Dexterous Grasping
Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）；RoboScience；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）
