Various robotic tool manipulation methods have been developed so far. However, to our knowledge, none of them have taken into account the fact that the grasping state such as grasping position and tool angle can change at any time during the tool manipulation. In addition, there are few studies that can handle deformable tools. In this study, we develop a method for estimating the position of a tool-tip, controlling the tool-tip, and handling online adaptation to changes in the relationship between the body and the tool, using a neural network including parametric bias. We demonstrate the effectiveness of our method for online change in grasping state and for deformable tools, in experiments using two different types of robots: axis-driven robot PR2 and tendon-driven robot MusashiLarm.
This study proposed a novel robotic gripper that can achieve grasping and infinite wrist twisting motions using a single actuator. The gripper is equipped with a differential gear mechanism that allows switching between the grasping and twisting motions according to the magnitude of the tip force applied to the finger. The grasping motion is activated when the tip force is below a set value, and the wrist twisting motion is activated when the tip force exceeds this value. "Twist grasping," a special grasping mode that allows the wrapping of a flexible thin object around the fingers of the grip…
Robotic Grasping has always been an active topic in robotics since grasping is one of the fundamental but most challenging skills of robots. It demands the coordination of robotic perception, planning, and control for robustness and intelligence. However, current solutions are still far behind humans, especially when confronting unstructured scenarios. In this paper, we survey the advances of robotic grasping, starting from the classical formulations and solutions to the modern ones. By reviewing the history of robotic grasping, we want to provide a complete view of this community, and perhaps…
Deep reinforcement learning (DRL) has been proven to be a powerful paradigm for learning complex control policy autonomously. Numerous recent applications of DRL in robotic grasping have successfully trained DRL robotic agents end-to-end, mapping visual inputs into control instructions directly, but the amount of training data required may hinder these applications in practice. In this paper, we propose a DRL based robotic visual grasping framework, in which visual perception and control policy are trained separately rather than end-to-end. The visual perception produces physical descriptions …
Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1)it is a comprehensive task to know what and how to grasp; (2)it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multi-task convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the ta…
In the robotic crop harvesting environment, foreign objects intrusion in the gripper workspace is frequently occurring and unignorable, however, rarely addressed. This paper presents a novel intelligent robotic grasping method capable of handling obstacle interference, which is the first of its kind in the literature. The proposed method combines deep learning algorithms with low-cost tactile sensing hardware on a multi-DoF soft robotic gripper. Through experimental validations, the proposed method demonstrated promising performance in distinguishing various grasping scenarios. The 4-finger in…
University of California, Merced（加州大学默塞德分校）；National Institute of Standards and Technology (NIST)（美国国家标准与技术研究院）
This technical report presents an introduction to different aspects of multi-fingered robot grasping. After having introduced relevant mathematical background for modeling, form and force closure are discussed. Next, we present an overview of various grasp planning algorithms with the objective of illustrating different approaches to solve this problem. Finally, we discuss grasp performance benchmarking.
Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Queensland University of Technology（昆士兰科技大学）；LYRO Robotics（LYRO机器人公司）
We present the Evolved Grasping Analysis Dataset (EGAD), comprising over 2000 generated objects aimed at training and evaluating robotic visual grasp detection algorithms. The objects in EGAD are geometrically diverse, filling a space ranging from simple to complex shapes and from easy to difficult to grasp, compared to other datasets for robotic grasping, which may be limited in size or contain only a small number of object classes. Additionally, we specify a set of 49 diverse 3D-printable evaluation objects to encourage reproducible testing of robotic grasping systems across a range of compl…
Safe yet stable grasping requires a robotic hand to apply sufficient force on the object to immobilize it while keeping it from getting damaged. Soft robotic hands have been proposed for safe grasping due to their passive compliance, but even such a hand can crush objects if the applied force is too high. Thus for safe grasping, regulating the grasping force is of uttermost importance even with soft hands. In this work, we present a force controlled soft hand and use it to achieve safe grasping. To this end, resistive force and bend sensors are integrated in a soft hand, and a data-driven cali…
Autonomous grasping is an important factor for robots physically interacting with the environment and executing versatile tasks. However, a universally applicable, cost-effective, and rapidly deployable autonomous grasping approach is still limited by those target objects with fuzzy-depth information. Examples are transparent, specular, flat, and small objects whose depth is difficult to be accurately sensed. In this work, we present a solution to those fuzzy-depth objects. The framework of our approach includes two major components: one is a soft robotic hand and the other one is a Fuzzy-dept…
We consider the problem of robotic grasping using depth + RGB information sampling from a real sensor. we design an encoder-decoder neural network to predict grasp policy in real time. This method can fuse the advantage of depth image and RGB image at the same time and is robust for grasp and observation height.We evaluate our method in a physical robotic system and propose an open-loop algorithm to realize robotic grasp operation. We analyze the result of experiment from multi-perspective and the result shows that our method is competitive with the state-of-the-art in grasp performance, real-…
A Hyperspectral Imaging Guided Robotic Grasping System
Department of Mechanical and Automation Engineering, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong SAR（机械与自动化工程系，T-Stone机器人研究所，香港中文大学，香港特别行政区）；School of Instrumentation and Optoelectronic Engineering, Beihang University（仪器与光电工程学院，北航大学）
Wuhan University（武汉大学）；University of Hamburg（汉堡大学）；Wuhan Institute of Technology（武汉工程大学）；The Chinese University of HongKong（香港中文大学）
Data-driven approaches have become a dominant paradigm for robotic grasp planning. However, the performance of these approaches is enormously influenced by the quality of the available training data. In this paper, we propose a framework to generate object shapes to improve the grasping dataset quality, thus enhancing the grasping ability of a pre-designed learning-based grasp planning network. In this framework, the object shapes are embedded into a low-dimensional feature space using an AutoEncoder (encoder-decoder) based structure network. The rarity and graspness scores are defined for eac…
In this work, we present a deep reinforcement learning based method to solve the problem of robotic grasping using visio-motor feedback. The use of a deep learning based approach reduces the complexity caused by the use of hand-designed features. Our method uses an off-policy reinforcement learning framework to learn the grasping policy. We use the double deep Q-learning framework along with a novel Grasp-Q-Network to output grasp probabilities used to learn grasps that maximize the pick success. We propose a visual servoing mechanism that uses a multi-view camera setup that observes the scene…
Universität Hamburg（汉堡大学）；Agile Robots AG（敏捷机器人公司）
Precise robotic grasping of several novel objects is a huge challenge in manufacturing, automation, and logistics. Most of the current methods for model-free grasping are disadvantaged by the sparse data in grasping datasets and by errors in sensor data and contact models. This study combines data generation and sim-to-real transfer learning in a grasping framework that reduces the sim-to-real gap and enables precise and reliable model-free grasping. A large-scale robotic grasping dataset with dense grasp labels is generated using domain randomization methods and a novel data augmentation meth…
DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Key Lab of CS&AUS of Zhejiang Province（浙江省计算机与应用系统安全重点实验室）；Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）
We present a novel approach to robotic grasp planning using both a learned grasp proposal network and a learned 3D shape reconstruction network. Our system generates 6-DOF grasps from a single RGB-D image of the target object, which is provided as input to both networks. By using the geometric reconstruction to refine the the candidate grasp produced by the grasp proposal network, our system is able to accurately grasp both known and unknown objects, even when the grasp location on the object is not visible in the input image. This paper presents the network architectures, training procedures,…
Transferring multiple objects between bins is a common task for many applications. In robotics, a standard approach is to pick up one object and transfer it at a time. However, grasping and picking up multiple objects and transferring them together at once is more efficient. This paper presents a set of novel strategies for efficiently grasping multiple objects in a bin to transfer them to another. The strategies enable a robotic hand to identify an optimal ready hand configuration (pre-grasp) and calculate a flexion synergy based on the desired quantity of objects to be grasped. This paper al…
Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of t…
Queensland University of Technology（昆士兰科技大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
We present the grasping system and design approach behind Cartman, the winning entrant in the 2017 Amazon Robotics Challenge. We investigate the design processes leading up to the final iteration of the system and describe the emergent solution by comparing it with key robotics design aspects. Following our experience, we propose a new design aspect, precision vs. redundancy, that should be considered alongside the previously proposed design aspects of modularity vs. integration, generality vs. assumptions, computation vs. embodiment and planning vs. feedback. We present the grasping system be…
Soft robotic hands and grippers are increasingly attracting attention as a robotic end-effector. Compared with rigid counterparts, they are safer for human-robot and environment-robot interactions, easier to control, lower cost and weight, and more compliant. Current soft robotic hands have mostly focused on the soft fingers and bending actuators. However, the palm is also essential part for grasping. In this work, we propose a novel design of soft humanoid hand with pneumatic soft fingers and soft palm. The hand is inexpensive to fabricate. The configuration of the soft palm is based on modul…
Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Queensland University of Technology（昆士兰科技大学）
This paper presents a real-time, object-independent grasp synthesis method which can be used for closed-loop grasping. Our proposed Generative Grasping Convolutional Neural Network (GG-CNN) predicts the quality and pose of grasps at every pixel. This one-to-one mapping from a depth image overcomes limitations of current deep-learning grasping techniques by avoiding discrete sampling of grasp candidates and long computation times. Additionally, our GG-CNN is orders of magnitude smaller while detecting stable grasps with equivalent performance to current state-of-the-art techniques. The light-we…
This paper presents a novel approach to numerically describe the interactions between rigid bodies, with a special focus on robotic grasping. Some of the more common approaches used to address such issues rely on satisfaction of a set of strict constraints, descriptive of the expected physical reality of such interactions in practice. However, application of constraint-based methods in a numerical setting may lead to problematic configurations in which, for instance, volumes occupied by distinct bodies may overlap. Such situations lying beyond the range of admissible configurations for constra…
University of Surrey（萨里大学）；University of Birmingham（伯明翰大学）
Inertial parameters characterise an object's motion under applied forces, and can provide strong priors for planning and control of robotic actions to manipulate the object. However, these parameters are not available a-priori in situations where a robot encounters new objects. In this paper, we describe and categorise the ways that a robot can identify an object's inertial parameters. We also discuss grasping and manipulation methods in which knowledge of inertial parameters is exploited in various ways. We begin with a discussion of literature which investigates how humans estimate the inert…
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；KUKA Deutschland GmbH（库卡德国有限公司）
Given the task of learning robotic grasping solely based on a depth camera input and gripper force feedback, we derive a learning algorithm from an applied point of view to significantly reduce the amount of required training data. Major improvements in time and data efficiency are achieved by: Firstly, we exploit the geometric consistency between the undistorted depth images and the task space. Using a relative small, fully-convolutional neural network, we predict grasp and gripper parameters with great advantages in training as well as inference performance. Secondly, motivated by the small …
We describe the grasping and manipulation strategy that we employed at the autonomous track of the Robotic Grasping and Manipulation Competition at IROS 2016. A salient feature of our architecture is the tight coupling between visual (Asus Xtion) and tactile perception (Robotic Materials), to reduce the uncertainty in sensing and actuation. We demonstrate the importance of tactile sensing and reactive control during the final stages of grasping using a Kinova Robotic arm. The set of tools and algorithms for object grasping presented here have been integrated into the open-source Robot Operatin…
University of South Florida（南佛罗里达大学）；National Institute of Standards and Technology (NIST)（美国国家标准与技术研究院）；German Aerospace Center (DLR)（德国航空航天中心）；Worcester Polytechnic Institute (WPI)（伍斯特理工学院）
This paper discusses recent research progress in robotic grasping and manipulation in the light of the latest Robotic Grasping and Manipulation Competitions (RGMCs). We first provide an overview of past benchmarks and competitions related to the robotics manipulation field. Then, we discuss the methodology behind designing the manipulation tasks in RGMCs. We provide a detailed analysis of key challenges for each task and identify the most difficult aspects based on the competing teams' performance in recent years. We believe that such an analysis is insightful to determine the future research …
This article presents a method for grasping novel objects by learning from experience. Successful attempts are remembered and then used to guide future grasps such that more reliable grasping is achieved over time. To generalise the learned experience to unseen objects, we introduce the dense geometric correspondence matching network (DGCM-Net). This applies metric learning to encode objects with similar geometry nearby in feature space. Retrieving relevant experience for an unseen object is thus a nearest neighbour search with the encoded feature maps. DGCM-Net also reconstructs 3D-3D corresp…
University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Control barrier functions have been demonstrated to be a useful method of ensuring constraint satisfaction for a wide class of controllers, however existing results are mostly restricted to continuous time systems of relative degree one. Mechanical systems, including robots, are typically second-order systems in which the control occurs at the force/torque level. These systems have velocity and position constraints (i.e. relative degree two) that are vital for safety and/or task execution. Additionally, mechanical systems are typically controlled digitally as sampled-data systems. The contribu…
This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape co…
University of Edinburgh（爱丁堡大学）；University of Groningen（格罗宁根大学）；University College London（伦敦大学学院）
Generating high-quality instance-wise grasp configurations provides critical information of how to grasp specific objects in a multi-object environment and is of high importance for robot manipulation tasks. This work proposed a novel \textbf{S}ingle-\textbf{S}tage \textbf{G}rasp (SSG) synthesis network, which performs high-quality instance-wise grasp synthesis in a single stage: instance mask and grasp configurations are generated for each object simultaneously. Our method outperforms state-of-the-art on robotic grasp prediction based on the OCID-Grasp dataset, and performs competitively on t…
Robotic grasping is one of the most fundamental robotic manipulation tasks and has been actively studied. However, how to quickly teach a robot to grasp a novel target object in clutter remains challenging. This paper attempts to tackle the challenge by leveraging object attributes that facilitate recognition, grasping, and quick adaptation. In this work, we introduce an end-to-end learning method of attribute-based robotic grasping with one-grasp adaptation capability. Our approach fuses the embeddings of a workspace image and a query text using a gated-attention mechanism and learns to predi…
We present a self-contained, soft robotic hand composed of soft pneumatic actuator modules that are equipped with strain and pressure sensing. We show how this data can be used to discern whether a grasp was successful. Co-locating sensing and embedded computation with the actuators greatly simplifies control and system integration. Equipped with a small pump, the hand is self-contained and needs only power and data supplied by a single USB connection to a PC. We demonstrate its function by grasping a variety of objects ranging from very small to large and heavy objects weighing more than the …
Hong Kong University of Science and Technology（香港科技大学）
Adaptation to various scene configurations and object properties, stability and dexterity in robotic grasping manipulation is far from explored. This work presents an origami-based shape morphing fingertip design to actively tackle the grasping stability and dexterity problems. The proposed fingertip utilizes origami as its skeleton providing degrees of freedom at desired positions and motor-driven four-bar-linkages as its transmission components to achieve a compact size of the fingertip. 3 morphing types that are commonly observed and essential in robotic grasping are studied and validated w…
