Vision-based grasping systems typically adopt an open-loop execution of a planned grasp. This policy can fail due to many reasons, including ubiquitous calibration error. Recovery from a failed grasp is further complicated by visual occlusion, as the hand is usually occluding the vision sensor as it attempts another open-loop regrasp. This work presents MAT, a tactile closed-loop method capable of realizing grasps provided by a coarse initial positioning of the hand above an object. Our algorithm is a deep reinforcement learning (RL) policy optimized through the clipped surrogate objective within a maximum entropy RL framework to balance exploitation and exploration. The method utilizes tactile and proprioceptive information to act through both fine finger motions and larger regrasp movements to execute stable grasps. A novel curriculum of action motion magnitude makes learning more tractable and helps turn common failure cases into successes. Careful selection of features that exhibit small sim-to-real gaps enables this tactile grasping policy, trained purely in simulation, to transfer well to real world environments without the need for additional learning. Experimentally, this methodology improves over a vision-only grasp success rate substantially on a multi-fingered robot hand. When this methodology is used to realize grasps from coarse initial positions provided by a vision-only planner, the system is made dramatically more robust to calibration errors in the camera-robot transform.
Technical University of Munich（慕尼黑工业大学）；Harvard University（哈佛大学）；RightHand Robotics, Inc.（RightHand机器人公司）
A long-standing question in robot hand design is how accurate tactile sensing must be. This paper uses simulated tactile signals and the reinforcement learning (RL) framework to study the sensing needs in grasping systems. Our first experiment investigates the need for rich tactile sensing in the rewards of RL-based grasp refinement algorithms for multi-fingered robotic hands. We systematically integrate different levels of tactile data into the rewards using analytic grasp stability metrics. We find that combining information on contact positions, normals, and forces in the reward yields the …
Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Bringing tactile sensation to robotic hands will allow for more effective grasping, along with the wide range of benefits of human-like touch. Here we present a 3D-printed, three-fingered tactile robot hand comprising an OpenHand Model O customized to house a TacTip soft biomimetic tactile sensor in the distal phalanx of each finger. We expect that combining the grasping capabilities of this underactuated hand with sophisticated tactile sensing will result in an effective platform for robot hand research -- the Tactile Model O (T-MO). The design uses three JeVois machine vision systems, each c…
Reliably planning fingertip grasps for multi-fingered hands lies as a key challenge for many tasks including tool use, insertion, and dexterous in-hand manipulation. This task becomes even more difficult when the robot lacks an accurate model of the object to be grasped. Tactile sensing offers a promising approach to account for uncertainties in object shape. However, current robotic hands tend to lack full tactile coverage. As such, a problem arises of how to plan and execute grasps for multi-fingered hands such that contact is made with the area covered by the tactile sensors. To address thi…
Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity
University of Southern California（南加州大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of Cincinnati（辛辛那提大学）
University of Bristol（布里斯托大学）；University of the West of England（西英格兰大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Tactile sensing is used by humans when grasping to prevent us dropping objects. One key facet of tactile sensing is slip detection, which allows a gripper to know when a grasp is failing and take action to prevent an object being dropped. This study demonstrates the slip detection capabilities of the recently developed Tactile Model O (T-MO) by using support vector machines to detect slip and test multiple slip scenarios including responding to the onset of slip in real time with eleven different objects in various grasps. We demonstrate the benefits of slip detection in grasping by testing tw…
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）；University of Utah（犹他大学）
Using simulation to train robot manipulation policies holds the promise of an almost unlimited amount of training data, generated safely out of harm's way. One of the key challenges of using simulation, to date, has been to bridge the reality gap, so that policies trained in simulation can be deployed in the real world. We explore the reality gap in the context of learning a contextual policy for multi-fingered robotic grasping. We propose a Grasping Objects Approach for Tactile (GOAT) robotic hands, learning to overcome the reality gap problem. In our approach we use human hand motion demonst…
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg（乌尔姆技术大学计算机科学与人工智能系）；Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学）；TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）
Tactile sensors provide useful contact data during the interaction with an object which can be used to accurately learn to determine the stability of a grasp. Most of the works in the literature represented tactile readings as plain feature vectors or matrix-like tactile images, using them to train machine learning models. In this work, we explore an alternative way of exploiting tactile information to predict grasp stability by leveraging graph-like representations of tactile data, which preserve the actual spatial arrangement of the sensor's taxels and their locality. In experimentation, we …
University of Alicante（阿利坎特大学）；Computer Science Research Institute, University of Alicante（阿利坎特大学计算机科学研究所）
Tactile sensors supply useful information during the interaction with an object that can be used for assessing the stability of a grasp. Most of the previous works on this topic processed tactile readings as signals by calculating hand-picked features. Some of them have processed these readings as images calculating characteristics on matrix-like sensors. In this work, we explore how non-matrix sensors (sensors with taxels not arranged exactly in a matrix) can be processed as tactile images as well. In addition, we prove that they can be used for predicting grasp stability by training a Convol…
TACTFUL: Tactile-Driven Exploration For Object Localization and Identification in Confined Environments
Amazon Fulfillment Technologies & Robotics（亚马逊履约技术与机器人）；Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learning-based approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of…
Most current works in Sim2Real learning for robotic manipulation tasks leverage camera vision that may be significantly occluded by robot hands during the manipulation. Tactile sensing offers complementary information to vision and can compensate for the information loss caused by the occlusion. However, the use of tactile sensing is restricted in the Sim2Real research due to no simulated tactile sensors being available. To mitigate the gap, we introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator. Similar to the real GelSight sensor, the sim…
High resolution tactile sensors are often bulky and have shape profiles that make them awkward for use in manipulation. This becomes important when using such sensors as fingertips for dexterous multi-fingered hands, where boxy or planar fingertips limit the available set of smooth manipulation strategies. High resolution optical based sensors such as GelSight have until now been constrained to relatively flat geometries due to constraints on illumination geometry.Here, we show how to construct a rounded fingertip that utilizes a form of light piping for directional illumination. Our sensors c…
Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）
In this paper we focus on analyzing the thermal modality of tactile sensing for material recognition using a large materials database. Many factors affect thermal recognition performance, including sensor noise, the initial temperatures of the sensor and the object, the thermal effusivities of the materials, and the duration of contact. To analyze the influence of these factors on thermal recognition, we used a semi-infinite solid based thermal model to simulate heat-transfer data from all the materials in the CES Edupack Level-1 database. We used support-vector machines (SVMs) to predict F1 s…
Institute for Dynamic Systems and Control, ETH Zurich（苏黎世联邦理工学院动态系统与控制研究所）
Data-driven approaches to tactile sensing aim to overcome the complexity of accurately modeling contact with soft materials. However, their widespread adoption is impaired by concerns about data efficiency and the capability to generalize when applied to various tasks. This paper focuses on both these aspects with regard to a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. Accurate models for the soft materials and the camera projection, derived via state-of-the-art techniques in the respective domains…
Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions
School of Computer Science, University of Bristol, 2 Bristol Robotics Laboratory 3 School of Engineering Mathematics and Technology, University of Bristol（1 计算机科学学院，布里斯托尔大学 2 布里斯托尔机器人实验室 3 工程数学与技术学院，布里斯托尔大学）
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics. One of the contributing factors that limit current robotic manipulation systems is the difficulty of precisely sensing contact forces -- sensing and reasoning about contact forces are crucial to accurately control interactions with the environment. As a step towards enabling better robotic manipulation, we introduce DIGIT, an inexpensive, compact, and high-resolution tactile sensor geared towards in-hand manipulation. DIGIT improves upon past vision-based tactile sensors by min…
University of Utah（犹他大学）；NVIDIA Corporation（英伟达公司）
While tactile skins have been shown to be useful for detecting collisions between a robotic arm and its environment, they have not been extensively used for improving robotic grasping and in-hand manipulation. We propose a novel sensor design for use in covering existing multi-fingered robot hands. We analyze the performance of four different piezoresistive materials using both fabric and anti-static foam substrates in benchtop experiments. We find that although the piezoresistive foam was designed as packing material and not for use as a sensing substrate, it performs comparably with fabrics …
Contact Status Recognition and Slip Detection with a Bio-inspired Tactile Hand
School of Mechanical Engineering, Southeast University Nanjing 211189, China；School of Electronic Science \& Engineering, Southeast University Nanjing 211189, China
Tactile SoftHand-A: 3D-Printed, Tactile, Highly-underactuated, Anthropomorphic Robot Hand with an Antagonistic Tendon Mechanism
School of Engineering Mathematics and Technology（工程数学与技术学院）；Bristol Robotics Laboratory（布里斯托尔机器人实验室）；University of Bristol（布里斯托尔大学）；School of Robotics（机器人学院）；Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Department of Information Engineering（信息工程系）；Research Center "E.Piaggio"（E.Piaggio研究中心）；University of Pisa（比萨大学）；Soft Robotics for Human Cooperation and Rehabilitation（人机合作与康复软机器人）；Istituto Italiano di Tecnologia（意大利技术研究院）
Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecture
DLR Institute of Robotics and Mechatronics（DLR机器人与 mechatronics研究所）；Technical University of Munich（慕尼黑工业大学）；Deggendorf Institute of Technology（代根多夫应用技术大学）
University of California, Merced（加州大学默塞德分校）；National Institute of Standards and Technology (NIST)（美国国家标准与技术研究院）
This technical report presents an introduction to different aspects of multi-fingered robot grasping. After having introduced relevant mathematical background for modeling, form and force closure are discussed. Next, we present an overview of various grasp planning algorithms with the objective of illustrating different approaches to solve this problem. Finally, we discuss grasp performance benchmarking.
ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, Universität Hamburg（乌尔姆大学信息学院技术多模态系统部门）；Agile Robots AG（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）
