Slip detection is essential for robots to make robust grasping and fine manipulation. In this paper, a novel dynamic vision-based finger system for slip detection and suppression is proposed. We also present a baseline and feature based approach to detect object slips under illumination and vibration uncertainty. A threshold method is devised to autonomously sample noise in real-time to improve slip detection. Moreover, a fuzzy based suppression strategy using incipient slip feedback is proposed for regulating the grip force. A comprehensive experimental study of our proposed approaches under uncertainty and system for high-performance precision manipulation are presented. We also propose a slip metric to evaluate such performance quantitatively. Results indicate that the system can effectively detect incipient slip events at a sampling rate of 2kHz ($Δt = 500μs$) and suppress them before a gross slip occurs. The event-based approach holds promises to high precision manipulation task requirement in industrial manufacturing and household services.
JingDong Group（京东集团）；Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Robotic grasping plays an important role in the field of robotics. The current state-of-the-art robotic grasping detection systems are usually built on the conventional vision, such as RGB-D camera. Compared to traditional frame-based computer vision, neuromorphic vision is a small and young community of research. Currently, there are limited event-based datasets due to the troublesome annotation of the asynchronous event stream. Annotating large scale vision dataset often takes lots of computation resources, especially the troublesome data for video-level annotation. In this work, we consider…
A Neuromorphic Incipient Slip Detection System using Papillae Morphology
School of Engineering Mathematics and Technology, University of Bristol（布里斯托大学工程数学与技术学院）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；School of Electrical and Electronic Engineering, University College Dublin（都柏林大学学院电子与电气工程学院）
Computer Science and Artificial Intelligence Laboratory (CSAIL), MIT（麻省理工学院计算机科学与人工智能实验室（CSAIL））
Slip detection plays a vital role in robotic manipulation and it has long been a challenging problem in the robotic community. In this paper, we propose a new method based on deep neural network (DNN) to detect slip. The training data is acquired by a GelSight tactile sensor and a camera mounted on a gripper when we use a robot arm to grasp and lift 94 daily objects with different grasping forces and grasping positions. The DNN is trained to classify whether a slip occurred or not. To evaluate the performance of the DNN, we test 10 unseen objects in 152 grasps. A detection accuracy as high as …
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）
Humans can accurately determine whether the object in hand has slipped or not by visual and tactile perception. However, it is still a challenge for robots to detect in-hand object slip through visuo-tactile fusion. To address this issue, a novel visuo-tactile fusion deep neural network is proposed to detect slip, which is a time-dependent continuous action. By using the multi-scale temporal convolution network (MS-TCN) to extract the temporal features of visual and tactile data, the slip can be detected effectively. In this paper, a 7-dregree-of-freedom (7-DoF) robot manipulator equipped with…
Perception and Robotics Group, University of Maryland, College Park（马里兰大学帕克分校感知与机器人小组）；University of Maryland, College Park（马里兰大学帕克分校）；University of Maryland（马里兰大学）
Grasping and manipulating a wide variety of objects is a fundamental skill that would determine the success and wide spread adaptation of robots in homes. Several end-effector designs for robust manipulation have been proposed but they mostly work when provided with prior information about the objects or equipped with external sensors for estimating object shape or size. Such approaches are limited to many-shot or unknown objects and are prone to estimation errors from external estimation systems. We propose an approach to grasp and manipulate previously unseen or zero-shot objects: the object…
Khalifa University（哈利法大学）；Khalifa University of Science and Technology（哈利法科学技术大学）；Strata Manufacturing PJSC（斯特拉制造有限公司）；Purdue University（普渡大学）；Kingston University（金斯顿大学）
Robotic vision plays a key role for perceiving the environment in grasping applications. However, the conventional framed-based robotic vision, suffering from motion blur and low sampling rate, may not meet the automation needs of evolving industrial requirements. This paper, for the first time, proposes an event-based robotic grasping framework for multiple known and unknown objects in a cluttered scene. Compared with standard frame-based vision, neuromorphic vision has advantages of microsecond-level sampling rate and no motion blur. Building on that, the model-based and model-free approache…
This paper presents a novel control approach to dealing with object slip during robotic manipulative movements. Slip is a major cause of failure in many robotic grasping and manipulation tasks. Existing works increase grip force to avoid/control slip. However, this may not be feasible when (i) the robot cannot increase the gripping force -- the max gripping force is already applied or (ii) increased force damages the grasped object, such as soft fruit. Moreover, the robot fixes the gripping force when it forms a stable grasp on the surface of an object, and changing the gripping force during r…
Agile Robots AG（敏捷机器人公司）；Technische Universität München（慕尼黑工业大学）；Universität Hamburg（汉堡大学）
An unstable grasp pose can lead to slip, thus an unstable grasp pose can be predicted by slip detection. A regrasp is required afterwards to correct the grasp pose in order to finish the task. In this work, we propose a novel regrasp planner with multi-sensor modules to plan grasp adjustments with the feedback from a slip detector. Then a regrasp planner is trained to estimate the location of center of mass, which helps robots find an optimal grasp pose. The dataset in this work consists of 1 025 slip experiments and 1 347 regrasps collected by one pair of tactile sensors, an RGB-D camera and …
Learning Stable Robot Grasping with Transformer-based Tactile Control Policies
Italian Institute of Technology（意大利技术研究院）；University of Genoa（热那亚大学）；National University of Singapore（新加坡国立大学）；Institute for Infocomm Research (IR), A*STAR（新加坡科技研究局信息通信研究所）；King’s College London（伦敦国王学院）
Grasping objects whose physical properties are unknown is still a great challenge in robotics. Most solutions rely entirely on visual data to plan the best grasping strategy. However, to match human abilities and be able to reliably pick and hold unknown objects, the integration of an artificial sense of touch in robotic systems is pivotal. This paper describes a novel model-based slip detection pipeline that can predict possibly failing grasps in real-time and signal a necessary increase in grip force. As such, the slip detector does not rely on manually collected data, but exploits physics t…
University of Bristol（布里斯托大学）；University of the West of England（西英格兰大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Tactile sensing is used by humans when grasping to prevent us dropping objects. One key facet of tactile sensing is slip detection, which allows a gripper to know when a grasp is failing and take action to prevent an object being dropped. This study demonstrates the slip detection capabilities of the recently developed Tactile Model O (T-MO) by using support vector machines to detect slip and test multiple slip scenarios including responding to the onset of slip in real time with eleven different objects in various grasps. We demonstrate the benefits of slip detection in grasping by testing tw…
Dexterous manipulation has broad applications in assembly lines, warehouses and agriculture. To perform large-scale manipulation tasks for various objects, a multi-fingered robotic hand sometimes has to sequentially adjust its grasping gestures, i.e. the finger gaits, to address the workspace limits and guarantee the object stability. However, realizing finger gaits planning in dexterous manipulation is challenging due to the complicated grasp quality metrics, uncertainties on object shapes and dynamics (mass and moment of inertia), and unexpected slippage under uncertain contact dynamics. In …
HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ImprintX Robotics（ImprintX机器人公司）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Carnegie Mellon University（卡内基梅隆大学）；Southern University of Science and Technology（南方科技大学）
A shared grasp is a grasp formed by contacts between the manipulated object and both the robot hand and the environment. By trading off hand contacts for environmental contacts, a shared grasp requires fewer contacts with the hand, and enables manipulation even when a full grasp is not possible. Previous research has used shared grasps for non-prehensile manipulation such as pivoting and tumbling. This paper treats the problem more generally, with methods to select the best shared grasp and robot actions for a desired object motion. The central issue is to evaluate the feasible contact modes: …
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（坎帕尼亚路易吉·凡维泰利大学工程系）；Institute for Artificial Intelligence, Universität Bremen（不来梅大学人工智能研究所）
Manipulation planning and control are relevant building blocks of a robotic system and their tight integration is a key factor to improve robot autonomy and allows robots to perform manipulation tasks of increasing complexity, such as those needed in the in-store logistics domain. Supermarkets contain a large variety of objects to be placed on the shelf layers with specific constraints, doing this with a robot is a challenge and requires a high dexterity. However, an integration of reactive grasping control and motion planning can allow robots to perform such tasks even with grippers with limi…
A Novel Bioinspired Neuromorphic Vision-based Tactile Sensor for Fast Tactile Perception
Khalifa University（哈利法大学）；University of Lincoln（林肯大学）；Khalifa University Center of Autonomous Robotic Systems (KUCARS)（哈利法大学自主机器人系统中心）；Healthcare Engineering and Innovation Group (HEIG)（医疗保健工程与创新集团）；Advanced Research and Innovation Center (ARIC)（先进研究与创新中心）
National Institute of Standards and Technology（美国国家标准与技术研究院）
The existence of tactile afferents sensitive to slip-related mechanical transients in the human hand augments the robustness of grasping through secondary force modulation protocols. Despite this knowledge and the fact that tactile-based slip detection has been researched for decades, robust slip detection is still not an out-of-the-box capability for any commercially available tactile sensor. This research seeks to bridge this gap with a comprehensive study addressing several aspects of slip detection. Key developments include a systematic data collection process yielding millions of sensory …
Robotics Institute（机器人研究所）；The Hong Kong University of Science and Technology（香港科技大学）
In essence, successful grasp boils down to correct responses to multiple contact events between fingertips and objects. In most scenarios, tactile sensing is adequate to distinguish contact events. Due to the nature of high dimensionality of tactile information, classifying spatiotemporal tactile signals using conventional model-based methods is difficult. In this work, we propose to predict and classify tactile signal using deep learning methods, seeking to enhance the adaptability of the robotic grasp system to external event changes that may lead to grasping failure. We develop a deep learn…
Robust Adaptive Safe Robotic Grasping with Tactile Sensing
Maersk Mc-Kinney Moller Institute, University of Southern Denmark（马士基麦金尼默勒研究所，丹麦南方大学）；Departments of Mechanical and Civil Engineering and Computing and Mathematical Sciences , California Institute of Technology（机械与土木工程系和计算与数学科学系，加州理工学院）
Hong Kong University of Science and Technology（香港科技大学）
Tactile sensing is essential to the human perception system, so as to robot. In this paper, we develop a novel optical-based tactile sensor "FingerVision" with effective signal processing algorithms. This sensor is composed of soft skin with embedded marker array bonded to rigid frame, and a web camera with a fisheye lens. While being excited with contact force, the camera tracks the movements of markers and deformation field is obtained. Compared to existing tactile sensors, our sensor features compact footprint, high resolution, and ease of fabrication. Besides, utilizing the deformation fie…
University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Control barrier functions have been demonstrated to be a useful method of ensuring constraint satisfaction for a wide class of controllers, however existing results are mostly restricted to continuous time systems of relative degree one. Mechanical systems, including robots, are typically second-order systems in which the control occurs at the force/torque level. These systems have velocity and position constraints (i.e. relative degree two) that are vital for safety and/or task execution. Additionally, mechanical systems are typically controlled digitally as sampled-data systems. The contribu…
Billions of people use chopsticks, a simple yet versatile tool, for fine manipulation of everyday objects. The small, curved, and slippery tips of chopsticks pose a challenge for picking up small objects, making them a suitably complex test case. This paper leverages human demonstrations to develop an autonomous chopsticks-equipped robotic manipulator. Due to the lack of accurate models for fine manipulation, we explore model-free imitation learning, which traditionally suffers from the covariate shift phenomenon that causes poor generalization. We propose two approaches to reduce covariate sh…
University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
For successful object manipulation with robotic hands, it is important to ensure that the object remains in the grasp at all times. In addition to grasp constraints associated with slipping and singular hand configurations, excessive rolling is an important grasp concern where the contact points roll off of the fingertip surface. Related literature focus only on a subset of grasp constraints, or assume grasp constraint satisfaction without providing guarantees of such a claim. In this paper, we propose a control approach that systematically handles all grasp constraints. The proposed controlle…
Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing
University of California San Diego（加利福尼亚大学圣迭戈分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；UC San Diego Health（加州大学圣迭戈分校健康中心）
