In this work, we report on the integrated sensorimotor control of the Pisa/IIT SoftHand, an anthropomorphic soft robot hand designed around the principle of adaptive synergies, with the BRL tactile fingertip (TacTip), a soft biomimetic optical tactile sensor based on the human sense of touch. Our focus is how a sense of touch can be used to control an anthropomorphic hand with one degree of actuation, based on an integration that respects the hand's mechanical functionality. We consider: (i) closed-loop tactile control to establish a light contact on an unknown held object, based on the structural similarity with an undeformed tactile image; and (ii) controlling the estimated pose of an edge feature of a held object, using a convolutional neural network approach developed for controlling other sensors in the TacTip family. Overall, this gives a foundation to endow soft robotic hands with human-like touch, with implications for autonomous grasping, manipulation, human-robot interaction and prosthetics. Supplemental video: https://youtu.be/ndsxj659bkQ
Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）
The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, able-bodied individuals inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we built two fabric-based tactile sensors that measure contact location along the palmar and…
Tactile SoftHand-A: 3D-Printed, Tactile, Highly-underactuated, Anthropomorphic Robot Hand with an Antagonistic Tendon Mechanism
School of Engineering Mathematics and Technology（工程数学与技术学院）；Bristol Robotics Laboratory（布里斯托尔机器人实验室）；University of Bristol（布里斯托尔大学）；School of Robotics（机器人学院）；Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Department of Information Engineering（信息工程系）；Research Center "E.Piaggio"（E.Piaggio研究中心）；University of Pisa（比萨大学）；Soft Robotics for Human Cooperation and Rehabilitation（人机合作与康复软机器人）；Istituto Italiano di Tecnologia（意大利技术研究院）
University of Bristol（布里斯托大学）；University of Pisa（比萨大学）；Istituto Italiano di Tecnologia（意大利技术研究院）
This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing asynchronous sensor data acquisition and processing, resulting in a fast control loop sufficient for real-time grasp control. We then develop a novel grasp controller that uses tactile feedback from all…
3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception
State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（数字制造装备与技术国家重点实验室，机械科学与工程学院，华中科技大学）
University of Maryland（马里兰大学）；Institute for Advanced Computer Studies（高级计算机研究所）
Grasping objects requires tight integration between visual and tactile feedback. However, there is an inherent difference in the scale at which both these input modalities operate. It is thus necessary to be able to analyze tactile feedback in isolation in order to gain information about the surface the end-effector is operating on, such that more fine-grained features may be extracted from the surroundings. For tactile perception of the robot, inspired by the concept of the tactile flow in humans, we present the computational tactile flow to improve the analysis of the tactile feedback in rob…
Soft robots are typically approximated as low-dimensional systems, especially when learning-based methods are used. This leads to models that are limited in their capability to predict the large number of deformation modes and interactions that a soft robot can have. In this work, we present a deep-learning methodology to learn high-dimensional visual models of a soft robot combining multimodal sensorimotor information. The models are learned in an end-to-end fashion, thereby requiring no intermediate sensor processing or grounding of data. The capabilities and advantages of such a modelling a…
The majority of artificial intelligence research, as it relates from which to biological senses has been focused on vision. The recent explosion of machine learning and in particular, dee p learning, can be partially attributed to the release of high quality data sets for algorithm s from which to model the world on. Thus, most of these datasets are comprised of images. We believe that focusing on sensorimotor systems and tactile feedback will create algorithms that better mimic human intelligence. Here we present SenseNet: a collection of tactile simulators and a large scale dataset of 3D obj…
The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation
VisLab, Institute for Systems；Soft Robotics Lab National University of Singapore, Singapore；BRAIR Lab, BioRobotics Institute Scuola Superiore Sant'Anna, Pisa, Italia
Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（电子工程、数学和计算机科学学院，代尔夫特理工大学）；Department of Electrical Engineering, Wright State University（电气工程系，怀特州立大学）
University of Southern California（南加州大学）；University of Washington（华盛顿大学）
In order to robustly execute a task under environmental uncertainty, a robot needs to be able to reactively adapt to changes arising in its environment. The environment changes are usually reflected in deviation from expected sensory traces. These deviations in sensory traces can be used to drive the motion adaptation, and for this purpose, a feedback model is required. The feedback model maps the deviations in sensory traces to the motion plan adaptation. In this paper, we develop a general data-driven framework for learning a feedback model from demonstrations. We utilize a variant of a radi…
Single-grasp deformable object discrimination: the effect of gripper morphology, sensing modalities, and action parameters
Czech Technical University in Prague（布拉格捷克理工大学）；Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克理工大学电气工程学院）；Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人学与控制论研究所）
Human-Inspired Soft Anthropomorphic Hand System for Neuromorphic Object and Pose Recognition Using Multimodal Signals
Institute for Cognitive Systems, Technical University of Munich（认知系统研究所，慕尼黑技术大学）；Department of Biomedical Engineering, Johns Hopkins University（生物医学工程系，约翰霍普金斯大学）
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology（南方科技大学）；University of Hong Kong（香港大学）；Tencent Robotics X（腾讯 Robotics X）；University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Southern California（南加利福尼亚大学）；Intrinsic Innovation LLC.（Intrinsic创新有限责任公司）；ETH Zurich（苏黎世联邦理工学院）；Google Brain（谷歌大脑）；Dexterity Inc.（Dexterity公司）；Meta Artificial Intelligence（Meta人工智能）
Robots need to be able to adapt to unexpected changes in the environment such that they can autonomously succeed in their tasks. However, hand-designing feedback models for adaptation is tedious, if at all possible, making data-driven methods a promising alternative. In this paper we introduce a full framework for learning feedback models for reactive motion planning. Our pipeline starts by segmenting demonstrations of a complete task into motion primitives via a semi-automated segmentation algorithm. Then, given additional demonstrations of successful adaptation behaviors, we learn initial fe…
Unsupervised Discovery of Behavioral Primitives from Sensorimotor Dynamic Functional Connectivity
Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑机器人与机器智能研究所，慕尼黑技术大学）；Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克技术大学电子工程学院）
Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Touchlab Limited（Touchlab有限公司）；Department of Engineering Mathematics, University of Bristol（布里斯托尔大学工程数学系）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Stanford University（斯坦福大学）；X, The Moonshot Factory（X，登月工厂）
A key challenge towards the goal of multi-part assembly tasks is finding robust sensorimotor control methods in the presence of uncertainty. In contrast to previous works that rely on a priori knowledge on whether two parts match, we aim to learn this through physical interaction. We propose a hierarchical approach that enables a robot to autonomously assemble parts while being uncertain about part types and positions. In particular, our probabilistic approach learns a set of differentiable filters that leverage the tactile sensorimotor trace from failed assembly attempts to update its belief …
The sense of touch is fundamental in several manipulation tasks, but rarely used in robot manipulation. In this work we tackle the problem of learning rich touch features from cross-modal self-supervision. We evaluate them identifying objects and their properties in a few-shot classification setting. Two new datasets are introduced using a simulated anthropomorphic robotic hand equipped with tactile sensors on both synthetic and daily life objects. Several self-supervised learning methods are benchmarked on these datasets, by evaluating few-shot classification on unseen objects and poses. Our …
Underactuated Robotic Hand with Grasp State Estimation Using Tendon-Based Proprioception
BioRobotics Laboratory (BRL), Soft Robotics Research Center (SRRC), Institute of Advanced Machines and Design (IAMD), Department of Mechanical Engineering, Institute of Engineering, Seoul National University (SNU)（生物机器人实验室（BRL）、软机器人研究中心（SRRC）、先进机械与设计研究所（IAMD）、机械工程系、工程学院、首尔国立大学（SNU））
Scuola Superiore Sant’Anna（圣安娜高等研究学院）；Università Campus Bio-Medico di Roma（罗马生物医学大学）；Sapienza University of Rome（罗马大学）；Istituto Italiano di Tecnologia（意大利理工学院）
Collaborative robots are expected to physically interact with humans in daily living and workplace, including industrial and healthcare settings. A related key enabling technology is tactile sensing, which currently requires addressing the outstanding scientific challenge to simultaneously detect contact location and intensity by means of soft conformable artificial skins adapting over large areas to the complex curved geometries of robot embodiments. In this work, the development of a large-area sensitive soft skin with a curved geometry is presented, allowing for robot total-body coverage th…
