In this paper we propose a novel method for in-hand object recognition. The method is composed of a grasp stabilization controller and two exploratory behaviours to capture the shape and the softness of an object. Grasp stabilization plays an important role in recognizing objects. First, it prevents the object from slipping and facilitates the exploration of the object. Second, reaching a stable and repeatable position adds robustness to the learning algorithm and increases invariance with respect to the way in which the robot grasps the object. The stable poses are estimated using a Gaussian mixture model (GMM). We present experimental results showing that using our method the classifier can successfully distinguish 30 objects.We also compare our method with a benchmark experiment, in which the grasp stabilization is disabled. We show, with statistical significance, that our method outperforms the benchmark method.
University of Bristol（布里斯托大学）；Ultraleap Ltd.（Ultraleap有限公司）
Mid-air haptics create a new mode of feedback to allow people to feel tactile sensations in the air. Ultrasonic arrays focus acoustic radiation pressure in space, to induce tactile sensation from the resulting skin deflection. In this work, we present a low-cost tactile robot to test mid-air haptics. By combining a desktop robot arm with a 3D-printed biomimetic tactile sensor, we developed a system that can sense, map, and visualize mid-air haptic sensations created by an ultrasonic transducer array. We evaluate our tactile robot by testing it on a variety of mid-air haptic stimuli, including …
University of California at Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by …
Koc University（科奇大学）；University of Lille（里尔大学）；École de Technologie Supérieure（高等技术学院）；Pohang University of Science and Technology (POSTECH)（浦项科技大学（POSTECH））
We review the current technology underlying surface haptics that converts passive touch surfaces to active ones (machine haptics), our perception of tactile stimuli displayed through active touch surfaces (human haptics), their potential applications (human-machine interaction), and finally the challenges ahead of us in making them available through commercial systems. This review primarily covers the tactile interactions of human fingers or hands with surface-haptics displays by focusing on the three most popular actuation methods: vibrotactile, electrostatic, and ultrasonic.
Wrist-worn haptic interfaces can deliver a wide range of tactile cues for communication of information and interaction with virtual objects. Unlike fingertips, the wrist and forearm provide a considerably large area of skin that allows the placement of multiple haptic actuators as a display for enriching tactile information transfer with minimal encumbrance. Existing multi-degree-of-freedom (DoF) wrist-worn devices employ traditional rigid robotic mechanisms and electric motors that limit their versatility, miniaturization, distribution, and assembly. Alternative solutions based on soft elasto…
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；University of Leeds（利兹大学）；Xi’an Jiaotong University（西安交通大学）
To assist robots in teleoperation tasks, haptic rendering which allows human operators access a virtual touch feeling has been developed in recent years. Most previous haptic rendering methods strongly rely on data collected by tactile sensors. However, tactile data is not widely available for robots due to their limited reachable space and the restrictions of tactile sensors. To eliminate the need for tactile data, in this paper we propose a novel method named as Vis2Hap to generate haptic rendering from visual inputs that can be obtained from a distance without physical interaction. We take …
Stanford University（斯坦福大学）；University of Southern California（南加州大学）
A common and effective form of social touch is stroking on the forearm. We seek to replicate this stroking sensation using haptic illusions. This work compares two methods that provide sequential discrete stimulation: sequential normal indentation and sequential lateral skin-slip using discrete actuators. Our goals are to understand which form of stimulation more effectively creates a continuous stroking sensation, and how many discrete contact points are needed. We performed a study with 20 participants in which they rated sensations from the haptic devices on continuity and pleasantness. We …
Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation
Queen Mary University of London（伦敦玛丽女王大学）；School of Engineering and Materials Science（工程与材料科学学院）；Humanoid AI（人形人工智能公司）；King’s College London（伦敦国王学院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is in stark contrast to robotics. In this field, the relative lack of good real-world interaction models - along with very restricted sensors and a scarcity of suitable training data to leverage machine l…
Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation
Technische Universität Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Centre for Artificial Intelligence (Hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science (CogSci)（认知科学中心）；Nagoya University（名古屋大学）
Nagoya University（名古屋大学）；Technical University of Darmstadt（达姆施塔特工业大学）
Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and…
For humans, our "senses of touch" have always been necessary for our ability to precisely and efficiently manipulate objects of all shapes in any environment, but until recently, not many works have been done to fully understand haptic feedback. This work proposed a novel method for getting a better shape descriptor than existing methods for classifying an object from multiple tactile data collected from a tactile glove. It focuses on improving previous works on object classification using tactile data. The major problem for object classification from multiple tactile data is to find a good wa…
A Haptic Robot Finger Designed for Guqin Instrument Playing
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong - Shenzhen（香港中文大学（深圳））；School of Physics and Optoelectronic Engineering, Shenzhen University（深圳大学物理与光电工程学院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity
University of Southern California（南加州大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of Cincinnati（辛辛那提大学）
Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（Skoltech））
TouchVR is a novel wearable haptic interface which can deliver multimodal tactile stimuli on the palm by DeltaTouch haptic display and vibrotactile feedback on the fingertips by vibration motors for the Virtual Reality (VR) user. DeltaTouch display is capable of generating 3D force vector at the contact point and presenting multimodal tactile sensation of weight, slippage, encounter, softness, and texture. The VR system consists of HTC Vive Pro base stations and head-mounted display (HMD), and Leap Motion controller for tracking the user's hands motion in VR. The MatrixTouch, BallFeel, and Rob…
LIRMM, University of Montpellier, CNRS（蒙彼利埃大学法国国家科学研究中心LIRMM实验室）；Stanford University（斯坦福大学）
The current generation of surgeons requires extensive training in teleoperation to develop specific dexterous skills, which are independent of medical knowledge. Training curricula progress from manipulation tasks to simulated surgical tasks but are limited in time. To tackle this, we propose to integrate surgical robotic training together with Haptic Feedback (HF) to improve skill acquisition. This paper present the initial but promising results of our haptic device designed to support in the training of surgical gestures. Our ongoing work is related to integrate the HF in the RAVEN II platfo…
Technical University of Munich（慕尼黑工业大学）；ABB Corporate Research（ABB集团研究院）；University of Campania Luigi Vanvitelli（坎帕尼亚大学路易吉·万维泰利）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
In this work, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception and, later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable …
Teleoperation of robotic systems for precise and delicate object grasping requires high-fidelity haptic feedback to obtain comprehensive real-time information about the grasp. In such cases, the most common approach is to use kinesthetic feedback. However, a single contact point information is insufficient to detect the dynamically changing shape of soft objects. This paper proposes a novel telemanipulation system that provides kinesthetic and cutaneous stimuli to the user's hand to achieve accurate liquid dispensing by dexterously manipulating the deformable object (i.e., pipette). The experi…
A Biomimetic Fingerprint for Robotic Tactile Sensing
Faculty of Mechatronic \& Micro-Mechatronic Systems Hochschule Karlsruhe (HKA) Karlsruhe, Germany；Deutsches Forschungszentrum für Künstliche Intelligenz GmbH；L3S Research Center, Leibniz Universität Hannover Hannover, Germany
Multimodal object recognition is still an emerging field. Thus, publicly available datasets are still rare and of small size. This dataset was developed to help fill this void and presents multimodal data for 63 objects with some visual and haptic ambiguity. The dataset contains visual, kinesthetic and tactile (audio/vibrations) data. To completely solve sensory ambiguity, sensory integration/fusion would be required. This report describes the creation and structure of the dataset. The first section explains the underlying approach used to capture the visual and haptic properties of the object…
Skolkovo Institute of Science and Technology（斯科尔科沃科技学院）
We propose a novel concept of augmented reality (AR) human-drone interaction driven by RL-based swarm behavior to achieve intuitive and immersive control of a swarm formation of unmanned aerial vehicles. The DroneARchery system developed by us allows the user to quickly deploy a swarm of drones, generating flight paths simulating archery. The haptic interface LinkGlide delivers a tactile stimulus of the bowstring tension to the forearm to increase the precision of aiming. The swarm of released drones dynamically avoids collisions between each other, the drone following the user, and external o…
We present a novel robot end-effector for gripping and haptic exploration. Tactile sensing through suction flow monitoring is applied to a new suction cup design that contains multiple chambers for air flow. Each chamber connects with its own remote pressure transducer, which enables both absolute and differential pressure measures between chambers. By changing the overall vacuum applied to this smart suction cup, it can perform different functions such as gentle haptic exploration (low pressure) and monitoring breaks in the seal during strong astrictive gripping (high pressure). Haptic explor…
Towards Tangible Immersion for Cobot Programming-by-Demonstration: Visual, Tactile and Haptic Interfaces for Mixed-Reality Cobot Automation in Semiconductor Manufacturing
Intel Labs, Intelligent System Research, Human-Robot Collaboration（英特尔实验室、智能系统研究、人机协作）
Inspired by the human ability to perform complex manipulation in the complete absence of vision (like retrieving an object from a pocket), the robotic manipulation field is motivated to develop new methods for tactile-based object interaction. However, tactile sensing presents the challenge of being an active sensing modality: a touch sensor provides sparse, local data, and must be used in conjunction with effective exploration strategies in order to collect information. In this work, we focus on the process of guiding tactile exploration, and its interplay with task-related decision making. W…
Creating haptic interfaces capable of rendering the rich sensation needed for dexterous manipulation is crucial for the advancement of human-in-the-loop telerobotic systems (HiLTS). One limiting factor has been the absence of detailed knowledge of the effect of operator limb dynamics and haptic exploration dynamics on haptic perception. We propose to begin investigations of these effects with single-joint haptic exploration and feedback of physical and virtual environments. Here, we present our experimental apparatus, a 1-DoF rotational kinesthetic haptic device and electromyography (EMG) syst…
A wearable haptic device for edge and surface simulation
Institute of Mechanical Intelligence, School of Advanced Studies Sant'Anna (SSSA)（圣安娜高等研究学院机械智能研究所）；CAS Key Laboratory of Mechanical Behavior and Design of Materials, Institute of Humanoid Robots, School of Engineering Sciences, University of Science and Technology of China（中国科学院材料力学行为和设计重点实验室，中国科学技术大学工程科学学院仿生机器人研究所）
Robotic teleoperation will allow us to perform complex manipulation tasks in dangerous or remote environments, such as needed for planetary exploration or nuclear decommissioning. This work proposes a novel telemanipulation architecture using a passive Fractal Impedance Controller (FIC), which does not depend upon an active viscous component for stability guarantees. Compared to a traditional impedance controller in ideal conditions (no delays and maximum communication bandwidth), our proposed method yields higher transparency in interaction and demonstrates superior dexterity and capability i…
Effect of Haptic Feedback on Avoidance Behavior and Visual Exploration in Dynamic VR Pedestrian Environment
Department of Mechano-informatics, The University of Tokyo（机械信息学系，东京大学）；Department of Electrical Engineering, Stanford University（电气工程系，斯坦福大学）；Department of Mechanical Engineering, Stanford University（机械工程系，斯坦福大学）
Robotic exploration under uncertain environments is challenging when optical information is not available. In this paper, we propose an autonomous solution of exploring an unknown task space based on tactile sensing alone. We first designed a whisker sensor based on MEMS barometer devices. This sensor can acquire contact information by interacting with the environment non-intrusively. This sensor is accompanied by a planning technique to generate exploration trajectories by using mere tactile perception. This technique relies on a hybrid policy for tactile exploration, which includes a proacti…
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑工业大学机器人与系统智能教席，慕尼黑机器人与机器智能研究所）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）
Tactile interaction plays a crucial role in interactions between people. Touch can, for example, help people calm down and lower physiological stress responses. Consequently, it is believed that tactile and haptic interaction matter also in human-robot interaction. We study if the intensity of the tactile interaction has an impact on people, and do so by studying whether different intensities of tactile interaction modulate physiological measures and task performance. We use a paradigm in which a small humanoid robot is used to encourage risk-taking behaviour, relying on peer encouragement to …
Skoltech（斯科尔科沃科技学院）；Imperial College London（伦敦帝国学院）
To achieve high fidelity haptic rendering of soft objects in a high mobility virtual environment, we propose a novel haptic display DandelionTouch. The tactile actuators are delivered to the fingertips of the user by a swarm of drones. Users of DandelionTouch are capable of experiencing tactile feedback in a large space that is not limited by the device's working area. Importantly, they will not experience muscle fatigue during long interactions with virtual objects. Hand tracking and swarm control algorithm allow guiding the swarm with hand motions and avoid collisions inside the formation. S…
King’s College London（伦敦国王学院）；University of Liverpool（利物浦大学）；Queen Mary University of London（伦敦玛丽女王大学）
For humans, both the proprioception and touch sensing are highly utilized when performing haptic perception. However, most approaches in robotics use only either proprioceptive data or touch data in haptic object recognition. In this paper, we present a novel method named Iterative Closest Labeled Point (iCLAP) to link the kinesthetic cues and tactile patterns fundamentally and also introduce its extensions to recognize object shapes. In the training phase, the iCLAP first clusters the features of tactile readings into a codebook and assigns these features with distinct label numbers. A 4D poi…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Developing artificial tactile sensing capabilities that rival human touch is a long-term goal in robotics and prosthetics. Gradually more elaborate biomimetic tactile sensors are being developed and applied to grasping and manipulation tasks to help achieve this goal. Here we present the neuroTac, a novel neuromorphic optical tactile sensor. The neuroTac combines the biomimetic hardware design from the TacTip sensor which mimicks the layered papillae structure of human glabrous skin, with an event-based camera (DAVIS240, iniVation) and algorithms which transduce contact information in the form…
TACTFUL: Tactile-Driven Exploration For Object Localization and Identification in Confined Environments
Amazon Fulfillment Technologies & Robotics（亚马逊履约技术与机器人）；Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Tactile recognition of 3D objects remains a challenging task. Compared to 2D shapes, the complex geometry of 3D surfaces requires richer tactile signals, more dexterous actions, and more advanced encoding techniques. In this work, we propose TANDEM3D, a method that applies a co-training framework for exploration and decision making to 3D object recognition with tactile signals. Starting with our previous work, which introduced a co-training paradigm for 2D recognition problems, we introduce a number of advances that enable us to scale up to 3D. TANDEM3D is based on a novel encoder that builds …
Finding safe 3D robot grasps through efficient haptic exploration with unscented Bayesian optimization and collision penalty
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）；Centro Universitario de la Defensa（国防大学中心）；SigOpt Inc.（SigOpt公司）；Queen Mary University of London（伦敦大学玛丽王后学院）
Shanghai Jiao Tong University（上海交通大学）；Massachusetts Institute of Technology（麻省理工学院）
Several robot manipulation tasks are extremely sensitive to variations of the physical properties of the manipulated objects. One such task is manipulating objects by using gravity or arm accelerations, increasing the importance of mass, center of mass, and friction information. We present SwingBot, a robot that is able to learn the physical features of a held object through tactile exploration. Two exploration actions (tilting and shaking) provide the tactile information used to create a physical feature embedding space. With this embedding, SwingBot is able to predict the swing angle achieve…
With the emergence of pen-enabled tablets and mobile devices, stylus-based interaction has been receiving increasing attention. Unfortunately, styluses available in the market today are all passive instruments that are primarily used for writing and pointing. In this paper, we describe a novel stylus capable of displaying certain vibro-tactile and inertial haptic effects to the user. Our stylus is equipped with two vibration actuators at the ends, which are used to create a tactile sensation of up and down movement along the stylus. The stylus is also embedded with a DC motor, which is used to…
